How to test the throughput/performance of your network using Netperf - Part 1

In this multi-part series, I will explain how to use GNU tools and Linux to have a free network throughput test setup. In Part 1 of the series we will use only one NIC to create single unicast TCP/UDP stream to saturate the linkspeed.
First of all what I used to have this test setup (you can adapt it to your needs)

HW :
- 2 HP Blade on same enclosure chassis with 10Gig HP Flex NICs. (servers)
- HP VC Flex 10 (the network device)

Software :- Centos 5.4 x64
- I only installed core GNU packages + development libraries. No unnecessary service/software loaded. Even I took XWindow(Gnome/KDE) out of the package and run the system in rulevel 3.
- Disable Firewall/SELinux
- netperf rpm (ftp://ftp.netperf.org/netperf/netperf-2.4.5.tar.gz)

I made the first run of the test using single Vnet (counterpart of VLANs in HP Virtual Connect). You can also use the same tools to create a setup that utilize Shared Uplink Sets (trunk links on VC). In order to set this up create a Vnet for your load VLAN(Vnet_LOAD). Create the profiles for the blades and assign one FlexNIC with 100Mb to management VNet and the other one to VNet_LOAD (10Gb)

Netperf is based on client server model. After installing the software on both blades you execute different processes on different nodes. netserver as the name states is the server part of the test suite. You can also use the

[root@SERVER ~]# netserver
Starting netserver at port 12865
Starting netserver at hostname 0.0.0.0 port 12865 and family AF_UNSPEC

while netperf is the tool that executes the test and gives output.

[root@CLIENT ~]# netperf -H SERVER -l 15
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to SERVER (*******) port 0 AF_INET
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec
 87380  16384  16384    15.00    9387.92

The default test is TCP_STREAM you can also define other tests like UDP Request Response to fully saturate Full Duplex Link :

[root@CLIENT ~]# netperf -t UDP_RR -H SERVER -l 15

UDP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to SERVER (*********) port 0 AF_INET

Local /Remote

Socket Size   Request  Resp.   Elapsed  Trans.

Send   Recv   Size     Size    Time     Rate

bytes  Bytes  bytes    bytes   secs.    per sec

129024 129024 1        1       15.00    21045.72

Also you can fetch the CPU utilization info while doing the test using -c (local) and -C(remote) parameters

[root@CLIENT ~]# netperf -t UDP_RR -H SERVER -l 15 -c -C

UDP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to SERVER (*******) port 0 AF_INET

Local /Remote

Socket Size   Request Resp.  Elapsed Trans.   CPU    CPU    S.dem   S.dem

Send   Recv   Size    Size   Time    Rate     local  remote local   remote

bytes  bytes  bytes   bytes  secs.   per sec  % S    % S    us/Tr   us/Tr

129024 129024 1       1      15.00   20964.03   2.06   2.03   15.759  15.481

For detailed documentation and command line options you can check :

http://www.netperf.org/netperf/training/Netperf.html#0.2.2Z141Z1.SUJSTF.8R2DBD.J

On the next parts of the series I will focus on different types of throughput/load tests like multi flow & multi IP throughput testing using netperf & some Linux tweaking and IP multicast testing using MGEN.

Windows 2008 Failover Clustering and HP EVA Storage

If you are using HP EVA and getting the below error during the WCS validation phase in Windows 2008 :


Failed to read drive layout of Cluster disk 0 from node ****, status 21
Cluster Disk 0 does not support Persistent Reservation
Test failed. Please look at the test log for more information

Make sure Windows LH Host Mode is configured for your LUN. Below article is a guidance for Win2k8 and HP EVA config :

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c01764063/c01764063.pdf

Page 17 :

Configuring EVA host connections

Make sure the Windows LH Host mode is enabled for each node on the Enterprise Virtual Array
(EVA) through command view. Depending on what command view you have installed, if in the drop-
down there is no "Microsoft Windows LH" then choose custom and type in the custom type field the
following HEX number 00000004198009A8 (Figure 1). If the wrong connection type is configured,
you cannot share the disks between the cluster nodes and the cluster check will fail with a non-SCSI-3
compliant message.

Hyper-V Tagging & Teaming with HP NCU

Yes, finally the first real post of my blog :) This article summarize the NIC Teaming & Tagging support on Hyper-V. Scenarios has been tested on HP Blade systems with HP NCU utility. Windows 2008 Datacenter Core Edition has been used for the parent partition.

In order to check VLAN tagging with teaming 2 scenarios have been tested :

1. NIC Teaming and Tagging with HP NCU (Works OK)
2. NIC Teaming with NCU and Tagging at HyperV Level (NOK)

As stated above only the first scenario works. This scenario creates lots of adapter overhead on the OS level. For instance lets assume that you have 2 physical interfaces which are teamed and you create 4 VLANs on top. After making the necessary configurations you have :

2 Interface for the actual pNICs.
1 Interface for Teamed NIC
4 Interface For the VLANs
4 Interface For the Virtual Switches

This creates some management overhead for the interfaces but this is the only supported scenario by Hyper-V currently.

Also with this setup the parent partition always have L2 access to all VLANs because the virtual network adapter at parent partition level is connected to the Virtual Switch by default. In order to create a External network without parent partition attached you can use the Poweshell scripts mentioned on the below pages.

http://blogs.msdn.com/virtual_pc_guy/archive/2009/02/19/script-creating-an-external-only-virtual-network-with-hyper-v.aspx

http://blogs.msdn.com/robertvi/archive/2008/08/27/howto-create-a-virtual-swich-for-external-without-creating-a-virtual-nic-on-the-root.aspx

Also after creating a virtual network you can disable this virtual interface. On server Core :

netsh interface show interface
netsh interface set interface name=”Name of Interface” disabled

HYPER-V NETWORKING

In order to understand the networking logic in Hyper-V it’s strongly recommended to check the below document :

http://www.microsoft.com/DOWNLOADS/details.aspx?FamilyID=3fac6d40-d6b5-4658-bc54-62b925ed7eea&displaylang=en

As stated by the above diagram when you bind a virtual network to a physical interface, a Virtual Network Adapter has been created on OS level. This virtual adapter has all the network binding like TCP/IP. After this operation the existing Network Adapter for the pNIC has only a binding for the HyperV Virtual Switch protocol.

In order to make OS level application work over the new created virtual adapter make sure appropriate tagging has been created also on host level.

IMPORTAT NOTE: Make sure you don’t create any Virtual Switch on the pNIC that is used for communication between SCVMM and Hyper-V host. Leave at least one NIC or Teamed Interface for this communication.

NIC Teaming and Tagging with HP NCU

HyperV has NO teaming capability at Hypervisor level like VmWare ESX/ESXi as mentioned in KB968703 (http://support.microsoft.com/kb/968703) :

Since Network Adapter Teaming is only provided by Hardware Vendors, Microsoft does not provide any support for this technology thru Microsoft Product Support Services. As a result, Microsoft may ask that you temporarily disable or remove Network Adapter Teaming software when troubleshooting issues where the teaming software is suspect.

If the problem is resolved by the removal of Network Adapter Teaming software, then further assistance must be obtained thru the Hardware Vendor.

This support has to be maintained at Hardware Level. For HP we used HP NCU for teaming purpose.

IMPORTANT NOTE : HP NCU have to be installed AFTER enabling HyperV role.

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c01663264/c01663264.pdf

http://support.microsoft.com/kb/950792

OS Level Settings For Teaming+Tagging

In order to check HyperV with Teaming + tagging :

1. Windows 2008 Datacenter Core installed
2. HyperV role activated with necessary KB Updates.

http://support.microsoft.com/?kbid=950050
http://support.microsoft.com/?kbid=956589
http://support.microsoft.com/?kbid=956774

3. Using HpTeam Utility NFT based teaming has been configured.
4. NCU Installed together with Broadcom and Intel Drivers.

5. VLAN1,1101,1102,1103,1104 has been setup on the teamed interface.

Hyper-V Level Settings For Teaming+Tagging

1. Create a Virtual Network on the Hyperv host for each VLAN and bind this network to each logical tagged interface that has been created by NCU.

NOTE : Access host through VLAN option enables the parent partition talk with that VLAN.

2. On the HOST created to VMs for testing. Each VM has been connected to different virtual switch as below

3. After setting tagging both on Host and VM level ping between different VLANs is possible. (The switch has been configured for interVLAN routing)

NIC Teaming with NCU and Tagging at HyperV Level

1. Windows 2008 Datacenter Core installed
2. HyperV role activated with necessary KB Updates

http://support.microsoft.com/?kbid=950050
http://support.microsoft.com/?kbid=956589
http://support.microsoft.com/?kbid=956774

3. NCU Installed together with Broadcom and Intel Drivers.
4. Only Teaming has been configured with NCU.
5. A virtual switch has been created at HyperV level and necessary tagging made for the Host Virtual Adapter.
6. Virtual guest machines has also configured with tagged vNICs.
7. Network connectivity between the VMs does NOT work.