How Can I Test the Network Performance of Linux ECSs?

Background

• Tested ECS: an ECS that is tested for network performance. Such an ECS functions as the client (TX end) or server (RX end) in netperf tests.
• Auxiliary ECS: an ECS that is used to exchange test data with the tested ECS. The auxiliary ECS functions as the client (TX end) or server (RX end) in netperf tests.
• Table 1 and Table 2 list the common netperf and iperf3 parameters.

  Table 1 Common netperf parameters

  Parameter	Description
  -p	Port number
  -H	IP address of the RX end
  -t	Protocol used in packet transmitting, the value of which is TCP_STREAM in bandwidth tests
  -l	Test duration
  -m	Data packet size, which is suggested to be 1440 in bandwidth tests

  Table 2 Common iperf3 parameters

  Parameter	Description
  -p	Port number
  -c	IP address of the RX end
  -u	UDP packets
  -b	TX bandwidth
  -t	Test duration
  -l	Data packet size, which is suggested to be 16 in PPS tests
  -A	ID of the vCPU used by iperf3 In this section, the maximum number of 16 vCPUs is used as an example for each ECS. If an ECS has 8 vCPUs, the -A value ranges from 0 to 7.

Test Preparations

1. Prepare ECSs.
   Ensure that both type and specifications of the tested ECS and auxiliary ECSs are the same. In addition, ensure that these ECSs are deployed in the same ECS group with anti-affinity enabled.

   Table 3 Preparations

   Category	Quantity	Image	Specifications	IP Address
   Tested ECS	1	CentOS 7.4 64bit (recommended)	At least eight vCPUs	192.168.2.10
   Auxiliary ECS	8	CentOS 7.4 64bit (recommended)	At least 8 vCPUs	192.168.2.11-192.168.2.18

2. Install the netperf, iperf3, and sar test tools on both the tested ECS and auxiliary ECSs.

   Table 4 lists the procedures for installing these tools.

   Table 4 Installing test tools

   Tool	Procedure
   netperf	Run the following command to install gcc: yum -y install unzip gcc gcc-c++ Run the following command to download the netperf installation package: wget https://github.com/HewlettPackard/netperf/archive/refs/tags/netperf-2.7.0.zip Run the following commands to decompress the installation package and install netperf: unzip netperf-2.7.0.zip cd netperf-netperf-2.7.0/ ./configure && make && make install
   iperf3	Run the following command to download the iperf3 installation package: wget --no-check-certificate https://codeload.github.com/esnet/iperf/zip/master -O iperf3.zip Run the following commands to decompress the installation package and install iperf3: unzip iperf3.zip cd iperf-master/ ./configure && make && make install
   sar	Run the following command to install sar: yum -y install sysstat

3. Enable NIC multi-queue.

   Perform the following operations on both tested ECS and auxiliary ECSs.

   1. Run the following command to check the number of queues supported by the ECSs:

      ethtool -l eth0 | grep -i Pre -A 5 | grep Combined

   2. Run the following command to enable NIC multi-queue:

      ethtool -L eth0 combined X

      In the preceding command, X is the number of queues obtained in 3.a.

TCP Bandwidth Test (Using netperf)

Perform the test on multiple flows. This section considers 16 flows that are evenly distributed to eight ECSs, as an example.

The TCP bandwidth test uses the multi-flow model.

• When testing the TCP transmission (TX) bandwidth, use the one-to-many model to ensure that the capability of the receiver is sufficient.
• When testing the TCP receiver (RX) bandwidth, use the many-to-one model to ensure that the capability of the sender is sufficient.

1. Test the TCP TX bandwidth.
   1. Run the following commands on all auxiliary ECSs to start the netserver process:

      netserver -p 12001

      netserver -p 12002

      In the preceding commands, -p specifies the listening port.

   2. Start the netperf process on the tested ECS and specify a netserver port for each auxiliary ECS. For details about common netperf parameters, see Table 1.

      ##The IP address is for the first auxiliary ECS.

      netperf -H 192.168.2.11 -p 12001 -t TCP_STREAM -l 300 -- -m 1440 &

      netperf -H 192.168.2.11 -p 12002 -t TCP_STREAM -l 300 -- -m 1440 &

      ##The IP address is for the second auxiliary ECS.

      netperf -H 192.168.2.12 -p 12001 -t TCP_STREAM -l 300 -- -m 1440 &

      netperf -H 192.168.2.12 -p 12002 -t TCP_STREAM -l 300 -- -m 1440 &

      ##The IP address is for the third auxiliary ECS.

      netperf -H 192.168.2.13 -p 12001 -t TCP_STREAM -l 300 -- -m 1440 &

      netperf -H 192.168.2.13 -p 12002 -t TCP_STREAM -l 300 -- -m 1440 &

      ##The IP address is for the fourth auxiliary ECS.

      netperf -H 192.168.2.14 -p 12001 -t TCP_STREAM -l 300 -- -m 1440 &

      netperf -H 192.168.2.14 -p 12002 -t TCP_STREAM -l 300 -- -m 1440 &

      ##The IP address is for the fifth auxiliary ECS.

      netperf -H 192.168.2.15 -p 12001 -t TCP_STREAM -l 300 -- -m 1440 &

      netperf -H 192.168.2.15 -p 12002 -t TCP_STREAM -l 300 -- -m 1440 &

      ##The IP address is for the sixth auxiliary ECS.

      netperf -H 192.168.2.16 -p 12001 -t TCP_STREAM -l 300 -- -m 1440 &

      netperf -H 192.168.2.16 -p 12002 -t TCP_STREAM -l 300 -- -m 1440 &

      ##The IP address is for the seventh auxiliary ECS.

      netperf -H 192.168.2.17 -p 12001 -t TCP_STREAM -l 300 -- -m 1440 &

      netperf -H 192.168.2.17 -p 12002 -t TCP_STREAM -l 300 -- -m 1440 &

      ##The IP address is for the eighth auxiliary ECS.

      netperf -H 192.168.2.18 -p 12001 -t TCP_STREAM -l 300 -- -m 1440 &

      netperf -H 192.168.2.18 -p 12002 -t TCP_STREAM -l 300 -- -m 1440 &

2. Test the TCP RX bandwidth.
   1. Start the netserver process on the tested ECS.

      ##The port number is for the first auxiliary ECS.

      netserver -p 12001

      netserver -p 12002

      ##The port number is for the second auxiliary ECS.

      netserver -p 12003

      netserver -p 12004

      ##The port number is for the third auxiliary ECS.

      netserver -p 12005

      netserver -p 12006

      ##The port number is for the fourth auxiliary ECS.

      netserver -p 12007

      netserver -p 12008

      ##The port number is for the fifth auxiliary ECS.

      netserver -p 12009

      netserver -p 12010

      ##The port number is for the sixth auxiliary ECS.

      netserver -p 12011

      netserver -p 12012

      ##The port number is for the seventh auxiliary ECS.

      netserver -p 12013

      netserver -p 12014

      ##The port number is for the eighth auxiliary ECS.

      netserver -p 12015

      netserver -p 12016

   2. Start the netperf process on all auxiliary ECSs.

      Log in to auxiliary ECS 1.

      netperf -H 192.168.2.10 -p 12001 -t TCP_STREAM -l 300 -- -m 1440 &

      netperf -H 192.168.2.10 -p 12002 -t TCP_STREAM -l 300 -- -m 1440 &

      Log in to auxiliary ECS 2.

      netperf -H 192.168.2.10 -p 12003 -t TCP_STREAM -l 300 -- -m 1440 &

      netperf -H 192.168.2.10 -p 12004 -t TCP_STREAM -l 300 -- -m 1440 &

      Log in to auxiliary ECS 3.

      netperf -H 192.168.2.10 -p 12005 -t TCP_STREAM -l 300 -- -m 1440 &

      netperf -H 192.168.2.10 -p 12006 -t TCP_STREAM -l 300 -- -m 1440 &

      Log in to auxiliary ECS 4.

      netperf -H 192.168.2.10 -p 12007 -t TCP_STREAM -l 300 -- -m 1440 &

      netperf -H 192.168.2.10 -p 12008 -t TCP_STREAM -l 300 -- -m 1440 &

      Log in to auxiliary ECS 5.

      netperf -H 192.168.2.10 -p 12009 -t TCP_STREAM -l 300 -- -m 1440 &

      netperf -H 192.168.2.10 -p 12010 -t TCP_STREAM -l 300 -- -m 1440 &

      Log in to auxiliary ECS 6.

      netperf -H 192.168.2.10 -p 12011 -t TCP_STREAM -l 300 -- -m 1440 &

      netperf -H 192.168.2.10 -p 12012 -t TCP_STREAM -l 300 -- -m 1440 &

      Log in to auxiliary ECS 7.

      netperf -H 192.168.2.10 -p 12013 -t TCP_STREAM -l 300 -- -m 1440 &

      netperf -H 192.168.2.10 -p 12014 -t TCP_STREAM -l 300 -- -m 1440 &

      Log in to auxiliary ECS 8.

      netperf -H 192.168.2.10 -p 12015 -t TCP_STREAM -l 300 -- -m 1440 &

      netperf -H 192.168.2.10 -p 12016 -t TCP_STREAM -l 300 -- -m 1440 &

3. Analyze the test result.

   After the test is complete, the output of the netperf process on one TX end is shown in Figure 1. The final result is the sum of the test results of the netperf processes on all TX ends.

   Figure 1 Output of the netperf process on one TX end
   
   

   There are a large number of netperf processes. To facilitate statistics collection, it is a good practice to run the following command to view test data on the tested ECS using sar:

   sar -n DEV 1 60

UDP PPS Test (Using iperf3)

1. Test the UDP TX PPS.
   1. Log in to an auxiliary ECS.
   2. Run the following commands on all auxiliary ECSs to start the server process:

      iperf3 -s -p 12001 &

      iperf3 -s -p 12002 &

      In the preceding commands, -p specifies the listening port.

   3. Start the client process on the tested ECS. For details about common iperf3 parameters, see Table 2.

      ##Auxiliary ECS 1

      iperf3 -c 192.168.2.11 -p 12001 -u -b 100M -t 300 -l 16 -A 0 &

      iperf3 -c 192.168.2.11 -p 12002 -u -b 100M -t 300 -l 16 -A 1 &

      ##Auxiliary ECS 2

      iperf3 -c 192.168.2.12 -p 12001 -u -b 100M -t 300 -l 16 -A 2 &

      iperf3 -c 192.168.2.12 -p 12002 -u -b 100M -t 300 -l 16 -A 3 &

      ##Auxiliary ECS 3

      iperf3 -c 192.168.2.13 -p 12001 -u -b 100M -t 300 -l 16 -A 4 &

      iperf3 -c 192.168.2.13 -p 12002 -u -b 100M -t 300 -l 16 -A 5 &

      ##Auxiliary ECS 4

      iperf3 -c 192.168.2.14 -p 12001 -u -b 100M -t 300 -l 16 -A 6 &

      iperf3 -c 192.168.2.14 -p 12002 -u -b 100M -t 300 -l 16 -A 7 &

      ##Auxiliary ECS 5

      iperf3 -c 192.168.2.15 -p 12001 -u -b 100M -t 300 -l 16 -A 8 &

      iperf3 -c 192.168.2.15 -p 12002 -u -b 100M -t 300 -l 16 -A 9 &

      ##Auxiliary ECS 6

      iperf3 -c 192.168.2.16 -p 12001 -u -b 100M -t 300 -l 16 -A 10 &

      iperf3 -c 192.168.2.16 -p 12002 -u -b 100M -t 300 -l 16 -A 11 &

      ##Auxiliary ECS 7

      iperf3 -c 192.168.2.17 -p 12001 -u -b 100M -t 300 -l 16 -A 12 &

      iperf3 -c 192.168.2.17 -p 12002 -u -b 100M -t 300 -l 16 -A 13 &

      ##Auxiliary ECS 8

      iperf3 -c 192.168.2.18 -p 12001 -u -b 100M -t 300 -l 16 -A 14 &

      iperf3 -c 192.168.2.18 -p 12002 -u -b 100M -t 300 -l 16 -A 15 &

2. Test the UDP RX PPS.
   1. Start the server process on the tested ECS. For details about common iperf3 parameters, see Table 2.

      ##The port number is for the first auxiliary ECS.

      iperf3 -s -p 12001 -A 0 -i 60 &

      iperf3 -s -p 12002 -A 1 -i 60 &

      ##The port number is for the second auxiliary ECS.

      iperf3 -s -p 12003 -A 2 -i 60 &

      iperf3 -s -p 12004 -A 3 -i 60 &

      ##The port number is for the third auxiliary ECS.

      iperf3 -s -p 12005 -A 4 -i 60 &

      iperf3 -s -p 12006 -A 5 -i 60 &

      ##The port number is for the fourth auxiliary ECS.

      iperf3 -s -p 12007 -A 6 -i 60 &

      iperf3 -s -p 12008 -A 7 -i 60 &

      ##The port number is for the fifth auxiliary ECS.

      iperf3 -s -p 12009 -A 8 -i 60 &

      iperf3 -s -p 12010 -A 9 -i 60 &

      ##The port number is for the sixth auxiliary ECS.

      iperf3 -s -p 12011 -A 10 -i 60 &

      iperf3 -s -p 12012 -A 11 -i 60 &

      ##The port number is for the seventh auxiliary ECS.

      iperf3 -s -p 12013 -A 12 -i 60 &

      iperf3 -s -p 12014 -A 13 -i 60 &

      ##The port number is for the eighth auxiliary ECS.

      iperf3 -s -p 12015 -A 14 -i 60 &

      iperf3 -s -p 12016 -A 15 -i 60 &

   2. Start the client process on all auxiliary ECSs. For details about common iperf3 parameters, see Table 2.

      Log in to auxiliary ECS 1.

      iperf3 -c 192.168.2.10 -p 12001 -u -b 100M -t 300 -l 16 -A 0 &

      iperf3 -c 192.168.2.10 -p 12002 -u -b 100M -t 300 -l 16 -A 1 &

      Log in to auxiliary ECS 2.

      iperf3 -c 192.168.2.10 -p 12003 -u -b 100M -t 300 -l 16 -A 0 &

      iperf3 -c 192.168.2.10 -p 12004 -u -b 100M -t 300 -l 16 -A 1 &

      Log in to auxiliary ECS 3.

      iperf3 -c 192.168.2.10 -p 12005 -u -b 100M -t 300 -l 16 -A 0 &

      iperf3 -c 192.168.2.10 -p 12006 -u -b 100M -t 300 -l 16 -A 1 &

      Log in to auxiliary ECS 4.

      iperf3 -c 192.168.2.10 -p 12007 -u -b 100M -t 300 -l 16 -A 0 &

      iperf3 -c 192.168.2.10 -p 12008 -u -b 100M -t 300 -l 16 -A 1 &

      Log in to auxiliary ECS 5.

      iperf3 -c 192.168.2.10 -p 12009 -u -b 100M -t 300 -l 16 -A 0 &

      iperf3 -c 192.168.2.10 -p 12010 -u -b 100M -t 300 -l 16 -A 1 &

      Log in to auxiliary ECS 6.

      iperf3 -c 192.168.2.10 -p 12011 -u -b 100M -t 300 -l 16 -A 0 &

      iperf3 -c 192.168.2.10 -p 12012 -u -b 100M -t 300 -l 16 -A 1 &

      Log in to auxiliary ECS 7.

      iperf3 -c 192.168.2.10 -p 12013 -u -b 100M -t 300 -l 16 -A 0 &

      iperf3 -c 192.168.2.10 -p 12014 -u -b 100M -t 300 -l 16 -A 1 &

      Log in to auxiliary ECS 8.

      iperf3 -c 192.168.2.10 -p 12015 -u -b 100M -t 300 -l 16 -A 0 &

      iperf3 -c 192.168.2.10 -p 12016 -u -b 100M -t 300 -l 16 -A 1 &

3. Analyze the test result.

   Figure 2 shows an example of the UDP PPS test result.

   Figure 2 UDP PPS test result
   
   

   There are a large number of iperf3 processes. To facilitate statistics collection, it is a good practice to run the following command to view test data on the tested ECS using sar:

   sar -n DEV 1 60

Latency Test

1. Run the following command to start the qperf process on the tested ECS:

   qperf &

2. Log in to auxiliary ECS 1 and run the following command to perform a latency test:

   qperf 192.168.2.10 -m 64 -t 60 -vu udp_lat

   After the test is complete, the lat value in the command output is the latency between ECSs.