Configuring the TOA Module

Scenario

Global Accelerator provides customized strategies for managing service access. Before these strategies can be customized, the clients' IP addresses contained in the requests are required. The TCP Option Address (TOA) kernel module is used to obtain the IP addresses of clients. It is installed on the server of the endpoint.

This section describes how you can compile the module in the OS if you use TCP to distribute IPv4 traffic.

The operations for Linux OSs with kernel version of 2.6.32 are different from those for Linux OSs with kernel version of 3.0 or later.

The TOA module cannot be used for UDP listeners.
The TOA module can work properly in the following OSs, and the methods for installing other kernel versions are similar:
- CentOS 6.8 (kernel version 2.6.32)
- SUSE 11 SP3 (kernel version 3.0.76)
- CentOS 7 or CentOS 7.2 (kernel version 3.10.0)
- Ubuntu 16.04.3 (kernel version 4.4.0)
- Ubuntu 18.04 (kernel version 4.15.0)
- OpenSUSE 42.2 (kernel version 4.4.36)
- Debian 8.2.0 (kernel version 3.16.0)

Constraints

The development environment for compiling the module must be the same as that of the current kernel. For example, if the kernel version is kernel-3.10.0-693.11.1.el7, the kernel development package version must be kernel-devel-3.10.0-693.11.1.el7.
The OS repositories are accessible to servers.
Users other than root must have sudo permissions.

Procedure

Linux kernel version 3.0 or later

Prepare the compilation environment.
- During the installation, download the required module development package from the Internet if it cannot be found in the source.
- If the kernel development package (kernel-devel) cannot be obtained, contact the image provider.
The following are operations for compiling the module in different Linux OSs.
- CentOS
  1. Install the gcc compiler.
    sudo yum install gcc
  2. Install the make tool.
    sudo yum install make
  3. Install the module development package (the package header and module library must have the same version as the kernel).
    sudo yum install kernel-devel-`uname -r`
    Download the required module development package from the following address if it cannot be found in the source:
    https://mirror.netcologne.de/oracle-linux-repos/ol7_latest/getPackage/
    
    For example, run the following command to install 3.10.0-693.11.1.el7.x86_64:
    
    rpm -ivh kernel-devel-3.10.0-693.11.1.el7.x86_64.rpm
    
    If the kernel development package (kernel-devel) cannot be obtained, contact the image provider.
- Ubuntu and Debian
  1. Install the gcc compiler.
    sudo apt-get install gcc
  2. Install the make tool.
    sudo apt-get install make
  3. Install the module development package (the package header and module library must have the same version as the kernel).
    sudo apt-get install linux-headers-`uname -r`
- SUSE
  1. Install the gcc compiler.
    sudo zypper install gcc
  2. Install the make tool.
    sudo zypper install make
  3. Install the module development package (the package header and module library must have the same version as the kernel).
    sudo zypper install kernel-default-devel
Compile the module.
1. Download the module source code.
  git clone https://github.com/Huawei/TCP_option_address.git
  
  If the git tool is not installed, download the module source code from the following link:
  
  https://github.com/Huawei/TCP_option_address
2. Enter the source code directory and compile the module.
  cd src
  
  make
  
  If no warning or error information is prompted, the compilation is successful. Verify that the toa.ko file has generated in the current directory.
  
  If error message "config_retpoline=y but not supported by the compiler, Compiler update recommended" is displayed, the GCC version is too old. Upgrade the GCC to a later version.
Load the module.
1. Load the module.
  sudo insmod toa.ko
2. Check the module loading and view the kernel output information.
  dmesg | grep TOA
  
  If "TOA: toa loaded" is displayed in the command output, the module has been loaded.
  
  After the CoreOS module is compiled in the container, copy it to the host system and then load it. The container for compiling the module shares the /lib/modules directory with the host system, so you can copy the module to this directory, allowing the host system to use it.
Set the script to enable the system to automatically load the module.
To make the module take effect when the system starts, add the command for loading the module to your startup script.

You can use either of the following methods to enable the module to automatically load:
- Add the command for the module to automatically load to the startup script as required.
- Perform the following operations to configure the startup script:
  1. Create the toa.modules file in the /etc/sysconfig/modules/ directory. This file contains the module loading script.
    The following is an example of the content in the toa.modules file.
    
    #!/bin/sh
    
    /sbin/modinfo -F filename /root/toa/toa.ko > /dev/null 2>&1
    
    if [ $? -eq 0 ]; then
    
    /sbin/insmod /root/toa/toa.ko
    
    fi
    
    /root/toa/toa.ko is the path of the module file. You need to replace it with their actual path.
  2. Add execution permissions for the toa.modules startup script.
    sudo chmod +x /etc/sysconfig/modules/toa.modules
    
    If the kernel is upgraded, the current module will no longer match. Compile the module again.
Install the module on servers.
To load the module in the same OSs, copy the toa.ko file to VMs where the module is to be loaded and then perform the operations in 3.

After the module is loaded, the IP address of a client can be obtained.

The OS version of each server must be the same as that of the kernel.
Verify the module.
After the module is installed, the source IP address can be directly obtained. You can perform the following operations to verify:

Start SimpleHTTPServer on the server of the endpoint where Python is installed:

python -m SimpleHTTPServer port

The value of port must be a port used by the server, and the default value is 80.

Access the anycast IP address provided by Global Accelerator. Access logs on the server are as follows:
```
192.168.0.90 - - [06/Aug/2020 14:24:21] "GET / HTTP/1.1" 200 –
```
192.168.0.90 is the source IP address and also the real IP address of the client that can be obtained by the backend server.

In the following operations, the Linux kernel version is 2.6.32.

The TOA module supports OSs (CentOS 6.8 image) with a kernel of 2.6.32-xx.

Obtain the kernel source code package Linux-2.6.32-220.23.1.el6.x86_64.rs.src.tar.gz containing the module from the following link:
http://kb.linuxvirtualserver.org/images/3/34/Linux-2.6.32-220.23.1.el6.x86_64.rs.src.tar.gz
Decompress the kernel source code package.
Modify compilation parameters.
1. Open the linux-2.6.32-220.23.1.el6.x86_64.rs directory.
2. Edit the net/toa/toa.h file.
  Change the value of #define TCPOPT_TOA200 to #define TCPOPT_TOA254.
3. On the Shell page, run the following commands:
  sed -i 's/CONFIG_IPV6=m/CONFIG_IPV6=y/g' .config
  
  echo -e '\n# toa\nCONFIG_TOA=m' >> .config
  
  The IPv6 module has been compiled into the kernel. TOA is compiled into a separate module and can be independently started and stopped.
4. Edit Makefile.
  You can add a description after EXTRAVERSION =. This description will be displayed in uname -r, for example, -toa.
Compile the software package.
make -j n

n indicates the number of vCPUs. For example, if there are four vCPUs, n must be set to 4.
Install the module.
make modules_install

Figure 1 shows the command output.

Figure 1 Installing the module
Install the kernel.
make install

Figure 2 shows the command output.

Figure 2 Installing the kernel
Open the /boot/grub/grub.conf file and configure the kernel to start up when the system starts.
1. Change the default startup kernel from the first kernel to the zeroth kernel. To do so, change the value of default to 0.
2. Add the nohz parameter (set it to off) to the end of the line containing the vmlinuz-2.6.32-toa kernel. If nohz is not disabled, the CPU0 utilization may be high and overload the kernel.
  Figure 3 Configuration file
3. Save the modification and exit. Restart the OS.
  During the restart, the system will load the vmlinuz-2.6.32-toa kernel.
After the restart, load the module.
modprobe toa

Add the modprobe toa command to both the startup script and the system scheduled monitoring script.

Figure 4 Adding the modprobe toa command

After the module is loaded, query the kernel information.

Figure 5 Querying the kernel
Verify the module.
After the module is installed, the source IP address can be directly obtained. You can perform the following operations to verify:

Start SimpleHTTPServer on the server of the endpoint where Python is installed:

python -m SimpleHTTPServer port

The value of port must be a port used by the server, and the default value is 80.

Access the anycast IP address provided by Global Accelerator. Access logs on the server are as follows:
```
192.168.0.90 - - [06/Aug/2020 14:24:21] "GET / HTTP/1.1" 200 –
```
192.168.0.90 is the source IP address and also the real IP address of the client that can be obtained by the backend server.