QingTian Enclave Network Proxy

Overview

QingTian Enclave proxy (qproxy) is a network proxy tool of QingTian Enclave. With this tool, you can smoothly migrate network services that are deployed on VMs based on the QingTian architecture to QingTian Enclave instances without any modifications.

The qproxy tool is an executable binary file. It needs to be executed using different commands on the parent instance and QingTian Enclave instance.

• On the parent instance, run /path/to/qproxy host --config=/path/to/config_qproxy.toml <cid> to enable qproxy.
• On a QingTian Enclave instance, run /path/to/qproxy host --config=/path/to/config_qproxy.toml <cid> to enable qproxy.

For example, a network service is directly deployed on a VM to process requests from end users. End users initiate network requests through the port exposed by the service and wait for the service to respond. After the network service is migrated to QingTian Enclave with qproxy, the same network service is provided.

You can run qlog host and qlog enclave on the parent instance and QingTian Enclave instance, respectively, to execute a given qproxy binary file. A local vsock-based communication link is established between them. The qproxy component in the parent instance listens to port 5050, receives user requests, and forwards the received requests to the qproxy component in the QingTian Enclave instance through the local vsock. The qproxy sends requests to the listened port 5050 in the QingTian Enclave instance. After the network service processing is complete, the response is returned along the original path.

Figure 1 Seamless migration of a network service to QingTian Enclave

The following describes how to use qproxy.

Prerequisites

1. You have obtained the qproxy source code by performing the following:

   Clone the QingTian Enclave code repository.

   git clone https://gitee.com/HuaweiCloudDeveloper/huawei-qingtian.git

2. You have obtained the cargo tool chain by performing the following:
   1. Install rustup.

      curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

   2. After the installation is complete, load rustup.

      source $HOME/.cargo/env

   3. Check that rustc and cargo are installed.

      rustc -V
      cargo -V

3. You have learned about the pre-dependency of qproxy.

   Table 1 Pre-dependency

   Dependency Item	Earliest Test Version
   cargo	1.77.0
   libcbor	0.10.2
   libssl (libcrypto)	3.0.0
   libcurl	4.0.0

4. You have prepared the QingTian Enclave environment by performing the following:
   1. Install the qt CLI tool and required RPM packages.
   2. Install Docker.
   3. Install Python 3 and required Python modules (docker and knack).

   For details, see Getting Started with QingTian Enclave and Installation of the qt CLI.

Procedure

1. Build qproxy.

   The generated qproxy binary file is compiled in qingtian-tools/qproxy/target/release.

2. Create a workspace.

   Create a workspace and copy the qproxy binary file to the workspace. Store the files generated subsequently in the workspace.

3. Configure the config_qproxy.toml file.

   In the workspace directory, create the config_qproxy.toml file with the following content:

   [[inbound_connections]]
   host_port = 5050 
   enclave_port = 5050 
   vsock_port = 9995  
   
   [[inbound_connections]]
   host_port = 5443 
   enclave_port = 5443 
   vsock_port = 9994  
   
   [log_location]
   host_log = "host.log" # qproxy host log name, e.g./var/log/qproxy/host.log
   enclave_log = "enclave.log" # qproxy enclave log name, e.g./var/log/qproxy/enclave.log
   log_level = "info" # qproxy logger level, e.g. "off", "info", "warn", "error", "debug", "trace"
   host_log_dir = "/var/log/qproxy" # qproxy host log dir, and its default value is "/var/log/qproxy"
   enclave_log_dir = "/var/log/qproxy" # qproxy enclave log dir, and its default value is "/var/log/qproxy"

4. Create a QingTian Enclave image that contains qproxy.
   1. In the workspace directory, create the HTTP request test script app.sh.

      #!/bin/bash 
      
      PORT=5050 
      
      while true; do    
           echo -e "HTTP/1.1 200 OK\nContent-Type: text/plain\n\nHello world!" | nc -l 127.0.0.1  $PORT
      done

   2. Run the following command in the workspace to grant app.sh execute permissions:

      chmod +x app.sh

   3. In the workspace directory, create the start.sh script.

      #!/bin/bash
      ip link set lo up 
      /root/qproxy enclave --config=/root/config_qproxy.toml &
      /root/app.sh

   4. Run the following command in the workspace to grant start.sh execute permissions:

      chmod +x start.sh

   5. In the workspace directory, create a Dockerfile.

      FROM ubuntu:latest 
      COPY ./qproxy /root/qproxy 
      COPY ./config_qproxy.toml /root/config_qproxy.toml 
      COPY ./start.sh /root/start.sh 
      COPY ./app.sh /root/app.sh 
      RUN apt-get update && \    
          apt-get install -y netcat-openbsd && \    
          apt-get install -y iproute2 
      CMD "/root/start.sh"

   6. Run the following command in the workspace to create a Docker image:

      docker build -f Dockerfile -t test_qproxy_enclave .

   7. Run the following command in the workspace to create a QingTian Enclave image:

      qt enclave make-img --docker-uri test_qproxy_enclave --eif test_qproxy_enclave.eif

5. Start qproxy.
   1. Run the following command in the workspace to start a QingTian Enclave instance:

      qt enclave start --cpus 2 --mem 1024 --cid 4 --eif test_qproxy_enclave.eif

   2. Run the following command in the workspace to start qproxy in the parent instance:

      ./qproxy host --config=./config_qproxy.toml 4 &

   3. Run the following curl command on the parent instance:

      curl localhost:5050

      "Hello world!" is displayed.

6. Set the qproxy environment variables.

   The qproxy contains two sub-commands, one is executed on the QingTian Enclave instance (qproxy enclave), and the other is executed on the parent instance (qproxy host).

   You can set the RUST_LOG environment variable to control the output of different levels of logs of a binary file.

   • RUST_LOG=OFF: All logs are not displayed.
   • RUST_LOG=info: Logs of the "info", "warn", and "error" levels are displayed.
   • By default, logs of the "warn" and "error" levels are displayed.
   • For more information, see the EnvFilter documentation.

qproxy Help Information

qproxy help

$ qproxy --help
Usage: qproxy <COMMAND> 

Commands:   
   host          The part of qproxy that runs outside the enclave   
   enclave       The part of qproxy that runs inside the enclave   
   check-config  Check the qproxy configuration file  
   help          Print this message or the help of the given subcommand(s) 

Options:   
   -h, --help     Print help  
   -V, --version  Print version

qproxy enclave help

$ qproxy enclave --help
The part of qproxy that runs inside the enclave  

Usage: qproxy enclave [OPTIONS] 

Options:       
      --parent-cid <PARENT_CID>          
          The CID of the parent VM of this enclave                      

          [env: QPROXY_PARENT_CID=]          
          [default: 3]       

      --config <CONFIG>          
           Path to the configuration file   

  -t, --threads <THREADS>          
           Number of threads the async runtime is allowed to use                      

           [env: TOKIO_WORKER_THREADS=]       

       --control-port <CONTROL_PORT>          
            The port where to listen for control messages from the enclave                      

            Leave at default value unless you know what you are doing                      

            [env: QPROXY_CONTROL_PORT=]          
            [default: 6666]          
            [0..=65535]   

  -h, --help          
            Print help (see a summary with '-h')

qproxy host help

$ qproxy host --help
The part of qproxy that runs outside the enclave  

Usage: qproxy host [OPTIONS] <CID> 

Arguments:   
  <CID>          
           The CID of the enclave                      

           [env: QPROXY_LISTEN_CID=] 

Options:       
        --config <CONFIG>          
             Path to the configuration file   

  -t, --threads <THREADS>          
             Number of threads the async runtime is allowed to use                      

             [env: TOKIO_WORKER_THREADS=]       

      --ipv4          
          Only resolve IPv4 addresses        

      --ipv6          
          Only resolve IPv6 addresses        

      --control-port <CONTROL_PORT>          
          The port where to listen for control messages from the enclave                      

          Leave at default value unless you know what you are doing                      

          [env: QPROXY_CONTROL_PORT=]          
          [default: 6666]          
          [0..=65535]   

  -h, --help          
          Print help (see a summary with '-h')

Configuration Information

Configuration Parameters

• outbound_connections: used to forward traffic from QingTian Enclave instances to specific external services (hostnames/IP addresses) and ports.
• inbound_connections: used to forward traffic received by the specified port of the parent instance to QingTian Enclave instances.

Log File Configuration

Configuration File Reference

[[outbound_connections]]
hostname = "api.myservice.com" 
vsock_port = 7777 
tcp_port = 443  

[[outbound_connections]]
hostname = "another.api.com" 
vsock_port = 7778 
tcp_port = 5555  

[[inbound_connections]]
host_port = 80  
enclave_port = 80  
vsock_port = 9000   

[[inbound_connections]]
host_port = 443 
enclave_port = 443 
vsock_port = 9001  

[log_location]
host_log = "host.log"  
enclave_log = "enclave.log"  
log_level = "info"  
host_log_dir = "/var/log/qproxy"  
enclave_log_dir = "/var/log/qproxy"