Collecting Logs from Third-Party Cloud Vendors, Internet Data Centers, and Other Huawei Cloud Regions to LTS

Solution Overview

Cloud users often need to collect logs across clouds or regions. There are two typical scenarios:

• Scenario 1: collecting logs from IDCs or third-party clouds to Huawei Cloud LTS
  Figure 1 Third-party cloud log collection
  
• Scenario 2: collecting logs from one Huawei Cloud region to LTS in another Huawei Cloud region
  Figure 2 Cross-region log collection
  

In both scenarios, you need to establish a network connection, install ICAgent, and follow the log ingestion wizard.

• ICAgent: the log collector of Huawei Cloud LTS. After being installed on a host, it collects logs from the host to LTS.
  

  Ensure that the time and time zone of your local browser are consistent with those of the host to install ICAgent.

• Networking
  • Scenario 1: Direct Connect is a typical method for connecting a customer-built IDC or third-party cloud to Huawei Cloud. If Direct Connect is unavailable, you can use a VPN or public IP address.
  • Scenario 2: Cloud Connect or Direct Connect is a typical method for interconnecting Huawei Cloud regions. You can also use a VPN or public IP address.
• Jump server
  • ICAgent installed in customer-built IDCs, third-party clouds, or other Huawei Cloud regions cannot directly access the network segment used by the Huawei Cloud management plane for log reporting, necessitating a jump server for data forwarding. Use the jump server solution for Proof of Concept (PoC) tests or when log traffic is light. If you do not want to use jump servers for heavy traffic scenarios in production environments, submit a service ticket to Huawei Cloud technical support to design a network passthrough solution.
  • A typical jump server configuration is 2 vCPUs and 4 GB memory, allowing it to forward traffic at approximately 30 MB/s. Configure a proper number of jump servers based on your log traffic and use a load balancer to distribute traffic among them.

This practice describes how to collect Alibaba Cloud host logs to Huawei Cloud LTS. The method is similar to that of collecting logs from customer-built IDCs or across Huawei Cloud regions.

Below are the steps to collect the logs from a Linux host in Alibaba Cloud's China (Beijing) region to LTS in Huawei Cloud's CN East-Shanghai1 region.

Planning Resources

Purchasing a Load Balancer and an ECS as a Jump Server in Huawei Cloud CN East-Shanghai1

1. Log in to the ECS console and buy an ECS.

   Before installing ICAgent on a non-Huawei Cloud host, buy an ECS as a jump server from Huawei Cloud.

2. Buy a load balancer, add TCP listeners, and associate a backend server group with it.
   1. Add listeners for ports 30200, 30201, 8149, 8923, and 8102. For details, see Adding a TCP Listener.
   2. Add the jump server to a backend server group. For details, see Backend Server.

3. Configure a security group rule for the jump server and open forwarding ports.
   1. Modify the security group rule used by the jump server.
      1. On the ECS console, click the name of the ECS used as the jump server to go to the details page.
      2. On the Security Groups tab page, click a security group name to go to the details page.
      3. Click the Inbound Rules tab and click Add Rule. Open the inbound ports 8149, 8102, 8923, 30200, 30201, and 80 to ensure that data can be transmitted from the non-Huawei Cloud host to the jump server.
      Figure 3 Modifying a security group rule
      
   2. On the LTS console, choose Host Management > Hosts in the navigation pane, and click Install ICAgent in the upper right corner. Set parameters as shown in the following figure. Set Private IP to the private IP address of the ECS to generate an installation password.
      Figure 4 Installing ICAgent
      
   3. Copy the command, log in to the jump server as user root, run the SSH tunneling command, and enter the password of user root as prompted.
   4. Run the netstat -lnp | grep ssh command to check whether the corresponding TCP ports are being listened to. If the command output similar to the following is returned, the ports are open.
      Figure 5 Viewing ports
      
      

      • Enter http://Jump server IP address in the address bar of a browser. If the access is successful, the security group rule has taken effect.
      • If the jump server is powered off and then restarted, run the installation command generated on the ICAgent installation page again. If you use the jump server in a production environment, configure the SSH tunneling command to run upon system startup.

Installing ICAgent on an Alibaba Cloud Host

1. Obtain an AK/SK. For details, see How Do I Obtain an Access Key (AK/SK)?
2. Enter the connection IP address of the jump server (the EIP of the ECS) to generate the ICAgent installation command.
   

   • Replace the AK/SK in the command with the correct AK/SK. Otherwise, ICAgent cannot be installed.
   • Connection IP: For EIP connection, enter the EIP of the jump server. For VPC peer connection, enter the internal IP address of the VPC where the jump server is located.

3. Log in to the Alibaba Cloud host as user root and run the ICAgent installation command. If the message ICAgent install success is displayed, ICAgent is successfully installed.
   Figure 6 Checking the ICAgent installation status
   
   

   If you use LTS to collect logs across Huawei Cloud regions, for example, collecting logs from the CN East-Shanghai1 region to the CN South-Guangzhou region, you need to buy a load balancer and an ECS used as a jump server in CN South-Guangzhou, and then run the ICAgent installation command on the jump server in CN East-Shanghai1.

4. Check wether the ICAgent status is Running by choosing Host Management > Hosts in the navigation pane of the LTS console.

Ingesting Logs to LTS

1. Log in to the LTS console and choose Host Management > Host Groups in the navigation pane. Click Create Host Group. On the displayed page, enter a host group name and select hosts.
2. Configure a log ingestion rule. For details, see Ingesting ECS Text Logs to LTS.

Viewing a Log Stream

On the Log Management page of LTS, click the target log stream to go to its details page. If there are logs, the Alibaba Cloud logs have been reported to LTS.