Compute
Elastic Cloud Server
Huawei Cloud Flexus
Bare Metal Server
Auto Scaling
Image Management Service
Dedicated Host
FunctionGraph
Cloud Phone Host
Huawei Cloud EulerOS
Networking
Virtual Private Cloud
Elastic IP
Elastic Load Balance
NAT Gateway
Direct Connect
Virtual Private Network
VPC Endpoint
Cloud Connect
Enterprise Router
Enterprise Switch
Global Accelerator
Management & Governance
Cloud Eye
Identity and Access Management
Cloud Trace Service
Resource Formation Service
Tag Management Service
Log Tank Service
Config
OneAccess
Resource Access Manager
Simple Message Notification
Application Performance Management
Application Operations Management
Organizations
Optimization Advisor
IAM Identity Center
Cloud Operations Center
Resource Governance Center
Migration
Server Migration Service
Object Storage Migration Service
Cloud Data Migration
Migration Center
Cloud Ecosystem
KooGallery
Partner Center
User Support
My Account
Billing Center
Cost Center
Resource Center
Enterprise Management
Service Tickets
HUAWEI CLOUD (International) FAQs
ICP Filing
Support Plans
My Credentials
Customer Operation Capabilities
Partner Support Plans
Professional Services
Analytics
MapReduce Service
Data Lake Insight
CloudTable Service
Cloud Search Service
Data Lake Visualization
Data Ingestion Service
GaussDB(DWS)
DataArts Studio
Data Lake Factory
DataArts Lake Formation
IoT
IoT Device Access
Others
Product Pricing Details
System Permissions
Console Quick Start
Common FAQs
Instructions for Associating with a HUAWEI CLOUD Partner
Message Center
Security & Compliance
Security Technologies and Applications
Web Application Firewall
Host Security Service
Cloud Firewall
SecMaster
Anti-DDoS Service
Data Encryption Workshop
Database Security Service
Cloud Bastion Host
Data Security Center
Cloud Certificate Manager
Edge Security
Managed Threat Detection
Blockchain
Blockchain Service
Web3 Node Engine Service
Media Services
Media Processing Center
Video On Demand
Live
SparkRTC
MetaStudio
Storage
Object Storage Service
Elastic Volume Service
Cloud Backup and Recovery
Storage Disaster Recovery Service
Scalable File Service Turbo
Scalable File Service
Volume Backup Service
Cloud Server Backup Service
Data Express Service
Dedicated Distributed Storage Service
Containers
Cloud Container Engine
SoftWare Repository for Container
Application Service Mesh
Ubiquitous Cloud Native Service
Cloud Container Instance
Databases
Relational Database Service
Document Database Service
Data Admin Service
Data Replication Service
GeminiDB
GaussDB
Distributed Database Middleware
Database and Application Migration UGO
TaurusDB
Middleware
Distributed Cache Service
API Gateway
Distributed Message Service for Kafka
Distributed Message Service for RabbitMQ
Distributed Message Service for RocketMQ
Cloud Service Engine
Multi-Site High Availability Service
EventGrid
Dedicated Cloud
Dedicated Computing Cluster
Business Applications
Workspace
ROMA Connect
Message & SMS
Domain Name Service
Edge Data Center Management
Meeting
AI
Face Recognition Service
Graph Engine Service
Content Moderation
Image Recognition
Optical Character Recognition
ModelArts
ImageSearch
Conversational Bot Service
Speech Interaction Service
Huawei HiLens
Video Intelligent Analysis Service
Developer Tools
SDK Developer Guide
API Request Signing Guide
Terraform
Koo Command Line Interface
Content Delivery & Edge Computing
Content Delivery Network
Intelligent EdgeFabric
CloudPond
Intelligent EdgeCloud
Solutions
SAP Cloud
High Performance Computing
Developer Services
ServiceStage
CodeArts
CodeArts PerfTest
CodeArts Req
CodeArts Pipeline
CodeArts Build
CodeArts Deploy
CodeArts Artifact
CodeArts TestPlan
CodeArts Check
CodeArts Repo
Cloud Application Engine
MacroVerse aPaaS
KooMessage
KooPhone
KooDrive
Help Center/ Log Tank Service/ Best Practices/ Log Ingestion/ Collecting Kubernetes Logs from Third-Party Clouds, IDCs, and Other Huawei Cloud Regions to LTS

Collecting Kubernetes Logs from Third-Party Clouds, IDCs, and Other Huawei Cloud Regions to LTS

Updated on 2025-02-17 GMT+08:00

Solution Overview

Cloud users often need to collect Kubernetes logs across clouds or regions. There are two typical scenarios:

  • Scenario 1: collecting logs from IDCs or third-party clouds to Huawei Cloud LTS
    Figure 1 Third-party cloud log collection
  • Scenario 2: collecting logs from one Huawei Cloud region to LTS in another Huawei Cloud region
    Figure 2 Cross-region log collection

In both scenarios, you need to establish a network connection, install ICAgent, and follow the log ingestion wizard.

  • ICAgent: the log collector of Huawei Cloud LTS. After being installed on a host, it collects logs from the host to LTS. Ensure that the time and time zone of your local browser are consistent with those of the host to install ICAgent.
  • Networking
    • Scenario 1: Direct Connect is a typical method for connecting a customer-built IDC or third-party cloud to Huawei Cloud. If Direct Connect is unavailable, you can use a VPN or public IP address.
    • Scenario 2: Cloud Connect or Direct Connect is a typical method for interconnecting Huawei Cloud regions. You can also use a VPN or public IP address.
  • Jump server
    • ICAgent installed in customer-built IDCs, third-party clouds, or other Huawei Cloud regions cannot directly access the network segment used by the Huawei Cloud management plane for log reporting, necessitating a jump server for data forwarding. Use the jump server solution for Proof of Concept (PoC) tests or when log traffic is light. If you do not want to use jump servers for heavy traffic scenarios in production environments, submit a service ticket to Huawei Cloud technical support to design a network passthrough solution.
    • A typical jump server configuration is 2 vCPUs and 4 GB memory, allowing it to forward traffic at approximately 30 MB/s. Configure a proper number of jump servers based on your log traffic and use a load balancer to distribute traffic among them.

This practice describes how to collect Alibaba Cloud host logs to Huawei Cloud LTS. The method is similar to that of collecting logs from customer-built IDCs or across Huawei Cloud regions.

Below are the steps to collect the logs from a Linux host in Alibaba Cloud's China (Beijing) region to LTS in Huawei Cloud's CN East-Shanghai1 region.

Planning Resources

Table 1 Planning resources

Region

Resource

Description

CN East-Shanghai1

ECS

You are advised to use CentOS 6.5 64bit or later images. The minimum specifications are 1 vCPU | 1 GB and the recommended ones are 2 vCPUs | 4 GB.

Load balancer

  • When buying a load balancer, select the same VPC as the ECS.
  • Create an EIP for connecting to the jump servers.
  • Buy the bandwidth based on the service requirements.

Purchasing a Load Balancer and an ECS as a Jump Server in Huawei Cloud CN East-Shanghai1

  1. Log in to the ECS console and buy an ECS.

    Before installing ICAgent on a non-Huawei Cloud host, buy an ECS as a jump server from Huawei Cloud.

  2. Buy a load balancer, add TCP listeners, and associate a backend server group with it.

    1. Add listeners for TCP ports 30200, 30201, 8149, 8923, and 8102. For details, see Adding a TCP Listener.
    2. Add the jump server to a backend server group. For details, see Backend Server.

  3. Configure a security group rule for the jump server and open forwarding ports.

    1. Modify the security group rule used by the jump server.
      1. On the ECS console, click the name of the ECS used as the jump server to go to the details page.
      2. On the Security Groups tab page, click a security group name to go to the details page.
      3. Click the Inbound Rules tab and click Add Rule. Open the inbound ports 8149, 8102, 8923, 30200, 30201, and 80 to ensure that data can be transmitted from the non-Huawei Cloud host to the jump server.
      Figure 3 Modifying a security group rule
    2. On the LTS console, choose Host Management > Hosts in the navigation pane, and click Install ICAgent in the upper right corner. Set parameters as shown in the following figure. Set Private IP to the private IP address of the ECS to generate an installation password.
      Figure 4 Installing ICAgent
    3. Copy the command, log in to the jump server as user root, run the SSH tunneling command, and enter the password of user root as prompted.
    4. Run the netstat -lnp | grep ssh command to check whether the corresponding TCP ports are being listened to. If the command output similar to the following is returned, the ports are open.
      • Enter http://Jump server IP address in the address bar of a browser. If the access is successful, the security group rule has taken effect.
      • If the jump server is powered off and then restarted, run the installation command generated on the ICAgent installation page again. If you use the jump server in a production environment, configure the SSH tunneling command to run upon system startup.
      Figure 5 Viewing ports

Configuring Log Ingestion

For a Kubernetes cluster, simply install ICAgent on one node, not all nodes.

Obtain an AK/SK in advance. For details, see How Do I Obtain an Access Key (AK/SK)?

  1. Configure the jump server.

    1. On the ECS console, locate the jump server and obtain its private IP address.
      Figure 6 Obtaining the private IP address
    2. On the LTS console, choose Host Management > Hosts in the navigation pane and click Install ICAgent. On the page displayed, set parameters as follows, set Private IP to the private IP address of the ECS to generate an installation command, and copy the command.
      Figure 7 Installing ICAgent
    3. Log in to the ECS, run the command copied in the previous step, and enter the node password as prompted. If no error is reported, the installation is successful.
      Figure 8 Running the generated installation command
    4. On the Install ICAgent page, set Connection IP to the public IP address of the jump server. Ensure that the checkbox next to Turn off command history to prevent the AK/SK from being stored is selected.
    5. Copy the ICAgent installation command and run it on the jump server. Enter the AK and SK of the current account as prompted. If the message ICAgent install success is displayed, ICAgent is successfully installed.

  2. Configure a log ingestion rule. For details, see Ingesting Self-Built Kubernetes Application Logs to LTS.

Viewing a Log Stream

On the Log Management page of LTS, click the target log stream to go to its details page. If there are logs, the Alibaba Cloud Kubernetes logs have been reported to LTS.

We use cookies to improve our site and your experience. By continuing to browse our site you accept our cookie policy. Find out more

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback