Overview

You can use SecMaster to collect security logs on and off the cloud, as well as transfer security logs from SecMaster to a third-party system and product.

**Table 1** Log access and transfer scenarios
Scenario	Operation Guide
Integrating logs of cloud services into SecMaster	For details, see Enabling Log Access. You can enable log integration for many cloud services all at once.
Integrating security logs from third parties (non-Huawei Cloud) into SecMaster	Security logs of third parties are integrated by data collection. This section describes the data collection function.
Transferring logs from SecMaster to a third-party system or product	The procedure is the same as that for integrating logs from a third-party (non-Huawei Cloud) system into SecMaster. Major differences are as follows: You need to create a log storage pipeline if you plan to integrate security logs from non-Huawei Cloud systems into SecMaster. No pipeline is required if you only need to transfer Huawei Cloud logs to a third-party system or product. You need to create a log storage pipeline if you plan to integrate security logs from non-current cloud systems into SecMaster. No pipeline is required if you only need to transfer logs on the current cloud to a third-party system or product. The data connection source and destination parameters are different when you configure a connector. If you integrate logs from other clouds into SecMaster, select SecMaster for Connection Type when configuring data connection destination. If you transfer SecMaster logs to a third-party system or product, select SecMaster for Connection Type when configuring data connection source.

This section describes how to integrate third-party logs into SecMaster. You can collect third-party logs using Logstash. Logstash can collect logs in various ways. After logs are collected, SecMaster can analyze historical data, associate data, and detect unknown threats.

Figure 1 Data collection

Data Collection Principles

The basic principle of data collection is as follows: SecMaster uses a component controller (isap-agent) that is installed on your ECSs to manage the collection component Logstash, and Logstash transfer security data in your organization or between you and SecMaster.

Figure 2 Functional architecture of the collection system

Basic Concepts

Collector: custom Logstash. A collector node is a custom combination of Logstash+ component controller (isap-agent).
Node: If you install SecMaster component controller isap-agent on an ECS, the ECS is called a node. You need to deliver data collection engine Logstash to managed nodes on the Components page.
Component: A component is a custom Logstash that works as a data aggregation engine to receive and send security log data.
Connector: A connector is a basic element for Logstash. It defines the way Logstash receives source data and the standards it follows during the process. Each connector has a source end and a destination end. Source ends and destination ends are used for data inputs and outputs, respectively. The SecMaster pipeline is used for log data transmission between SecMaster and your devices.
Parser: A parser is a basic element for configuring custom Logstash. Parsers mainly work as filters in Logstash. SecMaster preconfigures varied types of filters and provides them as parsers. In just a few clicks on the SecMaster console, you can use parsers to generate native scripts to set complex filters for Logstash. In doing this, you can convert raw logs into the format you need.
Collection channel: A collection channel is equivalent to a Logstash pipeline. Multiple pipelines can be configured in Logstash. Each pipeline consists of the input, filter, and output parts. Pipelines work independently and do not affect each other. You can deploy a pipeline for multiple nodes. A pipeline is considered one collection channel no matter how many nodes it is configured for.

Limitations and Constraints

Currently, the data collection component controller can run on Linux ECSs running the x86 or Arm architecture.

Prerequisites

You have purchased an ECS or have an available ECS for installing the log collector.
You have purchased a data disk and attached it to the ECS. The ECS has enough space to run the log collector.
You have obtained a non-administrator IAM account to log in to SecMaster as a tenant.
You have configured network connection to connect the tenant VPC to SecMaster.

Collector Specifications

The following table describes the specifications of the ECSs that are selected as nodes in collection management.

**Table 2** Collector Specifications
vCPUs	Memory	System Disk	Data Disk	Referenced Processing Capability
4 vCPUs	8 GB	50 GB	100 GB	2,000 EPS @ 1 KB 4,000 EPS @ 500 B
8 vCPUs	16 GB	50 GB	100 GB	5,000 EPS @ 1 KB 10,000 EPS @ 500 B
16 vCPUs	32 GB	50 GB	100 GB	10,000 EPS @ 1 KB 20,000 EPS @ 500 B
32 vCPUs	64 GB	50 GB	100 GB	20,000 EPS @ 1 KB 40,000 EPS @ 500 B
64 vCPUs	128 GB	50 GB	100 GB	40,000 EPS @ 1 KB 80,000 EPS @ 500 B
NOTE: The ECS must have at least two vCPUs and 4 GB of memory. A disk of at least 100 GB must be attached as the directory disk. The log volume usually increases in proportion to the server specifications. Generally, you are advised to increase the log volume based on the specifications in the table. If there is huge pressure on a collector, you can deploy multiple collectors and manage them centrally through collection channels. This can distribute the log forwarding pressure across collectors. Before installing the component controller, you are advised to mount a disk and use the disk partitioning script to allocate the disk. To ensure the installation and running of Logstash, the directory partition must have more than 100 GB of free space.

Log Source Limit

You can add as many as log sources you need to the collectors as long as your cloud resources can accommodate those logs. You can scale cloud resources anytime to meet your needs.

Parent topic: Log Data Collection

Previous topic: Log Data Collection

Next topic: Data Collection Process