Connecting EulerOS Logs to SecMaster
This section describes how to connect EulerOS logs to SecMaster.
Rsyslog is used to remotely collect logs, send logs on tenant ECSs to the corresponding SecMaster nodes over UDP ports after parsing using the EulerOS system log parser provided by collection management. After enabling log integration, you can query them on the
page. The procedure is as follows.
Step |
Description |
---|---|
(Optional) Step 1: Buy an ECS |
Buy an ECS for installing the log collector. |
(Optional) Step 2: Buy a Data Disk |
Buy a data disk to ensure that there is enough space for running the log collector. The data disk and ECS must be in the same AZ, and the data disk capacity cannot be less than 100 GB. |
(Optional) Step 3: Attach a Data Disk |
You need to attach a data disk to the ECS used for the log collector to make sure there is enough space for running the log collector. |
Create a non-administrator IAM user and password for logging in to SecMaster from the log collector. |
|
Step 5: Configure Network Connection |
Before collecting data, you need to establish the network connection between the tenant VPC and SecMaster. |
Manage the log collector node (ECS) on SecMaster. |
|
Configure the log collection process. |
|
Step 8: Create a Log Storage Pipeline |
Create a log storage location (pipeline) in SecMaster for log storage and analysis. |
Step 9: Configure a Connector |
Configure the log source and destination parameters. |
Step 10: Add a Parser |
Create a parser using the EulerOS system log parser template. |
Step 11: Configure a Log Collection Channel |
Connect all function components to ensure that SecMaster and the log collector work properly. |
Configure log forwarding on the tenant side. |
|
Step 13: Start Security Query and Analysis |
After logs are integrated to SecMaster pipeline, you can query logs in SecMaster. |
Notes and Constraints
- The component controller (isap-agent) of the log collector can only run on a Linux x86 or Arm ECS.
- Only a non-administrator IAM user can log in to the console and check information for installing the component controller.
Prerequisites
- You have enabled the paid SecMaster and created a workspace.
For details, see Buying SecMaster and Creating a Workspace.
- The ECSs that collect logs support Rsyslog.
Integrating EulerOS Logs into SecMaster
- Prepare an ECS and install the log collector on the ECS for log collection. Make sure the system disk capacity for the ECS is not less than 50 GB.
If you already have an ECS that meets the requirements, skip this step.
If you need to buy an ECS, see Buying an ECS.
- Buy data disks to ensure that the log collector has sufficient running space.
An idle data disk with capacity not less than 100 GB is required for the ECS you plan to install the log collector. This data disk is used for collection management. The data disk must be in the same AZ as the ECS.
If you have purchased an ECS and configured data disks by referring to Buying an ECS, skip this step. Otherwise, refer to Buying a Data Disk to buy for a data disk.
- Attach the data disk to the ECS that meets the requirements.
Attach the data disk to the ECS that meets the requirements to ensure that the log collector has sufficient running space. For details, see Attaching a Data Disk.
- Scenario 1: You have purchased an ECS and a data disk that meet the requirements by referring to (Optional) Step 1: Buy an ECS and the disk has been attached to the ECS.
- Scenario 2: You already have an ECS that meets the requirements (not purchased by referring to (Optional) Step 1: Buy an ECS), and a data disk that meets the requirements and is purchased based on (Optional) Step 2: Buy a Data Disk. The data disk has been attached to the ECS during the purchase.
- Create a non-administrator IAM account.
IAM authentication is used for tenant log collection. So you need to create an IAM user (machine-machine account) with the minimum permission to access SecMaster APIs. MFA must be disabled for the IAM user. This account is used by the tenant-side log collector to log in to and access SecMaster. For details, see Creating a Non-Administrator IAM Account.
- Before collecting data, you need to establish the network connection between the tenant VPC and SecMaster. For details, see Step 5: Configure Network Connection.
- Install the component controller (isap-agent) and manage log collector nodes (ECSs) to SecMaster. For details, see Installing the Component Controller (isap-agent).
- Install the log collection component (Logstash) and configure the log collection process. For details, see Installing the Log Collection Component (Logstash).
- Create a log storage location (pipeline) in SecMaster for log storage and analysis.
(Optional) Add a data space for importing data. If a third-party data storage space is available, skip this step.
(Optional) Add a storage pipeline in the new data space for importing data. If a storage pipeline is available, skip this step.- Go to the target workspace management page. In the navigation pane on the left, choose Figure 2 Accessing the Security Analysis tab
.
- Add a data space.
- In the upper left corner of the data space list, click Add. The Add Data Space panel is displayed on the right.
Figure 3 Add Data Space
- On the Add Data Space panel, set the parameters for the new data space.
Table 1 Adding a data space Parameter
Description
Data Space
Enter a data space name. The name must meet the following requirements:
- The name can contain 5 to 63 characters.
- Only letters, numbers, and hyphens (-) are allowed. The name cannot start or end with a hyphen (-) or contain consecutive hyphens (-).
- The name cannot be the same as any other data space name on Huawei Cloud.
Example: DataTransfer
Description
(Optional) Remarks of the data space.
- Click OK.
- In the upper left corner of the data space list, click Add. The Add Data Space panel is displayed on the right.
- In the data space navigation tree on the left, click
on the right of the data space name created in 8.b and select Create Pipeline.
Figure 4 Creating a pipeline - On the Create Pipeline page, configure pipeline parameters. For details about the parameters, see Table 2.
Table 2 Creating a pipeline Parameter
Description
Data Space
Data space to which the pipeline belongs, which is generated by the system by default.
Pipeline Name
Name of the pipeline. The name must meet the following requirements:
- The name can contain 5 to 63 characters.
- Only letters, numbers, and hyphens (-) are allowed. The name cannot start or end with a hyphen (-) or contain consecutive hyphens (-).
- The name must be unique in the data space.
Example: euleros_log
Shards
The number of shards of the pipeline. Value range: 1 to 64. Retain the default value.
An index can store data that exceeds the hardware limit of one node. To solve this problem, Elasticsearch subdivides your index into multiple pieces called shards. When creating an index, you can specify the number of shards as required. Each shard is hosted on any node in the cluster, and each shard is an independent and fully functional index.
Lifecycle
Lifecycle of data in the pipeline. Value range: 7 to 180.
Retain the default value.
Description
Remarks on the pipeline. This parameter is optional.
- Click OK. After the pipeline is created, you can click the data space name to check the created pipeline.
You are advised to use different pipelines to store data from different sources so that you can query and analyze the data.
- Go to the target workspace management page. In the navigation pane on the left, choose
- Configure the connector. You need to configure log source and log destination parameters.
- Go to the workspace management page. In the navigation pane on the left, choose Figure 5 Accessing the Connections page
.
- Add a source for the data connection.
- On the Connections tab, click Add.
- Configure the data connection source details.
Figure 6 Source
The following uses UDP as an example to describe how to add a log data source. For details about other connection types, see Connector Rules.
Table 3 Log source settings Parameter
Description
Connection Method
Select Source.
Connection Type
Select UDP.
Title
Name of the data connection source. In this example, in-euleros-log is used.
Description
Enter a brief description of the data source. In this example, "EulerOS logs" are used.
Port
Set the port for the data connection. In this example, 5157 is used.
Codec
Set the encoding format. You are advised to set it to Plain.
Advanced Settings
No configuration is required.
- After the setting is complete, click Confirm in the lower right corner of the page.
- Add a destination for the data connection.
- On the Connections tab, click Add.
- Configure the data connection destination details.
Figure 7 Destination
Table 4 Log transfer destination Parameter
Description
Connection Method
Select Destination.
Connection Type
Select SecMaster.
Title
Enter a custom name for the data connection destination. In this example, enter "out-pipe-euleros-log".
Description
Enter a custom description of the data connection destination. In this example, "EulerOS logs" are used.
Type
User-defined log destination type. Select Tenant.
Pipe
Select the pipe created in 8. In this example, select euleros_log.
Domain_name
Enter the domain account information of the IAM user used to log in to the console.
User_name
Enter the user information of the IAM user used to log in to the console.
User_password
Enter the password of the current login IAM user.
Advanced Settings
No configuration is required.
- After the setting is complete, click Confirm in the lower right corner of the page.
- Go to the workspace management page. In the navigation pane on the left, choose
- Add a parser created from the parser template euleros system log parsing.
- Go to the target workspace management page. In the navigation pane on the left, choose Parsers tab. . On the displayed page, click the
Figure 8 Accessing the Parsers tab- On the Parsers tab, click the Templates tab. In the Operation column of the parser template named Analysis of euleros system log, click Create by Template. The Create Parser page is displayed.
- On the Create Parser page, configure parser parameters.
- Name: Retain the default value Analysis of euleros system log.
- Description: Retain the default value. The default value is Analysis of euleros system log.
- Parsing rule: Retain the default value.
- Click OK. Return to the page to check the created parser.
Figure 9 Checking the created parser - Configure log collection channels, connect each functional component, and ensure that SecMaster and log collectors work properly.
- Go to the workspace management page. In the navigation pane, choose Collection Channels tab.
Figure 10 Accessing the Collection Channels tab
. On the Collections page, click the
- Add a log collection channel group.
- On the Collection Channels tab, click
on the right of Groups.
- Enter a group name, for example, EulerOS logs, and click
.
- On the Collection Channels tab, click
- Create a log collection channel.
- On the right of the group list, click Add.
- In the Configure Basic Information step, configure basic information.
Table 5 Basic information parameters Parameter
Description
Basic Information
Title
Enter a custom collection channel name. In this example, "EulerOS logs" are used.
Channel grouping
Select the group created in 11.b. In this example, "EulerOS logs" are used.
(Optional) Description
Enter the description of the collection channel.
Configure Source
Source Name
Select the log source name added in 9. In this example, select in-euleros-log.
After you select a source, the system automatically generates the information about the selected source.
Destination Configuration
Destination Name
Select the log destination name added in 9. In this example, select out-pipe-euleros-log.
After you select a destination, the system automatically generates the information about the selected destination.
- Click Next in the lower right corner of the page.
- On the Configure Parser page, select Analysis of euleros system log and add raw logs to the collection channel list. After the parser is configured, click Next in the lower right corner.
- On the Select Node page, click Create. In the displayed dialog box, select the ECS node managed in 6 and click OK.
Figure 11 Selecting a node
- Click Next in the lower right corner of the page.
- On the Preview Channel Details page, confirm the configuration and click Save and Execute.
- On the Collection Channels tab, if the health status of a collection channel is Normal, the collection channel is successfully delivered.
Figure 12 Collection channels configured
- Go to the workspace management page. In the navigation pane, choose Collection Channels tab.
- On the tenant side, configure ECS log forwarding. The Rsyslog service is used to forward logs to SecMaster.
- Remotely log in to your ECS for log collection.
- Select the target log type, run the data access command, monitor all ECS logs through Rsyslog, and specify the IP address and port of the SecMaster collection node to which logs are sent.
The supported log types of EulerOS are as follows. You can configure them based on your needs.
Table 6 EulerOS log types Log Type
Log Content
kern
Kernel information.
user
Information generated by user programs.
mail
Email system information.
daemon
Information generated by daemon processes.
auth
PAM authentication information.
syslog
Log system information.
authpriv
SSH and FTP login information.
cron
Information generated when the system executes scheduled tasks.
lpr
Printing related information.
mark
Internal service information, which is a time identifier.
uucp
Unix-to-Unix Copy (UUCP) for communication between two Unix systems
- Example 1: If all logs need to be integrated, run the following command:
echo "*.*@IP-address-of-the-SecMaster-collection-node:Port" >> /etc/rsyslog.conf systemctl restart rsyslog
IP-address-of-the-SecMaster-collection-node is the IP address of the ECS node added in 6, and Port is the port set in 9.b.ii.
Example:
echo "*.*@192.168.0.0:5157" >> /etc/rsyslog.conf systemctl restart rsyslog
Figure 13 Example 1 - Example 2: If you want to access auth logs, run the following command:
After logs are integrated into the SecMaster pipeline, you can query logs on SecMaster. systemctl restart rsyslog
IP-address-of-the-SecMaster-collection-node is the IP address of the ECS node added in 6, and Port is the port set in 9.b.ii.
Example:
echo "auth.*@192.168.0.0:5157" >> /etc/rsyslog.conf systemctl restart rsyslog
- After logs are integrated into the SecMaster pipeline, you can query logs on SecMaster.
- Go to the target workspace management page. In the navigation pane on the left, choose 8 to view the data after log parsing. . Select the pipeline (euleros_log selected in this example) created or used in
- For details about how to query and analyze logs, see Querying and Analyzing Logs.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot