Quick Data Access with the Default Parser in SecMaster
This chapter walks you through how to collect ECS logs in UDP mode, how to parse collected logs using the default parser configured for collectors, and how to send the parsed data to a SecMaster pipeline. After data access, you can query the information on the Security Analysis page.
Prerequisite
You have obtained the IAM account and its password for logging in to the console.
Step 1: Buy an ECS
For details, see Purchasing an ECS.
![](https://support.huaweicloud.com/eu/bestpractice-secmaster/public_sys-resources/caution_3.0-en-us.png)
Currently, the data collection agent can run only on Linux ECSs on x86_64 architecture. ECSs support the following OSs: Huawei Cloud EulerOS 2.5, Huawei Cloud EulerOS 2.9, EulerOS 2.5, EulerOS 2.9, and CentOS 7.9.
Note that you need to select the proper OSs and versions when you make a purchase.
![](https://support.huaweicloud.com/eu/bestpractice-secmaster/en-us_image_0000001704433034.png)
Step 2: Install an Agent
The agent is a client software that maintains the communication between SecMaster and an ECS. It can deliver commands and report heartbeat data.
- Pre-check before installing an agent.
- Run the ps -ef | grep salt command to check whether the salt-minion process exists on the host.
- If yes, stop it first.
- If no, go to 1.b.
Figure 2 Checking processes
- Before installing Logstash, run the df -h command to check whether there are at least 50 GB of disk space reserved for the root directory disk or opt disk, two CPU cores, and 4 GB of memory.
Figure 3 Disks
If the memory is insufficient, stop some applications with high memory usage or expand the memory capacity before the installation. For details about capacity expansion, see Modifying ECS Specifications.
- Run the ps -ef | grep salt command to check whether the salt-minion process exists on the host.
- Log in to the management console.
- Click
in the upper left corner of the page and choose Security & Compliance > SecMaster.
- In the navigation pane, choose Workspaces > Management. In the workspace list, click the name of the target workspace.
Figure 4 Workspace management page
- In the navigation tree on the left, choose Figure 5 Accessing the node management page
.
- On the Node Management tab page, click Create.
- On the Create Node page, set parameters.
Figure 6 Create Node
- In the Network Channel Configuration area, select the VPC and subnet the network channel belongs to.
- In the network channel list, locate the row that contains the target channel and click Config in the Operation column. In the displayed confirmation dialog box, click OK.
- Click Next in the lower right corner of the page. On the page for verifying the script installation, click
to copy the command for installing the Agent.
- Remotely log in to the ECS where you want to install the agent.
- Huawei Cloud servers
- Log in to the ECS console, locate the target server, and click Remote Login in the Operation column to log in to the server. For details, see Login Using VNC.
- If your server has an EIP bound, you can also use a remote management tool, such as PuTTY or Xshell, to log in to the server and install the agent on the server as user root.
- Non-Huawei Cloud servers
Use a remote management tool (such as PuTTY or Xshell) to connect to the EIP of your server and remotely log in to your server.
- Huawei Cloud servers
- Run the cd /opt/cloud command to go to the installation directory.
The recommended installation path is /opt/cloud. This section also uses this path as an example. If you want to install the Agent in another path, change the path based on site requirements.
- Run the command copied in 8 as user root to install the Agent on the ECS.
- Enter the IAM username and password for logging in to the console when prompted.
- If information similar to the following is displayed, the agent is successfully installed:
install isap-agent successfully
Step 3: Create a Node
- In the navigation tree on the left, choose Figure 7 Accessing the node management page
.
- On the Node Management tab page, click Create.
- On the Create Node page, set parameters.
Figure 8 Create Node
- In the Network Channel Configuration area, select the VPC and subnet the network channel belongs to.
- In the network channel list, locate the row that contains the target channel and click Config in the Operation column. In the displayed confirmation dialog box, click OK.
- Click Next in the lower right corner of the page to go to the Script Installation Verification page.
- After confirming that the installation is complete, click Confirm in the lower right corner of the page.
Step 4: Configure Components
Logstash is an open-source data collection engine that provides the real-time pipeline function. Logstash can dynamically collect data from different sources, convert the data, and output the data to different destinations.
- In the navigation pane on the left, choose Settings > Components and click the Components tab.
Figure 9 Accessing the Components tab page
- On the Components tab page, click Edit Configuration in the upper right corner of the component to be viewed. The configuration management page of the component is displayed on the right.
- In the Node Configuration area, click Add in the upper left corner of the node list. In the Add Node dialog box displayed, select a node and click OK.
- Click Save and Apply in the lower right corner of the page.
Step 5: (Optional) Create a Pipeline
You need to add a pipeline for storing incoming data. For details, see Creating a Pipeline.
Step 6: Create a Data Connection Source and Destination
Create a data connection, including the data source and the data destination where the parsed data is transferred to.
- In the navigation pane on the left, choose Figure 10 Collection Management
.
- Add a data connection source.
- On the Connection management page, click Add.
- On the Source tab page, select User data protocol UDP input as the source of the data source type and set UDP parameters.
Figure 11 Data source
Table 1 Data source parameters Parameter
Description
Title
Name of the data connection source.
Description
A brief description of the data connection source.
Port
Set the port over which you want to collect the data.
codec
Set the encoding format. You can select json or plain.
Optional Parameters
Customize other optional parameters.
- After the setting is complete, click Confirm in the lower right corner of the page.
- Add a data connection destination.
- On the Collection Management page, click the Connection management tab. On the displayed page, click Add.
- Click the Destination tab. Then, select Yunnao pipeline output for the data source type and configure the pipeline information.
Figure 12 Data source access destination
Table 2 Data source access destination parameters Parameter
Description
Title
Name of the data source destination.
Description
A brief description of the data connection destination.
type
Select tenant.
pipe
Select the name of the pipeline created in Step 5: (Optional) Create a Pipeline.
domain_name
Enter the account that creates the IAM user.
User_name
Enter the IAM username.
Password
Enter the password of the IAM user.
Optional Parameters
Customize other optional parameters.
- After the setting is complete, click Confirm in the lower right corner of the page.
Step 7: Add a Collection Channel
A collection channel connects the input, parsing, and output to form a pipeline and delivers the pipeline to collection nodes where the agent and Logstash are installed. In doing this, the data access and transfer process can then start.
- In the navigation pane on the left, choose Collection Channels tab.
Figure 13 Collection channel management tab page
. On the Collection Management page, click the
- Add a channel group.
- On the collection channel management page, click
on the right of the Group list.
- Enter a group name and click
.
- On the collection channel management page, click
- On the right of the group list, click Add.
- On the Basic Configuration page, configure basic information.
Table 3 Basic configuration parameters Parameter
Description
Basic Information
Title
The collection channel name you customize.
Channel grouping
Select the group created in 2.
Description
(Optional) Enter the description of the collection channel.
Source Configuration
Source Name
Select the source created in Step 6: Create a Data Connection Source and Destination.
Destination Configuration
Destination Name
Select the destination created in Step 6: Create a Data Connection Source and Destination.
- After the basic configuration is complete, click Next in the lower right corner of the page.
- On the Parser Configuration page, select Fast access.
In quick access mode, all raw logs are stored in the message field.
- After the parser is configured, click Next in the lower right corner of the page.
- On the Select Node page, click Add. In the Add Node dialog box displayed, select a node that has the agent and Logstash installed and click Confirm.
- After the node is selected, click Next in the lower right corner of the page.
- On the Channel Details Preview page, confirm the configuration and click OK.
After the collection channel is added, the pipeline will be delivered. Refresh the page. If the health status is Normal, the delivery is complete.
Step 8: Query and Analyze
As logs are transferred to SecMaster, you can query logs in SecMaster after data access completes.
- In the navigation pane on the left, choose Threat Operations > Security Analysis.
- Select the SecMaster pipeline added in Step 5: (Optional) Create a Pipeline. Then, you can view the parsed log data on SecMaster.
Figure 14 Analyze & Query
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.