Data Access with a Custom Parser

Prerequisites

You have obtained the IAM account and its password for logging in to the console.

Step 1: Buy an ECS

For details, see Purchasing an ECS.

Currently, the data collection agent can run only on Linux ECSs on x86_64 architecture. ECSs support the following OSs: Huawei Cloud EulerOS 2.5, Huawei Cloud EulerOS 2.9, EulerOS 2.5, EulerOS 2.9, and CentOS 7.9.

Note that you need to select the proper OSs and versions when you make a purchase.

Figure 1 Selecting an OS version

Step 2: Install an Agent

The agent is a client software that maintains the communication between SecMaster and an ECS. It can deliver commands and report heartbeat data.

1. Pre-check before installing an agent.
   1. Run the ps -ef | grep salt command to check whether the salt-minion process exists on the host.
      • If yes, stop it first.
      • If no, go to 1.b.
        Figure 2 Checking processes
        
   2. Before installing Logstash, run the df -h command to check whether there are at least 50 GB of disk space reserved for the root directory disk or opt disk, two CPU cores, and 4 GB of memory.
      Figure 3 Disks
      

      If the memory is insufficient, stop some applications with high memory usage or expand the memory capacity before the installation. For details about capacity expansion, see Modifying ECS Specifications.

2. Log in to the management console.
3. Click in the upper left corner of the page and choose Security & Compliance > SecMaster.
4. In the navigation pane, choose Workspaces > Management. In the workspace list, click the name of the target workspace.
   Figure 4 Workspace management page
   
5. In the navigation tree on the left, choose Settings > Components.
   Figure 5 Accessing the node management page
   
6. On the Node Management tab page, click Create.
7. On the Create Node page, set parameters.
   Figure 6 Create Node
   
   1. In the Network Channel Configuration area, select the VPC and subnet the network channel belongs to.
   2. In the network channel list, locate the row that contains the target channel and click Config in the Operation column. In the displayed confirmation dialog box, click OK.
8. Click Next in the lower right corner of the page. On the page for verifying the script installation, click to copy the command for installing the Agent.
9. Remotely log in to the ECS where you want to install the agent.
   • Huawei Cloud servers
     • Log in to the ECS console, locate the target server, and click Remote Login in the Operation column to log in to the server. For details, see Login Using VNC.
     • If your server has an EIP bound, you can also use a remote management tool, such as PuTTY or Xshell, to log in to the server and install the agent on the server as user root.
   • Non-Huawei Cloud servers

     Use a remote management tool (such as PuTTY or Xshell) to connect to the EIP of your server and remotely log in to your server.

10. Run the cd /opt/cloud command to go to the installation directory.
    

    The recommended installation path is /opt/cloud. This section also uses this path as an example. If you want to install the Agent in another path, change the path based on site requirements.

11. Run the command copied in 8 as user root to install the Agent on the ECS.
12. Enter the IAM username and password for logging in to the console when prompted.
13. If information similar to the following is displayed, the agent is successfully installed:

    install isap-agent successfully

Step 3: Create a Node

1. In the navigation tree on the left, choose Settings > Components.
   Figure 7 Accessing the node management page
   
2. On the Node Management tab page, click Create.
3. On the Create Node page, set parameters.
   Figure 8 Create Node
   
   1. In the Network Channel Configuration area, select the VPC and subnet the network channel belongs to.
   2. In the network channel list, locate the row that contains the target channel and click Config in the Operation column. In the displayed confirmation dialog box, click OK.
4. Click Next in the lower right corner of the page to go to the Script Installation Verification page.
5. After confirming that the installation is complete, click Confirm in the lower right corner of the page.

Step 4: Configure Components

Logstash is an open-source data collection engine that provides the real-time pipeline function. Logstash can dynamically collect data from different sources, convert the data, and output the data to different destinations.

1. In the navigation pane on the left, choose Settings > Components and click the Components tab.
   Figure 9 Accessing the Components tab page
   
2. On the Components tab page, click Edit Configuration in the upper right corner of the component to be viewed. The configuration management page of the component is displayed on the right.
3. In the Node Configuration area, click Add in the upper left corner of the node list. In the Add Node dialog box displayed, select a node and click OK.
4. Click Save and Apply in the lower right corner of the page.

Step 5: (Optional) Create a Pipeline

You need to add a pipeline for storing incoming data. For details, see Creating a Pipeline.

Step 6: Create a Data Connection Source and Destination

Create a data connection, including the data source and the data destination where the parsed data is transferred to.

1. In the navigation pane on the left, choose Settings > Collection Management.
   Figure 10 Collection Management
   
2. Add a data connection source.
   1. On the Connection management page, click Add.
   2. On the Source tab page, select User data protocol UDP input as the source of the data source type and set UDP parameters.
      Figure 11 Data source parameters
      

      Table 1 Data source parameters

      Parameter	Description
      Title	Name of the data connection source.
      Description	A brief description of the custom data connection source.
      Port	Set the port over which you want to collect the data.
      codec	Set the encoding format. You can select json or plain.
      Optional Parameters	Customize other optional parameters.

   3. After the setting is complete, click Confirm in the lower right corner of the page.
3. Add a data connection destination.
   1. On the Collection Management page, click the Connection management tab. On the displayed page, click Add.
   2. Click the Destination tab. Then, select Yunnao pipeline output for the data source type and configure the pipeline information.
      Figure 12 Data source access destination
      

      Table 2 Data source access destination parameters

      Parameter	Description
      Title	Name of the data source destination.
      Description	A brief description of the data connection destination.
      type	Select tenant.
      pipe	Select the name of the pipeline created in Step 5: (Optional) Create a Pipeline.
      domain_name	Enter the account that creates the IAM user.
      User_name	Enter the IAM username.
      Password	Enter the password of the IAM user.
      Optional Parameters	Customize other optional parameters.

   3. After the setting is complete, click Confirm in the lower right corner of the page.

Step 7: Configure a Parser

1. In the navigation pane on the left, choose Settings > Collection Management > Parser Management tab.
   Figure 13 Accessing the parser management page
   
2. On the Parser Management page, click Add. On the displayed page, set parameters and add a collection channel.
   • Name: Set a parser name.
   • (Optional) Description: Enter the parser description.
   • Rule list: Set parsing rules for the parser. Click Add and select a rule type.
     • Conditional control: Select the if condition to check whether the log exists.
     • Parsing rules: Select json to remove the original field (message).
     Figure 14 Rule list
     
3. Click OK in the lower right corner of the page.

Step 8: Add a Collection Channel

A collection channel connects the input, parsing, and output to form a pipeline and delivers the pipeline to collection nodes where the agent and Logstash are installed. In doing this, the data access and transfer process can then start.

1. In the navigation pane on the left, choose Settings > Collection Management. On the Collection Management page, click the Collection Channels tab.
   Figure 15 Collection channel management tab page
   
2. Add a channel group.
   1. On the collection channel management page, click on the right of the Group list.
   2. Enter a group name and click .
3. On the right of the group list, click Add.
4. On the Basic Configuration page, configure basic information.

   Table 3 Basic configuration parameters

   Parameter		Description
   Basic Information	Title	The collection channel name you customize.
   	Channel grouping	Select the group created in 2.
   	Description	(Optional) Enter the description of the collection channel.
   Source Configuration	Source Name	Select the source created in Step 6: Create a Data Connection Source and Destination.
   Destination	Destination Name	Select the name of the data destination created in Step 6: Create a Data Connection Source and Destination.

5. After the basic configuration is complete, click Next in the lower right corner of the page.
6. On the parser configuration page, select the parser configured in Step 7: Configure a Parser.
7. After the parser is configured, click Next in the lower right corner of the page.
8. On the Select Node page, click Add. In the Add Node dialog box displayed, select a node that has the agent and Logstash installed and click Confirm.
9. After the node is selected, click Next in the lower right corner of the page.
10. On the Channel Details Preview page, confirm the configuration and click OK.

After the collection channel is added, the pipeline will be delivered. Refresh the page. If the health status is Normal, the delivery is complete.

Step 9: Query and Analyze

As logs are transferred to SecMaster, you can query logs in SecMaster after data access completes.

1. In the navigation pane on the left, choose Threat Operations > Security Analysis.
2. Select the SecMaster pipeline added in Step 5: (Optional) Create a Pipeline. Then, you can view the parsed log data on SecMaster.
   Figure 16 Security Analysis