Quick Data Access with the Default Parser in SecMaster

Prerequisite

You have obtained the IAM account and its password for logging in to the console.

Step 1: Buy an ECS

For details, see Purchasing an ECS.

Currently, the data collection agent can run only on EulerOS Linux servers on x86_64 architecture. When purchasing an ECS, select an OS version SecMaster supports by referring to Supported OSs.

Figure 1 Selecting an OS version

Step 2: Create a Node and Install the Agent on the Node

The agent is a client software that maintains the communication between SecMaster and an ECS. It can deliver commands and report heartbeat data.

For details about how to install an agent on an ECS, see Installing the Agent.

For details about how to add a node, see Creating a Node.

Step 3: Configure Components

Logstash is an open-source data collection engine that provides the real-time pipeline function. Logstash can dynamically collect data from different sources, convert the data, and output the data to different destinations.

For details about how to configure components, see Configuring Components.

Step 4: (Optional) Create a Pipeline

You need to add a pipeline for storing incoming data. For details, see Creating a Pipeline.

Step 5: Create a Data Connection Source and Destination

Create a data connection, including the data source and the data destination where the parsed data is transferred to.

1. Go to the SecMaster console.
2. In the navigation pane, choose Workspace > Management. In the workspace list, click the name of the target workspace.
3. In the navigation pane on the left, choose Settings > Collection Management.
   Figure 2 Collection Management
   
4. Add a data connection source.
   1. On the Connection management page, click Add.
   2. On the Source tab page, select User data protocol UDP input as the source of the data source type and set UDP parameters.
      Figure 3 Data source
      

      Table 1 Data source parameters

      Parameter	Description
      Title	Name of the data connection source.
      Description	A brief description of the data connection source.
      Port	Set the port over which you want to collect the data.
      codec	Set the encoding format. You can select json or plain.
      Optional Parameters	Customize other optional parameters.

   3. After the setting is complete, click Confirm in the lower right corner of the page.
5. Add a data connection destination.
   1. On the Collection Management page, click the Connection management tab. On the displayed page, click Add.
   2. Click the Destination tab. Then, select Yunnao pipeline output for the data source type and configure the pipeline information.
      Figure 4 Data source access destination
      

      Table 2 Data source access destination parameters

      Parameter	Description
      Title	Name of the data source destination.
      Description	A brief description of the data connection destination.
      type	Select tenant.
      pipe	Select the name of the pipeline created in Step 4: (Optional) Create a Pipeline.
      domain_name	Enter the account that creates the IAM user.
      User_name	Enter the IAM username.
      Password	Enter the password of the IAM user.
      Optional Parameters	Customize other optional parameters.

   3. After the setting is complete, click Confirm in the lower right corner of the page.

Step 6: Add a Collection Channel

A collection channel connects the input, parsing, and output to form a pipeline and delivers the pipeline to collection nodes where the agent and Logstash are installed. In doing this, the data access and transfer process can then start.

1. Go to the SecMaster console.
2. In the navigation pane, choose Workspace > Management. In the workspace list, click the name of the target workspace.
3. In the navigation pane on the left, choose Settings > Collection Management. On the Collection Management page, click the Collection Channels tab.
   Figure 5 The collection channel management page is displayed.
   
4. Add a channel group.
   1. On the collection channel management page, click on the right of the Group list.
   2. Enter a group name and click .
5. On the right of the group list, click Add.
6. On the Basic Configuration page, configure basic information.

   Table 3 Basic configuration parameters

   Parameter		Description
   Basic Information	Title	The collection channel name you customize.
   	Channel grouping	Select the group created in 4.
   	Description	(Optional) Enter the description of the collection channel.
   Source Configuration	Source Name	Select the source created in Step 5: Create a Data Connection Source and Destination.
   Destination Configuration	Destination Name	Select the destination created in Step 5: Create a Data Connection Source and Destination.

7. After the basic configuration is complete, click Next in the lower right corner of the page.
8. On the Parser Configuration page, select Fast access.

   In quick access mode, all raw logs are stored in the message field.

9. After the parser is configured, click Next in the lower right corner of the page.
10. On the Select Node page, click Add. In the Add Node dialog box displayed, select a node that has the agent and Logstash installed and click Confirm.
11. After the node is selected, click Next in the lower right corner of the page.
12. On the Channel Details Preview page, confirm the configuration and click OK.

After the collection channel is added, the pipeline will be delivered. Refresh the page. If the health status is Normal, the delivery is complete.

Step 7: Query and Analyze

As logs are transferred to SecMaster, you can query logs in SecMaster after data access completes.

1. Go to the SecMaster console.
2. In the navigation pane, choose Workspace > Management. In the workspace list, click the name of the target workspace.
3. In the navigation pane on the left, choose Threat Operations > Security Analysis.
4. Select the SecMaster pipeline added in Step 4: (Optional) Create a Pipeline. Then, you can view the parsed log data on SecMaster.
   Figure 6 Analyze & Query