Help Center/ SecMaster/ Best Practices/ Log Access and Transfer Operation Guide/ Procedure/ (Optional) Step 8: Creating a Log Storage Pipeline
Updated on 2024-11-15 GMT+08:00

(Optional) Step 8: Creating a Log Storage Pipeline

This topic describes how to create a log storage location (pipeline) in SecMaster for log storage and analysis.

This step is required when you transfer security logs from non-Huawei Cloud systems to SecMaster. Skip this step if you only need to transfer Huawei Cloud logs to a third-party system or product.

Procedure

  1. Log in to the management console.
  2. Click in the upper left corner of the page and choose Security & Compliance > SecMaster.
  3. In the navigation pane on the left, choose Workspaces > Management. In the workspace list, click the name of the target workspace.

    Figure 1 Workspace management page

  4. In the navigation pane on the left, choose Threat Operations > Security Analysis. The security analysis page is displayed.

    Figure 2 Accessing the Security Analysis tab page

  5. Create a data space.

    1. In the upper left corner of the data space list, click Add. The Add Data Space page is displayed on the right.
      Figure 3 Adding a data space
    2. On the Add Data Space page, set the parameters for the new data space. For details about the parameters, see Table 1.
      Table 1 Adding a data space

      Parameter

      Description

      Data Space

      Data space name. It must meet the following requirements:

      • The name contains 5 to 63 characters.
      • The value can contain letters, numbers, and hyphens (-). The hyphen (-) cannot be used at the beginning or end, or used consecutively.
      • The name must be unique on Huawei Cloud and cannot be the same as any other data space name.

      Description

      You can make remarks on the data space. This parameter is optional.

    3. Click OK.

  6. In the data space navigation tree on the left, click on the right of the data space name created in 5 and select Create Pipeline.

    Figure 4 Creating a pipeline

  7. On the Create Pipeline page, configure pipeline parameters. For details about the parameters, see Table 2.

    Table 2 Creating a pipeline

    Parameter

    Description

    Data Space

    Data space to which the pipeline belongs, which is generated by the system by default.

    Pipeline Name

    Name of the pipeline. The name must meet the following requirements:

    • The name can contain 5 to 63 characters.
    • The value can contain letters, numbers, and hyphens (-). The name cannot start or end with a hyphen (-) or contain consecutive hyphens (-).
    • The name must be unique in the data space.

    Shards

    The number of shards of the pipeline. The value ranges from 1 to 64.

    An index can potentially store a large amount of data that exceeds the hardware limits of a single node. To solve this problem, Elasticsearch subdivides your index into multiple pieces called shards. When creating an index, you can specify the number of shards as required. Each shard is in itself a fully-functional and independent "index" that can be hosted on any node in the cluster.

    Lifecycle

    Life cycle of data in the pipeline. The value ranges from 7 to 180.

    Description

    Remarks on the pipeline. This parameter is optional.

  8. Click OK

    After the pipeline is created, you can click the data space name to view the created pipeline.