Updated on 2024-12-30 GMT+08:00

Managing Pipelines

A data transfer message topic and a storage index form a pipeline. This topic describes how to manage data pipelines. You can:

  • Creating a Pipeline: If you need to use security analysis, data analysis, and intelligent modeling features in SecMaster, you need to create pipelines.
  • Viewing Pipeline Details: You can view the pipeline details, including the pipeline name, data space, and creation time.
  • Editing a Pipeline: You can modify the pipeline information, such as the number of shards, description, and lifecycle.
  • Deleting a Pipeline: You can delete a pipeline. Data in the pipeline will also be deleted and cannot be restored. Exercise caution when performing this operation.

Limitations and Constraints

  • A maximum of 20 pipelines can be created in a data space in a region for an account.
  • Pipelines created by the system cannot be edited.
  • Pipelines created by the system cannot be deleted.

Creating a Pipeline

  1. Log in to the management console.
  2. Click in the upper left corner of the management console and select a region or project.
  3. Click in the upper left corner of the page and choose Security & Compliance > SecMaster.
  4. In the navigation pane on the left, choose Workspaces > Management. In the workspace list, click the name of the target workspace.

    Figure 1 Workspace management page

  5. In the navigation pane on the left, choose Threat Operations > Security Analysis. The security analysis page is displayed.

    Figure 2 Accessing the Security Analysis tab page

  6. (Optional) Add a data space. For details, see Adding a Data Space.
  7. In the data space navigation tree on the left, click on the right of the data space name and select Create Pipeline.

    Figure 3 Create Pipeline

  8. On the Create Pipeline page, configure pipeline parameters. For details about the parameters, see Table 1.

    Table 1 Parameters for creating a pipeline

    Parameter

    Description

    Data Space

    Data space the pipeline belongs to. This parameter is generated by the system by default.

    Pipeline Name

    Name of the pipeline. The naming rules are as follows:

    • The name can contain 5 to 63 characters.
    • The name can contain letters, numbers, and hyphens (-). The name cannot start or end with a hyphen (-) or contain consecutive hyphens (-).
    • The name must be unique in the data space.

    Shards

    The number of shards of the pipeline. The value ranges from 1 to 64.

    An index can store a large amount of data that exceeds the hardware limits of a node. To solve this problem, Elasticsearch subdivides your index into multiple pieces called shards. When creating an index, you can specify the number of shards as required. Each shard is in itself a fully-functional and independent "index" that can be hosted on any node in the cluster.

    Lifecycle

    Lifecycle of data in the pipeline. Value range: 7 to 180

    Description

    Remarks on the pipeline. This parameter is optional.

  9. Click OK.

    You can click the data space name or next to the data space to view the created pipeline.

Viewing Pipeline Details

  1. Log in to the management console.
  2. Click in the upper left corner of the management console and select a region or project.
  3. Click in the upper left corner of the page and choose Security & Compliance > SecMaster.
  4. In the navigation pane on the left, choose Workspaces > Management. In the workspace list, click the name of the target workspace.

    Figure 4 Workspace management page

  5. In the navigation pane on the left, choose Threat Operations > Security Analysis. The security analysis page is displayed.

    Figure 5 Accessing the Security Analysis tab page

  6. In the data space navigation tree on the left, click a data space name to show the pipeline list.

    Figure 6 Viewing pipeline details

  7. Click next to a pipeline name you want to view. The Pipeline Details pane is displayed on the right of the page.

    Table 2 Pipeline parameters

    Parameter

    Description

    Workspace Name

    Name of the workspace to which the pipeline belongs.

    Workspace ID

    ID of the workspace to which the pipeline belongs.

    Data Space Name

    Name of the data space to which the pipeline belongs.

    Data Space ID

    ID of the data space to which the pipeline belongs.

    Pipeline Name

    Name of the pipeline.

    Pipeline ID

    ID of the pipeline.

    Shards

    Number of shards of the pipeline.

    Lifecycle

    Retention period of the data in the pipeline.

    Created

    Time when the pipeline was created.

    Description

    Description of the pipeline.

Editing a Pipeline

  1. Log in to the management console.
  2. Click in the upper left corner of the management console and select a region or project.
  3. Click in the upper left corner of the page and choose Security & Compliance > SecMaster.
  4. In the navigation pane on the left, choose Workspaces > Management. In the workspace list, click the name of the target workspace.

    Figure 7 Workspace management page

  5. In the navigation pane on the left, choose Threat Operations > Security Analysis. The security analysis page is displayed.

    Figure 8 Accessing the Security Analysis tab page

  6. In the data space navigation tree on the left, click a data space name to show the pipeline list.

    Figure 9 Viewing pipeline details

  7. Click More > Edit next to the pipeline name.

    Figure 10 Entry for editing a pipeline

  8. On the Edit Pipeline page, configure pipeline parameters. For details about the parameters, see Table 3.

    Table 3 Parameters for editing a pipeline

    Parameter

    Description

    Data Space

    Data space to which the pipeline belongs. This parameter cannot be modified.

    Pipeline Name

    Name you specified for the pipeline. The name cannot be changed after the pipeline is created.

    Shards

    The number of shards of the pipeline. The value ranges from 1 to 64.

    Lifecycle

    Lifecycle of data in the pipeline. Value range: 7 to 180

    Description

    Remarks on the pipeline. This parameter is optional.

  9. Click OK.

Deleting a Pipeline

  1. Log in to the management console.
  2. Click in the upper left corner of the management console and select a region or project.
  3. Click in the upper left corner of the page and choose Security & Compliance > SecMaster.
  4. In the navigation pane on the left, choose Workspaces > Management. In the workspace list, click the name of the target workspace.

    Figure 11 Workspace management page

  5. In the navigation pane on the left, choose Threat Operations > Security Analysis. The security analysis page is displayed.

    Figure 12 Accessing the Security Analysis tab page

  6. In the data space navigation tree on the left, click a data space name to show the pipeline list.

    Figure 13 Viewing pipeline details

  7. Click More > Delete next to the pipeline name.

    Figure 14 Deleting a pipeline

  8. In the dialog box displayed, click OK.