Help Center/SecMaster/API Reference/API/Pipelines/Creating a Data Pipeline
Updated on 2025-03-20 GMT+08:00
View PDF

Creating a Data Pipeline

Function

create pipe

Calling Method

For details, see Calling APIs.

URI

POST /v1/{project_id}/workspaces/{workspace_id}/siem/pipes

Table 1 Path Parameters

Parameter

Mandatory

Type

Description

project_id

Yes

String

Project ID.

workspace_id

Yes

String

Workspace ID

Request Parameters

Table 2 Request header parameters

Parameter

Mandatory

Type

Description

X-Auth-Token

Yes

String

User token.

It can be obtained by calling the IAM API used to obtain a user token. The value of X-Subject-Token in the response header is a token.

Table 3 Request body parameters

Parameter

Mandatory

Type

Description

dataspace_id

Yes

String

Workspace ID

pipe_name

Yes

String

Data pipeline name.

description

No

String

Description.

storage_period

Yes

Integer

Data storage duration, in days. The default value is 30. The value ranges from 1 to 3600.

shards

Yes

Integer

Number of pipeline partitions. One partition is created by default. A maximum of 64 partitions can be created.

timestamp_field

No

String

Timestamp field.

mapping

No

Map<String,KeyIndex>

Index field mapping. Each key object carries information about a field. There are multiple key objects. The key is variable and indicates the field name. Nesting is supported.

Table 4 KeyIndex

Parameter

Mandatory

Type

Description

type

No

String

Field type. The options are text (full-text index field), keyword (structured field), Long, Integer, Double, Float (time field), and Date (time field).

is_chinese_exist

No

Boolean

Whether Chinese characters are contained.

properties

No

Map<String,KeyIndex>

Nested structure.

Response Parameters

Status code: 200

Table 5 Response body parameters

Parameter

Type

Description

domain_id

String

Account ID.

project_id

String

Project ID.

dataspace_id

String

Data space ID

dataspace_name

String

Data space name

pipe_id

String

Indicates the pipe ID.

pipe_name

String

Pipeline Name

pipe_type

String

Pipeline type. System-defined Preset types. User-defined Custom types.

description

String

Description.

storage_period

Integer

Index storage period by the day.

shards

Integer

Index shard quantity.

create_by

String

Created By

create_time

Integer

Creation time

update_by

String

Updated by

update_time

Integer

Update time.

Status code: 400

Table 6 Response body parameters

Parameter

Type

Description

error_msg

String

Invalid request message.

error_code

String

Error code

Status code: 401

Table 7 Response body parameters

Parameter

Type

Description

error_msg

String

Permissions error.

error_code

String

Error code

Status code: 403

Table 8 Response body parameters

Parameter

Type

Description

-

String

Status code: 500

Table 9 Response body parameters

Parameter

Type

Description

error_msg

String

Internal system error.

error_code

String

Error code

Example Requests

{
  "dataspace_id" : "a00106ba-bede-453c-8488-b60c70bd6aed",
  "pipe_name" : "pipe-01",
  "description" : "test pipe",
  "storage_period" : 30,
  "shards" : 3,
  "mapping" : {
    "name" : {
      "type" : "text"
    },
    "id" : {
      "type" : "text"
    },
    "publish_time" : {
      "type" : "data"
    }
  }
}

Example Responses

Status code: 200

Created pipeline returned.

{
  "domain_id" : "0531ed520xxxxxxebedb6e57xxxxxxxx",
  "project_id" : "2b31ed520xxxxxxebedb6e57xxxxxxxx",
  "dataspace_id" : "a00106ba-bede-453c-8488-b60c70bd6aed",
  "dataspace_name" : "dataspace-01",
  "pipe_id" : "b22106ba-bede-453c-8488-b60c70bd6aed",
  "pipe_name" : "pipe-01",
  "pipe_type" : "system-defined",
  "description" : "test pipe",
  "storage_period" : 30,
  "shards" : 3,
  "create_by" : "0642ed520xxxxxxebedb6e57xxxxxxxx",
  "create_time" : 1584883694354,
  "update_by" : "0642ed520xxxxxxebedb6e57xxxxxxxx",
  "update_time" : 1584883694354
}

Status Codes

Status Code

Description

200

Created pipeline returned.

400

Request error.

401

Authentication failed.

403

Access denied.

500

Internal system error.

Error Codes

See Error Codes.

Parent Topic: Pipelines