Creating a Data Pipeline

Updated on 2024-12-25 GMT+08:00

View PDF

Function

create pipe

Calling Method

For details, see Calling APIs.

URI

POST /v1/{project_id}/workspaces/{workspace_id}/siem/pipes

**Table 1** Path Parameters
Parameter	Mandatory	Type	Description
project_id	Yes	String	Project ID.
workspace_id	Yes	String	Workspace ID

Request Parameters

**Table 2** Request header parameters
Parameter	Mandatory	Type	Description
X-Auth-Token	Yes	String	User token. It can be obtained by calling the IAM API used to obtain a user token. The value of X-Subject-Token in the response header is a token.

**Table 3** Request body parameters
Parameter	Mandatory	Type	Description
dataspace_id	Yes	String	Workspace ID
pipe_name	Yes	String	Data pipeline name.
description	No	String	Description.
storage_period	Yes	Integer	Data storage duration, in days. The default value is 30. The value ranges from 1 to 3600.
shards	Yes	Integer	Number of pipeline partitions. One partition is created by default. A maximum of 64 partitions can be created.
timestamp_field	No	String	Timestamp field.
mapping	No	Map<String,KeyIndex>	Index field mapping. Each key object carries information about a field. There are multiple key objects. The key is variable and indicates the field name. Nesting is supported.

**Table 4** KeyIndex
Parameter	Mandatory	Type	Description
type	No	String	Field type. The options are text (full-text index field), keyword (structured field), Long, Integer, Double, Float (time field), and Date (time field).
is_chinese_exist	No	Boolean	Whether Chinese characters are contained.
properties	No	Map<String,KeyIndex>	Nested structure.

Response Parameters

Status code: 200

**Table 5** Response body parameters
Parameter	Type	Description
domain_id	String	Account ID.
project_id	String	Project ID.
dataspace_id	String	Data space ID
dataspace_name	String	Data space name
pipe_id	String	Indicates the pipe ID.
pipe_name	String	Pipeline Name
pipe_type	String	Pipeline type. System-defined Preset types. User-defined Custom types.
description	String	Description.
storage_period	Integer	Index storage period by the day.
shards	Integer	Index shard quantity.
create_by	String	Created By
create_time	Integer	Creation time
update_by	String	Updated by
update_time	Integer	Update time.

Status code: 400

**Table 6** Response body parameters
Parameter	Type	Description
error_msg	String	Invalid request message.
error_code	String	Error code

Status code: 401

**Table 7** Response body parameters
Parameter	Type	Description
error_msg	String	Permissions error.
error_code	String	Error code

Status code: 403

**Table 8** Response body parameters
Parameter	Type	Description
-	String

Status code: 500

**Table 9** Response body parameters
Parameter	Type	Description
error_msg	String	Internal system error.
error_code	String	Error code

Example Requests

{
  "dataspace_id" : "a00106ba-bede-453c-8488-b60c70bd6aed",
  "pipe_name" : "pipe-01",
  "description" : "test pipe",
  "storage_period" : 30,
  "shards" : 3,
  "mapping" : {
    "name" : {
      "type" : "text"
    },
    "id" : {
      "type" : "text"
    },
    "publish_time" : {
      "type" : "data"
    }
  }
}

Example Responses

Status code: 200

Created pipeline returned.

{
  "domain_id" : "0531ed520xxxxxxebedb6e57xxxxxxxx",
  "project_id" : "2b31ed520xxxxxxebedb6e57xxxxxxxx",
  "dataspace_id" : "a00106ba-bede-453c-8488-b60c70bd6aed",
  "dataspace_name" : "dataspace-01",
  "pipe_id" : "b22106ba-bede-453c-8488-b60c70bd6aed",
  "pipe_name" : "pipe-01",
  "pipe_type" : "system-defined",
  "description" : "test pipe",
  "storage_period" : 30,
  "shards" : 3,
  "create_by" : "0642ed520xxxxxxebedb6e57xxxxxxxx",
  "create_time" : 1584883694354,
  "update_by" : "0642ed520xxxxxxebedb6e57xxxxxxxx",
  "update_time" : 1584883694354
}