Updated on 2025-06-24 GMT+08:00

Creating a Data Pipeline

Function

This API is used to create a data pipeline.

Calling Method

For details, see Calling APIs.

URI

POST /v1/{project_id}/workspaces/{workspace_id}/siem/pipes

Table 1 Path Parameters

Parameter

Mandatory

Type

Description

project_id

Yes

String

Project ID.

workspace_id

Yes

String

Workspace ID.

Request Parameters

Table 2 Request header parameters

Parameter

Mandatory

Type

Description

X-Auth-Token

Yes

String

User token.

It can be obtained by calling the IAM API used to obtain a user token. The value of X-Subject-Token in the response header is the user token.

Table 3 Request body parameters

Parameter

Mandatory

Type

Description

dataspace_id

Yes

String

Workspace ID.

pipe_name

Yes

String

Data pipeline name.

description

No

String

Description.

storage_period

Yes

Integer

Data storage duration, in days. The default value is 30. The value ranges from 1 to 3,600.

shards

Yes

Integer

Number of pipeline partitions. One partition is created by default. A maximum of 64 partitions can be created.

timestamp_field

No

String

Timestamp.

mapping

No

Map<String,KeyIndex>

Index field mapping. Each key object carries information about a field. There are multiple key objects. The key is variable and indicates the field name. Nesting is supported.

Table 4 KeyIndex

Parameter

Mandatory

Type

Description

type

No

String

Field type. The options are text (full-text index field), keyword (structured field), Long, Integer, Double, Float (time field), and Date (time field).

is_chinese_exist

No

Boolean

Whether non-English characters are contained.

properties

No

Map<String,KeyIndex>

Nested structure.

Response Parameters

Status code: 200

Table 5 Response body parameters

Parameter

Type

Description

domain_id

String

User domain ID.

project_id

String

Project ID.

dataspace_id

String

Data space ID.

dataspace_name

String

Data space name.

pipe_id

String

Pipeline ID.

pipe_name

String

Pipeline name.

pipe_type

String

Pipeline type. system-defined: Predefined by the system; user-defined: Custom pipeline.

description

String

Description.

storage_period

Integer

Index storage duration, measured in days.

shards

Integer

Number of index shards.

create_by

String

Creator.

create_time

Integer

Creation time.

update_by

String

Updater.

update_time

Integer

Update time.

Status code: 400

Table 6 Response body parameters

Parameter

Type

Description

error_msg

String

Invalid request message.

error_code

String

Error code.

Status code: 401

Table 7 Response body parameters

Parameter

Type

Description

error_msg

String

Insufficient permission.

error_code

String

Error code.

Status code: 403

Table 8 Response body parameters

Parameter

Type

Description

error_msg

String

Invalid request message.

error_code

String

Error code.

Status code: 500

Table 9 Response body parameters

Parameter

Type

Description

error_msg

String

Internal system error.

error_code

String

Error code.

Example Requests

{
  "dataspace_id" : "a00106ba-bede-453c-8488-b60c70bd6aed",
  "pipe_name" : "pipe-01",
  "description" : "test pipe",
  "storage_period" : 30,
  "shards" : 3,
  "mapping" : {
    "name" : {
      "type" : "text"
    },
    "id" : {
      "type" : "text"
    },
    "publish_time" : {
      "type" : "data"
    }
  }
}

Example Responses

Status code: 200

Response value to a successful creation.

{
  "domain_id" : "0531ed520xxxxxxebedb6e57xxxxxxxx",
  "project_id" : "2b31ed520xxxxxxebedb6e57xxxxxxxx",
  "dataspace_id" : "a00106ba-bede-453c-8488-b60c70bd6aed",
  "dataspace_name" : "dataspace-01",
  "pipe_id" : "b22106ba-bede-453c-8488-b60c70bd6aed",
  "pipe_name" : "pipe-01",
  "pipe_type" : "system-defined",
  "description" : "test pipe",
  "storage_period" : 30,
  "shards" : 3,
  "create_by" : "0642ed520xxxxxxebedb6e57xxxxxxxx",
  "create_time" : 1584883694354,
  "update_by" : "0642ed520xxxxxxebedb6e57xxxxxxxx",
  "update_time" : 1584883694354
}

SDK Sample Code

The SDK sample code is as follows.

Java

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
package com.huaweicloud.sdk.test;

import com.huaweicloud.sdk.core.auth.ICredential;
import com.huaweicloud.sdk.core.auth.BasicCredentials;
import com.huaweicloud.sdk.core.exception.ConnectionException;
import com.huaweicloud.sdk.core.exception.RequestTimeoutException;
import com.huaweicloud.sdk.core.exception.ServiceResponseException;
import com.huaweicloud.sdk.secmaster.v2.region.SecMasterRegion;
import com.huaweicloud.sdk.secmaster.v2.*;
import com.huaweicloud.sdk.secmaster.v2.model.*;


public class CreatePipeSolution {

    public static void main(String[] args) {
        // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
        // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
        String ak = System.getenv("CLOUD_SDK_AK");
        String sk = System.getenv("CLOUD_SDK_SK");
        String projectId = "{project_id}";

        ICredential auth = new BasicCredentials()
                .withProjectId(projectId)
                .withAk(ak)
                .withSk(sk);

        SecMasterClient client = SecMasterClient.newBuilder()
                .withCredential(auth)
                .withRegion(SecMasterRegion.valueOf("<YOUR REGION>"))
                .build();
        CreatePipeRequest request = new CreatePipeRequest();
        request.withWorkspaceId("{workspace_id}");
        CreatePipeRequestBody body = new CreatePipeRequestBody();
        request.withBody(body);
        try {
            CreatePipeResponse response = client.createPipe(request);
            System.out.println(response.toString());
        } catch (ConnectionException e) {
            e.printStackTrace();
        } catch (RequestTimeoutException e) {
            e.printStackTrace();
        } catch (ServiceResponseException e) {
            e.printStackTrace();
            System.out.println(e.getHttpStatusCode());
            System.out.println(e.getRequestId());
            System.out.println(e.getErrorCode());
            System.out.println(e.getErrorMsg());
        }
    }
}

Python

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# coding: utf-8

import os
from huaweicloudsdkcore.auth.credentials import BasicCredentials
from huaweicloudsdksecmaster.v2.region.secmaster_region import SecMasterRegion
from huaweicloudsdkcore.exceptions import exceptions
from huaweicloudsdksecmaster.v2 import *

if __name__ == "__main__":
    # The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    # In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak = os.environ["CLOUD_SDK_AK"]
    sk = os.environ["CLOUD_SDK_SK"]
    projectId = "{project_id}"

    credentials = BasicCredentials(ak, sk, projectId)

    client = SecMasterClient.new_builder() \
        .with_credentials(credentials) \
        .with_region(SecMasterRegion.value_of("<YOUR REGION>")) \
        .build()

    try:
        request = CreatePipeRequest()
        request.workspace_id = "{workspace_id}"
        request.body = CreatePipeRequestBody(
        )
        response = client.create_pipe(request)
        print(response)
    except exceptions.ClientRequestException as e:
        print(e.status_code)
        print(e.request_id)
        print(e.error_code)
        print(e.error_msg)

Go

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
package main

import (
	"fmt"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/core/auth/basic"
    secmaster "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/secmaster/v2"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/services/secmaster/v2/model"
    region "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/secmaster/v2/region"
)

func main() {
    // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak := os.Getenv("CLOUD_SDK_AK")
    sk := os.Getenv("CLOUD_SDK_SK")
    projectId := "{project_id}"

    auth := basic.NewCredentialsBuilder().
        WithAk(ak).
        WithSk(sk).
        WithProjectId(projectId).
        Build()

    client := secmaster.NewSecMasterClient(
        secmaster.SecMasterClientBuilder().
            WithRegion(region.ValueOf("<YOUR REGION>")).
            WithCredential(auth).
            Build())

    request := &model.CreatePipeRequest{}
	request.WorkspaceId = "{workspace_id}"
	request.Body = &model.CreatePipeRequestBody{
	}
	response, err := client.CreatePipe(request)
	if err == nil {
        fmt.Printf("%+v\n", response)
    } else {
        fmt.Println(err)
    }
}

More

For SDK sample code of more programming languages, see the Sample Code tab in API Explorer. SDK sample code can be automatically generated.

Status Codes

Status Code

Description

200

Response value to a successful creation.

400

Request error.

401

Authentication failed.

403

Access denied.

500

Internal system error.

Error Codes

See Error Codes.