Help Center/ SecMaster/ API Reference/ API/ Pipelines/ Creating a Data Pipeline
Updated on 2024-07-25 GMT+08:00

Creating a Data Pipeline

Function

create pipe

Calling Method

For details, see Calling APIs.

URI

POST /v1/{project_id}/workspaces/{workspace_id}/siem/pipes

Table 1 Path Parameters

Parameter

Mandatory

Type

Description

project_id

Yes

String

Project ID.

Minimum: 32

Maximum: 36

workspace_id

Yes

String

Workspace ID

Minimum: 32

Maximum: 36

Request Parameters

Table 2 Request header parameters

Parameter

Mandatory

Type

Description

X-Auth-Token

Yes

String

User token. It can be obtained by calling the IAM API used to obtain a user token. The value of X-Subject-Token in the response header is a token.

Minimum: 0

Maximum: 2097152

Table 3 Request body parameters

Parameter

Mandatory

Type

Description

dataspace_id

Yes

String

Workspace ID

Minimum: 36

Maximum: 36

pipe_name

Yes

String

Data pipeline name.

Minimum: 5

Maximum: 63

description

No

String

Description.

Minimum: 1

Maximum: 255

storage_period

Yes

Integer

Data storage duration, in days. The default value is 30. The value ranges from 1 to 3600.

Minimum: 1

Maximum: 3600

shards

Yes

Integer

Number of pipeline partitions. One partition is created by default. A maximum of 64 partitions can be created.

Minimum: 1

Maximum: 64

timestamp_field

No

String

Timestamp field.

Default: __time

Minimum: 1

Maximum: 256

mapping

No

Map<String,KeyIndex>

Index field mapping. Each key object carries information about a field. There are multiple key objects. The key is variable and indicates the field name. Nesting is supported.

Table 4 KeyIndex

Parameter

Mandatory

Type

Description

type

No

String

Field type. The options are text (full-text index field), keyword (structured field), Long, Integer, Double, Float (time field), and Date (time field).

Enumeration values:

  • text

  • keyword

  • long

  • integer

  • double

  • float

  • date

is_chinese_exist

No

Boolean

Whether Chinese characters are contained.

properties

No

Map<String,KeyIndex>

Nested structure.

Response Parameters

Status code: 200

Table 5 Response body parameters

Parameter

Type

Description

domain_id

String

Account ID.

Minimum: 32

Maximum: 36

project_id

String

Project ID.

Minimum: 32

Maximum: 36

dataspace_id

String

Data space ID

Minimum: 32

Maximum: 36

dataspace_name

String

Data space name

Minimum: 32

Maximum: 36

pipe_id

String

Indicates the pipe ID.

Minimum: 32

Maximum: 36

pipe_name

String

Pipeline Name

Minimum: 32

Maximum: 36

pipe_type

String

Pipeline type. System-defined Preset types. User-defined Custom types.

Minimum: 5

Maximum: 128

description

String

Description.

Minimum: 5

Maximum: 128

storage_period

Integer

Index storage period by the day.

Minimum: 1

Maximum: 100000

shards

Integer

Index shard quantity.

Minimum: 1

Maximum: 128

create_by

String

Created By

Minimum: 5

Maximum: 128

create_time

Integer

Creation time

Minimum: 0

Maximum: 1010000000

update_by

String

Updated by

Minimum: 5

Maximum: 128

update_time

Integer

Update time.

Minimum: 0

Maximum: 10000000000

Status code: 400

Table 6 Response body parameters

Parameter

Type

Description

error_msg

String

Invalid request message.

Minimum: 1

Maximum: 128

error_code

String

Error code

Minimum: 1

Maximum: 128

Status code: 401

Table 7 Response body parameters

Parameter

Type

Description

error_msg

String

Permissions error.

Minimum: 1

Maximum: 128

error_code

String

Error code

Minimum: 1

Maximum: 128

Status code: 500

Table 8 Response body parameters

Parameter

Type

Description

error_msg

String

Internal system error.

Minimum: 1

Maximum: 128

error_code

String

Error code

Minimum: 1

Maximum: 128

Example Requests

{
  "dataspace_id" : "a00106ba-bede-453c-8488-b60c70bd6aed",
  "pipe_name" : "pipe-01",
  "description" : "test pipe",
  "storage_period" : 30,
  "shards" : 3,
  "mapping" : {
    "name" : {
      "type" : "text"
    },
    "id" : {
      "type" : "text"
    },
    "publish_time" : {
      "type" : "data"
    }
  }
}

Example Responses

Status code: 200

Created pipeline returned.

{
  "domain_id" : "0531ed520xxxxxxebedb6e57xxxxxxxx",
  "project_id" : "2b31ed520xxxxxxebedb6e57xxxxxxxx",
  "dataspace_id" : "a00106ba-bede-453c-8488-b60c70bd6aed",
  "dataspace_name" : "dataspace-01",
  "pipe_id" : "b22106ba-bede-453c-8488-b60c70bd6aed",
  "pipe_name" : "pipe-01",
  "pipe_type" : "system-defined",
  "description" : "test pipe",
  "storage_period" : 30,
  "shards" : 3,
  "create_by" : "0642ed520xxxxxxebedb6e57xxxxxxxx",
  "create_time" : 1584883694354,
  "update_by" : "0642ed520xxxxxxebedb6e57xxxxxxxx",
  "update_time" : 1584883694354
}

SDK Sample Code

The SDK sample code is as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
package com.huaweicloud.sdk.test;

import com.huaweicloud.sdk.core.auth.ICredential;
import com.huaweicloud.sdk.core.auth.BasicCredentials;
import com.huaweicloud.sdk.core.exception.ConnectionException;
import com.huaweicloud.sdk.core.exception.RequestTimeoutException;
import com.huaweicloud.sdk.core.exception.ServiceResponseException;
import com.huaweicloud.sdk.secmaster.v2.region.SecMasterRegion;
import com.huaweicloud.sdk.secmaster.v2.*;
import com.huaweicloud.sdk.secmaster.v2.model.*;


public class CreatePipeSolution {

    public static void main(String[] args) {
        // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
        // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
        String ak = System.getenv("CLOUD_SDK_AK");
        String sk = System.getenv("CLOUD_SDK_SK");

        ICredential auth = new BasicCredentials()
                .withAk(ak)
                .withSk(sk);

        SecMasterClient client = SecMasterClient.newBuilder()
                .withCredential(auth)
                .withRegion(SecMasterRegion.valueOf("<YOUR REGION>"))
                .build();
        CreatePipeRequest request = new CreatePipeRequest();
        CreatePipeRequestBody body = new CreatePipeRequestBody();
        request.withBody(body);
        try {
            CreatePipeResponse response = client.createPipe(request);
            System.out.println(response.toString());
        } catch (ConnectionException e) {
            e.printStackTrace();
        } catch (RequestTimeoutException e) {
            e.printStackTrace();
        } catch (ServiceResponseException e) {
            e.printStackTrace();
            System.out.println(e.getHttpStatusCode());
            System.out.println(e.getRequestId());
            System.out.println(e.getErrorCode());
            System.out.println(e.getErrorMsg());
        }
    }
}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# coding: utf-8

import os
from huaweicloudsdkcore.auth.credentials import BasicCredentials
from huaweicloudsdksecmaster.v2.region.secmaster_region import SecMasterRegion
from huaweicloudsdkcore.exceptions import exceptions
from huaweicloudsdksecmaster.v2 import *

if __name__ == "__main__":
    # The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    # In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak = os.environ["CLOUD_SDK_AK"]
    sk = os.environ["CLOUD_SDK_SK"]

    credentials = BasicCredentials(ak, sk)

    client = SecMasterClient.new_builder() \
        .with_credentials(credentials) \
        .with_region(SecMasterRegion.value_of("<YOUR REGION>")) \
        .build()

    try:
        request = CreatePipeRequest()
        request.body = CreatePipeRequestBody(
        )
        response = client.create_pipe(request)
        print(response)
    except exceptions.ClientRequestException as e:
        print(e.status_code)
        print(e.request_id)
        print(e.error_code)
        print(e.error_msg)
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
package main

import (
	"fmt"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/core/auth/basic"
    secmaster "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/secmaster/v2"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/services/secmaster/v2/model"
    region "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/secmaster/v2/region"
)

func main() {
    // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak := os.Getenv("CLOUD_SDK_AK")
    sk := os.Getenv("CLOUD_SDK_SK")

    auth := basic.NewCredentialsBuilder().
        WithAk(ak).
        WithSk(sk).
        Build()

    client := secmaster.NewSecMasterClient(
        secmaster.SecMasterClientBuilder().
            WithRegion(region.ValueOf("<YOUR REGION>")).
            WithCredential(auth).
            Build())

    request := &model.CreatePipeRequest{}
	request.Body = &model.CreatePipeRequestBody{
	}
	response, err := client.CreatePipe(request)
	if err == nil {
        fmt.Printf("%+v\n", response)
    } else {
        fmt.Println(err)
    }
}

For SDK sample code of more programming languages, see the Sample Code tab in API Explorer. SDK sample code can be automatically generated.

Status Codes

Status Code

Description

200

Created pipeline returned.

400

Request error.

401

Authentication failed.

403

Access denied.

500

Internal system error.

Error Codes

See Error Codes.