Help Center/ Data Replication Service/ API Reference/ APIs V3.0 (Deprecated)/ Real-Time Synchronization Management/ Processing Data in Batches
Updated on 2024-08-23 GMT+08:00
View PDF

Processing Data in Batches

Function

This API is used to add rules for data processing.

Constraints

  • You can call the API for starting a task only after the task is created and database objects are selected. For details, see Task Creation Process.
  • Each table has only one verification rule.
  • The Oracle source database supports a maximum of 20,000 tables at a time, and the MySQL source database supports a maximum of 10,000 tables at a time.
  • The filter criteria do not support the package, function, variable, and constant that are unique to a certain database engine. You must use standardized SQL.

URI

POST /v3/{project_id}/jobs/batch-transformation

Table 1 Path parameters

Parameter

Mandatory

Type

Description

project_id

Yes

String

Project ID of a tenant in a region

For details about how to obtain the project ID, see Obtaining a Project ID.

Request Parameters

Table 2 Request header parameters

Parameter

Mandatory

Type

Description

Content-Type

Yes

String

The content type.

The default value is application/json.

X-Auth-Token

Yes

String

User token obtained from IAM.

X-Language

No

String

Request language type

Default value: en-us

Values:

  • en-us
  • zh-cn
Table 3 Request body parameters

Parameter

Mandatory

Type

Description

jobs

Yes

Array of objects

Requests for adding data processing rules in batches.

For details, see Table 4.
Table 4 Data structure description of field jobs

Parameter

Mandatory

Type

Description

job_id

No

String

Task ID.

object_info

No

Array of objects

Object information. This parameter is mandatory when a processing rule is generated.

For details, see Table 5.

transformation_info

Yes

Object

Processing information.

For details, see Table 6.

config_transformation

No

Object
  • Configuration information. If there are multiple associated tables, generate multiple configuration rules. The data that meets the configuration conditions is temporarily stored in the cache and used in the data filtering scenario.
  • The database name and table name can contain digits, letters, and underscores (_).
  • Ensure that the column names, primary keys, and indexes are the same as those in the source database.

For details, see Table 7.
Table 5 Data structure description of field object_info

Parameter

Mandatory

Type

Description

id

No

String

Database name and database table name. For example, the format is lxl_test1-*-*-test_1, where lxl_test1 is the database name and test_1 is the table name.

select

No

String

Whether to select advanced configuration. The value is true.
Table 6 Data structure description of field transformation_info

Parameter

Mandatory

Type

Description

transformation_type

Yes

String
  • The processing rule value is contentConditionalFilter.
  • The configuration rule value is configConditionalFilter.

    Values:

    • contentConditionalFilter
    • configConditionalFilter

value

Yes

String

Filter criteria. The processing rule value is a SQL statement, and the configuration rule value is config. The value contains a maximum of 256 characters.
Table 7 Data structure description of field config_transformation

Parameter

Mandatory

Type

Description

db_table_name

Yes

String

Database-name.Table-name, for example, lxl_test1.test_1, where lxl_test1 is the database name and test_1 is the table name.

db_name

Yes

String

Database name. The value contains a maximum of 256 characters.

table_name

Yes

String

Table name The value contains a maximum of 256 characters.

col_names

Yes

String

Column name The value contains a maximum of 256 characters.

prim_key_or_index

Yes

String

Primary key or unique index The value contains a maximum of 256 characters.

indexs

Yes

String

Index that requires optimization. The value contains a maximum of 256 characters.

values

Yes

String

Filtering criteria. The value contains a maximum of 256 characters.

Response Parameters

Status code: 200

Table 8 Response body parameters

Parameter

Type

Description

results

Array of objects

Batch data processing response list.

For details, see Table 9.

count

integer

Total number.
Table 9 Data structure description of field results

Parameter

Type

Description

id

String

Task ID.

status

String

Status Values:

  • success
  • failed

error_code

String

Error code.

error_msg

String

Error message.

Example Request

  • Generating configuration rules for MySQL synchronization task data 
    https://{endpoint}/v3/054ba152d480d55b2f5dc0069e7ddef0/jobs/batch-transformation

{
  "jobs" : [ {
    "job_id" : "e894593d-5e0a-4652-af7e-1b0c239jb201",
    "object_info" : [ ],
    "transformation_info" : {
      "transformation_type" : "configConditionalFilter",
      "value" : "config"
    },
    "config_transformation" : {
      "col_names" : "id,name",
      "db_name" : "lxl_test1",
      "db_table_name" : "lxl_test1.test_1",
      "indexs" : "name",
      "prim_key_or_index" : "id",
      "table_name" : "test_1",
      "values" : "name like '%a%'"
    }
  } ]
}
  • Generating processing rules for MySQL synchronization task data, in which the table name is lxl_test1.test_1, and the filtering condition is id>5. 
    https://{endpoint}/v3/054ba152d480d55b2f5dc0069e7ddef0/jobs/batch-transformation

{
  "jobs" : [ {
    "job_id" : "e894593d-5e0a-4652-af7e-1b0c239jb201",
    "object_info" : [ {
      "id" : "lxl_test1-*-*-test_1",
      "select" : "true"
    } ],
    "transformation_info" : {
      "transformation_type" : "contentConditionalFilter",
      "value" : "id>5"
    }
  } ]
}

Example Response

Status code: 200

OK

{
  "count" : 1,
  "results" : [ {
    "id" : "e894593d-5e0a-4652-af7e-1b0c239jb201",
    "status" : "success"
  } ]
}

Status Code

Status Code

Description

200

OK

400

Bad Request

Error Code

For details, see Error Code.