Configuring Synchronization Policies in Batches

Function

This API is used to configure synchronization policies in batches, including conflict policies, DROP Database filtering, and object synchronization scope.
This API is used to configure Kafka synchronization policies.

Debugging

You can debug the API in API Explorer to support automatic authentication. API Explorer can automatically generate and debug example SDK code.

Constraints

This API can be called only after a task is created, the task status is CONFIGURATION, the test of connections to the source and destination databases is successful, and the API for modifying the task is successfully called.
Kafka synchronization policies can be configured for the following data flow scenarios: synchronization from PostgreSQL to Kafka, from Oracle to Kafka, from GaussDB to Kafka, from TaurusDB to Kafka, and from MySQL to Kafka.
TaurusDB-to-Kafka synchronization and MySQL-to-Kafka synchronization allow you to modify the Kafka policy configuration when the task is in the INCRE_TRANSFER_STARTED state. After the configuration is modified, you can edit the synchronization objects only when the task status changes to INCRE_TRANSFER_STARTED.

Authorization Information

Each account has all the permissions required to call all APIs, but IAM users must be assigned the required permissions. For details about the required permissions, see Permissions and Supported Actions.

URI

POST /v3/{project_id}/jobs/batch-sync-policy

**Table 1** URI parameters
Parameter	Mandatory	Type	Description
project_id	Yes	String	Definition Project ID of a tenant in a region. For details about how to obtain the project ID, see Obtaining a Project ID. Constraints N/A Range N/A Default Value N/A

Request Parameters

**Table 2** Request header parameters
Parameter	Mandatory	Type	Description
Content-Type	Yes	String	Definition MIME type of the request body. Use the default value application/json. For APIs used to upload objects or images, the value varies depending on the flow type. Constraints N/A Range application/json Default Value application/json
X-Auth-Token	Yes	String	Definition User token obtained from IAM. It is a response to the API for obtaining a user token. This API is the only one that does not require authentication. The token is the value of X-Subject-Token in the response header. Constraints N/A Range N/A Default Value N/A
X-Language	No	String	Definition Request language type. Constraints N/A Range en-us: English zh-cn: Chinese Default Value en-us

**Table 3** Request body parameters
Parameter	Mandatory	Type	Description
jobs	Yes	Array of objects	Definition Request list for configuring synchronization policies in batches, which includes information about the synchronization tasks for which policies need to be configured. For details, see Table 4.

**Table 4** Data structure description of field **jobs**
Parameter	Mandatory	Type	Description
job_id	Yes	String	Definition Synchronization task ID, which uniquely identifies a data replication task. When configuring synchronization policies in batches, you need to specify the tasks for which policies need to be configured. Constraints N/A Range N/A Default Value N/A
conflict_policy	No	String	Definition Conflict policy during data migration. This parameter specifies how to handle conflicts when the destination database already contains the same data. Constraints N/A Range ignore: Conflicting data is skipped and the migration continues. overwrite: The existing data in the destination database is overwritten by the source database data. stop: The task stops when a conflict occurs. Default Value N/A
filter_ddl_policy	No	String	Definition DDL filtering policy, which specifies the type of DDL operations to be filtered out during incremental synchronization. Constraints N/A Range drop_database: The DROP DATABASE operation is filtered out. Default Value N/A
ddl_trans	No	Boolean	Definition Whether to synchronize DDL operations during incremental synchronization. This parameter specifies whether to synchronize DDL operations from the source database to the destination database during incremental synchronization. Constraints N/A Range true: DDL operations are synchronized during incremental synchronization. false: DDL operations are not synchronized during incremental synchronization. Default Value N/A
index_trans	No	Boolean	Definition Whether to synchronize indexes during incremental synchronization. This parameter specifies whether to synchronize index changes from the source database to the destination database during incremental synchronization. Constraints N/A Range true: Indexes are synchronized during incremental synchronization. false: Indexes are not synchronized during incremental synchronization. Default Value N/A
topic_policy	No	String	Definition Topic synchronization policy, which specifies the topic generation mode of the destination Kafka database. Constraints This parameter is mandatory when the destination database is Kafka. The value range varies depending on the engine type. For details, see the value range description. Range Values for synchronization from GaussDB Distributed to Kafka: 0: A specified topic 1: Automatically generated using the database_name-schema_name-table_name format 2: Automatically generated based on the database name 3: Automatically generated using the database_name-schema_name format 4: Automatically generated using the database_name-DN_sequence_number format Values for synchronization from GaussDB Centralized to Kafka and from PostgreSQL to Kafka: 0: A specified topic 1: Automatically generated using the database_name-schema_name-table_name format 2: Automatically generated based on the database name 3: Automatically generated using the database_name-schema_name format Values for synchronization from Oracle to Kafka: 0: A specified topic 1: Automatically generated using the schema_name-table_name format 3: Automatically generated based on the schema name Values for synchronization from MySQL to Kafka: 0: A specified topic 1: Auto-generated topics Values for synchronization from TaurusDB to Kafka: 0: A specified topic 1: Auto-generated topics Default Value N/A
topic	No	String	Definition Kafka topic name, which specifies the destination topic to which data is synchronized. Constraints This parameter is mandatory when topic_policy is set to 0. Ensure that the topic already exists. Range N/A Default Value N/A
partition_policy	No	String	Definition Policy for synchronization to Kafka partitions, which specifies how data is synchronized to Kafka partitions. Constraints This parameter is mandatory when the destination database is Kafka. The available values of partition_policy are restricted by topic_policy. When topic_policy is set to 0, the value can be 0, 1, 2, 3, 4, or 5. When topic_policy is set to 1, the value can be 0, 1, 2, or 5. When topic_policy is set to 2, partition_policy can only be set to 0, 1, 2, or 3. For GaussDB Centralized as the source, partition_policy can be set to 0, 1, 2, or 3. For GaussDB Distributed as the source, partition_policy can be set to 0, 1, or 3. When topic_policy is set to 3, the value can be 0 or 1. When topic_policy is set to 4, the value can be 0, 1, or 3. Based on topic_policy, the available values of partition_policy are as follows in different scenarios: GaussDB Distributed -> Kafka: The value of partition_policy can be 0, 1, 2, 3, 4, or 5 based on topic_policy. GaussDB Centralized -> Kafka: The value of partition_policy can be 0, 1, 2, 3, or 5 based on topic_policy. TaurusDB -> Kafka: The value of partition_policy can be 0 or 2 based on topic_policy. MySQL -> Kafka: The value of partition_policy can be 0, 1, or 2 based on topic_policy. Oracle -> Kafka: The value of partition_policy can be 0, 1, 2, or 3 based on topic_policy. Range 0: Partitions are differentiated by the hash values of database_name.schema_name.table_name. 1: Topics are synchronized to partition 0. 2: Partitions are identified by the hash values of the primary key. 3: Partitions are differentiated by the hash values of database_name.schema_name. 4: Partitions are differentiated by the hash values of database_name.DN_sequence_number. (This value is only available for synchronization from GaussDB Distributed to Kafka.) 5: Partitions are differentiated by the hash values of non-primary-key columns. Default Value N/A
kafka_data_format	No	String	Definition Data format in Kafka, which specifies the serialization mode of data synchronized to Kafka. Constraints Incremental synchronization from MySQL to Kafka supports Avro, JSON, and JSON-C. Synchronization from TaurusDB to Kafka supports JSON and JSON-C. Other engines support JSON and Avro. Range json: JSON message format, which is easy to interpret but takes up more space. Avro: a binary encoded format that is efficient. json_c: a data format that is compatible with multiple batch and stream computing frameworks. Default Value json
topic_name_format	No	String	Definition Topic name format, which specifies the naming template used for automatically generating topic names. Constraints This parameter is mandatory when topic_policy is set to 1, 2, or 3. Values for synchronization from PostgreSQL to Kafka and synchronization from GaussDB Centralized to Kafka: If topic_policy is set to 1, the topic name supports the database and schema names as variables. Other characters are considered as constants. $database$ indicates the database name, and $schema$ indicates the schema name. The default value is $database$-$schema$. If topic_policy is set to 2, the topic name supports the database name as a variable. Other characters are regarded as constants. If this parameter is left blank, the default value is $database$. If topic_policy is set to 3, the topic name supports the names of database, schema, and table as variables. Other characters are considered as constants. $database$ indicates the database name, $schema$ indicates the schema name, and $tablename$ indicates the table name. The default value is $database$-$schema$-$tablename$. Values for synchronization from Oracle to Kafka: If topic_policy is set to 1, the topic name supports the schema and table names as variables. Other characters are considered as constants. Replace $schema$ with the schema name and $tablename$ with the table name. If this parameter is left blank, the default value is $schema$-$tablename$. If topic_policy is set to 3, the topic name supports the schema name as variables. Other characters are considered as constants. Replace $schema$ with the schema name. If this parameter is left blank, the default value is $schema$. Values for synchronization from MySQL to Kafka and TaurusDB to Kafka: If topic_policy is set to 1, the topic name supports the database and table names as variables. Other characters are considered as constants. Replace $database$ with the database name and $tablename$ with the table name. If this parameter is left blank, the default value is $database$-$tablename$. Range N/A Default Value N/A
partitions_num	No	String	Definition Number of partitions in the Kafka topic. Constraints This parameter is mandatory when topic_policy is set to 1, 2, or 3. Range 1–2147483647 Default Value 1
replication_factor	No	String	Definition Number of replicas in the Kafka topic. Constraints This parameter is mandatory when topic_policy is set to 1, 2, or 3. Range 1-32767 Default Value 1
is_fill_materialized_view	No	Boolean	Definition Whether to populate materialized views in the PostgreSQL full synchronization phase. This parameter specifies whether to populate the data of the materialized views in the source database to the destination database during full synchronization. Constraints N/A Range true: The materialized views are populated. false: The materialized views are not populated. Default Value false
export_snapshot	No	Boolean	Definition Whether to use the snapshot mode to export data from the source database in the PostgreSQL full synchronization phase. Constraints N/A Range true: The snapshot mode is used for data export. false: The snapshot mode is not used for data export. Default Value false
slot_name	No	String	Definition Replication slot name, which identifies the replication slot for PostgreSQL logical replication. Constraints This parameter is mandatory for primary/standby tasks from GaussDB Centralized to Kafka. This parameter is optional for an incremental synchronization task with PostgreSQL serving as the source database. Range N/A Default Value N/A
file_and_position	No	String	Definition Source database log position, which specifies the start point of incremental synchronization. Constraints When MySQL serves as the source database, run show master status to obtain the start point of the source database and set File and Position as prompted. For example, mysql-bin.000277:805, in which the file name can contain only 1 to 60 characters and cannot contain the following special characters <>&:"'/\\, the file number can contain only 3 to 20 digits, the binlog event position can contain only 1 to 20 digits, and the total length cannot exceed 100 characters. The value is in the format of File_name.file_number:Event_position. When MongoDB serves as the source database, the source database logs are obtained from within the time range of the oplog, starting with the current start position. To check whether the start position is within the oplog time range, run db.getReplicationInfo() for a non-cluster instance, and for a cluster instance, run db.watch([], {startAtOperationTime: Timestamp(xx, xx)}), where xx is the start position you specified. The value is in the format of timestamp:incre. The values of timestamp and incre are integers ranging from 1 to 2,147,483,647. Range N/A Default Value N/A
gtid_set	No	String	Definition MySQL GTID set, which specifies the start point of incremental synchronization. Constraints This parameter is mandatory for tasks whose source database is MySQL. Run show master status to obtain the start point of the source database and enter the value of Executed_Gtid_Set as prompted. If the source database version is MySQL 5.5, synchronization tasks are not supported. Enter a maximum of 2048 characters. Chinese characters and the following special characters are not allowed: < > & " ' / \\ Range N/A Default Value N/A
ddl_topic	No	String	Definition Topic for storing DDLs. This parameter specifies the destination topic to which DDL operation data is synchronized. Constraints This parameter is mandatory when Kafka is the destination database and ddl_trans is set to true. The value must be the name of an existing topic in the destination database. Ensure that the topic already exists. Range N/A Default Value N/A

Response Parameters

Status code: 200

**Table 5** Response body parameters
Parameter	Type	Description
count	Integer	Definition Total number of returned task records, which is consistent with the length of the results array. Constraints N/A Range N/A
results	Array of objects	Definition List of returned synchronization policies that are configured in batches. For details, see Table 6.

**Table 6** Data structure description of field **results**
Parameter	Type	Description
id	String	Definition ID of the task for configuring the synchronization policy, which is used to identify the task corresponding to the policy configuration result. Constraints N/A Range N/A
status	String	Definition Policy configuration result status, which indicates whether the synchronization policy is successfully configured. Constraints N/A Range success: The task is successful. failed: The task fails.
error_code	String	Definition Error code returned if the task fails. Constraints N/A Range The value is in the format of DRS.XXXXXX.
error_msg	String	Definition Error message returned if the task fails. Constraints N/A Range N/A

Example Request

Configuring synchronization task policies in batches, in which conflict_policy is set to ignore, ddl_trans is set to true, and filter_ddl_policy is set to drop_database

https://{endpoint}/v3/054ba152d480d55b2f5dc0069e7ddef0/jobs/batch-sync-policy

{
    "jobs": [{
	"conflict_policy": "ignore",
	"ddl_trans": true,
	"filter_ddl_policy": "drop_database",
	"index_trans": true,
	"job_id": "19557d51-1ee6-4507-97a6-8f69164jb201"
    }]
}

Configuring MySQL incremental synchronization task policies in batches:

https://{endpoint}/v3/054ba152d480d55b2f5dc0069e7ddef0/jobs/batch-sync-policy 
  
 { 
   "jobs": [ 
     { 
       "conflict_policy": "ignore", 
       "ddl_trans": true, 
       "filter_ddl_policy": "drop_database", 
       "index_trans": true, 
       "job_id": "19557d51-1ee6-4507-97a6-8f69164jb201",
       "file_and_position": "mysql-bin.000019:197", 
       "gtid_set":"e4979f26-4bc3-11ee-b279-fa163ef21d64:1-23" 
     } 
   ] 
 }

Example Response

Status code: 200

{
  "results" : [ {
    "id" : "19557d51-1ee6-4507-97a6-8f69164jb201",
    "status" : "success"
  } ],
  "count" : 1
}

Status Code

Status Code	Description
200	OK
400	Bad Request

For other statuses, see Status Code.

Error Code

For details, see Error Code.

Parent topic: Real-Time Synchronization Management

Previous topic: Processing Data in Batches

Next topic: Advanced Settings