Help Center/ ROMA Connect/ User Guide/ Data Integration Guide/ Creating a Flow Data Integration Task/ Configuring Source Information

Updated on 2024-05-07 GMT+08:00

View PDF

Configuring Source Information

Overview

This section describes how to configure source information for a flow task. Based on the source information, ROMA Connect integrates data, including the data source type, data format, and data range. The source information configuration varies depending on data source types.

Data Source Types Supported by Scheduled Integration Tasks	Data Source Types Supported by Real-Time Integration Tasks
API DB2 DWS MySQL MongoDB OBS Oracle PostgreSQL SQL Server	ActiveMQ ArtemisMQ Kafka RabbitMQ

API

If Integration Mode is set to Scheduled, API can be selected as the source data source.

**Table 1** API source information
Parameter	Description
Paging	This parameter specifies whether data is returned on multiple pages when ROMA Connect sends a request to the API data source to obtain data. Multiple data records can be returned for one API request. If Paging is enabled, all data that meets the conditions is displayed on multiple pages based on the fixed number of records on each page. Each time an integration task is executed, ROMA Connect sends multiple API requests to obtain all data. That is, each API request is sent to obtain data on one page. If Paging is disabled, ROMA Connect obtains all data that meets the conditions through one API request.
Page Number Field	This parameter is mandatory only if Paging is enabled. Enter a page number field defined in the API data source, for example, pageNo. This parameter is carried when ROMA Connect sends an API request to the source to specify the number of the page from which data is to be obtained. The page number field must be configured in Params or Body of Request Parameters.
Value	This parameter is mandatory only if Paging is enabled. Value indicates whether the page number starts from 0 or 1. Set Value based on the original definition of the API.
Page Size Field	This parameter is mandatory only if Paging is enabled. Enter a page size field defined in the API data source, for example, pageSize. This parameter is carried when ROMA Connect sends an API request to the source to specify the maximum number of records on each page.
Records	This parameter is mandatory only if Paging is enabled. Set the number of records on each page based on the original definition of the API.
Maximum Number of Pages	This parameter is mandatory only if Paging is enabled. This parameter specifies the maximum number of pages that can be queried in each scheduled task, for example, 10. If the number of pages exceeds the specified value, the task is stopped. The value 0 indicates that no restriction applies.
Pagination End	This parameter is mandatory only if Paging is enabled. Select the method to stop obtaining source data in pagination mode. Empty page list: If no data record is returned, ROMA Connect stops obtaining source data. Number of records: ROMA Connect compares the calculation result between the number of requested pages and the page size with the total number of records to determine whether to stop obtaining source data.
Pagination End Field Path	This parameter is mandatory only if Paging is enabled. Enter the path of the field in an API response, which is used to determine the end of pagination. In the API response, elements in different layers are separated by periods (.). For example, if element c in the {"a":{"b":{"c":"xxx"}}} response is the pagination end field, the pagination end field path is set to a.b.c. If Pagination End is set to Empty page list, set this parameter to the root path of the list field. If Pagination End is set to Number of records, set this parameter to the path of the total number of records.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For the first scheduling, the data between the initial timestamp and the current scheduling time is collected. For subsequent scheduling, the data between the last successful collection time and the current time is collected.
Start Time Field	Mandatory when Incremental Migration is enabled. Enter the start time field originally defined in the API data source, for example, startTime. This parameter is carried when ROMA Connect sends an API request to the source, indicating that data following the specified field will be obtained. The start time field and end time field must be both entered in Params or Body of the Request Parameters.
End Time Field	Mandatory when Incremental Migration is enabled. Enter the end time field originally defined in the API data source, for example, endTime. This parameter is carried when ROMA Connect sends an API request to the source, indicating that data before the specified value will be obtained.
Time Zone	Mandatory when Incremental Migration is enabled. Select the time zone used by the API data source so that ROMA Connect can identify the data timestamps.
Timestamp Initial Value	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated. Assume that Start Time Field is startTime, End Time Field is endTime, Timestamp Initial Value is 2020-11-01 12:00:00, Compensation Period is 0, and Period Settings is Default for incremental collection. If the first scheduling time of the task is 2020-11-01 13:00:00, the data collected for the first time is that the value of startTime is greater than or equal to 2020-11-01 12:00:00 and the value of endTime is less than or equal to 2020-11-01 13:00:00. For subsequent collection, the data collected each time is that the value of startTime is greater than or equal to the time when the task is successfully executed last time and the value of endTime is less than or equal to the time. Execution time of the current task.
Reset Initial Migration Time	This parameter can be set only when you edit an FDI task. This parameter specifies whether to enable the reset of the initial migration time. Enable: The start time of each scheduling task during incremental migration is the time specified by Timestamp Initial Value. Disable: The start time of each scheduling task during incremental migration is the end time of the previous task.
Compensation Period (ms)	Mandatory when Incremental Migration is enabled. This parameter specifies the period of time (in milliseconds) which will be used to compensate for any delay in data generation at the source when ROMA Connect queries incremental data. The end time for obtaining data is the current system time minus the value you specify here. For example, if the end time of the previous incremental migration task is 15:05, the current scheduled task is triggered at 17:00, and Compensation Period (ms) is set to 100, the time range of data to be integrated in the current incremental migration task is 15:05 to (17:00 – 100 ms).
Time Format	Mandatory when Incremental Migration is enabled. Select a timestamp format, for example, yyyy-MM-dd.
Period Settings	Mandatory when Incremental Migration is enabled. This parameter specifies the mode used for setting the time range for subsequent data integration after an incremental migration task is executed for the first time. Default: Data generated between the previous scheduling and current scheduling is integrated. When ROMA Connect obtains data from the source, it uses the triggering time of the two tasks as the start time and end time, respectively. Custom: The start time and end time are determined based on the configured period rules. This mode applies to common periodic tasks, for example, tasks executed once a day, a week, or a month.
Start Time Offset (Days)	This parameter is mandatory only if Period Settings is set to Default. Set the number of days before the start time of data collection. If data generated at the source changes in real time, such as alarm data, you can collect the data by setting this parameter. Start time of data collection = Data source system time – Start time offset
Time Interval	This parameter is mandatory only if Period Settings is set to Custom. Select the time granularity. The value must be the same as the unit configured in the task schedule so that the new data can be overwritten. For example, if Unit is set to Day in a task schedule, set this parameter to Day, indicating that data is obtained once a day.
Period	This parameter is mandatory only if Period Settings is set to Custom. Select the time period for obtaining source data. For example, if the task is executed once a day, Time Interval is set to Day, and Period is set to Previous period, data of the previous day is incrementally integrated once. If Period is set to Current period, data of the current day is incrementally integrated once.
Right Periodic Boundary	This parameter is mandatory only if Period Settings is set to Custom. This parameter specifies whether the end time is included in the time range for obtaining source data. Closed interval: The end time is included. Open interval: The end time is not included.
Request Parameters	Construct the parameter definition of the API request, for example, the page number and page size fields must be carried in Params or Body. Set this parameter based on the definition of the API data source.
Response Type	Select the format that will be used for the response of an API request. The value can be JSON or XML. Ensure that the format is the same as the actual response format of the API.
Data Root Field	This parameter specifies the path of the upper-layer common fields among all metadata in the data obtained from the source in JSON or XML format. Data Root Field and Parsing Path in Metadata form a complete metadata path. For details, see Description on Metadata Parsing Path Configuration.
Metadata	This parameter specifies each underlying key-value data element that is obtained from the source in JSON or XML format and needs to be integrated to the destination. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the response. Parsing Path: path of the metadata, which does not contain the data root field. For details, see Description on Metadata Parsing Path Configuration.

Description on Metadata Parsing Path Configuration

Data in JSON or XML format does not contain arrays:
For example, in the following JSON data (similar to XML data), the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b.c, and Parsing Path of element d must be set to b.d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to c, and Parsing Path of element d must be set to d.
Data in JSON or XML format contains arrays:
For example, in the following JSON data (similar to XML data), the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b[i].c, and Parsing Path of element d must be set to b[i].d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to [i].c, and Parsing Path of element d must be set to [i].d.

The preceding data in JSON or XML format that contains arrays is used as an example. The following describes the configuration when the destination is API:

In the example of pagination configuration. pageNo and pageSize are the pagination parameters of the API and need to be added to the Request Parameters.
Figure 1 API pagination configuration example
In the example of incremental migration configuration, startTime and endTime are the time parameters of the API and need to be added to the Request Parameters.
Figure 2 API incremental migration configuration example
In the example of metadata configuration, Data Root Field is set to a.
Figure 3 API metadata configuration example

ActiveMQ

Back to Overview

If Integration Mode is set to Real-Time, ActiveMQ can be selected as the source data source.

**Table 2** ActiveMQ information at the source
Parameter	Description
Destination Type	Select the message transfer model of the ActiveMQ data source. The value can be Topic or Queue.
Destination Name	Enter the name of an existing topic or queue from which data is obtained.
Data Root Field	This parameter specifies the path of the upper-layer common fields among all metadata in the data obtained from the source in JSON format. Data Root Field and Parsing Path in Metadata form a complete metadata path. For details, see Description on Metadata Parsing Path Configuration.
Metadata	This parameter specifies each underlying key-value data element that is obtained from the source in JSON format and needs to be integrated to the destination. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the source data. Parsing Path: path of the metadata, which does not contain the data root field. For details, see Description on Metadata Parsing Path Configuration.

Description on Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b.c, and Parsing Path of element d must be set to b.d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to c, and Parsing Path of element d must be set to d.
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b[i].c, and Parsing Path of element d must be set to b[i].d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to [i].c, and Parsing Path of element d must be set to [i].d.

The preceding JSON data that does not contain arrays is used as an example. The following describes the configuration when the source is ActiveMQ:

Figure 4 ActiveMQ configuration example
Click to enlarge

ArtemisMQ

Back to Overview

If Integration Mode is set to Real-Time, you can select ArtemisMQ as the data source type at the source.

On the Create Task page, configure source information.

**Table 3** ArtemisMQ source information
Parameter	Description
Destination Type	Select the message transfer model of the ArtemisMQ data source. The value can be Topic or Queue.
Destination Name	Enter the name of an existing topic or queue from which data is obtained.
Data Root Field	This parameter specifies the path of the upper-layer common fields among all metadata in the data obtained from the source in JSON format. Data Root Field and Parsing Path in Metadata form a complete metadata path. For details, see Description on Metadata Parsing Path Configuration.
Metadata	This parameter specifies each underlying key-value data element that is obtained from the source in JSON format and needs to be integrated to the destination. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the source data. Parsing Path: path of the metadata, which does not contain the data root field. For details, see Description on Metadata Parsing Path Configuration.

Description on Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b.c, and Parsing Path of element d must be set to b.d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to c, and Parsing Path of element d must be set to d.
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b[i].c, and Parsing Path of element d must be set to b[i].d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to [i].c, and Parsing Path of element d must be set to [i].d.

The configuration when the source is ArtemisMQ is similar to that when the source is ActiveMQ. For details, see ActiveMQ configuration example.

DB2

Back to Overview

If Integration Mode is set to Scheduled, DB2 can be selected as the source data source.

**Table 4** DB2 source information
Parameter	Description
Insert SQL	This parameter specifies whether SQL statements are used to obtain source data. If Insert SQL is enabled, ROMA Connect obtains source data based on the entered SQL statements. If Insert SQL is disabled, ROMA Connect obtains source data based on the conditions configured on the GUI. If Insert SQL is enabled, you need to enter an SQL statement for querying data. The statement must be a SELECT statement and contain the WHERE condition. The INSERT, UPDATE, DELETE, and DROP statements cannot be used. Click Check SQL to check the validity of the statement. For example, if you enter the statement SELECT col01, col02 FROM table01 WHERE col02 IN('A', 'B', 'C') ORDER BY col01, ROMA Connect selects the col01 and col02 columns from table01, displays data records whose values are A, B, or C in the col02 column, and sorts the data records in sequence based on the values in the col01 column.
Table	Mandatory when Insert SQL is disabled. Select the data table from which data is to be obtained in the DB2 data source. Then, click Select Table Field and select only the column fields that you want to integrate and synchronize, such as the ID.
Field Sorting	Mandatory when Insert SQL is disabled. This parameter specifies how the data to be integrated is sorted by field. To sort the integrated data, select a reference field and select Ascending or Descending.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For the first scheduling, the data between the initial timestamp and the current scheduling time is collected. For subsequent scheduling, the data between the last successful collection time and the current time is collected.
Time Zone	Mandatory when Incremental Migration is enabled. Select the time zone used by the DB2 data source so that ROMA Connect can identify the data timestamps.
Timestamp Field	This parameter is mandatory only if Incremental Migration is enabled and Insert SQL is disabled. Select a field of the DATE type as the timestamp, which can be used to check whether a data row meets incremental integration conditions. If the entered values of Timestamp field and Timestamp Initial Value are incomplete, full integration is used by default.
Timestamp Initial Value	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
Reset Initial Migration Time	This parameter can be set only when you edit an FDI task. This parameter specifies whether to enable the reset of the initial migration time. Enable: The start time of each scheduling task during incremental migration is the time specified by Timestamp Initial Value. Disable: The start time of each scheduling task during incremental migration is the end time of the previous task.
Compensation Period (ms)	Mandatory when Incremental Migration is enabled. This parameter specifies the period of time (in milliseconds) which will be used to compensate for any delay in data generation at the source when ROMA Connect queries incremental data. The end time for obtaining data is the current system time minus the value you specify here. For example, if the end time of the previous incremental migration task is 15:05, the current scheduled task is triggered at 17:00, and Compensation Period (ms) is set to 100, the time range of data to be integrated in the current incremental migration task is 15:05 to (17:00 – 100 ms).
Filter	Mandatory when Insert SQL is disabled. Add filter criteria for data to be integrated. Only the source data that meets the filter criteria will be integrated to the destination. For example, the condition and \| col02 \| equal \| A means that only the data records whose values are A in the col02 column will be integrated.
Extended Metadata	Mandatory when Insert SQL is disabled. This parameter is mandatory if the value of a field in the database is in JSON format and the underlying key-value data elements in the JSON field value need to be collected. Field Name: name of the data field whose child elements need to be collected in the table. Type: data type of the data element to be collected in the JSON field value. Parsing Path: complete path of the data element in the JSON field value. For details, see Description on Extended Metadata Parsing Path Configuration.

Description on Extended Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```

Incremental migration does not support synchronization of physical deletion operations on source data tables to the destination. Logical deletion is recommended at the source.
For data changes at the source, the timestamp of data rows needs to be updated synchronously. ROMA Connect compares the timestamp with the task execution time to identify the data to be incrementally migrated.

The configuration when the source is DB2 is similar to that when the source is MySQL. For details, see MySQL configuration example.

DWS

Back to Overview

If Integration Mode is set to Scheduled, DWS can be selected as the source data source.

**Table 5** DWS source information
Parameter	Description
Insert SQL	This parameter specifies whether SQL statements are used to obtain source data. If Insert SQL is enabled, ROMA Connect obtains source data based on the entered SQL statements. If Insert SQL is disabled, ROMA Connect obtains source data based on the conditions configured on the GUI. If Insert SQL is enabled, you need to enter an SQL statement for querying data. The statement must be a SELECT statement and contain the WHERE condition. The INSERT, UPDATE, DELETE, and DROP statements cannot be used. Click Check SQL to check the validity of the statement. For example, if you enter the statement SELECT col01, col02 FROM table01 WHERE col02 IN('A', 'B', 'C') ORDER BY col01, ROMA Connect selects the col01 and col02 columns from table01, filters data rows whose values are A, B, or C in the col02 column, and sorts the data rows in sequence based on the values in the col01 column.
Table	Mandatory when Insert SQL is disabled. Select the data table from which data is to be obtained in the DWS data source. Then, click Select Table Field and select only the column fields that you want to integrate and synchronize.
Field Sorting	Mandatory when Insert SQL is disabled. This parameter specifies how the data to be integrated is sorted by field. To sort the integrated data, select a reference field and select Ascending or Descending.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For the first scheduling, the data between the initial timestamp and the current scheduling time is collected. For subsequent scheduling, the data between the last successful collection time and the current time is collected.
Time Zone	Mandatory when Incremental Migration is enabled. Select the time zone used by the DWS data source so that ROMA Connect can identify the data timestamps.
Timestamp Field	This parameter is mandatory only if Incremental Migration is enabled and Insert SQL is disabled. Select a field of the DATE type in the data table as the timestamp of source data to determine whether the data meets the incremental integration conditions.
Timestamp Initial Value	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
Reset Initial Migration Time	This parameter can be set only when you edit an FDI task. This parameter specifies whether to enable the reset of the initial migration time. Enable: The start time of each scheduling task during incremental migration is the time specified by Timestamp Initial Value. Disable: The start time of each scheduling task during incremental migration is the end time of the previous task.
Compensation Period (ms)	Mandatory when Incremental Migration is enabled. This parameter specifies the period of time (in milliseconds) which will be used to compensate for any delay in data generation at the source when ROMA Connect queries incremental data. The end time for obtaining data is the current system time minus the value you specify here. For example, if the end time of the previous incremental migration task is 15:05, the current scheduled task is triggered at 17:00, and Compensation Period (ms) is set to 100, the time range of data to be integrated in the current incremental migration task is 15:05 to (17:00 – 100 ms).
Filter	Mandatory when Insert SQL is disabled. Add filter criteria for data to be integrated. Only the source data that meets the filter criteria will be integrated to the destination. For example, the condition and \| col02 \| equal \| A means that only the data records whose values are A in the col02 column will be integrated.
Extended Metadata	Mandatory when Insert SQL is disabled. This parameter is mandatory if the value of a field in the database is in JSON format and the underlying key-value data elements in the JSON field value need to be collected. Field Name: name of the data field whose child elements need to be collected in the table. Type: data type of the data element to be collected in the JSON field value. Parsing Path: complete path of the data element in the JSON field value. For details, see Description on Extended Metadata Parsing Path Configuration.

Description on Extended Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```

Incremental migration does not support synchronization of physical deletion operations on source data tables to the destination. Logical deletion is recommended at the source.
For data changes at the source, the timestamp of data rows needs to be updated synchronously. ROMA Connect compares the timestamp with the task execution time to identify the data to be incrementally migrated.

The configuration when the source is DWS is similar to that when the source is MySQL. For details, see MySQL configuration example.

Kafka

Back to Overview

If Integration Mode is set to Real-Time, Kafka can be selected as the source data source.

**Table 6** Kafka information at the source
Parameter	Description
Topic Name	Select the name of the topic whose data is to be obtained.
Data Root Field	This parameter specifies the path of the upper-layer common fields among all metadata in the data obtained from the source in JSON format. Data Root Field and Parsing Path in Metadata form a complete metadata path. For details, see Description on Metadata Parsing Path Configuration.
Data Type	Select the data type obtained from the Kafka data source and must be consistent with the actual data type stored in Kafka. The option is JSON.
Offset	Select whether to integrate the earliest message data or the latest message data.
Time Zone	Select the time zone used by the Kafka data source so that ROMA Connect can identify the data timestamps.
Metadata	This parameter specifies each underlying key-value data element that is obtained from the source in JSON format and needs to be integrated to the destination. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the source data. Parsing Path: path of the metadata, which does not contain the data root field. For details, see Description on Metadata Parsing Path Configuration.

Description on Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b.c, and Parsing Path of element d must be set to b.d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to c, and Parsing Path of element d must be set to d.
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b[i].c, and Parsing Path of element d must be set to b[i].d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to [i].c, and Parsing Path of element d must be set to [i].d.

The preceding JSON data that does not contain arrays is used as an example. The following describes the configuration when the source is Kafka:

Figure 5 Kafka configuration example
Click to enlarge

MySQL

Back to Overview

If Integration Mode is set to Scheduled, MySQL can be selected as the source data source.

**Table 7** MySQL source information
Parameter	Description
Insert SQL	This parameter specifies whether SQL statements are used to obtain source data. If Insert SQL is enabled, ROMA Connect obtains source data based on the entered SQL statements. If Insert SQL is disabled, ROMA Connect obtains source data based on the conditions configured on the GUI. If Insert SQL is enabled, you need to enter an SQL statement for querying data. The statement must be a SELECT statement and contain the WHERE condition. The INSERT, UPDATE, DELETE, and DROP statements cannot be used. Click Check SQL to check the validity of the statement. For example, if you enter the statement SELECT col01, col02 FROM table01 WHERE col02 IN('A', 'B', 'C') ORDER BY col01, ROMA Connect selects the col01 and col02 columns from table01, displays data records whose values are A, B, or C in the col02 column, and sorts the data records in sequence based on the values in the col01 column.
Table	Mandatory when Insert SQL is disabled. Select the data table from which data is to be obtained in the MySQL data source. Then, click Select Table Field and select only the column fields that you want to integrate and synchronize.
Field Sorting	Mandatory when Insert SQL is disabled. This parameter specifies how the data to be integrated is sorted by field. To sort the integrated data, select a reference field and select Ascending or Descending.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For the first scheduling, the data between the initial timestamp and the current scheduling time is collected. For subsequent scheduling, the data between the last successful collection time and the current time is collected.
Time Zone	Mandatory when Incremental Migration is enabled. Select the time zone used by the MySQL data source so that ROMA Connect can identify the data timestamps.
Timestamp Field	This parameter is mandatory only if Incremental Migration is enabled and Insert SQL is disabled. Select a field of the DATE type as the timestamp, which can be used to check whether a data line meets data integration conditions. If the entered values of Timestamp field and Timestamp Initial Value are incomplete, full integration is used by default.
Timestamp Initial Value	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
Reset Initial Migration Time	This parameter can be set only when you edit an FDI task. This parameter specifies whether to enable the reset of the initial migration time. Enable: The start time of each scheduling task during incremental migration is the time specified by Timestamp Initial Value. Disable: The start time of each scheduling task during incremental migration is the end time of the previous task.
Compensation Period (ms)	Mandatory when Incremental Migration is enabled. This parameter specifies the period of time (in milliseconds) which will be used to compensate for any delay in data generation at the source when ROMA Connect queries incremental data. The end time for obtaining data is the current system time minus the value you specify here. For example, if the end time of the previous incremental migration task is 15:05, the current scheduled task is triggered at 17:00, and Compensation Period (ms) is set to 100, the time range of data to be integrated in the current incremental migration task is 15:05 to (17:00 – 100 ms).
Filter	Mandatory when Insert SQL is disabled. Add filter criteria for data to be integrated. Only the source data that meets the filter criteria will be integrated to the destination. For example, the condition and \| col02 \| equal \| A means that only the data records whose values are A in the col02 column will be integrated.
Extended Metadata	Mandatory when Insert SQL is disabled. This parameter is mandatory if the value of a field in the database is in JSON format and the underlying key-value data elements in the JSON field value need to be collected. Field Name: name of the data field whose child elements need to be collected in the table. Type: data type of the data element to be collected in the JSON field value. Parsing Path: complete path of the data element in the JSON field value. For details, see Description on Extended Metadata Parsing Path Configuration.

Description on Extended Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```

Incremental migration does not support synchronization of physical deletion operations on source data tables to the destination. Logical deletion is recommended at the source.
For data changes at the source, the timestamp of data rows needs to be updated synchronously. ROMA Connect compares the timestamp with the task execution time to identify the data to be incrementally migrated.

The preceding JSON data that contains arrays is used as an example. The following describes the configuration when the destination is MySQL:

In the example of incremental migration configuration, the data table must contain a field of the DATE, TIME, or TIMESTAMP type as the timestamp field.
Figure 6 MySQL incremental migration configuration example
In the example of extended metadata configuration, the child elements c and d are obtained from the desc field in the data table.
Figure 7 MySQL extended metadata configuration example

MongoDB

Back to Overview

If Integration Mode is set to Scheduled, MongoDB can be selected as the source data source.

**Table 8** MongoDB source information
Parameter	Description
Table	Select the data set to be obtained from the MongoDB data source. (The data set is equivalent to a data table in a relational database.) Then, click Select Fields in Set and select only the column fields that you want to integrate and synchronize.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For the first scheduling, the data between the initial timestamp and the current scheduling time is collected. For subsequent scheduling, the data between the last successful collection time and the current time is collected.
Timestamp Field	Mandatory when Incremental Migration is enabled. Select a field of the DATE, TIME, or TIMESTAMP type in the data table as the timestamp of source data to determine whether the data meets the incremental integration conditions.
Time Zone	Mandatory when Incremental Migration is enabled. Select the time zone used by the MongoDB data source so that ROMA Connect can identify the data timestamps.
Timestamp Initial Value	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
Reset Initial Migration Time	This parameter can be set only when you edit an FDI task. This parameter specifies whether to enable the reset of the initial migration time. Enable: The start time of each scheduling task during incremental migration is the time specified by Timestamp Initial Value. Disable: The start time of each scheduling task during incremental migration is the end time of the previous task.
Compensation Period (ms)	Mandatory when Incremental Migration is enabled. This parameter specifies the period of time (in milliseconds) which will be used to compensate for any delay in data generation at the source when ROMA Connect queries incremental data. The end time for obtaining data is the current system time minus the value you specify here. For example, if the end time of the previous incremental migration task is 15:05, the current scheduled task is triggered at 17:00, and Compensation Period (ms) is set to 100, the time range of data to be integrated in the current incremental migration task is 15:05 to (17:00 – 100 ms).

Incremental migration does not support synchronization of physical deletion operations on source data tables to the destination. Logical deletion is recommended at the source.
For data changes at the source, the timestamp of data rows needs to be updated synchronously. ROMA Connect compares the timestamp with the task execution time to identify the data to be incrementally migrated.

The configuration when the source is MongoDB is similar to that when the source is MySQL. For details, see MySQL configuration example.

OBS

Back to Overview

If Integration Mode is set to Scheduled, OBS can be selected as the source data source.

**Table 9** OBS source information
Parameter	Description
Path	Enter the name of the object from which data is to be obtained in the OBS data source. The value of Path cannot end with a slash (/).
File Name Prefix	Enter a file name prefix. This parameter is used together with Time Format to filter the data files to be integrated.
Time Format	Select the time format to be used in the file name. This parameter is used together with File Name Prefix to filter the data files to be integrated.
File Type	Select the format of the data file obtained from the OBS data source. The value can be txt, csv, or zip.
Field Separator	Enter the field separator for the data file to distinguish different fields in each row of data.
Encoding Format	Select the encoding mode of data files obtained from the OBS data source. The value can be UTF-8 or GBK.
Skip Title	This parameter specifies whether to skip the title lines in the data file. The title is the first line or several lines at the beginning of a file, which helps identify and distinguish the file content.
Title Lines	Enter the number of rows in the title information in the data file so that ROMA Connect can identify the start row of the data in the file.
Metadata	This parameter specifies each data field in a data file obtained from the source to be integrated to the destination. Metadata must be filled according to the sequence of the fields in the file. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the source data.

The following describes the configuration when the source is OBS. The id, name, and info fields are obtained from the OBS data source and need to be integrated to the destination.

Figure 8 OBS configuration example
Click to enlarge

Oracle

Back to Overview

If Integration Mode is set to Scheduled, Oracle can be selected as the source data source.

**Table 10** Oracle source information
Parameter	Description
Insert SQL	This parameter specifies whether SQL statements are used to obtain source data. If Insert SQL is enabled, ROMA Connect obtains source data based on the entered SQL statements. If Insert SQL is disabled, ROMA Connect obtains source data based on the conditions configured on the GUI. If Insert SQL is enabled, you need to enter an SQL statement for querying data. The statement must be a SELECT statement and contain the WHERE condition. The INSERT, UPDATE, DELETE, and DROP statements cannot be used. Click Check SQL to check the validity of the statement. For example, if you enter the statement SELECT col01, col02 FROM table01 WHERE col02 IN('A', 'B', 'C') ORDER BY col01, ROMA Connect selects the col01 and col02 columns from table01, displays data records whose values are A, B, or C in the col02 column, and sorts the data records in sequence based on the values in the col01 column.
Table	Mandatory when Insert SQL is disabled. Select the data table from which data is to be obtained in the Oracle data source. Then, click Select Table Field and select only the column fields that you want to integrate and synchronize.
Field Sorting	Mandatory when Insert SQL is disabled. This parameter specifies how the data to be integrated is sorted by field. To sort the integrated data, select a reference field and select Ascending or Descending.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For the first scheduling, the data between the initial timestamp and the current scheduling time is collected. For subsequent scheduling, the data between the last successful collection time and the current time is collected.
Time Zone	Mandatory when Incremental Migration is enabled. Select the time zone used by the Oracle data source so that ROMA Connect can identify the data timestamps.
Timestamp Field	Mandatory when Incremental Migration is enabled. Select a field of the DATE, TIME, or TIMESTAMP type in the data table as the timestamp of source data to determine whether the data meets the incremental integration conditions.
Timestamp Initial Value	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
Reset Initial Migration Time	This parameter can be set only when you edit an FDI task. This parameter specifies whether to enable the reset of the initial migration time. Enable: The start time of each scheduling task during incremental migration is the time specified by Timestamp Initial Value. Disable: The start time of each scheduling task during incremental migration is the end time of the previous task.
Compensation Period (ms)	Mandatory when Incremental Migration is enabled. This parameter specifies the period of time (in milliseconds) which will be used to compensate for any delay in data generation at the source when ROMA Connect queries incremental data. The end time for obtaining data is the current system time minus the value you specify here. For example, if the end time of the previous incremental migration task is 15:05, the current scheduled task is triggered at 17:00, and Compensation Period (ms) is set to 100, the time range of data to be integrated in the current incremental migration task is 15:05 to (17:00 – 100 ms).
Filter	Mandatory when Insert SQL is disabled. Add filter criteria for data to be integrated. Only the source data that meets the filter criteria will be integrated to the destination. For example, the condition and \| col02 \| equal \| A means that only the data records whose values are A in the col02 column will be integrated.
Extended Metadata	Mandatory when Insert SQL is disabled. This parameter is mandatory if the value of a field in the database is in JSON format and the underlying key-value data elements in the JSON field value need to be collected. Field Name: name of the data field whose child elements need to be collected in the table. Type: data type of the data element to be collected in the JSON field value. Parsing Path: complete path of the data element in the JSON field value. For details, see Description on Extended Metadata Parsing Path Configuration.

Description on Extended Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```

Incremental migration does not support synchronization of physical deletion operations on source data tables to the destination. Logical deletion is recommended at the source.
For data changes at the source, the timestamp of data rows needs to be updated synchronously. ROMA Connect compares the timestamp with the task execution time to identify the data to be incrementally migrated.

The configuration when the source is Oracle is similar to that when the source is MySQL. For details, see MySQL configuration example.

PostgreSQL

Back to Overview

If Integration Mode is set to Scheduled, PostgreSQL can be selected as the source data source.

**Table 11** PostgreSQL source information
Parameter	Description
Insert SQL	This parameter specifies whether SQL statements are used to obtain source data. If Insert SQL is enabled, ROMA Connect obtains source data based on the entered SQL statements. If Insert SQL is disabled, ROMA Connect obtains source data based on the conditions configured on the GUI. If Insert SQL is enabled, you need to enter an SQL statement for querying data. The statement must be a SELECT statement and contain the WHERE condition. The INSERT, UPDATE, DELETE, and DROP statements cannot be used. Click Check SQL to check the validity of the statement. For example, if you enter the statement SELECT col01, col02 FROM table01 WHERE col02 IN('A', 'B', 'C') ORDER BY col01, ROMA Connect selects the col01 and col02 columns from table01, displays data records whose values are A, B, or C in the col02 column, and sorts the data records in sequence based on the values in the col01 column.
Table	Mandatory when Insert SQL is disabled. Select the data table from which data is to be obtained in the PostgreSQL data source. Then, click Select Table Field and select only the column fields that you want to integrate and synchronize.
Field Sorting	Mandatory when Insert SQL is disabled. This parameter specifies how the data to be integrated is sorted by field. To sort the integrated data, select a reference field and select Ascending or Descending.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For the first scheduling, the data between the initial timestamp and the current scheduling time is collected. For subsequent scheduling, the data between the last successful collection time and the current time is collected.
Time Zone	Mandatory when Incremental Migration is enabled. Select the time zone used by the PostgreSQL data source so that ROMA Connect can identify the data timestamps.
Timestamp Field	Mandatory when Incremental Migration is enabled. Select a field of the DATE, TIME, or TIMESTAMP type in the data table as the timestamp of source data to determine whether the data meets the incremental integration conditions.
Timestamp Initial Value	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
Reset Initial Migration Time	This parameter can be set only when you edit an FDI task. This parameter specifies whether to enable the reset of the initial migration time. Enable: The start time of each scheduling task during incremental migration is the time specified by Timestamp Initial Value. Disable: The start time of each scheduling task during incremental migration is the end time of the previous task.
Compensation Period (ms)	Mandatory when Incremental Migration is enabled. This parameter specifies the period of time (in milliseconds) which will be used to compensate for any delay in data generation at the source when ROMA Connect queries incremental data. The end time for obtaining data is the current system time minus the value you specify here. For example, if the end time of the previous incremental migration task is 15:05, the current scheduled task is triggered at 17:00, and Compensation Period (ms) is set to 100, the time range of data to be integrated in the current incremental migration task is 15:05 to (17:00 – 100 ms).
Filter	Mandatory when Insert SQL is disabled. Add filter criteria for data to be integrated. Only the source data that meets the filter criteria will be integrated to the destination. For example, the condition and \| col02 \| equal \| A means that only the data records whose values are A in the col02 column will be integrated.
Extended Metadata	Mandatory when Insert SQL is disabled. This parameter is mandatory if the value of a field in the database is in JSON format and the underlying key-value data elements in the JSON field value need to be collected. Field Name: name of the data field whose child elements need to be collected in the table. Type: data type of the data element to be collected in the JSON field value. Parsing Path: complete path of the data element in the JSON field value. For details, see Description on Extended Metadata Parsing Path Configuration.

Description on Extended Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```

Incremental migration does not support synchronization of physical deletion operations on source data tables to the destination. Logical deletion is recommended at the source.
For data changes at the source, the timestamp of data rows needs to be updated synchronously. ROMA Connect compares the timestamp with the task execution time to identify the data to be incrementally migrated.

The configuration when the source is PostgreSQL is similar to that when the source is MySQL. For details, see MySQL configuration example.

RabbitMQ

Back to Overview

If Integration Mode is set to Real-Time, RabbitMQ can be selected as the source data source.

**Table 12** RabbitMQ source information
Parameter	Description
New Queue Creation	This parameter specifies whether to create a queue in the RabbitMQ data source. If this parameter is set to Yes, a new queue is created and data is obtained from the new queue. If this parameter is set to No, data is obtained from the existing queue.
Exchange Mode	This parameter is mandatory only if New Queue Creation is set to Yes. Select a routing mode for the exchange in RabbitMQ to forward messages to the new queue. Direct: If the routing key in a message fully matches the queue, the message will be forwarded to the queue. Topic: If the routing key in a message approximately matches the queue, the message will be forwarded to the queue. Fanout: All messages will be forwarded to the queue. Headers: If the Headers attribute of a message fully matches the queue, the message will be forwarded to the queue.
Exchange Name	This parameter is mandatory only if New Queue Creation is set to Yes. Enter the exchange name of the new queue in RabbitMQ.
Routing Key	This parameter is mandatory only if Exchange Mode is set to Direct or Topic. Enter the routing key of the new queue. RabbitMQ uses the routing key as a condition and forwards messages that meet the condition to the new queue.
Message Parameters	This parameter is mandatory only if Exchange Mode is set to Headers. Enter the Headers key-value pair of the new queue. RabbitMQ uses Headers as a condition and forwards messages that meet the condition to the new queue.
Queue Name	Enter the name of the message queue whose data is to be obtained. If New Queue Creation is set to Yes, enter a new queue name. If New Queue Creation is set to No, enter the name of an existing queue in the RabbitMQ data source.
Automatic Deletion	This parameter specifies whether a queue will be automatically deleted if no client is connected.
Persistence	This parameter specifies whether messages in a queue are stored permanently.
Metadata	This parameter specifies each underlying key-value data element that is obtained from the source in JSON format and needs to be integrated to the destination. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the source data. Parsing Path: full path of metadata. For details, see Description on Metadata Parsing Path Configuration.

Description on Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```

The preceding JSON data that does not contain arrays is used as an example. The following describes the configuration when the source is RabbitMQ:

Figure 9 RabbitMQ configuration example
Click to enlarge

SQL Server

Back to Overview

If Integration Mode is set to Scheduled, SQL Server can be selected as the source data source.

**Table 13** SQL Server source information
Parameter	Description
Insert SQL	This parameter specifies whether SQL statements are used to obtain source data. If Insert SQL is enabled, ROMA Connect obtains source data based on the entered SQL statements. If Insert SQL is disabled, ROMA Connect obtains source data based on the conditions configured on the GUI. If Insert SQL is enabled, you need to enter an SQL statement for querying data. The statement must be a SELECT statement and contain the WHERE condition. The INSERT, UPDATE, DELETE, and DROP statements cannot be used. Click Check SQL to check the validity of the statement. For example, if you enter the statement SELECT col01, col02 FROM table01 WHERE col02 IN('A', 'B', 'C') ORDER BY col01, ROMA Connect selects the col01 and col02 columns from table01, displays data records whose values are A, B, or C in the col02 column, and sorts the data records in sequence based on the values in the col01 column.
Table	Mandatory when Insert SQL is disabled. Select the data table in the SQL Server data source. Then, click Select Table Field and select only the column fields that you want to integrate and synchronize.
Field Sorting	Mandatory when Insert SQL is disabled. This parameter specifies how the data to be integrated is sorted by field. To sort the integrated data, select a reference field and select Ascending or Descending.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For the first scheduling, the data between the initial timestamp and the current scheduling time is collected. For subsequent scheduling, the data between the last successful collection time and the current time is collected.
Time Zone	Mandatory when Incremental Migration is enabled. Select the time zone used by the SQL Server data source so that ROMA Connect can identify the data timestamps.
Timestamp Field	Mandatory when Incremental Migration is enabled. Select a field of the DATE, TIME, or TIMESTAMP type in the data table as the timestamp of source data to determine whether the data meets the incremental integration conditions.
Timestamp Initial Value	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
Reset Initial Migration Time	This parameter can be set only when you edit an FDI task. This parameter specifies whether to enable the reset of the initial migration time. Enable: The start time of each scheduling task during incremental migration is the time specified by Timestamp Initial Value. Disable: The start time of each scheduling task during incremental migration is the end time of the previous task.
Compensation Period (ms)	Mandatory when Incremental Migration is enabled. This parameter specifies the period of time (in milliseconds) which will be used to compensate for any delay in data generation at the source when ROMA Connect queries incremental data. The end time for obtaining data is the current system time minus the value you specify here. For example, if the end time of the previous incremental migration task is 15:05, the current scheduled task is triggered at 17:00, and Compensation Period (ms) is set to 100, the time range of data to be integrated in the current incremental migration task is 15:05 to (17:00 – 100 ms).
Filter	Mandatory when Insert SQL is disabled. Add filter criteria for data to be integrated. Only the source data that meets the filter criteria will be integrated to the destination. For example, the condition and \| col02 \| equal \| A means that only the data records whose values are A in the col02 column will be integrated.
Extended Metadata	Mandatory when Insert SQL is disabled. This parameter is mandatory if the value of a field in the database is in JSON format and the underlying key-value data elements in the JSON field value need to be collected. Field Name: name of the data field whose child elements need to be collected in the table. Type: data type of the data element to be collected in the JSON field value. Parsing Path: complete path of the data element in the JSON field value. For details, see Description on Extended Metadata Parsing Path Configuration.

Description on Extended Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```

Incremental migration does not support synchronization of physical deletion operations on source data tables to the destination. Logical deletion is recommended at the source.
For data changes at the source, the timestamp of data rows needs to be updated synchronously. ROMA Connect compares the timestamp with the task execution time to identify the data to be incrementally migrated.

The configuration when the source is SQL Server is similar to that when the source is MySQL. For details, see MySQL configuration example.

Parent topic: Creating a Flow Data Integration Task

Previous topic: Configuring a Flow Task Process

Next topic: Configuring Destination Information