Configuring Source Information

Overview

This topic describes how to configure source information for a data integration task. Based on the source information, ROMA Connect integrates data, including the data source type, data format, and data range. The source information configuration varies depending on data source types.

Data Source Types Supported by Scheduled Integration Tasks		Data Source Types Supported by Real-Time Integration Tasks
APIs DB2 DWS DM FTP Gauss100 HANA LDAP MySQL MongoDB	MRS Hive MRS HDFS MRS HBase OBS Oracle PostgreSQL SAP SNMP SQL Server GaussDB(for MySQL) Custom	ActiveMQ ArtemisMQ DIS HL7 IBM MQ Kafka MQS MRS Kafka RabbitMQ RocketMQ WebSocket

APIs

If Integration Mode is set to Scheduled, APIs can be selected as the source data source.

On the Create Task page, configure source information.

**Table 1** API source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the API data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select API.
Data Source Name	Select the API data source that you configured in Connecting to Data Sources.
Paging	This parameter specifies whether data is returned on multiple pages when ROMA Connect sends a request to the API data source to obtain data. Multiple data records can be returned for one API request. If Paging is enabled, all data that meets the conditions is displayed on multiple pages based on a fixed number of records on each page. Each time an integration task is executed, ROMA Connect sends multiple API requests to obtain all data. That is, each API request is sent to obtain data on one page. If Paging is disabled, ROMA Connect obtains all data that meets the conditions through one API request.
Page Number Field	This parameter is mandatory only if Paging is enabled. Enter a page number field defined in the API data source, for example, pageNo. This parameter is carried when ROMA Connect sends an API request to the source to specify the number of the page from which data is to be obtained. Value indicates whether the page number starts from 0 or 1. Set Value based on the original definition of the API. The page number field must be configured in Params or Body of Request Parameters.
Page Size Field	This parameter is mandatory only if Paging is enabled. Enter a page size field defined in the API data source, for example, pageSize. This parameter is carried when ROMA Connect sends an API request to the source to specify the maximum number of records on each page. Set the number of records on each page based on the original definition of the API.
Maximum Number of Pages	This parameter is mandatory only if Paging is enabled. This parameter specifies the maximum number of pages that can be queried in each scheduled task, for example, 10. If the number of pages exceeds the specified value, the task is stopped. The value 0 indicates that no restriction applies.
Pagination End	This parameter is mandatory only if Paging is enabled. Select the method to stop obtaining source data in pagination mode. Empty page list: If no data record is returned, ROMA Connect stops obtaining source data. Number of records: ROMA Connect compares the calculation result based on the number of requested pages and the page size with the total number of records, to determine whether data stops to be obtained.
Pagination End Field Path	This parameter is mandatory only if Paging is enabled. Enter the path of the field in an API response, which is used to determine the end of pagination. In the API response, elements in different layers are separated by periods (.). For example, if element c in the {"a":{"b":{"c":"xxx"}}} response is the pagination end field, the pagination end field path is set to a.b.c. If Pagination End is set to Empty page list, set this parameter to the root path of the list field. If Pagination End is set to Number of records, set this parameter to the path of the total number of records.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For the first scheduling, the data between the initial timestamp and the current scheduling time is collected. For subsequent scheduling, the data between the last successful collection time and the current time is collected.
Start Time Field	Mandatory when Incremental Migration is enabled. Enter the start time field originally defined in the API data source, for example, startTime. This parameter is carried when ROMA Connect sends an API request to the source, indicating that data following the specified field will be obtained. The start time field and end time field must be both entered in Params or Body of the Request Parameters.
End Time Field	Mandatory when Incremental Migration is enabled. Enter the end time field originally defined in the API data source, for example, endTime. This parameter is carried when ROMA Connect sends an API request to the source, indicating that data before the specified value will be obtained.
Time Zone	Mandatory when Incremental Migration is enabled. Select the time zone used by the API data source so that ROMA Connect can identify the data timestamps.
Timestamp Initial Value	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated. Assume that Start Time Field is startTime, End Time Field is endTime, Timestamp Initial Value is 2020-11-01 12:00:00, Compensation Period is 0, and Period Settings is Default for incremental collection. If the first scheduling time of the task is 2020-11-01 13:00:00, the data collected for the first time is that the value of startTime is greater than or equal to 2020-11-01 12:00:00 and the value of endTime is less than or equal to 2020-11-01 13:00:00. For subsequent collection, the data collected each time is that the value of startTime is greater than or equal to the time when the task is successfully executed last time and the value of endTime is less than or equal to the time. Execution time of the current task.
Reset Initial Migration Time	This parameter can be set only when you edit an FDI task. This parameter specifies whether to enable the reset of the initial migration time. Enable: The start time of each scheduling task during incremental migration is the time specified by Timestamp Initial Value. Disable: The start time of each scheduling task during incremental migration is the end time of the previous task.
Compensation Period (ms)	Mandatory when Incremental Migration is enabled. This parameter specifies the period of time (in milliseconds) which will be used to compensate for any delay in data generation at the source when ROMA Connect queries incremental data. The end time for obtaining data is the current system time minus the value you specify here. For example, if the end time of the previous incremental migration task is 15:05, the current scheduled task is triggered at 17:00, and Compensation Period (ms) is set to 100, the time range of data to be integrated in the current incremental migration task is 15:05 to (17:00 – 100 ms).
Time Format	Mandatory when Incremental Migration is enabled. Select a timestamp format, for example, yyyy-MM-dd.
Period Settings	Mandatory when Incremental Migration is enabled. This parameter specifies the mode used for setting the time range for subsequent data integration after an incremental migration task is executed for the first time. Default: Data generated between the previous scheduling and current scheduling is integrated. When ROMA Connect obtains data from the source, it uses the triggering time of the two tasks as the start time and end time, respectively. Custom: The start time and end time are determined based on the configured period rules. This mode applies to common periodic tasks, for example, tasks executed once a day, a week, or a month.
Start Time Offset (Days)	This parameter is mandatory only if Period Settings is set to Default. Set the number of days before the start time of data collection. If data generated at the source changes in real time, such as alarm data, you can collect the data by setting this parameter. Start time of data collection = Data source system time – Start time offset
Time Interval	This parameter is mandatory only if Period Settings is set to Custom. Select the time granularity. The value must be the same as the unit configured in the task schedule so that the new data can be overwritten. For example, if Unit is set to Day in a task schedule, set this parameter to Day, indicating that data is obtained once a day.
Period	This parameter is mandatory only if Period Settings is set to Custom. Select the time period for obtaining source data. For example, if the task is executed once a day, Time Interval is set to Day, and Period is set to Previous period, data of the previous day is incrementally integrated each time. If Period is set to Current period, data of the current day is incrementally integrated each time.
Right Periodic Boundary	This parameter is mandatory only if Period Settings is set to Custom. This parameter specifies whether the end time is included in the time range for obtaining source data. Closed interval: The end time is included. Open interval: The end time is not included.
Request Parameters	Construct the parameter definition of the API request, for example, the page number and page size fields must be carried in Params or Body. Set this parameter based on the definition of the API data source.
Parse	If Paging is enabled, Parse is set to Yes by default and cannot be changed. This parameter specifies whether ROMA Connect parses the obtained source data. If you select Yes, ROMA Connect parses the obtained source data based on the configured parsing rules and then integrates the data to the destination. If you select No, ROMA Connect transparently transmits the obtained source data and integrates the data to the destination.
Response Type	This parameter is mandatory only if Parse is set to Yes. Select the format that will be used for the response of an API request. The value can be JSON or XML. Ensure that the format is the same as the actual response format of the API.
Data Root Field	This parameter is mandatory only if Parse is set to Yes. This parameter specifies the path of the upper-layer common fields among all metadata in the data obtained from the source in JSON or XML format. Data Root Field and Parsing Path in Metadata form a complete metadata path. For details, see Description on Metadata Parsing Path Configuration.
Metadata	This parameter is mandatory only if Parse is set to Yes. This parameter specifies each underlying key-value data element that is obtained from the source in JSON or XML format and needs to be integrated to the destination. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the response. Parsing Path: path of the metadata, which does not contain the data root field. For details, see Description on Metadata Parsing Path Configuration.

Description on Metadata Parsing Path Configuration

Data in JSON or XML format does not contain arrays:
For example, in the following JSON data (similar to XML data), the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b.c, and Parsing Path of element d must be set to b.d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to c, and Parsing Path of element d must be set to d.
Data in JSON or XML format contains arrays:
For example, in the following JSON data (similar to XML data), the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b[i].c, and Parsing Path of element d must be set to b[i].d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to [i].c, and Parsing Path of element d must be set to [i].d.

The preceding data in JSON or XML format that contains arrays is used as an example. The following describes the configuration when the destination is API:

In the example of pagination configuration. pageNo and pageSize are the pagination parameters of the API and need to be added to the Request Parameters.
Figure 1 API pagination configuration example
In the example of incremental migration configuration, startTime and endTime are the time parameters of the API and need to be added to the Request Parameters.
Figure 2 API incremental migration configuration example
In the example of metadata configuration, Data Root Field is set to a.
Figure 3 API metadata configuration example

After configuring the source information, proceed with Configuring Destination Information.

ActiveMQ

Back to Overview

If Integration Mode is set to Real-Time, ActiveMQ can be selected as the source data source.

On the Create Task page, configure source information.

**Table 2** ActiveMQ information at the source
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the ActiveMQ data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select ActiveMQ.
Data Source Name	Select the ActiveMQ data source that you configured in Connecting to Data Sources.
Destination Type	Select the message transfer model of the ActiveMQ data source. The value can be Topic or Queue.
Destination Name	Enter the name of an existing topic or queue from which data is obtained.
Parse	This parameter specifies whether ROMA Connect parses the obtained source data. If you select Yes, ROMA Connect parses the obtained source data based on the configured parsing rules and then integrates the data to the destination. If you select No, ROMA Connect transparently transmits the obtained source data and integrates the data to the destination.
Data Root Field	This parameter is mandatory only if Parse is set to Yes. This parameter specifies the path of the upper-layer common fields among all metadata in the data obtained from the source in JSON format. Data Root Field and Parsing Path in Metadata form a complete metadata path. For details, see Description on Metadata Parsing Path Configuration.
Metadata	This parameter is mandatory only if Parse is set to Yes. This parameter specifies each underlying key-value data element that is obtained from the source in JSON format and needs to be integrated to the destination. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the source data. Parsing Path: path of the metadata, which does not contain the data root field. For details, see Description on Metadata Parsing Path Configuration.

Description on Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b.c, and Parsing Path of element d must be set to b.d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to c, and Parsing Path of element d must be set to d.
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b[i].c, and Parsing Path of element d must be set to b[i].d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to [i].c, and Parsing Path of element d must be set to [i].d.

The preceding JSON data that does not contain arrays is used as an example. The following describes the configuration when the source is ActiveMQ:

Figure 4 ActiveMQ configuration example
Click to enlarge

After configuring the source information, proceed with Configuring Destination Information.

ArtemisMQ

Back to Overview

If Integration Mode is set to Real-Time, ArtemisMQ can be selected as the source data source.

On the Create Task page, configure source information.

**Table 3** ArtemisMQ information at the source
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the ArtemisMQ data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select ArtemisMQ.
Data Source Name	Select the ArtemisMQ data source that you configured in Connecting to Data Sources.
Destination Type	Select the message transfer model of the ArtemisMQ data source. The value can be Topic or Queue.
Destination Name	Enter the name of an existing topic or queue from which data is obtained.
Parse	This parameter specifies whether ROMA Connect parses the obtained source data. If you select Yes, ROMA Connect parses the obtained source data based on the configured parsing rules and then integrates the data to the destination. If you select No, ROMA Connect transparently transmits the obtained source data and integrates the data to the destination.
Data Root Field	This parameter is mandatory only if Parse is set to Yes. This parameter specifies the path of the upper-layer common fields among all metadata in the data obtained from the source in JSON format. Data Root Field and Parsing Path in Metadata form a complete metadata path. For details, see Description on Metadata Parsing Path Configuration.
Metadata	This parameter is mandatory only if Parse is set to Yes. This parameter specifies each underlying key-value data element that is obtained from the source in JSON format and needs to be integrated to the destination. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the source data. Parsing Path: path of the metadata, which does not contain the data root field. For details, see Description on Metadata Parsing Path Configuration.

Description on Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b.c, and Parsing Path of element d must be set to b.d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to c, and Parsing Path of element d must be set to d.
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b[i].c, and Parsing Path of element d must be set to b[i].d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to [i].c, and Parsing Path of element d must be set to [i].d.

The configuration when the source is ArtemisMQ is similar to that when the source is ActiveMQ. For details, see ActiveMQ configuration example.

After configuring the source information, proceed with Configuring Destination Information.

DB2

Back to Overview

If Integration Mode is set to Scheduled, DB2 can be selected as the source data source.

On the Create Task page, configure source information.

**Table 4** DB2 source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the DB2 data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select DB2.
Data Source Name	Select the DB2 data source that you configured in Connecting to Data Sources.
Insert SQL	This parameter specifies whether SQL statements are used to obtain source data. If Insert SQL is enabled, ROMA Connect obtains source data based on the entered SQL statements. If Insert SQL is disabled, ROMA Connect obtains source data based on the conditions configured on the GUI. If Insert SQL is enabled, you need to enter an SQL statement for querying data. The statement must be a SELECT statement containing WHERE. INSERT, UPDATE, DELETE, and DROP statements cannot be used. Click Check SQL to check the validity of the statement. For example, if you enter the statement SELECT col01, col02 FROM table01 WHERE col02 IN('A', 'B', 'C') ORDER BY col01, ROMA Connect selects the col01 and col02 columns from table01, displays data records whose values are A, B, or C in the col02 column, and sorts the data records in sequence based on the values in the col01 column.
Table	Mandatory when Insert SQL is disabled. Select the data table from which data is to be obtained in the DB2 data source. Then, click Select Table Field and select only the column fields that you want to integrate and synchronize, such as the ID.
Field Sorting	Mandatory when Insert SQL is disabled. This parameter specifies how the data to be integrated is sorted by field. To sort the integrated data, select a reference field and select Ascending or Descending.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For first-time scheduling, data between the initial timestamp and the specified timestamp is collected. The subsequent collection starts from the time when the last data record is imported in the previous collection to the specified timestamp.
Time Zone	Mandatory when Incremental Migration is enabled. Select the time zone used by the DB2 data source so that ROMA Connect can identify the data timestamps.
Timestamp Field	This parameter is mandatory only if Incremental Migration is enabled and Insert SQL is disabled. Select a field of the DATE type as the timestamp, which can be used to check whether a data row meets incremental integration conditions. If the entered values of Timestamp field and Timestamp Initial Value are incomplete, full integration is used by default.
Timestamp Initial Value	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
Reset Initial Migration Time	This parameter can be set only when you edit an FDI task. This parameter specifies whether to enable the reset of the initial migration time. Enable: The start time of each scheduling task during incremental migration is the time specified by Timestamp Initial Value. Disable: The start time of each scheduling task during incremental migration is the end time of the previous task.
Filter	Mandatory when Insert SQL is disabled. Add filter criteria for data to be integrated. Only the source data that meets the filter criteria will be integrated to the destination. For example, the condition and \| col02 \| equal \| A means that only the data records whose values are A in the col02 column will be integrated.
Extended Metadata	Mandatory when Insert SQL is disabled. This parameter is mandatory if the value of a field in the database is in JSON format and the underlying key-value data elements in the JSON field value need to be collected. Field Name: name of the data field whose child elements need to be collected in the table. Type: data type of the data element to be collected in the JSON field value. Parsing Path: complete path of the data element in the JSON field value. For details, see Description on Extended Metadata Parsing Path Configuration.

Description on Extended Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```

Incremental migration does not support synchronization of physical deletion operations on source data tables to the destination. Logical deletion is recommended at the source.
For data changes at the source, the timestamp of data rows needs to be updated synchronously. ROMA Connect compares the timestamp with the task execution time to identify the data to be incrementally migrated.

The configuration when the source is DB2 is similar to that when the source is MySQL. For details, see MySQL configuration example.

After configuring the source information, proceed with Configuring Destination Information.

DWS

Back to Overview

If Integration Mode is set to Scheduled, DWS can be selected as the source data source.

On the Create Task page, configure source information.

**Table 5** DWS source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the DWS data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select DWS.
Data Source Name	Select the DWS data source that you configured in Connecting to Data Sources.
Insert SQL	This parameter specifies whether SQL statements are used to obtain source data. If Insert SQL is enabled, ROMA Connect obtains source data based on the entered SQL statements. If Insert SQL is disabled, ROMA Connect obtains source data based on the conditions configured on the GUI. If Insert SQL is enabled, you need to enter an SQL statement for querying data. The statement must be a SELECT statement containing WHERE. INSERT, UPDATE, DELETE, and DROP statements cannot be used. Click Check SQL to check the validity of the statement. For example, if you enter the statement SELECT col01, col02 FROM table01 WHERE col02 IN('A', 'B', 'C') ORDER BY col01, ROMA Connect selects the col01 and col02 columns from table01, displays data records whose values are A, B, or C in the col02 column, and sorts the data records in sequence based on the values in the col01 column.
Table	Mandatory when Insert SQL is disabled. Select the data table from which data is to be obtained in the DWS data source. Then, click Select Table Field and select only the column fields that you want to integrate and synchronize.
Field Sorting	Mandatory when Insert SQL is disabled. This parameter specifies how the data to be integrated is sorted by field. To sort the integrated data, select a reference field and select Ascending or Descending.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For first-time scheduling, data between the initial timestamp and the specified timestamp is collected. The subsequent collection starts from the time when the last data record is imported in the previous collection to the specified timestamp.
Time Zone	Mandatory when Incremental Migration is enabled. Select the time zone used by the DWS data source so that ROMA Connect can identify the data timestamps.
Timestamp Field	This parameter is mandatory only if Incremental Migration is enabled and Insert SQL is disabled. Select a field of the DATE type in the data table as the timestamp of source data to determine whether the data meets the incremental integration conditions.
Timestamp Initial Value	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
Reset Initial Migration Time	This parameter can be set only when you edit an FDI task. This parameter specifies whether to enable the reset of the initial migration time. Enable: The start time of each scheduling task during incremental migration is the time specified by Timestamp Initial Value. Disable: The start time of each scheduling task during incremental migration is the end time of the previous task.
Filter	Mandatory when Insert SQL is disabled. Add filter criteria for data to be integrated. Only the source data that meets the filter criteria will be integrated to the destination. For example, the condition and \| col02 \| equal \| A means that only the data records whose values are A in the col02 column will be integrated.

Incremental migration does not support synchronization of physical deletion operations on source data tables to the destination. Logical deletion is recommended at the source.
For data changes at the source, the timestamp of data rows needs to be updated synchronously. ROMA Connect compares the timestamp with the task execution time to identify the data to be incrementally migrated.

The configuration when the source is DWS is similar to that when the source is MySQL. For details, see MySQL configuration example.

After configuring the source information, proceed with Configuring Destination Information.

DIS

Back to Overview

If Integration Mode is set to Real-Time, DIS can be selected as the source data source type.

On the Create Task page, configure source information.

**Table 6** DIS source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the DIS data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select DIS.
Data Source Name	Select the DIS data source that you configured in Connecting to Data Sources.
Parse	This parameter specifies whether ROMA Connect parses the obtained source data. If you select Yes, ROMA Connect parses the obtained source data based on the configured parsing rules and then integrates the data to the destination. If you select No, ROMA Connect transparently transmits the obtained source data and integrates the data to the destination.
Data Root Field	This parameter is mandatory only if Parse is set to Yes. This parameter specifies the path of the upper-layer common fields among all metadata in the data obtained from the source in JSON format. Data Root Field and Parsing Path in Metadata form a complete metadata path. For details, see Description on Metadata Parsing Path Configuration.
Data Type	This parameter is mandatory only if Parse is set to Yes. Select the format of data obtained from the DIS data source. The data format must be the same as the actual data format stored in DIS.
Metadata	This parameter is mandatory only if Parse is set to Yes. This parameter specifies each underlying key-value data element that is obtained from the source in JSON format and needs to be integrated to the destination. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the source data. Parsing Path: path of the metadata, which does not contain the data root field. For details, see Description on Metadata Parsing Path Configuration.
Time Zone	Select the time zone used by the DIS data source so that ROMA Connect can identify the data timestamps.

Description on Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b.c, and Parsing Path of element d must be set to b.d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to c, and Parsing Path of element d must be set to d.
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b[i].c, and Parsing Path of element d must be set to b[i].d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to [i].c, and Parsing Path of element d must be set to [i].d.

The configuration when the source is DIS is similar to that when the source is ActiveMQ. For details, see ActiveMQ configuration example.

After configuring the source information, proceed with Configuring Destination Information.

DM

Back to Overview

If Integration Mode is set to Scheduled, DM can be selected as the source data source.

On the Create Task page, configure source information.

**Table 7** DM source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the DM data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select DM.
Data Source Name	Select the DM data source that you configured in Connecting to Data Sources.
Insert SQL	This parameter specifies whether SQL statements are used to obtain source data. If Insert SQL is enabled, ROMA Connect obtains source data based on the entered SQL statements. If Insert SQL is disabled, ROMA Connect obtains source data based on the conditions configured on the GUI. If Insert SQL is enabled, you need to enter an SQL statement for querying data. The statement must be a SELECT statement containing WHERE. INSERT, UPDATE, DELETE, and DROP statements cannot be used. Click Check SQL to check the validity of the statement. Example: select string_col,number_col from zyw_test where time_col >= ${{begin}} and time_col < ${{end}} ${{begin}} indicates the time the task was last executed (do not change the format). ${{end}} indicates the time when the current task starts execution (do not change the format).
Table	Mandatory when Insert SQL is disabled. Select the data table in the DM data source. Then, click Select Table Field and select only the column fields that you want to integrate and synchronize, such as the ID.
Field Sorting	Mandatory when Insert SQL is disabled. This parameter specifies how the data to be integrated is sorted by field. To sort the integrated data, select a reference field and select Ascending or Descending.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For first-time scheduling, data between the initial timestamp and the specified timestamp is collected. The subsequent collection starts from the time when the last data record is imported in the previous collection to the specified timestamp.
Time Zone	Mandatory when Incremental Migration is enabled. Select the time zone used by the DM data source so that ROMA Connect can identify the data timestamps.
Timestamp Field	This parameter is mandatory only if Incremental Migration is enabled and Insert SQL is disabled. Select a field of the DATE type as the timestamp, which can be used to check whether a data row meets incremental integration conditions. If the entered values of Timestamp field and Timestamp Initial Value are incomplete, full integration is used by default.
Timestamp Initial Value	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
Reset Initial Migration Time	This parameter can be set only when you edit an FDI task. This parameter specifies whether to enable the reset of the initial migration time. Enable: The start time of each scheduling task during incremental migration is the time specified by Timestamp Initial Value. Disable: The start time of each scheduling task during incremental migration is the end time of the previous task.
Filter	Mandatory when Insert SQL is disabled. Add filter criteria for data to be integrated. Only the source data that meets the filter criteria will be integrated to the destination. For example, the condition and \| col02 \| equal \| A means that only the data records whose values are A in the col02 column will be integrated.
Extended Metadata	Mandatory when Insert SQL is disabled. This parameter is mandatory if the value of a field in the database is in JSON format and the underlying key-value data elements in the JSON field value need to be collected. Field Name: name of the data field whose child elements need to be collected in the table. Type: data type of the data element to be collected in the JSON field value. Parsing Path: complete path of the data element in the JSON field value. For details, see Description on Extended Metadata Parsing Path Configuration.

Description on Extended Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```

Incremental migration does not support synchronization of physical deletion operations on source data tables to the destination. Logical deletion is recommended at the source.
For data changes at the source, the timestamp of data rows needs to be updated synchronously. ROMA Connect compares the timestamp with the task execution time to identify the data to be incrementally migrated.

The configuration when the source is DM is similar to that when the source is MySQL. For details, see MySQL configuration example.

After configuring the source information, proceed with Configuring Destination Information.

FTP

Back to Overview

If Integration Mode is set to Scheduled, FTP can be selected as the source data source.

On the Create Task page, configure source information.

**Table 8** FTP source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the FTP data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select FTP.
Data Source Name	Select the FTP data source that you configured in Connecting to Data Sources.
File Path	Enter the path of the folder to be accessed on the FTP server, for example, /data/FDI.
File Name	Enter the name of the data file from which data is to be obtained. If you do not specify this parameter, data of all files is obtained.
File Name Encoding	Select the encoding mode of the data file name.
Parse	This parameter specifies whether ROMA Connect parses the obtained source data. If you select Yes, ROMA Connect parses the obtained source data based on the configured parsing rules and then integrates the data to the destination. If you select No, ROMA Connect transparently transmits the obtained source data and integrates the data to the destination.
Maximum File Size	Enter the maximum size of the file to be obtained. If the size of a file exceeds the value you specify here, the file cannot be obtained.
File Content Encoding	This parameter is mandatory only if Parse is set to Yes. Select the encoding format of the data file content.
File Separator	This parameter is mandatory only if Parse is set to Yes. Enter the field separator for the data file to distinguish different fields in each row of data.
Space Format Character	This parameter is mandatory only if Parse is set to Yes. Enter the space characters defined in the data file so that ROMA Connect can identify the space characters in the file content. For example, if the space character is defined as a period (.), periods are considered as spaces during data reading.
Skip File Header	This parameter is mandatory only if Parse is set to Yes. This parameter specifies whether to skip the file header in the data file. The file header is the first line or several lines at the beginning of a file, which helps identify and distinguish the file content.
Skipped Header Lines	This parameter is mandatory only if Skip File Header is set to Yes. Enter the number of lines in the file header in a data file so that ROMA Connect can skip the header lines and identify the start line of the data in the file.
Migrated Data Quantity	Set the amount of data to be obtained. If Parse is set to Yes, this parameter indicates the number of data rows to be parsed. If Parse is set to No, this parameter indicates the number of files to be read.
Metadata	This parameter is mandatory only if Parse is set to Yes. This parameter specifies each data field in a data file obtained from the source to be integrated to the destination. Metadata must be filled according to the sequence of the fields in the file. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the source data.

The following describes the configuration when the source is FTP. The id, name, and info fields are obtained from the FTP data source and need to be integrated to the destination.

Figure 5 FTP configuration example
Click to enlarge

After configuring the source information, proceed with Configuring Destination Information.

Gauss100

Back to Overview

If Integration Mode is set to Scheduled, Gauss100 can be selected as the source data source.

On the Create Task page, configure source information.

**Table 9** Gauss100 source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the Gauss100 data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select Gauss100.
Data Source Name	Select the Gauss100 data source that you configured in Connecting to Data Sources.
Insert SQL	This parameter specifies whether SQL statements are used to obtain source data. If Insert SQL is enabled, ROMA Connect obtains source data based on the entered SQL statements. If Insert SQL is disabled, ROMA Connect obtains source data based on the conditions configured on the GUI. If Insert SQL is enabled, you need to enter an SQL statement for querying data. The statement must be a SELECT statement containing WHERE. INSERT, UPDATE, DELETE, and DROP statements cannot be used. Click Check SQL to check the validity of the statement. For example, if you enter the statement SELECT col01, col02 FROM table01 WHERE col02 IN('A', 'B', 'C') ORDER BY col01, ROMA Connect selects the col01 and col02 columns from table01, displays data records whose values are A, B, or C in the col02 column, and sorts the data records in sequence based on the values in the col01 column.
Schema	Mandatory when Insert SQL is disabled. Select a schema.
Table	Mandatory when Insert SQL is disabled. Select the data table from which data is to be obtained in the Gauss100 data source. Then, click Select Table Field and select only the column fields that you want to integrate and synchronize.
Field Sorting	Mandatory when Insert SQL is disabled. This parameter specifies how the data to be integrated is sorted by field. To sort the integrated data, select a reference field and select Ascending or Descending.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For first-time scheduling, data between the initial timestamp and the specified timestamp is collected. The subsequent collection starts from the time when the last data record is imported in the previous collection to the specified timestamp.
Time Zone	Mandatory when Incremental Migration is enabled. Select the time zone used by the Gauss100 data source so that ROMA Connect can identify the data timestamps.
Timestamp Field	This parameter is mandatory only if Incremental Migration is enabled and Insert SQL is disabled. Select a field of the DATE, TIME, or TIMESTAMP type in the data table as the timestamp of source data to determine whether the data meets the incremental integration conditions.
Timestamp Initial Value	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
Reset Initial Migration Time	This parameter can be set only when you edit an FDI task. This parameter specifies whether to enable the reset of the initial migration time. Enable: The start time of each scheduling task during incremental migration is the time specified by Timestamp Initial Value. Disable: The start time of each scheduling task during incremental migration is the end time of the previous task.
Filter	Mandatory when Insert SQL is disabled. Add filter criteria for data to be integrated. Only the source data that meets the filter criteria will be integrated to the destination. For example, the condition and \| col02 \| equal \| A means that only the data records whose values are A in the col02 column will be integrated.

Incremental migration does not support synchronization of physical deletion operations on source data tables to the destination. Logical deletion is recommended at the source.
For data changes at the source, the timestamp of data rows needs to be updated synchronously. ROMA Connect compares the timestamp with the task execution time to identify the data to be incrementally migrated.

The configuration when the source is Gauss100 is similar to that when the source is MySQL. For details, see MySQL configuration example.

After configuring the source information, proceed with Configuring Destination Information.

HANA

Back to Overview

If Integration Mode is set to Scheduled, HANA can be selected as the source data source.

On the Create Task page, configure source information.

**Table 10** HANA source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the HANA data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select HANA.
Data Source Name	Select the HANA data source that you configured in Connecting to Data Sources.
Insert SQL	This parameter specifies whether SQL statements are used to obtain source data. If Insert SQL is enabled, ROMA Connect obtains source data based on the entered SQL statements. If Insert SQL is disabled, ROMA Connect obtains source data based on the conditions configured on the GUI. If Insert SQL is enabled, you need to enter an SQL statement for querying data. The statement must be a SELECT statement containing WHERE. INSERT, UPDATE, DELETE, and DROP statements cannot be used. Click Check SQL to check the validity of the statement. For example, if you enter the statement SELECT col01, col02 FROM table01 WHERE col02 IN('A', 'B', 'C') ORDER BY col01, ROMA Connect selects the col01 and col02 columns from table01, displays data records whose values are A, B, or C in the col02 column, and sorts the data records in sequence based on the values in the col01 column.
Table	Mandatory when Insert SQL is disabled. Select the data table from which data is to be obtained in the HANA data source. Then, click Select Table Field and select only the column fields that you want to integrate and synchronize.
Field Sorting	Mandatory when Insert SQL is disabled. This parameter specifies how the data to be integrated is sorted by field. To sort the integrated data, select a reference field and select Ascending or Descending.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For first-time scheduling, data between the initial timestamp and the specified timestamp is collected. The subsequent collection starts from the time when the last data record is imported in the previous collection to the specified timestamp.
Time Zone	Mandatory when Incremental Migration is enabled. Select the time zone used by the HANA data source so that ROMA Connect can identify the data timestamps.
Timestamp Field	This parameter is mandatory only if Incremental Migration is enabled and Insert SQL is disabled. Select a field of the DATE, TIME, or TIMESTAMP type in the data table as the timestamp of source data to determine whether the data meets the incremental integration conditions.
Timestamp Initial Value	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
Reset Initial Migration Time	This parameter can be set only when you edit an FDI task. This parameter specifies whether to enable the reset of the initial migration time. Enable: The start time of each scheduling task during incremental migration is the time specified by Timestamp Initial Value. Disable: The start time of each scheduling task during incremental migration is the end time of the previous task.
Filter	Mandatory when Insert SQL is disabled. Add filter criteria for data to be integrated. Only the source data that meets the filter criteria will be integrated to the destination. For example, the condition and \| col02 \| equal \| A means that only the data records whose values are A in the col02 column will be integrated.

Incremental migration does not support synchronization of physical deletion operations on source data tables to the destination. Logical deletion is recommended at the source.
For data changes at the source, the timestamp of data rows needs to be updated synchronously. ROMA Connect compares the timestamp with the task execution time to identify the data to be incrementally migrated.

The configuration when the source is HANA is similar to that when the source is MySQL. For details, see MySQL configuration example.

After configuring the source information, proceed with Configuring Destination Information.

HL7

Back to Overview

If Integration Mode is set to Real-Time, HL7 can be selected as the source data source.

On the Create Task page, configure source information.

**Table 11** HL7 source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the HL7 data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select HL7.
Data Source Name	Select the HL7 data source that you configured in Connecting to Data Sources.
Encoding Format	Select the encoding mode of data files in the HL7 data source. The value can be UTF-8 or GBK.
Metadata	This parameter specifies the data fields in HL7 messages obtained from the source to be integrated to the destination. Alias: user-defined metadata name. Type: data type of metadata. Parsing Path: location of metadata in HL7 messages. For details, see the metadata path configuration description in the subsequent section.

Description on Metadata Path Configuration

MSH|^~\\&|hl7Integration|hl7Integration|||||ADT^A01|||2.3|
EVN|A01|20191212155644
PID|||PATID1234^5^M11||FN^Patrick^^^MR||19700101|1|||xx Street^^NY^^Ox4DP|||||||
NK1|1|FN^John^^^MR|Father||999-9999
NK1|2|MN^Georgie^^^MSS|Mother||999-9999

The metadata parsing path of HL7 messages must be set based on the Terser syntax specifications. In the preceding example HL7 message, each line represents an information segment. Each information segment starts with three uppercase letters, which are paragraph symbols of the information segment and are used to indicate content of the information segment. Information segments are separated by separators.

|: field separator, which is used to divide information segments into different fields. Each field in an information segment is numbered starting from 1 (excluding paragraph symbols). The rest may be deduced by analogy.
^: component separator, which divides the field content into different components. In the fields that are divided into components, the position of a component is identified by a number, starting from 1. The rest may be deduced by analogy.
~: subcomponent separator, which is used to divide a component into subcomponents.

For example, in the PID information segment, if the field position of 19700101 is 7, the parsing path is /PID-7. If the field position of xx Street is 11 and the component position is 1, the parsing path is /PID-11-1.

For the information segments with the same paragraph symbol in the HL7 message, the repeated paragraph symbol is identified by adding a number enclosed by brackets after the paragraph symbol. In repeated paragraph symbols, the first is (0), the second is (1), and so on.

For example, in the NK1 information segment, "Father" is located in the first NK1 information segment, and a field location is 3, a parsing path of the NK1 information segment is NK1(0)-3. Similarly, the parsing path of Mother is NK1(1)-3.

Reading the 19700101 and xx Street fields in the preceding HL7 message is used as an example. The following describes the configuration when the source is HL7:

Figure 6 HL7 configuration example
Click to enlarge

After configuring the source information, proceed with Configuring Destination Information.

IBM MQ

Back to Overview

If Integration Mode is set to Real-Time, IBM MQ can be selected as the source data source.

On the Create Task page, configure source information.

**Table 12** IBM MQ information at the source
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the IBM MQ data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select IBM MQ.
Data Source Name	Select the IBM MQ data source that you configured in Connecting to Data Sources.
Destination Type	Select the message transfer model of the IBM MQ data source. The value can be Topic or Queue.
Destination Name	Enter the name of an existing topic or queue from which data is obtained.
Parse	This parameter specifies whether ROMA Connect parses the obtained source data. If you select Yes, ROMA Connect parses the obtained source data based on the configured parsing rules and then integrates the data to the destination. If you select No, ROMA Connect transparently transmits the obtained source data and integrates the data to the destination.
Data Root Field	This parameter is mandatory only if Parse is set to Yes. This parameter specifies the path of the upper-layer common fields among all metadata in the data obtained from the source in JSON format. Data Root Field and Parsing Path in Metadata form a complete metadata path. For details, see Description on Metadata Parsing Path Configuration.
Metadata	This parameter is mandatory only if Parse is set to Yes. This parameter specifies each underlying key-value data element that is obtained from the source in JSON format and needs to be integrated to the destination. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the source data. Parsing Path: path of the metadata, which does not contain the data root field. For details, see Description on Metadata Parsing Path Configuration.

Description on Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b.c, and Parsing Path of element d must be set to b.d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to c, and Parsing Path of element d must be set to d.
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b[i].c, and Parsing Path of element d must be set to b[i].d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to [i].c, and Parsing Path of element d must be set to [i].d.

The configuration when the source is IBM MQ is similar to that when the source is ActiveMQ. For details, see ActiveMQ configuration example.

After configuring the source information, proceed with Configuring Destination Information.

Kafka

Back to Overview

If Integration Mode is set to Real-Time, Kafka can be selected as the source data source. ROMA Connect MQS is a Kafka data source.

On the Create Task page, configure source information.

**Table 13** Kafka source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the Kafka data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select Kafka.
Data Source Name	Select the Kafka data source that you configured in Connecting to Data Sources.
Topic Name	Select the name of the topic whose data is to be obtained.
Parse	This parameter specifies whether ROMA Connect parses the obtained source data. If you select Yes, ROMA Connect parses the obtained source data based on the configured parsing rules and then integrates the data to the destination. If you select No, ROMA Connect transparently transmits the obtained source data and integrates the data to the destination.
Data Root Field	This parameter is mandatory only if Parse is set to Yes. This parameter specifies the path of the upper-layer common fields among all metadata in the data obtained from the source in JSON format. Data Root Field and Parsing Path in Metadata form a complete metadata path. For details, see Description on Metadata Parsing Path Configuration.
Data Type	Select the format of data obtained from the Kafka data source. The data format must be the same as the actual data format stored in Kafka. If Parse is set to Yes, you can select JSON or XML. If Parse is set to No, you can select JSON, XML, or Binary file.
Offset	Select whether to integrate the earliest message data or the latest message data.
Metadata	This parameter is mandatory only if Parse is set to Yes. This parameter specifies each underlying key-value data element that is obtained from the source in JSON format and needs to be integrated to the destination. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the source data. Parsing Path: path of the metadata, which does not contain the data root field. For details, see Description on Metadata Parsing Path Configuration.
Time Zone	Select the time zone used by the Kafka data source so that ROMA Connect can identify the data timestamps.

Description on Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b.c, and Parsing Path of element d must be set to b.d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to c, and Parsing Path of element d must be set to d.
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b[i].c, and Parsing Path of element d must be set to b[i].d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to [i].c, and Parsing Path of element d must be set to [i].d.

The preceding JSON data that does not contain arrays is used as an example. The following describes the configuration when the source is Kafka:

Figure 7 Kafka configuration example
Click to enlarge

After configuring the source information, proceed with Configuring Destination Information.

LDAP

Back to Overview

If Integration Mode is set to Scheduled, LDAP can be selected as the source data source.

On the Create Task page, configure source information.

**Table 14** LDAP source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the LDAP data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select LDAP.
Data Source Name	Select the LDAP data source that you configured in Connecting to Data Sources.
DN	This parameter specifies the distinguished name (DN) in the character string for connecting to the server, including Domain Component (DC), Common Name (CN), User ID (UID), and Organizational Unit (OU). For example, if DN is set to CN=test,OU=roma,DC=domainname, it indicates the test object in the roma organization of the domainname domain.
Filter Criteria	Filter fields based on the LDAP syntax. For example, (\|(uid=user.1*)(&(createTimestamp>=${{begin}})(createTimestamp<=${{end}}}))) indicates that objects whose UID starts with user.1 within a specified period are queried. ${{begin}} indicates the time when the task was executed last time. ${{end}} indicates the time when the task is executed this time.
Time Format	Select a time format for the filter criterion.
Timestamp Initial Value	This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
Metadata	This parameter specifies each data field in a data file obtained from the source to be integrated to the destination. Metadata must be filled according to the sequence of the fields in the file. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the source data.

The following describes the configuration when the source is LDAP. The id, name, and info fields are obtained from the LDAP data source and need to be integrated to the destination.

Figure 8 LDAP configuration example
Click to enlarge

After configuring the source information, proceed with Configuring Destination Information.

MySQL

Back to Overview

If Integration Mode is set to Scheduled, MySQL can be selected as the source data source.

On the Create Task page, configure source information.

**Table 15** MySQL information at the source
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the MySQL data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select MySQL.
Data Source Name	Select the MySQL data source that you configured in Connecting to Data Sources.
Insert SQL	This parameter specifies whether SQL statements are used to obtain source data. If Insert SQL is enabled, ROMA Connect obtains source data based on the entered SQL statements. If Insert SQL is disabled, ROMA Connect obtains source data based on the conditions configured on the GUI. If Insert SQL is enabled, you need to enter an SQL statement for querying data. The statement must be a SELECT statement containing WHERE. INSERT, UPDATE, DELETE, and DROP statements cannot be used. Click Check SQL to check the validity of the statement. For example, if you enter the statement SELECT col01, col02 FROM table01 WHERE col02 IN('A', 'B', 'C') ORDER BY col01, ROMA Connect selects the col01 and col02 columns from table01, displays data records whose values are A, B, or C in the col02 column, and sorts the data records in sequence based on the values in the col01 column.
Table	Mandatory when Insert SQL is disabled. Select the data table from which data is to be obtained in the MySQL data source. Then, click Select Table Field and select only the column fields that you want to integrate and synchronize.
Field Sorting	Mandatory when Insert SQL is disabled. This parameter specifies how the data to be integrated is sorted by field. To sort the integrated data, select a reference field and select Ascending or Descending.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For first-time scheduling, data between the initial timestamp and the specified timestamp is collected. The subsequent collection starts from the time when the last data record is imported in the previous collection to the specified timestamp.
Time Zone	Mandatory when Incremental Migration is enabled. Select the time zone used by the MySQL data source so that ROMA Connect can identify the data timestamps.
Timestamp	This parameter is mandatory only if Incremental Migration is enabled and Insert SQL is disabled. Select a field of the DATE type as the timestamp, which can be used to check whether a data row meets data integration conditions. If the entered values of Timestamp field and Timestamp Initial Value are incomplete, full integration is used by default. If the timestamp field type is timestamp, set Time Zone to GMT to prevent query failures.
Timestamp Initial Value	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
Reset Initial Migration Time	This parameter can be set only when you edit an FDI task. This parameter specifies whether to enable the reset of the initial migration time. Enable: The start time of each scheduling task during incremental migration is the time specified by Timestamp Initial Value. Disable: The start time of each scheduling task during incremental migration is the end time of the previous task.
Filter	Mandatory when Insert SQL is disabled. Add filter criteria for data to be integrated. Only the source data that meets the filter criteria will be integrated to the destination. For example, the condition and \| col02 \| equal \| A means that only the data records whose values are A in the col02 column will be integrated.
Extended Metadata	Mandatory when Insert SQL is disabled. This parameter is mandatory if the value of a field in the database is in JSON format and the underlying key-value data elements in the JSON field value need to be collected. Field Name: name of the data field whose child elements need to be collected in the table. Type: data type of the data element to be collected in the JSON field value. Parsing Path: complete path of the data element in the JSON field value. For details, see Description on Extended Metadata Parsing Path Configuration.

Description on Extended Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```

Incremental migration does not support synchronization of physical deletion operations on source data tables to the destination. Logical deletion is recommended at the source.
For data changes at the source, the timestamp of data rows needs to be updated synchronously. ROMA Connect compares the timestamp with the task execution time to identify the data to be incrementally migrated.

The preceding JSON data that contains arrays is used as an example. The following describes the configuration when the source is MySQL:

In the example of incremental migration configuration, the data table must contain a field of the DATE, TIME, or TIMESTAMP type as the timestamp field.
Figure 9 MySQL incremental migration configuration example
In the example of extended metadata configuration, the child elements c and d are obtained from the desc field in the data table.
Figure 10 MySQL extended metadata configuration example

After configuring the source information, proceed with Configuring Destination Information.

MongoDB

Back to Overview

If Integration Mode is set to Scheduled, MongoDB can be selected as the source data source.

On the Create Task page, configure source information.

**Table 16** MongoDB information at the source
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the MongoDB data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select MongoDB.
Data Source Name	Select the MongoDB data source that you configured in Connecting to Data Sources.
Source Set	Select the data set to be obtained from the MongoDB data source. (The data set is equivalent to a data table in a relational database.) Then, click Select Fields in Set and select only the column fields that you want to integrate and synchronize.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For the first scheduling, the data between the initial timestamp and the current scheduling time is collected. For subsequent scheduling, the data between the last successful collection time and the current time is collected.
Timestamp Field	Mandatory when Incremental Migration is enabled. Select a field of the DATE, TIME, or TIMESTAMP type in the data table as the timestamp of source data to determine whether the data meets the incremental integration conditions.
Time Zone	Mandatory when Incremental Migration is enabled. Select the time zone used by the MongoDB data source so that ROMA Connect can identify the data timestamps.
Timestamp Initial Value	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
Reset Initial Migration Time	This parameter can be set only when you edit an FDI task. This parameter specifies whether to enable the reset of the initial migration time. Enable: The start time of each scheduling task during incremental migration is the time specified by Timestamp Initial Value. Disable: The start time of each scheduling task during incremental migration is the end time of the previous task.
Compensation Period (ms)	Mandatory when Incremental Migration is enabled. This parameter specifies the period of time (in milliseconds) which will be used to compensate for any delay in data generation at the source when ROMA Connect queries incremental data. The end time for obtaining data is the current system time minus the value you specify here. For example, if the end time of the previous incremental migration task is 15:05, the current scheduled task is triggered at 17:00, and Compensation Period (ms) is set to 100, the time range of data to be integrated in the current incremental migration task is 15:05 to (17:00 – 100 ms).

Incremental migration does not support synchronization of physical deletion operations on source data tables to the destination. Logical deletion is recommended at the source.
For data changes at the source, the timestamp of data rows needs to be updated synchronously. ROMA Connect compares the timestamp with the task execution time to identify the data to be incrementally migrated.

The configuration when the source is MongoDB is similar to that when the source is MySQL. For details, see MySQL configuration example.

After configuring the source information, proceed with Configuring Destination Information.

MQS

Back to Overview

If Integration Mode is set to Real-Time, MQS can be selected as the source data source. ROMA Connect MQS is an MQS data source.

On the Create Task page, configure source information.

**Table 17** MQS source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the MQS data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select MQS.
Data Source Name	Select the MQS data source that you configured in Connecting to Data Sources.
Topic Name	Select the name of the topic whose data is to be obtained.
Parse	This parameter specifies whether ROMA Connect parses the obtained source data. If you select Yes, ROMA Connect parses the obtained source data based on the configured parsing rules and then integrates the data to the destination. If you select No, ROMA Connect transparently transmits the obtained source data and integrates the data to the destination.
Data Root Field	This parameter is mandatory only if Parse is set to Yes. This parameter specifies the path of the upper-layer common fields among all metadata in the data obtained from the source in JSON format. Data Root Field and Parsing Path in Metadata form a complete metadata path. For details, see Description on Metadata Parsing Path Configuration.
Data Type	Select the format of data obtained from the MQS data source. The data format must be the same as the actual data format stored in MQS. If Parse is set to Yes, select JSON or XML. If Parse is set to No, select JSON, XML, or Binary file.
Offset	Select whether to integrate the earliest message data or the latest message data.
Metadata	This parameter is mandatory only if Parse is set to Yes. This parameter specifies each underlying key-value data element that is obtained from the source in JSON format and needs to be integrated to the destination. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the source data. Parsing Path: path of the metadata, which does not contain the data root field. For details, see Description on Metadata Parsing Path Configuration.
Time Zone	Select the time zone used by the MQS data source so that ROMA Connect can identify the data timestamps.

Description on Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b.c, and Parsing Path of element d must be set to b.d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to c, and Parsing Path of element d must be set to d.
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b[i].c, and Parsing Path of element d must be set to b[i].d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to [i].c, and Parsing Path of element d must be set to [i].d.

The preceding JSON data that does not contain arrays is used as an example. The following describes the configuration when the source is MQS:

Figure 11 MQS configuration example
Click to enlarge

After configuring the source information, proceed with Configuring Destination Information.

MRS Hive

Back to Overview

If Integration Mode is set to Scheduled, MRS Hive can be selected as the source data source.

On the Create Task page, configure source information.

**Table 18** MRS Hive information at the source
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the MRS Hive data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select MRS Hive.
Data Source Name	Select the MRS Hive data source that you configured in Connecting to Data Sources.
Database Name	Select the database in the MRS Hive data source. NOTE: You need to use a self-built database instead of the default database of MRS Hive.
Table	Select the data table from which data is to be obtained in the MRS Hive data source. Then, click Select Table Field and select only the column fields that you want to integrate and synchronize.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For the first scheduling, the data between the initial timestamp and the current scheduling time is collected. For subsequent scheduling, the data between the last successful collection time and the current time is collected.
Timestamp Field	Mandatory when Incremental Migration is enabled. Select a field of the DATE, TIME, or TIMESTAMP type in the data table as the timestamp of source data to determine whether the data meets the incremental integration conditions.
Time Zone	Mandatory when Incremental Migration is enabled. Select the time zone used by the MRS Hive data source so that ROMA Connect can identify the data timestamps.
Timestamp Initial Value	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
Reset Initial Migration Time	This parameter can be set only when you edit an FDI task. This parameter specifies whether to enable the reset of the initial migration time. Enable: The start time of each scheduling task during incremental migration is the time specified by Timestamp Initial Value. Disable: The start time of each scheduling task during incremental migration is the end time of the previous task.
Compensation Period (ms)	Mandatory when Incremental Migration is enabled. This parameter specifies the period of time (in milliseconds) which will be used to compensate for any delay in data generation at the source when ROMA Connect queries incremental data. The end time for obtaining data is the current system time minus the value you specify here. For example, if the end time of the previous incremental migration task is 15:05, the current scheduled task is triggered at 17:00, and Compensation Period (ms) is set to 100, the time range of data to be integrated in the current incremental migration task is 15:05 to (17:00 – 100 ms).
Filter	Add filter criteria for data to be integrated. Only the source data that meets the filter criteria will be integrated to the destination. For example, the condition and \| col02 \| equal \| A means that only the data records whose values are A in the col02 column will be integrated.

Incremental migration does not support synchronization of physical deletion operations on source data tables to the destination. Logical deletion is recommended at the source.
For data changes at the source, the timestamp of data rows needs to be updated synchronously. ROMA Connect compares the timestamp with the task execution time to identify the data to be incrementally migrated.

The configuration when the source is MRS Hive is similar to that when the source is MySQL. For details, see MySQL configuration example.

After configuring the source information, proceed with Configuring Destination Information.

MRS HDFS

Back to Overview

If Integration Mode is set to Scheduled, MRS HDFS can be selected as the source data source.

On the Create Task page, configure source information.

**Table 19** MRS HDFS source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the MRS HDFS data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select MRS HDFS.
Data Source Name	Select the MRS HDFS data source that you configured in Connecting to Data Sources.
Separator	Enter the field separator for the text data in the MRS HDFS data source. The separator is used to distinguish different fields in each row of data.
Storage Subpath	Enter the path of the data to be integrated in the hdfs:///hacluster directory of MRS HDFS.
Storage Block Size (M)	Select the size of data to be read each time ROMA Connect obtains data from the MRS HDFS data source.
Storage Type	Select the storage type of data in the MRS HDFS data source. The value must be the same as the actual data storage type of the MRS HDFS data source. The TEXT type is supported.
Parse	This parameter specifies whether ROMA Connect parses the obtained source data. If you select Yes, ROMA Connect parses the obtained source data based on the configured parsing rules and then integrates the data to the destination. If you select No, ROMA Connect transparently transmits the obtained source data and integrates the data to the destination.
Metadata	This parameter is mandatory only if Parse is set to Yes. This parameter specifies each data field in text data obtained from the source to be integrated to the destination. Metadata must be filled according to the sequence of the fields in the file. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the source data.

The following describes the configuration when the source is MRS HDFS. The id, name, and info fields are obtained from the MRS HDFS data source and need to be integrated to the destination.

Figure 12 MRS HDFS configuration example
Click to enlarge

After configuring the source information, proceed with Configuring Destination Information.

MRS HBase

Back to Overview

If Integration Mode is set to Scheduled, MRS HBase can be selected as the source data source.

On the Create Task page, configure source information.

**Table 20** MRS HBase information at the source
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the MRS HBase data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select MRS HBase.
Data Source Name	Select the MRS HBase data source that you configured in Connecting to Data Sources.
Table	Select the data table in the MRS HBase data source. Then, click Select Table Field and select only the column fields that you want to integrate and synchronize.

After configuring the source information, proceed with Configuring Destination Information.

MRS Kafka

Back to Overview

If Integration Mode is set to Real-Time, MRS Kafka can be selected as the source data source.

On the Create Task page, configure source information.

**Table 21** MRS Kafka source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the MRS Kafka data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select MRS Kafka.
Data Source Name	Select the MRS Kafka data source that you configured in Connecting to Data Sources.
Topic Name	Set this parameter to a topic that has been created in MRS Kafka.
Parse	This parameter specifies whether ROMA Connect parses the obtained source data. If you select Yes, ROMA Connect parses the obtained source data based on the configured parsing rules and then integrates the data to the destination. If you select No, ROMA Connect transparently transmits the obtained source data and integrates the data to the destination.
Data Root Field	This parameter is mandatory only if Parse is set to Yes. This parameter specifies the path of the upper-layer common fields among all metadata in the data obtained from the source in JSON format. Data Root Field and Parsing Path in Metadata form a complete metadata path. For details, see Description on Metadata Path Configuration.
Data Type	Select the format of data obtained from the MRS Kafka data source, which must be consistent with that of the data stored in MRS Kafka. For Parse set to Yes: JSON or XML For Parse set to No: JSON, XML, Binary file
Offset	Select whether to integrate the earliest message data or the latest message data.
Metadata	This parameter is mandatory only if Parse is set to Yes. This parameter specifies each underlying key-value data element that is obtained from the source in JSON format and needs to be integrated to the destination. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the source data. Parsing Path: path of the metadata, which does not contain the data root field. For details, see Description on Metadata Parsing Path Configuration.
Time Zone	Select the time zone used by the MRS Kafka data source so that ROMA Connect can identify the data timestamps.

Description on Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b.c, and Parsing Path of element d must be set to b.d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to c, and Parsing Path of element d must be set to d.
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b[i].c, and Parsing Path of element d must be set to b[i].d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to [i].c, and Parsing Path of element d must be set to [i].d.

The configuration when the source is MRS Kafka is similar to that when the source is Kafka. For details, see Kafka configuration example.

After configuring the source information, proceed with Configuring Destination Information.

OBS

Back to Overview

If Integration Mode is set to Scheduled, OBS can be selected as the source data source.

On the Create Task page, configure source information.

**Table 22** OBS source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the OBS data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select OBS.
Data Source Name	Select the OBS data source that you configured in Connecting to Data Sources.
Path	Enter the path of the data files to be obtained in the OBS data source. The path cannot end with a slash (/) or be set to the root directory of an OBS bucket.
File Name Prefix	Enter a file name prefix. This parameter is used together with Time Format to filter the data files to be integrated.
Time Format	Select the time format to be used in the file name. This parameter is used together with File Name Prefix to filter the data files to be integrated.
Parse	This parameter specifies whether ROMA Connect parses the obtained source data. If you select Yes, ROMA Connect parses the files based on the default sequence of OBS. By default, the file with the latest time is parsed first and then integrated into the destination. If you select No, ROMA Connect transparently transmits the obtained source data and integrates the data to the destination.
File Type	This parameter is mandatory only if Parse is set to Yes. Select the data file format of the OBS data source. The file format can be CSV, TXT, ZIP, XLS, or XLSX.
Field Separator	This parameter is mandatory only if Parse is set to Yes. Enter the field separator for the data file to distinguish different fields in each row of data.
Encoding Format	This parameter is mandatory only if Parse is set to Yes. Select the encoding mode of data files obtained from the OBS data source. The value can be UTF-8 or GBK.
Skip Title	This parameter is mandatory only if Parse is set to Yes. This parameter specifies whether to skip the title lines in the data file. The title is the first line or several lines at the beginning of a file, which helps identify and distinguish the file content.
Title Lines	This parameter is mandatory only if Parse is set to Yes and Skip Title is set to Yes. Enter the number of rows in the title information in the data file so that ROMA Connect can identify the start row of the data in the file.
Metadata	This parameter is mandatory only if Parse is set to Yes. This parameter specifies each data field in a data file obtained from the source to be integrated to the destination. Metadata must be filled according to the sequence of the fields in the file. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the source data. The following types are supported: String Double (floating point number) Date Boolean Long (integer) Select String if you are not sure about the data type.

The following describes the configuration when the source is OBS. The id, name, and info fields are obtained from the OBS data source and need to be integrated to the destination.

Figure 13 OBS configuration example
Click to enlarge

After configuring the source information, proceed with Configuring Destination Information.

Oracle

Back to Overview

If Integration Mode is set to Scheduled, Oracle can be selected as the source data source.

On the Create Task page, configure source information.

**Table 23** Oracle source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the Oracle data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select Oracle.
Data Source Name	Select the Oracle data source that you configured in Connecting to Data Sources.
Insert SQL	This parameter specifies whether SQL statements are used to obtain source data. If Insert SQL is enabled, ROMA Connect obtains source data based on the entered SQL statements. If Insert SQL is disabled, ROMA Connect obtains source data based on the conditions configured on the GUI. If Insert SQL is enabled, you need to enter an SQL statement for querying data. The statement must be a SELECT statement containing WHERE. INSERT, UPDATE, DELETE, and DROP statements cannot be used. Click Check SQL to check the validity of the statement. For example, if you enter the statement SELECT col01, col02 FROM table01 WHERE col02 IN('A', 'B', 'C') ORDER BY col01, ROMA Connect selects the col01 and col02 columns from table01, displays data records whose values are A, B, or C in the col02 column, and sorts the data records in sequence based on the values in the col01 column.
Table	Mandatory when Insert SQL is disabled. Select the data table from which data is to be obtained in the Oracle data source. Then, click Select Table Field and select only the column fields that you want to integrate and synchronize.
Field Sorting	Mandatory when Insert SQL is disabled. This parameter specifies how the data to be integrated is sorted by field. To sort the integrated data, select a reference field and select Ascending or Descending.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For first-time scheduling, data between the initial timestamp and the specified timestamp is collected. The subsequent collection starts from the time when the last data record is imported in the previous collection to the specified timestamp.
Time Zone	Mandatory when Incremental Migration is enabled. Select the time zone used by the Oracle data source so that ROMA Connect can identify the data timestamps.
Timestamp Field	This parameter is mandatory only if Incremental Migration is enabled and Insert SQL is disabled. Select a field of the DATE, TIME, or TIMESTAMP type in the data table as the timestamp of source data to determine whether the data meets the incremental integration conditions.
Timestamp Initial Value	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
Reset Initial Migration Time	This parameter can be set only when you edit an FDI task. This parameter specifies whether to enable the reset of the initial migration time. Enable: The start time of each scheduling task during incremental migration is the time specified by Timestamp Initial Value. Disable: The start time of each scheduling task during incremental migration is the end time of the previous task.
Filter	Mandatory when Insert SQL is disabled. Add filter criteria for data to be integrated. Only the source data that meets the filter criteria will be integrated to the destination. For example, the condition and \| col02 \| equal \| A means that only the data records whose values are A in the col02 column will be integrated.
Extended Metadata	Mandatory when Insert SQL is disabled. This parameter is mandatory if the value of a field in the database is in JSON format and the underlying key-value data elements in the JSON field value need to be collected. Field Name: name of the data field whose child elements need to be collected in the table. Type: data type of the data element to be collected in the JSON field value. Parsing Path: complete path of the data element in the JSON field value. For details, see Description on Extended Metadata Parsing Path Configuration.

Description on Extended Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```

Incremental migration does not support synchronization of physical deletion operations on source data tables to the destination. Logical deletion is recommended at the source.
For data changes at the source, the timestamp of data rows needs to be updated synchronously. ROMA Connect compares the timestamp with the task execution time to identify the data to be incrementally migrated.

The configuration when the source is Oracle is similar to that when the source is MySQL. For details, see MySQL configuration example.

After configuring the source information, proceed with Configuring Destination Information.

PostgreSQL

Back to Overview

If Integration Mode is set to Scheduled, PostgreSQL/openGauss can be selected as the source data source.

On the Create Task page, configure source information.

**Table 24** PostgreSQL/openGauss source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the PostgreSQL/openGauss data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select PostgreSQL/openGauss.
Data Source Name	Select the PostgreSQL/openGauss data source that you configured in Connecting to Data Sources.
Insert SQL	This parameter specifies whether SQL statements are used to obtain source data. If Insert SQL is enabled, ROMA Connect obtains source data based on the entered SQL statements. If Insert SQL is disabled, ROMA Connect obtains source data based on the conditions configured on the GUI. If Insert SQL is enabled, you need to enter an SQL statement for querying data. The statement must be a SELECT statement containing WHERE. INSERT, UPDATE, DELETE, and DROP statements cannot be used. Click Check SQL to check the validity of the statement. For example, if you enter the statement SELECT col01, col02 FROM table01 WHERE col02 IN('A', 'B', 'C') ORDER BY col01, ROMA Connect selects the col01 and col02 columns from table01, displays data records whose values are A, B, or C in the col02 column, and sorts the data records in sequence based on the values in the col01 column.
Table	Mandatory when Insert SQL is disabled. Select the data table from which data is to be obtained in the PostgreSQL/openGauss data source. Then, click Select Table Field and select only the column fields that you want to integrate and synchronize.
Field Sorting	Mandatory when Insert SQL is disabled. This parameter specifies how the data to be integrated is sorted by field. To sort the integrated data, select a reference field and select Ascending or Descending.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For first-time scheduling, data between the initial timestamp and the specified timestamp is collected. The subsequent collection starts from the time when the last data record is imported in the previous collection to the specified timestamp.
Time Zone	Mandatory when Incremental Migration is enabled. Select the time zone used by the PostgreSQL/openGauss data source so that ROMA Connect can identify the data timestamps.
Timestamp Field	This parameter is mandatory only if Incremental Migration is enabled and Insert SQL is disabled. Select a field of the DATE, TIME, or TIMESTAMP type in the data table as the timestamp of source data to determine whether the data meets the incremental integration conditions.
Timestamp Initial Value	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
Reset Initial Migration Time	This parameter can be set only when you edit an FDI task. This parameter specifies whether to enable the reset of the initial migration time. Enable: The start time of each scheduling task during incremental migration is the time specified by Timestamp Initial Value. Disable: The start time of each scheduling task during incremental migration is the end time of the previous task.
Filter	Mandatory when Insert SQL is disabled. Add filter criteria for data to be integrated. Only the source data that meets the filter criteria will be integrated to the destination. For example, the condition and \| col02 \| equal \| A means that only the data records whose values are A in the col02 column will be integrated.
Extended Metadata	Mandatory when Insert SQL is disabled. This parameter is mandatory if the value of a field in the database is in JSON format and the underlying key-value data elements in the JSON field value need to be collected. Field Name: name of the data field whose child elements need to be collected in the table. Type: data type of the data element to be collected in the JSON field value. Parsing Path: complete path of the data element in the JSON field value. For details, see Description on Extended Metadata Parsing Path Configuration.

Description on Extended Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```

Incremental migration does not support synchronization of physical deletion operations on source data tables to the destination. Logical deletion is recommended at the source.
For data changes at the source, the timestamp of data rows needs to be updated synchronously. ROMA Connect compares the timestamp with the task execution time to identify the data to be incrementally migrated.

The configuration when the source is PostgreSQL/openGauss is similar to that when the source is MySQL. For details, see MySQL configuration example.

After configuring the source information, proceed with Configuring Destination Information.

RabbitMQ

Back to Overview

If Integration Mode is set to Real-Time, RabbitMQ can be selected as the source data source.

On the Create Task page, configure source information.

**Table 25** RabbitMQ source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the RabbitMQ data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select RabbitMQ.
Data Source Name	Select the RabbitMQ data source that you configured in Connecting to Data Sources.
New Queue Creation	This parameter specifies whether to create a queue in the RabbitMQ data source. If this parameter is set to Yes, a new queue is created and data is obtained from the new queue. If this parameter is set to No, data is obtained from the existing queue.
Exchange Mode	This parameter is mandatory only if New Queue Creation is set to Yes. Select a routing mode for the exchange in RabbitMQ to forward messages to the new queue. Direct: If the routing key in a message fully matches the queue, the message will be forwarded to the queue. Topic: If the routing key in a message approximately matches the queue, the message will be forwarded to the queue. Fanout: All messages will be forwarded to the queue. Headers: If the Headers attribute of a message fully matches the queue, the message will be forwarded to the queue.
Exchange Name	This parameter is mandatory only if New Queue Creation is set to Yes. Enter the exchange name of the new queue in RabbitMQ.
Routing Key	This parameter is mandatory only if Exchange Mode is set to Direct or Topic. Enter the routing key of the new queue. RabbitMQ uses the routing key as a condition and forwards messages that meet the condition to the new queue.
Message Parameters	This parameter is mandatory only if Exchange Mode is set to Headers. Enter the Headers key-value pair of the new queue. RabbitMQ uses Headers as a condition and forwards messages that meet the condition to the new queue.
Queue Name	Enter the name of the message queue whose data is to be obtained. If New Queue Creation is set to Yes, enter a new queue name. If New Queue Creation is set to No, enter the name of an existing queue in the RabbitMQ data source.
Automatic Deletion	This parameter specifies whether a queue will be automatically deleted if no client is connected.
Persistence	This parameter specifies whether messages in a queue are stored permanently.
Parse	This parameter specifies whether ROMA Connect parses the obtained source data. If you select Yes, ROMA Connect parses the obtained source data based on the configured parsing rules and then integrates the data to the destination. If you select No, ROMA Connect transparently transmits the obtained source data and integrates the data to the destination.
Metadata	This parameter is mandatory only if Parse is set to Yes. This parameter specifies each underlying key-value data element that is obtained from the source in JSON format and needs to be integrated to the destination. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the source data. Parsing Path: full path of metadata. For details, see Description on Metadata Parsing Path Configuration.

Description on Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```

The preceding JSON data that does not contain arrays is used as an example. The following describes the configuration when the source is RabbitMQ:

Figure 14 RabbitMQ configuration example
Click to enlarge

After configuring the source information, proceed with Configuring Destination Information.

RocketMQ

Back to Overview

If Integration Mode is set to Real-Time, RocketMQ can be selected as the source data source.

The RocketMQ consumer group is named in the format of task ID+_consumer_group.

On the Create Task page, configure source information.

**Table 26** RocketMQ source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the RocketMQ data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select RocketMQ.
Data Source Name	Select the RocketMQ data source that you configured in Connecting to Data Sources.
Topic Name	Enter the topic character string or queue name.
Parse	This parameter specifies whether ROMA Connect parses the obtained source data. If you select Yes, ROMA Connect parses the obtained source data based on the configured parsing rules and then integrates the data to the destination. If you select No, ROMA Connect transparently transmits the obtained source data and integrates the data to the destination.
Data Root Field	This parameter is mandatory only if Parse is set to Yes. This parameter specifies the path of the upper-layer common fields among all metadata in the data obtained from the source in JSON format. Data Root Field and Parsing Path in Metadata form a complete metadata path. For details, see Description on Metadata Parsing Path Configuration.
Metadata	This parameter is mandatory only if Parse is set to Yes. This parameter specifies each underlying key-value data element that is obtained from the source in JSON format and needs to be integrated to the destination. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the source data. Parsing Path: full path of metadata. For details, see Description on Metadata Parsing Path Configuration.

Description on Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```

The following figure describes the configuration when the source is RocketMQ.

Figure 15 RocketMQ configuration example
Click to enlarge

After configuring the source information, proceed with Configuring Destination Information.

SAP

Back to Overview

If Integration Mode is set to Scheduled, SAP can be selected as the source data source.

On the Create Task page, configure source information.

**Table 27** SAP source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the SAP data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select SAP.
Data Source Name	Select the SAP data source that you configured in Connecting to Data Sources.
Connection Method	Select the connection mode of the SAP data source. Available Function: specifies the functions that are opened by the SAP data source for remote calling. Table Connector: connects to SAP data tables through a connector.
Available Function	This parameter is mandatory only if Connection Mode is set to Available Function. Click Query. In the Available Functions dialog box, enter the function name and group name to query and obtain the list of available functions of the SAP data source. In the Function List area, select the function you want to use and click OK. NOTE: The available functions can be queried on ROMA Connect only if the function that allows remote access has been enabled in the SAP data source. The function name and group name support fuzzy search by prefix.
Query Function	This parameter is mandatory only if Connection Mode is set to Table Connector. The default function is used generally. You can also select another query function as required, for example, /SAPDS/RFC_READ_TABLE2. NOTICE: SAP does not support cross-table query because the /SAPDS/RFC_READ_TABLE2 function can only be called to query one table.
Table	This parameter is mandatory only if Connection Mode is set to Table Connector. Select a source data table. After selecting a data table, click Select Table Field to select specific table fields.
Field Separator	This parameter is mandatory only if Connection Mode is set to Table Connector. Enter a field separator to separate fields.
Page Size	This parameter is mandatory only if Connection Mode is set to Table Connector. Specify the maximum number of data records that can be read from the SAP server at a time, in 10,000. NOTICE: A larger value indicates a higher read speed but may cause memory overflow, in which case the entire instance will fail to provide services. Retain the default value, or configure the parameter based on the actual service volume and instance specifications. In addition, to reduce the risk of memory overflow, set the number of concurrent tasks to be less than or equal to 5.
Incremental Migration	This parameter is mandatory only if Connection Mode is set to Table Connector. This parameter specifies whether only data generated in a specific period is integrated. For the first scheduling, the data between the initial timestamp and the current scheduling time is collected. For subsequent scheduling, the data between the last successful collection time and the current time is collected.
Partition Options	This parameter is mandatory only if Connection Mode is set to Table Connector. Configure data read by partition to increase read speed. Data can be read in partitions by year, month, or day, or read without partitions. For example, if you want to read data from the first day to the 30th day of a month, set this parameter to Day and data is grouped into partitions by these 30 days monthly. NOTE: Both full and incremental migration support partitioning.
Time Zone	This parameter is mandatory only if Incremental Migration is enabled, or if Partition Options is set to Day, Month, or Year. Select the time zone used by the SAP data source so that ROMA Connect can identify the data timestamps.
Timestamp Field	This parameter is mandatory only if Incremental Migration is enabled, or if Partition Options is set to Day, Month, or Year. Select a field of the DATE type in the data table as the timestamp of source data to determine whether the data meets the incremental integration conditions. If the values of Timestamp and Timestamp Initial Value are incomplete, full integration is used by default. NOTICE: SAP does not support incremental integration without the timestamp fields. That is, the table fields to be extracted must contain the timestamp fields.
Timestamp Initial Value	This parameter is mandatory only if Incremental Migration is enabled, or if Partition Options is set to Day, Month, or Year. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
End Timestamp	Mandatory when Incremental Migration is enabled. This parameter specifies the end time of the data to be integrated. That is, only the data generated before this time point will be integrated.
Time Format	This parameter is mandatory only if Incremental Migration is enabled, or if Partition Options is set to Day, Month, or Year. Select the time format to be used in the file name. This parameter is used together with File Name Prefix to filter the data files to be integrated. If the time format is selected and the file name prefix is not specified, files will not be filtered. After you select a time format, the task execution time will be converted into the selected time format. For example, if File Name Prefix is test, Time Format is yyyyMMdd, and the task execution time is 2021-01-14 10:00:00, data will be collected only from the files whose name prefix is test20210114.
Compensation Period (ms)	Mandatory when Incremental Migration is enabled. This parameter specifies the period of time (in milliseconds) which will be used to compensate for any delay in data generation at the source when ROMA Connect queries incremental data. The end time for obtaining data is the current system time minus the value you specify here. For example, if the end time of the previous incremental migration task is 15:05, the current scheduled task is triggered at 17:00, and Compensation Period (ms) is set to 100, the time range of data to be integrated in the current incremental migration task is 15:05 to (17:00 – 100 ms).
Filter	This parameter is mandatory only if Connection Mode is set to Table Connector. Add filter criteria for data to be integrated. Only the source data that meets the filter criteria will be integrated to the destination. For example, the condition and \| col02 \| equal \| A means that only the data records whose values are A in the col02 column will be integrated.

Description on Configuration of Table-type Request Parameters:

The following request parameter structure is used as an example. [] indicates the table-type data, and {} indicates a row of data in the table. There are three rows of data in a parameter table, and each row of data contains two parameters.

The key of parameter b is a[i].b, and the key of parameter c is a[i].c. i indicates the row number of the parameter in the parameter table. The value 0 indicates the first row, the value 1 indicates the second row, and so on.

a: [
   {b: xx, c: xx},
   {b: yy, c: yy},
   {b: zz, c: zz},
]

After you select an available function, only the parameters in the first row of a data table are displayed, that is, a[0].b and a[0].c. If you need to add the parameters of a row in the data table, add the parameters row by row in sequence. Ensure that the parameters of the previous row have been added. You are not allowed to add the parameters of only row 1 and row 3 without adding the parameters of row 2.

For example, in the preceding example, if you want to add the parameter in the second row, add a[1].b or a[1].c after a[0].c. To add the parameter in the third row, add at least one of parameters a[1].b and a[1].c in the second row after a[0].c, and then add parameter a[2].b or a[2].c in the third row.

The same parameter in the same row of data cannot be added repeatedly. Otherwise, the data integration task fails to be executed.

After configuring the source information, proceed with Configuring Destination Information.

SNMP

Back to Overview

If Integration Mode is set to Scheduled, SNMP can be selected as the source data source.

On the Create Task page, configure source information.

**Table 28** SNMP source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the SNMP data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select SNMP.
Data Source Name	Select the SNMP data source that you configured in Connecting to Data Sources.
Data Mode	Select a data mode from the SNMP data source. If you select Row for Data Mode, SNMP data is parsed by row. If you select Column for Data Mode, SNMP data is parsed by column.
Object Identifier	This parameter is mandatory only if Data Mode is set to Column. Enter the OID of the object to be obtained from the SNMP message. Separate multiple OIDs with commas (,).
Metadata	This parameter specifies the object (data field) to be integrated to the destination from the SNMP message data obtained from the source. Alias: user-defined metadata name. Type: data type of metadata. Parsing Path: OID of metadata in the SNMP message. This parameter is mandatory only if Data Mode is set to Row.

The following describes the configuration when the source is SNMP:

When Data Model is set to Row, user indicates obtaining system contact information and location indicates obtaining server location information.
Figure 16 Configuration example when the data model is Row
When Data Model is set to Column, the info column contains the system contact information and server location information.
Figure 17 Configuration example when the data model is Column

After configuring the source information, proceed with Configuring Destination Information.

SQL Server

Back to Overview

If Integration Mode is set to Scheduled, SQL Server can be selected as the source data source.

On the Create Task page, configure source information.

**Table 29** SQL Server source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the SQL Server data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select SQL Server.
Data Source Name	Select the SQL Server data source that you configured in Connecting to Data Sources.
Insert SQL	This parameter specifies whether SQL statements are used to obtain source data. If Insert SQL is enabled, ROMA Connect obtains source data based on the entered SQL statements. If Insert SQL is disabled, ROMA Connect obtains source data based on the conditions configured on the GUI. If Insert SQL is enabled, you need to enter an SQL statement for querying data. The statement must be a SELECT statement containing WHERE. INSERT, UPDATE, DELETE, and DROP statements cannot be used. Click Check SQL to check the validity of the statement. For example, if you enter the statement SELECT col01, col02 FROM table01 WHERE col02 IN('A', 'B', 'C') ORDER BY col01, ROMA Connect selects the col01 and col02 columns from table01, displays data records whose values are A, B, or C in the col02 column, and sorts the data records in sequence based on the values in the col01 column.
Table	Mandatory when Insert SQL is disabled. Select the data table in the SQL Server data source. Then, click Select Table Field and select only the column fields that you want to integrate and synchronize.
Field Sorting	Mandatory when Insert SQL is disabled. This parameter specifies how the data to be integrated is sorted by field. To sort the integrated data, select a reference field and select Ascending or Descending.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For first-time scheduling, data between the initial timestamp and the specified timestamp is collected. The subsequent collection starts from the time when the last data record is imported in the previous collection to the specified timestamp.
Time Zone	Mandatory when Incremental Migration is enabled. Select the time zone used by the SQL Server data source so that ROMA Connect can identify the data timestamps.
Timestamp Field	This parameter is mandatory only if Incremental Migration is enabled and Insert SQL is disabled. Select a field of the DATE, TIME, or TIMESTAMP type in the data table as the timestamp of source data to determine whether the data meets the incremental integration conditions.
Timestamp Initial Value	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
Reset Initial Migration Time	This parameter can be set only when you edit an FDI task. This parameter specifies whether to enable the reset of the initial migration time. Enable: The start time of each scheduling task during incremental migration is the time specified by Timestamp Initial Value. Disable: The start time of each scheduling task during incremental migration is the end time of the previous task.
Filter	Mandatory when Insert SQL is disabled. Add filter criteria for data to be integrated. Only the source data that meets the filter criteria will be integrated to the destination. For example, the condition and \| col02 \| equal \| A means that only the data records whose values are A in the col02 column will be integrated.
Extended Metadata	Mandatory when Insert SQL is disabled. This parameter is mandatory if the value of a field in the database is in JSON format and the underlying key-value data elements in the JSON field value need to be collected. Field Name: name of the data field whose child elements need to be collected in the table. Type: data type of the data element to be collected in the JSON field value. Parsing Path: complete path of the data element in the JSON field value. For details, see Description on Extended Metadata Parsing Path Configuration.

Description on Extended Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```

Incremental migration does not support synchronization of physical deletion operations on source data tables to the destination. Logical deletion is recommended at the source.
For data changes at the source, the timestamp of data rows needs to be updated synchronously. ROMA Connect compares the timestamp with the task execution time to identify the data to be incrementally migrated.

The configuration when the source is SQL Server is similar to that when the source is MySQL. For details, see MySQL configuration example.

After configuring the source information, proceed with Configuring Destination Information.

GaussDB(for MySQL)

Back to Overview

If Integration Mode is set to Scheduled, GaussDB(for MySQL) can be selected as the source data source.

On the Create Task page, configure source information.

**Table 30** GaussDB(for MySQL) source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the GaussDB(for MySQL) data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select GaussDB(for MySQL).
Data Source Name	Select the GaussDB(for MySQL) data source that you configured in Connecting to Data Sources.
Insert SQL	This parameter specifies whether SQL statements are used to obtain source data. If Insert SQL is enabled, ROMA Connect obtains source data based on the entered SQL statements. If Insert SQL is disabled, ROMA Connect obtains source data based on the conditions configured on the GUI. If Insert SQL is enabled, you need to enter an SQL statement for querying data. The statement must be a SELECT statement containing WHERE. INSERT, UPDATE, DELETE, and DROP statements cannot be used. Click Check SQL to check the validity of the statement. For example, if you enter the statement SELECT col01, col02 FROM table01 WHERE col02 IN('A', 'B', 'C') ORDER BY col01, ROMA Connect selects the col01 and col02 columns from table01, displays data records whose values are A, B, or C in the col02 column, and sorts the data records in sequence based on the values in the col01 column.
Table	Mandatory when Insert SQL is disabled. Select the data table from which data is to be obtained in the GaussDB(for MySQL) data source. Then, click Select Table Field and select only the column fields that you want to integrate and synchronize.
Field Sorting	Mandatory when Insert SQL is disabled. This parameter specifies how the data to be integrated is sorted by field. To sort the integrated data, select a reference field and select Ascending or Descending.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For first-time scheduling, data between the initial timestamp and the specified timestamp is collected. The subsequent collection starts from the time when the last data record is imported in the previous collection to the specified timestamp.
Time Zone	Mandatory when Incremental Migration is enabled. Select the time zone used by the GaussDB(for MySQL) data source so that ROMA Connect can identify the data timestamps.
Timestamp Field	This parameter is mandatory only if Incremental Migration is enabled and Insert SQL is disabled. Select a DATE field (down to the second) in the data table as the timestamp of the source data to determine whether the data meets the requirements of incremental integration.
Timestamp Initial Value	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
Reset Initial Migration Time	This parameter can be set only when you edit an FDI task. This parameter specifies whether to enable the reset of the initial migration time. Enable: The start time of each scheduling task during incremental migration is the time specified by Timestamp Initial Value. Disable: The start time of each scheduling task during incremental migration is the end time of the previous task.
Filter	Mandatory when Insert SQL is disabled. Add filter criteria for data to be integrated. Only the source data that meets the filter criteria will be integrated to the destination. For example, the condition and \| col02 \| equal \| A means that only the data records whose values are A in the col02 column will be integrated.
Extended Metadata	Mandatory when Insert SQL is disabled. This parameter is mandatory if the value of a field in the database is in JSON format and the underlying key-value data elements in the JSON field value need to be collected. Field Name: name of the data field whose child elements need to be collected in the table. Type: data type of the data element to be collected in the JSON field value. Parsing Path: complete path of the data element in the JSON field value. For details, see Description on Extended Metadata Parsing Path Configuration.

Description on Extended Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```

Incremental migration does not support synchronization of physical deletion operations on source data tables to the destination. Logical deletion is recommended at the source.
For data changes at the source, the timestamp of data rows needs to be updated synchronously. ROMA Connect compares the timestamp with the task execution time to identify the data to be incrementally migrated.

The preceding JSON data that contains arrays is used as an example. The following describes the configuration for the GaussDB(for MySQL) destination.

In the example of incremental migration configuration, the data table must contain a field of the DATE type as the timestamp field.
Figure 18 Incremental migration configuration example of GaussDB(for MySQL)
In the example of extended metadata configuration, the child elements c and d are obtained from the desc field in the data table.
Figure 19 GaussDB(for MySQL) extended metadata configuration example

After configuring the source information, proceed with Configuring Destination Information.

WebSocket

Back to Overview

If Integration Mode is set to Real-Time, WebSocket can be selected as the source data source.

On the Create Task page, configure source information.

**Table 31** WebSocket source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the WebSocket data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select WebSocket.
Data Source Name	Select the WebSocket data source that you configured in Connecting to Data Sources.
Data Root Field	This parameter specifies the path of the upper-layer common fields among all metadata in the data obtained from the source in JSON format. Data Root Field and Parsing Path in Metadata form a complete metadata path. For details, see Description on Metadata Path Configuration.
Heartbeat Mechanism	This parameter specifies whether the heartbeat mechanism is enabled between ROMA Connect and the WebSocket data source to ensure connection validity.
Heartbeat Message	This parameter is mandatory only if Heartbeat Mechanism is set to Yes. This parameter specifies the message content carried in heartbeat packets sent by ROMA Connect to the WebSocket data source.
Heartbeat Sending Interval	This parameter is mandatory only if Heartbeat Mechanism is set to Yes. This parameter specifies the interval for the ROMA Connect to send heartbeat packets to the WebSocket data source.
Parse	This parameter specifies whether ROMA Connect parses the obtained source data. If you select Yes, ROMA Connect parses the obtained source data based on the configured parsing rules and then integrates the data to the destination. If you select No, ROMA Connect transparently transmits the obtained source data and integrates the data to the destination.
Metadata	This parameter is mandatory only if Parse is set to Yes. This parameter specifies each underlying key-value data element that is obtained from the source in JSON format and needs to be integrated to the destination. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the source data. Parsing Path: path of the metadata, which does not contain the data root field. For details, see Description on Metadata Parsing Path Configuration.

Description on Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b.c, and Parsing Path of element d must be set to b.d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to c, and Parsing Path of element d must be set to d.
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```
In this scenario, three configuration solutions are available for Data Root Field and Parsing Path:
- Data Root Field is not specified.
  Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
- Data Root Field is set to a.
  Parsing Path starts from the underlying path of element a. Parsing Path of element c must be set to b[i].c, and Parsing Path of element d must be set to b[i].d.
- Data Root Field is set to a.b.
  Parsing Path starts from the underlying path of element b. Parsing Path of element c must be set to [i].c, and Parsing Path of element d must be set to [i].d.

The preceding JSON data that does not contain arrays is used as an example. The following describes the configuration when the source is WebSocket:

Figure 20 WebSocket configuration example
Click to enlarge

After configuring the source information, proceed with Configuring Destination Information.

Custom

Back to Overview

If Integration Mode is set to Scheduled, custom data sources can be selected at the source. ROMA Connect communicates with custom data sources through standard RESTful APIs.

On the Create Task page, configure source information.

**Table 32** Custom data source information
Parameter	Description
Instance	Select the ROMA Connect instance that is being used.
Integration Application	Select the integration application to which the custom data source belongs. Ensure that the integration application has been configured in Connecting to Data Sources.
Data Source Type	Select a custom data source type.
Data Source Name	Select the custom data source that you configured in Connecting to Data Sources.
Paging	Specifies whether data is returned on multiple pages when ROMA Connect sends a request to the custom data source to obtain data. Multiple data records can be returned for one API request. If Paging is enabled, all data that meets the conditions is displayed on multiple pages based on a fixed number of records on each page. ROMA Connect obtains data on one page each time and obtains all data by sending multiple requests. If Paging is disabled, ROMA Connect obtains all data that meets the conditions through one API request.
Start Page	This parameter is mandatory only if Paging is enabled. Enter the start page number for paging. The value can be 0 or 1. Set this parameter based on the original definition of the API.
Page Size	This parameter is mandatory only if Paging is enabled. Enter the maximum number of data records on each page. Set this parameter based on the original definition of the API.
Incremental Migration	This parameter specifies whether only data generated in a specific period is integrated. For the first scheduling, the data between the initial timestamp and the current scheduling time is collected. For subsequent scheduling, the data between the last successful collection time and the current time is collected.
Time Zone	Select the time zone used by the custom data source so that ROMA Connect can identify the data timestamps.
Start Time	Mandatory when Incremental Migration is enabled. This parameter specifies the time at which data is to be integrated for the first time. That is, only the data generated after this time point will be integrated.
Parse	This parameter specifies whether ROMA Connect parses the obtained source data. If you select Yes, ROMA Connect parses the obtained source data based on the configured parsing rules and then integrates the data to the destination. If you select No, ROMA Connect transparently transmits the obtained source data and integrates the data to the destination.
Metadata	This parameter is mandatory only if Parse is set to Yes. This parameter specifies each underlying key-value data element that is obtained from the source in JSON format and needs to be integrated to the destination. Alias: user-defined metadata name. Type: data type of metadata. The value must be the same as the data type of the corresponding parameter in the response. Parsing Path: full path of metadata. For details, see Description on Metadata Parsing Path Configuration.
In addition to the preceding parameters, different custom data sources define different reader parameters. Set the parameters based on the original definition specifications of the connector. You can locate the connector used by the custom data source on the Assets page of the ROMA Connect console and view the reader parameter definition of the connector.

Description on Metadata Parsing Path Configuration

Data in JSON format does not contain arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b.c, and a.b.d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b.c, and Parsing Path of element d must be set to a.b.d.
```
{
   "a": {
      "b": {
         "c": "xx",
         "d": "xx"
      }
   }
}
```
Data in JSON format contains arrays:
For example, in the following JSON data, the complete paths for elements a to d are a, a.b, a.b[i].c, and a.b[i].d, respectively. Elements c and d are underlying data elements, that is, the data to be integrated to the destination.

In this scenario, Parsing Path of element c must be set to a.b[i].c, and Parsing Path of element d must be set to a.b[i].d.
```
{
   "a": {
      "b": [{
         "c": "xx",
         "d": "xx"
      },
      {
         "c": "yy",
         "d": "yy"
      }
      ]
   }
}
```

The preceding JSON data that does not contain arrays is used as an example. The following figure describes the configuration when the source is a custom data source. In the example, the value of key is the reader parameter defined in the connector.

Figure 21 Custom data source configuration example
Click to enlarge

After configuring the source information, proceed with Configuring Destination Information.

Configuring Source Information

Overview

APIs

ActiveMQ

ArtemisMQ

DB2

DWS

DIS

DM

FTP

Gauss100

HANA

HL7

IBM MQ

Kafka

LDAP

MySQL

MongoDB

MQS

MRS Hive

MRS HDFS

MRS HBase

MRS Kafka

OBS

Oracle

PostgreSQL

RabbitMQ

RocketMQ

SAP

SNMP

SQL Server

GaussDB(for MySQL)

WebSocket

Custom

Feedback

Was this page helpful?