Connecting to a Custom Data Source

Overview

ROMA Connect can use a custom connector as a data source for data integration tasks. Before using a custom data source, you need to connect it to ROMA Connect.

ROMA Connect supports connections of common data source types, such as relational databases, message queues, APIs, NoSQL databases, and OBS. These data sources can be directly used in ROMA Connect. You can integrate your data from the source to the destination only by connecting to data sources and executing data integration tasks.

If ROMA Connect cannot access your data source, develop a plug-in to read and write the data source. Besides, the plug-in should serve as a RESTful API for ROMA Connect to access. Such data source plug-ins are also called connectors, and the data sources accessed through these connectors are custom data sources.

A connector can be directly connected to a data source, which is a data exchange channel between the data source and ROMA Connect. ROMA Connect reads and writes data sources by calling this RESTful API.

After developing and deploying a connector, create it on ROMA Connect and define related parameters for connection.

After creating a connector in ROMA Connect, you need to publish the connector so that ROMA Connect can connect to the deployed connector. ROMA Connect connects to the custom data source by a published connector instance, reading data from and writing data to the data source.

Prerequisites

• Each connected data source must belong to an integration application. Ensure that an integration application is available before connecting a data source, or create one first.
• A connector has been developed and deployed offline. For details about how to develop the API for connecting a connector to ROMA Connect, see RESTful API Specifications of Connectors.

Creating an FDI Connector

1. Log in to the ROMA Connect console. In the navigation pane on the left, choose Assets.
2. Click Create Connector in the upper right corner of the page.
3. On the Create Connector page, enter configuration information about a connector.

   Table 1 Connector configuration

   Parameter	Description
   Name	Enter a connector name. Using naming rules facilitates future search.
   Type	Select the permissions type of the connector for data operations. Reader & Writer: reads and writes data sources. Reader: reads data sources only. Writer: writes data sources only.
   Description	Brief description of the connector.
   Data Source Metadata	Define parameters of the data source. When configuring ROMA Connect to access the data source, you need to set the parameters defined here. The message data exchanged between ROMA Connect and the connector is in JSON format, and the parameters are transferred in the key-value format. Click Add Parameter to add the parameters for accessing the data source. Parameter Name: name of a parameter to be displayed on the data source configuration page. It is only used to identify a parameter and does not take effect in an actual data integration task. Parameter Key: parameter key transferred in a data source connection request. Parameter Value Type: type of the parameter value to be configured on the data source connection page. Text: text box. Select: drop-down list box. Date: date control, which allows you to select a date and time. Textarea: character text box, which supports line breaks. Verification Rule: whether to verify the input parameter values. None No special characters: The system checks whether the parameter value contains special characters. Special characters include digits, letters, hyphens (-), and underscores (_). Default Value: default value to be transferred if a parameter is not specified. If this parameter is left blank, no default value will be transferred. Mandatory: whether the parameter is mandatory.
   Reader Metadata	Reader parameters of a data source. When the data source is selected as the source of a data integration task, the reader parameters defined here must be set in the data integration task. The message data exchanged between ROMA Connect and the connector is in JSON format, and the parameters are transferred in the key-value format. Click Add Parameter to add the reader parameters of the data source. The parameter configuration description is similar to that of Data Source Metadata.
   Writer Metadata	Writer parameters of a data source. When the data source is selected as the destination of a data integration task, the writer parameters defined here must be set in the data integration task. The message data exchanged between ROMA Connect and the connector is in JSON format, and the parameters are transferred in the key-value format. Click Add Parameter to add the writer parameters of the data source. The parameter configuration description is similar to that of Data Source Metadata.

   Figure 1 Example of defining connector parameters
   
4. Click Create.

Publishing an FDI Connector

1. Log in to the ROMA Connect console. In the navigation pane on the left, choose Assets.
2. On the Connectors tab page, click Publish of the connector.
3. In the dialog box that is displayed, set related parameters and click OK.

   Table 2 Connector configuration

   Parameter	Description
   Instance Name	Enter a connector instance name. Using naming rules facilitates future search.
   Connection Address	Access address of the deployed connector.
   Authentication Mode	Select the security authentication mode used for accessing the connector instance. For simplicity, select None. None: Access requests are not authenticated. AppKey: Access requests are authenticated using AppKey and AppSecret.
   AppKey	Enter the AppKey used for connector authentication.
   AppSecret	Enter the AppSecret used for connector authentication.

   

   Only connectors in the Editable state can be edited. Connectors in the Published state cannot be edited.

Connecting to a Custom Data Source

1. Log in to the ROMA Connect console. In the navigation pane on the left, choose Resource. On the displayed page, click the instance to access its console.
2. In the navigation pane on the left, choose Data Sources. In the upper right corner of the page, click Access Data Source.
3. On the Custom tab page, select the connector to be used as the custom data source and click Next.
4. Configure the data source connection information.

   Table 3 Data source connection information

   Parameter	Description
   Data Source Name	Enter a data source name. It is recommended that you enter a name based on naming rules to facilitate search.
   Encoding Format	Default: utf-8
   Integration Application	Select the integration application to which the data source belongs.
   Description	Enter the descriptive information.
   Connector Instance	Select a published connector instance under the connector.
   Other parameters	Set the subsequent configuration items based on the data source information defined when you create the connector. You can find the connector on the Assets page and view the data source metadata of the connector.

   The following is an example of connecting to a custom data source for reading email data.

   The email server, protocol, user, and password are the data source parameters defined in the connector.

   Figure 2 Custom data source configuration example
   
5. Click Create.