On this page

Scenario
Overall Process
Prerequisites
Step 1: Obtain the Floating IP Address, Port Number, and Security Group of an External Data Source
Step 2: Obtain the CIDR Block of the DLI Queue
Step 3: Add a Rule to the Security Group of the External Data Source to Allow Access from the DLI Queue
Step 4: Create an Enhanced Datasource Connection
Step 5: Test Network Connectivity

Show all

Help Center/ Data Lake Insight/ User Guide/ Data Migration and Transmission/ Configuring DLI to Read and Write Data from and to External Data Sources/ Example Typical Scenario: Connecting DLI to a Data Source on a Private Network

Example Typical Scenario: Connecting DLI to a Data Source on a Private Network

Updated on 2024-12-28 GMT+08:00

View PDF

Scenario

Typically, connecting DLI to a data source on a private network means connecting it to a Huawei Cloud service, such as MRS, RDS, CSS, Kafka, or GaussDB(DWS). DLI's enhanced datasource connections use VPC peering connections to directly connect DLI to VPC networks of destination data sources.

This section describes how to connect DLI to a data source on a private network using an enhanced datasource connection.

If the network is disconnected when an enhanced datasource connection is created, you can rectify the fault based on the overall process and procedure in this section.

Overall Process

Figure 1 Configuration process of an enhanced datasource connection

Prerequisites

You have created a queue. For details about how to create a queue, see Creating a Queue.
CAUTION:

The queue billing mode must be Pay-per-use, and Dedicated Resource Mode must be selected after you select a queue type.

Enhanced datasource connections can be created only for pay-per-use resources in dedicated resource mode.

A cluster of the external data source has been created. You can select a data source as needed.

**Table 1** Reference for creating clusters of other data sources
Service Name	Reference Documents
RDS	Buying an RDS for MySQL DB Instance
GaussDB(DWS)	Creating a GaussDB(DWS) Cluster
DMS Kafka	Creating a Kafka Instance CAUTION: When you create the instance, do not enable Kafka SASL_SSL.
CSS	Creating a CSS Cluster
MRS	Creating an MRS Cluster

CAUTION:

The CIDR block of the DLI queue bound with a datasource connection cannot overlap with the CIDR block of other data sources.
Datasource connections cannot be bound with the default queue.

Step 1: Obtain the Floating IP Address, Port Number, and Security Group of an External Data Source

**Table 2** Data source information to be obtained
Data Source	Obtain Method
DMS Kafka	On the Kafka management console, click an instance name on the DMS for Kafka page. Basic information of the Kafka instance is displayed. In the Connection pane, obtain the Instance Address (Private Network) value. In the Network pane, obtain the VPC and subnet of the instance. In the Network pane, obtain the security group of the instance.
RDS	On the Instances page of the RDS console, click the target DB instance name. In the displayed page, locate the Connection Information pane and obtain the Floating IP Address, VPC, Subnet, Database Port, and Security Group.
CSS	On the CSS management console, choose Clusters > Elasticsearch. On the displayed page, click the name of the created CSS cluster to view basic information. On the Cluster Information page, obtain the Private Network Address, VPC, Subnet, and Security Group.
GaussDB(DWS)	On the GaussDB(DWS) management console, choose Clusters. On the displayed page, click the name of the created GaussDB(DWS) cluster to view basic information. On the Basic Information tab, locate the Database Attributes pane and obtain the private IP address and port number of the DB instance. In the Network pane, obtain the VPC, subnet, and security group information.
MRS HBase	An MRS 3.x cluster is used as an example. Log in to the MRS management console, click a cluster name on the Clusters > Active Clusters page to view basic information. On the dashboard, obtain VPC, subnet, and security group from the Basic Information pane. The ZooKeeper instance and its port of the MRS cluster are required for creating a job that connects DLI to MRS HBase. You need to obtain the host information of the MRS cluster. Log in to MRS Manager by referring to Accessing FusionInsight Manager. On MRS Manager, choose Cluster > Name of the desired cluster > Services > ZooKeeper. Click the Instance tab and obtain the ZooKeeper host information such as the host name and service IP address. On MRS Manager, choose Cluster and click the name of the desired cluster. Choose Services > ZooKeeper. Click the Configurations tab and select All Configurations, search for the clientPort parameter, and obtain its value, that is, the ZooKeeper port number. Log in to any MRS node as user root in SSH mode. For details, see Logging In to an ECS. Run the following command to obtain MRS hosts information. Copy and save the information. cat /etc/hosts An example query result is as follows:

Step 2: Obtain the CIDR Block of the DLI Queue

On the DLI management console, choose Resources > Queue Management from the navigation pane. Locate the queue you have created, and click next to the queue name to view the CIDR block of the queue.

Step 3: Add a Rule to the Security Group of the External Data Source to Allow Access from the DLI Queue

Log in to the VPC console.
In the navigation pane on the left, choose Access Control > Security Groups.
Click the name of the security group to which the external data source belongs.
Obtain the security group name of the data source on the management console of the data source by referring to Step 1: Obtain the Floating IP Address, Port Number, and Security Group of an External Data Source.

On the Inbound Rules tab, add a rule to allow access from the queue network segment.

Set the inbound rule parameters based on Table 3.

Figure 2 Adding an inbound rule

**Table 3** Inbound rule parameters
Parameter	Description	Example
Priority	The security group rule priority. The priority value ranges from 1 to 100. The default value is 1, indicating the highest priority. A smaller value indicates a higher priority of a security group rule.	1
Action	Action of the security group rule.	Select Allow.
Protocol &Port	Network protocol: The value can be All, TCP, UDP, ICMP, or GRE. Port: Port or port range over which the traffic can reach your instance. The port ranges from 1 to 65535.	In this example, select TCP. Leave the port blank or set it to the data source port obtained in Step 1: Obtain the Floating IP Address, Port Number, and Security Group of an External Data Source.
Type	Type of IP addresses.	IPv4
Source	Allow access from IP addresses or instances in another security group.	In this example, enter the queue CIDR block obtained in Step 2: Obtain the CIDR Block of the DLI Queue.
Description	Supplementary information about the security group rule. This parameter is optional.	_

Step 4: Create an Enhanced Datasource Connection

Log in to the DLI management console. In the navigation pane on the left, choose Datasource Connections. On the displayed page, click Create in the Enhanced tab.
In the displayed dialog box, set the following parameters:
- Connection Name: Name of the enhanced datasource connection
- Resource Pool: Select the target DLI queue. (Queues that are not added to a resource pool are displayed in this list.)
- VPC: VPC of the data source obtained in Step 1: Obtain the Floating IP Address, Port Number, and Security Group of an External Data Source
- Subnet: Subnet of the data source obtained in Step 1: Obtain the Floating IP Address, Port Number, and Security Group of an External Data Source
- Set other parameters as you need.
Click OK. Click the name of the created datasource connection to view its status. You can perform subsequent steps only after the connection status changes to Active.
To connect to MRS HBase, you need to add MRS host information. The procedure is as follows:
1. On the Datasource Connections page, click the Enhanced tab and locate the row that contains the created enhanced datasource connection. Click More > Modify Host in the Operation column.
2. In the dialog box that appears, enter the MRS HBase host information obtained in Step 1: Obtain the Floating IP Address, Port Number, and Security Group of an External Data Source to the Host Information box.
  Figure 3 Modifying host information
3. Click OK.

Step 5: Test Network Connectivity

Choose Resources > Queue Management from the left navigation pane, locate the target queue. In the Operation column, click More > Test Address Connectivity.
In the displayed dialog box, enter the IP address and port number of the data source obtained in Step 1: Obtain the Floating IP Address, Port Number, and Security Group of an External Data Source in the address box and click Test. If the queue passes the test, it can access the data source.
NOTE:

For MRS HBase, use ZooKeeper IP address:ZooKeeper port or ZooKeeper host information:ZooKeeper port for the test.

Parent topic: Configuring DLI to Read and Write Data from and to External Data Sources

Previous topic: Deleting an Enhanced Datasource Connection

Next topic: Example Typical Scenario: Connecting DLI to a Data Source on a Public Network

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot