Creating a Connection Between DataArts Studio and a GaussDB(DWS) Data Lake

Prerequisites

• You have created a data lake to connect, for example, a database or cloud service supported by DataArts Studio.
  • Before creating a DWS data connection, ensure that you have created a cluster in DWS and have the permissions required to view Key Management Service (KMS) keys.
  • Before creating an MRS connection such as an MRS HBase or MRS Hive connection, ensure that you have purchased an MRS cluster whose Kerberos encryption type is aes256-sha1,aes128-sha1, and that the cluster contains required components.
• You have obtained the required agent (CDM cluster). If no CDM cluster is available, create one by referring to Creating a CDM Cluster. The CDM cluster can communicate with the data lake to be connected.
  • If the data lake is an on-premises database, you need the Internet or Direct Connect. Ensure that the host where the data source is located and the CDM cluster can access the Internet, and the connection port has been enabled in the firewall rule.
  • If the data lake is a cloud service (such as DWS and MRS), the following requirements must be met for network interconnection:
    • If the CDM cluster and the cloud service are in different regions, a public network or a dedicated connection is required for enabling communication between the CDM cluster and the cloud service.
    • If the CDM cluster and the cloud service are in the same region, VPC, subnet, and security group, they can communicate with each other by default. If they are in the same VPC but in different subnets or security groups, you must configure routing rules and security group rules. For details about how to configure routing rules, see Configuring Routing Rules. For details about how to configure security group rules, see Configuring Security Group Rules.
    • The cloud service instance and the DataArts Studio workspace belong to the same enterprise project. If they do not, you can modify the enterprise project of the workspace.
• If the enterprise mode is used, pay attention to the following points:
  In enterprise mode, the development environment and production environment need to be distinguished. Therefore, you need to prepare two sets of data lake services for the production environment and development environment to isolate the development environment from the production environment.

  • If two clusters are used for clustered data sources, such as MRS, DWS, RDS, MySQL, Oracle, DIS, and ECS, you can create data connections in Management Center to distinguish data lake services in the development environment from those in the production environment. The data lake is automatically switched during development and production. Therefore, you need to prepare two sets of data lake services. The versions, specifications, components, regions, VPCs, subnets, and related configurations of the two sets of data lake services must be the same. For details on how to create data connections, see Creating a DataArts Studio Data Connection.
  • For serverless services (such as DLI), DataArts Studio configures the mapping between data lake services in the production environment and development environment through environment isolation in the management center. The corresponding data lake is automatically switched during the development and production processes. Therefore, you need to prepare two sets of queues and database resources in the serverless data lake service and distinguish them by name suffix. For details, see Configuring Environment Isolation for a DataArts Studio Workspace in Enterprise Mode.
  • For DWS, MRS Hive, and MRS Spark, if you select the same cluster when creating a data connection, you must configure database mapping on the Configure Data Source Resource Mapping page to isolate the development and production environments. For details, see DB configuration.
  • Offline processing migration jobs are not supported in enterprise mode.

  For example, if your data lake service is an MRS cluster, you need to prepare two MRS clusters with the same version, specifications, components, region, VPC, and subnet. If some configurations of an MRS cluster are modified, you also need to synchronize the modifications to the other MRS cluster.

Creating a Data Connection

1. Log in to the DataArts Studio console by following the instructions in Accessing the DataArts Studio Instance Console.
2. On the DataArts Studio console, locate a workspace and click Management Center.
3. On the displayed Manage Data Connections page, click Create Data Connection.
   Figure 1 Creating a data connection
   

4. On the Manage Data Connections page, click Create Data Connection. On the displayed page, select DWS for Data Connection Type and set other parameters based on the descriptions in Table 1.
   Figure 2 DWS connection parameters
   

   Table 1 DWS connection

   Parameter	Mandatory	Description
   Data Connection Type	Yes	DWS is selected and cannot be changed.
   Name	Yes	Name of the data connection to create. Data connection names can contain a maximum of 100 characters. They can contain only letters, digits, underscores (_), and hyphens (-).
   Description	No	A description which can help identify the data connection more easily. It can contain a maximum of 100 characters.
   Tag	No	Attribute of the data connection to create. Tags make management easier. NOTE: The tag name can contain only letters, digits, and underscores (_) and cannot start with an underscore (_) or contain more than 100 characters.
   Applicable Modules	Yes	Select the modules for which this connection is available. NOTE: When offline or real-time data migration jobs are enabled, you can select the DataArts Migration module. Then you can select this data connection when creating a data migration job in DataArts Factory. You can use offline or real-time data migration jobs only after you apply for the whitelist membership. To use this feature, contact customer service or technical support.
   Basic and Network Connectivity Configuration
   SSL Encryption	Yes	DWS supports SSL encryption and certificate authentication for communication between the client and server. You can set this parameter based on whether SSL connection is mandatory on the server. If this parameter is enabled, only SSL encryption can be used for communication. If this parameter is disabled, both SSL encryption and certificate authentication can be used for communication.
   Manual	Yes	Select either of the following modes: Cluster Name Mode: Select an existing cluster. Connection String Mode: Enter the IP address/domain name and port of the corresponding cluster and enable the communication between the connection's agent (CDM cluster) and the DWS cluster. NOTE: DataArts Security does not support DWS connections in string mode.
   DWS Cluster Name	Yes	This parameter is mandatory when Manual is set to Cluster Name Mode. Select a DWS cluster from all the DWS clusters with the same project ID and enterprise project.
   JDBC IP Address or Domain Name	No	This parameter is mandatory when Manual is set to Cluster Name Mode. Select a DWS ELB domain name or ELB IP address. The former is recommended. If it is not supported, select an ELB IP address.
   IP Address or Domain Name	Yes	This parameter is mandatory when Manual is set to Connection String Mode. This parameter indicates the address for accessing the cluster database through an internal network. Enter an IP address or domain name. The IP address or domain name is automatically generated during cluster creation. You can obtain them on the management console by performing the following operations: Log in to the DWS console. In the left navigation pane, choose Instances. Click the name of an instance to enter the basic information page. In the Connection Information area, you can obtain the private IP address, domain name, and port number.
   Port	Yes	This parameter is mandatory when Manual is set to Connection String Mode. This parameter indicates the database port number specified during the DWS cluster creation. Ensure that you have enabled this port in the security group rule so that the DataArts Studio instance can connect to the database in the DWS cluster through this port.
   KMS Key	Yes	KMS key used to encrypt and decrypt data source authentication information. Select a default or custom key. NOTE: When you use KMS for encryption through DataArts Studio or KPS for the first time, the default key dlf/default or kps/default is automatically generated. For more information about default keys, see What Is a Default Master Key?. Only symmetric keys are supported. Asymmetric keys are not supported.
   Agent	Yes	Data Warehouse Service (DWS) is not a fully managed service and thus cannot be directly connected to DataArts Studio. A CDM cluster can provide an agent for DataArts Studio to communicate with non-fully-managed services. Therefore, you need to select a CDM cluster when creating a DWS data connection. If no CDM cluster is available, create one first by referring to Creating a CDM Cluster. As a network proxy, the CDM cluster must be able to communicate with the DWS cluster. To ensure network connectivity, the CDM cluster must be in the same region and AZ and use the same VPC and subnet as the DWS cluster. The security group rule must also allow the CDM cluster communicate with the DWS cluster. NOTE: If a CDM cluster functions as the agent for a data connection in Management Center, the cluster supports a maximum of 200 concurrent active threads. If multiple data connections share an agent, a maximum of 200 SQL, Shell, and Python scripts submitted through the connections can run concurrently. Excess tasks will be queued. You are advised to plan multiple agents based on the workload.
   Data Source Authentication and Other Function Configuration
   Username	Yes	The database username, which is specified when the DWS cluster is created.
   Password	Yes	The password for accessing the database, which is specified when the DWS cluster is created.
   Data Integration Configuration
   Database Name	Yes	This parameter is displayed when DataArts Migration is selected for Applicable Modules. Name of the database to connect
   Fetch Size	No	This parameter is displayed when DataArts Migration is selected for Applicable Modules. Number of rows obtained by each request. Set this parameter based on the data source and the job's data size. If the value is either too large or too small, the job may run for a long time.
   Link Attributes	No	This parameter is displayed when DataArts Migration is selected for Applicable Modules. (Optional) Click Add to add the JDBC connector attributes of multiple specified data sources. For details, see the JDBC connector document of the corresponding database. The following are some examples: connectTimeout=360000 and socketTimeout=360000: When a large amount of data needs to be migrated or the entire table is retrieved using query statements, the migration fails due to connection timeout. In this case, you can customize the connection timeout interval (ms) and socket timeout interval (ms) to prevent failures caused by timeout. useCursorFetch=false: By default, useCursorFetch is enabled, indicating that the JDBC connector communicates with relational databases using a binary protocol. Some third-party systems may have compatibility issues, causing migration time conversion errors. In this case, you can disable this function. Open-source MySQL databases support the useCursorFetch parameter, and you do not need to set this parameter.
   Reference Sign	No	This parameter is displayed when DataArts Migration is selected for Applicable Modules. (Optional) Delimiter between the names of the referenced tables or columns. For details, see the product documentation of the corresponding database.

5. Click Test to test connectivity of the data connection. If the test fails, the data connection fails to be created.
6. After the test is successful, click Save to create the data connection.

Reference

1. What should I do if the connection test fails when I enable the SSL connection during the creation of a DWS data connection?
   On the DWS console, click the corresponding cluster, choose Security Settings, and disable Rights Separation.

   Figure 3 Disabling Rights Separation for the DWS cluster
   
2. Why does a DWS data connection fail to obtain information about databases or tables?

   The possible cause is that the CDM cluster is stopped or a concurrency conflict occurs. You can switch to another agent to temporarily avoid this issue.