Creating a Connection Between DataArts Studio and a GaussDB(DWS) Data Lake

Prerequisites

• You have created a data lake to connect, for example, a database or cloud service supported by DataArts Studio.
  • Before creating a DWS data connection, ensure that you have created a cluster in DWS and have the permissions required to view Key Management Service (KMS) keys.
  • Before creating an MRS connection such as an MRS HBase or MRS Hive connection, ensure that you have purchased an MRS cluster whose Kerberos encryption type is aes256-sha1,aes128-sha1, and that the cluster contains required components.
• You have obtained the required agent (CDM cluster). If no CDM cluster is available, create one by referring to Creating a CDM Cluster. The CDM cluster can communicate with the data lake to be connected.
  • If the data lake is an on-premises database, you need the Internet or Direct Connect. Ensure that the host where the data source is located and the CDM cluster can access the Internet, and the connection port has been enabled in the firewall rule.
  • If the data lake is a cloud service (such as DWS and MRS), the following requirements must be met for network interconnection:
    • If the CDM cluster and the cloud service are in different regions, a public network or a dedicated connection is required for enabling communication between the CDM cluster and the cloud service.
    • If the CDM cluster and the cloud service are in the same region, VPC, subnet, and security group, they can communicate with each other by default. If they are in the same VPC but in different subnets or security groups, you must configure routing rules and security group rules. For details about how to configure routing rules, see Configuring Routing Rules. For details about how to configure security group rules, see Configuring Security Group Rules.
    • The cloud service instance and the DataArts Studio workspace belong to the same enterprise project. If they do not, you can modify the enterprise project of the workspace.
• If the enterprise mode is used, pay attention to the following points:
  In enterprise mode, the development environment and production environment need to be distinguished. Therefore, you need to prepare two sets of data lake services for the production environment and development environment to isolate the development environment from the production environment.

  • If two clusters are used for clustered data sources, such as MRS, GaussDB(DWS), RDS, MySQL, Oracle, DIS, and ECS, you can create data connections in Management Center to distinguish data lake services in the development environment from those in the production environment. The data lake is automatically switched during development and production. Therefore, you need to prepare two sets of data lake services. The versions, specifications, components, regions, VPCs, subnets, and related configurations of the two sets of data lake services must be the same. For details on how to create data connections, see Creating a DataArts Studio Data Connection.
  • For serverless services (such as DLI), DataArts Studio configures the mapping between data lake services in the production environment and development environment through environment isolation in the management center. The corresponding data lake is automatically switched during the development and production processes. Therefore, you need to prepare two sets of queues and database resources in the serverless data lake service and distinguish them by name suffix. For details, see Configuring Environment Isolation for a DataArts Studio Workspace in Enterprise Mode.
  • For GaussDB(DWS), MRS Hive, and MRS Spark, if you select the same cluster when creating a data connection, you must configure database mapping on the Configure Data Source Resource Mapping page to isolate the development and production environments. For details, see DB configuration.
  • Offline processing migration jobs are not supported in enterprise mode.

  For example, if your data lake service is an MRS cluster, you need to prepare two MRS clusters with the same version, specifications, components, region, VPC, and subnet. If some configurations of an MRS cluster are modified, you also need to synchronize the modifications to the other MRS cluster.

Creating a Data Connection

1. Log in to the DataArts Studio console by following the instructions in Accessing the DataArts Studio Instance Console.
2. On the DataArts Studio console, locate a workspace and click Management Center.
3. On the displayed Manage Data Connections page, click Create Data Connection.
   Figure 1 Creating a data connection
   

4. On the Manage Data Connections page, click Create Data Connection. On the displayed page, select DWS for Data Connection Type and set other parameters based on the descriptions in Table 1.
   Figure 2 DWS connection parameters
   

   Table 1 DWS connection

   Parameter	Mandatory	Description
   Data Connection Type	Yes	DWS is selected and cannot be changed.
   Name	Yes	Name of the data connection to create. Data connection names can contain a maximum of 100 characters. They can contain only letters, digits, underscores (_), and hyphens (-).
   Tag	No	Attribute of the data connection to create. Tags make management easier. NOTE: The tag name can contain only letters, digits, and underscores (_) and cannot start with an underscore (_) or contain more than 100 characters.
   Applicable Modules	Yes	Select the modules for which this connection is available. NOTE: When the data migration job feature is enabled, you can select the DataArts Migration module. Then you can select this data connection when creating a data migration job in DataArts Factory. You can use offline processing migration jobs only after apply for the whitelist membership. To use this feature, contact customer service or technical support.
   Basic and Network Connectivity Configuration
   SSL Encryption	Yes	DWS supports SSL encryption and certificate authentication for communication between the client and server. You can set this parameter based on whether SSL connection is mandatory on the server. If this parameter is enabled, only SSL encryption can be used for communication. If this parameter is disabled, both SSL encryption and certificate authentication can be used for communication.
   Manual	Yes	Select either of the following modes: Cluster Name Mode: Select an existing cluster. Connection String Mode: Enter the IP address/domain name and port of the corresponding cluster and enable the communication between the connection's agent (CDM cluster) and the DWS cluster. NOTE: DataArts Security does not support GaussDB(DWS) connections in connection string mode.
   DWS Cluster Name	Yes	This parameter is mandatory when Manual is set to Cluster Name Mode. Select a DWS cluster from all the DWS clusters with the same project ID and enterprise project.
   IP Address or Domain Name	Yes	This parameter is mandatory when Manual is set to Connection String Mode. If you choose to manually enter an IP address or domain name, you must enter an internal IP address and a port that is accessible to the network segment of the resource group. Otherwise, the network is disconnected. This parameter indicates the address for accessing the cluster database through an internal network. Enter an IP address or domain name. The IP address or domain name is automatically generated during cluster creation. You can obtain them on the management console by performing the following operations: Log in to the GaussDB(DWS) console. In the left navigation pane, choose Instances. Click the name of an instance to enter the basic information page. In the Connection Information area, you can obtain the private IP address, domain name, and port number.
   Port	Yes	This parameter is mandatory when Manual is set to Connection String Mode. This parameter indicates the database port number specified during the DWS cluster creation. Ensure that you have enabled this port in the security group rule so that the DataArts Studio instance can connect to the database in the DWS cluster through this port.
   KMS Key	Yes	KMS key used to encrypt and decrypt data source authentication information. Select a default or custom key. NOTE: When you use KMS for encryption through DataArts Studio or KPS for the first time, the default key dlf/default or kps/default is automatically generated. For more information about default keys, see What Is a Default Master Key?.
   Agent	Yes	Data Warehouse Service (DWS) is not a fully managed service and thus cannot be directly connected to DataArts Studio. A CDM cluster can provide an agent for DataArts Studio to communicate with non-fully-managed services. Therefore, you need to select a CDM cluster when creating a DWS data connection. If no CDM cluster is available, create one first by referring to Creating a CDM Cluster. As a network proxy, the CDM cluster must be able to communicate with the DWS cluster. To ensure network connectivity, the CDM cluster must be in the same region and AZ and use the same VPC and subnet as the DWS cluster. The security group rule must also allow the CDM cluster communicate with the DWS cluster. NOTE: If a CDM cluster functions as the agent for a data connection in Management Center, the cluster supports a maximum of 200 concurrent active threads. If multiple data connections share an agent, a maximum of 200 SQL, Shell, and Python scripts submitted through the connections can run concurrently. Excess tasks will be queued. You are advised to plan multiple agents based on the workload.
   Data Source Authentication and Other Function Configuration
   Username	Yes	The database username, which is specified when the DWS cluster is created.
   Password	Yes	The password for accessing the database, which is specified when the DWS cluster is created.
   Metadata Collection Scope	No	Databases and data tables whose metadata will be synchronized in real time. If this parameter is not set, all metadata will be synchronized. The value can be in either of the following formats: database_name: databases whose names contain database_name database_name.table_name: databases whose names contain database_name and data tables whose names contain table_name Examples: If you enter datatest, metadata of the tables in the databases whose names contain datatest will be synchronized in real time. If you enter datatest.table1, metadata of the tables whose names contain table_name in the databases whose names contain datatest will be synchronized in real time.

5. Click Test to test connectivity of the data connection. If the test fails, the data connection fails to be created.
6. After the test is successful, click OK to create the data connection.

Reference

1. What should I do if the connection test fails when I enable the SSL connection during the creation of a DWS data connection?
   On the DWS console, click the corresponding cluster, choose Security Settings, and disable Rights Separation.

   Figure 3 Disabling Rights Separation for the DWS cluster
   
2. Why does a DWS data connection fail to obtain information about databases or tables?

   The possible cause is that the CDM cluster is stopped or a concurrency conflict occurs. You can switch to another agent to temporarily avoid this issue.