Help Center/ DataArts Studio/ User Guide/ Management Center/ Typical Scenarios for Using Management Center/ Creating a Connection Between DataArts Studio and a GaussDB(DWS) Data Lake
Updated on 2024-10-28 GMT+08:00
View PDF
Share

Creating a Connection Between DataArts Studio and a GaussDB(DWS) Data Lake

This section describes how to create a DWS connection between DataArts Studio and the data lake base.

Prerequisites

  • You have created a data lake to connect, for example, a database or cloud service supported by DataArts Studio.
    • Before creating a DWS data connection, ensure that you have created a cluster in DWS and have the permissions required to view Key Management Service (KMS) keys.
    • Before creating an MRS connection such as an MRS HBase or MRS Hive connection, ensure that you have purchased an MRS cluster whose Kerberos encryption type is aes256-sha1,aes128-sha1, and that the cluster contains required components.
  • The data lake to connect communicates with the DataArts Studio instance properly.
    • If the data lake is an on-premises database, a public network or a dedicated connection is required. Ensure that the host where the data source is located can access the public network and the port has been enabled in the firewall rule.
    • If the data lake is a cloud service (such as DWS and MRS), the following requirements must be met for network interconnection:
      • If the CDM cluster in the DataArts Studio instance and the cloud service are in different regions, a public network or a dedicated connection is required.
      • If the CDM cluster in the DataArts Studio instance and the cloud service are in the same region, VPC, subnet, and security group, they can communicate with each other by default. If they are in the same VPC but in different subnets or security groups, you must configure routing rules and security group rules. For details about how to configure routing rules, see Configuring Routing Rules. For details about how to configure security group rules, see Configuring Security Group Rules.
      • The cloud service instance and the DataArts Studio workspace belong to the same enterprise project. If they do not, you can modify the enterprise project of the workspace.
  • If the enterprise mode is used, pay attention to the following points:
    In enterprise mode, the development environment and production environment need to be distinguished. Therefore, you need to prepare two sets of data lake services for the production environment and development environment to isolate the development environment from the production environment.
    • If two clusters are used for clustered data sources, such as MRS, GaussDB(DWS), RDS, MySQL, Oracle, DIS, and ECS, you can create data connections in Management Center to distinguish data lake services in the development environment from those in the production environment. The data lake is automatically switched during development and production. Therefore, you need to prepare two sets of data lake services. The versions, specifications, components, regions, VPCs, subnets, and related configurations of the two sets of data lake services must be the same. For details on how to create data connections, see Creating a DataArts Studio Data Connection.
    • For serverless services (such as DLI), DataArts Studio configures the mapping between data lake services in the production environment and development environment through environment isolation in the management center. The corresponding data lake is automatically switched during the development and production processes. Therefore, you need to prepare two sets of queues and database resources in the serverless data lake service and distinguish them by name suffix. For details, see Configuring Environment Isolation for a DataArts Studio Workspace in Enterprise Mode.
    • For GaussDB(DWS), MRS Hive, and MRS Spark, if you select the same cluster when creating a data connection, you must configure database mapping on the Configure Data Source Resource Mapping page to isolate the development and production environments. For details, see DB configuration.
    • Offline processing migration jobs are not supported in enterprise mode.

    For example, if your data lake service is an MRS cluster, you need to prepare two MRS clusters with the same version, specifications, components, region, VPC, and subnet. If some configurations of an MRS cluster are modified, you also need to synchronize the modifications to the other MRS cluster.

Creating a Data Connection

  1. Log in to the DataArts Studio console by following the instructions in Accessing the DataArts Studio Instance Console.
  2. On the DataArts Studio console, locate a workspace and click Management Center.
  3. On the displayed Manage Data Connections page, click Create Data Connection.

    Figure 1 Creating a data connection

  4. On the Manage Data Connections page, click Create Data Connection. On the displayed page, select DWS for Data Connection Type and set other parameters based on the descriptions in Table 1.

    Figure 2 DWS connection parameters

    Table 1 DWS connection

    Parameter

    Mandatory

    Description

    Data Connection Type

    Yes

    DWS is selected and cannot be changed.

    Name

    Yes

    Name of the data connection to create. Data connection names can contain a maximum of 100 characters. They can contain only letters, digits, underscores (_), and hyphens (-).

    Tag

    No
    Attribute of the data connection to create. Tags make management easier.
    NOTE:

    The tag name can contain only letters, digits, and underscores (_) and cannot start with an underscore (_) or contain more than 100 characters.

    Applicable Modules

    Yes

    Select the modules for which this connection is available.

    NOTE:
    • When the data migration job feature is enabled, you can select the DataArts Migration module. Then you can select this data connection when creating a data migration job in DataArts Factory.
    • You can use offline processing migration jobs only after apply for the trustlist membership. To use this feature, contact customer service or technical support.

    Basic and Network Connectivity Configuration

    SSL Encryption

    Yes
    DWS supports SSL encryption and certificate authentication for communication between the client and server. You can set this parameter based on whether SSL connection is mandatory on the server.
    • If this parameter is enabled, only SSL encryption can be used for communication.
    • If this parameter is disabled, both SSL encryption and certificate authentication can be used for communication.

    Manual

    Yes

    Select either of the following modes:

    • Cluster Name Mode: Select an existing cluster.
    • Connection String Mode: Enter the IP address/domain name and port of the corresponding cluster and enable the communication between the connection's agent (CDM cluster) and the DWS cluster.

    DWS Cluster Name

    Yes

    This parameter is mandatory when Manual is set to Cluster Name Mode.

    Select a DWS cluster from all the DWS clusters with the same project ID and enterprise project.

    IP Address or Domain Name

    Yes

    This parameter is mandatory when Manual is set to Connection String Mode.

    If you choose to manually enter an IP address or domain name, you must enter an internal IP address and a port that is accessible to the network segment of the resource group. Otherwise, the network is disconnected.

    This parameter indicates the address for accessing the cluster database through an internal network. Enter an IP address or domain name. The IP address or domain name is automatically generated during cluster creation. You can obtain them on the management console by performing the following operations:
    1. Log in to the GaussDB(DWS) console.
    2. In the left navigation pane, choose Instances.
    3. Click the name of an instance to enter the basic information page. In the Connection Information area, you can obtain the private IP address, domain name, and port number.

    Port

    Yes

    This parameter is mandatory when Manual is set to Connection String Mode.

    This parameter indicates the database port number specified during the DWS cluster creation. Ensure that you have enabled this port in the security group rule so that the DataArts Studio instance can connect to the database in the DWS cluster through this port.

    KMS Key

    Yes
    KMS key used to encrypt and decrypt data source authentication information. Select a default or custom key.
    NOTE:

    When you use KMS for encryption through DataArts Studio or KPS for the first time, the default key dlf/default or kps/default is automatically generated. For more information about default keys, see What Is a Default Master Key?.

    Agent

    Yes

    Data Warehouse Service (DWS) is not a fully managed service and thus cannot be directly connected to DataArts Studio. A CDM cluster can provide an agent for DataArts Studio to communicate with non-fully-managed services. Therefore, you need to select a CDM cluster when creating a DWS data connection. If no CDM cluster is available, create one first.

    As a network proxy, the CDM cluster must be able to communicate with the DWS cluster. To ensure network connectivity, the CDM cluster must be in the same region and AZ and use the same VPC and subnet as the DWS cluster. The security group rule must also allow the CDM cluster communicate with the DWS cluster.

    NOTE:

    If a CDM cluster functions as the agent for a data connection in Management Center, the cluster supports a maximum of 200 concurrent active threads. If multiple data connections share an agent, a maximum of 200 SQL, Shell, and Python scripts submitted through the connections can run concurrently. Excess tasks will be queued. You are advised to plan multiple agents based on the workload.

    Data Source Authentication and Other Function Configuration

    Username

    Yes

    The database username, which is specified when the DWS cluster is created.

    Password

    Yes

    The password for accessing the database, which is specified when the DWS cluster is created.

  5. Click Test to test connectivity of the data connection. If the test fails, the data connection fails to be created.
  6. After the test is successful, click OK to create the data connection.

Reference

  1. What should I do if the connection test fails when I enable the SSL connection during the creation of a DWS data connection?
    On the DWS console, click the corresponding cluster, choose Security Settings, and disable Rights Separation.
    Figure 3 Disabling Rights Separation for the DWS cluster
  2. Why does a DWS data connection fail to obtain information about databases or tables?

    The possible cause is that the CDM cluster is stopped or a concurrency conflict occurs. You can switch to another agent to temporarily avoid this issue.