Creating a Connection Between DataArts Studio and a MySQL Database

Prerequisites

• You have created a data lake to connect, for example, a database or cloud service supported by DataArts Studio.
  • Before creating a DWS data connection, ensure that you have created a cluster in DWS and have the permissions required to view Key Management Service (KMS) keys.
  • Before creating an MRS connection such as an MRS HBase or MRS Hive connection, ensure that you have purchased an MRS cluster whose Kerberos encryption type is aes256-sha1,aes128-sha1, and that the cluster contains required components.
• You have obtained the required agent (CDM cluster). If no CDM cluster is available, create one by referring to Creating a CDM Cluster. The CDM cluster can communicate with the data lake to be connected.
  • If the data lake is an on-premises database, you need the Internet or Direct Connect. Ensure that the host where the data source is located and the CDM cluster can access the Internet, and the connection port has been enabled in the firewall rule.
  • If the data lake is a cloud service (such as DWS and MRS), the following requirements must be met for network interconnection:
    • If the CDM cluster and the cloud service are in different regions, a public network or a dedicated connection is required for enabling communication between the CDM cluster and the cloud service.
    • If the CDM cluster and the cloud service are in the same region, VPC, subnet, and security group, they can communicate with each other by default. If they are in the same VPC but in different subnets or security groups, you must configure routing rules and security group rules. For details about how to configure routing rules, see Configuring Routing Rules. For details about how to configure security group rules, see Configuring Security Group Rules.
    • The cloud service instance and the DataArts Studio workspace belong to the same enterprise project. If they do not, you can modify the enterprise project of the workspace.
• If the enterprise mode is used, pay attention to the following points:
  In enterprise mode, the development environment and production environment need to be distinguished. Therefore, you need to prepare two sets of data lake services for the production environment and development environment to isolate the development environment from the production environment.

  • If two clusters are used for clustered data sources, such as MRS, DWS, RDS, MySQL, Oracle, DIS, and ECS, you can create data connections in Management Center to distinguish data lake services in the development environment from those in the production environment. The data lake is automatically switched during development and production. Therefore, you need to prepare two sets of data lake services. The versions, specifications, components, regions, VPCs, subnets, and related configurations of the two sets of data lake services must be the same. For details on how to create data connections, see Creating a DataArts Studio Data Connection.
  • For serverless services (such as DLI), DataArts Studio configures the mapping between data lake services in the production environment and development environment through environment isolation in the management center. The corresponding data lake is automatically switched during the development and production processes. Therefore, you need to prepare two sets of queues and database resources in the serverless data lake service and distinguish them by name suffix. For details, see Configuring Environment Isolation for a DataArts Studio Workspace in Enterprise Mode.
  • For DWS, MRS Hive, and MRS Spark, if you select the same cluster when creating a data connection, you must configure database mapping on the Configure Data Source Resource Mapping page to isolate the development and production environments. For details, see DB configuration.
  • Offline processing migration jobs are not supported in enterprise mode.

  For example, if your data lake service is an MRS cluster, you need to prepare two MRS clusters with the same version, specifications, components, region, VPC, and subnet. If some configurations of an MRS cluster are modified, you also need to synchronize the modifications to the other MRS cluster.

Creating a Data Connection

1. Log in to the DataArts Studio console by following the instructions in Accessing the DataArts Studio Instance Console.
2. On the DataArts Studio console, locate a workspace and click Management Center.
3. On the displayed Manage Data Connections page, click Create Data Connection.
   Figure 1 Creating a data connection
   

4. On the Manage Data Connections page, click Create Data Connection. On the displayed page, select RDS for Data Connection Type and set other parameters based on the descriptions in Table 1.
   

   • You are not advised to select MySQL (pending offline) for Data Connection Type. Instead, You are advised to select RDS.
   • RDS data connections depend on OBS. If OBS is unavailable in the same region as DataArts Studio, RDS data connections are not supported.
   Figure 2 RDS connection parameters
   

   Table 1 RDS connection

   Parameter	Mandatory	Description
   Data Connection Type	Yes	RDS(MySQL) is selected by default and cannot be changed.
   Name	Yes	Name of the data connection to create. Data connection names can contain a maximum of 100 characters. They can contain only letters, digits, underscores (_), and hyphens (-).
   Description	No	A description which can help identify the data connection more easily. It can contain a maximum of 100 characters.
   Tag	No	Attribute of the data connection to create. Tags make management easier. NOTE: The tag name can contain only letters, digits, and underscores (_) and cannot start with an underscore (_) or contain more than 100 characters.
   Applicable Modules	Yes	Select the modules for which this connection is available. NOTE: When offline or real-time data migration jobs are enabled, you can select the DataArts Migration module. Then you can select this data connection when creating a data migration job in DataArts Factory. You can use offline or real-time data migration jobs only after you apply for the whitelist membership. To use this feature, contact customer service or technical support.
   Basic and Network Connectivity Configuration
   IP Address or Domain Name	Yes	Address for accessing the relational database through an internal network. Enter an IP address or domain name. If the data source is RDS, you can obtain the address on the management console by performing the following operations: Log in to the management console of the corresponding cloud service using the account you have obtained. In the left navigation pane, choose Instances. Click the name of an instance to enter the basic information page. In the Connection Information area, you can obtain the private IP address, domain name, and port number. If the data source is an on-premises MySQL, you can obtain the access address from the database administrator.
   Port	Yes	Port for accessing the relational database. Ensure that you have enabled this port in the security group rule so that the DataArts Studio instance can connect to this database through this port. If the data source is RDS, you can obtain the address on the management console by performing the following operations: Log in to the management console of the corresponding cloud service using the account you have obtained. In the left navigation pane, choose Instances. Click the name of an instance to enter the basic information page. In the Connection Information area, you can obtain the private IP address, domain name, and port number. If the data source is an on-premises MySQL, you can obtain the access address from the database administrator.
   KMS Key	Yes	KMS key used to encrypt and decrypt data source authentication information. Select a default or custom key. NOTE: When you use KMS for encryption through DataArts Studio or KPS for the first time, the default key dlf/default or kps/default is automatically generated. For more information about default keys, see What Is a Default Master Key?. Only symmetric keys are supported. Asymmetric keys are not supported.
   Agent	Yes	RDS is not a fully managed service and cannot be directly connected to DataArts Studio. A CDM cluster can provide an agent for DataArts Studio to communicate with non-fully-managed services. Therefore, you need to select a CDM cluster when creating an RDS data connection. If no CDM cluster is available, create one first by referring to Creating a CDM Cluster. As a network proxy, the CDM cluster must be able to communicate with RDS. To ensure network connectivity, the CDM cluster must be in the same region and AZ and use the same VPC and subnet as RDS. The security group rule must also allow the CDM cluster to communicate with RDS. NOTE: If a CDM cluster functions as the agent for a data connection in Management Center, the cluster supports a maximum of 200 concurrent active threads. If multiple data connections share an agent, a maximum of 200 SQL, Shell, and Python scripts submitted through the connections can run concurrently. Excess tasks will be queued. You are advised to plan multiple agents based on the workload.
   SSL Encryption	No	Whether to enable SSL encrypted transmission.
   Data Source Driver Configuration
   Driver Name	Yes	com.mysql.jdbc.Driver: Select this driver name for RDS for MySQL or MySQL.
   Driver File Path	Yes	It specifies the OBS path where the driver file is located. You need to download a .jar driver file from the corresponding official website and upload it to OBS. MySQL driver: Download it from https://downloads.mysql.com/archives/c-j/. The 5.1.48 version is recommended. NOTE: The OBS path of the driver file cannot contain Chinese characters. For security purposes, the system checks the SHA512 value and byte size of the selected driver file. If the driver file is not in the allowed list, an error message will be displayed. You are advised to use the driver file of the recommended version or contact customer service or technical support. To update the driver, you must restart the CDM cluster in DataArts Migration and then edit the data connection to upload the driver.
   Data Source Authentication and Other Function Configuration
   Username	Yes	The username of the database. The username is required for creating a cluster.
   Password	Yes	The password for accessing the database. The password is required for creating a cluster.
   Data Integration Configuration
   Database Name	Yes	Name of the database to connect
   Use Local API	No	(Optional) Whether to use the local API of the database for acceleration. When you create a MySQL link, CDM automatically enables the local_infile system variable of the MySQL database to enable the LOAD DATA function, which accelerates data import to the MySQL database. If this parameter is enabled, the date data that does not meet the format requirements will be stored as 0000-00-00. For details, visit the official MySQL website. If CDM fails to enable this function, contact the database administrator to enable the local_infile system variable. Alternatively, set Use Local API to No to disable API acceleration. If data is imported to RDS for MySQL, the LOAD DATA function is disabled by default. In such a case, you need to modify the parameter group of the MySQL instance and set local_infile to ON to enable the LOAD DATA function. NOTE: If local_infile on RDS is uneditable, it is the default parameter group. You need to create a parameter group, modify its values, and apply it to the RDS for MySQL instance. For details, see the Relational Database Service User Guide.
   local_infile Character Set	No	This parameter is displayed when Use Local API is enabled. When using local_infile to import data to MySQL, you can configure the encoding format. The default value is utf8.
   Fetch Size	No	Set the number of rows obtained by each request. Set this parameter based on the data source and the job's data size. This parameter is optional. If the value is either too large or too small, the job may run for a long time. The default value is 1,000.
   Commit Size	No	Number of records submitted each time. Set this parameter based on the data destination and the job's data size. If the value is either too large or too small, the job may run for a long time. The default value is 10,000.
   Batch Size	No	Number of rows written each time. This parameter is optional. It should be less than Commit Size. When the number of rows written reaches the value of Commit Size, the rows will be committed to the database. The default value is 100.
   Link Attributes	No	(Optional) Click Add to add the JDBC connector attributes of multiple specified data sources. For details, see the JDBC connector document of the corresponding database. Examples for MySQL: connectTimeout=360000 and socketTimeout=360000: When a large amount of data needs to be migrated or the entire table is retrieved using query statements, the migration fails due to connection timeout. In this case, you can customize the connection timeout interval (ms) and socket timeout interval (ms) to prevent failures caused by timeout. tinyInt1isBit=false or mysql.bool.type.transform=false: By default, tinyInt1isBit is true, indicating that TINYINT(1) is processed as a bit, that is, Types.BOOLEAN, and 1 or 0 is read as true or false. As a result, the migration fails. In this case, you can set tinyInt1isBit to false to avoid migration failures. useCursorFetch=false: By default, useCursorFetch is set to true, indicating that the JDBC connector communicates with relational databases using a binary protocol. Some third-party systems may have compatibility issues, causing migration time conversion errors. In this case, you can disable this function. Open-source MySQL databases support the useCursorFetch parameter, and you do not need to set this parameter. allowPublicKeyRetrieval=true: By default, public key retrieval is disabled for MySQL databases. If TLS is unavailable and an RSA public key is used for encryption, connection to a MySQL database may fail. In this case, you can enable public key retrieval to avoid connection failures.
   Reference Sign	No	(Optional) Delimiter between the names of the referenced tables or columns. For details, see the product documentation of the corresponding database. The default value is ".

5. Click Test to test connectivity of the data connection. If the test fails, the data connection fails to be created.
6. After the test is successful, click Save to create the data connection.

Reference

1. What Are the Precautions for Creating an RDS Data Connection?

   When creating an RDS data connection, you need to bind an agent provided by the CDM cluster. Currently, a version of the CDM cluster earlier than 1.8.6 is not supported.