Creating a Metadata Connection
Create a connection to your metadata source, so you can collect source metadata to extract the list of databases and tables.
You can create connections to big data cloud services and components listed in Table 1.
|
Type |
Metadata Source |
|---|---|
|
Big data components |
|
|
Big data cloud services |
|
- Before creating a connection to Hive Metastore, you need to obtain the IP address and port required for connecting to Hive Metastore
- Before creating a connection to MaxCompute, you need to obtain the endpoint for the region where your MaxCompute resources are hosted.
- Before creating a connection to Doris, you need to obtain the IP address and port required for accessing the Doris cluster
- Before creating a connection to HBase, you need to obtain the IP address and connection port of at least one ZooKeeper node in the HBase cluster.
Prerequisites
You have installed the MgC Agent on a server in the source environment.
Procedure
- Sign in to the MgC console. In the navigation pane, under Project, select a big data migration project from the drop-down list.
- In the navigation pane, choose Survey > Preparations.
- Click Create Connection.
Figure 1 Creating a connection
- Select a connection type and click Next.
- Based on the selected connection type, configure connection parameters.
- Parameters for creating a connection to Hive Metastore
- Parameters for creating a connection to Doris
- Parameters for creating a connection to HBase
- Parameters for creating a ClickHouse connection
- Parameters for creating a connection to Delta Lake (with metadata)
- Parameters for creating a connection to Delta Lake (without metadata)
- Parameters for creating a connection to Hudi (with metadata)
- Parameters for creating a connection to Hudi (without metadata)
- Parameters for creating a connection to MaxCompute
- Parameters for creating a connection to ApsaraDB for ClickHouse
Table 2 Parameters for creating a connection to Hive Metastore Parameter
Configuration
Connection To
Configure this parameter based on requirements.
Connection Name
The default name is Hive-Metastore-4 random characters (including letters and numbers). You can also customize a name.
MgC Agent
Select the MgC Agent installed in the source environment.
Secure Connection
Determine whether to enable secure connection.
- If Hive Metastore is deployed in an unsecured cluster, do not enable secure connection.
- If Hive Metastore is deployed in a secured cluster, enable secure connection and provide access credentials. For details about how to obtain and add credentials to MgC, see "Big data - Hive Metastore" in Adding Resource Credentials.
Hive Version
Select the Hive version at the source.
CAUTION:If the source Hive version is 2.1.1, select 1.x.
Hive Metastore IP Address
Enter the IP address for connecting to the Hive Metastore node.
Hive Metastore Thrift Port
Enter the port for connecting to the Hive Metastore Thrift service. The default port is 9083.
Connect to Metadata Database
During an incremental data verification, querying with Hive Metastore on more than 30,000 partitions may lead to a memory overflow (OOM) since all partition information is loaded into memory. Connecting to the MySQL metadata database can effectively prevent this issue.
- If you disable this option, the system queries the information of Hive tables and partitions using Hive Metastore.
- If you enable this option, configure the MySQL database information. The system will query the information of Hive tables and partitions through the MySQL database. You need to set the following parameters:
- Metadata Database Type: Only MySQL is supported.
- MySQL Credential: Select the credential for accessing the MySQL database. You need to add the credential to the MgC Agent and synchronize it to MgC. For details, see Adding Resource Credentials.
- MySQL Node IP Address: Enter the IP address of the MySQL database server.
- MySQL Port: Enter the port of the MySQL database service.
- Database Name: Enter the name of the database that stores the Hive table metadata.
NOTE:Ensure that the entered MySQL credential, node IP address, service port, and database name match the MySQL database used by Hive. Otherwise, data verification will fail.
Table 3 Parameters for creating a connection to Doris Parameter
Configuration
Connection To
Configure this parameter based on requirements.
Connection Name
The default name is Doris-4 random characters (including letters and numbers). You can also customize a name.
MgC Agent
Select the MgC Agent installed in the source environment.
Doris Credential
Select the Doris credential added to the MgC Agent. For details about how to add credentials, see "Big data - Doris" in Adding Resource Credentials.
Database IP Address
Enter the IP address for accessing the Doris cluster.
Database Port
Enter the port used for accessing the Doris cluster. The default value is 3306.
Database Name
Enter the name of the Doris database.
Collect Usage Metrics
This parameter is optional. If this option is enabled, usage metrics for your big data resources will be collected during the execution of tasks created using this connection. The collected information is used to generate reports on the MgC console and for performance optimization.
NOTICE:Before using this function, ensure that the Huawei Cloud account you added to the MgC Agent has the read-only permission for MRS and DLI.
- If the selected credential is the one you currently use to access MgC, you can select This is my MgC credential, and the projects in the region you choose will be listed.
- Under Region, select the region where the data to be verified is located.
- Under Project, select the project where the data to be verified is stored.
- Under Cluster ID, enter the ID of the cluster where the data to be verified is located.
- If the selected Doris credential is not the one you currently use to access MgC:
- Under Region ID, enter the ID of the region where the data to be verified is located. For example, if the region is CN South-Guangzhou, enter cn-south-1.
- Under Project ID, enter the project ID corresponding to the region.
- Under Cluster ID, enter the ID of the cluster where the data to be verified is located.
NOTE:- To view the region ID and project ID, choose My Credentials > API Credentials.
- For details about how to obtain the cluster ID, see Obtaining an MRS Cluster ID.
Table 4 Parameters for creating a connection to HBase Parameter
Configuration
Connection To
Configure this parameter based on requirements.
Connection Name
The default name is HBase-4 random characters (including letters and numbers). You can also customize a name.
MgC Agent
Select the MgC Agent installed in the source environment.
CAUTION:You need to configure /etc/hosts on the server where the MgC Agent is installed, so the HBase metadata connection can be established. For more information, see Preparing for Development and Operating Environments.
HBase Credential
Select the HBase credential added to the MgC Agent. For details about how to add credentials, see "Big data - HBase" in Adding Resource Credentials.
Secured Cluster
Choose whether the cluster is secured.
ZooKeeper IP Address
Enter the IP address for connecting to the ZooKeeper node. You can enter the public or private IP address of the ZooKeeper node.
ZooKeeper Connection Port
Enter the port for connecting to the ZooKeeper node.
HBase Version
Select the HBase version.
Table 5 Parameters for creating a connection to ClickHouse Parameter
Configuration
Connection To
Configure this parameter based on requirements.
Connection Name
The default name is ClickHouse-4 random characters (including letters and numbers). You can also customize a name.
MgC Agent
Select the MgC Agent installed in the source environment.
ClickHouse Credential (Optional)
Select the ClickHouse credential added to the MgC Agent. For details about how to add credentials, see "Big data - ClickHouse" in Adding Resource Credentials.
Secured Cluster
Choose whether the cluster is secured.
ClickHouse IP Address
Enter the IP address of the server that hosts ClickHouse.
HTTP Port
If the ClickHouse cluster is unsecured, enter the HTTP port for communicating with the ClickHouse server. The default value is 8123.
HTTP SSL/TLS Port
If the ClickHouse cluster is secured, enter the HTTPS port for communicating with the ClickHouse server.
Table 6 Parameters for creating a connection to Delta Lake (with metadata) Parameter
Configuration
Connection To
You only need to create a metadata connection to the source.
Connection Name
The default name is Delta-Lake-with-metadata-4 random characters (including letters and numbers). You can also customize a name.
MgC Agent
Select the MgC Agent installed in the source environment.
Executor Credential
Select the login credential of the executor. For details about how to add credentials, see "Big data - Executor" in Adding Resource Credentials.
Executor IP Address
Enter the IP address for connecting to the executor.
Executor Port
Enter the port for connecting to the executor.
Spark Client Directory
Enter the installation directory of the Spark client.
Environment Variable Address
Enter the absolute path of the environment variable file (configuration file), for example, /opt/bigdata/client/bigdata_env.
SQL File Location
Enter a directory for storing the SQL files generated for consistency verification. You must have the read and write permissions for the directory.
NOTICE:After the migration is complete, you need to manually clear the folders generated at this location to release storage space.
Table 7 Parameters for creating a connection to Delta Lake (without metadata) Parameter
Configuration
Connection To
You only need to create a metadata connection to the source.
Connection Name
The default name is Delta-Lake-without-metadata-4 random characters (including letters and numbers). You can also customize a name.
MgC Agent
Select the MgC Agent installed in the source environment.
Executor Credential
Select the login credential of the executor. For details about how to add credentials, see "Big data - Executor" in Adding Resource Credentials.
Executor IP Address
Enter the IP address for connecting to the executor.
Executor Port
Enter the port for connecting to the executor.
Spark Client Directory
Enter the installation directory of the Spark client.
Environment Variable Address
Enter the absolute path of the environment variable file (configuration file), for example, /opt/bigdata/client/bigdata_env.
SQL File Location
Enter a directory for storing the SQL files generated for consistency verification. You must have the read and write permissions for the directory.
NOTICE:After the migration is complete, you need to manually clear the folders generated at this location to release storage space.
Collect Usage Metrics
This parameter is optional. If this option is enabled, usage metrics for your big data resources will be collected during the execution of tasks created using this connection. The collected information is used to generate reports on the MgC console and for performance optimization.
NOTICE:Before using this function, ensure that the Huawei Cloud account you added to the MgC Agent has the read-only permission for MRS and DLI.
- If the selected credential is the one you currently use to access MgC, you can select This is my MgC credential, and the projects in the region you choose will be listed.
- Under Region, select the region where the data to be verified is located.
- Under Project, select the project where the data to be verified is stored.
- Under Cluster ID, enter the ID of the cluster where the data to be verified is located.
- If the selected Doris credential is not the one you currently use to access MgC:
- Under Region ID, enter the ID of the region where the data to be verified is located. For example, if the region is CN South-Guangzhou, enter cn-south-1.
- Under Project ID, enter the project ID corresponding to the region.
- Under Cluster ID, enter the ID of the cluster where the data to be verified is located.
NOTE:- To view the region ID and project ID, choose My Credentials > API Credentials.
- For details about how to obtain the cluster ID, see Obtaining an MRS Cluster ID.
Table 8 Parameters for creating a connection to Hudi (with metadata) Parameter
Configuration
Connection To
You only need to create a metadata connection to the source.
Connection Name
The default name is Hudi-with-metadata-4 random characters (including letters and numbers). You can also customize a name.
MgC Agent
Select the MgC Agent installed in the source environment.
Executor Credential
Select the login credential of the executor. For details about how to add credentials, see "Big data - Executor" in Adding Resource Credentials.
Executor IP Address
Enter the IP address for connecting to the executor.
Executor Port
Enter the port for connecting to the executor.
Spark Client Directory
Enter the installation directory of the Spark client.
Environment Variable Address
Enter the absolute path of the environment variable file (configuration file), for example, /opt/bigdata/client/bigdata_env.
SQL File Location
Enter a directory for storing the SQL files generated for consistency verification. You must have the read and write permissions for the directory.
NOTICE:After the migration is complete, you need to manually clear the folders generated at this location to release storage space.
Table 9 Parameters for creating a connection to Hudi (without metadata) Parameter
Configuration
Connection To
You only need to create a metadata connection to the source.
Connection Name
The default name is Hudi-without-metadata-4 random characters (including letters and numbers). You can also customize a name.
MgC Agent
Select the MgC Agent installed in the source environment.
Executor Credential
Select the login credential of the executor. For details about how to add credentials, see "Big data - Executor" in Adding Resource Credentials.
Executor IP Address
Enter the IP address for connecting to the executor.
Executor Port
Enter the port for connecting to the executor.
Spark Client Directory
Enter the installation directory of the Spark client.
Environment Variable Address
Enter the absolute path of the environment variable file (configuration file), for example, /opt/bigdata/client/bigdata_env.
SQL File Location
Enter a directory for storing the SQL files generated for consistency verification. You must have the read and write permissions for the directory.
NOTICE:After the migration is complete, you need to manually clear the folders generated at this location to release storage space.
Collect Usage Metrics
This parameter is optional. If this option is enabled, usage metrics for your big data resources will be collected during the execution of tasks created using this connection. The collected information is used to generate reports on the MgC console and for performance optimization.
NOTICE:Before using this function, ensure that the Huawei Cloud account you added to the MgC Agent has the read-only permission for MRS and DLI.
- If the selected credential is the one you currently use to access MgC, you can select This is my MgC credential, and the projects in the region you choose will be listed.
- Under Region, select the region where the data to be verified is located.
- Under Project, select the project where the data to be verified is stored.
- Under Cluster ID, enter the ID of the cluster where the data to be verified is located.
- If the selected Doris credential is not the one you currently use to access MgC:
- Under Region ID, enter the ID of the region where the data to be verified is located. For example, if the region is CN South-Guangzhou, enter cn-south-1.
- Under Project ID, enter the project ID corresponding to the region.
- Under Cluster ID, enter the ID of the cluster where the data to be verified is located.
NOTE:- To view the region ID and project ID, choose My Credentials > API Credentials.
- For details about how to obtain the cluster ID, see Obtaining an MRS Cluster ID.
Table 10 Parameters for creating a connection to MaxCompute Parameter
Configuration
Connection To
You only need to create a metadata connection to the source.
Connection Name
The default name is MaxCompute-4 random characters (including letters and numbers). You can also customize a name.
MgC Agent
Select the MgC Agent installed in the source environment.
Alibaba Cloud Credential
Select the credential of the source Alibaba Cloud account. For details about how to add credentials, see "Big data - MaxCompute" in Adding Resource Credentials.
MaxCompute Project
Enter the name of the MaxCompute project to be collected. You can obtain the project name from the MaxCompute console.
Endpoint
Enter the endpoint of the region where the MaxCompute project is located.
For details about MaxCompute endpoints in different regions, see MaxCompute Endpoints.
Table 11 Parameters for creating a connection to ApsaraDB for ClickHouse Parameter
Configuration
Connection To
You only need to create a metadata connection to the source.
Connection Name
The default name is ApsaraDB for ClickHouse-4 random characters (including letters and numbers). You can also customize a name.
MgC Agent
Select the MgC Agent installed in the source environment.
ClickHouse Credential
Select the ApsaraDB for ClickHouse credential added to the MgC Agent. For details about how to add credentials, see "Big data - ClickHouse" in Adding Resource Credentials.
Database URL
Enter the public address of the source ClickHouse cluster. You can view the IP address in the cluster details.
- Click Test. MgC verifies whether the connection can be established using the configuration information. If the test is successful, the connection can be established.
- After the connection test is successful, click Confirm. The metadata connection is created.
- Click Metadata connections in the Metadata Collection area. On the Connections tab, you can view the list of created metadata connections and their basic information. Locate the connection you created and in the Operation column, click Modify to modify the connection settings.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot
