Help Center/ Migration Center/ User Guide/ Big Data Lineage/ Collecting Metadata/ Creating a Metadata Connection
Updated on 2024-11-01 GMT+08:00
View PDF
Share

Creating a Metadata Connection

Create a connection to your metadata source, so you can collect source metadata to extract the list of databases and tables.

You can create connections to big data cloud services and components listed in Table 1.

Table 1 Supported metadata sources

Type

Metadata Source

Big data components
  • Hive Metastore
  • Doris
  • HBase
  • ClickHouse
  • Delta Lake (with metadata)
  • Delta Lake (without metadata)
  • Hudi (with metadata)
  • Hudi (without metadata)

Big data cloud services
  • MaxCompute
  • ApsaraDB for ClickHouse
  • Before creating a connection to Hive Metastore, you need to obtain the IP address and port required for connecting to Hive Metastore
  • Before creating a connection to MaxCompute, you need to obtain the endpoint for the region where your MaxCompute resources are hosted.
  • Before creating a connection to Doris, you need to obtain the IP address and port required for accessing the Doris cluster
  • Before creating a connection to HBase, you need to obtain the IP address and connection port of at least one ZooKeeper node in the HBase cluster.

Prerequisites

You have installed Edge on a server in the source environment.

Procedure

  1. Sign in to the MgC console.
  2. In the navigation pane on the left, choose Research > Data Lineage. Select a migration project in the upper left corner of the page.
  3. In the Metadata Collection area, click Create Connection.

  4. Select a connection type and click Next.
  5. Based on the selected connection type, configure connection parameters.

    Table 2 Parameters for creating a connection to Hive Metastore

    Parameter

    Configuration

    Connection Source

    You only need to create a metadata connection to the source.

    Connection Name

    The default name is Hive-Metastore-4 random characters (including letters and numbers). You can also customize a name.

    Edge Device

    Select the Edge device in the source environment.

    Secure Connection

    Determine whether to enable secure connection.

    • If Hive Metastore is deployed in an unsecured cluster, do not enable secure connection.
    • If Hive Metastore is deployed in a secured cluster, enable secure connection and provide access credentials. For details about how to obtain and add credentials to MgC, see "Big data - Hive Metastore" in Adding Resource Credentials.

    Hive Version

    Select the Hive version at the source.

    CAUTION:

    If the source Hive version is 2.1.1, select 1.x.

    Hive Metastore IP Address

    Enter the IP address for connecting to the Hive Metastore node.

    Hive Metastore Thrift Port

    Enter the port for connecting to the Hive Metastore Thrift service. The default port is 9083.
    Table 3 Parameters for creating a connection to Doris

    Parameter

    Configuration

    Connection Source

    You only need to create a metadata connection to the source.

    Connection Name

    The default name is Doris-4 random characters (including letters and numbers). You can also customize a name.

    Edge Device

    Select the Edge device in the source environment.

    Doris Credential

    Select the Doris credential added to the Edge device. For details about how to add credentials, see "Big data - Doris" in Adding Resource Credentials.

    Database IP Address

    Enter the IP address for accessing the Doris cluster.

    Database Port

    Enter the port used for accessing the Doris cluster. The default value is 3306.

    Database Name

    Enter the name of the Doris database.
    Table 4 Parameters for creating a connection to HBase

    Parameter

    Configuration

    Connection Source

    You only need to create a metadata connection to the source.

    Connection Name

    The default name is HBase-4 random characters (including letters and numbers). You can also customize a name.

    Edge Device

    Select the Edge device in the source environment.

    CAUTION:

    You need to configure /etc/hosts on the server where the Edge is installed, so the HBase metadata connection can be established. For more information, see Preparing for Development and Operating Environments.

    HBase Credential

    Select the HBase credential added to the Edge device. For details about how to add credentials, see "Big data - HBase" in Adding Resource Credentials.

    Secured Cluster

    Choose whether the cluster is secured.

    Zookeeper IP Address

    Enter the IP address for connecting to the ZooKeeper node. You can enter the public or private IP address of the ZooKeeper node.

    ZooKeeper Connection Port

    Enter the port for connecting to the ZooKeeper node.

    HBase Version

    Select the HBase version.
    Table 5 Parameters for creating a connection to ClickHouse

    Parameter

    Configuration

    Connection Source

    You only need to create a metadata connection to the source.

    Connection Name

    The default name is ClickHouse-4 random characters (including letters and numbers). You can also customize a name.

    Edge Device

    Select the Edge device in the source environment.

    ClickHouse Credential (Optional)

    Select the ClickHouse credential added to Edge. For details about how to add credentials, see "Big Data - ClickHouse" in Adding Resource Credentials.

    Secured Cluster

    Choose whether the cluster is secured.

    ClickHouse IP Address

    Enter the IP address of the server that hosts ClickHouse.

    HTTP Port

    If the ClickHouse cluster is unsecured, enter the HTTP port for communicating with the ClickHouse server. The default value is 8123.

    HTTP SSL/TLS Port

    If the ClickHouse cluster is secured, enter the HTTPS port for communicating with the ClickHouse server.
    Table 6 Parameters for creating a connection to Delta Lake (with metadata)

    Parameter

    Configuration

    Connection Source

    You only need to create a metadata connection to the source.

    Connection Name

    The default name is Delta-Lake-with-metadata-4 random characters (including letters and numbers). You can also customize a name.

    Edge Device

    Select the Edge device in the source environment.

    Executor Credential

    Select the login credential of the executor. For details about how to add credentials, see "Big Data - Executor" in Adding Resource Credentials.

    Executor IP Address

    Enter the IP address for connecting to the executor.

    Executor Port

    Enter the port for connecting to the executor.

    Spark Client Directory

    Enter the installation directory of the Spark client.

    Environment Variable Address

    Enter the absolute path of the environment variable file (configuration file), for example, /opt/bigdata/client/bigdata_env.

    SQL File Location

    Enter the directory for storing the SQL files generated for consistency verification. You must have the read and write permissions for the directory.

    NOTICE:

    After the migration is complete, you need to manually clear the folders generated at this location to release storage space.
    Table 7 Parameters for creating a connection to Delta Lake (without metadata)

    Parameter

    Configuration

    Connection Source

    You only need to create a metadata connection to the source.

    Connection Name

    The default name is Delta-Lake-without-metadata-4 random characters (including letters and numbers). You can also customize a name.

    Edge Device

    Select the Edge device in the source environment.

    Executor Credential

    Select the login credential of the executor. For details about how to add credentials, see "Big Data - Executor" in Adding Resource Credentials.

    Executor IP Address

    Enter the IP address for connecting to the executor.

    Executor Port

    Enter the port for connecting to the executor.

    Spark Client Directory

    Enter the installation directory of the Spark client.

    Environment Variable Address

    Enter the absolute path of the environment variable file (configuration file), for example, /opt/bigdata/client/bigdata_env.

    SQL File Location

    Enter the directory for storing the SQL files generated for consistency verification. You must have the read and write permissions for the directory.

    NOTICE:

    After the migration is complete, you need to manually clear the folders generated at this location to release storage space.
    Table 8 Parameters for creating a connection to Hudi (with metadata)

    Parameter

    Configuration

    Connection Source

    You only need to create a metadata connection to the source.

    Connection Name

    The default name is Hudi-with-metadata-4 random characters (including letters and numbers). You can also customize a name.

    Edge Device

    Select the Edge device in the source environment.

    Executor Credential

    Select the login credential of the executor. For details about how to add credentials, see "Big Data - Executor" in Adding Resource Credentials.

    Executor IP Address

    Enter the IP address for connecting to the executor.

    Executor Port

    Enter the port for connecting to the executor.

    Spark Client Directory

    Enter the installation directory of the Spark client.

    Environment Variable Address

    Enter the absolute path of the environment variable file (configuration file), for example, /opt/bigdata/client/bigdata_env.

    SQL File Location

    Enter the directory for storing the SQL files generated for consistency verification. You must have the read and write permissions for the directory.

    NOTICE:

    After the migration is complete, you need to manually clear the folders generated at this location to release storage space.
    Table 9 Parameters for creating a connection to Hudi (without metadata)

    Parameter

    Configuration

    Connection Source

    You only need to create a metadata connection to the source.

    Connection Name

    The default name is Hudi-without-metadata-4 random characters (including letters and numbers). You can also customize a name.

    Edge Device

    Select the Edge device in the source environment.

    Executor Credential

    Select the login credential of the executor. For details about how to add credentials, see "Big Data - Executor" in Adding Resource Credentials.

    Executor IP Address

    Enter the IP address for connecting to the executor.

    Executor Port

    Enter the port for connecting to the executor.

    Spark Client Directory

    Enter the installation directory of the Spark client.

    Environment Variable Address

    Enter the absolute path of the environment variable file (configuration file), for example, /opt/bigdata/client/bigdata_env.

    SQL File Location

    Enter the directory for storing the SQL files generated for consistency verification. You must have the read and write permissions for the directory.

    NOTICE:

    After the migration is complete, you need to manually clear the folders generated at this location to release storage space.
    Table 10 Parameters for creating a connection to MaxCompute

    Parameter

    Configuration

    Connection Source

    You only need to create a metadata connection to the source.

    Connection Name

    The default name is MaxCompute-4 random characters (including letters and numbers). You can also customize a name.

    Edge Device

    Select the Edge device in the source environment.

    Alibaba Cloud Credential

    Select the credential of the source Alibaba Cloud account. For details about how to add credentials, see "Big data - MaxCompute" in Adding Resource Credentials.

    MaxCompute Project

    Enter the name of the MaxCompute project to be collected. You can obtain the project name from the MaxCompute console.

    Endpoint

    Enter the endpoint of the region where the MaxCompute project is located.

    For details about the MaxCompute endpoints in different regions, see MaxCompute Endpoints.
    Table 11 Parameters for creating a connection to ApsaraDB for ClickHouse

    Parameter

    Configuration

    Connection Source

    You only need to create a metadata connection to the source.

    Connection Name

    The default name is ApsaraDB for ClickHouse-4 random characters (including letters and numbers). You can also customize a name.

    Edge Device

    Select the Edge device in the source environment.

    ClickHouse Credential

    Select the ApsaraDB for ClickHouse credential added to Edge. For details about how to add credentials, see "Big Data - ClickHouse" in Adding Resource Credentials.

    Database URL

    Enter the public address of the source ClickHouse cluster. You can view the IP address in the cluster details.

  6. Click Test. MgC verifies whether the connection can be established using the configuration information. If the test is successful, the connection can be established.
  7. After the connection test is successful, click Confirm. The metadata connection is created.
  8. Click Metadata connections in the Metadata Collection area. On the Connections tab, you can view the list of created metadata connections and their basic information. Locate the connection you created and in the Operation column, click Modify to modify the connection settings.

Parent topic: Collecting Metadata