Adding a Hudi Data Source

Prerequisites for Adding a Hudi Data Source

• You have created the proxy user of the Hudi data source. The proxy user is a human-machine user and must belong to the hive group.
• On all nodes in the cluster where HetuEngine is located, update the /etc/hosts file to include the host name and IP address mappings for the cluster that contains the data source to be connected. If a new node is added to the data source cluster, you must update the /etc/hosts file across all HetuEngine nodes. Ensure that external clients use the same mappings as the HetuEngine cluster. In addition, add the entry 10.10.10.10 hadoop.System domain name (for example, 10.10.10.10 hadoop.hadoop.com) from the data source cluster to the /etc/hosts file on all nodes in the cluster where HetuEngine is located. Without these settings, HetuEngine cannot connect to nodes outside its cluster using host names.
• You have created a HetuEngine administrator by referring to Creating a HetuEngine Permission Role.
• The cluster containing the data source must use the same time as the HetuEngine cluster.

Procedure for Adding a Hudi Data Source

1. Obtain the hdfs-site.xml and core-site.xml configuration files of the Hudi data source cluster.
   1. Log in to FusionInsight Manager of the cluster where the Hudi data source is located.
   2. In the upper right corner of the homepage, click Download Client to download the complete client to the local PC as prompted.
   3. Decompress the downloaded client file package and obtain the core-site.xml and hdfs-site.xml files from the FusionInsight_Cluster_1_Services_ClientConfig/HDFS/config directory.
   4. Check whether the core-site.xml file contains the fs.trash.interval configuration item. If no, add the following configuration items:

      <property>
      <name>fs.trash.interval</name>
      <value>2880</value>
      </property>

   5. Change the value of dfs.client.failover.proxy.provider.NameService name in the hdfs-site.xml file to org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.
      For example, if the NameService name is hacluster, the configuration is as follows:

      <property>
      <name>dfs.client.failover.proxy.provider.hacluster</name>
      <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
      </property>

      

      If the Hudi data source to be connected and HetuEngine are in the same Hadoop cluster, obtain the hdfs-site.xml and core-site.xml configuration files from the HDFS client of the cluster. To be specific, log in to the HDFS client by referring to Using the HDFS Client and run the following commands:

      hdfs dfs -get /user/hetuserver/fiber/restcatalog/hive/core-site.xml

      hdfs dfs -get /user/hetuserver/fiber/restcatalog/hive/hdfs-site.xml

2. Obtain the user.keytab and krb5.conf files of the proxy user of the Hudi data source.
   1. Log in to FusionInsight Manager of the cluster where the Hudi data source is located.
   2. Choose System > Permission > User.
   3. Locate the row that contains the target data source user, click More in the Operation column, and select Download Authentication Credential.
   4. Decompress the downloaded package to obtain the user.keytab and krb5.conf files.
      

      The proxy user of the Hudi data source must be associated with at least the hive user group.

3. Obtain the metastore URL and server principal.
   1. Decompress the client package of the cluster where the Hudi data source is located and obtain the hive-site.xml file from the FusionInsight_Cluster_1_Services_ClientConfig/Hive/config directory.
   2. Open the hive-site.xml file and search for hive.metastore.uris. The value of hive.metastore.uris is the value of the metastore URL. Search for hive.server2.authentication.kerberos.principal. The value of hive.server2.authentication.kerberos.principal is the value of the server principal.

4. Log in to FusionInsight Manager as a HetuEngine administrator and choose Cluster > Services > HetuEngine. The HetuEngine service page is displayed.
5. In the Basic Information area on the Dashboard page, click the link next to HSConsole WebUI. The HSConsole page is displayed.
6. Choose Data Source and click Add Data Source. Configure parameters on the Add Data Source page.
   1. In the Basic Configuration area, set Name and select Hudi for Data Source Type.
   2. In the Hudi Configuration area, configure the parameters according to Table 1.

      Table 1 Hudi configuration

      Parameter	Description	Example Value
      Driver	The default value is hudi.	hudi
      hdfs-site File	Select the hdfs-site.xml configuration file obtained in 1. The file name is fixed.	-
      core-site File	Select the core-site.xml configuration file obtained in 1. The file name is fixed.	-
      krb5 File	Configure this parameter when the security mode is enabled. It is the configuration file used for Kerberos authentication. Select the krb5.conf file obtained in 2.	krb5.conf

   3. In the Metastore Configuration area, configure the parameters according to Table 2.

      Table 2 Metastore configuration

      Parameter	Description	Example Value
      Metastore URL	URL of the metastore of the data source. For details about how to obtain the value, see 3.	thrift://10.92.8.42:21088,thrift://10.92.8.43:21088,thrift://10.92.8.44:21088
      Security Authentication Mechanism	After the security mode is enabled, the default value is KERBEROS.	KERBEROS
      Server Principal	Configure this parameter when the security mode is enabled. Specify the username with domain name used by meta to access the metastore. For details about how to obtain the value, see 3.	hive/hadoop.hadoop.com@HADOOP.COM
      Client Principal	Configure this parameter when the security mode is enabled. The parameter format is as follows: username for accessing the metastore@domain name in uppercase.COM. Username for accessing the metastore is the user to which the user.keytab file obtained in 2 belongs.	admintest@HADOOP.COM
      Keytab File	Configure this parameter when the security mode is enabled. Specify the keytab credential file of the username for accessing the metastore. The file name is fixed. Select the user.keytab file obtained in 2.	user.keytab

   4. (Optional) Modify custom configurations (in MRS 3.3.0 or later).
      • Click Add to add parameters and configure them by referring to Table 3.

      Table 3 Custom configurations

      Parameter	Description	Example Value
      hive.parquet.use-column-names	Whether to access Parquet storage files by column name. true: yes false (default value): no	false
      hive.hdfs.wire-encryption.enabled	This parameter needs to be added and set to false if the hadoop.rpc.protection parameter of the data source HDFS is set to authentication or integrity.	false
      hive.ignore-absent-partitions	Whether to ignore partitions when the file system location does not exist rather than failing the query. true: Ignore partitions when the file system location does not exist. false: Do not ignore partitions when the file system location does not exist. In this case, an error is reported. If this parameter is left blank when data sources are manually connected, false is used by default.	true
      hudi.has.operation.field	Whether to enable the function of writing Changlog messages to the Hudi MOR table. true: Writing Changelog messages to the Hudi MOR table is enabled. false: Writing Changelog messages to the Hudi MOR table is disabled. The corresponding session parameter is infer_operation_field. This parameter value is automatically calculated for Hudi clusters running on MRS 3.6.0 or later.	true

      • You can click Delete to delete custom configuration parameters.
   5. Click OK.

7. Log in to the node where the cluster client is located and run the following commands to switch to the client installation directory and authenticate the user:

   cd /opt/client

   source bigdata_env

   kinit User performing HetuEngine operations (If the cluster is in normal mode, skip this step.)

8. Log in to the catalog of the data source.

   hetu-cli --catalog Data source name --schema Database name

   For example, run the following command:

   hetu-cli --catalog hudi --schema default

9. Run the following command. If the database table information can be viewed or no error is reported, the connection is successful.

   show tables;

Hudi Data Type Mapping

Currently, the Hudi data source supports the following data types: INT, BIGINT, FLOAT, DOUBLE, DECIMAL, STRING, DATE, TIMESTAMP, BOOLEAN, BINARY, MAP, STRUCT, and ARRAY.

Performance Optimization

• Metadata caching

  Hudi connectors support metadata caching to provide metadata requests for various operations faster. For details, see Adjusting HetuEngine Metadata Caching.

• Dynamic filtering

  Enabling dynamic filtering helps optimize the calculation of the Join operator of Hudi connectors. For details, see Enabling Dynamic Filtering in HetuEngine.

• Query with partition conditions

  Creating a partitioned table and querying data with partitioning filter criteria help filter out some partition data, improving performance.

Constraints on Hudi Data Sources

You can only run queries on the Hudi data source. For other operations such as DDL, Insert, Update, and Delete, see Using Hudi.