On this page

Show all

Help Center/ MapReduce Service/ Component Operation Guide (LTS) (Ankara Region)/ Using HetuEngine/ Configuring Data Sources/ Configuring a Hudi Data Source

Configuring a Hudi Data Source

Updated on 2024-11-29 GMT+08:00

View PDF

Scenario

HetuEngine supports the query of COW/MOR table data. Configure a Hudi data source on HSConsole.

NOTE:

HetuEngine cannot read Hudi bootstrap tables.

Prerequisites

You have created the proxy user of the Hudi data source. The proxy user is a human-machine user and must belong to the hive group.
In the /etc/hosts file of all nodes in the cluster where HetuEngine is located, add the mapping between the host names and IP addresses of the cluster where the data source to be connected is located, and add 10.10.10.10 hadoop.System domain name in the /etc/hosts file (for example, 10.10.10.10 hadoop.hadoop.com). Otherwise, HetuEngine cannot connect to the nodes that are not in the cluster based on the host name.
You have created a HetuEngine administrator by referring to Creating a HetuEngine User.

Procedure

Obtain the hdfs-site.xml and core-site.xml configuration files of the Hudi data source cluster.
1. Log in to FusionInsight Manager of the cluster where the Hudi data source is.
2. In the upper right corner of the homepage, click Download Client to download the complete client to the local PC as prompted.
3. Decompress the downloaded client file package and obtain core-site.xml and hdfs-site.xml from the FusionInsight_Cluster_1_Services_ClientConfig/HDFS/config directory.
4. Check whether the core-site.xml file contains the fs.trash.interval configuration item. If not, add the following configuration items:
```
<property>
<name>fs.trash.interval</name>
<value>2880</value>
</property>
```
5. Change the value of dfs.client.failover.proxy.provider.NameService name in the hdfs-site.xml file to org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.
  For example, if the NameService name is hacluster, the configuration is as follows:
```
<property>
<name>dfs.client.failover.proxy.provider.hacluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
```
  NOTICE:
  
  If the Hudi data source to be connected and HetuEngine are in the same Hadoop cluster, obtain the hdfs-site.xml and core-site.xml configuration files from the HDFS client of the cluster. To be specific, log in to the HDFS client by referring to Using the HDFS Client and run the following commands:
  
  hdfs dfs -get /user/hetuserver/fiber/restcatalog/hive/core-site.xml
  
  hdfs dfs -get /user/hetuserver/fiber/restcatalog/hive/hdfs-site.xml
Obtain the user.keytab and krb5.conf files of the proxy user of the Hudi data source.
1. Log in to FusionInsight Manager of the cluster where the Hudi data source is.
2. Choose System > Permission > User.
3. Locate the row that contains the target data source user, click More in the Operation column, and select Download Authentication Credential.
4. Decompress the downloaded package to obtain user.keytab and krb5.conf.
  NOTE:
  
  The proxy user of the Hive data source must be associated with at least the hive user group.
Obtain the MetaStore URL and the Principal of the server.
1. Decompress the client package of the cluster where the Hudi data source is and obtain the hive-site.xml file from the FusionInsight_Cluster_1_Services_ClientConfig/Hive/config directory.
2. Open hive-site.xml and search for hive.metastore.uris. The value of hive.metastore.uris is the value of MetaStore URL. Search for hive.server2.authentication.kerberos.principal. The value of hive.server2.authentication.kerberos.principal is the value of Principal on the server.
Log in to FusionInsight Manager as a HetuEngine administrator and choose Cluster > Services > HetuEngine. The HetuEngine service page is displayed.
In the Basic Information area on the Dashboard page, click the link next to HSConsole WebUI. The HSConsole page is displayed.

Choose Data Source and click Add Data Source. Configure parameters on the Add Data Source page.

In the Basic Configuration area, configure Name and choose Hudi for Data Source Type.

In the Hudi Configuration area, configure the parameters according to Table 1.

**Table 1** Hudi configuration
Parameter	Description	Example Value
Driver	The default value is hudi.	hudi
hdfs-site File	Select the hdfs-site.xml configuration file obtained in 1. The file name is fixed.	-
core-site File	Select the core-site.xml configuration file obtained in 1. The file name is fixed.	-
krb5 File	Configure this parameter when the security mode is enabled. It is the configuration file used for Kerberos authentication. Select the krb5.conf file obtained in 2.	krb5.conf

In the Metastore Configuration area, configure the parameters according to Table 2.

**Table 2** MetaStore Configuration
Parameter	Description	Example Value
Metastore URL	URL of the MetaStore of the data source. For details, see 3.	thrift://10.92.8.42:21088,thrift://10.92.8.43:21088,thrift://10.92.8.44:21088
Security Authentication Mechanism	After the security mode is enabled, the default value is KERBEROS.	KERBEROS
Server Principal	Configure this parameter when the security mode is enabled. It specifies the username with domain name used by meta to access MetaStore. For details, see 3.	hive/hadoop.hadoop.com@HADOOP.COM
Client Principal	Configure this parameter when the security mode is enabled. The parameter format is as follows: Username for accessing MetaStore@Domain name (uppercase).COM. Username for accessing MetaStore is the user to which the user.keytab file obtained in 2 belongs.	admintest@HADOOP.COM
Keytab File	Configure this parameter when the security mode is enabled. It specifies the keytab credential file of the MetaStore username. The file name is fixed. Select the user.keytab file obtained in 2.	user.keytab

Click OK.

Log in to the node where the cluster client is located and run the following commands to switch to the client installation directory and authenticate the user:

cd /opt/client

source bigdata_env

kinit User performing HetuEngine operations (If the cluster is in normal mode, skip this step.)
Run the following command to log in to the catalog of the data source:

hetu-cli --catalog Data source name --schema Database name

For example, run the following command:

hetu-cli --catalog hudi_1 --schema default
Run the following command. If the database table information can be viewed or no error is reported, the connection is successful.

show tables;

Data Type Mapping

Currently, the Hudi data source supports the following data types: INT, BIGINT, FLOAT, DOUBLE, DECIMAL, STRING, DATE, TIMESTAMP, BOOLEAN, BINARY, MAP, STRUCT and ARRAY.

Performance Optimization

Metadata caching
Hudi connectors support metadata caching to provide metadata requests for various operations faster. For details, see Adjusting Metadata Cache.
Dynamic filtering
Enabling dynamic filtering helps optimize the calculation of the Join operator of Hudi connectors. For details, see Enabling Dynamic Filtering.
Query with partition conditions
Creating a partitioned table and querying data with partition filter criteria help filter out some partition data, improving performance.

Constraints

Hudi data sources support only the QUERY operation.

Parent topic: Configuring Data Sources

Previous topic: Configuring an Independently Deployed Hive Data Source

Next topic: Configuring a ClickHouse Data Source

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot