Adding an Iceberg Data Source

Scenarios

HetuEngine supports querying data in Iceberg v1/v2 tables. This section describes how to configure an Iceberg data source on HSConsole.

Prerequisites

• You have created the proxy user of the Iceberg data source. The proxy user is a human-machine user and has been added to the hive group.
• On all nodes in the cluster where HetuEngine is located, update the /etc/hosts file to include the host name and IP address mappings for the cluster that contains the data source to be connected. If a new node is added to the data source cluster, you must update the /etc/hosts file across all HetuEngine nodes. Ensure that external clients use the same mappings as the HetuEngine cluster. In addition, add the entry 10.10.10.10 hadoop.System domain name (for example, 10.10.10.10 hadoop.hadoop.com) from the data source cluster to the /etc/hosts file on all nodes in the cluster where HetuEngine is located. Without these settings, HetuEngine cannot connect to nodes outside its cluster using host names.
• You have created a HetuEngine administrator by referring to Creating a HetuEngine Permission Role.
• The time of the cluster where the data source is deployed must be the same as that of the HetuEngine cluster.
• This section applies only to MRS 3.6.0-LTS or later.

Procedure

1. Obtain the hdfs-site.xml and core-site.xml configuration files of the Iceberg data source cluster.
   1. Log in to Manager of the cluster where the Iceberg data source is located.
   2. In the upper right corner of the homepage, click Download Client to download the complete client to the local PC as prompted.
   3. Decompress the downloaded client file package and obtain core-site.xml and hdfs-site.xml from the FusionInsight_Cluster_1_Services_ClientConfig/HDFS/config directory.
   4. Check whether the core-site.xml file contains the fs.trash.interval configuration item. If not, add the following configuration items:

      <property>
      <name>fs.trash.interval</name>
      <value>2880</value>
      </property>

   5. Change the value of dfs.client.failover.proxy.provider.NameService name in the hdfs-site.xml file to org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.
      For example, if the NameService name is hacluster, the configuration is as follows:

      <property>
      <name>dfs.client.failover.proxy.provider.hacluster</name>
      <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
      </property>

      

      If the Iceberg data source to be interconnected is in the same Hadoop cluster with HetuEngine, you can log in to the HDFS client and run the following commands to obtain the hdfs-site.xml and core-site.xml configuration files. For details, see Using the HDFS Client.

      hdfs dfs -get /user/hetuserver/fiber/restcatalog/hive/core-site.xml

      hdfs dfs -get /user/hetuserver/fiber/restcatalog/hive/hdfs-site.xml

2. Obtain the user.keytab and krb5.conf files of the proxy user of the Iceberg data source.
   1. Log in to Manager of the cluster where the Iceberg data source is located.
   2. Choose System > Permission > User.
   3. Locate the row that contains the target data source user, click More in the Operation column, and select Download Authentication Credential.
   4. Decompress the downloaded package to obtain the user.keytab and krb5.conf files.
      

      The proxy user of the Iceberg data source must be associated with the hive user group.

3. Obtain the MetaStore URL and the Principal of the server.
   1. Decompress the client package of the cluster where the Iceberg data source is located and obtain the hive-site.xml file from the FusionInsight_Cluster_1_Services_ClientConfig/Hive/config directory.
   2. Open the hive-site.xml file and search for hive.metastore.uris. The value of hive.metastore.uris is the value of MetaStore URL. Search for hive.server2.authentication.kerberos.principal. Its value is the Principal on the server.

4. Log in to FusionInsight Manager as a HetuEngine administrator and choose Cluster > Services > HetuEngine. The HetuEngine service page is displayed.
5. In the Basic Information area on the Dashboard page, click the link next to HSConsole WebUI. The HSConsole page is displayed.
6. Choose Data Source and click Add Data Source. Configure parameters on the Add Data Source page.
   1. In the Basic Configuration area, configure Name and choose Iceberg for Data Source Type.
   2. In the Iceberg Configuration area, configure the parameters according to Table 1.

      Table 1 Iceberg configuration

      Parameter	Description	Example Value
      hdfs-site File	Select the hdfs-site.xml configuration file obtained in 1 from the local PC. The file name is fixed.	-
      core-site File	Select the core-site.xml configuration file obtained in 1 from the local PC. The file name is fixed.	-
      krb5 File	Configure this parameter when the security mode is enabled. It is the configuration file used for Kerberos authentication. Select the krb5.conf file obtained in 2.	krb5.conf

   3. In the MetaStore Configuration area, configure parameters by referring to Table 2 MetaStore configuration.

      Table 2 MetaStore configuration

      Parameter	Description	Example Value
      Metastore URL	URL of the MetaStore of the data source. For details about how to obtain the URL, see 3.	thrift://192.168.1.1:21088,thrift://192.168.1.11:21088,thrift://192.168.1.2:21088
      Security Authentication Mechanism	After the security mode is enabled, the default value is KERBEROS.	KERBEROS
      Server Principal	Configure this parameter when the security mode is enabled. It specifies the username with domain name used by meta to access MetaStore. For details about how to obtain the value, see 3.	hive/hadoop.hadoop.com@HADOOP.COM
      Client Principal	Configure this parameter when the security mode is enabled. The parameter format is as follows: Username for accessing MetaStore@Domain name (uppercase).COM. The username for accessing MetaStore is the user of the user.keytab file obtained in 2.	admintest@HADOOP.COM
      Keytab File	Configure this parameter when the security mode is enabled. It specifies the keytab credential file of the username for accessing the metastore. The file name is fixed. Select the user.keytab file obtained in 2 on the local PC.	user.keytab

   4. (Optional) Modify custom configurations.
      • Click Add to add custom parameters by referring to Table 3 Custom parameters.

      Table 3 Custom parameters

      Parameter	Description	Example Value
      hive.parquet.use-column-names	Determines whether to access Parquet storage files by column name. true: yes false (default value): no	false
      hive.hdfs.wire-encryption.enabled	This parameter needs to be added and set to false if the hadoop.rpc.protection parameter of the data source HDFS is set to authentication or integrity.	false
      hive.ignore-absent-partitions	Query whether any file is missing in a partition. Value options are as follows: true: Queries whether files are missing in the partition. false: Do not query whether files are missing in the partition. In this case, an error is reported. If this parameter is left blank when data sources are manually connected, false is used by default.	true
      iceberg.register-table-procedure.enabled	Allows users to invoke the register_table procedure. This parameter is used to register existing iceberg tables with metadata during data migration. The default value is false.	false

      • You can click Delete to delete custom parameters.
   5. Then, click OK.

7. Log in to the node where the cluster client is located and run the following commands to switch to the client installation directory and authenticate the user:

   cd /opt/client

   source bigdata_env

   kinit User performing HetuEngine operations (If the cluster is in normal mode, skip this step.)

8. Log in to the catalog of the data source.

   hetu-cli --catalog Data source name --schema Database name --tenant Tenant name

   For example, run the following command:

   hetu-cli --catalog iceberg --schema default --tenant default

9. Run the following command. If the database table information can be viewed or no error is reported, the connection is successful.

   show tables;

Data Type Mapping

Constraints

• Iceberg data sources do not support data source authentication. Only Ranger authentication is supported.
• The attribute partitioning is used to specify table partitions in the Iceberg data source. For example, if the table is partitioned into columns c1 and c2, the partitioning attribute is partitioning = ARRAY['c1', 'c2'].
• When using Iceberg's time travel feature to query table content by timestamp, the timestamp must be later than the table's creation time and earlier than the current system time.
• This section applies only to MRS 3.6.0 or later.