Configuring an Elasticsearch Data Source

Scenario

This section describes how to add an Elasticsearch data source on HSConsole.

Procedure

1. Obtain the ca.crt file of the Elasticsearch data source.
   1. Log in to FusionInsight Manager of the cluster where the Elasticsearch data source is located.
   2. In the upper right corner of the homepage, click Download Client to download the complete client to the local PC as prompted.
   3. Decompress the client file to obtain the ca.crt file in the FusionInsight_Cluster_1_Services_ClientConfig directory.

2. Generate the keystore file, in JKS format, of the Elasticsearch data source after TLS is enabled.
   1. Configure the Java environment for the node where the client file is obtained.
   2. Run the following command to generate the keystore.jks file in the directory where the ca.crt file is stored:

      keytool -import -alias es -file ca.crt -keystore keystore.jks -storepass Password of the custom keystore.jks file

3. Log in to FusionInsight Manager as a HetuEngine administrator and choose Cluster > Services > HetuEngine. The HetuEngine service page is displayed.
4. In the Basic Information area on the Dashboard tab page, click the link next to HSConsole WebUI. The HSConsole page is displayed.
5. Choose Data Source and click Add Data Source. Configure parameters on the Add Data Source page.
   1. In the Basic Configuration area, configure Name and choose Elasticsearch for Data Source Type.
   2. Configure Elasticsearch Configuration parameters. For details, see Table 1.

      Table 1 Elasticsearch Configuration

      Parameter	Description	Example Value
      Driver	The default value is elasticsearch.	elasticsearch
      Host IP Address	IP address of the Elasticsearch host. Log in to FusionInsight Manager and choose Cluster > Services > Elasticsearch > Instance. In the instance list, you can view the IP address of the host where the EsNode node is located. Select an IP address randomly. Currently, only one IP address can be entered.	IPV4: 10.10.10.11 IPV6: [10:10::10:11]
      HTTP Port	HTTP port of Elasticsearch. The default value is 24100.	24100
      Schema Name	Name of the default schema generated by HetuEngine.	default
      Security Authentication Mechanism	After the security mode is enabled, the default value is PASSWORD.	PASSWORD
      Username	Configure this parameter when the security mode is enabled. Username for connecting to Elasticsearch.	Change the value based on the username being connected with Elasticsearch.
      Password	Configure this parameter when the security mode is enabled. Password for connecting to Elasticsearch.	Change the value based on the username password for connecting to Elasticsearch.
      Enabling TLS	Specifies whether TLS is enabled in Elasticsearch.	Yes
      Keystore File	Configure this parameter when the security mode is enabled. Keystore file used for enabling TLS. Select the generated keystore.jks file generated in 2 on the local PC.	keystore.jks
      Keystore Password	This parameter is mandatory when the security mode is enabled. The value is the password of the custom keystore.jks file entered in 2.b. Password of the keystore file for enabling TLS.	N/A

   3. (Optional) Customize the configuration.
      • You can click Add to add custom configuration parameters. Configure custom parameters of the Elasticsearch data source. For details, see Table 2.

        Table 2 Custom parameters of the Elasticsearch data source

        Name	Description	Example Value
        allow-aggregation-pushdown	Whether to enable the aggregation pushdown function. The function is enabled by default. Value options are as follows: true: Enable the pushdown function. false: Disable the pushdown function.	true
        elasticsearch.query-data-immediate.enabled	Whether to enable the operation to take effect immediately after a table operation is performed on the Elasticsearch data source. The default value is false. Value options are as follows: true: Enable the function. false: Disable the function.	false

      • You can click Delete to delete custom configuration parameters.
   4. Click OK.

6. Log in to the node where the cluster client is located and run the following commands to switch to the client installation directory and authenticate the user:

   cd /opt/client

   source bigdata_env

   kinit User performing HetuEngine operations (If the cluster is in normal mode, skip this step.)

7. Run the following command to log in to the catalog of the data source:

   hetu-cli --catalog Data source name --schema Database name

   For example, run the following command:

   hetu-cli --catalog elasticsearch_1 --schema default

8. Run the following command. If the database table information can be viewed or no error is reported, the connection is successful.

   show tables;

Data Type Mapping

Performance Optimization

The query pushdown function is supported to improve query speed.

Constraints

• HetuEngine supports the following SQL syntax for interconnecting with Elasticsearch: SHOW CATALOGS/SCHEMAS/TABLES, SELECT, DROP TABLE, DELETE, UPDATE, and DESCRIBE.

• The following syntaxes are not supported: CREATE SCHEMA, CREATE TABLE, CREATE VIEW, ALTER TABLE, and ALTER VIEW.
• Tables with duplicate column names in Elasticsearch cannot be queried, for example, column names are name or NAME.
• You are not advised to query Elasticsearch tables whose names contain special characters, such as hyphens (-) and periods (.). To add special characters to the name of a table, use double quotation marks ("") to enclose the table name when you query the table.