Configuring an Elasticsearch Data Source

Updated on 2024-11-29 GMT+08:00

View PDF

Scenario

This section describes how to add an Elasticsearch data source on HSConsole.

NOTE:

If Ranger authentication is enabled for the connected Elasticsearch data source, you need to grant permissions to the user who accesses the Elasticsearch data source from HetuEngine on Ranger of the data source cluster.
HetuEngine is case insensitive to the metadata information of the Elasticsearch data source and can process only the metadata information in lowercase.

Procedure

Obtain the ca.crt file of the Elasticsearch data source.
1. Log in to FusionInsight Manager of the cluster where the Elasticsearch data source is located.
2. In the upper right corner of the homepage, click Download Client to download the complete client to the local PC as prompted.
3. Decompress the client file to obtain the ca.crt file in the FusionInsight_Cluster_1_Services_ClientConfig directory.
Generate the keystore file, in JKS format, of the Elasticsearch data source after TLS is enabled.
1. Configure the Java environment for the node where the client file is obtained.
2. Run the following command to generate the keystore.jks file in the directory where the ca.crt file is stored:
  keytool -import -alias es -file ca.crt -keystore keystore.jks -storepass Password of the custom keystore.jks file
Log in to FusionInsight Manager as a HetuEngine administrator and choose Cluster > Services > HetuEngine. The HetuEngine service page is displayed.
In the Basic Information area on the Dashboard tab page, click the link next to HSConsole WebUI. The HSConsole page is displayed.

Choose Data Source and click Add Data Source. Configure parameters on the Add Data Source page.

In the Basic Configuration area, configure Name and choose Elasticsearch for Data Source Type.

Configure Elasticsearch Configuration parameters. For details, see Table 1.

**Table 1** Elasticsearch Configuration
Parameter	Description	Example Value
Driver	The default value is elasticsearch.	elasticsearch
Host IP Address	IP address of the Elasticsearch host. Log in to FusionInsight Manager and choose Cluster > Services > Elasticsearch > Instance. In the instance list, you can view the IP address of the host where the EsNode node is located. Select an IP address randomly. Currently, only one IP address can be entered.	IPV4: 10.10.10.11 IPV6: [10:10::10:11]
HTTP Port	HTTP port of Elasticsearch. The default value is 24100.	24100
Schema Name	Name of the default schema generated by HetuEngine.	default
Security Authentication Mechanism	After the security mode is enabled, the default value is PASSWORD.	PASSWORD
Username	Configure this parameter when the security mode is enabled. Username for connecting to Elasticsearch.	Change the value based on the username being connected with Elasticsearch.
Password	Configure this parameter when the security mode is enabled. Password for connecting to Elasticsearch.	Change the value based on the username password for connecting to Elasticsearch.
Enabling TLS	Specifies whether TLS is enabled in Elasticsearch.	Yes
Keystore File	Configure this parameter when the security mode is enabled. Keystore file used for enabling TLS. Select the generated keystore.jks file generated in 2 on the local PC.	keystore.jks
Keystore Password	This parameter is mandatory when the security mode is enabled. The value is the password of the custom keystore.jks file entered in 2.b. Password of the keystore file for enabling TLS.	N/A

(Optional) Customize the configuration.

You can click Add to add custom configuration parameters. Configure custom parameters of the Elasticsearch data source. For details, see Table 2.

**Table 2** Custom parameters of the Elasticsearch data source
Name	Description	Example Value
allow-aggregation-pushdown	Whether to enable the aggregation pushdown function. The function is enabled by default. Value options are as follows: true: Enable the pushdown function. false: Disable the pushdown function.	true
elasticsearch.query-data-immediate.enabled	Whether to enable the operation to take effect immediately after a table operation is performed on the Elasticsearch data source. The default value is false. Value options are as follows: true: Enable the function. false: Disable the function.	false

You can click Delete to delete custom configuration parameters.

Click OK.

Log in to the node where the cluster client is located and run the following commands to switch to the client installation directory and authenticate the user:

cd /opt/client

source bigdata_env

kinit User performing HetuEngine operations (If the cluster is in normal mode, skip this step.)
Run the following command to log in to the catalog of the data source:

hetu-cli --catalog Data source name --schema Database name

For example, run the following command:

hetu-cli --catalog elasticsearch_1 --schema default
Run the following command. If the database table information can be viewed or no error is reported, the connection is successful.

show tables;

Data Type Mapping

Elasticsearch Data Type	HetuEngine Data Type
boolean	BOOLEAN
binary	VARBINARY
byte	TINYINT
short	SMALLINT
integer	INTEGER
long	BIGINT
float	REAL
double	DOUBLE
keyword	VARCHAR
text	VARCHAR
ip	IPADDRESS

Performance Optimization

The query pushdown function is supported to improve query speed.

Constraints

HetuEngine supports the following SQL syntax for interconnecting with Elasticsearch: SHOW CATALOGS/SCHEMAS/TABLES, SELECT, DROP TABLE, DELETE, UPDATE, and DESCRIBE.

The following syntaxes are not supported: CREATE SCHEMA, CREATE TABLE, CREATE VIEW, ALTER TABLE, and ALTER VIEW.
Tables with duplicate column names in Elasticsearch cannot be queried, for example, column names are name or NAME.
You are not advised to query Elasticsearch tables whose names contain special characters, such as hyphens (-) and periods (.). To add special characters to the name of a table, use double quotation marks ("") to enclose the table name when you query the table.

Parent topic: Configuring Data Sources

Previous topic: Configuring a ClickHouse Data Source

Next topic: Configuring a GaussDB Data Source