Using the Elasticsearch Client

Scenario

This section describes how to use the Elasticsearch client in O&M scenarios. In service scenarios, use the APIs provided by Elasticsearch.

Elasticsearch is incompatible with clients of other versions. If you reinstall the cluster or Elasticsearch service, or switch to the security mode, download and install the client again.

Prerequisites

• The client has been installed. The installation directory is /opt/client. The client directory in the following operations is only an example. Change it based on the actual installation directory onsite.

• Service component users have been created by the MRS cluster administrator.

  In security mode, machine-machine users need to download the keytab file. A human-machine user needs to change the password upon the first login. (This operation is not required in normal mode.)

Procedure

1. Log in to the node where the client is installed as the client installation user.
2. Run the following command to go to the client installation directory:

   cd /opt/client

3. Run the following command to configure environment variables:

   source bigdata_env

4. Check whether multiple Elasticsearch services are installed.
   • If yes, when you use the client to connect to a specific Elasticsearch service, run the following command to load environment variables of the service: For example, run the following command to load Elasticsearch-1 service variables: source Elasticsearch-1/component_env.
   • If no, skip this step.

5. If the cluster is in security mode, run the following command to authenticate the user. In normal mode, user authentication is not required.

   kinit Component service user

6. Run the Elasticsearch shell command:

   cd /opt/client/Elasticsearch/sbin

   The command format on the Elasticsearch client is as follows:

   esClient.sh <option> <command> [command-arg] [-p|u|s [args]]

   For example, run the following command to view the command help information:

   esClient.sh help

   Run the following command to view an example:

   esClient.sh example

   Run the following command to view the cluster health information:

   esClient.sh cluster health

   Run the following command to enter a text and the result is returned after being encrypted.

   esClient.sh encrypt

   Run the following command to view the reason why the primary shard of the myindex 1 index is not allocated:

   esClient.sh cluster allocation_explain -s index=myindex#shard=1#primary=true

   

   • If commands-arg or args contains special characters such as asterisks (*), parentheses (), hyphens (-), or spaces, enclose command-arg and args in quotation marks. For example: esClient.sh cluster health "in*" -p "level=shards#wait_for_status=yellow#timeout=50s#wait_for_nodes=ge(3)#pretty"
   • If an error occurs during the execution of the Elasticsearch shell command, view the execution log {client_home}/Elasticsearch service/logs/esClient.log. {client_home} indicates the installation directory of the Elasticsearch client. For example, if the Elasticsearch service is used, the log file path is /opt/client/Elasticsearch/logs/esClient.log.
   • esClient.sh encrypt receives parameters in interactive mode. After you press Enter, the system prompts you to enter a string. After you enter the string, confirm the string and press Enter again, the execution result {"cipherText":"xxxx"} is returned. xxxx is the encrypted string. For the -n in the esClient.sh encrypt [-n [args]]command parameter, add the username to the end of the parameter, press Enter, enter the password, and press Enter again. The encrypted user information encoded using Base64 is displayed. This command is used in scenarios where ciphertext is required, such as password encryption.
   • master_timeout: Some requests in Elasticsearch need to connect to EsMaster. The master_timeout parameter indicates the timeout interval of the Master. The default value is 120s. If the Master is overloaded, timeout may occur. In this case, you need to adjust the value of this parameter in the following format:

     Example: ./esClient.sh cat master -p master_timeout=300s

Running the Elasticsearch Rest Client Sample Command

1. Log in to the node where the client is installed as the client installation user.
2. Run the following command to go to the client installation directory:

   cd /opt/client

3. Run the following command to configure environment variables:

   source bigdata_env

4. Check whether multiple Elasticsearch services are installed.
   • If yes, when using the Elasticsearch Rest client sample command to connect to a specific Elasticsearch service, run the following command to load environment variables of the service. For example, run the following command to load Elasticsearch-1 service variables: source Elasticsearch-1/component_env.
   • If no, skip this step.

5. If the cluster is in security mode, upload the Kerberos authentication credentials krb5.conf and user.keytab of the component service user to the Elasticsearch client installation path/Elasticsearch service/elasticsearch-example/conf/ directory where the client is installed. For example, if the Elasticsearch service is used, the upload path is /opt/client/Elasticsearch/elasticsearch-example/conf/. The krb5.conf and user.keytab files need to be obtained by downloading the authentication credential files of the corresponding user on Manager.
6. Modify the es-rest-client-example.properties file. For example, if the Elasticsearch service is used, run the following command to modify es-rest-client-example.properties:

   cd /opt/client/Elasticsearch/elasticsearch-example/conf/

   vi es-rest-client-example.properties

   For details about parameter configuration, see Table 1. The following is a configuration example:

   esServerHost=10.1.1.1:24100,10.2.2.2:24100
   principal=esuser
   isSecureMode=true
   sslEnabled=true
   connectTimeout=5000
   socketTimeout=60000
   connectionRequestTimeout=100000
   maxConnPerRoute=100
   maxConnTotal=1000
   customJaasPath=

   Table 1 es-rest-client-example.properties parameter configuration

   Parameter	Default Value	Description
   esServerHost	IP address and port number of the Elasticsearch cluster instance downloaded from the Elasticsearch client	Specifies the instance configuration of the target Elasticsearch cluster. To facilitate load balancing, multiple groups of IP addresses and port numbers are configured. The parameter value is in the IP address:Port number format. Multiple groups of IP address and port number are separated by comma (,). The IP address is the IP address of the service plane of any DataNode in the Elasticsearch cluster, and the port number is the external HTTP(S) port number of the node. To view the instance configuration, log in to Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch > Configurations, and search for the INSTANCE_SERVER_PORT_LIST parameter. NOTE: Do not obtain this parameter value from the EsMaster instances.
   principal	esuser	Specifies the authentication user name of the target Elasticsearch cluster. Set this parameter if isSecureMode is set to true. The recommended format is user name. You can also use the format of user name@target cluster name.
   isSecureMode	true	Specifies the security mode of the target Elasticsearch cluster. true indicates that the cluster is in security mode, and false indicates that the cluster is in normal mode.
   sslEnabled	true	Indicates whether to enable SSL encryption on the client. true indicates that encryption is enabled, and false indicates that encryption is disabled. The value must be the same as that configured in the cluster.
   connectTimeout	5000	Specifies the connection timeout of the target cluster. The unit is millisecond.
   socketTimeout	60000	Specifies the socket timeout of the target cluster. The unit is millisecond.
   connectionRequestTimeout	100,000	Specifies the maximum timeout interval for obtaining available connections from the connection pool. The unit is millisecond.
   maxConnPerRoute	100	Specifies the maximum number of concurrent connections to the same route.
   maxConnTotal	1000	Specifies the maximum number of connections in a connection pool.
   customJaasPath	N/A	Specifies the user-defined path of the jaas.conf file, which must contain specific file name. If this parameter is not set, the system automatically generates a path.

7. Run the Elasticsearch Rest client sample command.

   cd /opt/client/Elasticsearch/sbin

   The command format is as follows:

   esRestClient.sh [-option [args]]

   For example, run the following command to view the command help information:

   esRestClient.sh -h

   Run the sample command:

   esRestClient.sh

   Table 2 lists parameters supported by the Elasticsearch Rest client sample command.

   Table 2 Parameters supported by the Elasticsearch Rest client sample command

   Parameter	Description
   -t	Indicates the type of the Rest client type. The value can be low, high, left empty, or unspecified. If this parameter is left empty or not specified, the default value high is used. low: run the low level rest client sample commands. high: run the high level rest client sample commands.
   -p	Indicates the path of the configuration file. This parameter can be left empty or not specified. If it is left empty or not specified, the default path is used. The default path is Elasticsearch client installation path/Elasticsearch service/elasticsearch-example/conf/es-rest-client-example.properties. If a configuration file path is specified, the user who executes the command must have the read and write permissions on the configuration file and the directory where the file is located. If Elasticsearch is in security mode, the Kerberos authentication credential must be placed in the directory where the configuration file is located and principal must be configured.
   -h	Prints the help information about the Elasticsearch Rest client sample command.

   

   The log file path of the Elasticsearch Rest client example command is {client_home}/Elasticsearch service/elasticsearch-example/logs/esRestExample.log. {client_home} indicates the installation directory of the Elasticsearch client. For example, if the Elasticsearch service is used, the log file path is /opt/client/Elasticsearch/elasticsearch-example/logs/esRestExample.log.

Running the Elasticsearch Transport Client Sample Command

1. Log in to the node where the client is installed as the client installation user.
2. Run the following command to go to the client installation directory:

   cd /opt/client

3. Run the following command to configure environment variables:

   source bigdata_env

4. Check whether multiple Elasticsearch services are installed.
   • If yes, when using the Elasticsearch Transport client sample command to connect to a specific Elasticsearch service, run the following command to load environment variables of the service. For example, run the following command to load Elasticsearch-1 service variables: source Elasticsearch-1/component_env.
   • If no, skip this step.

5. If the cluster is in security mode, upload the Kerberos authentication credentials krb5.conf and user.keytab of the component service user to the Elasticsearch client installation path/Elasticsearch instance/elasticsearch-example/conf/ directory. For example, if the Elasticsearch service is used, the upload path is /opt/client/Elasticsearch/elasticsearch-example/conf/. The krb5.conf and user.keytab files need to be obtained by downloading the authentication credential files of the corresponding user on Manager.
6. Modify es-transport-client-example.properties. For example, if the Elasticsearch service is used, run the following command to modify es-transport-client-example.properties:

   cd /opt/client/Elasticsearch/elasticsearch-example/conf/

   vi es-transport-client-example.properties

   For details about parameter configuration, see Table 3. The following is a configuration example:

   esServerHosts=10.1.1.1:24101,10.2.2.2:24101
   clusterName=elasticsearch_cluster
   principal=esuser
   isSecureMode=true
   sslEnabled=true
   isSniff=true
   krb5Path={client_home}/Elasticsearch/elasticsearch-example/conf/
   keytabPath={client_home}/Elasticsearch/elasticsearch-example/conf/
   customJaasPath=

   Table 3 es-transport-client-example.properties parameter configuration

   Parameter	Default Value	Description
   esServerHost	IP address and port number of the Elasticsearch cluster instance downloaded from the FusionInsight Elasticsearch client	Specifies the instance configuration of the target Elasticsearch cluster. To facilitate load balancing, multiple groups of IP addresses and port numbers are configured. The parameter value is in the IP address:Port number format. Multiple groups of IP address and port number are separated by comma (,). The IP address is the IP address of the service plane of any DataNode in the Elasticsearch cluster, and the port number is the external HTTP(S) port number of the node. To view the instance configuration, log in to Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch > Configurations, and search for the INSTANCE_TRANSPORT_TCP_PORT_LIST parameter. NOTE: Do not obtain this parameter value from the EsMaster instances.
   clusterName	Name of the FusionInsight Elasticsearch cluster downloaded from the Elasticsearch client	Specifies the name of the target Elasticsearch cluster.
   principal	esuser	Specifies the authentication user name of the target Elasticsearch cluster. Set this parameter if isSecureMode is set to true. The recommended format is user name. You can also use the format of user name@target cluster name.
   isSecureMode	true	Specifies the security mode of the target Elasticsearch cluster. true indicates that the cluster is in security mode, and false indicates that the cluster is in normal mode.
   sslEnabled	true	Indicates whether to enable SSL encryption. true indicates that encryption is enabled, and false indicates that encryption is disabled. The value must be the same as that configured in the cluster.
   isSniff	true	Indicates whether to start sniffing. true indicates that the function is enabled, and false indicates that the function is disabled. When sniffing is enabled, the Transport client automatically modifies the instance when sending a request when a new instance is added to or removed from the target Elasticsearch cluster.
   krb5Path	[client_home/Elasticsearch]/elasticsearch-example/conf/	Specifies the path for storing the krb5.conf file of the user principal. Set this parameter if isSecureMode is set to true. client_home indicates the client installation path, and Elasticsearch indicates the Elasticsearch service. For example, if the client installation path is /opt/client and the Elasticsearch-1 service is used, [client_home/Elasticsearch] is automatically replaced with /opt/client/Elasticsearch-1 after the Elasticsearch client is installed.
   keytabPath	[client_home/Elasticsearch]/elasticsearch-example/conf/	Specifies the path for storing the user.keytab file of the user principal. Set this parameter if isSecureMode is set to true. client_home indicates the client installation path, and Elasticsearch indicates the Elasticsearch service. For example, if the client installation path is /opt/client and the Elasticsearch-1 service is used, [client_home/Elasticsearch] is automatically replaced with /opt/client/Elasticsearch-1 after the Elasticsearch client is installed.
   customJaasPath	N/A	Specifies the user-defined path of the jaas.conf file, which must contain specific file name. If this parameter is not set, the system automatically generates a path.

7. Run the Elasticsearch Transport client sample command.

   cd /opt/client/Elasticsearch/sbin

   The command format is as follows:

   esTransportClient.sh [-option [args]]

   For example, run the following command to view the command help information:

   esTransportClient.sh -h

   Run the sample command:

   esTransportClient.sh

   Table 4 lists the parameters supported by the Elasticsearch Transport client sample command.

   Table 4 Parameters supported by the Elasticsearch Transport client sample command

   Parameter	Description
   -p	Configuration file path, which can be left empty or not specified. If this parameter is left empty or not specified, the default path is used. The default path is Elasticsearch client installation path/Elasticsearch service/elasticsearch-example/conf/es-transport-client-example.properties. If a configuration file path is specified, the user who executes the command must have the read and write permissions on the configuration file and the directory where the file is located. If Elasticsearch is in security mode, principal, krb5Path, and keytabPath must be configured.
   -h	Prints the help information about the Elasticsearch Transport client sample command.

   

   The log file path of the Elasticsearch Transport client example command is {client_home}/Elasticsearch service/elasticsearch-example/logs/esTransportExample.log. {client_home} indicates the installation directory of the Elasticsearch client. For example, if the Elasticsearch service is used, the log file path is /opt/client/Elasticsearch/elasticsearch-example/logs/esTransportExample.log.