Connecting Elasticsearch to Kibana

Scenario

Kibana provides a graphical user interface (GUI) for you to build, query, and analyze Elasticsearch indexes.

The basic authentication is used for Elasticsearch to connect Kibana, which is insecure. You are advised to use the client tool esClient.sh to encrypt the password.

Prerequisites

• The Elasticsearch client has been installed.
• The type compatibility function has been disabled by running a curl command. For details about how to use the curl command, see Running curl Commands in Linux.

  Run the following command to check whether the value of index.include.type.name is false:

  curl -XGET -s --tlsv1.2 --negotiate -k -u : "https://ip:httpport/_cluster/settings?pretty&include_defaults&flat_settings" |grep index.include.type.name

  "index.include.type.name" : "false"

  

  • ip: indicates the IP address of any EsNode instance in the Elasticsearch cluster.
  • httpport: indicates the HTTP port of the corresponding EsNode instance. To obtain the port number, log in to FusionInsight Manager, and choose Elasticsearch > Configurations > All Configurations, and search for the SERVER_PORT parameter in the upper right corner. If multiple HTTP ports (multiple EsNodes) exist on a node, select any of them.
  • In normal mode, delete the security authentication parameter --tlsv1.2 --negotiate -k -v -u :" and change https to http.

  • In security mode, if error message "curl: option --tlsv1.2: is unknown" is displayed when you run the curl command, the current operating system does not support TLSv1.2. Contact the system supplier to upgrade the version to TLSv1.2. If risks brought by earlier-versions can be accepted, you can disable DISABLE_TLS_LOW_PROTOCOL or DISABLE_TLSV1_PROTOCOL to enable the TLSv protocol of an earlier version.
  • Restart the service after you modify the configuration to enable the TLSv protocol of an earlier version, and remove --tlsv1.2 from the command.
  • If yes, go to Interconnecting Elasticsearch with Kibana 7.10.2.
  • If no, go to Interconnecting Elasticsearch with Kibana 6.7.1.

Interconnecting Elasticsearch with Kibana 7.10.2

Perform the following steps to interconnect Elasticsearch with Kibana 7.10.2 when type compatibility has been disabled:

1. Download and install Kibana 7.10.2.
   1. Download the Kibana installation package kibana-oss-7.10.2-linux-x86_64.tar.gz and upload it to a directory on the target node, for example, /opt. Check whether an x86 server or a TaiShan server is used, and select an open source community address to download the installation package.
      • x86 server: https://artifacts.elastic.co/downloads/kibana/kibana-oss-7.10.2-linux-x86_64.tar.gz
      • TaiShan server: https://artifacts.elastic.co/downloads/kibana/kibana-oss-7.10.2-linux-aarch64.tar.gz
   2. Decompress the installation package and go to the corresponding decompression directory. The following uses the x86 server as an example to describe the installation process.

      cd /opt

      tar -zxvf kibana-oss-7.10.2-linux-x86_64.tar.gz

      cd kibana-7.10.2-linux-x86_64/

      

      • Elasticsearch users who use Dev Tools must be in the elasticsearch group and have been assigned required operation permissions. For details about how to assign operation permissions, see Elasticsearch Authentication Mode.
      • Elasticsearch users who use other functions must be in the supergroup group.

2. Modify the config/kibana.yml file. The following is an example:

   #Name of the Kibana index in Elasticsearch.
   #If multiple Kibanas need to be started, ensure that the configuration items of different users are unique.
   kibana.index: ".kibana"
   #Kibana service port.
   server.port: 5601
   #IP address of the current node. The default value is localhost.
   server.host: "10.1.1.1"
   #IP address and port number of the Elasticsearch node to be accessed. It is recommended that the IP address and port number of EsNode1 instance be configured. In normal mode, change https to http.
   elasticsearch.hosts: ["https://10.1.1.1:24100"]
   #Enter none.
   elasticsearch.ssl.verificationMode: none
   #Change xxxxxx to username:password using the base64 coding mode. This parameter is not required in normal mode.
   elasticsearch.customHeaders: {"custom-proxy-username":"xxxxxx"}
   #Directory for storing Kibana logs. Set this parameter based on the site requirements. Create this file in advance.
   logging.dest: /opt/kibana-7.10.2-linux-x86_64/log/kibana.log
   #Kibana health check interval. Set this parameter based on the site requirements. The unit is ms.
   elasticsearch.healthCheck.delay: 60000

   

   • The IP address and port number of the EsMaster node cannot be configured in elasticsearch.hosts. The EsMaster instance is an important management process. To ensure the Elasticsearch cluster stability, you are not allowed to configure the IP address and port number of the EsMaster node.
   • Some Java libraries still comply with the Base64 rules defined in RFC. Therefore, newline characters are added during encryption. Ensure that the information in the elasticsearch.customHeaders configuration item does not contain such newline characters.

     The password is encrypted using the client tool esClient.sh encrypt -n <username>. For details, see 6 in Using the Elasticsearch Client.

     For example, if the current user name is test, the password is xxx, the esClient.sh encrypt -n test command is executed, and Encrypted password is displayed after you enter and confirm the password, set the configuration item as follows:

     elasticsearch.customHeaders: {"custom-proxy-username":"Encrypted password"}

3. Run the following command to start Kibana:

   ./bin/kibana --allow-root

   

   To start Kibana in the background, run the following command:

   nohup ./bin/kibana --allow-root &

4. View Kibana logs. After the Kibana is started, log in to Kibana.

   If a log similar to the following is found in the Kibana log file, Kibana is started successfully. Enter the link printed in the log in the browser, for example, http://10.1.1.1:5601. If the IPv6 environment is used, change the address to http://[ip]:port and enter the address in the browser.

   {"type":"log","@timestamp":"2019-06-07T07:07:32Z","tags":["status","plugin:elasticsearch@7.10.2","info"],"pid":44252,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
   {"type":"log","@timestamp":"2019-06-07T07:07:32Z","tags":["listening","info"],"pid":44252,"message":"Server running at http://10.1.1.1:5601"}

   

   • For operations such as creating index pattern, Kibana creates related indexes. By default, the indexes start with .kibana.
   • Kibana reports information based on the health check status. If the information changes (for example, the login user is switched) within the health check interval, the changed information may take effect until the next health check results. In addition, the health check sends multiple HTTP requests to Elasticsearch. To prevent the Elasticsearch cluster from being overloaded, configure the health check interval in the kibana.yml based on the application scenario, that is, the elasticsearch.healthCheck.delay parameter.

Interconnecting Elasticsearch with Kibana 6.7.1

Perform the following steps to interconnect Elasticsearch with Kibana 6.7.1 when type compatibility has been enabled:

1. Download and install Kibana 6.7.1.
   1. Download the Kibana installation package kibana-oss-6.7.1-linux-x86_64.tar.gz and upload it to a directory on the target node, for example, /opt.

      Download address: https://www.elastic.co/cn/downloads/past-releases/kibana-oss-6-7-1

   2. Run the following commands to decompress the installation package and go to the decompression directory:

      cd /opt

      tar -zxvf kibana-oss-6.7.1-linux-x86_64.tar.gz

      cd kibana-6.7.1-linux-x86_64/

      

      • Users who use the Dev Tools function must be in the elasticsearch group.
      • Users who use other functions must be in the supergroup group.

2. Check whether the OS of the current server runs on the x86 or TaiShan architecture.
   • If it runs on TaiShan, go to 3.
   • If runs on x86, go to 4.

3. Download node-v10.15.2-linux-arm64.tar.gz from https://nodejs.org/dist/v10.15.2/.

   Decompress the package and replace the node in kibana-6.7.1-linux-x86_64.

   tar -zxvf node-v10.15.2-linux-arm64.tar.gz -C /opt

   rm -rf /opt/kibana-6.7.1-linux-x86_64/node/*

   mv /opt/node-v10.15.2-linux-arm64/* /opt/kibana-6.7.1-linux-x86_64/node/

4. Modify the config/kibana.yml file. The following is an example:

   #Name of the Kibana index in Elasticsearch.
   #If multiple Kibanas need to be started, ensure that the configuration items of different users are unique.
   kibana.index: ".kibana"
   #Kibana service port.
   server.port: 5601
   #IP address of the current node. The default value is localhost.
   server.host: "10.1.1.1"
   #IP address and port number of the Elasticsearch node to be accessed. It is recommended that the IP address and port number of EsNode1 instance be configured. In normal mode, change https to http.
   elasticsearch.hosts: ["https://10.1.1.1:24100"]
   #Enter none.
   elasticsearch.ssl.verificationMode: none
   #Replace xxxxxx with a Base64-encoded username and password in the format of Username:Password. This parameter is not required in normal mode.
   elasticsearch.customHeaders: {"custom-proxy-username":"xxxxxx"}
   #Directory for storing Kibana logs. Set this parameter based on the site requirements. Create this file in advance.
   logging.dest: /opt/kibana-6.7.1-linux-x86_64/log/kibana.log
   #Kibana health check interval. Set this parameter based on the site requirements. The unit is ms.
   elasticsearch.healthCheck.delay: 60000

   

   • The IP address and port number of the EsMaster node cannot be configured in elasticsearch.hosts. The EsMaster instance is an important management process. To ensure the Elasticsearch cluster stability, you are not allowed to configure the IP address and port number of the EsMaster node.
   • Some Java libraries still comply with the Base64 rules defined in RFC. Therefore, newline characters are added during encryption. Ensure that the information in the elasticsearch.customHeaders configuration item does not contain such newline characters.
   • Download the Elasticsearch client, import the RestClient secondary development example, and use the com.*.fusioninsight.elasticsearch.example.util.Base64Utils tool for Base64 encryption.
   • Ensure that you replace only xxxxxx with the Base64-encoded result.

     For example, if the username is test and password is xxx, enter dGVzdDoxMjM= (Base64-encoded result of test:xxx).

     elasticsearch.customHeaders: {"custom-proxy-username":"dGVzdDoxMjM="}

5. Start Kibana.

   ./bin/kibana

   

   To start Kibana in the background, run the following command:

   nohup ./bin/kibana &

6. View Kibana logs. After the Kibana is started, log in to Kibana.

   If information similar to the following is found in Kibana logs, Kibana has been started. Enter the link printed in the log, for example, http://10.1.1.1:5601, in the browser address box.

   {"type":"log","@timestamp":"2022-06-07T07:07:32Z","tags":["status","plugin:elasticsearch@6.7.1","info"],"pid":44252,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
   {"type":"log","@timestamp":"2022-06-07T07:07:32Z","tags":["listening","info"],"pid":44252,"message":"Server running at http://10.1.1.1:5601"}

   

   • For operations such as creating index pattern, Kibana creates related indexes. By default, the indexes start with .kibana.
   • Kibana reports information based on the health check status. If the information changes (for example, the login user is switched) within the health check interval, the changed information may take effect until the next health check results. In addition, the health check sends multiple HTTP requests to Elasticsearch. To prevent the Elasticsearch cluster from being overloaded, configure the health check interval in the kibana.yml based on the application scenario, that is, the elasticsearch.healthCheck.delay parameter.