Configuring Interconnection Between MRS Hive and CSS
Scenarios
Use the Elasticsearch-Hadoop plug-in to exchange data between Hive and Elasticsearch of Cloud Search Service (CSS) so that Elasticsearch index data can be mapped to Hive tables.
The Elasticsearch-Hadoop (ES-Hadoop) connector combines Hadoop's massive data storage and in-depth processing capabilities with Elasticsearch's real-time search and analytical capabilities.
This combination allows you to gain insights from your data faster and work better within the Hadoop ecosystem.
Solution Architecture
Hive is a data warehouse built on Hadoop. It provides batch computing capabilities for big data platforms and can analyze and summarize structured and semi-structured data for large-scale calculations. Hive processes structured data using Hive Query Language (HQL), a SQL-like language. HQL automatically converts queries into MapReduce tasks to analyze massive datasets within a Hadoop cluster.
Cloud Search Service (CSS) is a fully managed, distributed search service based on open source Elasticsearch and OpenSearch. You can use it for structured and unstructured data search, and enable vector-based composite search, statistics generation, and reporting. CSS is a fully managed cloud service for the ELK Stack that supports open-source Elasticsearch, Logstash, Kibana, and Cerebro.
Notes and Constraints
- The Hive service of MRS and the Elasticsearch service of CSS have been installed, and the two clusters can communicate with each other.
- This section applies to MRS 3.x or later.
Step 1: Configure a CSS Index and Download the JAR File
- On the Clusters page of the CSS console, locate the row containing the target cluster and click Access Kibana in the Operation column. In the navigation pane of Kibana, click Dev Tools. On the console page that is displayed, run the following commands to create the index ddj_study_card_ratio_v12:
PUT ddj_study_card_ratio_v12 { "mappings" : { "properties" : { "uniq_id" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } } }If the following information is displayed, the index is created:
{ "acknowledged" : true, "shards_acknowledged" : true, "index" : "ddj_study_card_ratio_v12" } - Run the following commands to insert data into the ddj_study_card_ratio_v12 index:
POST /ddj_study_card_ratio_v12/_doc/_bulk {"index":{}} {"id":"1", "uniq_id":"23323"}If errors is false in the command output, the data is imported.
- Download the corresponding JAR file from Past Releases based on the Elasticsearch version in CSS.
For example, the JAR file corresponding to Elasticsearch 7.6.2 is elasticsearch-hadoop-7.6.2.jar.
- The JAR file and Elasticsearch of CSS must have the same version. This section uses an Elasticsearch 7.6.2 cluster with security mode enabled as an example.
- If there are any additional custom modules, pack them into a separate JAR file.
Step 2: Interconnect MRS to CSS
- For MRS 3.5.0-LTS or later, log in to each HiveServer node and run the following command to check if the elasticsearch-hadoop JAR file exists in the HiveServer installation directory:
ll ${BIGDATA_HOME}/FusionInsight_HD_*/install/FusionInsight-Hive-*/hive-*/lib/elasticsearch-hadoop-*.jar
If the elasticsearch-hadoop JAR file already exists, delete it.
- Upload the JAR file downloaded from Past Releases to the /opt/Bigdata/third_lib/Hive directory on each HiveServer node and run the following command to modify the permission:
chown omm:wheel -R /opt/Bigdata/third_lib/Hive
- Log in to FusionInsight Manager and choose Cluster > Services > Hive. On the page that is displayed, click the Instances tab. On this tab page, select all HiveServer instances, and choose More > Restart Instance.
- Download commons-httpclient-3.1.jar from the Maven Central Repository and upload this JAR file and the JAR file obtained from Past Releases to any node where the HDFS and Hive clients are installed in the cluster.
- Log in to the node where the JAR files were uploaded in the previous step as the client installation user.
- Run the following commands to authenticate the user:
cd Client installation directory
source bigdata_env
kinit Component service user (Skip this step for clusters with Kerberos authentication disabled.)
- Run the following command to create a directory for storing JAR files in HDFS:
- Run the following command to upload the JAR files downloaded from the Maven Central Repository and Past Releases to HDFS:
hdfs dfs -put JAR file storage path HDFS path for storing JAR files
- Run the following commands to enable Hive to load the specified JAR file when executing a command-line task:
add jar HDFS path for storing JAR files; (Execute this command once for each JAR file.)
- Run the following commands to create an Elasticsearch external table:
CREATE EXTERNAL TABLE `ddj_study_card_ratio_v12_test`( `uniq_id` string) ROW FORMAT SERDE 'org.elasticsearch.hadoop.hive.EsSerDe' STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' WITH SERDEPROPERTIES ( 'field.delim'='', 'serialization.format'='') TBLPROPERTIES ( 'bucketing_version'='2', 'es.index.auto.create'='false', 'es.mapping.date.rich'='false', 'es.net.http.auth.pass'='Pzh6537projectx', 'es.net.http.auth.user'='elastic', 'es.nodes'='vpcep-e0b33065-75b7-4193-8395-dbd00d10bc39.cn-east-3.huaweicloud.com', 'es.nodes.wan.only'='true', 'es.port'='9200', 'es.read.metadata'='true', 'es.resource'='ddj_study_card_ratio_v12', 'es.set.netty.runtime.available.processors'='false', 'es.write.operation'='index', 'last_modified_by'='root', 'last_modified_time'='1655264909', 'transient_lastDdlTime'='1655264909');
- The key parameters are described as follows:
- es.net.http.auth.pass and es.net.http.auth.user: indicate the password and username of the user created in Kibana who has the permissions to perform operations on index created in 1. For details, see Creating a User and Granting Permissions by Using Kibana.
- es.nodes: The IP address to connect to. You can log in to the CSS console and view the cluster IP address in the Internal Access Addresses column of the cluster list.
- es.port: The port for external access to the Elasticsearch cluster. The default value is 9200.
- es.resource: The name of the index created in 1.
For more details about parameter configurations, visit https://www.elastic.co/guide/en/elasticsearch/hadoop/6.1/hive.html.
- For MRS 3.5.0-LTS or later, if HTTPS access is enabled for CSS Elasticsearch, download the certificate from Elasticsearch, upload it to each HiveServer node, and add es.net.ssl.truststore.location to the TBLPROPERTIES parameter in the command for creating an Elasticsearch external table. The value must be the storage path of the certificate file. For example:
'es.net.ssl.truststore.location'='file:///tmp/truststore.jks'
- The key parameters are described as follows:
- Run the following command to view the created Elasticsearch external table:
select * from ddj_study_card_ratio_v12_test;
If no error information is displayed and the query is successful, Hive is interconnected with CSS. The command output is as follows.

What is your overall rating for this page?
Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot