Using Open Source Elasticsearch APIs to Import Data to Elasticsearch

Using an Open-Source Elasticsearch API to Import Data on Kibana

Using an Open-Source Elasticsearch API to Import Data on an ECS Server

On an ECS server, you can run cURL commands to use an open-source Elasticsearch API to import JSON files.

In the example below, a cluster in non-security mode is used to describe how to import data using cURL commands. For the commands for a security cluster, see Accessing an Elasticsearch Cluster Using cURL Commands.

1. Obtain the private network address of the cluster.
   1. Log in to the CSS management console.
   2. In the navigation pane on the left, choose Clusters > Elasticsearch.
   3. In the cluster list, obtain the target cluster's private IP address from the Private IP Address column. Generally, the IP address format is <host>:<port> or <host>:<port>,<host>:<port>.

      If the cluster has only one node, the IP address and port number of this one node are displayed, for example, 10.62.179.32:9200. If the cluster has multiple nodes, the IP addresses and port numbers of all nodes are displayed, for example, 10.62.179.32:9200,10.62.179.33:9200.

2. Log in to the ECS that you are going to use to import data.

   For how to buy and use an ECS, see Purchasing and Logging In to a Linux ECS.

3. Upload a JSON file to the ECS.

   For example, save the following data as a JSON file and upload the file to the ECS:

   For Elasticsearch 7.x or later:

   {"index": {"_index":"my_store"}}
   {"productName":"Autumn new woman blouses 2019","size":"M"}
   {"index": {"_index":"my_store"}}
   {"productName":"Autumn new woman blouses 2019","size":"L"}

   For Elasticsearch earlier than 7.x:

   {"index": {"_index":"my_store","_type":"products"}}
   {"productName":"Autumn new woman blouses 2019","size":"M"}
   {"index": {"_index":"my_store","_type":"products"}}
   {"productName":"Autumn new woman blouses 2019","size":"L"}

4. Run the following command in the path where the JSON file is stored in the ECS to import the JSON file to an Elasticsearch cluster.

   curl -X PUT "http://{Private network address and port number of the node}/_bulk" -H 'Content-Type: application/json' --data-binary @test.json

   Replace {Private network address and port number of the node} with the private network address and port number of a node in the cluster. The value of the -X parameter is a command and that of the -H parameter is a message header. In the preceding command, PUT is the value of the -X parameter and 'Content-Type: application/json' --data-binary @test.json is the value of the -H parameter. Do not add -k between a parameter and its value. test.json indicates the JSON file to be ingested.

   

   In the case of the failure of a cluster node, if the cluster contains multiple nodes, you can replace {Private network address and port number of the node} with the private network address and port number of any available node in the cluster; if the cluster contains only one node, restore this node and execute the command again.

   Example: In this example, assume that you need to import data in the test.json file to an Elasticsearch cluster, where communication encryption is disabled and the private network address and port number of one node are 192.168.0.90 and 9200 respectively.

   1. Run the following command to create an index named my_store:

      Sample code for Elasticsearch 7.x or later:

      curl -X PUT http://192.168.0.90:9200/my_store -H 'Content-Type: application/json' -d '
      {
          "settings": {
              "number_of_shards": 1
          },
          "mappings": {
              "properties": {
                  "productName": {
                      "type": "text"
                  },
                  "size": {
                      "type": "keyword"
                  }
              }
          }
      }'

      Sample code for Elasticsearch earlier than 7.x:

      curl -X PUT http://192.168.0.90:9200/my_store -H 'Content-Type: application/json' -d '
       { 
         "settings": { 
           "number_of_shards": 1 
         }, 
         "mappings": { 
           "products": { 
             "properties": { 
               "productName": { 
                 "type": "text" 
                 }, 
               "size": { 
                 "type": "keyword" 
               } 
             } 
           } 
         } 
       }'

   2. Run the following command to import the test.json file:

      curl -X PUT "http://192.168.0.90:9200/_bulk" -H 'Content-Type: application/json' --data-binary @test.json

      In this case, if the following information is displayed, the data is successfully imported:

      {"took":204,"errors":false,"items":[{"index":{"_index":"my_store","_type":"_doc","_id":"DJQkBIwBbJvUd2769Wi-","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1,"status":201}},{"index":{"_index":"my_store","_type":"_doc","_id":"DZQkBIwBbJvUd2769Wi_","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1,"status":201}}]}