Help Center/ MapReduce Service/ Component Operation Guide (LTS) (Ankara Region)/ Using Elasticsearch/ Running curl Commands in Linux
Updated on 2024-11-29 GMT+08:00
View PDF
Share

Running curl Commands in Linux

Overview

This section describes how to run common curl commands of Elasticsearch in Linux. In security mode, prepare the environment, log in to the curl client node in the Elasticsearch cluster, and run curl commands. In normal mode, log in to any node in the Elasticsearch cluster and run curl commands.

In openSUSE Leap 15.4, the shared link library files on the HDFS client are incompatible with curl commands. Download and install the Elasticsearch client separately and then run curl commands.

Environment Preparations

Before running curl commands in security mode, create a development user, bind the default user group and role to the user, download and install the client to the server, and configure environment variables. The procedure is as follows:

  1. On Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch > Configurations. Search for ELASTICSEARCH_SECURITY_ENABLE, check whether it is searchable, and if it is, check whether the value is true.

    • If yes, go to 2.
    • If no, you do not need to prepare the environment.

  2. Create a user.

    1. On Manager, choose System > Permission > User > Create and create a user, for example, user esuser.
    2. In user- and role-based authentication mode, add user esuser to the elasticsearch and supergroup user groups, set the primary group to supergroup, and bind the Manager_administrator role to the user to obtain related permissions. In Ranger-based authentication mode, add the Elasticsearch access permission policy for user esuser to Ranger. For details, see Adding a Ranger Access Permission Policy for Elasticsearch.
    3. Click OK.

  3. Download and install the client in a directory, for example, /opt/client.
  4. Run the following commands to go to the client installation directory and import environment variables:

    cd /opt/client

    source bigdata_env

  5. Run the following command to authenticate the user:

    kinit esuser

  6. Set both ELASTICSEARCH_SECURITY_ENABLE and DISABLE_TLS_LOW_PROTOCOL to true. Skip this step if their values are both true.

    1. On Manager, choose Cluster > Name of the desired cluster > Services > Elasticsearch > Configurations. In the search box, search for ELASTICSEARCH_SECURITY_ENABLE and set the parameter to true.
    2. Search for DISABLE_TLS_LOW_PROTOCOL and set this parameter to true.
    3. Choose Save > OK and manually restart the Elasticsearch service.
    • ELASTICSEARCH_SECURITY_ENABLE: indicates whether to enable the security mode for Elasticsearch. Switching from security mode to normal mode is a high-risk operation. Exercise caution when performing this operation.
    • DISABLE_TLS_LOW_PROTOCOL: indicates whether to disable the TLS protocol of an earlier version. If this parameter is set to true, the TLS protocol of an earlier version is disabled. Only TLSv1.2 is supported. If this parameter is set to false, TLSv1, TLSv1.1, and TLSv1.2 are all supported. Setting this value to false has security risks. Exercise caution when performing this operation.
      • In security mode, encryption is enabled for Elasticsearch. Currently, TLSv1.2 is recommended as the Secure Sockets Layer (SSL) protocol. If the protocol version is earlier than TLSv1.2, security risks may exist. To ensure data security, you need to disable protocols earlier than TLSv1.2. When running the curl command to access the Elasticsearch service using HTTPS, versions earlier than TLSv1.2 must be disabled for the curl command.
      • If error message "curl: option --tlsv1.2: is unknown" is displayed when you run the curl command, the current operating system does not support TLSv1.2. Contact the system supplier to upgrade the version to TLSv1.2. If risks brought by earlier-versions can be accepted, you can disable DISABLE_TLS_LOW_PROTOCOL to enable the TLSv protocol of an earlier version.

Command Description

  • Common parameters
    • ip: indicates the IP address of any EsNode in the Elasticsearch cluster.

      For Elasticsearch 8.0.2.1 and later versions, Elasticsearch cannot use 127.0.0.1 or localhost to access the cluster in security mode.

    • httpport: indicates the HTTP port number of the Elasticsearch instance. To obtain the port number, log in to Manager, select the Elasticsearch service of the cluster to be operated, choose Configurations > All Configurations, and search for SERVER_PORT in the upper right corner. Use the port of the EsNodeX instance for access.
    • --tlsv1.2: indicates that TLSv1.2 is used.
    • --negotiate: indicates that HTTP is used for authentication.
    • -k: indicates that SSL websites can be accessed without using a security certificate.
    • -u: indicates the username and password of the server.
    • pretty: formats the output result in JSON format.
    • -H: sends user-defined header information to the server.
    • -d: transmits data using HTTP POST. It refers to the document content here.
  • This section uses the curl command in security mode as an example. If the normal mode is used, delete the security authentication parameter --tlsv1.2 --negotiate -k -u: and change https to http. If the security mode is used but HTTP SSL encryption is disabled, retain the security authentication parameter --tlsv1.2 --negotiate -k -u: and change https to http.
  • master_timeout: Some requests in Elasticsearch need to connect to EsMaster. The master_timeout parameter indicates the timeout interval of the Master. The default value is 120s. If the Master is overloaded, timeout may occur. In this case, you need to adjust the value of this parameter.

    Example:

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_cat/master?master_timeout=300s'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_cat/master?master_timeout=300s'

_cluster Command

  • Query the cluster health status.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_cluster/health?pretty'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_cluster/health?pretty'

  • Query the global status of a cluster.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_cluster/stats?pretty'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_cluster/stats?pretty'

  • Explain cluster shards.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_cluster/allocation/explain?pretty' -H 'Content-Type:application/json' -d '{"index": "indexname","shard": number of shards,"primary": true}'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_cluster/allocation/explain?pretty' -H 'Content-Type:application/json' -d '{"index": "indexname","shard": 17,"primary": true}'

_cat Command

  • Query all command lists supported by the _cat API.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_cat/'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_cat/'

  • Query the cluster health status.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_cat/health?v'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_cat/health?v'

  • Query the index list.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_cat/indices?v'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_cat/indices?v'

  • Query the cluster instance list.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_cat/nodes?v'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_cat/nodes?v'

  • Query the help information about the nodes API.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_cat/nodes?help'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_cat/nodes?help'

  • Query the parameter values of a specified column in the nodes table.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_cat/nodes?v&h=http,dt,du,dup,hm,hc,hp,fdm,fdc,fdp,cpu'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_cat/nodes?v&h=http,dt,du,dup,hm,hc,hp,fdm,fdc,fdp,cpu'

  • Query the usage of the cluster thread pool.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_cat/thread_pool?v'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_cat/thread_pool?v'

  • Query the number of write threads in a cluster.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_cat/thread_pool/write?v'

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_cat/thread_pool/write?v&h=node_name,active,queue,rejected,c'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_cat/thread_pool/bulk?v'

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_cat/thread_pool/write?v&h=node_name,active,queue,rejected,c'

  • Query the master instance.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_cat/master?v'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_cat/master?v'

  • Query the number of shards of each instance.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_cat/allocation?v'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_cat/allocation?v'

  • Check the reason why the shard is not allocated.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_cat/shards?v&h=index,shard,prirep,state,ur'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_cat/shards?v&h=index,shard,prirep,state,ur'

_nodes Command

  • Query the jvm parameter of all EsMaster instances.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_nodes/EsMaster*/stats/jvm?pretty'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_nodes/EsMaster*/stats/jvm?pretty'

  • Query the jvm parameter of a specified EsMaster instance.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_nodes/EsMaster@ip/stats/jvm?pretty'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_nodes/EsMaster@10.131.112.121/stats/jvm?pretty'

  • Query the jvm parameter of all EsNode instances.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_nodes/EsNode*/stats/jvm?pretty'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_nodes/EsNode*/stats/jvm?pretty'

  • Query the jvm parameter of a specified EsNode1 instance.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_nodes/EsNode1@ip/stats/jvm?pretty'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_nodes/EsNode1@10.131.112.121/stats/jvm?pretty'

Index Operation

  • Create an index and set the number of replicas.

    curl -XPUT --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/indexName?pretty' -H 'Content-Type: application/json' -d' {"settings" : {"number_of_shards" : number of shards,"number_of_replicas" : number of replicas} }'

    Example: curl -XPUT --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/website?pretty' -H 'Content-Type: application/json' -d' {"settings" : {"number_of_shards" : 3,"number_of_replicas" : 1} }'

  • Query the specified index mappings.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/indexName/_mappings?pretty'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/website/_mappings?pretty'

  • Query the specified index settings.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/indexName/_settings?pretty'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/website/_settings?pretty'

  • Write data in batches.

    curl -XPOST --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/indexName/_doc/indexID?pretty' -H 'Content-Type: application/json' -d' { }'

    Example: curl -XPOST --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/website/_doc/123?pretty' -H 'Content-Type: application/json' -d' {"title": "My first blog entry", "text": "Just trying this out...","date": "2014/01/01"}'

  • Disable indexes.

    curl -XPOST --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/indexName/_close?pretty'

    Example: curl -XPOST --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/website/_close?pretty'

  • Enable indexes.

    curl -XPOST --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/indexName/_open?pretty'

    Example: curl -XPOST --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/website/_open?pretty'

  • Delete indexes.

    curl -XDELETE --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/indexName?pretty'

    Example: curl -XDELETE --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/website?pretty'

  • Modify the index update time.

    curl -XPUT --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/indexName/_settings?pretty' -H 'Content-Type: application/json' -d'{"refresh_interval" : "time (unit: seconds)"}'

    Example: curl -XPUT --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/website/_settings?pretty' -H 'Content-Type: application/json' -d'{"refresh_interval" : "60s"}'

  • Clear the query cache in Elasticsearch.

    curl -XPOST --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_cache/clear?pretty'

    Example: curl -XPOST --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_cache/clear?pretty'

  • Set the index replicas.

    curl -XPUT --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/indexName/_settings?pretty' -H 'Content-Type: application/json' -d'{"number_of_replicas" : number of replicas}'

    Example: curl -XPUT --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/website/_settings?pretty' -H 'Content-Type: application/json' -d'{"number_of_replicas" : 1}'

  • Run the refresh command to update the memory data to the disk cache.

    curl -XPOST --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_refresh?pretty'

    Example: curl -XPOST --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_refresh?pretty'

  • Run the flush command to update the disk cache to the file system.

    curl -XPOST --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/_flush?pretty'

    Example: curl -XPOST --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/_flush?pretty'

  • Run the synced flush command to update the disk cache to the file system and generate the same syncid.

    curl -XPOST --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/indexName/_flush/synced?pretty'

    Example: curl -XPOST --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/website/_flush/synced?pretty'

  • Merge segments.

    curl -XPOST --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/indexName/_forcemerge?only_expunge_deletes=false&max_num_segments=1&flush=true&pretty'

    Example: curl -XPOST --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/website/_forcemerge?only_expunge_deletes=false&max_num_segments=1&flush=true&pretty'

  • Set the number of shards of an index on each instance.

    curl -XPUT --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/indexName/_settings?pretty' -H 'Content-Type:application/json' -d '{"index.routing.allocation.total_shards_per_node":"number of shards"}'

    Example: curl -XPUT --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/website/_settings?pretty' -H 'Content-Type:application/json' -d '{"index.routing.allocation.total_shards_per_node":"2"}'

  • Check whether a file exists.

    curl -i -HEAD --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/indexName/_doc/indexID?pretty'

    Example: curl -i -HEAD --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/website/_doc/123'

  • Query all index data.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/indexName/_search?pretty'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/website/_search?pretty'

  • Use SQL statements to query indexes.

    curl -XPOST --tlsv1.2 --negotiate -k -u : "https://ip:httpport/_opendistro/_sql?pretty" -H 'Content-Type: application/json' -d'{"query": "your sql "}'

    Example: curl -XPOST --tlsv1.2 --negotiate -k -u: "https://10.131.112.121:24100/_opendistro/_sql?pretty" -H 'Content-Type: application/json' -d'{"query": "SELECT * FROM website"}'

  • Query data of a specified ID.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/indexName/_doc/indexID?pretty'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/website/_doc/123?pretty'

  • Query part of data in a document.

    curl -XGET --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/indexName/_doc/indexID?_source=documentAttributeName1,documentAttribute2&pretty'

    Example: curl -XGET --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/website/_doc/123?_source=title,text&pretty'

  • Delete a document.

    curl -XDELETE --tlsv1.2 --negotiate -k -u : 'https://ip:httpport/indexName/_doc/indexID?pretty'

    Example: curl -XDELETE --tlsv1.2 --negotiate -k -u : 'https://10.131.112.121:24100/website/_doc/123?pretty'

Parent topic: Using Elasticsearch