Using Elasticsearch for Data Search

Scenario Description

A women's clothing brand runs an e-commerce website. It has been using traditional databases to power a product search function on this website. However, as the website traffic increases, these traditional databases are struggling to keep up, leading to slow responses and low search accuracy. To improve shopping experience for customers, the e-commerce website plans to use Cloud Search Service (CSS) to provide the product search function.

Procedure

The following describes how to use an Elasticsearch cluster to implement a website search function.

1. Step 1: Creating a Cluster: Create a non-security mode Elasticsearch cluster for data search.
2. Step 2: Logging In to Kibana: Log in to the cluster through Kibana.
3. Step 3: Creating an Index: Create indexes in the cluster through Kibana.
4. Step 4: Importing Data: Use an open-source Elasticsearch API to import data on Kibana.
5. Step 5: Searching for Data: Perform full-text search and result aggregation and display on data in the Elasticsearch cluster.
6. Step 6: Deleting Indexes: Delete indexes that you no longer need to reclaim resources.

Step 1: Creating a Cluster

Create a non-security mode Elasticsearch cluster for data search.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. Click Create Cluster in the upper-right corner. The Create Cluster page is displayed.
4. Configure Billing Mode and AZ for the cluster.

   Table 2 Billing mode and AZ parameters

   Parameter	Description	Example Value
   Billing Mode	Select Yearly/Monthly or Pay-per-use. Yearly/monthly: You pay for the cluster by year or month, in advance. The service duration ranges from one month to three years. If you plan to use a cluster for more than nine months, you are advised to purchase a yearly package for a better price. Pay-per-use: You will be billed hourly by actual duration of use. Any partial hour of usage will be rounded up to one hour.	Pay-per-use
   Region	Select the region where the cluster is located. ECSs in different regions cannot communicate with each other over an intranet. For lower network latency and quicker resource access, select the nearest region.	xxx
   AZ	Select AZs associated with the cluster region. A maximum of three AZs can be configured.	AZ 1

5. Configure basic cluster information.
   Figure 1 Configuring cluster information
   

   Table 3 Basic configuration parameters

   Parameter	Description	Example Value
   Cluster Type	Select Elasticsearch.	Elasticsearch
   Cluster Version	Select a cluster version from the drop-down list.	7.10.2
   Cluster Name	User-defined cluster name.	Sample-ESCluster
   Cluster Description	Add a description for the cluster for easy recognition.	/

6. Configure the cluster's node specifications.
   Figure 2 Configuring the cluster's node specifications
   

   Table 4 Specification parameters

   Parameter	Description	Example Value
   Nodes	Set the number of nodes in the cluster. Select a number from 1 to 32.	1
   CPU Architecture	The CPU architectures actually supported vary depending on the regional environment.	x86
   Node Specifications	Select the specifications of cluster nodes.	ess.spec-4u8g
   Node Storage Type	Select the storage type of cluster nodes.	High I/O
   Node Storage Capacity	Node storage capacity. Its value range varies with node specifications. The node storage capacity must be a multiple of 20.	40GB
   Master node	The Master node manages all node tasks in the cluster.	Unselect it.
   Client node	Client nodes receive and coordinate external requests, such as search and write requests.	Unselect it.
   Cold data node	Cold data nodes are used to store data that is not particularly sensitive to query latency in large quantities.	Unselect it.

7. Set the enterprise project.

   When creating a CSS cluster, you can bind an enterprise project to the cluster if you have enabled the enterprise project function. In this example, default, the default enterprise project, is selected.

8. Click Next: Network to configure the cluster network.
   Figure 3 Configuring networking
   

   Table 5 Network configuration parameters

   Parameter	Description	Example Value
   VPC	Specify a VPC to isolate the cluster's network. NOTE: The VPC must contain CIDRs. Otherwise, cluster creation will fail. By default, a VPC will contain CIDRs.	vpc-default
   Subnet	A subnet provides dedicated network resources that are isolated from other networks, improving network security.	subnet-default
   Security Group	A security group serves as a virtual firewall that provides access control policies for clusters. NOTE: To enable cluster access, ensure that port 9200 is allowed by the security group.	default
   Security Mode	After the security mode is enabled, communication will be encrypted and authentication required for the cluster.	Disable

9. Click Next: Advanced Settings. Configure automatic snapshot creation and other functions.

   This cluster is used only for getting started. Cluster snapshots and advanced functions are not required.

10. Click Next: Confirm. Check the configuration and click Next to create a cluster.
11. Click Back to Cluster List to switch to the Clusters page. The cluster you created is now in the cluster list and its status is Creating. If the cluster is successfully created, its status changes to Available.
    Figure 4 Creating a cluster
    

Step 2: Logging In to Kibana

After an Elasticsearch cluster is created, you can access the cluster through Kibana.

1. From the Elasticsearch cluster list, select the created Sample-ESCluster cluster and click Access Kibana in the Operation column to access the Kibana console.
2. In the left navigation pane on the Kibana console, click Dev Tools.
   The left part of the console is the command input box, and the triangle icon in its upper-right corner is the execution button. The right part shows the execution result.

   Figure 5 Kibana console
   

Step 3: Creating an Index

Create an index in the Elasticsearch cluster to store data.

PUT /my_store
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "properties": {
      "productName": {
        "type": "text",
        "analyzer": "ik_smart"
        },
        "size": {
          "type": "keyword"
        }
      }
    }
  }

The command output is similar to the following:

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "my_store"
}

Step 4: Importing Data

There are several ways to import data to an Elasticsearch cluster. In this example, we use an open-source Elasticsearch API to import data on Kibana.

POST /my_store/_bulk
{"index":{}}
{"productName":"Latest art shirts for women in autumn 2017","size":"L"}
{"index":{}}
{"productName":"Latest art shirts for women in autumn 2017","size":"M"}
{"index":{}}
{"productName":"Latest art shirts for women in autumn 2017","size":"S"}
{"index":{}}
{"productName":"Latest jeans for women in spring 2018","size":"M"}
{"index":{}}
{"productName":"Latest jeans for women in spring 2018","size":"S"}
{"index":{}}
{"productName":"Latest casual pants for women in spring 2017","size":"L"}
{"index":{}}
{"productName":"Latest casual pants for women in spring 2017","size":"S"}

If the value of the errors field in the command output is false, the data is imported successfully.

Step 5: Searching for Data

Perform full-text search and result aggregation and display on data in the Elasticsearch cluster.

• Full-text search

  If you access the e-commerce website and want to search for items whose names include "spring jeans", enter "spring jeans" to begin your search.

  Run the following command on Kibana:

  GET /my_store/_search
  {
    "query": {"match": {
      "productName": "spring jeans"
    }}
  }

  The command output is similar to the following:

  {
    "took" : 3,
    "timed_out" : false,
    "_shards" : {
      "total" : 1,
      "successful" : 1,
      "skipped" : 0,
      "failed" : 0
    },
    "hits" : {
      "total" : {
        "value" : 4,
        "relation" : "eq"
      },
      "max_score" : 1.7965372,
      "hits" : [
        {
          "_index" : "my_store",
          "_type" : "_doc",
          "_id" : "9xf6VHIBfClt6SDjw7H5",
          "_score" : 1.7965372,
          "_source" : {
            "productName": "Latest jeans for women in spring 2018",
            "size" : "M"
          }
        },
        {
          "_index" : "my_store",
          "_type" : "_doc",
          "_id" : "-Bf6VHIBfClt6SDjw7H5",
          "_score" : 1.7965372,
          "_source" : {
            "productName": "Latest jeans for women in spring 2018",
            "size" : "S"
          }
        },
        {
          "_index" : "my_store",
          "_type" : "_doc",
          "_id" : "-Rf6VHIBfClt6SDjw7H5",
          "_score" : 0.5945667,
          "_source" : {
            "productName": "Latest casual pants for women in spring 2017",
            "size" : "L"
          }
        },
        {
          "_index" : "my_store",
          "_type" : "_doc",
          "_id" : "-hf6VHIBfClt6SDjw7H5",
          "_score" : 0.5945667,
          "_source" : {
            "productName": "Latest casual pants for women in spring 2017",
            "size" : "S"
          }
        }
      ]
    }
  }
  

  • Elasticsearch supports IK word segmentation. The search command above segments "spring jeans" into "spring" and "jeans".
  • Elasticsearch supports full-text search. The command above searches for all items whose names include "spring" or "jeans".
  • Unlike traditional databases, Elasticsearch can return results in milliseconds by using inverted indexes.
  • Elasticsearch supports ranking by score. In the command output, the first two items contains both "spring" and "jeans", while the last two items contain only "spring". Therefore, the first two items rank higher than the last two as they are more relevant to the search word.

• Aggregated result display

  The e-commerce website displays aggregated results. For example, it classifies items corresponding to "spring" based on sizes so that you can count the number of items of different sizes.

  Run the following result aggregation command on Kibana:

  GET /my_store/_search
  {
    "query": {
      "match": {
        "productName": "Spring",
      }
    },
    "size": 0,
    "aggs": {
      "sizes": {
        "terms": {
          "field": "size"
        }
      }
    }
  }

  The command output is similar to the following:

  {
    "took" : 3,
    "timed_out" : false,
    "_shards" : {
      "total" : 1,
      "successful" : 1,
      "skipped" : 0,
      "failed" : 0
    },
    "hits" : {
      "total" : {
        "value" : 4,
        "relation" : "eq"
      },
      "max_score" : null,
      "hits" : [ ]
    },
    "aggregations" : {
      "sizes" : {
        "doc_count_error_upper_bound" : 0,
        "sum_other_doc_count" : 0,
        "buckets" : [
          {
            "key" : "S",
            "doc_count" : 2
          },
          {
            "key" : "L",
            "doc_count" : 1
          },
          {
            "key" : "M",
            "doc_count" : 1
          }
        ]
      }
    }
  }

Step 6: Deleting Indexes

If an index is no longer used, run the following command on Kibana to delete it to reclaim resources:

DELETE /my_store

The command output is similar to the following:

{
  "acknowledged" : true
}

Follow-up Operations

You can delete the cluster if you no longer need it.

After you delete a cluster, its data cannot be restored. Exercise caution when deleting a cluster.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the cluster list, locate the Sample-ESCluster cluster, and choose More > Delete in the Operation column.
4. In the confirmation dialog box, type in DELETE, and click OK.