Using Elasticsearch for Data Search

Scenario Description

A women's clothing brand runs an e-commerce website. It has been using traditional databases to power a product search function for customers. However, as the website traffic increases, these traditional databases are struggling to keep up, leading to slow responses and low search accuracy. To improve shopping experience for customers, the e-commerce website plans to use Cloud Search Service (CSS) to provide the product search function.

Assume that the e-commerce website has the following data:

{
"products":[
{"productName":"Latest elegant shirts in autumn 2017","size":"L"}
{"productName":"Latest elegant shirts in autumn 2017","size":"M"}
{"productName":"Latest elegant shirts in autumn 2017","size":"S"}
{"productName":"Latest jeans in spring 2018","size":"M"}
{"productName":"Latest jeans in spring 2018","size":"S"}
{"productName":"Latest casual pants in spring 2017","size":"L"}
{"productName":"Latest casual pants in spring 2017","size":"S"}
]
}

Procedure

The following describes how to use an Elasticsearch cluster to implement a website search function.

Before starting to migrate data, make the necessary preparations. For details, see Preparations.

1. Step 1: Creating a Cluster: Create a non-security mode Elasticsearch cluster for data search.
2. Step 2: Importing Data: Use an open-source Elasticsearch API to import data on Kibana.
3. Step 3: Searching for Data: Perform full-text search and result aggregation and display on data in the Elasticsearch cluster.
4. Step 4: Deleting Indexes: Delete indexes that you no longer need to reclaim resources.

Preparations

You have registered with Huawei Cloud and performed real-name authentication. Make sure your account is not frozen or in arrears.

Step 1: Creating a Cluster

Create a non-security mode Elasticsearch cluster for data search.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. Click Create Cluster in the upper right corner. The Create Cluster page is displayed.
4. Configure Billing Mode and AZ for the cluster.

   Table 1 Billing mode and AZ parameters

   Parameter	Description	Example Value
   Billing Mode	Select Yearly/Monthly or Pay-per-use. Yearly/monthly: You pay for the cluster by year or month, in advance. The service duration ranges from one month to three years. If you plan to use a cluster for more than nine months, you are advised to purchase a yearly package for a better price. Pay-per-use: You are billed by actual duration of use, with a billing cycle of one hour. For example, 58 minutes of usage will be rounded up to an hour and billed.	Pay-per-use
   Region	Select the region where the cluster is located. ECSs in different regions cannot communicate with each other over an intranet. For lower network latency and quicker resource access, select the nearest region.	Hong Kong, China
   AZ	Select AZs associated with the cluster region. A maximum of three AZs can be configured.	AZ 1

5. Configure basic cluster information.
   Figure 1 Configuring cluster information
   

   Table 2 Basic configuration parameters

   Parameter	Description	Example Value
   Cluster Type	Select Elasticsearch.	Elasticsearch
   Version	Select a cluster version from the drop-down list box.	7.10.2
   Name	Cluster name, which contains 4 to 32 characters. Only letters, numbers, hyphens (-), and underscores (_) are allowed and the value must start with a letter.	Sample-ESCluster

6. Configure the cluster's node specifications
   Figure 2 Configuring the cluster's node specifications
   

   Table 3 Specification parameters

   Parameter	Description	Example Value
   Nodes	Set the number of nodes in the cluster. Select a number from 1 to 32.	1
   CPU Architecture	The CPU architectures actually supported vary depending on the regional environment.	x86
   Node Specifications	Select the specifications of cluster nodes.	ess.spec-4u8g
   Node Storage Type	Select the storage type of cluster nodes.	High I/O
   Node Storage Capacity	Node storage capacity. Its value range varies with node specifications. The node storage capacity must be a multiple of 20.	40GB
   Master node	The Master node manages all node tasks in the cluster.	Unselect it.
   Client node	Client nodes receive and coordinate external requests, such as search and write requests.	Unselect it.
   Cold data node	Cold data nodes are used to store data that is not particularly sensitive to query latency in large quantities.	Unselect it.

7. Set the enterprise project.

   When creating a CSS cluster, you can bind an enterprise project to the cluster if you have enabled the enterprise project function. In this example, default, the default enterprise project, is selected.

8. Click Next: Network. Configure the cluster network.
   Figure 3 Configuring networking
   

   Table 4 Network configuration parameters

   Parameter	Description	Example Value
   VPC	Specify a VPC to isolate the cluster's network. NOTE: The VPC must contain CIDRs. Otherwise, cluster creation will fail. By default, a VPC will contain CIDRs.	vpc-default
   Subnet	A subnet provides dedicated network resources that are isolated from other networks, improving network security.	subnet-default
   Security Group	A security group serves as a virtual firewall that provides access control policies for clusters. NOTE: For enable cluster access, ensure that port 9200 is allowed by the security group.	default
   Security Mode	After the security mode is enabled, communication will be encrypted and authentication required for the cluster.	Disable

9. Click Next: Advanced Settings. Configure automatic snapshot creation and other functions.

   This cluster is used only for getting started. Cluster snapshots and advanced functions are not required.

10. Click Next: Confirm. Check the configuration and click Next to create a cluster.
11. Click Back to Cluster List to switch to the Clusters page. The cluster you created is listed on the displayed page and its status is Creating. If the cluster is successfully created, its status will change to Available.
    Figure 4 Creating a cluster
    

Step 2: Importing Data

There are several ways to import data to an Elasticsearch cluster. In this example, we use an open-source Elasticsearch API to import data on Kibana.

1. On the Elasticsearch cluster management page, select the created Sample-ESCluster cluster and click Access Kibana in the Operation column to access the Kibana console.
2. In the Kibana navigation pane on the left, choose Dev Tools.
   The text box on the left is the input box. The triangle icon in the upper right corner of the input box is the command execution button. The text box on the right area is the result output box.

   Figure 5 Console page
   
3. On the Console page, run the following command to create an index named my_store:

   PUT /my_store
   {
     "settings": {
       "number_of_shards": 1
     },
     "mappings": {
       "properties": {
         "productName": {
           "type": "text",
           "analyzer": "ik_smart"
           },
           "size": {
             "type": "keyword"
           }
         }
       }
     }

   The command output is similar to the following:

   {
     "acknowledged" : true,
     "shards_acknowledged" : true,
     "index" : "my_store"
   }

4. On the Console page, run the following command to import data to the index named my_store:

   POST /my_store/_doc/_bulk
   {"index":{}}
   {"productName":"Latest elegant shirts in autumn 2017","size":"L"}
   {"index":{}}
   {"productName":"Latest elegant shirts in autumn 2017","size":"M"}
   {"index":{}}
   {"productName":"Latest elegant shirts in autumn 2017","size":"S"}
   {"index":{}}
   {"productName":"Latest jeans in spring 2018","size":"M"}
   {"index":{}}
   {"productName":"Latest jeans in spring 2018","size":"S"}
   {"index":{}}
   {"productName":"Latest casual pants in spring 2017","size":"L"}
   {"index":{}}
   {"productName":"Latest casual pants in spring 2017","size":"S"}

   If the value of the errors field in the command output is false, the data is imported successfully.

Step 3: Searching for Data

Perform full-text search and result aggregation and display on data in the Elasticsearch cluster.

• Full-text search

  If you access the e-commerce website and want to search for items whose names include "spring jeans", enter "spring jeans" to begin your search.

  Run the following command on Kibana:

  GET /my_store/_search
  {
    "query": {"match": {
      "productName": "spring jeans"
    }}
  }

  The command output is similar to the following:

  {
    "took" : 3,
    "timed_out" : false,
    "_shards" : {
      "total" : 1,
      "successful" : 1,
      "skipped" : 0,
      "failed" : 0
    },
    "hits" : {
      "total" : {
        "value" : 4,
        "relation" : "eq"
      },
      "max_score" : 1.7965372,
      "hits" : [
        {
          "_index" : "my_store",
          "_type" : "_doc",
          "_id" : "9xf6VHIBfClt6SDjw7H5",
          "_score" : 1.7965372,
          "_source" : {
            "productName": "Latest jeans in spring 2018",
            "size" : "M"
          }
        },
        {
          "_index" : "my_store",
          "_type" : "_doc",
          "_id" : "-Bf6VHIBfClt6SDjw7H5",
          "_score" : 1.7965372,
          "_source" : {
            "productName": "Latest jeans in spring 2018",
            "size" : "S"
          }
        },
        {
          "_index" : "my_store",
          "_type" : "_doc",
          "_id" : "-Rf6VHIBfClt6SDjw7H5",
          "_score" : 0.5945667,
          "_source" : {
            "productName": "Latest casual pants in spring 2017",
            "size" : "L"
          }
        },
        {
          "_index" : "my_store",
          "_type" : "_doc",
          "_id" : "-hf6VHIBfClt6SDjw7H5",
          "_score" : 0.5945667,
          "_source" : {
            "productName": "Latest casual pants in spring 2017",
            "size" : "S"
          }
        }
      ]
    }
  }
  

  • Elasticsearch supports IK word segmentation. The preceding command segments "spring jeans" into "spring" and "jeans".
  • Elasticsearch supports full-text search. The preceding command searches for the information about all items whose names include "spring" or "jeans".
  • Unlike traditional databases, Elasticsearch can return results in milliseconds by using inverted indexes.
  • Elasticsearch supports sorting by score. In the command output, information about the first two items contains both "spring" and "jeans", while that about the last two items contain only "spring". Therefore, the first two items rank higher than the last two due to high keyword match.

• Aggregated result display

  The e-commerce website displays aggregated results. For example, it classifies items corresponding to "spring" based on sizes so that you can count the number of items of different sizes.

  Run the following result aggregation command on Kibana:

  GET /my_store/_search
  {
  "query": {
  "match": { "productName": "spring" }
  },
  "size": 0,
  "aggs": {
  "sizes": {
  "terms": { "field": "size" }
  }
  }
  }

  The command output is similar to the following:

  {
    "took" : 3,
    "timed_out" : false,
    "_shards" : {
      "total" : 1,
      "successful" : 1,
      "skipped" : 0,
      "failed" : 0
    },
    "hits" : {
      "total" : {
        "value" : 4,
        "relation" : "eq"
      },
      "max_score" : null,
      "hits" : [ ]
    },
    "aggregations" : {
      "sizes" : {
        "doc_count_error_upper_bound" : 0,
        "sum_other_doc_count" : 0,
        "buckets" : [
          {
            "key" : "S",
            "doc_count" : 2
          },
          {
            "key" : "L",
            "doc_count" : 1
          },
          {
            "key" : "M",
            "doc_count" : 1
          }
        ]
      }
    }
  }

Step 4: Deleting Indexes

If an index is no longer used, run the following command on Kibana to delete the index to reclaim resources:

DELETE /my_store

The command output is similar to the following:

{
  "acknowledged" : true
}

Follow-up Operations

You can delete the cluster if you no longer need it.

After you delete a cluster, its data cannot be restored. Exercise caution when deleting a cluster.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the cluster list, locate the Sample-ESCluster cluster, and choose More > Delete in the Operation column.
4. In the confirmation dialog box, type in DELETE, and click OK.