Improving Keyword Search Accuracy with the IK Analyzer and Custom Dictionaries

Scenarios

In e-commerce, healthcare, enterprise knowledge bases, and similar scenarios, the default analyzer and the built-in IK dictionaries often cannot accurately recognize domain-specific words and phrases. For example, brand names may be split into multiple words and medical terms may be segmented incorrectly, resulting in poor search recall and precision. By combining the IK Analyzer with custom main word and stop word dictionaries, domain-specific words and phrases can be preserved as complete units while meaningless stop words are filtered out, enabling more accurate keyword search.

Typical applications

• E-commerce product search: Prevent brand names and product models from being segmented incorrectly, avoiding inaccurate search results.
• Healthcare information retrieval: Ensure that medical terminology, drug names, and disease names are recognized as complete units to improve search accuracy.
• Legal document search: Prevent legal terminology and organization names from being split into separate words, enabling precise document retrieval.
• Enterprise knowledge bases: Ensure that internal project names, technical terminology, and people's names are recognized as single terms.
• Content community operations: Quickly add newly emerging buzzwords and trending terms to the dictionary to improve content recall.

Solution Architecture

Figure 1 Solution architecture

• Indexing phase: Raw data is segmented by the IK Analyzer. The main word dictionary preserves domain-specific terms as complete words, while the stop word dictionary filters out meaningless words. The resulting terms are stored in the inverted index.
• Search phase: The query terms are analyzed by the IK Analyzer. The resulting terms are matched against the inverted index to retrieve relevant documents.
• Dictionary loading: Custom dictionaries (including the main word dictionary and stop word dictionary) are loaded dynamically from OBS and take effect without restarting the cluster. Preset dictionaries (static main word dictionary, static stop word dictionary, extra main word dictionary, and extra stop word dictionary) are maintained by CSS and provide basic coverage for commonly used terms.
• Combined analyzer strategy: Use the fine-grained ik_max_word analyzer during indexing to generate all possible terms for indexing, maximizing recall. Use the coarse-grained ik_smart analyzer during search to identify only the most appropriate terms for matching, reducing noise and improving search precision.

Advantages

• Dynamic updates with no service interruption: Custom dictionaries are loaded dynamically from OBS. Updates take effect without restarting the cluster.
• Balanced recall and precision: Using ik_max_word for indexing and ik_smart for search achieves an effective balance between search recall and search precision.
• Reduced index size: Meaningless stop words can be excluded from the inverted index, reducing index size and improving query performance.
• Easy dictionary maintenance: Dictionary files are maintained independently in OBS. New words and phrases can be added as business requirements evolve without modifying application code, allowing operations personnel to update dictionaries independently.

Constraints

• Cluster version requirements: Clusters created before March 10, 2018 do not support custom dictionaries. The sample code in this topic applies to OpenSearch clusters and Elasticsearch 7.x clusters. If you are using an Elasticsearch version earlier than 7.x, see Example code (Elasticsearch version < 7.x).
• The IK Analyzer is designed primarily for Chinese text. When used to analyze English text, only the following special characters are treated as regular characters and preserved: # & + - . @ _. Other special characters are treated as segmentation points where text can be split into different parts.

Prerequisites

• The target cluster is in the Available state and has no ongoing tasks.
• The account you are using has the following permissions (choose Permissions > Policies/Roles on the IAM console to check your permissions):
  • Permission to configure custom dictionaries:

    "css:IKThesaurus:*"

  • Permission to read OBS buckets and objects:

    obs:bucket:getBucketLocation
    obs:bucket:getBucketStoragePolicy
    obs:bucket:listAllMyBuckets
    obs:object:getObject

  • If the OBS bucket uses SSE-KMS encryption, the following KMS permissions are also required:

    "kms:cmk:create",
    "kms:dek:create",
    "kms:cmk:get",
    "kms:dek:decrypt",
    "kms:cmk:list"

Step 1: Prepare Dictionary Files

Prepare the custom main word dictionary and stop word dictionary, and upload them to an OBS bucket.

1. Prepare the following two dictionary files: The encoding format must be UTF-8 without BOM. Each line contains one entry (lowercase for English words). The file size cannot exceed 100 MB.
   • Example main word dictionary: main_custom.txt

     DR & backup
     high_availability

   • Example stop word dictionary: stop_custom.txt

     and
     can

2. Upload both dictionary files to an OBS bucket.

   The OBS bucket must be in the same region as the CSS cluster, and its storage class must be Standard.

Step 2: Configure Custom Dictionaries

Configure the custom main word dictionary and stop word dictionary so that the IK Analyzer can recognize important words and phrases while filtering out stop words.

1. Log in to the CSS management console.
2. In the navigation pane, choose Clusters > Elasticsearch or Clusters > OpenSearch.
3. In the cluster list, click the name of the target cluster. The cluster information page is displayed.
4. Choose Cluster Settings > Custom Word Dictionaries.
5. Configure custom dictionaries as required.

   Table 1 Configuring custom dictionaries

   Parameter	Description
   OBS Bucket	Select the OBS bucket that stores the dictionary files.
   Main Word Dictionary	Choose Update > Select, select main_custom.txt, and click OK.
   Stop Word Dictionary	Choose Update > Select, select stop_custom.txt, and click OK.
   Synonym Dictionary	Keep the default value No Update if no custom main word dictionary is required.

6. Click Save. In the displayed dialog box, click OK to start updating the dictionaries.

   The dictionary configuration information is displayed below. Wait for approximately 1 minute. If the dictionary status changes from Updating to Successful, the dictionary update is complete.

   Figure 2 Dictionary status
   

Step 3: Verify Word Segmentation

Step 4: Create an Index

Create an index and specify separate analyzers for indexing and search to implement the combined analyzer strategy.

Run the following request to create the book index. Use ik_max_word during indexing and ik_smart during search.

PUT /book
{
    "settings": {
        "number_of_shards": 2,
        "number_of_replicas": 1
    },
    "mappings": {
        "properties": {
            "content": {
                "type": "text",
                "analyzer": "ik_max_word",
                "search_analyzer": "ik_smart"
            }
        }
    }
}

Expected result:

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "book"
}

Step 5: Write Test Data to the Index

Write test data into the index to verify keyword search behavior.

Run the following command to write test text into the book index:

POST /book/_bulk
{"index":{"_id":"1"}}
{"content":"Sample Chinese text containing the term DR & backup"}
{"index":{"_id":"2"}}
{"content":"Sample Chinese text containing DR solution"}
{"index":{"_id":"3"}}
{"content":"Sample Chinese text containing data backup"}

Expected result:

{
  "took" : 5,
  "errors" : false,
  "items" : [
    { "index" : { "_index" : "book", "_type" : "_doc", "_id" : "1", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 0, "_primary_term" : 1, "status" : 201 } },
    { "index" : { "_index" : "book", "_type" : "_doc", "_id" : "2", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 1, "_primary_term" : 1, "status" : 201 } },
    { "index" : { "_index" : "book", "_type" : "_doc", "_id" : "3", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "_seq_no" : 2, "_primary_term" : 1, "status" : 201 } }
  ]
}

Step 6: Search by Keywords

Perform keyword searches to verify that the custom dictionaries work as expected.

1. Run the following command to search by the main word DR & backup:

   GET /book/_search
   {
     "query": {
       "match": {
         "content": "DR & backup"
       }
     }
   }

   Expected result:

   {
     "took" : 1,
     "timed_out" : false,
     "_shards" : {
       "total" : 2,
       "successful" : 2,
       "skipped" : 0,
       "failed" : 0
     },
     "hits" : {
       "total" : {
         "value" : 1,
         "relation" : "eq"
       },
       "max_score" : 0.89470756,
       "hits" : [
         {
           "_index" : "book",
           "_type" : "_doc",
           "_id" : "1",
           "_score" : 0.89470756,
           "_source" : {
             "content" : "Sample Chinese text containing the term DR & backup"
           }
         }
       ]
     }
   }

   Result interpretation: Because DR & backup is defined in the custom main word dictionary, only document 1 is returned.

   • During indexing, ik_max_word preserves DR & backup as a complete term in the inverted index. During search, ik_smart also recognizes it as a complete term, resulting in an exact match.
   • Document 2 contains DR solution, which includes DR but not the complete phrase DR & backup. Document 3 contains data backup, which includes backup but not the complete phrase DR & backup. Therefore, neither document is retrieved.
2. Run the following command to search for the stop word can:

   GET /book/_search
   {
     "query": {
       "match": {
         "content": "can"
       }
     }
   }

   Expected result:

   {
     "took" : 0,
     "timed_out" : false,
     "_shards" : {
       "total" : 2,
       "successful" : 2,
       "skipped" : 0,
       "failed" : 0
     },
     "hits" : {
       "total" : {
         "value" : 0,
         "relation" : "eq"
       },
       "max_score" : null,
       "hits" : [ ]
     }
   }

   Result interpretation: The stop word can is removed by the IK analyzer during indexing and is therefore not stored in the inverted index. As a result, the search returns no documents, confirming that the stop word dictionary is working properly.

Example code (Elasticsearch version < 7.x)

If your Elasticsearch cluster is running a version earlier than 7.x, you must define a custom document type.

1. Run the following command to create the book index and specify the IK analyzer:

   PUT /book
   {
       "settings": {
           "number_of_shards": 2,
           "number_of_replicas": 1
       },
       "mappings": {
           "type1": {
               "properties": {
                   "content": {
                       "type": "text",
                       "analyzer": "ik_max_word",
                       "search_analyzer": "ik_smart"
                   }
               }
           }
       }
   }

2. Run the following command to write test data to the book index:

   POST /_bulk
   { "index" : { "_index" : "book", "_type" : "type1", "_id" : "1" } }
   { "content" : "Sample Chinese text containing the term DR & backup" }
   { "index" : { "_index" : "book", "_type" : "type1", "_id" : "2" } }
   { "content" : "Sample Chinese text containing DR solution" }
   { "index" : { "_index" : "book", "_type" : "type1", "_id" : "3" } }
   { "content" : "Sample Chinese text containing data backup" }

3. Run the following command to verify the configuration by executing a keyword-based query:

   GET /book/type1/_search
   {
     "query": {
       "match": {
         "content": "DR & backup"
       }
     }
   }