Improving Search Recall with the Synonym Filter and a Custom Synonym Dictionary

Scenarios

In scenarios such as content search and customer service knowledge bases, users often use different words to express the same concept. For example, a document may contain the term notebook computer, while a user searches for laptop. Without synonym expansion, the document may not be returned. By configuring a custom synonym dictionary and using the dynamic_synonym filter, synonyms can be expanded automatically during both indexing and search, enabling synonym-based search across equivalent terms and significantly improving search recall.

Typical applications

• Content platform search: Match formal terminology when users search with everyday language. For example, searching for laptop can return documents containing notebook computer.
• E-commerce search: Match products regardless of whether users search by official names, abbreviations, or common aliases. For example, searching for cell phone can return products described as mobile device.
• Healthcare Q&A system: Match different names for the same disease or medication. For example, when patients search for heart attack, documents containing myocardial infarction are returned.
• Legal document search: Match both formal legal terminology and common expressions. For example, searching for divorce can return documents containing dissolution of marriage.
• Enterprise knowledge bases: Support abbreviations, aliases, and official names simultaneously. For example, searching for OA system can return documents containing office automation system.

Solution Architecture

Figure 1 Solution architecture

• Indexing phase: After text is analyzed, synonyms are automatically expanded before terms are stored in the inverted index. For example, if a document contains laptop, the index also stores notebook computer. Searches using either term can retrieve the document.
• Search phase: Query terms are expanded using the same synonym rules before being matched against the inverted index, providing bidirectional synonym matching.
• Dynamic dictionary loading: The synonym dictionary is stored in OBS and loaded dynamically. Updates take effect without restarting the cluster, and newly indexed documents immediately use the updated dictionary.
• Custom analyzer: A custom analyzer combines a base analyzer (standard, ik_smart, or ik_max_word) with the dynamic_synonym token filter.

Advantages

• Synonym-based search: Go beyond exact term matching by automatically expanding equivalent terms, significantly improving search recall.
• No-code configuration: Synonym rules are maintained in a text file. New synonym groups can be added without modifying application code.
• Dynamic updates: The synonym dictionary is loaded dynamically from OBS. Updates take effect without restarting the cluster.
• Flexible synonym management: Define one synonym group per line. Each group can contain any number of equivalent terms.
• Works together with main word dictionary: The synonym dictionary can be used together with the main word dictionary to ensure that multi-word expressions remain intact before synonym expansion is applied.

Constraints

• Cluster version requirements: Clusters created before March 10, 2018 do not support custom dictionaries. The sample code in this topic applies to OpenSearch clusters and Elasticsearch 7.x clusters. If you are using an Elasticsearch version earlier than 7.x, see Example code (Elasticsearch < 7.x).
• Analyzer granularity: Synonym expansion is performed after the base analyzer processes the input text. If a phrase is split into multiple terms during analysis, synonym expansion for that phrase may not work as expected. To ensure correct synonym expansion, add multi-word phrases in your synonym groups to the custom main word dictionary as well so they can be recognized as complete terms before synonym expansion is applied.

Prerequisites

• The target cluster is in the Available state and has no ongoing tasks.
• The account you are using has the following permissions (choose Permissions > Policies/Roles on the IAM console to check your permissions):
  • Permission to configure custom dictionaries:

    "css:IKThesaurus:*"

  • Permission to read OBS buckets and objects:

    obs:bucket:getBucketLocation
    obs:bucket:getBucketStoragePolicy
    obs:bucket:listAllMyBuckets
    obs:object:getObject

  • If the OBS bucket uses SSE-KMS encryption, the following KMS permissions are also required:

    "kms:cmk:create",
    "kms:dek:create",
    "kms:cmk:get",
    "kms:dek:decrypt",
    "kms:cmk:list"

Step 1: Prepare Dictionary Files

Prepare a custom synonym dictionary and upload it to an OBS bucket.

1. Prepare the synonym dictionary file synonym_custom.txt. The file must be encoded using UTF-8 without BOM. Each line contains a group of synonyms, which are separated by commas (,). Synonyms in English must use lowercase. The file size must not exceed 100 MB.

   happy,glad,joyful,pleased
   cell phone,mobile device

2. Upload the dictionary file to an OBS bucket.

   The OBS bucket must be in the same region as the CSS cluster, and its storage class must be Standard.

Step 2: Configure a Custom Synonym Dictionary

Configure a custom synonym dictionary so that the dynamic_synonym token filter can load the synonym rules.

1. Log in to the CSS management console.
2. In the navigation pane, choose Clusters > Elasticsearch or Clusters > OpenSearch.
3. In the cluster list, click the name of the target cluster. The cluster information page is displayed.
4. Choose Cluster Settings > Custom Word Dictionaries.
5. Configure the synonym dictionary as required.

   Table 1 Configuring the synonym dictionary

   Parameter	Description
   OBS Bucket	Select the OBS bucket that stores the dictionary file.
   Main Word Dictionary	Keep the default value No Update if no custom main word dictionary is required.
   Stop Word Dictionary	Keep the default value No Update if no custom main word dictionary is required.
   Synonym Dictionary	Choose Update > Select, select synonym_custom.txt, and click OK.

6. Click Save. In the displayed dialog box, click OK to start updating the synonym dictionary.

   The dictionary configuration information is displayed below. Wait for approximately 1 minute. If the dictionary status changes from Updating to Successful, the dictionary update is complete.

   Figure 2 Dictionary status
   

Step 3: Creating an Index

Create an index and configure a custom analyzer that combines word segmentation with synonym expansion so that synonyms are expanded during both indexing and search.

1. On the Custom Word Dictionaries page, click Kibana or Dashboards in the upper right corner.
2. In the left navigation pane of Kibana or Dashboards, choose Dev Tools.
3. Run the following command to create an index named book and define the custom analyzer ik_synonym, which combines the ik_smart analyzer with the dynamic_synonym token filter.

   PUT book
   {
     "settings": {
       "analysis": {
         "filter": {
           "my_synonym": {
             "type": "dynamic_synonym"
           }
         },
         "analyzer": {
           "ik_synonym": {
             "filter": [
               "my_synonym"
             ],
             "type": "custom",
             "tokenizer": "ik_smart"
           }
         }
       }
     },
     "mappings": {
       "properties": {
         "content": {
           "type": "text",
           "analyzer": "ik_synonym",
           "search_analyzer": "ik_synonym"
         }
       }
     }
   }

   Parameter description:

   • my_synonym: Name of the custom synonym token filter.
   • ik_synonym: Name of the custom analyzer.
   • ik_smart: The coarse-grained analyzer provided by the IK Analyzer.
   Expected result:

   {
     "acknowledged" : true,
     "shards_acknowledged" : true,
     "index" : "book"
   }

Step 4: Verify Synonym Expansion

After creating the index, verify that both word segmentation and synonym expansion work properly before importing data.

GET /book/_analyze
{
  "analyzer": "ik_synonym",
  "text": "I am very happy today"
}

{
  "tokens" : [
    { "token" : "I", "start_offset" : 0, "end_offset" : 1, "type" : "CN_WORD", "position" : 0 },
    { "token" : "today", "start_offset" : 1, "end_offset" : 3, "type" : "CN_WORD", "position" : 1 },
    { "token" : "very", "start_offset" : 3, "end_offset" : 5, "type" : "CN_WORD", "position" : 2 },
    { "token" : "happy", "start_offset" : 5, "end_offset" : 7, "type" : "CN_WORD", "position" : 3 },
    { "token" : "glad", "start_offset" : 5, "end_offset" : 7, "type" : "SYNONYM", "position" : 3 },
    { "token" : "joyful", "start_offset" : 5, "end_offset" : 7, "type" : "SYNONYM", "position" : 3 },
    { "token" : "pleased", "start_offset" : 5, "end_offset" : 7, "type" : "SYNONYM", "position" : 3 }
  ]
}

Result interpretation: The output contains the original term (happy) together with all three expanded synonyms at the same token position, indicating that synonym expansion is working correctly. A search using any synonym in the group can match documents containing any of the other synonyms.

Step 5: Write Test Data to the Index

Write test data into the index to verify synonym search.

Run the following command to write test text into the book index:

POST /book/_bulk
{"index":{"_id":"1"}}
{"content":"I am very happy today"}
{"index":{"_id":"2"}}
{"content":"The children were glad"}
{"index":{"_id":"3"}}
{"content":"He is so excited"}

Expected result:

{
  "took" : 9,
  "errors" : false,
  "items" : [
    { "index" : { "_index" : "book", "_type" : "_doc", "_id" : "1", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 2, "failed" : 0 }, "_seq_no" : 0, "_primary_term" : 1, "status" : 201 } },
    { "index" : { "_index" : "book", "_type" : "_doc", "_id" : "2", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 2, "failed" : 0 }, "_seq_no" : 1, "_primary_term" : 1, "status" : 201 } },
    { "index" : { "_index" : "book", "_type" : "_doc", "_id" : "3", "_version" : 1, "result" : "created", "_shards" : { "total" : 2, "successful" : 2, "failed" : 0 }, "_seq_no" : 2, "_primary_term" : 1, "status" : 201 } }
  ]
}

Step 6: Perform Synonym Search

Perform a synonym search to verify that the custom synonym dictionary and synonym filter work as expected.

GET /book/_search
{
  "query": {
    "match": {
      "content": "joyful"
    }
  }
}

Expected result:

{
  "took" : 15,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 0.8380518,
    "hits" : [
      {
        "_index" : "book",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 0.8380518,
        "_source" : {
          "content" : "I am very happy today"
        }
      },
      {
        "_index" : "book",
        "_type" : "_doc",
        "_id" : "2",
        "_score" : 0.8380518,
        "_source" : {
          "content" : "The children were glad"
        }
      }
    ]
  }
}

Result interpretation: Document 1 contains happy, and document 2 contains glad, which are configured synonyms for the query term joyful. Therefore, both documents are returned. Document 3 does not contain a configured synonym for the query term, so it is not returned. This indicates that synonym expansion has worked.

Example code (Elasticsearch < 7.x)

When the Elasticsearch version is earlier than 7.x, specify a custom type name. See the following command.

1. Run the following command to create the book index and specify the custom analyzer:

   PUT book
   {
     "settings": {
       "analysis": {
         "filter": {
           "my_synonym": {
             "type": "dynamic_synonym"
           }
         },
         "analyzer": {
           "ik_synonym": {
             "filter": [
               "my_synonym"
             ],
             "type": "custom",
             "tokenizer": "ik_smart"
           }
         }
       }
     },
     "mappings": {
       "type1": {
         "properties": {
           "content": {
             "type": "text",
             "analyzer": "ik_synonym",
             "search_analyzer": "ik_synonym"
           }
         }
       }
     }
   }

2. Run the following command to write test data to the book index:

   POST /_bulk
   { "index" : { "_index" : "book", "_type" : "type1", "_id" : "1" } }
   { "content" : "I am very happy today" }
   { "index" : { "_index" : "book", "_type" : "type1", "_id" : "2" } }
   { "content" : "The children were glad" }
   { "index" : { "_index" : "book", "_type" : "type1", "_id" : "3" } }
   { "content" : "He is so excited" }

3. Run the following command to verify the configuration by executing a synonym search:

   GET /book/type1/_search
   {
     "query": {
       "match": {
         "content": "happy"
       }
     }
   }