Configuring Custom Dictionaries

How the Feature Works

Dictionaries are core resources that guide analyzers in recognizing special terms. CSS supports the following three types of dictionaries:

• Main word dictionary: Defines special terms that should be recognized as independent terms rather than split into separate words by the analyzer.
• Stop word dictionary: Defines meaningless words that should be filtered out in order to reduce noise.
• Synonym dictionary: Defines groups of semantically equivalent terms to enable synonym-based search expansion.

Dictionaries take effect only when used together with analyzers. CSS provides a built-in standard analyzer, IK analyzers, and a synonym filter. IK analyzers are implemented through the IK analysis plugin, and the synonym filter is implemented through the dynamic synonym plugin. Table 1 describes the mapping between dictionaries and analyzers.

Table 1 Mapping between dictionaries and analyzers

Analyzer	Description	Dictionary Dependency
Default analyzer: Standard Analyzer	A standard analyzer based on Unicode text segmentation. It automatically converts text to lowercase and splits Chinese text into individual characters. For example, Hello World is split into hello,world. Advantage: Simple configuration; no dictionary maintenance required. Disadvantage: Poor performance in Chinese segmentation. Applicable scenario: Indexing Western languages or scenarios with low requirements for Chinese text analysis.	No dictionary required
IK Analyzer ik_smart	A coarse-grained analyzer that generates only the most reasonable segmentation results without enumerating all possible combinations. Example: Text is segmented into the most likely word combinations rather than all possible combinations. Advantage: Provides good Chinese text segmentation, reduces noise, and improves search precision. Disadvantage: May miss some less commonly used terms or word combinations. Applicable scenario: Query analysis during the search phase.	Requires main word and stop word dictionaries (built-in dictionaries are provided; custom dictionaries are also supported)
IK Analyzer ik_max_word	A fine-grained analyzer that enumerates all possible word combinations and generates the finest-grained segmentation results. Example: Text is segmented into all possible word combinations. Advantage: Provides good Chinese segmentation, covers more term combinations, and improves search recall. Disadvantage: Increases index size and slightly reduces write performance. Applicable scenario: Index building for full-text search during data ingestion.	Requires main word and stop word dictionaries (built-in dictionaries are provided; custom dictionaries are also supported)
Synonym filter dynamic_synonym	A synonym expansion filter that dynamically loads synonym dictionaries and performs semantic expansion or correction on segmentation results. Example: Searching for cell phone also returns documents containing mobile phone and mobile device. Advantage: Enables synonym expansion search and improves search recall. Disadvantage: Must be used together with the standard analyzer or IK analyzer. Applicable scenarios: Semantic expansion search and term correction.	Requires a synonym dictionary (supports custom dictionaries only)

The IK Analyzer is designed primarily for Chinese text. When used to analyze English text, only the following special characters are treated as regular characters and preserved: # & + - . @ _. Other special characters are treated as segmentation points where text can be split into different parts. Example:

• access address: example@domain.com is split into access,address,example@domain.com.
• set the parameter value to user|domain is split into set,parameter,value,user,domain. (the and to are in the built-in Static Stop Word Dictionary and are filtered out during word segmentation.)

How Custom Dictionaries Work

The workflow of custom dictionaries is as follows:

1. Dictionary loading: After uploading dictionary files to an OBS bucket, configure the dictionaries for the cluster. The system then loads the dictionaries to all nodes in the cluster.
2. Main word dictionary matching: The analyzer first matches entries in the main word dictionary and keeps matched entries as complete terms during segmentation, preventing special terms from being split.
3. Stop word filtering: Entries matching the stop word dictionary in the segmentation results are discarded and are not indexed.
4. Synonym expansion: The segmentation results are processed by the synonym filter. Entries matching the synonym dictionary are expanded into additional synonym terms to enable semantic expansion search.
5. Scope of update: After a dictionary is updated, newly written data is automatically analyzed using the new dictionary. Historical index data is not rebuilt automatically. To re-analyze historical data using the updated dictionary, manually run the _update_by_query command.

Dictionary File Specifications

Prepare dictionary files according to the following specifications. Files that do not meet these requirements will fail to be loaded.

Constraints

Cluster version requirements: Clusters created before March 10, 2018 do not support custom dictionaries.

Prerequisites

• The target cluster is in the Available state and has no ongoing tasks.
• The account you are using has the following permissions (choose Permissions > Policies/Roles on the IAM console to check your permissions):
  • Permission to configure custom dictionaries:

    "css:IKThesaurus:*"

  • To update dictionaries, the account must also have read access to the OBS bucket and objects:

    obs:bucket:getBucketLocation
    obs:bucket:getBucketStoragePolicy
    obs:bucket:listAllMyBuckets
    obs:object:getObject

  • If the OBS bucket uses SSE-KMS encryption, the following KMS permissions are also required:

    "kms:cmk:create",
    "kms:dek:create",
    "kms:cmk:get",
    "kms:dek:decrypt",
    "kms:cmk:list"

• To update dictionaries, prepare dictionary files that comply with the required format and upload them to an OBS bucket. For details, see Uploading an Object.

  The OBS bucket must be in the same region as the CSS cluster, and its storage class must be Standard.

Accessing the Custom Dictionary Configuration Page

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the cluster list, click the name of the target cluster. The cluster information page is displayed.
4. Choose Cluster Settings > Custom Word Dictionaries.

Configuring Custom Dictionaries

If built-in dictionaries do not meet your business requirements for text analysis, you can configure custom dictionaries.

1. On the Custom Word Dictionaries tab, configure custom dictionaries as required.

   Table 3 Configuring custom dictionaries

   Parameter	Description
   OBS Bucket	Select the OBS bucket that stores the dictionary file. This parameter is required only when you update dictionaries. If no OBS buckets meet your requirements, click Create Bucket to go to the OBS console and create one. For details, see Creating a Bucket.
   Main Word Dictionary	A custom collection of main words. It is empty initially. Default value: No Update, meaning not to configure this dictionary. To create a custom dictionary or update an existing one, choose Update > Select, select a dictionary file that complies with the required format from the OBS bucket, and click OK. To delete a dictionary, click Do Not Use.
   Stop Word Dictionary	A custom collection of stop words. It is empty initially. Default value: No Update, meaning not to configure this dictionary. To create a custom dictionary or update an existing one, choose Update > Select, select a dictionary file that complies with the required format from the OBS bucket, and click OK. To delete a dictionary, click Do Not Use.
   Synonym Dictionary	A custom collection of synonyms. It is empty initially. Default value: No Update, meaning not to configure this dictionary. To create a custom dictionary or update an existing one, choose Update > Select, select a dictionary file that complies with the required format from the OBS bucket, and click OK. To delete a dictionary, click Do Not Use.

2. Click Save. In the displayed dialog box, click OK to start updating dictionaries.

   The dictionary configuration information is displayed below. Wait for approximately 1 minute. If the dictionary status changes from Updating to Successful, the dictionary update is complete.

   Figure 1 Dictionary status
   
3. (Optional) After a dictionary is updated, newly written data uses the updated dictionary for analysis. Historical index data is not rebuilt automatically. To re-analyze and re-index historical data, manually run the following command:

   POST /{my_index}/_update_by_query

Updating Built-in Dictionaries

You can update built-in dictionaries if you need to modify or disable them.

1. On the Custom Word Dictionaries page, toggle on the switch next to Modify Preset Word Dictionaries.
   

   If Modify Preset Word Dictionaries is not displayed, the current cluster version does not support built-in dictionary updates. In this case, you are advised to upgrade the cluster version or create a new cluster and then migrate data to it.

2. Update built-in dictionaries as required.

   Table 4 Updating built-in dictionaries

   Parameter	Description
   OBS Bucket	Select the OBS bucket that stores the dictionary file. This parameter is required only when you update dictionaries. If no OBS buckets meet your requirements, click Create Bucket to go to the OBS console and create one. For details, see Creating a Bucket.
   Static Main Word Dictionary	A built-in collection of commonly used main words. Default value: No Update, meaning not to configure this dictionary. To update this dictionary, choose Update > Select, select a dictionary file that complies with the required format from the OBS bucket, and click OK. To delete this dictionary, click Do Not Use.
   Static Stop Word Dictionary	A built-in collection of commonly used stop words. Default value: No Update, meaning not to configure this dictionary. To update this dictionary, choose Update > Select, select a dictionary file that complies with the required format from the OBS bucket, and click OK. To delete this dictionary, click Do Not Use.
   Extra Main Word Dictionary	A built-in collection of uncommon main words. Default value: No Update, meaning not to configure this dictionary. To update this dictionary, choose Update > Select, select a dictionary file that complies with the required format from the OBS bucket, and click OK. To delete this dictionary, click Do Not Use.
   Extra Stop Word Dictionary	A built-in collection of uncommon stop words. Default value: No Update, meaning not to configure this dictionary. To update this dictionary, choose Update > Select, select a dictionary file that complies with the required format from the OBS bucket, and click OK. To delete this dictionary, click Do Not Use.

3. Click Save. In the displayed dialog box, click OK to start updating dictionaries.

   The dictionary configuration information is displayed below. Wait for approximately 1 minute. If the dictionary status changes from Updating to Restart required, the dictionary update is complete.

   Figure 2 Dictionary status
   
4. Click Restart in the upper-right corner of the page to restart the cluster and apply changes made to the built-in dictionaries. For details, see Restarting a Cluster.
5. (Optional) After a dictionary is updated, newly written data uses the updated dictionary for analysis. Historical index data is not rebuilt automatically. To re-analyze and re-index historical data, manually run the following command:

   POST /{my_index}/_update_by_query

Verifying Main Word and Stop Word Dictionaries

After configuring the main word dictionary and stop word dictionary, you can use the /_analyze API to test the text segmentation results.

Scenario: The custom main word dictionary and stop word dictionary have been configured with the necessary main words and stop words.

1. On the Custom Word Dictionaries page, click Kibana in the upper-right corner to log in to Kibana.
2. In the left navigation pane on Kibana, choose Dev Tools.
3. Run the following commands to check the segmentation results of ik_smart and ik_max_word:
   • ik_smart (coarse-grained) segmentation
     Example request:

     POST /_analyze
     {
       "analyzer":"ik_smart",
       "text":"Sample Chinese text"
     }

     Expected result: The main words in the provided text are recognized as independent terms, and stop words are filtered out.

     {
       "tokens": [
         {
           "token": "main_word_1",
           "start_offset": 0,
           "end_offset": 4,
           "type": "CN_WORD",
           "position": 0
         },
         {
           "token": "main_word_2",
           "start_offset": 5,
           "end_offset": 8,
           "type": "CN_WORD",
           "position": 1
         }
       ]
     }

   • ik_max_word (fine-grained) segmentation

     Example request:

     POST /_analyze
     {
       "analyzer":"ik_max_word",
       "text":"Sample Chinese text"
     }

     Expected result: The main words in the provided text are segmented into multiple possible terms, and stop words are filtered out.

     {
       "tokens": [
         { "token": "word_1", "start_offset": 0, "end_offset": 4, "type": "CN_WORD", "position": 0 },
         { "token": "word_2", "start_offset": 0, "end_offset": 2, "type": "CN_WORD", "position": 1 },
         { "token": "word_3", "start_offset": 0, "end_offset": 1, "type": "CN_WORD", "position": 2 },
         { "token": "word_4", "start_offset": 1, "end_offset": 3, "type": "CN_WORD", "position": 3 },
         { "token": "word_5", "start_offset": 2, "end_offset": 4, "type": "CN_WORD", "position": 4 },
         { "token": "word_6", "start_offset": 3, "end_offset": 4, "type": "CN_WORD", "position": 5 },
         { "token": "word_7", "start_offset": 5, "end_offset": 8, "type": "CN_WORD", "position": 6 },
         { "token": "word_8", "start_offset": 5, "end_offset": 7, "type": "CN_WORD", "position": 7 },
         { "token": "word_9", "start_offset": 6, "end_offset": 8, "type": "CN_WORD", "position": 8 },
         { "token": "word_10", "start_offset": 7, "end_offset": 8, "type": "CN_WORD", "position": 9 }
       ]
     }

Verifying Synonym Expansion

After configuring the synonym dictionary, you can test the dictionary through synonym search.

Scenario: The synonym dictionary contains synonyms "happy, glad".

1. On the Custom Word Dictionaries page, click Kibana in the upper-right corner to log in to Kibana.
2. In the left navigation pane on Kibana, choose Dev Tools.
3. Run the following command to create the test index and configure the synonym filter.

   PUT test
   {
       "settings": {
           "analysis": {
               "filter": {
                   "my_synonym": {
                       "type": "dynamic_synonym"
                   }
               },
               "analyzer": {
                   "ik_synonym": {
                       "filter": [
                           "my_synonym"
                       ],
                       "type": "custom",
                       "tokenizer": "ik_smart"
                   }
               }
           }
       },
       "mappings": {
           "properties": {
               "desc": {
                   "type": "text",
                   "analyzer": "ik_synonym"
               }
           }
       }
   }

4. Run the following command to verify synonym filter configuration.

   GET /test/_analyze
   {
     "analyzer": "ik_synonym",
     "text": ["I'm very happy"]
   }

   Expected result: Although the text uses "happy", the segmentation results also contain "glad" based on the synonym mapping.

   {
     "tokens": [
       { "token": "I'm", "start_offset": 0, "end_offset": 2, "type": "CN_WORD", "position": 0 },
       { "token": "happy", "start_offset": 2, "end_offset": 4, "type": "CN_WORD", "position": 1 },
       { "token": "glad", "start_offset": 2, "end_offset": 4, "type": "SYNONYM", "position": 1 }
     ]
   }

Preset Dictionaries

To help you quickly get started with the IK analyzer, CSS provides four built-in dictionaries, as listed in Table 5. If these preset main word and stop word dictionaries already meet your requirements, there is no need to configure custom dictionaries.

Related Documents

• Improving Keyword Search Accuracy with the IK Analyzer and Custom Dictionaries: Configure custom main word and stop word dictionaries with the IK analyzer to preserve domain-specific terms and filter irrelevant words, enabling more precise keyword search.
• Improving Search Recall with the Synonym Filter and a Custom Synonym Dictionary: Configure a custom synonym dictionary and use the dynamic_synonym synonym filter to automatically expand synonyms during query and indexing, enabling synonym-based search across equivalent terms.