Managing Word Dictionaries

Context

Custom word dictionaries use the IK and synonym analyzers.

The IK analyzer has a main word dictionary and a stop word dictionary. The synonym analyzer has a synonym word dictionary. Before configuring a custom word dictionary, upload the prepared word dictionary file to OBS. For details, see Uploading a Word Dictionary File to OBS.

The IK analyzer uses the ik_max_word and ik_smart word segmentation policies. The synonym analyzer uses the ik_synonym word segmentation policy.

• ik_max_word: splits the text at a fine granularity.
• ik_smart: splits the text at a coarse granularity.

Prerequisites

Uploading a Word Dictionary File to OBS

Before configuring a custom word dictionary, upload the word dictionary file to an OBS bucket.

1. Prepare the word dictionary file according to Table 1.

   Table 1 Word dictionary description

   Word Dictionary Type	Description	Requirement
   Main word dictionary	Main words are words that users want to use as basis for performing word segmentation. A main word dictionary is a custom collection of such words.	The word dictionary file must be a text file encoded using UTF-8 without BOM, with one word per row. The maximum size of a main word dictionary file is 100 MB. Letters must be in lowercase.
   Stop word dictionary	Stop words are words that users wish to be ignored during word segmentation. The stop word dictionary is a custom collection of stop words.	The word dictionary file must be a text file encoded using UTF-8 without BOM, with one word per row. The maximum size of a stop word dictionary file is 100 MB.
   Synonym dictionary	Synonyms are words with the same or similar meanings. A synonym dictionary is a custom collection of synonyms.	The word dictionary file must be a text file encoded using UTF-8 without BOM. Each row contains a group of synonyms separated by commas (,). The maximum size of a synonym dictionary file is 100 MB.
   Static main word dictionary	The static main word dictionary is a collection of common main words preconfigured in CSS. To view the static main word dictionary, visit https://github.com/infinilabs/analysis-ik/blob/master/config/main.dic.	The word dictionary file must be a text file encoded using UTF-8 without BOM, with one word per row. The maximum size of a static main word dictionary file is 100 MB.
   Static stop word dictionary	The static stop word dictionary is a collection of common stop words preconfigured in CSS. To view the static stop word dictionary, visit https://github.com/infinilabs/analysis-ik/blob/master/config/stopword.dic.	The word dictionary file must be a text file encoded using UTF-8 without BOM, with one word per row. The maximum size of a static stop word dictionary file is 100 MB.
   Extra main word dictionary	The Extra main word dictionary is a collection of uncommon main words preconfigured in CSS. To access the Extra main word dictionary, visit https://github.com/infinilabs/analysis-ik/blob/master/config/extra_main.dic.	The word dictionary file must be a text file encoded using UTF-8 without BOM, with one word per row. The maximum size of an Extra main word dictionary file is 100 MB.
   Extra stop word dictionary	The Extra stop word dictionary is a collection of uncommon stop words preconfigured in CSS. To access the Extra stop word dictionary, visit https://github.com/infinilabs/analysis-ik/blob/master/config/extra_stopword.dic.	The word dictionary file must be a text file encoded using UTF-8 without BOM, with one word per row. The maximum size of an Extra stop word dictionary file is 100 MB.

2. Upload the word dictionary file to an OBS bucket. For details, see Object Storage Service User Guide. The OBS bucket to which data is uploaded must be in the same region as the cluster.

Managing Word Dictionaries

1. Log in to the CSS management console.
2. In the navigation pane, choose Clusters > Elasticsearch.
3. On the Clusters page, click the name of the target cluster.
4. Click the Word Dictionaries tab.
5. On the Word Dictionaries page, configure custom word dictionaries for the cluster or modify preset ones.
   1. To configure custom word dictionaries, configure the required settings by referring to Table 2.

      Table 2 Configuring custom word dictionaries

      Parameter	Description
      OBS Bucket	Select the OBS location for storing the word dictionary file. You can click Create Bucket to create an OBS bucket. The new OBS bucket must be in the same region as the cluster, and Default Storage Class must be Standard or Infrequent Access.
      Main Word Dictionary	The main word dictionary is a custom word dictionary. Its initial status is empty. By default, No Update is selected, meaning not to configure this word dictionary. To add a custom main word dictionary, click Update and select a .txt word dictionary file. If you do not want to use a certain word dictionary, click Do Not Use.
      Stop Word Dictionary	Stop Word Dictionary is a custom word dictionary. Its initial status is empty. By default, No Update is selected, meaning not to configure this word dictionary. To add a custom stop word dictionary, click Update and select a .txt word dictionary file. If you do not want to use a certain word dictionary, click Do Not Use.
      Synonym Dictionary	Synonym Dictionary is a custom word dictionary. Its initial status is empty. By default, No Update is selected, meaning not to configure this word dictionary. To add a custom synonym dictionary, click Update and select a .txt word dictionary file. If you do not want to use a certain word dictionary, click Do Not Use.

   2. To modify a preset word dictionary, toggle on Modify Preset Word Dictionary, and modify the word dictionary by referring to Table 3.
      

      If the four preset word dictionaries (static & extra main word and stop word) are not displayed, the current cluster version does not support the deletion or modification of them. To use this function, you are advised to upgrade the cluster version, or create a new cluster and migrate data to it.

      Table 3 Configuring preset word dictionaries

      Parameter	Description
      Static Main Word Dictionary	Static Main Word Dictionary is a preset collection of common main words. By default, No Update is selected, meaning this word dictionary will be used. To modify Static Main Word Dictionary, click Update and select a .txt word dictionary file. If you do not want to use Static Main Word Dictionary, click Do Not Use.
      Static Stop Word Dictionary	Static Stop Word Dictionary is a preset collection of common stop words. By default, No Update is selected, meaning this word dictionary will be used. To modify Static Stop Word Dictionary, click Update and select a .txt word dictionary file. If you do not want to use Static Stop Word Dictionary, click Do Not Use.
      Extra Main Word Dictionary	Extra Main Word Dictionary is a preset collection of uncommon main words. By default, No Update is selected, meaning this word dictionary will be used. To modify Extra Main Word Dictionary, click Update and select a .txt word dictionary file. If you do not want to use Extra Main Word Dictionary, click Do Not Use.
      Extra Stop Word Dictionary	Extra Stop Word Dictionary is a preset collection of uncommon stop words. By default, No Update is selected, meaning this word dictionary will be used. To modify Extra Stop Word Dictionary, click Update and select a .txt word dictionary file. If you do not want to use Extra Stop Word Dictionary, click Do Not Use.

6. Click Save. In the dialog box that is displayed, click OK. The word dictionary information is displayed in the lower part of the page. The word dictionary status is Updating. In approximately 1 minute, the word dictionaries are configured, and Word Dictionary Status changes to Successful.
7. The deletion or update of the four preset dictionaries (static main word, static stop word, extra main word, and extra stop word) requires a cluster restart to take effect. The update of other word dictionaries happens dynamically, and there is no need for cluster restart. For details about how to restart a cluster, see Restarting a Cluster.