Introduction to Word Dictionaries for OpenSearch Clusters

Context

You can configure custom word dictionaries to support word segmentation. This gives the search engine improved performance when searching by keywords such as company names, for example, Huawei, and buzzwords from social media. You can also search text data based on a synonym dictionary.

CSS uses the IK and synonym analyzers. The IK analyzer uses a main word dictionary and a stop word dictionary. The synonym analyzer uses a synonym word dictionary. The IK analyzer uses the ik_max_word and ik_smart word segmentation policies. The synonym analyzer uses the ik_synonym word segmentation policy.

ik_max_word: splits the text at a fine granularity.
ik_smart: splits the text at a coarse granularity.

CSS provides four preset word dictionaries: static main word dictionary, static stop word dictionary, extra main word dictionary, and extra stop word dictionary.

If these preset word dictionaries can already support your application needs, there is no need to configure custom word dictionaries.
If these preset word dictionaries are no longer sufficient, you can add a custom main word dictionary, stop word dictionary, or synonym dictionary, or modify the four preset word dictionaries to enable search by keywords or synonyms.

Table 1 describes the different word dictionaries used by CSS.

**Table 1** Introduction to word dictionaries
Word Dictionary Type		Description	File Requirements
Custom	Main word dictionary	Main words are words that users want to use as basis for performing word segmentation. A main word dictionary is a custom collection of such words.	The word dictionary file must be a text file encoded using UTF-8 without BOM, with one word per row. The maximum size of a main word dictionary file is 100 MB. Letters must be in lowercase.
	Stop word dictionary	Stop words are words that users wish to be ignored during word segmentation. The stop word dictionary is a custom collection of stop words.	The word dictionary file must be a text file encoded using UTF-8 without BOM, with one word per row. The maximum size of a stop word dictionary file is 100 MB.
	Synonym dictionary	Synonyms are words with the same or similar meanings. A synonym dictionary is a custom collection of synonyms.	The word dictionary file must be a text file encoded using UTF-8 without BOM. Each row contains a group of synonyms separated by commas (,). The maximum size of a synonym dictionary file is 100 MB.
Built-in	Static main word dictionary	The static main word dictionary is a collection of common main words preconfigured in CSS. To view the static main word dictionary, visit https://github.com/infinilabs/analysis-ik/blob/master/config/main.dic.	The word dictionary file must be a text file encoded using UTF-8 without BOM, with one word per row. The maximum size of a static main word dictionary file is 100 MB.
	Static stop word dictionary	The static stop word dictionary is a collection of common stop words preconfigured in CSS. To view the static stop word dictionary, visit https://github.com/infinilabs/analysis-ik/blob/master/config/stopword.dic.	The word dictionary file must be a text file encoded using UTF-8 without BOM, with one word per row. The maximum size of a static stop word dictionary file is 100 MB.
	Extra main word dictionary	The Extra main word dictionary is a collection of uncommon main words preconfigured in CSS. To access the Extra main word dictionary, visit https://github.com/infinilabs/analysis-ik/blob/master/config/extra_main.dic.	The word dictionary file must be a text file encoded using UTF-8 without BOM, with one word per row. The maximum size of an Extra main word dictionary file is 100 MB.
	Extra stop word dictionary	The Extra stop word dictionary is a collection of uncommon stop words preconfigured in CSS. To access the Extra stop word dictionary, visit https://github.com/infinilabs/analysis-ik/blob/master/config/extra_stopword.dic.	The word dictionary file must be a text file encoded using UTF-8 without BOM, with one word per row. The maximum size of an Extra stop word dictionary file is 100 MB.

Constraints

You cannot use the custom word dictionary function for clusters created before the function was launched (March 10, 2018).
Custom word dictionaries are used for Chinese word segmentation. They can also be used to segment English words based on special characters other than #&+-.@_

The word dictionary files must meet the requirements described in Table 1.
The account that you use must have the permission to configure custom word dictionaries. On the IAM console, choose Permissions > Roles to check whether the account you are using has both of the following permissions. If the answer is no, contact the service administrator to add them.
- CSS Administrator
- OBS Administrator