Configuring Simplified-Traditional Chinese Conversion Search

How the Feature Works

The Simplified-Traditional Chinese conversion plugin (analysis-stconvert) supports bidirectional conversion between Simplified and Traditional Chinese. The plugin is pre-installed in CSS and ready for use out of the box. Its core capability is to convert characters during text analysis, unifying Simplified and Traditional Chinese at the index level to enable cross-form search and matching.

The Simplified-Traditional Chinese conversion plugin can be flexibly integrated into different stages of text analysis. The following four usage modes are supported:

How Simplified-Traditional Chinese Conversion Works

This topic uses the char_filter (tsconvert) mode to integrate Simplified-Traditional Chinese conversion before the IK analyzer, enabling character form normalization before word segmentation.

Figure 1 Solution architecture

The key workflow is as follows:

• Indexing phase: Original text is first processed by the char_filter (tsconvert) to convert Traditional Chinese characters into Simplified Chinese characters. The converted text is then analyzed by the IK analyzer (ik_smart), and the results are stored in the inverted index.
• Query phase: Search keywords undergo the same character normalization process before segmentation. The resulting terms are matched against the inverted index, enabling cross-form Simplified-Traditional Chinese search.
• Custom analyzer: A custom analyzer is created by combining the ik_smart analyzer with the tsconvert character filter. The same analyzer is used for both indexing and query processing.

Prerequisites

• The target cluster is in the Available state and has no ongoing tasks.
• If the custom analyzer depends on custom dictionaries, configure the custom dictionaries in advance. For details, see Configuring Custom Dictionaries.

The sample code in this topic applies to Elasticsearch 7.x clusters. If you are using an Elasticsearch version earlier than 7.x, see Example code (Elasticsearch < 7.x).

Logging In to Kibana

Log in to Kibana and go to the command execution page. Elasticsearch clusters support multiple access methods. This topic uses Kibana as an example to describe the operation procedures.

1. Log in to the CSS management console.
2. In the navigation pane on the left, choose Clusters > Elasticsearch.
3. In the cluster list, find the target cluster, and click Kibana in the Operation column to log in to the Kibana console.
4. In the left navigation pane, choose Dev Tools.

   The left part of the console is the command input box, and the triangle icon in its upper-right corner is the execution button. The right part shows the execution result.

Creating an Index

Create an index and define a custom analyzer with Simplified-Traditional Chinese conversion logic. The analyzer normalizes character forms before word segmentation during both indexing and query processing.

Run the following command to create the test index and configure the custom analyzer ts_ik (which combines ik_smart and tsconvert).

PUT /test
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0,
    "analysis": {
      "analyzer": {
        "ts_ik": {
          "tokenizer": "ik_smart",
          "char_filter": [
            "tsconvert"
          ]
        }
      },
      "char_filter": {
        "tsconvert": {
          "type": "stconvert",
          "convert_type": "t2s"
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "desc": {
        "type": "text",
        "analyzer": "ts_ik"
      }
    }
  }
}

Expected result:

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "test"
}

Verifying Character-Form Conversion and Text Analysis

After creating the index, verify the custom analyzer to ensure that Simplified-Traditional Chinese conversion and word segmentation are configured correctly before importing data.

Run the following command to check how the ts_ik custom analyzer converts and segments Traditional Chinese text.

POST /test/_analyze
{
  "analyzer": "ts_ik",
  "text": "Text in traditional Chinese"
}

Expected result:

{
  "tokens" : [
    { "token": "Simplified Chinese term 1", "start_offset": 0, "end_offset": 2, "type": "CN_WORD", "position": 0 },
    { "token" : "Simplified Chinese term 1", "start_offset" : 2, "end_offset" : 5, "type" : "CN_WORD", "position" : 1 }
  ]
}

Result interpretation: The Traditional Chinese text is converted into Simplified Chinese text first. The ik_smart analyzer then segments the converted text into terms. Both conversion and tokenization are working as expected.

Importing Data

Import both Traditional and Simplified Chinese content into the index to verify cross-form search with mixed Simplified and Traditional Chinese data.

Run the following command to write test data to the test index:

POST /test/_bulk
{"index":{"_id":"1"}}
{"desc":"Traditional Chinese sample text"}
{"index":{"_id":"2"}}
{"desc":"Simplified Chinese sample text"}
{"index":{"_id":"3"}}
{"desc":"Unrelated text"}

Expected result:

{
  "took" : 6,
  "errors" : false,
  "items" : [
    { "index" : { "_index" : "test", "_type" : "_doc", "_id" : "1", "_version" : 1, "result" : "created", "_shards" : { "total" : 1, "successful" : 1, "failed" : 0 }, "_seq_no" : 0, "_primary_term" : 1, "status" : 201 } },
    { "index" : { "_index" : "test", "_type" : "_doc", "_id" : "2", "_version" : 1, "result" : "created", "_shards" : { "total" : 1, "successful" : 1, "failed" : 0 }, "_seq_no" : 1, "_primary_term" : 1, "status" : 201 } },
    { "index" : { "_index" : "test", "_type" : "_doc", "_id" : "3", "_version" : 1, "result" : "created", "_shards" : { "total" : 1, "successful" : 1, "failed" : 0 }, "_seq_no" : 2, "_primary_term" : 1, "status" : 201 } }
  ]
}

Verifying Simplified-Traditional Chinese Conversion Search

Search using both Simplified Chinese and Traditional Chinese keywords to verify cross-form search.

GET /test/_search
{
  "query": {
    "match": {
      "desc": "Simplified Chinese keyword"
    }
  }
}

GET /test/_search
{
  "query": {
    "match": {
      "desc": "Traditional Chinese keyword"
    }
  }
}

The results of the two searches are expected to be the same:

{
...
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 0.43445712,
    "hits" : [
      {
        "_index" : "test",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 0.43445712,
        "_source" : {
          "desc": "Traditional Chinese document"
        }
      },
      {
        "_index" : "test",
        "_type" : "_doc",
        "_id" : "2",
        "_score" : 0.43445712,
        "_source" : {
          "desc" : "Simplified Chinese document"
        }
      }
    ]
  }
}

Result interpretation: Whether you search using the Simplified Chinese keyword or its Traditional Chinese equivalent, both Traditional Chinese and Simplified Chinese documents containing the keyword are returned. This indicates that bidirectional Simplified-Traditional Chinese search is working. Documents that do not contain the keyword are not returned.

FAQ

• Question 1: How do I configure Simplified Chinese search to match Traditional Chinese documents?

  When creating an index, replace the tsconvert (t2s) character filter with the stconvert (s2t) character filter. In this configuration, Simplified Chinese content is converted to Traditional Chinese during indexing, and user queries are also converted to Traditional Chinese during query processing. Configure the analyzer as follows:

  PUT /test1
  {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0,
      "analysis": {
        "analyzer": {
          "ts_ik": {
            "tokenizer": "ik_smart",
            "char_filter": [
              "stconvert"
            ]
          }
        },
        "char_filter": {
          "stconvert": {
            "type": "stconvert",
            "convert_type": "s2t"
          }
        }
      }
    },
    "mappings": {
      "properties": {
        "desc": {
          "type": "text",
          "analyzer": "ts_ik"
        }
      }
    }
  }

• Question 2: Do existing indexes need to be rebuilt to support Simplified-Traditional Chinese conversion?

  Analyzer configurations can only be specified when an index is created and cannot be modified for existing indexes. Whether an index needs to be rebuilt depends on the existing data:

  • If all existing data is in Simplified Chinese, no index rebuild is required after adding a Traditional-to-Simplified Chinese analyzer. Existing data is already in Simplified Chinese, and newly ingested Traditional Chinese data will be automatically converted.
  • If existing data contains Traditional Chinese content, the index must be rebuilt after adding a Traditional-to-Simplified Chinese analyzer. Otherwise, historical Traditional Chinese data cannot be retrieved by Simplified Chinese keywords.
  The following two methods can be used to rebuild an index:

  • Method 1: Update in place (where there is only a small amount of data)

    Run the following command:

    POST /{my_index}/_update_by_query

    This command re-analyzes existing documents but does not modify the analyzer configuration of the index. This method works only if the index was originally created with the Simplified-Traditional Chinese conversion analyzer.

  • Method 2: Switch indexes seamlessly using an alias
    1. Create a new index configured with the Simplified-Traditional Chinese conversion analyzer.
    2. Migrate data from the old index to the new index using _reindex.

       POST /_reindex
       {
         "source": { "index": "old_index" },
         "dest": { "index": "new_index" }
       }

    3. Point the alias to the new index.

       POST /_aliases
       {
         "actions": [
           {
             "remove": {
               "index": "old_index",
               "alias": "my_alias"
             }
           },
           {
             "add": {
               "index": "new_index",
               "alias": "my_alias"
             }
           }
         ]
       }

    4. (Optional) Delete the old index to release storage space.

       DELETE /old_index

Example code (Elasticsearch < 7.x)

When the Elasticsearch version is earlier than 7.x, specify a custom type name. See the following command.

1. Run the following command to create the test index and specify the custom analyzer:

   PUT /test
   {
     "settings": {
       "number_of_shards": 1,
       "number_of_replicas": 0,
       "analysis": {
         "analyzer": {
           "ts_ik": {
             "tokenizer": "ik_smart",
             "char_filter": [
               "tsconvert"
             ]
           }
         },
         "char_filter": {
           "tsconvert": {
             "type": "stconvert",
             "convert_type": "t2s"
           }
         }
       }
     },
     "mappings": {
       "type1": {
         "properties": {
           "desc": {
             "type": "text",
             "analyzer": "ts_ik"
           }
         }
       }
     }
   }

2. Run the following command to write test data to the test index:

   POST /_bulk
   { "index" : { "_index" : "test", "_type" : "type1", "_id" : "1" } }
   { "desc" : "Traditional Chinese sample text" }
   { "index" : { "_index" : "test", "_type" : "type1", "_id" : "2" } }
   { "desc" : "Simplified Chinese sample text" }
   { "index" : { "_index" : "test", "_type" : "type1", "_id" : "3" } }
   { "desc" : "Unrelated text" }

3. Run the following command to verify the configuration by executing a keyword-based query:

   GET /test/type1/_search
   {
     "query": {
       "match": {
         "desc": "Traditional Chinese keyword"
       }
     }
   }