Updated on 2025-09-07 GMT+08:00

Mapping and Enrichment Functions

This section describes mapping and enrichment functions, including their syntax, parameters, and usage examples.

Function List

Type

Function

Description

Field mapping

e_dict_map

Maps with the target data dictionary. A new field is mapped based on the input field. This function can be used together with other functions.

e_table_map

Maps with the target table and returns the field value based on the entered field name. This function can be used together with other functions.

Search mapping

e_search_dict_map

Maps with the dictionary data of the keyword (query string) and its matching value. This function can be used together with other functions.

e_search_table_map

Maps with the table data of a column (query string) and its matching value.

e_dict_map

Maps with the target data dictionary. A new field is mapped based on the input field.

  • Function format
    e_dict_map(data, field, output_field, case_insensitive=true, missing=None, mode="overwrite")
  • Parameter description

    Parameter

    Type

    Mandatory

    Description

    data

    Dict

    Yes

    Target data dictionary. The value must be a string in the standard {key01:value01,key01:value02,...} format. Example: {"1": "TCP", "2": "UDP", "3": "HTTP", "*": "Unknown"}

    field

    String or string list

    Yes

    A field name or a list of field names. If there are multiple fields:

    • The matched values are mapped in sequence.
    • If multiple logs are matched and mode is set to overwrite, the last log overwrites the previous logs.
    • If no field is matched, the value of the missing parameter is used as the matched value.

    output_field

    String

    Yes

    Name of the output field.

    case_insensitive

    Boolean

    No

    Whether the matching is case insensitive.

    If the dictionary contains different cases of the same keyword and case_insensitive is set to true, the value that completely matches the keyword is preferentially selected. If no such value exists, a random value is selected.

    • true (default value): case insensitive
    • false: case sensitive

    missing

    String

    No

    If no matched field is found, the value of this parameter is assigned to the output field output_field. The default value is None, indicating that no mapping assignment is performed.

    If the dictionary contains a matching asterisk (*), the asterisk takes precedence over missing. In this case, the missing parameter does not take effect.

    mode

    String

    No

    Field overwrite mode. The default value is overwrite.

    The options are fill, fill-auto, add, add-auto, overwrite, and overwrite-auto.

  • Returned result

    Logs containing the new field are returned.

  • Function example
    1. Example 1: Output the new field protocol based on the value of the pro field in the test data and the target data dictionary.
      • Test data
        {
         "data":  123,
         "pro":  1
        }
      • Processing rule
        e_dict_map(
            {"1": "TCP", "2": "UDP", "3": "HTTP", "6": "HTTPS", "*": "Unknown"},
            "pro",
            "protocol",
        )
      • Processing result
        data:  123
        pro:  1
        protocol:  TCP
    2. Example 2: Output the new field message based on the value of the status field in the test data and the target data dictionary.
      • Test data (three test logs)
        {
          "status":"500"
        }
        {
          "status":"400"
        }
        {
          "status":"200"
        }
      • Processing rule
        e_dict_map({"400": "Error", "200": "Normal", "*": "Other"}, "status", "message")
      • Processing result
        status:  500
        message: Other
        status:  400
        message: Error
        status:  200
        message: Normal
  • More

    This function can be used together with other functions.

e_table_map

This function maps with the target table and returns the field value based on the entered field name.

  • Function format
    e_table_map(data, field, output_fields, missing=None, mode="fill-auto")
  • Parameter description

    Parameter

    Type

    Mandatory

    Description

    data

    Table

    Yes

    Target table.

    field

    String, string list, or tuple list

    Yes

    Source field mapped to the table in the log. If the log does not contain the corresponding field, no operation is performed.

    output_fields

    String, string list, or tuple list

    Yes

    Mapped field. Example: ["province", "pop"]

    missing

    String

    No

    If no matched field is found, the value of this parameter is assigned to the output field output_fields. The default value is None, indicating that no mapping assignment is performed. If the target field contains multiple columns, missing can be a default value list whose length is the same as the number of target fields.

    Note: If the table contains a matching asterisk (*), the asterisk * has a higher priority than missing. In this case, the missing parameter does not take effect.

    mode

    String

    No

    Field overwrite mode. The default value is fill-auto.

  • Returned result

    Logs with new field values.

  • Function example
    1. Example 1: Search for the corresponding row in the mapping table and return the value of the province field based on the city field.
      • Test data
        {
         "data": 123,
         "city": "nj"
        }
      • Processing rule
        e_table_map(
            tab_parse_csv("city,pop,province\nnj,800,js\nsh,2000,sh"), "city", "province"
        )
      • Processing result
        data: 123
        city: nj
        province: js
    2. Example 2: Search for the corresponding row in the mapping table and return the values of the province and pop fields based on the city field.
      • Test data
        {
         "data": 123,
         "city": "nj"
        }
      • Processing rule
        e_table_map(
            tab_parse_csv("city,pop,province\nnj,800,js\nsh,2000,sh"),
            "city",
            ["province", "pop"],
        )
      • Processing result
        data: 123
        city: nj
        province: js
        pop: 800
    3. Example 3: Use the tab_parse_csv function to construct a mapping table and return the values of the province and pop fields based on the city field.
      • Test data
        {
         "data": 123,
         "city": "nj"
        }
      • Processing rule
        e_table_map(
            tab_parse_csv("city#pop#province\nnj#800#js\nsh#2000#sh", sep="#"),
            "city",
            ["province", "pop"],
        )
      • Processing result
        data: 123
        city: nj
        province: js
        pop: 800
    4. Example 4: Use the tab_parse_csv function to construct a mapping table and return the values of the province and pop fields based on the city field.
      • Test data
        {
         "data": 123,
         "city": "nj"
        }
      • Processing rule
        e_table_map(
            tab_parse_csv(
                "city,pop,province\n|nj|,|800|,|js|\n|shang hai|,2000,|SHANG,HAI|", quote="|"
            ),
            "city",
            ["province", "pop"],
        )
      • Processing result
        data: 123
        city: nj
        province: js
        pop: 800
    5. Example 5: The log matching fields are different from those in the mapping table. Search for the corresponding row in the mapping table and returns the value of the province field based on the cty or city field.
      • Test data
        {
         "data": 123,
         "city": "nj"
        }
      • Processing rule
        e_table_map(
            tab_parse_csv("city,pop,province\nnj,800,js\nsh,2000,sh"),
            [("city", "city")],
            "province"
        )
      • Processing result
        data: 123
        city: nj 
        province: js
    6. Example 6: The log matching field is different from the field in the mapping table, and the output field is renamed.
      • Test data
        {
         "data": 123,
         "city": "nj"
        }
      • Processing rule
        e_table_map(
            tab_parse_csv("city,pop,province\nnj,800,js\nsh,2000,sh"),
            [("city", "city")],
            [("province", "pro")],
        )
      • Processing result
        data: 123
        city: nj 
        pro: js
    7. Example 7: There are multiple log matching fields.
      • Test data
        {
         "data": 123,
         "city": "nj",
         "pop": 800
        }
      • Processing rule
        e_table_map(
            tab_parse_csv("city,pop,province\nnj,800,js\nsh,2000,sh"),
            ["city", "pop"],
            "province",
        )
      • Processing result
        data: 123
        city: nj 
        pop: 800
        province: js
    8. Example 8: There are multiple log matching fields, which are different from the fields in the mapping table.
      • Test data
        {
         "data": 123,
         "city": "nj",
         "pp": 800
        }
      • Processing rule
        e_table_map(
            tab_parse_csv("city,pop,province\nnj,800,js\nsh,2000,sh"),
            [("city", "city"), ("pp", "pop")],
            "province",
        )
      • Processing result
        data: 123
        city: nj 
        pp: 800
        province: js
  • More

    This function can be used together with other functions.

e_search_dict_map

This function maps with the dictionary data of the keyword (query string) and its matching value.

  • Function format
    e_search_dict_map(data, output_field, multi_match=false, multi_join=" ", missing=None, mode="overwrite")
  • Parameter description

    Parameter

    Type

    Mandatory

    Description

    data

    Dict

    Yes

    Dictionary of the mapping relationship. The value must be in the standard {key01:value01,key01:value02,...} format, and the keyword key must be a query string.

    output_field

    String

    Yes

    Name of the output field.

    multi_match

    Boolean

    No

    Whether to match multiple fields. The default value is false, indicating that the function does not match multiple fields and returns only the last matched field found. multi_join can be used to concatenate multiple matched values.

    multi_join

    String

    No

    Connection string of multiple values when multiple fields are matched. The default value is a space. This parameter is valid only when multi_match is set to true.

    missing

    String

    No

    If no matched field is found, the value of this parameter is assigned to the output field output_field. The default value is None, indicating that no mapping assignment is performed.

    If the dictionary contains the default match asterisk (*), the asterisk has a higher priority than missing. In this case, the missing parameter does not take effect.

    mode

    String

    No

    Field overwrite mode. The default value is overwrite.

  • Returned result

    Mapping result after query matching.

  • Function example
    1. Example 1: matching mode.
      • Test data
        {
         "data":123 ,
         "pro":1
        }
      • Processing rule
        e_search_dict_map ({"pro==1": "TCP", "pro==2": "UDP", "pro==3": "HTTP"}, "protocol")
      • Processing result
        data:123 
        pro:1 
        protocol:TCP
    2. Example 2: Performs mapping based on different starts of field values.
      • Test data
        {
         "status":"200,300"
        }
      • Processing rule
        e_search_dict_map(
            { 
               "status:2??": "ok", 
               "status:3??": "redirect",
                "status:4??": "auth",
                "status:5??": "server_error",
            },
            "status_desc", 
           multi_match=true, 
           multi_join="test",
        )
      • Processing result
        status:200,300 
        status_desc:ok test redirect
  • More

    This function can be used together with other functions.

e_search_table_map

This function maps with the table data of a column (query string) and its matching value.

  • Function format
    e_search_table_map(data, inpt, output_fields, multi_match=false, multi_join=" ", missing=None, mode="fill-auto")
  • Parameter description

    Parameter

    Type

    Mandatory

    Description

    data

    Table

    Yes

    Table of mappings. A column in the table must be a query string.

    inpt

    String

    Yes

    Field name used for matching and searching in the table.

    output_fields

    String, String List, or Tuple List

    Yes

    Fields mapped in the table. The fields can be strings, lists, or lists of name mapping tuples.

    multi_match

    Boolean

    No

    Whether to match multiple fields. The default value is false, indicating that the function does not match multiple fields and returns only the first matched field found. multi_join can be used to combine multiple matched values.

    multi_join

    String

    No

    Connection string of multiple values when multiple fields are matched. The default value is a space. This parameter is valid only when multi_match is set to true.

    missing

    String

    No

    If no matched field is found, the value of this parameter is assigned to the output field output_fields. The default value is None, indicating that no mapping assignment is performed.

    If the table contains the default match *, the priority of * is higher than that of missing. In this case, missing does not take effect.

    mode

    String

    No

    Field overwrite mode. The default value is fill-auto.

  • Returned result

    Mapping result after query matching.

  • Function example
    1. Example 1: Map the city field in the log to the pop and province fields based on the mapping table.
      • Test data
        {
         "data": 123,
         "city": "sh"
        }

        For example, the search column in the following table is a query string.

        search

        pop

        province

        city==nj

        800

        js

        city==sh

        2000

        sh

      • Processing rule
        e_search_table_map(
            tab_parse_csv("search,pop,province\ncity==nj,800,js\ncity==sh,2000,sh"),
            "search",
            ["pop", "province"],
        )
      • Processing result
        data: 123
        city: sh 
        province: sh 
        pop: 2000
    2. Example 2: overwrite mode.
      • Test data
        {
         "data": 123,
         "city": "nj",
         "province":""
        }
      • Processing rule
        e_search_table_map( 
           tab_parse_csv("search,pop,province\ncity==nj,800,js\ncity==sh,2000,sh"),
            "search",
            "province",
            mode="overwrite",
        )
      • Processing result
        pop: 800
        data: 123
        city: nj 
        province: js
    3. Example 3: If no match is found, the value of the target field is specified by missing.
      • Test data
        {
         "data": 123,
         "city": "wh",
         "province":""
        }
      • Processing rule
        e_search_table_map(
            tab_parse_csv("search,pop,province\ncity==nj,800,\ncity==sh,2000,sh"),
            "search",
            "province",
            missing="Unknown",
        )
      • Processing result
        data: 123
        city: wh 
        province: Unknown
    4. Example 4: Multiple fields can be matched (multi_match mode).
      • Test data
        {
         "data": 123,
         "city": "nj,sh",
         "province":""
        }
      • Processing rule
        e_search_table_map(
            tab_parse_csv("search,pop,province\ncity:nj,800,js\ncity:sh,2000,sh"),
            "search",
            "province",
            multi_match=true,
            multi_join=",",
        )
      • Processing result
        data: 123
        city: nj,sh
        province: js,sh