Mapping and Enrichment Functions
This section describes mapping and enrichment functions, including their syntax, parameters, and usage examples.
Function List
Type |
Function |
Description |
Field mapping |
e_dict_map |
Maps with the target data dictionary. A new field is mapped based on the input field. This function can be used together with other functions. |
Maps with the target table and returns the field value based on the entered field name. This function can be used together with other functions. |
||
Search mapping |
Maps with the dictionary data of the keyword (query string) and its matching value. This function can be used together with other functions. |
|
Maps with the table data of a column (query string) and its matching value. |
e_dict_map
Maps with the target data dictionary. A new field is mapped based on the input field.
- Function format
e_dict_map(data, field, output_field, case_insensitive=true, missing=None, mode="overwrite")
- Parameter description
Parameter
Type
Mandatory
Description
data
Dict
Yes
Target data dictionary. The value must be a string in the standard {key01:value01,key01:value02,...} format. Example: {"1": "TCP", "2": "UDP", "3": "HTTP", "*": "Unknown"}
field
String or string list
Yes
A field name or a list of field names. If there are multiple fields:
- The matched values are mapped in sequence.
- If multiple logs are matched and mode is set to overwrite, the last log overwrites the previous logs.
- If no field is matched, the value of the missing parameter is used as the matched value.
output_field
String
Yes
Name of the output field.
case_insensitive
Boolean
No
Whether the matching is case insensitive.
If the dictionary contains different cases of the same keyword and case_insensitive is set to true, the value that completely matches the keyword is preferentially selected. If no such value exists, a random value is selected.
- true (default value): case insensitive
- false: case sensitive
missing
String
No
If no matched field is found, the value of this parameter is assigned to the output field output_field. The default value is None, indicating that no mapping assignment is performed.
If the dictionary contains a matching asterisk (*), the asterisk takes precedence over missing. In this case, the missing parameter does not take effect.
mode
String
No
Field overwrite mode. The default value is overwrite.
The options are fill, fill-auto, add, add-auto, overwrite, and overwrite-auto.
- Returned result
Logs containing the new field are returned.
- Function example
- Example 1: Output the new field protocol based on the value of the pro field in the test data and the target data dictionary.
- Test data
{ "data": 123, "pro": 1 }
- Processing rule
e_dict_map( {"1": "TCP", "2": "UDP", "3": "HTTP", "6": "HTTPS", "*": "Unknown"}, "pro", "protocol", )
- Processing result
data: 123 pro: 1 protocol: TCP
- Test data
- Example 2: Output the new field message based on the value of the status field in the test data and the target data dictionary.
- Test data (three test logs)
{ "status":"500" }
{ "status":"400" }
{ "status":"200" }
- Processing rule
e_dict_map({"400": "Error", "200": "Normal", "*": "Other"}, "status", "message")
- Processing result
status: 500 message: Other
status: 400 message: Error
status: 200 message: Normal
- Test data (three test logs)
- Example 1: Output the new field protocol based on the value of the pro field in the test data and the target data dictionary.
- More
This function can be used together with other functions.
e_table_map
This function maps with the target table and returns the field value based on the entered field name.
- Function format
e_table_map(data, field, output_fields, missing=None, mode="fill-auto")
- Parameter description
Parameter
Type
Mandatory
Description
data
Table
Yes
Target table.
field
String, string list, or tuple list
Yes
Source field mapped to the table in the log. If the log does not contain the corresponding field, no operation is performed.
output_fields
String, string list, or tuple list
Yes
Mapped field. Example: ["province", "pop"]
missing
String
No
If no matched field is found, the value of this parameter is assigned to the output field output_fields. The default value is None, indicating that no mapping assignment is performed. If the target field contains multiple columns, missing can be a default value list whose length is the same as the number of target fields.
Note: If the table contains a matching asterisk (*), the asterisk * has a higher priority than missing. In this case, the missing parameter does not take effect.
mode
String
No
Field overwrite mode. The default value is fill-auto.
- Returned result
Logs with new field values.
- Function example
- Example 1: Search for the corresponding row in the mapping table and return the value of the province field based on the city field.
- Test data
{ "data": 123, "city": "nj" }
- Processing rule
e_table_map( tab_parse_csv("city,pop,province\nnj,800,js\nsh,2000,sh"), "city", "province" )
- Processing result
data: 123 city: nj province: js
- Test data
- Example 2: Search for the corresponding row in the mapping table and return the values of the province and pop fields based on the city field.
- Test data
{ "data": 123, "city": "nj" }
- Processing rule
e_table_map( tab_parse_csv("city,pop,province\nnj,800,js\nsh,2000,sh"), "city", ["province", "pop"], )
- Processing result
data: 123 city: nj province: js pop: 800
- Test data
- Example 3: Use the tab_parse_csv function to construct a mapping table and return the values of the province and pop fields based on the city field.
- Test data
{ "data": 123, "city": "nj" }
- Processing rule
e_table_map( tab_parse_csv("city#pop#province\nnj#800#js\nsh#2000#sh", sep="#"), "city", ["province", "pop"], )
- Processing result
data: 123 city: nj province: js pop: 800
- Test data
- Example 4: Use the tab_parse_csv function to construct a mapping table and return the values of the province and pop fields based on the city field.
- Test data
{ "data": 123, "city": "nj" }
- Processing rule
e_table_map( tab_parse_csv( "city,pop,province\n|nj|,|800|,|js|\n|shang hai|,2000,|SHANG,HAI|", quote="|" ), "city", ["province", "pop"], )
- Processing result
data: 123 city: nj province: js pop: 800
- Test data
- Example 5: The log matching fields are different from those in the mapping table. Search for the corresponding row in the mapping table and returns the value of the province field based on the cty or city field.
- Test data
{ "data": 123, "city": "nj" }
- Processing rule
e_table_map( tab_parse_csv("city,pop,province\nnj,800,js\nsh,2000,sh"), [("city", "city")], "province" )
- Processing result
data: 123 city: nj province: js
- Test data
- Example 6: The log matching field is different from the field in the mapping table, and the output field is renamed.
- Test data
{ "data": 123, "city": "nj" }
- Processing rule
e_table_map( tab_parse_csv("city,pop,province\nnj,800,js\nsh,2000,sh"), [("city", "city")], [("province", "pro")], )
- Processing result
data: 123 city: nj pro: js
- Test data
- Example 7: There are multiple log matching fields.
- Test data
{ "data": 123, "city": "nj", "pop": 800 }
- Processing rule
e_table_map( tab_parse_csv("city,pop,province\nnj,800,js\nsh,2000,sh"), ["city", "pop"], "province", )
- Processing result
data: 123 city: nj pop: 800 province: js
- Test data
- Example 8: There are multiple log matching fields, which are different from the fields in the mapping table.
- Test data
{ "data": 123, "city": "nj", "pp": 800 }
- Processing rule
e_table_map( tab_parse_csv("city,pop,province\nnj,800,js\nsh,2000,sh"), [("city", "city"), ("pp", "pop")], "province", )
- Processing result
data: 123 city: nj pp: 800 province: js
- Test data
- Example 1: Search for the corresponding row in the mapping table and return the value of the province field based on the city field.
- More
This function can be used together with other functions.
e_search_dict_map
This function maps with the dictionary data of the keyword (query string) and its matching value.
- Function format
e_search_dict_map(data, output_field, multi_match=false, multi_join=" ", missing=None, mode="overwrite")
- Parameter description
Parameter
Type
Mandatory
Description
data
Dict
Yes
Dictionary of the mapping relationship. The value must be in the standard {key01:value01,key01:value02,...} format, and the keyword key must be a query string.
output_field
String
Yes
Name of the output field.
multi_match
Boolean
No
Whether to match multiple fields. The default value is false, indicating that the function does not match multiple fields and returns only the last matched field found. multi_join can be used to concatenate multiple matched values.
multi_join
String
No
Connection string of multiple values when multiple fields are matched. The default value is a space. This parameter is valid only when multi_match is set to true.
missing
String
No
If no matched field is found, the value of this parameter is assigned to the output field output_field. The default value is None, indicating that no mapping assignment is performed.
If the dictionary contains the default match asterisk (*), the asterisk has a higher priority than missing. In this case, the missing parameter does not take effect.
mode
String
No
Field overwrite mode. The default value is overwrite.
- Returned result
Mapping result after query matching.
- Function example
- Example 1: matching mode.
- Test data
{ "data":123 , "pro":1 }
- Processing rule
e_search_dict_map ({"pro==1": "TCP", "pro==2": "UDP", "pro==3": "HTTP"}, "protocol")
- Processing result
data:123 pro:1 protocol:TCP
- Test data
- Example 2: Performs mapping based on different starts of field values.
- Test data
{ "status":"200,300" }
- Processing rule
e_search_dict_map( { "status:2??": "ok", "status:3??": "redirect", "status:4??": "auth", "status:5??": "server_error", }, "status_desc", multi_match=true, multi_join="test", )
- Processing result
status:200,300 status_desc:ok test redirect
- Test data
- Example 1: matching mode.
- More
e_search_table_map
This function maps with the table data of a column (query string) and its matching value.
- Function format
e_search_table_map(data, inpt, output_fields, multi_match=false, multi_join=" ", missing=None, mode="fill-auto")
- Parameter description
Parameter
Type
Mandatory
Description
data
Table
Yes
Table of mappings. A column in the table must be a query string.
inpt
String
Yes
Field name used for matching and searching in the table.
output_fields
String, String List, or Tuple List
Yes
Fields mapped in the table. The fields can be strings, lists, or lists of name mapping tuples.
multi_match
Boolean
No
Whether to match multiple fields. The default value is false, indicating that the function does not match multiple fields and returns only the first matched field found. multi_join can be used to combine multiple matched values.
multi_join
String
No
Connection string of multiple values when multiple fields are matched. The default value is a space. This parameter is valid only when multi_match is set to true.
missing
String
No
If no matched field is found, the value of this parameter is assigned to the output field output_fields. The default value is None, indicating that no mapping assignment is performed.
If the table contains the default match *, the priority of * is higher than that of missing. In this case, missing does not take effect.
mode
String
No
Field overwrite mode. The default value is fill-auto.
- Returned result
Mapping result after query matching.
- Function example
- Example 1: Map the city field in the log to the pop and province fields based on the mapping table.
- Test data
{ "data": 123, "city": "sh" }
For example, the search column in the following table is a query string.
search
pop
province
city==nj
800
js
city==sh
2000
sh
- Processing rule
e_search_table_map( tab_parse_csv("search,pop,province\ncity==nj,800,js\ncity==sh,2000,sh"), "search", ["pop", "province"], )
- Processing result
data: 123 city: sh province: sh pop: 2000
- Test data
- Example 2: overwrite mode.
- Test data
{ "data": 123, "city": "nj", "province":"" }
- Processing rule
e_search_table_map( tab_parse_csv("search,pop,province\ncity==nj,800,js\ncity==sh,2000,sh"), "search", "province", mode="overwrite", )
- Processing result
pop: 800 data: 123 city: nj province: js
- Test data
- Example 3: If no match is found, the value of the target field is specified by missing.
- Test data
{ "data": 123, "city": "wh", "province":"" }
- Processing rule
e_search_table_map( tab_parse_csv("search,pop,province\ncity==nj,800,\ncity==sh,2000,sh"), "search", "province", missing="Unknown", )
- Processing result
data: 123 city: wh province: Unknown
- Test data
- Example 4: Multiple fields can be matched (multi_match mode).
- Test data
{ "data": 123, "city": "nj,sh", "province":"" }
- Processing rule
e_search_table_map( tab_parse_csv("search,pop,province\ncity:nj,800,js\ncity:sh,2000,sh"), "search", "province", multi_match=true, multi_join=",", )
- Processing result
data: 123 city: nj,sh province: js,sh
- Test data
- Example 1: Map the city field in the log to the pop and province fields based on the mapping table.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot