Extracting Dynamic Key-Value Pairs from Strings

Comparison of Common Solutions

Dynamic key-value pair extraction from strings includes keyword extraction, value extraction, keyword processing, and value processing. Common solutions include using the e_kv function, e_kv_delimit, or e_regex function. These three solutions are applicable to different extraction scenarios.

Most key-value pairs can be extracted using the e_kv function with specific parameters, especially when brackets and backslashes need to be extracted and escaped. In other complex or advanced scenarios, the e_regex function can be used to extract key-value pairs. The e_kv_delimit function is easier to use in certain scenarios.

Keyword Extraction

1. Example 1

   Take k1: q=asd&a=1&b=2&__1__=3 logs as an example. If you want to extract keywords and values from logs in this format, the three solutions are as follows:

   • e_kv function
     • Raw log

       {
           "k1":"q=asd&a=1&b=2&__1__=3"
       }

     • Processing rule

       e_kv("k1")

     • Processing result

       {
       	"q": "asd",
       	"a": 1,
       	"b": 2,
       	"k1": "q=asd&a=1&b=2&__1__=3",
       	"__1__": 3
       }

   • e_kv_delimit function
     • Raw log

       {
           "k1":"q=asd&a=1&b=2&__1__=3"
       }

     • Processing rule

       # Use an ampersand (&) to separate keys and values and extract keywords.
       e_kv_delimit("k1", pair_sep=r"&")

     • Processing result

       {
       	"q": "asd",
       	"a": 1,
       	"b": 2,
       	"k1": "q=asd&a=1&b=2&__1__=3",
       	"__1__": 3
       }

   • e_regex function
     • Raw log

       {
           "k1":"q=asd&a=1&b=2&__1__=3"
       }

     • Processing rule

       # Specify the character set to extract keywords and values.
       e_regex("k1",r"(\w+)=([a-zA-Z0-9]+)",{r"\1": r"\2"})

     • Processing result

       {
       	"q": "asd",
       	"a": 1,
       	"b": 2,
       	"k1": "q=asd&a=1&b=2&__1__=3",
       	"__1__": 3
       }

2. Example 2

   Take content:k1=v1&k2=v2?k3:v3 as an example. To extract keywords using a specific regular expression, do as follows:

   • e_kv_delimit function
     • Raw log

       {
           "content":"k1=v1&k2=v2?k3:v3"
       }

     • Processing rule

       e_kv_delimit("content",pair_sep=r"&?",kv_sep="(?:=|:)")

     • Processing result

       {
       	"k1": "v1",
       	"k2": "v2",
       	"k3": "v3",
       	"content": "k1=v1&k2=v2?k3:v3"
       }

   • e_regex function
     • Raw log

       {
           "content":"k1=v1&k2=v2?k3:v3"
       }

     • Processing rule

       e_regex("content",r"([a-zA-Z0-9]+)[=|:]([a-zA-Z0-9]+)",{r"\1": r"\2"})

     • Processing result

       {
       	"k1": "v1",
       	"k2": "v2",
       	"k3": "v3",
       	"content": "k1=v1&k2=v2?k3:v3"
       }

Value Extraction

• If there are clear identifiers between dynamic key-value pairs and between keywords and values, for example, a=b or a="cxxx", you are advised to use the e_kv function. Example:
  • Raw log

    {
        "content":"k=\"helloworld\",the change world, k2=\"good\""
    }

  • Processing rule

    In this case, use the e_kv function to extract the content excluding the change world:

    e_kv("content")
    # e_regex function
    e_regex("content",r"(\w+)=\"(\w+)",{r"\1": r"\2"})

  • Processing result (extracted log):

    {
    	"k2": "good",
    	"k": "helloworld",
    	"content": "k=\"helloworld\",the change world, k2=\"good\""
    }

• You are advised to use the e_kv function to extract logs containing double quotation marks (") within the content format content:k1="v1=1"&k2=v2?k3=v3 format. Example:
  • Raw log

    {
        "content":"k1=\"v1=1\"&k2=v2?k3=v3"
    }

  • Processing rule

    e_kv("content",sep="=", quote="'")

  • Processing result

    Log after processing:

    {
    	"k1": "v1=1",
    	"k2": "v2",
    	"k3": "v3",
    	"content": "k1=\"v1=1\"&k2=v2?k3=v3"
    }

Keyword Processing

• Both the e_kv and e_kv_delimit functions can process keywords and values using prefix="" and suffix="".
  • Raw log

    {
        "content":"q=asd&a=1&b=2"
    }

  • Processing rule (Each statement is executed separately. They have the same function.)

    e_kv("content", sep="=", quote='"', prefix="start_", suffix="_end")
    e_kv_delimit("content", pair_sep=r"&", kv_sep="=", prefix="start_", suffix="_end")
    e_regex("content",r"(\w+)=([a-zA-Z0-9]+)",{r"start_\1_end": r"\2"})

  • Processing result

    The data after processing is in the keyword processing format, as shown below:

    {
    	"start_b_end": 2,
    	"start_a_end": 1,
    	"start_q_end": "asd",
    	"content": "q=asd&a=1&b=2"
    }

• The e_regex function has a stronger capability of processing keywords. Example:
  • Processing rule

    e_regex("content",r"(\w+)=([a-zA-Z0-9]+)",{r"\1_\1": r"\2"})

  • Processing result

    The data after processing is in the keyword processing format, as shown below:

    {
    	"q_q": "asd",
    	"a_a": 1,
    	"b_b": 2,
    	"content": "q=asd&a=1&b=2"
    }

Value Processing