Updated on 2025-09-07 GMT+08:00

Parsing Function

This section describes the User-Agent parsing functions, including their syntax, parameters, and usage examples.

Function List

Function

Description

ua_parse_device

Parses device information in User-Agent.

ua_parse_os

Parses OS information in User-Agent.

ua_parse_agent

Parses browser information in User-Agent.

ua_parse_all

Parses all information in User-Agent.

url_parse

Parses the components of the URL.

url_parse_qs

Parses parameters contained in the query string of the URL.

The User-Agent parsing function deletes the fields whose parsing result is None. For example, if the parsed device data is {'brand': None, 'family': 'Other', 'model': None}, the brand and model fields are deleted. The final parsing result is {'family': 'Other'}.

ua_parse_device

Use the ua_parse_device function to parse device information in User-Agent.

  • Function format
    ua_parse_device(value)
  • Parameter description

    Parameter

    Type

    Mandatory

    Description

    value

    String

    Yes

    User-Agent string to be parsed.

  • Returned result

    A JSON dataset.

  • Function example
    • Test data
      {
        "http_user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36"
      }
    • Processing rule
      e_set("new_column",ua_parse_device(v("http_user_agent")))
    • Processing result
      http_user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36
      new_column:{"family":"Mac","brand":"Apple","model":"Mac"}

ua_parse_os

Use the ua_parse_os function to parse OS information in User-Agent.

  • Function format
    ua_parse_os(value)
  • Parameter description

    Parameter

    Type

    Mandatory

    Description

    value

    String

    Yes

    User-Agent string to be parsed.

  • Returned result

    A JSON dataset.

  • Function example
    • Test data
      {
          "http_user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36"
      }
    • Processing rule
      e_set("new_column",ua_parse_os(v("http_user_agent")))
    • Processing result
      http_user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36
      new_column:{'family': 'Mac OS X', 
                            'major': '10',
                            'minor': '9', 
                            'patch': '4'}

ua_parse_agent

Use the ua_parse_agent function to parse browser information in User-Agent.

  • Function format
    ua_parse_agent(value)
  • Parameter description

    Parameter

    Type

    Mandatory

    Description

    value

    String

    Yes

    User-Agent string to be parsed.

  • Returned result

    A JSON dataset.

  • Function example
    • Test data
      {
         "http_user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36"
      }
    • Processing rule
      e_set("new_column",ua_parse_agent(v("http_user_agent")))
    • Processing result
      http_user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36
      new_column:{'family': 'Chrome', 'major': '192', 'minor': '168', 'patch': '0'

ua_parse_all

Use the ua_parse_all function to parse all information in User-Agent.

  • Function format
    ua_parse_all(value)
  • Parameter description

    Parameter

    Type

    Mandatory

    Description

    value

    String

    Yes

    User-Agent string to be parsed.

  • Returned result

    A JSON dataset.

  • Function example
    • Test data
      {
        "http_user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36"
      }
    • Processing rule
      e_set("new_column",ua_parse_all(v("http_user_agent")))
    • Processing result
      http_user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36
      new_column: { 
       "user_agent": {
         "family": "Chrome",
          "major": "192",
          "minor": "168",
          "patch": "0"
         },
        "os": {
          "family": "Mac OS X",
          "major": "10",
          "minor": "9",
          "patch": "4"
        },
        "device": { 
         "family": "Mac",
          "brand": "Apple",
          "model": "Mac" 
       }
      }

url_parse

Use the url_parse function to parse the components of the URL.

  • Function format
    url_parse(url, scheme="", allow_fragments=true)
  • Parameter description

    Parameter

    Type

    Mandatory

    Description

    value

    String

    Yes

    URL to be parsed.

    scheme

    String

    No

    Network protocol. The default value is an empty string. The scheme field in the returned result uses the value of this parameter only when the network protocol is not specified in the URL.

    allow_fragments

    Boolean

    No

    Whether to parse the fragment part in the URL.

    • true (default value): The fragment part in the URL is parsed, and the fragment field in the returned result is a specific value.
    • false: The fragment part in the URL is not parsed, and the fragment field in the returned result is an empty string.
  • Returned result

    The parsed JSON data is returned. The following table describes the fields.

    Field

    Description

    scheme

    Network protocol

    netloc

    Network location

    path

    Hierarchical path identifier

    query

    Query component

    fragment

    Fragment identifier

  • Function example
    1. Example 1: Use the default parameters to return the components of the URL.
      • Test data
        {
           "content":"https://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib"
        }
      • Processing rule
        e_set("url",url_parse(v("content")))
      • Processing result
        content:https://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib
        url:{
                "scheme": "https",
                 "netloc": "username:username@example.com:8083",
                 "path": "/hello/asdah/",
                 "params": "type=docx",
                 "query": "filename=python3.docx",
                 "fragment": "urllib",
             }
    2. Example 2: Set allow_fragments to false. In the returned result, the value of fragment is empty.
      • Test data
        {
           "content":"https://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib"
        }
      • Processing rule
        e_set("url",url_parse(v("content"),allow_fragments=false))
      • Processing result
        content:https://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib
        url:{ 
                       "scheme": "https",
                 "netloc": "username:username@example.com:8083", 
                "path": "/hello/asdah/",
                 "params": "type=docx",
                 "query": "filename=python3.docx", 
                "fragment": "",
             }
    3. Example 3: Set scheme to https and allow_fragments to false. In the returned result, the value of scheme is https and the value of fragment is empty.
      • Test data
        {
          "content":"//username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib"
        }
      • Processing rule
        e_set("url",url_parse(v("content"),scheme="https", allow_fragments=false))
      • Processing result
        content://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib
        url:{ 
                 "scheme": "https",
                 "netloc": "username:username@example.com:8083",
                 "path": "/hello/asdah/",
                 "params": "type=docx",
                 "query": "filename=python3.docx",
                 "fragment": "",
             }

url_parse_qs

Use the url_parse_qs function to parse the components of the query string in the URL.

  • Function format
    url_parse_qs(
        url_qs,
        keep_blank_values=false,
        strict_parsing=false,
        encoding="utf-8",
        errors="replace",
        ignore_multi_fields=true,
    )
  • Parameter description

    Parameter

    Type

    Mandatory

    Description

    url_qs

    String

    Yes

    URL query string to be parsed.

    keep_blank_values

    Boolean

    No

    Whether to return parameters whose values are empty.

    • false (default): no.
    • true: Return and process empty values as empty strings.

    strict_parsing

    Boolean

    No

    Whether to process parsing errors.

    • true: If a parsing error occurs, a ValueError exception is thrown.
    • false (default): Ignore the error.

    encoding

    String

    No

    Specifies the encoding mode. The escape characters containing percent signs (%) are parsed as Unicode characters. The default value is utf-8. ASCII is supported.

    errors

    String

    No

    Specifies the processing solution when characters cannot be identified according to the encoding mode. Options:

    • ignore: The characters are ignored.
    • strict: An error is reported and the log data is discarded.
    • replace (default): The unidentified part is replaced with the question mark (?).
    • xmlcharrefreplace: The unidentified part is replaced with the corresponding XML character.

    ignore_multi_fields

    Num

    No

    Specifies the number of values of a single return parameter.

    • true (default): Only the first value of each parameter is returned. The type is String.
    • false: All values of each parameter are returned. The type is List.
  • Returned result

    The parsed JSON data is returned. The following table describes the fields.

    Field

    Description

    logType

    Log type

    uid

    Unique ID of a log

    time

    Log time

    msg

    Log message

  • Function example
    1. Example 1: Set keep_blank_values to true. The returned result contains parameters whose values are empty.
      • Test data
        {
          "content":"logType=net_wheel_log&uid=62452****&vid=6.1.0_gf_pc&asb=1206427&git=&time=22-11-3+%e4%b8%8a11%e6%97%b649%e5%88%8633%e7%a7%92&operatingSystem=Windows+10++(10.0.0)+64bit&deviceModel=System+Product+Name+(System+manufacturer)&graphicsDeviceName=NVIDIA+GeForce+GTX+1650&graphicsDeviceType=Direct3D11&graphicsDeviceVendor=NVIDIA&graphicsDeviceVersion=Direct3D+11.0+%5blevel+11.1%5d&graphicsMemorySize=3962&systemMemorySize=8127&processorCount=6&processorFrequency=3000&processorType=Intel(R)+Core(TM)+i5-9500F+CPU+%40+3.00GHz&deviceID=96da5902a042a5f84118995f88373f73650e76be166589726****&guessUID=62452****&networkReachability=wifi&msg=GetAuthkeyRsp"
        }
      • Processing rule
        e_set("url",url_parse_qs(v("content"), keep_blank_values=true))
      • Processing result
        content:logType=net_wheel_log&uid=62452****&vid=6.1.0_gf_pc&asb=1206427&git=&time=22-11-3+%e4%b8%8a11%e6%97%b649%e5%88%8633%e7%a7%92&operatingSystem=Windows+10++(10.0.0)+64bit&deviceModel=System+Product+Name+(System+manufacturer)&graphicsDeviceName=NVIDIA+GeForce+GTX+1650&graphicsDeviceType=Direct3D11&graphicsDeviceVendor=NVIDIA&graphicsDeviceVersion=Direct3D+11.0+%5blevel+11.1%5d&graphicsMemorySize=3962&systemMemorySize=8127&processorCount=6&processorFrequency=3000&processorType=Intel(R)+Core(TM)+i5-9500F+CPU+%40+3.00GHz&deviceID=96da5902a042a5f84118995f88373f73650e76be166589726****&guessUID=62452****&networkReachability=wifi&msg=GetAuthkeyRsp
         url:{
                "logType": "net_wheel_log",
                "uid": "62452****", 
               "vid": "6.1.0_gf_pc", 
               "asb": "1206427", 
               "git": "", 
               "time": "2022-11-03 11:49:33 AM",
                "operatingSystem": "Windows 10  (10.0.0) 64bit",
                "deviceModel": "System Product Name (System manufacturer)",
                "graphicsDeviceName": "NVIDIA GeForce GTX 1650",
                "graphicsDeviceType": "Direct3D11",
                "graphicsDeviceVendor": "NVIDIA",
                "graphicsDeviceVersion": "Direct3D 11.0 [level 11.1]", 
               "graphicsMemorySize": "3962",
                "systemMemorySize": "8127", 
               "processorCount": "6",
                "processorFrequency": "3000",
                "processorType": "Intel(R) Core(TM) i5-9500F CPU @ 3.00GHz", 
               "deviceID": "96da5902a042a5f84118995f88373f73650e76be166589726****", 
               "guessUID": "62452****",
               "networkReachability": "wifi", 
               "msg": "GetAuthkeyRsp",
            }
    2. Example 2: Set keep_blank_values to the default false. The returned result does not contain parameters whose values are empty.
      • Test data
        {
           "content":"logType=net_wheel_log&uid=62452****&vid=6.1.0_gf_pc&asb=1206427&git=&time=22-11-3+%e4%b8%8a11%e6%97%b649%e5%88%8633%e7%a7%92&operatingSystem=Windows+10++(10.0.0)+64bit&deviceModel=System+Product+Name+(System+manufacturer)&graphicsDeviceName=NVIDIA+GeForce+GTX+1650&graphicsDeviceType=Direct3D11&graphicsDeviceVendor=NVIDIA&graphicsDeviceVersion=Direct3D+11.0+%5blevel+11.1%5d&graphicsMemorySize=3962&systemMemorySize=8127&processorCount=6&processorFrequency=3000&processorType=Intel(R)+Core(TM)+i5-9500F+CPU+%40+3.00GHz&deviceID=96da5902a042a5f84118995f88373f73650e76be166589726****&guessUID=62452****&networkReachability=wifi&msg=GetAuthkeyRsp"
        }
      • Processing rule
        e_set("url",url_parse_qs(v("content")))
      • Processing result
        content:logType=net_wheel_log&uid=62452****&vid=6.1.0_gf_pc&asb=1206427&git=&time=22-11-3+%e4%b8%8a11%e6%97%b649%e5%88%8633%e7%a7%92&operatingSystem=Windows+10++(10.0.0)+64bit&deviceModel=System+Product+Name+(System+manufacturer)&graphicsDeviceName=NVIDIA+GeForce+GTX+1650&graphicsDeviceType=Direct3D11&graphicsDeviceVendor=NVIDIA&graphicsDeviceVersion=Direct3D+11.0+%5blevel+11.1%5d&graphicsMemorySize=3962&systemMemorySize=8127&processorCount=6&processorFrequency=3000&processorType=Intel(R)+Core(TM)+i5-9500F+CPU+%40+3.00GHz&deviceID=96da5902a042a5f84118995f88373f73650e76be166589726****&guessUID=62452****&networkReachability=wifi&msg=GetAuthkeyRsp
         url:{
                "logType": "net_wheel_log",
                "uid": "62452****",
                "vid": "6.1.0_gf_pc",
                "asb": "1206427", 
               "time": "2022-11-03 11:49:33 AM",
                "operatingSystem": "Windows 10  (10.0.0) 64bit",
                "deviceModel": "System Product Name (System manufacturer)",
                "graphicsDeviceName": "NVIDIA GeForce GTX 1650",
                "graphicsDeviceType": "Direct3D11",
                "graphicsDeviceVendor": "NVIDIA", 
               "graphicsDeviceVersion": "Direct3D 11.0 [level 11.1]", 
               "graphicsMemorySize": "3962",
                "systemMemorySize": "8127", 
               "processorCount": "6", 
               "processorFrequency": "3000",
                "processorType": "Intel(R) Core(TM) i5-9500F CPU @ 3.00GHz",
                "deviceID": "96da5902a042a5f84118995f88373f73650e76be166589726****",
                "guessUID": "62452****",
                "networkReachability": "wifi",
                "msg": "GetAuthkeyRsp",
            }
    3. Example 3: Set ignore_multi_fields to the default true. Only the first value of each parameter is returned.
      • Test data
        {
           "content":"logType=net_log&uid=62452****&x=1&x=2&x=3&asb=123&asb=456"
        }
      • Processing rule
        e_set("url",url_parse_qs(v("content")))
      • Processing result
        content:logType=net_log&uid=62452****&x=1&x=2&x=3&asb=123&asb=456
        url:{
            "logType": "net_log", 
           "uid": "62452****",
            "x": "1"
            "asb": "123"
        }
    4. Example 4: Set ignore_multi_fields to false. All values of each parameter are returned.
      • Test data
        {
          "content":"logType=net_log&uid=62452****&x=1&x=2&x=3&asb=123&asb=456"
        }
      • Processing rule
        e_set("url",url_parse_qs(v("content"),ignore_multi_fields=false))
      • Processing result
        content:logType=net_log&uid=62452****&x=1&x=2&x=3&asb=123&asb=456
        url:{
            "logType": ["net_log"],
              "uid": ["62452****"],
              "x": ["1", "2", "3"],
              "asb": ["123", "456"],
        }