Parsing Function
This section describes the User-Agent parsing functions, including their syntax, parameters, and usage examples.
Function List
Function |
Description |
---|---|
Parses device information in User-Agent. |
|
Parses OS information in User-Agent. |
|
Parses browser information in User-Agent. |
|
Parses all information in User-Agent. |
|
Parses the components of the URL. |
|
Parses parameters contained in the query string of the URL. |
The User-Agent parsing function deletes the fields whose parsing result is None. For example, if the parsed device data is {'brand': None, 'family': 'Other', 'model': None}, the brand and model fields are deleted. The final parsing result is {'family': 'Other'}.
ua_parse_device
Use the ua_parse_device function to parse device information in User-Agent.
- Function format
ua_parse_device(value)
- Parameter description
Parameter
Type
Mandatory
Description
value
String
Yes
User-Agent string to be parsed.
- Returned result
A JSON dataset.
- Function example
- Test data
{ "http_user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36" }
- Processing rule
e_set("new_column",ua_parse_device(v("http_user_agent")))
- Processing result
http_user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36 new_column:{"family":"Mac","brand":"Apple","model":"Mac"}
- Test data
ua_parse_os
Use the ua_parse_os function to parse OS information in User-Agent.
- Function format
ua_parse_os(value)
- Parameter description
Parameter
Type
Mandatory
Description
value
String
Yes
User-Agent string to be parsed.
- Returned result
A JSON dataset.
- Function example
- Test data
{ "http_user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36" }
- Processing rule
e_set("new_column",ua_parse_os(v("http_user_agent")))
- Processing result
http_user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36 new_column:{'family': 'Mac OS X', 'major': '10', 'minor': '9', 'patch': '4'}
- Test data
ua_parse_agent
Use the ua_parse_agent function to parse browser information in User-Agent.
- Function format
ua_parse_agent(value)
- Parameter description
Parameter
Type
Mandatory
Description
value
String
Yes
User-Agent string to be parsed.
- Returned result
A JSON dataset.
- Function example
- Test data
{ "http_user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36" }
- Processing rule
e_set("new_column",ua_parse_agent(v("http_user_agent")))
- Processing result
http_user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36 new_column:{'family': 'Chrome', 'major': '192', 'minor': '168', 'patch': '0'
- Test data
ua_parse_all
Use the ua_parse_all function to parse all information in User-Agent.
- Function format
ua_parse_all(value)
- Parameter description
Parameter
Type
Mandatory
Description
value
String
Yes
User-Agent string to be parsed.
- Returned result
A JSON dataset.
- Function example
- Test data
{ "http_user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36" }
- Processing rule
e_set("new_column",ua_parse_all(v("http_user_agent")))
- Processing result
http_user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36 new_column: { "user_agent": { "family": "Chrome", "major": "192", "minor": "168", "patch": "0" }, "os": { "family": "Mac OS X", "major": "10", "minor": "9", "patch": "4" }, "device": { "family": "Mac", "brand": "Apple", "model": "Mac" } }
- Test data
url_parse
Use the url_parse function to parse the components of the URL.
- Function format
url_parse(url, scheme="", allow_fragments=true)
- Parameter description
Parameter
Type
Mandatory
Description
value
String
Yes
URL to be parsed.
scheme
String
No
Network protocol. The default value is an empty string. The scheme field in the returned result uses the value of this parameter only when the network protocol is not specified in the URL.
allow_fragments
Boolean
No
Whether to parse the fragment part in the URL.
- true (default value): The fragment part in the URL is parsed, and the fragment field in the returned result is a specific value.
- false: The fragment part in the URL is not parsed, and the fragment field in the returned result is an empty string.
- Returned result
The parsed JSON data is returned. The following table describes the fields.
Field
Description
scheme
Network protocol
netloc
Network location
path
Hierarchical path identifier
query
Query component
fragment
Fragment identifier
- Function example
- Example 1: Use the default parameters to return the components of the URL.
- Test data
{ "content":"https://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib" }
- Processing rule
e_set("url",url_parse(v("content")))
- Processing result
content:https://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib url:{ "scheme": "https", "netloc": "username:username@example.com:8083", "path": "/hello/asdah/", "params": "type=docx", "query": "filename=python3.docx", "fragment": "urllib", }
- Test data
- Example 2: Set allow_fragments to false. In the returned result, the value of fragment is empty.
- Test data
{ "content":"https://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib" }
- Processing rule
e_set("url",url_parse(v("content"),allow_fragments=false))
- Processing result
content:https://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib url:{ "scheme": "https", "netloc": "username:username@example.com:8083", "path": "/hello/asdah/", "params": "type=docx", "query": "filename=python3.docx", "fragment": "", }
- Test data
- Example 3: Set scheme to https and allow_fragments to false. In the returned result, the value of scheme is https and the value of fragment is empty.
- Test data
{ "content":"//username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib" }
- Processing rule
e_set("url",url_parse(v("content"),scheme="https", allow_fragments=false))
- Processing result
content://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib url:{ "scheme": "https", "netloc": "username:username@example.com:8083", "path": "/hello/asdah/", "params": "type=docx", "query": "filename=python3.docx", "fragment": "", }
- Test data
- Example 1: Use the default parameters to return the components of the URL.
url_parse_qs
Use the url_parse_qs function to parse the components of the query string in the URL.
- Function format
url_parse_qs( url_qs, keep_blank_values=false, strict_parsing=false, encoding="utf-8", errors="replace", ignore_multi_fields=true, )
- Parameter description
Parameter
Type
Mandatory
Description
url_qs
String
Yes
URL query string to be parsed.
keep_blank_values
Boolean
No
Whether to return parameters whose values are empty.
- false (default): no.
- true: Return and process empty values as empty strings.
strict_parsing
Boolean
No
Whether to process parsing errors.
- true: If a parsing error occurs, a ValueError exception is thrown.
- false (default): Ignore the error.
encoding
String
No
Specifies the encoding mode. The escape characters containing percent signs (%) are parsed as Unicode characters. The default value is utf-8. ASCII is supported.
errors
String
No
Specifies the processing solution when characters cannot be identified according to the encoding mode. Options:
- ignore: The characters are ignored.
- strict: An error is reported and the log data is discarded.
- replace (default): The unidentified part is replaced with the question mark (?).
- xmlcharrefreplace: The unidentified part is replaced with the corresponding XML character.
ignore_multi_fields
Num
No
Specifies the number of values of a single return parameter.
- true (default): Only the first value of each parameter is returned. The type is String.
- false: All values of each parameter are returned. The type is List.
- Returned result
The parsed JSON data is returned. The following table describes the fields.
Field
Description
logType
Log type
uid
Unique ID of a log
time
Log time
msg
Log message
- Function example
- Example 1: Set keep_blank_values to true. The returned result contains parameters whose values are empty.
- Test data
{ "content":"logType=net_wheel_log&uid=62452****&vid=6.1.0_gf_pc&asb=1206427&git=&time=22-11-3+%e4%b8%8a11%e6%97%b649%e5%88%8633%e7%a7%92&operatingSystem=Windows+10++(10.0.0)+64bit&deviceModel=System+Product+Name+(System+manufacturer)&graphicsDeviceName=NVIDIA+GeForce+GTX+1650&graphicsDeviceType=Direct3D11&graphicsDeviceVendor=NVIDIA&graphicsDeviceVersion=Direct3D+11.0+%5blevel+11.1%5d&graphicsMemorySize=3962&systemMemorySize=8127&processorCount=6&processorFrequency=3000&processorType=Intel(R)+Core(TM)+i5-9500F+CPU+%40+3.00GHz&deviceID=96da5902a042a5f84118995f88373f73650e76be166589726****&guessUID=62452****&networkReachability=wifi&msg=GetAuthkeyRsp" }
- Processing rule
e_set("url",url_parse_qs(v("content"), keep_blank_values=true))
- Processing result
content:logType=net_wheel_log&uid=62452****&vid=6.1.0_gf_pc&asb=1206427&git=&time=22-11-3+%e4%b8%8a11%e6%97%b649%e5%88%8633%e7%a7%92&operatingSystem=Windows+10++(10.0.0)+64bit&deviceModel=System+Product+Name+(System+manufacturer)&graphicsDeviceName=NVIDIA+GeForce+GTX+1650&graphicsDeviceType=Direct3D11&graphicsDeviceVendor=NVIDIA&graphicsDeviceVersion=Direct3D+11.0+%5blevel+11.1%5d&graphicsMemorySize=3962&systemMemorySize=8127&processorCount=6&processorFrequency=3000&processorType=Intel(R)+Core(TM)+i5-9500F+CPU+%40+3.00GHz&deviceID=96da5902a042a5f84118995f88373f73650e76be166589726****&guessUID=62452****&networkReachability=wifi&msg=GetAuthkeyRsp url:{ "logType": "net_wheel_log", "uid": "62452****", "vid": "6.1.0_gf_pc", "asb": "1206427", "git": "", "time": "2022-11-03 11:49:33 AM", "operatingSystem": "Windows 10 (10.0.0) 64bit", "deviceModel": "System Product Name (System manufacturer)", "graphicsDeviceName": "NVIDIA GeForce GTX 1650", "graphicsDeviceType": "Direct3D11", "graphicsDeviceVendor": "NVIDIA", "graphicsDeviceVersion": "Direct3D 11.0 [level 11.1]", "graphicsMemorySize": "3962", "systemMemorySize": "8127", "processorCount": "6", "processorFrequency": "3000", "processorType": "Intel(R) Core(TM) i5-9500F CPU @ 3.00GHz", "deviceID": "96da5902a042a5f84118995f88373f73650e76be166589726****", "guessUID": "62452****", "networkReachability": "wifi", "msg": "GetAuthkeyRsp", }
- Test data
- Example 2: Set keep_blank_values to the default false. The returned result does not contain parameters whose values are empty.
- Test data
{ "content":"logType=net_wheel_log&uid=62452****&vid=6.1.0_gf_pc&asb=1206427&git=&time=22-11-3+%e4%b8%8a11%e6%97%b649%e5%88%8633%e7%a7%92&operatingSystem=Windows+10++(10.0.0)+64bit&deviceModel=System+Product+Name+(System+manufacturer)&graphicsDeviceName=NVIDIA+GeForce+GTX+1650&graphicsDeviceType=Direct3D11&graphicsDeviceVendor=NVIDIA&graphicsDeviceVersion=Direct3D+11.0+%5blevel+11.1%5d&graphicsMemorySize=3962&systemMemorySize=8127&processorCount=6&processorFrequency=3000&processorType=Intel(R)+Core(TM)+i5-9500F+CPU+%40+3.00GHz&deviceID=96da5902a042a5f84118995f88373f73650e76be166589726****&guessUID=62452****&networkReachability=wifi&msg=GetAuthkeyRsp" }
- Processing rule
e_set("url",url_parse_qs(v("content")))
- Processing result
content:logType=net_wheel_log&uid=62452****&vid=6.1.0_gf_pc&asb=1206427&git=&time=22-11-3+%e4%b8%8a11%e6%97%b649%e5%88%8633%e7%a7%92&operatingSystem=Windows+10++(10.0.0)+64bit&deviceModel=System+Product+Name+(System+manufacturer)&graphicsDeviceName=NVIDIA+GeForce+GTX+1650&graphicsDeviceType=Direct3D11&graphicsDeviceVendor=NVIDIA&graphicsDeviceVersion=Direct3D+11.0+%5blevel+11.1%5d&graphicsMemorySize=3962&systemMemorySize=8127&processorCount=6&processorFrequency=3000&processorType=Intel(R)+Core(TM)+i5-9500F+CPU+%40+3.00GHz&deviceID=96da5902a042a5f84118995f88373f73650e76be166589726****&guessUID=62452****&networkReachability=wifi&msg=GetAuthkeyRsp url:{ "logType": "net_wheel_log", "uid": "62452****", "vid": "6.1.0_gf_pc", "asb": "1206427", "time": "2022-11-03 11:49:33 AM", "operatingSystem": "Windows 10 (10.0.0) 64bit", "deviceModel": "System Product Name (System manufacturer)", "graphicsDeviceName": "NVIDIA GeForce GTX 1650", "graphicsDeviceType": "Direct3D11", "graphicsDeviceVendor": "NVIDIA", "graphicsDeviceVersion": "Direct3D 11.0 [level 11.1]", "graphicsMemorySize": "3962", "systemMemorySize": "8127", "processorCount": "6", "processorFrequency": "3000", "processorType": "Intel(R) Core(TM) i5-9500F CPU @ 3.00GHz", "deviceID": "96da5902a042a5f84118995f88373f73650e76be166589726****", "guessUID": "62452****", "networkReachability": "wifi", "msg": "GetAuthkeyRsp", }
- Test data
- Example 3: Set ignore_multi_fields to the default true. Only the first value of each parameter is returned.
- Test data
{ "content":"logType=net_log&uid=62452****&x=1&x=2&x=3&asb=123&asb=456" }
- Processing rule
e_set("url",url_parse_qs(v("content")))
- Processing result
content:logType=net_log&uid=62452****&x=1&x=2&x=3&asb=123&asb=456 url:{ "logType": "net_log", "uid": "62452****", "x": "1" "asb": "123" }
- Test data
- Example 4: Set ignore_multi_fields to false. All values of each parameter are returned.
- Test data
{ "content":"logType=net_log&uid=62452****&x=1&x=2&x=3&asb=123&asb=456" }
- Processing rule
e_set("url",url_parse_qs(v("content"),ignore_multi_fields=false))
- Processing result
content:logType=net_log&uid=62452****&x=1&x=2&x=3&asb=123&asb=456 url:{ "logType": ["net_log"], "uid": ["62452****"], "x": ["1", "2", "3"], "asb": ["123", "456"], }
- Test data
- Example 1: Set keep_blank_values to true. The returned result contains parameters whose values are empty.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot