Help Center/ Log Tank Service/ Best Practices/ Log Jobs (Beta)/ Parsing Text/ Processing Complex JSON Data
Updated on 2026-05-12 GMT+08:00
View PDF
Share

Processing Complex JSON Data

This section describes how to use LTS's data processing feature to handle complex JSON data.

Processing Complex JSON Data with Multiple Sub-keys as Arrays

Logs generated by applications are written in a statistical JSON format. Generally, these logs contain basic information and multiple sub-keys structured as arrays. For example, a server writes a log every minute, including its current information status and the statistics of its related servers and client nodes.

  • Example log 
    {
    "content":{
     "service": "search_service",
     "overall_status": "yellow",
     "servers": [
         {
             "host": "192.0.2.1",
             "status": "green"
         },
         {
             "host": "192.0.2.2",
             "status": "green"
         }
     ],
     "clients": [
         {
             "host": "192.0.2.3",
             "status": "green"
         },
         {
             "host": "192.0.2.4",
             "status": "red"
         }
     ]
}
}
  • Processing requirements
    1. Split the raw log into three topics: overall_type, client_status, and server_status.
    2. Retain different information for each topic:
      • overall_type: retains the count of servers and clients, overall_status color, and service information.
      • client_status: retains the host address, status, and service information.
      • server_status: retains the host address, status, and service information.
  • Solution: The following describes how to use the processing syntax. The syntax in 1 to 7 must be used together.
    1. Split the log into three distinct entries by assigning them different topic values. Following the split, you will have three logs containing identical information, differentiated only by their respective topic fields. 
      e_set("topic", "server_status,client_status,overall_type")
e_split("topic")

      The log format after processing:

      topic: server_status         // The other two logs are marked as client_status and overall_type respectively; other parameters remain identical.
content:  {
    ...Same as above...
}
    2. Expand the JSON content. The content field is expanded at the first layer, and the original content field is deleted. 
      e_json('content',depth=1)
e_drop_fields("content")

      Log format after processing:

      topic: overall_type              // The other two logs are marked as client_status and server_status respectively; other parameters remain identical.
clients:  [{"host": "192.0.2.3", "status": "green"}, {"host": "192.0.2.4", "status": "red"}]
overall_status:  yellow
servers:  [{"host": "192.0.2.1", "status": "green"}, {"host": "192.0.2.2", "status": "green"}]
service:  search_service
    3. For the log whose topic is overall_type, collect statistics for client_count and server_count. 
      e_if(e_search("topic==overall_type"), 
     e_compose(
        e_set("client_count", json_select(v("clients"), "length([*])", default=0)), 
        e_set("server_count", json_select(v("servers"), "length([*])", default=0))
  ))

      Log after processing:

      topic:  overall_type
server_count:  2
client_count:  2
    4. Discard fields: 
      e_if(e_search("topic==overall_type"), e_drop_fields("clients", "servers"))
    5. Further split the log whose topic is server_status. 
      e_if(e_search("topic==server_status"), e_split("servers"))
e_if(e_search("topic==server_status"), e_json("servers", depth=1))

      Split log 1 after further splitting:

      topic:  server_status
servers:  {"host": "192.0.2.1", "status": "green"}
host: 192.0.2.1
status: green

      Split log 2 after further splitting:

      topic:  server_status
servers:  {"host": "192.0.2.2", "status": "green"}
host: 192.0.2.2
status: green
    6. Retain relevant fields: 
      e_if(e_search("topic==server_status"), e_compose(e_drop_fields("servers"),e_drop_fields("clients")))
    7. Further split the log whose topic is client_status and delete unnecessary fields. 
      e_if(e_search("topic==client_status"), e_split("clients"))
e_if(e_search("topic==client_status"), e_json("clients", depth=1))

      Split log 1 after further splitting:

      topic:  client_status
host: 192.0.2.3
status: green

      Split log 2 after further splitting:

      topic:  clients
host: 192.0.2.4
status: red
    8. The preceding syntax is combined as follows: 
      # Overall splitting
e_set("topic", "server_status,client_status,overall_type")
e_split("topic")
e_json('content',depth=1)
e_drop_fields("content")
# Process the overall_type log.
e_if(e_search("topic==overall_type"), 
     e_compose(
        e_set("client_count", json_select(v("clients"), "length([*])", default=0)), 
        e_set("server_count", json_select(v("servers"), "length([*])", default=0))
  ))
e_if(e_search("topic==overall_type"), e_drop_fields("clients", "servers"))
# Process the server_status log.
e_if(e_search("topic==server_status"), e_split("servers"))
e_if(e_search("topic==server_status"), e_json("servers", depth=1))
e_if(e_search("topic==server_status"), e_compose(e_drop_fields("servers"),e_drop_fields("clients")))
# Process the client_status log.
e_if(e_search("topic==client_status"), e_split("clients"))
e_if(e_search("topic==client_status"), e_json("clients", depth=1))
e_if(e_search("topic==client_status"), e_compose(e_drop_fields("servers"),e_drop_fields("clients")))

      Processing result:

      {
    "content":{
     "service": "search_service",
     "overall_status": "yellow",
     "servers": [
         {
             "host": "192.0.2.1",
             "status": "green"
         },
         {
             "host": "192.0.2.2",
             "status": "green"
         }
     ],
     "clients": [
         {
             "host": "192.0.2.3",
             "status": "green"
         },
         {
             "host": "192.0.2.4",
             "status": "red"
         }
     ]
}
}

Processing Complex JSON Data with Multi-Layer Array Object Nesting

Take a complex JSON object with multi-layer nested arrays as an example. Split each individual login record in login_histories of each object under users into a separate login event.

  • Raw log 
    {
"content":{
  "users": [
    {
        "name": "user1",
        "login_histories": [
          {
            "date": "2019-10-10 0:0:0",
            "login_ip": "192.0.2.6"
          },
          {
            "date": "2019-10-10 1:0:0",
            "login_ip": "192.0.2.6"
          },
      {
      ...More login information...
      }
       ]
    },
    {
        "name": "user2",
        "login_histories": [
          {
            "date": "2019-10-11 0:0:0",
            "login_ip": "192.0.2.7"
          },
          {
            "date": "2019-10-11 1:0:0",
            "login_ip": "192.0.2.9"
          },
      {
      ...More login information...
      }     
        ]
    },
  {
    ...More users...
  }
  ]
}
}
  • Expected split logs 
    name:  user1
date:  2019-10-11 1:0:0
login_ip:  192.0.2.6

name:  user1
date:  2019-10-11 0:0:0
login_ip:  192.0.2.6

name:  user2
date:  2019-10-11 0:0:0
login_ip:  192.0.2.7

name:  user2
date:  2019-10-11 1:0:0
login_ip:  192.0.2.9  

...More logs...
  • Solution
    1. Split and expand the users array in the content field. 
      e_split("content", jmes='users[*]', output='item')
e_json("item",depth=1)
      Logs returned after processing: 
      content:{...Same as above...}
item:  {"name": "user1", "login_histories": [{"date": "2019-10-10 0:0:0", "login_ip": "192.0.2.6"}, {"date": "2019-10-10 1:0:0", "login_ip": "192.0.2.6"}]}
login_histories:  [{"date": "2019-10-10 0:0:0", "login_ip": "192.0.2.6"}, {"date": "2019-10-10 1:0:0", "login_ip": "192.0.2.6"}]
name:  user1

content:{...Same as above...}
item:  {"name": "user2", "login_histories": [{"date": "2019-10-11 0:0:0", "login_ip": "192.0.2.7"}, {"date": "2019-10-11 1:0:0", "login_ip": "192.0.2.9"}]}
login_histories:  [{"date": "2019-10-11 0:0:0", "login_ip": "192.0.2.7"}, {"date": "2019-10-11 1:0:0", "login_ip": "192.0.2.9"}]
name:  user2
    2. Split and expand login_histories. 
      e_split("login_histories")
e_json("login_histories", depth=1)

      Logs returned after processing:

      content: {...Same as above...}
date:  2019-10-11 0:0:0
item:  {"name": "user2", "login_histories": [{"date": "2019-10-11 0:0:0", "login_ip": "192.0.2.7"}, {"date": "2019-10-11 1:0:0", "login_ip": "192.0.2.9"}]}
login_histories:  {"date": "2019-10-11 0:0:0", "login_ip": "192.0.2.7"}
login_ip:  192.0.2.7
name:  user2

content: {...Same as above...}
date:  2019-10-11 1:0:0
item:  {"name": "user2", "login_histories": [{"date": "2019-10-11 0:0:0", "login_ip": "192.0.2.7"}, {"date": "2019-10-11 1:0:0", "login_ip": "192.0.2.9"}]}
login_histories:  {"date": "2019-10-11 1:0:0", "login_ip": "192.0.2.9"}
login_ip:  192.0.2.9
name:  user2

content: {...Same as above...}
date:  2019-10-10 1:0:0
item:  {"name": "user1", "login_histories": [{"date": "2019-10-10 0:0:0", "login_ip": "192.0.2.6"}, {"date": "2019-10-10 1:0:0", "login_ip": "192.0.2.6"}]}
login_histories:  {"date": "2019-10-10 1:0:0", "login_ip": "192.0.2.6"}
login_ip:  192.0.2.6
name:  user1

content: {...Same as above...}
date:  2019-10-10 0:0:0
item:  {"name": "user1", "login_histories": [{"date": "2019-10-10 0:0:0", "login_ip": "192.0.2.6"}, {"date": "2019-10-10 1:0:0", "login_ip": "192.0.2.6"}]}
login_histories:  {"date": "2019-10-10 0:0:0", "login_ip": "192.0.2.6"}
login_ip:  192.0.2.6
name:  user1
    3. Delete irrelevant fields. 
      e_drop_fields("content", "item", "login_histories")

      Logs returned after processing:

      {
	"date": "2019-10-10 0:0:0",
	"name": "user1",
	"login_ip": "192.0.2.6"
}
{
	"date": "2019-10-10 1:0:0",
	"name": "user1",
	"login_ip": "192.0.2.6"
}
{
	"date": "2019-10-11 0:0:0",
	"name": "user2",
	"login_ip": "192.0.2.7"
}
{
	"date": "2019-10-11 1:0:0",
	"name": "user2",
	"login_ip": "192.0.2.9"
}
    4. The preceding DSL rules are combined as follows: 
      e_split("content", jmes='users[*]', output='item')
e_json("item",depth=1)
e_split("login_histories")
e_json("login_histories", depth=1)
e_drop_fields("content", "item", "login_histories")

      Summary: To meet the preceding requirements, the rules sequentially perform the splitting and expanding operations, and delete irrelevant information.

Parent topic: Parsing Text