Filtering Public Network Traffic in VPC Flow Logs Using DSL Statements

Prerequisites

• VPC flow logs have been ingested to LTS. For details, see Ingesting VPC Logs to LTS.
• A log group and a log stream have been created on the LTS console to store the public network traffic logs of VPC flow logs after data processing.

Scenarios

The following is an example VPC flow log ingested to LTS:

{
  "content": "1 5f67944957444bd6bb4fe3b367de8f3d 1d515d18-1b36-47dc-a983-bd6512aed4bd 192.168.0.154 192.168.3.25 38929 53 17 1 96 1548752136 1548752736 ACCEPT OK",
  "version": 1,
  "project_id": "5f67944957444bd6bb4fe3b367de8f3d",
  "interface_id": "1d515d18-1b36-47dc-a983-bd6512aed4bd",
  "srcaddr": "192.168.0.154",
  "dstaddr": "192.168.3.25",
  "srcport": 38929,
  "dstport": 53,
  "protocol": 17,
  "packets": 1,
  "bytes": 96,
  "start": 1548752136,
  "end": 1548752736,
  "action": "ACCEPT",
  "log_status": "ok"
}

When analyzing public network traffic in VPC flow logs, you need to process the raw logs as follows:

• Discard any log that does not contain the srcaddr and dstaddr fields.
• Discard any log that records traffic between private networks.

By following these requirements, you can use the DSL processing function to effectively process the collected VPC flow logs for issue analysis.

Procedure

1. Ensure that VPS flow logs have been ingested to LTS. For details, see Ingesting VPC Logs to LTS.
2. In the navigation pane of the LTS console, choose Log Jobs > DSL Processing. On the displayed page, click Create DSL Processing Task.
3. On the Basic Information page, configure these parameters:
   • Task Name: Enter a name for the DSL processing task, for example, test-vpc.
   • Status: The task is enabled by default.
   • Source Log Stream: Select the log group and log stream where your VPC flow logs are stored. This stream must be the same one you selected when creating the VPC log ingestion configuration and should already have a structuring configuration applied.
   • Target Log Stream: Click Add and select the log group and log stream for storing logs processed by DSL. Enter a log stream alias, which will be used when writing your DSL statements.
   • Advanced Settings: In this practice, leave these parameters blank.

4. Click Next. In the processing rule area, enter the following statements:

   # Discard logs that do not contain the srcaddr and dstaddr fields.
   e_if(e_not_has("srcaddr"), e_drop())
   e_if(e_not_has("dstaddr"), e_drop())
   # Discard logs whose srcaddr or dstaddr field does not comply with the specified IP address format
   e_if(op_not(e_match("srcaddr", r'^(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$')), e_drop())
   e_if(op_not(e_match("dstaddr", r'^(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$')), e_drop())
   # Discard logs that record traffic between private networks.
   e_if(op_and(
   op_or(ip_cidrmatch("10.0.0.0/8", v("srcaddr")),
         ip_cidrmatch("172.16.0.0/12", v("srcaddr")),
         ip_cidrmatch("192.168.0.0/16", v("srcaddr"))
   ),
   op_or(ip_cidrmatch("10.0.0.0/8", v("dstaddr")),
         ip_cidrmatch("172.16.0.0/12", v("dstaddr")),
         ip_cidrmatch("192.168.0.0/16", v("dstaddr"))
   )),e_drop())

   • To discard logs that do not contain the srcaddr and dstaddr fields, combine the following functions: e_if (see Process Control Functions) and e_not_has (see Event Check Functions).
   • To discard logs whose srcaddr or dstaddr field does not comply with the specified IP address format, combine the following functions: e_if (see Process Control Functions), op_not (see Operator Functions), and e_match (see Event Check Functions).
   • To discard private network logs, combine the following functions: e_if (see Process Control Functions), op_and and op_or (see Operator Functions), and ip_cidrmatch (see IP Parsing Functions).
   • For more information on processing syntax, see DSL Data Processing Syntax (Beta).

5. Enter test data on the Test Data tab page. The following is an example for your reference:

   When using this test data, you must replace the value of srcaddr with a public IP address.

   {
     "content": "1 5f67944957444bd6bb4fe3b367de8f3d 1d515d18-1b36-47dc-a983-bd6512aed4bd 192.168.0.154 192.168.3.25 38929 53 17 1 96 1548752136 1548752736 ACCEPT OK",
     "version": 1,
     "project_id": "5f67944957444bd6bb4fe3b367de8f3d",
     "interface_id": "1d515d18-1b36-47dc-a983-bd6512aed4bd",
     "srcaddr": "192.168.0.154",
     "dstaddr": "192.168.3.25",
     "srcport": 38929,
     "dstport": 53,
     "protocol": 17,
     "packets": 1,
     "bytes": 96,
     "start": 1548752136,
     "end": 1548752736,
     "action": "ACCEPT",
     "log_status": "ok"
   }

   Click Execution Preview. You will see that only flow logs of public traffic are retained, as shown below.

   In the result preview, the ip value of srcaddr is the public IP address.

   {
   "srcaddr": "ip",
   "dstport": 53,
   "start": "1548752136",
   "dstaddr": "192.168.3.25",
   "version": 1,
   "content": "1 5f67944957444bd6bb4fe3b367de8f3d 1d515d18-1b36-47dc-a983-bd6512aed4bd 192.168.0.154 192.168.3.25 38929 53 17 1 96 1548752136 1548752736 ACCEPT OK",
   "packets": 1,
   "protocol": 17,
   "interface_id": "1d515d18-1b36-47dc-a983-bd6512aed4bd",
   "log_status": "ok",
   "project_id": "5f67944957444bd6bb4fe3b367de8f3d",
   "bytes": 96,
   "srcport": "38929",
   "action": "ACCEPT",
   "end": "1548752736"
   }

6. Click OK to save the processing task. The created task and its details will then be displayed on the DSL processing page.