Using Hubble to Monitor Cluster TCP and UDP Traffic

Prerequisites

• A CCE cluster with DataPlane V2 enabled has been created.
• Cloud Native Cluster Monitoring has been installed, with data reporting to AOM enabled.
• Hubble-related open-source components have been deployed. For details, see Deploying Hubble for DataPlane V2 Network Observability.

Updating the Cilium Configuration

To conserve cluster resources, Cilium's default traffic monitoring is node-based and does not expose workload-specific metrics. Update cilium-config to enable workload metric labels.

1. Use kubectl to access the cluster. For details, see Accessing a Cluster Using kubectl.
2. Create cilium-config.yaml.

   apiVersion: v1 
   kind: ConfigMap 
   metadata: 
     name: cilium-config 
     namespace: kube-system 
   data: 
     enable-hubble: "true" 
     hubble-disable-tls: "true" 
     hubble-listen-address: :4244 
     hubble-metrics: dns drop tcp flow:labelsContext=source_ip,source_namespace,source_workload,source_pod,destination_ip,destination_namespace,destination_workload,destination_pod,traffic_direction port-distribution icmp http 
     hubble-metrics-server: :9965

3. Apply the updated configuration to enable workload metric labels.

   kubectl apply -f cilium-config.yaml

4. Restart the Cilium DaemonSet and wait for the rollout to complete.

   kubectl rollout restart daemonset yangtse-cilium -n kube-system

Configuring Metric Monitoring in AOM

1. Access AOM to view metrics.
   1. Log in to AOM console, choose Metric Browsing in the navigation pane, and choose the Prometheus instance reported by Cloud Native Cluster Monitoring from the drop-down list on the right.
   2. Click the Interaction tab, search for Hubble traffic metrics, and filter data using labels in the Conditions area.

      

      The table below lists the traffic metrics exposed by Hubble.

      Metric	Description
      hubble_drop_total	Total number of dropped packets
      hubble_flows_processed_total	Total traffic flows processed
      hubble_icmp_total	Total number of ICMP messages
      hubble_lost_events_total	Total number of dropped network flow events
      hubble_metrics_http_handler_request_duration_seconds_bucket	Duration buckets for Hubble metrics endpoint requests
      hubble_metrics_http_handler_request_duration_seconds_count	Total HTTP requests processed
      hubble_metrics_http_handler_request_duration_seconds_sum	Total HTTP request processing duration
      hubble_metrics_http_handler_requests_total	Total number of HTTP requests received
      hubble_port_distribution_total	Total traffic distribution by destination port
      hubble_relay_pool_peer_connection_status	Current connection status and node count
      hubble_tcp_flags_total	Total number of TCP flags observed

2. Monitor TCP traffic in the cluster.

   On the Metric Browsing page of AOM, select PromQL and use the following query to monitor TCP access frequency per workload:

   sum(irate(hubble_flows_processed_total[1m])) by (destination_workload)

   This query returns the per-second TCP access rate for each workload.

   

3. Add the monitoring template to a dashboard.
   1. In the AOM navigation pane, choose Dashboard, and select an existing dashboard.
   2. Choose the Prometheus instance reported by Cloud Native Cluster Monitoring from the drop-down list, and click Add Chart.

      

   3. Enter a chart title and the PromQL query in syntax mode.

      sum(irate(hubble_flows_processed_total[1m])) by (destination_workload)

      

   4. Click Save in the upper right corner to save the chart.

4. View the monitoring chart on the AOM dashboard page.