Monitoring and Alarm Configuration for Available IP Addresses of Subnets Associated with Clusters

Solution Principle

Figure 1 Principle of monitoring available subnet IP addresses

You can:

1. Enable VPC subnet instance monitoring and use Cloud Eye to collect metrics and report them to Prometheus instance A in AOM.
2. Install the Cloud Native Cluster Monitoring add-on in the CCE cluster, connect the add-on to Prometheus instance B in AOM, and enable subnet mapping for node pools or pods.
3. Add Prometheus instance C in AOM to aggregate the metrics of Prometheus instances A and B that are connected to VPC and CCE, respectively.
4. Use PromQL statements to query the metrics of instance C, monitor available subnet IP addresses, and configure alarms.

Prerequisites

• The cluster version must be v1.27.16-r50, v1.28.15-r40, v1.29.15-r0, v1.30.14-r0, v1.31.10-r0, v1.32.6-r0, v1.33.1-r0, or later.
• The Cloud Native Cluster Monitoring add-on (Cloud Native Cluster Monitoring) has been installed in the cluster and upgraded to 3.12.2 or later.

Notes and Constraints

• Only IPv4 subnet IP addresses can be monitored.
• When dynamic network interface pre-binding is configured for a CCE Turbo cluster, the pre-bound network interfaces also occupy subnet IP addresses.

Enabling VPC to Report Metrics

VPC reports the subnet IP address usage every 30 minutes.

1. Log in to the VPC console, click the VPC name, and enable Monitor Subnet IP Address Usage.

   

2. Check whether the subnet metrics are generated in Cloud Eye.
   1. Log in to the Cloud Eye console and choose Cloud Service Monitoring in the navigation pane.
   2. Click the Virtual Private Cloud VPC dashboard and select Subnets from the Bandwidths drop-down list on the top of the page.
   3. Click the subnet name to view the subnet metrics.

      

3. Connect VPC metrics to AOM.
   1. Log in to the AOM console.
   2. In the navigation pane, choose Access Center. Then, select VPC.

      

   3. Select an existing instance or create an instance. Then click Next.

      

   4. Enter an ingestion rule name and click OK. After the ingestion is successful, you can choose Access Management > Metric Ingestion Rules to check that the rule is enabled.

      

Enabling CCE to Collect Metrics

The mappings between nodes and pod subnets in CCE can be collected using the Cloud Native Cluster Monitoring add-on.

CCE reports node and pod subnet IP address usages every 10 minutes.

1. Log in to the CCE console and click the cluster name to access the cluster console.
2. In the navigation pane, choose Add-ons. On the displayed page, locate Cloud Native Cluster Monitoring.
3. Check that this add-on is running normally and enable Report Monitoring Data to AOM.

   

4. In the navigation pane, choose Settings. On the Monitoring tab, enable Subnet Mapping for Node Pools or Pods.

   

5. Click Confirm Settings.

Verifying that the Metrics Generated by VPC and CCE Are Collected

1. Log in to the AOM console and choose Prometheus Monitoring > Instances in the navigation pane.
2. Locate the Prometheus instance connected to the VPC in Enabling VPC to Report Metrics, click the instance name, and check the instance metrics.
   Check that the metrics listed in the following table are enabled.

   Metric	Description
   subnet_ipv4_availability_used_number	Number of used IPv4 addresses in a subnet
   subnet_ipv4_availability_remain_number	Number of available IPv4 addresses in a subnet
   subnet_ipv4_availability_total_number	Total number of IPv4 addresses in a subnet
   subnet_ipv4_availability_usage_percentage	The percentage of used IPv4 addresses

   

3. Locate the Prometheus instance connected to the Cloud Native Cluster Monitoring add-on in Enabling CCE to Collect Metrics, click the instance name, and check the instance metrics.
   Check that the metrics listed in the following table are enabled.

   Metric	Description
   subnet_relation_node	Relationship between a node pool in CCE and a VPC subnet. The available subnet IP addresses reported by VPC can be associated with the node pool, so that the available subnet IP addresses of a node pool can be queried.
   subnet_relation_nad	Relationship between a container network configuration (specified by NetworkAttachmentDefinition) in a CCE cluster and a subnet. The available subnet IP addresses reported by VPC can be associated with the container network configuration, so that the available subnet IP addresses of a container network configuration can be queried.

   

Aggregating VPC and CCE Metrics and Querying Available Subnet IP Addresses

1. Log in to the AOM console.
2. In the navigation pane, choose Instances. On the displayed page, click Add Prometheus Instance.
3. Enter the instance name and select Prometheus for multi-account aggregation or Prometheus for multi-account virtual aggregation for Instance Type.

   

4. Set Aggregation Mode to Current account's Prometheus instances, select the AOM instances connected to CCE and VPC, and click OK.

   

5. Query the following metrics in the new instance.
   

   VPC metrics are reported every 30 minutes. You are advised to query data within this 30-minute window. Shorter intervals may lead to inaccurate results.

   • PromQL for querying the number of available subnet IP addresses in a node pool:

     sum by (cluster_name, cluster_id, nodepool_id) ( max_over_time(huaweicloud_sys_vpc_subnet_ipv4_availability_remain_number[30m])   * on(subnet_id) group_left()   subnet_relation_node{nodepool_id="DefaultPool"})

   • PromQL for querying the number of available IP addresses of a pod subnet:

     sum by (cluster_name, cluster_id, network_attachment_definition_name)( max_over_time(huaweicloud_sys_vpc_subnet_ipv4_availability_remain_number[30m])* on(subnet_id) group_left()   subnet_relation_nad{network_attachment_definition_name="default-network"})

Configuring Alarms for Available Subnet IP Addresses

You can configure metric alarms in AOM. This part provides PromQL examples for configuring alarms for available subnet IP addresses in a node pool and available IP addresses of a pod subnet. For details about how to configure more alarm rules, see Creating an AOM Metric Alarm Rule.

1. Log in to the AOM console.
2. In the navigation pane, choose Alarm Center > Alarm Rules. Then, click Create Alarm Rule.
3. Enter basic information such as the alarm name and configure the alarm rule details.
   • Rule Type: Select Metric alarm rule.
   • Configuration Mode: Select PromQL.
   • Alarm Rule Details: Select Custom for Default Rule and enter the PromQL statement as required.
     • Example PromQL for configuring an alarm for available subnet IP addresses of nodes in a node pool

       An alarm is triggered if a node pool has fewer than 20 available subnet IP addresses.

       sum by (cluster_name, cluster_id, nodepool_id)(max_over_time(huaweicloud_sys_vpc_subnet_ipv4_availability_remain_number[30m])* on(subnet_id) group_left() subnet_relation_node{nodepool_id="973ab8c6-78f9-11f0-8729-0255ac100b0a"})<20

     • Example PromQL for configuring an alarm for available IP addresses of a pod subnet

       An alarm is generated when the default container subnet has fewer than 20 available IP addresses.

       (sum by (cluster_name, cluster_id, network_attachment_definition_name)( max_over_time(huaweicloud_sys_vpc_subnet_ipv4_availability_remain_number[30m])* on(subnet_id) group_left()   subnet_relation_nad{network_attachment_definition_name="default-network"})) < 20

     

4. Click OK.