Building an Intelligent O&M Agent for the CCE Production Environment Based on Hermes and Lark

Scenarios

In a production environment, a CCE cluster may continuously generate a large number of alarms, covering multiple dimensions such as workloads, pods, nodes, networks, storage, auto scaling, resource capacity, and availability risks. The larger the live network scale, the more complex the numbers of alarms, alarm sources, and handling paths. On-duty personnel need to frequently switch between the Lark alarms, AOM metrics, Kubernetes events, pod logs, workload configurations, HPA statuses, node capacity, and service ticket system. This can lead to issues such as alarm fatigue, slow response, incomplete evidence, lack of review for recovery actions, and difficulty in accumulating review materials.

By building an intelligent O&M Agent for the CCE production environment, you can consolidate alarm detection, alarm merging, context collection, root cause analysis, recovery preview, user confirmation, recovery execution, effect verification, and result archiving into a set of reusable ChatOps on-duty capabilities. The Agent can use Hermes or your existing ChatOps, AIOps, service ticket assistant, or self-developed on-duty robot. Huawei Cloud cloud-native Skills provide standard query, analysis, and controlled recovery capabilities for resources such as CCE, AOM, LTS, node pools, and workloads.

Solution Architecture

This solution uses the "Agent Runtime + cloud-native Skills + Lark confirmation" architecture. Agent Runtime is responsible for task scheduling, alarm distribution, context orchestration, and Lark interaction. Cloud-native Skills provide capabilities such as alarms, metrics, logs, events, root cause analysis, and recovery actions. Lark carries alarm notification, user review, and closed-loop results.

You are advised to split Agent permissions based on the capability boundary.

Constraints

• Use the recommended configurations of cloud services for alarm rules and metric specifications, and perform cross-verification based on real-time metrics.
• The preview and confirmation mechanisms must be retained for recovery actions, especially production operations such as scaling, rollback, and node pool changes.
• Lark messages should be readable to on-duty personnel. The conclusion, evidence, affected objects, optional solutions, and confirmation entry should be provided first.
• For complex recovery links, you are advised to make alarm fingerprints, solution IDs, target resources, and execution records persistent for confirmation and audit.
• Do not expose sensitive information such as AKs/SKs, tokens, certificates, and project IDs in prompts, Lark messages, screenshots, or documents.
• In the pilot phase, you are advised to start with read-only inspection and manual confirmation for recovery, and then gradually expand to more automatic recovery policies.

Prerequisites

Before performing this practice, you are advised to prepare the Agent running environment and CCE observability objects, and then gradually expand the automation scope.

• You have prepared the CCE clusters, namespaces, or service scope for inspection and diagnosis.
• AOM alarms, cloud native monitoring metrics, or existing inspection objects have been connected.
• You have prepared Hermes or your own Agent Runtime and connected it to Lark or the service ticket system.
• You have imported Huawei Cloud cloud-native Skills related to CCE to the Agent for querying alarms, metrics, events, workloads, pods, and nodes, and performing controlled recovery.
• You have prepared access credentials and have configured them securely to ensure that sensitive information is not exposed in prompts or documents.

Orchestratable Capabilities

You can combine the following Skills as needed to build an intelligent O&M process for countless alarms.

For live-network alarm governance, Agent capabilities can be classified into the following levels.

Procedure

This practice is based on the high CPU usage alarm of the default/chat-app workload in the demo-recovery cluster. It aims to verify the end-to-end capabilities of the intelligent O&M Agent in the CCE production environment. The high CPU usage alarm is just one type of alarm on the live network. The process focuses on demonstrating the complete link from normal inspection, alarm detection, automatic analysis, user confirmation, controlled recovery, to closed-loop verification.

Hermes Task Prompt Reference

The following prompts can be used as a task template for the Hermes ChatOps on-duty Agent. It downplays specific commands and environment variables, retaining only the role, process, output structure, and security boundaries.

You are the intelligent O&M Agent of the CCE production environment. You are responsible for performing alarm inspection, alarm merging, automatic analysis, recovery preview, recovery after user confirmation, recovery verification, and Lark closed-loop notification in the target CCE environment.

Prerequisites:
- CCE-related cloud-native Skills have been imported to the current Agent in advance.
- The target cluster, inspection scope, notification channel, and access credentials have been provided by the runtime environment.
- All notifications are sent to Lark or the on-duty channel specified by the customer.

Purposes:
1. Periodically scan active alarms and recent historical alarms in the target CCE environment.
2. Deduplicate, merge, classify, route, and summarize the impact scope of alarms.
3. When the inspection is normal, output a concise health summary and do not exit silently.
4. When alarms need to be handled, automatically aggregate the context, including AOM alarms, real-time metrics, Kubernetes events, pod/workload/node statuses, log summaries, and recent changes.
5. Generate a diagnosis report for on-duty personnel, including the alarm summary, affected objects, key evidence, possible causes, recommended solutions, and actions to be confirmed.
6. For actions involving resource changes, only a recovery preview is generated, and no action is directly performed on the live network.
7. The recovery action is executed only after the user clearly replies with a confirmation statement such as "Confirm execution" or confirms a specific solution in Lark.
8. After the execution, verify the alarm statuses, pod statuses, workload replicas, node capacity, and key metrics, and send the closed-loop result to Lark.

Recommended structure of the inspection report:
- Inspection summary: cluster, time window, number of active alarms, and key resource statuses
- Alarm discovery: alarm name, severity, status, affected objects, and current observed value
- System analysis: alarm statuses, real-time metrics, pod/workload/node statuses, related events, and recent changes
- Recovery solution: Provide two to three optional solutions, including the scenarios, impact scope, rollback method, and verification method.
- User confirmation: Prompt the user to reply "Confirm" or select a specific solution.

Security boundaries:
- Only read-only operations are allowed during alarm scanning, evidence collection, and root cause analysis.
- Recovery cannot be performed based on a single alarm.
- For all write operations, the recovery preview, impact scope, rollback method, and verification method must be provided first.
- The solution number and meaning in the same alarm must be consistent. After the user confirms the solution, the solution is executed.
- Before user confirmation, do not perform any operations that change the live network status, such as scaling, rollback, restart, or node pool change.
- Do not expose sensitive information such as AKs/SKs, tokens, certificates, and project IDs in the output.
- If the evidence is insufficient, list possible causes and information that requires manual review. Do not make judgments for the user without sufficient evidence.

Lark output requirements:
- When the inspection is normal, output a concise health summary.
- When an alarm is detected, output the alarm summary and affected objects first, and then output the key evidence and candidate solutions.
- If recovery is required, clearly prompt the user to reply "Confirm" or select a specific solution. Before receiving manual confirmation, wait for confirmation and do not perform the change.
- After the recovery is complete, output the execution actions, verification results, remaining risks, and subsequent suggestions.

Diagnosis Results

The focus of this case is not that capacity expansion is necessary when the CPU usage is high. Instead, it demonstrates a transferable method: Alarm detection by the Agent → Evidence aggregation by the Skill → Manual confirmation and recovery → System execution and verification. You can replace any of the phases with your own tools, approval processes, and business rules.

Extended Scenarios

You can extend this practice from multiple dimensions based on your requirements and existing tools to adapt to different O&M scenarios and service requirements. The following provides suggestions and examples for extending this practice from different dimensions.

The following table lists the typical extension scenarios.

Expected Results

After completing this practice, you can obtain the following benefits:

1. After a CCE alarm is reported to Lark, the Agent automatically starts the diagnosis link.
2. O&M personnel can view the alarm summary, evidence, and candidate solutions without repeatedly switching between multiple systems.
3. Recovery actions are confirmed on Lark before execution, reducing misoperations.
4. After the recovery is executed, the alarm convergence, pod status, node capacity, and metric trends are automatically verified.
5. The alarm handling process can be archived, audited, and reviewed.
6. The same Agent + Skill orchestration approach can be extended to more CCE O&M scenarios.