Creating and Managing Drill Tasks

Scenarios

Drill tasks allow you to simulate software or hardware faults to test the system's fault recovery capability. Drill task operations include managing chaos drill tasks, viewing drill records, and creating drill tasks. Setting a drill task include setting the basic information, adding an attack task group, selecting an attack task, and selecting an attack scenario. In addition, a drill task involves monitoring task configuration and post-drill review and improvement. This ensures that an excellent optimization policy can be applied when the system is under various pressures.

Automatic Task Termination Mechanism

• Automatic termination upon timeout: If a drill task fails and you do not manually close the task within 48 hours, the system automatically terminates the drill task.
• Automatic termination upon exceptions: During the drill, if a pod exception (for example, the pod has been deleted) is detected or a resource O&M ticket is manually closed, the system automatically terminates the current task immediately.

Creating a Drill Task

1. Log in to COC.
2. In the navigation tree on the left, choose Resilience Center > Chaos Drills.
3. Click the Drill Tasks tab.
4. Click Create Task.

   You can also use the drill plan ticket accepting function to access the page for creating a drill task. For details, see Creating and Managing Drill Plans.

5. Configure the basic information.

   Table 1 Parameters in the basic information

   Parameter	Description	Example Value
   Drill Task	Name of the drill task. Set it according to the naming rules.	test-drill
   Expected Recovery Duration (Minutes)	Expected time from the fault occurrence to the fault recovery, in minutes. Expected time for an application to automatically recover to the normal state during emergency plan execution after a fault is injected. This time does not affect the drill task.	3

6. Click Add Attack Task.
   By default, there is one attack task group. You can click Add Task Group to add a task group. After adding an attack task, you can click Add Attack Task to add another attack task.

   

   • Tasks in different task groups are executed in serial mode, and tasks in the same task group are executed in parallel mode.
   • Currently, multiple fault injection operations on the same resource in a task group are not supported.
   1. Set parameters for adding an attack task.
      • To add an existing task, click Select from Existing, select the existing task, and click OK.
      • To add a new attack task, perform the following steps.

        Table 2 Parameters for adding an attack task

        Parameter	Description	Example Value
        Vendor	Select a cloud vendor type.	Huawei Cloud
        Source of Attack Target	Select the source of the target instance. You can select attack targets by selecting instances, pods, or a specified number of targets if CCE instances are used.	Elastic Cloud Server (ECS)
        Attack Task	Customize the name of the attack task based on the naming rule.	test-attacktask
        Attack Target	Select the target instance.	-

   2. Click Next.
   3. Set parameters for selecting an attack scenario.
      For details, see Attack Scenarios.

      Table 3 Parameters for selecting an attack scenario

      Parameter	Description	Example Value
      Attack Type	Attack scenarios are classified based on attack scenario types.	Host Resource
      Attack Scenario	Select an attack scenario.	CPU usage increase
      Attack Parameters	Configure attack parameters based on attack scenarios.	CPU Usage (%): 80 Fault Duration (s): 60

   4. Click Next.
   5. (Optional) Set Configure Monitoring Tasks.

      Table 4 Parameters for configuring a monitoring task

      Parameter	Description
      Steady-State Metrics	Select the target resource, performance metric, lower limit, and upper limit from the drop-down lists one by one. If a service can perform well and stably when a performance monitoring metric is set to a certain value range, this metric is called stable-status metric. If this metric value is not in that value range before a drill, the drill will be canceled.
      Metric	Select the target resource, monitoring metric, lower limit, and upper limit from the drop-down lists one by one. These service metrics monitor the corresponding service data during fault drills. If the value of such a metric is within the allowed value range, the service is normal. Otherwise, you can determine whether to stop a drill.
      Automatic Rollback	Select whether to enable automatic rollback. Fault injection is automatically rolled back and restored to the status before fault injection. Automatic rollback cannot be configured for some disruptors for fault drills that do not support fault termination. If the value of a steady-state metric is not within the stable value range during a drill, the corresponding fault injection automatically stops after automatic rollback is enabled.

   6. Click Finish. The attack task is added.

7. Click OK. The drill task is created and the task status is to be drilled.

   If you click Save as Draft, the task status is draft, which does not allow you to start the drill task.

Modifying a Drill Task

If drill records have been generated for a drill task, the task cannot be modified.

Deleting a Drill Task

If a created drill task is no longer needed, you can delete it. Do not delete the drill task in the following scenarios:

• The drill task has generated drill records.
• The drill task is associated with a drill plan.

Starting a Drill Task

Start a drill task.

Viewing Drill Records

View the drill records of a drill task. A drill task that has not been drilled does not contain drill record.

Manually Clearing Data in the Environment

Forcibly terminating a drill task only closes the current drill ticket and does not automatically clear data in the environment. You need to manually clear the data.

The clearing methods vary depending on probes. For details, see Table 5.