When massive faults or major faults occur, a war room can be initiated to include O&M, R&D, and operation personnel for quick service recovery. You can add related members to the war room for fault rectification. The notification of fault rectification progress is sent to the personnel who are concerned about the fault in a timely manner. The application diagnosis and response plans can help quickly recover applications.
Constraints
A war room can be initiated only for incidents that are being processed. For details, see Initiating a War Room.
Precautions
If a group (WeCom or DingTalk group) needs to be set up when a war room is initiated, configure the following information:
Procedure
Figure 1 War room handling process
The war room handling process is as follows:
1. If the existing war room initiation rules do not meet the current scenario, create one by referring to Creating a War Room Initiation Rule. The corresponding rule can be matched based on the preset region, application, and incident level.
Creating a War Room Initiation Rule
When a war room is initiated, the system matches a rule based on the region, application, and incident level, and adds related personnel to the war room. The personnel responsible for fault rectification can receive a notification for response.
- Log in to COC.
- In the navigation pane, choose Fault Management > WarRoom.
- Click WarRoom Rules.
- Click Create WarRoom Rule.
- Set parameters for creating a war room initiation rule.
Table 1 Parameters for creating a war room initiation rule | Parameter | Description |
| Rule | Specify the rule name. |
| Region | (Optional) Select one or more regions from the drop-down list. If no region is selected, default is displayed, indicating that no region is required. |
| Application | Select one or more applications. |
| Incident Level | The options are P1, P2, P3, P4, and P5. Multiple options can be selected. P1 incidents are the most critical, while P5 incidents are the least severe. |
| Group | (Optional) Select shift roles. The shift roles are automatically added to the third-party application group when the war room is initiated. For details about how to configure a shift, see Shift Scheduling Management. For details about how to configure mobile apps, see Mobile App Management. |
- Click OK.
Trigger of a war room initiation rule: The region, application, and incident level of the war room initiation rule are matched with those of the incident. Related personnel will be invited into the war room on a mobile app.
2. If an incident meets a war room initiation rule, initiate a war room on the Incidents > Handle Incident page by referring to Initiating a War Room.
Initiating a War Room
If an incident meets the war room initiation rule, a war room can be initiated for incident handling.
- Log in to COC.
- In the navigation pane, choose Fault Management > Incidents.
- On the Pending tab page, locate the target incident ticket and click its title to go to the details page.
- Click Start WarRoom in the upper right corner. The page for starting a war room is displayed.
- Set parameters for initiating a war room.
Table 2 Parameters for initiating a war room | Parameter | Description |
| WarRoom Name | The preset value is the incident ticket name. You can specify one. |
| WarRoom Description | (Optional) Description of the war room. |
| War Room Administrator | Select a user from the drop-down list as the war room administrator. |
| Region | (Optional) Select a region for the war room. You can select multiple regions. |
| Enterprise Project | Select an enterprise project. |
| Application | Select an affected application. You can select multiple applications. |
| Group Creation Method | (Optional) Set this parameter to WeCom, or DingTalk. Configure the application notification channel in mobile app management. After the notification channel is selected, the shift roles and participants will be added to the corresponding group when the war room is initiated. |
| Notification Mode | (Optional) Set this parameter to SMS, WeCom, or DingTalk. To select WeCom or DingTalk, configure the application notification channel in mobile application management first. |
| Shift | (Optional) Select a shift scenario and corresponding roles from the drop-down lists. For details about how to configure a shift, see Shift Schedule Management. |
| Participant | (Optional) Select a participant. You can select multiple participants. |
- Click OK.
3. Analyze the fault impact scope and recover all affected applications.
If other applications are affected, add affected applications by referring to Adding an Affected Application.
Adding an Affected Application
If an application is affected when a fault occurs, you can add the affected application in the war room details page. You can use the application diagnosis function to check affected application details and execute response plans to quickly restore applications.
You can add affected applications when initiating war rooms, locating faults, and rectifying faults.
- Log in to COC.
- In the navigation pane, choose Fault Management > WarRoom.
- Click the title of the target war room to go to its details page.
- Click Add Affected Application. The page for adding an affected application is displayed on the right.
- Set information about the affected application by referring to Table 3.
Table 3 Parameters for adding an affected application | Parameter | Description |
| Affected Application | Select an affected application. |
| Start Time | Enter the time when the application starts to be affected. The preset value is the time when the war room is created. The start time cannot be later than the war room creation time. |
| Recovery Time | (Optional) Enter the application recovery time. The recovery time cannot be earlier than the war room creation time. |
| Description | Enter the impact description of the application. The value can contain a maximum of 500 characters. |
- Click OK. The affected application is added.
- Click an affected application to view the alarms, incidents, and changes of the application.
4. Locate the root cause of the fault using application diagnosis and rectify the fault by executing response plans. After the fault is rectified, change the application status to Restored.
Diagnosing an Application
- Log in to COC.
- In the navigation pane, choose Fault Management > WarRoom.
- Click the title of the target war room to go to its details page.
- Select the application to be handled and click Application Diagnosis.
- Click the time box and set the fault occurrence time.
The time entered in the time box is the end time. The start time is one hour earlier than the end time. After the time is selected, the number of alarms for the application and its sub-applications in the selected time period is displayed on the application topology dashboard, and the application fault details are displayed on the details page on the right.
- (Optional) Select Auto Refresh and select a refresh frequency.
After Auto Refresh is selected, the end time is updated to the current system time based on the refresh frequency.
- (Optional) If the application has sub-applications, click the target sub-application.
The application topology dashboard displays all components of the sub-application. The sub-application fault details are displayed on the details page on the right. You can switch to other sub-applications on the topology dashboard.
- Click a component under the application or its sub-application.
The application topology dashboard displays all resources of the component. The component fault details are displayed on the details page. You can switch to other components on the topology dashboard. Metrics of core cloud services can be displayed. If APM is associated in application management, you can also view link-related metrics.
- Click Alarm on the right of the application topology.
View application alarms. Alarms generated within the time range on the right axis are displayed in the list. After you selected a topology object on the left, its alarm data is automatically displayed.
- Click Change on the right of the application topology.
View application changes. Changes within the change time range on the right axis are displayed in the list.
- Click Fault Diagnosis on the right of the application topology.
View the fault diagnosis data for your resources. DCS, RDS, DMS, ECS, and ELB resources can be diagnosed. After you selected a topology object on the left, its diagnosis information is automatically displayed.
To create a diagnosis task, perform the following steps:
- Click Create Diagnosis Task.
- Select a resource type and resource.
- Click OK.
- Read and agree to Frontend Data Authorization Agreement on Guest OS Diagnosis Service, and click Agree.
You need to sign the agreement only if you select ECSs for fault diagnosis.
After the diagnosis is complete, click View Details on the right of the diagnosis result list to view the diagnosis report.
Executing a Response Plan
- Log in to COC.
- In the navigation pane, choose Fault Management > WarRoom.
- Click the title of the target war room to go to its details page.
- Select the application to be handled and click Execute Plan.
- Select a contingency plan.
If no proper contingency plan is available, create one by referring to Creating a Contingency Plan.
- Perform the corresponding step based on the task type associated with the response plan.
- Click OK.
5. Check the fault handling result and application status and set fault information by referring to Modifying Fault Information.
Note: The fault can be rectified only after all applications associated with the war room are recovered.
Modifying Fault Information
Fault information of war rooms records the fault occurrence time, rectification time, impacts, and causes for subsequent backtracking.
You can modify fault information during war room initiation as well as fault demarcation and rectification.
- Log in to COC.
- In the navigation pane, choose Fault Management > WarRoom.
- Click the title of the target war room to go to its details page.
- Click Modify.
- Set parameters for modifying the fault information.
Table 6 Parameters for modifying the fault information | Parameter | Description |
| Region | (Optional) Select one or more regions. If no region is selected, default is displayed, indicating that no region is required. |
| Fault Occurrence Time | Enter the time when the fault occurs. The preset value is the time when the war room is created. The fault occurrence time cannot be later than the war room creation time. |
| Fault Rectification Time | (Optional) This parameter is mandatory when the fault has been rectified. The fault rectification time cannot be earlier than the war room creation time. |
| Fault Impact | (Optional) This parameter is mandatory when the fault has been rectified. Enter the fault impact. The value can contain a maximum of 250 characters. |
| Fault Cause | (Optional) This parameter is mandatory when the fault has been rectified. Enter the fault cause. The value can contain a maximum of 250 characters. |
- Click OK.
6. Close the war room after the fault is rectified.
Note: The war room can be closed only after all mandatory parameters in Modifying Fault Information are set.
Inviting Members to a War Room
To quickly rectify a fault, you can invite fault rectification personnel to a war room or you can notify them through phone calls or SMS messages. After the invitation, you can set the administrator, owner, and support members.
The administrator is the personnel who creates the war room.
- Log in to COC.
- In the navigation pane, choose Fault Management > WarRoom.
- Click the title of the target war room to go to its details page.
- Click Invite on the right.
- Set the parameters for inviting members.
Table 7 Parameters for inviting members | Parameter | Description |
| Attendance Mode | The options are WeCom and DingTalk. After you select an attendance mode, the corresponding notification channel is automatically selected. You need to configure the application by referring to Mobile App Management and configure the email address of the application by referring to Overview. |
| Notification Mode | Supported options: SMS messages. You can select multiple options. The value can be WeCom or DingTalk. The notification channel is automatically selected when you select an attendance mode. For details about how to configure user information, see Personnel Management. |
| O&M Roles | Select a shift from the drop-down list. You can select multiple shift roles. After a shift is selected, all members in the shift are automatically selected. For details, see Shift Schedule Management. |
| O&M Personnel | Select one or more users from the drop-down list. |
- Click Add to WarRoom.
Setting Member Roles
- Log in to COC.
- In the navigation pane, choose Fault Management > WarRoom.
- Click the title of the target war room to go to its details page.
- Click
on the right of the participant to expand the participant list. - Select the member whose role is to be set and click
on the right. - Select a role.
Figure 2 Setting a member role
Table 8 Member role description | Member Role | Description |
| Administrator | The administrator role can be transferred to other members without roles. The war room administrator is a hidden administrator. Even if other members are set as administrators, the war room administrator still retains the permission to manage members. |
| Fault Rectification Owner | You can set a member without a role as a fault rectification owner. The fault rectification owner can be canceled by the administrator but cannot be removed. |
| Fault Rectification Personnel | You can set a member without a role as a fault rectification personnel. The fault rectification personnel can be canceled or removed by the administrator. |
| Member Without a Role | You can set a member as an administrator, fault rectification owner, or fault rectification personnel. A member without a role can be removed by the administrator. |
Updating a Progress Notice
When a fault occurs or is handled, the progress notice is synchronized to related personnel in a timely manner. People who are concerned about the fault can quickly learn the fault handling progress.
- Log in to COC.
- In the navigation pane, choose Fault Management > WarRoom.
- Click the title of the target war room to go to its details page.
- Click Update Notice on the right.
- Set parameters for updating a progress notice.
Table 9 Parameters for updating a progress notice | Parameter | Description |
| Notification Template | The options are First Notification, Progress Notification, and Fault Recovery Notification. |
| Content | Enter the content based on the selected notification template. The value can contain a maximum of 1,000 characters. |
- Click OK.
The latest notice is displayed in the Progress Notices area.
- Click Release and set the information in the displayed dialog box.
Table 10 Parameters for releasing a notice | Parameter | Description |
| Notice Topic | Specify the notice topic. |
| Recipient | Select Shift or Individual. - Shift: Select a shift scenario and corresponding roles from the drop-down lists based on the configured values.
Click Create Schedule to configure the shift schedule. For details, see Shift Schedule Management. - Individual: Select an individual that you want to notify.
Click Synchronize to configure users. For details, see Personnel Management. |
| Sending Mode | (Optional) Set this parameter to SMS, WeCom or DingTalk. Before setting this parameter, configure WeCom or DingTalk by referring to Mobile App Management. |
| Content | The content cannot be modified. The content is the latest. |
- Click OK.
Viewing the Status of a War Room
After a war room is initiated, you need to view and update the war room status during fault handling. In this way, you can record the time when a fault is rectified and learn about the fault handling progress. A war room can be in the Start, Demarcate and Rectify the Fault, Confirm Fault Recovery, and Finish states.
- Before the fault is rectified, ensure that the affected application is in the Restored state.
- Before closing a war room, enter the fault information of the war room.
- Log in to COC.
- In the navigation pane, choose Fault Management > WarRoom.
On the displayed page, view the war room list.
- Click the title of the target war room to go to its details page.
The war room details page is displayed. You can view the status of the war room in the upper right corner of the page.
- If some product, O&M, or management items need to be improved during fault handling, you can create improvement tickets and handle them through improvement management.
More Operations
You can also perform the following operations.
Table 11 More operations | Function | Scenario | Operation |
| Modifying a war room initiation rule | You can modify a created war room initiation rule. | - Choose Fault Management > WarRoom > WarRoom Rules. On the displayed page, locate the target war room initiation rule and click Modify in the Operation column.
- Modify the war room initiation rule based on the service scenario.
- Click OK.
|
| Deleting a war room initiation rule | If a created war room initiation rule is no longer needed, you can delete it. Note: Deleted rules cannot be restored. Exercise caution when performing this operation. | - Choose Fault Management > WarRoom > WarRoom Rules. On the displayed page, locate the target war room initiation rule and click Delete in the Operation column.
- In the displayed dialog box, click OK.
|
Helpful Links
On COC, you can create and query war rooms by calling APIs. For details, see War Rooms.