ALM-33006 Failure Rate of Containers Calls Exceeds the Threshold

Alarm Description

The system checks the failure rate of service calls every 5 minutes and compares the rate with the threshold. This alarm is generated when the failure rate exceeds the threshold.

This alarm is automatically cleared when the failure rate is less than or equal to the threshold.

Alarm Attributes

Alarm Parameters

Impact on the System

If the failure rate is too high, the application service fails to be called for many times within 5 minutes. As a result, the service calls be slow or fail.

Possible Causes

• A timeout occurs due to a long network latency.
• The service volume is too large, and the number of concurrent requests exceeds the upper limit.
• The service code is incorrect.

Handling Procedure

Check whether the network latency is too long.

1. Check whether the service for which the alarm is generated is a provider based on the IP address.
   • If yes, go to 9.
   • If no, go to 2.

2. On FusionInsight Manager, choose Cluster > Services > Containers, and click the SGP Management tab.
3. Choose App Service List and click the link of the service for which the alarm is generated to access the service status page.
4. View the processing time of the service consumer and check whether it is too long.
   • If yes, go to 5.
   • If no, go to 9.

5. View the processing time of the service provider and compare it with that of the consumer to check whether the difference is large.
   • If yes, check the network configuration, reduce the network latency, and go to 6.
   • If no, go to 7.

6. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 7.

7. Check whether the processing time of user's upper-layer service logic code is too long.
   • If yes, optimize the upper-layer service logic code and execute 8.
   • If no, go to 9.

8. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 9.

Check whether the service volume is too large and the number of concurrent requests exceeds the upper limit.

9. On FusionInsight Manager, choose Cluster > Services > Containers, and click the SGP Management tab.
10. Click the name of the service for which the alarm is generated to access its State page.
11. Check whether the number of concurrent requests exceeds the upper limit.
    • If yes, go to 12.
    • If no, go to 14.

12. Choose Cluster > Services > Containers, and click the Business Manager tab. Select the BLU that carries services. On the page that is displayed, click the BLU Instance tab and click Add Instance to add a BLU instance for the service provider.
13. Check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 14.

Check whether the service code is correct.

14. Check whether the upper-layer service code is incorrect.
    • If yes, correct the service code and go to 15.
    • If no, go to 16.

15. Check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 16.

Collect fault information.

16. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
17. Expand the Service drop-down list and select Containers.
18. Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the hosts to which the role belongs, and click OK.
19. Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 20 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
20. Contact O&M personnel and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.