ALM-33008 Average Latency of Containers Exceeds the Threshold

Alarm Description

The system checks the average latency of service calls every 5 minutes and compares the latency with the threshold. This alarm is generated when the average latency exceeds the threshold.

This alarm is automatically cleared when the average latency is less than or equal to the threshold.

Alarm Attributes

Alarm Parameters

Impact on the System

The BLU service processing becomes slow. As a result, user requests may be stacked, and even upper-layer services may be blocked or unavailable.

Possible Causes

• It takes a long time to process the upper-layer service logic code.
• The service volume is too large.
• The service code is incorrect.

Handling Procedure

Check whether the processing time of the upper-layer service logic code is too long.

1. Check whether the service for which the alarm is generated is a provider based on the IP address.
   • If yes, go to 9.
   • If no, go to 2.

2. On FusionInsight Manager, choose Cluster > Services > Containers, and click the SGP Management tab.
3. Choose App Service List and click the link of the service for which the alarm is generated to access the service status page.
4. View the processing time of the service consumer and check whether it is too long.
   • If yes, go to 5.
   • If no, go to 9.

5. View the processing time of the service provider and compare it with that of the consumer to check whether the difference is large.
   • If yes, check the network configuration, reduce the network latency, and go to 6.
   • If no, go to 7.

6. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 7.

7. Check whether the processing time of user's upper-layer service logic code is too long.
   • If yes, optimize the upper-layer service logic code and execute 8.
   • If no, go to 9.

8. Check whether the alarm is cleared.
   • If yes, no further action is required.
   • If no, go to 9.

Check whether the service volume is too large.

9. On FusionInsight Manager, choose Cluster > Services > Containers, and click the SGP Management tab.
10. Click the name of the service for which the alarm is generated to access its State page.
11. Check whether the number of concurrent requests exceeds the upper limit.
    • If yes, go to 12.
    • If no, go to 14.

12. On FusionInsight Manager, choose Cluster > Services > Containers, and click the Business Manager tab. Select the BLU that carries services. On the page that is displayed, click the BLU Instance tab and click Add Instance to add a BLU instance for the service provider.
13. Check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 14.

Check whether the service code is correct.

14. Check whether the upper-layer service code is incorrect.
    • If yes, correct the service code and go to 15.
    • If no, go to 16.

15. Check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 16.

Change the alarm threshold.

16. On FusionInsight Manager, choose O&M > Alarm > Thresholds. On the Thresholds page, click the name of the desired cluster, choose Containers > Other > Application Service Average Elapse Statistics (SGP), and change the alarm threshold based on the actual average latency.
17. Check whether the alarm is cleared.
    • If yes, no further action is required.
    • If no, go to 18.

Collect fault information.

18. On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
19. Expand the Service drop-down list and select Containers.
20. Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the hosts to which the role belongs, and click OK.
21. Click the edit icon in the upper right corner, and set Start Date and End Date for log collection to 1 hour ahead of and after the alarm generation time, respectively. Then, click Download.
22. Contact O&M personnel and provide the collected logs.

Alarm Clearance

This alarm is automatically cleared after the fault is rectified.

Related Information

None.