What Should I Do If the Agent Status Switches Between Running and Faulty?
Symptom
If the Cloud Eye Agent status switches between Running and Faulty, the Agent may be overloaded and unstable.
Constraints
The restoration method in this section only supports the new Agent version. If your Agent is of an earlier version, you are advised to upgrade it to the new version.
- Linux:
- Log in to the server as user root.
- Check the Agent version:
if [[ -f /usr/local/uniagent/extension/install/telescope/bin/telescope ]]; then /usr/local/uniagent/extension/install/telescope/bin/telescope -v; elif [[ -f /usr/local/telescope/bin/telescope ]]; then echo "old agent"; else echo 0; fi
- If old agent is returned, the earlier version of the Agent is used. Perform operations by referring to the instructions of the earlier version.
- If a version is returned, the new version of the Agent is in use. Perform operations by referring to the instructions of the new version.
- If 0 is returned, the Agent is not installed.
- Windows: Determine the Agent version based on the installation path. Default installation paths:
- New version: C:\Program Files\uniagent\extension\install\telescope
- Earlier version: C:\Program Files\telescope
Possible Causes
To prevent the Agent from being overloaded and affecting other services on servers, Cloud Eye provides a circuit breaker mechanism when the CPU or memory usage of the Agent is too high. When the Agent is overloaded, the circuit breaker mechanism is automatically triggered. After that, the Agent stops working and does not report monitoring data.
Circuit Breaker Principles
By default, the Agent detection mechanism is as follows:
The system checks whether the Agent process exceeds the second threshold (CPU usage exceeds 30% or memory usage exceeds 700 MB) every minute. If either the CPU or memory usage exceeds the second threshold, the Agent process exits. If the second threshold is not exceeded, the system checks whether the Agent process exceeds the first threshold (CPU usage exceeds 10% or memory usage exceeds 200 MB). If the first threshold is exceeded for three consecutive times, the Agent process exits and the event is recorded.
After the Agent exits, the daemon process automatically starts the Agent process and checks the exit records. If there are three consecutive exit records, the Agent will hibernate for 20 minutes, during which monitoring data will not be collected.
When too many disks are attached to a server, the CPU or memory usage of the Agent process will become high. You can configure the tier-1 and tier-2 thresholds based on Solution to trigger the circuit-breaker pattern according to the actual resource usages.
Solution
- Log in to the ECS or BMS where the Agent does not report data.
- Optional: Modify the conf.json configuration file.
- Go to the Agent installation path bin by running the following command: Windows:
cd C:\Program Files\uniagent\extension\install\telescope\bin
Linux:cd /usr/local/uniagent/extension/install/telescope/bin
- Open conf.json.
vi conf.json
- Add the following parameters to the conf.json file.
{ "cpu_first_pct_threshold": xx, "memory_first_threshold": xxx, "cpu_second_pct_threshold": xx, "memory_second_threshold": xxx }Table 1 Parameters in the conf.json file Parameter
Description
How to Query Parameter Values
cpu_first_pct_threshold
Tier-1 threshold of CPU usage. The default value is 10 (%).
To query the CPU usage and memory usage of the Agent process, use either of the following methods:
memory_first_threshold
Tier-1 threshold of memory usage. The default value is 209715200 (200 MB). The unit is byte.
cpu_second_pct_threshold
Tier-2 threshold of CPU usage. The default value is 30 (%).
memory_second_threshold
Tier-2 threshold of memory usage. The default value is 734003200 (700 MB). The unit is byte.
- Save the conf.json file and exit.
:wq
- Go to the Agent installation path bin by running the following command:
- Modify the manifest.json configuration file.
- Switch to the manifest.json file directory. Windows:
cd C:\Program Files\uniagent\extension\holder\telescope
Linux:cd /usr/local/uniagent/extension/holder/telescope
- Open manifest.json.
vi manifest.json
- In the manifest.json file, modify mem and cpuUsage in resourceLimit. There are three groups of parameters. Retain the default values of other parameters.
[ { "arch": "amd64", ... "resourceLimit": { "mem": 200, "cpuUsage": 10 } }, { "arch": "arm64", ... "resourceLimit": { "mem": 200, "cpuUsage": 10 } }, { "arch": "amd64", "os": "linux", ... "resourceLimit": { "mem": 200, "cpuUsage": 10 } } ]Table 2 Parameters in the manifest.json file Parameter
Description
"resourceLimit": {
"mem": 200,
"cpuUsage": 10
}
mem: memory threshold. The default value is 200, in MB.
cpuUsage: CPU usage threshold (%). The default value is 10.
- Save the manifest.json file and exit.
:wq
- Switch to the manifest.json file directory.
- Restart the Agent.
- Windows:
- In the C:\Program Files\uniagent\extension\install\telescope directory, double-click shutdown.bat to stop the Agent, and then run start.bat to start the Agent.
- In the C:\Program Files\uniagent\script directory, double-click shutdown.bat to stop the Agent, and then run start.bat to start the Agent.
- Linux:
service ces-uniagent stop service ces-uniagent start /usr/local/uniagent/extension/install/telescope/telescoped restart
- Windows:
What is your overall rating for this page?
Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot