What Should I Do If Excessive Docker Audit Logs Affect the Disk I/O?
Symptom
There are a large number of Docker audit logs on existing nodes in some clusters. Due to OS kernel defects, it is slightly possible that I/Os are suspended. You can optimize the audit log rules to avoid this problem.
Impact
Affected cluster versions:
- v1.15.11-r1
- v.1.17.9-r0
- You only need to fix this issue for existing nodes, not for newly created nodes.
- The auditd component needs to be restarted during the upgrade.
Check Method
- Log in to the worker node as user root.
- Run the following command to check whether the problem exists on the current node:
auditctl -l | grep "/var/lib/docker -p rwxa -k docker"
If information similar to the following is displayed, the problem exists and needs to be rectified. If no command output is displayed, the node is not affected.
Solution
- Log in to the worker node as user root.
- Run the following commands:
sed -i "/\/var\/lib\/docker -k docker/d" /etc/audit/rules.d/docker.rules
service auditd restart
Verification Method
Run the following command to check whether the fault is rectified:
auditctl -l | grep "/var/lib/docker -p rwxa -k docker"
If no command output is displayed, the problem has been resolved.
Node Running FAQs
- What Should I Do If a Cluster Is Available But Some Nodes Are Unavailable?
- How Do I Troubleshoot the Failure to Remotely Log In to a Node in a CCE Cluster?
- How Do I Log In to a Node Using a Password and Reset the Password?
- How Do I Collect Logs of Nodes in a CCE Cluster?
- What Can I Do If the Container Network Becomes Unavailable After yum update Is Used to Upgrade the OS?
- What Should I Do If the vdb Disk of a Node Is Damaged and the Node Cannot Be Recovered After Reset?
- Which Ports Are Used to Install kubelet on CCE Cluster Nodes?
- How Do I Configure a Pod to Use the Acceleration Capability of a GPU Node?
- What Should I Do If I/O Suspension Occasionally Occurs When SCSI EVS Disks Are Used?
- What Should I Do If Excessive Docker Audit Logs Affect the Disk I/O?
- How Do I Fix an Abnormal Container or Node Due to No Thin Pool Disk Space?
- Which Ports Does a Node Listen On?
- How Do I Rectify Failures When the NVIDIA Driver Is Used to Start Containers on GPU Nodes?
- What Should I Do If a Node Does Not Synchronize with the NTP Clock Source?
- What Should I Do If the Data Disk Usage Is High Because a Large Volume of Data Is Written Into the Log File?
- Why Does My Node Memory Usage Obtained by Running the kubelet top node Command Exceeds 100%?
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbotmore