How Do I Configure atop and kdump on Linux ECSs for Performance Analysis?
Scenarios
This section describes how you can configure atop and kdump on Linux ECSs for performance analysis.
The method for configuring atop varies with the OS version.
atop
- Introduction to atop
- Preparing for atop Installation
- Configuring atop for CentOS 7/8, AlmaLinux, and Rocky Linux
- Configuring atop for CentOS 6
- Configuring atop for Ubuntu 20/22 and Debian 10/11
- Configuring atop for Ubuntu 18 and Debian 8/9
- Configuring atop for Ubuntu 16
- Configuring atop for SUSE 12 or SUSE 15
- Installing atop by Compiling the Source Code (for CentOS Stream 9, openEuler or EulerOS)
- Analyzing atop Logs
kdump
Introduction to atop
atop is a monitor for Linux that can report the activity of all processes and resource consumption by all processes at regular intervals. It shows system-level activity related to the CPU, memory, disks, and network layers for every process. It also logs system and process activities daily and saves the logs in disks for long-term analysis.
Preparing for atop Installation
- Ensure that the target ECS already has an EIP bound.
- Ensure that the target ECS can access YUM.
Configuring atop for CentOS 7/8, AlmaLinux, and Rocky Linux
- Run the following command to install atop:
- Run the following command to modify the configuration file of atop:
Modify the following parameters, save the modification, and exit:
- Change the value of LOGINTERVAL to, for example, 15. The default value of LOGINTERVAL is 600, in seconds.
- Change the value of LOGGENERATIONS to, for example, 3. The default retention period of atop logs is 28 days.
LOGINTERVAL=15 LOGGENERATIONS=3
- Run the following command to start atop:
- Run the following command to check the status of atop. If active (running) is displayed in the output, atop is running properly.
systemctl status atop
atop.service - Atop advanced performance monitor Loaded: loaded (/usr/lib/systemd/system/atop.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2024-03-6 11:49:47 CST; 2h 27min ago
Configuring atop for CentOS 6
- Run the following command to install atop:
- Run the following command to modify the configuration file of atop:
Modify the following parameters, save the modification, and exit:
The default value of LOGINTERVAL is 600 (seconds), but you can change it to, for example, 15.
LOGINTERVAL=15
vi /etc/logrotate.d/atop
Modify the following parameters, save the modification, and exit:
You can change the value of -mtime to, for example, 3. The default retention period of atop logs is 40 days.
postrotate /usr/bin/find /var/log/atop/ -maxdepth 1 -mount -name atop_\[0-9\]\[0-9\]\[0-9\]\[0-9\]\[0-9\]\[0-9\]\[0-9\]\[0-9\]\* -mtime +3 -exec /bin/rm {} \; endscript - Run the following command to start atop:
- Run the following command to check the status of atop. is running indicates that atop is running properly.
service atop status
atop (pid 3170) is running
Configuring atop for Ubuntu 20/22 and Debian 10/11
- Run the following command to install atop:
- Run the following command to modify the configuration file of atop:
Modify the following parameters, save the modification, and exit:
- Change the value of LOGINTERVAL to, for example, 15. The default value of LOGINTERVAL is 600, in seconds.
- Change the value of LOGGENERATIONS to, for example, 3. The default retention period of atop logs is 28 days.
LOGINTERVAL=15 LOGGENERATIONS=3
- Run the following command to start atop:
- Run the following command to check the status of atop. active (running) indicates that atop is running properly.
systemctl status atop
atop.service - Atop advanced performance monitor Loaded: loaded (/etc/init.d/atop; bad; vendor preset: disabled) Active: active (running) since Sat 2024-03-11 14:09:47 CST; 16s ago
Configuring atop for Ubuntu 18 and Debian 8/9
- Run the following command to install atop:
- Run the following command to modify the configuration file of atop:
Modify the following parameters, save the modification, and exit:
- The default value of LOGINTERVAL is 600 (seconds), but you can change it to, for example, 15.
- You can change the value of -mtime to, for example, 3. The default retention period of atop logs is 28 days.
LOGINTERVAL=15 …… ( (sleep 3; find $LOGPATH -name 'atop_*' -mtime +3 -exec rm {} \;)& )
- Run the following command to start atop:
- Run the following command to check the status of atop. active (running) indicates that atop is running properly.
systemctl status atop
atop.service - Atop advanced performance monitor Loaded: loaded (/etc/init.d/atop; bad; vendor preset: disabled) Active: active (running) since Sat 2024-03-6 14:09:47 CST; 15s ago
Configuring atop for Ubuntu 16
- Run the following command to install atop:
- Run the following command to modify the configuration file of atop:
Modify the following parameters, save the modification, and exit:
- The default value of LOGINTERVAL is 600 (seconds), but you can change it to, for example, 15.
- The default retention period of atop logs is 28 days and cannot be modified.
LOGINTERVAL=15
- Run the following command to start atop:
- Run the following command to check the status of atop. active (running) indicates that atop is running properly.
systemctl status atop
atop.service - LSB: Monitor for system resources and process activity Loaded: loaded (/etc/init.d/atop; bad; vendor preset: enabled) Active: active (running) since Mon 2024-04-29 19:33:22 CST; 38s ago
Configuring atop for SUSE 12 or SUSE 15
- Run the following command to download the atop source package:
- Run the following command to install the package:
- Run the following command to install atop dependencies.
- Run the following command to compile atop:
rpmbuild -bb atop-2.6.0.spec
- Run the following command to install atop:
cd /usr/src/packages/RPMS/x86_64
rpm -ivh atop-2.6.0-1.x86_64.rpm
- Run the following command to modify the configuration file of atop:
Modify the following parameters, save the modification, and exit:
- Change the value of LOGINTERVAL to, for example, 15. The default value of LOGINTERVAL is 600, in seconds.
- Change the value of LOGGENERATIONS to, for example, 3. The default retention period of atop logs is 28 days.
LOGINTERVAL=15 LOGGENERATIONS=3
- Run the following command to restart atop:
- Run the following command to check the status of atop. active (running) indicates that atop is running properly.
systemctl status atop
atop.service - Atop advanced performance monitor Loaded: loaded (/usr/lib/systemd/system/atop.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2021-06-19 16:50:01 CST; 6s ago
Installing atop by Compiling the Source Code (for CentOS Stream 9, openEuler or EulerOS)
- Decompress the source package.
- Query the systemctl version.
If the version is 220 or later, go to the next step.
Otherwise, delete parameter --now from the Makefile of atop.
vi atop-2.6.0/Makefile
Delete parameter --now following the systemctl command.
then /bin/systemctl disable atop 2> /dev/null; \ /bin/systemctl disable atopacct 2> /dev/null; \ /bin/systemctl daemon-reload; \ /bin/systemctl enable atopacct; \ /bin/systemctl enable atop; \ /bin/systemctl enable atop-rotate.timer; \ - Install atop dependencies.
- Run the following commands to compile and install atop.
make systemdinstall
- Modify the configuration file of atop.
Modify the following parameters, save the modification, and exit:
- Change the value of LOGINTERVAL to, for example, 15. The default value of LOGINTERVAL is 600, in seconds.
- Change the value of LOGGENERATIONS to, for example, 3. The default retention period of atop logs is 28 days.
LOGOPTS="" LOGINTERVAL=15 LOGGENERATIONS=3 LOGPATH=/var/log/atop
- Restart atop.
- Run the following command to check the status of atop. active (running) indicates that atop is running properly.
systemctl status atop
atop.service - Atop advanced performance monitor Loaded: loaded(/lib/systemd/system/atop.service; enabled) Active: active (running) since Sun2021-07-25 19:29:40 CST; 4s ago .
Analyzing atop Logs
After startup, atop stores collection records in /var/log/atop.
Run the following command to check the log file:
atop -r /var/log/atop/atop_2024XXXX
- Common atop commands
After opening the log file, you can use the following commands to sort data.
- c: used to sort processes by CPU usage in descending order.
- m: used to sort processes by memory usage in descending order.
- d: used to sort processes by disk usage in descending order.
- a: used to sort processes by the overall resource usage in descending order.
- n: used to sort processes by network usage in descending order.
- t: used to go to the next monitoring collection point.
- T: used to go to the previous monitoring collection point.
- b: used to specify a time point in the format of YYYYMMDDhhmm.
- System resource monitoring fields
The following figure shows some monitoring fields and values. The values vary according to the sampling period and atop version. The figure is for reference only.
Figure 1 System resource monitoring fields
Description of major fields is as follows:- ATOP row: Specifies the host name and information sampling date and time.
- PRC row: Specifies the running status of a process.
- #sys and user: Specifies how long the CPU is occupied when the system is running in kernel mode and user mode.
- #proc: Specifies the total number of processes.
- #zombie: Specifies the number of zombie processes.
- #exit: Specifies the number of processes that exited during the sampling period.
- CPU row: Specifies the overall CPU usage (multi-core CPU as a whole CPU). The sum of the values in the CPU row is N x 100%. N indicates the number of vCPUs.
- #sys and user: Specifies the percentage of how long the CPU is occupied when the system is running in kernel mode and user mode.
- #irq: Specifies the percentage of time when CPU is servicing interrupts.
- #idle: Specifies the percentage of time when CPU is idle.
- #wait: Specifies the percentage of time when CPU is idle due to I/O wait.
- CPL row: Specifies CPU load.
- #avg1, avg5 and avg15: Specifies the average number of running processes in the past 1, 5, and 15 minutes, respectively.
- #csw: Specifies the number of context exchanges.
- #intr: Specifies the number of interruptions.
- MEM row: Specifies the memory usage.
- #tot: Specifies the physical memory size.
- #free: Specifies the size of available physical memory.
- #cache: Specifies the memory size used for page cache.
- #buff: Specifies the memory size used for file cache.
- #slab: Specifies the memory size occupied by the system kernel.
- SWP row: Specifies the usage of swap space.
- #tot: Specifies the total swap space.
- #free: Specifies the size of available swap space.
- DSK row: Specifies the disk usage. Each disk device corresponds to a column. If there is an sdb device, a DSK row should be added.
- #sda: Specifies the disk device identifier.
- #busy: Specifies the percentage of time when the disk is busy.
- #read and write: Specifies the number of read and write requests.
- NET row: Displays the network status, covering the transport layer (TCP and UDP), IP layer, and active network ports.
- #xxxxxi: Specifies the number of packets received by each layer or active network port.
- #xxxxxo: Specifies the number of packets sent by each layer or active network port.
- Stopping atop
Running atop occupies extra system and disk resources. You are not advised to run it for a long time in the service environment. After faults are rectified, run the following command to stop atop:
systemctl stop atop
For CentOS 6, run the following command to stop atop:
service atop stop
Precautions for Configuring kdump
The method for configuring kdump described in this section applies to KVM ECSs running EulerOS or CentOS 7.x. For details, see Documentation for kdump.
Introduction to kdump
kdump is a feature of the Linux kernel that creates crash dumps in the event of a kernel crash. In the event of a kernel crash, kdump boots another Linux kernel and uses it to export an image of RAM, which is known as vmcore and can be used to debug and determine the cause of the crash.
Configuring kdump
- Run the following command to check whether kexec-tools is installed:
If it is not installed, run the following command to install it:
yum install -y kexec-tools
- Run the following command to enable kdump to run at system startup:
- Configure the parameters for the crash kernel to reserve the memory for the capture kernel.
Check whether the parameters are configured.
grep crashkernel /proc/cmdline
If the command output is displayed, this parameter has been configured.
Edit the /etc/default/grub file to configure the following parameters:GRUB_TIMEOUT=5 GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel00/root rd.lvm.lv=rhel00/swap rhgb quiet" GRUB_DISABLE_RECOVERY="true"
Locate parameter GRUB_CMDLINE_LINUX and add crashkernel=auto after it.
- Run the following command for the configuration to take effect:
- Open the /etc/kdump.conf file, locate parameter path, and add /var/crash after it.
path /var/crash
By default, the file is saved in the /var/crash directory.
You can save the file to another directory, for example, /home/kdump. Then add /home/kdump after parameter path:path /home/kdump
There must be enough space in the specified path for storing the vmcore file. It is recommended that the available space be greater than or equal to the RAM size. You can also store the vmcore file on a shared device such as SAN or NFS.
- Set the vmcore dump level.
Add the following content to file /etc/kdump.conf. If the content already exists, skip this step.
core_collector makedumpfile -d 31 -c
where
-c indicates compressing the vmcore file.
-d indicates leaving out irrelevant data. Generally, the value following -d is 31, which is calculated based on the following values. You can adjust the value if needed.
zero pages = 1 cache pages = 2 cache private = 4 user pages = 8 free pages = 16
- Run the following command to restart the system for the configurations to take effect:
Checking Whether kdump Configurations Have Taken Effect
- Run the following command and check whether crashkernel=auto is displayed:
cat /proc/cmdline |grep crashkernel
BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.44.5.10.h142.x86_64 root=UUID=6407d6ac-c761-43cc-a9dd-1383de3fc995 ro crash_kexec_post_notifiers softlockup_panic=1 panic=3 reserve_kbox_mem=16M nmi_watchdog=1 rd.shell=0 fsck.mode=auto fsck.repair=yes net.ifnames=0 spectre_v2=off nopti noibrs noibpb crashkernel=auto LANG=en_US.UTF-8
- Run the following command and check whether the configuration in the output is correct:
grep core_collector /etc/kdump.conf |grep -v ^"#"
core_collector makedumpfile -l --message-level 1 -d 31
- Run the following command and check whether the path configuration in the output is correct:
grep path /etc/kdump.conf |grep -v ^"#"
path /var/crash
- Run the following command and check whether the value of Active in the output is active (exited):
systemctl status kdump
● kdump.service - Crash recovery kernel arming Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; vendor preset: enabled) Active: active (exited) since Tue 2019-04-09 19:30:24 CST; 8min ago Process: 495 ExecStart=/usr/bin/kdumpctl start (code=exited, status=0/SUCCESS) Main PID: 495 (code=exited, status=0/SUCCESS) CGroup: /system.slice/system-hostos.slice/kdump.service
- Run the following test command:
After the command is executed, kdump will be triggered, the system will be restarted, and the generated vmcore file will be saved to the path specified by path.
- Run the following command to check whether the vmcore file has been generated in the specified path, for example, /var/crash/:
ll /var/crash/
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot
