How to Use
You can use A-Tune through the CLI client atune-adm. This section describes how to use the A-Tune client.
Overview
- To use A-Tune, you must have the root permissions.
- You can run atune-adm help/--help/-h to query atune-adm commands.
- define, update, undefine, collection, train, and upgrade cannot be executed remotely.
- In the command syntax, a parameter enclosed within square brackets ([]) is optional, and a parameter enclosed within angle brackets (<>) is mandatory.
Querying Workload Types
list
Description
This command is used to query the profiles supported by the system and the profiles in active state.
Command format
atune-adm list
Example
# atune-adm list Support profiles: +------------------------------------------------+-----------+ | ProfileName | Active | +================================================+===========+ | arm-native-android-container-robox | false | +------------------------------------------------+-----------+ | basic-test-suite-euleros-baseline-fio | false | +------------------------------------------------+-----------+ | basic-test-suite-euleros-baseline-lmbench | false | +------------------------------------------------+-----------+ | basic-test-suite-euleros-baseline-netperf | false | +------------------------------------------------+-----------+ | basic-test-suite-euleros-baseline-stream | false | +------------------------------------------------+-----------+ | basic-test-suite-euleros-baseline-unixbench | false | +------------------------------------------------+-----------+ | basic-test-suite-speccpu-speccpu2006 | false | +------------------------------------------------+-----------+ | basic-test-suite-specjbb-specjbb2015 | false | +------------------------------------------------+-----------+ | big-data-hadoop-hdfs-dfsio-hdd | false | +------------------------------------------------+-----------+ | big-data-hadoop-hdfs-dfsio-ssd | false | +------------------------------------------------+-----------+ | big-data-hadoop-spark-bayesian | false | +------------------------------------------------+-----------+ | big-data-hadoop-spark-kmeans | false | +------------------------------------------------+-----------+ | big-data-hadoop-spark-sql1 | false | +------------------------------------------------+-----------+ | big-data-hadoop-spark-sql10 | false | +------------------------------------------------+-----------+ | big-data-hadoop-spark-sql2 | false | +------------------------------------------------+-----------+ | big-data-hadoop-spark-sql3 | false | +------------------------------------------------+-----------+ | big-data-hadoop-spark-sql4 | false | +------------------------------------------------+-----------+ | big-data-hadoop-spark-sql5 | false | +------------------------------------------------+-----------+ | big-data-hadoop-spark-sql6 | false | +------------------------------------------------+-----------+ | big-data-hadoop-spark-sql7 | false | +------------------------------------------------+-----------+ | big-data-hadoop-spark-sql8 | false | +------------------------------------------------+-----------+ | big-data-hadoop-spark-sql9 | false | +------------------------------------------------+-----------+ | big-data-hadoop-spark-tersort | false | +------------------------------------------------+-----------+ | big-data-hadoop-spark-wordcount | false | +------------------------------------------------+-----------+ | cloud-compute-kvm-host | false | +------------------------------------------------+-----------+ | database-mariadb-2p-tpcc-c3 | false | +------------------------------------------------+-----------+ | database-mariadb-4p-tpcc-c3 | false | +------------------------------------------------+-----------+ | database-mongodb-2p-sysbench | false | +------------------------------------------------+-----------+ | database-mysql-2p-sysbench-hdd | false | +------------------------------------------------+-----------+ | database-mysql-2p-sysbench-ssd | false | +------------------------------------------------+-----------+ | database-postgresql-2p-sysbench-hdd | false | +------------------------------------------------+-----------+ | database-postgresql-2p-sysbench-ssd | false | +------------------------------------------------+-----------+ | default-default | false | +------------------------------------------------+-----------+ | docker-mariadb-2p-tpcc-c3 | false | +------------------------------------------------+-----------+ | docker-mariadb-4p-tpcc-c3 | false | +------------------------------------------------+-----------+ | hpc-gatk4-human-genome | false | +------------------------------------------------+-----------+ | in-memory-database-redis-redis-benchmark | false | +------------------------------------------------+-----------+ | middleware-dubbo-dubbo-benchmark | false | +------------------------------------------------+-----------+ | storage-ceph-vdbench-hdd | false | +------------------------------------------------+-----------+ | storage-ceph-vdbench-ssd | false | +------------------------------------------------+-----------+ | virtualization-consumer-cloud-olc | false | +------------------------------------------------+-----------+ | virtualization-mariadb-2p-tpcc-c3 | false | +------------------------------------------------+-----------+ | virtualization-mariadb-4p-tpcc-c3 | false | +------------------------------------------------+-----------+ | web-apache-traffic-server-spirent-pingpo | false | +------------------------------------------------+-----------+ | web-nginx-http-long-connection | true | +------------------------------------------------+-----------+ | web-nginx-https-short-connection | false | +------------------------------------------------+-----------+
If the value of Active is true, the profile is activated. In this example, the activated profile is web-nginx-http-long-connection.
Analyzing Workload Types and Performing Automated Tuning
analysis
Description
This command is used to collect real-time statistics of the system to identify workload types and then perform automated tuning.
Command format
atune-adm analysis [OPTIONS]
Parameters
- OPTIONS
Parameter |
Description |
---|---|
--model, -m |
New model trained by a user |
--characterization, -c |
Application identification using the default model, without automated tuning |
--times value, -t value |
Data collection duration |
--script value, -s value |
File to be executed |
Example
- Use the default model for application identification.
# atune-adm analysis --characterization
- Use the default model for application identification and perform automated tuning.
# atune-adm analysis
- Use a user-trained model for application identification.
# atune-adm analysis --model /usr/libexec/atuned/analysis/models/new-model.m
Custom Models
A-Tune allows users to define a new model for learning. To define a new model, perform the following steps:
- Run define to define a profile for a new application.
- Run collection to collect system data of the application.
- Run train to train a model.
define
Description
This command is used to add a user-defined application scenario and tuning items to a profile.
Command format
atune-adm define <service_type> <application_name> <scenario_name> <profile_path>
Example
Add a profile. Set service_type to test_service, application_name to test_app, scenario_name to test_scenario, and tuning item configuration file to example.conf.
# atune-adm define test_service test_app test_scenario ./example.conf
You can write the example.conf file as below (the tuning items are optional and for reference only). Alternatively, run atune-adm info to see how existing profiles are written.
[main] # list its parent profile [kernel_config] # to change the kernel config [bios] # to change the bios config [bootloader.grub2] # to change the grub2 config [sysfs] # to change the /sys/* config [systemctl] # to change the system service status [sysctl] # to change the /proc/sys/* config [script] # the script extension of cpi [ulimit] # to change the resources limit of user [schedule_policy] # to change the schedule policy [check] # check the environment [tip] # the recommended optimization, which should be performed manunaly
collection
Description
This command is used to collect resource usages of the entire system and the OS status when services are running and save the collected information into a CSV file as the input dataset for model training.
Note:
- This command depends on perf, mpstat, vmstat, iostat, and sar.
- Currently, only Kunpeng 920 is supported. You can run dmidecode -t processor to check the CPU model.
Command format
atune-adm collection <OPTIONS>
Parameters
- OPTIONS
Parameter
Description
--filename, -f
Name of the generated CSV file used for training (format: Name-Timestamp.csv)
--output_path, -o
Absolute path for storing the generated CSV file
--disk, -b
Disks used for running services, for example, /dev/sda
--network, -n
Network interface used for running services, for example, eth0
--app_type, -t
Application type of a service, which is used as a label during training
--duration, -d
Data collection duration when services are running, in seconds (default: 1,200s)
--interval, -i
Interval for collecting data, in seconds (default: 5s)
Example
# atune-adm collection --filename name --interval 5 --duration 1200 --output_path /home/data --disk sda --network eth0 --app_type test_service-test_app-test_scenario
Note:
In the example, data is collected every 5s, and the collection lasts for 1,200s. The collected data is stored in the name file in the /home/data directory. The service application type is specified by atune-adm define and the value is test_service-test_app-test_scenario.
train
Description
This command is used to use the collected data to train a model. Use data of at least two application types for training. Otherwise, an error may occur during training.
Command format
atune-adm train <OPTIONS>
Parameters
- OPTIONS
Parameter
Description
--data_path, -d
Directory for storing the CSV file required for model training
--output_file, -o
Trained model
Example
Use the CSV file in the data directory as the training input and save the generated model new-model.m in the model directory.
# atune-adm train --data_path /home/data --output_file /usr/libexec/atuned/analysis/models/new-model.m
undefine
Description
This command is used to delete a user-defined profile.
Command format
atune-adm undefine <profile>
Example
Delete a user-defined profile.
# atune-adm undefine test_service-test_app-test_scenario
info
Description
This command is used to view information about a profile.
Command format
atune-adm info <profile>
Example
View the web-nginx-http-long-connection profile.
# atune-adm info web-nginx-http-long-connection *** web-nginx-http-long-connection: # # nginx http long connection A-Tune configuration # [main] include = default-default [kernel_config] #TODO CONFIG [bios] #TODO CONFIG [bootloader.grub2] iommu.passthrough = 1 [sysfs] #TODO CONFIG [systemctl] sysmonitor = stop irqbalance = stop [sysctl] fs.file-max = 6553600 fs.suid_dumpable = 1 fs.aio-max-nr = 1048576 kernel.shmmax = 68719476736 kernel.shmall = 4294967296 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_syncookies = 1 net.ipv4.ip_local_port_range = 1024 65500 net.ipv4.tcp_max_tw_buckets = 5000 net.core.somaxconn = 65535 net.core.netdev_max_backlog = 262144 net.ipv4.tcp_max_orphans = 262144 net.ipv4.tcp_max_syn_backlog = 262144 net.ipv4.tcp_timestamps = 0 net.ipv4.tcp_synack_retries = 1 net.ipv4.tcp_syn_retries = 1 net.ipv4.tcp_fin_timeout = 1 net.ipv4.tcp_keepalive_time = 60 net.ipv4.tcp_mem = 362619 483495 725238 net.ipv4.tcp_rmem = 4096 87380 6291456 net.ipv4.tcp_wmem = 4096 16384 4194304 net.core.wmem_default = 8388608 net.core.rmem_default = 8388608 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 [script] prefetch = off ethtool = -X {network} hfunc toeplitz [ulimit] {user}.hard.nofile = 102400 {user}.soft.nofile = 102400 [schedule_policy] #TODO CONFIG [check] #TODO CONFIG [tip] SELinux provides extra control and security features to linux kernel. Disabling SELinux will improve the performance but may cause security risks. = kernel disable the nginx log = application
Updating a Profile
You can update a profile as needed.
update
Description
This command is used to update the tuning items in an existing profile to those in the new.conf file.
Command format
atune-adm update <profile> <profile_path>
Example
Update the tuning items in the test_service-test_app-test_scenario profile to those in the new.conf file.
# atune-adm update test_service-test_app-test_scenario ./new.conf
Activating a Profile
profile
Description
This command is used to manually activate a profile.
Command format
atune-adm profile <profile>
Description
For the profile name, see the query result of the list command.
Example
Activate the web-nginx-http-long-connection profile.
# atune-adm profile web-nginx-http-long-connection
Rolling Back a Profile
rollback
Description
This command is used to roll back a profile to the initial settings.
Command format
atune-adm rollback
Example
# atune-adm rollback
Updating the Database
upgrade
Description
This command is used to update the system database.
Command format
atune-adm upgrade <DB_FILE>
Description
Example
Update the database to new_sqlite.db.
# atune-adm upgrade ./new_sqlite.db
Querying System Information
check
Description
This command is used to query system information such as CPU, BIOS, OS, and NICs.
Command format
atune-adm check
# atune-adm check cpu information: cpu:0 version: Kunpeng 920-6426 speed: 2600000000 HZ cores: 64 cpu:1 version: Kunpeng 920-6426 speed: 2600000000 HZ cores: 64 system information: DMIBIOSVersion: 20.47 OSRelease: 5.10.0-182.0.0.95.r2055_140.hce2.aarch64 network information: name: eth0 product: HNS GE/10GE/25GE RDMA Network Controller name: eth1 product: HNS GE/10GE/25GE Network Controller name: eth2 product: HNS GE/10GE/25GE RDMA Network Controller name: eth3 product: HNS GE/10GE/25GE Network Controller name: eth4 product: HNS GE/10GE/25GE RDMA Network Controller name: eth5 product: HNS GE/10GE/25GE Network Controller name: eth6 product: HNS GE/10GE/25GE RDMA Network Controller name: eth7 product: HNS GE/10GE/25GE Network Controller
Performing Automated Parameter Tuning
A-Tune automatically searches for the optimal settings. This saves time and efforts by eliminating the need of manual adjustments and performance evaluations.
tuning
Description
This command is used to specify a project file to dynamically search for the optimal settings for the current environment.
Command format
Before running the command, ensure that the following conditions are met:
- The YAML configuration file of the A-Tune server has been edited and stored in the **/etc/atuned/tuning/** directory of the server.
- The YAML configuration file of the A-Tune client has been edited and stored in any directory of the client.
atune-adm tuning [OPTIONS] <PROJECT_YAML>
Parameters
- OPTIONS
Parameter
Description
--restore, -r
Initial settings before tuning
--project, -p
Name of the project to be restored in the YAML file
--restart, -c
Tuning based on historical tuning results
--detail, -d
Details about the tuning process
-p must be followed by a specific project name. The YAML file of the project must also be specified.
- PROJECT_YAML: YAML configuration file on the client
Configuration
Parameter |
Description |
Type |
Value |
---|---|---|---|
project |
Project name. |
String |
- |
startworkload |
Script for starting the service to be tuned. |
String |
- |
stopworkload |
Script for stopping the service to be tuned. |
String |
- |
maxiterations |
Maximum number of tuning iterations, which is used to limit the number of iterations on the client. Generally, more iterations can lead to better tuning results, but they also require more time. You can configure it based on service requirements. |
Integer |
>10 |
object |
Parameters to be adjusted and related information. For details, see Table 2. |
- |
- |
Parameter |
Description |
Type |
Value |
---|---|---|---|
name |
Name of the parameter to be tuned. |
String |
- |
desc |
Description of the parameter to be tuned. |
String |
- |
get |
Script for querying the parameter value. |
- |
- |
set |
Script for setting the parameter value. |
- |
- |
needrestart |
Whether to restart the service for the parameter to take effect. |
Enumerated |
"true", "false" |
type |
Parameter type, which can be discrete or continuous. |
Enumerated |
"discrete", "continuous" |
dtype |
This parameter needs to be configured only when type is set to discrete. The value can be int, float, and string. |
Enumerated |
"int","float","string" |
scope |
Parameter value range. This parameter is valid only when type is set to discrete and dtype is set to int or float, or when type is set to continuous. |
Integer/Float |
User-defined. It specifies the valid value range of the tunned parameter. |
step |
Parameter value step, which is used when dtype is set to int or float. |
Integer/Float |
User-defined |
items |
Enumerated parameter values beyond the range defined by scope. It is used when dtype is set to int or float. |
Integer/Float |
User-defined. It specifies the valid value range of the tunned parameter. |
options |
Enumerated value range of the tunned parameter. It is used when dtype is set to string. |
String |
User-defined. It specifies the valid value range of the tunned parameter. |
Parameter |
Description |
Type |
Value |
---|---|---|---|
project |
Project name, which must be the same as that in the configuration file of the server. |
String |
- |
engine |
Tunning algorithm |
String |
"random", "forest", "gbrt", "bayes", "extraTrees" |
iterations |
Tuning iterations. |
Integer |
>=10 |
random_starts |
Number of random iterations. |
Integer |
<iterations |
feature_filter_engine |
(Optional) Parameter search algorithm. It is used to select important parameters. |
String |
"lhs" |
feature_filter_cycle |
Number of parameter search rounds. It is used to select important parameters and must be used together with feature_filter_engine. |
Integer |
- |
feature_filter_iters |
Number of iterations for each cycle of parameter search. It is used to select important parameters and must be used together with feature_filter_engine. |
Integer |
- |
split_count |
Number of evenly selected parameters in the value range of the tuned parameters. It is used to select important parameters and must be used together with feature_filter_engine. |
Integer |
- |
benchmark |
Performance test script. |
- |
- |
evaluations |
Performance evaluation metrics. For details, see Table 4. |
- |
- |
Parameter |
Description |
Type |
Value |
---|---|---|---|
name |
Evaluation metric name. |
String |
- |
get |
Script for obtaining performance evaluation results. |
- |
- |
type |
How evaluation results are judged. For positive, a larger number indicates better performance. For negative, a smaller number indicates better performance. |
Enumerated |
"positive","negative" |
weight |
Metric weight. |
Integer |
0-100 |
threshold |
Minimum performance required by a metric. |
Integer |
User-defined |
Configuration examples
YAML configuration file on the server:
project: "compress" maxiterations: 500 startworkload: "" stopworkload: "" object : - name : "compressLevel" info : desc : "The compresslevel parameter is an integer from 1 to 9 controlling the level of compression" get : "cat /root/A-Tune/examples/tuning/compress/compress.py | grep 'compressLevel=' | awk -F '=' '{print $2}'" set : "sed -i 's/compressLevel=\\s*[0-9]*/compressLevel=$value/g' /root/A-Tune/examples/tuning/compress/compress.py" needrestart : "false" type : "continuous" scope : - 1 - 9 dtype : "int" - name : "compressMethod" info : desc : "The compressMethod parameter is a string controlling the compression method" get : "cat /root/A-Tune/examples/tuning/compress/compress.py | grep 'compressMethod=' | awk -F '=' '{print $2}' | sed 's/\"//g'" set : "sed -i 's/compressMethod=\\s*[0-9,a-z,\"]*/compressMethod=\"$value\"/g' /root/A-Tune/examples/tuning/compress/compress.py" needrestart : "false" type : "discrete" options : - "bz2" - "zlib" - "gzip" dtype : "string"
YAML configuration file on the client:
project: "compress" engine : "gbrt" iterations : 20 random_starts : 10 benchmark : "python3 /root/A-Tune/examples/tuning/compress/compress.py" evaluations : - name: "time" info: get: "echo '$out' | grep 'time' | awk '{print $3}'" type: "positive" weight: 20 - name: "compress_ratio" info: get: "echo '$out' | grep 'compress_ratio' | awk '{print $3}'" type: "negative" weight: 80
Usage examples
- Download test data.
wget http://cs.fit.edu/~mmahoney/compression/enwik8.zip
- Create a prepare.sh file for preparing the tunning environment.
#!/usr/bin/bash if [ "$#" -ne 1 ]; then echo "USAGE: $0 the path of enwik8.zip" exit 1 fi path=$( cd "$(dirname "$0")" pwd ) echo "unzip enwik8.zip" unzip "$path"/enwik8.zip echo "set FILE_PATH to the path of enwik8 in compress.py" sed -i "s#compress/enwik8#$path/enwik8#g" "$path"/compress.py echo "update the client and server yaml files" sed -i "s#python3 .*compress.py#python3 $path/compress.py#g" "$path"/compress_client.yaml sed -i "s# compress/compress.py# $path/compress.py#g" "$path"/compress_server.yaml echo "copy the server yaml file to /etc/atuned/tuning/" cp "$path"/compress_server.yaml /etc/atuned/tuning/
Run a script.sh prepare.sh enwik8.zip
- Perform tuning.
atune-adm tuning --project compress --detail compress_client.yaml
- Restore to the initial settings before the tuning. compress is the project name in the YAML file.
atune-adm tuning --restore --project compress
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot