HetuEngine Log Overview
Log Description
Log paths
The HetuEngine logs are stored in /var/log/Bigdata/hetuengine/ and /var/log/Bigdata/audit/hetuengine/.
Log archiving rules
- When the size of a single file exceeds the default maximum value, a new compressed archive file is generated. The naming rule of the compressed archive log file is as follows: <Original log name>.[ID].log.gz.
- Log deletion rules
- When the total size of compressed run logs of the HetuEngine compute instance reaches the maximum value, the earliest log file is deleted.
Run logs of HetuEngine compute instances are synchronized to HDFS and retained for 30 days (log.clean.task.expire-time.day) by default. The archive path is hdfs://hacluster/hetuserverhistory/tenant/coordinator.
- When the number of other archived log files reaches the maximum value or the total size of compressed log files reaches the maximum value, the earliest log files are deleted.
- When the total size of compressed run logs of the HetuEngine compute instance reaches the maximum value, the earliest log file is deleted.
By default, the maximum size of an audit log file is 30 MB, and the maximum number of log archive files is 20.
By default, the maximum size of a single run log file is 100 MB, and the maximum number of archived log files is 20. The maximum size of a single HetuEngine compute instance run log file is 100 MB, and logs archived in HDFS are retained for 30 days by default.
To change the maximum size of a single run log file or audit log file or change the maximum number of log archive files of an instance, perform the following operations:
- Log in to Manager.
- Choose Cluster > Services > HetuEngine > Configurations > All Configurations.
- In the parameter list of log levels, search for logback.xml to view the current run log and audit log configurations of HSBroker, HSConsole, HSFabric, and QAS.
Parameters related to HetuEngine compute instance run logs
- log.clean.task.enabled: indicates whether to enable scheduled compute instance log clearing.
- log.clean.task.expire-time.day: indicates the expiration time of compute instance logs archived in HDFS. The default value is 30 days.
- log.max-history: indicates the maximum retention period of compute instance logs locally. The default value is 7 days.
- log.clean.task.schedule.plan: indicates the schedule for automatically clearing compute instance logs. The value is a cron expression. Only a fixed triggering time in a day can be specified.
- log.max-size: indicates the maximum size of a single log file of a HetuEngine compute instance. The default value is 100 MB.
- log.max-total-size: indicates the maximum size of compressed HetuEngine compute instance log files. The default value is 5 GB.
- Select the configuration item to be modified and modify it.
- Click Save, and then click OK. The configuration automatically takes effect after about 30 seconds.
Log Category |
Log File |
Description |
---|---|---|
Installation, startup, and stop log |
prestart.log |
Preprocessing script log before startup |
start.log |
Startup log |
|
stop.log |
Stop log |
|
postinstall.log |
Installation log |
|
Run log |
Instance name.log |
Run log |
Instance name_wsf.log |
Interface parameter verification log |
|
hdfs://hacluster/hetuserverhistory/Tenant/Coordinator or worker/application_ID/container_ID/yyyyMMdd/server.log |
Run log of the HetuEngine compute instance |
|
Status check log |
service_check.log |
Health check log |
service_getstate.log |
Status check log |
|
availability-check.log |
HetuEngine status check log |
|
haCheck.log |
Log generated when the QAS checks the HA status |
|
Audit log |
Instance name-audit.log |
Audit log |
hsbroker-audit.log |
Audit log of HSBroker operations |
|
hsconsole-audit.log |
Audit log of HSConsole operations |
|
hsfabric-audit.log |
Audit log of HetuEngine operations performed across domains |
|
hdfs://hacluster/hetuserverhistory/Tenant/coordinator/application_ID/container_ID/yyyyMMdd/hetuserver-engine-audit.log |
Audit log of the HetuEngine compute instance |
|
queryInfo log |
hdfs://hacluster/hetuserverhistory/Tenant/Coordinator/application_ID/container_ID/yyyyMMdd/queryinfo.log |
queryInfo log of HetuEngine compute instances, which records SQL running statistics. |
Clean log |
cleanup.log |
Script cleanup log |
Initialization log |
hetupg.log |
Metadata initialization log |
ranger-trino-plugin-enable.log |
Operation log of integrating Ranger plug-ins to the HetuEngine kernel |
|
Client log |
qas_client.log |
ZooKeeper client log of the QAS instance |
Stack information log |
threadDump-<DATE>.log |
Log printed when instances are restarted or stopped |
Other |
hetu-updateKrb5.log |
Log generated when the Hive data source configuration is automatically updated after the Hive cluster domain is changed. |
hetu_utils.log |
Log generated when the preprocessing script calls the tool class to upload files to the HDFS during startup. |
Log Level
Table 2 describes the log levels provided by HetuEngine. The priorities of log levels are OFF, ERROR, WARN, INFO, and DEBUG in descending order. Logs whose levels are higher than or equal to the specified level are printed. The number of printed logs decreases as the specified log level increases.
Level |
Description |
---|---|
OFF |
Logs of this level record no logs. |
ERROR |
Logs of this level record error information about the current event processing. |
WARN |
Logs of this level record exception information about the current event processing. |
INFO |
Logs of this level record normal running status information about the system and events. |
DEBUG |
Logs of this level record the system information and system debugging information. |
To change the run log or audit log level of an instance, perform the following steps:
- Log in to FusionInsight Manager.
- Choose Cluster > Services > HetuEngine > Configurations > All Configurations.
- In the parameter list of log levels, search for logback.xml to view the current run log and audit log levels of HSBroker, HSConsole, and HSFabric.
- Select a desired log level.
- Click Save, and then click OK. The configuration automatically takes effect after about 30 seconds.
To change the HetuEngine Coordinator/Worker log level, perform the following steps:
- Log in to FusionInsight Manager.
- Choose Cluster > Services > HetuEngine > Configurations > All Configurations.
- In the parameter list of log levels, search for log.properties to view the current log levels.
- Select a desired log level.
- Click Save, and then click OK. Wait until the operation is successful.
- Choose Cluster > Services > HetuEngine > Instance, click the HSBroker instance in the role list, and choose More > Restart Instance.
- After the HSBroker instance is restarted, choose Cluster > Services > HetuEngine. On the overview page, click the link next to HSConsole WebUI to go to the compute instance page.
- Select the compute instances to be restarted and click Stop. After all instances are stopped, click Start to restart the compute instances.
Log Format
The following table lists the HetuEngine log formats.
Log Type |
Format |
Example |
---|---|---|
Run log |
<yyyy-MM-dd HH:mm:ss,SSS>|<Log level>|<Name of the thread that generates the log>|<Message in the log>|<Location where the log event occurs> |
2024-05-22 06:03:35,696 | INFO | main | Construct zooKeeper helper finished. | com.xxx.hetuserver.hsbroker.core.zookeeper.ZooKeeperClient (ZooKeeperClient.java:312) |
Audit log |
<yyyy-MM-dd HH:mm:ss,SSS>|<Log level>|<Name of the thread that generates the log>| UserName=<User name>UserIP=<User IP>Time=<Event time>Operation=<Operation>Result=<Operation result>Detail=<Details> | xxx |
2024-05-22 14:12:24,967 | INFO | https-jsse-nio-192.168.43.244-29860-exec-10 | UserName=hetuserver/hadoop.eef78bf6_bce3_47ff_b808_ec90ae6f6a2a.com@EEF78BF6_BCE3_47FF_B808_EC90AE6F6A2A.COM UserIP=192.168.43.244 Time=2024-05-22 14:12:24 Operation=Login stmt={kerberos login} Result=SUCCESS Detail=SUCCESS | audit xxx |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot