ClickHouse Log Overview

Log Description

Log path: ClickHouse logs are stored in ${BIGDATA_LOG_HOME}/clickhouse by default.

ClickHouse run logs: /var/log/Bigdata/clickhouse/clickhouseServer/ *.log
Balancer run logs: /var/log/Bigdata/clickhouse/balance/*.log
Data migration logs: /var/log/Bigdata/clickhouse/migration/${task_name}/clickhouse-copier_{timestamp}_{processId}/copier.log
ClickHouse audit logs: /var/log/Bigdata/audit/clickhouse/clickhouse-server-audit.log

Log archiving rules:

The automatic compression and archiving function has been enabled for ClickHouse logs. By default, when the size of log files exceeds 100 MB, the log files will be automatically compressed.
The file generated after log files are compressed is named in the format of <Original log name>.[ID].gz.
A maximum of 10 latest compressed files are reserved by default. The number of compressed files can be configured on Manager.

**Table 1** ClickHouse log list
Log Type	Log File Name	Description
ClickHouse log	/var/log/Bigdata/clickhouse/clickhouseServer/clickhouse-server.err.log	Path of ClickHouseServer error log files
	/var/log/Bigdata/clickhouse/clickhouseServer/checkService.log	Path of key ClickHouseServer run log files
	/var/log/Bigdata/clickhouse/clickhouseServer/clickhouse-server.log	Path of key ClickHouseServer run log files
	/var/log/Bigdata/clickhouse/clickhouseServer/ugsync.log	User role synchronization tool log
	/var/log/Bigdata/clickhouse/clickhouseServer/prestart.log	ClickHouse prestart log
	/var/log/Bigdata/clickhouse/clickhouseServer/start.log	ClickHouse startup log
	/var/log/Bigdata/clickhouse/clickhouseServer/checkServiceHealthCheck.log	ClickHouse health check log
	/var/log/Bigdata/clickhouse/clickhouseServer/checkugsync.log	User role synchronization check log
	/var/log/Bigdata/clickhouse/clickhouseServer/checkDisk.log	Path of ClickHouse disk check log files
	/var/log/Bigdata/clickhouse/clickhouseServer/backup.log	Path of log files generated when ClickHouse performs the backup and restoration operations on Manager
	/var/log/Bigdata/clickhouse/clickhouseServer/stop.log	ClickHouse stop log
	/var/log/Bigdata/clickhouse/clickhouseServer/postinstall.log	postinstall.sh script invoking log of ClickHouse
	/var/log/Bigdata/clickhouse/balance/start.log	Path of ClickHouseBalancer startup log files
	/var/log/Bigdata/clickhouse/balance/error.log	Path of ClickHouseBalancer error log files
	/var/log/Bigdata/clickhouse/balance/access_http.log	Path of the HTTP log files generated during ClickHouseBalancer running
	/var/log/Bigdata/clickhouse/balance/access_tcp.log	Path of the TCP log files generated during ClickHouseBalancer running
	/var/log/Bigdata/clickhouse/balance/checkService.log	ClickHouseBalancer service check log
	/var/log/Bigdata/clickhouse/balance/postinstall.log	Invoking log of the postinstall.sh script of ClickHouseBalancer
	/var/log/Bigdata/clickhouse/balance/prestart.log	Path of prestart log files of ClickHouseBalancer
	/var/log/Bigdata/clickhouse/balance/stop.log	Path of stop log files of ClickHouseBalancer
	/var/log/Bigdata/clickhouse/clickhouseServer/auth.log	ClickHouse service authentication log
	/var/log/Bigdata/clickhouse/clickhouseServer/cleanService.log	Log generated when an instance fails to reinstall
	/var/log/Bigdata/clickhouse/clickhouseServer/offline_shard_table_manager.log	ClickHouse recommissioning/decommissioning log
	/var/log/Bigdata/clickhouse/clickhouseServer/traffic_control.log	ClickHouse active/standby DR traffic control log
	/var/log/Bigdata/clickhouse/clickhouseServer/clickhouse_migrate_metadata.log	ClickHouse metadata migration log
	/var/log/Bigdata/clickhouse/clickhouseServer/clickhouse_migrate_data.log	ClickHouse service data migration log
	/var/log/Bigdata/clickhouse/clickhouseServer/changePassword.log	ClickHouse user password change log
	/var/log/coredump/clickhouse-*.core.gz	Compressed package of memory dump files generated after the ClickHouse process breaks down
Data migration log	/var/log/Bigdata/clickhouse/migration/Data migration task name/clickhouse-copier_{timestamp}_{processId}/copier.log	Run log generated when you use the migration tool by referring to Using the ClickHouse Data Migration Tool
	/var/log/Bigdata/clickhouse/migration/Data migration task name/clickhouse-copier_{timestamp}_{processId}/copier.err.log	Error log generated when you use the migration tool by referring to Using the ClickHouse Data Migration Tool
	/var/log/Bigdata/tomcat/clickhouse/auto_balance/Data migration task name/balance_manager.log	Run log generated when one-click balancing is selected by referring to Using the ClickHouse Data Migration Tool
clickhouse-tomcat log	/var/log/Bigdata/tomcat/clickhouse/web_clickhouse.log	ClickHouse custom UI run log
clickhouse-tomcat log	/var/log/Bigdata/tomcat/audit/clickhouse/clickhouse_web_audit.log	Clickhouse data migration audit log
ClickHouse audit log	/var/log/Bigdata/audit/clickhouse/clickhouse-server-audit.log	Path of ClickHouse audit log files

Log Level

Table 2 describes the log levels supported by ClickHouse.

Levels of run logs are error, warning, trace, information, and debug from the highest to the lowest priority. Run logs of equal or higher levels are recorded. The higher the specified log level, the fewer the logs recorded.

**Table 2** Log levels
Level	Description
error	Logs of this level record error information about system running
warning	Logs of this level record exception information about the current event processing
trace	Logs of this level record trace information about the current event processing
information	Logs of this level record normal running status information about the system and events
debug	Logs of this level record system running and debugging information

To modify log levels, perform the following operations:

Log in to FusionInsight Manager.
Choose Cluster > Services > ClickHouse > Configurations.
Select All Configurations.
On the menu bar on the left, select the log menu of the target role.
Select a desired log level.
Click Save. Then, click OK.

The configurations take effect immediately without the need to restart the service.

Log Format

The following table lists the ClickHouse log format:

**Table 3** Log formats
Log Type	Format	Example
ClickHouse run log	<yyyy-MM-dd HH:mm:ss,SSS>\|<Log level>\|<Name of the thread that generates the log>\|<Message in the log>\|<Location where the log event occurs>	2021.02.23 15:26:30.691301 [ 6085 ] {} <Error> DynamicQueryHandler: Code: 516, e.displayText() = DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name, Stack trace (when copying this message, always include the lines below): 0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x1250e59c