Updated on 2023-05-15 GMT+08:00

ClickHouse Log Overview

Log Description (MRS 3.2.0 or later)

Log path: ClickHouse logs are stored in ${BIGDATA_LOG_HOME}/clickhouse by default.

  • ClickHouse run logs: /var/log/Bigdata/clickhouse/clickhouseServer/ *.log
  • Balancer run logs: /var/log/Bigdata/clickhouse/balance/*.log
  • Data migration logs: /var/log/Bigdata/clickhouse/migration/${task_name}/clickhouse-copier_{timestamp}_{processId}/copier.log
  • ClickHouse audit logs: /var/log/Bigdata/audit/clickhouse/clickhouse-server.audit.log

Log archiving rules:

  • The automatic compression and archiving function has been enabled for ClickHouse logs. By default, when the size of log files exceeds 100 MB, the log files will be automatically compressed.
  • The file generated after log files are compressed is named in the format of <Original log name>.[ID].gz.
  • A maximum of 10 latest compressed files are reserved by default. The number of compressed files can be configured on Manager.
Table 1 ClickHouse log list

Log Type

Log File Name

Description

ClickHouse log

/var/log/Bigdata/clickhouse/clickhouseServer/clickhouse-server.err.log

Path of ClickHouseServer error log files

/var/log/Bigdata/clickhouse/clickhouseServer/checkService.log

Path of key ClickHouseServer run log files

/var/log/Bigdata/clickhouse/clickhouseServer/clickhouse-server.log

/var/log/Bigdata/clickhouse/clickhouseServer/ugsync.log

User role synchronization tool log

/var/log/Bigdata/clickhouse/clickhouseServer/prestart.log

ClickHouse prestart log

/var/log/Bigdata/clickhouse/clickhouseServer/start.log

ClickHouse startup log

/var/log/Bigdata/clickhouse/clickhouseServer/checkServiceHealthCheck.log

ClickHouse health check log

/var/log/Bigdata/clickhouse/clickhouseServer/checkugsync.log

User role synchronization check log

/var/log/Bigdata/clickhouse/clickhouseServer/checkDisk.log

Path of ClickHouse disk check log files

/var/log/Bigdata/clickhouse/clickhouseServer/backup.log

Path of log files generated when ClickHouse performs the backup and restoration operations on Manager

/var/log/Bigdata/clickhouse/clickhouseServer/stop.log

ClickHouse stop log

/var/log/Bigdata/clickhouse/clickhouseServer/postinstall.log

postinstall.sh script invoking log of ClickHouse

/var/log/Bigdata/clickhouse/balance/start.log

Path of ClickHouseBalancer startup log files

/var/log/Bigdata/clickhouse/balance/error.log

Path of ClickHouseBalancer error log files

/var/log/Bigdata/clickhouse/balance/access_http.log

Path of the HTTP log files generated during ClickHouseBalancer running

/var/log/Bigdata/clickhouse/balance/access_tcp.log

Path of the TCP log files generated during ClickHouseBalancer running

/var/log/Bigdata/clickhouse/balance/checkService.log

ClickHouseBalancer service check log

/var/log/Bigdata/clickhouse/balance/postinstall.log

Invoking log of the postinstall.sh script of ClickHouseBalcer

/var/log/Bigdata/clickhouse/balance/prestart.log

Path of prestart log files of ClickHouseBalancer

/var/log/Bigdata/clickhouse/balance/stop.log

Path of stop log files of ClickHouseBalancer

Data migration log

/var/log/Bigdata/clickhouse/migration/Data migration task name/clickhouse-copier_{timestamp}_{processId}/copier.log

Run log generated when you use the migration tool by referring to Using the ClickHouse Data Migration Tool

/var/log/Bigdata/clickhouse/migration/Data migration task name/clickhouse-copier_{timestamp}_{processId}/copier.err.log

Error log generated when you use the migration tool by referring to Using the ClickHouse Data Migration Tool

clickhouse-tomcat log

/var/log/Bigdata/tomcat/clickhouse/web_clickhouse.log

ClickHouse custom UI run log

/var/log/Bigdata/tomcat/audit/clickhouse/clickhouse_web_audit.log

Clickhouse data migration audit log

Log Description (Versions earlier than MRS 3.2.0)

Log path: The default storage path of ClickHouse log files is as follows: ${BIGDATA_LOG_HOME}/clickhouse

Log archive rule: The automatic log compression and archiving function are enabled. By default, when the size of logs exceeds 100 MB, logs are automatically compressed into a log file named in the format of <Original log file name>.[No.].gz. A maximum of 10 latest compressed files are reserved by default. The number of compressed files can be configured on Manager.

Table 2 ClickHouse log list

Log Type

Log File Name

Description

Run log

/var/log/Bigdata/clickhouse/clickhouseServer/clickhouse-server.err.log

Path of ClickHouseServer error log files

/var/log/Bigdata/clickhouse/clickhouseServer/checkService.log

Path of key ClickHouseServer run log files

/var/log/Bigdata/clickhouse/clickhouseServer/clickhouse-server.log

/var/log/Bigdata/clickhouse/balance/start.log

Path of ClickHouseBalancer startup log files

/var/log/Bigdata/clickhouse/balance/error.log

Path of ClickHouseBalancer error log files

/var/log/Bigdata/clickhouse/balance/access_http.log

Path of ClickHouseBalancer run log files

Data migration log

/var/log/Bigdata/clickhouse/migration/Data migration task name/clickhouse-copier_{timestamp}_{processId}/copier.log

Run log generated when you use the migration tool by referring to Using the ClickHouse Data Migration Tool

/var/log/Bigdata/clickhouse/migration/Data migration task name/clickhouse-copier_{timestamp}_{processId}/copier.err.log

Error log generated when you use the migration tool by referring to Using the ClickHouse Data Migration Tool

Log Level

Table 3 describes the log levels supported by ClickHouse.

Levels of run logs are error, warning, trace, information, and debug from the highest to the lowest priority. Run logs of equal or higher levels are recorded. The higher the specified log level, the fewer the logs recorded.

Table 3 Log levels

Level

Description

error

Logs of this level record error information about system running

warning

Logs of this level record exception information about the current event processing

trace

Logs of this level record trace information about the current event processing

information

Logs of this level record normal running status information about the system and events

debug

Logs of this level record system running and debugging information

To modify log levels, perform the following operations:

  1. Log in to FusionInsight Manager.
  2. Choose Cluster > Services > ClickHouse > Configurations.
  3. Select All Configurations.
  4. On the menu bar on the left, select the log menu of the target role.
  5. Select a desired log level.
  6. Click Save. Then, click OK.

The configurations take effect immediately without the need to restart the service.

Log Format

The following table lists the ClickHouse log format:

Table 4 Log formats

Log Type

Format

Example

ClickHouse run log

<yyyy-MM-dd HH:mm:ss,SSS>|<Log level>|<Name of the thread that generates the log>|<Message in the log>|<Location where the log event occurs>

2021.02.23 15:26:30.691301 [ 6085 ] {} <Error> DynamicQueryHandler: Code: 516, e.displayText() = DB::Exception: default: Authentication failed: password is incorrect or there is no user with such name, Stack trace (when copying this

message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x1250e59c