Pre-Caching ClickHouse Metadata to the Memory
Scenarios
RocksDB is a high performance embedded database for key-value data. It is widely used in scenarios that require fast data read and write. ClickHouse leverages RocksDB to cache metadata, greatly accelerating metadata loading.
When there are many ClickHouse service tables and large volumes of table data, metadata loading during a rolling restart may take a considerable amount of time. To speed up metadata loading, you can use RocksDB to pre-cache metadata in memory. This reduces startup time and enhances query performance.
Notes and Constraints
This section applies only to MRS 3.3.1-LTS or later.
Enabling Metadata Pre-Caching
You can set use_metadata_cache to 1 or true to cache metadata to the memory through RocksDB.
- Install the ClickHouse client.
For details, see Installing a Client.
- Use the ClickHouse client to connect to the ClickHouse server by referring to ClickHouse Client Practices.
- Enable metadata pre-caching.
- Enable metadata pre-caching for historical tables.
ALTER TABLE <table name> MODIFY SETTING use_metadata_cache=1;
or
ALTER TABLE <table name> MODIFY SETTING use_metadata_cache=true;
- Enable metadata pre-caching when you create a table.
CREATE TABLE <table name> ( `x` UInt32, `y` UInt32, `z` UInt32, `t` UInt32 ) ENGINE = MergeTree PARTITION BY x % 10 ORDER BY (x, y) SETTINGS index_granularity = 8192, use_metadata_cache = 1
or
CREATE TABLE <table name> ( `x` UInt32, `y` UInt32, `z` UInt32, `t` UInt32 ) ENGINE = MergeTree PARTITION BY x % 10 ORDER BY (x, y) SETTINGS index_granularity = 8192, use_metadata_cache = true
- Enable metadata pre-caching for historical tables.
Parameter Tuning
Metadata pre-caching can be optimized as follows:
Log in to FusionInsight Manager, choose Cluster > Services > ClickHouse, click Configurations and then All Configurations, and modify the following parameters.
Parameter |
Description |
Example Value |
---|---|---|
merge_tree_metadata_cache.continue_if_corrupted |
If the local RocksDB directory fails to be read, you can either exit the process or clear dirty data.
|
true |
merge_tree_metadata_cache.lru_cache_size |
Size of the LRU in the RocksDB instance used to cache part metadata. Value 0 indicates that there is no limit in the size. |
1 GB |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot