Redis-based CacheStore of HiveMetaStore

Updated on 2022-12-14 GMT+08:00

View PDF

Scenario

The MetaStore service of Hive can cache the metadata of some tables in Redis.

Prerequisites

The Redis service has been installed in a cluster.

Configure Parameters Related to Metastore

Log in to FusionInsight Manager. For details, see Accessing FusionInsight Manager. Choose Cluster > Name of the desired cluster > Services > Hive > Configurations > All Configurations > MetaStore(Role) > Customization. Modify the following parameters to interconnect the cache of MetaStore to the Redis service.

**Table 1** Parameters
Parameter	Value	Description
hive.metastore.rawstore.impl	org.apache.hadoop.hive.metastore.cache.redis.RedisCachedStore	(Mandatory) Implementation class of CachedStore. Use the customized RedisCachedStore.
redis.cluster.host.and.port	xxx.xxx.xxx.xxx:22400;xxx.xxx.xxx.xxx.xxx:22401	(Mandatory) IP address and port number of any node in the Redis cluster. The format is ip:port. ip:port. The value cannot end with a semicolon (;).
metastore.cached.rawstore.cached.object.whitelist	catalog.database.table,catalog.database.table	(Optional) Cache table whitelist. The configured tables are cached to Redis. Multiple tables are separated by commas (,). The default separator is ., that is, all tables are cached. NOTE: The table name consists of catalog.database.table. The default catalog is hive*.
metastore.cached.rawstore.cached.object.blacklist	catalog.database.table,catalog.database.table	(Optional) Cache table blacklist. Tables that are not configured in the blacklist are not cached in Redis. Multiple tables are separated by commas (,). By default, this parameter is left blank. NOTE: The table name consists of catalog.database.table. The default catalog is hive.
redis.cache.prewarm.cron	cronTab expression, for example, 0 0 16 * *?	(Optional) Periodically execute the corn expression of prewarm to update the data cached in the metabase to redisCache for synchronization.
metastore.cached.rawstore.catalogs	hive	(Optional) Catalog to be cached. The default value is hive.
jedis.pool.max.wait.mills	30,000	(Optional) Obtain the Redis connection timeout interval. In security mode, the timeout interval can be longer. The unit is ms. The default value is 30,000 ms.
jedis.pool.max.idle	200	(Optional) Maximum number of idle connections in the Jedis connection pool. You are advised to set this parameter to the value of max.total. The default value is 200.
jedis.pool.max.total	200	(Optional) Maximum number of connections in the Jedis connection pool. The default value is 200.
redis.security.enabled	true or false	(Optional) Whether to enable the Redis cache security mode. The default value is true, indicating that the security mode is enabled.

NOTE:

If the cluster is installed in non-security mode, choose Cluster > Name of the desired cluster > Services > Redis > Configurations > All Configurations > Redis > Security and check whether the value of REDIS_SECURITY_ENABLED is false. If not, change it to false.Otherwise, the Redis service does not comply with the non-security mode of the current Metastore.

Save the configuration and choose Dashboard > More > Restart Service to restart the Hive service.

Precautions

If Redis is switched back to the native non-cache mode and then switched back after a period of time, the added, deleted, or modified metadata cannot be synchronized to the Redis when the database is used. Therefore, before switching back, you must clear the cache table in the Redis and synchronize the metadata again in either of the following two clearing modes:

Log in to the Redis client and run the flushall command on all Redis nodes.
Log in to the Redis client and run the following commands to change the two Redis identifiers. {hiveServiceName} is the value of HIVE_DEFAULT_GROUP in the Metastore configuration file ENV_VARS. The default value is hive.
set {hiveServiceName}-hive-isRedisAvailable false

del {hiveServiceName}-hive-isCanPrewarm

Parent topic: Using Hive

Previous topic: Connecting Hive with External RDS

Next topic: Hive Materialized View

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot