How Do I Detect and Resolve BigKey and HotKey Issues?

BigKey Issue

• Possible causes

  The main cause of the BigKey issue is that the primary key design is improper. As a result, a single partition contains too many records or data. Once a partition becomes extremely large, the access to this partition increases the load of the server where the partition is located, and even causes the out of memory (OOM) issue.

• Troubleshooting

  You can use either of the following methods to rectify BigKey issues:

  • Add caches and optimize the table structure.
  • Add a new partition key for hashing data. Split data to avoid too much data in a single partition.
• Check method

  You can specify a threshold based on your service requirement. If any threshold is exceeded, a BigKey is generated.

  1. The number of rows of a single partition key cannot exceed 100,000.
  2. The size of a single partition cannot exceed 100 MB.

  GeminiDB Cassandra supports BigKey detection and alarms. On the Cloud Eye console, you can configure BigKey alarms for instances. For details, see Configuring Alarm Rules.

  When a BigKey event occurs, the system sends a warning notification immediately. You can view the event data on the Cloud Eye page and handle the event in a timely manner to prevent service fluctuation.

  Figure 1 BigKey alarm
  

  The alarm is described as follows:

  [
    {
      "partition_size": "1008293497",             //Total size of oversized partition keys
      "timestamp": "2021-09-08 07:08:18,240",      //Time when a BigKey is generated
      "partition_num": "676826",         //Total number of rows for oversized partition keys
      "keyspace_name": "ssss",      //keyspace name
      "table_name": "zzzz",        //Table name
      "table_id": "024a1070-0064-11eb-bdf3-d3fe5956183b",      //Table ID
      "partition_key": "{vin=TESTW3YWZD2021003}"      //Partition key
    }
  ]

• Common cases and solutions

  Case 1: The data volume of a cluster is too large. As a result, the cluster has large partition keys (more than 2,000 partition keys are checked), and the maximum size of a partition key has reached 38 GB. When services frequently access these large partition keys, the node load remains high, affecting the service request success rate.

  The table structure is designed as follows.

  

  Table design analysis:

  The movie table stores information about short videos. The partition key is movieid, and stores user information (uid). If movieid is a popular short video and tens of millions or even hundreds of millions of users like this short video, the size of the partition where the short video is located is large (38 GB).

  Solution:

  To solve the problem, perform the following steps:

  1. Optimize the table structure.

     Create a table to store the short video information. Only public short video information is retained, and user information is not included. This ensures that the table does not generate large partition keys. Write the short video information to the table.

     

  2. Add caches.

     A service application first reads popular file information from the cache. If no information is found, the service application queries the database to reduce the number of database query times.

     The overall optimization logic is as follows:

     

     1. The service applications query the cache first. If the data to be queried already exists in the cache, the results are directly returned.
     2. If the data is not in the cache, the popular video cache, the hot table, and the hotmovieaccess table will be accessed in sequence.
     3. If the hotmovieaccess table contains the results, the results are directly returned. If the hotmovieaccess table does not contain any record, the movie table is queried.
     4. Cache the query results.

  Case 2: The movie_meta table is created by month, and each table stores only the data of the current month. The initial design can reduce or avoid large partition keys. Due to frequent service writes, a large number of popular video records are stored, generating large partitions.

  

  Solution:

  A random number (0 to 999) is added to the new partition key. The information stored in the original partition is randomly and discretely stored to 1,000 partitions. After the new partition key is used, no new partition key whose size exceeds 100 MB is formed. The old partition key data whose size exceeds 100 MB expires as time goes by.

HotKey Problem

• Hazards of HotKey:

  In daily life, when the hot news is clicked, viewed, and commented for tens of thousands of times in an application, large number of requests will be generated. In this case, the same key is frequently accessed within a short period of time. As a result, the CPU usage and load of the node where the key is located suddenly increase, affecting other requests on the node and decreasing the service success rate. Such scenarios include promotion of popular products and Internet celebrity live streaming. In these read-intensive scenarios, HotKey issues will be generated.

  The HotKey issue has the following impacts:

  1. The traffic is centralized and reaches the upper limit of the physical NICs.
  2. Too many requests may cause the cache service to break down.
  3. The database breaks down, causing service avalanche.
• Troubleshooting
  To solve the HotKey issue, perform the following steps:

  1. HotKeys must be considered in design to prevent them from being generated in a database.
  2. Add caches in the service side to reduce HotKey issues. Multi-level cache should be used to solve the HotKey issue (such as Redis + local level-2 cache).
  3. Disable hotspot keys. For example, configure a whitelist for HotKeys on the service side to shield HotKeys as required.
• Check method

  A key whose access frequency is greater than 100,000 times per minute is defined as a HotKey.

  HotKey events are classified into the following types: One is the Writes event, indicating a write hotspot, and the other is the Reads event, indicating a read hotspot.

  GeminiDB Cassandra provides HotKey monitoring and alarms. On the Cloud Eye console, you can configure HotKey alarms for instances. For details, see Configuring Alarm Rules.

  When a BigKey event occurs, the system sends a warning notification immediately. You can view the event data on the Cloud Eye page and handle the event in a timely manner to prevent service fluctuation.

  Figure 2 HotKey alarm
  

  HotKey alarm description:

  {
    "sampler_type": "WRITES",         //Sampling type. The value can be WRITES or READS. WRITES indicates write, and READS indicates read.
    "partition_num": "2969",          //Hotspot times of a partition key
    "keyspace_name": "performance",   //Keyspace name
    "table_id": "a10f3bb0-3626-11ec-bbdf-63e05bbb4391",   //Table ID
    "table_name": "stresstable",   //Table name
    "partition_key": "85897376"   //The value of the hotspot partition key.
  }

Summary

If you use Cassandra for online services, you must follow related rules to minimize risks in the development and design phase. Proper design can reduce the probability of most risks.

• The design of any table must consider whether HotKey or BigKey will be generated and whether load skew will occur.
• A data expiration mechanism must be established. Data in a table cannot increase indefinitely without being deleted or expired.
• In read-intensive scenarios, a cache mechanism needs to be added to handle read hotspots and improve query performance.
• A threshold must be set for each primary key and row. Otherwise, the database performance and stability will be affected. If the threshold is exceeded, optimize the settings in a timely manner.