Help Center/ MapReduce Service/ Component Operation Guide (LTS) (Ankara Region)/ Using MemArtsCC/ Configuring the Connection Between Hive and MemArtsCC
Updated on 2024-11-29 GMT+08:00
View PDF
Share

Configuring the Connection Between Hive and MemArtsCC

Scenario

MemArtsCC stores hotspot data in computing clusters to reduce the required bandwidth on the OBS server. With the local storage of MemArtsCC, hotspot data does not need to be accessed across networks, improving the data read efficiency of Hive. This topic describes how to integrate Hive into HetuEngine tasks for a system where storage and compute are decoupled.

Prerequisites

  • The Guardian service is running properly, and decoupled storage and compute have been used.
  • Hive has been connected to OBS.

Modifying Hive Configurations

  1. Log in to FusionInsight Manager and choose Cluster > Services > Hive, click Configurations and then All Configurations, and choose Hive(Service) > OBS.
  2. Set fs.obs.readahead.policy to memArtsCC.
  3. Click Save. In the displayed dialog box, click OK to save the configuration. Click Dashboard and choose More > Service Rolling Restart to restart the Hive service.

Verifying the Configuration

  1. Log in to FusionInsight Manager and choose Cluster > Services > MemArtsCC > Chart > Capacity.
  2. View and record the number of shards in the cluster.
  3. Log in to the Hive client node, use Beeline to create a table, and ensure that Location is an OBS path.

    Run the following statement in Beeline to execute MapReduce tasks:

    select count(*) from tablename;

  4. Repeat 1 to 2. If there are more shards in the cluster than there were in 2, the interconnection is successful.
Parent topic: Using MemArtsCC