Configuring the Connection Between Hive and MemArtsCC

Scenario

MemArtsCC stores hotspot data in computing clusters to reduce the required bandwidth on the OBS server. With the local storage of MemArtsCC, hotspot data does not need to be accessed across networks, improving the data read efficiency of Hive. This topic describes how to integrate MemArtsCC into Hive tasks for a system where storage and compute are decoupled.

Hive cannot be interconnected with MemArtsCC for clusters containing both x86 and Arm servers.

Prerequisites

• The Guardian service is running properly, and decoupled storage and compute have been used. For details, see Interconnecting Guardian with OBS.

• Hive has been connected to OBS. For details, see Accessing OBS Using Hive Through Guardian.

Modifying Hive Configurations

1. Log in to FusionInsight Manager and choose Cluster > Services > Hive, click Configurations and then All Configurations, and choose Hive(Service) > OBS.
2. Set fs.obs.readahead.policy to memArtsCC.
3. Click Save. In the displayed dialog box, click OK to save the configuration. Click Dashboard and choose More > Service Rolling Restart to restart the Hive service.

Verifying the Configuration

1. Log in to FusionInsight Manager and choose Cluster > Services > MemArtsCC > Chart > Capacity.
2. View and record the number of shards in the cluster.
3. Log in to the Hive client node, use Beeline to create a table, and ensure that Location is an OBS path. For details, see Accessing OBS Using Hive Through Guardian.

   Run the following statement in Beeline to execute MapReduce tasks:

   select count(*) from tablename;

4. Repeat 1 and 2. If there are more shards in the cluster than there were in 2, the interconnection is successful.