Storing Hive Table Partitions to OBS and HDFS

Updated on 2024-10-25 GMT+08:00

View PDF

Scenario

In the scenario where storage and compute resources are separated, you can specify different storage sources, for example, OBS or HDFS, for partitions in a Hive partitioned table.

NOTE:

This feature applies only to MRS 3.2.0 or later. This section describes the capability of specifying storage sources for partitioned tables. For details about how to connect Hive to OBS in the storage-compute decoupling scenario, see Interconnecting Hive with OBS.

Prerequisites

The Hive client has been installed.

Example

Log in to the node where the Hive client is installed as the Hive client installation users.
Run the following command to go to the client installation directory:
cd Client installation directory

For example, if the client installation directory is /opt/client, run the following command:

cd /opt/client
Run the following command to configure environment variables:
source bigdata_env
Check whether the cluster authentication mode is in security mode.
- If yes, run the following command to authenticate the user:
  kinit Hive service user
- If no, go to 5.
Run the following command to log in to the Hive client:
beeline
Run the following commands to create a Hive partitioned table named table_1, and set the path of partitions pt=?2021-12-12 and pt='2021-12-18 to hdfs//xxx and obs://xxx respectively:
create table table_1(id string) partitioned by(pt string) [stored as [orc|textfile|parquet|...]];

alter table table_1 add partition(pt='2021-12-12') location 'hdfs://xxx';

alter table table_1 add partition(pt='2021-12-18') location 'obs://xxx';
After data is inserted into table_1, it is stored in the corresponding storage source. You can run the desc command to view the location of each partition.
desc formatted table_1 partition(pt='2021-12-18');

Parent topic: Enterprise-Class Enhancements of Hive

Previous topic: Enterprise-Class Enhancements of Hive

Next topic: Configuring Automatic Removal of Old Data in the Hive Directory to the Recycle Bin

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot