Interconnecting Hive with OBS Using an IAM Agency
After configuring decoupled storage and compute for a cluster by referring to Interconnecting an MRS Cluster with OBS Using an IAM Agency, you can create tables with OBS paths as their location on the Hive client.
Setting the Location to an OBS Path When Creating a Table
- Log in to the client installation node as the client installation user.
- Run the following command to initialize environment variables:
source Client installation directory/bigdata_env
- Authenticate the user of the cluster with Kerberos authentication enabled. (The user must have the permission to perform operations on Hive.) Skip this step for the user of the cluster with Kerberos authentication disabled.
kinit User performing Hive operations
- Log in to FusionInsight Manager of a cluster earlier than MRS 3.2.0-LTS, choose Cluster > Services > Hive, and click Configurations then All Configurations.
In the navigation pane on the left, choose Hive(Service) > Customization. In the custom parameters, add dfs.namenode.acls.enabled to hdfs.site.customized.configs and set its value to false to disable the HDFS ACL function.
Figure 1 Adding custom parameters - Click Save to save the configuration for versions earlier than MRS 3.2.0-LTS. On the Dashboard page, click More and select Restart Service. Enter the password of the current user, click OK, and select Restart upper-layer services. Click OK to restart Hive.
- Log in to the beeline client and set Location to the OBS file system path when creating a table.
beeline
For example, run the following command to create the table test in obs://OBS parallel file system name/user/hive/warehouse/Database name/Table name:
create table test(name string) location "obs://OBS parallel file system name/user/hive/warehouse/Database name/Table name";
Check the table location.
desc formatted test;
You need to attach the component operation user to the URL policy within the Ranger policy. Enter the complete OBS path of the object for URL and select Read and Write permissions. Other permissions are not involved in the URL policy. For versions earlier than MRS 3.x, see Configuring Hive Access Permissions in Ranger. For MRS 3.x or later, see Adding a Ranger Access Permission Policy for Hive.
Interconnecting Hive with OBS Through MetaStore
- Log in to FusionInsight Manager and choose Cluster > Services > Hive > Configurations > All Configurations.
For details about how to log in to MRS Manager, see Accessing MRS Manager.
- For versions earlier than MRS 3.2.0-LTS:
- In the navigation pane on the left, choose MetaStore (role) > Customization. Add the configuration item hive.metastore.warehouse.dir to the custom parameter hive.metastore.customized.configs and set the value to an OBS path, for example, obs://hivetest/user/hive/warehouse/, where hivetest is the name of the OBS parallel file system.
Figure 2 Configuring hive.metastore.warehouse.dir
- In the navigation pane on the left, choose HiveServer (role) > Customization. Add the configuration item hive.metastore.warehouse.dir to custom parameters hive.metastore.customized.configs and hive.server.customized.configs and set the values to an OBS path, for example, obs://hivetest/user/hive/warehouse/, where hivetest is the name of the OBS parallel file system.
Figure 3 Configuring hive.metastore.warehouse.dir
- In the navigation pane on the left, choose MetaStore (role) > Customization. Add the configuration item hive.metastore.warehouse.dir to the custom parameter hive.metastore.customized.configs and set the value to an OBS path, for example, obs://hivetest/user/hive/warehouse/, where hivetest is the name of the OBS parallel file system.
- For MRS 3.2.0-LTS and later versions:
- For versions earlier than MRS 3.2.0-LTS:
- Save the change and restart Hive.
- (Optional) Install the client by referring to Installing an MRS Cluster Client. If the client has been installed in the cluster, go to Step 4.
- Update the client configuration file.
- Run the following command to open hivemetastore-site.xml in the Hive configuration file directory on the client:
vim Client installation directory/Hive/config/hivemetastore-site.xml
- Change the value of hive.metastore.warehouse.dir to the corresponding OBS path, for example, obs://hivetest/user/hive/warehouse/, where hivetest is the OBS bucket name.
Figure 5 Configuring the OBS Path
- For MRS 3.2.0-LTS and later versions, change the value of hive.metastore.warehouse.dir in hivemetastore-site.xml file to the corresponding OBS path, for example, obs://hivetest/user/hive/warehouse/. The XML file is stored in the HCatalog client configuration file directory.
vi Client installation directory/Hive/HCatalog/conf/hivemetastore-site.xml
- Run the following command to open hivemetastore-site.xml in the Hive configuration file directory on the client:
- Log in to the beeline client, create a table, and check whether the location is the OBS path.
- Go to the client installation directory.
cd Client installation directory
- Load the environment variables.
source bigdata_env
- Authenticate the user. Skip this step for clusters with Kerberos authentication disabled.
kinit User performing Hive operations
- Log in to the Hive client.
beeline
- Create the test table.
create table test(name string);
- Check the table location.
desc formatted test;
Location of the table is the OBS path.
Figure 6 Viewing the location of a Hive table
- Go to the client installation directory.
- If the location of the current database points to HDFS, tables created in the database are located in HDFS by default. You do not need to specify the location. To modify the default table creation policy, update the location of the database to OBS. The procedure is as follows:
- Query the location of the database.
show create database obs_test;
Figure 7 Viewing the location of a Hive table - Change the database location.
alter database obs_test set location 'obs://OBS parallel file system name/user/hive/warehouse/Database name';
Check whether the database location is changed to an OBS path.
show create database obs_test;
Figure 8 Checking the new location of the Hive table - Change the table location.
alter table user_info set location 'obs://OBS parallel file system name/user/hive/warehouse/Database name/Table name';
If there is already data in the table, migrate the data to the new location.
- Query the location of the database.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot