Connecting to the OBS File System with an MRS Hive Table

Updated on 2024-08-12 GMT+08:00

View PDF

MRS allows you to store data in OBS and use an MRS cluster for data computing only. In this way, storage and compute are decoupled. You can use the IAM service to perform simple configurations to access OBS.

This section describes how to create a Hive table to store data to OBS.

Creating an ECS Agency
Configuring an Agency for an MRS Cluster
Creating an OBS File System
Accessing the OBS File System Through Hive

Creating an ECS Agency

Log in to the Huawei Cloud management console.
Choose Service List > Management & Governance > Identity and Access Management.
Click Agencies. On the displayed page, click Create Agency.
Enter an agency name, for example, mrs_ecs_obs.
Set Agency Type to Cloud service and select ECS BMS to authorize ECS or BMS to invoke OBS.
Set Validity Period to Unlimited and click Next.
Figure 1 Creating an agency
On the page that is displayed, search for OBS OperateAccess in the search box and select it in the result list.
Figure 2 Assigning permissions
Click Next. On the page that is displayed, select the desired scope for the permissions you selected. By default, All resources is selected. Click Show More, select Global resources, and click OK.
In the dialog box that is displayed, click OK to start authorization. After the message "Authorization successful." is displayed, click Finish. The agency is created successfully.

Configuring an Agency for an MRS Cluster

You can configure an agency when creating a cluster or bind an agency to an existing cluster to decouple storage and compute. This section uses an existing cluster as an example to describe how to configure an agency.

Log in to the MRS management console. In the left navigation pane, choose Active Clusters.
Click the name of a cluster to go to the cluster details page.
On the Dashboard page, click Synchronize on the right side of IAM User Sync to synchronize IAM users.
On the Dashboard page, click Manage Agency on the right side of Agency to select the agency created in Creating an ECS Agency, and click OK to bind it to the cluster. Alternatively, click Create Agency to go to the IAM console to create an agency and bind it to the cluster.
Figure 3 Binding an agency

Creating an OBS File System

Log in to the OBS console.
Choose Parallel File System > Create Parallel File System.
Enter the file system name, for example, mrs-demo01.
Set other parameters as required.

Figure 4 Creating a parallel file system
Click Create Now.
In the parallel file system list on the OBS console, click a file system name to go to the details page.
In the navigation pane, choose Files and create program and input folders.
- program: Upload the program package to this folder.
- input: Upload the input data to this folder.

Accessing the OBS File System Through Hive

Log in to a master node as user root. For details, see Logging In to an ECS.
Verify that Hive can access OBS.
1. Log in to the master node of the cluster as user root and run the following commands:
  cd /opt/Bigdata/client
  
  source bigdata_env
  
  source Hive/component_env
2. View the list of files in file system mrs-demo01.
  hadoop fs -ls obs://mrs-demo01/
3. Check whether the file list is returned. If it is returned, access to OBS is successful.
  Figure 5 Viewing the file list in mrs-demo01
4. Run the following command to authenticate the user (skip this step for a normal cluster, that is, with Kerberos authentication disabled):
  kinit hive
  
  Enter the password of user hive. The default password is Hive@123. Change the password upon the first login.
5. Run the Hive client command.
  beeline
6. Access the OBS directory in the Beeline. For example, run the following command to create a Hive table and specify that data is stored in the test_demo01 table of file system mrs-demo01:
  create table test_demo01(name string) location "obs://mrs-demo01/test_demo01";
7. Run the following command to query all tables. If the test_demo01 table is displayed in the command output, the access to OBS is successful.
  show tables;
  
  Figure 6 Checking whether the test_demo01 table exists
8. Run the following command to check the table location.
  show create table test_demo01;
  
  Check whether the location of the table starts with obs://OBS bucket name/.
  
  Figure 7 Checking the location of the test_demo01 table
9. Run the following command to write data into the table.
  insert into test_demo01 values('mm'),('ww'),('ww');
  
  Run the select * from test_demo01; command to check whether the data is written successfully.
  
  Figure 8 Viewing data in the test_demo01 table
10. Run the !q command to exit the Beeline client.
11. Log in to the OBS console again.
12. Click Parallel File System and select the created file system.
13. Click Files to check whether the data exists in the created table.
  Figure 9 Viewing data

Parent topic: Interconnection with Other Cloud Services

Previous topic: Interconnecting Hive with CSS

Next topic: Interconnection with Ecosystem Components

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot