Storing Ranger Metadata to RDS

Notes and Constraints

When multiple MRS clusters connect to the same RDS database and share the same metadata, the clusters must be in the same mode.

Disabling Ranger Authentication for Cluster Components

This operation is required only for MRS 3.1.0 or later.

1. Log in to FusionInsight Manager of the MRS cluster.

   For details about how to log in to FusionInsight Manager, see Accessing MRS Manager.

2. Choose Cluster > Services > Service name.

   Currently, the following components in an MRS 3.1.x or later cluster support Ranger authentication: HDFS, HBase, Hive, Spark, Impala, YARN, Kafka, and HetuEngine.

3. In the upper right corner of the Dashboard page, click More and select Disable Ranger. If Disable Ranger is dimmed, Ranger authentication is disabled, as shown in Figure 1.
   Figure 1 Disabling Ranger authentication
   

4. (Optional) To use an existing authentication policy, perform this step to export the authentication policy on the Ranger web page. After the Ranger metadata is switched, you can import the existing authentication policy again. The following uses Hive as an example. After the export, a policy file in JSON format is generated in a local directory.
   1. Log in to MRS Manager.
   2. Choose Cluster > Services > Ranger to go to the Ranger service overview page.
   3. Click RangerAdmin in the Basic Information area to go to the Ranger Web UI.

      For clusters with Kerberos authentication enabled, the admin user in Ranger belongs to the User type. To view all management pages, click the username in the upper right corner and select Log Out to log out of the system.

      Log in to the system as user rangeradmin (default password: Rangeradmin@123) or another user who has the Ranger administrator permissions. For details about the user and its default password, see User Account List.

   4. Click the export button in the row where the Hive component is located to export the authentication policy.
      Figure 2 Exporting authentication policies
      
   5. Click Export. After the export is complete, a policy file in JSON format is generated in a local directory.
      Figure 3 Exporting Hive authentication policies
      

Creating and Configuring an RDS DB Instance

1. Log in to the RDS console and buy an RDS DB instance. For details, see Buying a DB Instance.
   • To ensure network communications between the cluster and the MySQL database, create the instance in the same VPC and subnet as the MRS cluster.
   • Security group rules of the RDS DB instance must allow inbound access from the MySQL database port (3306 by default).
   • Ranger can interconnect with RDS for MySQL databases of the MySQL 5.7.x and 8.0 versions only.

2. In the navigation pane of the RDS console, choose Instances. Locate the row containing the RDS DB instance used by an MRS data connection, click Log In in the Operation column to log in to the DB instance as user root.
   Figure 4 Logging in to an RDS DB instance
   

3. On the home page of the instance, click Create Database to create a database.
   

   • If no new database is created, the MRS data connections will fail to set up.
   • To connect MRS clusters in different modes to the same RDS database, you must create a new database.
   Figure 5 Creating a database
   

4. On the top of the page, choose Account Management > User Management.
   • For clusters earlier than MRS 3.x, if the selected data connection type is RDS MySQL database, ensure that the database user is root. If the user is not root, create a user and grant permissions to the user by referring to Step 5 to Step 6.
   • For MRS 3.x or later clusters, if the selected data connection type is RDS MySQL database, the database user must not be root. In this case, create a user and grant permissions to the user by referring to Step 5 to Step 6.

5. Click Create User to create a non-root user, select all permissions in the Global Permissions area, and set other parameters as required.

   If you are configuring an external RDS data connection for Ranger, you can select only the SELECT, INSERT, CREATE, RELOAD, CREATE USER, and GRANT permissions.

   Figure 6 Creating a user
   

6. On the top of the page, choose SQL Operations > SQL Query, switch to the target database by database name, and run the following SQL statements to grant permissions to the database user. In the following statements, ${db_name} and ${db_user} indicate the name of the database to be connected to MRS and the name of the new user, respectively.

   grant all privileges on ${db_name}.* to '${db_user}'@'%' with grant option;
   grant reload on *.* to '${db_user}'@'%' with grant option;
   flush privileges;

   Figure 7 Assigning permissions to a database user
   

Creating an RDS Data Connection for an Existing MRS Cluster

Perform the following steps to create an RDS data connection for an existing MRS cluster.

1. Log in to the MRS console.
2. In the navigation pane, choose Data Connections.
3. Click Create Data Connection.
4. Set parameters by referring to Table 1.

   Table 1 Parameters for creating a data connection

   Parameter	Example Value	Description
   Type	-	The type of an external source connection. Value options are as follows: RDS PostgreSQL database. Clusters with Hive installed can connect to this type of database. RDS MySQL database. Clusters with Hive or Ranger installed can connect to this type of database.
   Name	newtest	The name of a data connection.
   Database Instance	-	The RDS database instance. This instance must be created in RDS before being used here, and the database must have been created. For details, see Creating and Configuring an RDS DB Instance. Click View DB Instance to view the created DB instances.
   Database	dataname	The name of the database to be connected.
   Username	datauser	The username for logging in to the database to be connected.
   Password	-	The password for logging in to the database to be connected.

   If the selected data connection type is RDS MySQL database, ensure that the database user is user root. If the user is not root, perform operations by referring to Creating and Configuring an RDS DB Instance.

5. Click OK.

Configuring a Ranger Data Connection

1. Log in to the MRS console.
2. Click the name of a cluster to go to the cluster details page.
3. Click Manage on the right of Data Connection to go to the data connection configuration page.
4. Click Configure Data Connection and set related parameters.
   • Type: Ranger
   • Module Type: Ranger metadata
   • Connection Type: RDS MySQL database
   • Connection Instance: Select a created RDS MySQL DB instance. For details about how to create a data connection, see Creating an RDS Data Connection for an Existing MRS Cluster.

5. Select I understand the consequences of performing the scale-in operation and click Test.
6. After the test is successful, click OK to complete the data connection configuration.
7. Log in to MRS Manager. For details, see Accessing MRS Manager.
8. Choose Cluster > Services > Ranger to go to the Ranger service overview page.
9. Choose More > Restart Service or More > Service Rolling Restart.

   If you choose Restart Service, services will be interrupted during the restart. If you select Service Rolling Restart, rolling restart can minimize the impact or do not affect service running.

   Restarting Ranger will affect the permissions of all components controlled by Ranger and may affect service running. Restart Ranger when the cluster is idle or during off-peak hours. Before the Ranger component is restarted, the policies in the Ranger component still take effect.

   Figure 8 Restarting a service
   

10. Enable Ranger authentication for the component to be authenticated. The Hive component is used as an example.
    Currently, the following components in an MRS 3.1.x or later cluster support Ranger authentication: HDFS, HBase, Hive, Spark, Impala, Storm, Kafka, HetuEngine, and YARN.

    1. Log in to MRS Manager and choose Cluster > Services > Service Name.
    2. In the upper right corner of the Dashboard page, click More and select Enable Ranger.
       Figure 9 Enabling Ranger authentication
       

11. Log in to the Ranger Web UI as a user with the Ranger administrator permissions and click for the Hive component.
    Figure 10 Clicking the import button
    

12. Import parameters.
    • Click Select file and select the authentication policy file downloaded in 4.e.
    • Select Merge If Exist Policy.
    Figure 11 Importing authentication policies
    

13. Restart the component for which Ranger authentication is enabled.
    1. Log in to MRS Manager. For details, see Accessing MRS Manager.
    2. Choose Cluster > Services > Service name. The service overview page is displayed.
    3. Choose More > Restart Service or More > Service Rolling Restart.
       Figure 12 Restarting a service
       

       If you choose Restart Service, services will be interrupted during the restart. If you select Service Rolling Restart, rolling restart can minimize the impact or do not affect service running.

Verifying Ranger Metadata Storage in an External Database

1. In MRS 3.5.0-LTS or earlier, for a cluster with Kerberos authentication disabled, the Ranger data source can be accessed by Unix users of the cluster by default. To enable the data source to be accessed by the MRS Manager LDAP users, perform the following operations:
   1. Log in to MRS Manager by referring to Accessing MRS Manager. Choose Cluster > Services > Ranger, click Configurations and then All Configurations, click UserSync (Role) and then Customization, and add the following custom parameters to set the Ranger data source to the MRS Manager LDAP:
      • For versions later than MRS 3.1.0, in the ranger.usersync.config.expandor area, set ranger.usersync.sync.source to ldap and ranger.usersync.cookie.enabled to false, as shown in the following figure.

        

      • For MRS 3.1.0, in the ranger.ugsync.site.customized.configs area, set ranger.usersync.sync.source to ldap and ranger.usersync.cookie.enabled to false.
   2. Click Save.
   3. In the upper right corner of the Ranger Dashboard page, click More and choose Synchronize Configuration.

      

   4. On the Ranger instance page, select the UserSync instance and choose More > Restart Instance.

      

   5. On the Ranger Dashboard page, click RangerAdmin and choose Settings > Users/Groups/Roles to check whether LDAP users exist.

2. Enable Ranger authentication for Hive.

   On MRS Manager, choose Cluster > Services > Hive. In the upper right corner of the page, click More and check whether Ranger authentication is enabled for Hive. If Ranger authentication is not enabled, select Enable Ranger to enable it.

3. On MRS Manager of the cluster, choose System > Permission > User and add a human-machine user, for example, user test. Users in a cluster with Kerberos authentication enabled must have the permissions to create Hive tables. That is, the user must belong to either the hive or hadoop user group.
4. Log in to the node where the Hive client is installed and run the following commands:

   Switch to the client installation directory.

   cd Client installation directory

   Load the environment variables.

   source bigdata_env

   Authenticate the user. Skip this step for clusters with Kerberos authentication disabled.

   kinit Service user

5. Run the following command to log in to the Hive client:
   • For clusters with Kerberos authentication enabled:

     beeline

   • For clusters with Kerberos authentication disabled:

     beeline -n Service user

6. Run the following commands to create a Hive table and insert data to the table:

   Create a table:

   create table user_info(id string,name string,gender string,age int,addr string);

   Insert data:

   insert into table user_info(id,name,gender,age,addr) values("12005000201","A","man",19,"city");

7. On MRS Manager of the cluster, choose System > Permission > User and add another human-machine user, for example, user test1. Users in a cluster with Kerberos authentication enabled can have only the permissions on the hive user group.
8. Log in to the Ranger Web UI as user rangeradmin (for a cluster with Kerberos authentication enabled) or admin (for a cluster with Kerberos authentication disabled), and add the permission to query the table created in Step 6 for the user.
   1. On the Service Manager page, click Hive in Hadoop SQL, click Add New Policy to add a Hive permission control policy, set the following parameters, and click Add.
      • Policy Name: Enter a policy name.
      • database: Set it to default.
      • table: Set it to the name of the table to be accessed, for example, user_info.
      • column: Set it to *.
      • In the Allow Conditions area, select the user you want to authorize, for example, test1, in the Select User column, and select the select permission in the Permissions column.
   2. For a cluster with Kerberos authentication enabled, return to the Service Manager page, click hacluster in HDFS, click Add New Policy to add an HDFS permission control policy, set the following parameters, and click Add.
      • Policy Name: Enter a policy name.
      • Resource Path: Enter the specific path of the table to be accessed in HDFS, for example, /user/hive/warehouse/user_info.
      • In the Allow Conditions area, select the user you want to authorize, for example, test1, in the Select User column, and select Read and Execute in the Permissions column.

9. Log in to the Hive client as the user created in Step 7 and run the following command to query data in the table created in Step 6:

   select * from user_info;

   If the table data can be queried, the Ranger metadata is successfully stored to the external RDS MySQL database.

   Figure 13 Hive table data queried