Introduction to MRS Multi-Tenancy

Overview

• Background:

  Modern enterprises' data clusters are becoming more and more centralized and cloud-based. Enterprise-class big data clusters must meet the following requirements:

  • Carry data of different types and formats and run jobs and applications of different types (such analysis, query, and stream processing).
  • Isolate data of a user from that of another user who has demanding requirements on data security, such as a bank or government institute.

  The preceding requirements bring the following challenges to the big data clusters:

  • Proper allocation and scheduling of resources to ensure stable operating of applications and jobs.
  • Strict access control to ensure data and service security.

  Multi-tenancy isolates the resources of a big data cluster into resource sets. Users can lease desired resource sets to run applications and jobs and store data. In a big data cluster, multiple resource sets can be deployed to meet diverse requirements of multiple users.

  MRS provides a complete enterprise-class big data multi-tenant solution.

• Multi-tenancy:

  An MRS cluster provides various resources and services that can be shared among organizations, departments, or applications. The cluster provides logical entities, that is, tenants, to use these resources and services. Currently, only the analysis cluster supports tenant management.

  A mode involving different tenants is called multi-tenant mode. Multi-tenancy refers to multiple resource sets (a resource set is a tenant) in the MRS cluster and is able to allocate and schedule resources. The resources include computing resources and storage resources. The MRS cluster offers multi-tenancy, supporting a layered tenant model and enabling dynamic tenant creation or deletion to isolate resources. It dynamically allocates and configures compute and storage resources for each tenant.

  The computing resources indicate tenants' Yarn task queue resources. The task queue quota can be modified, and the task queue usage status and statistics can be viewed.

  The storage resources can be stored on HDFS. You can add and delete the HDFS storage directories of tenants, and set the quotas of file quantity and the storage space of the directories.

  Tenants can create and manage tenants in a cluster based on service requirements.

  • Roles, computing resources, and storage resources are automatically created when tenants are created. By default, all permissions of the new computing resources and storage resources are allocated to a tenant's roles.
  • Permissions to view the current tenant's resources, add a subtenant, and manage the subtenant's resources are granted to the tenant's roles by default.
  • After you have modified the tenant's computing or storage resources, permissions of the tenant's roles are automatically updated.

  MRS supports a maximum of 512 tenants. The default tenants created by the system include default. Tenants that are in the topmost layer with the default tenant are called level-1 tenants.

• Resource pool:

  YARN task queues support only one scheduling policy, namely label based scheduling. This policy allows YARN task queues to select NodeManagers with specific node labels, ensuring that tasks run on designated nodes and utilize target hardware resources. For example, YARN tasks requiring a large memory capacity can run on nodes labeled as large memory resources, preventing poor service performance.

  In an MRS cluster, tenants logically partition nodes to form a resource pool with multiple NodeManagers. YARN task queues can be associated with specified resource pools by configuring queue capacity policies, ensuring efficient and independent resource utilization in the resource pools.

  MRS supports a maximum of 50 resource pools. The system has a default resource pool.

• Advantages:
  • Proper resource configuration and isolation

    The resources of a tenant are isolated from those of another tenant. The resource use of a tenant does not affect other tenants. This mechanism ensures that each tenant can configure resources based on service requirements, improving resource utilization.

  • Resource consumption measurement and statistics

    Tenants are system resource applicants and consumers. System resources are planned and allocated based on tenants. Resource consumption by tenants can be measured and collected.

  • Assured data security and access security

    In multi-tenant scenarios, the data of each tenant is stored separately to ensure data security. The access to tenants' resources is controlled to ensure access security.

Multi-Tenant Model

• Multi-tenant model

  The following figure shows a multi-tenant model.

  Figure 1 Multi-tenant model
  

  Table 1 describes the concepts involved in Figure 1.

  Table 1 Concepts in the model

  Concept	Description
  Users	A natural person who has a username and password and uses the big data cluster. There are three different users in the figure: users A, B, and C.
  Role	A role is a carrier of one or more permissions. Permissions are assigned to specific objects, for example, access permissions for the /tenant directory in HDFS. There are four roles: t1, t2, t3, and Manager_tenant. Roles t1, t2, and t3 are automatically generated when tenants are created. The role names are the same as the tenant names. That is, roles t1, t2, and t3 map to tenants t1, t2, and t3. Role names and tenant names need to be used in pair. Role Manager_tenant is defaulted in the cluster and cannot be used separately.
  Tenant	A tenant is a resource set in a big data cluster. Multiple tenants are referred to as multi-tenancy. The resource sets further divided under a tenant are called sub-tenants. There are three tenants: t1, t2, and t3.
  Resource	Compute resources include CPUs and memory. Compute resources are allocated to a tenant from the cluster's total resources. Each tenant's resources are isolated from others. According to the figure, compute resources 1, 2, and 3 are allocated to tenants t1, t2, and t3 respectively from the cluster's total compute resources. Storage resources include disks and third-party storage systems. Storage resources are allocated to each tenant from the cluster's total storage. Each tenant's storage is isolated from others. According to the figure, storage resources 1, 2, and 3 are allocated for tenants t1, t2, and t3 respectively from the cluster's total storage resources.

  To use a tenant's resources or add/delete sub-tenants, a user must be assigned both the tenant role and the Manager_tenant role. Table 2 lists the roles assigned to each user in Figure 1.

  Table 2 Roles assigned to each user

  User	Role	Permission
  User A	Role t1 Role t2 Role Manager_tenant	Uses the resources of tenants t1 and t2. Adds or deletes sub-tenants of tenants t1 and t2.
  User B	Role t3 Role Manager_tenant	Uses the resources of tenant t3. Adds or deletes sub-tenants of tenant t3.
  User C	Role t1 Role Manager_tenant	Uses the resources of tenant t1. Adds or deletes sub-tenants of tenant t1.

  A user can be assigned multiple roles, and one role can also be assigned to multiple users. Users are associated to tenants after being assigned the tenant roles. Therefore, tenants and users form a many-to-many relationship. Users can access resources across multiple tenants, and multiple users can share resources within the same tenant. For example, in Figure 1, user A uses the resources of tenants t1 and t2, and users A and C uses the resources of tenant t1.

  

  The concepts of a parent tenant, sub-tenant, level-1 tenant, and level-2 tenant are all designed for the multi-tenant service scenarios. Pay attention to the differences between these concepts and the concepts of a leaf tenant resource and non-leaf tenant resource on FusionInsight Manager.

  • Level-1 tenant: determined based on the tenant's level. For example, the first created tenant is a level-1 tenant and its sub-tenant is a level-2 tenant.
  • Parent tenant and sub-tenant: indicates the hierarchical relationship between tenants.
  • Non-leaf tenant resource: indicates the tenant type selected during tenant creation. This tenant type can be used to create sub-tenants.
  • Leaf tenant resource: indicates the tenant type selected during tenant creation. This tenant type cannot be used to create sub-tenants.
• Multi-tenant platform:

  The MRS big data platform's core concept is the tenant, which enables the transition from user-centered to multi-tenant architecture to support enterprise multi-tenant applications. Figure 2 shows the transformation of big data platforms.

  Figure 2 Platform transformation from user-centered to multi-tenant
  

  On a user-centered big data platform, users can directly access and use all resources and services.

  • However, user applications may use only partial cluster resources, resulting in low resource utilization.
  • The data of different users may be stored together, decreasing data security.

  On a multi-tenant big data platform, users use required resources and services by accessing the tenants.

  • Resources are allocated and scheduled based on application requirements and used based on tenants, increasing resource utilization.
  • Users can access the resources of tenants only after being associated with tenant roles, enhancing access security.
  • The data of tenants is isolated, ensuring data security.

Multi-Tenant Resource

MRS cluster resources are classified into compute resources and storage resources. The multi-tenant architecture implements resource isolation.

• Compute Resources

  Compute resources include CPUs and memory. One tenant cannot occupy the compute resources of another tenant.

  Compute resources are divided into static service resources and dynamic resources.

  

  The resources allocated to YARN in a big data cluster are static service resources but can be dynamically allocated to job queues by YARN.

  • Static Service Resources

    Static service resources are compute resources allocated to each service and are not shared between services. The total compute resources of each service are fixed. These services include Flume, HBase, HDFS, and Yarn.

  • Dynamic Resources

    YARN provides distributed resource management for a big data cluster. The total volume of resources allocated to YARN can be configured. Then YARN allocates and schedules compute resources for job queues. For MapReduce, Spark, Flink, and Hive task queues, compute resources are allocated and scheduled by YARN.

    YARN queues are fundamental units of scheduling compute resources.

    The resources obtained by tenants using YARN queues are dynamic resources. Users can dynamically create and modify the queue quotas and view the status and statistics of the queues.

    • Resource pool:

      Enterprise IT systems often face complex cluster environments and diverse upper-layer requirements, such as the following scenarios:

      • Heterogeneous cluster: The computing speed, storage capacity, and network performance of each node in the cluster are different. All the tasks of complex applications need to be properly allocated to each compute node in the cluster based on service requirements.
      • Computing isolation: Data must be shared among multiple departments but computing resources must be distributed onto different compute nodes.

      These require that the compute nodes be further partitioned.

      Resource pools are used to specify the configuration of dynamic resources. YARN queues are associated with resource pools for resource allocation and scheduling.

      One tenant can have only one default resource pool. Users can be assigned the role of a tenant to use the resources in the resource pool of the tenant. To use resources in multiple resource pools, a user can be assigned the roles of multiple tenants.

    • Dynamic resource scheduling:

      YARN dynamic resources support label-based scheduling. This policy creates labels for compute nodes (YARN NodeManagers) and adds the compute nodes with the same label into the same resource pool. Then YARN dynamically associates the queues with resource pools based on the resource requirements of the queues.

      For example, a cluster has more than 40 nodes which are labeled by Normal, HighCPU, HighMEM, or HighIO based on their hardware and network configurations and added into four resource pools, respectively. Table 3 describes the performance of each node in the resource pool.

      Table 3 Performance of each node in a resource pool

      Label	Node Count	Hardware and Network Configuration	Added To	Associated With
      Normal	10	Minor	Resource pool A	Common queue
      HighCPU	10	High-performance CPU	Resource pool B	Computing-intensive queue
      HighMEM	10	Large memory	Resource pool C	Memory-intensive queue
      HighIO	10	High-performance network	Resource pool D	I/O-intensive queue

      A queue can use only the compute nodes in its associated resource pool.

      • A common queue is associated with resource pool A and uses Normal nodes with general hardware and network configurations.
      • A computing-intensive queue is associated with resource pool B and uses HighCPU nodes with high-performance CPUs.
      • A memory-intensive queue is associated with resource pool C and uses HighMEM nodes with large memory.
      • An I/O-intensive queue is associated with resource pool D and uses HighIO nodes with high-performance network.

      YARN queues are associated with specified resource pools to efficiently utilize resources in resource pools and maximize node performance.

      FusionInsight Manager supports a maximum of 50 resource pools. The system has a default resource pool.

• Storage resources

  Storage resources include disks and third-party storage systems. One tenant cannot access the data of another tenant.

  As a distributed file storage service in a big data cluster, HDFS stores all the user data of the upper-layer applications in the big data cluster, including the data written to HBase tables or Hive tables.

  A directory is the basic unit of allocating HDFS storage resources. HDFS supports the conventional hierarchical file structure. Users or applications can create directories and create, delete, move, or rename files in directories. Tenants can obtain storage resources from specified directories in the HDFS file system.

  The storage resource scheduling mechanism is as follows:

  • HDFS directories can be stored on nodes with specified labels or disks of specified hardware types.
    • When both real-time query and data analysis tasks are running in the same cluster, the real-time query tasks need to be deployed only on certain nodes, and the task data must also be stored on these nodes.
    • Based on actual service requirements, key data needs to be stored on highly reliable nodes.
  • Administrators can flexibly configure HDFS data storage policies based on actual service requirements and data features to store data on specified nodes.
  • For tenants, storage resources refer to the HDFS resources they use. Data of specified directories can be stored to the tenant-specified storage paths, thereby implementing storage resource scheduling and ensuring data isolation between tenants.
  • Users can add or delete HDFS storage directories of tenants and set the file quantity quota and storage capacity quota of directories to manage storage resources.

Schedulers

Multi-tenant schedulers are classified into the open-source Capacity scheduler and enhanced Superior scheduler. By default, the Superior scheduler is enabled for the MRS cluster.

• The Capacity scheduler is an open-source scheduler.
• The Superior scheduler is an enhanced version and named after the Lake Superior, indicating that the scheduler can manage a large amount of data.

You can check the scheduler type of YARN from the yarn.resourcemanager.scheduler.class value. For details about how to switch the scheduler, see Switching the MRS Tenant Resource Scheduler.

To meet enterprise requirements and tackle scheduling challenges faced by the YARN community, the Superior scheduler makes the following enhancements in addition to inheriting the advantages of the Capacity scheduler and Fair scheduler:

• Enhanced resource sharing policy

  The Superior scheduler supports queue hierarchy. It integrates the functions of open-source schedulers and shares resources based on configurable policies. In terms of instances, administrators can use the Superior scheduler to configure an absolute value or percentage policy for queue resources. The resource sharing policy of the Superior scheduler enhances label-based scheduling of YARN as a resource pool feature. The nodes in the YARN cluster can be grouped based on the capacity or service type to ensure that queues can more efficiently utilize resources.

• Tenant-based resource reservation policy

  Some tenants may run critical tasks at some time, and their resource requirements must be preferentially addressed. The Superior scheduler builds a mechanism to support the resource reservation policy. Reserved resources can be allocated to the critical tasks running in the specified tenant queues in a timely manner to ensure proper task execution.

• Fair sharing among tenants and resource pool users

  The Superior scheduler allows shared resources to be configured for users in a queue. Each tenant may have users with different weights. Heavily weighted users may require more shared resources.

• Ensured scheduling performance in a big cluster

  The Superior scheduler receives heartbeats from each NodeManager and saves resource information in memory, which enables the scheduler to control cluster resource usage globally. The Superior scheduler uses the push scheduling model, which makes the scheduling more precise and efficient and remarkably improves cluster resource utilization. Additionally, the Superior scheduler delivers excellent performance when the interval between NodeManager heartbeats is long and prevents heartbeat storms in big clusters.

• Priority policy

  If the minimum resource requirement of a service cannot be met after the service obtains all available resources, a preemption occurs. The preemption function is disabled by default.

Multi-Tenant Management

• Unified multi-tenant management

  Log in to the MRS console or FusionInsight Manager and click Tenant Resources. The displayed page integrates multiple functions such as tenant lifecycle management, tenant resource configuration, tenant service association, and tenant resource usage statistics, delivering a mature multi-tenant management model and achieving centralized tenant and service management.

  • Intuitive UI: MRS provides the graphical multi-tenant management interface and manages and operates multiple levels of tenants using the tree structure. Additionally, it integrates the basic information and resource quota of the current tenant in one interface to facilitate O&M and management, as shown in Figure 3.
    Figure 3 Multi-tenant management (using Manager 3.x as an example)
    
  • Hierarchical tenant management: MRS allows you to add sub-tenants to an existing tenant to re-configure resources. Sub-tenants of level-1 tenants are level-2 tenants. So on and so forth. FusionInsight Manager provides enterprises with a field-tested multi-tenant management model, enabling centralized tenant and service management.
• Simplified permission management

  MRS hides internal permission management details from common users and simplifies permission management operations for administrators, improving usability and user experience of tenant permission management.

  • MRS employs role-based access control (RBAC) to configure different permissions for users based on service scenarios during multi-tenant management.
  • The administrator of tenants has tenant management permissions, including viewing resources and services of the current tenant, adding or deleting sub-tenants of the current tenant, and managing permissions of sub-tenants' resources. MRS supports setting of the administrator for a single tenant so that the management over this tenant can be delegated to a user who is not the system administrator.
  • Roles of a tenant have all permissions on the computing resources and storage resources of the tenant. When a tenant is created, the system automatically creates roles for this tenant. You can add a user and assign the tenant roles to the user so that the user can use the resources of the tenant.
• Easy resource management
  • Free resource configuration

    You can configure the compute resources and storage resources during the creation of a tenant and add, modify, or delete the resources of the tenant.

    Permissions of the roles that are granted to a tenant are updated automatically when you modify the computing or storage resources of the tenant.

  • Resource usage statistics

    Resource usage statistics are critical for administrators to determine O&M activities based on the status of cluster applications and services, improving the cluster O&M efficiency. MRS displays the resource statistics of tenants in Resource Quota, including the vCores, memory, and HDFS storage resources.

    

    • The available resources of the Capacity scheduler and Superior scheduler are calculated as follows:
      • Capacity

        Available YARN resources (memory and CPU) = Resource capacity (%) x Total capacity of the resource pool

        When a queue spans multiple resource pools, its available resources are the combined total of resources allocated by each pool.

      • Superior

        The available YARN resources (memory and CPU) are allocated in proportion based on the queue weight.

    • When the tenant administrator is bound to a tenant role, the tenant administrator has the permissions to manage the tenant and use all resources of the tenant.
  • Graphical resource monitoring

    Graphical resource monitoring supports the graphical display of monitoring metrics listed in Table 4, as shown in Figure 4.

    Figure 4 Refined monitoring (using Manager 3.x as an example)
    

    By default, real-time monitoring data is displayed. You can click to customize a time range. Click and choose Export from the shortcut menu to export monitoring information.

    Table 4 Monitoring metrics

    Service	Metric Item	Description
    HDFS	HDFS Tenant Space Details Allocated Space Used Space	HDFS can monitor a specified storage directory. The storage directory is the same as the directory added by the current tenant in Resource.
    	HDFS Tenant File Object Details Number of Used File Objects
    YARN	Yarn Allocated Cores Maximum Number of CPU Cores in an AM Allocated Cores Number of Used CPU Cores in an AM	Monitoring information of the current tenant is displayed. If no sub-item is configured for a tenant, this information is not displayed. The monitoring data is obtained from Scheduler > Application Queues > Queue: Tenant name on the native web UI of YARN.
    	Yarn Allocated Memory Allocated Maximum AM Memory Allocated Memory Used AM Memory