Configuring Static Resources for an MRS Cluster

Impact on the System

• After a static service pool is configured, the configuration of affected services expires. You need to restart the services. Services are unavailable during restart.
• After a static service pool is configured, the maximum number of resources used by each service and role instance cannot exceed the upper limit.

Configuring Static Resources (MRS 3.x and Later)

Modify the Resource Adjustment Base

1. Log in to FusionInsight Manager and choose Cluster > Static Service Pool Configurations.
2. Click Configuration in the upper right corner. The page for configuring resource pools is displayed.
3. Change the values of CPU (%) and Memory (%) in the System Resource Adjustment Base area.

   Modifying the system resource adjustment base changes the maximum physical CPU and memory usage on nodes by services. If multiple services are deployed on the same node, the maximum physical resource usage of all services cannot exceed the adjusted CPU or memory usage.

4. Click Next.

   To modify parameters again, click Previous.

Modify the Default Resource Configuration Group

5. Click default. In the Configure weight table, set CPU LIMIT(%), CPU SHARE(%), I/O(%), and Memory(%) for each service.
   Figure 1 Weight Configuration
   
   

   • The sum of CPU LIMIT(%) and CPU SHARE(%) used by all services can exceed 100%.
   • The sum of I/O(%) used by all services can exceed 100% but cannot be 0.
   • The sum of Memory(%) used by all services can be greater than, smaller than, or equal to 100%.
   • Memory(%) cannot take effect dynamically and can only be modified in the default configuration group.
   • CPU LIMIT(%) is used to configure the ratio of the number of CPU cores that can be used by a service to those can be allocated to related nodes.
   • CPU SHARE(%) is used to configure the ratio of the time when a service uses a CPU core to the time when other services use the CPU core. That is, the ratio of time when multiple services compete for the same CPU core.

6. Click Generate detailed configurations based on weight configurations. FusionInsight Manager generates the actual values of the parameters in the default weight configuration table based on the cluster hardware resources and allocation information.
7. Click OK.

   In the displayed dialog box, click OK.

Add a Customized Resource Configuration Group

8. Determine whether to automatically adjust resource configurations at different time segments.
   • If yes, go to 9.
   • If no, use the default configurations, and no further action is required.

9. Click Configuration, change the system resource adjustment base values, and click Next.
10. Click Add to add a resource configuration group.
    Figure 2 Adding a resource configuration group
    

11. In Step 1: Scheduling Time, click Configuration.

    The page for configuring the time policy is displayed.

    Modify the following parameters based on service requirements and click OK.

    • Repeat: If this parameter is selected, the customized resource configuration is applied repeatedly based on the scheduling period. If this parameter is not selected, set the date and time when the configuration of the group of resources can be applied.
    • Repeat Policy: The available values are Daily, Weekly, and Monthly. This parameter is valid only when Repeat is selected.
    • On: indicates the time period between the start time and end time when the resource configuration is applied. Set a unique time range. If the time range overlaps with that of an existing group of resource configuration, the time range cannot be saved.
    

    • The default group of resource configuration takes effect in all undefined time segments.
    • The newly added resource group is a parameter set that takes effect dynamically in a specified time range.
    • The newly added resource group can be deleted. A maximum of four resource configuration groups that take effect dynamically can be added.
    • Select a repetition policy. If the end time is earlier than the start time, the resource configuration ends in the next day by default. For example, if a validity period ranges from 22:00 to 06:00, the customized resource configuration takes effect from 22:00 on the current day to 06:00 on the next day.
    • If the repetition policy types of multiple configuration groups are different, the time ranges can overlap. The policy types are listed as follows by priority from low to high: daily, weekly, and monthly. The following is an example. There are two resource configuration groups using the monthly and daily policies, respectively. Their application time ranges in a day overlap as follows: 04:00 to 07:00 and 06:00 to 08:00. In this case, the configuration of the group that uses the monthly policy prevails.
    • If the repetition policy types of multiple resource configuration groups are the same, the time ranges of different dates can overlap. For example, if there are two weekly scheduling groups, you can set the same time range on different day for them, such as to 04:00 to 07:00, on Monday and Wednesday, respectively.

12. Modify the resource configuration of each service in Step 2: Weight Configuration.
13. Click Generate detailed configuration. FusionInsight Manager generates the actual values of the parameters in the default weight configuration table based on the cluster hardware resources and allocation information.
14. Click OK.

    In the displayed dialog box, click OK.

Configuring Static Resources (MRS 2.x and Earlier)

1. Modify the system resource adjustment base.
   1. On MRS Manager, click System. In the Resource area, click Configure Static Service Pool.
   2. Click Configuration. The service pool configuration group management page is displayed.
   3. In the System Resource Adjustment Base area, change the values of CPU(%) and Memory(%).

      Modifying System Resource Adjustment Base limits the maximum physical CPU and memory resource percentage of nodes that can be used by the Flume, HBase, HDFS, Impala and YARN services. If multiple services are deployed on the same node, the maximum physical resource usage of all services cannot exceed the adjusted CPU or memory usage.

   4. Click Next.

      If you need to modify the parameters again, click Previous in the lower part of the page.

2. Modify the default configuration group of the service pool.
   1. In the Service Pool Configuration table, set CPU LIMIT(%), CPU SHARE(%), I/O(%), and Memory(%) for the Flume, HBase, HDFS, Impala, and YARN services.
   

   • The sum of CPU LIMIT(%) used by all services can exceed 100%.
   • The sum of CPU SHARE(%) and I/O(%) used by all services must be 100%. For example, if CPU resources are allocated to the HDFS and YARN services, the total CPU resources allocated to the two services are 100%.
   • The sum of Memory(%) used by all services can be greater than, smaller than, or equal to 100%.
   • Memory(%) cannot take effect dynamically and can only be modified in the default configuration group.
   2. Click in the blank area of the page to complete the editing. MRS Manager generates the correct values of service pool parameters in the Detailed Configuration area based on the cluster hardware resources and allocation information.
   3. You can click the edit icon on the right of Detailed Configuration to modify the parameter values of the service pool based on service requirements.

      In the Service Pool Configuration area, click the specified service name. The Detailed Configuration area displays only the parameters of the service. Manually changing parameter values does not refresh the service resource usage. In added configuration groups, the configuration group numbers of the parameters that take effect dynamically will be displayed. For example, HBase: RegionServer: dynamic-config1.RES_CPUSET_PERCENTAGE. The parameter functions do not change.

      Table 1 Parameters of the static service pool

      Parameter	Description
      RES_CPUSET_PERCENTAGE dynamic-configX.RES_CPUSET_PERCENTAGE	Configures the service CPU percentage.
      RES_CPU_SHARE dynamic-configX.RES_CPU_SHARE	Configures the service CPU usage share.
      RES_BLKIO_WEIGHT dynamic-configX.RES_BLKIO_WEIGHT	Configures service I/O usage.
      HBASE_HEAPSIZE	Configures the maximum JVM memory for RegionServer.
      HADOOP_HEAPSIZE	Configures the maximum JVM memory of a DataNode.
      yarn.nodemanager.resource.memory-mb	Configures the memory that can be used by NodeManager on the current node.
      dfs.datanode.max.locked.memory	Configures the maximum memory that can be used by a DataNode as the HDFS cache.
      FLUME_HEAPSIZE	Configures the maximum JVM memory that can be used by each Flume instance.
      IMPALAD_MEM_LIMIT	Configures the maximum memory that can be used by an Impalad instance.

3. Add a customized resource configuration group.
   1. Determine whether to automatically adjust resource configurations based on the time.

      If yes, go to 3.b.

      If no, go to 4.

   2. Click to add a resource configuration group. In the Scheduling Time area, click the edit icon. The time policy configuration page is displayed.

      Modify the following parameters based on service requirements and click OK.

      • Repeat: If selected, the resource configuration group runs repeatedly based on the scheduling period. If not selected, set the date and time when the configuration of the group of resources can be applied.
      • Repeat Policy: can be set to Daily, Weekly, and Monthly. This parameter is valid only when Repeat is selected.
      • Between: indicates the time period between the start time and end time when the resource configuration is applied. Set a unique time range. If the time range overlaps with that of an existing group of resource configuration, the time range cannot be saved. This parameter is valid only when Repeat is selected.
      

      • The default group of resource configuration takes effect in all undefined time segments.
      • The newly added resource group is a parameter set that takes effect dynamically in a specified time range.
      • The newly added resource group can be deleted. A maximum of four resource configuration groups that take effect dynamically can be added.
      • Select a repetition policy. If the end time is earlier than the start time, the resource configuration ends in the next day by default. For example, if a validity period ranges from 22:00 to 06:00, the customized resource configuration takes effect from 22:00 on the current day to 06:00 on the next day.
      • If the repetition policy types of multiple configuration groups are different, the time ranges can overlap. The policy types are listed as follows by priority from low to high: daily, weekly, and monthly. The following is an example. There are two resource configuration groups using the monthly and daily policies, respectively. Their application time ranges in a day overlap as follows: 04:00 to 07:00 and 06:00 to 08:00. In this case, the configuration of the group that uses the monthly policy prevails.
      • If the repetition policy types of multiple resource configuration groups are the same, the time ranges of different dates can overlap. For example, if there are two weekly scheduling groups, you can set the same time range on different day for them, such as to 04:00 to 07:00, on Monday and Wednesday, respectively.
   3. On the Service Pool Configuration page, modify the resource configuration of each service. Click the blank area on the page to complete the editing, and go to 4.

      You can click the edit icon on the right of Service Pool Configuration to modify the parameters. Click the edit icon in the Detailed Configuration area to manually update the parameter values generated by the system based on service requirements.

4. Saves the settings.

   Click Save. In the Save Configuration dialog box, select Restart the affected services or instances. Click OK to save the settings and restart related services.

   After the system displays "Operation succeeded", click Finish. The service is started successfully.