Help Center/ MapReduce Service/ Component Operation Guide (ME-Abu Dhabi Region)/ Using Impala/ Enabling and Configuring a Dynamic Resource Pool for Impala
Updated on 2024-07-19 GMT+08:00

Enabling and Configuring a Dynamic Resource Pool for Impala

This section describes how to enable and configure a dynamic resource pool to control Impala concurrency.

Background

Use a dynamic resource pool to control Impala concurrency.

  1. Log in to the master1 node of the cluster, switch to user omm, and create the fair-scheduler.xml and llama-site.xml files in the /home/omm directory.

  2. Open the fair-scheduler.xml file and add the following configurations:
    <allocations>
        <queue name="root">
            <aclSubmitApps> </aclSubmitApps>
            <queue name="default">
                <maxResources> 1mb, 0 vcores</maxResources><!--This parameter is for reference only.-->
                <aclSubmitApps>*</aclSubmitApps>
            </queue>
            <queue name="development">
                <maxResources>2048 mb, 0 vcores</maxResources><!--This parameter is for reference only.-->
                <aclSubmitApps>admin</aclSubmitApps>
            </queue>
            <queue name="production">
                <maxResources>7168 mb, 0 vcores</maxResources><!--This parameter is for reference only.-->
                <aclSubmitApps>omm</aclSubmitApps>
            </queue>
        </queue>
        <queuePlacementPolicy>
            <rule name="specified" create="false"/>
            <rule name="default" />
        </queuePlacementPolicy>
    </allocations>
  3. Open the llama-site.xml file and add the following configurations:
    <?xml version="1.0" encoding="UTF-8"?>
    <configuration>
      <property>
        <name>llama.am.throttling.maximum.placed.reservations.root.default</name>
        <value>1</value>
      </property>
      <property>
        <name>llama.am.throttling.maximum.queued.reservations.root.default</name>
        <value>2</value><!--This parameter is for reference only.-->
      </property>
      <property>
        <name>impala.admission-control.pool-default-query-options.root.default</name>
        <value>mem_limit=128m,query_timeout_s=20,max_io_buffers=10</value>
      </property>
      <property>
        <name>impala.admission-control.pool-queue-timeout-ms.root.default</name>
        <value>30000</value><!--This parameter is for reference only.-->
      </property>
      <property>
        <name>impala.admission-control.max-query-mem-limit.root.default</name>
        <value>1024000000</value><!--1GB--><!--This parameter is for reference only.-->
      </property>
      <property>
        <name>impala.admission-control.min-query-mem-limit.root.default</name>
        <value>2048000000</value><!--2GB-->
      </property>
      <property>
        <name>impala.admission-control.clamp-mem-limit-query-option.root.default.regularPool</name>
        <value>true</value>
      </property>
    </configuration>
  4. Run the following commands to synchronize fair-scheduler.xml and llama-site.xml to the etc folder in the installation directory on all Impalad nodes, respectively:

    scp fair-scheduler.xml {IP address of the Impalad instance}:/opt/Bigdata/FusionInsight_Impala_***/***_Impalad/etc/

    scp llama-site.xml {IP address of the Impalad instance}:/opt/Bigdata/FusionInsight_Impala_***/***_Impalad/etc/

  5. Log in to FusionInsight Manager, find the Impala component, and add the following custom configuration items and values to the Impala instance:

    --fair_scheduler_allocation_path The value is, for example, /opt/Bigdata/FusionInsight_Impala_***/***_Impalad/etc/fair-scheduler.xml

    --llama_site_path The value is, for example, /opt/Bigdata/FusionInsight_Impala_***/***_Impalad/etc/llama-site.xml

  6. Restart the Impalad instance.

  7. Log in to the node where the Impala client is installed, run the source command to obtain the environment variables, and run the following command:

    impala-shell -i {IP address of the Impalad instance:Port number} -Q request_pool=root.default (resource pool configured in fair-scheduler.xml and llama-site.xml)

    Execute SQL statements.

  8. Log in to the Impalad WebUI to check the resource pool usage and ensure that the configuration has taken effect.

    https://{Cluster console address}:9022/component/Impala/Impalad/95/