Enabling and Configuring a Dynamic Resource Pool for Impala

This section describes how to enable and configure a dynamic resource pool to control Impala concurrency.

Background

Use a dynamic resource pool to control Impala concurrency.

Log in to the master1 node of the cluster, switch to user omm, and create the fair-scheduler.xml and llama-site.xml files in the /home/omm directory.

Open the fair-scheduler.xml file and add the following configurations:

<allocations>
    <queue name="root">
        <aclSubmitApps> </aclSubmitApps>
        <queue name="default">
            <maxResources>4096 mb, 0 vcores</maxResources><!--For reference only-->
            <aclSubmitApps>*</aclSubmitApps>
        </queue>
        <queue name="development">
            <maxResources>2048 mb, 0 vcores</maxResources><!--This parameter is for reference only.-->
            <aclSubmitApps>admin</aclSubmitApps>
        </queue>
        <queue name="production">
            <maxResources>7168 mb, 0 vcores</maxResources><!--This parameter is for reference only.-->
            <aclSubmitApps>omm</aclSubmitApps>
        </queue>
    </queue>
    <queuePlacementPolicy>
        <rule name="specified" create="false"/>
        <rule name="default" />
    </queuePlacementPolicy>
</allocations>

Open the llama-site.xml file and add the following configurations:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <property>
    <name>llama.am.throttling.maximum.placed.reservations.root.default</name>
    <value>1</value>
  </property>
  <property>
    <name>llama.am.throttling.maximum.queued.reservations.root.default</name>
    <value>2</value><!--This parameter is for reference only.-->
  </property>
  <property>
    <name>impala.admission-control.pool-default-query-options.root.default</name>
    <value>mem_limit=128m,query_timeout_s=20,max_io_buffers=10</value>
  </property>
  <property>
    <name>impala.admission-control.pool-queue-timeout-ms.root.default</name>
    <value>30000</value><!--This parameter is for reference only.-->
  </property>
  <property>
    <name>impala.admission-control.max-query-mem-limit.root.default</name>
    <value>3072000000</value><!--3GB--><!--For reference only-->
  </property>
  <property>
    <name>impala.admission-control.min-query-mem-limit.root.default</name>
    <value>2048000000</value><!--2GB-->
  </property>
  <property>
    <name>impala.admission-control.clamp-mem-limit-query-option.root.default.regularPool</name>
    <value>true</value>
  </property>
</configuration>

Run the following commands to synchronize fair-scheduler.xml and llama-site.xml to the etc folder in the installation directory on all Impalad nodes, respectively:
scp fair-scheduler.xml {IP address of the Impalad instance}:/opt/Bigdata/FusionInsight_Impala_***/***_Impalad/etc/

scp llama-site.xml {IP address of the Impalad instance}:/opt/Bigdata/FusionInsight_Impala_***/***_Impalad/etc/
Log in to FusionInsight Manager, find the Impala component, and add the following custom configuration items and values to the Impala instance:
--fair_scheduler_allocation_path The value is, for example, /opt/Bigdata/FusionInsight_Impala_***/***_Impalad/etc/fair-scheduler.xml

--llama_site_path The value is, for example, /opt/Bigdata/FusionInsight_Impala_***/***_Impalad/etc/llama-site.xml
Restart the Impalad instance.
Log in to the node where the Impala client is installed, run the source command to obtain the environment variables, and run the following command:
impala-shell -i {IP address of the Impalad instance:Port number} -Q request_pool=root.default (resource pool configured in fair-scheduler.xml and llama-site.xml)

Execute SQL statements.
Log in to the Impalad web UI to check the resource pool usage and apply the configurations.
https://{Cluster console address}:9022/component/Impala/Impalad/95/