Enabling and Configuring a Dynamic Resource Pool for Impala

Updated on 2024-10-08 GMT+08:00

View PDF

This section describes how to enable and configure a dynamic resource pool to control Impala concurrency.

Background

Use a dynamic resource pool to control Impala concurrency.

Log in to the master1 node of the cluster, switch to user omm, and create the fair-scheduler.xml and llama-site.xml files in the /home/omm directory.

Open the fair-scheduler.xml file and add the following configurations:

<allocations>
    <queue name="root">
        <aclSubmitApps> </aclSubmitApps>
        <queue name="default">
            <maxResources>4096 mb, 0 vcores</maxResources><!--For reference only-->
            <aclSubmitApps>*</aclSubmitApps>
        </queue>
        <queue name="development">
            <maxResources>2048 mb, 0 vcores</maxResources><!--This parameter is for reference only.-->
            <aclSubmitApps>admin</aclSubmitApps>
        </queue>
        <queue name="production">
            <maxResources>7168 mb, 0 vcores</maxResources><!--This parameter is for reference only.-->
            <aclSubmitApps>omm</aclSubmitApps>
        </queue>
    </queue>
    <queuePlacementPolicy>
        <rule name="specified" create="false"/>
        <rule name="default" />
    </queuePlacementPolicy>
</allocations>

Open the llama-site.xml file and add the following configurations:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <property>
    <name>llama.am.throttling.maximum.placed.reservations.root.default</name>
    <value>1</value>
  </property>
  <property>
    <name>llama.am.throttling.maximum.queued.reservations.root.default</name>
    <value>2</value><!--This parameter is for reference only.-->
  </property>
  <property>
    <name>impala.admission-control.pool-default-query-options.root.default</name>
    <value>mem_limit=128m,query_timeout_s=20,max_io_buffers=10</value>
  </property>
  <property>
    <name>impala.admission-control.pool-queue-timeout-ms.root.default</name>
    <value>30000</value><!--This parameter is for reference only.-->
  </property>
  <property>
    <name>impala.admission-control.max-query-mem-limit.root.default</name>
    <value>3072000000</value><!--3GB--><!--For reference only-->
  </property>
  <property>
    <name>impala.admission-control.min-query-mem-limit.root.default</name>
    <value>2048000000</value><!--2GB-->
  </property>
  <property>
    <name>impala.admission-control.clamp-mem-limit-query-option.root.default.regularPool</name>
    <value>true</value>
  </property>
</configuration>

Run the following commands to synchronize fair-scheduler.xml and llama-site.xml to the etc folder in the installation directory on all Impalad nodes, respectively:
scp fair-scheduler.xml {IP address of the Impalad instance}:/opt/Bigdata/FusionInsight_Impala_***/***_Impalad/etc/

scp llama-site.xml {IP address of the Impalad instance}:/opt/Bigdata/FusionInsight_Impala_***/***_Impalad/etc/
Log in to FusionInsight Manager, find the Impala component, and add the following custom configuration items and values to the Impala instance:
--fair_scheduler_allocation_path The value is, for example, /opt/Bigdata/FusionInsight_Impala_***/***_Impalad/etc/fair-scheduler.xml

--llama_site_path The value is, for example, /opt/Bigdata/FusionInsight_Impala_***/***_Impalad/etc/llama-site.xml
Restart the Impalad instance.
Log in to the node where the Impala client is installed, run the source command to obtain the environment variables, and run the following command:
impala-shell -i {IP address of the Impalad instance:Port number} -Q request_pool=root.default (resource pool configured in fair-scheduler.xml and llama-site.xml)

Execute SQL statements.
Log in to the Impalad web UI to check the resource pool usage and apply the configurations.
https://{Cluster console address}:9022/component/Impala/Impalad/95/

Parent topic: Using Impala

Previous topic: Interconnecting Impala with External LDAP

Next topic: Using the Impala Query Management Page

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot