Help Center/ MapReduce Service/ Component Operation Guide (Normal)/ Using Spark2x (for MRS 3.x or Later)/ Spark Core Enterprise-Class Enhancements/ Configuring Spark HA to Enhance HA/ Configuring the Spark Multi-Tenant Mode
Updated on 2024-10-08 GMT+08:00
View PDF
Share

Configuring the Spark Multi-Tenant Mode

Scenarios

In multi-tenant mode, JDBCServers are bound with tenants. Each tenant corresponds to one or more JDBCServers, and a JDBCServer provides services for only one tenant. Different tenants can be configured with different Yarn queues to implement resource isolation.

If YARN resources are insufficient, you are advised not to enable the multi-tenant mode.

Configuration Description

Log in to Manager, choose Cluster > Name of the desired cluster > Services > Spark2x > Configurations, click All Configurations, and search for and modify the following parameters.

Table 1 Parameter description

Parameter

Description

Default Value

spark.proxyserver.hash.enabled

Specifies whether to connect to ProxyServer using the Hash algorithm.

  • true indicates using the Hash algorithm. In multi-tenant mode, this parameter must be configured to true.
  • false indicates using random connection. In multi-active instance mode, this parameter must be configured to false.

true

NOTE:

After this parameter is modified, you need to download the client again.

spark.thriftserver.proxy.enabled

Specifies whether to use the multi-tenant mode.

  • false: The multi-instance mode is used.
  • true: The multi-tenant mode is used.

true

spark.thriftserver.proxy.maxThriftServerPerTenancy

Specifies the maximum number of JDBCServer instances that can be started by a tenant in multi-tenant mode.

1

spark.thriftserver.proxy.maxSessionPerThriftServer

Specifies the maximum number of sessions in a single JDBCServer instance in multi-tenant mode. If the number of sessions exceeds this value and the number of JDBCServer instances does not exceed the upper limit, a new JDBCServer instance is started. Otherwise, an alarm log is output.

50

spark.thriftserver.proxy.sessionWaitTime

Specifies the wait time before a JDBCServer instance is stopped when it has no session connections in multi-tenant mode.

180000

spark.thriftserver.proxy.sessionThreshold

In multi-tenant mode, when the session usage (formula: number of current sessions/spark.thriftserver.proxy.maxSessionPerThriftServer x number of current JDBCServer instances) of the JDBCServer instance reaches the threshold, a new JDBCServer instance is automatically added.

100

spark.thriftserver.proxy.healthcheck.period

Specifies the period of JDBCServer health checks conducted by the JDBCServer proxy in multi-tenant mode.

60000

spark.thriftserver.proxy.healthcheck.recheckTimes

Specifies the number of JDBCServer health check retries conducted by the JDBCServer proxy in multi-tenant mode.

3

spark.thriftserver.proxy.healthcheck.waitTime

Specifies the wait time for JDBCServer to respond to a health check request sent by the JDBCServer proxy.

10000

spark.thriftserver.proxy.session.check.interval

Specifies the period of JDBCServer proxy sessions in multi-tenant mode.

6h

spark.thriftserver.proxy.idle.session.timeout

Specifies the idle time interval of a JDBCServer proxy session in multi-tenant mode. If no operation is performed within this period, the session is closed.

7d

spark.thriftserver.proxy.idle.session.check.operation

Specifies whether to check that operations still exist on a JDBCServer proxy session when the session is checked for expiration in multi-tenant mode.

true

spark.thriftserver.proxy.idle.operation.timeout

Specifies the timeout interval of an operation in multi-tenant mode. An operation that times out is closed.

5d

hive.spark.client.server.connect.timeout

Specifies the timeout duration for client connections in multi-tenant mode.

5min