Help Center/ MapReduce Service/ Component Operation Guide (LTS)/ Using Spark/Spark2x/ Spark Core Enterprise-Class Enhancements/ Configuring Spark HA to Enhance HA/ Configuring the Spark Multi-Tenant Mode
Updated on 2024-10-09 GMT+08:00
View PDF
Share

Configuring the Spark Multi-Tenant Mode

Scenarios

In multi-tenant mode, JDBCServers are bound with tenants. Each tenant corresponds to one or more JDBCServers, and a JDBCServer provides services for only one tenant. Different tenants can be configured with different Yarn queues to implement resource isolation.

Configuration Description

Log in to Manager and choose Cluster > Services > Spark2x. Click Configurations, click All Configurations, and search for and modify the following parameters:

Table 1 Parameter description

Parameter

Description

Default Value

spark.proxyserver.hash.enabled

Specifies whether to connect to ProxyServer using the Hash algorithm.

  • true indicates using the Hash algorithm. In multi-tenant mode, this parameter must be configured to true.
  • false indicates using random connection. In multi-active instance mode, this parameter must be configured to false.

true

NOTE:

After this parameter is modified, you need to download the client again.

spark.thriftserver.proxy.enabled

Specifies whether to use the multi-tenant mode.

  • false: The multi-instance mode is used.
  • true: The multi-tenant mode is used.

true

spark.thriftserver.proxy.maxThriftServerPerTenancy

Specifies the maximum number of JDBCServer instances that can be started by a tenant in multi-tenant mode.

1

spark.thriftserver.proxy.maxSessionPerThriftServer

Specifies the maximum number of sessions in a single JDBCServer instance in multi-tenant mode. If the number of sessions exceeds this value and the number of JDBCServer instances does not exceed the upper limit, a new JDBCServer instance is started. Otherwise, an alarm log is output.

50

spark.thriftserver.proxy.sessionWaitTime

Specifies the wait time before a JDBCServer instance is stopped when it has no session connections in multi-tenant mode.

180000

spark.thriftserver.proxy.sessionThreshold

In multi-tenant mode, when the session usage (formula: number of current sessions/spark.thriftserver.proxy.maxSessionPerThriftServer x number of current JDBCServer instances) of the JDBCServer instance reaches the threshold, a new JDBCServer instance is automatically added.

100

spark.thriftserver.proxy.healthcheck.period

Specifies the period of JDBCServer health checks conducted by the JDBCServer proxy in multi-tenant mode.

60000

spark.thriftserver.proxy.healthcheck.recheckTimes

Specifies the number of JDBCServer health check retries conducted by the JDBCServer proxy in multi-tenant mode.

3

spark.thriftserver.proxy.healthcheck.waitTime

Specifies the wait time for JDBCServer to respond to a health check request sent by the JDBCServer proxy.

10000

spark.thriftserver.proxy.session.check.interval

Specifies the period of JDBCServer proxy sessions in multi-tenant mode.

6h

spark.thriftserver.proxy.idle.session.timeout

Specifies the idle time interval of a JDBCServer proxy session in multi-tenant mode. If no operation is performed within this period, the session is closed.

7d

spark.thriftserver.proxy.idle.session.check.operation

Specifies whether to check that operations still exist on a JDBCServer proxy session when the session is checked for expiration in multi-tenant mode.

true

spark.thriftserver.proxy.idle.operation.timeout

Specifies the timeout interval of an operation in multi-tenant mode. An operation that times out is closed.

5d