Configuring CDL ENV Variables

Updated on 2024-10-25 GMT+08:00

View PDF

Scenario

To capture data to or from Hudi, create and manage Hudi environment variables by performing the operations in this section.

Prerequisites

A user with the CDL management permission has been created for the cluster with Kerberos authentication enabled. For details, see CDL User Permission Management.

Procedure

Access the CDLService web UI as a user with the CDL management permissions or the admin user (for the cluster where Kerberos authentication is not enabled). For details, see Logging In to the CDLService WebUI.

Choose ENV Management and click Add Env. In the displayed dialog box, set related parameters.

**Table 1** Parameters for adding an ENV
Parameter	Description	Example Value
Name	ENV name	spark-env
Type	ENV type	spark
Driver Memory	Memory for the driver process, in GB ( by default).	1 GB
Executor Memory	Memory size for each Executor process, in GB by default. Its string format is the same as that of JVM.	1 GB
Executor Cores	Number of CPU cores occupied by each Executor	1
Number Executors	Number of Executors	1
Queue	Name of the Yarn tenant queue. Jobs are submitted to the default queue if this parameter is not specified.	-
Description	ENV description	-