Managing ENV

Scenario

To capture data to or from Hudi, create and manage Hudi environment variables by performing the operations in this section.

Prerequisites

A user with CDL management operation permissions has been created by referring to Creating a CDL User for the cluster with Kerberos authentication enabled.

Procedure

1. Log in to the CDLService web UI as a user with CDL management operation permissions or as user admin (for clusters with Kerberos authentication disabled) by referring to Logging In to the CDLService Web UI.
2. Choose ENV Management and click Add Env. In the displayed dialog box, set related parameters.

   Table 1 Parameters for adding an ENV

   Parameter	Description	Example Value
   Name	ENV name	spark-env
   Type	ENV type	spark
   Driver Memory	Memory for the driver process, in GB ( by default).	1 GB
   Executor Memory	Memory size for each Executor process, in GB by default. Its string format is the same as that of JVM.	1 GB
   Executor Cores	Number of CPU cores occupied by each Executor	1
   Number Executors	Number of Executors	1
   Queue	Name of the Yarn tenant queue. Kerberos authentication is enabled for the cluster (the cluster is in security mode). If this parameter is left blank, by default, the request is submitted to the queue bound to the user specified by Pricipal during Hudi connection creation. If the user is not bound to any queue, the request is submitted to the default queue. Kerberos authentication is disabled for the cluster (the cluster is in normal mode). Jobs are submitted to the default queue if this parameter is not specified.	-
   Description	ENV description	-

3. Click OK.

   After the ENV is created, you can click Edit or Delete in the Operation column to edit or delete the ENV, respectively.