Help Center/ PanguLargeModels/ User Guide/ Using Data Engineering to Create a Dataset/ Processing Datasets/ Managing Processing Task Resources
Updated on 2025-07-02 GMT+08:00
View PDF
Share

Managing Processing Task Resources

When creating MRS-related processing or publishing tasks, you can set task resource parameters. For details about the parameters, see Table 1.

Table 1 Task resource parameters

Parameter Name

Description

executorCores

Number of CPU cores of each Executor in a Spark application.

numExecutors

Number of Executors in a Spark application.

executorMemory

Executor memory of a Spark application, for example, 2 GB or 2048 MB.

driverCores

Number of CPU cores of the Spark application driver. This configuration will replace the default setting in sc_type.

driverMemory

Driver memory of a Spark application, for example, 2 GB or 2048 MB.
Parent topic: Processing Datasets