Custom Parameters

spark.dli.metaAccess.enable

-

Enter true.

Yes

spark.dli.job.agency.name

Enter the DLI agency name.

mgc.mc2dli.data.migration.dli.file.path

Enter the OBS path for storing the migration-dli-spark-1.0.0.jar package. For example, obs://mgc-test/data/migration-dli-spark-1.0.0.jar

mgc.mc2dli.data.migration.dli.spark.jars

Enter the OBS path for storing the fastjson-1.2.54.jar and datasource.jar packages. The value is transferred in array format. Package names must be enclosed using double quotation marks and be separated with commas (,) For example, ["obs://mgc-test/data/datasource.jar","obs://mgc-test/data/fastjson-1.2.54.jar"]

spark.sql.catalog.mc_catalog.tableWriteProvider

Enter tunnel.

Mandatory for migration over private lines

spark.sql.catalog.mc_catalog.tableReadProvider

Enter tunnel.

spark.hadoop.odps.end.point

Enter the VPC endpoint of the region where the source MaxCompute service is provisioned. For details about the MaxCompute VPC endpoint in each region, see Endpoints in different regions (VPC). For example, if the source MaxCompute service is located in Hong Kong, China, enter http://service.cn-hongkong.maxcompute.aliyun-inc.com/api.

spark.hadoop.odps.tunnel.end.point

Enter the VPC Tunnel endpoint of the region where the source MaxCompute service is located. For details about the MaxCompute VPC Tunnel endpoint in each region, see Endpoints in different regions (VPC). For example, if the source MaxCompute service is located in Hong Kong, China, enter http://dt.cn-hongkong.maxcompute.aliyun-inc.com.

sc_type

A, B, and C

Compute resource type. Currently, resource types A, B, and C are available. If this parameter is not specified, the minimum configuration (type A) is used.

• A: (vCPUs: 8; memory: 32 GB; driverCores: 2; executorCores: 1; driverMemory: 7 GB; executorMemory: 4 GB; and numExecutor: 6)
• B: (vCPUs: 16; memory: 64 GB; driverCores: 2; executorCores: 2; driverMemory: 7 GB; executorMemory: 8 GB; and numExecutor: 7)
• C: (vCPUs: 32; memory: 128 GB; driverCores: 4; executorCores: 2; driverMemory: 15 GB; executorMemory: 8 GB; and numExecutor: 14)

No

executorCores

1- 4

Number of CPU cores of each Executor in the Spark application. This configuration will replace the default setting in sc_type.

numExecutors

1-100

Number of Executors in a Spark application. This configuration will replace the default setting in sc_type.

executorMemory

1-16 G

Executor memory of the Spark application, for example, 2 GB or 2048 MB. This configuration will replace the default setting in sc_type. The unit must be provided. Otherwise, the startup fails.

driverCores

1-4

Number of CPU cores of the Spark application driver. This configuration will replace the default setting in sc_type.

driverMemory

1-16 G

Driver memory of the Spark application, for example, 2 GB or 2048 MB. This configuration will replace the default setting in sc_type. The unit must be provided. Otherwise, the startup fails.