更新时间:2026-11-03 GMT+08:00
Spark SQL作业提交
功能说明
本节操作介绍使用AI DataLake SDK提交SQL作业的操作方法。
前提条件
- 已完成配置Java SDK环境并完成了初始化,详情请参见Spark SDK的获取与初始化
- API和SQL作业相关权限已配置,权限详情请参见权限及授权项说明。
- 使用的计算资源和端点为可用状态。
函数
SparkSqlJob
# 同步提交函数 submit(); # 异步提交函数 asyncSubmit();
样例代码
- 异步提交SQL
public static void asyncSubmit(AIDataLakeSparkSqlClient sparkSqlClient) throws AIDataLakeException { String databaseName = "DATABASE_NAME"; String catalogName = "CATALOG_NAME"; String sqlStatement = "SELECT xxx"; Map<String, String> sparkConfig = new HashMap<>(); // spark conf starting with "spark." sparkConfig.put("spark.sql.shuffle.partitions", "200"); SparkSqlJob sqlJob = sparkSqlClient.newSparkSqlJobBuilder(sqlStatement) .catalogName(catalogName) .databaseName(databaseName) .sparkConfig(sparkConfig) .build(); sqlJob.asyncSubmit(); }
- 同步提交SQL
public static void submit(AIDataLakeSparkSqlClient sparkSqlClient) throws AIDataLakeException { String databaseName = "DATABASE_NAME"; String catalogName = "CATALOG_NAME"; String sqlStatement = "SELECT xxx"; Map<String, String> sparkConfig = new HashMap<>(); // spark conf starting with "spark." sparkConfig.put("spark.sql.shuffle.partitions", "200"); SparkSqlJob sqlJob = sparkSqlClient.newSparkSqlJobBuilder(sqlStatement) .catalogName(catalogName) .databaseName(databaseName) .sparkConfig(sparkConfig) .build(); sqlJob.submit(); }
- SQL查询
public static void submit(AIDataLakeSparkSqlClient sparkSqlClient) throws AIDataLakeException { String databaseName = "DATABASE_NAME"; String catalogName = "CATALOG_NAME"; String sqlStatement = "SELECT xxx"; String statementId; Map<String, String> sparkConfig = new HashMap<>(); // spark conf starting with "spark." sparkConfig.put("spark.sql.shuffle.partitions", "200"); SparkSqlJob sqlJob = sparkSqlClient.newSparkSqlJobBuilder(sqlStatement) .catalogName(catalogName) .databaseName(databaseName) .sparkConfig(sparkConfig) .build(); sqlJob.submit(); statementId = sqlJob.getStatementId(); ResultSet resultSet = sparkSqlClient.getResultsWithStatementId(statementId).getResultSet(); while (resultSet.hasNext()) { Row row = resultSet.read(); List<Column> schema = row.getSchema(); List<Object> record = row.getRecord(); for (int i = 0; i < schema.size(); i++) { DataType.TypeName typeName = DataType.TypeName.fromName(schema.get(i).getType().getName()); switch (typeName) { case STRING: String strV = (String) record.get(i); break; case DATE: Date dtV = (Date) record.get(i); break; } } } }
参数说明
| 参数 | 必须配置 | 默认值 | 描述 |
|---|---|---|---|
| databaseName | 是 | 无 | 默认LakeFormation数据库名称。 |
| catalogName | 是 | 无 | 默认LakeFormation Catalog名称。 |
| sqlStatement | 是 | 无 | 待提交的SQL语句。 |
父主题: Spark SDK(Java)