更新时间:2026-11-03 GMT+08:00
分享

Spark SQL作业提交

功能说明

本节操作介绍使用AI DataLake SDK提交SQL作业的操作方法。

前提条件

函数

SparkSqlJob
# 同步提交函数
submit();

# 异步提交函数
asyncSubmit();

样例代码

  • 异步提交SQL
    public static void asyncSubmit(AIDataLakeSparkSqlClient sparkSqlClient) throws AIDataLakeException {
        String databaseName = "DATABASE_NAME";
        String catalogName = "CATALOG_NAME";
        String sqlStatement = "SELECT xxx";
        Map<String, String> sparkConfig = new HashMap<>();
        // spark conf starting with "spark."
        sparkConfig.put("spark.sql.shuffle.partitions", "200");
        SparkSqlJob sqlJob = sparkSqlClient.newSparkSqlJobBuilder(sqlStatement)
                        .catalogName(catalogName)
                        .databaseName(databaseName)
                        .sparkConfig(sparkConfig)
                        .build();
        sqlJob.asyncSubmit();
    }
  • 同步提交SQL
    public static void submit(AIDataLakeSparkSqlClient sparkSqlClient) throws AIDataLakeException {
        String databaseName = "DATABASE_NAME";
        String catalogName = "CATALOG_NAME";
        String sqlStatement = "SELECT xxx";
        Map<String, String> sparkConfig = new HashMap<>();
        // spark conf starting with "spark."
        sparkConfig.put("spark.sql.shuffle.partitions", "200");
        SparkSqlJob sqlJob = sparkSqlClient.newSparkSqlJobBuilder(sqlStatement)
                        .catalogName(catalogName)
                        .databaseName(databaseName)
                        .sparkConfig(sparkConfig)
                        .build();
        sqlJob.submit();
    }
  • SQL查询
    public static void submit(AIDataLakeSparkSqlClient sparkSqlClient) throws AIDataLakeException {
        String databaseName = "DATABASE_NAME";
        String catalogName = "CATALOG_NAME";
        String sqlStatement = "SELECT xxx";
        String statementId;
        Map<String, String> sparkConfig = new HashMap<>();
        // spark conf starting with "spark."
        sparkConfig.put("spark.sql.shuffle.partitions", "200");
        SparkSqlJob sqlJob = sparkSqlClient.newSparkSqlJobBuilder(sqlStatement)
                        .catalogName(catalogName)
                        .databaseName(databaseName)
                        .sparkConfig(sparkConfig)
                        .build();
        sqlJob.submit();
        statementId = sqlJob.getStatementId();
        ResultSet resultSet = sparkSqlClient.getResultsWithStatementId(statementId).getResultSet();
        while (resultSet.hasNext()) {
            Row row = resultSet.read();
            List<Column> schema = row.getSchema();
            List<Object> record = row.getRecord();
            for (int i = 0; i < schema.size(); i++) {
                DataType.TypeName typeName = DataType.TypeName.fromName(schema.get(i).getType().getName());
                switch (typeName) {
                    case STRING:
                        String strV = (String) record.get(i);
                        break;
                    case DATE:
                        Date dtV = (Date) record.get(i);
                        break;
                }
            }
        }
    }

参数说明

表1 参数详情

参数

必须配置

默认值

描述

databaseName

是

无

默认LakeFormation数据库名称。

catalogName

是

无

默认LakeFormation Catalog名称。

sqlStatement

是

无

待提交的SQL语句。

相关文档