SQL Syntax Overview of Batch Jobs

This section describes the Spark SQL syntax list provided by DLI. For details about the parameters and examples, see the syntax description.

Database-related Syntax

Table 1 Database-related Syntax

Function

Syntax

Creating a Database

1
2
3
CREATE [DATABASE | SCHEMA] [IF NOT EXISTS] db_name
  [COMMENT db_comment]
  [WITH DBPROPERTIES (property_name=property_value, ...)];

Deleting a Database

1
DROP [DATABASE | SCHEMA] [IF EXISTS] db_name [RESTRICT|CASCADE];

Viewing a Specified Database

1
DESCRIBE DATABASE [EXTENDED] db_name;

Viewing All Databases

1
SHOW [DATABASES | SCHEMAS] [LIKE regex_expression];

Syntax for Creating an OBS Table

Table 2 Syntax for creating an OBS table

Function

Syntax

Creating an OBS Table Using the Datasource Syntax

1
2
3
4
5
6
7
CREATE TABLE [IF NOT EXISTS] [db_name.]table_name 
  [(col_name1 col_type1 [COMMENT col_comment1], ...)]
  USING file_format 
  [OPTIONS (path 'obs_path', key1=val1, key2=val2, ...)] 
  [PARTITIONED BY (col_name1, col_name2, ...)]
  [COMMENT table_comment]
  [AS select_statement];

Creating an OBS Table Using the Hive Syntax

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name 
  [(col_name1 col_type1 [COMMENT col_comment1], ...)]
  [COMMENT table_comment] 
  [PARTITIONED BY (col_name2 col_type2, [COMMENT col_comment2], ...)] 
  [ROW FORMAT row_format]
  [STORED AS file_format] 
  LOCATION 'obs_path'
  [AS select_statement];

row_format:
  : SERDE serde_cls [WITH SERDEPROPERTIES (key1=val1, key2=val2, ...)]
  | DELIMITED [FIELDS TERMINATED BY char [ESCAPED BY char]]
      [COLLECTION ITEMS TERMINATED BY char]
      [MAP KEYS TERMINATED BY char]
      [LINES TERMINATED BY char]
      [NULL DEFINED AS char]

Syntax for Creating a DLI Table

Table 3 Syntax for creating a DLI table

Function

Syntax

Creating a DLI Table Using the Datasource Syntax

1
2
3
4
5
6
7
CREATE TABLE [IF NOT EXISTS] [db_name.]table_name 
  [(col_name1 col_type1 [COMMENT col_comment1], ...)]
  USING file_format 
  [OPTIONS (key1=val1, key2=val2, ...)] 
  [PARTITIONED BY (col_name1, col_name2, ...)]
  [COMMENT table_comment]
  [AS select_statement];

Creating a DLI Table Using the Hive Syntax

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
CREATE TABLE [IF NOT EXISTS] [db_name.]table_name 
  [(col_name1 col_type1 [COMMENT col_comment1], ...)]
  [COMMENT table_comment] 
  [PARTITIONED BY (col_name2 col_type2, [COMMENT col_comment2], ...)] 
  [ROW FORMAT row_format]
  STORED AS file_format 
  [TBLPROPERTIES (key1=val1, key2=val2, ...)]
  [AS select_statement];

row_format:
  : SERDE serde_cls [WITH SERDEPROPERTIES (key1=val1, key2=val2, ...)]
  | DELIMITED [FIELDS TERMINATED BY char [ESCAPED BY char]]
      [COLLECTION ITEMS TERMINATED BY char]
      [MAP KEYS TERMINATED BY char]
      [LINES TERMINATED BY char]
      [NULL DEFINED AS char]

Syntax for Deleting a Table

Table 4 Syntax for Deleting a Table

Function

Syntax

Deleting a Table

1
DROP TABLE [IF EXISTS] [db_name.]table_name;

Syntax for Viewing a Table

Table 5 Syntax for viewing a table

Function

Syntax

Viewing All Tables

1
SHOW TABLES [IN | FROM db_name] [LIKE regex_expression];

Viewing Table Creation Statements

1
SHOW CREATE TABLE table_name;

Viewing Table Properties

1
SHOW TBLPROPERTIES table_name [('property_name')];

Viewing All Columns in a Specified Table

1
SHOW COLUMNS {FROM | IN} table_name [{FROM | IN} db_name];

Viewing All Partitions in a Specified Table

1
2
SHOW PARTITIONS [db_name.]table_name
  [PARTITION partition_specs];

Viewing Table Statistics

1
DESCRIBE [EXTENDED|FORMATTED] [db_name.]table_name;

Syntax for Modifying a Table

Table 6 Syntax for modifying a table

Function

Syntax

Adding a Column

1
ALTER TABLE [db_name.]table_name ADD COLUMNS (col_name1 col_type1 [COMMENT col_comment1], ...);

Syntax for Partitioning a Table

Table 7 Syntax for partitioning a table

Function

Syntax

Adding a Partition (Only OBS Tables Supported)

1
2
3
4
5
ALTER TABLE table_name ADD [IF NOT EXISTS]
  PARTITION partition_specs1
  [LOCATION 'obs_path1']
  PARTITION partition_specs2
  [LOCATION 'obs_path2'];

Renaming a Partition

1
2
3
ALTER TABLE table_name
  PARTITION partition_specs
  RENAME TO PARTITION partition_specs;

Deleting a Partition

1
2
3
ALTER TABLE [db_name.]table_name
  DROP [IF EXISTS]
  PARTITION partition_spec1[,PARTITION partition_spec2,...];

Altering the Partition Location of a Table (Only OBS Tables Supported)

1
2
3
ALTER TABLE table_name
  PARTITION partition_specs
  SET LOCATION obs_path;

Modifying the SerDe Attribute of a Table Partition (Only OBS Tables Supported)

1
2
3
4
ALTER TABLE table_name
 [PARTITION partition_specs]
 SET SERDE serde
 [WITH SERDEPROPERTIES (property_name=property_value,...)];
ALTER TABLE table_name
 [PARTITION partition_specs]
 SET SERDEPROPERTIES (property_name=property_value,...);

Updating Partitioned Table Data (Only OBS Tables Supported)

1
2
MSCK REPAIR TABLE table_name;
ALTER TABLE table_name RECOVER PARTITIONS;

Syntax for Importing Data

Table 8 Syntax for importing data

Function

Syntax

Importing Data

1
2
LOAD DATA INPATH 'folder_path' INTO TABLE [db_name.]table_name
  OPTIONS(property_name=property_value, ...);

Syntax for Inserting Data

Table 9 Syntax for inserting data

Function

Syntax

Inserting Data

  • Insert the SELECT query result into a table.
    1
    2
    INSERT INTO [TABLE] [db_name.]table_name
      [PARTITION part_spec] select_statement;
    
    1
    2
    INSERT OVERWRITE TABLE [db_name.]table_name
      [PARTITION part_spec] select_statement;
    
    part_spec:  
      : (part_col_name1=val1 [, part_col_name2=val2, ...])
  • Insert a data record into a table.
    1
    2
    INSERT INTO [TABLE] [db_name.]table_name
      [PARTITION part_spec] VALUES values_row [, values_row ...];
    
    1
    2
    INSERT OVERWRITE TABLE [db_name.]table_name
      [PARTITION part_spec] VALUES values_row [, values_row ...];
    
    values_row:
      : (val1 [, val2, ...])

Syntax for Clearing Data

Table 10 Syntax for clearing data

Function

Syntax

Clearing Data

1
TRUNCATE TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)];

Syntax for Exporting Query Results

Table 11 Syntax for exporting query results

Function

Syntax

Exporting Query Result

1
2
3
4
INSERT OVERWRITE DIRECTORY path
  USING file_format
  [OPTIONS(key1=value1)]
  select_statement;

Syntax for Datasource Connections

Table 12 Syntax related to HBase tables

Function

Syntax

Creating a DLI Table and Associating It with HBase

  • Single row key
    1
    2
    3
    4
    5
    6
    7
    8
    9
    CREATE TABLE [IF NOT EXISTS] TABLE_NAME (
      ATTR1 TYPE,
      ATTR2 TYPE,
      ATTR3 TYPE)
      USING [CLOUDTABLE | HBASE] OPTIONS (
      'ZKHost'='xx',
      'TableName'='TABLE_IN_CLOUDTABLE',
      'RowKey'='ATTR1',
      'Cols'='ATTR2:CF1.C1, ATTR3:CF1.C2');
    
  • Combined row key
     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    CREATE TABLE [IF NOT EXISTS] TABLE_NAME (
      ATTR1 String,
      ATTR2 String,
      ATTR3 TYPE)
      USING [CLOUDTABLE | HBASE] OPTIONS (
      'ZKHost'='xx',
      'TableName'='TABLE_IN_CLOUDTABLE',
      'RowKey'='ATTR1:2, ATTR2:10',
      'Cols'='ATTR2:CF1.C1, ATTR3:CF1.C2',
      'krbauth'='KRB_AUTH_NAME');
    

Inserting Data to an HBase Table

1
2
3
4
5
6
7
INSERT INTO DLI_TABLE
  SELECT field1,field2...
  [FROM DLI_TEST]
  [WHERE where_condition]
  [LIMIT num]
  [GROUP BY field]
  [ORDER BY field] ...;

Querying an HBase Table

1
SELECT * FROM table_name LIMIT number;
Table 13 Syntax related to OpenTSDB tables

Function

Syntax

Creating a DLI Table and Associating It with OpenTSDB

1
2
3
4
5
CREATE TABLE [IF NOT EXISTS] UQUERY_OPENTSDB_TABLE_NAME
  USING OPENTSDB OPTIONS (
  'host' = 'xx;xx',
  'metric' = 'METRIC_NAME',
  'tags' = 'TAG1,TAG2');

Inserting Data to the OpenTSDB Table

1
INSERT INTO TABLE TABLE_NAME SELECT * FROM DLI_TABLE;
1
INSERT INTO TABLE TABLE_NAME VALUES(XXX);

Querying an OpenTSDB Table

1
SELECT * FROM table_name LIMIT number;
Table 14 Syntax related to DWS tables

Function

Syntax

Creating a DLI Table and Associating It with DWS

1
2
3
4
5
6
CREATE TABLE [IF NOT EXISTS] TABLE_NAME
  USING DWS OPTIONS (
  'url'='xx',
  'dbtable'='db_name_in_DWS.table_name_in_DWS',
  'passwdauth' = 'xxx',
  'encryption' = 'true');

Inserting Data to the DWS Table

  • Insert the SELECT query result into a table.
    1
    2
    3
    4
    5
    6
    7
    INSERT INTO DLI_TABLE
      SELECT field1,field2...
      [FROM DLI_TEST]
      [WHERE where_condition]
      [LIMIT num]
      [GROUP BY field]
      [ORDER BY field] ...;
    
  • Insert a data record into a table.
    1
    2
    INSERT INTO DLI_TABLE
      VALUES values_row [, values_row ...];
    

Querying the DWS Table

1
SELECT * FROM table_name LIMIT number;
Table 15 Syntax related to RDS tables

Function

Syntax

Creating a DLI Table and Associating It with RDS

1
2
3
4
5
6
7
CREATE TABLE [IF NOT EXISTS] TABLE_NAME
  USING JDBC OPTIONS (
  'url'='xx',
  'driver'='DRIVER_NAME',
  'dbtable'='db_name_in_RDS.table_name_in_RDS',
  'user'='YOUR_RDS_USER_NAME',
  'password'='YOUR_PASS_WORD');

Inserting Data to the RDS Table

  • Insert the SELECT query result into a table.
    1
    2
    3
    4
    5
    6
    7
    INSERT INTO DLI_TABLE
      SELECT field1,field2...
      [FROM DLI_TEST]
      [WHERE where_condition]
      [LIMIT num]
      [GROUP BY field]
      [ORDER BY field] ...;
    
  • Insert a data record into a table.
    1
    2
    INSERT INTO DLI_TABLE
      VALUES values_row [, values_row ...];
    

Querying the RDS Table

1
SELECT * FROM table_name LIMIT number;
Table 16 Syntax related to CSS tables

Function

Syntax

Creating a DLI Table and Associating It with CSS

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
CREATE TABLE [IF NOT EXISTS] TABLE_NAME(
  FIELDNAME1 FIELDTYPE1,
  FIELDNAME2 FIELDTYPE2)
  USING CSS OPTIONS (
  'es.nodes'='xx',
  'resource'='type_path_in_CSS',
  'pushdown'='true',
  'strict'='false',
  'batch.size.entries'= '1000',
  'batch.size.bytes'= '1mb',
  'es.nodes.wan.only' = 'true',
  'es.mapping.id' = 'FIELDNAME');

Inserting Data to the CSS Table

  • Insert the SELECT query result into a table.
    1
    2
    3
    4
    5
    6
    7
    INSERT INTO DLI_TABLE
      SELECT field1,field2...
      [FROM DLI_TEST]
      [WHERE where_condition]
      [LIMIT num]
      [GROUP BY field]
      [ORDER BY field] ...;
    
  • Insert a data record into a table.
    1
    2
    INSERT INTO DLI_TABLE
      VALUES values_row [, values_row ...];
    

Querying the CSS Table

1
SELECT * FROM table_name LIMIT number;
Table 17 Syntax related to DCS tables

Function

Syntax

Creating a DLI Table and Associating It with DCS

  • Specified key
     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    CREATE TABLE [IF NOT EXISTS] TABLE_NAME(
        FIELDNAME1 FIELDTYPE1,
        FIELDNAME2 FIELDTYPE2)
      USING REDIS OPTIONS (
      'host'='xx',
      'port'='xx',
      'passwdauth' = 'xxx',
      'encryption' = 'true',
      'table'='namespace_in_redis:key_in_redis',
      'key.column'= 'FIELDNAME1'
    );
    
  • Wildcard key
     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    CREATE TABLE [IF NOT EXISTS] TABLE_NAME(
        FIELDNAME1 FIELDTYPE1,
        FIELDNAME2 FIELDTYPE2)
      USING REDIS OPTIONS (
      'host'='xx',
      'port'='xx',
      'passwdauth' = 'xxx',
      'encryption' = 'true',
      'keys.pattern'='key*:*',
      'key.column'= 'FIELDNAME1'
    );
    

Inserting Data to the DCS Table

  • Insert the SELECT query result into a table.
    1
    2
    3
    4
    5
    6
    7
    INSERT INTO DLI_TABLE
      SELECT field1,field2...
      [FROM DLI_TEST]
      [WHERE where_condition]
      [LIMIT num]
      [GROUP BY field]
      [ORDER BY field] ...;
    
  • Insert a data record into a table.
    1
    2
    INSERT INTO DLI_TABLE
      VALUES values_row [, values_row ...];
    

Querying the DCS Table

1
SELECT * FROM table_name LIMIT number;
Table 18 Syntax related to DDS tables

Function

Syntax

Creating a DLI Table and Associating It with DDS

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
CREATE TABLE [IF NOT EXISTS] TABLE_NAME(
    FIELDNAME1 FIELDTYPE1,
    FIELDNAME2 FIELDTYPE2)
  USING MONGO OPTIONS (
  'url'='IP:PORT[,IP:PORT]/[DATABASE][.COLLECTION][AUTH_PROPERTIES]',
  'database'='xx',
  'collection'='xx',
  'passwdauth' = 'xxx',
  'encryption' = 'true'
);

Inserting Data to the DDS Table

  • Insert the SELECT query result into a table.
    1
    2
    3
    4
    5
    6
    7
    INSERT INTO DLI_TABLE
      SELECT field1,field2...
      [FROM DLI_TEST]
      [WHERE where_condition]
      [LIMIT num]
      [GROUP BY field]
      [ORDER BY field] ...;
    
  • Overwriting the inserted data
    1
    2
    3
    4
    5
    6
    7
    INSERT OVERWRITE TABLE DLI_TABLE
      SELECT field1,field2...
      [FROM DLI_TEST]
      [WHERE where_condition]
      [LIMIT num]
      [GROUP BY field]
      [ORDER BY field] ...;
    

Querying the DDS Table

1
SELECT * FROM table_name LIMIT number;

View-related Syntax

Table 19 View-related syntax

Function

Syntax

Creating a View

1
CREATE [OR REPLACE] VIEW view_name AS select_statement;

Deleting a View

1
DROP VIEW [IF EXISTS] [db_name.]view_name;

Syntax for Viewing the Execution Plan

Table 20 Syntax for viewing the execution plan

Function

Syntax

Viewing the Execution Plan

1
EXPLAIN [EXTENDED | CODEGEN] statement;

Syntax Related to Data Permissions

Table 21 Syntax related to data permissions

Function

Syntax

Creating a Role

1
CREATE ROLE [db_name].role_name;

Deleting a Role

1
DROP ROLE [db_name].role_name;

Binding a Role

1
GRANT ([db_name].role_name,...) TO (user_name,...);

Unbinding a Role

1
REVOKE ([db_name].role_name,...) FROM (user_name,...);

Displaying a Role

1
SHOW [ALL] ROLES [user_name];

Granting a Permission

1
GRANT (privilege,...) ON (resource,..) TO ((ROLE [db_name].role_name) | (USER user_name)),...);

Revoking a Permission

1
REVOKE (privilege,...) ON (resource,..) FROM ((ROLE [db_name].role_name) | (USER user_name)),...);

Displaying the Granted Permissions

1
SHOW GRANT ((ROLE [db_name].role_name) | (USER user_name)) ON resource;

Displaying the Binding Relationship Between All Roles and Users

1
SHOW PRINCIPALS ROLE;

Syntax related to user-defined functions

Table 22 Syntax related to user-defined functions

Function

Syntax

Creating a Function

1
2
3
4
5
CREATE [TEMPORARY] FUNCTION [db_name.]function_name AS class_name
  [USING resource,...]

resource: 
  : (JAR|FILE|ARCHIVE)file_uri

Deleting a Function

1
DROP [TEMPORARY] FUNCTION [IF EXISTS] [db_name.] function_name;

Displaying Function Details

1
DESCRIBE FUNCTION [EXTENDED] [db_name.] function_name;

Displaying All Functions

1
SHOW [USER|SYSTEM|ALL] FUNCTIONS ([LIKE] regex | [db_name.] function_name);