Source Table

Function

The file system connector can be used to read single files or entire directories into a single table.

When using a directory as the source path, there is no defined order of ingestion for the files inside the directory. For more information, see FileSystem SQL Connector.

Syntax

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

CREATE TABLE sink_table (
   name string,
   num INT,
   p_day string,
   p_hour string
) partitioned by (p_day, p_hour) WITH (
   'connector' = 'filesystem',
   'path' = 'obs://*** ',
   'format' = 'parquet',
   'source.monitor-interval'=''
);

Parameter Description

• Directory watching

  By default, the file system connector is bounded, that is it will scan the configured path once and then close itself.

  You can enable continuous directory watching by configuring the source.monitor-interval parameter:

  Key	Default Value	Data Type	Description
  source.monitor-interval	None	Duration	The interval in which the source checks for new files. The interval must be greater than 0. Each file is uniquely identified by its path, and will be processed once, as soon as it's discovered. The set of files already processed is kept in state during the whole lifecycle of the source, so it's persisted in checkpoints and savepoints together with the source state. Shorter intervals mean that files are discovered more quickly, but also imply more frequent listing or directory traversal of the file system/object store. If this config option is not set, the provided path will be scanned once, hence the source will be bounded.

Example

Read data from the OBS table as the data source and output it to the Print connector.

CREATE TABLE obs_source(
   name string,
   num INT,
   `file.path` STRING NOT NULL METADATA
) WITH (
   'connector' = 'filesystem',
   'path' = 'obs://demo/sink_parquent_obs',
   'format' = 'parquet',
   'source.monitor-interval'='1 h'
);


CREATE TABLE print (
   name string,
   num INT,
   path  STRING
) WITH (
   'connector' = 'print'
);

insert into print
select * from obs_source;