Updated on 2024-07-19 GMT+08:00

How Do I Migrate Data from OBS/S3 to ClickHouse?

Question

How do I migrate data from OBS/S3 to ClickHouse?

Answer

  1. Query data stored in S3.
    select * from s3(path [,aws_access_key_id, aws_secret_access_key] [,format] [,structure])

    Parameter description:

    • path: The bucket URL with a file path.
    • format: The file format.
    • aws_access_key_id and aws_secret_access_key: Long-term credentials of an account. Credentials can be used to authenticate requests. These parameter are optional. If no credential is specified, credentials are read from the configuration file.
    • structure: The table schema.

  2. ClickHouse obtains data from S3 to create a table.

    CREATE TABLE test1_s3 (name String, value UInt32) ENGINE = S3(path, [aws_access_key_id, aws_secret_access_key,] format)

  3. View the created table.

    select * from test1_s3