Updated on 2026-06-27 GMT+08:00

Using Iceberg from Scratch

Apache Iceberg is an open table format for huge analytic datasets. Iceberg adds table functions to compute engines including Spark, HetuEngine, and Flink using a high-performance table format that works just like a SQL table.

Iceberg was built for huge tables. Iceberg is used in production where a single table can contain tens of petabytes of data and even these huge tables can be read without a distributed SQL engine. It provides the following functions:

Fast scan planning enables table reads and file searches without relying on a distributed SQL engine.
Advanced filtering uses table metadata to prune data files based on partition and column-level statistics.
Iceberg was designed to solve correctness problems in eventually-consistent cloud object stores. It works with any cloud store and reduces NameNode congestion when in HDFS, by avoiding listing and renames.
Serializable isolation enables table changes to be atomic, and readers never see partial or uncommitted changes.
Multiple concurrent writers use optimistic concurrency and will retry to ensure that compatible updates succeed, even when writes conflict.

Prerequisites

The Spark component has been installed.
If you need to register tables with the Hive MetaStore during catalog creation, the Hive component must have been installed in the cluster.

Procedure

Install the client. For details, see .
Log in to the node where the client is installed as the client installation user.
Run the following command to switch to the client installation directory:
```
cd Client installation directory
```
Run the following commands to initialize the environment variables:
```
source bigdata_env
source Spark/component_env
```
Run the following command to perform Kerberos authentication if Kerberos authentication is enabled for the cluster (the cluster is in security mode). If Kerberos authentication is disabled for the cluster (the cluster is in normal mode), skip this step.
```
kinit Created component service user
```
Start Iceberg.
- Scenario 1
  If you need to register tables with the Hive MetaStore and use table formats (such as Parquet and ORC) supported by Spark and Iceberg tables, you can use SparkSessionCatalog. The catalog name can only be spark_catalog.
```
spark-sql \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
--conf spark.sql.catalog.spark_catalog.type=hive \
--conf spark.sql.storeAssignmentPolicy=ANSI
```
- Scenario 2
  If you need to register tables with the Hive MetaStore and use only Iceberg tables, you can use SparkCatalog. You can specify a custom catalog name, such as prod.
```
spark-sql \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.prod=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.prod.type=hive \
--conf spark.sql.storeAssignmentPolicy=ANSI
```
In the preceding command:
- org.apache.iceberg.spark.SparkSessionCatalog adds support for Iceberg tables and Spark's native table formats. When SparkSessionCatalog is used, the catalog name can only be spark_catalog.
- org.apache.iceberg.spark.SparkCatalog supports only Iceberg tables.
Create an Iceberg table.
1. Use the catalog (for example, prod) specified when spark-sql is started in Scenario 2. Create a database, for example, db.
```
create database prod.db;
use prod.db;
```
2. Create an Iceberg table.
```
CREATE TABLE prod.db.table (id bigint, data string) USING iceberg;
```

Write data to the Iceberg table.

Insert data.

Method 1:

INSERT INTO prod.db.table VALUES (1, 'a'), (2, 'b'), (3, 'c');

Method 2:

Create the table named source.

CREATE TABLE IF NOT EXISTS source (id bigint, data string) USING iceberg;

Insert data into the table named source.

INSERT INTO source (id, data) VALUES(1, 'a'), (2, 'bc'), (3, 'd');

Insert data into the prod.db.table table.

INSERT INTO prod.db.table SELECT id, data FROM source WHERE length(data) = 1;

Row-level SQL data update:

Create the table named prod.db.target.

CREATE TABLE IF NOT EXISTS prod.db.target (id bigint,  count INT) USING iceberg;

Create the table named updates.

CREATE TABLE IF NOT EXISTS updates (id bigint,  count INT)  USING parquet;

Insert data into the updates table.

INSERT INTO updates VALUES (1, 2), (3, 4);

Perform the row-level SQL data update.

MERGE INTO prod.db.target t USING (SELECT * FROM updates) u ON t.id = u.id
WHEN MATCHED THEN UPDATE SET t.count = t.count + u.count
WHEN NOT MATCHED THEN INSERT *;

Insert data by column.

INSERT INTO prod.db.table  SELECT id, data FROM source;

Read data from the Iceberg table.
- Query data in the table.
```
SELECT count(1) as count, data FROM prod.db.table GROUP BY data;
```
- View all metadata in the table.
```
DESCRIBE EXTENDED prod.db.table;
```
- Obtain the total number of rows in a specified table.
```
SELECT COUNT(*) FROM prod.db.table;
```

Parent topic: Using Iceberg (MRS 3.6.0.1 or Later)

Using Iceberg from Scratch

Prerequisites

Procedure

What is your overall rating for this page?