Apache Iceberg — The Lakehouse Handbook

Most Iceberg material is either a vendor's marketing page, a dense spec, or a $200 course you watch once and forget. You don't need any of them. This is the whole Apache Iceberg table format — creating tables, writing, time travel, MERGE, schema and partition evolution, maintenance and tuning — 100% free, no signup, no paywall.

Treat it as a handbook you keep coming back to. Googled "iceberg time travel" or "iceberg merge into example"? Land on the exact topic, get a plain-English definition, the command syntax, and the N ways to actually use it — then stay and learn the module around it. Start below, or jump to any module in the contents on the left.

For years, a "table" on a data lake was really just a directory of files plus a Hive metastore entry pointing at it. That worked until it didn't. The Hive-table model has four expensive problems:

1. No atomic commits. A job that writes 500 files can be seen half-finished by a reader — there's no single moment where the change becomes visible. 2. Slow, unreliable listing. To plan a query the engine has to LIST directories on object storage, which is slow and eventually consistent on S3. 3. Unsafe schema changes. Renaming or reordering a column can silently corrupt reads, because columns are matched by name or position in the files. 4. Partitioning is rigid and leaky. You physically lay data out as dt=2025-01-01/ and every query must know to filter on that column, or it scans everything.

Apache Iceberg is an open table format: a specification for the metadata that turns a pile of data files into a real, versioned, transactional table. Any engine that implements the spec — Spark, Flink, Trino, Dremio, Snowflake, DuckDB, StarRocks — reads and writes the same table with consistent results. No single vendor owns it.

Table format vs file format

A file format (Parquet) decides how one file is encoded. A table format (Iceberg) decides how many files become one table — which files are current, what the schema is, how it's partitioned, and what every past version looked like. Iceberg sits above Parquet, not instead of it.

This is the one diagram worth memorising. Everything Iceberg does — atomic commits, time travel, hidden partitioning, cheap schema changes — falls out of this four-level tree. A read or write walks it top to bottom:

catalog                      → points to the CURRENT metadata file for the table
  metadata.json              → schema, partition spec, snapshot list, current snapshot
    manifest list (avro)     → one entry per manifest, with partition range stats
      manifest (avro)        → one entry per data file, with per-column stats
        data files (parquet) → the actual rows

A commit is a pointer swap. When you write, Iceberg creates new data files, a new manifest, a new manifest list, and a new metadata.json that references them — then atomically tells the catalog "the current version is now this one." Readers already running keep using the old version; new readers get the new one. Nobody ever sees a half-written table.

Two moving parts sit around an Iceberg table. The engine is what runs your query (this course uses Spark SQL, the most common). The catalog is what stores the pointer to each table's current metadata — so the engine can find it and commits stay atomic.

Common catalogs: Hive metastore, Hadoop (pointer file on the filesystem), REST (the modern open standard), AWS Glue, and Nessie (which adds Git-like branches across many tables). You'll pick one in Module 9; for now just know the catalog is the thing that says "the current version of db.orders is this metadata file."

To follow along on a real cluster later, you add the Iceberg runtime and configure a catalog. In a Spark shell that's one --packages flag plus a few configs wiring the built-in spark_catalog to Iceberg:

spark-shell \
  --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.0 \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
  --conf spark.sql.catalog.spark_catalog.type=hive

You don't need any of that to learn here, though. The cells below run a tiny SQL engine inside your browser so you see real output instantly. The Spark SQL shown on top is exactly what you'd run against Iceberg.

Here are the two tables every module uses. Run this to see the orders we'll be creating, writing, upserting and time-travelling as an Iceberg table — edit it and press Run:

-- The orders we'll manage as an Iceberg table all course long
SELECT order_id, customer_id, product, category, amount, qty, status, order_date
FROM orders
ORDER BY order_id;

And the customers they belong to — five customers across four countries, each with a loyalty tier we'll use for joins and upserts:

SELECT * FROM customers ORDER BY customer_id;