Iceberg Catalogs Compared: REST, Polaris, Unity, Glue and Nessie

Quick Answer: An Iceberg catalog is the service that maps a table name to its current metadata file and performs the atomic compare-and-swap that makes a commit visible. It is a transaction coordinator, not a search tool. Choose by platform gravity: AWS Glue if your engines are Athena, EMR and Redshift; Unity Catalog if you are Databricks-centred; Apache Polaris when you want genuine engine and cloud neutrality; Nessie when you need Git-style branching across tables. The one rule that overrides all of it: exactly one catalog may own writes for a given table.

Most teams pick an Iceberg catalog the way they pick a logging library — quickly, and by whatever the platform defaults to. Then eighteen months later they discover the catalog is the only component in the lakehouse they cannot swap without touching every job, every dashboard and every access policy they own.

That is not an accident. Apache Iceberg deliberately made the storage layer portable and left the catalog as the one stateful, opinionated, vendor-shaped piece. Your data files are yours; your catalog is where the lock-in went.

This guide covers what a catalog actually does at the byte level, an honest comparison of the real options in 2026, the concurrency mistake that silently deletes committed data, a security advisory most catalog articles never mention, and what a migration genuinely costs. If you are new to the format itself, start with our Apache Iceberg table guide with PySpark and come back.

What an Iceberg catalog actually is

An Iceberg catalog is a pointer store with a lock. It answers one question — "for table sales.orders, what is the path of the current metadata JSON file?" — and it guarantees that only one writer at a time may change that answer.

That is the whole job. It is much smaller than people expect, and much more critical.

Everything else in an Iceberg table lives in object storage as immutable files: the metadata JSON that describes the schema and lists snapshots, the manifest lists, the manifest files, and the Parquet data files. None of it ever changes in place. A write produces new files and a new metadata JSON.

So the only mutable state in the entire system is one string: which metadata file is current. The catalog owns that string.

Why "catalog" is a badly overloaded word

Three different things get called a catalog and only one of them is what Iceberg means.

When a vendor says "our catalog gives you data discovery and governance," check whether it can serve POST /v1/namespaces/{ns}/tables/{table} and commit a snapshot. If it cannot, it is a business catalog and you still need a technical one underneath it. Some products — Unity Catalog and Polaris among them — are genuinely both, which is why the confusion persists.

The commit: an atomic compare-and-swap

Every Iceberg write ends with a compare-and-swap against the catalog, and that single operation is what gives Iceberg its ACID guarantees. Here is the exact sequence.

1. The engine asks the catalog to load the table. It receives the current metadata location, call it v41.metadata.json.
2. The engine writes new Parquet data files to object storage. Nothing is visible yet; these are just files nobody references.
3. The engine writes new manifests and a new metadata file, v42.metadata.json, which contains a new snapshot pointing at those data files.
4. The engine calls the catalog: "swap the pointer from v41 to v42." It sends both the expected current value and the new one.
5. The catalog checks that its stored pointer still equals v41. If it does, it updates to v42 and the commit succeeds — atomically and instantly visible to every reader. If another writer got there first and the pointer now says v42b, the catalog rejects the commit.
6. On rejection the engine re-reads the new current metadata, re-validates its changes against it, and retries.

That check-then-set in step 5 is a compare-and-swap, and it is the entire concurrency-control mechanism of Apache Iceberg. There is no in-between state: a reader either sees the old snapshot or the new one, never a half-written table.

The consequence people miss: because correctness lives in the catalog's compare-and-swap, the catalog is not an optional convenience layer around your table. It is the table's transaction manager. A catalog outage does not degrade your lakehouse; it stops all writes. Size and monitor it accordingly.

Why this makes catalog choice a correctness decision

Once you see the commit path, several practical things follow immediately, and they explain most of the trade-offs in the rest of this guide:

• Latency of the catalog is latency of every commit. Hive Metastore does its compare-and-swap via a round trip to a relational database and a lock, which is exactly why it becomes the bottleneck under high write concurrency.
• Availability of the catalog is availability of writes. No catalog, no commits, for every engine at once.
• Access control belongs in the catalog — it is the one component every engine must pass through. That is why credential vending was added to the REST spec.
• The catalog is a hard boundary. A compare-and-swap is only atomic within one catalog's own state. Two catalogs cannot coordinate. This is the source of the most dangerous mistake in this whole article.

The REST catalog spec is not a catalog

"Iceberg REST Catalog" is an API specification, not software you can deploy. This trips up more teams than any other point in the topic, because "just use the REST catalog" is offered as if it were an answer.

The REST Catalog API was introduced in Apache Iceberg 0.14.0 in July 2022, and defines the HTTP contract between an Iceberg client and a catalog server: how to list namespaces, load a table, and commit an update. You can read the full contract in the Apache Iceberg REST Catalog spec{target="_blank" rel="noopener"}.

What it buys you is real, and it is the reason the whole ecosystem converged on it:

• Thin, language-agnostic clients. Before REST, catalog support meant a JVM library per catalog type. PyIceberg, Go, and Rust clients all became straightforward once the contract was HTTP and JSON.
• Server-side logic. Because the server performs the commit, it can also do conflict resolution, caching, auditing and authorisation — none of which a client-side library could enforce.
• Portability. Point a Spark, Trino, DuckDB or Flink job at a different REST catalog by changing a URL and a credential, not a dependency.

What the spec deliberately does not define is just as important, because it is exactly where implementations differ:

So "we support the Iceberg REST catalog" tells you an engine can connect. It tells you almost nothing about whether the implementation can express the permissions your security team asked for.

Credential vending and remote signing

Credential vending is the catalog handing an engine short-lived, table-scoped storage credentials instead of the engine holding long-lived bucket keys. It is the main reason the REST catalog era is more secure than the Hive Metastore era.

The flow: an engine authenticates to the catalog, the catalog checks whether that identity may read sales.orders, and if so it returns temporary credentials scoped to just that table's storage prefix. The engine never receives credentials for the rest of the bucket, and the credentials expire in minutes.

Remote signing is the stricter sibling. Rather than handing over credentials at all, the catalog signs each individual storage request the engine wants to make. The engine never holds a credential of any kind. It costs a round trip per request, so it is used where the security posture demands it.

Hive Metastore supports neither, which on its own is a strong argument for leaving it. But vended credentials are not automatically safe either — see Catalog security below for a live advisory that caught teams who assumed otherwise.

Iceberg catalog comparison

Here is the honest state of the field in August 2026. "Lock-in cost" is my estimate of how painful it is to leave, which is the column most comparisons omit and the one that matters most in three years.

Two things stand out. First, view support is not universal, and views are how most organisations expose curated data — check this before you commit. Second, the two options with the highest lock-in cost are also the two with the least operational work. That is the actual trade you are making.

The catalogs in depth

AWS Glue Data Catalog and S3 Tables

Glue is the right default if every engine you use is an AWS service, and the wrong one the moment that stops being true. It is IAM-native, has no servers to run, and Athena, EMR, Glue ETL, Redshift and Lake Formation all already trust it. If you are building on the stack described in our guide to data engineering on AWS with S3, EMR and Glue, you are probably already using it.

Since AWS shipped the Glue Iceberg REST endpoint, external engines like Trino, Spark on Kubernetes, PyIceberg and DuckDB can talk to it through the standard spec rather than an AWS-specific SDK. That closed the biggest gap.

The limitations are specific and worth reading before you commit, straight from the AWS Glue Iceberg REST API considerations{target="_blank" rel="noopener"}:

• View APIs from the Iceberg REST specification are not supported. If your governance model depends on Iceberg views, this is a hard blocker today.
• Metadata size is capped at 50 MB per table — and only 5 MB per REST API call for tables reached through catalog federation. Requests above the cap are rejected outright.
• RenameTable works for Redshift-backed tables but not S3-backed ones.
• DDL against Redshift-managed namespaces is asynchronous, because it waits on the managed workgroup and any conflicting transaction.

That 50 MB cap deserves a moment. Metadata grows with snapshot count and manifest count, so a busy table that never expires snapshots will drift towards it. AWS's own advice is to enable compaction and snapshot retention — which is precisely the routine described in our Iceberg table maintenance guide. A metadata size limit is a maintenance problem wearing a catalog costume.

Amazon S3 Tables is the newer, more managed option: a bucket type that stores Iceberg tables and runs compaction and snapshot expiry for you, reachable through the same Glue Iceberg REST endpoint. It trades control for less operational work, and it is a genuinely good deal for teams without a platform engineer. Note that S3 Tables and the classic Glue Data Catalog have different permission models even though they share the endpoint — verify yours in a non-production account first.

Apache Polaris

Polaris is the neutral choice, and in 2026 it is the closest thing the ecosystem has to a default. It was co-created by Snowflake and Dremio, donated to the Apache Software Foundation, and graduated from the Incubator to a Top-Level Project in February 2026. Graduation matters here: it signals that governance is community-controlled rather than a single vendor's roadmap, which is the whole reason to pick a neutral catalog.

It implements the full REST spec and then adds the parts the spec leaves out and every production deployment needs:

• Multi-catalog management — one Polaris instance hosting many logical catalogs with isolated namespaces.
• Role-based access control at catalog, namespace and table granularity, expressed in the catalog rather than in cloud IAM.
• Credential vending and remote signing, so engines never hold long-lived storage keys.
• Federation to Hive Metastore, AWS Glue and other Iceberg REST catalogs — Polaris acts as a routing layer for tables that still live elsewhere. This is the feature that makes incremental adoption possible without a big-bang migration.
• Generic tables, which let Polaris catalog Delta Lake and Hudi tables alongside Iceberg in the same namespace.
• Metrics reporting, where engines push query-level execution stats back to the catalog through the REST API.

Engine support is broad — Spark, Flink, Trino, Dremio, StarRocks and Apache Doris among others. If you would rather not operate it, Snowflake Open Catalog and Dremio's managed offering are Polaris under the hood, which means the exit path stays open.

The honest cost: self-hosting Polaris means running a JVM service and a relational store such as PostgreSQL, with backups and high availability, because it sits on the write path of every table you own.

Databricks Unity Catalog

Unity Catalog is excellent inside Databricks and asymmetric outside it — and understanding that asymmetry is the whole decision.

Unity governs Delta Lake and Iceberg tables, plus ML models, features, volumes and notebooks, with column-level lineage and fine-grained access control. If Databricks is your platform, nothing else comes close for breadth. The core has been open-sourced, and external engines including Trino, DuckDB, Spark, Daft and Dremio can reach it through the Iceberg REST endpoint with credential vending.

The asymmetry is in how it treats two classes of table:

Read that table again with the compare-and-swap in mind and it stops looking like a product limitation and starts looking like correct engineering: a table whose pointer is owned by Glue cannot safely be written by Unity, because the two catalogs cannot coordinate a compare-and-swap. The read-only restriction is Databricks refusing to let you corrupt your own table.

There is a second asymmetry worth knowing: through the Iceberg REST endpoint, external clients get read, write and create on managed Iceberg tables, but only read-only access to Delta tables exposed as Iceberg via UniForm. If your plan is "we'll keep everything in Delta and let Trino write through UniForm," that plan does not work.

For the wider platform question this sits inside, see our comparison of Snowflake vs Databricks.

Project Nessie

Nessie is the only mainstream catalog that gives you Git semantics over your entire lakehouse, and it exists for one problem the others do not solve: multi-table atomicity.

You create a branch, run a pipeline that writes to eight tables, validate the results on the branch where no consumer can see them, and then merge — atomically, across all eight. If validation fails, you delete the branch and production never saw a partial state. You can also tag a commit and query the lakehouse exactly as it looked at that tag.

This is genuinely powerful for a specific set of jobs:

• Multi-table transactions that must land together or not at all.
• Testing a pipeline change against production data without a production copy.
• Reproducible experiments pinned to an exact catalog-wide state.
• Isolating a large backfill on a branch before merging it.

The trade-offs are real. Access control is thinner than Polaris or Unity. Engine support is narrower. And branching is a discipline, not a feature — teams that adopt Nessie without agreeing on a branching model end up with dozens of stale branches pinning snapshots that snapshot expiry can never clean up, which quietly inflates storage. Note that Iceberg itself has table-level branches and tags; Nessie's distinction is that its branches span all tables at once.

Lakekeeper, Gravitino and the rest

Two newer options are worth knowing because they solve problems the big four do not.

Lakekeeper is the minimalist: written in Rust, shipping as a single binary with no JVM and no Python runtime, backed by PostgreSQL. Point it at a database and it serves REST requests. Authorisation is fine-grained via OpenFGA. If you run Kubernetes and want a catalog whose operational footprint is one container and one database, it is the lightest credible option.

Apache Gravitino goes the other way: a federated metadata lake that fronts many underlying catalogs and formats, exposing a native Iceberg REST catalog service among other interfaces. It is the right shape when you have inherited several catalogs across teams and clouds and need one place to reason about them. The cost is a large configuration surface — a JVM server, a connector layer, and a federation topology to maintain.

Hive Metastore — why you are leaving

Hive Metastore still runs a large share of the world's tables and it works. But the reasons to leave have accumulated into a clear case:

1. Thrift, not HTTP. The protocol is JVM-shaped, so non-JVM clients like PyIceberg need workarounds.
2. A database round trip per operation. Under write concurrency the relational store behind HMS becomes the bottleneck — the commit path was never designed for many engines committing at once.
3. No credential vending. Every engine needs its own long-lived storage credentials, which is the security model the REST spec was written to replace.
4. A single point of failure unless you have carefully configured high availability — and remember, when the catalog is down, writes stop everywhere.
5. Ecosystem drift. Vendors have begun removing packaged HMS images from their distributions, and new features land in REST implementations first.

You do not need a big-bang migration. Both Polaris and Gravitino can federate to an existing Hive Metastore, letting you route new work to the new catalog while old tables keep working — which is by far the safest exit path.

How to choose an Iceberg catalog

Choose by platform gravity first, then check the three things that can veto your choice. In practice, the first question decides it for 80% of teams.

The decision path

1. Is Databricks your primary compute? → Unity Catalog. Fighting this is expensive and you lose lineage, ML asset governance and managed-table performance work.
2. Are all your engines AWS services? → AWS Glue Data Catalog, or S3 Tables if you want AWS to run maintenance too.
3. Do you need multiple engines or multiple clouds to write the same tables? → Apache Polaris, self-hosted or via Snowflake Open Catalog / Dremio.
4. Do you need atomic changes across several tables, or Git-style branching of the whole lakehouse? → Project Nessie. Nothing else does this.
5. Do you want the smallest possible thing to operate on Kubernetes? → Lakekeeper.
6. Are you consolidating several existing catalogs you cannot retire? → Apache Gravitino, or Polaris federation.

The three vetoes

Before you commit, check each of these against your actual requirements. Any one of them can override the decision above:

• Views. Do you expose curated data through Iceberg views? AWS Glue's Iceberg REST endpoint does not support the View APIs. This has changed more than one AWS-native decision.
• Access control expressiveness. Write down your two hardest permission rules — the ones involving a specific team, a specific namespace and a time-bound exception. Can the candidate express them? Cloud IAM is excellent at coarse-grained access and awkward at namespace-level RBAC.
• Metadata scale. If you have very high-frequency writers, check the metadata size ceiling. Glue's 50 MB cap, and 5 MB per call under federation, is generous until a table with a thousand daily commits and no snapshot expiry approaches it.

Do not over-think the reversible parts

One more piece of perspective. The data files are portable, the table format is portable, and register_table is a millisecond operation. What is genuinely expensive to change is the access-control model and the connection configuration spread across every job you own. So spend your evaluation time on permissions and on how engines authenticate — not on feature-matrix rows you will never use.

The mistake that actually loses data: two catalogs, one table

If two catalogs both accept writes to the same Iceberg table, you will lose committed data, and nothing will report an error. This is the most important operational rule in this article and it is barely discussed anywhere.

The mechanism follows directly from the commit path. Catalog A holds pointer v41. Catalog B, registered against the same table, also holds v41. Writer 1 commits through A: A checks v41, swaps to v42. Writer 2 commits through B: B checks its own stored pointer, still v41, sees a match, and swaps to v42b.

Both compare-and-swaps succeeded. Both writers were told their commit landed. But v42 and v42b each descend from v41, and whichever pointer a reader resolves determines which set of rows exists. The other writer's data is now orphaned on disk — present in object storage, referenced by nothing, invisible to every query. Worse, a subsequent remove_orphan_files maintenance run will happily delete it for good.

There is no exception thrown, no warning logged, and no metric that moves. You find out during a reconciliation weeks later.

The rules that prevent it

1. Exactly one catalog owns writes for any given table. Write this down in your platform docs. It is not a guideline.
2. Everything else reads through the owning catalog, or is registered explicitly read-only — which is precisely what Unity Catalog's foreign tables and Polaris's federation do for you.
3. Never run register_table against a live table in a second catalog "just to see it." Registration is what creates the second owner.
4. During a migration, freeze writes. The window in which both catalogs know about a table and both accept writes is the whole risk.
5. Audit for duplicate registrations the same way you audit for orphaned resources — periodically, and automatically.

We hit the near-miss version of this on Telemetrix, our infrastructure monitoring platform. The device-telemetry tables were owned by Glue, and while evaluating an alternative catalog someone registered a handful of them into the second catalog to run read benchmarks. Nothing wrote through the second catalog, so nothing broke. But the review that followed was uncomfortable, because the only thing standing between us and silent data loss was that the benchmark happened to be read-only. We now treat register_table as a privileged operation with the same care as a DROP.

Catalog security: vended credentials are not automatically safe

Credential vending removes long-lived storage keys from your engines; it does not guarantee the short-lived ones are handled safely once vended. The distinction is not theoretical.

In 2026, Trino disclosed CVE-2026-34214{target="_blank" rel="noopener"} (CVSS 7.7, High), affecting Trino versions 439 through 479 when the Iceberg connector is configured against a REST catalog. The advisory states it plainly:

"The storage credentials are stored in those handles when performing write operations, or table maintenance operations. They are serialized in query JSON. A user with write access to data in Iceberg connector configured to use REST Catalog with static or vended credentials can retrieve those credentials."

In other words, the credentials — static keys or vended temporary ones — ended up embedded in query JSON that the submitting user could read back through the Trino UI and API. Trino 480 fixes it. AWS's own remediation guidance is the useful shape of the lesson: static credentials must be rotated, while vended credentials expire on their own but you still have to assess what the holder could have reached before they did.

That last clause is the point. A short-lived credential limits the duration of an exposure, not its blast radius. If the vended credential is scoped to an entire bucket rather than one table's prefix, "temporary" buys you very little.

A short checklist worth running against your own deployment:

• Patch the engines, not just the catalog. Credential handling bugs live on the client side.
• Verify the vending scope. Ask your catalog what prefix the vended credential actually covers, then test it — try to read a table you are not entitled to with a vended credential and confirm the failure.
• Prefer remote signing where the security posture justifies the extra round trip, because the engine then holds nothing at all.
• Treat query-history surfaces as sensitive. Query plans, JSON dumps and debug endpoints are credential-adjacent.
• Keep credential TTLs short and make sure your long-running jobs refresh rather than pinning one credential for hours.

Access control at the catalog is one half of a governance story; the other half is agreeing what the data should contain in the first place, which is where data contracts and pipeline quality tests do their work.

What migrating between catalogs really costs

Moving a table between catalogs is a metadata-only operation that finishes in milliseconds; moving a platform between catalogs takes weeks. Both halves of that sentence are true and teams usually only hear the first.

Because data files never move, migration is register_table: you point the new catalog at the existing metadata JSON, and it takes ownership of the pointer. In Spark:

-- Register an existing Iceberg table into a new catalog.
-- The data files do not move. This is a metadata-only transaction.
CALL new_catalog.system.register_table(
  table    => 'analytics.orders',
  metadata_file => 's3://lake/warehouse/analytics/orders/metadata/00041-a1b2c3.metadata.json'
);

That is the easy part. Here is what actually determines the timeline:

The two bolded rows are the migration. Everything else is a script.

A safe migration sequence

1. Inventory. Every table, every engine that touches it, every job config that names the old catalog, every view. The configs are what bite you.
2. Translate the permission model and have it reviewed before you move anything. This is the long pole. Do not discover on cutover day that your IAM policies do not map onto namespace RBAC.
3. Prove it on non-critical tables. Register a low-stakes namespace in the new catalog, point one engine at it, and run a real workload.
4. Freeze writes on the tables being cut over. This is the non-negotiable step — it is the only thing preventing the two-catalog problem during the window.
5. Register, verify, then repoint engines. Confirm snapshot history and row counts match before any job changes.
6. Deregister from the old catalog so no path back to a second writer exists, then unfreeze.

Or do not migrate at all: federate

The alternative most teams should consider first is federation. Apache Polaris can federate to Hive Metastore, AWS Glue and other Iceberg REST catalogs, and Unity Catalog can mount external Iceberg tables as read-only foreign tables. In both cases a single endpoint serves tables that still physically belong to their original catalog.

That means new work targets the new catalog, old tables keep working unchanged, and you migrate individual tables when there is a reason to — rather than scheduling a weekend to move everything at once. Given that the risky window in any catalog migration is precisely the moment two catalogs both know about a table, an approach that never opens that window is worth serious consideration.

Common mistakes to avoid

1. Treating the catalog as a lookup service. It is the transaction coordinator. Give it the monitoring, alerting and high-availability budget you would give a database, because writes stop when it does.
2. Registering one table in two catalogs. Silent data loss, no error. See the rules above.
3. Using HadoopCatalog in production on S3. It depends on atomic rename semantics that object storage does not provide. It is fine for a laptop demo and dangerous anywhere else.
4. Ignoring metadata growth until you hit a ceiling. Snapshot expiry and compaction are not optional housekeeping — they are what keeps you under Glue's 50 MB metadata cap and keeps planning fast. Our Iceberg maintenance guide has the routine.
5. Assuming views are portable. They are the least standardised part of the ecosystem and AWS Glue's REST endpoint does not support the View APIs at all.
6. Choosing on a feature matrix instead of on your permission model. You will use four catalog features. You will fight your access-control model every week.
7. Assuming vended credentials are safe by construction. Check the scope, patch the engine, and keep TTLs short — CVE-2026-34214 exposed vended credentials through a client-side bug, not a catalog one.
8. Skipping the write freeze during migration. Ten minutes of frozen writes is cheaper than one orphaned snapshot you find in a reconciliation two months later.

Where this fits in your lakehouse

If you are assembling the wider picture, the catalog is one of five decisions and the least reversible of them: the table format, the file format, the partitioning strategy, the compute engine, and the catalog. Change any of the first four and you rewrite files. Change the catalog and you renegotiate access control across the organisation.

If you are working out whether a lakehouse is the right architecture in the first place, start with data lake vs data warehouse vs lakehouse before choosing any of this.

Designing a lakehouse and want a second opinion before it carries production traffic? Post your project on solutiongigs.in and get matched with a vetted data engineer — it's free to post.

Frequently Asked Questions

What is an Iceberg catalog?

An Iceberg catalog is the service that maps a table name to the location of that table's current metadata file, and that performs the atomic compare-and-swap which makes a commit visible. It is not a search or discovery tool. It is the transaction coordinator for the table — without it, two writers can overwrite each other's snapshots and you lose data.

Do I need a catalog to use Apache Iceberg?

For anything beyond a single-writer experiment, yes. Iceberg does support a HadoopCatalog that stores the pointer as a file in object storage, but S3 and most object stores lack the atomic rename that approach depends on, so concurrent commits can corrupt the table. Use a real catalog service: Glue, Polaris, Unity Catalog, Nessie or Lakekeeper.

What is the difference between the Iceberg REST catalog and a catalog implementation?

The Iceberg REST Catalog is a specification — an HTTP API contract introduced in Iceberg 0.14.0 in July 2022 — describing how a client asks a catalog to load, create and commit tables. It is not software you run. Implementations are the servers that speak it: Polaris, Unity Catalog, AWS Glue's REST endpoint, Lakekeeper, Gravitino. "Use the REST catalog" still leaves you the real decision.

Which Iceberg catalog should I use on AWS?

If your engines are Athena, EMR, Glue ETL and Redshift, use the AWS Glue Data Catalog through its Iceberg REST endpoint — it is IAM-native and every AWS analytics service already trusts it. Choose Amazon S3 Tables if you also want AWS running compaction and snapshot expiry. Choose Polaris instead when you need multi-cloud neutrality, Iceberg views, or RBAC that IAM cannot express cleanly.

Can two catalogs manage the same Iceberg table?

Not for writes, and this is the most dangerous mistake in lakehouse operations. Iceberg's correctness comes from an atomic compare-and-swap on the metadata pointer, and that swap is only atomic within one catalog. If two catalogs both accept writes, neither sees the other's commit and one silently overwrites the other's snapshots. Exactly one catalog must own writes per table.

Is Apache Polaris production ready?

Yes. Polaris graduated from the Apache Incubator to a Top-Level Project in February 2026, and both Snowflake and Dremio offer it as a managed service if you would rather not operate it. Self-hosting is realistic — a JVM service plus a relational store such as PostgreSQL — but treat it as production infrastructure with backups and HA, because every write goes through it.

How hard is it to migrate between Iceberg catalogs?

The data does not move and register_table completes in milliseconds. The hard part is everything the catalog holds that is not the pointer: access-control policies, engine connection configs in every job and notebook, audit history, lineage and views. Budget days of coordination rather than a data migration, and freeze writes during cutover so no table ever has two owners.

Does Unity Catalog work with non-Databricks engines?

Yes, through its Iceberg REST endpoint with credential vending — Trino, DuckDB, Spark, Daft and Dremio can all connect. But note the asymmetry: external clients get read, write and create on Unity's managed Iceberg tables, and only read-only access to Delta tables exposed as Iceberg through UniForm. Foreign Iceberg tables owned by another catalog are read-only inside Databricks.

Conclusion

The catalog is the smallest component in your lakehouse and the one you will regret choosing carelessly. Everything else — Parquet files, the Iceberg format, even the compute engine — is genuinely portable in a way the industry promised and mostly delivered. The catalog is where the coupling went, because it holds the one piece of mutable state in the whole design and, with it, your access-control model.

So make the decision on the two things that are actually hard to change. Can this catalog express the permissions your organisation needs? And how many engine configurations would have to change if you left? Feature matrices are mostly noise next to those two questions.

For most teams the answer is dictated by platform gravity and that is fine: Unity Catalog if you live in Databricks, AWS Glue if you live in AWS, Apache Polaris when you genuinely need neutrality — and Polaris is a stronger default in 2026 than it was a year ago, now that it is an Apache Top-Level Project with federation and a real managed-service ecosystem. Nessie remains the answer to one specific question about multi-table atomicity that nothing else answers.

Whichever you choose, take three things from this article into your platform docs: exactly one catalog owns writes per table, vended credentials still need scope verification and patched engines, and federation is usually a better first move than migration.

If you are building a lakehouse, evaluating catalogs, or want an experienced data engineer to review the design before it carries production traffic, post your project on solutiongigs.in and get matched with a vetted expert — it's free to post.