You need to know who can see which data, where a number came from and who touched a table last. On Databricks, Unity Catalog answers all three. It controls access to tables, files, models and functions, records lineage automatically and logs every access for audits. A good setup comes down to four decisions: metastore and storage layout, catalog structure, permission model and how you classify sensitive data. This guide walks through each, then covers migration from the legacy Hive metastore.

How does the Unity Catalog hierarchy work?

Unity Catalog uses a three-level namespace, catalog.schema.object, under a metastore:

• Metastore: the top-level container, one per region, attached to all workspaces in that region.

• Catalog: the main unit of isolation, typically per environment, business unit or domain. Catalogs can be bound to specific workspaces.

• Schema: a group of related objects inside a catalog, for example a medallion layer or a subject area.

• Objects: tables, views, materialised views, volumes (for files), functions and registered ML models.

Around this hierarchy sit storage credentials and external locations, which grant governed access to cloud storage, and connections for query federation to external databases. Give each catalog its own managed storage location so production and development data land in separate storage accounts or buckets.

What catalog layout works for most enterprises?

A layout that works for most enterprises combines environments at the catalog level with medallion layers at the schema level:

In a larger organisation with domain ownership, add the domain to the catalog name (prod_finance, prod_supply_chain) and give each domain team ownership of its catalogs. Don’t create a catalog per project. Catalogs are hard to rename, and projects come and go.

How should you set up groups and service principals?

1. Sync groups from your identity provider. Use SCIM provisioning (or automatic identity management on Azure) from Microsoft Entra ID or Okta to create account-level groups. Never grant permissions to individual users.

2. Grant on catalogs and schemas, not tables. Privileges inherit downwards, so grant USE CATALOG, USE SCHEMA and SELECT at the schema level for readers, and MODIFY or CREATE TABLE for writers.

3. Run production jobs as service principals. Only service principals should write to production catalogs, and only through CI/CD pipelines.

4. Set clear ownership. Make a group, not a person, the owner of each catalog and schema, so access keeps working when people leave.

5. Use workspace-catalog bindings. Bind the prod catalog read-write only to the production workspace and read-only to analytics workspaces, so development work cannot touch production data.

What we see in delivery: govern before the first login

On a HIPAA-aligned data platform we built on Azure and Databricks for a nonprofit human-services organization, Unity Catalog carried the governance load from the start: central access control, lineage and role-based rights for teams working with protected health information. We followed Databricks’ Security Reference Architecture, kept the whole environment on private networking with no public exposure, and trained the client’s teams on secure data handling.

The lesson: set up groups, ownership and access rules before the first data scientist logs in. Retrofitting permissions onto tables people already use is slower and riskier than starting governed.

How do lineage and auditing work?

Unity Catalog captures table and column-level lineage automatically for queries, notebooks, jobs and dashboards running on Unity Catalog compute. You can view it in Catalog Explorer or query the system.access.table_lineage and system.access.column_lineage system tables. The system.access.audit table records who accessed what and when, which is the evidence auditors ask for under frameworks such as ISO 27001, HIPAA or DORA.

How do tags, classification and ABAC fit together?

Tagging is where governance scales. Unity Catalog now offers three features that work together, all generally available in 2026:

• Governed tags: account-level tag definitions with allowed values and control over who can apply them, for example a sensitivity tag with the values public, internal, confidential and pii.

• Data classification: automatic scanning that detects sensitive data such as emails or national identifiers and suggests or applies tags.

• ABAC policies: row filter and column mask policies defined once at catalog or schema level and applied to every table or column with a matching tag.

For example, one policy can mask all columns tagged pii for everyone except members of a privacy-approved group, across hundreds of tables. You can still use table-level row filters and column masks for special cases. For regulated data, combine this with the controls in our HIPAA-aligned data platform checklist.

How do you migrate from the Hive metastore?

From 30 September 2026, new Databricks workspaces are Unity Catalog-only, without the Hive metastore, DBFS root and mounts, or no-isolation shared clusters. Existing workspaces aren’t affected, but staying on the Hive metastore means missing out on lineage, ABAC, system tables and most new features. There are two main paths:

A typical migration runs in five steps: assess with UCX, migrate groups to account level, create the target catalogs and external locations, upgrade tables and views, then repoint jobs, notebooks and BI connections. For a mid-sized workspace (a few hundred tables and jobs) expect 4 to 12 weeks, with most of the effort in updating and testing code rather than moving data. Our data platform migration checklist covers the testing and cutover steps.

Dates and feature status reflect Databricks announcements at the time of writing. This guide is general information, not legal advice.

How RUBICON helps with Unity Catalog

We design catalog layouts and permission models, set up tagging and ABAC, and run Hive metastore migrations together with your team. Our work includes a HIPAA-aligned data platform on Azure and Databricks. We’re a Databricks Partner and a Microsoft Solutions Partner for Cloud & AI Platforms, about 55 people with 40+ engineers, working in CET under ISO 27001:2022 certified processes.

See our data engineering services. If you’re planning a Unity Catalog setup or migration, our architects can map the layout with you.

Frequently asked questions