Databricks PII governance: governed tags, ABAC, and Terraform

Layered PII and data governance on Unity Catalog: classify sensitive data, tag it with a small governed taxonomy, enforce masks and row filters with ABAC, and keep the policy control plane in Terraform. This is the governance layer you add on top of a platform foundation (Unity Catalog as code, Entra-backed groups, group-based RBAC, managed-identity storage access, PR-reviewed Terraform). See identity-entra-scim.md and terraform-vs-dabs.md for that foundation.

The governance layers#

1. Classification, tagging, and masking#

• Use Databricks Data Classification (GA 2026-04-20) to detect sensitive data in Unity Catalog.
• Use governed tags to standardize labels such as classification=restricted, pii=email, pii=phone, pii=ssn, row_security=region.
• Require data-steward review before broad auto-tagging in production.
• Use ABAC policies to apply masks or row filters based on governed tags.
• Use DABs or SQL migrations to apply workload-specific tags when pipelines create new tables.
• Use Terraform for the stable governance plane: tag policies, ABAC policies, Entra groups, grants, and stable tag assignments.

The pattern: Unity Catalog + Entra-backed groups + group-based grants is the foundation. To extend it into PII governance, add a governed tag taxonomy, enable Data Classification as the discovery layer, require steward review for sensitive detections, and enforce column masks and row filters with ABAC policies at the catalog or schema level.

2. ABAC vs governed tags vs tag policies#

The clearest way to keep these straight:

• Governed tags describe the data.
• Tag policies define allowed tag keys and values.
• ABAC policies enforce row filters or column masks based on those tags.
• Entra-backed groups decide who is exempt or allowed to see unmasked data.

Governed tag:
  pii=email on prod_catalog.gold.customer.email_address

ABAC policy:
  Any column tagged pii=* is masked for account users
  except approved PII-reader groups

Entra ID:
  Membership in a "PII Unmasked Readers" group
  determines whether the UDF returns raw or masked values

Rows are not the governance target — don't try to tag rows. Tables or columns are tagged, and row-filter UDFs evaluate row values such as region_code, tenant_id, or business_unit.

3. Row filters and column masks#

• RBAC gets a user to the catalog/schema/table.
• Column masks decide whether sensitive column values are raw or masked.
• Row filters decide which rows are returned.
• ABAC applies those controls centrally when matching governed tags are present.

Table:   prod_catalog.gold.customer
Column:  region_code
Tags:    table row_security=region
         column security_key=region_code
Policy:  Pass region_code to governance.allow_region(region_code)
UDF:     Return true only if current user is in the matching Entra-backed group

Don't create a policy per table or tag every row. Define a small governed tag taxonomy, apply tags at the highest useful Unity Catalog object level, tag sensitive columns specifically, and use catalog-level ABAC policies so the same mask/filter logic follows the tags.

4. Terraform's role at scale#

Terraform should own the stable, repeatable governance objects:

• Entra security groups and Databricks account groups.
• Unity Catalog grants and service principal permissions.
• Governed tag definitions and allowed values.
• Catalog/schema/table/column tag assignments for stable, high-value assets.
• ABAC policy resources (databricks_policy_info).
• External locations, storage credentials, and managed-identity access paths.

Terraform should not own every fast-moving table or generated column if those change frequently — that creates noisy plans and stale governance metadata.

Terraform:            platform controls and stable governance
DABs / pipeline SQL:  workload-owned table creation and workload-specific tags
Data Classification:  sensitive-data discovery and optional auto-tagging
Data stewards:        review false positives, approve tag activation, exceptions

5. DABs in the governance model#

DABs are not a replacement for Terraform-owned ABAC.

• Terraform owns the platform and control plane.
• DABs own jobs, pipelines, notebooks, SQL tasks, dashboards, and deployment targets.
• DABs run production jobs as service principals that Terraform created and granted.
• Pipelines can apply tags to the tables they produce, but central ABAC policies stay in Terraform or a restricted governance repo.

Terraform defines the guardrails; DABs deploy the workloads inside them. A bundle can write a gold customer table, but Unity Catalog grants, governed tags, ABAC policies, and the service principal's access should be centrally controlled.

6. Periodic access reviews#

Make access reviews identity-led, not workspace-led:

• Entra ID is the source of truth for group membership.
• Databricks account groups are synced or mapped from Entra.
• Terraform grants permissions to groups, not individual users.
• A quarterly review exports group membership, UC grants, service principals, PAT usage, and job ownership.
• Business data owners approve or remove access.
• Removals happen in Entra first, then propagate to Databricks.

Review Entra group membership and service principal ownership first, then validate Unity Catalog grants and workspace permissions against those approved groups.

7. PAT tokens, service principals, and secrets#

The modern direction is workload identity federation and service principals instead of long-lived secrets:

• Prefer OAuth / service principal auth for automation.
• Avoid human PATs for production automation.
• Restrict PAT creation and lifetime where PATs are still allowed.
• Store secrets in Azure Key Vault-backed secret scopes or an external secret system.
• Use managed identities for Azure resource access where possible.
• Rotate secrets and tokens, audit usage, and alert on stale credentials.

Feature status (GA dates)#

The Unity Catalog governance stack went GA in April 2026 (verified against Microsoft Learn release notes):

• Governed tags GA: 2026-04-02
• Databricks Data Classification GA: 2026-04-20
• ABAC GA: 2026-04-28
• SQL DDL for governed tags (CREATE/ALTER/DROP/DESCRIBE GOVERNED TAG, SHOW GOVERNED TAGS) documented 2026-04-13

The ABAC GA release introduced a breaking change: row filters and column masks on views/functions now evaluate as the session user, not the view/function owner. Existing customers got a 3-month grace period; new customers get the new behavior by default.

• ABAC compute requirements: serverless, standard compute on DBR 16.4+, or dedicated compute on DBR 16.4+ with fine-grained access control enabled. Older runtimes cannot read ABAC-protected tables.
• Governed tags SQL DDL (CREATE GOVERNED TAG, ALTER GOVERNED TAG) requires DBR 18.1+ or Databricks SQL.
• Governed tags and ABAC should use non-sensitive metadata. Do not put actual PII values in tag keys or values.

Tag inheritance — the rule that trips people up#

• Tags applied to a catalog inherit to schemas and tables underneath.
• Tags applied to a schema inherit to tables underneath.
• Tags applied to a table do NOT inherit to its columns.
• Column tags must always be applied directly.

So the PII tagging pattern is: classify schemas with data_classification, then tag individual columns with pii type. Don't expect column tags to fall out of schema tags.

SQL DDL cheatsheet#

Built-in functions for ABAC WHEN and MATCH COLUMNS clauses are snake_case: has_tag('pii'), has_tag_value('pii', 'ssn'). Not hasTag().

-- 1. Create the governed tag taxonomy (account-level, one time)
CREATE GOVERNED TAG pii
  DESCRIPTION 'Type of PII contained in the column'
  VALUES ('ssn', 'email', 'phone', 'dob', 'address');

CREATE GOVERNED TAG data_classification
  VALUES ('public', 'internal', 'confidential', 'restricted');

-- 2. Define the masking UDF (one per masking strategy)
CREATE FUNCTION governance.mask_ssn(val STRING)
  RETURNS STRING
  RETURN CASE
    WHEN is_account_group_member('pii_readers') THEN val
    ELSE '***-**-' || right(val, 4)
  END;

-- 3. Define the ABAC policy at the highest useful scope (catalog)
CREATE POLICY mask_ssn_columns
ON CATALOG prod_catalog
COLUMN MASK governance.mask_ssn
TO `account users`
FOR TABLES
MATCH COLUMNS has_tag_value('pii', 'ssn') AS ssn_col
ON COLUMN ssn_col;

-- 4. Row filter pattern (uses USING COLUMNS, not ON COLUMN)
CREATE FUNCTION governance.allow_region(region STRING, allowed STRING)
  RETURNS BOOLEAN
  RETURN region = allowed;

CREATE POLICY regional_access_emea
ON CATALOG sales
ROW FILTER governance.allow_region
TO `emea team`
FOR TABLES
MATCH COLUMNS has_tag('region') AS rgn
USING COLUMNS (rgn, 'EMEA');

Terraform resources — exact names#

Current Databricks Terraform provider resource names. There is one unified tag assignment resource (earlier notes referencing databricks_schema_tag / databricks_column_tag were wrong).

databricks_entity_tag_assignment takes entity_type (catalogs, schemas, tables, columns, volumes) and entity_name (fully qualified, e.g. prod_catalog.customers.users.email_address).

Quotas and limits#

10,000 policies per metastore is enormous because the design assumes you write a handful of policies and let tags do the work. If you're approaching the limit, you're using ABAC wrong.

Separation of duties — who does what#

Databricks publishes a four-role split:

1. Tag taxonomy owner (security or governance team) — CREATE on tags at account level. Defines pii, data_classification, etc.
2. Data stewards / classifiers — ASSIGN on the tag + APPLY TAG on the object. They stamp tags on data.
3. Governance admins — MANAGE on the catalog/schema + EXECUTE on UDFs. They write the ABAC policies.
4. Data consumers — query through the enforced policies.

Tagging is a security boundary. If a user can change tags, they can change which policies apply. Audit tag changes.

Gotchas (from the ABAC limitations page)#

The operational sharp edges:

1. No information schema view for ABAC policies. information_schema.row_filters and information_schema.column_masks show only table-level filters/masks, not ABAC. To list ABAC policies use the Unity Catalog REST API; audit trail comes from system.access.audit. Build "periodic access review" on UC system tables + REST API, not info_schema for policy enumeration.
2. Materialized views and streaming tables require the pipeline owner in the EXCEPT clause. If the pipeline run-as identity is subject to the ABAC policy, the refresh fails. Every policy covering tables a pipeline reads must exempt the pipeline's run-as service principal.
3. Time travel and clones fail on ABAC-protected tables unless the principal is in EXCEPT. ETL service principals that RESTORE or CREATE TABLE ... CLONE need explicit exemption.
4. Vector search indexes don't enforce ABAC. The index syncs all rows from the source and serves them without applying filters/masks. For PII columns, use the "columns to sync" setting to exclude masked columns — otherwise PII leaks out the vector search path.
5. Delta Sharing requires the share owner in EXCEPT. Tables with ABAC policies (or views over them) can only be shared if the share owner is exempt. The policy doesn't govern the recipient — recipients enforce their own ABAC.
6. Multiple conflicting policies on the same column for the same user → query blocked. Only one row filter resolves per (table, user) and one column mask per (column, user). If two distinct policies resolve, Databricks blocks access. Same UDF + same args is fine; different masks for the same column for the same user is not.
7. Standard/dedicated compute on DBR < 16.4 can't read ABAC tables at all. The escape hatch is to scope the policy to a specific group via TO and put the legacy workload's principal in EXCEPT. Useful migration pattern.

Two that bite teams adopting ABAC: pipeline refreshes and Delta Sharing both need the operating identity in the EXCEPT clause or refreshes/shares break. And vector search indexes don't inherit ABAC, so for a PII workload exclude masked columns from the index at sync time rather than rely on policy enforcement.

Operating principles#

• Don't tag at the row level — row filters use column values. Tag schemas with classification, then columns with PII type only where it matters. Policies are written against tags, so adding a new PII table doesn't touch policy code. A few PII-type policies cover the whole platform, and a CI check against information_schema.column_tags catches columns that should have been tagged but weren't.
• Confirm all production compute is on DBR 16.4+ before adopting ABAC, otherwise ABAC-protected tables are unreadable. If a team has dynamic views built around owner identity, revisit them for the session-user GA change.

Microsoft Learn doc links#

Concept + setup: - ABAC overview — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/abac/ - ABAC core concepts — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/abac/core-concepts - Governed tags overview — https://learn.microsoft.com/azure/databricks/admin/governed-tags/ - Create and manage governed tags — https://learn.microsoft.com/azure/databricks/admin/governed-tags/manage-governed-tags - Apply tags to UC securable objects — https://learn.microsoft.com/azure/databricks/database-objects/tags

Hands-on tutorials: - Tutorial: Configure ABAC (UI) — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/abac/tutorial - Tutorial: Configure ABAC with SQL — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/abac/tutorial-sql

SQL DDL reference: - CREATE GOVERNED TAG — https://learn.microsoft.com/azure/databricks/sql/language-manual/sql-ref-syntax-ddl-create-governed-tag - ALTER GOVERNED TAG — https://learn.microsoft.com/azure/databricks/sql/language-manual/sql-ref-syntax-ddl-alter-governed-tag - Row filter syntax — https://learn.microsoft.com/azure/databricks/sql/language-manual/sql-ref-syntax-ddl-row-filter - Column mask syntax — https://learn.microsoft.com/azure/databricks/sql/language-manual/sql-ref-syntax-ddl-column-mask

Best practices, perf, limits: - Create and manage ABAC policies — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/abac/policies - Best practices — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/abac/best-practices - Common patterns (VARIANT UDFs, struct redaction) — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/abac/common-patterns - Performance — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/abac/performance - Requirements, quotas, limitations — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/abac/requirements - Resource limits (governed tag, tag-assignment, and policy quotas) — https://learn.microsoft.com/azure/databricks/resources/limits - ABAC vs table-level filters/masks — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/abac/abac-vs-rls-cm - Access control overview — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/access-control/

Audit + observability: - System tables reference — https://learn.microsoft.com/azure/databricks/admin/system-tables/ - Audit log system table (system.access.audit) — https://learn.microsoft.com/azure/databricks/admin/system-tables/audit-logs - Information schema overview — https://learn.microsoft.com/azure/databricks/sql/language-manual/sql-ref-information-schema - INFORMATION_SCHEMA.TABLE_TAGS — https://learn.microsoft.com/azure/databricks/sql/language-manual/information-schema/table_tags - INFORMATION_SCHEMA.SCHEMA_TAGS — https://learn.microsoft.com/azure/databricks/sql/language-manual/information-schema/schema_tags - INFORMATION_SCHEMA.ROW_FILTERS — https://learn.microsoft.com/azure/databricks/sql/language-manual/information-schema/row_filters

Terraform provider: - databricks_policy_info — https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/policy_info - databricks_tag_policy — https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/tag_policy - databricks_entity_tag_assignment — https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/entity_tag_assignment - databricks_access_control_rule_set — https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/access_control_rule_set