Databricks Entra ID Identity Provisioning (Automatic Identity Management)

Background#

Use Automatic Identity Management (AIM) to make Entra ID the identity source for Databricks. AIM creates an account group when an administrator adds the Entra group to Databricks. Databricks sets the account group's externalId from the Entra ObjectId and refreshes it each day.

Do not create a second Databricks group for the same Entra group. Do not build a workflow that depends on externalId, because Databricks can update it. Manage the group and its membership in Entra ID. Manage the workspace assignment and Unity Catalog grants in Databricks, with Terraform where useful.

A typical naming convention for the paired groups is App Access - Databricks <Role> <Env> so the Entra group and the Databricks group are obviously the same thing.

Two provisioning paths exist:

• AIM is the Databricks recommendation for supported identity providers. It does not need an enterprise application or a SCIM token.
• SCIM remains the fallback for an unsupported identity provider, a non-federated workspace, or a cross-tenant Entra design.

Current state to aim for#

• AIM: enabled on the account.
• Identity federation: enabled for each workspace that uses AIM.
• JIT provisioning: enabled by AIM. Users become active on first login.
• Entra ID groups: managed in Entra ID, with Terraform if the team uses it.
• Databricks account groups: created and synced by AIM.
• Workspace assignments: managed by an administrator or Terraform.
• Grants: in Terraform (databricks_grants / databricks_grant) at catalog, schema, and external-location levels.
• Verification: audit logs confirm AIM provisioning users and syncing group membership on login (queries below).

How the pieces fit together#

Entra ID                          Terraform                           Databricks
─────────                         ─────────                           ──────────
azuread_group ──→ Entra group ──AIM──→ Account group ──→ Workspace access
                         │                    │
                         │                    └──databricks_grants──→ UC privileges
                         │
                         └──authentication──→ User or service principal sync

1. Terraform can create the Entra ID groups.
2. AIM creates the Databricks account groups and syncs their membership.
3. Terraform can read the existing account group by name.
4. Terraform can assign the group to workspaces and grant catalog privileges.
5. AIM activates users and service principals when they authenticate.

A naming trap to watch for#

An app registration named like an Entra connector (for example sp_<something>_entraid) is not necessarily a SCIM provisioning connector. A plain app registration with User.Read.All Graph permissions that's registered in Databricks as an account admin is just a service principal, not an identity sync path. AIM does the syncing; don't assume a suggestively named SP is wiring identities.

How Automatic Identity Management works#

Overview#

AIM syncs users, service principals, and groups (including nested groups) from Entra ID to Databricks without an enterprise app, SCIM token, or extra Terraform. Databricks uses Entra ID as the source of record.

Sync triggers#

AIM does not run on a fixed schedule. It syncs lazily based on user activity:

When a sync triggers, Databricks fetches transitive (nested) group memberships from Entra ID for groups that have been added to Databricks.

What gets synced#

• Users (on first login via JIT)
• Group memberships (on each login/activity, resolving nested groups)
• Service principals (on first use — token auth or job run)

What stays inactive or incomplete until use#

• Users and service principals that have not authenticated
• Parent group hierarchy that no added Databricks group needs
• Group name renames (only update when an admin opens the group detail page in the account console)

User and group statuses#

Sharing and permissions#

• Account-level assets (Unity Catalog objects, dashboards, Genie spaces, workspace assignment): Entra ID groups are available immediately.
• Workspace-level assets (notebooks, jobs, SQL warehouses, alerts): a workspace admin must first add the group to the workspace.

Terraform architecture#

AIM owns the Databricks account group and its membership. Do not declare an AIM-managed group as a databricks_group resource. Do not declare its Entra membership as databricks_group_member resources.

Terraform can manage the boundaries around the AIM-owned group:

Identity resources (groups.tf)#

modules/entra_id_group  → Creates Entra ID security group
data.databricks_group   → Reads the AIM-created account group by display_name
permission_assignment   → Assigns the existing group to a workspace

The account group must exist before Terraform plans the data-source lookup. Add the Entra group to Databricks first, then apply the workspace assignment and grants.

Grant resources (grants.tf)#

• databricks_grants.catalog — catalog-level grants per role group
• databricks_grants.schema — schema-level grants per role group + per-schema group
• databricks_grants.external_location — external location grants

Example role-group model#

A workable six-tier RBAC model, with privileges tuned per role:

Verifying AIM is working#

Audit logs (SQL)#

-- Users provisioned by AIM
SELECT event_time, action_name, request_params.targetUserName
FROM system.access.audit
WHERE action_name = 'add'
  AND request_params.endpoint = 'autoUserCreation'
  AND event_time > current_timestamp() - INTERVAL 7 DAYS
ORDER BY event_time DESC;

-- Group membership synced from Entra ID
SELECT event_time, request_params.targetUserName, request_params.targetGroupName
FROM system.access.audit
WHERE action_name IN ('addPrincipalToGroup', 'removePrincipalFromGroup')
  AND request_params.groupMembershipType = 'IdentityProvider'
  AND event_time > current_timestamp() - INTERVAL 7 DAYS
ORDER BY event_time DESC;

-- User logins
SELECT DISTINCT user_identity.email
FROM system.access.audit
WHERE action_name = 'aadBrowserLogin'
  AND event_time > current_timestamp() - INTERVAL 7 DAYS;

API verification#

Use a Databricks CLI profile to check whether a user is active in the workspace:

databricks users list \
  --filter 'userName eq "<USER_NAME>"' \
  --profile "$P" \
  --output json

Use the audit queries above to confirm that AIM created the user and refreshed the group membership.

Known behaviors and limitations#

1. Users become active on first login — an unused identity stays inactive.
2. Nested groups work for permissions but not API/Terraform — nested groups inherit permissions automatically, but aren't retrievable via API or Terraform unless explicitly provisioned to the account.
3. Service principals provision on first use — adding a group containing SPs doesn't provision those SPs until they authenticate or run a job.
4. Group name renames aren't proactive — only update when an admin opens the group detail page in the account console.
5. Cross-tenant Entra ID is not supported — use SCIM with Entra B2B collaboration for cross-tenant scenarios.
6. Do not mix AIM and SCIM — adding the same identity through both causes duplicate entries and permission conflicts. Use AIM as the single source of truth.
7. Do not depend on externalId — Databricks can update it from the Entra ObjectId.

Identifiers you'll need#

Keep these per environment (in your own secret/secrets-of-record store, not here):

References#

• Sync users and groups automatically from Microsoft Entra ID (AIM)
• Establish your first account admin
• Identity best practices
• Identity federation
• databricks_group data source
• databricks_group Terraform resource
• databricks_mws_permission_assignment Terraform resource