Runbook: Databricks Environment Audit (Read-Only)

Owner: Jonathan Moss | Frequency: As Needed (new-environment onboarding); re-run quarterly Last Updated: 2026-06-23 | Last Run: —

Generic, reusable across any Databricks tenant (Azure / AWS / GCP). Every command is list / get / query / SELECT — read-only, nothing mutates state. The only non-read action is authentication (a prerequisite, not a change to the environment).

---

Executive Summary — Pillar Scorecard#

Fill this in after running the audit. It's the one-page roll-up for stakeholders; everything below it is the working detail. Keep it to this page. Replace every italic placeholder.

Environment: tenant / workspaces audited Date: YYYY-MM-DD Auditor: name Coverage: phases run (A–K) · environments covered · what was skipped and why

Bottom line: 2–3 sentences — overall posture and the single most important thing to fix.

Scorecard (RAG by Well-Architected pillar)#

🟢 no material issues · 🟡 issues to address · 🔴 urgent / business risk

Top 5 findings (severity-ranked)#

Quick wins vs. strategic#

• Quick wins (≤1 day): e.g. enable auto-termination; add failure webhooks; stop abandoned apps
• Strategic (needs a project): e.g. migrate hive_metastore → UC; adopt DABs for all resources

Coverage gaps / caveats#

• Products not present (phases skipped): …
• Phases blocked by missing permissions or disabled system tables: …

---

Purpose#

Map and assess an unfamiliar Databricks tenant across the full platform surface — inventory, cost/waste, jobs, pipelines, deployment hygiene, serverless opportunities, model serving, vector search, apps, and Lakebase — and produce an actionable findings list. Built so someone unfamiliar with the environment can run it top-to-bottom and end up with a defensible picture of "what's here and what's hurting."

Best-Practice Anchor — Databricks Well-Architected Lakehouse#

Every check below maps to a pillar of the official Databricks Well-Architected Lakehouse Framework (learn.microsoft.com/azure/databricks/lakehouse-architecture/well-architected), so findings are grounded in Databricks' own guidance rather than opinion. The seven pillars:

Prerequisites#

• [ ] Databricks CLI ≥ v0.292.0 (databricks --version)
• [ ] An authenticated profile (databricks auth profiles shows Valid: YES)
• [ ] Ideally account-admin read + workspace-admin read. Partial access still works — note gaps.
• [ ] Unity Catalog system tables enabled (system.billing, system.compute, system.lakeflow). If disabled, the SQL steps fall back to slower per-resource CLI loops (noted inline).

Conventions#

• Replace $P with the profile under audit; $ACCT with the account-console profile.
• Run the whole runbook once per environment (dev / test / stage / prod) and once for the account.
• Names are literal. Never normalize hyphens to underscores. Backtick-quote any name part with a hyphen in SQL: `my-catalog`.schema.table.
• ⚠️ Unity Catalog commands take POSITIONAL args, not flags. schemas list <CATALOG> — schemas list --catalog-name X does not exist and will fail.
• Capture resource IDs with every finding so it stays actionable later.

---

Permissions Required (per phase)#

Databricks access spans four independent permission planes: account roles, workspace object ACLs, Unity Catalog privileges, and system-table grants. This audit reads only metadata and system tables — never business data — so SELECT on user tables is never required.

Hard preconditions for the system-table phases (B, C):

• An account admin must have enabled the relevant schemas (system.billing, system.compute, system.lakeflow, and system.access for audit logs). They're not all on by default.
• The metastore must be on Unity Catalog Privilege Model v1.0, and you must query from a UC-enabled workspace.
• System tables only contain data for workspaces in the same cloud region. To audit a workspace in another region, run Phases B/C from a workspace deployed in that region.

Recommended "auditor" grant bundle (read-only, least privilege):

1. Workspace admin (read) in each environment workspace — the practical way to CAN_VIEW every job/pipeline/cluster/notebook at once (A2, C3, D).
2. USE CATALOG on system + USE SCHEMA + SELECT on system.billing, system.compute, system.lakeflow, system.access (B, C).
3. CAN_USE on one SQL warehouse (to run the queries).
4. BROWSE on all catalogs — or metastore admin if you also need external locations / storage credentials (A3).
5. Account admin — only required for Phase A1 (account-console inventory). This is the one genuinely high-privilege ask; if it's denied, drop A1 and audit per-workspace.

Grant to a group, not your user. Per UC best practice (and consistent with this audit's own findings), assign the bundle to an auditors group. Whoever grants it must themselves be both an account admin and a metastore admin.

---

Procedure#

Step 0 — Authenticate (prerequisite, not a mutation)#

databricks auth login --host <WORKSPACE_URL> --profile $P
databricks current-user me --profile $P

Expected: OAuth flow completes; current-user me returns your identity JSON. If it fails: configuration does not support OAuth tokens → re-run the login to convert a PAT/azure-cli profile. On Azure with auth_type = azure-cli, run az login --tenant <TENANT_ID> first — the Databricks profile borrows that identity.

---

PHASE A — Inventory ("what exists")#

A1 — Account-level (run with $ACCT)#

databricks account workspaces list --profile $ACCT
databricks account metastores list --profile $ACCT
databricks account groups list --profile $ACCT
databricks account users list --profile $ACCT

Look for: more metastores than regions; users granted directly instead of via groups; orphaned or empty workspaces; duplicate workspace configs. If it fails: subcommand names vary by CLI build — confirm with databricks account --help. No account-admin → skip A1, audit per-workspace only and flag the coverage gap.

A2 — Per-workspace resources (run for each environment)#

databricks clusters list --profile $P
databricks warehouses list --profile $P
databricks instance-pools list --profile $P
databricks cluster-policies list --profile $P
databricks jobs list --profile $P
databricks pipelines list --profile $P
databricks serving-endpoints list --profile $P

Expected: one block per resource type; capture counts + IDs. If it fails: databricks sql-warehouses list does not exist — use warehouses list.

A3 — Unity Catalog (governance)#

databricks catalogs list --profile $P
databricks schemas list <CATALOG> --profile $P
databricks tables list <CATALOG> <SCHEMA> --profile $P
databricks external-locations list --profile $P
databricks storage-credentials list --profile $P

Look for: a hive_metastore catalog still in active use (un-migrated legacy data — major governance gap); external locations with broad access; storage credentials shared too widely.

---

PHASE B — Cost / Waste#

Prefer system tables — one query beats iterating the CLI over every resource. Get the warehouse first:

databricks experimental aitools tools get-default-warehouse --profile $P

B1 — Top DBU spend by SKU, last 30 days#

databricks experimental aitools tools query "SELECT sku_name, usage_unit, ROUND(SUM(usage_quantity),1) AS dbus FROM system.billing.usage WHERE usage_date >= DATE_SUB(CURRENT_DATE,30) GROUP BY 1,2 ORDER BY dbus DESC LIMIT 20" --profile $P

This is the "what's going on" headline. TABLE_OR_VIEW_NOT_FOUND → billing schema not enabled; flag "enable system tables" as a finding and use B3.

B2 — Idle / always-on clusters (classic leak)#

databricks experimental aitools tools query "SELECT cluster_name, auto_termination_minutes, worker_node_type, num_workers FROM system.compute.clusters WHERE delete_time IS NULL ORDER BY auto_termination_minutes DESC" --profile $P

Red flag: auto_termination_minutes = 0 or NULL → burning DBUs 24/7. Top remediation target.

B3 — Fallback: per-cluster config sweep (no system tables)#

databricks clusters list --profile $P --output json

Scan for autotermination_minutes (0/missing), spark_version (outdated runtime = security + perf debt), oversized node_type_id.

---

Active incident triage (first 15 minutes)#

Use this path before you restart a failed job or pipeline. Preserve evidence first.

1. Record the UTC time, workspace, workload name, resource ID, and the person who reported it.
2. Record the last successful run, the failed run ID, and the exact failure time.
3. Capture the current job or pipeline configuration and note changes since the last success.
4. Capture the exact error before you retry: - Jobs: databricks jobs list-runs --job-id <ID> --limit 10 --profile $P --output json. - Pipelines: databricks pipelines list-pipeline-events <PIPELINE_ID> --max-results 50 --profile $P --output json.
5. Check whether the failure affects one workload or a shared dependency, then continue with C3 for jobs or D1/D2 for pipelines. Record every command and result in the findings table.

Do not restart until you capture the failed run ID, error, and current configuration. A retry can replace the useful failure context.

---

PHASE C — Jobs Reliability (deep dive)#

The job timeline contains hourly slices. The result_state value exists only on the final slice. Count final slices as runs, and combine all slices before you calculate a run duration.

Source: https://learn.microsoft.com/azure/databricks/admin/system-tables/jobs

C1 — Failure rates, last 30 days (system tables)#

databricks experimental aitools tools query "SELECT job_id, COUNT(*) AS runs, COUNT_IF(result_state <> 'SUCCEEDED') AS failures, ROUND(100.0 * COUNT_IF(result_state <> 'SUCCEEDED') / COUNT(*), 1) AS fail_pct FROM system.lakeflow.job_run_timeline WHERE period_start_time >= DATE_SUB(CURRENT_DATE, 30) AND result_state IS NOT NULL GROUP BY 1 HAVING failures > 0 ORDER BY fail_pct DESC, failures DESC LIMIT 30" --profile $P

Worst offenders by failure rate. Column names vary by region — if one errors, run databricks experimental aitools tools discover-schema system.lakeflow.job_run_timeline --profile $P.

C2 — Long-running / drifting jobs (duration outliers)#

databricks experimental aitools tools query "WITH runs AS (SELECT job_id, run_id, (UNIX_TIMESTAMP(MAX(period_end_time)) - UNIX_TIMESTAMP(MIN(period_start_time))) / 60.0 AS duration_min FROM system.lakeflow.job_run_timeline WHERE period_start_time >= DATE_SUB(CURRENT_DATE, 30) AND period_end_time IS NOT NULL GROUP BY job_id, run_id) SELECT job_id, ROUND(AVG(duration_min), 1) AS avg_min, ROUND(MAX(duration_min), 1) AS max_min, COUNT(*) AS runs FROM runs GROUP BY job_id ORDER BY max_min DESC LIMIT 20" --profile $P

Red flag: max_min ≫ avg_min → runaway runs (likely missing a timeout_seconds).

C3 — Per-job config review (anti-pattern scan)#

For each high-risk job ID from C1/C2:

databricks jobs get <JOB_ID> --profile $P --output json

Score each job against the table below (these are the patterns that quietly create a poor experience):

Quick jq to surface the two most common offenders:

databricks jobs get <JOB_ID> --profile $P --output json | jq '{name, tasks: [.settings.tasks[] | {task_key, uses_all_purpose: (.existing_cluster_id != null), retries: .max_retries}], on_failure: .settings.email_notifications.on_failure}'

C4 — Fallback: recent runs per job (no system tables)#

databricks jobs list --profile $P
databricks jobs list-runs --job-id <ID> --limit 25 --profile $P

---

PHASE D — Pipelines Reliability (deep dive — SDP / formerly DLT)#

D1 — Inventory + health#

databricks pipelines list --profile $P
databricks pipelines get <PIPELINE_ID> --profile $P --output json

Score each pipeline against the table below:

D2 — Recent failures + the REAL error message#

databricks pipelines list-pipeline-events <PIPELINE_ID> --max-results 50 --profile $P

⚠️ Read the real cause from error.exceptions[0].message, NOT the top-level .message (which only says "Update X is FAILED"):

databricks pipelines list-pipeline-events <PIPELINE_ID> --max-results 50 --profile $P --output json | jq -r '.events[] | select(.level=="ERROR") | .error.exceptions[0].message' | sort | uniq -c | sort -rn

Note: a pipeline stuck INITIALIZING on serverless is a normal cold start (a few minutes), not a failure — don't flag it as broken.

D3 — Source-code anti-patterns (if pipeline source is accessible)#

Pull the pipeline's source notebooks/files and scan. Legacy DLT syntax is migration debt and a reliability/perf risk — flag every hit:

rg -n "import dlt|@dlt\.|dlt\.apply_changes|LIVE\.|CREATE LIVE TABLE|CREATE STREAMING LIVE TABLE|CREATE OR REPLACE (STREAMING TABLE|MATERIALIZED VIEW)" <pipeline_src_dir>

---

PHASE E — Cross-Cutting Reliability#

• Alerting coverage: cross-reference C3 (jobs) + D1 (pipelines) — what fraction of production workloads have any failure notification? A low number is usually the single biggest finding.
• Full-refresh exposure: note any pipeline/job that triggers a pipeline full refresh on a schedule — it reprocesses streaming sources from scratch and can cause data loss.
• Runtime/version drift: outdated spark_version (jobs) and channel: PREVIEW (pipelines) → security + reproducibility risk.
• Tagging: are clusters/jobs/warehouses tagged (team/cost-center)? Untagged = no cost attribution.

---

PHASE F — Deployment / IaC Hygiene (DABs) · Operational excellence#

Run from each bundle root (the directory containing databricks.yml). The audit question: is this environment managed as code, or hand-clicked?

databricks bundle validate --strict --profile $P
databricks bundle validate --strict -t <TARGET> --profile $P
databricks bundle summary -t <TARGET> --profile $P

Then cross-reference what's bundle-managed against the live inventory to find hand-created resources:

databricks bundle summary -t <TARGET> --profile $P   # IaC-managed resources
databricks jobs list --profile $P                    # everything that exists
# anything in `jobs list` but absent from `bundle summary` was created by hand

Caveats (flagged by reader): there is no bundle list-targets command — read targets from databricks.yml. pipelines list-pipelines / apps list aren't shown in the DABs skill — confirm with databricks <group> --help.

---

PHASE G — Serverless Migration Opportunities · Performance efficiency + Cost#

"Use serverless architectures" is the #1 principle of the Performance Efficiency pillar. This phase finds classic compute that should move.

databricks experimental aitools tools query "SELECT sku_name, usage_type, ROUND(SUM(usage_quantity),1) AS dbus FROM system.billing.usage WHERE usage_date >= DATE_SUB(CURRENT_DATE,90) GROUP BY 1,2 ORDER BY dbus DESC" --profile $P

Workloads whose usage_type is not SERVERLESS_COMPUTE are the candidate spend. Then per job/pipeline, the migration signal is in the spec JSON:

databricks jobs get <JOB_ID> --profile $P --output json   # job_clusters/new_cluster = classic; environments/environment_key = serverless
databricks pipelines get <PIPELINE_ID> --profile $P --output json

Candidates: all-purpose clusters backing notebook/Python work; jobs on classic job_clusters; spark_jar_task on classic; outdated DBR (13.x/14.x); no scale-to-zero. Compatibility blockers (don't over-recommend): RDD/SparkContext APIs, %scala/%r cells, custom Spark data-source JARs, unsupported spark.conf.set(...), DBFS mounts / Hive Metastore, continuous-trigger streaming, ML libs not pre-installed on serverless.

Caveat: system.compute.clusters is a real table but not referenced by the serverless skill — confirm columns (cluster_source, dbr_version, auto_termination_minutes) in-env before relying on them.

---

PHASE H — Model Serving (deepens A2) · Cost + Reliability + Governance#

databricks serving-endpoints list --profile $P -o json
databricks serving-endpoints get <NAME> --profile $P -o json
databricks serving-endpoints get <NAME> --profile $P -o json | jq '.config.served_entities[] | {entity_name, entity_version, scale_to_zero_enabled, workload_size, min_provisioned_throughput, max_provisioned_throughput}'
databricks serving-endpoints get-permissions <ENDPOINT_ID> --profile $P   # takes ID, not name
databricks serving-endpoints export-metrics <NAME> --profile $P           # Prometheus metrics

Cost: system.billing.usage WHERE billing_origin_product = 'MODEL_SERVING'. Caveat: the skill exposes no CLI for UC registered-model/alias listing (it uses the MLflow client); served_entities[].entity_name/entity_version from get covers the serving side.

---

PHASE I — Vector Search · Cost + Performance + Governance#

databricks vector-search-endpoints list-endpoints --profile $P
databricks vector-search-endpoints get-endpoint <ENDPOINT_NAME> --profile $P
databricks vector-search-indexes list-indexes <ENDPOINT_NAME> --profile $P
databricks vector-search-indexes get-index <CATALOG>.<SCHEMA>.<INDEX> --profile $P

Health states to watch on get-endpoint: YELLOW_STATE, RED_STATE, OFFLINE, stuck PROVISIONING. Caveat: the VECTOR_SEARCH billing-SKU string and system-table mappings aren't documented in the skill — confirm in-env before quoting cost numbers.

---

PHASE J — Databricks Apps · Security + Cost + Usability#

databricks apps list --profile $P -o json          # verify with --help (not shown in skill)
databricks apps get <APP_NAME> --profile $P -o json # app_status.state, url, compute, resources[], service_principal
databricks apps logs <APP_NAME> --profile $P        # OAuth auth only — silently fails on PAT

Limits: 100 apps/workspace; 10-min start; 120s proxy timeout (504 not in app logs); ephemeral FS — only stdout/stderr survive. UX (from databricks-app-design): missing loading/empty/error states; Genie/AI surfaces that hide the generated SQL, execution identity, or "verify" disclaimer.

---

PHASE K — Lakebase (Postgres / OLTP) · Cost + Reliability + Security#

CLI group is postgres (not lakebase); needs CLI ≥ v0.294.0. Discover first — the surface is narrow: databricks postgres -h.

databricks postgres list-branches projects/<PROJECT_ID> --profile $P
databricks postgres list-endpoints projects/<PROJECT_ID>/branches/<BRANCH_ID> --profile $P
databricks postgres list-databases projects/<PROJECT_ID>/branches/<BRANCH_ID> --profile $P
databricks postgres get-endpoint projects/<PROJECT_ID>/branches/<BRANCH_ID>/endpoints/<ENDPOINT_ID> --profile $P
databricks postgres get-synced-table "synced_tables/<CATALOG>.<SCHEMA>.<TABLE>" --profile $P

Caveats (flagged by reader): no documented list-projects/get-project or list-synced-tables command — enumerate via the Python SDK (w.postgres), REST /database/, or Catalog Explorer UI. Provisioned-era instances use the separate databricks database group. Never run generate-database-credential during an audit — it mints a live DB token. Cost: CU-hours (~2 GB RAM/CU) + synced-table DLT pipeline cost; both surface under system.billing.usage (confirm exact SKU in-env).

---

Verification#

• [ ] One completed inventory block per environment + the account
• [ ] B1 (spend) and C1 (job failures) returned data, or their fallbacks ran
• [ ] Every high-risk job (C3) and pipeline (D1) scored against its anti-pattern table
• [ ] Phases F–K run only for products that exist in the environment (skip absent ones; note which were skipped)
• [ ] Each finding tagged with its Well-Architected pillar (so the report rolls up by pillar)
• [ ] Findings captured with resource IDs

Findings Template#

Troubleshooting#

Rollback#

None — every step is read-only. Nothing is created or modified, so there is nothing to undo.

Escalation#

History#