Operate

Databricks platform lessons

Operate5 min readreview needed

Things I've learned running an Azure Databricks platform as code. Generalized so they apply beyond any one engagement. Grounded in real work, not theory.

Plan before anyone builds#

  • Decide dev/test/prod and a workspace + catalog strategy up front. A workspace per environment lets you test infra changes (storage, compute policies, init scripts) in a lower environment before they hit prod.
  • env + medallion catalogs work well: each workspace gets bronze/silver/gold. Bronze is schema-per-source; silver and gold are schema-per-data-product.
  • The temptation is to spin up a workspace and start writing notebooks. Don't. The platform (catalogs, groups, policies, service principals) should exist before teams build on it.

Two tools, one boundary: Terraform vs DABs#

  • Terraform answers "what does the workspace look like?" — networking, storage, workspace, Unity Catalog structure, grants, identity, cluster policies, account groups, service principals.
  • DABs answer "what runs inside the workspace?" — jobs, pipelines, notebooks, the compute and secrets tied to a workload. The asset's author (data engineering) owns the bundle.
  • Don't put workload resources in Terraform. Keeping two repos in sync is painful and you get state conflicts. See best-practices/terraform-vs-dabs.md.

Unity Catalog grants will bite you#

  • databricks_grants (plural) is authoritative — it owns every grant on the object and removes anything it doesn't declare. DABs sets its own grants on catalogs/schemas when it deploys, so plural Terraform grants wipe them out on every apply. Use databricks_grant (singular) for objects DABs also touches, so the two coexist. See best-practices/unity-catalog-grants.md.
  • Grant to groups, not users. Bind Entra ID (or Okta) groups to Databricks account groups and let Terraform grant on the group.

Compute policies need sane defaults#

  • The default cluster auto-termination of 4320 minutes is not sensible. Override it in a cluster policy.
  • Size SQL warehouses deliberately and give them a short timeout (they start and stop in seconds). Don't allow unrestricted compute.

Identity: machines run jobs, not people#

  • Run production jobs as service principals, one per environment, scoped to only the catalogs they need.
  • Manage access groups in Entra ID. Use Automatic Identity Management to create and sync the Databricks account groups. Use SCIM only when AIM does not apply.
  • For CI/CD, use workload identity federation (OIDC) instead of long-lived PAT or service-principal secrets.

Secrets: split by where they're used#

  • Databricks secret scopes for anything read inside a job's context.
  • An external store (Key Vault, GitHub Secrets, Doppler) for secrets used by CI/CD. For GitHub-based CI, GitHub Secrets is usually enough.

Networking: measure twice, you can't resize#

  • In Azure you cannot resize a subnet once it's assigned to a Databricks workspace. To change it you tear the workspace down and recreate it. Size generously: /21 per VNet has proven plenty.
  • Clusters don't need public IPs as long as they have an egress route via NAT gateway or firewall.
  • Azure has two tiers, Standard and Premium. There is no Enterprise tier — that one is AWS. I had this backwards, sourced from an AWS-flavoured Reddit comment (things-you-wish-you-knew.md, item 12), and it is wrong on Azure. Premium is the prerequisite for Private Link, not the expensive alternative to it: every Private Link leg on Azure requires the Premium plan.
  • Standard tier is going away. New workspaces have been Premium-only since 2026-04-01, and any Standard workspace left on 2026-10-01 is upgraded to Premium automatically. On Azure, plan for one tier. Check a client's workspace tier in the account console and price the change before the date.
  • The cost argument underneath the Reddit comment still holds, just without the tier framing: every non-serverless cluster pulls a ~15 GB image from the control plane at start, and that NAT data-processing cost adds up. Private Link avoids it. Price Private Link endpoints against your NAT egress bill.

Sources for the tier correction, checked 2026-08-08: Manage your subscription, End of life for Standard tier workspaces, Private Link concepts.

Cost and observability are platform work#

  • Many system tables are not enabled by default. Turn them on to emit compute usage to UC tables and build real cost dashboards beyond the built-in reports. See best-practices/auditing-and-system-tables.md.
  • Watch the SKU. PREMIUM_SERVERLESS_REAL_TIME_INFERENCE is one of the most expensive DBU classes — confirm any workload on it is intentional and budgeted.

Cost spike triage (first 15 minutes)#

When someone asks "why is the bill up?", do not start in the account console UI. Use system tables, in this order. Full audit commands live in best-practices/auditing-and-system-tables.md Phase B and C.

  1. Spend by SKU (last 30 days). Query system.billing.usage grouped by sku_name. One expensive SKU (model serving, serverless inference, oversized SQL) usually explains the jump.
  2. Idle / always-on classic compute. Query system.compute.clusters for auto_termination_minutes of 0 or NULL on live clusters. That is the classic leak. Override the 4320-minute default in cluster policy before the next incident.
  3. Jobs on all-purpose clusters. For high-spend job IDs, check existing_cluster_id on tasks. Job compute or serverless is usually cheaper and isolates failure. See the anti-pattern table in auditing Phase C3.
  4. Tags and budget policies. Untagged serverless has no chargeback path. Budget policies exist so serverless notebooks and jobs still land tags in billing — give every user at least one. Detail in best-practices/community-best-practices.md (Cost and FinOps).

If system tables are disabled, enable system.billing and system.compute first; the CLI fallback is slower and incomplete.

Onboarding a brownfield platform to Terraform#

  • Import in phases — networking, then storage, then workspace, then grants, then cleanup — so you never interrupt running workloads.
  • Manual changes to prod will happen (a grant here, a cluster policy there). They have to be reverse-engineered back into Terraform. Codify them and move to PR-based change control so the drift stops.

Operational discipline#

  • PR-driven Terraform with rendered plan comments, environment CODEOWNERS, and concurrency-locked applies.
  • Pin provider versions; SHA-pin GitHub Actions; retain plan artifacts (90 days) for audit.

Resist service sprawl#

  • Don't add ADF if DABs already orchestrates your Databricks jobs. Don't add Purview if Unity Catalog already catalogs. Don't add Azure ML until you need it. Every extra service is a moving part that earns its place only by solving something Databricks can't.

Small things#

  • Databricks has a dark theme. It's in settings.