Learn

Databricks community and industry best practices

Learn16 min read10 days old

The vendor and community canon for Azure Databricks. Azure-focused, mostly cloud-agnostic. This is the published guidance, not my engagement notes — for those see lessons-learned/platform-lessons.md. Where a topic overlaps with my own docs I point at them rather than repeat.

Search this file for: liquid clustering, ZORDER, predictive optimization, Photon, serverless vs classic, cluster policies, FinOps, budget policies, SQL warehouse sizing, Asset Bundles / DABs, secure cluster connectivity, Private Link, Lakeflow expectations, table constraints.

Compiled: 2026-06-24. Features are marked GA or Public Preview where it matters. (verify) means confirm against current docs before you rely on it.

Refreshed: 2026-08-08 — catalog design, managed-table retention, ABAC status, compute sizing, serverless limits, predictive optimization, and secure cluster connectivity checked against current docs. The rest of the file still dates from the compile.

Well-Architected Lakehouse Framework#

Seven pillars: five mirror the Azure Well-Architected Framework (operational excellence; security, compliance and privacy; reliability; performance efficiency; cost optimization) and two are lakehouse-specific (data and AI governance; interoperability and usability).

The practice-level detail lives in well-architected.md, generated from the seven Microsoft Learn pillar pages. Read that when you need the checklist; this file covers the wider vendor and community canon.

Unity Catalog governance#

UC is the governance layer and is on by default for workspaces created after 2023-11-09. Object model is a three-level namespace: catalog.schema.object.

Catalog and schema design. Databricks recommends domain-based catalogs, such as sales or finance. Environment-based, project-based, and combined patterns also work when those boundaries match the required isolation. Use catalogs as the primary isolation unit. Do not use a fixed catalog count. Create a catalog for a real domain, environment, team, or project boundary. (My own env+medallion catalog layout is in lessons-learned/platform-lessons.md.)

Medallion. Current governance guidance puts bronze, silver, and gold schemas inside a domain catalog. Use a different layout only when the lifecycle layers need different isolation.

Managed vs external — prefer managed. Databricks recommends managed tables and volumes for all new tables and most use cases. Managed tables get auto-compaction, auto-optimize, metadata caching, intelligent file sizing, and automatic upgrades to new features; new platform features increasingly land on managed tables first. Managed tables are Delta or Iceberg only. Dropping a managed table soft-deletes it: you can UNDROP for about 7 days, and the data files are removed after that. The window is configurable — ALTER CATALOG my_catalog RETAIN DROPPED TO 30 DAYS, also available on ALTER SCHEMA and at create time as RETAIN DROPPED FOR. Managed does not mean lock-in: external engines read managed tables through open APIs (Unity REST, Iceberg REST Catalog).

Use external tables/volumes when: upgrading from Hive metastore without moving data, non-Delta/non-Iceberg formats (Parquet, Avro, ORC), DR needs managed tables can't meet, or external writers must touch the files. Even then, limit external access to reads and route writes through Databricks. Create external tables/volumes from one external location per schema, and put external locations at the highest common path prefix (one per catalog or per schema is a clean default).

Tags and ABAC. ABAC row filtering and column masking policies, governed tags, and data classification went GA in April 2026 (per-feature dates and the breaking session-user change in governance-pii-abac.md). Governed tags are account-level key/value pairs with an enforced policy: allowed values, and who can assign them. They apply across all workspaces and metastores in the account. They drive data classification, cost attribution (cost center / project), discovery, and attribute-based access control via has_tag() / has_tag_value() in policy conditions. Tags inherit down catalog → schema → table, but not to columns — column tags are applied directly. Max 1,000 governed tags/account, 500 allowed values each, 50 tags/object. Tag data is plain text and may be replicated globally, so never put sensitive values in tag names or values. Note: governed tags don't apply to compute (warehouses, jobs) — those use the separate compute-tag mechanism that feeds billing.

Lineage is captured automatically to table and column level, no setup. Known limits worth keeping in mind: nothing before 2024-09-01; lineage is lost on rename of any object; path-referenced reads/writes (delta."abfss://...") and UDFs break column-level capture; runs submit / spark submit jobs capture table/column lineage but not the run link.

Delta Lake and table optimization#

Liquid clustering is GA for Delta tables on Databricks Runtime 15.4 LTS and above (Public Preview for Iceberg on 16.4 LTS+). It replaces both partitioning and ZORDER and is the recommendation for all new tables. Unlike partitioning you can change clustering keys without rewriting data. Use it for high-cardinality filter columns, skewed or fast-growing tables, and concurrent-write tables. Rules: up to four keys, chosen from the columns most used in query filters; keys must have statistics (Delta collects stats on the first 32 columns by default); CLUSTER BY is mutually exclusive with PARTITIONED BY.

Z-order is legacy. Migration is low-friction: stop running OPTIMIZE ... ZORDER BY, reuse the same columns as clustering keys, and run plain OPTIMIZE. On DBR 18.1+ there's an in-place ALTER TABLE ... REPLACE PARTITIONED BY WITH CLUSTER BY [(...) | AUTO] that converts a partitioned table with minimal read/write downtime. CLUSTER BY AUTO seeds from the existing partition columns and lets predictive optimization take over (UC managed tables only). After enabling clustering or changing keys, run OPTIMIZE FULL once to recluster existing data; subsequent OPTIMIZE runs are incremental.

Automatic liquid clustering (CLUSTER BY AUTO, DBR 15.4 LTS+, UC managed tables) lets Databricks pick and adapt keys from your query history, and only changes keys when predicted data-skipping savings beat the reclustering cost. It won't select keys when the table is too small, already well-clustered, or rarely queried.

Predictive Optimization runs OPTIMIZE and VACUUM automatically for UC managed tables. If you turn it on, disable your scheduled OPTIMIZE/VACUUM jobs — don't run both. Its optimized VACUUM path reads the Delta log to find removable files instead of doing a full directory listing, which matters a lot on tables with millions of files (verify current numbers).

OPTIMIZE / VACUUM / file sizing. Predictive Optimization runs OPTIMIZE, VACUUM, and ANALYZE for you on Unity Catalog managed tables. It is on by default for accounts created on or after 2024-11-11, and now for all new managed tables, so check before you assume a client needs manual jobs. Without it, schedule OPTIMIZE regularly — every 1-2 hours for tables with heavy updates/inserts (incremental, so usually fast). VACUUM removes unreferenced files for storage and compliance; respect the default retention floor (7 days) unless you have a specific reason. Let the platform manage file sizing on managed tables rather than hand-tuning spark.databricks.delta configs.

Deletion vectors are enabled by default with liquid clustering. They mark rows deleted without rewriting files, so DELETE/MERGE/UPDATE are faster, and they enable row-level concurrency (fewer write conflicts). Turning them off also turns off row tracking and row-level concurrency, so leave them on unless an old reader forces your hand (deletion-vector reads need DBR 12.2 LTS+). Liquid clustering bumps the table to Delta writer v7 / reader v3 — older Delta clients can't read it and you can't downgrade.

Performance, compute sizing, and serverless#

Photon is the native vectorized C++ engine, Spark-API compatible, no code changes. On by default on all SQL warehouses and on serverless. Most benefit on SQL and DataFrame workloads with wide transformations — joins, aggregations, large scans. Minimal benefit on simple sub-2-second ETL. Evaluate recurring jobs for whether Photon is both faster and cheaper.

Serverless vs classic. Databricks recommends serverless for new SQL warehouses, jobs, notebooks, and pipelines: near-zero startup, autoscale and Photon on automatically, no cloud-infra management, pay only for compute time. Serverless compute has no public IPs and doesn't use secure cluster connectivity (it's isolated differently — see Security). Keep classic where you need specific instance types, init scripts, long-running custom-config clusters, or predictable steady usage where reserved capacity / DBCU commitments win on price.

Classic compute sizing. Size from executor cores, executor memory, and local storage, not only worker count. Start with general-purpose workers, then read the workload evidence. For a shuffle-heavy job, try fewer, larger workers to reduce network transfer. Increase worker memory when the Spark UI shows spill or out-of-memory errors. Add workers when the job needs more parallel tasks. Measure the next run before you keep the change.

Serverless compatibility check. Serverless notebooks and jobs use Spark Connect. They do not support R, RDD APIs, the Spark UI, Spark logs, compute policies, init scripts, instance pools, or most Spark settings. A serverless job can run for no more than seven days. For Structured Streaming, use Trigger.AvailableNow() or a Lakeflow pipeline. Serverless jobs do not support processing-time or continuous Spark triggers. Use classic compute when the workload needs one of these features, or change the workload first.

Cluster policies are the guardrail layer. Enforce sizing standards, restrict expensive instance types, require autoscaling, cap worker counts, and define T-shirt sizes (S/M/L). Override the default auto-termination (the 4320-minute default is absurd — see lessons-learned/platform-lessons.md). Add libraries through the policy rather than init scripts.

Autoscaling. Enable it with a sane minimum (often 2 workers) and a max within workspace quota; target 70-80% utilization. Always pair with auto-termination (1 hour is a reasonable classic default; aggressive for dev).

Spot / Fleet and pools. Apply a spot-instance strategy for fault-tolerant workloads (Spark reschedules tasks lost to a revoked spot VM). Instance/cluster pools keep prewarmed VMs to cut startup from minutes to under ~30-60 seconds; useful for interactive and job clusters but the idle instances accrue infra cost, so size pools to real concurrency.

Compute hygiene (these also ease a later move to serverless): avoid init scripts and compute-scoped libraries (drift + conflicts); don't hardcode Spark configs (you override built-in optimizations and waste spend); don't write to compute-local paths or use DBFS mounts — use UC volumes. Turn on AQE.

Cost and FinOps#

System tables are the foundation. system.billing.usage carries per-record attribution: usage_metadata (the resource), identity_metadata (who ran it), and custom_tags. Join to system.billing.list_prices for dollars and system.access.workspaces_latest for workspace context. The usage schema follows the FinOps FOCUS standard, so it lines up with cloud-provider billing. Note many system tables aren't enabled by default — see best-practices/auditing-and-system-tables.md.

Tag from day one. At minimum, custom tags for business unit and project on workspaces, clusters, warehouses, and pools, which then flow into custom_tags in billing. This enables showback (visibility) and chargeback (direct billing). Governed tags (above) keep the taxonomy consistent.

Budget policies solve serverless attribution: serverless compute has no cluster to tag, so a budget policy auto-applies tags to whatever a user creates (notebook/job/pipeline) on serverless, and those tags land in billing. Give every user at least one budget policy. Account/workspace budgets add spend tracking and alerts — a common cadence is alerting at 50/75/90% of the monthly threshold (verify exact feature behavior).

SQL warehouse sizing. Two independent levers: cluster size (scale up) for query complexity and disk spills — if the query profile shows spill, size up; cluster count (scale out, max-clusters) for concurrency. Start Small/Medium with autoscaling and separate warehouses by workload (dev small, prod larger; don't share one warehouse across mismatched patterns). Prefer serverless warehouses: they start and scale in seconds, scale down sooner than classic, and Intelligent Workload Management right-sizes resources per query — so you get instant availability and aggressive idle termination at once. Set short auto-stop (1 minute is fine for dev).

Serverless tradeoffs. Upside: no idle cost, no infra management, fast startup, auto-optimization. Watch: it's consumption-billed, so a runaway query or always-on warehouse still costs — budgets and policies are your control. And confirm expensive SKUs are intentional (PREMIUM_SERVERLESS_REAL_TIME_INFERENCE is among the priciest DBU classes — see lessons-learned/platform-lessons.md).

CI/CD and DevOps#

Declarative Automation Bundles (DABs; formerly Databricks Asset Bundles) are the standard. dbx is deprecated and no longer maintained; there's an official dbx-to-bundles migration guide. Bundles describe jobs, pipelines, and other resources as source files and are the recommended CI/CD path on Databricks. Docs: https://learn.microsoft.com/azure/databricks/dev-tools/bundles/. My Terraform-vs-bundles boundary is in best-practices/terraform-vs-dabs.md — that's the load-bearing decision; this section is the mechanics around it.

  • Repo structure. Single repo for code + bundle config when they're tightly coupled (most cases); separate repos for large teams with independent release cycles, with CI enforcing version compatibility.
  • Versioned artifacts. Upload with Git commit hashes for traceability and rollback. Parameterize environment-specific values (cluster size, secrets) instead of hardcoding.
  • Environment isolation. Separate dev/staging/prod workspaces; bundle deployment modes (dev vs prod) handle the differences.
  • Testing. Develop notebooks/tests locally or in-workspace; databricks bundle validate for config; pytest (Python wheels) / JUnit (JARs) for units; chispa for Spark DataFrame assertions; integration tests for full pipelines. Lint with Pylint plus the Databricks Labs pylint plugin.
  • Lighter option. Git folders (repos) source-control notebooks without full bundle CI/CD, for teams not ready for pipelines.
  • Known gaps (2025 list — verify against current docs): lifecycle hooks (pre/post-deploy scripts), dependency management beyond wheels, and failure output vs Terraform. The product moved since that note; re-check before you design around a gap.

Security baseline (Azure)#

This is the published baseline; my hard-won networking notes (subnet sizing, Private Link cost math) are in lessons-learned/platform-lessons.md.

  • No public IPs — secure cluster connectivity (NPIP). Enable it on every classic workspace. It is the default for new workspaces and will become mandatory. A new workspace can use the Databricks-managed VNet or VNet injection. An existing workspace must use VNet injection before you add SCC. Serverless has no public IPs but does not use SCC.
  • Private Link, three independent legs: inbound/front-end (users → workspace), back-end/classic (classic compute → control plane), and outbound/serverless (serverless → your Azure resources via NCC private endpoints). Enforce private connectivity to make the workspace reject public connections. The "Isolated environment" reference architecture adds VPN/inbound-Private-Link-only access plus a required egress firewall — that's the pattern for HIPAA/PCI/FedRAMP-style workloads.
  • Egress note (Azure-specific). After 2026-03-31, new Azure VNets default to no outbound internet, so new workspaces need an explicit egress method (NAT gateway). Existing workspaces are unaffected.
  • TLS pinning. Do not run TLS inspection (decrypt/re-encrypt) on cluster ↔ control-plane traffic — certificate pinning will fail the cluster. Use service-endpoint policies or a private package repo (Artifactory/Nexus) to cut firewall rules and artifact-download egress cost.
  • Secrets. Databricks secret scopes for anything read inside a job; an external store (Key Vault, GitHub Secrets) for CI/CD secrets. Detail in lessons-learned/platform-lessons.md.
  • UC over legacy ACLs. Govern through Unity Catalog, not workspace-local table ACLs, DBFS mounts, or Hive metastore. Avoid DBFS mounts entirely — no proper ACLs; use UC volumes. Grant to groups, not users (see best-practices/unity-catalog-grants.md). Enhanced security and compliance settings exist for regulated frameworks (HIPAA, PCI DSS, SOC 2).

Data quality and reliability#

Lakeflow expectations are the primary data-quality mechanism in declarative pipelines (Lakeflow Spark Declarative Pipelines, formerly DLT). An expectation is a SQL boolean per row with one of three violation policies:

  • warn (default) — invalid rows are written and flagged in metrics.
  • drop (ON VIOLATION DROP ROW) — invalid rows dropped before write, count logged.
  • fail (ON VIOLATION FAIL UPDATE) — stops the flow on the first bad record; needs manual intervention.

Metrics emit to the pipeline event log regardless of policy, so track quality trends and alert on regressions by querying event_log(). For records you don't want to silently drop, use the quarantine pattern: two flows, one writing clean rows to the target and one routing failures to a separate table for investigation and reprocessing.

Table constraints (outside pipelines too): NOT NULL and CHECK are enforced on Delta tables (CHECK can do regex via REGEXP/RLIKE and range checks). Primary-key and foreign-key constraints are informational only — not enforced, used for documentation and query optimization. In UC-enabled pipelines you can declare PK/FK on streaming tables and materialized views; to validate PK uniqueness for real, use an EXPECT (num_entries = 1) expectation over a GROUP BY.

Pipeline defaults worth adopting: serverless for new pipelines (enhanced autoscaling, UC and lineage on by default); CLUSTER BY over PARTITIONED BY; Auto Loader for incremental file ingestion; predictive optimization runs OPTIMIZE/VACUUM on pipeline tables automatically. Use the event log plus event hooks (Python functions on pipeline events) to push failures and quality breaches to Slack/PagerDuty.

Sources#