Databricks knowledge base

Best practices, lessons, and consulting playbooks for Databricks (Azure-focused). The place to come and ask questions, and to drop new learnings as they happen.

Start here#

Use the full-text search in the generated site, or search the Markdown source directly. HTML is disposable output under _site/, so agents see only one copy of each claim in the repository.

How to trust a claim#

• Treat a source URL as the evidence for an external fact. Open it again for a version, price, tier, limit, or feature status.
• Treat (verify) as unconfirmed. Do not quote that claim to a client.
• Treat Compiled and Refreshed dates as the claim scope. A partial refresh does not make the full document current.
• Treat first-person guidance as Jonathan's field judgment. It can be useful without being a Databricks recommendation.
• Treat generated watch pages as discovery indexes. Follow their links before you make a design decision.
• Treat best-practices/well-architected.md as a dated Microsoft Learn snapshot. The source links at the end remain authoritative.

Watch pages#

Generated. They live in the Watch section. Each one polls a feed so you do not have to.

• azure-release-notes.md — the official Azure Databricks release feed, last 90 days.
• release-notes.md — the AWS Databricks release feed, retained for cross-cloud parity checks.
• blog-watch.md — Databricks blog posts about the platform. The filter is narrow, because the blog is mostly AI marketing.
• community-watch.md — relevant posts from the Databricks Community Technical Blog. Useful practitioner guidance, but verify product claims against official docs.
• microsoft-watch.md — what Microsoft says about Databricks. Partnership news from Microsoft Source, Azure Databricks engineering guidance from Tech Community, Fabric interoperability from the Fabric blog. Databricks writes about Databricks; Microsoft writes about Azure Databricks, and an Azure client reads the second one.
• keywords.md — what people type into Google, and which of those terms this knowledge base already covers.
• skills-watch.md — every Databricks agent skill in the latest release and what it covers. It intentionally excludes local machine state.
• source-status.md — availability and source timestamps for the seven Well-Architected pages.
• content-candidates.md — one review queue assembled from Azure releases, Community, practitioner discussions, and search-demand gaps.

best-practices/#

How to do a thing well.

• terraform-vs-dabs.md — what Terraform owns vs what DABs owns, with a resource-by-resource table.
• unity-catalog-grants.md — the authoritative plural grant vs principal-scoped singular grant, plus traversal, BROWSE, inheritance, no DENY, MANAGE, and grant visibility limits.
• identity-entra-scim.md — Entra ID → Databricks identity via Automatic Identity Management, with the SCIM migration boundary and Terraform ownership.
• governance-pii-abac.md — PII, ABAC, and group-based grant governance.
• auditing-and-system-tables.md — audit/review procedure and system tables for cost and usage visibility.
• community-best-practices.md — the broader vendor/community canon: Well-Architected Lakehouse pillars, UC governance, Delta optimization (liquid clustering, predictive optimization), FinOps, CI/CD, Azure security baseline.
• reddit-dataengineering.md — practitioner signal from the r/dataengineering subreddit (Databricks-leaning): platform comparisons, governance/cost takeaways, and Delta Lake operational gotchas, with cited threads. Dated capture; re-run the sweep to refresh.
• new-features-to-watch.md — recent and emerging features to evaluate (compiled 2026-06-24, goes stale): Lakeflow, ABAC GA, Lakebase, managed Iceberg, Mosaic AI, and Data + AI Summit 2026 announcements.

lessons-learned/#

What I'd tell someone starting out, and what bit me.

• platform-lessons.md — planning, compute policies, networking (you can't resize a subnet), secrets, brownfield Terraform onboarding, cost, resisting service sprawl.
• things-you-wish-you-knew.md — captured practitioner advice on standing up a platform end to end.

migrations/#

Technical playbooks for moving companies onto Databricks.

• README.md — how the migration guides fit together.
• migration-playbook.md — source-agnostic methodology (assess → design → migrate → validate → cut over → operate), strategy choices, and validation.
• sql-server-to-databricks.md — SQL Server + SSIS/T-SQL migration, Lakebridge/Lakeflow Connect, conversion gotchas.
• azure-synapse-to-databricks.md — Synapse (dedicated/serverless SQL, Spark, pipelines) migration and the Synapse-vs-Fabric positioning.

Keeping this current#

./refresh.sh      # refresh unattended feeds and rebuild the searchable site
./build.sh        # build _site/ and the Pagefind search index
./open.sh         # build, serve locally, and open the field manual

Then read the diff and commit what you want to keep:

git diff --stat

refresh.sh wraps refresh.py, which pulls the unattended public sources independently:

The public, keyless commands run unattended. .github/workflows/refresh.yml pulls them every Monday and opens a pull request when any of them moved. blog-watch.md, community-watch.md, and microsoft-watch.md append rather than replace, because those feeds only hold a few days and a weekly poll would otherwise drop posts. The blog filter is deliberately narrow — the Databricks blog is mostly AI marketing. The Microsoft filter is just the word databricks, across Microsoft Source, the official blog, the Azure blog, the Fabric blog, and the Azure Databricks board on Tech Community.

The Community and Reddit feeds are deliberately keyless. Reddit's JSON and HTML endpoints block hosted runners, so the refresh uses one combined subreddit RSS request instead. LinkedIn still needs a signed-in browser session and remains local-only.

Source refreshes rewrite tracked Markdown. The build rewrites only ignored _site/ output. Nothing is committed for you.

Content registry#

content.toml is the source of truth for navigation, product-area coverage, content type, cloud scope, status, and review dates. coverage.md becomes a live coverage/freshness dashboard during the build. A review date means a human checked the product claims; it is never inferred from a Git timestamp.

Notes#

• Currency: new-features-to-watch.md and the migration tooling references move over time. Several items are tagged (verify) — confirm against the live source before quoting to a client.
• Adding to this: drop notes into the right folder, or ask me to distill a messy note, scrub client specifics, or pull fresh Databricks guidance on a topic.

---

Reorganized 2026-06-24. Client-specific delivery files are intentionally kept out of this knowledge base — everything here is generalized and safe to share.