Reddit Capture: Databricks Things You Wish You Knew

Source capture, not the field manual. The 536tech distillation of this comment is platform-lessons.md. Prefer that file for platform advice. Keep this page for provenance and the full quote.

Captured: 2026-06-05
Thread: r/databricks - What are some things you wish you knew?
Comment permalink: First top-level comment
Author display: [deleted] on Reddit page
Why saved: Strong practitioner framing for Azure Databricks platform ownership: workspace design, Unity Catalog, Terraform, DABs, CI/CD, compute policies, service principals, secrets, system tables, network design, and resisting unnecessary service sprawl.
Known error, do not quote: item 12 says "Databricks Enterprise + privatelink will, in most cases, be cheaper than Premium". That is AWS. Azure has only Standard and Premium, and Premium is a prerequisite for Private Link. The author hedges this himself in the same item. The corrected version is in platform-lessons.md. The quote below stays verbatim as the capture; read it against that correction.

Why This Matters#

This is useful interview and architecture prep because it lines up with a real platform-owner view of Databricks:

Plan environments, workspaces, catalogs, access groups, policies, and service principals before teams start building ad hoc notebooks.
Use Terraform for platform infrastructure and Unity Catalog guardrails.
Use Databricks Asset Bundles, source control, and CI/CD for deployable workloads.
Enforce compute policies, short timeouts, SQL warehouse sizing, and restricted personal compute.
Run production jobs as service principals, not human users.
Treat cost reporting, system tables, and network sizing as first-class platform responsibilities.
Avoid adding ADF, Purview, Azure ML, or other services until the need is specific.

Saved Comment#

Tldr Summary:

Plan your workspace design, catalogs, access groups, compute policies, service principals, etc

Deploy the above with terraform

Prototype in notebooks, use DAB + source control + CICD actions for deployment and promotion across environments.

—- Databricks is an extremely powerful platform. The temptation will be there to spin up a workspace, start creating catalogs; and begin writing and scheduling notebooks. Don’t do this.

This is what you’ll want to think about: 1. make sure you have dev/test/production environments. Choose a workspace and catalog strategy that supports these environments. We use env+medallion catalogs (so each workspace has three), and a workspace per environment. Bronze organized with schema-per-source, silver and gold with schema-per-data product/use case. Some people will say that a workspace per environment is overkill. The main advantage is that you’ll have the opportunity to test changes to your underlying infrastructure in a controlled manner (e.g. testing Terraform updates to your storage accounts, compute policies, or init scripts in your lower environment before promoting to test/production environments). 2) I alluded to it above, but I strongly advise deploying your Databricks infrastructure with Terraform. Deploy your core cloud infrastructure, your Unity Catalog metastore, your Databricks catalogs and compute policies, environment service principals for jobs, and your external storage locations via Terraform. If you ever need to quickly spin up new workspaces for whatever reason, like in a disaster recovery scenario, Terraform will make your life much easier. Use source control. Create an infra-specific repository, and CICD actions that deploy a workspace to a target environment based on branch (dev, test, prod) 3) make sure your compute policies have sensible defaults. The Databricks default cluster timeout of 4320 minutes is not sensible. Make sure you add an override to this policy. Make sure your SQL warehouse is appropriately sized and has a minimal timeout (it spins up and down in seconds). Don’t allow unrestricted compute. Create an access group with access to a sensible personal compute policy - ideally provisioned from AD or Okta via SCIM. 4) figure out what your development flow is going to look like. You can start with prototyping in notebooks for your core data logic, but eventually, you will need to orchestrate your code. Databricks Asset Bundles are great for defining jobs as code and orchestrating them in Databricks. They can be easily templated and integrated with any source control system that offers CICD scripting. This can be as easy or as complex as you want it to be, but you basically want a controlled and audited release process to make sure code goes through at least some form of review before progressing from dev to test and test to prod. Getting used to defining job configs as code with DAB will save you tons of time in the future in a multi-environment setup. 5) take the time to plan and model how your data will look in the silver layer as you onboard new bronze sources. Silver should be resilient to source system changes upstream (ie we don’t have to blow up the entire customers table because we switched from Salesforce to Hubspot and never abstracted away the source-specific fields). This is not a Databricks-specific task but still something worth considering. 6) run jobs as service principals. Don’t run jobs as users. Create a service principal for each environment. Give it access to only the catalogs it needs. This can all be handled through terraform. 7) use secrets. Databricks secrets for anything that is accessed within the context of a Databricks job, and an external secret store (azure key vault, AWS secrets manager, Doppler, GitHub secrets, etc) for secrets that are used by CICD. If you are using GitHub, you’ll probably find GitHub Secrets to be more than sufficient for CICD secrets. 8) Databricks System Tables are great, but many are not enabled by default. They allow you to emit things like compute usage history to UC tables, enabling you to build some pretty cool cost dashboards beyond the out-of-the-box cost reports.

9) Databricks has a dark theme. You can enable it in settings. 10) put thought into your network design for each Databricks workspace. Your clusters don’t need public IP addresses as long as they have a route to the internet via a NAT gateway or firewall. Make sure your IP range for each VNet is large enough to accommodate future usage. /21 is what we use and it’s proving to be plenty sufficient. In Azure, you can’t resize a subnet once it’s assigned to a Databricks workspace - you’ll have to blow the workspace away and recreate it. While it’s not the end of the world if your jobs and infrastructure code all live in source control, it’s still a disruptive pain in the ass. 11) resist the urge to add other services into Databricks until you run into a wall. You don’t need ADF if you’re using Databricks built-in orchestration via DAB to solely orchestrate Databricks jobs. It’s just an additional moving part that doesn’t add value unless you need to orchestrate things outside of Databricks together with things inside of Databricks. Same applies to Purview for cataloging, Azure ML for machine learning, etc. don’t introduce complexity unless you’re solving for something specific that Databricks can’t do, or can’t do well for your use case. 12) Databricks Enterprise + privatelink will, in most cases, be cheaper than Premium, solely because of Databricks cluster behaviour on startup. Every non-serverless cluster downloads a 15GB image from Databricks from the control plane at start time, and this can add up quickly in NAT data processing costs. The slight DBU increase is usually offset by the decrease in data processing costs between private link and NAT/firewall services. An S3 gateway endpoint will also suffice if you are on AWS, since the image comes from a regional S3 bucket hosted by Databricks. Not sure if Azure functions the same way. Sorry for the info dump. Stream of consciousness from my phone. It’s an awesome platform with tons of capabilities but also a hell of a lot of out of box complexity if you aren’t sure where to start.