Well-Architected Lakehouse · 536tech Databricks

Databricks organizes lakehouse guidance into seven pillars. Five match the Azure Well-Architected Framework: operational excellence, security, reliability, performance efficiency, and cost optimization. Two are specific to a lakehouse: data governance, and interoperability and usability. Use this file to review an existing Azure Databricks environment, or to set the target state before a build.

This file is generated from the seven Microsoft Learn pillar pages. Do not edit it by hand. Regenerate it with uv run refresh.py waf.

Operational excellence#

1. Optimize build and release processes#

Create a dedicated Databricks operations team — A central platform team writes the blueprints, supplies the automation and self-service access, and owns security and compliance. Distributed data teams then work only with data. Why it matters: When no team owns the platform, each project rebuilds its own workspace standard, and a review finds five different cluster patterns.
Use enterprise source code management (SCM) — Track changes, keep code integrity, find bugs, and roll back. Databricks Git folders hold notebooks and files in a Git repository, with clone, commit, push, pull, branch management, and diffs. Why it matters: Notebooks edited only in the workspace have no history and no review, which is the first finding in most maturity assessments.
Standardize DevOps processes (CI/CD) — Build, test, and deploy in short automated cycles. Databricks Asset Bundles drive deployment from GitHub Actions, Azure DevOps pipelines, or Jenkins. Why it matters: It converts "who deployed this?" into a pipeline run, which is what an audit asks for. See terraform-vs-dabs.md.
Standardize MLOps processes — MLOps manages models, data, and code across data preparation, EDA, feature engineering, training, validation, deployment, and monitoring. Keep business goals first, use MLflow as the open tool, and build in modules. Why it matters: Modular ML code is testable, so the client can change a training step without a full rewrite.
Define environment isolation strategy — The workspace is the primary processing environment. Split workspaces to separate business units, to separate development, staging, and production, and to escape subscription and platform resource limits. Why it matters: Isolation costs money — notebook collaboration stops at the workspace edge, and per-workspace network controls get expensive. Set the balance early with the client.
Define catalog strategy for your enterprise — Structure metadata in the Unity Catalog three-level namespace catalog > schema > tables/views/volumes. Set storage locations at the metastore, catalog, or schema level, and bind catalogs to specific workspaces. Why it matters: Catalog-workspace bindings keep production data out of a development workspace, whatever the object grants say. See unity-catalog-grants.md.

2. Automate deployments and workloads#

Use infrastructure as code (IaC) for deployments and maintenance — The Databricks Terraform provider manages workspaces, clusters, jobs, and data access. It targets full coverage of the Databricks REST API. Why it matters: Terraform is the recommended tool for workspace provisioning and data access configuration. Compare it against bundles in terraform-vs-dabs.md.
Standardize compute configurations — Use IaC for setup, and compute policies for ongoing control. Workspace admins limit compute creation per user or group, enforce Spark configuration settings, and enforce cluster-scoped library installation. Why it matters: T-shirt size policies (S, M, L) give the client one decision instead of twenty cluster fields.
Use Unity Catalog managed tables — Managed tables give Databricks control of the data lifecycle, the storage location, and the file layout, and enable predictive optimization. External tables put maintenance and optimization back on you. Why it matters: Managed tables remove the hand-written OPTIMIZE and VACUUM jobs that fill most legacy workspaces.
Use automated workflows for jobs — Lakeflow Jobs orchestrates tasks with dependencies and handles cluster management, monitoring, and error reporting. Lakeflow pipelines extend Apache Spark Declarative Pipelines for declarative ETL. External orchestrators such as Apache Airflow and Azure Data Factory drive Databricks through the REST API. Why it matters: Use Lakeflow Jobs for task dependencies inside Databricks, then call the whole job from the enterprise orchestrator. Do not rebuild the DAG in Azure Data Factory.
Use automated and event-driven file ingestion — Auto Loader processes new files in cloud storage as they arrive, across JSON, CSV, PARQUET, AVRO, ORC, TEXT, and BINARYFILE. Use COPY INTO for one-off ingestion. Why it matters: A job that runs on an event, not a schedule, does not burn compute on an empty folder.
Use ETL frameworks for data pipelines — Lakeflow pipelines take the transformation definition in SQL or Python, then plan execution, set up infrastructure, and handle orchestration, monitoring, data quality, and errors. Why it matters: The framework supplies the error handling and quality checks the client would otherwise hand-code per pipeline.
Follow the deploy-code approach for ML workloads — Move code from development to staging to production and run training there, instead of moving a trained model artifact. Why it matters: Only production needs read access to production training data, and the whole flow uses Git and CI/CD the client already runs.
Use a model registry to decouple code and model lifecycle — Models in Unity Catalog host the MLflow Model Registry, with centralized access control, auditing, lineage, and cross- workspace discovery. It stays compatible with the open source MLflow Python client. Why it matters: Model lifecycles do not match code lifecycles, so a shared registry is the only place both meet.
Automate ML experiment tracking — Databricks Autologging extends MLflow automatic logging and captures parameters, metrics, files, and lineage with no code. Why it matters: Consistent metadata across every run is what makes a result reproducible six months later.
Reuse the same infrastructure to manage ML pipelines — Deploy ML environments with the Databricks Terraform provider, run ML pipelines as jobs, ingest with Auto Loader, compute features with Lakeflow pipelines, and serve with Model Serving. Why it matters: One infrastructure pattern for data and ML halves what the platform team must support.
Utilize declarative management for complex data and ML projects — Databricks Asset Bundles describe a project in one declarative YAML syntax and add CI/CD to the development workflow. Why it matters: Templated bundle projects spread the standard without a review of every repo.

3. Manage capacity and quotas#

Manage service limits and quotas — Three limit types apply: Databricks platform resource limits, Unity Catalog resource quotas, and Azure subscription quotas such as VM-family vCPU limits. Why it matters: A vCPU quota, not the design, is what stops the first large production run. Raise it before go-live.
Invest in capacity planning — Plan for load variation, test unexpected variation, and make sure each region can carry the total load if another region fails. Weigh technology limits, SLAs, and cost against the improvement bought. Why it matters: If resources run out during a high-volume event, the workload does not slow down — it fails.

4. Set up monitoring, alerting, and logging#

Establish monitoring processes — Monitoring finds data quality problems, performance bottlenecks, and system failures early, and supplies audit trails of data access and usage. Why it matters: The audit trail requirement is usually already in the client's compliance policy, unsigned by anyone on the data team.
Platform monitoring using Azure monitoring solutions — Azure Databricks sends custom application metrics, streaming query events, and application log messages to Azure logging services. Why it matters: It puts Databricks signals in the same place as the rest of the client's Azure estate.
Databricks Data Quality Monitoring — Tracks metrics for every table in the account, detects anomalies, and runs at the update frequency of the table. Results land in system tables, and per-table data profiling tracks statistical properties over time. Why it matters: Alerts keyed to anomaly severity and downstream lineage beat a hand-written row-count check.
System tables for monitoring — System tables carry billing and usage data, cluster and warehouse events, job and task run history, pipeline events, and audit logs across the account. Why it matters: This is the reporting layer for cost, access review, and job SLA in one place. See auditing-and-system-tables.md.
SQL warehouse monitoring — Shows the load profile over time, including queries handled and clusters allocated to the warehouse. Why it matters: It tells you whether a warehouse is undersized or simply left running.
Databricks SQL alerts — Run a query on a schedule, evaluate a condition, and notify. Alerts can read a monitor metrics table to fire on drift against a baseline table. Why it matters: Business-condition alerts reach the data owner, not only the platform team.
Auto Loader monitoring — A SQL API returns metadata about files an Auto Loader stream has discovered. The Apache Spark Streaming Query Listener interface adds deeper monitoring. Why it matters: "Did the file arrive or did we skip it?" is answerable without opening the storage account.
Job monitoring — Surfaces failures, delays, and bottlenecks in Lakeflow Jobs. Configure alerts on failure, delay, or success, send webhooks to PagerDuty or Opsgenie, and query system.workflow.job_runs and system.workflow.task_runs for trends and SLA compliance. Why it matters: Those two system tables answer "how often does this pipeline miss?" with data rather than opinion.
Lakeflow pipelines monitoring — Every pipeline keeps an event log with audit records, data quality checks, progress, and lineage. Why it matters: One log holds the quality result and the run history, so root cause analysis stays in one query.
Streaming monitoring — Monitor Structured Streaming queries for low-latency ingestion and analysis. Why it matters: Streaming failures are silent until the downstream table goes stale.
Spark monitoring — Enable cluster log delivery to persist Spark event logs to cloud storage, inspect stages, tasks, storage, and executors in the Spark UI, and export Spark metrics to an external system. Why it matters: Without persisted event logs, the evidence of a slow stage disappears when the cluster terminates.
ML and AI monitoring — Inference tables log serving request inputs and responses from Model Serving endpoints into a Delta table in Unity Catalog. Why it matters: Model quality debate ends when the requests and predictions are queryable.
Security monitoring — See the security pillar, "Monitor system security". Why it matters: Security signals belong in the same observability plan as job failures.
Cost monitoring — See the cost optimization pillar, "Monitor and control cost". Why it matters: Cost is an operational metric, not a finance report.

Security, compliance, and privacy#

The named practices below come from the Databricks Security Best Practices guide on the Security & Trust Center. The Learn page lists the practices; the linked PDF holds the detail. Platform-level lessons from real environments are in ../lessons-learned/platform-lessons.md.

1. Manage identity and access using least privilege#

Account setup and identity configuration

Assign account admin roles to 2-3 trusted individuals only — Keep the account admin group to two or three people. Why it matters: Account admin is the widest role in the platform, and a long admin list is an immediate audit finding.
Configure SSO with Azure Active Directory (Azure AD) using OIDC or SAML — Federate workspace sign-in to the corporate identity provider. Why it matters: Joiner-mover-leaver control only works when Databricks reads from the same directory as everything else.
Enable automatic identity management or SCIM provisioning — Synchronize users and groups from Azure AD instead of creating them in Databricks. Why it matters: Manual groups drift, and stale members keep grants they should have lost. See identity-entra-scim.md conventions in this knowledge base.
Set up identity federation — Link corporate identities across workspaces. Why it matters: One identity across workspaces makes a cross-workspace access review possible.
Configure multifactor authentication at the Azure AD level — Enforce MFA in the identity provider, not in the platform. Why it matters: MFA set at Azure AD covers every Databricks entry point at once.
Define emergency access procedures for account recovery — Document how to recover the account when normal sign-in fails. Why it matters: An SSO misconfiguration can lock out every admin, and the break-glass path must exist before that day.

Identity and access management best practices

Leverage multi-factor authentication — Require a second factor for platform access. Why it matters: It is the single control that blocks a stolen password.
Use automatic identity management to synchronize users and groups — Keep principals in step with the directory. Why it matters: Grants stay meaningful only if group membership is accurate.
Limit the number of admin users — Keep admin counts low at every level. Why it matters: Fewer admins mean fewer paths to an unreviewed change.
Enforce segregation of duties between administrative accounts — Separate administrative roles so one account cannot both grant and use access. Why it matters: Regulators ask for this by name.
Restrict workspace admins — Limit who holds the workspace admin role. Why it matters: Workspace admin can reconfigure the environment your controls depend on.
Manage access according to the principle of least privilege — Grant the minimum privilege that the task needs. Why it matters: It is the baseline every other access control assumes. See unity-catalog-grants.md.
Use OAuth or Entra ID token authentication — Authenticate integrations with OAuth or Entra ID tokens. Why it matters: Token-based identity beats shared credentials in a pipeline.
Enforce token management — Control who can create personal access tokens and how long they live. Why it matters: An unmanaged personal access token outlives the employee who created it.
Restrict cluster creation rights — Do not give every user the compute creation privilege. Why it matters: Free cluster creation is both a security hole and the top cost overrun.
Use compute policies — Constrain compute configuration through policies. Why it matters: A policy enforces the security settings a documented standard only requests.
Use service principals to run administrative tasks and production workloads — Run production and admin automation as a service principal, not as a person. Why it matters: Production stops depending on one employee's account staying active.
Use compute that supports user isolation — Choose compute modes that isolate users from each other. Why it matters: Shared compute without isolation lets one user reach another user's credentials.
Store and use secrets securely — Keep credentials in a secret store, not in notebook code. Why it matters: Hard-coded secrets end up in Git history and in an incident report.

2. Protect data in transit and at rest#

Centralise data governance with Unity Catalog — Put all data and AI assets under one governance layer. Why it matters: Controls outside Unity Catalog are the ones nobody reviews.
Use Azure Managed Identities to access storage — Authenticate storage access with a managed identity. Why it matters: No stored key means no key to rotate or leak.
Plan your data isolation model — Decide which data lives where, logically and physically, before the build. Why it matters: Retrofitting isolation means moving data and rewriting every path.
Avoid storing production data in DBFS — Keep production data out of DBFS. Why it matters: DBFS lacks the access controls the rest of the environment relies on.
Configure Azure Storage firewalls — Restrict which networks may reach the storage account. Why it matters: The lakehouse perimeter is only as strong as the storage account behind it.
Prevent anonymous read access and apply other protections — Turn off anonymous access on storage. Why it matters: A public container makes every other control irrelevant.
Enable soft deletes and other data protection features — Use the storage account's data protection features. Why it matters: Soft delete is what turns an accidental delete into a support ticket.
Backup your Azure Storage data — Take backups of the underlying storage. Why it matters: Delta time travel does not survive a deleted container.
Configure customer-managed keys for managed services — Supply your own key for managed services. Why it matters: Many regulated clients cannot sign off without key ownership.
Configure customer-managed keys for storage — Supply your own key for workspace storage. Why it matters: Same requirement, different scope. Check both.
Use OpenSharing — Share data through the OpenSharing protocol instead of copying files. Why it matters: Governed sharing keeps an audit trail that an emailed extract does not.
Configure an OpenSharing recipient token lifetime — Set an expiry on recipient tokens. Why it matters: A share with no expiry becomes permanent access to a partner.
Additionally encrypt sensitive data at rest using Advanced Encryption Standard (AES) — Add an application-level AES layer for sensitive fields. Why it matters: It protects the field even from a reader who holds storage access.
Leverage data exfiltration prevention settings within the workspace — Turn on the workspace settings that block data from leaving. Why it matters: Most data loss is an authorized user with an unmonitored egress path.
Use Clean Rooms to collaborate in a privacy-safe environment — Run joint analysis without exposing raw data to the other party. Why it matters: It is the answer when a client wants partner analytics but cannot share records.

3. Secure your network and protect endpoints#

Network deployment considerations for Azure

Create a VNet with minimum /18 address space — Size the VNet at /18 or larger for workspace deployments. Why it matters: An undersized VNet caps cluster scale, and you cannot resize it after deployment.
Provision public and private subnets delegated to Microsoft.Databricks/workspaces — Both subnets need the delegation. Why it matters: Missing delegation fails the deployment, and the error is not obvious.
Configure Network Security Groups (NSGs) — Control traffic with NSGs; Databricks manages the default NSG rules. Why it matters: Editing Databricks-managed rules breaks the workspace. Add, do not replace.
Deploy Azure Private Link — Give private connectivity to the Databricks control plane and web application. Why it matters: Private Link removes the public path that a network review will flag.
Enable Secure Cluster Connectivity (No Public IP) — Remove public IP addresses from cluster nodes. Why it matters: It is the default expectation for any regulated Azure workload.
Configure VPN Gateway or ExpressRoute for on-premises connectivity — Add a private route to on-premises systems where needed. Why it matters: Hybrid sources are common, and the network path is usually the long-lead item.
Implement network segmentation — Separate production and non-production networks. Why it matters: Segmentation is what stops a development mistake reaching production data.

Network security best practices

Use Secure Cluster Connectivity (No Public IP) — Run clusters with no public IP. Why it matters: See above; confirm it on every existing workspace during a review.
Deploy Azure Databricks into your own Azure virtual network — Use VNet injection instead of the managed VNet. Why it matters: Only a customer VNet lets you apply the client's own network policy.
Configure IP access lists — Restrict which source addresses may reach the workspace. Why it matters: It closes the workspace to the open internet even when credentials leak.
Use Azure PrivateLink — Keep workspace and control plane traffic on the Azure backbone. Why it matters: Pair it with IP access lists; neither is sufficient alone.
Implement network exfiltration protections — Control egress from the workspace network. Why it matters: Egress control is where data loss prevention actually happens.
Isolate Azure Databricks workspaces into different networks — Give sensitive workspaces their own network. Why it matters: Network isolation costs money, so size it against the actual risk.
Configure a firewall for serverless compute access — Apply firewall rules to serverless compute egress. Why it matters: Serverless does not inherit the classic compute network path. Check it separately.
Restrict access to valuable codebases to only trusted networks — Limit repository access by network. Why it matters: Source code carries the same business value as the data.
Use virtual network encryption — Encrypt traffic between VMs in the VNet. Why it matters: It answers the in-transit question for traffic inside Azure.

4. Meet compliance and data privacy requirements#

Restart compute on a regular schedule — Recycle long-running compute. Why it matters: Restarts pick up runtime patches; a cluster running for months does not.
Isolate sensitive workloads into different workspaces — Put regulated workloads in their own workspace. Why it matters: A separate workspace lets you apply stricter settings without slowing everyone else.
Assign Unity Catalog securables to specific workspaces — Bind catalogs and other securables to the workspaces allowed to use them. Why it matters: Binding enforces data residency and environment rules above the grant layer.
Implement fine-grained access controls — Use row filters, column masks, and attribute-based rules. Why it matters: This is where PII policy becomes enforcement. See governance-pii-abac.md.
Apply tags — Tag securable objects in Unity Catalog. Why it matters: Tags drive both discovery and attribute-based policy, so a tag scheme is a prerequisite, not a nicety.
Use lineage — Keep column-level lineage across queries. Why it matters: Lineage answers the GDPR and SOX question "where else does this field go?".
Use Enhanced Security Monitoring or Compliance Security Profile — Turn on the hardened workspace profiles. Why it matters: Some compliance frameworks require the profile, not just the equivalent controls.
Control and monitor workspace access for Azure Databricks personnel — Govern vendor access to the workspace. Why it matters: Clients ask who at Databricks can see their data. Have the answer.
Implement and test a Disaster Recovery strategy — Build a DR plan and exercise it. Why it matters: An untested DR plan is a document, not a capability.
Consider the use of Azure Confidential Compute — Use confidential compute VMs for the most sensitive workloads. Why it matters: It covers data in use, which the other controls do not.

5. Monitor system security#

Leverage system tables — Query account-wide activity from system tables. Why it matters: System tables are the evidence source for access reviews. See auditing-and-system-tables.md.
Monitor system activities via Azure logs — Send platform activity to Azure logging. Why it matters: The client's SOC watches Azure logs, not the Databricks UI.
Enable verbose audit logging — Record an audit entry for every query or command run in the workspace. Why it matters: Standard audit logs do not show what a user actually ran.
Manage code versions with Git folders — Keep workspace code under Git. Why it matters: Version control is a security control, not only a developer convenience.
Restrict usage to trusted code repositories — Allow only approved Git remotes. Why it matters: It stops production code arriving from an unreviewed source.
Provision infrastructure via infrastructure-as-code — Deploy through Terraform or bundles. Why it matters: A reviewable, repeatable deployment is also a recoverable one. See terraform-vs-dabs.md.
Manage code via CI/CD — Deploy code through a pipeline. Why it matters: The pipeline is where tests, scans, and approvals attach.
Control library installation — Restrict which libraries can be installed. Why it matters: An arbitrary PyPI install is an unreviewed supply chain entry.
Use models and data from only trusted or reputable sources — Vet external models and datasets. Why it matters: A downloaded model is executable code with the cluster's permissions.
Implement DevSecOps processes — Put security checks in the delivery pipeline. Why it matters: Controls that run automatically are the only ones that keep running.
Use data quality monitoring — Watch quality metrics for anomalies. Why it matters: An unexplained distribution shift can be a pipeline bug or a data incident.
Use inference tables and AI Guardrails — Log serving requests and responses, and apply guardrails. Why it matters: Model endpoints are a public surface, and the request log is the only record.
Use tagging as part of your cost monitoring and charge-back strategy — Tag for attribution. Why it matters: Unattributed spend is also unattributed activity.
Use budgets to monitor account spending — Set budgets on the account. Why it matters: A cost spike is often the first visible sign of misuse.
Use Azure Policy to create "upper limit" resource controls — Cap resources with Azure Policy. Why it matters: Azure Policy holds even when a Databricks-level control is changed.

Reliability#

1. Design for failure#

Use a data format that supports ACID transactions — Delta Lake gives ACID transactions, schema enforcement, scalable metadata handling, and one copy of data for both batch and streaming. Why it matters: Without ACID, a failed write leaves a partial table, and every downstream job inherits the corruption.
Use a resilient distributed data engine for all workloads — Apache Spark reschedules failed tasks automatically, through both the SQL API and the DataFrame API. Photon is a native vectorized C++ engine compatible with Spark APIs. Why it matters: Task-level retry absorbs a brief network problem or a revoked Spot VM with no code.
Automatically rescue invalid or nonconforming data — Auto Loader supports rescued data for JSON and CSV, and returns the _rescued_data column by default under schema inference. Lakeflow pipelines support retain, drop, and fail modes, and expectation rules can quarantine bad records in another table. Why it matters: A single malformed record should not stop the pipeline, and rescued data means nothing is lost while it is investigated.
Configure jobs for automatic retries and termination — Lakeflow Jobs support a retry policy, an expected completion time, and a maximum completion time per task. Lakeflow pipelines use escalating retries. Why it matters: The timeout matters as much as the retry — a hanging task holds compute and bills for it.
Use scalable and production-grade model serving infrastructure — For batch and streaming inference, deploy models as Spark UDFs through Lakeflow Jobs and MLflow. Model Serving exposes models as REST endpoints on serverless compute. Why it matters: Endpoint scaling, retries, and compute management stop being the client's problem.
Use managed services where possible — Serverless SQL warehouses, Model Serving, serverless jobs, serverless compute for notebooks, and Lakeflow pipelines are run by Databricks. Why it matters: Every managed service is one less failure mode the platform team must operate.

2. Manage data quality#

Use a layered storage architecture — The medallion pattern raises quality through three layers. Bronze holds ingested raw data in external tables so raw data survives a dropped table. Silver holds cleansed and filtered data in managed tables with schema enforcement and quality checks. Gold holds business data products in managed tables with business logic validation and SLA guarantees. Why it matters: Because bronze persists, every later layer can be rebuilt. That is the recovery plan for a logic bug.
Improve data integrity by reducing data redundancy — A throwaway copy is fine; a copy that becomes operational is a data silo with its own permissions and its own lineage gap. Why it matters: When two datasets disagree, the business stops trusting both.
Actively manage schemas — Delta Lake enforces and validates schema on write. Auto Loader stops a stream with UnknownFieldException when a new column appears, and supports several schema evolution modes. Why it matters: The failure is deliberate. Choose the evolution mode per source instead of turning enforcement off.
Use constraints and data expectations — Delta tables support SQL constraint clauses and raise InvariantViolationException on a violation. Lakeflow pipeline expectations pair a description, an invariant, and an action, through Python decorators or SQL constraint clauses. Why it matters: The rule lives with the table, so it applies to every writer, not only the pipeline that defined it.
Take a data-centric approach to machine learning — Feature engineering, training, deployment, inference, and monitoring are all data pipelines. Run them on the platform that manages the production data. Why it matters: Splitting ML off onto another platform recreates the data silo problem the lakehouse just solved.

Implementation considerations — Organize bronze, silver, and gold as Unity Catalog schemas (sales.bronze_transactions, sales.silver_transactions, sales.gold_metrics). Set delta.enableChangeDataFeed to track changes between layers, enable auto optimization for file sizes, process incrementally with Delta Live Tables or Structured Streaming, and set explicit quality expectations at each layer boundary.

3. Design for autoscaling#

Enable autoscaling for ETL workloads — Clusters resize to the workload. For streaming, use Lakeflow pipelines with Databricks enhanced autoscaling, which allocates on workload volume with minimal latency impact. Why it matters: Classic autoscaling does not scale streaming down well. Enhanced autoscaling is the reason to move a stream to Lakeflow pipelines.
Enable autoscaling for SQL warehouse — The scaling parameter sets the minimum and maximum cluster count for a warehouse. The default is one cluster with no autoscaling. Why it matters: The default cannot absorb concurrent users, and the symptom the client reports is "the dashboard is slow".

4. Test recovery procedures#

Recover from Structured Streaming query failures — Checkpointing lets a restarted query continue where the failed query stopped. Configure automatic restart through Lakeflow Jobs. Why it matters: Without a checkpoint there is no restart, only a reprocess.
Recover ETL jobs using data time travel capabilities — Delta time travel rolls a table back to an older version or timestamp. The RESTORE command does it in one statement. Why it matters: It converts a bad production run from an outage into a rollback.
Leverage a job automation framework with built-in recovery — Lakeflow Jobs show a matrix view of runs, and a repair run reruns only the failed and dependent tasks while keeping earlier successful results. Why it matters: A repair run saves both the reprocessing time and the compute bill.
Implement high availability strategies — Databricks provides a 99.9% SLA for the control plane, deployed across availability zones with no customer configuration. For compute, supply subnets in different zones and configure job retries. For storage, use ZRS on Azure. For network, spread subnets, NAT gateways, and VPN connections across zones. Why it matters: The control plane SLA is given; everything below it is the client's design decision.
Configure a disaster recovery pattern — Define RTO and RPO, rebuild workspaces with Terraform or Asset Bundles, replicate Unity Catalog metadata through metastore backup and import, replicate critical Delta tables with DEEP CLONE, configure cloud storage replication, and test the procedure regularly. Why it matters: Metastore replication is the step most DR plans miss. Without it the data arrives in the DR region ungoverned.

5. Automate deployments and workloads#

See "Operational excellence — Automate deployments and workloads".

6. Monitor systems and workloads#

See "Operational excellence — Set up monitoring, alerting, and logging".

Performance efficiency#

1. Vertical scaling, horizontal scaling, and linear scalability#

Vertical scaling — Add CPU, memory, or GPU to one machine. This usually means stopping the workload, moving it, and restarting. Why it matters: There is always a largest machine, and the next size up is often priced beyond the benefit.
Horizontal scaling — Add or remove nodes in a distributed system. Workloads must be parallel, which Apache Spark and Photon support. Why it matters: There is no hard node limit, but large node counts raise management complexity.
Linear scalability — Throughput rises in proportion to resources only when parallel tasks are independent. Data exchange between nodes costs network time and adds distribution overhead. Why it matters: A small dataset can run slower on a cluster than on one node. Check the data size before you recommend a cluster.

2. Use serverless architectures#

Use serverless compute — Serverless SQL warehouses, serverless jobs, and serverless compute for notebooks are fully managed. Cloud admins stop adjusting quotas and maintaining network resources, and users get near-zero cluster startup latency and better query concurrency. Why it matters: If serverless interactive compute is enabled for a workspace, all users in that workspace already have access with no extra permission. Confirm that is intended.
Use an enterprise grade model serving service — Model Serving deploys, governs, and queries models behind a REST API, and scales up and down automatically on serverless compute. Why it matters: It trades a fixed serving cluster for demand-shaped cost and latency.

3. Design workloads for performance#

Understand your data ingestion and access patterns — Large files suit scan queries; small files suit searches. DML performs best on clustered data, so keep a natural time sort order and push as many filters onto the target table as possible. Append-only and overwrite ingestion need little tuning. Why it matters: The access pattern usually dictates the layout. If it does not, pick the goal that matters to the business and optimize for it.
Use parallel computation where it is beneficial — Spark parallelizes SQL across all cluster nodes, and Python and Scala libraries do the same. Structured Streaming distributes streaming jobs. Pandas API on Spark scales pandas code, and MLlib, DeepSpeed Distributor, and TorchDistributor parallelize training. Why it matters: Pandas does not scale out, and Pandas API on Spark is the low-friction fix for a data science team.
Analyze the whole chain of execution — BI performance depends on the BI tool, the connector, and the SQL engine. Why it matters: Tuning the warehouse does nothing when the connector is the bottleneck.
Prefer larger clusters — With linear scaling, two workers for one hour costs the same as four workers for half an hour, and finishes sooner. An autoscaling cluster is usually cheapest where the SLA is flexible, but not fastest. This does not apply to serverless compute. Why it matters: "Use a smaller cluster to save money" is wrong for linear workloads, and this is the argument that shows why.
Use predictive optimization — Unity Catalog governs all reads and writes, so it knows the query patterns and can optimize layout, clean up old files, and update clustering automatically. Why it matters: Enable it at account, catalog, or schema level and retire the hand-built maintenance jobs.
Use Unity Catalog managed tables — Managed tables get automatic file layout management and predictive optimization enabled by default. Why it matters: External tables put layout and maintenance work back on the client's team.
Use native Spark operations — Do not write a Python or Scala UDF when a Spark SQL or PySpark native function exists. Where a Python UDF is unavoidable, use Pandas UDFs, which move data through Apache Arrow. Why it matters: Python-to-Spark serialization slows queries badly, and this is a common finding in a slow-pipeline review.
Use native platform engines — Photon speeds up ingestion, ETL, streaming, data science, and interactive queries, with no code change and no lock-in. It is on by default in Databricks SQL warehouses. Why it matters: It is a switch, not a project. Check whether job clusters have it on.
Understand your hardware and workload type — VM families differ in RAM, cores, processor type and generation, network bandwidth guarantees, and local high-speed storage versus local or remote disk. Spot markets differ too. Serverless compute does not need this. Why it matters: Instance choice is a real lever, and most clients picked a default years ago.
Use disk cache — The disk cache (formerly "Delta cache") keeps copies of remote data on local SSDs and updates itself when files change. Choose a worker type with SSD volumes and it is enabled and configured. Why it matters: Picking the right worker type is the entire configuration step.
Avoid Spark caching — .persist() and .unpersist() can cache any subquery and non-Parquet formats, but wrong placement consumes all memory and slows queries. Why it matters: The rule of thumb is to avoid it. Look for it in inherited notebooks.
Query result cache — Per-cluster caching of results for queries through SQL warehouses. Deterministic queries benefit; predicates such as = NOW() do not. Why it matters: One non-deterministic predicate disables the cache for a whole dashboard.
Databricks SQL UI caching — Per-user caching of query results in the Databricks SQL UI. Why it matters: It explains why one user sees an instant result and another waits.
Use compaction — OPTIMIZE coalesces small files. Auto compact merges small files inside Delta partitions after a successful write, synchronously on the writing cluster, and only for files not compacted before. Optimized writes improve file size at write time and help most on partitioned tables. Why it matters: Small files are the most common cause of a slow table, and both fixes are table properties.
Use data skipping — Delta Lake collects statistics on the first 32 columns of the schema by default and uses minimum and maximum values to skip files. Use liquid clustering, which lets you redefine clustering keys without rewriting data. It suits tables filtered on high-cardinality columns, with skewed distribution, growing fast, with concurrent writes, with changing access patterns, or where a partition key gives too many or too few partitions. Why it matters: The 32-column default means a filter column placed late in a wide schema gets no statistics at all.
Avoid over-partitioning — Do not partition tables below 1 TB, and only partition on a column when each partition holds at least 1 GB. Use liquid clustering instead of partitioning for new Delta tables. Why it matters: These two numbers settle most partitioning arguments on the spot.
Optimize join performance — Range join optimization can improve interval and overlap joins by orders of magnitude but needs careful manual tuning. Adaptive query execution (AQE) re-optimizes at run time: it converts sort merge join to broadcast hash join, coalesces partitions after a shuffle exchange, handles skew in sort merge and shuffle hash joins, and detects and propagates empty relations. Keep AQE enabled. Why it matters: AQE handles skew that would otherwise need a hand-written salting fix.
Run analyze table to collect table statistics — ANALYZE TABLE gathers statistics that the cost-based optimizer uses to choose join type, hash-join build side, and join order. Predictive optimization runs ANALYZE automatically on Unity Catalog managed tables (Public Preview). Why it matters: A bad multi-way join order is usually missing statistics, not a bad query.

4. Run performance testing in the scope of development#

Test on data representative of production data — Test on production data read-only, or on data with similar volume, file layout, and skew. Why it matters: A test on clean sample data proves nothing about a skewed production table.
Consider prewarming resources — The first query on a cluster is always slower. Databricks pools hold idle, ready-to-use instances and cut cluster start and autoscaling times. Prewarm caches by running specific queries after a restart. Test first execution with and without prewarming, and subsequent executions too. Why it matters: A benchmark that ignores prewarming produces a number nobody can reproduce.
Identify bottlenecks — Find the areas that will degrade as production load rises, at design time, by testing above expected load. Why it matters: A bottleneck found in design is a config change; found in production it is an incident.

5. Monitor performance#

Configure performance monitoring at deployment — Enable system tables and query system.compute for cluster utilization, system.workflow for job performance, and system.query for SQL warehouse query metrics. Deliver cluster logs to persist Spark event logs for historical analysis of long stages, skew, excessive shuffles, and memory pressure. Monitor jobs and pipelines for execution time and failures, and alert on SLA violations. Enable query history and use query profiles. Why it matters: A baseline captured at deployment is what later makes "it got slower" measurable. See auditing-and-system-tables.md.
Monitor query performance — The query profile visualizes each query task with time spent, rows processed, and memory used. SQL warehouse monitoring shows live statistics, peak query count, running clusters, and query history. Why it matters: The query profile names the expensive stage, which shortens most tuning engagements to one session.
Monitor streaming workloads — Use built-in Structured Streaming monitoring in the Spark UI, or push metrics out with the Streaming Query Listener interface. Why it matters: Streaming latency drifts slowly, and only a trend line shows it.
Monitor job performance — Job monitoring surfaces failures, delays, and bottlenecks in Lakeflow Jobs. Why it matters: It is where wasted compute in a scheduled workload becomes visible.

Cost optimization#

1. Choose optimal resources#

Use performance optimized data formats — Use Delta Lake as the storage framework. It builds simpler ETL pipelines and runs faster than Parquet, ORC, and JSON. Why it matters: On job compute, a faster run is a shorter compute uptime, so format choice is a direct cost lever.
Use job compute — Non-interactive workloads on job compute cost significantly less than on all-purpose compute. Each job can run on a new compute instance for isolation, or a multitask job can reuse compute so startup happens once. Why it matters: Scheduled ETL left on all-purpose compute is the most common overspend, and the fix is a job configuration change.
Use SQL warehouse for SQL workloads — A Databricks SQL warehouse is the most cost-efficient engine for interactive SQL. All SQL warehouses include Photon. Serverless SQL warehouses add intelligent workload management (IWM). Why it matters: Analysts running SQL on an all-purpose cluster pay more and wait longer.
Use up-to-date runtimes for your workloads — Databricks Runtime and Databricks Runtime for Machine Learning ship regularly with performance improvements between major releases. Why it matters: Runtime upgrades cut cost through efficiency, and pinned old runtimes are common in environments nobody maintains.
Only use GPUs for the right workloads — GPU VMs speed up deep learning but cost far more. Most workloads use no GPU-accelerated libraries. Workspace admins can restrict GPU machines. Why it matters: An unrestricted GPU instance type in a policy is a large invoice waiting to happen.
Use serverless services for your workloads — BI workloads arrive in bursts. Non-serverless SQL warehouses take minutes to start, so users leave them running; serverless warehouses start and scale in seconds and scale down earlier. Model Serving uses serverless compute and scales to demand. Why it matters: Serverless makes idle termination acceptable to users, which is where the saving actually comes from.
Use the right instance type — Use the latest instance generation, then match the family to the workload: memory optimized for ML, heavy shuffle, and spill workloads; compute optimized for structured streaming and maintenance jobs such as optimize and vacuum; storage optimized for cache-friendly ad-hoc and interactive analysis; GPU optimized for specific ML and DL; general purpose when no specific requirement applies. Why it matters: This list settles instance-family questions in a design review in one minute.
Choose the most efficient compute size — Databricks runs one executor per worker node. Size on total executor cores (maximum parallelism), total executor memory (data held before spill), and executor local storage (shuffle spill and caching). Ask how much data the workload consumes, how complex it is, where it reads from, how the source is partitioned, and how much parallelism it needs. Why it matters: "How many workers?" is the wrong first question, and these five are the right ones.
Right-size compute resources at deployment — Development and test: single node or 2-4 workers with autoscaling. Batch ETL: 8-16 workers, memory-optimized, autoscaling on. Streaming: 4-8 workers with autoscaling for variable throughput. Machine learning: GPU instances sized to model and data volume. Size SQL warehouses on concurrent users and query complexity, start at Small or Medium with autoscaling, and prefer serverless. Use classic compute policies to enforce sizing, block expensive instance types, require autoscaling, cap worker counts, and define T-shirt sizes. Why it matters: These starting sizes give a defensible number on day one, before real telemetry exists.
Evaluate performance-optimized query engines — Photon is a vectorized engine for SQL and DataFrame API calls, with no code change and no lock-in. Why it matters: Regularly scheduled jobs deserve an explicit test — faster does not always mean cheaper, so measure both.

2. Dynamically allocate resources#

Use auto-scaling compute — Databricks adds workers during intensive pipeline phases and removes them afterwards. Compute autoscaling scales down poorly for structured streaming, so use Lakeflow pipelines with enhanced autoscaling for streaming. Why it matters: A statically sized cluster pays for the peak all day.
Use auto termination — Configure automatic termination on all interactive compute. Where compute is needed only in business hours, add a scheduled restart and prewarm with CACHE SELECT. Where startup is too slow, use cluster pools of idle instances. Databricks does not charge DBUs for idle pool instances, though the instance provider still bills. Why it matters: The pool DBU exemption is the detail that makes prewarming affordable. Say it out loud in the cost conversation.
Use compute policies to control costs — Policies can require autoscaling with a set minimum worker count, require auto termination at a reasonable value such as 1 hour, allow only cost-efficient VM instance types, and apply a spot instance strategy. Why it matters: A policy enforces the cost standard on every cluster created after it, without another training session.

3. Monitor and control cost#

Setup tagging for cost attribution — Tag workspaces, clusters, SQL warehouses, and pools. Beyond the Databricks default tags, set custom tags Business Units and Projects as a minimum, and add Environment to separate development, QA, and production. Tags propagate to usage logs and to cloud provider resources. Total cost is DBUs plus VM, disk, and network; for serverless, the DBU price already includes the VM. Why it matters: Tags only affect future usage. Missing tags cannot be added to past events, so start detailed and drop what proves useless.
Set up budgets and alerts to enable monitoring of account spending — Budgets track account-wide spend or filter to a team, project, or workspace. Use usage policies to attribute serverless usage. Set email notification when the monthly budget is reached. Why it matters: Serverless spend is invisible to tag-based attribution without usage policies.
Monitor costs to align spending with expectations — Import the cost management AI/BI dashboards from the account console into any Unity Catalog-enabled workspace. Watch OpenSharing egress charges when sharing across clouds or regions. Query system.billing.usage, which carries the custom tags, plus the serverless, job, and model serving cost tables. Azure Cost Analysis reads the Azure Databricks resource tags. Why it matters: One tag scheme feeding both system.billing.usage and Azure Cost Analysis lets finance and the platform team agree on a number.
Manage costs to align usage with organizational needs — Schedule a housekeeping job to apply and clean up tags incrementally, resilient to single-resource failures, writing all changes to an audit log. Run regular cost audits of active resources, share monthly cost reports, apply autoscaling and auto termination, educate teams on cost implications, and use compute policies to control what each group can create. Why it matters: Cost optimization is an ongoing process. Revisit it at every scaling event, new project, or unexplained spike.

4. Design cost-effective workloads#

Balance always-on and triggered streaming — Continuous streams need compute 24 hours a day. Where the business needs fresh data only every few hours, run Structured Streaming with the AvailableNow trigger for incremental batch processing. Why it matters: Asking "how fresh does this really need to be?" turns a 24/7 cluster into a few runs a day.
Balance between on-demand and capacity excess instances — Spot instances use excess cloud capacity at a lower price. Keep the first instance, the Spark driver, on an on-demand VM. Why it matters: A spot driver kills the whole job. This is the single rule to check in an existing spot configuration.

Data and AI governance#

1. Unify data and AI management#

Establish a data and AI governance process — Governance manages availability, usability, integrity, and security of data and AI assets. It supports privacy regulation compliance, reduces breach and penalty risk, and removes redundancy. Why it matters: Without a named process, Unity Catalog becomes a permissions tool instead of a governance program.
Design Unity Catalog for your organization — Pick a governance model: centralized, where governance admins own the metastore and can take ownership of any object, for strong central IT and strict compliance; distributed (federated), where each catalog is a data domain owned independently, for large organizations with autonomous business units; or hybrid, central for sensitive data and federated for operational data, which suits most enterprises. Deploy one metastore per cloud region and assign workspaces by data residency. Choose a catalog pattern: domain-based (recommended, such as sales, marketing, finance), environment-based, or data lifecycle-based. Use schemas for data products, such as sales.bronze_transactions. Why it matters: The catalog pattern decides who can grant access without a ticket. Choose it with the client's org chart on the table. See unity-catalog-grants.md.
Manage metadata for all data and AI assets in one place — Run Databricks in a single account with Unity Catalog. The metastore is the top-level container and holds tables, views, volumes, features, and AI models with their permissions. Use one metastore per cloud region and do not access metastores across regions, to avoid latency. Catalogs usually map to a development environment scope, a team, or a business unit. Why it matters: Cross-region metastore access is a latency problem that looks like a performance problem.
Track data and AI lineage to drive visibility of the data — Unity Catalog captures runtime lineage across queries, in all languages, down to the column level, including notebooks, jobs, and dashboards. Model lineage links a model to the datasets used to train and evaluate it. Lineage is visible in near real time in Catalog Explorer. Why it matters: Lineage covers four asks at once: compliance evidence for GDPR, CCPA, HIPAA, BCBS 239, and SOX; impact analysis before a change; quality context; and root cause analysis.
Add consistent descriptions to your metadata — Add comments to tables and columns, and tags (key with optional value) to any Unity Catalog securable. AI-generated comments speed the work, and Databricks strongly recommends human review before saving. Why it matters: Comments carry sensitivity and compliance context, and tags drive both search and attribute-based policy. See governance-pii-abac.md.
Allow easy data discovery for data consumers — Catalog Explorer explores and manages data, schemas, tables, permissions, owners, external locations, and credentials. The Insights tab shows the most frequent recent queries and users for any registered table. Why it matters: The Insights tab tells you which tables actually matter before you tune or redesign anything.
Govern AI assets together with data — Unity Catalog governs feature tables and models alongside data. Models in Unity Catalog add centralized access control, auditing, chronological lineage, versioning, and deployment by alias. Why it matters: Splitting AI governance off means two access reviews and two answers to the same compliance question.

2. Unify data and AI security#

Centralize access control for all data and AI assets — Unity Catalog centralizes control for all supported securables. Every securable has an owner, who holds all privileges and can grant them on. Manage privileges with SQL DDL. Row filters return only rows where the filter predicate is true; column masks substitute a masking function result at query run time. Why it matters: Row filters and column masks replace the shadow copies teams build to hide columns. See governance-pii-abac.md and unity-catalog-grants.md.
Configure audit logging — Databricks provides workspace-level audit logs for workspace events and account-level audit logs for account events. Verbose audit logs add a record for every query or command run in the workspace. Why it matters: Audit logs are for compliance and policy enforcement, not for debugging. Do not accept application logs as a substitute. See auditing-and-system-tables.md.
Audit data platform events — Unity Catalog records actions performed against the metastore, so admins can see who accessed a dataset and what they did. OpenSharing audit logs record share and recipient create, modify, update, and delete; activation link access and credential download; recipient access to shares and shared tables; and credential rotation or expiry. Why it matters: The OpenSharing event list is exactly what a partner data-sharing review asks for.

3. Establish data quality standards#

Define clear data quality standards — Base the standards on business need and document them across accuracy (data reflects real-world values), completeness (no critical data missing), consistency (no contradiction across systems), timeliness (updated and available in time), and reliability (dependable sourcing and processing). Why it matters: These five dimensions give a review a scoring frame instead of an opinion.
Use data quality tools for profiling, cleansing, validating, and monitoring data — Automate detection and correction so quality work scales to lake-sized datasets. In Lakeflow pipelines, use expectations to define constraints on dataset contents and to report quality per pipeline update. Why it matters: Expectations produce a quality metric per run, which is what an SLA needs.
Implement and enforce standardized data formats and definitions — Build and enforce a data dictionary with definitions, formats, and acceptable values for every data element. Use consistent naming conventions, date formats, and measurement units across all databases and applications. Why it matters: Most "the numbers do not match" escalations are a unit or date-format disagreement.

Implementation detail for these standards sits in the reliability pillar, "Manage data quality".

Interoperability and usability#

1. Define standards for integration#

Use standard and reusable integration patterns for external integration — The Databricks REST API runs in the control plane and manages nearly the whole platform. Prefer the higher abstractions: the Databricks SDKs, and the shell-based CLI tools for CI/CD and MLOps workflows. Why it matters: The REST API always works, but SDK and CLI code survives platform changes with less rework.
Use optimized connectors to ingest data sources into Databricks — Optimized connectors cover stream messaging services such as Apache Kafka. Built-in integrations cover many cloud-native data systems, with extensible JDBC support beyond them. Lakehouse Federation queries multiple sources with no migration, governed and audited through Unity Catalog. Why it matters: Every federated query is sent to the source system, and a cross-region or cross-cloud source charges egress per query. Offload through materialized views to protect the operational database and cut egress.
Use certified partner tools — Partner Connect creates trial accounts with Databricks technology partners and connects the workspace to partner solutions from the UI, across ingestion, preparation, BI, ML, and data quality. Why it matters: It shortens a tool evaluation to a working trial on the client's own data.
Reduce complexity of data engineering pipelines — Lakeflow pipelines take the transformation definition and handle orchestration, cluster management, monitoring, quality, and errors. Auto Loader reads new cloud files incrementally and reliably. Why it matters: Both are declarative, so the client stops maintaining a hand-built notification and queue service to detect new files.
Use infrastructure as code (IaC) for deployments and maintenance — Terraform builds safe and predictable cloud infrastructure across providers. Why it matters: See the operational excellence pillar and terraform-vs-dabs.md for the choice between Terraform and Asset Bundles.

2. Utilize open interfaces and open data formats#

Use open data formats — Delta Lake is a fully open format with ACID transactions and unified streaming and batch processing, supported by dozens of third-party tools. Delta tables can be read by Apache Iceberg reader clients, with one copy of the data files serving both formats. Why it matters: Iceberg read support answers the lock-in objection without a second copy of the data.
Enable secure data and AI sharing for all data assets — OpenSharing is an open protocol for sharing live data with any computing platform, with no replication and no need for the recipient to use Databricks or any cloud. Databricks-to-Databricks OpenSharing shares with recipients on a Unity Catalog-enabled workspace, across accounts, regions, and cloud providers. Shares can carry tables, views, volumes, models, and notebooks. Why it matters: Sharing without replication removes the copy that would otherwise fall out of governance, and out of date.
Use open standards for developing agents, LLMs, and ML models — MLflow is the open platform for agents, LLMs, and ML models. Databricks hosts a managed version with enterprise security, high availability, experiment and run management, and notebook revision tracking. It covers tracing, evaluation, prompt management, an AI Gateway for cost and model access, experiment tracking, a production model registry in Unity Catalog, and deployment tools. Why it matters: The AI Gateway is where AI cost and model access control belong. Raise it before the client builds their own.

3. Simplify new use case implementation#

Provide a self-service experience across the platform — Automate the whole setup when a business unit requests access, even where an approval step stays manual: provision the environment, synchronize users, use SSO, set access control on shared data, and give separate object stores for their own data. Why it matters: Self-service plus a central catalog of business-ready datasets is what lets the platform team stop processing access tickets.
Use serverless compute — Cloud admins stop managing quotas, network resources, and billing connections. Users get near-zero cluster startup latency and better query concurrency. Why it matters: It removes the platform team from the critical path of every new use case.
Use predefined compute templates — Offer shared clusters as immediate environments, with autoscaling down to a very small node count to avoid idle cost. Use compute policies for a standardized environment: restrict cluster size and features, or define T-shirt sizes (S, M, L). Why it matters: Templates simplify the UI and cap the maximum cost per cluster at the same time.
Use AI capabilities to increase productivity — Databricks IQ uses Unity Catalog metadata — tables, columns, descriptions, and popular assets — to give personalized answers. Genie Code generates SQL from an English description, explains complex code, and fixes errors. AI-generated comments speed up metadata work, and Databricks strongly recommends human review for inaccuracies. Why it matters: Databricks IQ is only as good as the catalog metadata, which is a direct reason to fund the comments and tags work.

4. Ensure data consistency and usability#

Offer reusable data-as-products that the business can trust — Apply product thinking to published assets: the right data, at the right time, with the right quality, in the right format. Why it matters: Data products create the standard and the trusted foundation that a data mesh or federated model depends on.
Publish data products semantically consistent across the enterprise — Source systems name the same concept differently (customer versus account) or reuse an identifier for different concepts. Make the data homogeneous, apply internal business rules such as revenue recognition correctly, publish the result to Unity Catalog, and restrict source data access to teams that understand its correct use. Why it matters: Restricting raw source access is the half of this that clients skip, and it is why two teams still report two revenue numbers.
Provide a central catalog for discovery and lineage — Unity Catalog manages users and access centrally across all workspaces in the account. Catalog Explorer shows schema details, sample data, table details and properties, and lets owners grant and revoke. Databricks Search returns only assets the user can access. Column-level lineage across every cluster and SQL warehouse query is visible in Catalog Explorer and retrievable through the REST API. Unity Catalog also integrates with enterprise data catalogs, the "catalog of catalogs". Why it matters: The enterprise catalog integration is the answer when the client already owns a catalog tool and does not want a second one.