Watch

Practitioner notes from r/dataengineering and r/databricks

Watch2026-08-1811 min read10 days old

Top/month posts from the combined subreddit RSS feed, filtered to tooling and platform signal. The feed preserves Reddit's ranking but does not expose scores or comment counts.

This is a dated reading list, not product guidance. Career, compensation, hiring, and event posts are excluded. Verify technical claims against official documentation.

Databricks and lakehouse#

  • Is Databricks and Fabric overtaking Snowflake? — r/dataengineering · 2026-07-24 Everywhere I have been hearing companies moving to Databricks or Fabric. 2-3 years ago all I heard of was Snowflake. Is Snowflake losing the race? A former employee at Snowflake told me that their work culture has gotten really bad and they cannot hold on to good employees and their stock keeps crashing.
  • Lost track of Databricks product renames? Here's a community tracker — r/databricks · 2026-07-20 I recently contributed to REbricked, a community project that tracks Databricks product renames. It includes: Product rename history Links to official documentation A short quiz to test yourself https://rebricked.org We're still improving it, so feedback is very welcome. If we missed a rename or you have ideas, let us
  • An Example of How Much It Costs to Build a Dashboard or Semantic Model Using Databricks Genie Code — r/databricks · 2026-07-29 How much does it cost to build a dashboard or semantic model with Databrick Genie Code? In my tests, a standalone dashboard came in at $2.10 (30 Genie DBUs), while a standalone semantic view cost $1.12 (16 Genie DBUs). To be clear, the tests I ran weren't designed to produce an award-winning dashboard or semantic model
  • I couldn't find a practical Data Mesh guide for Databricks, so I wrote one — r/databricks · 2026-08-10 I've been looking for a good guide on implementing Data Mesh in Databricks. There is plenty of content explaining what Data Mesh is, its principles, domains, data products, etc. But when you actually try to build it, the questions are much more practical: How should you organize catalogs and schemas? What should the ce
  • Open sourcing the Apache Spark skills we used on OpenAI’s data platform — r/dataengineering · 2026-08-11 I used to work on OpenAI’s data platform team, where we ran Airflow, Spark, Databricks, and dbt at a pretty serious scale. The rate at which the number of jobs was growing outpaced how much our team could manage pretty quickly w/ everyone using AI to write jobs, so we had to invest in making sure that our team's agents
  • Lakehouse RT vs. SQL Warehouse — r/databricks · 2026-08-05 I benchmarked the new Lakehouse RT for billion-record tables, and it is crazy fast and crazy cheap, making it the first real-time database with open-format storage. https://www.sunnydata.ai/blog/databricks-rt-lakehouse-benchmark-results https://medium.com/databrickscommunity/i-benchmarked-the-new-lakehouse-rt-for-billi
  • Genie One is cost effective ChatGPT - but for businesses — r/databricks · 2026-08-16 I feel really positive about all the recent progress of the Genie One which is backed up by all the business contexts, data, metrics views, pages and driven by Genie Ontology. It can get us answers cost effectively compared to external models connected to Databricks. Earlier it didn't have much unstructured data suppor
  • Databricks is leading the way in real-time data - so why is the billing still living in the past? — r/databricks · 2026-07-30 I love Databricks' platform evolution: Lakehouse//RT is delivering sub-second latency. But why does the system.billing.usage table still have up to a 4-hour delay? In a world of serverless compute that scales instantly, 4 hours is enough time for an "explosive query" to cause major bill shock before we even see it happ
  • To what extent do Data Engineers need to master Agentic AI? — r/dataengineering · 2026-08-03 I'm an Azure Data Engineer with 4.5 years of experience, primarily focused on Databricks. Since most of my role involves designing business solutions rather than writing raw code, I feel I might be falling behind on leveraging agentic AI workflows (such as custom skills, Markdown instructions, and orchestrators). As a
  • Operational feedbacks on Databricks vs. Snowflake — r/dataengineering · 2026-08-14 Hi, I am a lead on a data engineering team and looking for feedbacks on Databricks and Snowplake from an operational perspective. My team will chose a data platform and it’s going to be one of those two. I have experience with Databricks (and a bit with IaC and config of data platforms for Databricks, Domino, Rstudio s
  • What would you change in a Databricks architecture built 2–3 years ago? — r/databricks · 2026-08-13 I recently revisited a Databricks architecture I worked on 2–3 years ago, and it made me realize how much the platform has evolved. At the time, we used traditional Jobs, custom ingestion and data-quality frameworks, and separate components for orchestration and governance. If I were designing the same solution today,
  • 🔴 Unity AI Gateway is Generally Available. 🔴 — r/databricks · 2026-08-04 Unity AI Gateway is the Databricks governance solution for AI and is part of Unity Catalog. You can: ⚡ Control which AI services teams can use. ⚡ Route and manage AI traffic across providers. ⚡ Govern MCP servers to control access and costs. ⚡ Monitor usage, cost, access, and lineage from one place. FYI: Some capabilit
  • Databricks Genie Ontology — r/databricks · 2026-08-14 Having read through and seeing some demos I still don’t understand if Genie Ontology is a real thing or some marketing fluff, we have been asked to compare against Palantir foundry’s ontology and I find very few comparisons apart from the data model and relationships, for example how do I show that Genie Ontology Knowl

Governance, operations, and cost#

  • Netflix Adopts Cloud-Native Job Queueing System Kueue to Replace an In-House Solution — r/dataengineering · 2026-08-14 Netflix migrated most of its batch workloads onto Kueue, an open-source cloud-native batch job execution system that has outgrown its homegrown solution over the years. The company mapped the capabilities previously created in-house to Kueue’s functionality and also benefited from new features that would have been cost
  • Genie cost tracking — r/databricks · 2026-07-22 🔴 Genie Cost tracking update🔴 You can now track GENIE_FREE_USAGE SKU (only starts appearing on July 20, 2026) FYI: Free usage consumed before this date is not visible in the system tables. All free Genie usage appears under sku_name = 'GENIE_FREE_USAGE' it does include usage under the free allowance( 150 DBUs) which re

Platform engineering and delivery#

  • 3 Python + Data Warehouse Patterns I Use in Every Production Pipeline — r/dataengineering · 2026-07-20 Shared some content on data engineering based on my experiences. If you are also new to this learn from the article.
  • How to learn streaming? — r/dataengineering · 2026-07-29 Hey everyone. I'm a software engineer who's been working as a data engineer in my current role for the past four years. I learned data engineering from scratch on the job, and I've worked extensively with Spark, including designing and building new data processing systems. The challenge is that my company is almost ent
  • Apache Spark 4.2: What Data Engineers Need to Know About Auto CDC and Metric Views — r/dataengineering · 2026-07-28 Apache Spark 4.2 moves change data capture into the engine, plus metric views, Real-Time Mode, and Arrow-first Python for your pipelines.
  • How we built a pipeline for ingesting millions of records daily from Postgres shards into a Redshift Data Warehouse using Kafka, Debezium and Airflow — r/dataengineering · 2026-07-20 I work on the insights team at TouchBistro (a Canadian restaurant software company), and I led a project to pipe restaurant data from a sharded postgres operational database into Redshift using Debezium. I found articles on the web which talked about Debezium and CDC more generally but not much on how to best use it wi
  • Python usage patterns in data pipelines — r/dataengineering · 2026-07-23 Hello everyone, People trying to learn Python for data engineering ask me, “What libraries to learn?”, but the answer is not a list of libraries but patterns of usage. Especially with AI being able to generate so much code, I believe its critical to know exactly how the data is moved & processed. So I wrote this post t
  • When do you actually use ephemeral materialization in dbt? — r/dataengineering · 2026-07-30 I’m trying to understand the practical use case for materialized='ephemeral'. It doesn’t create a table or view, which seems to make the model harder to inspect and debug. If the goal is simply to avoid creating another table, why not materialize the model as a view instead? What advantages does an ephemeral model prov

Other practitioner signal#

  • What to read next? Learning Spark or Data Warehouse Toolkit? — r/dataengineering · 2026-08-07 I just finished reading Fundamentals of Data Engineering by Joe Reis and Matt Housley. Now, I'm trying to decide what makes the most sense to read next. I'm torn between reading Learning Spark or The Data Warehouse Toolkit and curious for those of you who have read them what would you recommend next? For context, I'm a
  • Microsoft Fabric - How is it? Is it worth getting into? — r/dataengineering · 2026-07-22 I recently saw a video where they were talking about Fabric Apps and how that would basically replace Power BI. They had it connected to some AI, would provide it with a link to a data model and it would spit out a fully rendered Fabric App page. The demo was decently impressive, as most demos are, but demos always lac
  • To what extent should data engineering work be centralized in a large org? — r/dataengineering · 2026-08-15 I work as a data engineer at a bank. My team is supposed to be the one centralizing data collection and management for the whole org. I've realized that there's this tension between centralizing and decentralizing data work -- whether we, as the org-level data team, should implement a flow or expect the analysts to imp
  • Genie Ontology — r/databricks · 2026-07-21 https://preview.redd.it/cnkpxe9f6keh1.jpg?width=1260&format=pjpg&auto=webp&s=4ce68cad93912763ffba1956eb690f1f6940552a Genie Ontology is coming soon. You can already get ready by preparing the following components: 🛑UC Metric Views: Build measures, dimensions, and relationships that map to your business KPIs with AI-ass
  • Identical data in DEV/TST/PROD? — r/dataengineering · 2026-07-31 I am doing a summer internship, and my company loads the exact same data into the development, testing, and production environments in Snowflake (although development and testing have some extra tables compared to production.) Is this normal? Google seems ambiguous on whether this is a massive antipattern. What do y'al
  • How we built a DuckDB transpiler — r/dataengineering · 2026-07-24 Hi there, I’ve been working with DuckDB extensively for the last several years. Used it across languages from Python, Go, JS via WASM, Swift, C & C++ and even Rust. As part of a product we're building, we built a DuckDB Transpiler that converts every line of SQL from DuckDB syntax into the syntax of the database you’re
  • Data Migration Horror Stories: What's your worst nightmare? — r/dataengineering · 2026-07-31 As we all know...data migrations can be a beast. But it's also a necessary evil. Tell me about the worst data migration you were on (or heard about) that would wake up even the most seasoned data engineer in a cold sweat
  • Genie one — r/databricks · 2026-07-27 Can we please stop with the constant UI changes to Genie? Literally every 2 weeks the UI/UX changes for it. We basically drop all of our users there (made a dns record for genie.company.com) and the amount times I have to tell someone "well the apps used to be under the prompt box, now click on apps" and today it's lik
  • Deploying Genie at Enterprise Scale — r/databricks · 2026-08-01 Posting another topic that came to us from a customer conversation that I thought would be good to discuss here on the sub ---------------------- Our team has a few technical questions regarding Genie usage and its limitations, especially as we plan to integrate Genie into our Enterprise application. Current QPM (Queri