Designing good data engineering flows and not avoiding the real tradeoffs
I have been building data pipelines for five years now. Multiple clouds, enough SQL to fill a novel, and more Spark jobs than I can count. Every time I come back to Databricks, it makes me rethink something I thought I already understood.
Not because it is complicated. Because it makes the hard stuff impossible to ignore.
A few months back, I was helping a team migrate their batch jobs into a lakehouse setup. Nothing exotic; just standard ingestion, some cleanup, aggregate tables for downstream teams. On paper, the Databricks pieces fit perfectly. Delta Lake, Spark jobs, Unity Catalog for permissions. The demo was clean. Everyone nodded. Then we went to production, and the gaps showed up. Not in Databricks itself. In how we had designed around it.
That is the real story. The platform gives you an incredibly flexible surface area. Your job is to not screw it up.
The architecture you bring matters more than the tool
People keep asking me whether Databricks is better than Snowflake or BigQuery or whatever the new managed warehouse is this quarter. I think that is the wrong question. I have used all of them. They all work. The difference is what kind of discipline you bring to the table.
With Databricks specifically, I keep seeing teams treat the flexibility as a license to improvise. They build a notebook. It works. They build another one. It also works. Six months later, there are forty jobs, nobody knows which table is fresh, and the “lakehouse” has become a data swamp with better marketing.
In that migration I mentioned, we had to pause and fix the basics. Raw data lands in one place. Cleaned data lives in another. Curated outputs for analytics are strictly separate. You would think this is obvious, but in the rush to ship, teams skip it. I get it. I have skipped it too.
The difference is that Databricks lets you skip it for longer, which is actually a problem. By the time the mess catches up with you, the surface area is huge. Fixing it means refactoring pipelines, retraining analysts, and untangling permissions. Much better to set the boundaries early and be boring about it.
Governance is not paperwork; it is the product
I used to roll my eyes at governance discussions. Access control, lineage, cataloging. It felt like bureaucracy slowing down real engineering. That changed when I started working across multiple teams who all touched the same data.
Without a central way to track who owns what, you end up with tables that have seventeen downstream consumers and zero upstream owners. Someone changes a column type on a Friday, and three dashboards break on Monday. Then you spend your Monday in Slack apologizing.
Unity Catalog is the piece that made me stop rolling my eyes. Not because it is magical, but because it forces the conversation early. Who can read this? Who can modify it? Where did this data come from? Those questions used to be afterthoughts. Now they are part of the setup, and honestly, that is where they should have been all along.
When governance is bolted on later, it is always a mess. When it is woven into the architecture, it is almost invisible. That is the state I want: where the data is secure and traceable without engineers constantly filling out forms.
Performance is a decision, not a tuning knob
The Spark tuning rabbit hole is deep. I have gone down it. Partitioning, bucketing, AQE, shuffle optimizations. There is always another knob to turn.
But here is the thing. In most cases, the performance problem is not that Spark is running slowly. The performance problem is that the data was structured poorly in the first place. I have seen pipelines that run for hours because someone is doing a full merge on a billion-row table every run, when they only needed to process the last two hours of changes. You can tune the executor memory all day. It will not fix a bad strategy.
The fastest wins I have gotten on Databricks were not from tuning. They were asking basic questions. Do we need all this data? Can we partition by date instead of doing a global scan? Is this transformation happening at the right layer? Can we make this incremental?
Those questions sound less exciting than optimizing the Catalyst engine. But they save more money and more sanity.
I also think about cost differently now. Fast and expensive is not a win. Predictable and reasonable is the win. Your finance team does not care about your clever Spark query. They care whether the compute bill is twice what they budgeted. Databricks gives you the tools to spend a lot of money very efficiently. Choose wisely.
It works best when it is part of the neighborhood
Another thing I like is that Databricks does not pretend the rest of your cloud infrastructure does not exist. On AWS, it works with S3, IAM, and the broader security model. On GCP, it integrates with BigQuery and GCS. On Azure, it fits into the Microsoft ecosystem.
That matters because no one starts with a blank slate. Real companies have real systems. They have compliance requirements, existing storage, and monitoring tools they already paid for. A platform that forces you to rip everything out and start over is a platform that gets rejected by the security team.
I usually think of Databricks as the execution and governance layer sitting on top of cloud storage. The cloud handles the raw infrastructure. Databricks handles the data processing, cataloging, and access. That separation is healthy. It keeps responsibilities clear.
My advice if you are starting out
If you are adopting Databricks now, here is my honest advice. Do not start with the advanced features. Start with one pipeline. Make it clean. Make it observable. Make it owned by someone specific.
Then build a second pipeline using the same pattern. Then a third. Resist the urge to let every team invent their own conventions. The platform is flexible enough to support a hundred different patterns, which means without standards, you will get a hundred different patterns.
I have made this mistake. I have let teams go off and build custom solutions because they were in a hurry. It always comes back. Technical debt in data platforms is just like technical debt in code, except harder to refactor because more people are reading the data than reading the code.
The teams that scale well are boring in the best way. Same folder structure. Same naming. Same testing approach. Same review process. When something breaks, they know where to look. That is the goal.
Why I still bother writing about this
Data engineering is changing. It used to be about moving rows from a database to a warehouse. Now it is about building trustworthy data products that serve analytics, operations, and AI at the same time. That is a much harder job, but it is also more interesting.
Databricks sits right at that intersection. It is not the only platform that does, but it is one of the few that makes the intersection feel navigable. Recent improvements around Unity Catalog, lakehouse federation, and governance patterns show that the platform is still maturing in the right direction.
For me, writing about it is less about the tool and more about the mindset. Good data engineering is not about picking the right platform. It is about building systems that do not fall apart when you are not looking.
Questions people ask me
Do I need to know Spark deeply to use Databricks?
Not anymore. SQL is enough for a lot of use cases now. Spark helps for scale, but you can start without being a distributed systems expert.
Is it overkill for small teams?
Depends. If you have one dataset and one dashboard, maybe. But if you know complexity is coming, setting up the foundation early is cheaper than migrating later.
What is the biggest mistake you see?
Treating Databricks like a hosted notebook service. It is a platform. Use it like one, or you will regret it later.