For the second year in a row, I had the privilege of attending the Databricks Data + AI Summit in San Francisco.
For the second year in a row, I had the opportunity to stand on stage and speak to a packed room about something I’m incredibly passionate about, helping organisations unlock value from their data.
The shift we are seeing in analytics is not technical. It is organisational.
For years, organisations have invested heavily in dashboards, semantic layers, and modern data platforms. And yet, the same question keeps coming back from the business:
“Why does it still take days to get a simple answer?”
At first glance, it feels like a tooling problem. It isn’t.
It is a consumption problem.
Databricks has just taken a major step forward in something that rarely gets headlines but absolutely should.
Cost transparency.
In a recent post, Raman captured a pain every data leader knows too well. The moment a C-level executive asks:
“How much does our Customer Intelligence platform actually cost us?”
What follows is rarely a clean answer. Instead, it’s hours in spreadsheets, portal hopping, infrastructure mapping, and educated estimates. For years, the biggest challenge in Total Cost of Ownership (TCO) conversations hasn’t been performance. It hasn’t been scalability. It’s been visibility.
I was in the room at Databricks Summit San Francisco when Lakebase was first announced. At the time, it felt bold, almost provocative, to talk about operational databases as ephemeral, serverless infrastructure running directly on the lake.
Now, Lakebase PostgreSQL is generally available on AWS and beta in Azure, and the significance of that moment is much clearer. This isn’t just a new Databricks product reaching GA, it’s a signal that databases themselves must adapt to the realities of an AI-native world.
Within Databricks, Dashboards are frequently used to surface data to this audience and provide quick internal visibility.
However, anyone who has relied on Databricks Dashboards as a presentation layer will be familiar with their limitations. Interactivity is constrained, large datasets are often truncated, version control is minimal, cross-dashboard interactions are not supported, and geospatial visualisation options are fairly limited and difficult to customise.
Picture the scene…
You’re three years in, millions spent and a technically impressive data platform has now been sitting proudly in production for a few months.
And yet, wider adoption is anaemic. Business units are still building shadow systems and the data team spends most of their time fielding complaints and one-off manual requests.
Unfortunately, this is not as unique a case as it should be with industry research suggesting anywhere from 60-85% of data platform deliveries fail to deliver on their promise. The interesting wrinkle, though, is that these failures are not primarily technical but are rooted in process and mindset, especially neglecting user needs, misaligned outcomes, lack of ownership and absence of effective feedback loops between technology teams and end users.
This is where product thinking can bring real tangible benefits. At Unifeye we’ve built our practice around a simple principle: data platforms succeed when they’re treated as products, not standalone projects. That means dedicated ownership, user-centric design, continuous feedback and value-based measurement all underpinned by adaptive delivery practices and organisational change support.
The reality is your data platform is a product and even if you don’t treat it that way your users will. They’re making adoption decisions, comparing it to alternatives and voting with their feet when it doesn’t meet their needs.
For the Databricks Free Edition Hackathon, I wanted to show that traditional machine learning still has a big role to play today, and how it can work hand in hand with Databricks’ newer AI tooling. As a concrete use case, I built a recipe recommendation engine that suggests relevant recipes to users: classic natural language processing (NLP) and topic modelling structure the data, and AI/BI Genie helps surface that value for end users. Both approaches work together rather than replacing one another.
I have always been interested in using NLP tools to analyse classical Arabic texts, but I had never built an end to end solution in Databricks that brings an NLP pipeline to life. This felt like the perfect opportunity to do exactly that.
Hackathons give you the freedom to approach problems from different angles. Most of the time you’re handed a problem and asked to solve it, but with a broad scope I decided to flip that – find interesting data first and then discover what problem it could solve.
I’m drawn to data engineering because I like understanding how systems work, and that curiosity extends to physics in my spare time. Solar flares seemed like fascinating territory to explore. I’m no expert, but I knew they can cause serious problems for electrical grids, especially aging infrastructure. The question formed: what if we could predict these events days in advance and help grid operators prepare?
In today’s data-driven ecosystems, silos are the enemy of agility. Whether it’s marketing assets buried in SharePoint folders or operational logs scattered across cloud drives, fragmented data slows down insights and complicates governance. The Lakehouse architecture, pioneered by Databricks, offers a unified solution, combining the reliability of data warehouses with the flexibility of data lakes.
But to unlock its full potential, we need seamless ingestion pipelines that bridge these silos. That’s where Lakeflow Connect comes in.
With the recent explosion of AI models and the rapid ongoing innovation in that space, enterprises are trying to make use of the technology to gain a competitive edge. But while base AI models offer general capabilities, they fall short when it comes to deeply understanding proprietary data, workflows, and domain-specific nuances. Enterprises need intelligent agents that capitalize on their most valuable resource – their data. They don’t just need Artificial Intelligence; they need Data Intelligence. This could be for many reasons, including building innovative products, getting quick insights for better decision making, and for improving productivity.
However, the task of augmenting base models with proprietary data is not easy and the path to production remains treacherous. A staggering 90% of enterprise Gen AI projects fail to reach production. Why? The reasons may be familiar…
