For the Databricks Free Edition Hackathon, I wanted to show that traditional machine learning still has a big role to play today, and how it can work hand in hand with Databricks’ newer AI tooling. As a concrete use case, I built a recipe recommendation engine that suggests relevant recipes to users: classic natural language processing (NLP) and topic modelling structure the data, and AI/BI Genie helps surface that value for end users. Both approaches work together rather than replacing one another. 

I have always been interested in using NLP tools to analyse classical Arabic texts, but I had never built an end to end solution in Databricks that brings an NLP pipeline to life. This felt like the perfect opportunity to do exactly that.

Hackathons give you the freedom to approach problems from different angles. Most of the time you’re handed a problem and asked to solve it, but with a broad scope I decided to flip that – find interesting data first and then discover what problem it could solve. 

I’m drawn to data engineering because I like understanding how systems work, and that curiosity extends to physics in my spare time. Solar flares seemed like fascinating territory to explore. I’m no expert, but I knew they can cause serious problems for electrical grids, especially aging infrastructure. The question formed: what if we could predict these events days in advance and help grid operators prepare?

With the recent explosion of AI models and the rapid ongoing innovation in that space, enterprises are trying to make use of the technology to gain a competitive edge. But while base AI models offer general capabilities, they fall short when it comes to deeply understanding proprietary data, workflows, and domain-specific nuances. Enterprises need intelligent agents that capitalize on their most valuable resource – their data. They don’t just need Artificial Intelligence; they need Data Intelligence. This could be for many reasons, including building innovative products, getting quick insights for better decision making, and for improving productivity.

However, the task of augmenting base models with proprietary data is not easy and the path to production remains treacherous. A staggering 90% of enterprise Gen AI projects fail to reach production. Why? The reasons may be familiar…

Operations teams have always been the heartbeat of execution, people who make things happen when strategy meets reality. But in many organisations, their dashboards haven’t evolved.

Leaders still rely on fragmented reporting tools, weekly Excel dumps, and delayed performance insights. The result? Decisions made on stale data and entire teams waiting for “the next refresh.”

It’s time to change that.

This blog explores how organisations can move from manual to autonomous dashboards using Databricks AI/BI Genie, Unity Catalog, and the new Lakehouse-native Dashboards, without depending on third-party tools like Power BI, Tableau, or Palantir.

Because intelligence shouldn’t live in a licence. It should live where your data already does, inside the Lakehouse.

Every organisation, whether managing homes, hospitals, or highways, depends on location. Where things happen shapes why they happen and what we should do next. 

Yet, many teams still treat geospatial data as a specialist discipline locked inside GIS (Geographic Information Systems) tools, accessible to only a few experts This creates silos and delays, when in reality, location is everyone’s business. 

It’s time to bring geospatial intelligence out of the GIS corner and into the Databricks Lakehouse, governed, scalable, and available to every data-driven team. 

With Spatial SQL, Lakeflow, and Unity Catalog, organisations can now turn coordinates into context and context into action, moving beyond static maps to living, real-time insights that evolve with the world.