I build data platforms on the Microsoft stack for a living. This is where I write down what works, what breaks, and what I would do differently. No fluff, just the things I wish someone had told me first.
Spark on Fabric needs a capacity, and that capacity runs all day to serve your reports. For a nightly job on data that is not actually big, an Azure Container Apps job in Python bills per second and scales to zero. A cost comparison: capacity 24/7 vs pay-and-shut-off, OneLake vs ADLS, and when each wins.
Read the postThe textbook says silver is a cleansed, queryable copy of bronze. Under real transformation complexity it almost never is. It becomes a staging area, or you sprout a fourth layer. An opinion piece on why medallion architecture is a metaphor, not a law, and how I actually use the silver layer.
Read the postAndrej Karpathy's LLM Wiki has the agent compile knowledge into interlinked markdown pages. I run the opposite: atomic notes the agent searches by meaning. Same goal, opposite bet. An honest pros, cons, and which one wins for which job.
Read the postUseful AI on a consulting project is not a model pointed at your data lake. It is a model reading the ordinary materials of the work: a local copy of the repo, discovery and design transcripts, the statement of work and pre-sales handoff, the deliverables, and a private knowledge base that ties them together. Here is how they fit, and the line they all share: none of them is your production data.
Read the postFabric lets you attach a data agent to a lakehouse, a warehouse, or a Power BI semantic model. Most teams aim it straight at the gold layer. I think the semantic model is usually the better home, and the reasons go well past convenience.
Read the postMore posts on the way. I write about real problems from real builds across healthcare, retail, manufacturing, and finance data work, with client details kept out of it. Want to talk shop or hire the practice? Reach me at admin@westarkdata.com.