WestarkData
Back to all posts
Microsoft Fabric June 26, 2026 9 min read

You probably don't need Spark for that nightly batch

JM
John MorrisonSenior data architect, Westark Data

Spark needs a capacity, and that capacity tends to run all day to serve your reports. For a nightly job on data that is not actually big, a Python container that wakes up, runs, and shuts off does the same work for a fraction of the cost.

When you build a data pipeline on Microsoft Fabric, the default move for the heavy lifting is a Spark notebook. It is right there, it is what the tutorials use, and it works. I want to argue that for a large share of real nightly batch it is the wrong default, and the reason is mostly about cost.

This is not a complaint about Spark. Spark is a fine distributed compute engine. It is a complaint about reaching for a distributed compute engine to do work that was never distributed in the first place, and about what running it on Fabric quietly forces you to keep paying for.

What you are really paying for with Spark on Fabric

Spark in Fabric runs on a Fabric capacity. To run your nightly batch you need that capacity live. So far so reasonable. The catch is that the same capacity is what serves your daytime workspace browsing and your Direct Lake reports, and those need it up during business hours. So the capacity ends up running all day, every day.

Now look at where the heavy compute actually happens. Say the batch runs at night for three hours, which covers a lot of real workloads. The other twenty-one hours the capacity is mostly idling so a report renders quickly and people can click around the workspace. You are paying for a big always-on capacity, and the part that genuinely needs heavy compute is a slice of the day.

The alternative: a container that wakes up, runs, and disappears

Here is the other way to do the batch. Azure Container Apps lets you run a job, a container that starts on a schedule, runs to completion, and stops. You put plain Python in it. It reads from storage, does the transformations, writes Delta tables, and exits.

The cost shape is the opposite of a capacity. The job bills per second only while it runs, and scales to zero the rest of the time, so it costs nothing when it is not running. There is even a monthly free allotment of compute per subscription that a modest nightly job can fit inside. So your two-in-the-morning batch spins up, runs, writes its output, shuts off, and bills you for the minutes it used and nothing else.

For a single nightly run, spin-up barely matters. One cold start in front of a three-hour batch is noise, and the cost argument here rests on the all-day capacity, not on the wait. I will not pretend the few minutes a Spark cluster takes to warm up is what sinks it.

Where the cluster wait does hurt

The wait stops being noise in two situations, and they are common enough to name.

The first is a pipeline with several notebooks. Each notebook tends to want its own Spark session, so the cold start you paid once gets paid again and again down the chain. To avoid that you reach for tricks. You tag the notebooks so they share one session through the pipeline, or you call runMultiple so a parent notebook runs its children inside the session it already has. None of that is hard once you know it, but it is real engineering effort spent on nothing but dodging the spin-up tax. A plain container has no session to keep warm or hand around. It just runs.

The second is frequency. A job that runs every hour, all day, pays the cluster wait on every single run. Twenty-some cold starts a day, each one warming up before any real work happens, and a capacity pinned on all day anyway to service the schedule. There the gap between a few minutes and a few seconds stops being a rounding error and turns into a steady drag on every cycle. A container that starts clean in seconds, runs, and bills only for the run is a different experience entirely. My own least favorite version is the all-day hourly job, where I can spend more design effort keeping a Spark session alive than I spend on the actual transformation.

For a once-a-night batch, this is a footnote. The more notebooks your job spans and the more often it runs, the louder the footnote gets.

But isn't that what Spark is for?

The honest objection is scale. Spark exists to spread a job across many machines when one machine cannot hold it. If your batch genuinely processes terabytes with big distributed shuffles and joins, Spark is the right tool and a single container is not.

But be honest about your data. A very large share of enterprise nightly batch is tens of gigabytes, sometimes low hundreds. That fits comfortably on one decent node, and modern single-node Python tools chew through it fast. Polars, DuckDB, and Arrow, with delta-rs to write Delta, will handle that volume without a cluster. If your big data is not actually big, Spark is a distributed engine you are renting to do single-node work, and paying cluster and capacity prices for the privilege.

OneLake or ADLS

Storage is the other half of the decision. OneLake is Fabric's storage, and it is what makes Direct Lake reporting work. It is also tied to Fabric. ADLS is standalone Azure storage, cheap, and completely independent of any compute capacity. A Python container can write Delta straight to ADLS.

You do not have to give up Fabric reporting to do this. A OneLake shortcut can point at the ADLS location, so the Delta tables your container wrote show up inside a Fabric lakehouse with no copy, and Direct Lake can read them. That is the move that decouples the batch and the storage from the reporting capacity. Compute and storage run cheap and independent. Fabric still sees the data when it needs to serve a report.

Where the savings actually come from

Put it together for the common shape, which is nightly batch plus daytime reporting and browsing.

  • The batch moves to a container job that runs at night and costs you only the few hours it runs.
  • The capacity now only has to cover what daytime reporting truly needs. You can size it smaller, pause it off hours, or, if you can live with Import mode refreshed overnight, drop it a lot further.

The win is not shaving the batch bill. It is no longer running heavy batch compute on a capacity that has to stay on all day for an entirely different reason. You stop renting a cluster around the clock so that a report renders in the afternoon.

A rough cost example

Numbers make it concrete. These are list prices in one US region, in dollars, and they move around by region and over time, so treat them as the shape of the answer and not a quote. Storage is left out because it costs about the same either way.

The stock Spark setup needs an F8 capacity. That is the size that can feed a Spark session and still leave room to serve the workspace and the reports at the same time. To cover daytime reporting, it runs all day.

  • F8, pay as you go, all month. About $1.44 an hour, which is roughly $1,050 a month running around the clock.
  • F8, one-year reservation. About $625 a month. You give up the ability to pause it, but a capacity that runs most of the day comes out cheaper reserved.

Now the container. Give it the same shape of compute the job actually uses, say eight vCPUs and thirty-two gigs of memory, and run it three hours a night. Azure Container Apps bills about $0.000024 per vCPU-second and $0.000003 per GiB-second while it runs, and nothing the rest of the time.

  • Container, 8 vCPU and 32 GiB, 3 hours a night. About $90 a month. A per-subscription monthly free compute grant trims a few dollars more off the top.

So the same nightly work lands near $90 a month on the container against $625 to $1,050 on the always-on F8. Seven to twelve times cheaper for the identical job.

Here is the part worth sitting with. Per running hour the two are close. The F8 is about $1.44 an hour and that container is about a dollar an hour. If you suspected the rates were basically a tie, you were right. The savings do not come from a cheaper compute rate. They come from running three hours instead of twenty-four, and from not having to oversize the box so it can run batch and serve a website at the same time. Pay for the compute, then turn it off.

What you give up

This is not free. Saving the capacity money means giving up real things, and the biggest one has a name: Direct Lake.

Direct Lake is the Power BI mode that reads your Delta tables straight from OneLake, with no import copy and no query round-trip. You get import-like speed on always-current data. When the batch writes new rows, the next click on the report sees them, with no dataset refresh to schedule. It is genuinely good, and it only runs on a Fabric or Premium capacity. Drop the capacity and Direct Lake goes with it.

Without it, your reports do not stop updating. They just update a different way. The usual answer is Import mode. Power BI pulls a copy of the data into the model on a refresh, which is fast to query, and you schedule that refresh to fire as soon as the nightly batch finishes. The reports are fresh for the business day. The other option is DirectQuery, which hits a live endpoint for always-current data at the cost of speed. You author in Power BI Desktop either way. The real change is that you publish to a normal Pro workspace and refresh from your storage, instead of pointing a Direct Lake model at OneLake.

Here is the kicker for a nightly workload. Direct Lake's headline feature is that the data is always current, with no refresh needed. But if your data only changes once a night, you are giving up freshness the data never had. A scheduled refresh right after the batch lands you in the same place. Always-current only earns its keep when the data actually changes through the day, which is the hourly-job case, not the nightly-batch one.

Then there is licensing, and the line that matters is F64.

  • At F64 and up, report viewers need no license of their own. The capacity covers them. With a few hundred viewers, that free viewing can be worth more than the capacity costs, which is a strong reason to keep the capacity rather than chase container savings.
  • Below F64, there is no free viewing. Every viewer already needs a Power BI Pro license, about fourteen dollars a user per month, on top of the capacity. So a workload sized for something like an F8 was paying for the capacity and the per-viewer licenses both.

That second case is where the container story is cleanest. You were already buying the Pro licenses, so dropping the capacity adds no license cost. You shed the capacity, keep the Pro licenses you already needed, and serve Import-mode reports from the same cheap storage the container writes to. The savings are real precisely because you were below the line where a capacity earns its keep on licensing alone.

When to just use Spark

To be fair, plenty of cases still point straight at Spark and Fabric.

  • The data really is big. Distributed shuffles and joins over terabytes are exactly what Spark is good at, and a single container will not cut it.
  • You are all in on Fabric. One governed platform, a single security model, and a lot less integration surface to manage. That has real value, and the container route adds operational pieces: images, a registry, deployment, monitoring, and the ADLS and shortcut wiring.
  • Your team lives in notebooks. Skillset matters. The cheapest architecture nobody can maintain is not cheap.

The container route trades some of that simplicity for cost and decoupling. It is a trade, not a free win.

The short version

Spark is a distributed answer to a distributed problem. A lot of nightly batch is not that problem. When yours is not, a Python container that wakes up at two in the morning, runs, and shuts off does the same work for a fraction of the cost, and it frees you from paying for a capacity around the clock just to keep the lights on for a report. The one caveat is licensing. If your reporting alone would justify an F64, where viewers are free, keep the capacity and let it earn its keep. Below that line, the container is usually the cheaper and simpler home for the batch. Look at your real data volume and your real reporting needs before you reach for the notebook.

Staring at a Fabric capacity bill and wondering how much of it is just batch that could run somewhere cheaper? That is a worthwhile afternoon. admin@westarkdata.com.