Streaming reliability at batch cost, without hand-written tracking.
Many teams keep a streaming cluster running all day. The data, though, only lands a few times a day.
That means the cluster sits idle most of the time, and you still pay for it.
The usual fix is to switch to a batch job. But batch jobs bring their own problems. You have to track what you already loaded, and that tracking breaks in quiet ways.
There is a simpler middle path. It is called availableNow.
Most hand-written batch jobs remember the last run time and load anything newer.
This works until it does not. A job fails halfway. A file arrives late with an old timestamp. Two runs overlap. Suddenly rows are missing or loaded twice, and nobody notices for weeks.
The real issue is that you are building your own bookkeeping. Spark already has a better one.
A streaming query in Spark keeps a checkpoint. It is a small folder where Spark writes down exactly which files or records it has already processed.
When the query starts again, Spark reads the checkpoint and picks up where it left off. No timestamps. No guessing.
availableNow lets you keep that bookkeeping, while running the job like a batch job.
When you start a query with availableNow=True, Spark does three things:
flowchart LR
A[New files land in Volume] --> B[Job starts]
B --> C[Read checkpoint]
C --> D[Process backlog in micro-batches]
D --> E[Write to Delta table]
E --> F[Update checkpoint]
F --> G[Query stops, compute is released]You might have seen the older once=True trigger. It tried to process the whole backlog in one giant batch. With a large backlog, that could run out of memory. availableNow splits the work, so it is the one to use today.
This works in Databricks Free Edition. It reads CSV files from a Unity Catalog Volume and appends them to a Delta table.
from pyspark.sql.types import StructType, StructField, StringType, IntegerType
orders_schema = StructType([
StructField("order_id", StringType()),
StructField("customer_id", StringType()),
StructField("amount", IntegerType()),
])
source_path = "/Volumes/workspace/default/book_data/orders_incoming/"
checkpoint_path = "/Volumes/workspace/default/book_data/_checkpoints/orders_bronze/"
orders_stream_df = (
spark.readStream
.format("csv")
.option("header", "true")
.schema(orders_schema)
.load(source_path)
)
query = (
orders_stream_df.writeStream
.format("delta")
.option("checkpointLocation", checkpoint_path)
.trigger(availableNow=True)
.toTable("workspace.default.orders_bronze")
)
query.awaitTermination()Try this. Drop one CSV file into the folder and run the cell. Check the row count. Run it again without adding files. The count does not change, because the checkpoint knows that file is done. Add a second file and run once more. Only the new rows arrive.
That small experiment is the whole idea.
To check the result:
SELECT
COUNT(*) AS order_count
FROM workspace.default.orders_bronzeGive every query its own checkpoint folder. Two queries sharing one checkpoint will confuse each other. Never delete a checkpoint unless you really want to reload everything.
Control batch size on big backlogs. If a lot of data has piled up, add .option("maxFilesPerTrigger", 100) so each micro-batch stays a reasonable size.
Keep the schema stable. Streaming reads need a fixed schema. Declare it, as in the example, instead of letting Spark guess.
Most "streaming" data in companies is not truly real time. Orders, logs and app events often only need to be fresh within an hour.
For that kind of work, availableNow gives you streaming reliability at batch cost. It is also the pattern behind many bronze layer ingestion jobs, and it pairs naturally with Auto Loader when you move to larger workloads.
On a paid workspace, you would usually schedule this notebook in Jobs & Pipelines to run every hour. In Free Edition, you can run it by hand to learn exactly how it behaves.
availableNow=True processes everything new, then stops.Once you see a checkpoint working, timestamp-based batch loads start to feel fragile. That is a good instinct to build.