Five features worth knowing and why production should plan around 4.5
Apache Spark used to change slowly. A big release would arrive, teams would wait, and upgrades felt like a once-a-year event.
That rhythm just changed. Apache Spark 4.3 is out, and it is the first release on Spark's new quarterly release train.
This article explains what the new schedule means for you, walks through the five features worth knowing, and gives you a short checklist before you upgrade anything.
The new plan is simple. A new feature release ships every three months. Each release gets about six months of bug fixes.
That means you get new features sooner. It also means each release has a shorter life, so you need to know which one to build on.
Here is the key point. The long-term support (LTS) release for the 4.x line will be 4.5, not 4.3. So 4.3 is a great release to learn and test with. For production, plan around 4.5.
flowchart LR
A["Spark 4.3<br/>Feature release"] --> B["Spark 4.4<br/>Feature release"]
B --> C["Spark 4.5<br/>LTS release"]
A -.-> D["~6 months of fixes"]
B -.-> E["~6 months of fixes"]
C -.-> F["Long-term support<br/>Plan production here"]Imagine you have a table of trades and a table of prices. For each trade, you want the latest price at or before the trade time. That is a point-in-time join.
Before 4.3, you usually wrote a join, added ROW_NUMBER(), and then filtered to row 1. It worked, but it was long and easy to get wrong.
Now SQL has ASOF JOIN. You say what you mean in one clause. This matters for finance, IoT sensors, and any audit trail where "what was true at that moment" is the question.
Python UDFs are friendly to write but costly to run. Each row travels from the JVM to a Python worker and back.
In 4.3, simple Python UDFs can be translated into native SQL expressions. When that happens, there is no hop to Python at all. Your code stays the same, and the plan gets faster.
Not every UDF qualifies. Simple logic, like string cleanup or basic math, is the best candidate. Check the query plan to see if your UDF was translated.
Nested JSON is everywhere: API responses, event logs, app payloads. Flattening it often meant several explode calls and careful column picking.
JSON_TABLE is a SQL-standard function that turns nested JSON into rows and columns in one step. It is easier to read and easier to review in a pull request.
VARIANT is the type for semi-structured data that does not fit a fixed schema. Until now, changing one field inside it was awkward.
Spark 4.3 lets you update values inside a VARIANT in place. You no longer need to convert to a string, edit, and parse it back. Fewer steps means fewer chances to break the data.
Keeping history of changing records is a classic data engineering task. A customer moves city, and you want both the old and new address, each with its valid dates. That is SCD Type 2.
Spark Declarative Pipelines can now handle this with Auto CDC. You declare the keys and the ordering column, and the pipeline keeps the change history for you. No hand-written MERGE logic needed.
If you have read Chapter 14 of our book on slowly changing dimensions, this is the same idea, now handled by the engine.
You can try the old pattern today in Databricks Free Edition. It is worth understanding, because it shows exactly what ASOF JOIN replaces.
CREATE OR REPLACE TEMP VIEW trades AS
SELECT * FROM VALUES
('AAPL', TIMESTAMP '2026-11-02 10:00:05', 100),
('AAPL', TIMESTAMP '2026-11-02 10:00:12', 50)
AS t(symbol, trade_time, quantity);
CREATE OR REPLACE TEMP VIEW prices AS
SELECT * FROM VALUES
('AAPL', TIMESTAMP '2026-11-02 10:00:00', 221.10),
('AAPL', TIMESTAMP '2026-11-02 10:00:10', 221.40)
AS p(symbol, price_time, price);
-- The old way: join, rank, filter
WITH ranked AS (
SELECT
t.symbol,
t.trade_time,
t.quantity,
p.price,
ROW_NUMBER() OVER (
PARTITION BY t.symbol, t.trade_time
ORDER BY p.price_time DESC
) AS row_num
FROM trades AS t
JOIN prices AS p
ON t.symbol = p.symbol
AND p.price_time <= t.trade_time
)
SELECT symbol, trade_time, quantity, price
FROM ranked
WHERE row_num = 1;The first trade should match 221.10, and the second should match 221.40. Once your runtime supports Spark 4.3 SQL features, the same intent becomes one ASOF JOIN clause instead of a window and a filter. Check the Spark 4.3 SQL reference for the exact syntax before you rely on it.
To explore 4.3 on your own laptop, install it locally:
pip install pyspark==4.3.0Databricks Free Edition runs the Databricks Runtime, which picks up Spark versions on its own schedule. A feature in open source Spark 4.3 may arrive in Databricks at a different time. Test locally first if you want to try everything today.
Three checks will save you trouble.
Test existing jobs first. Some default settings changed in 4.3. A job that ran fine before may behave a little differently. Run your important jobs on a copy of the data and compare results.
Check your UDFs. If a UDF is translated to SQL, its behavior should match, but edge cases like nulls and types deserve a quick test.
Plan production around 4.5. Use 4.3 and 4.4 to learn and prepare. Move long-running production work to the 4.5 LTS release when it arrives.
A faster release train rewards engineers who keep learning in small steps. You do not need to read every release note. You need to know the few features that remove real work from your day.
ASOF JOIN, JSON_TABLE, and Auto CDC all do that. They replace clever code with clear intent. That is good for you, and good for whoever reads your code next.
Spark keeps moving, and so do we. The book walks you through Spark, Delta Lake and SCDs with hands-on labs in Databricks Free Edition. Start with the free lessons, watch short explainers on /watch, and check your understanding with the Data Engineer Associate practice exam.
Big thanks to the Apache Spark community and Databricks for another strong release. Happy upgrading, carefully.