Announcing Apache Spark 4.2.0: Geospatial Intelligence, First-Class CDC, DSv2 Transactions, and Arrow-Powered PySpark

The Apache Spark community has officially released Apache Spark 4.2.0. As the third major iteration in the Spark 4.x line, this release represents a significant milestone for data platform engineers, analytics teams, and AI/ML practitioners. With over 1,700 issues resolved and contributions from more than 250 open-source contributors, Spark 4.2 elevates modern data architectures to new heights.

From native geospatial types and built-in Change Data Capture (CDC) to Data Source V2 transactions and default Arrow acceleration for PySpark, Spark 4.2 delivers substantial enhancements across performance, developer experience, and governance.

---

1. Native Geospatial Intelligence

Historically, performing geospatial analytics in Spark required third-party libraries such as Apache Sedona or Magellan. Spark 4.2 introduces native geospatial data support directly into core Spark SQL and DataFrame APIs - enabled by default.

Key Highlights

• Native Data Types: Introduces GEOMETRY and GEOGRAPHY data types directly into Spark SQL.
• Spatial Functions: Out-of-the-box ST_* functions for spatial operations (distance, containment, intersection, bounding box, etc.).
• I/O Support: Native Well-Known Text (WKT), Well-Known Binary (WKB), and GeoParquet / Parquet read and write integration.
• SRID Registry: Full Spatial Reference System Identifier (SRID) registry support for precise coordinate system transformations.

Whether handling logistics, geospatial telemetry, or IoT device mapping, spatial workloads are now fully integrated into Spark SQL pipelines as first-class citizens.

---

2. First-Class Change Data Capture (CDC) & Auto CDC

Tracking incremental row-level changes across data lakes has historically required custom merge logic or complex framework orchestration. Spark 4.2 simplifies CDC into a native, declarative engine.

-- Read row-level changes seamlessly via SQL
SELECT * FROM CHANGES(my_catalog.db.orders) AT (TIMESTAMP => '2026-01-01 00:00:00');

Key Highlights

• SQL CHANGES Clause: Query row-level inserts, updates, and deletes directly in batch and structured streaming using SQL, DataFrame, PySpark, and Spark Connect APIs.
• Auto CDC in Declarative Pipelines: Automated Slowly Changing Dimension (SCD) Type 1 upserts within Spark Declarative Pipelines, handling record deduplication, ordering, and state management out of the box.

---

3. Data Source V2 (DSv2) Transactions & Storage Upgrades

Spark 4.2 extends the capabilities of Data Source V2 (DSv2), enabling custom connectors and table formats (such as Apache Iceberg, Delta Lake, and Apache Hudi) to integrate more tightly with the Spark execution engine.

Key Highlights

• DSv2 Transaction Management: Enables connectors to execute multi-operation transactions atomically, ensuring data platform consistency across multi-step writes.
• Partition-Stats Filtering: Enhanced partition-level statistical pruning, dramatically accelerating query planning and scan times on large partitioned tables.
• Schema Evolution in INSERT Statements: Introduces the WITH SCHEMA EVOLUTION clause to INSERT statements, seamlessly accommodating source schema drift without breaking target lakehouse tables.

---

4. Default PySpark Performance: Arrow Engine Enabled

PySpark execution receives a massive out-of-the-box performance improvement in Spark 4.2 by making Apache Arrow-based execution the default across all workloads.

Key Highlights

• Arrow-based PySpark Inter-Process Communication (IPC): Enabled out of the box for ultra-fast data transfer between the JVM host and Python worker processes.
• Arrow-optimized Python UDFs: Enabled by default to accelerate custom Python functions and Pandas operations.

Why it matters: By removing serialization overhead between the JVM host and Python worker processes, PySpark users automatically benefit from faster DataFrame transfers, low-overhead UDF execution, and significantly reduced memory consumption - without requiring custom configuration flags.

---

5. SQL Surface & Semantic Modeling Upgrades

Spark 4.2 introduces several powerful extensions to Spark SQL for analytical processing, declarative modeling, and syntax flexibility:

• NEAREST BY Top-K Ranking Join: A new join primitive for top-K nearest-neighbor queries, optimized within Catalyst for vector search, spatial proximity, and similarity workflows.
• Declarative Metric Views: Allows platform teams to define metrics directly within Spark view definitions using the CREATE VIEW ... WITH METRICS syntax, bridging semantic modeling layers (such as dbt or Looker) directly into Spark SQL.
• Path-Based Name Resolution: Enables dynamic configuration of search paths using SET PATH, CURRENT_PATH(), and qualified function/session name resolution, eliminating catalog boilerplate in multi-tenant environments.
• Pipe Operators & SQL Extensions:
• Expanded SQL Pipe syntax with support for | alongside |>, as well as aggregations and GROUP BY within pipe operators.
• Added support for the QUALIFY clause to filter window function results easily.
• Added TABLESAMPLE SYSTEM block sampling with DSv2 pushdown.

---

6. Modernized UI, Observability & Cloud-Native Infrastructure

All these powerful engine features demand equally capable tooling to manage, debug, and deploy them. Spark 4.2 delivers a thoroughly modernized user interface alongside cloud-native infrastructure improvements.

Modernized Web UI

• Dark Mode Support: Native dark and light themes for improved usability during extended debugging sessions.
• Interactive Query Plan Visualization: Searchable, zoomable, side-by-side Spark SQL physical query plan diagrams.

Infrastructure & Cloud-Native Execution

• Java 25 Support: Spark 4.2 can now be built and run on Java 25 runtime environments.
• K8s Control Plane Optimization: Reduced API server overhead and a dedicated Kubernetes Resource Manager API for large-scale cloud-native clusters.
• Heterogeneous K8s Executor Management: Dynamic allocation now supports mixed node selectors and resource spec execution pools, including mixed CPU and GPU worker pods.

---

Summary: What Spark 4.2.0 Means for Your Data Platform

Together, these six core updates translate into practical value across enterprise data stacks:

Feature Area	Key Upgrade	Data Platform Benefit
Geospatial	GEOMETRY / GEOGRAPHY native types	Native spatial analysis without external dependency fragmentation
CDC	CHANGES clause & Auto CDC	Simplified ingestion pipelines, automated SCD-1 upserts
Storage Engine	DSv2 Multi-op Transactions	Atomic commits, safer batch writes, flexible schema evolution
PySpark	Default Arrow IPC & UDFs	Instant performance gains for Python/Pandas workflows
SQL & Modeling	NEAREST BY, Metric Views, Pipe Syntax	Expressive analytical queries, vector search, and unified metrics
Observability & Infrastructure	Modernized UI & Dynamic K8s Allocation	Faster query debugging and optimized cloud-native executor footprint

---

Dataverses Data Platform: Spark 4.2 Integration

The Dataverses Data Platform delivers full production support for Apache Spark 4.2.0 out of the box across all deployment models:

• Native Geospatial & CDC Pipelines: Build location-aware, real-time ingestion pipelines directly in pure SQL without managing external plugins.
• Arrow-Native PySpark Performance: Enjoy immediate execution speedups for Python and Pandas workloads with zero code modifications required.
• Transactional Storage & Modern SQL: Combine DSv2 multi-operation transactions with enriched SQL constructs like NEAREST BY, QUALIFY, and Metric Views for safer, highly expressive analytics.

Spark 4.2.0 represents a strategic inflection point for the ecosystem. By bringing native geospatial analytics, declarative CDC, Arrow-powered PySpark, and multi-op transactions together in a managed environment, the Dataverses Data Platform bridges the gap between open-source innovation and enterprise-grade execution.