Skip to content
Spark Acceleration Compare

Side by side, read 10 Nov 2026

Ways to speed up Spark without rewriting your jobs, compared

Open the LedgerCompare two options

Latest in the Ledger

  1. 2026-11-03Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000Run by Apache DataFusion Comet project, its maker
  2. 2026-11-03Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000Run by Apache DataFusion Comet project, its maker
  3. 2026-09Flarion TPC-DS SF100, six engines on one machineRun by Flarion, its maker
  4. 2026-09Flarion TPC-DS SF1000, six engines on one machineRun by Flarion, its maker

21 entries, 1 run by someone other than the option's maker. The full Ledger.

The Ledger: the 10 newest of 21 published benchmarks

Ordered by publication date, newest first. 20 of 21 entries were run by the maker of the option tested (its vendor or its own project). Undated entries come last. Full list: /ledger/.
PublishedOptionTestResult as publishedRan byMethod published
2026-11-03Apache DataFusion CometComet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000Spark 4.1.3 1,336 s; Comet 1.1.0 613 s (2.18x); Comet 1.2.0 532 s (2.51x)Apache DataFusion Comet project (its maker)Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository
2026-11-03Apache DataFusion CometComet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000Spark 4.2.0 994 s; Comet 1.1.0 601 s; Comet 1.2.0 488 s (published as 2.0x faster than Spark and 1.2x faster than Comet 1.1.0)Apache DataFusion Comet project (its maker)Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository
2026-09FlarionFlarion TPC-DS SF100, six engines on one machineTotal query time: Databricks Standard 1,180 s; Spark 3.5.5 1,135 s; Comet 1.0.0 652 s; Gluten with Velox 1.7.0 557 s; Flarion (build 2026-09-12) 386 s; Databricks Photon 362 sFlarion (its maker)Partly: machine, builds and settings published; Flarion is not downloadable and no scripts were found
2026-09FlarionFlarion TPC-DS SF1000, six engines on one machineTotal query time: Databricks Standard 103.1 min; Spark 3.5.5 135.6 min; Comet 84.4 min; Gluten with Velox 46.9 min; Flarion 38.3 min; Databricks Photon 30.4 minFlarion (its maker)Partly: machine, builds and settings published; Flarion is not downloadable and no scripts were found
2026-08-25NVIDIA cuDF plugin for Apache SparkGPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TBGPU 281 s (4.7 min); CPU 1,010 to 1,043 s (16.8 to 17.4 min); published speedup 3.7xAWS with NVIDIA (its maker)Yes, cluster, Spark config, EMR release and pricing method in the post (no scripts linked)
2026-06-11DualBirdDualBird sort-based Iceberg compaction, 100 GBAbout 12x to 20x faster task execution against vanilla Spark and an unnamed C++ accelerated Spark (DualBird's wording; no absolute times published)DualBird (its maker)Behind a sign-up form (white paper dated 2026-07-01); no public repository found
2026-06-11SpeedataSpeedata APU on TPC-H SF100, Microsoft CoddSpeed paperAPU speedup as relayed by Speedata from Table 2 of Microsoft's paper: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x, against single-core SQL ServerMicrosoft (paper); numbers read from Speedata's blog (another party)Yes, in the paper; the paper itself was not fetched
2025-08-21DataPelagoNucleus v3.0 against NVIDIA cuDF 25.04.00, operator testsUp to 10.5x on project, 10.1x on filter, 4.3x on aggregate, 4.5x on hash aggregate and 8.2x on K-largest selection, against cuDFDataPelago (its maker)No
2025-05-30Google Cloud Lightning EngineLightning Engine against open source Spark, TPC-H-like 10 TB3.6x acceleration over open source SparkGoogle Cloud (its maker)No
2022Databricks PhotonPhoton customer speedups, SIGMOD 2022 paperAverage speedup of 3x over the previous Databricks Runtime; maximum speedup over 10xDatabricks authors (its maker)No

Four ways into the facts

What each option does well, and where it is limited

Alphabetical. First sentence of each; the full text and sources are on the option's profile.
OptionKindDoes wellLimited
Amazon EMR runtime for Apache SparkAWS-tuned Spark runtime on EMRAWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes.The runtime is proprietary to AWS and available only where EMR, EMR on EKS and EMR Serverless are, so jobs on other Spark hosts must migrate first.
Apache AuronOpen-source Rust engine plugin for SparkThe project publishes its TPC-DS 1 TB method, with data generation and runner tools in the auron-project/tpcds-tools repository and the full Spark configuration.You build the jar yourself (Rust nightly and a JDK are required), and the getting-started page describes building from source.
Apache DataFusion CometOpen-source Spark plugin on DataFusionThe project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores.The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported.
Apache Gluten with VeloxOpen-source Spark plugin offloading to VeloxGluten needs no DataFrame or SQL changes, only configuration: the GlutenPlugin class, off-heap memory and the columnar shuffle manager.The docs name Spark 3.4.4, 3.5.5, 4.0.2 and 4.1.1 as officially supported and warn that an unofficial Spark version can throw NoSuchMethodError at runtime.
Databricks PhotonDatabricks's C++ vectorized enginePer Databricks's documentation, Photon works with existing Spark API code with no changes, and it is on by default for classic all-purpose and jobs compute.Photon is documented only as part of Databricks compute and is not offered as a standalone engine for open source Spark or EMR.
DataPelagoGluten-based CPU and GPU acceleratorPer DataPelago's documentation, DPA-S runs in the customer's own cloud account on EKS, Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, and GKE.DataPelago's 10X and 80% claims, and its customer quotes of 50% lower cost, carry no baseline, workload or method.
DualBirdFPGA Spark plugin on AWS F2DualBird publishes a worked example: a sort-based Iceberg compaction benchmark on 100 GB, with the dataset shape (about 370 million rows, 64 columns, ZSTD level 3) and two baselines (vanilla Spark and an unnamed C++ accelerated Spark).DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own.
FlarionRust and DataFusion engine as one JARFlarion lists prices: $0.009 per CPU hour for self-hosted Spark, $0.012 for fully managed Spark such as EMR, $0.095 per DBU on Databricks and $0.75 per CPU per month on premises.Flarion is proprietary and not downloadable (the page says contact us), and no repository or scripts were found for its benchmarks.
Google Cloud Lightning EnginePremium Spark engine on Google CloudGoogle's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning.Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings.
Microsoft Fabric native execution engineVelox-based engine inside Fabric SparkPer Microsoft's documentation, no code modifications are required, and the engine is enabled with an Acceleration tab toggle or spark.native.enabled.The engine runs only inside Microsoft Fabric on Azure and does not run on AWS EMR or EKS.
NVIDIA cuDF plugin for Apache SparkGPU plugin, formerly RAPIDS AcceleratorAWS's August 2026 post publishes runtime and cost per run: 281 seconds and $2.06 on GPU versus 1,010 to 1,043 seconds and $2.93 to $3.18 on CPU clusters.The plugin needs NVIDIA GPUs of the Volta generation or newer, so the cluster moves to GPU instances such as g7.4xlarge.
SpeedataPCIe analytics chip plus Dash Spark pluginDash plugs into the Spark Catalyst optimizer, and per Speedata it delegates SQL elements it cannot accelerate, such as UDFs, to the CPU.The APU is a PCIe card, so it needs physical servers; Speedata Acceleration for AWS runs the work in a Speedata-hosted environment, not in the customer's own EC2 fleet.

Five situations

What Spark acceleration means here

Spark acceleration, on this site, means a product or project that makes existing Apache Spark jobs run faster or cheaper while leaving the job code in place. The usual form is a plugin or a runtime setting that takes over the operators it supports, such as scans, joins and aggregations, and hands the rest back to Spark.

The twelve options fall into four groups. Native-engine plugins (Apache Auron, Apache DataFusion Comet, Apache Gluten with Velox, Flarion, DataPelago) run supported operators in C++ or Rust. Hardware accelerators (the NVIDIA cuDF plugin on GPUs, DualBird on FPGAs, Speedata on its own chip) move work off the CPU. Platform runtimes (Databricks Photon, the Amazon EMR runtime, Google Cloud Lightning Engine, the Microsoft Fabric native execution engine) are tied to one provider's service.

Options that need the jobs rewritten, such as moving from Spark to another engine or from RDD code to SQL, are outside this site. Each option page records what the vendor says it changes, where it runs and what has been published about its speed and price, with a source and a read date on every fact.

Data catalogs, lineage, semantic layers and MCP servers for data are outside this site's subject.

Guides and glossary

Sources

  1. Amazon EMR Spark features page, read 10 Nov 2026
  2. NVIDIA cuDF plugin download page, read 10 Nov 2026
  3. Apache Gluten homepage, read 10 Nov 2026
  4. Comet TPC-DS benchmark results, read 10 Nov 2026

Last reviewed 10 Nov 2026.