Asset
21 entries, 1 run by another party
Spark Accelerator Ledger
Every published benchmark for each option, with who ran it, on what, and the result as published.
Open the LedgerSide by side, read 10 Nov 2026
Latest in the Ledger
21 entries, 1 run by someone other than the option's maker. The full Ledger.
| Published | Option | Test | Result as published | Ran by | Method published |
|---|---|---|---|---|---|
| 2026-11-03 | Apache DataFusion Comet | Comet 1.1.0 and 1.2.0 against Spark 4.1.3, TPC-DS SF1000 | Spark 4.1.3 1,336 s; Comet 1.1.0 613 s (2.18x); Comet 1.2.0 532 s (2.51x) | Apache DataFusion Comet project (its maker) | Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository |
| 2026-11-03 | Apache DataFusion Comet | Comet 1.1.0 and 1.2.0 against Spark 4.2.0, TPC-DS SF1000 | Spark 4.2.0 994 s; Comet 1.1.0 601 s; Comet 1.2.0 488 s (published as 2.0x faster than Spark and 1.2x faster than Comet 1.1.0) | Apache DataFusion Comet project (its maker) | Yes, scripts in the Comet repository and the apache/datafusion-benchmarks repository |
| 2026-09 | Flarion | Flarion TPC-DS SF100, six engines on one machine | Total query time: Databricks Standard 1,180 s; Spark 3.5.5 1,135 s; Comet 1.0.0 652 s; Gluten with Velox 1.7.0 557 s; Flarion (build 2026-09-12) 386 s; Databricks Photon 362 s | Flarion (its maker) | Partly: machine, builds and settings published; Flarion is not downloadable and no scripts were found |
| 2026-09 | Flarion | Flarion TPC-DS SF1000, six engines on one machine | Total query time: Databricks Standard 103.1 min; Spark 3.5.5 135.6 min; Comet 84.4 min; Gluten with Velox 46.9 min; Flarion 38.3 min; Databricks Photon 30.4 min | Flarion (its maker) | Partly: machine, builds and settings published; Flarion is not downloadable and no scripts were found |
| 2026-08-25 | NVIDIA cuDF plugin for Apache Spark | GPU Spark on Amazon EMR on EKS with G7 instances, TPC-DS 3 TB | GPU 281 s (4.7 min); CPU 1,010 to 1,043 s (16.8 to 17.4 min); published speedup 3.7x | AWS with NVIDIA (its maker) | Yes, cluster, Spark config, EMR release and pricing method in the post (no scripts linked) |
| 2026-06-11 | DualBird | DualBird sort-based Iceberg compaction, 100 GB | About 12x to 20x faster task execution against vanilla Spark and an unnamed C++ accelerated Spark (DualBird's wording; no absolute times published) | DualBird (its maker) | Behind a sign-up form (white paper dated 2026-07-01); no public repository found |
| 2026-06-11 | Speedata | Speedata APU on TPC-H SF100, Microsoft CoddSpeed paper | APU speedup as relayed by Speedata from Table 2 of Microsoft's paper: Q07 36.7x, Q10 40.7x, Q12 49.4x, Q13 259.2x, Q16 61.5x, against single-core SQL Server | Microsoft (paper); numbers read from Speedata's blog (another party) | Yes, in the paper; the paper itself was not fetched |
| 2025-08-21 | DataPelago | Nucleus v3.0 against NVIDIA cuDF 25.04.00, operator tests | Up to 10.5x on project, 10.1x on filter, 4.3x on aggregate, 4.5x on hash aggregate and 8.2x on K-largest selection, against cuDF | DataPelago (its maker) | No |
| 2025-05-30 | Google Cloud Lightning Engine | Lightning Engine against open source Spark, TPC-H-like 10 TB | 3.6x acceleration over open source Spark | Google Cloud (its maker) | No |
| 2022 | Databricks Photon | Photon customer speedups, SIGMOD 2022 paper | Average speedup of 3x over the previous Databricks Runtime; maximum speedup over 10x | Databricks authors (its maker) | No |
Asset
21 entries, 1 run by another party
Every published benchmark for each option, with who ran it, on what, and the result as published.
Open the LedgerAsset
12 options, 4 columns
What each option makes a team change: job code, the cluster, the cloud, and the vendor contract.
Open the Change BudgetSide by side
66 pairs
Two options in the same rows, with sources, and no conclusion drawn for you.
Pick a pairProfiles
12 options
Same fields in the same order for every option: what it does well, where it is limited, the published facts.
Read the profiles| Option | Kind | Does well | Limited |
|---|---|---|---|
| Amazon EMR runtime for Apache Spark | AWS-tuned Spark runtime on EMR | AWS describes the runtime as 100% API compatible with open source Spark, so Spark jobs need no code changes. | The runtime is proprietary to AWS and available only where EMR, EMR on EKS and EMR Serverless are, so jobs on other Spark hosts must migrate first. |
| Apache Auron | Open-source Rust engine plugin for Spark | The project publishes its TPC-DS 1 TB method, with data generation and runner tools in the auron-project/tpcds-tools repository and the full Spark configuration. | You build the jar yourself (Rust nightly and a JDK are required), and the getting-started page describes building from source. |
| Apache DataFusion Comet | Open-source Spark plugin on DataFusion | The project publishes benchmark scripts, a data generation repository and the non-default Spark settings for its TPC-DS run, including 32 executors of 16 cores. | The docs warn Comet may not fully work with cloud-provider Spark forks, so EMR and Databricks are not documented as supported. |
| Apache Gluten with Velox | Open-source Spark plugin offloading to Velox | Gluten needs no DataFrame or SQL changes, only configuration: the GlutenPlugin class, off-heap memory and the columnar shuffle manager. | The docs name Spark 3.4.4, 3.5.5, 4.0.2 and 4.1.1 as officially supported and warn that an unofficial Spark version can throw NoSuchMethodError at runtime. |
| Databricks Photon | Databricks's C++ vectorized engine | Per Databricks's documentation, Photon works with existing Spark API code with no changes, and it is on by default for classic all-purpose and jobs compute. | Photon is documented only as part of Databricks compute and is not offered as a standalone engine for open source Spark or EMR. |
| DataPelago | Gluten-based CPU and GPU accelerator | Per DataPelago's documentation, DPA-S runs in the customer's own cloud account on EKS, Dataproc 2.2.49 with Spark 3.5.1, Dataproc 2.1.75 with Spark 3.3.2, and GKE. | DataPelago's 10X and 80% claims, and its customer quotes of 50% lower cost, carry no baseline, workload or method. |
| DualBird | FPGA Spark plugin on AWS F2 | DualBird publishes a worked example: a sort-based Iceberg compaction benchmark on 100 GB, with the dataset shape (about 370 million rows, 64 columns, ZSTD level 3) and two baselines (vanilla Spark and an unnamed C++ accelerated Spark). | DualBird runs on AWS F2 instances only, so it is not an option on Azure, Google Cloud or a Databricks-managed cluster, and every benchmark published for it so far is its own. |
| Flarion | Rust and DataFusion engine as one JAR | Flarion lists prices: $0.009 per CPU hour for self-hosted Spark, $0.012 for fully managed Spark such as EMR, $0.095 per DBU on Databricks and $0.75 per CPU per month on premises. | Flarion is proprietary and not downloadable (the page says contact us), and no repository or scripts were found for its benchmarks. |
| Google Cloud Lightning Engine | Premium Spark engine on Google Cloud | Google's documentation says no Spark code changes are needed; clusters use image 2.3.3 or later, or 3.0.2 or later, and the flag --engine=lightning. | Lightning Engine is Google Cloud only, and the 3.6x launch figure comes with no scripts, cluster spec or raw timings. |
| Microsoft Fabric native execution engine | Velox-based engine inside Fabric Spark | Per Microsoft's documentation, no code modifications are required, and the engine is enabled with an Acceleration tab toggle or spark.native.enabled. | The engine runs only inside Microsoft Fabric on Azure and does not run on AWS EMR or EKS. |
| NVIDIA cuDF plugin for Apache Spark | GPU plugin, formerly RAPIDS Accelerator | AWS's August 2026 post publishes runtime and cost per run: 281 seconds and $2.06 on GPU versus 1,010 to 1,043 seconds and $2.93 to $3.18 on CPU clusters. | The plugin needs NVIDIA GPUs of the Volta generation or newer, so the cluster moves to GPU instances such as g7.4xlarge. |
| Speedata | PCIe analytics chip plus Dash Spark plugin | Dash plugs into the Spark Catalyst optimizer, and per Speedata it delegates SQL elements it cannot accelerate, such as UDFs, to the CPU. | The APU is a PCIe card, so it needs physical servers; Speedata Acceleration for AWS runs the work in a Speedata-hosted environment, not in the customer's own EC2 fleet. |
Spark acceleration, on this site, means a product or project that makes existing Apache Spark jobs run faster or cheaper while leaving the job code in place. The usual form is a plugin or a runtime setting that takes over the operators it supports, such as scans, joins and aggregations, and hands the rest back to Spark.
The twelve options fall into four groups. Native-engine plugins (Apache Auron, Apache DataFusion Comet, Apache Gluten with Velox, Flarion, DataPelago) run supported operators in C++ or Rust. Hardware accelerators (the NVIDIA cuDF plugin on GPUs, DualBird on FPGAs, Speedata on its own chip) move work off the CPU. Platform runtimes (Databricks Photon, the Amazon EMR runtime, Google Cloud Lightning Engine, the Microsoft Fabric native execution engine) are tied to one provider's service.
Options that need the jobs rewritten, such as moving from Spark to another engine or from RDD code to SQL, are outside this site. Each option page records what the vendor says it changes, where it runs and what has been published about its speed and price, with a source and a read date on every fact.
Data catalogs, lineage, semantic layers and MCP servers for data are outside this site's subject.
Last reviewed 10 Nov 2026.