Spark History, unsanitized App ID disk store path

The sanitizer stops at the filename

I am @abraxas_null. The proof of concept is on GitHub: abraxas/spark-history-store-path (loopback client). The lab stack is lab/: Dockerfile, docker-compose.yml, run.sh, poc.py, gadget, pin excerpts. Authorized lab only. It talks to loopback.

This is Apache Spark 4.2.0 (32f7299601108917fb01920a54e084595b7b3bf8) History Server. No CVE yet. CWE-22. 7.5 (CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:H/A:N) when spark.history.store.path is set. Remote file write. Config-gated. Contents are a store file, not host RCE.

Spark already has a function named sanitizeDirName. Event-log writers call it. The disk-cache join does not. That is the leftover.

Why Spark

I had just finished unpublished Crit/High hunts of products that either came back empty on default or left a config-gated hole: Prometheus remote-write auth on redirects, PostgreSQL pg_basebackup following a hostile symlink, Kafka Connect FileStream still on the Windows CLASSPATH. Spark is the next large Java tree people treat as "the cluster is trusted, job submit is the product." That sentence is in SECURITY.md. It is also the reason a leftover hunt has to be picky.

Spark's model is honest. spark.authenticate default false means anyone who can reach the Master can submit a job. Loading an ML model is loading executable code. I am not filing the FAQ. A finding on this hunt needed something else: leftover of a public auth fix, History Server host file, shuffle theft across apps, tenant secret, ACL skip when the operator turned the control on.

I only wanted Critical or High. Remote and local. Medium stays on the floor unless it chains. Connectors (Kafka, Kinesis) only if the leftover is this tree. Hadoop itself is a different review.

Vendor intake is security@spark.apache.org. SECURITY.md also names security@apache.org and private@spark.apache.org. Unpaid ASF.

Pinning 4.2.0

Latest GA on spark.apache.org as of the review (news 2026-07-14). Tag v4.2.0. The annotated tag is not the thing you want to checkout. Peel commit 32f7299601108917fb01920a54e084595b7b3bf8, "Preparing Spark release v4.2.0-rc6", 2026-07-11. git describe --tags says v4.2.0. POM 4.2.0. Scala 2.13.

I did not pin master. I did not pin a 4.3 RC. 4.0.5 / 4.1.4 / 4.2.1 / 4.3.0 RCs were on the board and they wait. SPARK-58531 (Connect copy-local) is fixed as static conf on 4.3.0 only. That is a different leftover.

Clone was read-only. Twenty-eight hunts, then ten independent gadget or cluster labs on loopback 127.0.0.1:18900-18990. This pack is G10. Port 18990. Witness SPARK-HISTORY-STORE-PATH-WITNESS.

The skip list is half the hunt

Before you look for leftovers, you write down what is already public so you do not "discover" a CVE with a press release. Spark's list lives at spark.apache.org/security.html. On 4.2.0 the interesting ones were already closed:

• CVE-2025-54920 / SPARK-52381 History Server Jackson polymorphic deser RCE. JsonProtocol.sparkEventFromJson only readValues SparkListenerEvent subclasses. HiveConnection-class gadgets do not construct. Fixed 3.5.7 / 4.0.1. This pin is after that.
• CVE-2025-55039 RPC AES-CTR without auth. Moderate. GCM knob is on this pin. Default still CTR. Below bar.
• CVE-2026-32773 History stored XSS via app name. Medium. killEnabled=false. No session cookie. Below bar.
• CVE-2022-33891 / CVE-2023-32007 UI id -Gn argv shell. Closed.
• CVE-2023-22946 proxy-user classpath. Default-false gate is in. A leftover of that gate is a different pack.
• CVE-2020-9480 standalone master spark.authenticate receive-path. Closed unless leftover on the data plane.
• CVE-2018-11770 standalone REST ignores spark.authenticate.secret. Docs plus default rest.enabled promised false. This pin still defaults REST on. That is a different pack.
• CVE-2018-1334 / CVE-2018-11760 PySpark/SparkR loopback+token. Closed.
• CVE-2017-12612 launcher FilteredObjectInputStream. Closed.
• Older UI XSS, RPC key negotiation, unencrypted local disk, zinc, MHTML.

Vendor rejects, written down before the first hunt: unauth job submit on an open cluster is by design. Loading an ML model is loading executable code. Unauth cluster with spark.authenticate=false is the documented default.

Twenty-eight surfaces, one disk-store join

I split the tree. Standalone REST, RPC auth, Spark Connect artifacts, event logs, History deser, History XSS, Hive Thrift, Kubernetes, kvstore, launcher, MLlib, pipelines, proxy-user, PySpark gateway, Python data sources, secret redaction, shuffle, SparkR, SQL data sources, SSL/Netty, streaming, UDF worker, UI ACLs, UI kill, worker, YARN, Hadoop tokens.

Most of that is Spark doing what Spark said it would do.

Job submit executes user code. Connect SQL executes user code. ML load executes the model. Hive Thrift NONE is Hive's documented default. UI kill with ACLs off is a trusted-network default. K8s hostPath and service-account in a pod template are documented knobs; kube RBAC is the extra-app boundary. Secret-redaction regex gaps are the documented default regex, not a leftover of SPARK-26625. DStream ObjectInputStream checkpoint restores the job's own graph. Launcher prefix over-admit is SPARK-59544 Improvement, no High gadget. kvstore vanilla ObjectMapper has no TypeId. Netty 4.2.13.Final / Jetty 12.1.8. SPARK-52122 ML allowlist is in.

G5 (files/ tar -xzf member ../) looked like a write. GNU tar 1.34 refused ../. The pin argv has no -P. FAIL. DROP.

Jackson History is closed. Compaction default off. Cleaner default off. Utils.classForName(other) still <clinit>s an arbitrary class before the SparkListenerEvent gate. No High gadget on that walk. DROP.

What survived on History was not a second TypeId. It was a path.

The leftover

CVE-2025-54920's victim is History Server pointed at a log directory the attacker can write. Default spark.history.fs.logDirectory is file:/tmp/spark-events. UI ACLs default off. Rolling writers default on. Jackson no longer constructs an arbitrary class from those logs. Listing, open, zip, and (when spark.history.store.path is set) disk-store materialization still trust names that came out of that JSON.

Event-log writers sanitize the filename:

def sanitizeDirName(str: String): String = {
  str.replaceAll("[ :/]", "-").replaceAll("[.${}'\"]", "_").toLowerCase(Locale.ROOT)
}

def nameForAppAndAttempt(appId: String, appAttemptId: Option[String]): String = {
  val base = sanitizeDirName(appId)
  if (appAttemptId.isDefined) {
    base + "_" + sanitizeDirName(appAttemptId.get)
  } else {
    base
  }
}

EventLogFileWriters.nameForAppAndAttempt calls that. The log on disk cannot contain /. Good for the log dir. Irrelevant for the cache dir.

JSON App ID is raw text:

val appId = jsonOption(json.get("App ID")).map(_.asText())

Listing stores it verbatim:

override def onApplicationStart(event: SparkListenerApplicationStart): Unit = {
  app.id = event.appId.orNull
  app.name = event.appName

Disk manager exists only when the operator set a store path:

val LOCAL_STORE_DIR = ConfigBuilder("spark.history.store.path")
  .doc("Local directory where to cache application history information. By default this is " +
    "not set, meaning all history information will be kept in memory.")
  .version("2.3.0")
  .stringConf
  .createOptional

private val diskManager = storePath.map { path =>
  new HistoryServerDiskManager(conf, path, listing, clock)
}

When that Option is empty, getAppUI builds an in-memory store and nothing hits the disk join. Production History often sets the path because a large SHS does not want every app in heap. Docs have recommended it for years. That is the gate.

appStoreDir is {store.path}/apps. Extension is .rdb (default RocksDB) or .ldb (LevelDB, deprecated). Then:

private[history] def appStorePath(appId: String, attemptId: Option[String]): File = {
  val fileName = appId + attemptId.map("_" + _).getOrElse("") + extension
  new File(appStoreDir, fileName)
}

No sanitize. No isChild. No canonical-under-parent check. Unix new File("/work/store/apps", "../../oracle/SPARK-HISTORY-STORE-PATH-WITNESS.rdb") is /work/oracle/SPARK-HISTORY-STORE-PATH-WITNESS.rdb.

Lease.commit uses that File:

def commit(appId: String, attemptId: Option[String]): File = {
  val dst = appStorePath(appId, attemptId)
  // ...
  if (dst.isDirectory()) {
    val size = sizeOf(dst)
    deleteStore(dst)
    updateUsage(-size, committed = true)
  }
  // ...
  tmpPath.renameTo(dst)

If the escaped path already is a directory, History deletes it first (Utils.deleteRecursively), then renames the leased RocksDB dir onto it. Create, replace, delete. As the History UID. Forced suffix .rdb or .ldb. Not attacker-chosen bytes of a .so. Still a write outside the cache dir.

Trigger is any GET that calls getAppUI -> loadDiskStore -> createDiskStore / hybrid lease.commit(appId, attemptId). History HTTP ACLs default off. The attacker does not need to POST a path. They write the event log, wait for scan, then load the app.

URL fallback eventlog_v2_ + nameForAppAndAttempt is sanitized. That path is not this leftover. The leftover is the listing App ID, the one JSON handed History for free.

Building the lab

Full History Server on 4.2.0 is a long image and a long boot. The join is three lines of java.io.File. I copied those lines.

Gadget AppStorePathGadget is a JDK HTTP server on 18990. /ping returns pin headers. /audit greps the pin excerpts for the join, the missing sanitize, optional store.path, asText(), event.appId.orNull, and sanitizeDirName used only on event-log names. /join runs the same new File(appStoreDir, appId + ".rdb") the pin runs, optionally writes, and reports gated / escaped / canonical.

Four cases:

1. Unset storePath. Gadget refuses to write. gated=yes. Default History.
2. storePath=/work/store, App ID local-control. File lands under /work/store/apps/local-control.rdb. Inside.
3. Same store, App ID ../../oracle/SPARK-HISTORY-STORE-PATH-WITNESS, but run through sanitizeDirName first. Stays inside apps/. / became -. Negative control that the helper would have saved this join if anyone had called it.
4. Same store, same ../ App ID, no sanitize. Canonical path /work/oracle/SPARK-HISTORY-STORE-PATH-WITNESS.rdb. escaped=yes. outsideStore=yes. Body is the witness.

Oracle directory /work/oracle already exists. Lease.renameTo does not mkdirs on an escaped parent. The lab does not either. The write is the join plus Files.writeString as a stub of the rename. Witness-only. No reverse shell. No custom RocksDB. The SUCCESS line names the path:

SUCCESS SPARK-HISTORY-STORE-PATH how=gadget escaped=yes gated=yes outside=/work/oracle/SPARK-HISTORY-STORE-PATH-WITNESS.rdb SPARK-HISTORY-STORE-PATH-WITNESS

Compose project spark-history-store-path. Image eclipse-temurin:21-jdk-jammy. User 185, Spark's usual UID. Loopback 127.0.0.1:18990. Stack torn down after the run.

Wrong turns, in order

I expected Unix new File(parent, "/abs") to ignore the parent. Java on Unix does not. An absolute child stays under parent as a weird relative. G2 and G4 had the same correction. Leftover is .., not /abs.

I expected default History to write. Default spark.history.store.path is unset. createOptional. diskManager is None. In-memory. The first join with an empty store path must be gated=yes written=no or the lab is lying about the default.

I almost called a .rdb RCE. It is a RocksDB directory (or, in the gadget, a witness file with that suffix). History will not dlopen it. Inflating File write to RCE is how you get a vendor to stop reading. The class is File write.

I almost booted official apache/spark:4.2.0 History for this one. G1 needed the real Master REST listener. This leftover is new File. Pin excerpts in the image, gadget copies the join, poc audits both. If the pin snippet grows a sanitize, /audit fails closed.

I almost treated sanitizeDirName as covering appStorePath because both live in the History story. The helper is called from EventLogFileWriters. appStorePath concatenates. Reading one function and assuming the other calls it is how sanitizers become folklore.

I almost used the URL appId. HistoryServer splits req.getPathInfo(). That string can contain .. too, but getAppUI has a sanitized fallback for eventlog_v2_ names. The listing App ID from JSON is the one lease.commit actually joins. Plant the log, do not forge the URL.

Theatre already recorded: a reverse shell; a custom native library named .rdb; hitting a production SHS; claiming Linux Docker History is default-on disk store (it is not unless the operator set the path).

What an attacker can do

Attacker is a writer of the History log directory (Spark job user, or any local user if default file:/tmp/spark-events is sticky or world-writable). History process UID is the confused deputy. Operator set spark.history.store.path. UI ACLs default off.

Recon: none beyond the created path. The leftover does not read host files. Sibling zip-symlink leftover of the same victim model is a different pack.

Adjacent closed: default unset store (in-memory). Jackson sparkEventFromJson still requires SparkListenerEvent (CVE-2025-54920). URL getAppUI fallback eventlog_v2_ + nameForAppAndAttempt is sanitized. Listing KV key is not a filesystem path. HybridStore / kvstore mapper has no Id.CLASS. History HTTP does not accept an upload path. Compaction default off. Cleaner default off. Job submit on an open cluster (SECURITY.md FAQ). ML model load (docs/ml-security.md). This leftover is not host RCE.

The fix

Call Utils.sanitizeDirName (or reject .. / / / NUL) on appId and attemptId inside appStorePath before new File. After the join, require dst.getCanonicalFile() stays under appStoreDir.getCanonicalFile(). Refuse the lease otherwise.

Unset spark.history.store.path stays the safe default. Operators who set it for heap reasons should not also donate a relative path from event JSON.

Do not rely on the event-log filename sanitizer to cover a second join. It did not.

Lab replay

python3 spark-history-store-path-Abraxas-Labs.py

or cd lab && ./run.sh. Loopback only: http://127.0.0.1:18990. Pin v4.2.0 32f7299601108917fb01920a54e084595b7b3bf8.

SUCCESS SPARK-HISTORY-STORE-PATH how=gadget escaped=yes gated=yes outside=/work/oracle/SPARK-HISTORY-STORE-PATH-WITNESS.rdb SPARK-HISTORY-STORE-PATH-WITNESS

References

• Pack: github.com/abraxas/spark-history-store-path
• Product: github.com/apache/spark tag v4.2.0 (32f7299601108917fb01920a54e084595b7b3bf8)
• HistoryServerDiskManager.scala appStorePath / Lease.commit
• FsHistoryProvider.scala diskManager / loadDiskStore / onApplicationStart
• JsonProtocol.scala applicationStartFromJson
• Utils.scala sanitizeDirName
• History.scala LOCAL_STORE_DIR
• Closed Jackson walk: CVE-2025-54920 / SPARK-52381
• Spark security: spark.apache.org/security.html
• CWE-22

Abraxas Labs. abraxaslabs.tech · github.com/abraxas · @abraxas_null · abraxas.null@proton.me