Apache Spark 4.2.0, History rolling zip follows planted symlinks

Jackson closed the constructor. The zip still follows the inode.

I am @abraxas_null. The proof of concept is on GitHub: abraxas/spark-history-zip-symlink (loopback client). The lab stack is lab/: Dockerfile, docker-compose.yml, run.sh, poc.py, stub.py. Authorized lab only. It talks to loopback.

This is Apache Spark 4.2.0 (32f7299601108917fb01920a54e084595b7b3bf8). No CVE yet. CWE-59 / CWE-22. 7.5 High (CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:N/A:N). Remote file read. Leftover of the CVE-2025-54920 victim model after the Jackson lock.

CVE-2025-54920 taught History Server not to construct HiveConnection out of an Event field. That is the right lesson. The same process still lists a rolling log directory, opens every child, and ships the bytes as eventLogs-{appId}.zip. FileSystem.open follows a POSIX symlink. The zip entry is named after the symlink, so the archive looks like a normal download. The body is whatever the History UID could read.

I did not plant a gadget class. The oracle is a witness string in /secret/host-secret.

Why Spark

I had just finished unpublished Crit/High hunts of other large Java trees. Default installs came back empty or config-gated. Spark is honest in a way that wastes a week if you do not read SECURITY.md first. Unauthenticated job submit on an open cluster is by design. Loading an ML model is loading executable code. spark.authenticate defaults false. Standalone REST defaults on. Spark Connect gRPC has no PSK unless you set one. Hive Thrift NONE is Hive's documented default.

A finding on this hunt needed something else: leftover of a public auth fix, or a walk that reads or writes outside the provisioned application. History Server host files sit outside the application. That is the FAQ exception.

I only wanted Critical or High. Local is in-scope. Medium stays on the floor unless it chains. Hadoop itself is a different tree. Spark 4.3 RCs wait.

Pinning v4.2.0

Latest GA on spark.apache.org as of the review (news 2026-07-14). GitHub also had 4.3.0 RCs and 4.2.1 / 4.0.5 / 4.1.4 RCs. Those wait. Master moves. I do not pin master.

The tag object is v4.2.0. Peel commit 32f7299601108917fb01920a54e084595b7b3bf8, "Preparing Spark release v4.2.0-rc6", 2026-07-11. git describe --tags says v4.2.0. POM 4.2.0. Scala 2.13.

Intake is security@spark.apache.org. Unpaid ASF.

The skip list is half the hunt

Before you look for leftovers, you write down what is already public so you do not "discover" a CVE with a press release. Spark's list lives at spark.apache.org/security.html. On 4.2.0 the interesting ones were already closed:

• CVE-2025-54920 / GHSA-jwp6-cvj8-fw65 / SPARK-52381. History Server Jackson polymorphic deser. Fixed 3.5.7 / 4.0.1. JsonProtocol.sparkEventFromJson only readValues SparkListenerEvent subclasses. HiveConnection-class gadgets throw Unknown event type.
• CVE-2025-55039 RPC AES-CTR without auth. Moderate. GCM knob present. Default still CTR.
• CVE-2026-32773 History stored XSS via app name. Medium. No High chain on this pin.
• CVE-2022-33891 UI HttpSecurityFilter shell via id -Gn. Closed.
• CVE-2023-22946 proxy-user classpath. Default-false gate. Closed unless leftover.
• CVE-2020-9480 standalone master spark.authenticate bypass on the RPC receive path. Closed unless leftover.
• CVE-2018-11770 standalone REST / Mesos REST ignore spark.authenticate.secret. Docs plus default rest.enabled promised false. Closed unless leftover.
• CVE-2018-1334 / CVE-2018-11760 PySpark / SparkR local impersonation. Closed.
• CVE-2017-12612 launcher FilteredObjectInputStream. Closed.
• UI XSS, MHTML, RPC key negotiation, unencrypted local disk: closed or below bar.

Vendor rejects: unauth spark-submit / REST / Connect running a jar on an open cluster. ML load. I did not KEEP those.

Twenty-eight surfaces, one leftover in this post

I split the tree. Standalone REST, RPC auth, Spark Connect session, Connect artifacts, proxy-user, Hive Thrift, Python data sources, SQL data sources, event-log paths, History Jackson, History XSS, kvstore, UI ACLs, UI kill, worker, YARN, Kubernetes, launcher, MLlib, pipelines, PySpark gateway, secret redaction, shuffle ESS, SparkR, SSL/Netty, streaming, UDF worker, Hadoop tokens.

Most of that is Spark doing what Spark said it would do.

Unauth job submit is the product. Connect SQL is the product. Loading PipelineModel is the product. Hive NONE is Hive's default. UI kill with ACLs off is documented. History UI ACLs default off, so listing other people's apps is the documented open UI, not a leftover. Compaction default retain is Integer.MAX_VALUE, so it does not run. Cleaner default off. Utils.classForName still <clinit>s an unknown Event name before the SparkListenerEvent gate fails; no High gadget on this classpath. Compression codec token is the last filename dot-segment, so a FQCN cannot survive split("\\."). Client-side zip slip of an entry named eventlog_v2_.. is a legal single path component and a UI:R extract. Medium. I dropped it.

The Jackson leftover hunt came back NONE. @JsonTypeInfo(Id.CLASS) is still on SparkListenerEvent. The unmatched class walk is closed. kvstore's ObjectMapper has no default typing. HybridStore dumps already-constructed wrappers. History HTTP is GET. There is no request-body readValue of Event JSON.

That is when I stopped staring at Jackson and opened the files History still opens.

The victim model did not move

CVE-2025-54920's attacker is a writer of spark.history.fs.logDirectory. Default is file:/tmp/spark-events. Shared sticky tmp, NFS home, or a job user who can already drop event logs: same person. Rolling writers default on (spark.eventLog.rolling.enabled true), so the on-disk layout is eventlog_v2_<appId>/ with appstatus_* and events_*. History UI ACLs default off, so GET /api/v1/applications/{id}/logs does not ask who you are.

SPARK-52381 locked construct. It did not lock open.

EventLogFileReader.apply accepts any non-directory whose name does not start with . as a single event log, or any directory named eventlog_v2_* that contains an events_* file and an appstatus_* file. The rolling reader lists once:

private lazy val files: Seq[FileStatus] = {
  val ret = fs.listStatus(rootPath).toImmutableArraySeq
  require(ret.exists(isEventLogFile), "Log directory must contain at least one event log file!")
  require(ret.exists(isAppStatusFile), "Log directory must contain an appstatus file!")
  ret
}

eventLogFiles filters. zipEventLogFiles does not.

override def zipEventLogFiles(zipStream: ZipOutputStream): Unit = {
  val dirEntryName = rootPath.getName + "/"
  zipStream.putNextEntry(new ZipEntry(dirEntryName))
  files.foreach { file =>
    addFileAsZipEntry(zipStream, file.getPath, dirEntryName + file.getPath.getName)
  }
}

addFileAsZipEntry is the whole leftover:

protected def addFileAsZipEntry(
    zipStream: ZipOutputStream,
    path: Path,
    entryName: String): Unit = {
  Utils.tryWithResource(fileSystem.open(path, 1 * 1024 * 1024)) { inputStream =>
    zipStream.putNextEntry(new ZipEntry(entryName))
    inputStream.transferTo(zipStream)
    zipStream.closeEntry()
  }
}

Hadoop LocalFileSystem.open follows a POSIX symlink. There is no O_NOFOLLOW. There is no realpath-under-log-dir check. The zip entry name is Path.getName(), so the archive cannot name /etc/passwd. It can still contain /etc/passwd.

Download sink: unauth GET /api/v1/applications/{appId}/logs in OneApplicationResource builds eventLogs-$appId.zip, then FsHistoryProvider.writeEventLogs -> resolveLogPath -> reader.zipEventLogFiles. resolveLogPath prefix-joins the configured log dir with attempt.logPath (the rolling directory basename). The Path object stays under the log dir. The escape is the inode.

getLogDirInfo comments "Uses FileSystem path resolution for security and handles symbolic links properly." I read that comment twice. The helper picks a display name. isDescendantOf walks unresolved Path parents. It does not lstat.

Single-file zip is the same open on rootPath. A symlink whose target is another tenant's event log still parses as SparkListenerApplicationStart and is downloadable. A symlink whose target is not JSON fails listing replay, so the extra-file rolling walk is the host-file read. That is the one I labbed.

The lab

I did not boot the official spark:4.2.0 History Server for this leftover. History bring-up is a cluster's worth of XML for a twelve-line zip dump. The oracle is whether FileSystem.open of a listStatus child follows a planted symlink into the HTTP zip. That is a gadget of EventLogFileReaders.scala ~267-272, not a second Jackson parser.

lab/stub.py plants:

• /tmp/spark-events/eventlog_v2_app-2026-0001/appstatus_app-2026-0001
• events_1_app-2026-0001 with only SparkListenerLogStart and SparkListenerApplicationStart (no witness)
• planted-secret -> /secret/host-secret (witness)

It speaks GET /health, GET /api/v1/applications, and GET /api/v1/applications/{id}/logs. It does not call JsonProtocol. CVE-2025-54920 stays closed in this process because it is never opened.

Loopback 127.0.0.1:18980. Compose project spark-history-zip-symlink. Image python:3.13-slim, UID 1000. poc.py checks:

• health 200
• apps list contains app-2026-0001
• rolling dir listing shows the symlink
• cat /secret/host-secret has the witness
• events_* does not have the witness (otherwise I planted the oracle in the log)
• symlink realpath is outside /tmp/spark-events/
• GET /logs is 200 application/octet-stream
• zip member eventlog_v2_app-2026-0001/planted-secret contains SPARK-HISTORY-ZIP-SYMLINK-WITNESS
• events zip member does not
• no zip entry uses the target path or an absolute name

Last line:

SUCCESS SPARK-HISTORY-ZIP-SYMLINK SPARK-HISTORY-ZIP-SYMLINK-WITNESS

Replay: python3 spark-history-zip-symlink-Abraxas-Labs.py or cd lab && ./run.sh. Stack tears down.

Wrong turns, in the order they happened

I treated SPARK-52381 as the whole History story and almost walked away. The Jackson hunt note says NONE. That is correct for construct. It is not correct for open.

I tried to re-open a HiveConnection Event class on this pin. isAssignableFrom(SparkListenerEvent) throws. The suite already has SPARK-52381: only read classes that extend SparkListenerEvent. That is closed. Planting it in this lab would have been theatre.

I expected the zip entry to carry the target path, so a client extract would write /secret/host-secret. Path.getName() refuses. POSIX filenames cannot contain /. The leftover is the body, not the name. The lab fails if an entry is named after the target.

I treated resolveLogPath prefix-join as the escape. new Path(logDir, name) keeps the Path under the log dir. The confused deputy is open, not new Path.

I planted the witness in events_* on the first draft. The zip then "leaked" a string that was already a log line. poc.py now fails events-file-contains-witness and events-zip-member-contains-witness. The oracle has to live outside the log dir.

I thought about booting full SHS from apache/spark:4.2.0. Scan interval, UI bind, log dir permissions, and a real SparkListenerApplicationStart later. The gadget is the cited Scala. Full SHS would have proven the same open. It would also have mixed this leftover with listing replay, disk store, and XSS. This post is the zip.

I looked at HDFS. Namenode symlinks are off unless you turn them on. Local and NFS follow POSIX. The default log dir is file:/tmp/spark-events. That is the lab FS.

I read getLogDirInfo as a realpath sandbox because the comment said "security" and "symbolic links". Display names are not sandboxes.

I considered compaction as a write primitive. Compact file is originalPath + ".compact" in the same dir. Default retain is MaxValue. Local delete of a symlink unlinks the link. Dropped.

I considered Content-Disposition: filename=eventLogs-$appId.zip header injection from JSON App ID. Medium. Dropped.

I considered a Hadoop Path child with a : scheme (file:foo as a filename). Linux allows :. Basename still cannot contain /. Symlink walk is the file read. Not a separate leftover.

What an attacker can do

Someone who can write the directory History scans can plant a rolling application the UI will zip. The SHS UID is the confused deputy. ACLs default off, so the GET is unauthenticated once the app is listed.

PR:L on the vector is write to the log directory, the same bar CVE-2025-54920 used. Confidentiality is High. Integrity and availability are none. Do not inflate this to host RCE.

The fix

Pack only events_* and appstatus_* children, not every listStatus name. Open with no-follow (O_NOFOLLOW on local, Hadoop no-symlink options on others) and reject a child whose realpath is outside spark.history.fs.logDirectory. A display-name helper is not that check. Single-file readers should refuse a rootPath that is a symlink out of the log dir as well.

I am not going to print a curl you can paste at someone else's History Server. The missing realpath check is the useful part. If you own the SHS, run the proof of concept against loopback.

References

• Proof of concept: abraxas/spark-history-zip-symlink · spark-history-zip-symlink-Abraxas-Labs.py
• Lab: lab/ · Dockerfile · docker-compose.yml · run.sh · poc.py · stub.py
• @abraxas_null · github.com/abraxas · abraxaslabs.tech · abraxas.null@proton.me
• CWE-59 · CWE-22
• Tree: apache/spark v4.2.0 (32f7299601108917fb01920a54e084595b7b3bf8) · EventLogFileReaders.scala · FsHistoryProvider.scala · OneApplicationResource.scala · JsonProtocol.scala
• Nearby patched (Jackson construct, not this inode walk): CVE-2025-54920 / SPARK-52381
• CVE list: spark.apache.org/security.html