Apache Spark Connect archives hash-fragment dest

The fragment was never concatenatePaths

I am @abraxas_null. The proof of concept is on GitHub: abraxas/spark-connect-archive-fragment (loopback client). The lab stack is lab/: Dockerfile, docker-compose.yml, run.sh, poc.py, Gadget.java. Authorized lab only. It talks to loopback.

This is Apache Spark 4.2.0 (32f7299601108917fb01920a54e084595b7b3bf8). No CVE yet. CWE-22. 8.1 High (CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:N/I:H/A:H). Remote File write. Spark Connect client. Not RCE.

SPARK-33530 taught Spark that path#foo means unpack into a directory named foo under SparkFiles. The Connect handler still comments Workaround to keep the fragment. It splits the proto name on the hash / number-sign character. The left side is a zip, and that zip is staged through concatenatePaths. The right side is a URI fragment. SparkContext.addFile(..., isArchive=true) then does new File(root, uri.getFragment), deleteRecursively(dest), and unpack. foo was supposed to be a name. ../oracle is also a fragment.

That is the leftover.

Why Spark

I wanted a large JVM data plane with a security model that is honest on the brochure and still has sockets that grew a second life. Spark's FAQ is unusually clear: if you can submit a job, you can run code. Loading an ML model is loading executable code. An unauthenticated cluster with spark.authenticate=false is the documented default. Those are not findings. They are the product.

A leftover on this hunt had to be something else. A public auth fix that still has a data-plane hole. A testing flag a session can flip. A path join the name sanitizer does not see. History host-file. Cross-app shuffle. Tenant secret. Not "I posted SQL to Connect."

I only wanted Critical or High. Remote preferred. Local is in-scope. Medium stays on the floor unless it chains. Connectors that are not this tree are out. Livy is another repo. 4.3 release candidates wait.

Pinning 4.2.0

Latest GA on spark.apache.org, news 2026-07-14. GitHub will happily check out v4.2.0 onto an annotated tag object that is not a commit. The peel is 32f7299601108917fb01920a54e084595b7b3bf8, "v4.2.0-rc6", 2026-07-11. POM says 4.2.0. Scala 2.13. Apache-2.0.

Master is not the pin. 4.3.0 RCs are not the pin. 4.0.5 / 4.1.4 / 4.2.1 RCs wait. SPARK-58531 (session SET of a local-dest testing flag) is static on 4.3.0 only. This peel still has that class as a different leftover. This post is the archive fragment.

The skip list is half the hunt

spark.apache.org/security.html plus GHSA siblings. On 4.2.0 the interesting public rows were already closed:

• CVE-2025-54920 History Server Jackson polymorphic deser RCE. JsonProtocol wants a SparkListenerEvent. Closed unless a new unmatched walk.
• CVE-2026-32773 History stored XSS via app name. Medium. No High chain on this pin.
• CVE-2025-55039 RPC AES-CTR without auth. Moderate. GCM knob present. Default still CTR.
• CVE-2022-33891 UI HttpSecurityFilter shell via ACL impersonation. id -Gn argv. Closed.
• CVE-2018-1334 / CVE-2018-11760 PySpark / SparkR local impersonation. Loopback plus token. Closed.
• CVE-2017-12612 launcher FilteredObjectInputStream. Closed.
• CVE-2020-9480 standalone Master receive / upload. Closed on the receive path. Data-plane stream open is a different leftover.
• CVE-2018-11770 standalone REST ignores spark.authenticate. Docs plus default rest.enabled promised false. 4.2.0 REST is on by default. Different leftover, different post.
• CVE-2023-22946 proxy-user classpath. Default-false gate. Leftover if the gate still injects.
• SPARK-52122 DefaultParamsReader ML RCE. Allowlist plus safeMLClassLoader on this pin.

Vendor rejects, correctly: unauth job submit on an open cluster. ML model load. I am not filing those.

Twenty-eight surfaces, then this dest

I split the tree. Standalone REST. RPC auth and shuffle ESS. Spark Connect session, artifacts, Python, Arrow. Proxy-user. JDBC serde. Python data sources. History zip and store path. Hive Thrift. Kubernetes. YARN. UI ACLs and kill. SSL / Netty. Launcher. Worker. Streaming. SparkR. Pipelines. Secret redaction. Hadoop tokens. KV store. Event log. MLlib. UDF worker. SQL datasources.

Most of that is Spark doing what Spark said it would do.

Hive Thrift NONE is Hive's documented default. History UI ACLs default off is documented. Kubernetes and YARN in this pin did not give me an unpublished Crit/High walk that was not already a FAQ job. SSL hostname is not a leftover here. Launcher unsafe deser is the 2017 CVE. Unauth Connect ExecutePlan SQL / Python UDF / own-session JAR is job submit. Cross-session jar load is closed: Connect forces spark.sql.artifact.isolation.enabled=true, isolated sessions read addedJars(sessionUUID), not allAddedJars.

The artifacts hunt was the interesting one. concatenatePaths is a real sanitizer. Tests reject .. and absolute names with INVALID_ARTIFACT_PATH. The comment on StagedArtifact is the tell. If you have to work around keeping a fragment, the fragment is not going through the join you just wrote.

How the hash split actually works

SparkConnectAddArtifactsHandler.StagedArtifact (SparkConnectAddArtifactsHandler.scala:202-213):

val (canonicalFileName: String, fragment: Option[String]) =
  if (name.startsWith(s"archives${File.separator}")) {
    val splits = name.split("#")
    assert(splits.length <= 2, "'#' in the path is not supported for adding an archive.")
    if (splits.length == 2) {
      (splits(0), Some(splits(1)))
    } else {
      (splits(0), None)
    }
  } else {
    (name, None)
  }

Only archives/ plus the server File.separator. files/ does not split. jars/ does not split. Two hashes assert. One hash: left is canonicalFileName, right is fragment.

canonicalFileName is staged:

ArtifactUtils.concatenatePaths(stagingDir, path)

concatenatePaths refuses an absolute child, concatenates strings, normalizes, and requires the result still start with basePath/. files/../../oracle/x dies here. archives/../../oracle/x dies here. That is why the artifact name is not the leftover.

fragment is not passed through concatenatePaths. ArtifactManager.addArtifact for archives/ attaches it to the session file-server URI and calls SparkContext.addArchive (ArtifactManager.scala:269-274).

SparkContext.addFile with isArchive=true (SparkContext.scala:1891-1900):

val dest = new File(
  root,
  if (uri.getFragment != null) uri.getFragment else source.getName)
Utils.deleteRecursively(dest)
Utils.unpack(source, dest)

root for a Connect session is SparkFiles.getRootDirectory()/jobArtifactUUID. SPARK-33530 tests sc.addArchive(s"${jarFile.getAbsolutePath}#foo") as a directory name under that root. Official PySpark already copies a non-empty URI fragment into the archive name (python/pyspark/sql/connect/client/artifact.py:223-236). The client is not bending the API. It is using the documented #name workaround.

Executors build the same dest (Executor.scala:1468-1477). Driver first. Later each executor that fetches the archive.

Utils.unpack into that dest is Hadoop FileUtil.unZip for .zip (HADOOP-13649 class). Member zip-slip under a contained dest is not this leftover. The dest itself is.

Wrong turns, in the order I made them

Pinning master. Master has 4.3 work. SPARK-58531 is already static there. A leftover that exists on 4.2.0 and is gone on master is still a leftover on the GA operators run.

Checking out the tag object. git switch v4.2.0 after a shallow clone can land on an annotated tag. The useful SHA is 32f72996.

Treating open Connect SQL as a finding. SECURITY.md FAQ. Drop.

Treating concatenatePaths as covering the fragment. It covers the zip bytes. The comment Workaround to keep the fragment is the opposite of a stay-under-root check.

Unix absolute fragment ignores root. java.io.File(parent, child) documentation says an absolute child ignores parent. On Unix, new File("/lab/root", "/lab/oracle/abs") is /lab/root/lab/oracle/abs. The leading slash is stripped and the child is joined. Parent is ignored on Windows. The hunt write-up claimed absolute dests. The lab said no. The High walk on this pin is #../.

Zip-slip of members. Hadoop 3.5.0 unZip keeps members under dest. I wrote the gadget to reject a member that escaped dest so SUCCESS could not be a zip-slip in disguise. The leftover is dest, not entry names.

Inflating dest write to RCE. A dest that names a path on the driver classpath is an inferred chain. JVM does not reload already-linked classes. A later executor start or a not-yet-loaded class is the realistic overwrite window. The lab writes a witness file. File write. Not RCE.

files/ .tar.gz tar -xzf member escape. Same artifacts hunt, different path. addFile still shells out tar -xzf with cwd = SparkFiles. SPARK-38631 closed Utils.unpack .tar. GNU tar 1.34 on the lab host printed Removing leading ../ from member names and Member name contains '..' and exited 2. Pin argv has no -P. That claim dropped. This post is not that claim.

Running a full Spark Connect cluster for a new File join. The oracle is dest canonical outside root, deleteRecursively of a canary, and INVALID_ARTIFACT_PATH on name concat. A gadget of the pinned Scala is the right size. Official apache/spark:4.2.0 is the right size for REST. Different leftover.

A reverse shell. Theatre. The oracle is SPARK-CONNECT-ARCHIVE-FRAGMENT-WITNESS in pwned.txt outside /lab/root.

The lab

Java gadget of the pinned split, join, delete, unpack. eclipse-temurin:21-jdk-jammy. Runtime uid 1000. Loopback 127.0.0.1:18930. Compose project spark-connect-archive-fragment.

CONTROL: archives/payload.zip unpacks under /lab/root. Witness inside. Fragment absent.

Name concat: files/../../oracle/files-escape.zip and archives/../../oracle/name-escape.zip both 400 INVALID_ARTIFACT_PATH.

ATTACK: name archives/payload-frag.zip#../oracle/SPARK-CONNECT-ARCHIVE-FRAGMENT-WITNESS. Dest canonical /lab/oracle/SPARK-CONNECT-ARCHIVE-FRAGMENT-WITNESS. Outside root. Dest canary deleted. pwned.txt written as uid 1000. Sibling /lab/oracle/canary-keep.txt kept.

Unix absolute: fragment /lab/oracle/abs-SPARK-CONNECT-ARCHIVE-FRAGMENT-WITNESS. Dest /lab/root/lab/oracle/abs-.... Outside false. Abs canary kept.

Last line:

SUCCESS SPARK-CONNECT-ARCHIVE-FRAGMENT how=gadget dest-outside=yes deleteRecursively=yes files-concat=rejected uid=1000 SPARK-CONNECT-ARCHIVE-FRAGMENT-WITNESS

Replay:

git clone https://github.com/abraxas/spark-connect-archive-fragment
cd spark-connect-archive-fragment/lab
./run.sh

Bind it to loopback. The loopback client is lab/run.sh with a banner on it. Stack tears down on EXIT.

What an attacker can do

Reach Spark Connect as a client that can AddArtifacts with archive=True. Default Connect has no PSK (spark.connect.authenticate.token unset). PR:L if someone put a token or a network ACL in front. PR:N on the documented default socket.

Lab-proven vs inferred: hash-fragment dest outside SparkFiles, dest canary deleted, sibling canary kept, name concat rejected, Unix absolute fragment joins under root, uid 1000, dest-contained zip unpack: lab. Executor fetch of the same dest: inferred (same constructor). Overwrite of a later-loaded class on the driver classpath: inferred chain, not demonstrated, not inflated to RCE.

You get the Spark UID, not a new principal, and you do not get ExecutePlan for free (you already had that on open Connect). You get a dest outside SparkFiles, deleted first, then filled with the archive you uploaded. That is enough to plant files the service account can write and to remove a directory that was already there. It is not a shell. I am not going to print a Connect client snippet you can paste at someone else's :15002. The missing stay-under-root check on the fragment is the useful part. If you own the driver, run the proof of concept against loopback.

The fix

Pass the archive fragment through the same stay-under-root check as the artifact name. concatenatePaths already exists. Or reject .. and absolute and keep SPARK-33530 #name as a single directory component under SparkFiles. Do not deleteRecursively a dest that failed that check. Executors use the same constructor. Fix both.

Until then, do not expose Spark Connect to anyone you do not already trust with the Spark UID filesystem. A PSK on spark.connect.authenticate.token raises the bar to a token holder. It does not sanitize the fragment.

References

• Proof of concept: abraxas/spark-connect-archive-fragment · spark-connect-archive-fragment-Abraxas-Labs.py
• Lab: lab/ · Dockerfile · docker-compose.yml · run.sh · poc.py · Gadget.java
• @abraxas_null · github.com/abraxas · abraxaslabs.tech · abraxas.null@proton.me
• CWE-22 · CVSS 8.1
• Tree: apache/spark v4.2.0 (32f7299601108917fb01920a54e084595b7b3bf8) · SparkConnectAddArtifactsHandler.scala · ArtifactUtils.scala · ArtifactManager.scala · SparkContext.scala · Executor.scala · artifact.py
• SPARK-33530 (SparkContextSuite #foo / #bar as SparkFiles directory names)
• Security: spark.apache.org/security.html