Spark RPC StreamRequest still reads host files

The lock went on the door that starts a driver

I am @abraxas_null. The proof of concept is on GitHub: abraxas/spark-rpc-stream-unauth (loopback client). The lab stack is lab/: Dockerfile, docker-compose.yml, run.sh, poc.py, Gadget.java. Authorized lab only. It talks to loopback.

This is Apache Spark 4.2.0 (32f7299601108917fb01920a54e084595b7b3bf8). No CVE yet. CWE-862 / CWE-22. 7.5 High (CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N). Remote. Privilege leftover / File read. Operator-enabled spark.authenticate. Not 9480 RCE.

In 2020 someone noticed that a standalone Master with spark.authenticate=true would still start a driver if you sent the right OneWayMessage. CVE-2020-9480 is the Important that followed. The patch is a wrapper: AbstractAuthRpcHandler. Unauthenticated receive() throws. Unauthenticated receiveStream() throws. RequestSubmitDriver and RegisterApplication sit behind that gate. Six years later, on 4.2.0, those throws still fire. I watched them fire. HTTP 401, body Unauthenticated call to receive()., header x-auth-gated=true.

Then I asked the wrapper for its stream manager.

It handed me the real one.

Why Spark

I had just finished a Windows leftover on Kafka Connect: Unix kafka-run-class.sh dropped connect-file after KAFKA-13748. The .bat never got the exclude. Unauth REST plus FileStream is a file API. Spark is the next large ASF Java tree with the same kind of honesty in its docs. Unauthenticated job submit on a cluster with spark.authenticate=false is by design. SECURITY.md says so. Loading an ML model is loading executable code. docs/ml-security.md says so. A finding on this hunt needed something else: an operator-enabled control that still leaks, a leftover of a public auth fix, a host file the provisioned application was not supposed to serve, shuffle theft across apps, a tenant secret.

Spark's RPC story is the 9480 story. The FAQ says "turn spark.authenticate on." Operators who followed that FAQ deserve the control they think they bought.

I only wanted Critical or High. Remote preferred. Medium stays on the floor unless it chains. Job submit executing user code is out. History XSS is out. UI kill with ACLs off is out. Hive Thrift NONE is Hive's documented default. Kubernetes hostPath in a pod template is a kube RBAC conversation.

Pinning 4.2.0, not a 4.3 RC

spark.apache.org listed 4.2.0 as the current GA (news 2026-07-14). GitHub also offered 4.3.0 RCs, 4.2.1 RCs, 4.1.4 RCs, 4.0.5 RCs. Those wait. Master waits. A leftover that only exists on an RC is a moving target. A leftover that only exists on master is a pull request.

The annotated tag is not always the commit you want. Peel is 32f7299601108917fb01920a54e084595b7b3bf8, "Preparing Spark release v4.2.0-rc6", 2026-07-11. git describe --tags says v4.2.0. POM 4.2.0. Scala 2.13. Apache-2.0.

Clone stayed read-only. Twenty-eight hunts, then labs. Loopback 127.0.0.1:18900 through 18990. This leftover is 18910.

The skip list is half the hunt

Before you look for leftovers, you write down what already has a press release so you do not "discover" it again.

On 4.2.0 the interesting public rows were already closed on their original walks:

• CVE-2020-9480 Important. Standalone Master crafted RPC starts application resources without the secret. AuthRpcHandler / SaslRpcHandler extend AbstractAuthRpcHandler. Unauth OneWayMessage throws. Unauth UploadStream throws. Unauth RpcRequest stays in doAuthChallenge until the shared secret unwraps. Master RequestSubmitDriver is receiveAndReply. RegisterApplication is receive. Both need a deserialized body after that gate. DROP as a re-file of 9480 RCE. KEEP only an unmatched walk.
• CVE-2021-38296 handshake key recovery. This pin's AuthEngine is X25519 plus AES-GCM wrapping of ephemeral public keys under HKDF(PSK). Replay of a sniffed challenge without the PSK cannot derive the session cipher. DROP.
• CVE-2025-55039 / GHSA-6p6v-m64v-jx8q Moderate. RPC AES-CTR without authentication. 3.4.4 / 3.5.2 / 4.0.0 added GCM as an option. Default remains AES/CTR/NoPadding. Documented. Same Moderate class. DROP as unpublished High.
• CVE-2025-54920 History Jackson RCE. JsonProtocol still requires SparkListenerEvent. Closed.
• CVE-2026-32773 History XSS. Medium. No High chain on this pin.
• CVE-2022-33891 UI shell via id -Gn argv. Closed.
• CVE-2018-1334 / CVE-2018-11760 PySpark / SparkR loopback plus token. Closed.
• CVE-2017-12612 launcher FilteredObjectInputStream. Closed.
• CVE-2018-11770 REST ignores spark.authenticate.secret. Different surface. Different leftover. Not this post.
• CVE-2023-22946 proxy-user classpath. Default-false gate is still --conf-writable, and that is also a different leftover. Not this post.
• spark.authenticate default false. Documented trusted-network default. Unauth job submit on an open cluster is CVE-2018-17190 adjacent and by design. DROP.
• spark.network.crypto.authEngineVersion default 1 skips a final HKDF. Documented. v2 recommended. Hygiene, not 38296. DROP.
• SASL fallback (spark.network.crypto.saslFallback default true). Failed AuthMessage parse falls back to DIGEST-MD5, which still needs the PSK. Downgrade, not a skip. DROP.

Vendor rejects: unauth spark-submit / Connect SQL / ML model load. I am not filing those.

Twenty-eight surfaces, one data-plane hole

I split the tree. Standalone REST, RPC auth, Spark Connect artifacts, Connect copy-to-local, tar unpack, proxy-user classpath, JDBC SERDE, Python Data Sources, History zip, History store path, ESS / BlockManager, UI ACLs, UI kill, Hive Thrift, Kubernetes, YARN, launcher, MLlib, pipelines, PySpark gateway, SparkR, secret redaction, SSL / Netty, streaming, UDF workers, kvstore, Hadoop tokens, event log.

Most of that is Spark doing what Spark said it would do.

REST on 6066 still ignores spark.authenticate. That is the 11770 leftover, and it is a different finding. Connect session SET of copyFromLocalToFs.allowDestLocal is a different finding. History zip following a planted symlink is a different finding. This post is the RPC data plane.

The RPC-auth hunt was supposed to close the book on 9480. NettyRpcEnv.startServer still installs AuthServerBootstrap when securityManager.isAuthenticationEnabled(). The wrapper still exists. I read receive. I read receiveStream. Then I read the methods the wrapper did not wrap.

Three lines that do not ask who you are

AbstractAuthRpcHandler.java lines 85-87:

@Override
public StreamManager getStreamManager() {
  return delegate.getStreamManager();
}

No isAuthenticated check. getMergedBlockMetaReqHandler() is the same shape at 108-111.

TransportRequestHandler.handle is a type switch. ChunkFetchRequest goes to ChunkFetchRequestHandler. RpcRequest and OneWayMessage go to rpcHandler.receive. UploadStream goes to receiveStream. StreamRequest goes to processStreamRequest. That method never calls receive. It does:

streamManager.checkAuthorization(reverseClient, req.streamId);
buf = streamManager.openStream(req.streamId);

Default StreamManager.checkAuthorization is a no-op. SPARK-57889 taught StreamRequest to call the string-id variant so ESS would authorize the same way as chunk fetch. OneForOneStreamManager.checkAuthorization still does:

if (client.getClientId() != null) {
  // require clientId.equals(state.appId)
}

A client that skipped SASL has clientId == null. The suite documents that as the auth-off path. It is also the never-authenticated path when auth is on.

Driver RpcEnv does not use OneForOneStreamManager for /classes. It uses NettyStreamManager. That class does not override checkAuthorization at all.

new File(dir, fname) is not a jail

openStream splits the attacker string:

val Array(ftype, fname) = streamId.stripPrefix("/").split("/", 2)
val file = ftype match {
  case "files" => files.get(fname)
  case "jars" => jars.get(fname)
  case other =>
    val dir = dirs.get(ftype)
    require(dir != null, s"Invalid stream URI: $ftype not found.")
    new File(dir, fname)
}

files and jars are map lookups of a basename. SparkContext.addFile / addJar publish exact names. You cannot join a path through those maps. Lab: files//oracle/secret is 404.

Any other ftype that was registered via addDirectory does new File(dir, fname) with no canonical-prefix check. If the file exists, it comes back as FileSegmentManagedBuffer.

Who registers a directory?

• REPL: SparkContext sees spark.repl.class.outputDir and calls fileServer.addDirectory("/classes", new File(path)) (SparkContext.scala:504-508). spark-shell does this.
• Spark Connect: ArtifactManager registers artifacts (ArtifactManager.scala:62-66).

Standalone master RpcEnv typically has neither. This host-file walk is a driver / REPL / Connect process, not port 7077 on a bare Master. ESS and BlockManager are the shuffle half, inferred, same ungated getStreamManager().

The JavaDoc lied to me, or I lied to myself

The hunt note said: Java treats an absolute child as the whole path. new File(dir, "/etc/passwd") ignores dir. Unauth StreamRequest therefore reads Spark-UID files. Confidence 84%.

JavaDoc for File(File parent, String child) does say that. On Windows, C:\oracle\secret as the child really does ignore the parent. On Unix, UnixFileSystem.resolve concatenates:

parent="/opt/repl-classes"
child="/oracle/secret"
result="/opt/repl-classes/oracle/secret"

I found that out the way you find out anything you were too sure of. The lab returned 404. Header x-opened-path=/opt/repl-classes/oracle/secret. Header x-parent-ignored=false. Body Stream 'classes//oracle/secret' was not found. Witness not in the body. That is a closed walk, recorded as a closed walk, because the hunt was wrong about Unix.

The leftover that actually fires is classes/../../oracle/secret. new File("/opt/repl-classes", "../../oracle/secret") is /opt/repl-classes/../../oracle/secret. Canonical /oracle/secret. file.isFile() is true. Bytes come back.

Relative .. through an unsanitized join is the oldest path bug that still ships. I almost missed it because I had already written "absolute child" into the KEEP and wanted that sentence to be the finding. The lab is the adult in the room.

What I almost filed instead

Re-filing 9480 RCE. Unauth receive() still throws. Unauth receiveStream() still throws. Authed receiveStream() with the lab secret returns receiveStream-ok. The control plane is the patch. The data plane is the leftover. Inflating a file read into job submit is how you get a WONTFIX that you deserved.

Unauth shuffle with spark.authenticate=false. Documented default. SECURITY.md FAQ. Not a leftover of a control the operator turned on.

files/<escape> as a join. It is a ConcurrentHashMap.get. 404. Do not claim a map lookup is new File.

Standalone Master:7077 host-file read. Master does not register /classes unless someone pointed a REPL at it. The victim is a driver RpcEnv, a spark-shell, a Connect driver, or ESS.

ESS shuffle drain as the primary. OneForOneStreamManager stream ids start at Random.nextInt(Integer.MAX_VALUE)*1000 and then increment. Blind enumeration of that space is not a practical High walk. A neighbor sample (own OpenBlocks, on-path SASL auth not auth-conf, small window around N) makes drain real. I did not speak live ESS. I marked it inferred. The lab-proven High is the .. File join.

Booting official spark:4.2.0 for this gadget. G1 REST needed a real Master and Worker because the leftover is servlet wiring plus _SPARK_AUTH_SECRET inject. G2 is three copied functions. A faithful gadget of pin source is enough. HTTP stands in for Netty framing so the oracle is a status and a body, not a handwritten Spark protocol client. The SUCCESS line still names the leftover. It does not pretend the gadget is a cluster.

AES-CTR default as High. CVE-2025-55039 already called it Moderate. GCM is a knob. I am not renaming a Moderate.

AuthEngine v1 HKDF skip as 38296. It is not.

The lab

Host is macOS. One container. eclipse-temurin:21-jdk-jammy. User lab uid 65532. Compose project spark-rpc-stream-unauth. Loopback 127.0.0.1:18910.

The image plants two files. /opt/repl-classes/hello.txt is IN-DIR-OK. /oracle/secret is SPARK-RPC-STREAM-UNAUTH-WITNESS. Gadget.java copies AbstractAuthRpcHandler.getStreamManager / receive / receiveStream and NettyStreamManager.openStream. It registers classes on /opt/repl-classes and files/hello.txt on the map. com.sun.net.httpserver.HttpServer is the stand-in for TransportRequestHandler.

The PoC, in order:

1. GET /ping 200 pong.
2. Unauth POST /rpc/receive 401, body Unauthenticated call to receive()., x-auth-gated=true.
3. Unauth POST /rpc/receive-stream 401, body Unauthenticated call to receiveStream()..
4. GET /stream?id=classes/hello.txt 200 IN-DIR-OK, x-authenticated=false, x-parent-ignored=false. Witness not in body.
5. GET /stream?id=files/hello.txt 200 map get. Witness not in body.
6. GET /stream?id=files//oracle/secret 404.
7. GET /stream?id=classes//oracle/secret 404. Unix absolute child stayed under the registered dir. x-opened-path=/opt/repl-classes/oracle/secret.
8. GET /stream?id=classes/../../oracle/secret 200. x-lab-how=gadget. x-auth-gated=false. x-authenticated=false. x-ftype=classes. x-fname=../../oracle/secret. x-canonical-path=/oracle/secret. x-escaped=true. Body contains SPARK-RPC-STREAM-UNAUTH-WITNESS.
9. Authed POST /rpc/receive-stream with the lab secret 200 receiveStream-ok.

Last run:

unauth-receive status=401 gated='true'
unauth-receiveStream status=401
unix-absolute status=404 opened='/opt/repl-classes/oracle/secret'
leftover status=200 how='gadget' gated='false' auth='false'
  ftype='classes' fname='../../oracle/secret'
  canonical='/oracle/secret' escaped='true'
  body='SPARK-RPC-STREAM-UNAUTH-WITNESS'
SUCCESS SPARK-RPC-STREAM-UNAUTH SPARK-RPC-STREAM-UNAUTH-WITNESS

Replay:

git clone https://github.com/abraxas/spark-rpc-stream-unauth
cd spark-rpc-stream-unauth
python3 spark-rpc-stream-unauth-Abraxas-Labs.py

Or cd lab && ./run.sh. Bind it to loopback. The loopback client is lab/run.sh with a banner on it.

Wrong turns, in the order I made them

Pinning master. Master had 4.3 work and SPARK-58531 already static on a Connect flag that is still session-scoped on 4.2.0. The pin is the GA tag.

Pinning a 4.3 RC. Same temptation as Kafka 4.4.0-rc3. GA is 4.2.0.

Treating default-off auth as the bug. I spent a morning confirming unauth shuffle with spark.authenticate=false is the product. Spark already told everyone that.

Treating 9480's receive-path as still open. It is not. The PoC fails if unauth receive() is not 401. That check is there so nobody can pretend I missed the patch.

Absolute child as the High walk. JavaDoc, Windows semantics, hunt KEEP. Unix lab 404. I corrected the hunt writeup. The leftover is ...

files/ join. Map get. 404. Same correction.

Calling a gadget a cluster. The gadget copies the sinks. It does not boot a Worker. The headers say x-lab-how=gadget so the last-run file cannot be quoted as "we exploited spark-shell." We exploited the join and the ungated getter.

ESS as lab-proven. I did not send a ChunkFetchRequest at a live External Shuffle Service. Stream-id theft stays inferred. Confidence 80% on that half. The file read is 84% source plus lab SUCCESS.

A reverse shell. Theatre. The oracle is a witness string in /oracle/secret.

Claiming Master:7077 is this bug. Master REST is the 11770 leftover. Master RPC control plane is 9480, closed. This leftover needs a registered directory.

What an attacker can do

TCP to a Spark RPC port that has spark.authenticate=true and a registered directory. No shared secret. No Authorization header. No spark.network.crypto.enabled. Send a transport StreamRequest (message type 6) with streamId classes/../../<path>. The bytes of that path come back if the Spark UID can read them.

That is a host-file read. Config files. Keytabs sitting on disk. spark-env.sh. Anything the process user can open. It is not a write. It is not Runtime.exec. It is not RequestSubmitDriver.

Confidentiality (lab-proven)

Confidentiality (inferred)

Closed on this leftover (lab-proven)

Lab-proven vs inferred: unauth .. file read, Unix absolute 404, files/ 404, gated receive / receiveStream are lab. ESS shuffle drain is inferred.

Adjacent closed (do not inflate):

• CVE-2020-9480 RCE: unauth RequestSubmitDriver / RegisterApplication. Still needs the shared secret.
• Default spark.authenticate=false unauth shuffle and job submit (FAQ).
• Authenticated app A fetching app B (AppIsolationSuite / SPARK-57889 non-null clientId).
• CVE-2021-38296 handshake.
• CVE-2025-55039 Moderate CTR default.
• Standalone Master without a registered directory.
• Does not execute the file. Does not write. Does not dump spark.authenticate.secret. Does not complete AuthEngine or SASL.

Reach is Remote because the RPC port is a TCP service. Victim is whoever followed the FAQ and turned authentication on, then ran a REPL or Connect driver that registered a directory. Privilege is the Spark UID's read set. Class stays Privilege leftover / File read.

The fix

Gate getStreamManager() and getMergedBlockMetaReqHandler() until isAuthenticated, or return a manager that fails closed. In NettyStreamManager.openStream, resolve fname under dir with getCanonicalFile and reject anything that does not stay under the registered root. On ESS, treat null clientId as unauthenticated when spark.authenticate is on. SPARK-57889 already calls checkAuthorization on StreamRequest. The skip is the leftover.

Until then, do not register /classes or artifacts on an RpcEnv whose TCP port is reachable from anyone you do not already trust with the Spark UID's files. spark.authenticate=true does not close this data plane. Network ACLs on the driver RPC port still help. They are not the patch.

I am not going to print a Netty frame you can paste at someone else's driver. The ungated getter and the File join are the useful part. If you own the cluster, run the proof of concept against loopback.

References

• Proof of concept: abraxas/spark-rpc-stream-unauth · spark-rpc-stream-unauth-Abraxas-Labs.py
• Lab: lab/ · Dockerfile · docker-compose.yml · run.sh · poc.py · Gadget.java
• @abraxas_null · github.com/abraxas · abraxaslabs.tech · abraxas.null@proton.me
• CWE-862 · CWE-22
• Tree: apache/spark v4.2.0 (32f7299601108917fb01920a54e084595b7b3bf8)
• AbstractAuthRpcHandler.java · TransportRequestHandler.java · NettyStreamManager.scala · OneForOneStreamManager.java · SparkContext.scala
• CVE-2020-9480 · SPARK-57889
• Vendor: spark.apache.org/security.html