Spark 4.2.0 REST still ignores spark.authenticate

The FAQ said turn on spark.authenticate. REST did not get the memo.

I am @abraxas_null. The proof of concept is on GitHub: abraxas/spark-rest-auth-bypass (loopback client). The lab stack is lab/: Dockerfile, docker-compose.yml, run.sh, poc.py, WitnessMain.java, spark-defaults.conf. Authorized lab only. It talks to loopback.

This is Apache Spark 4.2.0 (32f7299601108917fb01920a54e084595b7b3bf8). No CVE yet. CWE-306 / CWE-287. 8.8 High (CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H). Remote privilege leftover of CVE-2018-11770. Unauthenticated HTTP to standalone Master 6066.

Spark's own FAQ is honest: job submit executing user code is by design. An unauthenticated cluster with spark.authenticate=false is the documented default. I am not filing that. The leftover is the other cluster. The operator followed the FAQ, set the shared secret, closed RPC 7077, and left REST at the 4.1 default. REST never asked for the secret. The Worker then handed the secret to the attacker driver so it could join 7077 anyway.

That is not a new CVE-2020-9480 RCE. RPC still has AuthServerBootstrap. This is the HTTP door next to it.

Why Spark

I had just finished a Kafka Connect leftover on Windows CLASSPATH. Spark is the other large ASF cluster product people put on a network and then forget which port is the "real" one. Standalone Master speaks RPC on 7077, UI on 8080, REST on 6066. Operators who read spark.apache.org/security.html turn on spark.authenticate because CVE-2018-17190 told them an unsecured Master will run user code, and CVE-2018-11770 told them REST used to ignore the secret.

I only wanted Critical or High. Remote and local. Medium stays on the floor unless it chains. Loading an ML model is loading executable code. docs/ml-security.md already said that. Hive Thrift NONE is Hive's documented default. Spark Connect without a PSK is the product. A finding on this hunt needed something else: leftover of a public auth fix, a High walk that writes or reads outside the provisioned application, History Server host file, cross-app shuffle, tenant secret, ACL skip.

Standalone REST was the first surface that still answered after the skip list.

Pinning 4.2.0, not a 4.3 RC

Latest GA on spark.apache.org the day I cloned was v4.2.0, news 2026-07-14. GitHub tags offered 4.0.5 / 4.1.4 / 4.2.1 / 4.3.0 RCs. Those wait. Master moves. 4.3 is where SPARK-58531 makes a Connect conf static. That is a different leftover and a different pin.

The tag object is not always a commit. Peel landed on 32f7299601108917fb01920a54e084595b7b3bf8, "Preparing Spark release v4.2.0-rc6". POM says 4.2.0. Scala 2.13. Kubernetes and YARN are in the tree. Mesos is gone as of 4.0.

I cloned that peel and did not update it when a later RC appeared. The lab uses the official spark:4.2.0 image, not a from-source rebuild of master.

The skip list is half the hunt

Before you look for leftovers, you write down what is already public so you do not "discover" a CVE with a press release. Spark's list lives at spark.apache.org/security.html. On 4.2.0 the interesting ones were already closed, or closed unless a new unmatched walk:

• CVE-2025-55039 RPC AES-CTR without auth. Moderate. GCM knob present. Default still CTR.
• CVE-2025-54920 History Jackson polymorphic deser RCE. JsonProtocol requires SparkListenerEvent.
• CVE-2026-32773 History stored XSS via app name. Medium. No High chain.
• CVE-2022-33891 UI HttpSecurityFilter shell via id -Gn argv. Closed on this pin.
• CVE-2023-22946 proxy-user classpath. Default-false gate. A leftover of that gate is a different finding.
• CVE-2020-9480 standalone Master RPC receive/upload bypass. AuthServerBootstrap still installed when spark.authenticate is true. Closed on the RPC path.
• CVE-2018-11770 standalone / Mesos REST ignore spark.authenticate.secret. Docs plus default rest.enabled promised false. Closed unless leftover. That is this post.
• CVE-2018-1334 / CVE-2018-11760 PySpark / SparkR loopback plus token. Closed.
• CVE-2017-12612 launcher FilteredObjectInputStream. Closed.
• CVE-2018-17190 unsecured standalone Master executes user code. By-design adjacent. Mitigation was "enable spark.authenticate."

Vendor FAQ: unauth job submit on an open cluster is by design. Do not re-file "REST runs a jar" against spark.authenticate=false.

Twenty-eight surfaces, then this door

I split the tree. Standalone REST, RPC auth, shuffle ESS, Spark Connect artifacts, SQL datasources, Python datasources, proxy-user, History deser, History XSS, Hive Thrift, Kubernetes, YARN, launcher, MLlib, pipelines, PySpark gateway, SparkR, SSL/Netty, streaming, UDF worker, UI ACLs, UI kill, Worker, Hadoop tokens, secret redaction, event log, kvstore.

Most of that is Spark doing what Spark said it would do.

Connect copyFromLocalToFs.allowDestLocal is a session buildConf on this pin and static in 4.3.0. That is a different leftover. Connect archive #../ fragment is a different leftover. History zip-follows-symlink is a different leftover. RPC getStreamManager .. through new File(dir, fname) is a leftover of 9480's data plane, not this REST listener. I am not bundling them.

Standalone REST is the default-on listener an operator hits when they followed the FAQ.

CVE-2018-11770 promised two things. This pin reversed both.

In 2018, standalone and Mesos REST ignored spark.authenticate.secret. The code fix (SPARK-25088, "fixed" as of 2.4.0) was:

1. Default spark.master.rest.enabled false.
2. require Master fail-closed if the RPC secret was set while REST was on.

The advisory said REST would stay off unless you opted in, and you could not combine the secret with REST at all.

This peel:

MASTER_REST_SERVER_ENABLED is createWithDefault(true) in package.scala:2045-2048. The 4.1 migration guide is explicit: "Since Spark 4.1, Spark Master daemon provides REST API by default. To restore the behavior before Spark 4.1, you can set spark.master.rest.enabled to false."

Fail-closed is absent from Master.scala. The test is the opposite. MasterSuite.scala:253-261:

test("SPARK-57451: Allows REST server and spark.authenticate.secret to be enabled together") {
  val conf = new SparkConf()
    .set(MASTER_REST_SERVER_ENABLED, true)
    .set(AUTH_SECRET, "secret")
  // Creating a Master must not fail when both the REST server and an auth secret are set.
  noException should be thrownBy {
    makeMaster(conf)
  }
}

SPARK-57451 treats REST and the RPC secret as independent concerns. Independence is a product decision. It is also how 11770 comes back: the operator who only set spark.authenticate still has an unauthenticated job-submit HTTP API.

Master.onStart always starts StandaloneRestServer when that flag is true (Master.scala:161-166). Port 6066. Host is spark.master.rest.host or the Master's RPC host. Jetty is plaintext HTTP only. HttpConnectionFactory, no SSL.

The servlet map in RestSubmissionServer.scala:71-79:

• POST /v1/submissions/create
• POST /v1/submissions/kill/{id}
• POST /v1/submissions/killall
• GET /v1/submissions/status/{id}
• POST /v1/submissions/clear
• GET /v1/submissions/readyz

addFilters applies only spark.master.rest.filters (:144-151). Default Nil (package.scala:2067-2072). JWSFilter is a different control: set spark.master.rest.filters=org.apache.spark.ui.JWSFilter plus a secretKey. Tests: missing JWT is HTTP 403. Nothing maps spark.authenticate onto those filters.

RequestSubmitDriver (Master.scala:402-420) has no secret check. REST talks to Master via in-process RpcEndpointRef. SASL on 7077 never sees the HTTP client.

Then the Worker hands over the secret

The 2018 leftover had a cap: a REST-submitted driver might not get executors, because it did not know the RPC secret. That cap does not hold on this launch path.

Worker CommandUtils.buildLocalCommand:

if (securityMgr.isAuthenticationEnabled()) {
  newEnvironment = newEnvironment ++
    Map(SecurityManager.ENV_AUTH_SECRET -> securityMgr.getSecretKey())
}

DriverWrapper then classForName(mainClass).getMethod("main").invoke (DriverWrapper.scala:61-64). Attacker main runs as the Worker UID, with _SPARK_AUTH_SECRET already in the environment.

CLI spark-submit still defaults useRest from the string "false", so the CLI still uses RPC 7077. --kill / --status on spark:// always use REST. The leftover is the server listener, not the submit CLI default.

The lab

Host is macOS. Official spark:4.2.0 image. Two containers, one compose project spark-rest-auth-bypass. Loopback 127.0.0.1:18900 published to Master 6066.

spark-defaults.conf on both Master and Worker:

spark.authenticate true
spark.authenticate.secret lab-rpc-secret
spark.master.rest.enabled true
spark.master.rest.host 0.0.0.0
spark.master.rest.port 6066

REST filters unset. That is the default. JWSFilter is not installed.

The image compiles WitnessMain.java into /opt/lab/witness.jar. main writes SPARK-REST-AUTH-BYPASS-WITNESS and whether _SPARK_AUTH_SECRET was non-empty. Dest is /oracle/..., bind-mounted from ./oracle.

The PoC:

1. GET /v1/submissions/readyz until HTTP 200 success=true.
2. Wait until Worker logs registered with master.
3. Unauth POST /v1/submissions/create with action=CreateSubmissionRequest, appResource=file:/opt/lab/witness.jar, mainClass=lab.WitnessMain. No Authorization header.
4. Poll GET /v1/submissions/status/{id} and the oracle file.

Create returned HTTP 200 success=true submissionId=driver-20261012231831-0000. Status reached RUNNING. Oracle contained the witness and injected=yes.

Last line:

SUCCESS SPARK-REST-AUTH-BYPASS create-http=200 success=true submissionId=driver-20261012231831-0000 injected=yes state=RUNNING SPARK-REST-AUTH-BYPASS-WITNESS

Replay:

git clone https://github.com/abraxas/spark-rest-auth-bypass
cd spark-rest-auth-bypass
python3 spark-rest-auth-bypass-Abraxas-Labs.py

or cd lab && ./run.sh. Bind it to loopback. The loopback client is lab/run.sh with a banner on it.

I used the official image on purpose. A gadget of RestSubmissionServer would have proved the servlet map. It would not have proved that a stock 4.2.0 Master with spark.authenticate=true still schedules DriverWrapper on a live Worker and that the Worker still injects the secret. That is the leftover an operator actually ships.

Wrong turns, in the order I made them

Pinning a 4.3 RC. 4.3.0-rc looks like "latest." It is a release candidate. SPARK-58531 lives there. GA is 4.2.0.

Checking out the tag object. Annotated tags are not commits. The useful SHA is 32f72996.

Treating unauth job submit as a CVE. On spark.authenticate=false it is the product. The FAQ already said so. KEEP only because 11770's auth fix no longer holds when the operator did enable the secret.

Claiming this reopens CVE-2020-9480. 9480 was crafted RPC to receive / upload on an auth-enabled Master. NettyRpcEnv.startServer still installs AuthServerBootstrap. REST is a different in-process Jetty port. Inflating this leftover into a new 9480 RCE is wrong. The data-plane getStreamManager walk is a sibling, not this post.

Treating JWSFilter default-off as the leftover. JWSFilter is documented opt-in. The leftover is that spark.authenticate is not that control and is not fail-closed against REST.

Treating SPARK-57451 as a new CVE by itself. The JIRA says REST and the RPC secret are independent. The unpublished leftover is that independence re-opens 11770 for operators who only set spark.authenticate.

Expecting the CLI to use REST. spark-submit string-defaults useRest=false. I spent a pass confirming RPC submit still wants the secret. It does. The HTTP listener is still on.

Building Spark from the peel. The official spark:4.2.0 image is the thing operators pull. If that image did not show the leftover, the finding would be a from-source curiosity. It showed it.

A reverse shell. Theatre. The oracle is a witness string plus injected=yes. Kill / killall / clear were not exercised. They sit on the same unauth map. I marked them inferred.

Mesos REST. Removed in 4.0. Half of 11770 is gone. The standalone half is not.

What an attacker can do

Reach standalone Master REST on 6066. No Authorization header. spark.authenticate is on. spark.master.rest.filters is unset. At least one Worker is live.

Lab-proven vs inferred: create, secret inject, own-id status, and readyz are lab. Kill / killall / clear are inferred from the same unauth servlet map.

Requires: standalone Master (not YARN/K8s). Operator set spark.authenticate=true. REST left at default (spark.master.rest.enabled true, spark.master.rest.filters unset). Network path to Master 6066. At least one Worker. Not the documented-open cluster (spark.authenticate=false).

You get the Worker UID, not a new Spark CVE-2020-9480 on RPC 7077. You do get attacker main on a cluster the operator thought was gated by the shared secret. You get that secret in the driver environment. You do not need JWS. You do not need UI admin. You do not need --proxy-user.

Adjacent surface that stays closed

The fix

Default spark.master.rest.enabled back to false, or fail-closed when spark.authenticate is on and REST filters are empty. Map spark.authenticate onto REST, or require JWSFilter whenever the RPC secret is set.

Until then, operators who turn RPC auth on still have to set spark.master.rest.enabled=false or install spark.master.rest.filters=org.apache.spark.ui.JWSFilter plus a secretKey. Leaving REST at the 4.1 default is the leftover.

I am not going to print a curl you can paste at someone else's :6066. The missing mapping is the useful part. If you own the Master, run the proof of concept against loopback.

References

• Proof of concept: abraxas/spark-rest-auth-bypass · spark-rest-auth-bypass-Abraxas-Labs.py
• Lab: lab/ · Dockerfile · docker-compose.yml · run.sh · poc.py
• @abraxas_null · github.com/abraxas · abraxaslabs.tech · abraxas.null@proton.me
• CWE-306 · CWE-287
• Tree: apache/spark v4.2.0 (32f7299601108917fb01920a54e084595b7b3bf8) · package.scala · Master.scala · RestSubmissionServer.scala · CommandUtils.scala · DriverWrapper.scala · NettyRpcEnv.scala
• CVE-2018-11770 · CVE-2020-9480 · CVE-2022-33891 · CVE-2018-17190
• Security FAQ: spark.apache.org/security.html