Spark proxy-user cluster still loads defaultExtraClassPath

The gate still closes. The jar is already on -cp.

I am @abraxas_null. The proof of concept is on GitHub: abraxas/spark-proxyuser-classpath (loopback client). The lab stack is lab/: Dockerfile, docker-compose.yml, run.sh, poc.py, gadget. Authorized lab only. It talks to loopback.

This is Apache Spark 4.2.0 (32f7299601108917fb01920a54e084595b7b3bf8). No CVE yet. Leftover of CVE-2023-22946 / SPARK-41958. CWE-269 / CWE-427. 8.8 High (CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H). Remote. Privilege leftover. Livy-style tenant versus the gateway Unix UID.

In 2023 Spark taught spark-submit --proxy-user a lesson. Cluster mode plus a proxy user plus --jars used to put tenant jars on the submit JVM. That JVM is the Livy host. Hadoop doAs changes the Hadoop identity. It does not change the Unix UID. SPARK-41958 defaulted spark.submit.proxyUser.allowCustomClasspathInClusterMode to false and emptied childClasspath before runMain loaded anything. The public CVE is that --jars walk. On 4.2.0 clear() still runs.

Spark 4.0 then added spark.driver.defaultExtraClassPath. The launcher always prepends it to the spark-submit -cp. Including cluster mode. SPARK-41958 never looks at that list. The gate still closes. The jar is already on -cp.

Why Spark

I had been walking large Java trees whose security model looks like a hole until you read the FAQ. Kafka Connect had a Windows CLASSPATH leftover of a Unix fix. MySQL had SET ROLE ALL EXCEPT leaving FILE in m_master_access. Spark was next because it is the other half of that story: a cluster that is honest about job submit being code execution, and a long skip list of public CVEs that were supposed to have finished the proxy-user classpath job.

Spark's model is also honest. Unauthenticated job submit on an open cluster is by design. Loading an ML model is loading executable code. spark.authenticate defaults false. Standalone REST defaults on. A finding on this hunt needed something else: leftover of a public auth fix, a walk that writes or reads outside the provisioned application, a History Server host file, a tenant secret, an ACL skip when the operator turned the control on.

I only wanted Critical or High. Local is in-scope. Medium stays on the floor unless it chains. Livy, Kyuubi, and Oozie are other trees. This leftover is in Spark's submit path. Those gateways are how a tenant reaches it.

Pinning 4.2.0

Latest GA on spark.apache.org as of the review, news 2026-07-14. GitHub tags offered 4.3.0 RCs and 4.2.1 / 4.1.4 / 4.0.5 RCs. Those wait. Master is not a pin. The annotated tag is not a commit either. Git will tell you, with the patience of a librarian who has had this conversation before. Peel landed on 32f7299601108917fb01920a54e084595b7b3bf8, v4.2.0-rc6, 2026-07-11. POM says 4.2.0. Scala 2.13. Clone is that tree, read-only.

SPARK-58531 (a Connect copyFromLocalToFs leftover) is static on 4.3.0 only. That is a sibling. This post is G6.

The skip list is half the hunt

Before you look for leftovers, you write down what is already public so you do not "discover" a CVE with a press release. Spark's list lives at spark.apache.org/security.html. On 4.2.0 the interesting ones were already closed:

• CVE-2025-55039 RPC AES-CTR without auth. Moderate. GCM knob present. Default still CTR.
• CVE-2025-54920 History Server Jackson polymorphic deser RCE. JsonProtocol still requires SparkListenerEvent.
• CVE-2026-32773 History stored XSS via app name. Medium. No High chain.
• CVE-2022-33891 / CVE-2023-32007 UI id -Gn shell. Closed.
• CVE-2023-22946 proxy-user classpath. Fixed 3.4.0 by allowCustomClasspathInClusterMode default false. Closed unless leftover. This post is that leftover.
• CVE-2020-9480 standalone Master spark.authenticate bypass on receive/upload. Closed unless leftover (data plane is a sibling).
• CVE-2018-11770 standalone REST ignores spark.authenticate.secret. Closed unless leftover (REST default-on is a sibling).
• CVE-2018-1334 / CVE-2018-11760 PySpark/SparkR local impersonation. Closed.
• CVE-2017-12612 launcher FilteredObjectInputStream. Closed.
• CVE-2018-17190 unsecured standalone executes user code. By design. Adjacent.

Vendor rejects: unauth job submit / driver code exec on an open cluster is the FAQ. Loading an ML model is loading executable code (docs/ml-security.md). I did not KEEP those as unpublished RCE.

Twenty-eight surfaces, one leftover of 22946

I split the tree. Standalone REST, RPC auth, Spark Connect artifacts, event logs, Hadoop tokens, History deser, History XSS, Hive Thrift, Kubernetes, kvstore, launcher, MLlib, pipelines, proxy-user, PySpark gateway, Python data sources, secret redaction, shuffle ESS, SparkR, SQL datasources, SSL/Netty, streaming, UDF worker, UI ACLs, UI kill, worker, YARN.

Most of that is Spark doing what Spark said it would do.

Job submit on an open cluster is code execution. Connect SQL is the same. Hive Thrift NONE is Hive's documented default. UI kill with ACLs off is a trusted-network default. Kubernetes hostPath and service-account knobs in a pod template are documented. Jackson History RCE is closed. UI shell leftover is closed. PySpark loopback+token is closed. Unix new File(dir, "/abs") stays under the parent; the High walks on those surfaces are .., and they are other leftovers.

The YARN hunt almost dropped this one. CVE-2023-22946's cluster-mode --jars walk is gated. childClasspath.clear() runs. runMain asserts empty classpath when the flag is false. Client-mode driver-as-superuser is documented as not a CVE. Shuffle auth is off by default and that is the FAQ. I almost wrote "closed unless leftover" and moved on.

Then I opened the launcher.

SPARK-41958 still does its job

SparkSubmit.scala prepareSubmitEnvironment still computes:

val isCustomClasspathInClusterModeDisallowed =
  !sparkConf.get(ALLOW_CUSTOM_CLASSPATH_BY_PROXY_USER_IN_CLUSTER_MODE) &&
  args.proxyUser != null &&
  (isYarnCluster || isStandAloneCluster || isKubernetesCluster)

YARN cluster still copies the primary jar and --jars onto childClasspath (:738-743). Then:

if (childClasspath.nonEmpty && isCustomClasspathInClusterModeDisallowed) {
  childClasspath.clear()
  logWarning(...)
}

runMain asserts the empty list (:980-983) before addJarToClasspath. The flag still defaults false (package.scala:2849-2854). k8s cluster plus proxy-user tests still expect length 0. Public CVE-2023-22946 remains this path. The lab control walk is that path. --jars with the flag false: childClasspath-cleared=yes, constructed -cp has no attacker jar.

.internal() on the flag is docs-only. ConfigBuilder.internal() sets _public = false. SparkSubmitArguments stores --conf first (:136-137, :474-476) and toSparkConf still conf.set (:672-676). A tenant --conf spark.submit.proxyUser.allowCustomClasspathInClusterMode=true turns the gate off. YARN cluster then keeps --jars. That is a second walk. It is also the walk the original advisory told gateways to drop. I labbed it because --conf still writes the key. I am not re-filing "3.3.x loads --jars on submit."

Then 4.0 added a key the gate never saw

SparkSubmitCommandBuilder.java:317-327:

boolean isClientMode = isClientMode(config);
String extraClassPath = isClientMode ? config.get(SparkLauncher.DRIVER_EXTRA_CLASSPATH) : null;
String defaultExtraClassPath = config.get(SparkLauncher.DRIVER_DEFAULT_EXTRA_CLASS_PATH);
if (extraClassPath == null || extraClassPath.trim().isEmpty()) {
  extraClassPath = defaultExtraClassPath;
} else {
  extraClassPath += File.pathSeparator + defaultExtraClassPath;
}

List<String> cmd = buildJavaCommand(extraClassPath);

Cluster mode sets extraClassPath to null. That is the SPARK-41958-era omit of spark.driver.extraClassPath / --driver-class-path. The suite even asserts a cluster command has no /driver entry. Then the 4.0.0 key is concatenated anyway. buildJavaCommand puts it on the spark-submit JVM -cp. First in buildClassPath. Before SparkSubmit starts. Before childClasspath.clear(). Before Hadoop UserGroupInformation.createProxyUser.

The key is .internal() (package.scala:80-85). --conf still writes it (SparkSubmitCommandBuilder.java:572-576). --driver-default-class-path is the CLI alias. Livy stages uploads on the gateway disk. After staging, --jars is a file: path. defaultExtraClassPath is the same idea with a different name: a local jar the gateway JVM can open.

Hadoop Configuration resource overlay and FileSystem impl lookup then run as the gateway UID. That is the public CVE-2023-22946 mechanism. I did not re-run a Hadoop overlay. I loaded a witness class from the constructed -cp and wrote a string. Same classloader, same UID, less theatre.

spark.driver.extraClassPath in cluster is omitted. Lab: extraClassPath-omitted=yes. That close still holds. The leftover is the other key.

The lab

Host is macOS. One container. Gadget of the cited Scala/Java, not a full Spark distribution. Official spark:4.2.0 is a sibling leftover. This one does not need a YARN RM. It needs the launcher -cp builder and prepareSubmitEnvironment to agree with the pin.

Image is eclipse-temurin:21-jdk-jammy. Attacker jar is lab.WitnessMain packed as /opt/lab/evil.jar. It is not on the gadget classpath. It only lands via constructed -cp. Readiness HTTP on the Docker bridge, published 127.0.0.1:18950. Oracle volume ./oracle to /work/oracle. Compose project spark-proxyuser-classpath.

Four walks in one java -cp /opt/lab/classes lab.LabMain:

1. Control. --proxy-user tenant --master yarn --deploy-mode cluster --jars /opt/lab/evil.jar. Flag default false. childClasspath.clear(). Constructed submit -cp has no evil jar. Witness class does not load. This is public CVE-2023-22946, closed.
2. Closed extraClassPath. Same plus --conf spark.driver.extraClassPath=/opt/lab/evil.jar. Cluster mode omits it from submit -cp. No load. The 3.4 omit still holds.
3. Leftover defaultExtraClassPath. Same plus --conf spark.driver.defaultExtraClassPath=/opt/lab/evil.jar. Gate still closed. childClasspath still cleared. Submit -cp is /opt/lab/evil.jar. Child JVM loads lab.WitnessMain. Writes SPARK-PROXYUSER-CLASSPATH-WITNESS. HTTP GET /witness returns the token.
4. Flag flip. --conf spark.submit.proxyUser.allowCustomClasspathInClusterMode=true plus --jars. childClasspath keeps the jars. Witness loads from the child list. This is the --conf bypass of the internal flag. Not the ungated leftover. Labbed so nobody can pretend .internal() filters --conf.

Last run:

walk=control_jars_cleared child_cleared=true submit_cp_has_evil=false loaded_submit_cp=false
walk=closed_extra_classpath submit_cp_has_evil=false loaded_submit_cp=false
walk=leftover_default_extra child_cleared=true submit_cp_has_evil=true loaded_submit_cp=true extra=/opt/lab/evil.jar
walk=leftover_flag_flip child_cleared=false child_has_evil=true loaded_child_cp=true
SUCCESS SPARK-PROXYUSER-CLASSPATH defaultExtraClassPath-on-cp=yes childClasspath-cleared=yes extraClassPath-omitted=yes conf-flag-keeps-jars=yes SPARK-PROXYUSER-CLASSPATH-WITNESS

Replay:

git clone https://github.com/abraxas/spark-proxyuser-classpath
cd spark-proxyuser-classpath
python3 spark-proxyuser-classpath-Abraxas-Labs.py

or cd lab && ./run.sh. Bind it to loopback. The loopback client is lab/run.sh with a banner on it.

Wrong turns, in the order I made them

Pinning master or a 4.3 RC. 4.3.0-rc looks like "latest." It is a release candidate. SPARK-58531 is already static there. 4.2.0 is the GA.

Checking out the annotated tag. git switch v4.2.0 after a shallow clone can land on a tag object that is not a commit. The useful SHA is 32f72996.

Re-filing CVE-2023-22946. clear() still runs. The control walk is green. Filing "YARN --jars on submit" is the 2023 advisory. The unmatched walk is defaultExtraClassPath on the launcher -cp.

Treating client-mode driver-as-superuser as a CVE. docs/security.md Proxy User section says the client-mode driver JVM is the superuser and that this is not considered a CVE. Livy and Kyuubi are told to force cluster mode. I dropped it.

Scoring the --conf flag flip as a new High. It is the same class as 22946 with the gate turned off. The original advisory said "ensure this is not overridden by submitted applications." Gateways that forward --conf still do. I kept it as a second walk in the lab, not a second leftover.

Ivy spark.jars.ivySettings. MavenUtils.loadIvySettings loads a local Ivy XML before the SPARK-41958 clear. A hostile ivysettings typedef is a different gadget, needs a local settings file plus --packages, and is not the Hadoop configuration-class class. Dropped.

Standalone cluster --packages. resolveMavenDependencies is skipped when isStandAloneCluster. Standalone cluster also does not copy --jars onto childClasspath. Flag-flip walk is YARN/k8s. defaultExtraClassPath still hits standalone cluster because it is launcher -cp.

The log line after clear(). SparkSubmit.scala:918-923 logs the already-cleared buffer. Logging bug. Not a leftover.

spark.hadoop.fs.*.impl without a tenant jar. Points at a class that must already be loadable. No High gadget on the default Spark/Hadoop jars from this review.

SPARK_DIST_CLASSPATH / SPARK_SUBMIT_OPTS. Gateway env. Operator-controlled, not Livy --conf.

Running full spark-submit against a YARN RM. Faithful, slow, and unnecessary. The leftover is the constructed -cp plus prepareSubmitEnvironment. A gadget of those two functions is the pin. A reverse shell is theatre. The oracle is a witness string written when that -cp loads lab.WitnessMain.

Claiming Hadoop doAs drops the Unix UID. It does not. UserGroupInformation.createProxyUser then proxyUser.doAs wraps runMain. The process is still the Livy UID. That is the whole point of 22946.

Inflating this to YARN AM RCE. The YARN AM / k8s driver of the submitted app is still the proxy user after a clean submit. This leftover is the gateway JVM only.

What an attacker can do

Hold a tenant account on a multi-tenant Spark gateway (Livy, Kyuubi, Oozie, and friends) that runs spark-submit --proxy-user <tenant> --deploy-mode cluster as a superuser UID. Stage a jar the gateway JVM can open. Send --conf spark.driver.defaultExtraClassPath=<staged jar>. The SPARK-41958 gate can stay false. The constructed spark-submit -cp still starts with that jar. Classes in it load as the gateway Unix UID.

That UID is the Hadoop proxy superuser. A later local spark-submit --proxy-user as any allowed principal is in reach. Files the gateway can read (keytab dir, spark-defaults, local staging) are in reach. Hadoop identity of later RPCs is still the proxy user. The YARN AM of the submitted app is still the tenant. Do not inflate this to "I am yarn on the cluster."

--conf spark.submit.proxyUser.allowCustomClasspathInClusterMode=true plus --jars is the flag-flip walk. .internal() does not filter --conf. childClasspath keeps the jars. Same UID. Public 22946 remains the default --jars path that clear() already covers.

Lab-proven vs inferred: constructed -cp contains defaultExtraClassPath in cluster mode; childClasspath.clear() still runs on the default flag; spark.driver.extraClassPath is omitted in cluster; --jars with flag false is cleared; a witness class on that -cp writes the oracle. Hadoop config-class execution is the public CVE-2023-22946 mechanism, not re-run here. Livy/Kyuubi forwarding --conf is the architecture those projects document.

Adjacent closed (do not inflate)

You get the gateway UID, not yarn on the cluster for free, and you do not get a shell in this lab. You get a classloader the 2023 fix was supposed to keep tenant-free.

The fix

Stop putting spark.driver.defaultExtraClassPath on the spark-submit JVM -cp in cluster mode the same way spark.driver.extraClassPath is already omitted. Drop tenant --conf of spark.submit.proxyUser.allowCustomClasspathInClusterMode at the gateway (Livy/Kyuubi) and/or refuse that internal key in SparkSubmitArguments. Keep childClasspath.clear() for the public CVE-2023-22946 --jars path.

Until then, a multi-tenant gateway that forwards --conf into spark-submit --proxy-user --deploy-mode cluster still has an ungated launcher classpath. Filter spark.driver.defaultExtraClassPath and the SPARK-41958 flag on the way in. Cluster mode was supposed to be the secure Livy setting. The driver JVM on YARN is the tenant. The submit JVM on the gateway is not.

I am not going to print a Livy POST you can paste at someone else's REST. The unmatched key is the useful part. If you own the gateway, run the proof of concept against loopback.

References

• Proof of concept: abraxas/spark-proxyuser-classpath · spark-proxyuser-classpath-Abraxas-Labs.py
• Lab: lab/ · Dockerfile · docker-compose.yml · run.sh · poc.py · gadget
• @abraxas_null · github.com/abraxas · abraxaslabs.tech · abraxas.null@proton.me
• CWE-269 · CWE-427
• Tree: apache/spark v4.2.0 (32f7299601108917fb01920a54e084595b7b3bf8) · SparkSubmitCommandBuilder.java · SparkSubmit.scala
• CVE-2023-22946 · SPARK-41958 · GHSA-329j-jfvr-rhr6
• Security model: spark.apache.org/security.html · docs/security.md Proxy User