Spark Connect session SET still writes the driver

The testing flag is still a session SET

I am @abraxas_null. The proof of concept is on GitHub: abraxas/spark-connect-copy-local (loopback client). The lab stack is lab/: Dockerfile, docker-compose.yml, run.sh, poc.py, gadget.py. Authorized lab only. It talks to loopback.

This is Apache Spark 4.2.0 (32f7299601108917fb01920a54e084595b7b3bf8). No CVE yet. Leftover of SPARK-58531. CWE-22 / CWE-732. 8.1 High (CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:N/I:H/A:H). Remote file write. Spark Connect. Dest must be Hadoop LocalFileSystem. Not RCE.

Spark Connect has an API named copyFromLocalToFs. The name is honest. You give it a local blob and an absolute dest, and the driver copies through Hadoop copyFromLocalFile(..., overwrite=true). For a cloud filesystem that is the point. For the driver's own disk it is a loaded gun. The method that loads it, ArtifactManager.uploadArtifactToFs, already knows that. The comment in the 4.2.0 tree is the threat model in one paragraph: by default we do not support uploading to a local dest, otherwise a user overwrites an arbitrary file on the Spark driver node. Enable the testing flag at driver start. Testing purpose only.

The 3.5 flag was static. Spark 4.0 replaced it with spark.sql.artifact.copyFromLocalToFs.allowDestLocal so the old Connect key could be deprecated without breaking people who already set it. The replacement is buildConf. buildConf is session-settable. A Connect client who can spark.conf.set can turn the testing flag on for that session, then forward_to_fs a dest the driver process can write.

SPARK-58531 already named this leftover. JIRA, 2026-08-05, Fix Version 4.3.0, PR 57730: convert the key to a static conf, because it "controls whether a caller can write to an arbitrary path on the driver's local filesystem, so it is meant to be set by whoever starts the driver." The 4.0 replacement was runtime, so "a session could set it and override the value the driver was started with." 4.2.0 is listed Affects. This pin still has the runtime conf.

Why Spark

I had just finished leftover hunts of Kafka Connect, Tomcat RewriteValve, and a stack of history-server-shaped Java trees. Spark is the next large Apache compute runtime: Scala 2.13, a security page that tells you job submit is by design, and a Connect gRPC socket that looks like a hole until you read the FAQ. The product is honest about unauthenticated clusters. A finding on this hunt needed something else: leftover of a published auth fix, a walk that writes outside the provisioned application, History host file, tenant secret, ACL skip when auth is on.

I only wanted Critical or High. Remote preferred. Local is in-scope. Medium stays on the floor unless it chains. Unauth SQL on an open Connect server is DROP. Loading an ML model is loading executable code. Those are the FAQ. I am not filing them.

Connectors, Livy, Kubernetes operators, and cloud vendor overlays are other trees. This post is vanilla Spark 4.2.0, Spark Connect, one leftover of SPARK-58531.

Pinning 4.2.0, not a 4.3 RC

spark.apache.org listed 4.2.0 as the latest GA (news 2026-07-14). RCs for 4.0.5 / 4.1.4 / 4.2.1 / 4.3.0 were on the board. Those wait. Master is not a pin. A leftover that only exists on an RC is a moving target. A leftover that is already static on 4.3.0 and still runtime on the GA people actually run is the post.

The tag object is not always a commit. Peel 32f7299601108917fb01920a54e084595b7b3bf8, "v4.2.0-rc6", 2026-07-11. POM says 4.2.0. Scala 2.13. Apache-2.0.

The clone is the tagged tree. Twenty-eight parallel hunts: standalone REST, RPC auth, Spark Connect (auth, session, artifacts, Python, Arrow, executePlan), Connect artifacts, Python data sources, SQL data sources, History deser, History XSS, event log, Hive Thrift, Kubernetes, YARN, proxy-user, PySpark gateway, SparkR, UI ACLs, UI kill, worker, launcher, MLlib, pipelines, streaming, shuffle, Hadoop tokens, secret redaction, SSL/Netty, KVStore, UDF worker. Parent checked this leftover against SQLConf.ARTIFACT_COPY_FROM_LOCAL_TO_FS_ALLOW_DEST_LOCAL, ArtifactManager.uploadArtifactToFs, SparkConnectConfigHandler.handleSet, and classic RuntimeConfig.requireNonStaticConf. One independent lab on loopback. Witness-only.

The skip list is half the hunt

Before you look for leftovers, you write down what is already public so you do not "discover" a CVE with a press release. Spark's list lives at spark.apache.org/security.html. On 4.2.0 the interesting ones were already closed:

• CVE-2025-55039 Moderate RPC AES-CTR without auth. GCM knob present. Default still CTR. Closed unless a new unmatched walk.
• CVE-2025-54920 History Server Jackson polymorphic deser RCE. JsonProtocol requires SparkListenerEvent. Closed on this pin.
• CVE-2026-32773 History stored XSS via app name. Medium. No High chain on this pin.
• CVE-2023-32007 / CVE-2022-33891 UI shell. id -Gn argv. Closed.
• CVE-2023-22946 proxy-user classpath. Default-false gate on this pin.
• CVE-2020-9480 standalone master spark.authenticate bypass on the receive path. Closed unless leftover.
• CVE-2018-11770 standalone REST ignores spark.authenticate. Connect gRPC is a different socket.
• CVE-2018-17190 unsecured standalone master executes user code. By-design adjacent.
• CVE-2018-1334 / CVE-2018-11760 PySpark/SparkR local impersonation. Closed.
• CVE-2017-12612 launcher unsafe deser. FilteredObjectInputStream. Closed.

SPARK-52122 DefaultParamsReader ML RCE: allowlist plus safeMLClassLoader on this pin. SPARK-44981 already filtered the old Connect key from local-mode SET. That is not this leftover.

Vendor FAQ: unauth job submit and driver code exec on an open cluster are by design. Loading an ML model is loading executable code. I wrote those on the floor before I opened ArtifactManager.scala.

SPARK-58531 itself is public as an Improvement, not a CVE. Fix Version 4.3.0. The class is not a secret. The pin still ships the session-settable replacement.

Twenty-eight surfaces, then this gate

Standalone REST, RPC auth, History, Hive, Kubernetes, and UI ACLs were their own hunts. This post is Spark Connect copyFromLocalToFs.

Default Connect has no PSK. spark.connect.authenticate.token is optional static. Unset: no interceptor. Set: PreSharedKeyAuthenticationInterceptor compares Authorization to Bearer $token with MessageDigest.isEqual. Missing header is UNAUTHENTICATED. That is a shared secret, not per-user. It also does not keep allowDestLocal off. Unauth SQL / Python UDF / Scala UdfPacket ObjectInputStream on that socket is job submit. DROP.

Sessions are SessionKey(userId, sessionId). Both fields are client-supplied. Docs: session_id exists to share sessions across languages. Same-token attach to a known UUID is the documented share model, not a leftover of an isolation control.

Artifact staging is bounded. ArtifactUtils.concatenatePaths requires a relative name, concatenates, normalize(), then startsWith(basePath + "/"). Suites reject /absolute/path/ and .. / ../sibling / a/../../b. Class, jar, py, archive land under artifactRootDirectory/<sessionUUID>/. That DROP is staging escape. This leftover is the intended forward_to_fs dest after the local gate is flipped.

pythonExec is env PYSPARK_PYTHON / python3, not client-supplied. Arrow is result framing. Plan decompression is size-bounded. ML backend classes still need an MLBackendPlugin on the classpath. None of that is this post.

Config RPC handleGetAll dumps session SQL confs. Static Spark core keys cannot be SET. spark.connect.grpc.interceptor.classes / binding / token are static. The session-flip of an operator-off control that survived is this one key.

The 4.0 replacement is not static

copyFromLocalToFs / AddArtifacts name prefix forward_to_fs copies a staged blob through Hadoop copyFromLocalFile(..., overwrite=true) to an absolute dest. LocalFileSystem dest is blocked unless the operator enabled the testing flag at driver start:

def uploadArtifactToFs(
    remoteRelativePath: Path,
    serverLocalStagingPath: Path): Unit = {
  ...
  if (fs.isInstanceOf[LocalFileSystem]) {
    val allowDestLocalConf =
      session.sessionState.conf.getConf(SQLConf.ARTIFACT_COPY_FROM_LOCAL_TO_FS_ALLOW_DEST_LOCAL)
        .getOrElse(
          session.conf.get("spark.connect.copyFromLocalToFs.allowDestLocal").contains("true"))

    if (!allowDestLocalConf) {
      // To avoid security issue, by default,
      // we don't support uploading file to local file system
      // destination path, otherwise user is able to overwrite arbitrary file
      // on spark driver node.
      throw new SparkUnsupportedOperationException("_LEGACY_ERROR_TEMP_3161")
    }
  }
  fs.copyFromLocalFile(false, true, new FSPath(localPath.toString), destFSPath)
}

Handler path: CRC-ok artifact whose path starts with forward_to_fs/ (SparkConnectAddArtifactsHandler). Python client concatenates FORWARD_TO_FS_PREFIX + dest_path and refuses os.path.join because dest is absolute (python/pyspark/sql/connect/client/artifact.py). Staging itself is concatenatePaths-bounded.

Old conf spark.connect.copyFromLocalToFs.allowDestLocal is static, default false (Connect.scala). RuntimeConfig.set of that key throws cannotModifyValueOfStaticConfigError. SPARK-44981 already filtered it from local-mode SET.

The 4.0 replacement is not static:

val ARTIFACT_COPY_FROM_LOCAL_TO_FS_ALLOW_DEST_LOCAL =
  buildConf("spark.sql.artifact.copyFromLocalToFs.allowDestLocal")
    .internal()
    ...
    .booleanConf
    .createOptional

buildConf does not call registerStaticConfigKey. isModifiable is containsConfigKey && !isStaticConfigKey. Connect Config RPC SET:

private def handleSet(
    operation: proto.ConfigRequest.Set,
    conf: RuntimeConfig): proto.ConfigResponse.Builder = {
  ...
      conf.set(key, value.orNull)

Classic RuntimeConfig.set only rejects static keys and Spark-core keys. Internal is still settable. After SET, getConf(ARTIFACT_...) returns Some(true) and the getOrElse fallback to the static old conf is skipped. SQL SET spark.sql.artifact.copyFromLocalToFs.allowDestLocal=true via ExecutePlan SqlCommand is the same SQLConf.setConfString.

The docs on the new key still say testing purpose only. They also still say "when starting spark driver." The 4.0 type of the key does not enforce that sentence.

What I almost filed instead

Unauth Connect ExecutePlan SQL. The FAQ already took that. Filing "port 15002 runs SQL" is a product review.

SET of spark.connect.copyFromLocalToFs.allowDestLocal. That is the old key. It is static. The error class is CANNOT_MODIFY_STATIC_CONFIG. That is the control I wanted, on the name nobody uses after 4.0.

Staging ... concatenatePaths already rejects it. INVALID_ARTIFACT_PATH. I spent a pass proving the sandbox holds for class/jar/py/archive names so I would not confuse a dest-gate leftover with a staging leftover.

Unix new File(root, "/abs"). People remember that an absolute child ignores the parent. On Unix it does not. It joins under root. That myth ate a different Connect walk. This leftover does not need it. forward_to_fs dest is built as Paths.get(File.separator).resolve(subpath) from the name after the prefix. The dest is absolute on purpose. The gate is the only thing standing in front of LocalFileSystem.

HDFS/S3/ABFS dest without the flag. That is the documented API. The leftover is the local dest after a session SET. Cloud dest never consults the testing flag.

Inflating file write to RCE. Spark UID overwrite of a path the process can write is a file write. A later-exec chain (cron, already-executable script, keys) is a different lab. I did not run that lab. The SUCCESS line is a witness file outside the artifact sandbox, uid 185.

Pinning 4.3.0-rc because SPARK-58531 is resolved. The JIRA is an Improvement with Fix Version 4.3.0. GA is 4.2.0. Affects includes 4.2.0. That is the pin.

The lab

Host is macOS. One container. The leftover is SQLConf plus ArtifactManager, not a Spark build. The gadget copies the session-conf gate and the LocalFileSystem dest copy. Pin commit is in /health. Server runs as uid 185, the uid the official Spark image uses. Artifact root is /opt/spark/work/artifacts/<sessionUUID>/. Oracle dest is /oracle/SPARK-CONNECT-COPY-LOCAL-WITNESS. Loopback 127.0.0.1:18920.

The PoC:

1. GET /health. Pin v4.2.0, commit 32f7299601108917fb01920a54e084595b7b3bf8.
2. GET /whoami. uid 185. Artifact root recorded so the dest can be proved outside it.
3. POST /copyFromLocalToFs with dest /oracle/SPARK-CONNECT-COPY-LOCAL-WITNESS and no SET. Expect 400 _LEGACY_ERROR_TEMP_3161. Dest absent.
4. POST /config/set of the old static key spark.connect.copyFromLocalToFs.allowDestLocal=true. Expect 400 CANNOT_MODIFY_STATIC_CONFIG.
5. POST /config/set of spark.sql.artifact.copyFromLocalToFs.allowDestLocal=true. Expect 200, modifiable=True.
6. POST /copyFromLocalToFs again. Expect 200, dest equal to the oracle path, dest outside the artifact root.
7. GET /stat. exists, content WITNESS, uid 185.

Last run:

health pin=v4.2.0 commit=32f7299601108917fb01920a54e084595b7b3bf8
server uid=185 artifact_root=/opt/spark/work/artifacts/<sessionUUID>
control-no-set status=400 condition=_LEGACY_ERROR_TEMP_3161
control-no-set dest-absent=yes
control-static-set status=400 error_class=CANNOT_MODIFY_STATIC_CONFIG
session-set status=200 modifiable=True
copy-after-set status=200 dest=/oracle/SPARK-CONNECT-COPY-LOCAL-WITNESS
oracle dest=/oracle/SPARK-CONNECT-COPY-LOCAL-WITNESS uid=185 outside-sandbox=yes content=SPARK-CONNECT-COPY-LOCAL-WITNESS
SUCCESS SPARK-CONNECT-COPY-LOCAL SPARK-CONNECT-COPY-LOCAL-WITNESS

Replay:

git clone https://github.com/abraxas/spark-connect-copy-local
cd spark-connect-copy-local/lab
./run.sh

Bind it to loopback. The loopback client is lab/run.sh with a banner on it.

The gadget is not a Spark compile. It is the gate: isModifiable / requireNonStaticConf / getConf optional then fallback to the static old key / copyFromLocalFile(overwrite=true) as the process UID. The SUCCESS line names the pin commit so nobody can pretend I labbed 4.3.0.

Wrong turns, in the order I made them

Pinning an RC. 4.3.0-rc looks like "the fix is out." It is a release candidate. SPARK-58531 Fix Version is 4.3.0. GA people run is 4.2.0. Affects includes this pin.

Checking out the tag object. Annotated tags are not commits. The useful SHA is 32f72996.

Treating unauth Connect SQL as a finding. I spent a morning confirming ExecutePlan runs SQL when no token is set. Spark already told everyone that. SECURITY.md FAQ. DROP.

SET of the old static key. Muscle memory from the 3.5 name. The error class is the control working. The leftover is the 4.0 name.

Staging ... I wanted a path traversal. Staging already refuses it. The leftover is a dest the API is designed to write, behind a gate a session can open.

Unix absolute child ignores parent. That is a Windows story. On Unix new File(dir, "/abs") stays under dir. Different Connect walk. Not this one.

Cloud dest without the flag. copyFromLocalToFs to HDFS is the documented feature. The testing flag is only consulted for LocalFileSystem. Filing cloud dest is filing the API.

Inflating File write to RCE. Overwrite as Spark UID is a file write. A later-exec chain is inferred and not in the lab. The oracle is witness bytes, uid 185, outside the artifact sandbox.

Thinking SPARK-44981 already closed this. That JIRA filtered the old Connect key from local-mode SET. The 4.0 replacement was still buildConf. SPARK-58531 is the one that makes the replacement static, and only on 4.3.0.

Compiling Spark for the lab. The leftover is two methods and a conf type. A gadget of those methods plus the pin SHA in /health is the faithful lab. Pulling apache/spark:4.2.0 and waiting for Connect to come up would have tested Docker more than isModifiable.

What an attacker can do

A Spark Connect client who can reach gRPC (default bind port 15002, no PSK unless the operator set spark.connect.authenticate.token) can flip a testing-only gate the driver was started with off, then overwrite a chosen local path on the driver as the Spark UID.

Lab-proven vs inferred: session SET of the buildConf key, then LocalFileSystem dest write as uid 185 outside the artifact sandbox, is lab. Spark UID overwrite of a path the process can write (driver work dir, a world-writable drop, an already-executable script the operator later runs) is inferred. No RIP control. Do not inflate File write to RCE.

Requires: Spark Connect listening. Client that can SET session SQL conf and AddArtifacts. Dest must be Hadoop LocalFileSystem. Default Connect has no PSK; if the operator set a token, the leftover still needs that shared Bearer. It is a bypass of the local-dest control, not a PSK bypass.

Adjacent surface that stays closed

No shell. No heap dump. No Connect PSK skip. The dest still has to be LocalFileSystem. The write is as the Spark UID, not as root.

The fix

Move spark.sql.artifact.copyFromLocalToFs.allowDestLocal to StaticSQLConf so a session cannot SET it. That is SPARK-58531 / PR 57730 on 4.3.0. Add a test that Config RPC SET of that key throws CANNOT_MODIFY_STATIC_CONFIG, the same way the old Connect key already does.

Until operators can run 4.3.0: do not expose Connect without a token and a network ACL, and do not treat the testing flag as off just because the driver was started with it unset. The 4.2.0 type of the key does not keep that promise.

Replay:

cd lab
./run.sh

Loopback 127.0.0.1:18920. Witness SPARK-CONNECT-COPY-LOCAL-WITNESS. Last line SUCCESS SPARK-CONNECT-COPY-LOCAL SPARK-CONNECT-COPY-LOCAL-WITNESS.

References

• Pack: github.com/abraxas/spark-connect-copy-local
• Product: github.com/apache/spark tag v4.2.0 (32f7299601108917fb01920a54e084595b7b3bf8)
• ArtifactManager.scala uploadArtifactToFs
• SQLConf.scala ARTIFACT_COPY_FROM_LOCAL_TO_FS_ALLOW_DEST_LOCAL
• Connect.scala old static key
• SparkConnectConfigHandler.scala handleSet
• RuntimeConfig.scala requireNonStaticConf
• SPARK-58531 · PR 57730
• spark.apache.org/security.html
• CWE-22 · CWE-732

Abraxas Labs https://abraxaslabs.tech https://github.com/abraxas https://x.com/abraxas_null (@abraxas_null) abraxas.null@proton.me