Spark SET SERDEPROPERTIES retargets JDBC with stored passwords

The password rides

I am @abraxas_null. The proof of concept is on GitHub: abraxas/spark-jdbc-serde-url (loopback client). The lab stack is lab/: Dockerfile, docker-compose.yml, run.sh, poc.py, SerdeJdbcGadget.java. Authorized lab only. It talks to loopback.

This is Apache Spark 4.2.0 (32f7299601108917fb01920a54e084595b7b3bf8). No CVE yet. CWE-15 / CWE-522. 8.1 (CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:C/C:H/I:L/A:N). Remote JDBC URL injection. Shared-catalog SQL user. Not a job-submit jar.

Spark already closed the obvious door. Scan-time extra options cannot override a stored JDBC url. That is SPARK-31710. ALTER TABLE SET TBLPROPERTIES writes a different map. CREATE TABLE USING jdbc OPTIONS (url ...) is the owner naming the database. SHOW CREATE redacts password. Logs redact the JDBC URL unconditionally. All of that is real work.

ALTER TABLE SET SERDEPROPERTIES is still allowed on a datasource table. It merges into storage.properties. That map is the JDBC scan option map. Stored user and password are not Spark JDBC option names, so they are not stripped before Driver.connect. Change the URL. Keep the secret. The next SELECT hands both to whatever host you pointed at.

I pointed it at a PostgreSQL protocol listener on loopback. The driver connected. The startup packet said user=spark_jdbc. Cleartext auth replayed the stored password. The oracle wrote SPARK-JDBC-SERDE-URL-WITNESS. Extra-options clash refused. SET TBLPROPERTIES left storage on 127.0.0.1:1 and never hit the listener.

Why Spark

I had just finished a Windows leftover in Kafka Connect. FileStream was supposed to be off the default CLASSPATH. The .bat never got the memo. Spark is the other large ASF Java tree that people put a shared SQL session in front of and then forget who can ALTER TABLE.

Spark's security model is honest if you read it. SECURITY.md FAQ: submitting a job on an open cluster executes user code. That is the product. Loading an ML model is loading executable code. spark.authenticate=false is the documented default. Filing "unauth spark-submit is RCE" is a product review, not a leftover.

A finding on this hunt needed something else: leftover of a public auth fix, a walk that writes or reads outside the provisioned application, History Server host file, cross-app shuffle, tenant secret, ACL skip. JDBC stored credentials in a shared catalog sit in that last bucket. The SQL user is not supposed to be a job submitter. They are supposed to query tables someone else provisioned.

I only wanted Critical or High. Remote preferred. Medium stays on the floor unless it chains. Databricks Unity, Ranger plugins, and third-party JDBC catalogs are other trees.

Pinning 4.2.0, not a 4.3 RC

The day of the hunt, spark.apache.org listed 4.2.0 as the current GA (news 2026-07-14). GitHub also had 4.3.0 RCs and 4.2.1 / 4.1.4 / 4.0.5 candidates. Those wait. A leftover that only exists on master is a moving target. A leftover that only exists on an RC is a press release waiting for a tag.

The annotated tag is not always the useful SHA. Peel commit 32f7299601108917fb01920a54e084595b7b3bf8 is v4.2.0-rc6, 2026-07-11. POM says 4.2.0. Scala 2.13. Kubernetes and YARN are in the pin. Connectors only when the leftover is this tree.

I cloned the tagged tree. Twenty-eight parallel hunts: standalone REST, RPC auth, Spark Connect, Connect artifacts, Python data sources, proxy-user classpath, SQL data sources, History deser, History XSS, Hive Thrift, Kubernetes, kvstore, launcher, MLlib, pipelines, PySpark gateway, secret redaction, shuffle, SparkR, SSL/Netty, streaming, UDF worker, UI ACLs, UI kill, worker, YARN, Hadoop tokens, event log. Parent checked KEEP claims against the cited Scala. One independent gadget lab for this leftover, loopback, witness-only.

The skip list is half the hunt

Before you look for leftovers, you write down what is already public so you do not "discover" a CVE with a press release. Spark's list lives at spark.apache.org/security.html. On 4.2.0 the interesting ones were already closed:

• CVE-2025-54920 History Server Jackson polymorphic deser. JsonProtocol requires SparkListenerEvent. Closed unless a new unmatched walk.
• CVE-2025-55039 RPC AES-CTR without auth. Moderate. GCM knob present. Default still CTR. Below this bar.
• CVE-2026-32773 History stored XSS via app name. Medium. No High chain on this pin.
• CVE-2022-33891 / CVE-2023-32007 UI shell via id -Gn. Closed.
• CVE-2023-22946 proxy-user classpath. spark.submit.proxyUser.allowCustomClasspathInClusterMode defaults false. Closed unless leftover.
• CVE-2020-9480 standalone master spark.authenticate bypass on the RPC receive path. Closed unless leftover.
• CVE-2018-11770 standalone REST / Mesos REST ignore spark.authenticate.secret. Docs plus default rest.enabled promised false. Closed unless leftover.
• CVE-2018-11760 / CVE-2018-1334 PySpark / SparkR local impersonation. Closed.
• CVE-2017-12612 launcher FilteredObjectInputStream. Closed.
• CVE-2018-17190 unsecured standalone master executes user code. By-design adjacent. The FAQ.

Vendor rejects: unauth job submit / driver code exec on an open cluster is by design. Loading an ML model is loading executable code (docs/ml-security.md). I am not filing those.

Twenty-eight surfaces, one leftover of SPARK-31710

Most of that tree is Spark doing what Spark said it would do.

Job submit on an open cluster runs user code. Spark Connect Python and Java UDFs run as the Spark UID. ADD JAR is the same class. Hive TRANSFORM is the same class. Unauth History UI with ACLs off is the documented default. Hive Thrift NONE is Hive's documented default.

Standalone REST, RPC data-plane, Connect copy/archive, proxy-user classpath, Python data-source registry, and History log-dir walks are other hunts. They are not this post. This post is a SQL v1 JDBC table in a shared catalog.

The SQL data-sources hunt was looking for a non-admin SQL session that could retarget an existing table to driver-local files, IMDS, or another tenant's sink without being a job submitter. Session-named spark.read.format("jdbc").option("url", ...) is intended data-plane. SET spark.sql.catalog.<name>.url in that session is the same data-plane. File-table path lives in locationUri. XML StAX DTD is off. ${env:} substitution is documented Hive-compat.

SPARK-31710 is the named fix. DataFrameReader.table extra options, default spark.sql.legacy.extraOptionsBehavior.enabled=false, refuse a different value for an existing serde key (url / user / password / dbtable). New keys such as sessionInitStatement still merge. That is documented. OSS Spark has no table ACL and no Unity externalOptionsAllowList. The clash check is the control. The leftover is a write path the clash check never sees.

SET TBLPROPERTIES was the first door I tried

ALTER TABLE t SET TBLPROPERTIES ('url' = 'jdbc:postgresql://...') looks like it should work. It writes CatalogTable.properties. JDBC v1 scan does not read that map for the connection URL. AlterTableSetPropertiesCommand even invalidates the relation cache. Storage stays put.

val newTable = table.copy(
  properties = table.properties ++ properties,
  comment = properties.get(TableCatalog.PROP_COMMENT).orElse(table.comment))
catalog.alterTable(newTable)
catalog.invalidateCachedTable(tableName)

Lab: tblproperties-storage-url-unchanged=yes. Stored URL still 127.0.0.1:1. Connection refused. Oracle never wrote a witness. Closed for this walk.

Extra options was the second door, and SPARK-31710 locked it

Scan-time extras go through generateDatasourceOptions. If a stored serde key exists and the extra value differs, it throws.

table.storage.properties.foreach { case (k, v) =>
  if (extraOptions.containsKey(k) && extraOptions.get(k) != v) {
    throw QueryCompilationErrors.failToResolveDataSourceForTableError(table, k)
  }
}

That is SPARK-31710. Lab: extra-options-clash on url. No attacker connect. Closed.

Someone who only reads that function would file "JDBC url is protected." They would be half right.

SET SERDEPROPERTIES is the unmatched write

Datasource tables reject SET SERDE (class name) and partition serde. They allow SET SERDEPROPERTIES. The merge is storage.properties ++ serdeProperties. Later keys win.

if (serdeClassName.isDefined && DDLUtils.isDatasourceTable(table)) {
  throw QueryCompilationErrors.alterTableSetSerdeNotSupportedError(table.qualifiedName)
}
if (partSpec.isEmpty) {
  val newTable = table.withNewStorage(
    serde = serdeClassName.orElse(table.storage.serde),
    properties = table.storage.properties ++ serdeProperties.getOrElse(Map()))
  catalog.alterTable(newTable)
}

CREATE TABLE USING jdbc OPTIONS already stuffed every option except path into that same map:

def buildStorageFormatFromOptions(options: Map[String, String]): CatalogStorageFormat = {
  val path = CaseInsensitiveMap(options).get("path")
  val optionsWithoutPath = options.filter { case (k, _) => k.toLowerCase(Locale.ROOT) != "path" }
  CatalogStorageFormat.empty.copy(
    locationUri = path.map(CatalogUtils.stringToURI), properties = optionsWithoutPath)
}

So after create, storage.properties holds url, user, password, dbtable, driver. After SET SERDEPROPERTIES ('url' = '<attacker-jdbc>'), it holds the new url and the old user / password / dbtable. In-tree AlterTableSetSerdeSuite even asserts the merge on a datasource table. That is not a hidden API. It is a test.

SessionCatalog.alterTable writes externalCatalog and does not refresh the relation cache. AlterTableSerDePropertiesCommand does not call invalidateCachedTable (unlike SET TBLPROPERTIES). REFRESH TABLE invalidates. A new Connect session, or the same session after REFRESH TABLE, resolves from the updated metastore row.

Connect ExecutePlan SQL is the ordinary SparkSqlParser path. No extra block on SET SERDEPROPERTIES. Hive Thrift SQL is the same parser with a different transport.

user and password are not Spark JDBC option names

JDBCOptions.asConnectionProperties keeps every key that is not in jdbcOptionNames and hands the rest to the driver. The named set is url, dbtable, query, driver, partition bounds, pushdowns, keytab, principal, sessionInitStatement, prepareQuery, and friends. There is no user. There is no password.

val asConnectionProperties: Properties = {
  val properties = new Properties()
  parameters.originalMap
    .filter { case (key, _) => !jdbcOptionNames(key.toLowerCase(Locale.ROOT)) }
    .foreach { case (k, v) => properties.setProperty(k, v) }
  properties
}

BasicConnectionProvider.getConnection then:

driver.connect(jdbcOptions.url, properties)

The new URL is Spark's. The leftover properties are the driver's. PostgreSQL JDBC puts user and password on the wire. SHOW CREATE redacts password via conf.redactOptions(metadata.storage.properties). JDBC URL log redaction keeps only jdbc:<subprotocol>: and replaces the rest. The SQL user cannot read the secret. They can still spend it.

That is CWE-15 (external control of the JDBC URL) plus CWE-522 (insufficiently protected credentials that still get used). Scope is changed because the credential leaves the Spark process for a host the attacker chose.

The lab

Host is macOS. Two containers. Compose project spark-jdbc-serde-url. Loopback 127.0.0.1:18960 (PostgreSQL protocol) and 127.0.0.1:18961 (HTTP health / hits).

oracle is python:3.13-slim running poc.py --role oracle. It binds a PG listener. SSLRequest and GSSRequest get N. Startup is parsed. Witness is written on startup, not on a bare TCP probe, so a healthcheck cannot fake SUCCESS. Then it asks for cleartext password (auth type 3), records the p message, answers AuthenticationOk plus ReadyForQuery.

gadget is eclipse-temurin:21-jdk-jammy plus Debian libpostgresql-jdbc-java. It does not boot a Spark cluster. It is a gadget of the cited 4.2.0 Scala: buildStorageFormatFromOptions, SET SERDEPROPERTIES merge, SET TBLPROPERTIES non-merge, SPARK-31710 clash, asConnectionProperties, then real org.postgresql.Driver.connect. Pin SHA is in the Java comment.

Stored URL is jdbc:postgresql://127.0.0.1:1/provisioned?sslmode=disable&gssEncMode=disable. Port 1 refuses. Attack URL is jdbc:postgresql://oracle:18960/stolen?.... User spark_jdbc. Password stored-secret-SPARK-JDBC-SERDE-URL. Table secret_accounts. Driver org.postgresql.Driver.

The PoC:

1. Extra-options gadget with attack url. Must clash. Must not write WITNESS.
2. TBLPROPERTIES gadget with attack url. Storage URL must stay on port 1. Must not write WITNESS. Connect to the stored URL is connection refused, which is the control.
3. SERDE gadget. Merge must replace url and keep user. Driver.connect must hit the oracle. Witness file must contain SPARK-JDBC-SERDE-URL-WITNESS. Hits JSON must show user=spark_jdbc and the stored password.

Last run:

SUCCESS SPARK-JDBC-SERDE-URL connect=yes user-replay=true password-replay=true SPARK-JDBC-SERDE-URL-WITNESS

Replay:

git clone https://github.com/abraxas/spark-jdbc-serde-url
cd spark-jdbc-serde-url/lab
./run.sh

Bind it to loopback. The loopback client is lab/run.sh with a banner on it.

Wrong turns, in the order I made them

Pinning master. Spark master moves. 4.3 RCs move faster. GA is 4.2.0. SPARK-58531 (a different leftover class) is static on 4.3.0 only. That is a reason to pin GA, not a reason to skip 4.2.0.

Checking out the tag object. Annotated tags are not commits. The useful SHA is 32f72996.

Filing job submit. I spent a morning confirming that an open cluster will run a jar. Spark already wrote that down. The FAQ is the skip list.

Treating CREATE TABLE url as the bug. The owner of the table is allowed to name a JDBC URL. That is the data-plane. The leftover is changing the URL later, on a table someone else provisioned, while the stored password stays.

Filing extra-options clash again. SPARK-31710 is closed. Lab-proven. Re-filing it is a skip-list miss.

Expecting SET TBLPROPERTIES to move storage. The names are confusing on purpose. TBLPROPERTIES is CatalogTable.properties. SERDEPROPERTIES is storage.properties. JDBC v1 reads the second map. The first command even invalidates the cache. The second does not. I almost scored TBLPROPERTIES as the write and stopped.

Claiming SHOW CREATE leaks the password. It redacts. Hive serde dump redacts too. Default redaction regex matches password. The bug is use, not display.

Booting a full Spark cluster for a merge gadget. A cluster would prove the SQL parser reaches the command. The command is unit-tested on datasource tables. The interesting sink is Driver.connect with the merged map. A Temurin container plus the PostgreSQL JDBC driver is the faithful sink. A cluster would also collide with other loopback labs.

Unity / Ranger as the control. OSS Spark has neither. An operator who put a SQL ACL in front of ALTER TABLE is running a different product. Default Connect has no SQL ACL.

A reverse shell. Theatre. The oracle is a PG startup plus a password replay on loopback.

Inflating to RCE. driver as a serde key is loadClass plus a no-arg ctor of a class already on the session classpath. That is not a default-classpath gadget on 4.2.0. I am not calling this RCE. JDBC URL injection plus stored-credential reuse is the class.

Single-user pyspark. No shared catalog, no tenant. Out of model.

What an attacker can do

Reach Spark Connect gRPC SQL or Hive Thrift SQL as a catalog user who can ALTER TABLE a persistent USING jdbc table that already stores url plus credentials. Not a job submitter. Not the session that created the table, unless that session is also a later tenant. Default Connect has no SQL ACL.

Adjacent closed: session-named JDBC / file paths. spark.sql.legacy.extraOptionsBehavior clash check. OSS Spark has no Unity externalOptionsAllowList. CREATE TABLE URL is owner-controlled (not this leftover). JDBC URL log redaction is unconditional. File-table path lives in locationUri; serde path does not win over LOCATION. JDBCTableCatalog connections come from spark.sql.catalog.<name>.* (same data-plane as choosing a URL). V2SessionCatalog.getDataSourceOptions uses table.properties plus LOCATION; JDBC v1 stays on storage.properties. Single-user pyspark with no shared catalog is out of model.

You get the database password the operator stored for that table. You get a connection to a host you choose, as that database user. You do not get Spark UID code execution from this leftover. You do not get SHOW CREATE to print the password. You do not get scan-time extras to override url. You do not get SET TBLPROPERTIES to move the JDBC URL.

The fix

Refuse SET SERDEPROPERTIES keys that clash with stored JDBC url / user / password, the same check generateDatasourceOptions already runs for scan-time extras. Or drop stored user / password when url changes, so a retarget cannot ride the secret. Or treat datasource JDBC serde as table properties, not storage, matching the TBLPROPERTIES map JDBC v1 already ignores.

Call invalidateCachedTable from AlterTableSerDePropertiesCommand either way, so a persist is not a stale-cache lottery.

Until then, do not grant untrusted SQL ALTER TABLE on a persistent USING jdbc table that stores a password. Prefer a catalog that does not put database credentials in storage.properties. Prefer a JDBC catalog whose connections come from operator conf, not table serde.

I am not going to print a SQL snippet you can paste at someone else's Connect endpoint. The unmatched merge is the useful part. If you own the catalog, run the proof of concept against loopback.

References

• Proof of concept: abraxas/spark-jdbc-serde-url · spark-jdbc-serde-url-Abraxas-Labs.py
• Lab: lab/ · Dockerfile · docker-compose.yml · run.sh · poc.py · SerdeJdbcGadget.java
• @abraxas_null · github.com/abraxas · abraxaslabs.tech · abraxas.null@proton.me
• CWE-15 · CWE-522
• Tree: apache/spark v4.2.0 (32f7299601108917fb01920a54e084595b7b3bf8) · ddl.scala · DataSource.scala · DataSourceUtils.scala · BasicConnectionProvider.scala · JDBCOptions.scala
• SPARK-31710
• Security: spark.apache.org/security.html