Repository navigation
Credential Rotation #58475
Description
Hello,
In my company we are building an internal platform that runs many ETL processes over Spark 4.1 with PySpark and spark-connect.
We are using an AWS RDS Postgres DB as well as custom Postgres deployments for usage with AGE etc., all read/write via Spark.
We would like to have an option to change the credentials for connections where the pipeline configuration is coming from config files we have that are dynamically translated into Spark code.
For writers this is not a problem as we can add our own retry mechanism that recreates the writer with the updated credentials.
However, For readers, this is a problem because at the point of materialization, the reader is already baked into the pipeline with the original credentials and it does not have any way to change the credentials without restarting the entire ETL process, which could be running for several hours before the plan reaches the point where it needs to use the JDBC connection.
I'm wondering if it would be possible to add some way of reconfiguring just the reader/writer credentials from PySpark code without having to write dedicated Java code, compile into a JAR and load when running Spark?
Thanks!
Activity
konjac commentedon Sep 17, 2026
The problem does not seem to be Spark-specific, but rather a general reader/writer integration scenario (not necessarily Spark) involving connection credentials.
CoderNasim2023 commentedon Oct 5, 2026
CoderNasim2023 commentedon Oct 7, 2026
May I start on working on it ? @tmaoz