Repository navigation
Gateway panics on secret CRUD: workspace resolution omits signing_key → nil deref in ParseSecretKey (v0.1.772) #1906
Description
Summary
On gateway v0.1.772, any workspace secret operation through the RPC/CLI (secret create, secret update, or any decrypt path) panics the gateway process with a nil pointer dereference, because the types.Workspace object on the auth path is resolved by queries that do not SELECT signing_key, and the secret repository then dereferences it unconditionally:
// pkg/repository/backend_postgres.go:2247 (also 2295, 2316, 2362)
secretKey, err := pkgCommon.ParseSecretKey(*workspace.SigningKey)GetWorkspaceByExternalId / GetWorkspace omit the column:
// pkg/repository/backend_postgres.go:212
query := `SELECT id, external_id, name, created_at, updated_at, concurrency_limit_id, volume_cache_enabled, multi_gpu_enabled, storage_id FROM workspace WHERE external_id = $1;`GetWorkspaceByExternalIdWithSigningKey (line 251) already exists and SELECTs signing_key, but WorkspaceSecretService (pkg/abstractions/secret/secret.go) passes the auth-resolved authInfo.Workspace straight into backendRepo.CreateSecret, so the field is always nil on that path.
Impact
beta9 secret create NAME VALUE(or the gRPC equivalent) panics and kills the gateway — in Kubernetes this presents as CrashLoopBackOff, taking down the shared gateway for all workspaces, even workspaces whosesigning_keyis set in the DB (the value is simply never loaded).- Decrypt paths (
GetSecretByNameDecrypted,GetSecretsByNameDecrypted) andUpdateSecrethave the same deref, so once a deployment is in this state, reads fail the same way. - We hit this in production today (self-hosted k3s, image
beta9-gateway:0.1.772): twosecret createcalls crash-looped the shared gateway serving ~12 users.
Repro
- Run gateway
v0.1.772against postgres. beta9 secret create TEST_X hello(any workspace token).- Gateway panics:
invalid memory address or nil pointer dereferenceatpkgCommon.ParseSecretKey(nil*workspace.SigningKey), container exits, restarts, and the CLI retry loops it.
Suggested fix
Two complementary changes (we run the first one live):
- Load the key on the secret path — add
signing_keyto theGetWorkspaceByExternalId/GetWorkspaceSELECTs (or route the secret service throughGetWorkspaceByExternalIdWithSigningKey):
--- a/pkg/repository/backend_postgres.go
+++ b/pkg/repository/backend_postgres.go
@@ -212,7 +212,7 @@
func (r *PostgresBackendRepository) GetWorkspaceByExternalId(ctx context.Context, externalId string) (types.Workspace, error) {
var workspace types.Workspace
- query := `SELECT id, external_id, name, created_at, updated_at, concurrency_limit_id, volume_cache_enabled, multi_gpu_enabled, storage_id FROM workspace WHERE external_id = $1;`
+ query := `SELECT id, external_id, name, created_at, updated_at, concurrency_limit_id, signing_key, volume_cache_enabled, multi_gpu_enabled, storage_id FROM workspace WHERE external_id = $1;`
err := r.client.GetContext(ctx, &workspace, query, externalId)
@@ -225,7 +225,7 @@
query := `
- SELECT w.id, w.external_id, w.name, w.created_at, w.concurrency_limit_id, w.volume_cache_enabled, w.multi_gpu_enabled,
+ SELECT w.id, w.external_id, w.name, w.created_at, w.concurrency_limit_id, w.volume_cache_enabled, w.multi_gpu_enabled, w.signing_key,
ws.id "storage.id", ws.bucket_name "storage.bucket_name", ws.access_key "storage.access_key",- Guard the deref so a workspace without a signing key returns an error instead of a panic (applies to
CreateSecret:2247,GetSecretByNameDecrypted:2295,GetSecretsByNameDecrypted:2316,UpdateSecret:2362):
+ if workspace == nil || workspace.SigningKey == nil {
+ return nil, errors.New("workspace signing key not loaded")
+ }
secretKey, err := pkgCommon.ParseSecretKey(*workspace.SigningKey)Happy to open a PR with the full patch if useful.
Activity