Visitar URL original
Gateway panics on secret CRUD: workspace resolution omits signing_key → nil deref in ParseSecretKey (v0.1.772) · Issue #1906 · beam-cloud/beta9 · GitHub
Skip to content

Gateway panics on secret CRUD: workspace resolution omits signing_key → nil deref in ParseSecretKey (v0.1.772) #1906

Open

Description

@pjatcreditgenie

Summary

On gateway v0.1.772, any workspace secret operation through the RPC/CLI (secret create, secret update, or any decrypt path) panics the gateway process with a nil pointer dereference, because the types.Workspace object on the auth path is resolved by queries that do not SELECT signing_key, and the secret repository then dereferences it unconditionally:

// pkg/repository/backend_postgres.go:2247 (also 2295, 2316, 2362)
secretKey, err := pkgCommon.ParseSecretKey(*workspace.SigningKey)

GetWorkspaceByExternalId / GetWorkspace omit the column:

// pkg/repository/backend_postgres.go:212
query := `SELECT id, external_id, name, created_at, updated_at, concurrency_limit_id, volume_cache_enabled, multi_gpu_enabled, storage_id FROM workspace WHERE external_id = $1;`

GetWorkspaceByExternalIdWithSigningKey (line 251) already exists and SELECTs signing_key, but WorkspaceSecretService (pkg/abstractions/secret/secret.go) passes the auth-resolved authInfo.Workspace straight into backendRepo.CreateSecret, so the field is always nil on that path.

Impact

  • beta9 secret create NAME VALUE (or the gRPC equivalent) panics and kills the gateway — in Kubernetes this presents as CrashLoopBackOff, taking down the shared gateway for all workspaces, even workspaces whose signing_key is set in the DB (the value is simply never loaded).
  • Decrypt paths (GetSecretByNameDecrypted, GetSecretsByNameDecrypted) and UpdateSecret have the same deref, so once a deployment is in this state, reads fail the same way.
  • We hit this in production today (self-hosted k3s, image beta9-gateway:0.1.772): two secret create calls crash-looped the shared gateway serving ~12 users.

Repro

  1. Run gateway v0.1.772 against postgres.
  2. beta9 secret create TEST_X hello (any workspace token).
  3. Gateway panics: invalid memory address or nil pointer dereference at pkgCommon.ParseSecretKey (nil *workspace.SigningKey), container exits, restarts, and the CLI retry loops it.

Suggested fix

Two complementary changes (we run the first one live):

  1. Load the key on the secret path — add signing_key to the GetWorkspaceByExternalId / GetWorkspace SELECTs (or route the secret service through GetWorkspaceByExternalIdWithSigningKey):
--- a/pkg/repository/backend_postgres.go
+++ b/pkg/repository/backend_postgres.go
@@ -212,7 +212,7 @@
 func (r *PostgresBackendRepository) GetWorkspaceByExternalId(ctx context.Context, externalId string) (types.Workspace, error) {
 	var workspace types.Workspace
-	query := `SELECT id, external_id, name, created_at, updated_at, concurrency_limit_id, volume_cache_enabled, multi_gpu_enabled, storage_id FROM workspace WHERE external_id = $1;`
+	query := `SELECT id, external_id, name, created_at, updated_at, concurrency_limit_id, signing_key, volume_cache_enabled, multi_gpu_enabled, storage_id FROM workspace WHERE external_id = $1;`
 	err := r.client.GetContext(ctx, &workspace, query, externalId)
@@ -225,7 +225,7 @@
 	query := `
-	SELECT w.id, w.external_id, w.name, w.created_at, w.concurrency_limit_id, w.volume_cache_enabled, w.multi_gpu_enabled,
+	SELECT w.id, w.external_id, w.name, w.created_at, w.concurrency_limit_id, w.volume_cache_enabled, w.multi_gpu_enabled, w.signing_key,
 	ws.id "storage.id", ws.bucket_name "storage.bucket_name", ws.access_key "storage.access_key",
  1. Guard the deref so a workspace without a signing key returns an error instead of a panic (applies to CreateSecret :2247, GetSecretByNameDecrypted :2295, GetSecretsByNameDecrypted :2316, UpdateSecret :2362):
+	if workspace == nil || workspace.SigningKey == nil {
+		return nil, errors.New("workspace signing key not loaded")
+	}
 	secretKey, err := pkgCommon.ParseSecretKey(*workspace.SigningKey)

Happy to open a PR with the full patch if useful.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions