Visitar URL original
Validating each generated SDK module scope uploads the whole host workspace · Issue #15029 · dagger/dagger · GitHub
Skip to content

Validating each generated SDK module scope uploads the whole host workspace #15029

Open

Description

@yasyf

What happens. Generating a workspace's SDK scopes (WorkspaceSDK.generate, which also backs the SDK's generate/stale check) runs each [sdks.<name>.scopes."<path>"] entry in turn. After each module scope, validateGeneratedModuleConfig in core/schema/workspace_sdk_module.go calls filteredWorkspaceRootfs. For a host workspace, that issues Host.directory(path: <workspace root>, gitignore: true, gitIgnoreRoot: <workspace root>) with no include filter, which is a full upload of the checkout. All of it is there just to stat <scope>/dagger-module.toml through moduleConfigInDir.

Reproduction. A workspace with several Dang module scopes in dagger.toml, for example seven [sdks.dang.scopes.".dagger/modules/<name>"] entries with is-module = true. Run dagger check -vvv dag://?check=dang-sdk/generate/stale after a README-only edit. Under WorkspaceSDK.generate there is one Host.directory of the whole workspace root per scope, run one after another.

Cost. On a large monorepo checkout the check took 16.6-17.7 s against a quiet engine, nearly all of it in those seven uploads. In CI, ten dagger check clients upload from one agent at once, and the same check took 87-106 s on every push, which made it the slowest check of the run.

Proposed fix. Resolve only the scope's config filenames (modules.ConfigFilenames() under the scope path) instead of the whole rootfs. With that change the check took 1.6 s on the same engine. PR to follow.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions