RELEASING.md

Releasing

This describes how to release Dagger:

This is a high-level diagram of how all the pieces fit together:

flowchart TD
    repo(["🐙 github.com/dagger/dagger"])
    docs["📒 Documentation"]
    playground["🛝 Playground"]
    repo -.-> docs & playground

    subgraph Dagger
        engine("🚙 Engine")
        cli("🚗 CLI &nbsp;")
    end

    repo ==> engine & cli

    S3["🗄 dl.dagger.io/dagger"]
    brew-tap["🐙 github.com/dagger/homebrew-tap"]
    cli --> S3 --> brew-tap

    registry["📦 registry.dagger.io/engine"]
    ghcr["🐙 ghcr.io/dagger/engine"]
    engine --> ghcr --> registry

    go["🐹 Go SDK"]
    go-repo["🐙 github.com/dagger/dagger-go-sdk"]
    go-pkg["🐹 dagger.io/dagger"]
    go-ref["🐹 pkg.go.dev/dagger.io/dagger"]

    repo ==> go --> go-repo --> go-pkg & go-ref
    registry -.- S3 -.- go & python & nodejs & elixir

    python["🐍 Python SDK"]
    pypi["🐍 pypi.org/project/dagger-io"]
    readthedocs["📖 dagger-io.readthedocs.io"]
    repo ==> python --> pypi & readthedocs

    nodejs["⬢ Node.js SDK"]
    npm["⬢ npmjs.com/@dagger.io/dagger"]
    repo ==> nodejs --> npm

    elixir["🧪 Elixir SDK"]
    hex["🧪 hex.pm/packages/dagger"]
    repo ==> elixir --> hex

Let the team know

Before you go ahead and produce a new release, remember that it's a team effort. The first step is to let the team know what is going to happen, preferably a few days in advance so that they can react. We do this by:

Create a new post in Discord #ask-the-team, e.g. v0.6.4 release - July 19, 2023

This allows others to weigh in whether:

we should go for a patch / minor bump
there are any PRs that people are waiting to get merged
any big features which need to remain experimental?
etc.

Maybe there are breaking changes which we should be aware of and message accordingly. Giving other team members a day or two to react - because timezones! - will make this entire process smoother.

Most importantly, patch vs minor is not a technical decision. If you want to read more about this, see this (private) Discord thread.

Note

Once you know what type of release we are producing - patch vs minor - remember to edit the ? in the Discord thread.

Improve this doc while releasing 改善

In order to keep this relevant & accurate, we improve this doc during the release process. It's the best time to pause, observe how it all fits together, and improve it. We want small, constant improvements which compound. Therefore:

Open this doc in your code editor. As you go through these steps, edit the parts which could be better. As inspiration, see the PR for these changes.
Update the date in the shields.io badge, first line in this file.

Note

We believe in documentation first, automation second. Documenting a process forces us to understand it well. Continuously editing this documentation refines our understanding. Once we have a good grasp of the problem space, and reach an elegant solution, it comes natural to automate & speed things up, to make the process more efficient. We should still be able to perform things manually if we need to - because sometimes automation fails 🤷. This is when everyone wishes they had good documentation, the original author(s) or both! It's also worth mentioning that when it's time to improve this automation, we want to be looking at the blueprint - this doc right here - not the implementation. If you ever had to migrate from Chef/Puppet to Ansible/Terraform, you know what it was like to migrate the implementation.

🚙 Engine + 🚗 CLI ⏱ `30mins`

Warning

It is important to always do an Engine + CLI release prior to releasing any SDK. This will ensure that all the APIs in the SDK are also available in the Engine it depends on.

Create .changes/<ENGINE_VERSION>.md by either running changie batch patch (or changie batch minor if this is a new minor). ENGINE_VERSION will be automatically generated.

Note If you do not have changie installed, see https://changie.dev

Update CHANGELOG.md by running changie merge.
Submit a PR with the resulting changes so that release notes can be generated correctly. The merge commit is what gets tagged in the next step.
Ensure that all checks are green ✅ for the <ENGINE_GIT_SHA> on the main branch that you are about to release.
20mins When you have confirmed that all checks are green, run the following:

git checkout main
git pull

export ENGINE_GIT_SHA="$(git rev-parse --verify HEAD)"
export ENGINE_VERSION="$(changie latest)"
git tag "${ENGINE_VERSION:?must be set}" "${ENGINE_GIT_SHA:?must be set}"

git push origin "${ENGINE_VERSION:?must be set}"

This will kick off .github./workflows/publish.yml. After the publish job in this workflow passes, a new draft PR will automatically be created to bump the Engine version in the various SDKs.

Run export CHANGIE_ENGINE_VERSION=${ENGINE_VERSION:?must be set} so that the release changelog will be correctly generated.
Checkout this branch locally & generate changelogs for all SDKs:

git fetch origin
git checkout bump-engine

cd sdk/go
changie new --kind "Dependencies" --body "Bump Engine to $ENGINE_VERSION" --custom "Author=github-actions" --custom "PR=5613"
changie batch patch
changie merge

cd ../python
changie new --kind "Dependencies" --body "Bump Engine to $ENGINE_VERSION" --custom "Author=github-actions" --custom "PR=5613"
changie batch patch
changie merge

cd ../nodejs
changie new --kind "Dependencies" --body "Bump Engine to $ENGINE_VERSION" --custom "Author=github-actions" --custom "PR=5613"
changie batch patch
changie merge

cd ../elixir
changie new --kind "Dependencies" --body "Bump Engine to $ENGINE_VERSION" --custom "Author=github-actions" --custom "PR=5613"
changie batch patch
changie merge

10mins Open this draft PR in github.com/dagger/dagger/pulls & click on Ready to review so that all checks run.
After all checks pass, merge this PR. Tip: go to the Files changed tab on the PR to review without an explicit request.

🐹 Go SDK ⏱ `30mins`

⚠️ Ensure that all SDKs have the same Engine version

Warning

If we publish one SDK with an updated Engine version, we must do the same for all other SDKs. This is important as currently our automatic provisioning code enforces the existence of a single Engine running at a time. Users will not be able to use multiple SDKs at the same time if the Engine version that they reference differs.

Ensure that all checks are green ✅ for the <SDK_GIT_SHA> on the main branch that you are about to release. This will usually be the commit that bumps the Engine version, the one that you merged earlier.
Tag & publish:

export SDK_GIT_SHA="$(git rev-parse --verify HEAD)"
cd sdk/go && export GO_SDK_VERSION=$(changie latest) && cd ../..
git tag "sdk/go/${GO_SDK_VERSION:?must be set}" "${SDK_GIT_SHA:?must be set}"
git push origin "sdk/go/${GO_SDK_VERSION:?must be set}"

This will trigger the publish-sdk-go workflow which publishes to 🐙 github.com/dagger/dagger-go-sdk.

After the newly published tag appears on 🐙 github.com/dagger/dagger-go-sdk, double-check that is was picked up by pkg.go.dev. You can manually request this new version via open https://pkg.go.dev/dagger.io/dagger@${GO_SDK_VERSION:?must be set}. The new version can take up to 15mins to appear, it's OK to move on.
20mins Bump the Go SDK version in our internal mage CI targets. Submit a new PR now, so that we can dogfood the first version ourselves, before creating a new GitHub release and making it widely public.

cd internal/mage
go get -u dagger.io/dagger
go mod tidy

# Check that everything works as expected:
go run main.go -w ../.. engine:lint

Note

To upload the release notes, we need to have the gh CLI installed, e.g. brew install gh

Upload the release notes by running:

gh release create "sdk/go/${GO_SDK_VERSION:?must be set}" \
    --draft --verify-tag --title sdk/go/$GO_SDK_VERSION \
    --notes-file sdk/go/.changes/$GO_SDK_VERSION.md

Check that release notes look good in Preview
⚠️ De-select Set as the latest release (only used for 🚙 Engine + 🚗 CLI releases)
Click on Publish release

🐍 Python SDK ⏱ `5mins`

⚠️ Ensure that all SDKs have the same Engine version

Warning

If we publish one SDK with an updated Engine version, we must do the same for all other SDKs. This is important as currently our automatic provisioning code enforces the existence of a single Engine running at a time. Users will not be able to use multiple SDKs at the same time if the Engine version that they reference differs.

Ensure that all checks are green ✅ for the <SDK_GIT_SHA> on the main branch that you are about to release. This will usually be the commit that bumps the Engine version, the one that you merged earlier.
Tag & publish:

cd sdk/python && export PYTHON_SDK_VERSION=$(changie latest) && cd ../..
git tag "sdk/python/${PYTHON_SDK_VERSION:?must be set}" "${SDK_GIT_SHA:?must be set}"
git push origin sdk/python/${PYTHON_SDK_VERSION}

This will trigger the Publish Python SDK workflow which publishes dagger-io to 🐍 PyPI

Note

To upload the release notes, we need to have the gh CLI installed, e.g. brew install gh

Upload the release notes by running:

gh release create "sdk/python/${PYTHON_SDK_VERSION:?must be set}" \
    --draft --verify-tag --title sdk/python/$PYTHON_SDK_VERSION \
    --notes-file sdk/python/.changes/$PYTHON_SDK_VERSION.md

⚠️ De-select Set as the latest release (only used for 🚙 Engine + 🚗 CLI releases)
Check that release notes look good in Preview. FWIW: https://readthedocs.org/projects/dagger-io/builds/
Click on Publish release

⬢ Node.js SDK ⏱ `5mins`

⚠️ Ensure that all SDKs have the same Engine version

Warning

If we publish one SDK with an updated Engine version, we must do the same for all other SDKs. This is important as currently our automatic provisioning code enforces the existence of a single Engine running at a time. Users will not be able to use multiple SDKs at the same time if the Engine version that they reference differs.

Ensure that all checks are green ✅ for the <SDK_GIT_SHA> on the main branch that you are about to release. This will usually be the commit that bumps the Engine version, the one that you merged earlier.
Tag & publish:

cd sdk/nodejs && export NODEJS_SDK_VERSION=$(changie latest) && cd ../..
git tag "sdk/nodejs/${NODEJS_SDK_VERSION:?must be set}" "${SDK_GIT_SHA:?must be set}"
git push origin sdk/nodejs/${NODEJS_SDK_VERSION}

This will trigger the Publish Node.js SDK workflow which publishes a new version to ⬢ npmjs.com/package/@dagger.io/dagger

Note

To upload the release notes, we need to have the gh CLI installed, e.g. brew install gh

Upload the release notes by running:

gh release create "sdk/nodejs/${NODEJS_SDK_VERSION:?must be set}" \
    --draft --verify-tag --title sdk/nodejs/$NODEJS_SDK_VERSION \
    --notes-file sdk/nodejs/.changes/$NODEJS_SDK_VERSION.md

Check that release notes look good in Preview
⚠️ De-select Set as the latest release (only used for 🚙 Engine + 🚗 CLI releases)
Click on Publish release

🧪 Elixir SDK ⏱ `5mins`

⚠️ Ensure that all SDKs have the same Engine version

Warning

If we publish one SDK with an updated Engine version, we must do the same for all other SDKs. This is important as currently our automatic provisioning code enforces the existence of a single Engine running at a time. Users will not be able to use multiple SDKs at the same time if the Engine version that they reference differs.

Ensure that all checks are green ✅ for the <SDK_GIT_SHA> on the main branch that you are about to release. This will usually be the commit that bumps the Engine version, the one that you merged earlier.
Tag & publish:

cd sdk/elixir && export ELIXIR_SDK_VERSION=$(changie latest) && cd ../..
git tag "sdk/elixir/${ELIXIR_SDK_VERSION:?must be set}" "${SDK_GIT_SHA:?must be set}"
git push origin sdk/elixir/${ELIXIR_SDK_VERSION}

This will trigger the Publish Elixir SDK workflow which publishes a new version to 🧪 hex.pm/packages/dagger

Note

To upload the release notes, we need to have the gh CLI installed, e.g. brew install gh

Upload the release notes by running:

gh release create "sdk/elixir/${ELIXIR_SDK_VERSION:?must be set}" \
    --draft --verify-tag --title sdk/elixir/$ELIXIR_SDK_VERSION \
    --notes-file sdk/elixir/.changes/$ELIXIR_SDK_VERSION.md

Check that release notes look good in Preview
⚠️ De-select Set as the latest release (only used for 🚙 Engine + 🚗 CLI releases)
Click on Publish release

📒 Documentation ⏱ `5mins`

Warning

Merging a documentation PR does NOT automatically deploy the new documentation to the production website.

There are two websites for documentation:

Staging: https://devel.docs.dagger.io - Netlify dashboard
Production: https://docs.dagger.io - Netlify dashboard

Staging release

When a PR is merged, a new deployment is created for the documentation site and it is automatically published to https://devel.docs.dagger.io via Netlify.

Use this staging website to test the documentation, including:

verifying that the new content appears in the navigation
verifying internal and external links work correctly
verifying that images appear correctly
etc.

Production release

When a PR is merged, a new production deployment is also created for https://docs.dagger.io. However, this deployment is not automatically published.

After testing the documentation using the staging website and if you are satisfied with it, manually publish the production deployment via Netlify as follows:

Log in to the Netlify dashboard for https://docs.dagger.io.
Refer to the list of "production deploys" and select the one you wish to deploy. Usually, this will be the most recent one. You can confirm this by checking the deployment hash against the latest commit hash in the dagger/dagger repository main branch.
On the deployment page, click the "Preview" button to once again preview/check the deployment. You can also check the deployment log to confirm there were no errors during the documentation build process.
If you are satisfied with the preview, click the "Publish deploy" button. This will publish the selected deployment on https://docs.dagger.io

Note

There have been cases where Netlify builds have failed with errors, but the same build succeeds when performed locally. In the past, one reason for this has been Netlify's use of a stale cache. In case you encounter this error, click "Options -> Clear cache and retry with latest branch commit" to recreate the deployment with a clean cache.

🛝 Playground ⏱ `2mins`

The Dagger Playground is set to automatically update once there's a new release of the Dagger Engine. In order to verify which Dagger version the Playground is using, check the x-dagger-engine HTTP header with the deployed Dagger Engine version is returned for each playground query:

Follow these steps to retrieve and verify the Playground Dagger version:

Login with your GitHub account at https://play.dagger.cloud
Open your browser's Developer Tools, and then the Network tab
Click the Execute query button
Click in the /playgrounds POST request row in the Network tab
Verify that the x-dagger-engine response header commit value matches the ENGINE_GIT_SHA value from the beginning of this guide

Last step

When all the above done, remember to open a PR with all the changes as a result of going through these steps. Here is an example: dagger#5490
Remember to toggle all the checkboxes back to [ ]

Navigation Menu

Repository navigation

RELEASING.md

RELEASING.md

Releasing

Let the team know

Improve this doc while releasing 改善

🚙 Engine + 🚗 CLI ⏱ `30mins`

🐹 Go SDK ⏱ `30mins`

🐍 Python SDK ⏱ `5mins`

⬢ Node.js SDK ⏱ `5mins`

🧪 Elixir SDK ⏱ `5mins`

📒 Documentation ⏱ `5mins`

Staging release

Production release

🛝 Playground ⏱ `2mins`

Last step

FilesExpand file tree

RELEASING.md

Latest commit

History

RELEASING.md

File metadata and controls

Releasing

Let the team know

Improve this doc while releasing 改善

🚙 Engine + 🚗 CLI ⏱ 30mins

🐹 Go SDK ⏱ 30mins

🐍 Python SDK ⏱ 5mins

⬢ Node.js SDK ⏱ 5mins

🧪 Elixir SDK ⏱ 5mins

📒 Documentation ⏱ 5mins

Staging release

Production release

🛝 Playground ⏱ 2mins

Last step

🚙 Engine + 🚗 CLI ⏱ `30mins`

🐹 Go SDK ⏱ `30mins`

🐍 Python SDK ⏱ `5mins`

⬢ Node.js SDK ⏱ `5mins`

🧪 Elixir SDK ⏱ `5mins`

📒 Documentation ⏱ `5mins`

🛝 Playground ⏱ `2mins`