Dagster Multi-Region ECS Demo

This repo demonstrates an example of how Dagster OSS can be used on AWS ECS Fargate so that tasks can be dispatched across multiple regions from a single cluster. Infrastructure managed with Pulumi (Python).

Cost warning: This demo requires provisioning resources on AWS, so expect costs of ~$100+/month if left running. Always make destroy when done.

Security notice: The ALB provisioned by this demo is open to the public internet (port 3000, no authentication). This means anyone can access the Dagster UI and trigger runs on your AWS account. No intentional effort has been made to restrict access or mitigate this risk, as it is outside the scope of this demo. If you choose to deploy this, consider adding IP allowlisting, VPN, or an authentication layer in front of the ALB.

• uv (Python package manager)
• Pulumi CLI
• Docker
• some (preferably sandbox) AWS account credentials

cd code

# 1. Configure AWS credentials
cp .env.example .env
# Edit .env with your AWS credentials
source .env

# 2. Install deps, login to Pulumi, init stack
make init

# 3. Deploy everything (infra + Docker images + ECS redeploy)
make deploy

# 4. Get the Dagster UI URL
cd infra && pulumi stack output alb_dns_name
# Open http://<alb_dns_name>:3000 in your browser

1. Open the Dagster UI at http://<alb_dns_name>:3000
2. Go to Launchpad and launch local_job -- it runs in the home region and should succeed immediately.
3. Launch remote_job -- it runs in the remote cluster via cross-region dispatch. Check the logs for hello from remote cluster (eu-central-1).
4. Materialise remote_asset (via remote_asset_job) to verify asset-level cross-region dispatch. The metadata panel will show the remote region and hostname.
5. To test cancellation across regions, launch long_remote_job and terminate it from the UI while it is running.

├── README.md                       # This file
└── code/
    ├── Makefile                    # init, deploy, docker-push, destroy, status
    ├── infra/                      # Pulumi project (Python)
    │   ├── __main__.py             # Entry point
    │   ├── vpc.py                  # VPCs for both regions
    │   ├── peering.py              # VPC peering + routes
    │   ├── security_groups.py      # All security groups
    │   ├── ecs.py                  # ECS clusters, services, ALB
    │   ├── rds.py                  # RDS Postgres
    │   ├── iam.py                  # IAM roles
    │   ├── ecr.py                  # ECR repository
    │   ├── Pulumi.yaml             # Pulumi project config
    │   └── Pulumi.dev.yaml         # Stack config (regions, CIDRs)
    ├── docker/                     # Dagster container images
    │   ├── dagster.yaml            # Dagster instance config
    │   ├── workspace.yaml          # Code location config
    │   ├── patches/                # Patched EcsRunLauncher (PR #33437)
    │   │   └── ecs_launcher.py
    │   ├── Dockerfile.daemon       # Dagster daemon
    │   └── Dockerfile.webserver    # Dagster webserver
    ├── code_location/              # Sample Dagster code location
    │   ├── definitions.py          # local_job, remote_job, etc
    │   └── Dockerfile              
    └── pyproject.toml              # Python project config (uv)

• Home region (e.g., us-west-2): Dagster cluster (webserver, daemon, code location) + RDS Postgres + ALB
• Remote region (e.g., eu-central-1): Worker-only ECS cluster, tasks launched on demand via ecs/* run tags
• Cross-region: VPC peering for RDS connectivity, cross-region ECR pull

sequenceDiagram
    actor User as Browser
    participant ALB
    participant Web as Webserver
    participant Daemon
    participant RDS as RDS Postgres
    participant ECR
    participant S3 as S3 Logs
    participant ECS as ECS API (remote cluster)
    participant Task as Fargate Task (remote cluster)
    participant CW as CloudWatch (remote cluster)

    Note over User,CW: 1. Launch Run

    User->>ALB: HTTPS (internet)
    ALB->>Web: forward (SG: alb -> services)
    Web->>RDS: GraphQL launchRun -> INSERT run (QUEUED)

    Note over User,CW: 2. Daemon Dispatches to Remote Region

    Daemon->>RDS: poll QUEUED runs
    RDS-->>Daemon: run with ecs/region=eu-central-1
    Daemon->>ECS: ecs:RunTask (via NAT GW -> internet)
    ECS-->>Daemon: task ARN
    Daemon->>RDS: store ecs/task_arn tag

    Note over User,CW: 3. Task Provisions and Executes

    ECS->>Task: place in private subnet
    Task->>ECR: pull image (NAT GW -> internet -> us-west-2)
    Task->>RDS: read run config (VPC peering)
    loop Execute ops/assets
        Task->>RDS: write events (VPC peering)
        Task->>CW: stdout/stderr (NAT GW, awslogs driver)
        Task->>S3: compute logs (NAT GW -> internet)
    end

    Note over User,CW: 4. Health Monitoring (parallel)

    loop Every 30s
        Daemon->>ECS: ecs:DescribeTasks (NAT GW -> internet)
        ECS-->>Daemon: task status
    end

    Note over User,CW: 5. Task Completes

    Task->>RDS: final status SUCCESS/FAILURE (VPC peering)
    Task->>CW: final logs
    Task--xTask: container exits

    Note over User,CW: 6. User Views Result

    User->>ALB: HTTPS
    ALB->>Web: forward
    Web->>RDS: query run status + events
    Web->>S3: fetch compute logs
    Web-->>ALB: render result
    ALB-->>User: run status + logs

Cross-region task dispatch requires a patched EcsRunLauncher from PR #33437, which is not yet in a released dagster-aws version. The patch adds tag-based multi-region support (ecs/region, ecs/cluster, ecs/security_groups, ecs/subnets) with zero changes to dagster.yaml config schema.

This demo bundles the patched launcher as docker/patches/ecs_launcher.py and loads it via dagster.yaml:

run_launcher:
  module: ecs_launcher          # patched from dagster_aws.ecs
  class: EcsRunLauncher

The Dockerfiles for daemon and webserver copy the patch into the container and add it to PYTHONPATH:

COPY patches/ecs_launcher.py /opt/dagster/patches/ecs_launcher.py
ENV PYTHONPATH=/opt/dagster/patches

This works because the patched file uses absolute imports (from dagster_aws.ecs.container_context import ...) that resolve against the installed dagster-aws package.

Once PR #33437 is merged and released, revert to the standard module by changing dagster.yaml back to module: dagster_aws.ecs and removing the COPY patches/ and PYTHONPATH lines from both Dockerfiles.

Dagster compute logs (stdout/stderr from ops) are stored in S3 using S3ComputeLogManager. The bucket name is passed to daemon and webserver containers via the DAGSTER_BUCKET env var (set by Pulumi from the provisioned S3 bucket), and logs are written under the compute_logs/ prefix:

compute_logs:
  module: dagster_aws.s3.compute_log_manager
  class: S3ComputeLogManager
  config:
    bucket:
      env: DAGSTER_BUCKET
    prefix: compute_logs

This demo uses local state (~/.pulumi/ on disk) -- no account or external storage needed to get started. This is fine for a single-developer demo but not for team or production use, because:

• The state file is only on your machine -- if lost, Pulumi loses track of all provisioned resources.
• No locking -- concurrent pulumi up from two machines can corrupt state or cause drift.
• No audit trail of who changed what and when.

# Local backend (this demo) -- state stored in ~/.pulumi/
pulumi login --local

For production-like environments, configure a remote backend so state is centralized, locked, and durable:

# S3 backend (recommended for AWS-native setups)
pulumi login s3://my-pulumi-state-bucket

# Or use Pulumi Cloud (managed SaaS -- free tier available)
pulumi login

See Pulumi docs: Self-managed backends for S3 backend setup details including state locking and encryption.

cd code
make destroy

Disclaimer: This code is provided as-is for educational purposes only. You are solely responsible for any AWS costs, data loss, or other damages resulting from its use. The author(s) and maintainer(s) accept no liability whatsoever.