Automate Dagster User Deployments with Flux: A GitOps Approach

User-specific deployments in a data orchestration platform like Dagster can become brittle and hard to manage at scale especially when user pipelines evolve independently. In this post, we’ll show how to automate Dagster user deployments using Flux, Helm, and an S3-backed ConfigMap to externalize the user deployment configuration. We’ll also walk through a purpose-built GitHub Actions pipeline that updates the config and Flux reconciliation automatically.

Whether you’re orchestrating data pipelines for internal teams or delivering managed environments to users, this setup ensures deployments are dynamic, declarative, and version-controlled—the GitOps trifecta.

Approach Summary

Automate Dagster user deployments by externalizing workspace.yaml in S3 and controlling updates via GitHub Actions. Flux continuously syncs both the Dagster HelmRelease and the centralized config from S3, ensuring GitOps consistency across environments.

Dagster Deployment with external ConfigMap

We deploy Dagster to our Kubernetes cluster using Flux’s HelmRelease CRD. The magic lies in activating workspace.enabled: true and sourcing Dagster’s workspace.yaml from an external ConfigMap (dagster-workspace-yaml):

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: dagster
  namespace: dagster
spec:
  interval: 10m
  chart:
    spec:
      chart: dagster
      version: 1.11.0
      sourceRef:
        kind: HelmRepository
        name: dagster-repo
        namespace: flux-system
  values:
    generatePostgresqlPasswordSecret: true
    generateCeleryConfigSecret: true
    dagster-user-deployments:
      enabled: true
      enableSubchart: false
      deployments: []  # No user deployments as part of helm release
    workspace:
      enabled: true # enbale workspace
      servers: []
      externalConfigmap: dagster-workspace-yaml # User deployments will be defined in external configmap

Scripts to update configmap and upload to S3

To ensure Dagster deployments are dynamic and self-service, we maintain a shared workspace.yaml in an S3 bucket. Two purpose-built scripts help safely modify this configuration and push updates without race conditions:

patch_workspace_cm.py

A Python script that:

• Locates the embedded workspace.yaml string inside a Kubernetes ConfigMap manifest

• Deserializes its contents using ruamel.yaml

• Adds or removes a grpc_server block for a user code server, based on --action, --host, --port, and --location-name

• Overwrites the manifest in-place so it's ready to apply

It’s smart enough to avoid duplicates, skip if no change is needed, and gracefully handle malformed or empty input.

s3-patch-with-etag.sh

A Bash wrapper that makes the process atomic and safe:

• Fetches the latest workspace.yaml and its ETag from S3

• Downloads and runs the Python patcher locally

• Uploads the patched file back to S3 only if the ETag matches (via --if-match), preventing conflicting writes

• Retries the whole operation with backoff if an ETag collision occurs

This ensures that even in multi-developer or multi-pipeline environments, your S3-stored configuration remains consistent and race-free.

If S3 versioning is enabled on the bucket, every update becomes trackable and reversible—great for audit trails or rollbacks.

GitHub Actions Pipeline for User Deployments

The user deployment repo which has actual pipeline code triggers a pipeline on push. It builds the Dagster user image, deploys it with Helm (dagster-user-deployments Helm chart), and then updates workspace.yaml in S3.

# Github Actions steps
  
- name: Deploy Dagster user code via Helm
  run: |
    helm upgrade --install ${{ env.DEPLOY_NAME }} dagster/dagster-user-deployments \
      --namespace ${{ env.NAMESPACE }} \
      --create-namespace \
      --set deployments[0].name=${{ env.DEPLOY_NAME }} \
      --set deployments[0].image.repository=843986362912.dkr.ecr.${{ env.AWS_REGION }}.amazonaws.com/${{ env.ECR_REPO }} \
      --set deployments[0].image.tag=${{ env.IMAGE_TAG }} \
      --set deployments[0].image.pullPolicy=IfNotPresent \
      --set deployments[0].port=4000 \
      --wait

- name: Install Python dependencies
  run: pip install ruamel.yaml

- name: Update Dagster Workspace Config in S3
  run: |
    export BUCKET=dagster-workspace-config
    export HOST="${{ env.DEPLOY_NAME }}"
    export PORT=4000
    export LOCATION_NAME="${{ env.DEPLOY_NAME }}"
    chmod +x ./s3-patch-with-etag.sh
    ./s3-patch-with-etag.sh --action add

Flux to install and reconcile configmap using Bucket Source

The workspace.yaml lives in an AWS S3 bucket. Flux tracks this object using the Bucket source and applies it into the cluster via a Kustomization. This allows updates to be pushed outside Git and still reconciled declaratively:

# Bucket Source
apiVersion: source.toolkit.fluxcd.io/v1
kind: Bucket
metadata:
  name: dagster-workspace
  namespace: flux-system
spec:
  interval: 2m
  provider: aws
  bucketName: dagster-workspace-config
  region: eu-west-2
  endpoint: s3.amazonaws.com

# Kustomization to apply the updated ConfigMap
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: dagster-workspace-cm
  namespace: flux-system
spec:
  interval: 3m
  sourceRef:
    kind: Bucket
    name: dagster-workspace
  path: ./
  prune: true
  wait: true
  targetNamespace: dagster

Refresh Dagster UI to Load New Deployments

After the updated workspace.yaml ConfigMap is applied to the cluster Flux as part of its reconcile process , the Dagster UI will not reflect the changes immediately. To ensure the new user code servers are recognized:

• Navigate to the Deployments tab in the Dagster UI.

• Click Reload All to trigger a full refresh of the workspace.yaml.

• The updated list of code locations will appear, including any newly added or removed deployments.

Dagster dynamically reads the workspace.yaml on reload, so there's no need to restart the instance just a single click to activate the latest config.