Deploying Your Data Pipeline to Kubernetes with Helm

You’ve built a complete data pipeline with ingestion, transformation, validation, and monitoring. Now it’s time to deploy it to Kubernetes and make it production-ready. This chapter covers packaging, deployment, and verification of your entire stack.

Architecture Overview

The Kubernetes deployment consists of:

Namespace: ais-pipeline
├── Dagster Components
│   ├── dagster-daemon (orchestrator)
│   ├── dagster-webserver (UI)
│   ├── dagster-run-worker (K8s jobs)
│   └── PostgreSQL (metadata storage)
├── Storage
│   ├── MinIO (S3-compatible storage)
│   └── PersistentVolumes (1TB for data lake)
├── Monitoring
│   ├── Prometheus (metrics collection)
│   ├── Grafana (dashboards)
│   └── Alertmanager (notifications)
└── Networking
    └── MetalLB LoadBalancers (external IPs)

Prerequisites

Before deploying, ensure you have:

• MicroK8s cluster running with dns, storage, and metallb addons
• MetalLB configured with an IP pool
• Helm 3.x installed
• kubectl configured to access the cluster
• Docker for building custom images
• 1TB+ disk available for persistent volumes

Step 1: Create Namespace

kubectl create namespace ais-pipeline
kubectl config set-context --current --namespace=ais-pipeline

Step 2: Build Custom Dagster Image

Dagster needs access to your pipeline code. Create a Docker image with all dependencies:

# docker/Dockerfile.dagster
FROM python:3.11-slim

WORKDIR /opt/dagster/app

# Install system dependencies
RUN apt-get update && apt-get install -y \
    gcc g++ libpq-dev \
    && rm -rf /var/lib/apt/lists/*

# Copy and install Python dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy pipeline source code
COPY src/ /opt/dagster/app/src/

# Set PYTHONPATH for imports
ENV PYTHONPATH=/opt/dagster/app

# Verify installation
RUN python -c "import dagster; import duckdb; import great_expectations; print('Dependencies installed')"

# Default command (overridden by Helm)
CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-m", "pipeline.repository"]

Build and import to MicroK8s:

# Build the image
docker build -f docker/Dockerfile.dagster -t ais-pipeline:latest .

# For MicroK8s local registry
microk8s enable registry
docker tag ais-pipeline:latest localhost:32000/ais-pipeline:latest
docker push localhost:32000/ais-pipeline:latest

Step 3: Configure Secrets

Store credentials securely in Kubernetes:

# k8s/dagster/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: ais-pipeline-secrets
  namespace: ais-pipeline
type: Opaque
stringData:
  MINIO_ENDPOINT: "minio.ais-pipeline.svc.cluster.local:9000"
  MINIO_ACCESS_KEY: "your-access-key"
  MINIO_SECRET_KEY: "your-secret-key"
  MINIO_SECURE: "false"
  POSTGRES_USER: "dagster"
  POSTGRES_PASSWORD: "your-postgres-password"
  POSTGRES_DB: "dagster"

Important: Never commit secrets to Git. Add to .gitignore.

kubectl apply -f k8s/dagster/secrets.yaml

Step 4: Deploy MinIO

If not already deployed from Chapter 2:

helm repo add minio https://charts.min.io/
helm repo update

helm install minio minio/minio \
  --namespace ais-pipeline \
  --values k8s/minio/values.yaml

Create the data lake bucket:

# Install MinIO client
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc && sudo mv mc /usr/local/bin/

# Configure and create bucket
MINIO_IP=$(kubectl get svc minio-console -n ais-pipeline -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
mc alias set ais-minio http://$MINIO_IP minioadmin minioadmin
mc mb ais-minio/ais-lake

Step 5: Deploy Dagster with Helm

Create a Helm values file for Dagster:

# k8s/dagster/values.yaml
dagsterWebserver:
  service:
    type: LoadBalancer
    port: 80

dagsterDaemon:
  enabled: true

postgresql:
  enabled: true
  persistence:
    enabled: true
    size: 10Gi

runLauncher:
  type: K8sRunLauncher
  config:
    k8sRunLauncher:
      resources:
        requests:
          memory: "4Gi"
          cpu: "2000m"
        limits:
          memory: "8Gi"
          cpu: "4000m"

dagster-user-deployments:
  enabled: true
  deployments:
    - name: "ais-pipeline"
      image:
        repository: "localhost:32000/ais-pipeline"
        tag: "latest"
        pullPolicy: Always
      dagsterApiGrpcArgs:
        - "--module-name"
        - "pipeline.repository"
      port: 3030
      envSecrets:
        - name: ais-pipeline-secrets

Deploy:

helm repo add dagster https://dagster-io.github.io/helm
helm repo update

helm install dagster dagster/dagster \
  --namespace ais-pipeline \
  --values k8s/dagster/values.yaml \
  --timeout 10m

Monitor the deployment:

kubectl get pods -n ais-pipeline -w

Expected pods:

• dagster-daemon-xxx - Running
• dagster-webserver-xxx - Running
• dagster-postgresql-xxx - Running
• dagster-user-deployments-xxx - Running

Step 6: Deploy Monitoring Stack

Deploy Prometheus and Grafana:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace ais-pipeline \
  --set prometheus.service.type=LoadBalancer \
  --set grafana.service.type=LoadBalancer \
  --set prometheus.prometheusSpec.retention=15d \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi

Deploy the custom metrics server:

# k8s/monitoring/metrics-server.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ais-metrics-server
  namespace: ais-pipeline
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ais-metrics-server
  template:
    metadata:
      labels:
        app: ais-metrics-server
    spec:
      containers:
      - name: metrics-server
        image: localhost:32000/ais-pipeline:latest
        command: ["python", "-m", "monitoring.metrics_server"]
        ports:
        - containerPort: 9091
        envFrom:
        - secretRef:
            name: ais-pipeline-secrets
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "200m"
---
apiVersion: v1
kind: Service
metadata:
  name: ais-metrics-server
  namespace: ais-pipeline
spec:
  selector:
    app: ais-metrics-server
  ports:
  - port: 9091
    targetPort: 9091

kubectl apply -f k8s/monitoring/metrics-server.yaml

Step 7: Access Your Services

Get all LoadBalancer IPs:

kubectl get svc -n ais-pipeline -o json | \
  jq -r '.items[] | select(.spec.type=="LoadBalancer") | "\(.metadata.name): http://\(.status.loadBalancer.ingress[0].ip)"'

Expected services:

• Dagit UI: Web interface for pipeline management
• MinIO Console: Object storage management
• Grafana: Monitoring dashboards
• Prometheus: Metrics queries

Step 8: Verify Deployment

Test Asset Materialization

1. Open Dagit UI
2. Navigate to Assets tab
3. Select dma_ais_bronze
4. Click Materialize → Select partition → Launch run

Watch the logs for:

• “Downloading DMA AIS data to bronze”
• “Uploaded to MinIO: s3://ais-lake/bronze/…”

Verify Data in MinIO

mc ls ais-minio/ais-lake/bronze/dma/

Check Prometheus Metrics

Open Prometheus UI and query:

ais_rows_ingested_total{source="dma"}

Resource Configuration

Recommended resource limits for a 32GB server:

Component	CPU Request	Memory Request	CPU Limit	Memory Limit
Dagster Daemon	500m	1Gi	1000m	2Gi
Dagster Webserver	500m	1Gi	1000m	2Gi
Run Worker	2000m	4Gi	4000m	8Gi
PostgreSQL	250m	512Mi	500m	1Gi
MinIO	2000m	4Gi	4000m	8Gi
Prometheus	500m	2Gi	1000m	4Gi
Grafana	200m	512Mi	500m	1Gi

Total baseline: ~6 CPU cores, ~14GB RAM (leaves headroom for burst and additional workloads)

Adjust in values.yaml if needed:

runLauncher:
  config:
    k8sRunLauncher:
      resources:
        requests:
          memory: "6Gi"  # Increase for larger datasets
          cpu: "3000m"

Apply changes:

helm upgrade dagster dagster/dagster \
  --namespace ais-pipeline \
  --values k8s/dagster/values.yaml

Quick Reference Commands

# Get all service IPs
kubectl get svc -n ais-pipeline -o wide

# Restart a deployment
kubectl rollout restart deployment/<name> -n ais-pipeline

# View logs (follow mode)
kubectl logs -f deployment/<name> -n ais-pipeline

# Execute command in pod
kubectl exec -it deployment/<name> -n ais-pipeline -- /bin/bash

# Check resource usage
kubectl top pods -n ais-pipeline
kubectl top nodes

# Delete and redeploy Dagster
helm uninstall dagster -n ais-pipeline
helm install dagster dagster/dagster --namespace ais-pipeline --values k8s/dagster/values.yaml

What You’ve Built

Your pipeline is now running on Kubernetes:

• Custom Docker image with all pipeline dependencies
• Helm-managed Dagster with PostgreSQL for metadata
• Secure secrets for MinIO credentials
• LoadBalancer services for clean external access
• Monitoring stack with Prometheus and Grafana
• End-to-end verification from ingestion to visualization

What’s Next

With the pipeline deployed, Chapter 11 covers automating daily runs with schedules and processing historical data with backfills.

---

Deploying data infrastructure to Kubernetes? We help organizations design and implement production-ready data platforms. Get in touch to discuss your project.