Your AIS pipeline is deployed, scheduled, and running automatically. But production systems require ongoing operations—monitoring health, responding to failures, and maintaining performance. This is your operational playbook.

Daily Operations Checklist

Morning Check (5 minutes)

1. Pipeline Health

kubectl get pods -n ais-pipeline | grep -v Running | grep -v Completed

Check for:

  • All pods Running (except completed CronJobs)
  • Yesterday’s partition materialized successfully
  • No alerts in Grafana
  • Storage usage < 80%

2. Yesterday’s Run

In Dagit → Runs tab:

  • Find yesterday’s partition
  • Status: SUCCESS
  • Duration: ~30-60 minutes (normal range)
  • All assets green in lineage view

3. Data Quality

Check validation metrics in Grafana:

ais_validation_success_rate{source="dma"}

Expected: ≥95%. If lower, investigate failed expectations.

4. Storage Usage

kubectl logs -n ais-pipeline deployment/ais-storage-monitor --tail=20

Expected output:

bronze         :     2.45 GB
silver         :   145.67 GB
gold_tracks    :    12.34 GB
TOTAL          :   169.60 GB
Storage usage: 16.9% of 1000 GB capacity

If >80%, schedule cleanup.

Weekly Review (15 minutes)

Monday morning:

  1. Success Rate: Last 7 days should be >95% successful runs
  2. Performance Trends: Asset duration increasing? Investigate optimization
  3. Capacity Planning: Storage growth rate ~8GB/day. Calculate days until full
  4. Alert Review: Were alerts actionable? Adjust thresholds if too noisy

Common Issues and Solutions

Issue 1: Schedule Didn’t Run

Symptoms:

  • Expected run not in Runs tab
  • Schedule shows green but no recent executions

Diagnosis:

kubectl logs -n ais-pipeline deployment/dagster-daemon --tail=100 | grep schedule

Common Causes:

A. Daemon Not Running

kubectl get pods -n ais-pipeline | grep dagster-daemon

# If not running:
kubectl rollout restart deployment/dagster-daemon -n ais-pipeline

B. Schedule Not Enabled

dagster schedule start daily_dma_ingestion -m pipeline.repository

C. Cron Expression Wrong

Test at https://crontab.guru. Our schedule: 0 6 * * * = 06:00 UTC daily

Issue 2: Asset Materialization Failed

Symptoms:

  • Run shows FAILURE status
  • Asset appears red in lineage view

Common Failure Types:

A. Download Failed (Bronze)

Error: HTTP 404: File not found

# Check if file exists
curl -I https://web.ais.dk/aisdata/aisdk-2024-01-15.zip

# If 404, data delayed - will auto-retry tomorrow
# Manual retry once available:
dagster asset materialize --select dma_ais_bronze --partition 2024-01-15

B. Out of Memory (Silver)

Error: DuckDB error: Out of memory

# Increase in repository.py
"memory_limit": "20GB"  # Was 16GB

Or reduce parallel processing—process one partition at a time.

C. Validation Failed

Error: Data quality validation failed: 92.3% success (threshold: 95%)

Check validation details in Dagit logs. If transient, lower threshold temporarily. If persistent, investigate data source quality.

D. S3 Connection Error

Error: MinIO error: Connection refused

# Test MinIO connectivity
kubectl exec -it deployment/dagster-daemon -n ais-pipeline -- \
  curl http://minio.ais-pipeline.svc.cluster.local:9000/minio/health/live

# Restart MinIO if unhealthy
kubectl rollout restart deployment/minio -n ais-pipeline

Issue 3: Slow Performance

Symptoms:

  • Asset materialization taking >2 hours (normally 30-60 min)
  • System becoming unresponsive

Diagnosis:

kubectl top pods -n ais-pipeline
kubectl top nodes

Solutions:

CPU Throttling:

# Increase CPU limits in values.yaml
runLauncher:
  config:
    k8sRunLauncher:
      resources:
        limits:
          cpu: "6000m"  # Was 4000m

Disk I/O Saturation:

kubectl exec -it deployment/minio -n ais-pipeline -- \
  dd if=/dev/zero of=/data/test.bin bs=1M count=1000

# If <100MB/s, storage is bottleneck
# Solutions: faster storage, reduce compression

Query Optimization:

# Add WHERE clause to filter early
query = f"""
COPY (
    SELECT * FROM read_csv_auto('{csv_path}')
    WHERE timestamp::DATE = '{partition_date}'
      AND mmsi BETWEEN 200000000 AND 799999999
) TO '{parquet_path}' (FORMAT PARQUET, COMPRESSION 'SNAPPY')
"""

Issue 4: Storage Full

Symptoms:

  • No space left on device error
  • MinIO returns 507 Insufficient Storage

Immediate Fix:

# Delete bronze layer (temporary staging)
mc rm --recursive --force ais-minio/ais-lake/bronze/

This typically frees ~50GB.

Long-term Solutions:

Implement Retention Policy:

# After successful Parquet conversion:
minio_client.remove_object("ais-lake", f"bronze/dma/{partition_date}.csv.gz")

Archive Old Data:

mc cp --recursive ais-minio/ais-lake/gold/vessel_tracks/dt=2023-*/ \
  archive-bucket/ais-lake/gold/vessel_tracks/
mc rm --recursive --force ais-minio/ais-lake/gold/vessel_tracks/dt=2023-*/

Issue 5: Alerts Firing

AISNoDataIngested (No data in 2 hours):

  1. Check if schedule ran
  2. Check data source availability
  3. Check network connectivity
  4. If source down, wait for recovery

AISValidationQualityDrop (<95% success):

  1. Check validation details in logs
  2. Compare with historical data
  3. If transient, monitor
  4. If persistent, investigate source

AISStorageNearFull (>80% capacity):

  1. Run storage cleanup
  2. Review retention policy
  3. Plan storage expansion

Maintenance Tasks

Monthly

1. Update Dependencies

helm repo update
helm search repo dagster/dagster --versions | head -10

helm upgrade dagster dagster/dagster \
  --namespace ais-pipeline \
  --values k8s/dagster/values.yaml \
  --version 1.6.1

2. Database Maintenance

kubectl exec -it deployment/dagster-postgresql -n ais-pipeline -- psql -U dagster -d dagster

# In psql:
DELETE FROM runs WHERE created_at < NOW() - INTERVAL '90 days';
DELETE FROM event_logs WHERE timestamp < NOW() - INTERVAL '90 days';
VACUUM FULL;

3. Backup Critical Data

# Backup silver layer
mc mirror ais-minio/ais-lake/silver/ backup-bucket/ais-lake/silver/

# Backup Dagster database
kubectl exec deployment/dagster-postgresql -n ais-pipeline -- \
  pg_dump -U dagster dagster | gzip > dagster-backup-$(date +%Y%m%d).sql.gz

Quarterly

  1. Cost Analysis: Storage costs, compute usage, optimization opportunities
  2. Performance Baseline: Document current durations and success rates
  3. Schema Evolution: Check for data source changes, update expectations

Annual

  1. Data Retention Review: What can be archived or deleted?
  2. Security Audit: Rotate credentials, review access logs
  3. Disaster Recovery Test: Simulate data loss, verify backup restoration

Performance Optimization

Query Optimization

Before (slow):

SELECT * FROM read_parquet('s3://ais-lake/silver/ais/**/*.parquet')
WHERE dt = '2024-01-15' AND source = 'dma';

After (40x faster):

SELECT * FROM read_parquet('s3://ais-lake/silver/ais/source=dma/dt=2024-01-15/*.parquet');

Compression Tuning

# Faster writes, larger files
COMPRESSION 'SNAPPY'

# Slower writes, 50% smaller files
COMPRESSION 'ZSTD'

# For AIS data: SNAPPY is best balance

Memory Allocation

# Too low: OOM errors
"memory_limit": "8GB"

# Too high: wastes RAM
"memory_limit": "28GB"

# Goldilocks (50% of 32GB server):
"memory_limit": "16GB"

Disaster Recovery

Scenario 1: Corrupted Silver Layer

# 1. Delete corrupted partitions
mc rm --recursive ais-minio/ais-lake/silver/ais/source=dma/dt=2024-01-15/

# 2. Reprocess from bronze
dagster asset materialize --select dma_ais_silver --partition 2024-01-15

# 3. If bronze deleted, re-download from source
dagster asset materialize --select dma_ais_bronze --partition 2024-01-15

Scenario 2: Lost PostgreSQL Database

Impact:

  • Can’t view historical runs
  • Assets still work (stateless)
  • Schedules still run

Recovery:

# Restore from backup
gunzip -c dagster-backup-20240115.sql.gz | \
  kubectl exec -i deployment/dagster-postgresql -n ais-pipeline -- \
  psql -U dagster -d dagster

# Restart Dagster
kubectl rollout restart deployment/dagster-webserver -n ais-pipeline
kubectl rollout restart deployment/dagster-daemon -n ais-pipeline

Scenario 3: Complete Storage Loss

# 1. Restore silver from backup
mc mirror backup-bucket/ais-lake/silver/ ais-minio/ais-lake/silver/

# 2. Rebuild gold layer
dagster asset materialize --select vessel_tracks port_density daily_stats \
  --partition-range 2024-01-01 2024-12-31

# 3. If no backup, reprocess from source (takes days)
dagster asset materialize --select dma_ais_bronze+ \
  --partition-range 2024-01-01 2024-12-31

Quick Reference

Essential Commands

# Health check
kubectl get pods -n ais-pipeline
kubectl top pods -n ais-pipeline

# View logs
kubectl logs -f deployment/dagster-daemon -n ais-pipeline

# Restart components
kubectl rollout restart deployment/dagster-daemon -n ais-pipeline

# Manual run
dagster asset materialize --select dma_ais_bronze+ --partition 2024-01-15

# Storage check
mc du ais-minio/ais-lake/

What You’ve Learned

This operational guide covers:

  • Daily monitoring: Quick health checks and validation
  • Troubleshooting: Common issues and step-by-step solutions
  • Maintenance: Regular tasks to keep the pipeline healthy
  • Performance tuning: Optimizations for speed and cost
  • Disaster recovery: Handling data loss and corruption

What’s Next

Chapter 13 explores extending the pipeline—adding new data sources, ML integration, and advanced analytics.

Running production data infrastructure? We help organizations build and maintain reliable data platforms. Get in touch to discuss your project.