Blog

Notes from the work

Longer pieces on data platforms, MLOps and AI adoption, and the occasional argument about what these models can and cannot do.

Building an AI-Ready Data Platform

A series in 13 parts. Start at part one.

  1. 1 Ubuntu Server and Kubernetes Setup for Your Home AI Platform Set up a production-grade machine learning infrastructure on a single-node Kubernetes cluster using affordable mini PC hardware. The foundation for your AI data platform.
  2. 2 Setting Up Your MLOps Infrastructure: MinIO, Dagster, and Monitoring on Kubernetes Deploy production-grade object storage, orchestration, and observability tools on your home Kubernetes cluster using Helm charts and LoadBalancer services.
  3. 3 Designing Your Data Lake: The Medallion Architecture Pattern Learn how to structure a scalable data lake using the bronze, silver, and gold medallion pattern with MinIO, Parquet, and Hive-style partitioning.
  4. 4 Connecting to AIS Data Sources: Implementing the Bronze layer Learn how to download and stage maritime AIS data from public sources with streaming downloads, retry logic, and direct upload to object storage.
  5. 5 Dagster Fundamentals: Asset-Based Orchestration for Data Pipelines Learn Dagster's asset-based paradigm and how it differs from task-based orchestrators. Build your first data assets with partitions, resources, and dependencies.
  6. 6 Bronze to Silver: Transforming CSV to Parquet with DuckDB Learn how to convert raw CSV files to optimized Parquet format using DuckDB, with data validation, partitioning, and direct S3 integration.
  7. 7 Automated Data Quality with Great Expectations Implement production-grade data validation using Great Expectations 1.x to catch bad data before it corrupts downstream analytics and ML models.
  8. 8 Silver to Gold: Building Business-Ready Curated Datasets Transform validated silver layer data into pre-aggregated gold datasets including vessel tracks, geographic density maps, and daily statistics.
  9. 9 Pipeline Observability: Prometheus Metrics, Grafana Dashboards, and Alerting Implement comprehensive monitoring for your data pipeline with custom Prometheus metrics, real-time Grafana dashboards, and proactive alerting.
  10. 10 Deploying Your Data Pipeline to Kubernetes with Helm Package and deploy a production-ready data pipeline to Kubernetes using Docker images, Helm charts, secrets management, and LoadBalancer services.
  11. 11 Automating Your Pipeline: Daily Schedules and Historical Backfills Configure Dagster schedules for automated daily ingestion and use backfills to process historical data efficiently.
  12. 12 Production Operations: Health Checks, Troubleshooting, and Maintenance Your operational playbook for running a data pipeline in production—daily monitoring, common issues, performance tuning, and disaster recovery.
  13. 13 Extending Your Pipeline: New Data Sources, ML Integration, and Advanced Analytics Learn how to extend your data pipeline with additional data sources, machine learning workflows, custom analytics, and external integrations.

AI Agents

A series in 2 parts. Start at part one.

  1. 1 AI Agents in Software Development: What Engineering Organisations Are Actually Running A tour of the background agents Google, Spotify, Meta, Airbnb and others run today, what they achieved, how they wired them together, and what the evidence does and does not support.
  2. 2 Multi-Agent Architecture for Customer Support Chatbots: What Actually Works Multi-agent is a spectrum, not an architecture. The evidence on where distributed support bots fail, the shape that has won in production, and the questions to ask in any architecture review.