Designing Machine Learning Systems

An Engineer's Field Guide to Building ML That Ships, Scales, and Survives Production

By: Jordan O'Neal

Narrated by: Virtual Voice

Inside this book, readers will learn how to:

• Apply a five-question framework before writing a single line of architecture — identifying what decision the system supports, who consumes its output, how fresh that output must be, what a wrong output costs, and what the system does when it is unavailable
• Design the data layer for ML workloads from first principles — covering data contracts, schema versioning, lineage tracking, and the boundary discipline that prevents the silent cross-team failures most ML incidents trace back to
• Build feature engineering pipelines with point-in-time correctness — understanding the offline-online gap, the training-serving skew, and the feature store patterns that prevent well-evaluated models from silently underperforming in production
• Select the right serving architecture — batch, real-time, streaming, or hybrid — using a requirements-first decision method that maps latency, freshness, and cost constraints onto architecture before any tool is chosen
• Design deployment and release strategies for ML systems — including blue-green deployments, canary releases, shadow mode testing, behavioral evaluation gates, and the rollback procedures that make model updates safe to ship
• Instrument ML systems for production reliability — monitoring for data drift, concept drift, prediction distribution shift, and the silent degradation patterns that make offline metrics unreliable predictors of production performance
• Architect for regulated environments — translating HIPAA, GDPR, SOC 2, and EU AI Act requirements into specific architectural hooks, audit trail designs, and explainability patterns that satisfy compliance from the first design document rather than the first audit finding
• Structure and deliver a complete ML system design in an interview setting — using the book's five-question opening, context diagram, and layered design method that mirrors exactly what senior interviewers test at ML-focused companies
• Walk two production systems from blank page to full design — a recommendation engine and a fraud detection system, each annotated with the decisions made, alternatives rejected, and trade-offs accepted

The eleven chapters follow the arc of a system design engagement. The first three chapters establish the mental model and the foundational disciplines: the system context diagram, the five-question framework, the architect-engineer-lead triangle, requirements and constraints documentation, and data layer design with explicit boundary contracts. The middle chapters build the model-facing and serving layers: feature stores and point-in-time correctness, model development as an engineering discipline, and four serving architecture patterns with decision criteria for each. The final chapters cover operations, interviews, regulated environments, and two fully worked reference designs.
Every chapter closes with a Quick Check designed to test whether the concepts have settled, and a Think About This section that applies the chapter's method to a real design scenario. The patterns are tool-neutral. Specific platforms appear as examples of underlying patterns, not as the patterns themselves — because tools change and the design thinking that selects them does not.
The question the staff engineer asked in that design review is not a hard question. It just needs to be automatic. By the last page of this book, it will be.