DenCT/test-storage / system-design-aiml-roadmap.md
DenCT's picture
|
download
raw
38.4 kB

System Design & AI/ML System Design — Complete Roadmap

[!info] Target audience: 3+ years of ML/AI engineering experience, weak on production design & system design interviews. Goal: Become production-grade engineer + nail system design rounds (incl. ML, LLM, RAG, Agentic, VLM).


How to Use This Roadmap

  • Phase 0–1: Build the vocabulary of distributed systems. Without this, ML system design discussions are shallow.
  • Phase 2–3: Learn classic system design and practice 10–15 problems.
  • Phase 4–5: ML system design — frameworks, pipelines, serving, MLOps.
  • Phase 6–8: Modern AI — LLM serving, RAG, Agentic, multimodal, evals, guardrails.
  • Phase 9: Interview drilling.

Estimated calendar time: 10–14 weeks part-time (1.5–2 hrs/day), 5–6 weeks full-time.

Method per topic: (1) Read the concept, (2) Read a real-world implementation (engineering blog), (3) Draw the architecture from memory, (4) Explain it aloud in <5 min, (5) Identify trade-offs and failure modes.


PART 1 — CLASSIC SYSTEM DESIGN (A–Z)

Phase 0 — Foundations of Computing & Networks

0.1 Computer architecture refresh

  • CPU vs GPU vs TPU; memory hierarchy (L1/L2/L3 cache, RAM, disk, network).
  • Latency numbers every engineer must memorize (Jeff Dean's table):
    • L1 cache: ~0.5 ns; L2: ~7 ns; main memory: ~100 ns
    • SSD random read: ~150 µs; disk seek: ~10 ms
    • Round trip same datacenter: ~500 µs; cross-continent: ~150 ms
  • Throughput vs latency vs concurrency.
  • Big-O of common operations; back-of-envelope estimation.

0.2 Networking essentials

  • OSI model and TCP/IP stack (Layers 3, 4, 7 are interview-relevant).
  • TCP vs UDP, three-way handshake, congestion control (TCP slow start), head-of-line blocking.
  • HTTP/1.1, HTTP/2 (multiplexing), HTTP/3 (QUIC over UDP), keep-alive.
  • DNS: recursive vs iterative resolution, TTL, A/AAAA/CNAME/MX records, GeoDNS.
  • TLS handshake, mTLS, certificate chains.
  • CDNs: edge caching, push vs pull, origin shielding, Anycast, BGP routing.
  • WebSockets, Server-Sent Events (SSE), long polling vs short polling.
  • gRPC (HTTP/2 + protobuf, streaming modes: unary, server, client, bidirectional).
  • REST vs gRPC vs GraphQL vs WebSocket — when to use each.

0.3 Operating systems & concurrency

  • Processes vs threads vs coroutines.
  • Context switching cost.
  • Blocking vs non-blocking I/O, sync vs async, epoll/kqueue.
  • Memory mapping, page cache.
  • Locking: mutex, semaphore, read-write lock, optimistic locking, deadlock.
  • Concurrency models: thread-per-request, event loop (Node, Nginx), actor (Erlang/Akka), CSP (Go).

Phase 1 — Building Blocks of Distributed Systems

1.1 Theoretical foundations

  • CAP theorem — partition-tolerance is non-negotiable, so it's really CP vs AP.
  • PACELC — extension to CAP that adds latency-vs-consistency trade-off.
  • ACID vs BASE.
  • Consistency models (strongest → weakest):
    • Linearizable
    • Sequential
    • Causal
    • Read-your-writes / monotonic reads / monotonic writes / writes-follow-reads
    • Eventual
  • Isolation levels: read uncommitted, read committed, repeatable read, serializable, snapshot isolation; phenomena: dirty read, non-repeatable read, phantom read, write skew.

1.2 Distributed primitives

  • Consensus: Paxos, Raft, ZAB; leader election; quorum (R+W>N).
  • Logical clocks: Lamport timestamps, vector clocks; hybrid logical clocks (HLC).
  • Two-phase commit (2PC), three-phase commit (3PC); their failure modes.
  • Distributed transactions, Saga pattern (orchestration vs choreography).
  • Idempotency, exactly-once vs at-least-once vs at-most-once delivery.
  • Failure modes: crash, omission, Byzantine; gray failures, partial failures.
  • Time: NTP, clock skew, TrueTime (Spanner).

1.3 Hashing & partitioning

  • Consistent hashing (with virtual nodes); jump hash; rendezvous hashing.
  • Range-based vs hash-based partitioning vs directory-based.
  • Hotspot mitigation: salting, key randomization.
  • Bloom filters, Count-min sketch, HyperLogLog (when interviewers ask "how to check membership/count of billions cheaply").

Phase 2 — Core Components in Depth

2.1 Databases (SQL)

  • Storage engines: B-tree vs LSM-tree (RocksDB).
  • Indexing: clustered vs secondary; covering index; multi-column index; partial index; index selectivity.
  • Query planning, EXPLAIN, statistics.
  • MVCC, undo/redo logs, WAL.
  • Read replicas, replication lag, synchronous vs asynchronous replication.
  • Failover, split-brain, fencing tokens.
  • Connection pooling (PgBouncer); prepared statements.
  • Sharding: vertical (by table), horizontal (by row), functional. Shard key selection.
  • Postgres-specific tips: TOAST, vacuum, table partitioning, logical replication.

2.2 NoSQL

  • Wide-column (Cassandra, ScyllaDB, HBase, BigTable) — data modeling around query patterns.
  • Document (MongoDB, DynamoDB) — denormalization, single-table design.
  • Key-value (Redis, DynamoDB, Riak).
  • Graph (Neo4j, Neptune, JanusGraph) — when to use.
  • Search engines (Elasticsearch, OpenSearch, Solr) — inverted index, BM25 scoring, analyzers/tokenizers, replicas/shards.
  • Time-series (InfluxDB, TimescaleDB, Prometheus TSDB) — downsampling, retention.
  • NewSQL (Spanner, CockroachDB, TiDB, YugabyteDB).
  • Vector DBs (covered in Part 3).

2.3 Caching

  • Levels: browser, CDN, reverse proxy (Nginx, Varnish), application, database, OS page cache.
  • Strategies:
    • Cache-aside (lazy loading)
    • Read-through
    • Write-through
    • Write-back / write-behind
    • Refresh-ahead
  • Eviction policies: LRU, LFU, ARC, FIFO, TTL, random.
  • Cache problems: stampede / thundering herd (mitigations: request coalescing, jittered TTL, probabilistic early expiration), penetration, avalanche.
  • Distributed cache: Redis (cluster, sentinel), Memcached (consistent hashing client-side).
  • Cache consistency: invalidation vs TTL; double-write hazards.

2.4 Message queues, streaming, pub/sub

  • Queues vs logs (RabbitMQ vs Kafka).
  • Kafka deep-dive: topics, partitions, offsets, consumer groups, ISR, leader/follower, log compaction, exactly-once semantics, KRaft (Zookeeper-less).
  • Alternatives: Pulsar, RabbitMQ (AMQP), NATS, ActiveMQ, AWS SQS/SNS/Kinesis, GCP Pub/Sub.
  • Stream processing: Flink, Spark Streaming, Kafka Streams, Beam — windowing (tumbling, sliding, session), watermarks, exactly-once.
  • Backpressure, dead-letter queues, retry with exponential backoff + jitter.

2.5 Load balancing & traffic management

  • L4 (TCP, e.g., AWS NLB, HAProxy TCP) vs L7 (HTTP, e.g., ALB, Envoy, Nginx).
  • Algorithms: round-robin, weighted RR, least connections, least response time, IP hash, consistent hash, power-of-two-choices.
  • Health checks, sticky sessions, draining, blue/green & canary deployments.
  • Reverse proxy vs API gateway vs service mesh.
  • Rate limiting algorithms: token bucket, leaky bucket, fixed window, sliding window log, sliding window counter.
  • Throttling, circuit breakers (Hystrix pattern), bulkheads, retries with backoff + jitter, timeouts.

2.6 Storage

  • Block (EBS), file (EFS, NFS), object (S3, GCS) — semantics and use cases.
  • S3 internals at high level: consistency model (now strong read-after-write), multipart upload, lifecycle policies, storage classes.
  • HDFS, Ceph, GlusterFS.
  • Data lakes vs warehouses vs lakehouses (Iceberg, Delta Lake, Hudi).

2.7 Microservices & architecture patterns

  • Monolith → modular monolith → microservices — don't start with microservices.
  • Service discovery (client-side: Eureka; server-side: AWS ELB/Consul).
  • API gateway responsibilities: auth, rate limit, routing, aggregation, protocol translation.
  • Service mesh (Istio, Linkerd) — sidecar proxies, mTLS, traffic shaping, observability.
  • Patterns: BFF (backend-for-frontend), CQRS, Event Sourcing, Saga, Outbox, Strangler Fig, Sidecar, Ambassador, Anti-Corruption Layer.
  • Synchronous vs asynchronous communication trade-offs.

2.8 Reliability & operations

  • SLI / SLO / SLA / error budgets.
  • High availability: redundancy, replication, multi-AZ, multi-region (active-active vs active-passive).
  • DR: RTO (recovery time objective), RPO (recovery point objective), backup strategies (full, incremental, differential).
  • Chaos engineering (Chaos Monkey, Gremlin).
  • Graceful degradation, feature flags (LaunchDarkly, Unleash, OpenFeature).
  • Deployment strategies: rolling, blue/green, canary, shadow, A/B.

2.9 Observability

  • Three pillars: logs, metrics, traces.
  • Metrics: Prometheus (pull-based), Datadog, CloudWatch; PromQL basics; RED (Rate, Errors, Duration) and USE (Utilization, Saturation, Errors) methods.
  • Logs: structured logging (JSON), centralized (ELK / Loki / OpenSearch).
  • Distributed tracing: OpenTelemetry, Jaeger, Zipkin, Tempo. Trace ID propagation, spans.
  • On-call, runbooks, post-mortems (blameless), incident severity levels.

2.10 Security

  • AuthN vs AuthZ.
  • OAuth2 flows (auth code, PKCE, client credentials), OIDC, SAML, JWT (signing vs encryption, JWS vs JWE), session cookies vs tokens.
  • RBAC vs ABAC vs ReBAC; Open Policy Agent (OPA).
  • Secrets management (Vault, AWS Secrets Manager, KMS).
  • Encryption: at rest (envelope encryption), in transit (TLS), in use (confidential compute, homomorphic — rare).
  • OWASP Top 10; SQLi, XSS, CSRF, SSRF, RCE.
  • DDoS mitigation: WAF, rate limiting, anti-bot, scrubbing centers.
  • Zero-trust architecture, network segmentation, VPCs, security groups.
  • Supply-chain security (SBOM, dependency scanning, signed images — Sigstore, Cosign).

2.11 Containers & orchestration

  • Docker fundamentals: images, layers, registries, multi-stage builds, BuildKit.
  • Kubernetes core: Pods, Deployments, StatefulSets, DaemonSets, Jobs/CronJobs, Services (ClusterIP/NodePort/LoadBalancer), Ingress, HPA/VPA, ConfigMaps, Secrets, PV/PVC, namespaces, RBAC.
  • Operators, CRDs; Helm; Kustomize.
  • Networking: CNI, kube-proxy, service mesh integration.
  • GPU scheduling: device plugins, NVIDIA Operator, KubeRay, MIG partitioning.

Phase 3 — Classic System Design Practice

Drill these 15 problems. For each: (a) clarify scope, (b) capacity & QPS estimates, (c) API design, (d) data model, (e) high-level architecture, (f) deep-dive on 2 components, (g) failure handling, (h) scaling next step.

# Problem Key concepts
1 URL shortener (TinyURL/bit.ly) Hashing, base62, DB choice, cache, analytics pipeline
2 Pastebin Storage tier, expiry, CDN
3 Rate limiter Token bucket in Redis, distributed counters
4 Web crawler BFS/DFS, politeness, dedup with Bloom filter, robots.txt, frontier
5 Notification system Fan-out, queues, retries, priority
6 News feed / Twitter Fan-out on write vs read, hybrid, timeline cache
7 Chat (WhatsApp/Slack) WebSocket, presence, message ordering, E2EE basics
8 Video streaming (YouTube/Netflix) CDN, adaptive bitrate, HLS/DASH, encoding pipeline
9 Ride sharing (Uber) Geohash/H3 indexing, matching, surge pricing
10 Food delivery (DoorDash) Geo, ETA, dispatch optimization
11 Distributed file store (Dropbox/Drive) Chunking, dedup, sync, conflict resolution
12 Search autocomplete (typeahead) Trie, ranking, prefix cache
13 Distributed counter / metrics (analytics) Sharded counters, approximate algorithms
14 Payment system Idempotency, double-entry ledger, exactly-once, reconciliation
15 Ticketing (Ticketmaster) High-contention seat locking, queues, anti-bot

Practice resource priority:

  1. Alex Xu — System Design Interview Vol. 1 & 2 (read end-to-end).
  2. Designing Data-Intensive Applications (Martin Kleppmann) — the book; chapters 1–9, 11.
  3. ByteByteGo channel, System Design Primer GitHub repo, Hello Interview on YouTube.
  4. Engineering blogs: Uber, Netflix, Stripe, Discord, Cloudflare, Meta, LinkedIn, Pinterest, Shopify.

PART 2 — ML SYSTEM DESIGN

Phase 4 — ML System Design Framework

4.1 The 7-step interview framework (use this every time)

  1. Clarify the problem — business goal, success metric, constraints (latency, cost, freshness, scale).
  2. Frame as ML — supervised/unsupervised/RL; classification/regression/ranking/generation; online vs batch.
  3. Metrics — offline (AUC, NDCG, MAP, BLEU, perplexity); online (CTR, engagement, revenue); guardrail (latency p99, fairness, hallucination rate).
  4. Data — sources, volume, labels (human, semi-supervised, weak, self-supervised), privacy/PII, class imbalance, leakage.
  5. Features & model — feature engineering, embeddings, baselines first (logistic regression, GBDT), then DL; tune for bias-variance.
  6. System architecture — training pipeline, feature store, model registry, serving stack, monitoring.
  7. Iteration & scale — A/B testing, drift detection, retraining cadence, cost optimization.

4.2 Problem framing pitfalls

  • Solving the wrong problem (proxy metric divorced from business).
  • Label leakage (future info in features).
  • Train/serve skew (different code paths).
  • Sampling bias (selection effects).
  • Survivorship bias.
  • Concept of counterfactual evaluation for recommender/ads systems.

Phase 5 — ML Production Infrastructure

5.1 Data engineering for ML

  • Batch vs streaming ingestion; CDC (Debezium); ETL vs ELT.
  • Orchestration: Airflow, Prefect, Dagster, Argo Workflows.
  • Data warehouse (Snowflake, BigQuery, Redshift) vs lakehouse (Databricks, Iceberg, Delta).
  • Data quality: Great Expectations, Soda, Deequ; schema enforcement; data contracts.
  • Labeling: Label Studio, Scale AI, Snorkel (weak supervision), active learning loops.
  • Privacy: PII detection, anonymization, differential privacy, k-anonymity, federated learning (when relevant).

5.2 Feature stores

  • Why: solves online/offline parity, reuse, point-in-time correctness.
  • Components: offline store (warehouse), online store (Redis/DynamoDB), registry, materialization engine.
  • Tools: Feast (OSS), Tecton, Hopsworks, Vertex AI Feature Store, SageMaker Feature Store.
  • Point-in-time joins / time-travel — critical to avoid leakage.
  • Feature freshness, TTL, on-demand vs precomputed features.

5.3 Training infrastructure

  • Single-GPU → multi-GPU (DataParallel, DistributedDataParallel) → multi-node.
  • Parallelism strategies:
    • Data parallelism
    • Tensor / model parallelism (Megatron)
    • Pipeline parallelism (GPipe, PipeDream)
    • Sequence parallelism (long context)
    • 3D parallelism (combine all)
    • FSDP / ZeRO (DeepSpeed) — sharded optimizer/gradient/parameter.
  • Mixed precision (FP16, BF16, FP8), gradient checkpointing, gradient accumulation.
  • Distributed training infra: Ray, Horovod, NCCL, RDMA, InfiniBand.
  • Hyperparameter tuning: grid, random, Bayesian (Optuna), population-based (Ray Tune), Hyperband/ASHA.
  • Reproducibility: seed everything, pin versions, log artifacts.

5.4 Model serving patterns

  • Batch inference — Spark/Beam pipelines, write to DB; cheap, simple, high-latency.
  • Online / real-time — gRPC/REST endpoint, <100 ms latency.
  • Near-real-time / streaming — Kafka → consumer with model → output topic.
  • Edge — on-device (TFLite, Core ML, ONNX Runtime Mobile).
  • Patterns: shadow mode, canary, multi-armed bandit, champion–challenger.
  • Model servers: TorchServe, TensorFlow Serving, Triton Inference Server (NVIDIA), BentoML, KServe, Ray Serve, Seldon.
  • Triton specifics: dynamic batching, model ensembles, multiple backends (TRT, ONNX, PyTorch, Python).

5.5 Inference optimization

  • Hardware: GPU vs CPU vs TPU vs Inferentia; FP16/INT8/INT4 tensor cores.
  • Graph compilation: TorchScript, ONNX, TensorRT, OpenVINO, XLA, Apache TVM.
  • Quantization: post-training (static, dynamic), quantization-aware training (QAT); INT8/INT4.
  • Pruning (structured vs unstructured), distillation (teacher-student).
  • Operator fusion, kernel autotuning.
  • Caching: result cache, embedding cache.
  • Batching: static vs dynamic vs continuous (covered under LLM).

5.6 MLOps & lifecycle

  • Experiment tracking: MLflow, Weights & Biases, Neptune, Comet.
  • Model registry & versioning: MLflow registry, SageMaker, Vertex; data versioning with DVC, LakeFS, Pachyderm.
  • CI/CD for ML: GitHub Actions/GitLab CI + DVC + tests; data tests, model tests (slice metrics, behavioral tests à la Checklist).
  • Pipeline orchestration: Kubeflow Pipelines, Vertex AI Pipelines, Metaflow, ZenML, SageMaker Pipelines.
  • Reproducibility: deterministic builds, container images per experiment, lockfiles.
  • Cost ops (FinOps for ML): spot/preemptible instances, autoscaling, right-sizing GPUs, fractional GPU sharing.

5.7 Monitoring & observability for ML

  • Operational: latency, throughput, error rate, GPU/CPU/memory.
  • Data quality: missing values, schema drift, range violations.
  • Data drift: covariate shift (input distribution) — PSI, KL, KS test, Wasserstein.
  • Concept drift: P(y|x) changes — tracked via labeled feedback or proxy metrics.
  • Label drift: P(y) changes.
  • Performance: live metric vs ground truth (with lag).
  • Fairness & bias: subgroup metrics, disparate impact.
  • Tools: Evidently, Arize, WhyLabs, Fiddler, Aporia, Datadog ML Monitoring.
  • Retraining triggers: scheduled, drift-based, performance-based.

5.8 ML system case studies (drill these)

Problem Anchor company / paper
YouTube video recommendation Two-tower retrieval + ranking, YouTube DNN paper
Netflix recommendations Personalized ranking + contextual bandits
Instagram/TikTok feed ranking Multi-task DNN, real-time features
Twitter/X "For You" Open-sourced algo (Heavy Ranker, SimClusters)
Ad CTR prediction DCN-v2, DeepFM, DLRM (Meta)
Fraud detection GBDT + graph features + rules
Search ranking LTR (LambdaMART), neural rerankers
Spam / abuse Two-stage (cheap filter + heavy model)
Image classification at scale Distributed training, edge deploy
Document understanding OCR + layout LM
Speech recognition Streaming ASR, RNN-T / Conformer
Machine translation NMT, beam search
Personalization (Spotify) Embedding-based retrieval
Anomaly detection Statistical + isolation forest + autoencoders
Forecasting (Uber demand) Hierarchical, Prophet/DeepAR

PART 3 — LLM, RAG, AGENTIC & VLM SYSTEM DESIGN

Phase 6 — LLM System Design

6.1 LLM fundamentals (what designers must know)

  • Transformer recap: attention, multi-head, KV cache, positional encodings (RoPE, ALiBi).
  • Decoder-only (GPT family) vs encoder-decoder (T5, mT5) vs encoder-only (BERT, used for embeddings).
  • Tokenizers: BPE, SentencePiece, tiktoken; token cost implications.
  • Context window: techniques to extend (RoPE scaling, YaRN, sliding window, sparse attention, Mamba/SSMs as alternative).
  • Sampling: temperature, top-k, top-p, repetition penalty, beam search vs greedy vs sampling, logit bias.

6.2 LLM serving stack (deep)

  • vLLM — paged attention, continuous (in-flight) batching, prefix caching; default OSS choice.
  • TensorRT-LLM — NVIDIA-optimized, highest throughput; supports speculative decoding, in-flight batching.
  • TGI (Text Generation Inference) — HuggingFace; production-ready.
  • SGLang — fast for structured generation and agents.
  • Triton + TensorRT-LLM backend — production pattern at scale.
  • llama.cpp / Ollama — CPU/edge.
  • Ray Serve + vLLM — orchestrated multi-replica.
  • Key concepts:
    • KV cache — memory dominates inference; managed by paged attention.
    • Prefill vs decode — prefill is compute-bound, decode is memory-bound → different scaling.
    • Continuous batching — replaces static batching, dramatic throughput gains.
    • Speculative decoding — small draft model proposes, large verifies.
    • Prefix caching / prompt caching — reuse KV for shared system prompts.
    • Chunked prefill — overlap prefill and decode to smooth tail latency.
    • Disaggregated serving — separate prefill and decode pools (Splitwise pattern).

6.3 Inference optimization for LLMs

  • Quantization: GPTQ, AWQ, SmoothQuant, GGUF (llama.cpp), FP8.
  • KV-cache quantization (INT8/FP8 KV).
  • Flash Attention v1/v2/v3, Flash Decoding.
  • LoRA at inference: dynamic LoRA loading (vLLM supports multi-LoRA).
  • Speculative decoding variants: Medusa, EAGLE, lookahead decoding.
  • Distillation to smaller model (e.g., 70B → 8B).
  • Structured output: outlines, lm-format-enforcer, JSON-mode, grammar-constrained decoding.

6.4 Training & fine-tuning (for designers, not researchers)

  • Pre-training — usually not done in product teams; know the cost (millions of $).
  • Continued pre-training — domain adaptation on raw corpus.
  • SFT (Supervised Fine-Tuning) — instruction tuning on (prompt, response) pairs.
  • PEFT methods: LoRA, QLoRA, DoRA, prefix-tuning, P-tuning, IA³. LoRA/QLoRA is the default.
  • Preference optimization: RLHF (PPO), DPO, IPO, KTO, ORPO, SimPO.
  • Reasoning RL: GRPO (DeepSeek), RLAIF.
  • Data pipeline: deduplication (MinHash), quality filtering, decontamination, balancing.
  • Eval harness: lm-eval-harness, HELM, MT-Bench, Arena-Hard, custom domain evals.

6.5 Cost & latency math for LLM systems

  • Tokens/sec per GPU per model size — memorize ballparks for Llama-8B/70B on H100.
  • Cost = (input_tokens × input_$/1M) + (output_tokens × output_$/1M).
  • Latency = TTFT (time-to-first-token, dominated by prefill) + (output_tokens × time-per-output-token).
  • Levers: smaller model, distillation, quantization, caching, batching, regional deploy, model routing.
  • Throughput vs latency trade-off curve — design choice.

Phase 7 — RAG System Design

7.1 RAG architecture stages

  1. Ingestion — load, parse, clean, chunk, embed, index, store metadata.
  2. Retrieval — query understanding, embedding, search, filter, rerank.
  3. Generation — prompt assembly, LLM call, citation, post-processing.
  4. Evaluation — offline + online.

7.2 Document processing

  • Parsing: PDFs (PyMuPDF, Unstructured, LlamaParse, Marker, Docling), HTML, Office docs, images (OCR via Tesseract, PaddleOCR, TrOCR).
  • Layout-aware parsing critical for tables, multi-column.
  • Cleaning: remove boilerplate, normalize whitespace, dedup.

7.3 Chunking strategies

  • Fixed-size (token / char) with overlap.
  • Recursive character splitter.
  • Sentence/paragraph aware (NLTK, spaCy).
  • Semantic chunking (embedding-similarity-based splits).
  • Document-structure aware (markdown headers, code blocks).
  • Late chunking (embed whole doc, then chunk the embeddings).
  • Hierarchical (parent-child): retrieve small chunks, return larger context.
  • Choose per content type — code, tables, prose differ.

7.4 Embedding models

  • Closed: OpenAI text-embedding-3, Cohere embed-v3, Voyage.
  • Open: BGE, E5, GTE, Nomic, Jina, mxbai, Stella, Snowflake Arctic, Mistral embed.
  • Dimensions (Matryoshka representations let you truncate).
  • Specialized: code (CodeBERT, Voyage-code), multilingual, multimodal (CLIP, SigLIP).
  • Eval: MTEB benchmark — but build a domain eval set; MTEB rank does not transfer.
  • Fine-tuning embeddings on your data (contrastive, hard negatives) often beats picking a better OSS model.

7.5 Vector databases & indexes

  • Algorithms: HNSW (most common), IVF, IVF-PQ, ScaNN, DiskANN.
  • Trade-offs: recall vs latency vs memory vs build cost.
  • Products:
    • Pinecone (managed, serverless)
    • Weaviate
    • Qdrant
    • Milvus / Zilliz
    • Chroma (dev / small)
    • pgvector (Postgres extension) — strong default if already on Postgres
    • Elasticsearch / OpenSearch (BM25 + dense in one)
    • Vespa (production search at scale)
    • LanceDB, Turbopuffer (cheap object-storage-backed)
  • Filters: pre-filter vs post-filter vs hybrid filter — performance differs sharply.
  • Multi-tenancy: namespace per tenant; collection per tenant; metadata filter — pick based on tenant count.

7.6 Retrieval techniques

  • Sparse: BM25, SPLADE (learned sparse).
  • Dense: bi-encoder embeddings.
  • Hybrid: combine via RRF (reciprocal rank fusion) or weighted sum.
  • Multi-vector: ColBERT / ColPali — token-level late interaction; higher quality, more storage.
  • Query transformation:
    • HyDE (hypothetical document embeddings)
    • Multi-query (generate N variants)
    • Step-back prompting
    • Query decomposition for multi-hop
    • Routing (which index / source)
  • Reranking: cross-encoder (e.g., bge-reranker, Cohere Rerank, Voyage rerank); LLM-as-reranker (cost).
  • Contextual retrieval (Anthropic): prepend chunk-specific context using LLM at ingest.
  • GraphRAG (Microsoft): build entity/community graph, summarize, retrieve by community.
  • Agentic RAG: agent loops over retrieval, reads, decides whether to keep searching.

7.7 Advanced RAG patterns

  • Naive RAG → Advanced RAG (pre+post retrieval) → Modular RAG.
  • Self-RAG, CRAG (corrective RAG), Self-correcting loops.
  • Long-context RAG vs RAG vs fine-tuning — when to choose each.
  • Lost-in-the-middle problem; reorder retrieved chunks.
  • Citation enforcement (return spans, validate).
  • Multi-modal RAG (text + image, ColPali for screenshots/PDFs).

7.8 RAG evaluation

  • Components to eval independently: retrieval (recall@k, MRR, NDCG, hit rate), generation (faithfulness, answer relevance), end-to-end.
  • Frameworks: Ragas, TruLens, DeepEval, Arize Phoenix, LangSmith.
  • LLM-as-judge — calibrate vs human labels; use pairwise > pointwise.
  • Build a golden eval set (50–200 questions with reference answers) before scaling anything.
  • Online: thumbs up/down, downstream task completion, edit distance from user edits.

7.9 RAG production concerns

  • Freshness: incremental ingestion, change data capture from source.
  • Permissions: row/document-level ACLs in vector DB filters.
  • PII handling and redaction.
  • Caching: semantic cache (GPTCache), exact prompt cache.
  • Cost: embedding cost (one-time at ingest), retrieval cost (cheap), generation cost (dominant).

Phase 8 — Agentic AI System Design

8.1 Agent paradigms

  • ReAct (reason + act): interleave thought / action / observation.
  • Plan-and-Execute: plan first, then execute steps.
  • Reflexion / Self-critique: agent reviews its own output.
  • Tree-of-Thoughts / Graph-of-Thoughts.
  • Multi-agent: supervisor/worker, debate, role-based (CrewAI), graph-based (LangGraph).

8.2 Building blocks

  • Tool use / function calling: schema (JSON), parallel calls, tool error handling, retries.
  • Memory:
    • Short-term: conversation buffer, summary buffer.
    • Long-term: vector store of past interactions, episodic memory.
    • Semantic memory: facts, KG.
    • Working memory: scratchpad.
    • Mem0, Letta (MemGPT), Zep, LangMem — frameworks.
  • Planning: HTN, LLM-generated DAG, code-as-plan (CodeAct).
  • Reflection / verification: separate critic LLM, deterministic verifier (unit tests, JSON-schema validation).

8.3 Frameworks (know strengths, not just names)

  • LangChain / LangGraph — LangGraph is the production-grade graph runtime (state, branching, human-in-loop, checkpointing).
  • LlamaIndex — data-centric, agents add-on.
  • CrewAI — role-based multi-agent.
  • AutoGen (Microsoft) — multi-agent conversation.
  • Semantic Kernel — Microsoft, .NET friendly.
  • DSPy — declarative, optimizable pipelines and prompts.
  • Pydantic AI, Atomic Agents — typed agent frameworks.
  • OpenAI Agents SDK, Anthropic SDK with tool use.
  • Smolagents (HF) — code-as-action minimalism.

8.4 Agent orchestration & infra

  • State management: graph state, checkpoints (LangGraph SqliteSaver/Postgres), replayability.
  • Concurrency: parallel tool calls, fan-out/fan-in.
  • Human-in-the-loop: approval gates, edit-and-resume.
  • Streaming: stream tokens + tool events to UI; SSE/WebSocket.
  • Timeouts, max-step limits, cost caps.
  • Sandboxed code execution: E2B, Modal, Daytona, Firecracker microVMs.
  • Browser agents: Browserbase, Playwright, computer-use models.

8.5 Evaluating agents

  • Trajectory-level vs final-answer-level metrics.
  • Tool-call accuracy, plan correctness, cost per task.
  • Benchmarks: AgentBench, GAIA, SWE-bench, WebArena, OSWorld, τ-bench, AppWorld.
  • Build domain-specific task suites — golden trajectories.

8.6 Agentic production challenges

  • Non-determinism — pin seeds where possible, snapshot test inputs/outputs.
  • Cost — agents loop; enforce step + token budgets.
  • Latency — parallelize tool calls; cache plan steps.
  • Safety — restrict tool capabilities, sandbox writes, allow-list URLs, output filters.
  • Observability — full trace per run (LangSmith, Langfuse, Arize, Helicone).
  • Versioning — versions of prompts, tools, models, schemas; replay old traces against new versions.

8.7 Protocols

  • MCP (Model Context Protocol) — Anthropic-led standard for tool/resource servers.
  • A2A (Agent-to-Agent) — emerging inter-agent protocols.
  • OpenAPI as tool spec — common pattern.

Phase 9 — VLM / Multimodal System Design

9.1 Multimodal models

  • CLIP / SigLIP — contrastive vision-text; embeddings for search.
  • BLIP-2 / InstructBLIP — Q-Former bridge.
  • LLaVA family, Qwen-VL, Llama 3.2 Vision, InternVL, Pixtral, Phi-Vision, Molmo, Idefics — open VLMs.
  • GPT-4o, Claude (vision), Gemini — closed.
  • Video VLMs: VideoLLaMA, Video-LLaVA, Gemini.
  • Document VLMs: Donut, Nougat, ColPali, GOT-OCR.

9.2 Use cases & patterns

  • Visual question answering, document understanding, chart/table extraction.
  • Multimodal RAG (retrieve images by text/embedding; pass images into VLM).
  • OCR-free document QA (ColPali + VLM).
  • UI agents / screen understanding (Anthropic computer use, OmniParser).
  • Robotics / embodied AI is its own world — out of scope.

9.3 Multimodal infra specifics

  • Image preprocessing: resize, tile/patch (high-res), normalization; influences token count drastically.
  • Storage: object store + CDN; signed URLs; dedup by perceptual hash.
  • Inference cost: image tokens often dominate; bound max images, downsample.
  • Caching image embeddings.
  • Safety: NSFW filters, CSAM detection (hash matching, e.g., PhotoDNA), face redaction.

PART 4 — CROSS-CUTTING: PRODUCTION-GRADE AI

10.1 Evaluation (broader)

  • Offline: dataset → metric. Build golden sets per use case.
  • Online: A/B, interleaving, guardrail metrics.
  • Human eval: pairwise > Likert; calibrate raters; inter-rater agreement (Cohen's kappa).
  • LLM-as-judge: known biases (position, verbosity, self-preference) — mitigate with randomization & rubric.
  • Continuous evals: run on every release; alert on regressions.

10.2 Safety, guardrails, alignment-in-product

  • Input filters: PII detection, prompt-injection detection, jailbreak detection (Lakera, Prompt Guard, Llama Guard).
  • Output filters: toxicity (Perspective API, Detoxify), hallucination check (faithfulness vs sources), PII leakage.
  • Constrained generation: JSON schema, regex grammars, function calling.
  • Tools: NeMo Guardrails, Guardrails AI, Llama Guard, ShieldGemma.
  • Policy/red-teaming process; abuse monitoring; rate limiting & per-user quotas.
  • Prompt-injection defenses: separate trusted and untrusted contexts, sandbox tools, never let model execute arbitrary tools on untrusted input.

10.3 Cost & latency engineering for AI

  • Model routing / cascading (cheap → expensive based on confidence).
  • Distillation to smaller in-house model after collecting traffic.
  • Semantic cache for repeated queries.
  • Prompt caching (Anthropic, OpenAI, Gemini all support).
  • Batched embedding ingestion.
  • Async / streaming UX to hide latency.
  • Region-local serving to cut RTT.
  • Speculative decoding, smaller draft models.
  • Right-size GPUs (A10G < L4 < L40S < A100 < H100 < H200 < B200); fractional MIG for small models.

10.4 Data flywheel / continuous learning

  • Capture user feedback (explicit + implicit).
  • Curate logs into training/eval data with PII scrubbing.
  • Active learning: surface uncertain or edge-case examples for labeling.
  • Closed-loop: prod logs → eval set → fine-tune → A/B → ship.
  • Synthetic data: use stronger model to bootstrap weaker model's data.

10.5 LLMOps stack (concrete)

  • Gateway: LiteLLM, Portkey, OpenRouter, Kong AI Gateway (multi-provider, fallback, cost tracking).
  • Tracing/Observability: LangSmith, Langfuse, Arize Phoenix, Helicone, Honeyhive, Braintrust.
  • Eval: Ragas, DeepEval, Promptfoo, Inspect, Braintrust.
  • Vector DB + Embedding service.
  • Prompt versioning: PromptLayer, Helicone, Langfuse, in-repo + git.
  • Feature flags & experimentation: LaunchDarkly, Statsig.

10.6 Common production reference architectures (memorize these blueprints)

(a) Enterprise RAG chatbot

  • Ingest: connectors (S3, SharePoint, Confluence, Slack) → parser → chunker → embedder → vector DB + metadata.
  • Auth & ACL propagation into retriever filters.
  • Query: query rewrite → hybrid search → rerank → prompt assembly → LLM (with prompt cache) → citations → post-filter.
  • Observability + feedback capture + eval harness.

(b) Agentic copilot

  • LangGraph state machine; tools (DB query, API calls, code exec).
  • Per-user memory store.
  • Sandbox for code; allow-list for HTTP.
  • Streaming + interrupts for human approval.
  • Trace store + replay.

(c) Real-time recommendation system

  • Candidate generation (two-tower, ANN) → ranking (DLRM-style) → re-ranking with business rules.
  • Feature store (online + offline parity).
  • Logging for counterfactual eval.
  • Online learning / hourly retrain.

(d) Fraud / abuse detection

  • Streaming features (Flink) → low-latency model (GBDT/DL) + rules engine → action service → human review queue.

(e) Search with semantic + lexical

  • Indexer (BM25 + dense embeddings).
  • Multi-stage: retrieval → light ranker → heavy cross-encoder.
  • Personalization layer.

PART 5 — INTERVIEW PREPARATION

Phase 10 — Interview Drilling

11.1 ML system design interview template (use this verbatim)

  1. Clarify: business problem, scope, scale (DAU, QPS, data volume), latency budget, freshness, cost.
  2. Metrics: business → ML → guardrail. Trade-offs.
  3. Data: sources, labeling, volume, biases, privacy.
  4. Architecture overview: pipelines (data, training, serving, monitoring).
  5. Modeling: baseline → enhanced; features; loss; eval.
  6. Serving: latency, throughput, scaling, fallback.
  7. Monitoring & feedback.
  8. Trade-offs and what you'd improve with more time.

11.2 Common ML/LLM interview questions to drill

  • Design YouTube recommendations.
  • Design ad CTR prediction at Meta.
  • Design fraud detection for Stripe.
  • Design semantic search across enterprise docs.
  • Design ChatGPT-style assistant with citations over a private corpus (RAG).
  • Design an agent that books travel end-to-end.
  • Design a code assistant (Copilot-like).
  • Design a multimodal product-search system (image+text).
  • Design hallucination-detection / fact-checking pipeline.
  • Design model serving for a 70B LLM with <1s p50 latency and 100 QPS.
  • Design A/B test framework for LLM products (ratios, novelty effects, guardrails).
  • Build a personalization system with cold-start.
  • Design content moderation at scale.

11.3 Behavioral / staff-level signals

  • Talk about trade-offs, not just choices.
  • Lead with metrics and constraints.
  • Surface failure modes proactively.
  • Distinguish "what I'd build now" vs "what I'd build at 100× scale."
  • Discuss organizational and rollout aspects: ownership, on-call, ramp-up, rollback.

11.4 Practice cadence

  • Solo: 1 problem/day, 60 min timed, then 30 min review.
  • Peer mocks: 2/week with feedback.
  • Record yourself once a week and rewatch.
  • After each problem: write a 1-page "would do differently" note.

Resources (curated, in order)

Books

  1. Designing Data-Intensive Applications — Kleppmann (mandatory).
  2. System Design Interview Vol. 1 & 2 — Alex Xu.
  3. Designing Machine Learning Systems — Chip Huyen (mandatory).
  4. Machine Learning System Design Interview — Ali Aminian & Alex Xu.
  5. AI Engineering — Chip Huyen (2024).
  6. Building LLMs for Production — Louis-François Bouchard.
  7. Hands-On Large Language Models — Jay Alammar & Maarten Grootendorst.
  8. Site Reliability Engineering — Google (free online).

Courses / channels

  • ByteByteGo (YouTube + newsletter).
  • Hello Interview (YouTube).
  • Stanford CS329S — ML Systems Design.
  • Full Stack Deep Learning.
  • DeepLearning.AI short courses (RAG, agents, evals, function calling).
  • LangChain Academy (LangGraph).

Engineering blogs to read weekly

  • Anthropic Engineering ("Building effective agents", "Contextual retrieval").
  • OpenAI cookbook, Google Research, Meta AI, Microsoft Research blogs.
  • Netflix Tech Blog, Uber Engineering, Stripe, Cloudflare, Discord, Pinterest, LinkedIn, Shopify, DoorDash, Airbnb.
  • Eugene Yan, Chip Huyen, Hamel Husain, Jason Liu, Lilian Weng, Sebastian Raschka.

Papers (read at least the abstracts of these)

  • Attention Is All You Need, GPT-3, LLaMA, RAG (Lewis et al.), ReAct, Toolformer, Self-RAG, ColBERT/ColPali, FlashAttention, vLLM/PagedAttention, LoRA, DPO, DLRM, Wide & Deep, YouTube DNN, Two Tower, DCN-v2.

Final Tracker (check off as you go)

  • Phase 0 — Foundations of computing & networks
  • Phase 1 — Building blocks of distributed systems
  • Phase 2 — Core components in depth
  • Phase 3 — 15 classic system design problems
  • Phase 4 — ML system design framework
  • Phase 5 — ML production infra (data, training, serving, MLOps, monitoring)
  • Phase 6 — LLM system design + serving
  • Phase 7 — RAG system design
  • Phase 8 — Agentic AI system design
  • Phase 9 — VLM / multimodal
  • Phase 10 — Cross-cutting production AI (cost, safety, evals, LLMOps)
  • Phase 11 — Interview drilling (20+ mocks)

When you can sketch any of the reference architectures in section 10.6 on a whiteboard in 10 minutes and defend every choice — you're ready.

Xet Storage Details

Size:
38.4 kB
·
Xet hash:
cc903ba652b0d33b853a10abc7ae48c4c51c9deecb52da701ea69d0a61297973

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.