DenCT/test-storage / system-design-aiml-roadmap.md
DenCT's picture
|
download
raw
38.4 kB
# System Design & AI/ML System Design — Complete Roadmap
> [!info] Target audience: 3+ years of ML/AI engineering experience, weak on production design & system design interviews.
> Goal: Become production-grade engineer + nail system design rounds (incl. ML, LLM, RAG, Agentic, VLM).
---
## How to Use This Roadmap
- **Phase 0–1**: Build the *vocabulary* of distributed systems. Without this, ML system design discussions are shallow.
- **Phase 2–3**: Learn classic system design and practice 10–15 problems.
- **Phase 4–5**: ML system design — frameworks, pipelines, serving, MLOps.
- **Phase 6–8**: Modern AI — LLM serving, RAG, Agentic, multimodal, evals, guardrails.
- **Phase 9**: Interview drilling.
**Estimated calendar time**: 10–14 weeks part-time (1.5–2 hrs/day), 5–6 weeks full-time.
**Method per topic**: (1) Read the *concept*, (2) Read a *real-world implementation* (engineering blog), (3) *Draw* the architecture from memory, (4) *Explain it aloud* in <5 min, (5) Identify *trade-offs and failure modes*.
---
# PART 1 — CLASSIC SYSTEM DESIGN (A–Z)
## Phase 0 — Foundations of Computing & Networks
### 0.1 Computer architecture refresh
- CPU vs GPU vs TPU; memory hierarchy (L1/L2/L3 cache, RAM, disk, network).
- Latency numbers every engineer must memorize (Jeff Dean's table):
- L1 cache: ~0.5 ns; L2: ~7 ns; main memory: ~100 ns
- SSD random read: ~150 µs; disk seek: ~10 ms
- Round trip same datacenter: ~500 µs; cross-continent: ~150 ms
- Throughput vs latency vs concurrency.
- Big-O of common operations; back-of-envelope estimation.
### 0.2 Networking essentials
- OSI model and TCP/IP stack (Layers 3, 4, 7 are interview-relevant).
- TCP vs UDP, three-way handshake, congestion control (TCP slow start), head-of-line blocking.
- HTTP/1.1, HTTP/2 (multiplexing), HTTP/3 (QUIC over UDP), keep-alive.
- DNS: recursive vs iterative resolution, TTL, A/AAAA/CNAME/MX records, GeoDNS.
- TLS handshake, mTLS, certificate chains.
- CDNs: edge caching, push vs pull, origin shielding, Anycast, BGP routing.
- WebSockets, Server-Sent Events (SSE), long polling vs short polling.
- gRPC (HTTP/2 + protobuf, streaming modes: unary, server, client, bidirectional).
- REST vs gRPC vs GraphQL vs WebSocket — when to use each.
### 0.3 Operating systems & concurrency
- Processes vs threads vs coroutines.
- Context switching cost.
- Blocking vs non-blocking I/O, sync vs async, epoll/kqueue.
- Memory mapping, page cache.
- Locking: mutex, semaphore, read-write lock, optimistic locking, deadlock.
- Concurrency models: thread-per-request, event loop (Node, Nginx), actor (Erlang/Akka), CSP (Go).
## Phase 1 — Building Blocks of Distributed Systems
### 1.1 Theoretical foundations
- **CAP theorem** — partition-tolerance is non-negotiable, so it's really CP vs AP.
- **PACELC** — extension to CAP that adds latency-vs-consistency trade-off.
- **ACID** vs **BASE**.
- Consistency models (strongest → weakest):
- Linearizable
- Sequential
- Causal
- Read-your-writes / monotonic reads / monotonic writes / writes-follow-reads
- Eventual
- Isolation levels: read uncommitted, read committed, repeatable read, serializable, snapshot isolation; phenomena: dirty read, non-repeatable read, phantom read, write skew.
### 1.2 Distributed primitives
- Consensus: Paxos, Raft, ZAB; leader election; quorum (R+W>N).
- Logical clocks: Lamport timestamps, vector clocks; hybrid logical clocks (HLC).
- Two-phase commit (2PC), three-phase commit (3PC); their failure modes.
- Distributed transactions, Saga pattern (orchestration vs choreography).
- Idempotency, exactly-once vs at-least-once vs at-most-once delivery.
- Failure modes: crash, omission, Byzantine; gray failures, partial failures.
- Time: NTP, clock skew, TrueTime (Spanner).
### 1.3 Hashing & partitioning
- Consistent hashing (with virtual nodes); jump hash; rendezvous hashing.
- Range-based vs hash-based partitioning vs directory-based.
- Hotspot mitigation: salting, key randomization.
- Bloom filters, Count-min sketch, HyperLogLog (when interviewers ask "how to check membership/count of billions cheaply").
## Phase 2 — Core Components in Depth
### 2.1 Databases (SQL)
- Storage engines: B-tree vs LSM-tree (RocksDB).
- Indexing: clustered vs secondary; covering index; multi-column index; partial index; index selectivity.
- Query planning, EXPLAIN, statistics.
- MVCC, undo/redo logs, WAL.
- Read replicas, replication lag, synchronous vs asynchronous replication.
- Failover, split-brain, fencing tokens.
- Connection pooling (PgBouncer); prepared statements.
- Sharding: vertical (by table), horizontal (by row), functional. Shard key selection.
- Postgres-specific tips: TOAST, vacuum, table partitioning, logical replication.
### 2.2 NoSQL
- Wide-column (Cassandra, ScyllaDB, HBase, BigTable) — data modeling around query patterns.
- Document (MongoDB, DynamoDB) — denormalization, single-table design.
- Key-value (Redis, DynamoDB, Riak).
- Graph (Neo4j, Neptune, JanusGraph) — when to use.
- Search engines (Elasticsearch, OpenSearch, Solr) — inverted index, BM25 scoring, analyzers/tokenizers, replicas/shards.
- Time-series (InfluxDB, TimescaleDB, Prometheus TSDB) — downsampling, retention.
- NewSQL (Spanner, CockroachDB, TiDB, YugabyteDB).
- Vector DBs (covered in Part 3).
### 2.3 Caching
- Levels: browser, CDN, reverse proxy (Nginx, Varnish), application, database, OS page cache.
- Strategies:
- Cache-aside (lazy loading)
- Read-through
- Write-through
- Write-back / write-behind
- Refresh-ahead
- Eviction policies: LRU, LFU, ARC, FIFO, TTL, random.
- Cache problems: stampede / thundering herd (mitigations: request coalescing, jittered TTL, probabilistic early expiration), penetration, avalanche.
- Distributed cache: Redis (cluster, sentinel), Memcached (consistent hashing client-side).
- Cache consistency: invalidation vs TTL; double-write hazards.
### 2.4 Message queues, streaming, pub/sub
- Queues vs logs (RabbitMQ vs Kafka).
- Kafka deep-dive: topics, partitions, offsets, consumer groups, ISR, leader/follower, log compaction, exactly-once semantics, KRaft (Zookeeper-less).
- Alternatives: Pulsar, RabbitMQ (AMQP), NATS, ActiveMQ, AWS SQS/SNS/Kinesis, GCP Pub/Sub.
- Stream processing: Flink, Spark Streaming, Kafka Streams, Beam — windowing (tumbling, sliding, session), watermarks, exactly-once.
- Backpressure, dead-letter queues, retry with exponential backoff + jitter.
### 2.5 Load balancing & traffic management
- L4 (TCP, e.g., AWS NLB, HAProxy TCP) vs L7 (HTTP, e.g., ALB, Envoy, Nginx).
- Algorithms: round-robin, weighted RR, least connections, least response time, IP hash, consistent hash, power-of-two-choices.
- Health checks, sticky sessions, draining, blue/green & canary deployments.
- Reverse proxy vs API gateway vs service mesh.
- Rate limiting algorithms: token bucket, leaky bucket, fixed window, sliding window log, sliding window counter.
- Throttling, circuit breakers (Hystrix pattern), bulkheads, retries with backoff + jitter, timeouts.
### 2.6 Storage
- Block (EBS), file (EFS, NFS), object (S3, GCS) — semantics and use cases.
- S3 internals at high level: consistency model (now strong read-after-write), multipart upload, lifecycle policies, storage classes.
- HDFS, Ceph, GlusterFS.
- Data lakes vs warehouses vs lakehouses (Iceberg, Delta Lake, Hudi).
### 2.7 Microservices & architecture patterns
- Monolith → modular monolith → microservices — *don't start with microservices*.
- Service discovery (client-side: Eureka; server-side: AWS ELB/Consul).
- API gateway responsibilities: auth, rate limit, routing, aggregation, protocol translation.
- Service mesh (Istio, Linkerd) — sidecar proxies, mTLS, traffic shaping, observability.
- Patterns: BFF (backend-for-frontend), CQRS, Event Sourcing, Saga, Outbox, Strangler Fig, Sidecar, Ambassador, Anti-Corruption Layer.
- Synchronous vs asynchronous communication trade-offs.
### 2.8 Reliability & operations
- SLI / SLO / SLA / error budgets.
- High availability: redundancy, replication, multi-AZ, multi-region (active-active vs active-passive).
- DR: RTO (recovery time objective), RPO (recovery point objective), backup strategies (full, incremental, differential).
- Chaos engineering (Chaos Monkey, Gremlin).
- Graceful degradation, feature flags (LaunchDarkly, Unleash, OpenFeature).
- Deployment strategies: rolling, blue/green, canary, shadow, A/B.
### 2.9 Observability
- Three pillars: logs, metrics, traces.
- Metrics: Prometheus (pull-based), Datadog, CloudWatch; PromQL basics; RED (Rate, Errors, Duration) and USE (Utilization, Saturation, Errors) methods.
- Logs: structured logging (JSON), centralized (ELK / Loki / OpenSearch).
- Distributed tracing: OpenTelemetry, Jaeger, Zipkin, Tempo. Trace ID propagation, spans.
- On-call, runbooks, post-mortems (blameless), incident severity levels.
### 2.10 Security
- AuthN vs AuthZ.
- OAuth2 flows (auth code, PKCE, client credentials), OIDC, SAML, JWT (signing vs encryption, JWS vs JWE), session cookies vs tokens.
- RBAC vs ABAC vs ReBAC; Open Policy Agent (OPA).
- Secrets management (Vault, AWS Secrets Manager, KMS).
- Encryption: at rest (envelope encryption), in transit (TLS), in use (confidential compute, homomorphic — rare).
- OWASP Top 10; SQLi, XSS, CSRF, SSRF, RCE.
- DDoS mitigation: WAF, rate limiting, anti-bot, scrubbing centers.
- Zero-trust architecture, network segmentation, VPCs, security groups.
- Supply-chain security (SBOM, dependency scanning, signed images — Sigstore, Cosign).
### 2.11 Containers & orchestration
- Docker fundamentals: images, layers, registries, multi-stage builds, BuildKit.
- Kubernetes core: Pods, Deployments, StatefulSets, DaemonSets, Jobs/CronJobs, Services (ClusterIP/NodePort/LoadBalancer), Ingress, HPA/VPA, ConfigMaps, Secrets, PV/PVC, namespaces, RBAC.
- Operators, CRDs; Helm; Kustomize.
- Networking: CNI, kube-proxy, service mesh integration.
- GPU scheduling: device plugins, NVIDIA Operator, KubeRay, MIG partitioning.
## Phase 3 — Classic System Design Practice
Drill these 15 problems. For each: (a) clarify scope, (b) capacity & QPS estimates, (c) API design, (d) data model, (e) high-level architecture, (f) deep-dive on 2 components, (g) failure handling, (h) scaling next step.
| # | Problem | Key concepts |
|---|---|---|
| 1 | URL shortener (TinyURL/bit.ly) | Hashing, base62, DB choice, cache, analytics pipeline |
| 2 | Pastebin | Storage tier, expiry, CDN |
| 3 | Rate limiter | Token bucket in Redis, distributed counters |
| 4 | Web crawler | BFS/DFS, politeness, dedup with Bloom filter, robots.txt, frontier |
| 5 | Notification system | Fan-out, queues, retries, priority |
| 6 | News feed / Twitter | Fan-out on write vs read, hybrid, timeline cache |
| 7 | Chat (WhatsApp/Slack) | WebSocket, presence, message ordering, E2EE basics |
| 8 | Video streaming (YouTube/Netflix) | CDN, adaptive bitrate, HLS/DASH, encoding pipeline |
| 9 | Ride sharing (Uber) | Geohash/H3 indexing, matching, surge pricing |
| 10 | Food delivery (DoorDash) | Geo, ETA, dispatch optimization |
| 11 | Distributed file store (Dropbox/Drive) | Chunking, dedup, sync, conflict resolution |
| 12 | Search autocomplete (typeahead) | Trie, ranking, prefix cache |
| 13 | Distributed counter / metrics (analytics) | Sharded counters, approximate algorithms |
| 14 | Payment system | Idempotency, double-entry ledger, exactly-once, reconciliation |
| 15 | Ticketing (Ticketmaster) | High-contention seat locking, queues, anti-bot |
**Practice resource priority**:
1. Alex Xu — *System Design Interview* Vol. 1 & 2 (read end-to-end).
2. Designing Data-Intensive Applications (Martin Kleppmann) — *the* book; chapters 1–9, 11.
3. ByteByteGo channel, *System Design Primer* GitHub repo, *Hello Interview* on YouTube.
4. Engineering blogs: Uber, Netflix, Stripe, Discord, Cloudflare, Meta, LinkedIn, Pinterest, Shopify.
---
# PART 2 — ML SYSTEM DESIGN
## Phase 4 — ML System Design Framework
### 4.1 The 7-step interview framework (use this every time)
1. **Clarify the problem** — business goal, success metric, constraints (latency, cost, freshness, scale).
2. **Frame as ML** — supervised/unsupervised/RL; classification/regression/ranking/generation; online vs batch.
3. **Metrics** — offline (AUC, NDCG, MAP, BLEU, perplexity); online (CTR, engagement, revenue); guardrail (latency p99, fairness, hallucination rate).
4. **Data** — sources, volume, labels (human, semi-supervised, weak, self-supervised), privacy/PII, class imbalance, leakage.
5. **Features & model** — feature engineering, embeddings, baselines first (logistic regression, GBDT), then DL; tune for bias-variance.
6. **System architecture** — training pipeline, feature store, model registry, serving stack, monitoring.
7. **Iteration & scale** — A/B testing, drift detection, retraining cadence, cost optimization.
### 4.2 Problem framing pitfalls
- Solving the wrong problem (proxy metric divorced from business).
- Label leakage (future info in features).
- Train/serve skew (different code paths).
- Sampling bias (selection effects).
- Survivorship bias.
- Concept of *counterfactual evaluation* for recommender/ads systems.
## Phase 5 — ML Production Infrastructure
### 5.1 Data engineering for ML
- Batch vs streaming ingestion; CDC (Debezium); ETL vs ELT.
- Orchestration: Airflow, Prefect, Dagster, Argo Workflows.
- Data warehouse (Snowflake, BigQuery, Redshift) vs lakehouse (Databricks, Iceberg, Delta).
- Data quality: Great Expectations, Soda, Deequ; schema enforcement; data contracts.
- Labeling: Label Studio, Scale AI, Snorkel (weak supervision), active learning loops.
- Privacy: PII detection, anonymization, differential privacy, k-anonymity, federated learning (when relevant).
### 5.2 Feature stores
- Why: solves online/offline parity, reuse, point-in-time correctness.
- Components: offline store (warehouse), online store (Redis/DynamoDB), registry, materialization engine.
- Tools: Feast (OSS), Tecton, Hopsworks, Vertex AI Feature Store, SageMaker Feature Store.
- Point-in-time joins / time-travel — critical to avoid leakage.
- Feature freshness, TTL, on-demand vs precomputed features.
### 5.3 Training infrastructure
- Single-GPU → multi-GPU (DataParallel, DistributedDataParallel) → multi-node.
- Parallelism strategies:
- Data parallelism
- Tensor / model parallelism (Megatron)
- Pipeline parallelism (GPipe, PipeDream)
- Sequence parallelism (long context)
- 3D parallelism (combine all)
- FSDP / ZeRO (DeepSpeed) — sharded optimizer/gradient/parameter.
- Mixed precision (FP16, BF16, FP8), gradient checkpointing, gradient accumulation.
- Distributed training infra: Ray, Horovod, NCCL, RDMA, InfiniBand.
- Hyperparameter tuning: grid, random, Bayesian (Optuna), population-based (Ray Tune), Hyperband/ASHA.
- Reproducibility: seed everything, pin versions, log artifacts.
### 5.4 Model serving patterns
- **Batch inference** — Spark/Beam pipelines, write to DB; cheap, simple, high-latency.
- **Online / real-time** — gRPC/REST endpoint, <100 ms latency.
- **Near-real-time / streaming** — Kafka → consumer with model → output topic.
- **Edge** — on-device (TFLite, Core ML, ONNX Runtime Mobile).
- Patterns: shadow mode, canary, multi-armed bandit, champion–challenger.
- Model servers: TorchServe, TensorFlow Serving, Triton Inference Server (NVIDIA), BentoML, KServe, Ray Serve, Seldon.
- Triton specifics: dynamic batching, model ensembles, multiple backends (TRT, ONNX, PyTorch, Python).
### 5.5 Inference optimization
- Hardware: GPU vs CPU vs TPU vs Inferentia; FP16/INT8/INT4 tensor cores.
- Graph compilation: TorchScript, ONNX, TensorRT, OpenVINO, XLA, Apache TVM.
- Quantization: post-training (static, dynamic), quantization-aware training (QAT); INT8/INT4.
- Pruning (structured vs unstructured), distillation (teacher-student).
- Operator fusion, kernel autotuning.
- Caching: result cache, embedding cache.
- Batching: static vs dynamic vs continuous (covered under LLM).
### 5.6 MLOps & lifecycle
- **Experiment tracking**: MLflow, Weights & Biases, Neptune, Comet.
- **Model registry & versioning**: MLflow registry, SageMaker, Vertex; data versioning with DVC, LakeFS, Pachyderm.
- **CI/CD for ML**: GitHub Actions/GitLab CI + DVC + tests; data tests, model tests (slice metrics, behavioral tests à la Checklist).
- **Pipeline orchestration**: Kubeflow Pipelines, Vertex AI Pipelines, Metaflow, ZenML, SageMaker Pipelines.
- **Reproducibility**: deterministic builds, container images per experiment, lockfiles.
- **Cost ops (FinOps for ML)**: spot/preemptible instances, autoscaling, right-sizing GPUs, fractional GPU sharing.
### 5.7 Monitoring & observability for ML
- **Operational**: latency, throughput, error rate, GPU/CPU/memory.
- **Data quality**: missing values, schema drift, range violations.
- **Data drift**: covariate shift (input distribution) — PSI, KL, KS test, Wasserstein.
- **Concept drift**: P(y|x) changes — tracked via labeled feedback or proxy metrics.
- **Label drift**: P(y) changes.
- **Performance**: live metric vs ground truth (with lag).
- **Fairness & bias**: subgroup metrics, disparate impact.
- Tools: Evidently, Arize, WhyLabs, Fiddler, Aporia, Datadog ML Monitoring.
- Retraining triggers: scheduled, drift-based, performance-based.
### 5.8 ML system case studies (drill these)
| Problem | Anchor company / paper |
|---|---|
| YouTube video recommendation | Two-tower retrieval + ranking, *YouTube DNN* paper |
| Netflix recommendations | Personalized ranking + contextual bandits |
| Instagram/TikTok feed ranking | Multi-task DNN, real-time features |
| Twitter/X "For You" | Open-sourced algo (Heavy Ranker, SimClusters) |
| Ad CTR prediction | DCN-v2, DeepFM, DLRM (Meta) |
| Fraud detection | GBDT + graph features + rules |
| Search ranking | LTR (LambdaMART), neural rerankers |
| Spam / abuse | Two-stage (cheap filter + heavy model) |
| Image classification at scale | Distributed training, edge deploy |
| Document understanding | OCR + layout LM |
| Speech recognition | Streaming ASR, RNN-T / Conformer |
| Machine translation | NMT, beam search |
| Personalization (Spotify) | Embedding-based retrieval |
| Anomaly detection | Statistical + isolation forest + autoencoders |
| Forecasting (Uber demand) | Hierarchical, Prophet/DeepAR |
---
# PART 3 — LLM, RAG, AGENTIC & VLM SYSTEM DESIGN
## Phase 6 — LLM System Design
### 6.1 LLM fundamentals (what designers must know)
- Transformer recap: attention, multi-head, KV cache, positional encodings (RoPE, ALiBi).
- Decoder-only (GPT family) vs encoder-decoder (T5, mT5) vs encoder-only (BERT, used for embeddings).
- Tokenizers: BPE, SentencePiece, tiktoken; token cost implications.
- Context window: techniques to extend (RoPE scaling, YaRN, sliding window, sparse attention, Mamba/SSMs as alternative).
- Sampling: temperature, top-k, top-p, repetition penalty, beam search vs greedy vs sampling, logit bias.
### 6.2 LLM serving stack (deep)
- **vLLM** — paged attention, continuous (in-flight) batching, prefix caching; default OSS choice.
- **TensorRT-LLM** — NVIDIA-optimized, highest throughput; supports speculative decoding, in-flight batching.
- **TGI (Text Generation Inference)** — HuggingFace; production-ready.
- **SGLang** — fast for structured generation and agents.
- **Triton + TensorRT-LLM backend** — production pattern at scale.
- **llama.cpp** / Ollama — CPU/edge.
- **Ray Serve + vLLM** — orchestrated multi-replica.
- Key concepts:
- **KV cache** — memory dominates inference; managed by paged attention.
- **Prefill vs decode** — prefill is compute-bound, decode is memory-bound → different scaling.
- **Continuous batching** — replaces static batching, dramatic throughput gains.
- **Speculative decoding** — small draft model proposes, large verifies.
- **Prefix caching / prompt caching** — reuse KV for shared system prompts.
- **Chunked prefill** — overlap prefill and decode to smooth tail latency.
- **Disaggregated serving** — separate prefill and decode pools (Splitwise pattern).
### 6.3 Inference optimization for LLMs
- Quantization: GPTQ, AWQ, SmoothQuant, GGUF (llama.cpp), FP8.
- KV-cache quantization (INT8/FP8 KV).
- Flash Attention v1/v2/v3, Flash Decoding.
- LoRA at inference: dynamic LoRA loading (vLLM supports multi-LoRA).
- Speculative decoding variants: Medusa, EAGLE, lookahead decoding.
- Distillation to smaller model (e.g., 70B → 8B).
- Structured output: outlines, lm-format-enforcer, JSON-mode, grammar-constrained decoding.
### 6.4 Training & fine-tuning (for designers, not researchers)
- **Pre-training** — usually not done in product teams; know the cost (millions of $).
- **Continued pre-training** — domain adaptation on raw corpus.
- **SFT (Supervised Fine-Tuning)** — instruction tuning on (prompt, response) pairs.
- **PEFT methods**: LoRA, QLoRA, DoRA, prefix-tuning, P-tuning, IA³. LoRA/QLoRA is the default.
- **Preference optimization**: RLHF (PPO), DPO, IPO, KTO, ORPO, SimPO.
- **Reasoning RL**: GRPO (DeepSeek), RLAIF.
- Data pipeline: deduplication (MinHash), quality filtering, decontamination, balancing.
- Eval harness: lm-eval-harness, HELM, MT-Bench, Arena-Hard, custom domain evals.
### 6.5 Cost & latency math for LLM systems
- Tokens/sec per GPU per model size — memorize ballparks for Llama-8B/70B on H100.
- Cost = (input_tokens × input_$/1M) + (output_tokens × output_$/1M).
- Latency = TTFT (time-to-first-token, dominated by prefill) + (output_tokens × time-per-output-token).
- Levers: smaller model, distillation, quantization, caching, batching, regional deploy, model routing.
- Throughput vs latency trade-off curve — design choice.
## Phase 7 — RAG System Design
### 7.1 RAG architecture stages
1. **Ingestion** — load, parse, clean, chunk, embed, index, store metadata.
2. **Retrieval** — query understanding, embedding, search, filter, rerank.
3. **Generation** — prompt assembly, LLM call, citation, post-processing.
4. **Evaluation** — offline + online.
### 7.2 Document processing
- Parsing: PDFs (PyMuPDF, Unstructured, LlamaParse, Marker, Docling), HTML, Office docs, images (OCR via Tesseract, PaddleOCR, TrOCR).
- Layout-aware parsing critical for tables, multi-column.
- Cleaning: remove boilerplate, normalize whitespace, dedup.
### 7.3 Chunking strategies
- Fixed-size (token / char) with overlap.
- Recursive character splitter.
- Sentence/paragraph aware (NLTK, spaCy).
- Semantic chunking (embedding-similarity-based splits).
- Document-structure aware (markdown headers, code blocks).
- Late chunking (embed whole doc, then chunk the embeddings).
- Hierarchical (parent-child): retrieve small chunks, return larger context.
- Choose per content type — code, tables, prose differ.
### 7.4 Embedding models
- Closed: OpenAI text-embedding-3, Cohere embed-v3, Voyage.
- Open: BGE, E5, GTE, Nomic, Jina, mxbai, Stella, Snowflake Arctic, Mistral embed.
- Dimensions (Matryoshka representations let you truncate).
- Specialized: code (CodeBERT, Voyage-code), multilingual, multimodal (CLIP, SigLIP).
- Eval: MTEB benchmark — but build a *domain* eval set; MTEB rank does not transfer.
- Fine-tuning embeddings on your data (contrastive, hard negatives) often beats picking a better OSS model.
### 7.5 Vector databases & indexes
- Algorithms: HNSW (most common), IVF, IVF-PQ, ScaNN, DiskANN.
- Trade-offs: recall vs latency vs memory vs build cost.
- Products:
- Pinecone (managed, serverless)
- Weaviate
- Qdrant
- Milvus / Zilliz
- Chroma (dev / small)
- pgvector (Postgres extension) — *strong default if already on Postgres*
- Elasticsearch / OpenSearch (BM25 + dense in one)
- Vespa (production search at scale)
- LanceDB, Turbopuffer (cheap object-storage-backed)
- Filters: pre-filter vs post-filter vs hybrid filter — performance differs sharply.
- Multi-tenancy: namespace per tenant; collection per tenant; metadata filter — pick based on tenant count.
### 7.6 Retrieval techniques
- **Sparse**: BM25, SPLADE (learned sparse).
- **Dense**: bi-encoder embeddings.
- **Hybrid**: combine via RRF (reciprocal rank fusion) or weighted sum.
- **Multi-vector**: ColBERT / ColPali — token-level late interaction; higher quality, more storage.
- **Query transformation**:
- HyDE (hypothetical document embeddings)
- Multi-query (generate N variants)
- Step-back prompting
- Query decomposition for multi-hop
- Routing (which index / source)
- **Reranking**: cross-encoder (e.g., bge-reranker, Cohere Rerank, Voyage rerank); LLM-as-reranker (cost).
- **Contextual retrieval** (Anthropic): prepend chunk-specific context using LLM at ingest.
- **GraphRAG** (Microsoft): build entity/community graph, summarize, retrieve by community.
- **Agentic RAG**: agent loops over retrieval, reads, decides whether to keep searching.
### 7.7 Advanced RAG patterns
- Naive RAG → Advanced RAG (pre+post retrieval) → Modular RAG.
- Self-RAG, CRAG (corrective RAG), Self-correcting loops.
- Long-context RAG vs RAG vs fine-tuning — when to choose each.
- Lost-in-the-middle problem; reorder retrieved chunks.
- Citation enforcement (return spans, validate).
- Multi-modal RAG (text + image, ColPali for screenshots/PDFs).
### 7.8 RAG evaluation
- Components to eval independently: retrieval (recall@k, MRR, NDCG, hit rate), generation (faithfulness, answer relevance), end-to-end.
- Frameworks: Ragas, TruLens, DeepEval, Arize Phoenix, LangSmith.
- LLM-as-judge — calibrate vs human labels; use pairwise > pointwise.
- Build a *golden eval set* (50–200 questions with reference answers) before scaling anything.
- Online: thumbs up/down, downstream task completion, edit distance from user edits.
### 7.9 RAG production concerns
- Freshness: incremental ingestion, change data capture from source.
- Permissions: row/document-level ACLs in vector DB filters.
- PII handling and redaction.
- Caching: semantic cache (GPTCache), exact prompt cache.
- Cost: embedding cost (one-time at ingest), retrieval cost (cheap), generation cost (dominant).
## Phase 8 — Agentic AI System Design
### 8.1 Agent paradigms
- **ReAct** (reason + act): interleave thought / action / observation.
- **Plan-and-Execute**: plan first, then execute steps.
- **Reflexion / Self-critique**: agent reviews its own output.
- **Tree-of-Thoughts / Graph-of-Thoughts**.
- **Multi-agent**: supervisor/worker, debate, role-based (CrewAI), graph-based (LangGraph).
### 8.2 Building blocks
- **Tool use / function calling**: schema (JSON), parallel calls, tool error handling, retries.
- **Memory**:
- Short-term: conversation buffer, summary buffer.
- Long-term: vector store of past interactions, episodic memory.
- Semantic memory: facts, KG.
- Working memory: scratchpad.
- Mem0, Letta (MemGPT), Zep, LangMem — frameworks.
- **Planning**: HTN, LLM-generated DAG, code-as-plan (CodeAct).
- **Reflection / verification**: separate critic LLM, deterministic verifier (unit tests, JSON-schema validation).
### 8.3 Frameworks (know strengths, not just names)
- **LangChain / LangGraph** — LangGraph is the production-grade graph runtime (state, branching, human-in-loop, checkpointing).
- **LlamaIndex** — data-centric, agents add-on.
- **CrewAI** — role-based multi-agent.
- **AutoGen** (Microsoft) — multi-agent conversation.
- **Semantic Kernel** — Microsoft, .NET friendly.
- **DSPy** — declarative, optimizable pipelines and prompts.
- **Pydantic AI**, **Atomic Agents** — typed agent frameworks.
- **OpenAI Agents SDK**, **Anthropic SDK with tool use**.
- **Smolagents** (HF) — code-as-action minimalism.
### 8.4 Agent orchestration & infra
- State management: graph state, checkpoints (LangGraph SqliteSaver/Postgres), replayability.
- Concurrency: parallel tool calls, fan-out/fan-in.
- Human-in-the-loop: approval gates, edit-and-resume.
- Streaming: stream tokens + tool events to UI; SSE/WebSocket.
- Timeouts, max-step limits, cost caps.
- Sandboxed code execution: E2B, Modal, Daytona, Firecracker microVMs.
- Browser agents: Browserbase, Playwright, computer-use models.
### 8.5 Evaluating agents
- Trajectory-level vs final-answer-level metrics.
- Tool-call accuracy, plan correctness, cost per task.
- Benchmarks: AgentBench, GAIA, SWE-bench, WebArena, OSWorld, τ-bench, AppWorld.
- Build domain-specific task suites — golden trajectories.
### 8.6 Agentic production challenges
- Non-determinism — pin seeds where possible, snapshot test inputs/outputs.
- Cost — agents loop; enforce step + token budgets.
- Latency — parallelize tool calls; cache plan steps.
- Safety — restrict tool capabilities, sandbox writes, allow-list URLs, output filters.
- Observability — full trace per run (LangSmith, Langfuse, Arize, Helicone).
- Versioning — versions of prompts, tools, models, schemas; replay old traces against new versions.
### 8.7 Protocols
- **MCP (Model Context Protocol)** — Anthropic-led standard for tool/resource servers.
- **A2A (Agent-to-Agent)** — emerging inter-agent protocols.
- **OpenAPI as tool spec** — common pattern.
## Phase 9 — VLM / Multimodal System Design
### 9.1 Multimodal models
- **CLIP / SigLIP** — contrastive vision-text; embeddings for search.
- **BLIP-2 / InstructBLIP** — Q-Former bridge.
- **LLaVA family, Qwen-VL, Llama 3.2 Vision, InternVL, Pixtral, Phi-Vision, Molmo, Idefics** — open VLMs.
- **GPT-4o, Claude (vision), Gemini** — closed.
- **Video VLMs**: VideoLLaMA, Video-LLaVA, Gemini.
- **Document VLMs**: Donut, Nougat, ColPali, GOT-OCR.
### 9.2 Use cases & patterns
- Visual question answering, document understanding, chart/table extraction.
- Multimodal RAG (retrieve images by text/embedding; pass images into VLM).
- OCR-free document QA (ColPali + VLM).
- UI agents / screen understanding (Anthropic computer use, OmniParser).
- Robotics / embodied AI is its own world — out of scope.
### 9.3 Multimodal infra specifics
- Image preprocessing: resize, tile/patch (high-res), normalization; influences token count drastically.
- Storage: object store + CDN; signed URLs; dedup by perceptual hash.
- Inference cost: image tokens often dominate; bound max images, downsample.
- Caching image embeddings.
- Safety: NSFW filters, CSAM detection (hash matching, e.g., PhotoDNA), face redaction.
---
# PART 4 — CROSS-CUTTING: PRODUCTION-GRADE AI
### 10.1 Evaluation (broader)
- Offline: dataset → metric. Build *golden sets* per use case.
- Online: A/B, interleaving, guardrail metrics.
- Human eval: pairwise > Likert; calibrate raters; inter-rater agreement (Cohen's kappa).
- LLM-as-judge: known biases (position, verbosity, self-preference) — mitigate with randomization & rubric.
- Continuous evals: run on every release; alert on regressions.
### 10.2 Safety, guardrails, alignment-in-product
- Input filters: PII detection, prompt-injection detection, jailbreak detection (Lakera, Prompt Guard, Llama Guard).
- Output filters: toxicity (Perspective API, Detoxify), hallucination check (faithfulness vs sources), PII leakage.
- Constrained generation: JSON schema, regex grammars, function calling.
- Tools: NeMo Guardrails, Guardrails AI, Llama Guard, ShieldGemma.
- Policy/red-teaming process; abuse monitoring; rate limiting & per-user quotas.
- Prompt-injection defenses: separate trusted and untrusted contexts, sandbox tools, never let model execute arbitrary tools on untrusted input.
### 10.3 Cost & latency engineering for AI
- Model routing / cascading (cheap → expensive based on confidence).
- Distillation to smaller in-house model after collecting traffic.
- Semantic cache for repeated queries.
- Prompt caching (Anthropic, OpenAI, Gemini all support).
- Batched embedding ingestion.
- Async / streaming UX to hide latency.
- Region-local serving to cut RTT.
- Speculative decoding, smaller draft models.
- Right-size GPUs (A10G < L4 < L40S < A100 < H100 < H200 < B200); fractional MIG for small models.
### 10.4 Data flywheel / continuous learning
- Capture user feedback (explicit + implicit).
- Curate logs into training/eval data with PII scrubbing.
- Active learning: surface uncertain or edge-case examples for labeling.
- Closed-loop: prod logs → eval set → fine-tune → A/B → ship.
- Synthetic data: use stronger model to bootstrap weaker model's data.
### 10.5 LLMOps stack (concrete)
- Gateway: LiteLLM, Portkey, OpenRouter, Kong AI Gateway (multi-provider, fallback, cost tracking).
- Tracing/Observability: LangSmith, Langfuse, Arize Phoenix, Helicone, Honeyhive, Braintrust.
- Eval: Ragas, DeepEval, Promptfoo, Inspect, Braintrust.
- Vector DB + Embedding service.
- Prompt versioning: PromptLayer, Helicone, Langfuse, in-repo + git.
- Feature flags & experimentation: LaunchDarkly, Statsig.
### 10.6 Common production reference architectures (memorize these blueprints)
**(a) Enterprise RAG chatbot**
- Ingest: connectors (S3, SharePoint, Confluence, Slack) → parser → chunker → embedder → vector DB + metadata.
- Auth & ACL propagation into retriever filters.
- Query: query rewrite → hybrid search → rerank → prompt assembly → LLM (with prompt cache) → citations → post-filter.
- Observability + feedback capture + eval harness.
**(b) Agentic copilot**
- LangGraph state machine; tools (DB query, API calls, code exec).
- Per-user memory store.
- Sandbox for code; allow-list for HTTP.
- Streaming + interrupts for human approval.
- Trace store + replay.
**(c) Real-time recommendation system**
- Candidate generation (two-tower, ANN) → ranking (DLRM-style) → re-ranking with business rules.
- Feature store (online + offline parity).
- Logging for counterfactual eval.
- Online learning / hourly retrain.
**(d) Fraud / abuse detection**
- Streaming features (Flink) → low-latency model (GBDT/DL) + rules engine → action service → human review queue.
**(e) Search with semantic + lexical**
- Indexer (BM25 + dense embeddings).
- Multi-stage: retrieval → light ranker → heavy cross-encoder.
- Personalization layer.
---
# PART 5 — INTERVIEW PREPARATION
## Phase 10 — Interview Drilling
### 11.1 ML system design interview template (use this verbatim)
1. Clarify: business problem, scope, scale (DAU, QPS, data volume), latency budget, freshness, cost.
2. Metrics: business → ML → guardrail. Trade-offs.
3. Data: sources, labeling, volume, biases, privacy.
4. Architecture overview: pipelines (data, training, serving, monitoring).
5. Modeling: baseline → enhanced; features; loss; eval.
6. Serving: latency, throughput, scaling, fallback.
7. Monitoring & feedback.
8. Trade-offs and what you'd improve with more time.
### 11.2 Common ML/LLM interview questions to drill
- Design YouTube recommendations.
- Design ad CTR prediction at Meta.
- Design fraud detection for Stripe.
- Design semantic search across enterprise docs.
- Design ChatGPT-style assistant with citations over a private corpus (RAG).
- Design an agent that books travel end-to-end.
- Design a code assistant (Copilot-like).
- Design a multimodal product-search system (image+text).
- Design hallucination-detection / fact-checking pipeline.
- Design model serving for a 70B LLM with <1s p50 latency and 100 QPS.
- Design A/B test framework for LLM products (ratios, novelty effects, guardrails).
- Build a personalization system with cold-start.
- Design content moderation at scale.
### 11.3 Behavioral / staff-level signals
- Talk about *trade-offs*, not just choices.
- Lead with metrics and constraints.
- Surface failure modes proactively.
- Distinguish "what I'd build now" vs "what I'd build at 100× scale."
- Discuss organizational and rollout aspects: ownership, on-call, ramp-up, rollback.
### 11.4 Practice cadence
- Solo: 1 problem/day, 60 min timed, then 30 min review.
- Peer mocks: 2/week with feedback.
- Record yourself once a week and rewatch.
- After each problem: write a 1-page "would do differently" note.
---
## Resources (curated, in order)
**Books**
1. *Designing Data-Intensive Applications* — Kleppmann (mandatory).
2. *System Design Interview Vol. 1 & 2* — Alex Xu.
3. *Designing Machine Learning Systems* — Chip Huyen (mandatory).
4. *Machine Learning System Design Interview* — Ali Aminian & Alex Xu.
5. *AI Engineering* — Chip Huyen (2024).
6. *Building LLMs for Production* — Louis-François Bouchard.
7. *Hands-On Large Language Models* — Jay Alammar & Maarten Grootendorst.
8. *Site Reliability Engineering* — Google (free online).
**Courses / channels**
- ByteByteGo (YouTube + newsletter).
- Hello Interview (YouTube).
- Stanford CS329S — ML Systems Design.
- Full Stack Deep Learning.
- DeepLearning.AI short courses (RAG, agents, evals, function calling).
- LangChain Academy (LangGraph).
**Engineering blogs to read weekly**
- Anthropic Engineering ("Building effective agents", "Contextual retrieval").
- OpenAI cookbook, Google Research, Meta AI, Microsoft Research blogs.
- Netflix Tech Blog, Uber Engineering, Stripe, Cloudflare, Discord, Pinterest, LinkedIn, Shopify, DoorDash, Airbnb.
- Eugene Yan, Chip Huyen, Hamel Husain, Jason Liu, Lilian Weng, Sebastian Raschka.
**Papers (read at least the abstracts of these)**
- *Attention Is All You Need*, *GPT-3*, *LLaMA*, *RAG (Lewis et al.)*, *ReAct*, *Toolformer*, *Self-RAG*, *ColBERT/ColPali*, *FlashAttention*, *vLLM/PagedAttention*, *LoRA*, *DPO*, *DLRM*, *Wide & Deep*, *YouTube DNN*, *Two Tower*, *DCN-v2*.
---
## Final Tracker (check off as you go)
- [ ] Phase 0 — Foundations of computing & networks
- [ ] Phase 1 — Building blocks of distributed systems
- [ ] Phase 2 — Core components in depth
- [ ] Phase 3 — 15 classic system design problems
- [ ] Phase 4 — ML system design framework
- [ ] Phase 5 — ML production infra (data, training, serving, MLOps, monitoring)
- [ ] Phase 6 — LLM system design + serving
- [ ] Phase 7 — RAG system design
- [ ] Phase 8 — Agentic AI system design
- [ ] Phase 9 — VLM / multimodal
- [ ] Phase 10 — Cross-cutting production AI (cost, safety, evals, LLMOps)
- [ ] Phase 11 — Interview drilling (20+ mocks)
When you can sketch any of the reference architectures in section 10.6 on a whiteboard in 10 minutes and defend every choice — you're ready.

Xet Storage Details

Size:
38.4 kB
·
Xet hash:
cc903ba652b0d33b853a10abc7ae48c4c51c9deecb52da701ea69d0a61297973

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.