fix: ship the collection files — data/ was excluded from git AND the image

The collections feature was committed without its content. `git ls-files data/`
was empty: data/ is gitignored for the 2.7GB metadata sidecar, so the eight
curated JSON files were never tracked. A fresh clone, and the deployed image,
would have rendered an empty Collections page while every test passed --
the tests load the real directory from the working tree, which had the files.

.dockerignore had the same blanket `data/` rule, so even once tracked they
would not have reached the image.

git: `data/*` rather than `data/`, because git cannot re-include a file whose
parent DIRECTORY is excluded -- with `data/`, the `!data/collections/`
negation silently does nothing. Verified with git check-ignore: the JSON files
are visible, data/metadata.sqlite is still ignored via *.sqlite.

docker: no negation at all. Docker's nested-path exception semantics are easy
to get subtly wrong and untestable without a daemon, and the *.sqlite rule
already excluded the sidecar -- the blanket data/ rule was redundant for its
stated purpose. Now excludes data/*.sqlite and data/*.db explicitly and lets
everything else through.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.dockerignore

@@ -7,7 +7,18 @@ __pycache__
 *.pyo
 *.db
 *.sqlite
-data/
+# data/ is deliberately NOT excluded wholesale. A blanket `data/` rule shipped
+# an image with an empty collections page: data/collections/*.json is curated
+# content that must be in the image, and a negation exception (`!data/...`)
+# depends on Docker's nested-path semantics, which are easy to get subtly wrong
+# and cannot be tested without a Docker daemon.
+#
+# The thing that actually needs excluding is the 2.7GB metadata sidecar, which
+# the image fetches at build time rather than copying from the host. The
+# *.sqlite and *.db rules above already cover it; these are explicit for
+# anyone reading this file in isolation.
+data/*.sqlite
+data/*.db
 *.ipynb
 pdf_images/
 notebooks/

.gitignore

@@ -40,7 +40,13 @@ e2e_output.txt
 
 # Metadata sidecar — ~1.5GB, built by scripts/build_metadata_sidecar.py and
 # fetched at image build time. Never commit it.
-data/
+#
+# `data/*` rather than `data/`: git cannot re-include a file whose parent
+# DIRECTORY is excluded, so excluding the directory itself would make the
+# negation below silently do nothing. Curated collections are small, hand-
+# written content and must be version-controlled.
+data/*
+!data/collections/
 *.sqlite
 
 # Generated / scratch

data/collections/agents-tools.json

@@ -0,0 +1,15 @@
+{
+  "slug": "agents-tools",
+  "title": "Agents & Tool Use",
+  "blurb": "From prompting tricks to systems that act. The through-line is giving a model a scratchpad, then a tool, then a loop.",
+  "anchors": [
+    {"id": "2201.11903", "note": "Ask for the reasoning and accuracy jumps on multi-step problems. The prompt as a scratchpad."},
+    {"id": "2203.11171", "note": "Sample several reasoning paths and take the majority. Cheap, and reliably better than one greedy chain."},
+    {"id": "2210.03629", "note": "Interleaves reasoning with actions so the model can look things up mid-thought. The template most agents still follow."},
+    {"id": "2302.04761", "note": "The model teaches itself when to call an API, in a self-supervised way. Tool use as a learned skill."},
+    {"id": "2303.11366", "note": "Verbal self-critique carried across attempts, as an alternative to weight updates."},
+    {"id": "2305.10601", "note": "Search over a tree of partial reasoning states rather than committing to one chain."},
+    {"id": "2304.03442", "note": "Memory, reflection and planning in a simulated society. Where the agent-architecture vocabulary comes from."},
+    {"id": "2305.16291", "note": "An open-ended agent that writes and stores its own skills. Lifelong learning without gradient updates."}
+  ]
+}

data/collections/diffusion-vision.json

@@ -0,0 +1,15 @@
+{
+  "slug": "diffusion-vision",
+  "title": "Diffusion & Generative Vision",
+  "blurb": "The theory that made image generation work, then the engineering that made it fast enough to ship.",
+  "anchors": [
+    {"id": "2006.11239", "note": "The paper that made diffusion competitive. A simple noise-prediction objective, and suddenly it trains."},
+    {"id": "2011.13456", "note": "Unifies score matching and diffusion as one SDE. The theory the rest of the field is written in."},
+    {"id": "2207.12598", "note": "Drops the separate classifier and gets better control from the model itself. Every image model uses this."},
+    {"id": "2112.10752", "note": "Run diffusion in a compressed latent space instead of pixels. This is what made Stable Diffusion possible."},
+    {"id": "2205.11487", "note": "Shows a large frozen text encoder matters more than a bigger image model."},
+    {"id": "2212.09748", "note": "Replaces the U-Net with a transformer and shows it scales predictably. The backbone of current video models."},
+    {"id": "2302.05543", "note": "Adds spatial conditioning to a frozen model without destroying it. Turned generation into something directable."},
+    {"id": "2307.01952", "note": "The engineering write-up: what actually had to change to make latent diffusion production-grade."}
+  ]
+}

data/collections/embeddings-representation.json

@@ -0,0 +1,14 @@
+{
+  "slug": "embeddings-representation",
+  "title": "Embeddings & Representation",
+  "blurb": "How text became vectors you can search. This is the stack ResearchIT itself runs on.",
+  "anchors": [
+    {"id": "1810.04805", "note": "Bidirectional pretraining. The starting point for almost every encoder that follows."},
+    {"id": "1908.10084", "note": "BERT embeddings are poor for similarity out of the box; a siamese objective fixes it. The paper that made semantic search practical."},
+    {"id": "2104.08821", "note": "Contrastive learning where dropout alone provides the positive pair. Startlingly simple, startlingly strong."},
+    {"id": "2212.03533", "note": "Weakly-supervised contrastive pretraining at scale, and a demonstration that data curation carries most of the gain."},
+    {"id": "2309.07597", "note": "The BGE family, including the BGE-M3 model this app encodes with. Multilingual, multi-granularity, dense plus sparse."},
+    {"id": "2210.07316", "note": "The benchmark that made embedding models comparable. Also a caution: leaderboard rank does not transfer to your domain."},
+    {"id": "2205.13147", "note": "Train one vector that can be truncated to many sizes. The way to cut vector storage without re-encoding a corpus."}
+  ]
+}

data/collections/evaluation.json

@@ -0,0 +1,13 @@
+{
+  "slug": "evaluation",
+  "title": "Evaluating Language Models",
+  "blurb": "How the field decides whether a model is any good — and the steady realisation that static benchmarks are not enough.",
+  "anchors": [
+    {"id": "2009.03300", "note": "57 subjects of multiple-choice knowledge. Flawed and saturating, but still the number everyone quotes."},
+    {"id": "2107.03374", "note": "Functional correctness on real programs instead of text overlap. Introduced pass@k."},
+    {"id": "2206.04615", "note": "204 tasks built to be hard for current models, and an honest look at where scaling does not help."},
+    {"id": "2211.09110", "note": "Argues evaluation should be multi-metric and transparent, not one accuracy number. As much a methodology paper as a benchmark."},
+    {"id": "2303.16634", "note": "Using a strong model as the judge, with the correlation study to justify it — and the biases that come with it."},
+    {"id": "2403.04132", "note": "Pairwise human preference at scale. The current answer to benchmark saturation."}
+  ]
+}

data/collections/finetuning-alignment.json

@@ -0,0 +1,15 @@
+{
+  "slug": "finetuning-alignment",
+  "title": "Fine-tuning & Alignment",
+  "blurb": "How raw language models become usable assistants, and how tuning them got cheap enough to do on one GPU.",
+  "anchors": [
+    {"id": "1707.06347", "note": "The policy-gradient method the whole RLHF stack was built on. Read it before InstructGPT."},
+    {"id": "2009.01325", "note": "Human feedback beats supervised fine-tuning on summarisation. The proof of concept for RLHF."},
+    {"id": "2203.02155", "note": "RLHF applied at scale to instruction following. This is the recipe behind ChatGPT."},
+    {"id": "2212.08073", "note": "Replaces most human feedback with a written constitution and model self-critique."},
+    {"id": "2305.18290", "note": "Shows the RLHF objective has a closed-form solution — no reward model, no RL loop. Removed most of the machinery."},
+    {"id": "2101.00190", "note": "Tune a small continuous prefix rather than the model. The start of parameter-efficient tuning."},
+    {"id": "2106.09685", "note": "Low-rank updates to frozen weights. The single most widely deployed fine-tuning method."},
+    {"id": "2305.14314", "note": "4-bit base model plus LoRA adapters, so a 65B model fine-tunes on one consumer GPU."}
+  ]
+}

data/collections/llm-inference.json

@@ -0,0 +1,16 @@
+{
+  "slug": "llm-inference",
+  "title": "The LLM Inference Stack",
+  "blurb": "How a forward pass became an engineering discipline. Read in order: memory first, then precision, then decoding.",
+  "anchors": [
+    {"id": "2205.14135", "note": "Removes the quadratic memory cost by never materialising the attention matrix. Everything after this assumes it."},
+    {"id": "2307.08691", "note": "The rewrite that actually saturates the GPU. Same idea, better work partitioning."},
+    {"id": "2309.06180", "note": "Applies OS-style paging to the KV cache. This is the paper vLLM is built on."},
+    {"id": "2208.07339", "note": "Showed 8-bit inference can be lossless if you handle outlier features separately. Started the quantisation era."},
+    {"id": "2210.17323", "note": "One-shot post-training quantisation to 3-4 bits. No retraining, which is why it spread so fast."},
+    {"id": "2211.10438", "note": "Migrates quantisation difficulty from activations to weights, making 8-bit work where it previously did not."},
+    {"id": "2306.00978", "note": "Protects the small fraction of salient weights instead of quantising uniformly. Strong accuracy at 4 bits."},
+    {"id": "2211.17192", "note": "Decode several tokens per step with a cheap draft model, verify with the big one. Latency without accuracy loss."},
+    {"id": "2401.10774", "note": "Speculative decoding without a separate draft model — extra heads on the model you already have."}
+  ]
+}

data/collections/rag-retrieval.json

@@ -0,0 +1,17 @@
+{
+  "slug": "rag-retrieval",
+  "title": "RAG & Retrieval",
+  "blurb": "Retrieval-augmented generation from first principles: how to find the right passage, then how to use it.",
+  "anchors": [
+    {"id": "2004.04906", "note": "Dense passage retrieval beat BM25 with a dual encoder and hard negatives. The paper that made dense retrieval credible."},
+    {"id": "2002.08909", "note": "Trains the retriever jointly with the language model rather than bolting it on afterwards."},
+    {"id": "2005.11401", "note": "The paper that named RAG. Parametric memory plus a non-parametric index, trained end to end."},
+    {"id": "2007.01282", "note": "Encode passages separately, fuse in the decoder. Still the strongest simple recipe for reading many passages."},
+    {"id": "2004.12832", "note": "Late interaction: keep per-token vectors and match them at query time. More expressive than a single vector."},
+    {"id": "2112.01488", "note": "Makes late interaction affordable through residual compression. The version people actually deploy."},
+    {"id": "2112.09118", "note": "Unsupervised contrastive retrieval that transfers without in-domain labels — the zero-shot baseline to beat."},
+    {"id": "2212.10496", "note": "Have the LLM hallucinate a fake answer, then embed that instead of the query. Absurd and effective."},
+    {"id": "2310.11511", "note": "Teaches the model when to retrieve and to critique what it retrieved, rather than always retrieving."},
+    {"id": "2401.18059", "note": "Recursive summarisation into a tree, so retrieval can operate at several levels of abstraction."}
+  ]
+}

data/collections/scaling-architecture.json

@@ -0,0 +1,17 @@
+{
+  "slug": "scaling-architecture",
+  "title": "Scaling & Architecture",
+  "blurb": "Why models got bigger, what the scaling laws actually said, and how anyone trains something that no longer fits on a GPU.",
+  "anchors": [
+    {"id": "1706.03762", "note": "The architecture everything else in this list is built on."},
+    {"id": "2001.08361", "note": "Loss falls as a power law in compute, data and parameters. The paper that made scaling a plan rather than a hope."},
+    {"id": "2005.14165", "note": "Few-shot behaviour emerging from scale alone. The empirical shock that set the agenda."},
+    {"id": "2203.15556", "note": "Corrects the earlier scaling laws: most large models were badly undertrained on data. Changed how everyone allocates compute."},
+    {"id": "1909.08053", "note": "Tensor parallelism — how to split a single layer across GPUs."},
+    {"id": "1910.02054", "note": "Shards optimiser state, gradients and parameters. The memory arithmetic behind most large-scale training."},
+    {"id": "2101.03961", "note": "Mixture of experts with a single expert per token, making trillion-parameter models trainable."},
+    {"id": "2401.04088", "note": "A sparse MoE that was genuinely good and openly released. Made MoE a practical choice rather than a curiosity."},
+    {"id": "2302.13971", "note": "Strong models trained only on public data, released to researchers. The open-weights era starts here."},
+    {"id": "2307.09288", "note": "The follow-up with the full RLHF pipeline documented, including the safety work."}
+  ]
+}