The old README was a 581-line internal onboarding document; it now lives at
docs/DEEP_DIVE.md. The card that replaces it opens with what the model does and
how to use it, adds HF frontmatter (license, base_model, library_name, tags),
and documents the parts a user cannot guess:
- the head expects the specific 6-view 12288-d embedding and standardises its
input internally, so callers must pass RAW embeddings
- the emitted (A,B) follow PEFT's LoRA convention exactly
- which checkpoint to use and why the removed ones were removed
Results are reported as deltas against the frozen base on identical data,
because absolute losses across differing eval sets are not comparable -- the
confusion that hid a checkpoint scoring worse than no adapter at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Drop duplicate all_lora/head.best.pt (identical to all_lora_best_cpt.pth)
sixview_v1/v2 predate the input-standardisation fix; sixview_v2 measurably
degrades the base model. metrics.jsonl retained for provenance.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Removes ~11.6GB of weights that should not be published:
- sixview_v1/v2 checkpoints. sixview_v2 is the pre-standardisation head that
scored WORSE than applying no adapter (and worse than random noise); leaving
it downloadable invites someone to use it. Their metrics.jsonl stay for
provenance.
- duplicate all_lora/head.best.pt (same bytes as all_lora_best_cpt.pth) and a
stray h200_best/ copy.
- 49 files of snapshot_download bookkeeping under data/**/.cache/.
Keeps the two checkpoints worth having: all_lora_best_cpt.pth (merged corpus,
best) and h200_run/head.best.pt (prose-only, fallback).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
all_lora final checkpoint: cr_val 2.6811 on merged corpus
One resident model answers both sides: serve_fallback loads base+adapter and
routes on the request's model id, toggling via PeftModel.disable_adapter().
Serving a merged copy instead would need a second ~10GB model just to have
something to compare against.
- /compare: repo URL -> build -> two chat columns from the same server, so the
only variable is whether the adapter is on.
- pipeline now defaults to the un-merged path (also skips a ~10GB artifact).
- config picks the newest checkpoint that actually beats the baseline.
sixview_v2 was the default and scored WORSE than no adapter, so every demo
built so far used an adapter that made the model worse.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Merge branch 'main' of hf.co:moncefem/memory-lora-gemma4
El-Mouden Moncif
7f1b9d6
Add RepoPeftBench corpus merge
El-Mouden Moncif+1: Claude Opus 5
4f41179
Joins Code2LoRA's benchmark (~500K assertion-completion items over 512 repos)
to aligned6 (~27K prose QA over 2066 repos). The two are complementary: the
benchmark showed the head learns a repo's stack and conventions but not what the
project does, and exact-recall data is what addresses that.
Carries RepoPeftBench's own by-repo partition through unchanged, and lets an
eval assignment always win over a train one, so no repo can leak between train
and cross-repo eval. Caps QA per repo (evo alone has ~1000/repo) so the exact
recall task cannot swamp the prose QA.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add checkpoint autopush for non-stoppable instances
El-Mouden Moncif+1: Claude Opus 5
09dc9c5
digitalocean_H200_sxm5 has no stop -- only 'brev delete', which destroys the
disk. Pushing only at the end means an instance failure late in the run loses
everything, so push the best checkpoint to the HF remote as training proceeds.
Skips the push when the checkpoint has not changed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add H200 deploy kit with pre-flight validation
El-Mouden Moncif+1: Claude Opus 5
ddefab2
Targets a single H200 SXM5 (141GB) for a time- and cost-boxed run.
- deploy/h200/preflight.py: fails fast on anything that would waste GPU-hours
-- git-LFS pointers masquerading as data, embeddings that are already
standardized (double-applying the transform), a head without fit_input_stats
(the old collapsing code), missing eval splits -- then auto-tunes the largest
micro-batch that fits and measures steps achievable in the budget.
--data-only validates off-GPU so bad data is caught before provisioning.
- deploy/h200/setup.sh: targeted LFS pull. An unfiltered 'git lfs pull' fetches
~10GB of raw corpora; training reads ~0.14GB.
- deploy/h200/train_h200.sh: raises max-qna-per-doc alongside lm-micro-batch --
a step is one document, so batch alone stops helping past the QA-per-doc cap.
Gradient checkpointing off (not memory-bound at 141GB).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The trained head emitted a near-identical adapter for every repo (pairwise
cosine 0.96) and scored worse than random noise on held-out repo text. Cause:
~64% of each 6-view embedding is a constant vector shared by all repos, which
dominated the trunk (input cosine 0.73 -> 0.978 after the trunk).
- MemoryLoRAHead: standardize the conditioning input via input_mean/input_std
buffers fitted on train docs only and stored in the checkpoint, so training
and inference apply the same transform. After 40 steps the emitted-adapter
cosine dropped 0.96 -> 0.21.
- Training: log the no-adapter baseline and delta_vs_baseline every eval, plus
diag/adapter_cosine. Tracking only the adapted loss is why the collapse went
unnoticed.
- app/: Next.js app serving repo-personalized models over OpenAI- and
Anthropic-compatible APIs for coding CLIs.
- deploy/h100/: provisioning + training scripts for a single-GPU cloud run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Push ALL raw datasets: commitpack (9 langs), RepoPeftBench/code2lora, combined corpora
moncefem+1: Claude Opus 4.8
cd50fed
Adds the full raw training corpora (~9.5GB) previously kept local-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Update aligned6 training dataset to v4 (2066 repos / 27501 QA incl. batch2 novel + enhanced QA)
moncefem+1: Claude Opus 4.8
02f08b1
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Push current state: v3 checkpoints + metrics + batch2 scripts
moncefem+1: Claude Opus 4.8
f1c3402
v3 trained on complete 1647-repo dataset (best cr_val 2.5995); batch2 novel
dataset scripts (enhanced QA + orchestrator).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
LFS-track data/*.jsonl (multiview_sources grew past 10MB with 2083 repos).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Memory-LoRA hypernetwork for Gemma-4-E2B: code, curated data, sixview checkpoints
moncefem+1: Claude Opus 4.8
481fbb6
Hypernetwork that generates repo-specific LoRA adapters for a frozen
google/gemma-4-E2B (zero inference-time token overhead). Includes the
memory_lora package, all data/build/train/eval scripts, the aligned
6-view dataset, and the sixview_v1/v2 checkpoints.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CeUvTz4vLw4ni2ZNR6pS1p
