Spaces:
Running
Running
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>Distill yourself onto a small model — Digital Twin Evals</title> | |
| <style> | |
| :root { color-scheme: dark; } | |
| * { box-sizing: border-box; } | |
| body { margin: 0; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; | |
| background: #101418; color: #e8e6e1; line-height: 1.6; } | |
| .wrap { max-width: 720px; margin: 0 auto; padding: 24px 18px 64px; } | |
| h1 { font-size: 30px; line-height: 1.2; margin: 8px 0 4px; } | |
| h2 { font-size: 21px; margin: 36px 0 8px; border-bottom: 1px solid #2a3138; padding-bottom: 6px; } | |
| h3 { font-size: 17px; margin: 22px 0 6px; } | |
| p, li { font-size: 15px; color: #c9c7c2; } | |
| p.lede { font-size: 17px; color: #e8e6e1; } | |
| a { color: #7cc4ff; } | |
| .btn { display: inline-block; background: #4f8ff7; color: #fff ; text-decoration: none; | |
| font-weight: 700; padding: 14px 28px; border-radius: 12px; margin: 14px 8px 14px 0; font-size: 16px; } | |
| .btn.ghost { background: #1e242b; border: 1px solid #39424c; } | |
| .card { background: #161b21; border: 1px solid #2a3138; border-radius: 12px; padding: 16px; margin: 14px 0; } | |
| .step { display: flex; gap: 12px; margin: 14px 0; } | |
| .step .n { flex: 0 0 32px; height: 32px; border-radius: 50%; background: #4f8ff7; color: #fff; | |
| display: flex; align-items: center; justify-content: center; font-weight: 700; } | |
| code { background: #1e242b; padding: 2px 6px; border-radius: 6px; font-size: 13.5px; color: #ffd9a0; } | |
| pre { background: #0b0e11; border: 1px solid #2a3138; border-radius: 10px; padding: 14px; | |
| overflow-x: auto; font-size: 13px; line-height: 1.5; } | |
| pre code { background: none; padding: 0; color: #d6e2f0; } | |
| table { width: 100%; border-collapse: collapse; font-size: 14px; margin: 12px 0; } | |
| th, td { text-align: left; padding: 9px 10px; border-bottom: 1px solid #2a3138; } | |
| th { color: #e8e6e1; } | |
| td { color: #c9c7c2; } | |
| .note { font-size: 13.5px; color: #9aa0a8; } | |
| ul { padding-left: 20px; } | |
| </style> | |
| </head> | |
| <body> | |
| <div class="wrap"> | |
| <h1>Distill yourself onto a small model</h1> | |
| <p class="lede">RLHF distills the crowd into big models. This distills <i>you</i> — your taste, your | |
| trade-offs, your moral instincts — into a small model that runs on hardware you own.</p> | |
| <a class="btn" href="game/">Play Twin Builder</a> | |
| <a class="btn ghost" href="https://github.com/ctrimm/digital-twin-evals">Eval suite on GitHub</a> | |
| <h2>The loop</h2> | |
| <div class="step"><div class="n">1</div><div><b>Play.</b> 60 rapid-fire this-or-that choices in the | |
| <a href="game/">Twin Builder</a> game (~5 minutes). The twin predicts each answer live.</div></div> | |
| <div class="step"><div class="n">2</div><div><b>Export.</b> Download your answers as training data — | |
| SFT and DPO JSONL in the eval's exact format.</div></div> | |
| <div class="step"><div class="n">3</div><div><b>Fine-tune.</b> Train a LoRA adapter on a small base model, | |
| on your machine or a rented GPU. Minutes, not hours.</div></div> | |
| <div class="step"><div class="n">4</div><div><b>Evaluate.</b> Run the eval suite against base vs tuned. | |
| If tuning internalized <i>your</i> taste, the tuned model should match or beat the base model's | |
| best in-context score on held-out items.</div></div> | |
| <h2>Step 1 — Play the game</h2> | |
| <p>Open <a href="game/">Twin Builder</a>, pick a starting prior (your MBTI type, or blank slate — | |
| it's just a starting point), and tap through 60 choices. Answer fast and honestly; there are no right | |
| answers. A few questions repeat later on purpose — that measures how consistent you are with yourself, | |
| which is the ceiling your twin is scored against. Want more training data? Hit "keep going" for endless mode.</p> | |
| <h2>Step 2 — Export your data</h2> | |
| <p>The end screen offers three downloads. Know which is which:</p> | |
| <ul> | |
| <li><b>twin-profile.json</b> — the <i>measurement</i>: your fitted 10-trait vector plus stats. | |
| Good for inspection, not for training.</li> | |
| <li><b>twin-sft.jsonl</b> — the <i>training data</i>: one line per answer, | |
| <code>{"messages": [{"role":"user","content": prompt}, {"role":"assistant","content":"A"}]}</code>. | |
| Train on this.</li> | |
| <li><b>twin-dpo.jsonl</b> — same answers as preference pairs, | |
| <code>{"prompt","chosen","rejected"}</code>, for a DPO second stage.</li> | |
| </ul> | |
| <p class="note">Prompts are byte-identical to the eval suite's forced-choice format, so fine-tuned | |
| agreement is directly comparable to in-context scores. Consistency retests are excluded from training | |
| files. Your answers never leave your device until <i>you</i> download them.</p> | |
| <h2>Step 3 — Fine-tune</h2> | |
| <p>You're training a LoRA adapter (a small set of extra weights on a frozen base model), not the full | |
| model. With 60–500 short examples this is a minutes-scale job. Pick your platform:</p> | |
| <h3>A. Your Mac (Apple Silicon) — recommended</h3> | |
| <p>Private, free, no uploads. <a href="https://github.com/ml-explore/mlx-lm">MLX</a> is the native path.</p> | |
| <pre><code>pip install mlx-lm | |
| # split into train/valid (hold out ~10 for validation) | |
| mkdir -p twin-data | |
| head -n 50 twin-sft.jsonl > twin-data/train.jsonl | |
| tail -n 10 twin-sft.jsonl > twin-data/valid.jsonl | |
| python -m mlx_lm.lora --model mlx-community/Qwen3-4B-Instruct-2507-4bit \ | |
| --train --data twin-data --adapter-path ./twin-adapter \ | |
| --iters 300 --batch-size 4 | |
| # run it: | |
| python -m mlx_lm.generate --model mlx-community/Qwen3-4B-Instruct-2507-4bit \ | |
| --adapter-path ./twin-adapter --prompt "Scenario: ..." --max-tokens 5</code></pre> | |
| <p class="note">Check <code>mlx_lm.lora --help</code> for current flags. Any small instruct model with an | |
| MLX 4-bit build works (Qwen3, Llama 3.2, Gemma 3 in the 1–8B range).</p> | |
| <h3>B. Google Colab (free) — Unsloth + QLoRA</h3> | |
| <p>No Mac or no Apple Silicon? A free Colab T4 handles ≤8B models in 4-bit via | |
| <a href="https://github.com/unslothai/unsloth">Unsloth</a>'s turnkey notebooks:</p> | |
| <ul> | |
| <li>New notebook → Runtime → Change runtime type → T4 GPU.</li> | |
| <li><code>pip install</code> per Unsloth's Colab instructions, load a 4-bit base | |
| (e.g. an Unsloth Qwen3/Llama 3 4-bit build), attach LoRA adapters.</li> | |
| <li>Upload <code>twin-sft.jsonl</code>, format with the model's chat template, train with TRL's | |
| <code>SFTTrainer</code>, download the adapter.</li> | |
| </ul> | |
| <p class="note">Start from Unsloth's official Colab notebooks — they track library changes better than any | |
| static guide can.</p> | |
| <h3>C. Rented GPU — RunPod, Modal, Lambda Labs</h3> | |
| <p>For bigger models or faster turnaround, rent an A100/H100 by the hour and run the same | |
| Unsloth/TRL script you would on Colab. A short LoRA run like this typically costs a few dollars | |
| at current rates — check pricing before you start.</p> | |
| <h3>D. Fine-tuning APIs</h3> | |
| <p>Services like Together AI and Fireworks offer hosted fine-tuning: upload the JSONL, get back a | |
| model endpoint. Least setup, least control, and your data leaves your machine — the trade-off to weigh.</p> | |
| <div class="card"> | |
| <b>A note on Replicate.</b> Replicate's turnkey fine-tuning is built around image models | |
| (LoRA trainers for Stable Diffusion / Flux). For LLM LoRA it isn't the push-button path — | |
| use B, C, or D above instead. Replicate <i>is</i> useful afterward if you want to host your | |
| tuned model behind an API. | |
| </div> | |
| <h3>Rough time estimates</h3> | |
| <p class="note">For this workload specifically — 60–500 short examples, LoRA rank 8–16. Dataset size | |
| and step count dominate; these are ballparks, not benchmarks.</p> | |
| <table> | |
| <tr><th>Setup</th><th>Rough time</th></tr> | |
| <tr><td>MacBook M1–M3 (MLX), 3–8B model</td><td>~5–30 min</td></tr> | |
| <tr><td>Mac Studio (MLX), 8B+</td><td>~5–20 min</td></tr> | |
| <tr><td>Colab free T4 (Unsloth QLoRA)</td><td>~15–45 min</td></tr> | |
| <tr><td>Colab Pro A100</td><td>~5–15 min</td></tr> | |
| <tr><td>Rented A100 / H100</td><td>~5–15 min + setup</td></tr> | |
| </table> | |
| <h2>Step 4 — Evaluate</h2> | |
| <p>Clone the eval suite and compare base vs tuned on held-out items:</p> | |
| <pre><code>export OPENAI_BASE_URL=http://localhost:1234/v1 | |
| export OPENAI_API_KEY=lm-studio | |
| # in-context baseline (no tuning): how well does the base model predict you from K examples? | |
| inspect eval src/inspect_evals/tasks/daily_decisions.py@daily_decisions_predict \ | |
| --model openai/<base-model-id> -T profile=twin -T k=30 | |
| # then the same task against your fine-tuned model -- agreement should meet or beat it</code></pre> | |
| <p class="note">The eval items are held out from the game by design — the game draws from a separate | |
| 504-item capture pool, so a tuned model can't win by memorization. Score twins against your | |
| self-agreement rate (the retest ceiling), not against 100%.</p> | |
| <h2>Step 5 — Keep it learning (feedback mode)</h2> | |
| <p>Your twin doesn't freeze after the first fine-tune. On the game's end screen, | |
| <b>Test your twin</b> starts feedback mode: the twin shows its prediction <i>first</i>, | |
| and you agree or disagree with each one.</p> | |
| <ul> | |
| <li><b>Agree</b> — confirms the twin; it keeps learning from the answer.</li> | |
| <li><b>Disagree</b> — you pick the right answer. That correction becomes a DPO pair | |
| for free: the twin's wrong pick is the <i>rejected</i> response, yours the | |
| <i>chosen</i> one.</li> | |
| </ul> | |
| <p>The end screen reports your twin's feedback accuracy, and the DPO download now includes | |
| your corrections alongside the base pairs. Re-tune on the accumulated file every so often — | |
| that's the continual loop: play, tune, judge, correct, re-tune.</p> | |
| <h2>Notes</h2> | |
| <ul> | |
| <li><b>Privacy:</b> the local path (Mac + MLX, or LM Studio) never sends your answers anywhere. | |
| Your twin can live entirely on your hardware.</li> | |
| <li><b>Data size:</b> 60 answers is a start. Endless mode in the game grows the set; a few hundred | |
| diverse answers beats dozens.</li> | |
| <li><b>DPO:</b> optional second stage after SFT, using <code>twin-dpo.jsonl</code>.</li> | |
| <li><b>Eval vs train:</b> never train on the 240-item eval set — it's the held-out test.</li> | |
| </ul> | |
| <p class="note">Part of <a href="https://github.com/ctrimm/digital-twin-evals">digital-twin-evals</a> — | |
| eval suites for moral dilemmas, preferences, and individual digital twins.</p> | |
| </div> | |
| </body> | |
| </html> | |