halle01/coder-fake

coder-training — fine-tune our own threejs coder checkpoint

Implements ../CODER_TRAINING_PLAN.md (what data, why, how it was verified) using ../CONTRASTIVE_TRAINING_GUIDE.md's established format (algorithm comparison + hyperparameter tables, see TRAINING_GUIDE.md), adapted from a custom Gaussian-decoder trainer to a real LLM fine-tune via TRL. This is the one lever the competitive-intelligence research in ../past-winner/ and ../miner-submission/ identified as untried and highest-leverage: every recent leader forks the same architecture and even, in 4 of 5 cases, the same underlying coder checkpoint lineage — see ../past-winner/README.md.

Status: SFT verified end-to-end on a real Thunder A6000 (loss drops, eval runs, LoRA adapter checkpoint saves) on the curated data; KTO uses the same machinery. Throughput and deployment bugfixes documented — see DEPLOYING_ON_THUNDER.md and SPEEDUP_FINDINGS.md. Full multi-day run not yet started. Phase 0/1 (scrape + build datasets) has actually been executed — 142,647 usable duel records archived, datasets/{sft,kto,preference}.jsonl built from them — but that data isn't in this git repo (see "Getting this onto an instance" below). Training code is written and reviewed for syntax/import correctness, but there's no GPU in the environment it was built in — see "Pre-flight checklist" below for what to verify before trusting a real training run, especially the multimodal model/TRL API surface, which moves fast and couldn't be checked against a live install.

What's here

coder-training/
  data/
    curate_manifest.py     THROUGHPUT — subsample the manifest to ~10%
                            by prompt-dedup + judge-stage selection before
                            build_datasets.py (see SPEEDUP_FINDINGS.md)
    scrape_duels.py        Phase 0 — archive the competition's real per-prompt
                            duel data before its CDN retention window closes
                            (see CODER_TRAINING_PLAN.md — time-sensitive)
    build_datasets.py      Phase 1 — shape the archive into sft/preference/kto JSONL
    dataset_utils.py        shared HF-Datasets loading (lazy image loading)
  common.py                  shared model/LoRA/QLoRA/wandb setup
  train_sft.py                Stage 1: SFT distillation on winners
  train_kto.py                Stage 2, RECOMMENDED — see TRAINING_GUIDE.md
  train_dpo.py                Stage 2, alternative
  train_simpo.py              Stage 2, alternative (reference-free)
  muon.py                     opt-in Muon–AdamW hybrid optimizer (experimental)
  train_sft_unsloth.py        FALLBACK SFT via Unsloth (see SPEEDUP_FINDINGS.md)
  train_kto_unsloth.py        FALLBACK KTO via Unsloth
  unsloth_common.py           Unsloth FastVisionModel loading
  configs/{sft,kto,dpo,simpo}.yaml   hyperparameters, each cited to a paper
  eval/
    self_judge_eval.py     Phase 3 — held-out eval against REAL archived
                            winners using the REAL validator judge
    judge_challenge.py       vendored copy of that judge (see its docstring)
  scripts/
    setup_env.sh              run once on a fresh instance
    setup_fast_kernels.sh     THROUGHPUT — fla 0.4.2 + fused MoE kernel + verify
    setup_unsloth.sh          FALLBACK training stack (alternative to fla path)
    run_pipeline.sh            scrape -> build -> SFT -> preference-tune
  requirements.txt
  NEXT_INSTANCE_TRAINING.md  START HERE — master handoff: the fla/triton-3.6
                            speedup fix, ≤7-hour recipe, verified vs pending
  DEPLOY_A100_AGENT_RUNBOOK.md  step-by-step agent runbook for an A100 80GB
                            (self-contained; VERIFY gates + failure table)
  DEPLOYING_ON_THUNDER.md    runbook + verified bugfixes for a fresh Thunder
                            GPU instance (SFT + KTO both verified end-to-end)
  SPEEDUP_FINDINGS.md        why the naive run was ~165 days and how the
                            same 48GB GPU gets it to days (research-cited)
  TRAINING_GUIDE.md          algorithm choice + hyperparameters, paper-cited

Getting this onto an instance

Code and data travel separately — code is small (~150KB) and lives in this git repo; data is large (archive/ 3.2GB + datasets/ 6.5GB) and is not committed (see .gitignore) — scp it directly instead:

# on the instance
git clone https://github.com/karvachiik-lgtm/threejstraining.git coder-training
cd coder-training
bash scripts/setup_env.sh   # venv + deps + pre-flight checks; tells you what's next

# from your local machine, in a separate step (this is the ~9.7GB part)
scp -r /Users/xavier/bittensor/404-gen-subnet/404-agent/coder-training/{archive,datasets} \
    you@instance:~/coder-training/

eval/self_judge_eval.py additionally imports RendererModule from ../miner-submission/pipeline_service (it reuses the real renderer instead of reimplementing headless-Chromium rendering) — scp miner-submission/ (2.8MB) over too if you'll run Phase 3 eval, as a sibling of coder-training/, or pass --pipeline-service-dir to point at wherever you put it. Nothing else needs it — scrape_duels.py, build_datasets.py, and all four train_*.py scripts are fully self-contained.

scp -r /Users/xavier/bittensor/404-gen-subnet/404-agent/miner-submission you@instance:~/

Running it

ssh you@instance
cd coder-training
source .venv/bin/activate
export WANDB_API_KEY=...   # wandb login also works

# datasets/ already scp'd over (see above) — skip straight to training:
python train_sft.py --config configs/sft.yaml
python train_kto.py --config configs/kto.yaml     # or train_dpo.py / train_simpo.py — see TRAINING_GUIDE.md

Building the dataset fresh instead of scp'ing it (not needed if you scp'd datasets/ — only relevant for picking up new rounds later):

python data/scrape_duels.py --rounds-dir <path-to-404-active-competition/rounds> --out-dir ./archive
python data/build_datasets.py --manifest ./archive/duels_index.jsonl --out-dir ./datasets

Watch progress in wandb (WANDB_PROJECT defaults to 404-coder-training, set in each config's wandb: block) — loss curves, and once the preference stage runs, TRL's own logged reward-accuracy/margin metrics. See TRAINING_GUIDE.md's "What to watch in wandb" for what those numbers should be doing.

Then, once you have a checkpoint served behind an OpenAI-compatible endpoint (e.g. point vLLM at checkpoints/kto/):

python eval/self_judge_eval.py \
    --checkpoint-base-url http://localhost:8000/v1 --checkpoint-model <served-name> \
    --judge-base-url https://openrouter.ai/api/v1 --judge-api-key-env OPENROUTER_API_KEY \
    --wandb

This is Phase 3 of CODER_TRAINING_PLAN.md — the actual bar to clear before this is worth putting into ../miner-submission/configuration.yaml and testing further per ../miner-submission/TESTING_WITHOUT_GPU.md: does the new checkpoint beat the REAL archived competition winners on held-out prompts, according to the REAL validator judge — not a proxy metric.

Pre-flight checklist — verify before trusting a real run

Nothing below is a known bug — these are the specific places this repo makes an assumption about an environment (a live GPU + specific package versions) that doesn't exist where it was written, so they're unverified, not wrong:

1. transformers support for the qwen3_5 architecture. Tooony133/Qwen-3.6-27B-* and Qwen/Qwen3.6-27B report architectures: ["Qwen3_5ForConditionalGeneration"] / model_type: "qwen3_5" (checked via the HF API). scripts/setup_env.sh checks for this and tells you if you need transformers from source.
2. TRL's exact VLM dataset/collator conventions for your installed version. data/dataset_utils.py + the chat-message shape in data/build_datasets.py follow TRL's documented pattern as of when this was written, but TRL's vision-language support is newer and moves faster than its text-only DPO/SFT — check python -c "from trl import DPOConfig; help(DPOConfig)" (and KTOConfig/CPOConfig/SFTConfig) against whatever version pip install actually resolves.
3. The PEFT-adapter-disable trick for the reference model (ref_model= None + peft_config in train_dpo.py/train_kto.py) — documented TRL behavior, not invented here, but confirm it's actually skipping a second full model load (watch VRAM at startup) rather than silently loading a full reference copy anyway.
4. eval/self_judge_eval.py's RendererModule/PipelineTask integration — written by reading ../miner-submission/pipeline_service/modules/ renderer/module.py and pipeline/task.py's source, not by running it. If render_js() raises or task.rendered_png stays None, that's the first place to look — see the docstring at the top of that file.
5. VRAM sizing. No instance to test against here. Tooony133/Qwen-3.6- 27B-AronHorn reports ~56GB BF16 weights (HF API usedStorage) — QLoRA's 4-bit quantization (enabled by default in every config here) should bring the base model to ~14-16GB, leaving room for LoRA adapters + activations
   • optimizer states on a single 40-80GB GPU, per QLoRA's own headline claim of fitting a 65B model on a single 48GB GPU — but size your actual instance against real nvidia-smi numbers on a short run before committing to a long one.