--- license: apache-2.0 base_model: convaiinnovations/laya base_model_relation: finetune pipeline_tag: text-classification language: [en] tags: [laya, code-search, reranker, code-retrieval, calibrated, claude-code, laya-codex] --- # laya-code A code-relevance re-ranker fine-tuned from [Laya](https://huggingface.co/convaiinnovations/laya) (ModernBERT-large encoder + typed-decision head, 421M parameters). Given a task description and a source-code chunk, it answers one yes/no (`noul`) question with a **calibrated probability**: ``` question: Is this source code relevant to the software change: "{task}"? state: file: (lines a-b)\n (truncated to 128 tokens in production) ``` It is the default re-ranker of [laya-codex](https://github.com/pilotspace/laya-codex), which feeds Claude Code the most relevant code spans for a prompt (tree-sitter chunks, then Moon BM25 candidates, then laya-code re-ranking). ## Model details | | | |---|---| | Base model | `convaiinnovations/laya`, root checkpoint (revision `1c5edc17a7acd8701df6fc341c0d179f1c62c982`), Apache-2.0 | | Architecture | unchanged: same safetensors keys, shapes and dtypes as the base (205 F16 tensors + `temperature` F32), 842,609,210 bytes | | What changed | `model.safetensors` (fine-tuned weights) and `rl_agent_config.json` (`model_name`, `noul:2` temperature **0.9410**, was 1.9834; `finetune` block). `encoder/config.json`, `tokenizer/*`, `rl_agent_api.py` and `rl_common.py` are byte-identical to the base. The `training` block of `rl_agent_config.json` is inherited from the base and describes the base's training, not this fine-tune. | | Context | 512 tokens (`max_len`), 192 for the question head (`head_max_len`) | | Runtime | Python: `rl_agent_api.RLAgent` from this repo (same as the base). Rust: `laya-model` crate of laya-codex (candle; Metal F16 on macOS, CPU F32 elsewhere), parity-tested against the Python reference | | License | Apache-2.0 (see `LICENSE` and `NOTICE`) | ## Training data Weak supervision from git history. Nothing was hand-labelled. - **8 training repositories** (mixed Rust, Python, TypeScript and JavaScript): openai/codex, TinDang97/velos, MervinPraison/PraisonAI, badlogic/pi-mono, Portkey-AI/gateway (local `ai-guard` checkout), TinDang97/python-dependency-injector, Netflix/dispatch and pilotspace/hydroa (local `ai-proxy` checkout). Source: `finetune/repos.py`. - **Held out** (never used for training or calibration): pilotspace/moon and pilot-space. Moon client codebases (helios, helios-mono, lunaris) were left out so that Moon vocabulary does not leak into the Moon eval. A root-commit check rejects forks and clones of each other and of the held-out repos. - **Examples**: up to 600 non-merge commits per repo (the `build_data.py` default) that touch 1–4 source files and have an informative subject (at least 20 characters). The task is the subject plus a short first body line. The candidate list is the BM25 top 12 over other files plus the parent-revision windows of the touched files (40-line windows, stride 30). Labels: 1.0 for a window that overlaps a changed hunk, 0.7 for another window of a touched file, 0 for other files. Extra examples: hunks BM25 missed (label 1.0), a random same-file window (label 0.4) and random windows from other files (label 0). - **Size**: 50,926 pairs, split by commit hash into 45,790 train and 5,136 validation pairs. Train pairs: 2,779 candidate positives, 5,609 candidate same-file, 24,768 candidate negatives, 4,656 extra positives, 2,392 same-file, 5,586 random negatives. Per-repo counts (v2 data; the warm start used v1 data from the same repos): | repo | train commits | val commits | train pairs | val pairs | |---|---|---|---|---| | PraisonAI | 231 | 18 | 3,711 | 280 | | ai-guard | 460 | 40 | 7,404 | 665 | | ai-proxy | 200 | 23 | 3,293 | 380 | | codex | 446 | 54 | 7,676 | 932 | | dispatch | 447 | 53 | 7,347 | 863 | | pi-mono | 439 | 61 | 7,386 | 1,013 | | python-dependency-injector | 453 | 47 | 7,066 | 744 | | velos | 117 | 16 | 1,907 | 259 | - **Training**: top 8 of 28 encoder layers, the final norm and the decision head. fp32 on an M4 Pro (MPS). AdamW; learning rate 2e-5 for the encoder and 1e-4 for the head. 32 sequences per update. Log loss against soft targets. 536 updates on v2 data, warm-started from 300 updates on v1 data (about 2.3 h in total). Checkpoint chosen by lowest validation NLL. The `noul:2` temperature was then refitted on the validation split, using the exported F16 weights. ## Evaluation All numbers come from files in the laya-codex repository and are quoted as recorded. Gold labels are file-level: the files the commit touched. Candidates are BM25 windows at HEAD. ### Re-ranker comparison, 128-token state (production setting) `spike/results/compare_models.json` (`spike/compare_models.py`): the 40 most recent qualifying moon commits, BM25 top 24, state truncated to 128 tokens, probabilities pooled over all candidates (base rate 0.309). | model | Laya-only MRR | Laya-only P@10 | RRF MRR | AUROC | ECE | mean P | |---|---|---|---|---|---|---| | BM25 alone | 0.480 (MRR) | 0.340 | – | – | – | – | | laya-base (`convaiinnovations/laya`) | 0.479 | 0.348 | 0.505 | 0.586 | 0.362 | 0.671 | | laya-typed-decisions | 0.441 | 0.288 | 0.456 | 0.539 | 0.239 | 0.545 | | **laya-code** | **0.702** | **0.405** | **0.630** | **0.713** | **0.049** | 0.328 | ### Held-out evaluation, 256-token state (training protocol) `spike/results/finetune_eval.json` (`finetune/eval.py`): 40 tasks per held-out repo, BM25 top 32, state truncated to 256 tokens. Paired bootstrap over the tasks. | repo | model | Laya-only MRR | RRF MRR | RRF P@10 | ECE (15 bins) | AUROC pooled | |---|---|---|---|---|---|---| | moon | laya-base | 0.497 | 0.586 | 0.343 | 0.461 | 0.584 | | moon | laya-code | 0.526 | 0.519 | 0.398 | 0.060 | 0.677 | | pilot-space | laya-base | 0.519 | 0.762 | 0.323 | 0.509 | 0.536 | | pilot-space | laya-code | 0.646 | 0.714 | 0.393 | 0.016 | 0.709 | Under this protocol, laya-code clearly improves calibration and discrimination. It puts more gold-file spans in the top 10: RRF P@10 rose by +0.055 (95% CI [0.013, 0.098]) on moon and by +0.070 ([0.033, 0.108]) on pilot-space. Its **RRF MRR did not beat laya-base's**: −0.066 ([−0.195, 0.061]) on moon and −0.049 ([−0.172, 0.073]) on pilot-space. For that reason, laya-codex fuses laya-code by score (`(1−w)·lexical + w·P`, w = 0.5) rather than by RRF. End to end, the laya-codex paired Claude Code benchmark (20 moon tasks, `docs/RESULTS.md`) measured −45% code-reading tokens and −25% wall-clock time for the whole pipeline, with no loss of answer recall. The same document reports that the model's **marginal** contribution over lexical-only ranking is within run-to-run noise at n = 20. Do not read the pipeline numbers as a property of this model. ## Intended use - Re-ranking lexical (BM25) candidates of source-code chunks for a natural-language software-change task, as a calibrated `P(relevant)`. - Gating or fusing retrieval results by probability; P is calibrated to the training distribution (ECE ≤ 0.06 on held-out repos). ## Out of scope and limitations - **Weak labels.** A commit touching a file does not make every window of it relevant, and the file-level gold is coarse. - **Small evaluation.** Each held-out set has 40 tasks, and most CIs are wide. The two protocols (128 vs 256 tokens, top 24 vs 32) give different absolute numbers, as shown above. - **Low probabilities.** P rarely exceeds 0.5 (max 0.49 on moon, 0.50 on pilot-space at 256 tokens). Use rank or score fusion, or a threshold near 0.4, not "P ≥ 0.5 means relevant". - **Under-trained.** Only about half an epoch of the v2 data was used, on a shared laptop. Validation AUROC was still rising when training stopped. - **English prompts only.** Training covered Rust, Python, TypeScript and JavaScript; other languages are untested. - **Other tasks untested.** It is not a general Laya replacement: `choice`/`score` questions (for example, task scope) were not trained, and zero-shot scope accuracy is poor (`spike/results/scope_eval.json`). - **Too slow for interactive CPU use.** At 421M parameters, CPU re-ranking of 24 candidates is too slow for interactive use. laya-codex runs it on Metal, or falls back to lexical ranking. - **Legal status of training data.** The model was trained on permissively licensed public code plus the author's own repositories. It is a classifier and cannot reproduce that code, but the legal status of weights trained on source code is not settled. ## License and attribution Apache-2.0, like the base model. laya-code is a Derivative Work of [convaiinnovations/laya](https://huggingface.co/convaiinnovations/laya) (Apache-2.0, © Convai Innovations), which builds on [answerdotai/ModernBERT-large](https://huggingface.co/answerdotai/ModernBERT-large) (Apache-2.0). The modified files are `model.safetensors` and `rl_agent_config.json`; every other file is unchanged from the base. See `NOTICE`. ## Files See `MANIFEST.sha256` for the sha256 of every uploaded file.