GhostScientist's picture
Model card: wiki generation task, training details, paired eval results
e96a46f verified
|
Raw History Blame Contribute Delete
2.88 kB
---
base_model: Qwen/Qwen2.5-Coder-14B-Instruct
library_name: transformers
model_name: semanticwiki-coder-14b-v3
tags:
- generated_from_trainer
- trl
- sft
- hf_jobs
- code-documentation
- technical-writing
- semanticwiki
licence: license
license: apache-2.0
---
# SemanticWiki Coder 14B v3
LoRA adapter fine-tuned from [Qwen/Qwen2.5-Coder-14B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-14B-Instruct)
to generate **DeepWiki-style architectural wiki pages** with **verifiable `path:line`
source citations**, Mermaid diagrams, and structured sections.
## What it does
Given a line-numbered code context and a query, it writes a wiki page whose factual
claims carry strict `path/file.ext:line` or `path:start-end` citations that resolve
against the real repository.
Input format (same as training):
```
<START_OF_CONTEXT>
# Repository: org/name (language: ...)
# File tree ...
### FILE: src/module.py (lines 1-N)
1| <source with line numbers>
<END_OF_CONTEXT>
<query>
Generate a wiki page describing ...
</query>
```
## Training
| Parameter | Value |
|---|---|
| Base model | Qwen/Qwen2.5-Coder-14B-Instruct |
| Method | LoRA SFT (r=64, alpha=128, dropout 0.05, all linear projections) |
| Data | [GhostScientist/semanticwiki-data-v3](https://huggingface.co/datasets/semanticwiki-data-v3) - 313 pages from 97 real GitHub repos, every citation deterministically verified |
| Loss | completion-only (prompt-completion format; the code context is not trained on) |
| Epochs / LR / effective batch | 2 / 1e-4 cosine / 16 |
| Max sequence length | 24576 |
| Final loss / token accuracy | 0.5101 / 0.823 |
## Paired eval (SemanticWiki-Eval v3, 13 held-out repos, 26 pages)
| Arm | Format | Citation validity | Fidelity (judge 1-5) | Citations/page |
|---|---:|---:|---:|---:|
| **this model** | **0.812** | **0.562** | 3.39 | 4.9 |
| Qwen2.5-Coder-14B-Instruct (base) | 0.755 | 0.308 | 3.15 | 2.0 |
Fine-tuning improved every axis; citation validity rose +83% relative. Full protocol
and per-page results: [GhostScientist/semanticwiki-eval-v3](https://huggingface.co/datasets/GhostScientist/semanticwiki-eval-v3).
## Limitations
Citation validity on unseen repos is 0.56 vs the teacher dataset's ~1.0 - hallucinated
line numbers remain the main failure mode. Fidelity is judge-based (judge shares the
Qwen3 family; see benchmark card caveats).
## Framework versions
TRL 1.14.1 - Transformers 5.18.0 - PyTorch 2.14.1 - PEFT 0.21.2 - Datasets 5.1.0
## Citation
```bibtex
@software{vonwerra2020trl,
title = {{TRL: Transformers Reinforcement Learning}},
author = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallou{\'e}dec, Quentin},
license = {Apache-2.0},
url = {https://github.com/huggingface/trl},
year = {2020}
}
```