| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3.5-9B-Base |
| language: |
| - en |
| tags: |
| - hardware |
| - rtl |
| - chip-design |
| - domain-adaptive-pretraining |
| - pulp-platform |
| - knowledge-injection |
| --- |
| |
| # Qwen3.5-9B-PULP-DAPT |
|
|
| A 9B model domain-adaptively pretrained on the [PULP platform](https://pulp-platform.org/) |
| (Carfield/Cheshire RISC-V SoC) corpus, as a public proxy for the "inject a proprietary chip's |
| knowledge into an LLM" problem (the ChipNeMo setting). |
|
|
| **Closed-book factual accuracy about the platform: 92.8% vs Claude Opus 5's 72.0%** |
| (125-question layered audit benchmark: register maps, memory maps, dependency pins, |
| documentation semantics, driver↔register correspondences, issue history) — and **81.6% vs |
| 72.2%** on the full 1,776-question bank (base model: 41.2%), only 125 of which were ever |
| seen during development. |
|
|
| | Model | Audit subset (125q) | Full bank (1,776q) | |
| |---|---|---| |
| | Qwen3.5-9B-Base | 43.2% | 41.2% | |
| | Claude Opus 5 (closed-book) | 72.0% | 72.2% | |
| | **This model** | **92.8%** | **81.6%** (register offsets 97%) | |
|
|
| The interesting part is the recipe: **raw-corpus DAPT alone produced zero gain on |
| memorization questions** (loss 0.80→0.35, offsets 1/42 → 1/42). All gains come from a |
| knowledge-rewriting augmentation stage: every fact restated through 24 LLM-generated |
| templates (⅓ reversed forms), full-coverage over all facts (not just evaluated ones), plus |
| whole-table narrative documents to fight similar-fact interference. Full recipe, data |
| pipeline, benchmark, and findings: **https://github.com/ARA-Labs/PULP-LLM** |
|
|
| ## Training |
|
|
| - Continued pretraining (`stage=pt`), full-parameter, bf16, LLaMA-Factory + DeepSpeed ZeRO-3 |
| - 31.2M tokens pulp-platform org corpus + 3.49M tokens augmentation + 8% wikitext replay |
| - LR 5e-6 cosine, global batch 16, packing @4096, 2 epochs — 77 min on 4×H100 (~$20) |
|
|
| ## Usage |
|
|
| This is a **base-style** model (no chat template) tuned for knowledge extraction via |
| few-shot completion: |
|
|
| ```python |
| from vllm import LLM, SamplingParams |
| llm = LLM(model="AgentNativeResearchLab/Qwen3.5-9B-PULP-DAPT", dtype="bfloat16") |
| prompt = ( |
| "The following are factual questions about the PULP Carfield/Cheshire hardware platform " |
| "(ETH Zurich). Answer each with only the value, on one line.\n\n" |
| "Q: In the PULP Carfield SoC memory map (car_memory_map.h), what is the value of CAR_BOOTROM_BASE_ADDR?\nA: 0x2000000\n\n" |
| "Q: In the axi_llc register block, what is the byte offset of the CFG_FLUSH_LOW register?\nA:" |
| ) |
| print(llm.generate([prompt], SamplingParams(temperature=0, max_tokens=16))[0].outputs[0].text) |
| ``` |
|
|
| ## Limitations |
|
|
| - Knowledge snapshot of the pulp-platform repos as of 2026-08; no forgetting audit (MMLU) yet. |
| - Weak at numeric range-membership reasoning (memory-map region ownership). |
| - Not instruction-tuned; use few-shot completion or apply your own SFT. |
|
|
