File size: 2,846 Bytes
3efc892 d7cf737 9cf71e3 3efc892 d7cf737 3efc892 d7cf737 9cf71e3 d7cf737 3efc892 11447fa 3efc892 d7cf737 3efc892 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 | ---
license: apache-2.0
base_model: Qwen/Qwen3.5-9B-Base
language:
- en
tags:
- hardware
- rtl
- chip-design
- domain-adaptive-pretraining
- pulp-platform
- knowledge-injection
---
# Qwen3.5-9B-PULP-DAPT
A 9B model domain-adaptively pretrained on the [PULP platform](https://pulp-platform.org/)
(Carfield/Cheshire RISC-V SoC) corpus, as a public proxy for the "inject a proprietary chip's
knowledge into an LLM" problem (the ChipNeMo setting).
**Closed-book factual accuracy about the platform: 92.8% vs Claude Opus 5's 72.0%**
(125-question layered audit benchmark: register maps, memory maps, dependency pins,
documentation semantics, driver↔register correspondences, issue history) — and **81.6% vs
72.2%** on the full 1,776-question bank (base model: 41.2%), only 125 of which were ever
seen during development.
| Model | Audit subset (125q) | Full bank (1,776q) |
|---|---|---|
| Qwen3.5-9B-Base | 43.2% | 41.2% |
| Claude Opus 5 (closed-book) | 72.0% | 72.2% |
| **This model** | **92.8%** | **81.6%** (register offsets 97%) |
The interesting part is the recipe: **raw-corpus DAPT alone produced zero gain on
memorization questions** (loss 0.80→0.35, offsets 1/42 → 1/42). All gains come from a
knowledge-rewriting augmentation stage: every fact restated through 24 LLM-generated
templates (⅓ reversed forms), full-coverage over all facts (not just evaluated ones), plus
whole-table narrative documents to fight similar-fact interference. Full recipe, data
pipeline, benchmark, and findings: **https://github.com/ARA-Labs/PULP-LLM**
## Training
- Continued pretraining (`stage=pt`), full-parameter, bf16, LLaMA-Factory + DeepSpeed ZeRO-3
- 31.2M tokens pulp-platform org corpus + 3.49M tokens augmentation + 8% wikitext replay
- LR 5e-6 cosine, global batch 16, packing @4096, 2 epochs — 77 min on 4×H100 (~$20)
## Usage
This is a **base-style** model (no chat template) tuned for knowledge extraction via
few-shot completion:
```python
from vllm import LLM, SamplingParams
llm = LLM(model="AgentNativeResearchLab/Qwen3.5-9B-PULP-DAPT", dtype="bfloat16")
prompt = (
"The following are factual questions about the PULP Carfield/Cheshire hardware platform "
"(ETH Zurich). Answer each with only the value, on one line.\n\n"
"Q: In the PULP Carfield SoC memory map (car_memory_map.h), what is the value of CAR_BOOTROM_BASE_ADDR?\nA: 0x2000000\n\n"
"Q: In the axi_llc register block, what is the byte offset of the CFG_FLUSH_LOW register?\nA:"
)
print(llm.generate([prompt], SamplingParams(temperature=0, max_tokens=16))[0].outputs[0].text)
```
## Limitations
- Knowledge snapshot of the pulp-platform repos as of 2026-08; no forgetting audit (MMLU) yet.
- Weak at numeric range-membership reasoning (memory-map region ownership).
- Not instruction-tuned; use few-shot completion or apply your own SFT.
|