| --- |
| base_model: Qwen/Qwen2.5-Coder-3B-Instruct |
| license: other |
| license_name: qwen-research |
| license_link: https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct/blob/main/LICENSE |
| library_name: transformers |
| pipeline_tag: text-generation |
| datasets: |
| - ESCAD/OpenRTLSet |
| tags: |
| - verilog |
| - systemverilog |
| - rtl |
| - code |
| --- |
| |
| # verilog-coder (3B, full fine-tuning) |
|
|
| Qwen/Qwen2.5-Coder-3B-Instruct fine-tuned for Verilog/SystemVerilog generation from a natural-language |
| spec (optionally with the module interface). Code, data pipeline, and evaluation harness: |
| [https://github.com/Henil-08/verilog-coder-ft](https://github.com/Henil-08/verilog-coder-ft). |
|
|
| ## Training data |
|
|
| [OpenRTLSet](https://huggingface.co/datasets/ESCAD/OpenRTLSet): 131k Verilog modules from |
| GitHub, each paired with an LLM-written description. After dropping 54.6k fuzzer-generated |
| modules, deduplicating, and capping examples at 2048 tokens, 53.7k remain; this run used |
| 40,000 of them (~93% Verilog, ~7% SystemVerilog). About 30% of |
| training prompts describe the ports in prose instead of giving the header as code, which |
| keeps the model usable when no header is provided. |
|
|
| The raw corpus contains UVM testbench code in only 3 rows, and none of them survive |
| filtering. This model was not trained or evaluated on UVM and makes no claim of UVM |
| capability. |
|
|
| ## Training setup |
|
|
| - Method: full fine-tuning, 1 epoch, 29,906,052 supervised tokens |
| - LR 2e-05 (cosine, 3% warmup), effective batch 64, max length 2048, bf16 |
| - HF Transformers + PEFT + DeepSpeed ZeRO-2 on 2× H100 (Modal); |
| 27 min of training, ~$3.63 |
| - Loss on the code (assistant) tokens only; final eval loss 0.2705 |
| - Selected among LoRA/QLoRA/full-fine-tune configs by pass@1 on a validation split, so the |
| test numbers below were not used for model selection. |
|
|
| ## Evaluation (pass@1, greedy decoding) |
|
|
| | Benchmark | Base | Fine-tuned | Items gained / lost | |
| |---|---|---|---| |
| | OpenRTLSet held-out (diff-sim) | 25.6% (101/394, 95% CI 21.6–30.2) | 44.7% (176/394, 95% CI 39.8–49.6) | +81 / -6 (McNemar p<0.001) | |
| | VerilogEval v2 spec-to-RTL (published prompts) | 18.6% (29/156, 95% CI 13.3–25.4) | 26.9% (42/156, 95% CI 20.6–34.4) | +21 / -8 (McNemar p=0.024) | |
| | VerilogEval v2 + interface code block | 23.1% (36/156, 95% CI 17.2–30.3) | 28.8% (45/156, 95% CI 22.3–36.4) | +20 / -11 (McNemar p=0.150) | |
|
|
| - **OpenRTLSet held-out**: 394 standalone modules not seen in training. A generated module |
| passes if it matches the reference's ports and its outputs equal the reference's on every |
| half-cycle of 500 cycles of random stimulus (Verilator). Random stimulus is evidence of |
| equivalence, not proof. |
| - **VerilogEval v2**: the official testbenches and Icarus Verilog v12 pass rule, with this |
| project's chat prompt (scores are not directly comparable to leaderboard numbers that use |
| other prompts). |
|
|
| ## Limitations |
|
|
| - Greedy decoding sometimes falls into repetition loops on long modules; about |
| 14% of eval outputs hit the 2048-token cap. |
| - Trained on single-module completions; multi-module designs and testbenches are out of |
| scope. Not evaluated for synthesis quality, timing, or area. |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| tok = AutoTokenizer.from_pretrained("henil08/verilog-coder-3b") |
| model = AutoModelForCausalLM.from_pretrained("henil08/verilog-coder-3b", dtype="auto", device_map="auto") |
| ``` |
|
|
| Prompt format (system + user), as in training: |
|
|
| ```text |
| Specification: |
| <description> |
| |
| Module interface: |
| ```verilog |
| module ...(...); |
| ``` |
| ``` |
|
|
