Text Generation
Transformers
Safetensors
qwen3_5
image-text-to-text
verilog
systemverilog
design-verification
testbench-generation
trl
sft
rlvr
grpo
conversational
Instructions to use AI-Institute-BSC/CheckmateRTL-27B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use AI-Institute-BSC/CheckmateRTL-27B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="AI-Institute-BSC/CheckmateRTL-27B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("AI-Institute-BSC/CheckmateRTL-27B") model = AutoModelForMultimodalLM.from_pretrained("AI-Institute-BSC/CheckmateRTL-27B", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use AI-Institute-BSC/CheckmateRTL-27B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "AI-Institute-BSC/CheckmateRTL-27B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AI-Institute-BSC/CheckmateRTL-27B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/AI-Institute-BSC/CheckmateRTL-27B
- SGLang
How to use AI-Institute-BSC/CheckmateRTL-27B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "AI-Institute-BSC/CheckmateRTL-27B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AI-Institute-BSC/CheckmateRTL-27B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "AI-Institute-BSC/CheckmateRTL-27B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AI-Institute-BSC/CheckmateRTL-27B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use AI-Institute-BSC/CheckmateRTL-27B with Docker Model Runner:
docker model run hf.co/AI-Institute-BSC/CheckmateRTL-27B
Commit ·
013ed78
1
Parent(s): c2d12f6
Hide readme
Browse files
INTRO.md
ADDED
|
@@ -0,0 +1,102 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Qwen3.6-27B-RL-Verif
|
| 2 |
+
|
| 3 |
+
A finetune of Qwen3.6-27B specialized for SystemVerilog test-stimulus and test-checking
|
| 4 |
+
|
| 5 |
+
Training was done in two stages:
|
| 6 |
+
|
| 7 |
+
- Supervised Finetuning (SFT) on 70k examples distilled from Gemma-4-31B
|
| 8 |
+
- Reinforcement Learning (RL) on 12k SystemVerilog environments
|
| 9 |
+
|
| 10 |
+
A smaller 9B variant is trained similarly.
|
| 11 |
+
|
| 12 |
+
# CVDP Results
|
| 13 |
+
|
| 14 |
+
<img src="https://cdn-uploads.huggingface.co/production/uploads/64da082a79c87e13ca232042/MZwGhx1CoxyJ7F1rUHZCL.png" width=800px />
|
| 15 |
+
|
| 16 |
+
Full results on the [NVIDIA CVDP benchmark](https://github.com/NVlabs/cvdp_benchmark).
|
| 17 |
+
|
| 18 |
+
| | GLM-5.2 | Kimi-K2.5 | R1-0528 | Gemma-4-31B | Qwen3.6-27B | **Our 27B model** |
|
| 19 |
+
|:----|--:|--:|--:|--:|--:|--:|
|
| 20 |
+
| **CVDP Design Verification** | | | | | | |
|
| 21 |
+
| `cid012*` Testbench Stimulus Gen. | 34.03% | 22.09% | 22.99% | 24.78% | 19.40% | **39.40%** |
|
| 22 |
+
| `cid013*` Testbench Checker Gen. | 13.96% | 23.40% | 18.87% | 17.74% | 15.09% | **31.32%** |
|
| 23 |
+
| `cid014*` Assertion Generation | 33.13% | 36.12% | 22.69% | 33.13% | 22.39% | **39.70%** |
|
| 24 |
+
| `cid016` Debugging / Bug Fixing | 44.57% | **64.57%** | 54.29% | 63.43% | 58.86% | 56.57% |
|
| 25 |
+
| **CVDP Code Generation** | | | | | | |
|
| 26 |
+
| `cid002` Code Completion | **48.51%** | 44.04% | 34.47% | 39.57% | 32.55% | 29.57% |
|
| 27 |
+
| `cid003` Natural Language Spec to Code | 52.82% | **56.15%** | 41.28% | 48.21% | 37.18% | 50.00% |
|
| 28 |
+
| `cid004` Code Modification | 50.91% | 50.55% | 37.45% | **54.55%** | 35.27% | 41.45% |
|
| 29 |
+
| `cid007` Code Improvement (Linting/QoR) | 51.50% | 59.50% | 39.50% | **64.50%** | 50.00% | 40.50% |
|
| 30 |
+
| **CVDP Code Comprehension** | | | | | | |
|
| 31 |
+
| `cid006` RTL ↔ Specification | 69.17% | 70.53% | **72.77%** | 67.66% | 68.25% | 72.69% |
|
| 32 |
+
| `cid008` Testbench ↔ Test Plan | 61.83% | **66.69%** | 45.65% | 63.10% | 62.06% | 58.08% |
|
| 33 |
+
|
| 34 |
+
Note that those categories with asterisks in the title were converted from Commercial tools (XCelium) to OSS tools (Verilator)
|
| 35 |
+
|
| 36 |
+
# Deployment
|
| 37 |
+
|
| 38 |
+
## Serving with vLLM
|
| 39 |
+
|
| 40 |
+
Install [vLLM](https://github.com/vllm-project/vllm) `>= 0.23`, then serve the model:
|
| 41 |
+
|
| 42 |
+
```bash
|
| 43 |
+
vllm serve HPAI-BSC/Qwen3.6-27B-RL-Verif \
|
| 44 |
+
--trust-remote-code \
|
| 45 |
+
--tensor-parallel-size 1 \
|
| 46 |
+
--reasoning-parser qwen3 \
|
| 47 |
+
--language-model-only \
|
| 48 |
+
--enable-auto-tool-choice \
|
| 49 |
+
--tool-call-parser qwen3_coder
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
+
## MareNostrum 5
|
| 53 |
+
|
| 54 |
+
From a login node, submit [`launch.slurm`](launch.slurm) with `sbatch`:
|
| 55 |
+
|
| 56 |
+
```bash
|
| 57 |
+
sbatch launch.slurm
|
| 58 |
+
```
|
| 59 |
+
|
| 60 |
+
Once the model is loaded, the job output prints the connection details:
|
| 61 |
+
|
| 62 |
+
```
|
| 63 |
+
============================================
|
| 64 |
+
vLLM Server is UP and READY!
|
| 65 |
+
============================================
|
| 66 |
+
Job ID: 43755135
|
| 67 |
+
Node IP: 10.2.101.26
|
| 68 |
+
vLLM Port: 41211
|
| 69 |
+
|
| 70 |
+
To test from a login node:
|
| 71 |
+
curl http://10.2.101.26:41211/v1/models
|
| 72 |
+
|
| 73 |
+
To connect from your laptop (replace bscXXXXXX with your BSC username):
|
| 74 |
+
ssh -N -L 8000:10.2.101.26:41211 bscXXXXXX@glogin1.bsc.es
|
| 75 |
+
|
| 76 |
+
Then test from your laptop:
|
| 77 |
+
curl http://localhost:8000/v1/models
|
| 78 |
+
|
| 79 |
+
Server will keep running until time limit or scancel 43755135
|
| 80 |
+
============================================
|
| 81 |
+
```
|
| 82 |
+
|
| 83 |
+
Open the SSH tunnel it gives you (forwards local port `8000` to the compute node):
|
| 84 |
+
|
| 85 |
+
```bash
|
| 86 |
+
ssh -N -L 8000:10.2.101.26:41211 bscXXXXXX@glogin1.bsc.es
|
| 87 |
+
```
|
| 88 |
+
|
| 89 |
+
Then send a request from your laptop:
|
| 90 |
+
|
| 91 |
+
```bash
|
| 92 |
+
curl http://localhost:8000/v1/chat/completions \
|
| 93 |
+
-H "Content-Type: application/json" \
|
| 94 |
+
-d '{
|
| 95 |
+
"model": "Qwen3.6-27B-RL-Verif",
|
| 96 |
+
"messages": [{"role": "user", "content": "Hello world"}]
|
| 97 |
+
}'
|
| 98 |
+
```
|
| 99 |
+
|
| 100 |
+
# Contact
|
| 101 |
+
|
| 102 |
+
For questions, reach out to [cristian.gutierrez@bsc.es](mailto:cristian.gutierrez@bsc.es).
|
README.md
CHANGED
|
@@ -1,102 +1 @@
|
|
| 1 |
-
|
| 2 |
-
|
| 3 |
-
A finetune of Qwen3.6-27B specialized for SystemVerilog test-stimulus and test-checking
|
| 4 |
-
|
| 5 |
-
Training was done in two stages:
|
| 6 |
-
|
| 7 |
-
- Supervised Finetuning (SFT) on 70k examples distilled from Gemma-4-31B
|
| 8 |
-
- Reinforcement Learning (RL) on 12k SystemVerilog environments
|
| 9 |
-
|
| 10 |
-
A smaller 9B variant is trained similarly.
|
| 11 |
-
|
| 12 |
-
# CVDP Results
|
| 13 |
-
|
| 14 |
-
<img src="https://cdn-uploads.huggingface.co/production/uploads/64da082a79c87e13ca232042/MZwGhx1CoxyJ7F1rUHZCL.png" width=800px />
|
| 15 |
-
|
| 16 |
-
Full results on the [NVIDIA CVDP benchmark](https://github.com/NVlabs/cvdp_benchmark).
|
| 17 |
-
|
| 18 |
-
| | GLM-5.2 | Kimi-K2.5 | R1-0528 | Gemma-4-31B | Qwen3.6-27B | **Our 27B model** |
|
| 19 |
-
|:----|--:|--:|--:|--:|--:|--:|
|
| 20 |
-
| **CVDP Design Verification** | | | | | | |
|
| 21 |
-
| `cid012*` Testbench Stimulus Gen. | 34.03% | 22.09% | 22.99% | 24.78% | 19.40% | **39.40%** |
|
| 22 |
-
| `cid013*` Testbench Checker Gen. | 13.96% | 23.40% | 18.87% | 17.74% | 15.09% | **31.32%** |
|
| 23 |
-
| `cid014*` Assertion Generation | 33.13% | 36.12% | 22.69% | 33.13% | 22.39% | **39.70%** |
|
| 24 |
-
| `cid016` Debugging / Bug Fixing | 44.57% | **64.57%** | 54.29% | 63.43% | 58.86% | 56.57% |
|
| 25 |
-
| **CVDP Code Generation** | | | | | | |
|
| 26 |
-
| `cid002` Code Completion | **48.51%** | 44.04% | 34.47% | 39.57% | 32.55% | 29.57% |
|
| 27 |
-
| `cid003` Natural Language Spec to Code | 52.82% | **56.15%** | 41.28% | 48.21% | 37.18% | 50.00% |
|
| 28 |
-
| `cid004` Code Modification | 50.91% | 50.55% | 37.45% | **54.55%** | 35.27% | 41.45% |
|
| 29 |
-
| `cid007` Code Improvement (Linting/QoR) | 51.50% | 59.50% | 39.50% | **64.50%** | 50.00% | 40.50% |
|
| 30 |
-
| **CVDP Code Comprehension** | | | | | | |
|
| 31 |
-
| `cid006` RTL ↔ Specification | 69.17% | 70.53% | **72.77%** | 67.66% | 68.25% | 72.69% |
|
| 32 |
-
| `cid008` Testbench ↔ Test Plan | 61.83% | **66.69%** | 45.65% | 63.10% | 62.06% | 58.08% |
|
| 33 |
-
|
| 34 |
-
Note that those categories with asterisks in the title were converted from Commercial tools (XCelium) to OSS tools (Verilator)
|
| 35 |
-
|
| 36 |
-
# Deployment
|
| 37 |
-
|
| 38 |
-
## Serving with vLLM
|
| 39 |
-
|
| 40 |
-
Install [vLLM](https://github.com/vllm-project/vllm) `>= 0.23`, then serve the model:
|
| 41 |
-
|
| 42 |
-
```bash
|
| 43 |
-
vllm serve HPAI-BSC/Qwen3.6-27B-RL-Verif \
|
| 44 |
-
--trust-remote-code \
|
| 45 |
-
--tensor-parallel-size 1 \
|
| 46 |
-
--reasoning-parser qwen3 \
|
| 47 |
-
--language-model-only \
|
| 48 |
-
--enable-auto-tool-choice \
|
| 49 |
-
--tool-call-parser qwen3_coder
|
| 50 |
-
```
|
| 51 |
-
|
| 52 |
-
## MareNostrum 5
|
| 53 |
-
|
| 54 |
-
From a login node, submit [`launch.slurm`](launch.slurm) with `sbatch`:
|
| 55 |
-
|
| 56 |
-
```bash
|
| 57 |
-
sbatch launch.slurm
|
| 58 |
-
```
|
| 59 |
-
|
| 60 |
-
Once the model is loaded, the job output prints the connection details:
|
| 61 |
-
|
| 62 |
-
```
|
| 63 |
-
============================================
|
| 64 |
-
vLLM Server is UP and READY!
|
| 65 |
-
============================================
|
| 66 |
-
Job ID: 43755135
|
| 67 |
-
Node IP: 10.2.101.26
|
| 68 |
-
vLLM Port: 41211
|
| 69 |
-
|
| 70 |
-
To test from a login node:
|
| 71 |
-
curl http://10.2.101.26:41211/v1/models
|
| 72 |
-
|
| 73 |
-
To connect from your laptop (replace bscXXXXXX with your BSC username):
|
| 74 |
-
ssh -N -L 8000:10.2.101.26:41211 bscXXXXXX@glogin1.bsc.es
|
| 75 |
-
|
| 76 |
-
Then test from your laptop:
|
| 77 |
-
curl http://localhost:8000/v1/models
|
| 78 |
-
|
| 79 |
-
Server will keep running until time limit or scancel 43755135
|
| 80 |
-
============================================
|
| 81 |
-
```
|
| 82 |
-
|
| 83 |
-
Open the SSH tunnel it gives you (forwards local port `8000` to the compute node):
|
| 84 |
-
|
| 85 |
-
```bash
|
| 86 |
-
ssh -N -L 8000:10.2.101.26:41211 bscXXXXXX@glogin1.bsc.es
|
| 87 |
-
```
|
| 88 |
-
|
| 89 |
-
Then send a request from your laptop:
|
| 90 |
-
|
| 91 |
-
```bash
|
| 92 |
-
curl http://localhost:8000/v1/chat/completions \
|
| 93 |
-
-H "Content-Type: application/json" \
|
| 94 |
-
-d '{
|
| 95 |
-
"model": "Qwen3.6-27B-RL-Verif",
|
| 96 |
-
"messages": [{"role": "user", "content": "Hello world"}]
|
| 97 |
-
}'
|
| 98 |
-
```
|
| 99 |
-
|
| 100 |
-
# Contact
|
| 101 |
-
|
| 102 |
-
For questions, reach out to [cristian.gutierrez@bsc.es](mailto:cristian.gutierrez@bsc.es).
|
|
|
|
| 1 |
+
See [`INTRO.md`](/INTRO.md)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|