Blame for README.md

---
license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-3B-Instruct
tags:
  - prisma
  - orm
  - text-to-code
  - code-generation
  - qlora
  - dpo
  - grpo
  - reinforcement-learning
language:
  - en
library_name: transformers
pipeline_tag: text-generation
---

# PrismaCoder-3B

PrismaCoder-3B is a [Qwen2.5-Coder-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct)
fine-tune specialized for generating and editing [Prisma ORM](https://www.prisma.io/)
schemas and client code from natural-language instructions. It was trained
with a full SFT → DPO → GRPO pipeline, using a verifier (gate) as both the
data-quality filter and the reinforcement-learning reward signal.

**Baseline comparison — 200-task PrismaBench subset, static V0 gates, greedy decoding, identical prompts:**

| Model | pass@1 | Wilson 95% CI |
|---|---|---|
| **PrismaCoder-3B** | **1.0000** | [0.9812, 1.0000] |
| StarCoder2-3B | 0.8950 | [0.8448, 0.9303] |
| DeepSeek-Coder-1.3B-Instruct | 0.8550 | [0.7995, 0.8971] |
| CodeLlama-7B-Instruct | 0.7400 | [0.6751, 0.7959] |
| Qwen2.5-Coder-3B-Instruct (base model) | 0.6700 | [0.6022, 0.7314] |

Run with the same evaluator, same prompt construction, same decoding settings
(greedy, `max_new_tokens=768`) across every model — see Evaluation Methodology.


**This 200-task result is consistent with, not a replacement for, the
authoritative full-benchmark number: 0.9992 pass@1 (Wilson [0.9953, 0.9999])
on the complete 1,200-task PrismaBench**, reported below.

## Model Details

- **Base model:** Qwen/Qwen2.5-Coder-3B-Instruct
- **Parameters:** ~3B
- **Fine-tuning method:** QLoRA (r=16, α=32, all linear projections) at each
  stage, LoRA weights merged into the base model between stages
- **Training stages:** SFT → DPO → GRPO (online RL, verifier reward) →
  iterative self-play RL round (attempted; see Limitations)
- **Precision:** 4-bit NF4 (bitsandbytes) during training, fp16/bf16 merged
  weights for release
- **License:** apache-2.0 (matches the base model)

## Training Procedure

| Stage | Method | Key hyperparameters |
|---|---|---|
| SFT | Completion-only loss, soft-ordering difficulty mixture | QLoRA r16/α32, lr 1e-4, cosine schedule, 3% warmup, effective batch 16, 3 epochs |
| DPO | Preference optimization on gate-differential pairs (static + on-policy mined) | β=0.05, lr 5e-7, 1 epoch, effective batch 16 |
| GRPO | Online RL, reward = the same verifier used to filter training data | lr 1e-6, β=0.04, group size 2, effective batch 8 completions/step |
| RL round 2 | Best-of-*k* self-play distilled into a short DPO round | Attempted; skipped this run (see Limitations) |

All stages used seed `20260830` and ran on a single Kaggle T4 GPU (16GB) with
gradient checkpointing and paged 8-bit AdamW.

**Reward/gate design:** every stage is filtered or scored by the same static
verifier (`LightValidator`, "V0" in the project's internal gate numbering):
it checks that generated Prisma schema/client code parses, has balanced code
fences, and that referenced models/fields are actually grounded in the
provided schema. It does **not** check that the generated client code
compiles or executes correctly against a real database — that requires the
full gate chain (V1–V5, Node-based), which did not run in this training
environment. See Limitations.

## Training Data

- **Corpus:** PrismaForge, a two-track dataset — a small mined track (real
  `schema.prisma` files and call sites crawled from permissively-licensed
  public repos) and a larger synthetic track (mechanically generated
  task/schema pairs across families, domains, and difficulty levels).
- **Scale actually used for this model:** ~3,000 of ~17,500 available
  finalized training records (`MAX_TRAIN_SAMPLES=3000`). This is smaller
  than the project's full-scale design target (120,000 synthetic records);
  results should be read as a proof-of-concept at this data scale, not the
  ceiling of the method.
- **Contamination firewall:** training records whose provenance hash
  matches a PrismaBench benchmark task are excluded from the training
  splits before SFT.

 


**Practical implication:** always run `prisma validate` (or the full
V1–V5 gate chain, not yet run against this model — see Limitations) on any
output before using it. Do not treat a high pass@1 score alone as a
guarantee of correct, usable Prisma code, particularly for relation fields.

## Evaluation

**Benchmark:** PrismaBench, a 1,200-task benchmark (categories T1–T12,
difficulty levels L1–L3) built by the same project, with canary strings and
a provenance registry used for the contamination firewall above.

**Gate mode:** static V0 gates only (`static evaluation mode (no Node or
--full-gates not passed)` — printed by the evaluator on every run in this
environment). This distinction matters: V0-only pass@1 measures well-formed,
schema-grounded output; full-gate pass@1 (not yet run) would measure whether
the generated code actually compiles, migrates, and executes correctly.

| Stage | pass@1 (40-prompt internal proxy, V0 gates) |
|---|---|
| Base model (untrained) | 0.700 |
| + SFT | 0.975 |
| + DPO | 0.975 |
| + GRPO | 0.975 |
| + RL round 2 | 0.975 |

**Full PrismaBench, N=1,200, static V0 gates — checkpoint: `merged/grpo`**

| Metric | Value |
|---|---|
| pass@1 | 0.9992 |
| Wilson 95% CI | [0.9953, 0.9999] |
| Exact match | 0.0175 |
| CodeBLEU | 0.9332 |

The low exact-match alongside near-perfect pass@1 is expected, not a
red flag: pass@1 checks gate-level correctness (parses, balanced fences,
schema-grounded references), not verbatim agreement with a single reference
solution — code that is correct but phrased differently from the reference
scores low on exact-match by design.

**Per-category pass@1 (1,200 tasks):**

| Category | pass@1 |
|---|---|
| T1_schema_greenfield | 120/120 (1.00) |
| T2_schema_edit | 120/120 (1.00) |
| T3_migration_plan | 90/90 (1.00) |
| T4_client_crud | 150/150 (1.00) |
| T5_complex_reads | 139/140 (0.99) |
| T6_nested_writes_txn | 120/120 (1.00) |
| T7_aggregations | 100/100 (1.00) |
| T8_repo_completion | 120/120 (1.00) |
| T9_bug_fix | 100/100 (1.00) |
| T10_cross_file | 60/60 (1.00) |
| T11_performance | 60/60 (1.00) |
| T12_security | 20/20 (1.00) |

**Error taxonomy:** 1 failure classified `E4`; no other error classes
observed across 1,200 tasks.

**Which checkpoint this reflects:** RL round 2 also completed successfully
in training (self-play produced enough separable pairs this run, unlike an
earlier attempt) and reached the same 0.975 proxy score as GRPO, but the
1,200-task benchmark above was run against the **GRPO** checkpoint
(`merged/grpo`), not `merged/rl2`. The RL round 2 checkpoint has not yet
been scored on the full benchmark — see Next Steps in the project repo
before treating GRPO as the final, best checkpoint.

**Baseline comparison:** the only benchmark comparison currently valid for
this card is against the *untrained* Qwen2.5-Coder-3B-Instruct base model,
scored on the identical 1,200-task set (0.700 proxy pass@1 pre-training;
full-benchmark base-model pass@1 not separately run). Comparisons against
other released coding models are not included here because they have not
been run through this same evaluator on this same held-out task set — a
fair comparison requires that, and until it's done, any such claim on this
card would be a number this project didn't actually measure.

## Intended Use

- Generating and editing Prisma schema definitions and Prisma Client
  TypeScript code from natural-language instructions.
- Assistance with schema migrations, CRUD operations, and query
  composition against an existing Prisma schema.


## How to Use

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sandeeprdy1729/PrismaCoder-3B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

messages = [
    {"role": "system", "content": "You are a Prisma ORM code generation assistant."},
    {"role": "user", "content": "Add a `Comment` model related to an existing `Post` model, with a body field and a createdAt timestamp."},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=640, do_sample=False)  # greedy - matches published eval
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

**Always validate generated schemas before use**, given the known
`@relation` issue above:
```bash
npx prisma validate --schema=./generated_schema.prisma
```

A GGUF export (`PrismaCoder-3B-f16.gguf`) is also provided for
llama.cpp/Ollama-based inference.

## Training Infrastructure

Trained on Kaggle (single NVIDIA T4, 16GB), 4-bit QLoRA throughout, with
step-level checkpointing every 10–25 steps and cross-session resume to fit
within Kaggle's 12-hour session limit.

## Citation

If you use this model, please cite the base model and note this fine-tune:

```
@misc{prismacoder3b,
  title  = {PrismaCoder-3B: QLoRA/DPO/GRPO fine-tune of Qwen2.5-Coder-3B for Prisma ORM code generation},
  author = {Thummala, Sandeep},
  year   = {2026},
  url    = {https://huggingface.co/sandeeprdy1729/PrismaCoder-3B}
}
```

Base model: [Qwen2.5-Coder-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct).