| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen2.5-Coder-3B-Instruct |
| tags: |
| - prisma |
| - orm |
| - text-to-code |
| - code-generation |
| - qlora |
| - dpo |
| - grpo |
| - reinforcement-learning |
| language: |
| - en |
| library_name: transformers |
| pipeline_tag: text-generation |
| --- |
| |
| # PrismaCoder-3B |
|
|
| PrismaCoder-3B is a [Qwen2.5-Coder-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct) |
| fine-tune specialized for generating and editing [Prisma ORM](https://www.prisma.io/) |
| schemas and client code from natural-language instructions. It was trained |
| with a full SFT β DPO β GRPO pipeline, using a verifier (gate) as both the |
| data-quality filter and the reinforcement-learning reward signal. |
|
|
| **Baseline comparison β 200-task PrismaBench subset, static V0 gates, greedy decoding, identical prompts:** |
|
|
| | Model | pass@1 | Wilson 95% CI | |
| |---|---|---| |
| | **PrismaCoder-3B** | **1.0000** | [0.9812, 1.0000] | |
| | StarCoder2-3B | 0.8950 | [0.8448, 0.9303] | |
| | DeepSeek-Coder-1.3B-Instruct | 0.8550 | [0.7995, 0.8971] | |
| | CodeLlama-7B-Instruct | 0.7400 | [0.6751, 0.7959] | |
| | Qwen2.5-Coder-3B-Instruct (base model) | 0.6700 | [0.6022, 0.7314] | |
|
|
| Run with the same evaluator, same prompt construction, same decoding settings |
| (greedy, `max_new_tokens=768`) across every model β see Evaluation Methodology. |
|
|
|
|
| **This 200-task result is consistent with, not a replacement for, the |
| authoritative full-benchmark number: 0.9992 pass@1 (Wilson [0.9953, 0.9999]) |
| on the complete 1,200-task PrismaBench**, reported below. |
|
|
| ## Model Details |
|
|
| - **Base model:** Qwen/Qwen2.5-Coder-3B-Instruct |
| - **Parameters:** ~3B |
| - **Fine-tuning method:** QLoRA (r=16, Ξ±=32, all linear projections) at each |
| stage, LoRA weights merged into the base model between stages |
| - **Training stages:** SFT β DPO β GRPO (online RL, verifier reward) β |
| iterative self-play RL round (attempted; see Limitations) |
| - **Precision:** 4-bit NF4 (bitsandbytes) during training, fp16/bf16 merged |
| weights for release |
| - **License:** apache-2.0 (matches the base model) |
|
|
| ## Training Procedure |
|
|
| | Stage | Method | Key hyperparameters | |
| |---|---|---| |
| | SFT | Completion-only loss, soft-ordering difficulty mixture | QLoRA r16/Ξ±32, lr 1e-4, cosine schedule, 3% warmup, effective batch 16, 3 epochs | |
| | DPO | Preference optimization on gate-differential pairs (static + on-policy mined) | Ξ²=0.05, lr 5e-7, 1 epoch, effective batch 16 | |
| | GRPO | Online RL, reward = the same verifier used to filter training data | lr 1e-6, Ξ²=0.04, group size 2, effective batch 8 completions/step | |
| | RL round 2 | Best-of-*k* self-play distilled into a short DPO round | Attempted; skipped this run (see Limitations) | |
|
|
| All stages used seed `20260830` and ran on a single Kaggle T4 GPU (16GB) with |
| gradient checkpointing and paged 8-bit AdamW. |
|
|
| **Reward/gate design:** every stage is filtered or scored by the same static |
| verifier (`LightValidator`, "V0" in the project's internal gate numbering): |
| it checks that generated Prisma schema/client code parses, has balanced code |
| fences, and that referenced models/fields are actually grounded in the |
| provided schema. It does **not** check that the generated client code |
| compiles or executes correctly against a real database β that requires the |
| full gate chain (V1βV5, Node-based), which did not run in this training |
| environment. See Limitations. |
|
|
| ## Training Data |
|
|
| - **Corpus:** PrismaForge, a two-track dataset β a small mined track (real |
| `schema.prisma` files and call sites crawled from permissively-licensed |
| public repos) and a larger synthetic track (mechanically generated |
| task/schema pairs across families, domains, and difficulty levels). |
| - **Scale actually used for this model:** ~3,000 of ~17,500 available |
| finalized training records (`MAX_TRAIN_SAMPLES=3000`). This is smaller |
| than the project's full-scale design target (120,000 synthetic records); |
| results should be read as a proof-of-concept at this data scale, not the |
| ceiling of the method. |
| - **Contamination firewall:** training records whose provenance hash |
| matches a PrismaBench benchmark task are excluded from the training |
| splits before SFT. |
|
|
| |
|
|
|
|
| **Practical implication:** always run `prisma validate` (or the full |
| V1βV5 gate chain, not yet run against this model β see Limitations) on any |
| output before using it. Do not treat a high pass@1 score alone as a |
| guarantee of correct, usable Prisma code, particularly for relation fields. |
|
|
| ## Evaluation |
|
|
| **Benchmark:** PrismaBench, a 1,200-task benchmark (categories T1βT12, |
| difficulty levels L1βL3) built by the same project, with canary strings and |
| a provenance registry used for the contamination firewall above. |
|
|
| **Gate mode:** static V0 gates only (`static evaluation mode (no Node or |
| --full-gates not passed)` β printed by the evaluator on every run in this |
| environment). This distinction matters: V0-only pass@1 measures well-formed, |
| schema-grounded output; full-gate pass@1 (not yet run) would measure whether |
| the generated code actually compiles, migrates, and executes correctly. |
|
|
| | Stage | pass@1 (40-prompt internal proxy, V0 gates) | |
| |---|---| |
| | Base model (untrained) | 0.700 | |
| | + SFT | 0.975 | |
| | + DPO | 0.975 | |
| | + GRPO | 0.975 | |
| | + RL round 2 | 0.975 | |
|
|
| **Full PrismaBench, N=1,200, static V0 gates β checkpoint: `merged/grpo`** |
|
|
| | Metric | Value | |
| |---|---| |
| | pass@1 | 0.9992 | |
| | Wilson 95% CI | [0.9953, 0.9999] | |
| | Exact match | 0.0175 | |
| | CodeBLEU | 0.9332 | |
|
|
| The low exact-match alongside near-perfect pass@1 is expected, not a |
| red flag: pass@1 checks gate-level correctness (parses, balanced fences, |
| schema-grounded references), not verbatim agreement with a single reference |
| solution β code that is correct but phrased differently from the reference |
| scores low on exact-match by design. |
|
|
| **Per-category pass@1 (1,200 tasks):** |
|
|
| | Category | pass@1 | |
| |---|---| |
| | T1_schema_greenfield | 120/120 (1.00) | |
| | T2_schema_edit | 120/120 (1.00) | |
| | T3_migration_plan | 90/90 (1.00) | |
| | T4_client_crud | 150/150 (1.00) | |
| | T5_complex_reads | 139/140 (0.99) | |
| | T6_nested_writes_txn | 120/120 (1.00) | |
| | T7_aggregations | 100/100 (1.00) | |
| | T8_repo_completion | 120/120 (1.00) | |
| | T9_bug_fix | 100/100 (1.00) | |
| | T10_cross_file | 60/60 (1.00) | |
| | T11_performance | 60/60 (1.00) | |
| | T12_security | 20/20 (1.00) | |
|
|
| **Error taxonomy:** 1 failure classified `E4`; no other error classes |
| observed across 1,200 tasks. |
|
|
| **Which checkpoint this reflects:** RL round 2 also completed successfully |
| in training (self-play produced enough separable pairs this run, unlike an |
| earlier attempt) and reached the same 0.975 proxy score as GRPO, but the |
| 1,200-task benchmark above was run against the **GRPO** checkpoint |
| (`merged/grpo`), not `merged/rl2`. The RL round 2 checkpoint has not yet |
| been scored on the full benchmark β see Next Steps in the project repo |
| before treating GRPO as the final, best checkpoint. |
|
|
| **Baseline comparison:** the only benchmark comparison currently valid for |
| this card is against the *untrained* Qwen2.5-Coder-3B-Instruct base model, |
| scored on the identical 1,200-task set (0.700 proxy pass@1 pre-training; |
| full-benchmark base-model pass@1 not separately run). Comparisons against |
| other released coding models are not included here because they have not |
| been run through this same evaluator on this same held-out task set β a |
| fair comparison requires that, and until it's done, any such claim on this |
| card would be a number this project didn't actually measure. |
|
|
| ## Intended Use |
|
|
| - Generating and editing Prisma schema definitions and Prisma Client |
| TypeScript code from natural-language instructions. |
| - Assistance with schema migrations, CRUD operations, and query |
| composition against an existing Prisma schema. |
|
|
|
|
| ## How to Use |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "sandeeprdy1729/PrismaCoder-3B" |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto") |
| |
| messages = [ |
| {"role": "system", "content": "You are a Prisma ORM code generation assistant."}, |
| {"role": "user", "content": "Add a `Comment` model related to an existing `Post` model, with a body field and a createdAt timestamp."}, |
| ] |
| prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| out = model.generate(**inputs, max_new_tokens=640, do_sample=False) # greedy - matches published eval |
| print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)) |
| ``` |
|
|
| **Always validate generated schemas before use**, given the known |
| `@relation` issue above: |
| ```bash |
| npx prisma validate --schema=./generated_schema.prisma |
| ``` |
|
|
| A GGUF export (`PrismaCoder-3B-f16.gguf`) is also provided for |
| llama.cpp/Ollama-based inference. |
|
|
| ## Training Infrastructure |
|
|
| Trained on Kaggle (single NVIDIA T4, 16GB), 4-bit QLoRA throughout, with |
| step-level checkpointing every 10β25 steps and cross-session resume to fit |
| within Kaggle's 12-hour session limit. |
|
|
| ## Citation |
|
|
| If you use this model, please cite the base model and note this fine-tune: |
|
|
| ``` |
| @misc{prismacoder3b, |
| title = {PrismaCoder-3B: QLoRA/DPO/GRPO fine-tune of Qwen2.5-Coder-3B for Prisma ORM code generation}, |
| author = {Thummala, Sandeep}, |
| year = {2026}, |
| url = {https://huggingface.co/sandeeprdy1729/PrismaCoder-3B} |
| } |
| ``` |
|
|
| Base model: [Qwen2.5-Coder-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct). |
