Model card

README.md

@@ -0,0 +1,263 @@
+---
+license: apache-2.0
+base_model: KBBridge/KBBridge-v3
+base_model_relation: quantized
+tags: [genexus, code-generation, qwen3, fp8, vllm]
+language: [es, en]
+pipeline_tag: text-generation
+---
+# KBBridge-v3 (FP8)
+
+A fine-tune of [`Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B) specialised in
+**GeneXus** programming, in the native `.gxSource` export format.
+
+Frontier models do not know this format. Without the GeneXus documentation injected into the
+prompt they produce syntactically invalid output almost every time (parse rate 0.5–3.1%).
+KBBridge writes it natively, runs on your own hardware, and never sends your Knowledge Base
+code to an external API.
+
+---
+
+## ⚠️ Read this before your first prompt
+
+Two settings, or the model will look broken. Both are measured, not stylistic.
+
+### 1. Turn reasoning OFF
+
+```bash
+# vLLM: pass it per request
+#   "chat_template_kwargs": {"enable_thinking": false}
+```
+
+The Qwen chat template enables a `<think>` block by default. If that block does not close
+within your token budget, the answer comes back **empty or half-finished** — the client sees
+"the model did not respond". Measured on this GGUF: with reasoning on, 300 tokens were not
+enough to even begin the object; with it off, the same prompt returned a complete, valid
+`Procedure`.
+
+Our own gateway sets this by default for exactly this reason.
+
+### 2. Ask for the format explicitly
+
+Write **"in `.gxSource` format"** in your prompt.
+
+Measured on v3: the bare request *"a Procedure that adds two numbers"* returns generic **SQL**.
+Naming the format returns the GeneXus object, consistently. If you use a harness with its own
+system prompt, put the instruction there once.
+
+### 3. Give it enough room
+
+`max_tokens` ≥ 4096. A `.gxSource` object consumes roughly **340 tokens per KB** of source, and
+most tools default to 512–1024, which truncates the object mid-body.
+
+---
+
+## Results
+
+580 held-out items (191 codegen + 329 MCQ + 60 data-model) that no model saw during training.
+Syntax validated with the official GeneXus ANTLR parser. Same protocol for every model:
+temperature 0.1, reasoning off, concurrency 8.
+
+### v3 vs v2 — an honest comparison
+
+**v3 is not a clean win over v2.** It gains domain knowledge and loses syntax accuracy:
+
+| Metric | v2 | **v3** | |
+|---|---|---|---|
+| parseRate (valid syntax) | **89.0** | 84.8 | −4.2 |
+| parmMatch (exact signature) | 78.6 | 78.6 | = |
+| MCQ (GeneXus knowledge) | 76.0 | **79.0** | +3.0 |
+| methodValidity | 90.0 | **91.1** | +1.1 |
+
+**What these numbers do NOT establish.** v3 changed three things at once — the base model
+(Qwen3.6 → 3.8), the corpus (4× larger, per-KB cap removed) and the teacher (v1 → v2). The
+parseRate drop **cannot be attributed** to any one of them without a control arm that was never
+run. Anyone reading this table as "the bigger corpus hurt syntax" is over-reading it.
+
+Choose v3 if domain knowledge matters more to you; v2 still leads on raw syntax validity.
+
+### Generalisation to unseen Knowledge Bases
+
+Three entire KBs were held out — different domains, never in the pipeline:
+
+| | held-out from training KBs | 3 completely new KBs |
+|---|---|---|
+| v2 | 89.0 | 89.9 |
+| **v3** | 84.8 | **87.4** |
+
+v3's *relative* gap to unseen KBs is larger than v2's (+2.6 vs +0.9), i.e. it generalises
+better in relative terms, even though two KBs make up 54.7% of its corpus.
+
+### Fairness note on the frontier comparison
+
+In our benchmark the frontier models were run **with** ~21,600 tokens of GeneXus documentation
+injected into every request; KBBridge was run **without** any. That is not a handicap we
+imposed — injecting the same documentation into KBBridge makes it *worse* (76.4 → 73.3
+parseRate), because the fine-tune already internalised that knowledge and the extra context
+gets in the way. Still, the setups differ, and you should know that when reading any
+head-to-head number.
+
+### About quantisation quality
+
+The 4-bit GGUF build was measured against the bf16 master on the same 580 items, and
+**excluding items where either run hit the token ceiling the two are indistinguishable**
+(parseRate 93.0 vs 93.6 over 171 items). FP8 is a lighter quantisation than that, so the same
+conclusion applies with room to spare — though we did not benchmark this FP8 build separately.
+
+---
+
+## Files
+
+FP8 dynamic quantisation, **29 GB**, single shard. Built for vLLM on Hopper/Blackwell-class
+GPUs. This is the build we run in production.
+
+```bash
+vllm serve KBBridge/KBBridge-v3-FP8 --served-model-name kbbridge-v3 \
+  --max-model-len 262144 --kv-cache-dtype fp8 \
+  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder
+```
+
+> ### ⚠️ This build has no MTP head
+>
+> It was quantised before the multi-token-prediction head was restored to the bf16 master, so
+> its 1,680 tensors include the vision tower but **not** the 15 MTP tensors. Speculative
+> decoding is therefore unavailable here.
+>
+> That costs real speed: on the GGUF build, enabling the MTP head took generation from 62 to
+> **109 tok/s (+71%)**. If you need that, quantise your own FP8 from
+> [`KBBridge/KBBridge-v3`](https://huggingface.co/KBBridge/KBBridge-v3), which does include it.
+> We will replace this build when we re-quantise.
+
+## Intended use
+
+Assisting GeneXus developers: generating objects (Procedures, Transactions, Data Providers,
+SDTs, WebPanels), explaining existing code, completion, and documentation questions.
+
+**Out of scope:** not a general-purpose model, not a replacement for validating in the GeneXus
+IDE, and it does not know any particular Knowledge Base (see *Limitations*).
+
+---
+
+## Limitations
+
+- **It does not know your KB.** It learned the style and syntax of the format, not the contents
+  of any specific base. Ask it about a transaction you did not paste in, and it will **invent
+  plausible attribute names and present them as fact**. Always give it the context and validate
+  the output in the IDE.
+- **Runaway generation on very large objects.** For objects over ~10 KB the model can fall into
+  degenerate repetition — the same line hundreds of times without closing the object. Measured
+  on v2 at ~1.6% of benchmark items; **not re-measured on v3**. Raising `max_tokens` does not
+  fix it. Generate large objects section by section.
+- **Spanish bias** in explanations, reflecting the corpus.
+- **Specialised**: worse than the base model at general tasks.
+- The limitations above other than the first were measured on **v2** and are carried over as
+  working assumptions, not verified properties of v3.
+
+### If you also use a hosted KBBridge endpoint
+
+The raw GGUF and a gateway-fronted deployment **do not behave the same by default**. Our
+gateway applies four corrections the plain model does not have: a `max_tokens` floor, reasoning
+disabled, a `reasoning_content` fallback when `content` comes back empty, and
+`repetition_penalty` 1.05 to suppress runaway. If you compare "what I tried on your server"
+against "what I downloaded", the difference is those four settings, not the weights.
+
+---
+
+## Training
+
+| | |
+|---|---|
+| Method | QLoRA 4-bit (bitsandbytes) + Liger kernel |
+| LoRA | r=64, α=128, dropout=0.05, all projections |
+| Context | 12,288 tokens |
+| Effective batch | 16 (1 × 16 grad accum) |
+| LR | 1.0e-4, cosine, 3% warmup |
+| Epochs | 2 complete (14,108 steps) |
+| Hardware | 1× RTX PRO 6000 Blackwell 96 GB |
+| Duration | 7 days 4:41 |
+| Framework | LLaMA-Factory, transformers 5.6.0 |
+
+train_loss **0.2618** (v2: 0.3344) · eval_loss **0.3723** (v2: 0.4675), minimum at the **last**
+step — no overfitting across 71 evaluations, which suggests there was room for more epochs.
+
+Note that these losses are much better than v2's and yet parseRate went *down*: `eval_loss`
+measures fit to the corpus, not GeneXus quality.
+
+### Data
+
+80,344 examples derived from GeneXus objects across 25 real Knowledge Bases (GX16/17/17U8/18/
+Evo1, multi-domain) — 129% more than v2, with the per-KB cap removed. Sanitised, deduplicated
+and split by deterministic hash. **The datasets are not published**: they contain customer
+proprietary code.
+
+---
+
+## Training-data privacy
+
+The model was trained on real customer Knowledge Bases, so we audited whether it can leak them.
+This is the strongest result of the project.
+
+### Canaries: no memorisation threshold found
+
+12 synthetic objects containing unguessable 16-character secrets were inserted at four
+frequencies, and verified to have reached `train.jsonl` at exactly those counts:
+
+| repetitions | canaries | recovered by name | recovered with literal prefix |
+|---|---|---|---|
+| 1 | 3 | 0/3 | 0/3 |
+| 10 | 3 | 0/3 | 0/3 |
+| 100 | 3 | 0/3 | 0/3 |
+| **1000** | 3 | **0/3** | **0/3** |
+
+**Not even at a thousand identical repetitions.** A control rules out a broken probe: asked for
+the canary, the model returns a structurally valid but **empty** object — no token, no secret.
+And it does generate real bodies when the request has content, so the empty skeleton is not an
+inability to generate.
+
+### Membership inference: marginal signal
+
+| | |
+|---|---|
+| mean loss, seen examples | 3.4130 |
+| mean loss, unseen | 3.7711 |
+| mean length | 3,133 vs 3,117 chars — comparable, so the AUC is meaningful |
+| **AUC** | **0.5539** |
+
+0.554 against 0.50 for indistinguishable. There is a statistical trace of having seen the data,
+but the distributions overlap almost entirely.
+
+**Conclusion: customer code is not recoverable from the weights.**
+
+**Caveat, stated plainly:** absence of evidence is not proof of absence. These audits cover the
+attacks we ran, not every attack that exists.
+
+---
+
+## Reproducibility
+
+Full external reproduction is **not possible**, and it is worth saying so directly:
+
+1. The 25 Knowledge Bases are customer code and are not distributed.
+2. The `parseRate` scorer uses the KBEditor's ANTLR parser — proprietary, not distributable.
+3. The teacher that generated v3's data is KBBridge-v2, which is not published.
+
+What a third party *can* verify: the raw benchmark outputs (one model response per item) and
+the scoring over them.
+
+---
+
+## Citation
+
+```bibtex
+@misc{kbbridge-v3,
+  title  = {KBBridge-v3: a GeneXus code assistant fine-tuned from Qwen3.8-27B},
+  author = {Nardone, Angelo},
+  year   = {2026},
+  url    = {https://huggingface.co/KBBridge/KBBridge-v3-FP8}
+}
+```
+
+## License
+
+Apache 2.0, inherited from the base model `Qwen/Qwen3.8-27B`. This is a modified derivative
+work; see `NOTICE`.