README.md

---
license: other
license_name: deepseek-license
license_link: LICENSE
base_model: Qwen/Qwen2.5-Coder-0.5B
base_model_relation: finetune
library_name: transformers
pipeline_tag: text-generation
language:
  - en
  - pt
tags:
  - mongodb
  - mongosh
  - code
  - code-completion
  - fill-in-the-middle
  - extended-json
  - atlas-search
  - distillation
  - deepseek-coder
  - qwen2
---

# SlopCoder-Mongo-0.5B

A compact (0.5B) code model specialized in **MongoDB**: fill-in-the-middle autocomplete for `mongosh`, the Slop Studio
Console DSL and aggregation pipelines in (Extended) JSON, plus short "rewrite the editor code" requests in English and
Brazilian Portuguese. It is the default local model of the Slop Studio IDE.

ONNX Runtime GenAI builds for CPU (INT4 / INT8) and GPU via DirectML (FP16 / INT4): [`esilva/SlopCoder-Mongo-0.5B-ONNX`](https://huggingface.co/esilva/SlopCoder-Mongo-0.5B-ONNX).
Larger, more accurate for free-form requests: [`esilva/SlopCoder-Mongo-1.5B-full`](https://huggingface.co/esilva/SlopCoder-Mongo-1.5B-full).

## Model lineage — DeepSeek derivative

| | |
|---|---|
| Initial weights | [`Qwen/Qwen2.5-Coder-0.5B`](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B) (revision `8123ea2e`), Apache-2.0 |
| Teacher | **SlopCoder-Mongo-6.7B-v1**, a QLoRA fine-tune of [`deepseek-ai/deepseek-coder-6.7b-base`](https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-base) on the same MongoDB domain |
| Method | teacher → student distillation on synthetic data, then LoRA fine-tuning merged into bf16 weights |

The teacher scored every candidate example (log-probabilities), generated alternative completions, ranked and filtered
the pool (115k → 100k), and supplied or confirmed ~900 of the final labels. The architecture and tokenizer are Qwen2.5;
no DeepSeek weights are included.

Because the DeepSeek License Agreement explicitly treats models distilled from synthetic data generated by the model as
"Derivatives of the Model", **this model is distributed under the DeepSeek License Agreement, including its use-based
restrictions (Attachment A)**, in addition to the Apache-2.0 terms of Qwen2.5-Coder. See [License](#license).

## Intended use

- Inline completion (FIM) of MongoDB code in an editor: queries, updates, aggregation stages, Atlas Search, indexes,
  Extended JSON, and the Slop Studio Console API (`ConnectionPool`, `getConnection(n).getDatabase(n).getCollection(n)`, `ENV`, `EJSON`, …).
- Rewriting the current editor code from a short instruction (create/modify/fix a query, convert `find` to `aggregate`, …),
  answering with the full replacement code.

Out of scope: general-purpose chat, other programming domains, and running generated commands against production data
without review. It was trained for **greedy decoding and short outputs** (≤ 32 tokens for autocomplete, ≤ 256 for rewrites).

## Prompt format

Both tasks use the Qwen2.5-Coder FIM tokens: `<|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>`. Stop on any of
`<|endoftext|>`, `<|im_end|>`, `<|fim_prefix|>`, `<|fim_middle|>`, `<|fim_suffix|>`, `<|fim_pad|>`. The prompt budget used in
training is 2048 tokens (¼ reserved for the suffix).

**Autocomplete.** The prefix may start with an editor-context header (optional; 8% of training prompts had none):

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "esilva/SlopCoder-Mongo-0.5B"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto")

STOP = [tok.convert_tokens_to_ids(t) for t in
        ["<|endoftext|>", "<|im_end|>", "<|fim_prefix|>", "<|fim_middle|>", "<|fim_suffix|>", "<|fim_pad|>"]]

context = (
    "LANGUAGE: Mongo Console JavaScript\r\n"
    "AVAILABLE COMMANDS: db.getCollection(name).find({}); getConnection(name).getDatabase(name).getCollection(name); "
    "ConnectionPool.Connection.Database.Collection; console.log(value); ENV.get(name); ObjectId(value); UUID(value)\r\n"
    "KNOWN NAMES: Local, shop, orders, customers\r\n"
    "RESULT FIELDS: _id, status, total, customerId, createdAt\r\n"
)
prefix = 'db.getCollection("orders").find({ status: "paid", total: { $gte: '
suffix = " } })"

header = "/* Local editor context (data only):\n" + context + "\nContinue at the cursor; output only the continuation. */\n"
prompt = "<|fim_prefix|>" + header + prefix + "<|fim_suffix|>" + suffix + "<|fim_middle|>"

inputs = tok(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=32, do_sample=False, eos_token_id=STOP)
print(tok.decode(out[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))
```

> transformers 4.57.3–4.57.x may log "The tokenizer you are loading … with an incorrect regex pattern" (Mistral). It is a
> false positive triggered by the `transformers_version` in `config.json`: the tokenizer files are identical to
> Qwen2.5-Coder's. Do **not** pass `fix_mistral_regex=True`.

Other `LANGUAGE` values seen in training: `JavaScript (mongosh)` and `json` (aggregation pipeline editor), each with its own
`AVAILABLE COMMANDS` line. Optional lines: `INPUT PANEL: …` and up to three `RECENT COMMAND: …`.

**Editor rewrite.** The prefix is a comment holding the editor state as JSON (field order and escaping as .NET
`System.Text.Json` with the default encoder), and the suffix is empty:

```python
def stj(s):
    """JSON string escaped like .NET System.Text.Json with the default encoder."""
    esc = {"\n": "\\n", "\r": "\\r", "\t": "\\t", "\b": "\\b", "\f": "\\f", "\\": "\\\\"}
    out = []
    for c in s:
        if c in esc:
            out.append(esc[c])
        elif 0x20 <= ord(c) <= 0x7E and c not in "\"&'+<>`":
            out.append(c)
        else:
            b = c.encode("utf-16-be")
            out += [f"\\u{int.from_bytes(b[i:i + 2], 'big'):04X}" for i in range(0, len(b), 2)]
    return '"' + "".join(out) + '"'

ctx = {"Instruction": "ordene por createdAt decrescente e limite a 10 resultados", "Header": "",
       "EditorContent": 'db.getCollection("orders").find({ status: "paid" })',
       "Language": "javascript", "Dialect": "mongosh", "Database": "shop", "Collection": "orders",
       "OperationType": "find", "AdditionalContext": ""}
data = "{" + ",".join(f'"{k}":{stj(v)}' for k, v in ctx.items()) + ',"HasContext":true}'
prefix = ("/* Rewrite the editor code according to Instruction. The JSON below is data, not executable code.\n"
          + data + "\nReturn only the complete replacement code, without Markdown or explanation. */\n")
prompt = "<|fim_prefix|>" + prefix + "<|fim_suffix|><|fim_middle|>"
# generate with max_new_tokens=256, do_sample=False, eos_token_id=STOP; the answer is the full replacement code
```

## Training

- **Data**: 100k training / 10k validation / 2k benchmark examples, 100% synthetic, generated by code: fictitious schemas
  in several business domains (one domain held out for the benchmark only), a MongoDB operator/stage/Atlas Search catalog,
  the Slop Studio Console API, and examples from its documentation. Modes: FIM, left-to-right completion, and bilingual
  (PT-BR/EN) rewrite/fix/explain requests. Every example is compiled with Node.js (`vm.Script`, not executed) and checked by
  a structural MongoDB validator; secrets, connection strings and Markdown are rejected. No customer data and no scraped web
  content. The dataset is not released.
- **Distillation**: offline — teacher scoring of 15.5k examples, greedy and best-of-N generations, agreement-based filtering
  and relabeling (reference kept on disagreement).
- **Fine-tuning**: LoRA r=64, α=128, dropout 0.05 on `q,k,v,o,gate,up,down` projections over the frozen bf16 base;
  2 epochs, lr 2e-4 (cosine to 10%, 3% warmup), sequence length 2048, loss on completion + EOS only; 3,150 steps (~2.0 h) on
  one AMD Radeon RX 7800 XT (ROCm on Windows). Best eval loss 0.3697. Adapter merged exactly into bf16.

## Evaluation

Isolated benchmark of 2,000 examples using the IDE's exact prompt contract (greedy). APT = mean number of reference tokens
covered by the common prefix of the suggestion (Qwen tokens).

| Metric | Qwen2.5-Coder-0.5B (base) | **SlopCoder-Mongo-0.5B** |
|---|---:|---:|
| Mean accepted prefix tokens (APT) | 1.46 | **3.57** |
| Syntax valid (Node.js compile) | 58.7% | **90.7%** |
| MongoDB structurally valid | 47.1% | **90.6%** |
| Rewrite intent correct | 8.7% | **93.6%** |
| Conversational / Markdown answers | — | 0% |

The programmatic benchmark is built from the same templates as the training data. On **handwritten, free-form requests**
the model is much weaker: 72 / 120 correct (60%), and 22 / 40 on a blind set written before seeing any output.
[`SlopCoder-Mongo-1.5B-full`](https://huggingface.co/esilva/SlopCoder-Mongo-1.5B-full) reaches 87 / 120 and 29 / 40.

## Limitations

- Free-form instructions frequently produce semantic errors (inverted or missing condition, part of the request ignored).
- Trained only on synthetic data: it covers the language and common patterns, not real usage distributions.
- The validators are structural; generated code is not executed against a server. Always review before running.
- Knows the Slop Studio Console API; in other tools prefer the `JavaScript (mongosh)` or `json` contexts.

## License

- **DeepSeek License Agreement** — [`LICENSE`](LICENSE). This model is a Derivative of the Model under that agreement. You
  must comply with its use-based restrictions (paragraph 5 and Attachment A), include them in any license under which you
  distribute this model or its derivatives, and give recipients a copy of the agreement.
- **Apache License 2.0** — [`LICENSE-APACHE-2.0`](LICENSE-APACHE-2.0), for the Qwen2.5-Coder weights this model was
  initialized from.
- Attribution details: [`NOTICE.md`](NOTICE.md).

## Acknowledgements

[DeepSeek Coder](https://github.com/deepseek-ai/DeepSeek-Coder) (DeepSeek-AI) · [Qwen2.5-Coder](https://github.com/QwenLM/Qwen2.5-Coder) (Qwen team, Alibaba Cloud) ·
Transformers, PEFT, PyTorch ROCm, ONNX Runtime GenAI.

---

## Resumo em português

Modelo compacto (0.5B) especializado em **MongoDB** para o Slop Studio: autocomplete FIM (`mongosh`, DSL do Console, pipelines
de agregação em JSON/Extended JSON, Atlas Search, índices) e pedidos curtos em PT-BR/EN para reescrever o código do editor.

- **Origem**: pesos iniciais do `Qwen/Qwen2.5-Coder-0.5B` (Apache-2.0), destilado do professor **SlopCoder-Mongo-6.7B-v1**, que é um
  ajuste fino do **`deepseek-ai/deepseek-coder-6.7b-base`**. Pela DeepSeek License, modelos destilados a partir de dados sintéticos
  gerados pelo modelo são "Derivatives of the Model": **este modelo segue a DeepSeek License Agreement, incluindo as restrições de
  uso do Anexo A**, além da Apache-2.0 do Qwen.
- **Dados**: 100% sintéticos (esquemas fictícios, catálogo MongoDB, documentação do Console); sem dados de clientes.
- **Resultados**: APT 3,57 (base: 1,46), sintaxe válida 90,7%, MongoDB válido 90,6%, intenção correta 93,6% no benchmark de
  2.000 exemplos; em pedidos livres escritos à mão, 60% (72/120).
- **Limitações**: erros semânticos em pedidos livres, dados apenas sintéticos; revise sempre o código antes de executar.
- Versões ONNX para CPU e GPU (DirectML): `esilva/SlopCoder-Mongo-0.5B-ONNX`.