How to use from the
Use from the
Transformers library
# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="esilva/KapiCoder-Mongo-Nano")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)
# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("esilva/KapiCoder-Mongo-Nano")
model = AutoModelForCausalLM.from_pretrained("esilva/KapiCoder-Mongo-Nano", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))
Quick Links

KapiCoder-Mongo-Nano (0.5B)

Status: PREVIEW / experimental, public preview. Published for inspection and feedback, not for production use. Internal evaluation on synthetic data; no blind human-written set. Read Limitations and risks before using it.

KapiCoder-Mongo-Nano is a small code-completion model (about 494 M parameters) fine-tuned from Qwen/Qwen2.5-Coder-0.5B to complete MongoDB queries at the cursor (fill-in-the-middle, FIM). It was built for the Kapibara Studio editor, which sends it the text before and after the cursor plus a short context header (the editor-context-v1 contract described below). This is version v3 of the weights (repository tag v0.1.0-preview).

Read this first

  • Internal evaluation, synthetic data, no human-written queries. Every number in this card comes from the authors' own evaluation on synthetic data (a validation split of the training generator and a homologation set written by an LLM and a grammar); nothing was measured on queries written by people.
  • Invented field names on unseen schemas. With the schema visible (full), 10.81% of this model's suggestions on the homologation set use a field that does not exist, against 7.02% for the zero-shot base and 15.35% for the previous internal checkpoint (not released): still above the base. Validate field names before running a suggestion.
  • The homologation set is not fully independent of the v3 training data. The v3 corpus contains queries from the same LLM generator (Qwen3-14B) and grammar templates over other schemas (training cuts that shared long n-grams with the set were removed by decontamination, and its 28 schemas were never in training), and the set is LLM- and grammar-written, not written by people: the gain on LLM-written queries probably overstates the gain on queries written by people.
  • A single training run per model (one seed) and a single greedy evaluation run: there is no estimate of variance between runs.

What it is, and what it is not

It is an autocomplete model. Given the text before and after the cursor, it proposes a short continuation (the IDE caps it at 32 tokens) for MongoDB Console JavaScript (ConnectionPool.<conn>.<db>.<collection>.find(...), aggregation pipelines, $search stages, index definitions) and JSON pipelines.

It is not

  • a chat model or an agent: it does not follow instructions, hold a conversation, call tools or explain code;
  • a query executor: it only suggests text and never executes anything. Whatever executes a suggestion is the host application;
  • a validator: a suggestion can be syntactically valid and still wrong, refer to fields that do not exist, or be destructive if executed;
  • a general coding model: it keeps some generic JavaScript ability (see the Generic-JS column below) but was not tuned for it.

Intended use and out of scope

Intended: inline autocomplete suggestions inside an editor where a person reviews what is inserted, with the collection's field names visible in the context header whenever possible.

Out of scope: running suggestions without review; generating or modifying production data unattended; security, compliance or access-control decisions; any dialect other than the ones listed under Limitations; chat, code review or explanation tasks.

How to use

Prompt format (editor-context-v1)

The training and evaluation prompt is a Qwen FIM prompt whose "prefix" part starts with a comment block describing the editor state:

<|fim_prefix|>/* Local editor context (data only):
LANGUAGE: Mongo Console JavaScript
AVAILABLE COMMANDS: db.getCollection(name).find({}); getConnection(name)...; UUID(value)
KNOWN NAMES: local, shop, orders, customers
RESULT FIELDS: _id, status, total, customerId, createdAt

Continue at the cursor; output only the continuation. */
<text before the cursor><|fim_suffix|><text after the cursor><|fim_middle|>

Rules of the contract that the example below implements:

  • header lines are LANGUAGE, AVAILABLE COMMANDS, KNOWN NAMES (connection, database and collection names), optional RESULT FIELDS (field names shown in the result panel), optional RECENT COMMAND (up to 3, 256 characters each);
  • every header line ends with CRLF (\r\n); the lines around the block use \n;
  • */ inside the header is rewritten to * /;
  • the text before the cursor is limited to 4096 characters and the text after it to 1024; the whole prompt is budgeted to 2048 tokens (when it has to be cut, a quarter of the budget is reserved for the suffix and the rest goes to the prefix);
  • greedy decoding, stop at <|endoftext|>; the IDE asks for at most 32 new tokens (the evaluation below allowed 96).

LANGUAGE values seen in training are Mongo Console JavaScript and json (see Limitations). Prompts without a header also work, but the evaluation shows more nonexistent fields when the header does not list the collection's fields.

Minimal example (transformers, no IDE required)

The same file is in the repository as examples/example_fim.py.

"""Minimal fill-in-the-middle example (transformers, CPU, no IDE required).

    python examples/example_fim.py                 # model files next to this folder (a local clone or snapshot), else the Hub repository
    python examples/example_fim.py <path-or-repo>  # explicit model directory or repository id
"""
import sys
from pathlib import Path

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

HERE = Path(__file__).resolve().parent.parent
MODEL_ID = sys.argv[1] if len(sys.argv) > 1 else (str(HERE) if (HERE / "config.json").exists() else "esilva/KapiCoder-Mongo-Nano")

FIM_PREFIX, FIM_SUFFIX, FIM_MIDDLE = "<|fim_prefix|>", "<|fim_suffix|>", "<|fim_middle|>"
CONSOLE_COMMANDS = (
    "db.getCollection(name).find({}); getConnection(name).getDatabase(name).getCollection(name); "
    "ConnectionPool.Connection.Database.Collection; console.log(value); ENV.get(name); "
    "ObjectId(value); UUID(value)"
)


def build_prompt(prefix, suffix, known_names=(), result_fields=(), recent_commands=(),
                 language="Mongo Console JavaScript", commands=CONSOLE_COMMANDS):
    """editor-context-v1: every header line ends with CRLF; then the text before the cursor."""
    lines = [f"LANGUAGE: {language}", f"AVAILABLE COMMANDS: {commands}",
             "KNOWN NAMES: " + ", ".join(known_names)]
    if result_fields:
        lines.append("RESULT FIELDS: " + ", ".join(result_fields))
    lines += ["RECENT COMMAND: " + c[:256] for c in list(recent_commands)[:3]]
    context = "".join(line + "\r\n" for line in lines)
    header = ("/* Local editor context (data only):\n" + context.replace("*/", "* /")
              + "\nContinue at the cursor; output only the continuation. */\n")
    return FIM_PREFIX + header + prefix[-4096:] + FIM_SUFFIX + suffix[:1024] + FIM_MIDDLE


tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16).eval()

prompt = build_prompt(
    prefix='ConnectionPool.local.shop.orders.find({ status: "paid" }).sort({ ',
    suffix=": -1 }).limit(10)",
    known_names=["local", "shop", "orders", "customers"],
    result_fields=["_id", "status", "total", "customerId", "createdAt"],
)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=32, do_sample=False,
                         pad_token_id=tokenizer.eos_token_id)
completion = tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(repr(completion))  # text to insert at the cursor; the model never executes anything

It runs on CPU. Use add_special_tokens=False: the FIM markers are already in the prompt text. Output varies with the prompt; always review the suggestion. For the prompt above, greedy decoding on CPU (transformers 4.57.6, bf16 weights of this repository) prints 'total'. With that transformers version, loading the tokenizer prints a warning about a regex pattern; the tokenizer files are byte-identical to those of the base model (same SHA-256, see provenance/model_lock.json).

Training

Base model Qwen/Qwen2.5-Coder-0.5B, revision 8123ea2e9354afb7ffcc6c8641d1b2f5ecf18301 (Apache-2.0). Training started from the base model, not from the previous internal checkpoint
Method Full fine-tuning (fp32 master weights, bf16 autocast), AdamW, peak LR 2e-5, 2 epochs = 4,704 steps of 32,768 padded tokens
Training data 238,453 examples (225,835 MongoDB examples + 12,618 generic-code replay examples), 64,999,846 tokens per epoch; 129,999,692 tokens seen in total
Sources of the MongoDB examples 58.0% distilled corpus of the previous version (130,984 examples; targets written by a 7B teacher, KapiTeacher-Mongo-7B, a QLoRA fine-tune of Qwen2.5-Coder-7B, and kept only when they pass an execution check against a real MongoDB with read-only commands; raw-FIM cuts keep the original reference); 26.0% queries written by Qwen3-14B (58,808 examples; reference label, validated by executing the real text); 16.0% queries from a grammar over 61 synthetic v3 training schemas (36,043 examples; reference label). LLM + grammar = 42.0% of the MongoDB cuts
Context kinds 39.1% of the MongoDB cuts have no schema or an unrelated schema in the header (none + irrelevant), with conservative targets that use only names visible in the text
Replay 12.0% of the tokens are permissive JavaScript/TypeScript from npm packages (MIT, Apache-2.0, BSD, ISC), used to limit forgetting of generic code
Data origin Synthetic: fictitious schemas (110 schemas in 14 domains from the previous version plus 61 synthetic v3 training schemas) and queries generated from templates, a grammar and an LLM. No personal data by construction. No human review of the training samples was done
Decontamination The v3 corpus is the first training run decontaminated against the sealed homologation set (n-gram check; the previous checkpoint was trained without it, so its homologation numbers may be slightly optimistic)
Hardware One AMD Radeon RX 7800 XT (16 GB), PyTorch ROCm, transformers 4.57.6
Training time 4.38 hours of training time
Runs A single training run (seed 20261018); no repetition with other seeds
Output Full model in bf16, safetensors

The hyper-parameters are those of the previous run; the v3 change is the corpus. The previous internal checkpoint (v1, not released) was trained on 159,865 examples of the distilled corpus only and appears below as a comparison row. The training set is not published.

Evaluation

Internal evaluation on synthetic data; no blind human-written set. The first two tables below come from one internal evaluation: 2,000 examples of the validation split of the same synthetic dataset (stratified by cut kind and domain, 12 domains, seed 20261008, schemas disjoint from training), prompt in the editor-context-v1 format with a 2048-token context, greedy decoding, up to 96 new tokens, bf16 on the RX 7800 XT. 600 examples of this validation split were also used to monitor the validation loss during training. They describe behaviour on synthetic, template-generated queries and are not an estimate of quality on real queries.

Metric definitions:

  • FIM accuracy: the generated middle equals the reference after whitespace normalization.
  • Exec-valid: the completion, put back in the query, parses and runs read-only against a local MongoDB (2 s limit). It means "runs", not "answers the intended question".
  • Ceiling: only 0.6255 of the reference completions are accepted by this validator (its Console parser covers a subset of the dialect), so Exec รท ceiling shows how close the model is to what the validator can accept.
  • Nonexistent fields: share of suggestions that use a field name absent from the collection schema. full = the header lists the collection's names and result fields (n=1647); irrelevant = the header describes another collection (n=198); none = no header (n=155).
  • Generic-JS: FIM accuracy on 500 cuts of permissive npm JavaScript from packages disjoint from the replay training packages; a regression check, not a goal.
Model FIM acc Exec-valid Exec รท ceiling Syntax-valid Generic-JS FIM acc (n=500)
Qwen2.5-Coder-0.5B, zero-shot base 0.076 0.213 0.3405 0.498 0.340
Previous internal checkpoint, v1 (not released) 0.403 0.582 0.9305 0.995 0.518
KapiCoder-Mongo-Nano (this model, v3) 0.405 0.5945 0.9504 0.9955 0.530

Internal evaluation, synthetic data, n=2,000 (Generic-JS: n=500). Validator ceiling for exec-valid: 0.6255.

Nonexistent fields (%) by context_kind full (n=1647) irrelevant (n=198) none (n=155)
Qwen2.5-Coder-0.5B, zero-shot base 4.13 8.59 7.74
Previous internal checkpoint, v1 (not released) 2.06 11.11 12.90
KapiCoder-Mongo-Nano (this model, v3) 1.34 8.59 9.03

Reading the table:

  • Relative to the zero-shot base model, FIM accuracy, exec-valid and syntax-valid rise strongly on this synthetic set. Between the previous internal checkpoint and this model the validation numbers are close (see the table); the v3 corpus was built to change the behaviour on unseen schemas, which the next section measures.
  • The base model produces mostly unusable output on this task (exec-valid 0.213 overall), so rates of nonexistent fields are not directly comparable with it.
  • On this validation set the nonexistent-field rate is higher without a visible schema (irrelevant, none) than with it; the suggestion quality depends on the host sending field names.
  • The 7B teacher scored 0.41 FIM accuracy on the same validation set in the previous version's evaluation; this model is a student of that data, not a replacement for it.

Out-of-distribution validation (homologation v1)

Internal evaluation, synthetic out-of-distribution homologation. The queries were not written by people. The set has 6,030 fill-in-the-middle cuts from 28 schemas (telecom and adtech) that were never used in training, sealed as one set (SHA-256 starting with aca0aae5). The queries were written by Qwen3-14B and by a grammar, and Qwen is the same model family as the base models, so the set measures unseen schemas and unseen phrasing, not real use. The set is not fully independent of the v3 training data: the v3 corpus includes queries from the same LLM generator and grammar templates over other schemas (training cuts that shared long n-grams with the set were removed from the corpus by decontamination, and the 28 schemas were never in training, but the style was not removed), so the gain on LLM-written queries probably overstates the gain on queries written by people. No evaluation on human-written queries has been run: quality on real queries is still unmeasured. Protocol: the same editor-context-v1 prompt, greedy decoding, up to 96 new tokens, 2048-token context, bf16 on the RX 7800 XT; exec-valid runs the completed text in mongosh behind a read-only proxy (no write command was forwarded). All models were run on the same 6,030 items. Intervals: Wilson for FIM accuracy, an item bootstrap for exec รท ceiling; paired comparisons use McNemar and a paired bootstrap with Holm adjustment (95% CI).

Model FIM acc (%) [95% CI] Exec-valid (%) Exec รท ceiling (%) [95% CI] Syntax-valid (%)
Qwen2.5-Coder-0.5B, zero-shot base 12.62 [11.81, 13.48] 36.80 39.54 [38.24, 40.81] 56.09
Previous internal checkpoint, v1 (not released) 33.10 [31.92, 34.30] 69.27 74.43 [73.25, 75.57] 95.59
KapiCoder-Mongo-Nano (this model, v3) 42.34 [41.10, 43.59] 80.20 86.17 [85.26, 86.99] 99.27

Internal evaluation, synthetic out-of-distribution homologation, n=6,030. Validator ceiling for exec-valid: 93.07%.

Nonexistent fields (%) by context_kind, homologation full (n=3,720) irrelevant (n=1,155) none (n=1,155)
Qwen2.5-Coder-0.5B, zero-shot base 7.02 8.40 10.91
Previous internal checkpoint, v1 (not released) 15.35 14.55 14.72
KapiCoder-Mongo-Nano (this model, v3) 10.81 9.09 9.18
Model FIM acc (%): validation โ†’ homologation (ฮ” pp) Exec รท ceiling (%): validation โ†’ homologation (ฮ” pp) Nonexistent fields full (%): validation โ†’ homologation (ฮ” pp)
Qwen2.5-Coder-0.5B, zero-shot base 7.60 โ†’ 12.62 (+5.02) 34.05 โ†’ 39.54 (+5.49) 4.13 โ†’ 7.02 (+2.89)
Previous internal checkpoint, v1 (not released) 40.30 โ†’ 33.10 (-7.20) 93.05 โ†’ 74.43 (-18.62) 2.06 โ†’ 15.35 (+13.29)
KapiCoder-Mongo-Nano (this model, v3) 40.50 โ†’ 42.34 (+1.84) 95.04 โ†’ 86.17 (-8.87) 1.34 โ†’ 10.81 (+9.47)

The exec รท ceiling column compares two validators (validation: ceiling 62.55%, homologation: ceiling 93.07%); read trends, not decimals.

Reading the homologation tables:

  • Generalisation gap. From the synthetic validation set to unseen schemas, FIM accuracy goes 40.50% โ†’ 42.34% (+1.84 pp) and exec รท ceiling 95.04% โ†’ 86.17% (-8.87 pp). For the previous internal checkpoint (v1) the same step was 40.30% โ†’ 33.10% in FIM accuracy and 93.05% โ†’ 74.43% in exec รท ceiling. FIM accuracy is higher here than on the validation set, which is not evidence of better generalisation: see the next sections: the homologation is not fully independent of the v3 training data. The zero-shot base model goes 7.60% โ†’ 12.62%, so the homologation queries are easier for a model that was not trained here (interpretation, not tested).
  • Field names are the main risk. Do not trust field names unless the schema is in the context, and validate a suggestion before running it. On schemas never seen in training this model still invents field names at a rate above its base (paired difference in items without a nonexistent field, full: -3.8 pp [-5.0, -2.6], Holm p<0.001; negative means more invented fields than the base): 10.81% of suggestions with the schema visible (full) against 7.02% for the zero-shot base and 15.35% for the previous internal checkpoint (on the synthetic validation set it was 1.34%). Without the schema the rates are 9.09% (irrelevant) and 9.18% (none), against 8.40% and 10.91% for the base. Having the schema in the header did not lower the rate here (10.81% with it, 9.18% without); why is a hypothesis that was not tested. The rate is counted only on suggestions whose command reached the server, and the base model more often fails to parse (syntax-valid 56.09% against 99.27%), so the base rates cover fewer suggestions and are not directly comparable.
  • Against the base and the previous checkpoint (same 6,030 items, paired): over the zero-shot base, FIM accuracy +29.7 pp [+28.5, +31.0] and exec-valid +43.4 pp [+42.0, +44.8]; over the previous internal checkpoint, FIM accuracy +9.2 pp [+8.3, +10.1], exec-valid +10.9 pp [+9.9, +11.9] and items without a nonexistent field in full +4.5 pp [+3.4, +5.7] (95% CIs, Holm-adjusted McNemar tests). The previous checkpoint is not released.
  • grammar against llm. The gain over the zero-shot base in FIM accuracy is +27.9 pp [+26.4, +29.5] on grammar and +31.8 pp [+29.9, +33.6] on llm; exec รท ceiling is 88.72% on grammar and 83.63% on llm (previous checkpoint: 82.80% and 66.09%). The gain of the v3 model over the previous checkpoint in FIM accuracy is +1.7 pp [+0.8, +2.7] on grammar and +17.8 pp [+16.3, +19.4] on llm. Both parts of the set resemble what the v3 corpus was built from (queries from the same LLM and grammar templates over other schemas), so these are not independent tests of style; the gain on LLM-written queries probably overstates the gain on queries written by people.
  • Long cuts. FIM accuracy on stage and multiline cuts is 8.06% and 11.04% (exact match against a long target within 96 new tokens); do not read the overall number as evenly spread over cut kinds.

Limits of this evaluation: the authors are in the same family as the models (see above); the homologation is not fully independent of the v3 training data (see above); exec-valid means "runs", not "does what was intended"; the validator ceiling is 93.07% because 418 items refer to helpers (ConnectionPool, getConnection) that mongosh does not define; the smallest cut_kind has n=269 and the confidence intervals of the group tables are not shown; there is no multiplicity correction between groups; it is a single greedy run, with no variance across training runs.

Limitations and risks

  • Synthetic data only. Schemas are fictitious; the evaluation split uses unseen schemas of the same generator and templates. Expect lower scores on real collections, naming styles and query shapes.
  • Invented field names. It can propose fields, collections or operators that do not exist, and on unseen schemas the rate is not below the zero-shot base's in every setting (homologation section above, rates by context_kind). Validate field names before running a suggestion; do not rely on them unless the schema is in the context.
  • The homologation numbers are not an estimate for real use: the set is LLM- and grammar-written, not human-written, and is not fully independent of the v3 training data.
  • A single training run per model (one seed): differences between this model and the previous checkpoint are not backed by repeated runs.
  • Dialects. Trained on Mongo Console JavaScript (ConnectionPool... style) and JSON pipelines only. mongosh style prompts, other languages and chat are untested.
  • No safety guarantee. Behaviour around destructive operations (for example deletes or drops) was neither trained nor tested; during evaluation only read commands were executed and a guard blocked writes (0 write executions). The host application must keep a human in the loop and must not run suggestions blindly.
  • Exec-valid is not correctness. A validator-accepted completion may still not do what the user wanted; the validator itself accepts only 0.6255 of the references on the validation set.
  • Autocomplete scope. Short continuations (the IDE caps them at 32 tokens); long stage or multi-line targets are rarely matched exactly.
  • Not evaluated: bias or toxicity, long-context behaviour beyond 2048 tokens, multi-turn use, adversarial prompts, latency of the safetensors weights.

Formats

Format Where Notes
safetensors, bf16 repository root Reference weights, loaded with transformers.

This repository contains only the safetensors weights, the tokenizer and metadata (provenance/, evaluation/, examples/). ONNX builds for the editor are not part of it; they are planned as a separate repository (esilva/KapiCoder-Mongo-Nano-ONNX) and are not published yet. No GGUF is provided. Neither the teacher model nor the training data are part of this repository.

License, modification notice and attribution

  • License: Apache License 2.0 (see LICENSE).
  • Modification notice: this model is a modified version of Qwen/Qwen2.5-Coder-0.5B (revision above). Changes: full fine-tuning (SFT) for MongoDB fill-in-the-middle autocomplete, with part of the labels distilled from a 7B teacher; export to safetensors bf16. It is not a product of Alibaba Cloud or the Qwen team and is not endorsed by them. See NOTICE.md.
  • Attribution: base model Qwen2.5-Coder by the Qwen team (Hui et al., Qwen2.5-Coder Technical Report, arXiv:2409.12186), Apache-2.0. The teacher was derived from Qwen2.5-Coder-7B (Apache-2.0); its weights are not part of this repository. Part of the training queries was written by Qwen3-14B (Apache-2.0).
  • The generic-code replay used npm packages under MIT, Apache-2.0, BSD-2/3-Clause and ISC licenses. The code itself is not redistributed; the package names, versions and licenses are listed in REPLAY_ATTRIBUTION.md.
  • The editor-context-v1 prompt format is documented above so that others can build prompts for this model. Its use is not restricted by this card beyond the model license.

Related model: KapiCoder-Mongo-Small (1.5B), same task and contract.


Resumo em portuguรชs (pt-BR)

Status: PREVIEW / experimental, preview pรบblico. KapiCoder-Mongo-Nano รฉ um modelo pequeno (cerca de 494 M de parรขmetros) ajustado a partir do Qwen/Qwen2.5-Coder-0.5B para completar consultas MongoDB no cursor (preenchimento no meio, FIM) no editor Kapibara Studio. Esta รฉ a versรฃo v3 dos pesos.

  • O que รฉ: autocomplete. Sugere texto; nunca executa consultas. Nรฃo รฉ chat nem agente, e nรฃo valida o que sugere.
  • Uso: prompt FIM do Qwen (<|fim_prefix|>, <|fim_suffix|>, <|fim_middle|>) com o cabeรงalho de contexto editor-context-v1 (veja o exemplo em inglรชs acima, que roda sem a IDE, tambรฉm em examples/example_fim.py).
  • Treino: Ajuste completo (full fine-tuning), 2 รฉpocas. 238.453 exemplos sintรฉticos (225.835 de MongoDB e 12.618 de replay de cรณdigo JS/TS permissivo, 12,0% dos tokens), 64.999.846 tokens por รฉpoca, 129.999.692 tokens vistos, 4.704 passos, 4,38 horas em uma RX 7800 XT. Os rรณtulos vรชm em parte de um professor de 7B (mantidos sรณ quando passam por validaรงรฃo de execuรงรฃo somente leitura), em parte de consultas escritas pelo Qwen3-14B e de uma gramรกtica (rรณtulo de referรชncia). Sem dados pessoais por construรงรฃo; nรฃo houve revisรฃo humana das amostras. Uma รบnica execuรงรฃo de treino (uma semente).
  • Avaliaรงรฃo: avaliaรงรฃo interna sobre dados sintรฉticos; sem conjunto cego humano. As tabelas acima trazem os nรบmeros reais (FIM acc, exec-valid, exec รท teto, campos inexistentes por context_kind, Generic-JS, base ร— modelo ร— checkpoint interno anterior nรฃo publicado). A homologaรงรฃo sintรฉtica fora da distribuiรงรฃo estรก no item seguinte; nada foi medido com consultas escritas por humanos.
  • Homologaรงรฃo (avaliaรงรฃo interna, homologaรงรฃo sintรฉtica fora da distribuiรงรฃo): 6.030 cortes de 28 schemas de telecom e adtech nunca vistos no treino, escritos pelo Qwen3-14B e por gramรกtica, nรฃo por humanos (mesma famรญlia Qwen dos modelos). A homologaรงรฃo nรฃo รฉ totalmente independente do treino v3 (o corpus inclui consultas do mesmo gerador LLM e templates de gramรกtica de outros schemas), entรฃo o ganho em consultas escritas pelo LLM provavelmente superestima o ganho em consultas humanas. FIM acc 42,34% (validaรงรฃo sintรฉtica: 40,50%; base: 12,62%; checkpoint interno anterior, nรฃo publicado: 33,10%); exec รท teto 86,17% (validaรงรฃo: 95,04%, com validadores diferentes). Alerta: com o schema visรญvel (full), 10,81% das sugestรตes usam campos inexistentes, contra 7,02% da base zero-shot (validaรงรฃo sintรฉtica: 1,34%); sem schema, 9,09% (irrelevant) e 9,18% (none). Valide os nomes de campo antes de executar e nรฃo confie neles sem o schema no contexto. Executar nรฃo รฉ acertar a intenรงรฃo.
  • Limites: sรณ dado sintรฉtico; em schemas novos hรก campos inexistentes mesmo com o schema visรญvel (veja a homologaรงรฃo), e no conjunto de validaรงรฃo a taxa รฉ maior sem schema visรญvel; sรณ dialetos Console e JSON; sem garantia de seguranรงa. Valide os nomes de campo antes de executar.
  • Formatos: apenas safetensors bf16 neste repositรณrio; o ONNX para a IDE serรก um repositรณrio separado, ainda nรฃo publicado.
  • Licenรงa: Apache-2.0, com aviso de modificaรงรฃo: modelo derivado e modificado do Qwen2.5-Coder-0.5B (equipe Qwen, Alibaba Cloud), sem endosso deles. O replay usa pacotes npm MIT/Apache-2.0/BSD/ISC, listados em REPLAY_ATTRIBUTION.md.
Downloads last month
207
Safetensors
Model size
0.5B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for esilva/KapiCoder-Mongo-Nano

Finetuned
(47)
this model
Quantizations
2 models

Paper for esilva/KapiCoder-Mongo-Nano

Evaluation results

  • FIM accuracy (normalized exact match) on KapiCoder v2c validation split, internal and synthetic (n=2000)
    self-reported
    0.405
  • Exec-valid (completion parses and runs read-only) on KapiCoder v2c validation split, internal and synthetic (n=2000)
    self-reported
    0.595
  • Exec-valid divided by validator ceiling (0.6255) on KapiCoder v2c validation split, internal and synthetic (n=2000)
    self-reported
    0.950
  • Syntax-valid on KapiCoder v2c validation split, internal and synthetic (n=2000)
    self-reported
    0.996
  • Nonexistent fields, schema visible (context_kind=full), percent on KapiCoder v2c validation split, internal and synthetic (n=2000)
    self-reported
    1.340
  • FIM accuracy (normalized exact match) on Permissive npm JavaScript FIM cuts, internal (n=500)
    self-reported
    0.530
  • FIM accuracy (normalized exact match), homologation on Internal evaluation, synthetic out-of-distribution homologation v1 (n=6030, LLM-written, 28 unseen telecom/adtech schemas; not fully independent of the v3 training data)
    self-reported
    0.423
  • Exec-valid (completion parses and runs read-only), homologation on Internal evaluation, synthetic out-of-distribution homologation v1 (n=6030, LLM-written, 28 unseen telecom/adtech schemas; not fully independent of the v3 training data)
    self-reported
    0.802