qsql-qwen35-9b

A Qwen3.5-9B fine-tune that writes q/kdb+ (qSQL) from a natural-language instruction plus the schemas of the tables involved.

Model details

Base model	Qwen/Qwen3.5-9B (Apache-2.0)
Architecture	Qwen3_5ForCausalLM, model_type: qwen3_5_text
Parameters	9B, hidden_size 4096, 32 layers
Attention	hybrid — 24 linear_attention + 8 full_attention layers; every 4th layer is full attention (full_attention_interval: 4)
Weights dtype	bfloat16
Context	max_position_embeddings 262144 (see Serving for what we actually serve)
Vocab	vocab_size 248320, tokenizer length 248259 — see Known quirks
Fine-tuning	LoRA (r=64, α=128, dropout 0.05) over 3 epochs, merged into the base weights and published as a single dense model
Modality	text only. The base model's multimodal tower is not present in these weights

Intended use

Drafting qSQL for a caller that verifies the result. Concretely: you supply an instruction and the schemas of the tables it refers to, the model returns q code, and you check it — parse it, run it, and compare it against a reference computed some other way.

Out of scope

• Unverified generation. Do not put its output straight into a production query path. See Limitations.
• Tool / function calling. The fine-tuning data contained none. Passing tools=[...] to apply_chat_template renders identically to omitting it — the argument is silently ignored, not honoured.
• Thinking / reasoning traces. No thinking mode; there is no reasoning block in the chat template and none in the training data.
• Anything other than q. It is a narrow specialist.

Prompt contract

The model was trained on one prompt shape, and it is sensitive to it. Use a system turn plus a user turn built like this:

System turn — verbatim:

You are an expert q/kdb+ programmer.

User turn — the instruction, then a blank line, then a Table schemas: block with two-space-indented name: coldefs lines, then optionally a blank line and one Expected result schema: line:

For each symbol, compute the volume-weighted average price.

Table schemas:
  trade: sym:symbol; time:timestamp; price:float; size:long

Expected result schema: sym:symbol; vwap:float

Table schemas are effectively required: the model was trained with them in context, and quality falls off without them. The expected result schema is optional and acts as a shape hint.

Rendered through the bundled chat template, that becomes:

<|im_start|>system
You are an expert q/kdb+ programmer.<|im_end|>
<|im_start|>user
For each symbol, compute the volume-weighted average price.

Table schemas:
  trade: sym:symbol; time:timestamp; price:float; size:long

Expected result schema: sym:symbol; vwap:float<|im_end|>
<|im_start|>assistant

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "KxSystems/qsql-qwen35-9b"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, dtype="bfloat16", device_map="auto")

user = (
    "For each symbol, compute the volume-weighted average price.\n"
    "\nTable schemas:\n"
    "  trade: sym:symbol; time:timestamp; price:float; size:long\n"
    "\nExpected result schema: sym:symbol; vwap:float"
)
prompt = tok.apply_chat_template(
    [{"role": "system", "content": "You are an expert q/kdb+ programmer."},
     {"role": "user", "content": user}],
    tokenize=False, add_generation_prompt=True)

out = model.generate(**tok(prompt, return_tensors="pt").to(model.device),
                     max_new_tokens=512, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

Generation parameters

generation_config.json deliberately carries no sampling defaults — only eos_token_id and use_cache — so the decoding strategy is entirely the caller's choice. Two that we use:

• Greedy (do_sample=False) for a single deterministic draft.
• Temperature 0.8, several independent samples when the caller intends to pick between them — e.g. drawing n candidates, discarding the ones that fail to parse or to execute, and keeping the one the survivors agree on. This is what the sampling temperature is for; it is not a quality knob on a single draft.

eos_token is <|im_end|>; pad_token is <|endoftext|>.

Serving

Requires vLLM >= 0.27.0 — the first release with native support for text-only Qwen3_5ForCausalLM. Earlier versions need source patches and are not supported here.

vllm serve KxSystems/qsql-qwen35-9b \
    --served-model-name qsql-qwen35-9b \
    --quantization fp8 \
    --max-model-len 32768 --gpu-memory-utilization 0.92

--max-model-len 32768 is a deliberate reduction from the architectural 262144: the prompt shape above is short, and the KV cache is what a single card runs out of first. Raise it if you have the memory and need it.

Hardware requirements

One 24 GB GPU. Measured on a single NVIDIA L4 (24 GB) with vLLM 0.27.1:

Load	Flags	Weights in VRAM	KV cache
fp8 (recommended)	the command above	10.3 GiB	8.4 GiB (~257k tokens)
bfloat16	drop --quantization fp8; use --gpu-memory-utilization 0.98 --max-num-seqs 16	16.7 GiB	1.3 GiB (~40k tokens)

The download is 16.7 GiB (four bfloat16 safetensors shards). fp8 is applied by vLLM at load time from those same files, so there is no separate fp8 download.

To serve in bfloat16 on a single 24 GB card:

vllm serve KxSystems/qsql-qwen35-9b \
    --served-model-name qsql-qwen35-9b \
    --max-model-len 32768 --gpu-memory-utilization 0.98 \
    --max-num-seqs 16

bfloat16 needs both flag changes on a 24 GB card. At the default --gpu-memory-utilization 0.92 the weights plus CUDA-graph memory leave no room for a KV cache, and startup fails. This is a hybrid linear-attention model, so each decode sequence also needs its own recurrent-state cache block; at the default --max-num-seqs 256 there are too few blocks, and the server fails during CUDA-graph capture. --max-num-seqs 16 still allows 16 parallel samples of one prompt.

Throughput

Single NVIDIA L4, vLLM 0.27.1, one client and no other load. These are the medians over 5 qSQL prompts × 3 repeats, with outputs of about 55–60 tokens per sample. Latency is for the whole request, with n samples drawn in a single request.

Request	fp8 latency	fp8 output tok/s	bf16 latency	bf16 output tok/s
1 sample, greedy	2.4 s	24	3.6 s	16
n=4, temperature 0.8	3.0 s	78	4.3 s	51
n=8, temperature 0.8	3.1 s	146	4.6 s	93
n=16, temperature 0.8	3.9 s	247	5.3 s	165

Time to first token is about 100 ms (fp8) and 145 ms (bf16). Drawing 16 samples costs about 1.5 s more than drawing one, because decoding is memory-bandwidth-bound and extra sequences share each pass over the weights.

On which weights you are actually serving: the first argument is a repo id, so vLLM resolves it against the Hub and serves whatever main points at — a later checkpoint published here is picked up silently on your next restart. The weights are cached (~/.cache/huggingface/hub), so restarting does not re-download them, but it does contact the Hub to check the revision and fails if that is unreachable. For a long-lived deployment, pin the weights with --revision <commit-sha> using a commit from this repo's history, or pass a local directory instead of the repo id; HF_HUB_OFFLINE=1 uses the cache without the revision check.

On reproducing greedy output exactly: --quantization fp8 quantizes at load time, and that path is sensitive to how the checkpoint is split across files, so greedy decoding is not guaranteed bit-identical between this sharded repo and an otherwise identical single-file copy of the same weights. Measured on 10 prompts: identical in bfloat16, and 8 of 10 identical under fp8, with both differences landing on tokens whose top two candidates were tied to three decimal places. If you need bit-reproducibility, load in bfloat16.

On fp8 and accuracy: fp8 changes the exact output but not the pass rate. Greedy on the 70-task human-audited set, fp8 served by vLLM passed 48/70 and 16-bit served by transformers passed 46/70. The two agreed on pass/fail for 64 of the 70 tasks. The completions themselves were byte-identical on only 41 of the 70. That comparison changes the inference engine as well as the precision, so it shows that the recommended fp8 setup matches the 16-bit model in accuracy. It does not isolate the effect of fp8 alone.

Chat template

The published chat_template.jinja renders under plain Jinja2, so it works outside transformers as well: GGUF conversion, llama.cpp, Ollama, TGI's Rust renderer, and the JS renderer behind this model page.

This is not automatic. A freshly trained checkpoint's template contains {% generation %} / {% endgeneration %}, which mark the assistant span for assistant-only loss. They are a transformers extension, not standard Jinja, and every renderer that does not go through transformers raises TemplateSyntaxError: Encountered unknown tag 'generation' on them. They have been removed here. The removal is provably prompt-neutral: the tags render to zero characters, and the rendered prompt is byte-identical before and after.

If you continue training from these weights, you must put the tags back, or your trainer will compute loss over an empty assistant span. Whether it tells you depends on the trainer: TRL >= 0.29 raises a RuntimeError per example, but apply_chat_template on its own returns an all-zero assistant mask with only a log line — a hand-rolled loop gets no usable warning.

Limitations

Like other large language models, this model may occasionally generate code that parses and runs but does not fully capture the intended logic. We recommend reviewing or testing generated queries before relying on them in production.

No safety tuning of any kind was applied. It is a code model for a trusted internal audience; treat generated q as untrusted input and never value it in a privileged process.

Known quirks

Two things a loader may warn about. Both are known and neither is a defect in the weights:

1. vocab_size (248320) is larger than the tokenizer (248259) — a 61-slot embedding pad from extending the tokenizer. This is the safe direction: no id the tokenizer can emit is out of range for the embedding matrix. Some tools warn about the mismatch anyway.
2. No multi-token prediction. The base config declares mtp_num_hidden_layers, but no MTP weights were trained or merged, so the key has been removed from the published config.json rather than left to advertise a layer that does not exist. Ordinary serving is identical either way; the difference shows up only if you enable speculative decoding / MTP, where the honest answer — this model has none — is better than a declaration pointing at absent weights. Do not add the key back.

Evaluation

Two internal test sets, neither of them published, so everything below is self-reported on private data — read it as such, and read Limitations for what "pass" does and does not establish. Each set answers one question: the first compares this model with general-purpose frontier models, the second measures what the self-consistency wrapper adds on top of this model.

The tasks are synthetic: finance-domain instructions over generated table schemas, each paired with a reference q implementation whose output defines the answer. They are held back deliberately, because releasing them would put them straight into the next model's training data and retire the benchmark.

What counts as a pass, on both sets: the generated q is executed against dummy tables built from the task's declared schemas, and its result is compared for execution-equivalence against that task's reference implementation run over the same tables. Code that fails to parse, raises, times out, or returns a degenerate result (a bare function, an unexpected atom) counts as a failure. Tasks the harness could not evaluate at all are counted as failures rather than dropped from the denominator. Note what this makes a "pass": agreement with a generated reference — vetted by a person on one set and by automated checks on the other — not with an independently written expert solution.

Neither set leaks into training. Of the 70 human-audited instructions and the 1,078 machine-audited ones, zero appear in this model's training data, by exact instruction match and by exact (instruction, response) pair.

1. Against general-purpose frontier models — human-audited set

70 held-out tasks. Instructions and references were LLM-generated, filtered to those whose reference executed, then selected by a human as the final gate. No task in it was skipped by the harness, so the denominator is 70 exactly.

One sample each (pass@1). All six models received the same system prompt and the same user prompt — the contract under Prompt contract, schemas and expected output shape included. This model was decoded greedily (temperature 0). The five Claude models were called one-shot through the claude CLI, which exposes no temperature control, with no tools and no verification loop.

model	pass@1	95% CI (Wilson)
Claude Haiku 5.5	31/70 — 44.3%	33.2%–55.9%
Claude Sonnet 5.5	34/70 — 48.6%	37.2%–60.0%
Claude Opus 5	34/70 — 48.6%	37.2%–60.0%
Claude Opus 5.5	37/70 — 52.9%	41.3%–64.1%
Claude Fable 5	42/70 — 60.0%	48.3%–70.7%
qsql-qwen35-9b	46/70 — 65.7%	54.0%–75.8%

All six ran on the same tasks, so the comparisons are paired and an exact McNemar test says more than the intervals above. Tasks only this model got, against tasks only the other model got:

• vs Haiku 5.5 — 20 against 5. p = 0.004: a real difference.
• vs Sonnet 5.5 — 18 against 6. p = 0.023: a real difference.
• vs Opus 5 — 19 against 7. p = 0.029: a real difference.
• vs Opus 5.5 — 16 against 7. p = 0.093: not a distinguishable difference at the conventional 0.05 level.
• vs Fable 5 — 12 against 8. p = 0.50: not a distinguishable difference.

So the honest claim is that a 9B specialist is ahead of Haiku 5.5, Sonnet 5.5 and Opus 5, and matches Opus 5.5 and Fable 5 on this task — not that it beats all five. At n = 70 a four-task gap carries no signal and a nine-task gap is suggestive at most, and we would rather say so than round it up. This comparison is also on one small held-out set; we do not claim the ranking generalises to every qSQL workload.

2. What the self-consistency wrapper adds — machine-audited set

1,078 held-out tasks. References were LLM-generated with q-interpreter access and gated by automated checks only: execution throughout, plus static lint and an LLM-judge verdict on parts of it. No human reviewed them. They are the held-out split of the same corpus this model's training data was drawn from, so this is an in-distribution measurement.

The same pass criterion, and this model in every row: the variable here is decoding strategy, not model choice. The harness evaluated every one of the 1,078 tasks, so the denominator is 1,078 exactly.

decoding	pass	rate	95% CI (Wilson)
single sample, greedy (temperature 0)	606/1078	56.2%	53.2%–59.1%
single sample, temperature 0.8	555/1078	51.5%	48.5%–54.5%
self-consistency, temperature 0.8	690/1078	64.0%	61.1%–66.8%
pass@16 ceiling — any of 16 samples passes	850/1078	78.8%	76.3%–81.2%

Self-consistency here is the pipeline implemented by the MCP server: draw candidates in growing rounds of 4 → 8 → 16 at temperature 0.8, drop any that fail to parse or that raise on execution, cluster the survivors by whether they produce identical results across three independently seeded dummy-table draws (seeds 42/43/44, 20 rows each), and return the shortest code in the largest cluster. A round finalises early when its winning cluster is unanimous at 4 or reaches weight 6 at 8, which here cut sample generation by 34% while passing the same number of tasks as running all 16.

Paired again, and both gaps are wide:

• vs the greedy single sample: +84 tasks, 105 won against 21 lost, p ≈ 1 × 10⁻¹⁴.
• vs the first sample of the same temperature-0.8 draw: +135, 162 against 27, p ≈ 1 × 10⁻²⁴.

The greedy row is deterministic: two independent runs of it produced byte-identical completions on all 1,078 tasks, so it is a fixed reference point rather than one draw among many.

Two things are worth reading off that table. Raising the temperature to get diverse candidates costs 4.7 points on its own (56.2% → 51.5%, greedy against one sample at 0.8); the clustering has to pay that back before it earns anything, and it does — the net gain over the better single-sample baseline is +7.8 points. And the wrapper closes 34% of the distance between one sample and the pass@16 ceiling: a majority vote over the model's own samples recovers a third of what a perfect oracle picker would. The ceiling itself is an upper bound, not a deployable system — it needs a perfect picker.

Two caveats on how this was measured, both of which make it an approximation of the live server rather than a recording of it. The rounds were simulated offline by re-windowing a single 16-sample draw, where the server draws fresh candidates each round; and the offline harness applies the parse and execution gates but not the server's optional KX qlint check, whose contribution is narrow but not nil.

What none of this measures is semantic correctness. Execution-equivalence against a reference is a strong signal and it is not a proof: agreement between independent samples cannot catch an error the model makes systematically on a given instruction, because then every sample makes it — and on the machine-audited set the reference itself was generated and machine-filtered rather than written or checked by hand. That limit is the subject of Limitations, and it is why this is a drafting model.

No model-index: block is published. Those entries are keyed to a named, resolvable dataset so that leaderboards can consume them, and this test set is not published — a machine-readable claim nobody can reproduce would be worse than the prose above.

Training

• Method: LoRA on Qwen/Qwen3.5-9B, merged into the base weights for release — the published files are a single dense bfloat16 model, so no PEFT adapter and no separate base download is needed at inference time.
• LoRA configuration: rank 64, α 128, dropout 0.05, task_type: CAUSAL_LM, no bias. Applied to 13 module types — the attention and MLP projections (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj), the linear-attention projections this architecture adds (in_proj_qkv, in_proj_a, in_proj_b, in_proj_z, out_proj), and lm_head.
• Schedule: 3 epochs, 849 optimizer steps, effective batch size 48.
• Data: not released. The training corpus and the pipeline that produced it are not part of this publication, and there are no plans to release them. Please do not open issues asking for them — see Support and issues for what the issue tracker is for.
• Precision: bfloat16 throughout; no quantised weights are published here.

Support and issues

Community discussions and pull requests are turned off on this Hugging Face repository, so questions asked here will not reach anyone. They go to the code repository instead:

• Issues, bugs and questions: https://github.com/KxSystems/mcp-qsql-suggester/issues. That repository holds the MCP server this model was built for, and documents what it does and does not verify about the model's output.
• General inquiries: incubation@kx.com.
• Support for licensed kdb+ subscribers: support.kx.com.
• Security reports: privately, per that repository's SECURITY.md — never as a public issue.

Pull requests are not accepted on either repository. The reason is in the code repository's CONTRIBUTING.md: most of its test suite needs licensed software, so a change to the parts most likely to break cannot be verified by public CI or by a contributor without a kdb+ licence. A well-described issue is more useful to us than a patch.

License and attribution

These weights are released under the Apache License, Version 2.0; the full text is in LICENSE.

Copyright 2026 KX Systems, Inc.

They are a modified derivative work of Qwen/Qwen3.5-9B, Copyright 2026 Alibaba Cloud, also licensed under the Apache License, Version 2.0. The modification is the fine-tune described under Training, merged into the base weights. Apache-2.0 places no copyleft obligation on a fine-tune; the attribution above is what it does require, and this section is where it is recorded.