Hugging Face's logo

KxSystems
/
qsql-qwen35-9b

Text Generation
Transformers
Safetensors
English
qwen3_5_text
q
kdb
qsql
code-generation
conversational
Model card Files
xet
qsql-qwen35-9b / README.md
lanstonchu's picture
lanstonchu
Card: clearer explanation on --max-model-len and max_tokens
c6690b0 verified
preview code
|
Raw History Blame
23.7 kB
---
license: apache-2.0
base_model: Qwen/Qwen3.5-9B
base_model_relation: finetune
library_name: transformers
pipeline_tag: text-generation
language:
- en
tags:
- q
- kdb
- qsql
- code-generation
---
<!-- The block above is REAL frontmatter and must stay unfenced and first.
It used to be wrapped in a ```yaml fence with a stray `---` above it,
which renders more tidily in this draft and is silently wrong in the
published file: a hub parses frontmatter only from a literal `---` block
at the very top of README.md, so the fenced form publishes with no
licence tag, no base_model link, and the YAML shown as a code sample.
`prepare_release.py stage --card` now fails on the fenced shape. -->
# qsql-qwen35-9b
A [Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) fine-tune that writes
**financial qSQL** for kdb+ and KDB-X from a natural-language instruction plus the
schemas of the tables involved.
## Model details
| | |
|---|---|
| Base model | `Qwen/Qwen3.5-9B` (Apache-2.0) |
| Architecture | `Qwen3_5ForCausalLM`, `model_type: qwen3_5_text` |
| Parameters | 9B, `hidden_size` 4096, 32 layers |
| Attention | hybrid β€” 24 `linear_attention` + 8 `full_attention` layers; every 4th layer is full attention (`full_attention_interval: 4`) |
| Weights dtype | `bfloat16` |
| Context | `max_position_embeddings` 262144 (see [Serving](#serving) for what we actually serve) |
| Vocab | `vocab_size` 248320, tokenizer length 248259 β€” see [Known quirks](#known-quirks) |
| Fine-tuning | LoRA (r=64, Ξ±=128, dropout 0.05) over 3 epochs, merged into the base weights and published as a single dense model |
| Modality | **text only.** The base model's multimodal tower is not present in these weights |
## Intended use
Drafting financial qSQL queries β€” across market data and execution analytics,
portfolio and risk, fixed income, FX and derivatives, and compliance and
reference-data work β€” for a caller that verifies the result. Concretely: you
supply an instruction and the schemas of the tables it refers to, the model
returns q code, and **you** check it β€” parse it, run it, and compare it against
a reference computed some other way.
### Out of scope
- **Unverified generation.** Do not put its output straight into a production
query path. See [Limitations](#limitations).
- **Tool / function calling.** The fine-tuning data contained none. Passing
`tools=[...]` to `apply_chat_template` renders identically to omitting it β€”
the argument is silently ignored, not honoured.
- **Thinking / reasoning traces.** No thinking mode; there is no reasoning
block in the chat template and none in the training data.
- **General q questions.** It writes queries; it was not trained to explain q
syntax or semantics, or to work out what an expression returns, and it can
answer such questions wrongly.
- **Use as an agent's primary model** (e.g. in Claude Code). It has never seen
an agent system prompt or tool definitions. To use it from an agent, keep a
general-purpose model as the primary and call this one as a specialist β€” the
[`mcp-qsql-suggester`](https://github.com/KxSystems/mcp-qsql-suggester) MCP
server does exactly that.
- **Anything other than q.** It is a narrow specialist.
## Prompt contract
The model was trained on one prompt shape, and it is sensitive to it. Use a
system turn plus a user turn built like this:
**System turn** β€” verbatim:
```
You are an expert q/kdb+ programmer.
```
**User turn** β€” the instruction, then a blank line, then a `Table schemas:`
block with **two-space-indented** `name: coldefs` lines, then optionally a blank
line and one `Expected result schema:` line:
```
For each symbol, compute the volume-weighted average price.
Table schemas:
trade: sym:symbol; time:timestamp; price:float; size:long
Expected result schema: sym:symbol; vwap:float
```
Table schemas are effectively **required**: the model was trained with them in
context, and quality falls off without them. The expected result schema is
optional and acts as a shape hint.
Rendered through the bundled chat template, that becomes:
```
<|im_start|>system
You are an expert q/kdb+ programmer.<|im_end|>
<|im_start|>user
For each symbol, compute the volume-weighted average price.
Table schemas:
trade: sym:symbol; time:timestamp; price:float; size:long
Expected result schema: sym:symbol; vwap:float<|im_end|>
<|im_start|>assistant
```
## How to use
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL = "KxSystems/qsql-qwen35-9b"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, dtype="bfloat16", device_map="auto")
user = (
"For each symbol, compute the volume-weighted average price.\n"
"\nTable schemas:\n"
" trade: sym:symbol; time:timestamp; price:float; size:long\n"
"\nExpected result schema: sym:symbol; vwap:float"
)
prompt = tok.apply_chat_template(
[{"role": "system", "content": "You are an expert q/kdb+ programmer."},
{"role": "user", "content": user}],
tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(prompt, return_tensors="pt").to(model.device),
max_new_tokens=512, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))
```
### Generation parameters
`generation_config.json` deliberately carries **no sampling defaults** β€” only
`eos_token_id` and `use_cache` β€” so the decoding strategy is entirely the
caller's choice. Two that we use:
- **Greedy** (`do_sample=False`) for a single deterministic draft.
- **Temperature 0.8, several independent samples** when the caller intends to
pick between them β€” e.g. drawing *n* candidates, discarding the ones that fail
to parse or to execute, and keeping the one the survivors agree on. This is
what the sampling temperature is for; it is not a quality knob on a single
draft.
`eos_token` is `<|im_end|>`; `pad_token` is `<|endoftext|>`.
**`max_tokens` 2048** is ample. Fine-tuning used a 2048-token sequence limit
for prompt and answer together, and a typical answer is one query of tens of
tokens. A large cap does not make answers better; it only lets a runaway
generation run longer before it is cut off.
## Serving
Requires **vLLM >= 0.27.0** β€” the first release with native support for
text-only `Qwen3_5ForCausalLM`. Earlier versions need source patches and are
not supported here.
```bash
vllm serve KxSystems/qsql-qwen35-9b \
--served-model-name qsql-qwen35-9b \
--quantization fp8 \
--max-model-len 32768 --gpu-memory-utilization 0.92
```
`--max-model-len 32768` is a deliberate reduction from the architectural
262144, and is already far more than this model needs: it was fine-tuned with
a 2048-token sequence limit. The headroom costs little, and it fits a
single 24 GB card with an unquantised KV cache, so there is no need for
`--kv-cache-dtype fp8`. Raising it buys nothing for the intended use β€” in
particular, a long context sized for an agent harness's system prompt does not
help, since that use is [out of scope](#out-of-scope).
### Hardware requirements
**One 24 GB GPU.** Measured on a single NVIDIA L4 (24 GB) with vLLM 0.27.1:
| Load | Flags | Weights in VRAM | KV cache |
|---|---|---|---|
| **fp8** (recommended) | the command above | 10.3 GiB | 8.4 GiB (~257k tokens) |
| bfloat16 | drop `--quantization fp8`; use `--gpu-memory-utilization 0.98 --max-num-seqs 16` | 16.7 GiB | 1.3 GiB (~40k tokens) |
The download is 16.7 GiB (four bfloat16 safetensors shards). fp8 is applied by
vLLM at load time from those same files, so there is no separate fp8 download.
To serve in bfloat16 on a single 24 GB card:
```bash
vllm serve KxSystems/qsql-qwen35-9b \
--served-model-name qsql-qwen35-9b \
--max-model-len 32768 --gpu-memory-utilization 0.98 \
--max-num-seqs 16
```
bfloat16 needs both flag changes on a 24 GB card. At the default
`--gpu-memory-utilization 0.92` the weights plus CUDA-graph memory leave no room
for a KV cache, and startup fails. This is a hybrid linear-attention model, so
each decode sequence also needs its own recurrent-state cache block; at the
default `--max-num-seqs 256` there are too few blocks, and the server fails
during CUDA-graph capture. `--max-num-seqs 16` still allows 16 parallel samples
of one prompt.
### Throughput
Single NVIDIA L4, vLLM 0.27.1, one client and no other load. These are the
medians over 5 qSQL prompts Γ— 3 repeats, with outputs of about 55–60 tokens per
sample. Latency is for the whole request, with `n` samples drawn in a single
request.
| Request | fp8 latency | fp8 output tok/s | bf16 latency | bf16 output tok/s |
|---|---|---|---|---|
| 1 sample, greedy | 2.4 s | 24 | 3.6 s | 16 |
| `n=4`, temperature 0.8 | 3.0 s | 78 | 4.3 s | 51 |
| `n=8`, temperature 0.8 | 3.1 s | 146 | 4.6 s | 93 |
| `n=16`, temperature 0.8 | 3.9 s | 247 | 5.3 s | 165 |
Time to first token is about 100 ms (fp8) and 145 ms (bf16). Drawing 16
samples costs about 1.5 s more than drawing one, because decoding is
memory-bandwidth-bound and extra sequences share each pass over the weights.
**On which weights you are actually serving**: the first argument is a repo id,
so vLLM resolves it against the Hub and serves whatever `main` points at β€” a
later checkpoint published here is picked up silently on your next restart. The
weights are cached (`~/.cache/huggingface/hub`), so restarting does not
re-download them, but it does contact the Hub to check the revision and fails if
that is unreachable. For a long-lived deployment, pin the weights with
`--revision <commit-sha>` using a commit from this repo's history, or pass a
local directory instead of the repo id; `HF_HUB_OFFLINE=1` uses the cache
without the revision check.
**On reproducing greedy output exactly**: `--quantization fp8` quantizes at load
time, and that path is sensitive to how the checkpoint is split across files, so
greedy decoding is not guaranteed bit-identical between this sharded repo and an
otherwise identical single-file copy of the same weights. Measured on 10 prompts:
identical in bfloat16, and 8 of 10 identical under fp8, with both differences
landing on tokens whose top two candidates were tied to three decimal places. If
you need bit-reproducibility, load in bfloat16.
**On fp8 and accuracy**: fp8 changes the exact output but not the pass rate.
Greedy on the 70-task
[human-audited set](#1-against-general-purpose-frontier-models--human-audited-set),
fp8 served by vLLM passed 48/70 and 16-bit served by transformers passed 46/70.
The two agreed on pass/fail for 64 of the 70 tasks. The completions themselves
were byte-identical on only 41 of the 70. That comparison changes the inference
engine as well as the precision, so it shows that the recommended fp8 setup
matches the 16-bit model in accuracy. It does not isolate the effect of fp8
alone.
## Chat template
The published `chat_template.jinja` renders under **plain Jinja2**, so it works
outside transformers as well: GGUF conversion, llama.cpp, Ollama, TGI's Rust
renderer, and the JS renderer behind this model page.
This is not automatic. A freshly trained checkpoint's template contains
`{% generation %}` / `{% endgeneration %}`, which mark the assistant span for
assistant-only loss. They are a **transformers extension, not standard Jinja**,
and every renderer that does not go through transformers raises
`TemplateSyntaxError: Encountered unknown tag 'generation'` on them. They have
been removed here. The removal is provably prompt-neutral: the tags render to
zero characters, and the rendered prompt is byte-identical before and after.
**If you continue training from these weights, you must put the tags back**, or
your trainer will compute loss over an empty assistant span. Whether it tells
you depends on the trainer: TRL >= 0.29 raises a `RuntimeError` per example, but
`apply_chat_template` on its own returns an all-zero assistant mask with only a
log line β€” a hand-rolled loop gets no usable warning.
## Limitations
Like other large language models, this model may occasionally generate code
that parses and runs but does not fully capture the intended logic. We
recommend reviewing or testing generated queries before relying on them in
production.
No safety tuning of any kind was applied. It is a code model for a trusted
internal audience; treat generated q as untrusted input and never `value` it
in a privileged process.
## Known quirks
Two things a loader may warn about. Both are known and neither is a defect in
the weights:
1. **`vocab_size` (248320) is larger than the tokenizer (248259)** β€” a 61-slot
embedding pad from extending the tokenizer. This is the safe direction: no id
the tokenizer can emit is out of range for the embedding matrix. Some tools
warn about the mismatch anyway.
2. **No multi-token prediction.** The base config declares
`mtp_num_hidden_layers`, but no MTP weights were trained or merged, so the
key has been **removed** from the published `config.json` rather than left to
advertise a layer that does not exist. Ordinary serving is identical either
way; the difference shows up only if you enable speculative decoding / MTP,
where the honest answer β€” this model has none β€” is better than a declaration
pointing at absent weights. Do not add the key back.
## Evaluation
Two internal test sets, neither of them published, so everything below is
**self-reported on private data** β€” read it as such, and read
[Limitations](#limitations) for what "pass" does and does not establish. Each set
answers one question: the first compares this model with general-purpose
frontier models, the second measures what the self-consistency wrapper adds on
top of this model.
**The tasks** are synthetic: finance-domain instructions over generated table
schemas, each paired with a reference q implementation whose output defines the
answer. They are held back deliberately, because releasing them would put them
straight into the next model's training data and retire the benchmark.
**What counts as a pass**, on both sets: the generated q is executed
against dummy tables built from the task's declared schemas, and its result is
compared for execution-equivalence against that task's reference implementation
run over the same tables. Code that fails to parse, raises, times out, or returns
a degenerate result (a bare function, an unexpected atom) counts as a failure.
Tasks the harness could not evaluate at all are counted as failures rather than
dropped from the denominator. Note what this makes a "pass": agreement with a
generated reference β€” vetted by a person on one set and by automated checks on
the other β€” not with an independently written expert solution.
**Neither set leaks into training.** Of the 70 human-audited instructions and the
1,078 machine-audited ones, zero appear in this model's training data, by exact
instruction match and by exact `(instruction, response)` pair.
### 1. Against general-purpose frontier models β€” human-audited set
70 held-out tasks. Instructions and references were LLM-generated, filtered to
those whose reference executed, then selected by a **human** as the final gate.
No task in it was skipped by the harness, so the denominator is 70 exactly.
One sample each (pass@1). All six models received the **same** system prompt
and the same user prompt β€” the contract under [Prompt contract](#prompt-contract),
schemas and expected output shape included. This model was decoded **greedily**
(temperature 0). The five Claude models were called one-shot through the `claude`
CLI, which exposes no temperature control, with no tools and no verification
loop.
| model | pass@1 | 95% CI (Wilson) |
|---|---|---|
| Claude Haiku 5.5 | 31/70 β€” 44.3% | 33.2%–55.9% |
| Claude Sonnet 5.5 | 34/70 β€” 48.6% | 37.2%–60.0% |
| Claude Opus 5 | 34/70 β€” 48.6% | 37.2%–60.0% |
| Claude Opus 5.5 | 37/70 β€” 52.9% | 41.3%–64.1% |
| Claude Fable 5 | 42/70 β€” 60.0% | 48.3%–70.7% |
| **qsql-qwen35-9b** | **46/70 β€” 65.7%** | 54.0%–75.8% |
All six ran on the same tasks, so the comparisons are paired and an exact
McNemar test says more than the intervals above. Tasks only this model got,
against tasks only the other model got:
- vs **Haiku 5.5** β€” 20 against 5. *p* = 0.004: a real difference.
- vs **Sonnet 5.5** β€” 18 against 6. *p* = 0.023: a real difference.
- vs **Opus 5** β€” 19 against 7. *p* = 0.029: a real difference.
- vs **Opus 5.5** β€” 16 against 7. *p* = 0.093: **not** a distinguishable
difference at the conventional 0.05 level.
- vs **Fable 5** β€” 12 against 8. *p* = 0.50: **not** a distinguishable
difference.
So the honest claim is that a 9B specialist **is ahead of Haiku 5.5, Sonnet 5.5
and Opus 5, and matches Opus 5.5 and Fable 5** on this task β€” not that it beats
all five. At n = 70 a four-task gap carries no signal and a nine-task gap is
suggestive at most, and we would rather say so than round it up. This comparison is also
on one small held-out set; we do not claim the ranking generalises to every qSQL
workload.
### 2. What the self-consistency wrapper adds β€” machine-audited set
1,078 held-out tasks. References were LLM-generated with q-interpreter access and
gated by automated checks only: execution throughout, plus static lint and an
LLM-judge verdict on parts of it. No human reviewed them. They are the held-out
split of the same corpus this model's training data was drawn from, so this is an
**in-distribution** measurement.
The same pass criterion, and this model in every row: the variable here is
decoding strategy, not model choice. The harness evaluated every one of the
1,078 tasks, so the denominator is 1,078 exactly.
| decoding | pass | rate | 95% CI (Wilson) |
|---|---|---|---|
| single sample, greedy (temperature 0) | 606/1078 | 56.2% | 53.2%–59.1% |
| single sample, temperature 0.8 | 555/1078 | 51.5% | 48.5%–54.5% |
| **self-consistency, temperature 0.8** | **690/1078** | **64.0%** | 61.1%–66.8% |
| *pass@16 ceiling β€” any of 16 samples passes* | *850/1078* | *78.8%* | *76.3%–81.2%* |
Self-consistency here is the pipeline implemented by the
[MCP server](https://github.com/KxSystems/mcp-qsql-suggester): draw candidates
in growing rounds of 4 β†’ 8 β†’ 16 at temperature 0.8, drop any that fail to parse
or that raise on execution, cluster the survivors by whether they produce
identical results across **three** independently seeded dummy-table draws (seeds
42/43/44, 20 rows each), and return the shortest code in the largest cluster. A
round finalises early when its winning cluster is unanimous at 4 or reaches
weight 6 at 8, which here cut sample generation by 34% while passing the same
number of tasks as running all 16.
Paired again, and both gaps are wide:
- vs the greedy single sample: +84 tasks, 105 won against 21 lost,
*p* β‰ˆ 1 Γ— 10⁻¹⁴.
- vs the first sample of the same temperature-0.8 draw: +135, 162 against 27,
*p* β‰ˆ 1 Γ— 10⁻²⁴.
The greedy row is deterministic: two independent runs of it produced
byte-identical completions on all 1,078 tasks, so it is a fixed reference point
rather than one draw among many.
Two things are worth reading off that table. Raising the temperature to get
diverse candidates **costs** 4.7 points on its own (56.2% β†’ 51.5%, greedy against
one sample at 0.8); the clustering has to pay that back before it earns anything,
and it does β€” the net gain over the better single-sample baseline is +7.8
points. And the wrapper closes 34% of the distance between one sample and the
pass@16 ceiling: a majority vote over the model's own samples recovers a third of
what a perfect oracle picker would. The ceiling itself is an upper bound, not a
deployable system β€” it needs a perfect picker.
Two caveats on how this was measured, both of which make it an approximation of
the live server rather than a recording of it. The rounds were simulated offline
by re-windowing a single 16-sample draw, where the server draws fresh candidates
each round; and the offline harness applies the parse and execution gates but not
the server's optional KX qlint check, whose contribution is narrow but not nil.
**What none of this measures is semantic correctness.** Execution-equivalence
against a reference is a strong signal and it is not a proof: agreement between
independent samples cannot catch an error the model makes *systematically* on a
given instruction, because then every sample makes it β€” and on the
machine-audited set the reference itself was generated and machine-filtered
rather than written or checked by hand. That limit is the subject of
[Limitations](#limitations), and it is why this is a drafting model.
No `model-index:` block is published. Those entries are keyed to a named,
resolvable dataset so that leaderboards can consume them, and this test set is
not published β€” a machine-readable claim nobody can reproduce would be worse than
the prose above.
## Training
- **Method**: LoRA on `Qwen/Qwen3.5-9B`, merged into the base weights for
release β€” the published files are a single dense bfloat16 model, so no PEFT
adapter and no separate base download is needed at inference time.
- **LoRA configuration**: rank 64, Ξ± 128, dropout 0.05, `task_type: CAUSAL_LM`,
no bias. Applied to 13 module types β€” the attention and MLP projections
(`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`),
the linear-attention projections this architecture adds (`in_proj_qkv`,
`in_proj_a`, `in_proj_b`, `in_proj_z`, `out_proj`), and `lm_head`.
- **Schedule**: 3 epochs, 849 optimizer steps, effective batch size 48.
- **Data**: not released. The training corpus and the pipeline that produced it
are not part of this publication, and there are no plans to release them.
Please do not open issues asking for them β€” see
[Support and issues](#support-and-issues) for what the issue tracker is for.
- **Precision**: bfloat16 throughout; no quantised weights are published here.
## Support and issues
Community discussions and pull requests are turned off on this Hugging Face
repository, so questions asked here will not reach anyone. They go to the code
repository instead:
- **Issues, bugs and questions**:
<https://github.com/KxSystems/mcp-qsql-suggester/issues>. That repository holds the MCP server
this model was built for, and documents what it does and does not verify about
the model's output.
- **General inquiries**: [`incubation@kx.com`](mailto:incubation@kx.com).
- **Support for licensed kdb+ subscribers**:
[support.kx.com](https://client.support.kx.com/).
- **Security reports**: privately, per that repository's `SECURITY.md` β€” never
as a public issue.
Pull requests are not accepted on either repository. The reason is in the code
repository's `CONTRIBUTING.md`: most of its test suite needs licensed software,
so a change to the parts most likely to break cannot be verified by public CI or
by a contributor without a kdb+ licence. A well-described issue is more useful to
us than a patch.
## License and attribution
These weights are released under the **Apache License, Version 2.0**;
the full text is in `LICENSE`.
Copyright 2026 KX Systems, Inc.
They are a **modified derivative work** of
[`Qwen/Qwen3.5-9B`](https://huggingface.co/Qwen/Qwen3.5-9B), Copyright 2026
Alibaba Cloud, also licensed under the Apache License, Version 2.0. The
modification is the fine-tune described under [Training](#training), merged into
the base weights. Apache-2.0 places no copyleft obligation on a fine-tune; the
attribution above is what it does require, and this section is where it is
recorded.