rag-gate-27b-p300

A tt-model v6 thin bundle that serves ThakiCloud/RAG-Gate-27B (revision 3f008df4d711f1aca5014d6b5e59646a0eca9c7d) on 2 Tenstorrent blackhole chips (mesh P150x2). The tt-orchard harness staged it from a run of that model.

License

The weights this bundle points to are licensed Apache 2.0 (https://www.apache.org/licenses/LICENSE-2.0). The bundle ships no weights. tt-model downloads them from ThakiCloud/RAG-Gate-27B with your own Hugging Face account.

What it runs

• Weights: ThakiCloud/RAG-Gate-27B at revision 3f008df4d711f1aca5014d6b5e59646a0eca9c7d.
• Model code: the Tenstorrent implementation of Qwen/Qwen3.8-27B, taken from the bundle qwen3.8-27b-dflash2-p300. ThakiCloud/RAG-Gate-27B has the same text architecture, so only the weights differ.
• At each start the launcher builds model-dir/ from the configuration files of Qwen/Qwen3.8-27B (shipped in base_config/) and the tokenizer and weights of ThakiCloud/RAG-Gate-27B. It sets MODEL_WEIGHTS_DIR and HF_MODEL to that directory, so the chips load the weights of ThakiCloud/RAG-Gate-27B.
• Context length 262144 tokens; up to 4 sequences at a time.
• Speculative decoding is off. The drafter of Qwen/Qwen3.8-27B needs the MTP tensors of the model, and ThakiCloud/RAG-Gate-27B has none.
• Sampling runs on the host, because on-device sampling is not used with the drafter off on this mesh.

Intended use

RAG-Gate-27B sits after retrieval and before generation in a retrieval-augmented pipeline. It reads a question, the retrieved passages and whether more retrieval is possible, and picks one action: Answer (the evidence holds a complete support chain), Retrieve (it does not, and you can search again) or Stop (it does not, and you cannot). ThakiCloud trained it as a LoRA fine-tune of Qwen/Qwen3.8-27B, merged into bf16 weights. This bundle serves those weights on Tenstorrent hardware. The model card at https://huggingface.co/ThakiCloud/RAG-Gate-27B holds the author's training data, results and limits.

How to use it as a gate

Build the prompt as the author's card gives it (the question, the numbered passages, Retrieval available: YES or NO, the policy and the action list), apply the chat template with thinking turned off, and end the prompt with Final action:. The action is the first token generated.

curl -s http://<host>:<port>/v1/completions -H 'Content-Type: application/json' -d '{
  "model": "<the model name the server reports>", "prompt": "<the prompt, ending with Final action:>",
  "max_tokens": 1, "temperature": 0, "logprobs": 5 }'

The reply holds the chosen token and the five most likely first tokens with their log probabilities, so you can set a threshold on the probability of Answer. On this server logprobs works on /v1/completions and on /v1/chat/completions. prompt_logprobs is refused ("Not yet supporting prompt_logprobs on tt"). This was checked with speculative decoding off and sampling on the host, which is how this bundle runs.

Expected performance

Every number is labelled. A measured number names its evidence files; the first file holds the value. The files are in this repository under evidence/, at the paths shown, with the run's directory, the home directory and the host name replaced by <RUN_DIR>, <HOME> and <HOST>. TODO means not measured.

Number	Value	Label	Evidence
top1 agreement with the CPU reference, stage 2 (2 chips, bundle qwen3.8-27b-dflash2-p300)	0.96875 fraction	measured	stages/2/result.json, stages/2/evidence/swap-check.json, stages/2/evidence/server.log
server ready after start, stage 2 (empty tensor cache)	1482.3 s	measured	stages/2/result.json, stages/2/evidence/swap-check.json, stages/2/evidence/server.log
top1 agreement with the CPU reference, this package (stage 7)	0.96875 fraction	measured	stages/7/verify/evidence/verify.json, stages/7/verify/evidence/server.log
server ready after start, this package (stage 7, fresh install, empty tensor cache)	1546.3 s	measured	stages/7/verify/evidence/verify.json, stages/7/verify/evidence/server.log
gate accuracy, 200 ChainCheck items (2 chips)	0.930 fraction	measured	gate-check/RESULTS.md, gate-check/results.jsonl
over-refusal, P(not Answer given sufficient evidence), 135 items	0.074 fraction	measured	gate-check/RESULTS.md, gate-check/results.jsonl
unsupported answers, P(Answer given insufficient evidence), 65 items	0.062 fraction	measured	gate-check/RESULTS.md, gate-check/results.jsonl
chip label equals CPU label, 25 of the same prompts	25 of 25 items	measured	gate-check/RESULTS.md, gate-check/cpu_ref.json
time per request with 4 requests in flight, median	2.5 s	measured	gate-check/RESULTS.md, gate-check/results.jsonl

What the gate check measured

The check ran on 2 Blackhole chips with this bundle's serving setup (speculative decoding off, sampling on the host). It sent 200 prompts built from ChainCheck, the author's own evaluation set (https://huggingface.co/datasets/ThakiCloud/ChainCheck): 70 with an intact chain, 65 with an intact chain and a consistent entity swap, and 65 with a broken chain, each at most 2,048 tokens. Intact chains should give Answer and broken chains Retrieve. The files named in the table above hold every result.

• The interval for gate accuracy is 0.886 to 0.958 (Wilson, 95 per cent confidence), so 200 items can show a large gap and cannot confirm the author's figure. The author reports accuracy of .938, over-refusal of .071 and unsupported answers of .055 on a different, larger test set. The numbers here are consistent with that. They do not replicate it, because the sets differ.
• The chips and the CPU model gave the same label on all 25 prompts compared (every 8th of the 200). Both were wrong on the same one.
• The first prompt's token probabilities were also compared: the chips returned a probability of 0.795 for Answer and 0.201 for Retrieve on the first prompt. The probabilities for the other prompts were not collected from the chips.

Compared with the same author's smaller models

ThakiCloud also publishes RAG-Gate-4B, 8B and 9B. Their cards report action accuracy of .950, .949 and .955, against .938 for this 27B model, and unsupported-answer rates of .037, .047 and .033, against .055. We did not check that all four were scored on identical test items. If you do not need this base model, the smaller models are at least as accurate on those figures and are smaller models.

Limitations

• Tested on 2 chips (mesh P150x2) only; other configurations are not claimed by this bundle.
• The boot check compared 32 tokens of one fixed prompt with the CPU reference. The gate check used 200 prompts from one evaluation set, in English, built from Wikipedia-based multi-hop questions. Other domains, languages, tables and code were not measured.
• Inputs longer than 2,048 tokens were not measured, here or by the author.
• The model judges whether the evidence is sufficient. It does not judge whether the evidence is true.
• Speculative decoding is off, so decode speed is that of plain decoding.
• Sampling runs on the host.

Risks and safety considerations

The checks above do not cover long outputs, tool calling or safety behaviour. A wrong Answer lets an unsupported answer through, and the measured rate for that is in the table. Output can differ from ThakiCloud/RAG-Gate-27B run on a CPU or GPU in ways it does not show.

Not measured

• a download of the weights through tt-model pull, and a boot of the package from the Hub
• the gate on 1 or 4 chips, and latency for a single request
• the Stop action (every prompt offered retrieval), and any threshold on the Answer probability
• the same labels with speculative decoding on

Boot check

Stage 7 of the run installed this bundle, served it on a leased board and compared its tokens with the CPU reference. The Expected performance table has the result.

How to serve

tt-model serve episod/rag-gate-27b-p300