ThinCtx-Qwen3.5-0.8B

ThinCtx by thinletter. ThinCtx-Qwen3.5-0.8B is the ThinCtx conversion of Qwen3.5-0.8B that keeps the GPU memory for the context small: it grows 30 to 40 times more slowly than the base model's cache (0.33 GB instead of 6.0 GB at half a million tokens).

Not recommended for tool calling. The conversion lost much of the base model's function calling: BFCL non-live 28.7 % against the base model's 53.5 %, also on short prompts. It also loses more than the larger sizes on long context (RULER at 64k tokens: 71 against 83). For agents and tool use, use ThinCtx-Qwen3.5-2B or -4B.

The part of the context kept on the GPU is small and grows slowly with the length of the conversation or document; the full context is kept in ordinary system memory. ThinCtx reads long prompts more slowly than the base model (see Speed below). Chat and tool calling use the Qwen3.5 chat template.

Runtime required. ThinCtx models run on the thinctx runtime (pip install thinctx, below). The runtime is what keeps the context memory small; other inference libraries do not do that with these weights. This model requires thinctx >= 0.2.0.

Install and run

pip install "thinctx>=0.2.0"
# an OpenAI-compatible endpoint (add --auth USER:PASS for HTTP basic auth)
thinctx serve thinletter/ThinCtx-Qwen3.5-0.8B --port 8000
# or chat in the terminal
thinctx chat thinletter/ThinCtx-Qwen3.5-0.8B

import thinctx
engine = thinctx.load("thinletter/ThinCtx-Qwen3.5-0.8B")

The server speaks the OpenAI chat-completions API (/v1/chat/completions, streaming and non-streaming, tools, enable_thinking). Point any OpenAI client at http://host:8000/v1.

Results

Every number compares ThinCtx with the unchanged Qwen3.5-0.8B on the same items.

GPU memory for the context

Per conversation, model weights excluded, measured on one RTX 3090. The base model keeps its whole context cache on the GPU; ThinCtx keeps a small part on the GPU, growing 30 to 40 times more slowly, and the full context in system memory (RAM). During the prefill of a long input the GPU peak is higher. GB.

Test	Qwen3.5-0.8B	ThinCtx
GPU, 128k tokens of context	1.5	0.18
GPU, 512k tokens of context	6.0	0.33
System memory, 512k tokens, 16-bit storage	–	6.5
System memory, 512k tokens, 8-bit storage	–	3.9
System memory, 512k tokens, 4-bit storage	–	2.7

Speed

Tokens per second on one RTX 3090, bf16, a single conversation; for ThinCtx the steady decode rate after the first 16 tokens. The base model runs in plain transformers without CUDA graphs, which ThinCtx's runtime uses, so the decode rows do not compare like with like: an optimized serving engine would run the base model faster. The base model was not measured at 512k tokens.

Test	Qwen3.5-0.8B (transformers)	ThinCtx
Prefill, 8k tokens	23 207	9 848
Prefill, 128k tokens	5 761	4 883
Prefill, 512k tokens	–	2 364
Decode, 8k tokens	44.5	96.7
Decode, 128k tokens	43.0	94.3
Decode, 512k tokens	–	77.6

Long context

BABILong: reasoning over facts hidden in long text. RULER: needle search, tracking and aggregation. LongBench v1: 16 English tasks with their official metrics. Scores in %.

Test	Qwen3.5-0.8B	ThinCtx
BABILong, 4k tokens	45	42
BABILong, 16k tokens	44	40
BABILong, 64k tokens	36	34
RULER, 4k tokens	90	86
RULER, 16k tokens	88	81
RULER, 64k tokens	83	71
LongBench v1, mean of 16 tasks	36.0	33.7

Tools and agent behaviour

BFCL: the Berkeley Function Calling Leaderboard, non-live categories. Facts: planted facts asked 32k tokens later. Rules: planted policy rules for an agent's commands, their effect kept far back in the context (1.00 = as the base model). Divergence: KL of the next-token distribution from the base model's, lower is better.

Test	Qwen3.5-0.8B	ThinCtx
Function calling (BFCL, non-live), %	53.5	28.7
Facts recalled at 32k tokens, %	67.9	66.1
Rules kept far back, worst case (1.00 = as the original)	1.00	0.81
Distant tool calls, divergence from the original (lower is better)	reference	0.010
Plain text, divergence from the original (lower is better)	reference	0.0071

Czech and German

Bits per byte: lower is better. Belebele: multiple-choice reading comprehension. Needle search: RULER-style tasks in Czech and German. RAG: answers from planted documents.

Test	Qwen3.5-0.8B	ThinCtx
Czech web text, bits per byte (lower is better)	1.476	1.479
German web text, bits per byte	1.198	1.207
Czech EU law, 8k tokens, bits per byte	0.754	0.757
German EU law, 8k tokens, bits per byte	0.645	0.644
Czech reading comprehension (Belebele), %	51.3	51.7
German reading comprehension (Belebele), %	62.0	63.3
Czech needle search at 64k tokens, %	88.9	71.1
German needle search at 64k tokens, %	93.1	80.0
Czech rules kept far back, worst case	1.00	0.84
German rules kept far back, worst case	1.00	0.85
Czech RAG answers at 8k tokens, %	100	100
German RAG answers at 8k tokens, %	94	94

Full results for all sizes: https://thinletter.io/thinctx/

Requirements

Linux x86_64, CPython 3.12, an NVIDIA GPU, PyTorch 2.11. pip install thinctx pulls torch, transformers, fla-core and safetensors. The default PyTorch 2.11 wheel targets CUDA 13; with an older NVIDIA driver, install the CUDA 12.8 build first: pip install "torch==2.11.*" --index-url https://download.pytorch.org/whl/cu128. causal_conv1d is optional and makes prefill faster (without it a slower pure-PyTorch fallback is used).

License

Released under the PolyForm Small Business License 1.0.0 (LICENSE, full text in LICENSE-PolyForm-Small-Business-1.0.0.txt): free for companies with fewer than 100 people and under USD 1 million revenue (2019 dollars, adjusted for inflation). LICENSE adds free use, for anyone, for personal non-commercial purposes, for non-commercial research and education, and for evaluation and testing (non-production). Production use by larger organisations needs a commercial licence. The weights are a derivative of Qwen3.5-0.8B (Apache-2.0); Qwen's license (LICENSE-Apache-2.0-Qwen.txt) and a NOTICE with the attribution and the changes are included.

Using ThinCtx at work, or above the free threshold? Write to info@thinletter.io; we will also let you know about new releases.