ThinCtx-Qwen3.5-2B

ThinCtx by thinletter. ThinCtx-Qwen3.5-2B is the ThinCtx conversion of Qwen3.5-2B that keeps the GPU memory for the context small: it grows 20 to 30 times more slowly than the base model's cache (0.50 GB instead of 6.0 GB at half a million tokens), with accuracy on par with the base model (on average 73.4 % against the base model's 72.4 % on BABILong and RULER at 4k, 16k and 64k tokens and BFCL non-live).

The part of the context kept on the GPU is small and grows slowly with the length of the conversation or document; the full context is kept in ordinary system memory. ThinCtx trades some speed for this (see Speed below). Chat and tool calling use the Qwen3.5 chat template.

Runtime required. ThinCtx models run on the thinctx runtime (pip install thinctx, below). The runtime is what keeps the context memory small; other inference libraries do not do that with these weights.

Install and run

pip install thinctx
# an OpenAI-compatible endpoint (add --auth USER:PASS for HTTP basic auth)
thinctx serve thinletter/ThinCtx-Qwen3.5-2B --port 8000
# or chat in the terminal
thinctx chat thinletter/ThinCtx-Qwen3.5-2B

import thinctx
engine = thinctx.load("thinletter/ThinCtx-Qwen3.5-2B")

The server speaks the OpenAI chat-completions API (/v1/chat/completions, streaming and non-streaming, tools, enable_thinking). Point any OpenAI client at http://host:8000/v1.

Results

Every number compares ThinCtx with the unchanged Qwen3.5-2B on the same items.

GPU memory for the context

Per conversation, model weights excluded, measured on one RTX 3090. The base model keeps its whole context cache on the GPU; ThinCtx keeps a small part on the GPU, growing 20 to 30 times more slowly, and the full context in system memory (RAM). During the prefill of a long input the GPU peak is higher. GB.

Test	Qwen3.5-2B	ThinCtx
GPU, 128k tokens of context	1.5	0.29
GPU, 512k tokens of context	6.0	0.50
System memory, 512k tokens, 16-bit storage	–	6.0
System memory, 512k tokens, 8-bit storage	–	3.3
System memory, 512k tokens, 4-bit storage	–	2.2

Speed

Tokens per second on one RTX 3090, bf16, a single conversation. ThinCtx trades speed for memory; the base model was not measured at 512k tokens.

Test	Qwen3.5-2B	ThinCtx
Prefill, 8k tokens	13 542	6 504
Prefill, 32k tokens	9 974	4 223
Prefill, 128k tokens	4 939	2 304
Prefill, 512k tokens	–	996
Decode, 8k tokens	45.0	34.5
Decode, 32k tokens	45.3	33.2
Decode, 128k tokens	44.7	28.8
Decode, 512k tokens	–	20.1

Long context

BABILong: reasoning over facts hidden in long text. RULER: needle search, tracking and aggregation. LongBench v1: 16 English tasks with their official metrics. Scores in %.

Test	Qwen3.5-2B	ThinCtx
BABILong, 4k tokens	61	60
BABILong, 16k tokens	55	54
BABILong, 64k tokens	43	52
BABILong, 128k tokens	43	45
RULER, 4k tokens	95	95
RULER, 16k tokens	97	97
RULER, 64k tokens	95	92
RULER, 128k tokens	91.6	90.6
LongBench v1, mean of 16 tasks	42.6	42.6

Tools and agent behaviour

BFCL: the Berkeley Function Calling Leaderboard, non-live categories. Facts: planted facts asked 32k tokens later. Rules: planted policy rules for an agent's commands, their effect kept far back in the context (1.00 = as the base model). Divergence: KL of the next-token distribution from the base model's, lower is better.

Test	Qwen3.5-2B	ThinCtx
Function calling (BFCL, non-live), %	68.5	70.1
Function calling with thinking (BFCL subset), %	87.8	88.8
Facts recalled at 32k tokens, %	78.6	73.2
Rules kept far back, worst case (1.00 = as the original)	1.00	0.85
Distant tool calls, divergence from the original (lower is better)	reference	0.0032
Plain text, divergence from the original (lower is better)	reference	0.0021

Czech and German

Bits per byte: lower is better. Belebele: multiple-choice reading comprehension. Needle search: RULER-style tasks in Czech and German. RAG: answers from planted documents.

Test	Qwen3.5-2B	ThinCtx
Czech web text, bits per byte (lower is better)	1.298	1.292
German web text, bits per byte	1.063	1.059
Czech EU law, 8k tokens, bits per byte	0.637	0.636
German EU law, 8k tokens, bits per byte	0.563	0.558
Czech reading comprehension (Belebele), %	70.7	69.7
German reading comprehension (Belebele), %	71.0	74.3
Czech needle search at 64k tokens, %	97.6	92.6
German needle search at 64k tokens, %	98.9	95.4
Czech rules kept far back, worst case	1.00	0.82
German rules kept far back, worst case	1.00	0.81
Czech RAG answers at 8k tokens, %	100	100
German RAG answers at 8k tokens, %	94	94

Full results for all sizes: https://thinletter.io/thinctx/

Requirements

Linux x86_64, CPython 3.12, an NVIDIA GPU, PyTorch 2.11. pip install thinctx pulls torch, transformers, fla-core and safetensors. The default PyTorch 2.11 wheel targets CUDA 13; with an older NVIDIA driver, install the CUDA 12.8 build first: pip install "torch==2.11.*" --index-url https://download.pytorch.org/whl/cu128. causal_conv1d is optional and makes prefill faster (without it a slower pure-PyTorch fallback is used).

License

Released under the PolyForm Small Business License 1.0.0 (LICENSE, full text in LICENSE-PolyForm-Small-Business-1.0.0.txt): free for companies with fewer than 100 people and under USD 1 million revenue (2019 dollars, adjusted for inflation). LICENSE adds free use, for anyone, for personal non-commercial purposes, for non-commercial research and education, and for evaluation and testing (non-production). Production use by larger organisations needs a commercial licence. The weights are a derivative of Qwen3.5-2B (Apache-2.0); Qwen's license (LICENSE-Apache-2.0-Qwen.txt) and a NOTICE with the attribution and the changes are included.

Using ThinCtx at work, or above the free threshold? Write to info@thinletter.io; we will also let you know about new releases.