Spaces:

thinletter
/

README

Running

App Files Files Community

README / README.md

honza-rosecky

GPU memory for the context: the thinctx 1.0.2 measurement (grows about 15 to 20 times more slowly)

b7b7b2e verified 2 days ago

preview code

Raw History Blame Contribute Delete

2.32 kB

metadata

title: thinletter
emoji: 🪶
colorFrom: pink
colorTo: yellow
sdk: static
pinned: false

thinletter is a small research group. We make existing models lighter to run and measure them against the originals. thinletter.io · info@thinletter.io

ThinCtx

ThinCtx-small: the quality of Qwen3.5 in a small GPU memory. Qwen3.5 converted so that the GPU memory for the context stays small and grows about 15 to 20 times more slowly than the original's cache, with the weights stored at 5.4 to 6.4 bits per weight and the context kept compressed in system memory. At 128k tokens the 4B-small needs 4.0 GB of GPU memory instead of 12.0 GB for the original, and at 540k tokens 4.7 GB instead of 24.5 GB.

ThinCtx-Qwen3.5-4B-small · full precision: ThinCtx-Qwen3.5-4B
ThinCtx-Qwen3.5-2B-small · full precision: ThinCtx-Qwen3.5-2B
ThinCtx-Qwen3.5-0.8B-small · full precision: ThinCtx-Qwen3.5-0.8B (not recommended for tool calling)
Runtime: pip install thinctx · results: thinletter.io/thinctx

ThinEmbed

The query side of an embedding model, compressed into a small file that runs in the browser on WebGPU, against your unchanged document index. Open source, Apache-2.0.

harrier-0.6b (English): WebGPU clients · GGUF clients
Qwen3-Embedding-0.6B (Czech): WebGPU client · GGUF client
bge-m3 (Czech, multilingual): GGUF clients
Demo · code · results: thinletter.io/query-encoders