Spaces:

thinletter
/

README

Running

App Files Files Community

README / README.md

honza-rosecky

GPU memory for the context: the thinctx 1.0.2 measurement (grows about 15 to 20 times more slowly)

b7b7b2e verified 3 days ago

preview code

Raw History Blame Contribute Delete

2.32 kB

	---
	title: thinletter
	emoji: 🪶
	colorFrom: pink
	colorTo: yellow
	sdk: static
	pinned: false
	---

	thinletter is a small research group. We make existing models lighter to run and measure them against the originals. [thinletter.io](https://thinletter.io) · info@thinletter.io

	## ThinCtx

	ThinCtx-small: the quality of Qwen3.5 in a small GPU memory. Qwen3.5 converted so that the GPU memory for the context stays small and grows about 15 to 20 times more slowly than the original's cache, with the weights stored at 5.4 to 6.4 bits per weight and the context kept compressed in system memory. At 128k tokens the 4B-small needs 4.0 GB of GPU memory instead of 12.0 GB for the original, and at 540k tokens 4.7 GB instead of 24.5 GB.

	- [ThinCtx-Qwen3.5-4B-small](https://huggingface.co/thinletter/ThinCtx-Qwen3.5-4B-small) · full precision: [ThinCtx-Qwen3.5-4B](https://huggingface.co/thinletter/ThinCtx-Qwen3.5-4B)
	- [ThinCtx-Qwen3.5-2B-small](https://huggingface.co/thinletter/ThinCtx-Qwen3.5-2B-small) · full precision: [ThinCtx-Qwen3.5-2B](https://huggingface.co/thinletter/ThinCtx-Qwen3.5-2B)
	- [ThinCtx-Qwen3.5-0.8B-small](https://huggingface.co/thinletter/ThinCtx-Qwen3.5-0.8B-small) · full precision: [ThinCtx-Qwen3.5-0.8B](https://huggingface.co/thinletter/ThinCtx-Qwen3.5-0.8B) (not recommended for tool calling)
	- Runtime: [`pip install thinctx`](https://pypi.org/project/thinctx/) · results: [thinletter.io/thinctx](https://thinletter.io/thinctx/)

	## ThinEmbed

	The query side of an embedding model, compressed into a small file that runs in the browser on WebGPU, against your unchanged document index. Open source, Apache-2.0.

	- harrier-0.6b (English): [WebGPU clients](https://huggingface.co/thinletter/harrier-0.6b-vq-clients) · [GGUF clients](https://huggingface.co/thinletter/harrier-0.6b-query-clients)
	- Qwen3-Embedding-0.6B (Czech): [WebGPU client](https://huggingface.co/thinletter/qwen3-embedding-0.6b-vq-clients) · [GGUF client](https://huggingface.co/thinletter/qwen3-embedding-0.6b-query-clients)
	- bge-m3 (Czech, multilingual): [GGUF clients](https://huggingface.co/thinletter/bge-m3-query-clients)
	- [Demo](https://thinletter.io/demo) · [code](https://github.com/rosecky/embedding-quantization-public) · results: [thinletter.io/query-encoders](https://thinletter.io/query-encoders/)