|
Download README.md from StillDeadcode/ling-3.0-tiny-iq8r: direct link, hf CLI and curl.
- Browser
- Download file 1.89 kB
-
https://huggingface.co/StillDeadcode/ling-3.0-tiny-iq8r/resolve/main/README.md
- Command line
-
hf download hf://StillDeadcode/ling-3.0-tiny-iq8r/README.md
-
curl -L -o README.md https://huggingface.co/StillDeadcode/ling-3.0-tiny-iq8r/resolve/main/README.md
1.89 kB
| license: mit | |
| base_model: inclusionAI/Ling-3.0-tiny | |
| base_model_relation: quantized | |
| pipeline_tag: text-generation | |
| tags: | |
| - radiance | |
| # Ling-3.0-tiny, IQ8R, for radiance | |
| `ling-3.0-tiny-iq8r.rad` (7.9 GiB) is | |
| [inclusionAI/Ling-3.0-tiny](https://huggingface.co/inclusionAI/Ling-3.0-tiny) converted for the | |
| [radiance](https://codeberg.org/StillDeadcode/radiance) inference engine, version 2.1.0 or later. | |
| It is a quick conversion for trying the model: round to nearest with a scale search, no calibration | |
| data. | |
| ## What is in it | |
| - IQ8R trunk: the KDA projections (fused q|k|v, forget, gate, output), the latent attention's | |
| output, the dense layer's feed-forward and the shared expert's gate/up as int8 codes with one u8 | |
| scale per (row, 256 columns) under one bf16 scale a row, each 128 columns rotated by a | |
| Walsh-Hadamard transform before rounding. | |
| - The 128 routed experts of every layer and the shared expert's down projection as int8 on the same | |
| rotated grid, one bf16 scale per (row, 128 columns). | |
| - In bf16: the KDA beta projection, the latent attention's low-rank projections and gate, the | |
| convolutions, the router, the embedding and the lm_head. | |
| Converted with the recipe the engine ships: | |
| ``` | |
| rad-convert <Ling-3.0-tiny checkpoint> --recipe data/recipes/ling-3.0-tiny-iq8r.recipe \ | |
| -o ling-3.0-tiny-iq8r.rad | |
| ``` | |
| ## Serving | |
| One card, or two: | |
| ```yaml | |
| models: | |
| ling-3.0-tiny: | |
| model: /models/ling-3.0-tiny-iq8r.rad | |
| tp: 1 | |
| ``` | |
| ## What was checked | |
| On Radeon AI PRO R9700 cards: the container loads at `tp: 1` and `tp: 2`, answers short factual | |
| and coding prompts correctly, finds a string hidden in a 4.8k-token prompt, and decodes one stream | |
| at 4.3 ms a step on one card (230 tokens/s) and 3.7 ms on two (270 tokens/s). The model has no | |
| draft head. Not measured: KL divergence against the bf16 model, and any benchmark. | |
| ## License | |
| MIT, as the source model. | |