|
Download README.md from StillDeadcode/flint-iq8r: direct link, hf CLI and curl.
- Browser
- Download file 1.84 kB
-
https://huggingface.co/StillDeadcode/flint-iq8r/resolve/main/README.md
- Command line
-
hf download hf://StillDeadcode/flint-iq8r/README.md
-
curl -L -o README.md https://huggingface.co/StillDeadcode/flint-iq8r/resolve/main/README.md
1.84 kB
| license: apache-2.0 | |
| base_model: RecursiveSI/flint | |
| base_model_relation: quantized | |
| pipeline_tag: text-generation | |
| tags: | |
| - radiance | |
| # Recursive Flint, IQ8R, for radiance | |
| `flint-iq8r.rad` (37.9 GiB) is [RecursiveSI/flint](https://huggingface.co/RecursiveSI/flint) | |
| converted for the [radiance](https://codeberg.org/StillDeadcode/radiance) inference engine, | |
| version 2.1.0 or later. It is a quick conversion for trying the model: round to nearest with a | |
| scale search, no calibration data. | |
| ## What is in it | |
| - IQ8R trunk: the attention projections, the two dense layers and the shared expert's gate/up as | |
| int8 codes with one u8 scale per (row, 256 columns) under one bf16 scale a row, each 128 columns | |
| rotated by a Walsh-Hadamard transform before rounding. | |
| - The 256 routed experts of every layer and the shared expert's down projection as int8 on the same | |
| rotated grid, one bf16 scale per (row, 128 columns). | |
| - The lm_head as plain int8 with a bf16 scale per (row, 128 columns). | |
| - The router, the norms, the attention sinks and the embedding in bf16. | |
| Converted with the recipe the engine ships: | |
| ``` | |
| rad-convert <flint checkpoint> --add data/flint-zero-bias.safetensors \ | |
| --recipe data/recipes/flint-iq8r.recipe \ | |
| --chat-template data/flint-chat-template.jinja -o flint-iq8r.rad | |
| ``` | |
| ## Serving | |
| Two 32 GB cards: | |
| ```yaml | |
| models: | |
| flint: | |
| model: /models/flint-iq8r.rad | |
| tp: 2 | |
| kv-cache-dtype: i8 | |
| ``` | |
| ## What was checked | |
| On two Radeon AI PRO R9700 at `tp: 2` with the int8 KV cache: the container loads, answers short | |
| factual and coding prompts correctly, finds a string hidden in a 4.5k-token prompt, and decodes at | |
| 6.8 ms a step for one stream (no drafter). Not measured: KL divergence against the bf16 model, and | |
| any benchmark. | |
| ## License | |
| Apache-2.0, as the source model; its LICENSE file is included. | |