--- license: apache-2.0 base_model: RecursiveSI/flint base_model_relation: quantized pipeline_tag: text-generation tags: - radiance --- # Recursive Flint, IQ8R, for radiance `flint-iq8r.rad` (37.9 GiB) is [RecursiveSI/flint](https://huggingface.co/RecursiveSI/flint) converted for the [radiance](https://codeberg.org/StillDeadcode/radiance) inference engine, version 2.1.0 or later. It is a quick conversion for trying the model: round to nearest with a scale search, no calibration data. ## What is in it - IQ8R trunk: the attention projections, the two dense layers and the shared expert's gate/up as int8 codes with one u8 scale per (row, 256 columns) under one bf16 scale a row, each 128 columns rotated by a Walsh-Hadamard transform before rounding. - The 256 routed experts of every layer and the shared expert's down projection as int8 on the same rotated grid, one bf16 scale per (row, 128 columns). - The lm_head as plain int8 with a bf16 scale per (row, 128 columns). - The router, the norms, the attention sinks and the embedding in bf16. Converted with the recipe the engine ships: ``` rad-convert --add data/flint-zero-bias.safetensors \ --recipe data/recipes/flint-iq8r.recipe \ --chat-template data/flint-chat-template.jinja -o flint-iq8r.rad ``` ## Serving Two 32 GB cards: ```yaml models: flint: model: /models/flint-iq8r.rad tp: 2 kv-cache-dtype: i8 ``` ## What was checked On two Radeon AI PRO R9700 at `tp: 2` with the int8 KV cache: the container loads, answers short factual and coding prompts correctly, finds a string hidden in a 4.5k-token prompt, and decodes at 6.8 ms a step for one stream (no drafter). Not measured: KL divergence against the bf16 model, and any benchmark. ## License Apache-2.0, as the source model; its LICENSE file is included.