File size: 1,836 Bytes
77f74a3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
---
license: apache-2.0
base_model: RecursiveSI/flint
base_model_relation: quantized
pipeline_tag: text-generation
tags:
- radiance
---

# Recursive Flint, IQ8R, for radiance

`flint-iq8r.rad` (37.9 GiB) is [RecursiveSI/flint](https://huggingface.co/RecursiveSI/flint)
converted for the [radiance](https://codeberg.org/StillDeadcode/radiance) inference engine,
version 2.1.0 or later. It is a quick conversion for trying the model: round to nearest with a
scale search, no calibration data.

## What is in it

- IQ8R trunk: the attention projections, the two dense layers and the shared expert's gate/up as
  int8 codes with one u8 scale per (row, 256 columns) under one bf16 scale a row, each 128 columns
  rotated by a Walsh-Hadamard transform before rounding.
- The 256 routed experts of every layer and the shared expert's down projection as int8 on the same
  rotated grid, one bf16 scale per (row, 128 columns).
- The lm_head as plain int8 with a bf16 scale per (row, 128 columns).
- The router, the norms, the attention sinks and the embedding in bf16.

Converted with the recipe the engine ships:

```
rad-convert <flint checkpoint> --add data/flint-zero-bias.safetensors \
    --recipe data/recipes/flint-iq8r.recipe \
    --chat-template data/flint-chat-template.jinja -o flint-iq8r.rad
```

## Serving

Two 32 GB cards:

```yaml
models:
  flint:
    model: /models/flint-iq8r.rad
    tp: 2
    kv-cache-dtype: i8
```

## What was checked

On two Radeon AI PRO R9700 at `tp: 2` with the int8 KV cache: the container loads, answers short
factual and coding prompts correctly, finds a string hidden in a 4.5k-token prompt, and decodes at
6.8 ms a step for one stream (no drafter). Not measured: KL divergence against the bf16 model, and
any benchmark.

## License

Apache-2.0, as the source model; its LICENSE file is included.