StillDeadcode commited on
Commit
77f74a3
·
verified ·
1 Parent(s): 8fdf200

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +56 -0
README.md ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: RecursiveSI/flint
4
+ base_model_relation: quantized
5
+ pipeline_tag: text-generation
6
+ tags:
7
+ - radiance
8
+ ---
9
+
10
+ # Recursive Flint, IQ8R, for radiance
11
+
12
+ `flint-iq8r.rad` (37.9 GiB) is [RecursiveSI/flint](https://huggingface.co/RecursiveSI/flint)
13
+ converted for the [radiance](https://codeberg.org/StillDeadcode/radiance) inference engine,
14
+ version 2.1.0 or later. It is a quick conversion for trying the model: round to nearest with a
15
+ scale search, no calibration data.
16
+
17
+ ## What is in it
18
+
19
+ - IQ8R trunk: the attention projections, the two dense layers and the shared expert's gate/up as
20
+ int8 codes with one u8 scale per (row, 256 columns) under one bf16 scale a row, each 128 columns
21
+ rotated by a Walsh-Hadamard transform before rounding.
22
+ - The 256 routed experts of every layer and the shared expert's down projection as int8 on the same
23
+ rotated grid, one bf16 scale per (row, 128 columns).
24
+ - The lm_head as plain int8 with a bf16 scale per (row, 128 columns).
25
+ - The router, the norms, the attention sinks and the embedding in bf16.
26
+
27
+ Converted with the recipe the engine ships:
28
+
29
+ ```
30
+ rad-convert <flint checkpoint> --add data/flint-zero-bias.safetensors \
31
+ --recipe data/recipes/flint-iq8r.recipe \
32
+ --chat-template data/flint-chat-template.jinja -o flint-iq8r.rad
33
+ ```
34
+
35
+ ## Serving
36
+
37
+ Two 32 GB cards:
38
+
39
+ ```yaml
40
+ models:
41
+ flint:
42
+ model: /models/flint-iq8r.rad
43
+ tp: 2
44
+ kv-cache-dtype: i8
45
+ ```
46
+
47
+ ## What was checked
48
+
49
+ On two Radeon AI PRO R9700 at `tp: 2` with the int8 KV cache: the container loads, answers short
50
+ factual and coding prompts correctly, finds a string hidden in a 4.5k-token prompt, and decodes at
51
+ 6.8 ms a step for one stream (no drafter). Not measured: KL divergence against the bf16 model, and
52
+ any benchmark.
53
+
54
+ ## License
55
+
56
+ Apache-2.0, as the source model; its LICENSE file is included.