StillDeadcode commited on
Commit
b03abcf
·
verified ·
1 Parent(s): 1c28c5b

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +56 -0
README.md ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ base_model: inclusionAI/Ling-3.0-tiny
4
+ base_model_relation: quantized
5
+ pipeline_tag: text-generation
6
+ tags:
7
+ - radiance
8
+ ---
9
+
10
+ # Ling-3.0-tiny, IQ8R, for radiance
11
+
12
+ `ling-3.0-tiny-iq8r.rad` (7.9 GiB) is
13
+ [inclusionAI/Ling-3.0-tiny](https://huggingface.co/inclusionAI/Ling-3.0-tiny) converted for the
14
+ [radiance](https://codeberg.org/StillDeadcode/radiance) inference engine, version 2.1.0 or later.
15
+ It is a quick conversion for trying the model: round to nearest with a scale search, no calibration
16
+ data.
17
+
18
+ ## What is in it
19
+
20
+ - IQ8R trunk: the KDA projections (fused q|k|v, forget, gate, output), the latent attention's
21
+ output, the dense layer's feed-forward and the shared expert's gate/up as int8 codes with one u8
22
+ scale per (row, 256 columns) under one bf16 scale a row, each 128 columns rotated by a
23
+ Walsh-Hadamard transform before rounding.
24
+ - The 128 routed experts of every layer and the shared expert's down projection as int8 on the same
25
+ rotated grid, one bf16 scale per (row, 128 columns).
26
+ - In bf16: the KDA beta projection, the latent attention's low-rank projections and gate, the
27
+ convolutions, the router, the embedding and the lm_head.
28
+
29
+ Converted with the recipe the engine ships:
30
+
31
+ ```
32
+ rad-convert <Ling-3.0-tiny checkpoint> --recipe data/recipes/ling-3.0-tiny-iq8r.recipe \
33
+ -o ling-3.0-tiny-iq8r.rad
34
+ ```
35
+
36
+ ## Serving
37
+
38
+ One card, or two:
39
+
40
+ ```yaml
41
+ models:
42
+ ling-3.0-tiny:
43
+ model: /models/ling-3.0-tiny-iq8r.rad
44
+ tp: 1
45
+ ```
46
+
47
+ ## What was checked
48
+
49
+ On Radeon AI PRO R9700 cards: the container loads at `tp: 1` and `tp: 2`, answers short factual
50
+ and coding prompts correctly, finds a string hidden in a 4.8k-token prompt, and decodes one stream
51
+ at 4.3 ms a step on one card (230 tokens/s) and 3.7 ms on two (270 tokens/s). The model has no
52
+ draft head. Not measured: KL divergence against the bf16 model, and any benchmark.
53
+
54
+ ## License
55
+
56
+ MIT, as the source model.