File size: 1,894 Bytes
b03abcf
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
---
license: mit
base_model: inclusionAI/Ling-3.0-tiny
base_model_relation: quantized
pipeline_tag: text-generation
tags:
- radiance
---

# Ling-3.0-tiny, IQ8R, for radiance

`ling-3.0-tiny-iq8r.rad` (7.9 GiB) is
[inclusionAI/Ling-3.0-tiny](https://huggingface.co/inclusionAI/Ling-3.0-tiny) converted for the
[radiance](https://codeberg.org/StillDeadcode/radiance) inference engine, version 2.1.0 or later.
It is a quick conversion for trying the model: round to nearest with a scale search, no calibration
data.

## What is in it

- IQ8R trunk: the KDA projections (fused q|k|v, forget, gate, output), the latent attention's
  output, the dense layer's feed-forward and the shared expert's gate/up as int8 codes with one u8
  scale per (row, 256 columns) under one bf16 scale a row, each 128 columns rotated by a
  Walsh-Hadamard transform before rounding.
- The 128 routed experts of every layer and the shared expert's down projection as int8 on the same
  rotated grid, one bf16 scale per (row, 128 columns).
- In bf16: the KDA beta projection, the latent attention's low-rank projections and gate, the
  convolutions, the router, the embedding and the lm_head.

Converted with the recipe the engine ships:

```
rad-convert <Ling-3.0-tiny checkpoint> --recipe data/recipes/ling-3.0-tiny-iq8r.recipe \
    -o ling-3.0-tiny-iq8r.rad
```

## Serving

One card, or two:

```yaml
models:
  ling-3.0-tiny:
    model: /models/ling-3.0-tiny-iq8r.rad
    tp: 1
```

## What was checked

On Radeon AI PRO R9700 cards: the container loads at `tp: 1` and `tp: 2`, answers short factual
and coding prompts correctly, finds a string hidden in a 4.8k-token prompt, and decodes one stream
at 4.3 ms a step on one card (230 tokens/s) and 3.7 ms on two (270 tokens/s). The model has no
draft head. Not measured: KL divergence against the bf16 model, and any benchmark.

## License

MIT, as the source model.