Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
base_model: inclusionAI/Ling-3.0-tiny
|
| 4 |
+
base_model_relation: quantized
|
| 5 |
+
pipeline_tag: text-generation
|
| 6 |
+
tags:
|
| 7 |
+
- radiance
|
| 8 |
+
---
|
| 9 |
+
|
| 10 |
+
# Ling-3.0-tiny, IQ8R, for radiance
|
| 11 |
+
|
| 12 |
+
`ling-3.0-tiny-iq8r.rad` (7.9 GiB) is
|
| 13 |
+
[inclusionAI/Ling-3.0-tiny](https://huggingface.co/inclusionAI/Ling-3.0-tiny) converted for the
|
| 14 |
+
[radiance](https://codeberg.org/StillDeadcode/radiance) inference engine, version 2.1.0 or later.
|
| 15 |
+
It is a quick conversion for trying the model: round to nearest with a scale search, no calibration
|
| 16 |
+
data.
|
| 17 |
+
|
| 18 |
+
## What is in it
|
| 19 |
+
|
| 20 |
+
- IQ8R trunk: the KDA projections (fused q|k|v, forget, gate, output), the latent attention's
|
| 21 |
+
output, the dense layer's feed-forward and the shared expert's gate/up as int8 codes with one u8
|
| 22 |
+
scale per (row, 256 columns) under one bf16 scale a row, each 128 columns rotated by a
|
| 23 |
+
Walsh-Hadamard transform before rounding.
|
| 24 |
+
- The 128 routed experts of every layer and the shared expert's down projection as int8 on the same
|
| 25 |
+
rotated grid, one bf16 scale per (row, 128 columns).
|
| 26 |
+
- In bf16: the KDA beta projection, the latent attention's low-rank projections and gate, the
|
| 27 |
+
convolutions, the router, the embedding and the lm_head.
|
| 28 |
+
|
| 29 |
+
Converted with the recipe the engine ships:
|
| 30 |
+
|
| 31 |
+
```
|
| 32 |
+
rad-convert <Ling-3.0-tiny checkpoint> --recipe data/recipes/ling-3.0-tiny-iq8r.recipe \
|
| 33 |
+
-o ling-3.0-tiny-iq8r.rad
|
| 34 |
+
```
|
| 35 |
+
|
| 36 |
+
## Serving
|
| 37 |
+
|
| 38 |
+
One card, or two:
|
| 39 |
+
|
| 40 |
+
```yaml
|
| 41 |
+
models:
|
| 42 |
+
ling-3.0-tiny:
|
| 43 |
+
model: /models/ling-3.0-tiny-iq8r.rad
|
| 44 |
+
tp: 1
|
| 45 |
+
```
|
| 46 |
+
|
| 47 |
+
## What was checked
|
| 48 |
+
|
| 49 |
+
On Radeon AI PRO R9700 cards: the container loads at `tp: 1` and `tp: 2`, answers short factual
|
| 50 |
+
and coding prompts correctly, finds a string hidden in a 4.8k-token prompt, and decodes one stream
|
| 51 |
+
at 4.3 ms a step on one card (230 tokens/s) and 3.7 ms on two (270 tokens/s). The model has no
|
| 52 |
+
draft head. Not measured: KL divergence against the bf16 model, and any benchmark.
|
| 53 |
+
|
| 54 |
+
## License
|
| 55 |
+
|
| 56 |
+
MIT, as the source model.
|