Flare-Preview / README.md
arthu1's picture
Upload README.md with huggingface_hub
c38fe21 verified
|
Raw History Blame Contribute Delete
6.76 kB
---
license: apache-2.0
pipeline_tag: text-generation
language:
- en
tags:
- conversational
- deltanet
- flare
- hybrid
- north-ml
- safetensors
- starlight
thumbnail: flare.png
---
# Flare (preview)
Flare is North ML's merged hybrid architecture: a Qwen3.5-derived backbone of
mostly gated DeltaNet linear-attention layers with periodic full GQA attention,
a stateful agent-memory register after every full-attention layer, a Fiber
draft/memory trunk with delta-rule associative memory blocks, a learned bridge
between the two widths, and a small search/think router head β€” all in one
`model.safetensors` (arch `flare-1`; legacy artifacts read as
`fiber-starlight-1`).
```
token ids
β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Ί shared embed (tied LM head)
β”‚
β”œβ”€β–Ί Starlight backbone β”œβ”€β–Ί Fiber trunk (158M)
β”‚ 24 layers: β”‚ 24 blocks: 21 attn +
β”‚ ~18 gated DeltaNet β”‚ 3 DeltaMemory layers
β”‚ ~6 full GQA attention β”‚ (delta-rule S[H,D,D])
β”‚ +AgentRegister per attn β”€β”€β”€β”€β”˜
β”‚ (carried mem slots)
β”‚
β”œβ”€β–Ί router head β†’ {p_search, p_think}
└─► Era: lossless state snapshots / prefix checkpoints / rollback
over attention KV + recurrent S + register mem + Fiber state
```
## Tiers
```
──────────────────────┬───────────┬──────────────┬────────────────────────────
API id β”‚ Params β”‚ Backbone β”‚ Extra
──────────────────────┼───────────┼──────────────┼────────────────────────────
ink-1 β”‚ ~725M β”‚ Small (1024h)β”‚ router + retrieval
fiber-1 β”‚ ~1.63B β”‚ Pro (2048h)β”‚ router + retrieval
fiber-1-pro β”‚ ~1.63B β”‚ Pro (2048h)β”‚ + fact-verify pass
──────────────────────┴───────────┴──────────────┴────────────────────────────
```
Weights ship 8-bit quantized in a single safetensors file with JSON metadata
(prompts, generation config, router thresholds) plus the tokenizer.
## What is actually measured
Scores below are the merged artifacts running through the MLX runtime, not the
raw backbones. Subsets and settings are noted; treat them as indicative, not
official leaderboard numbers.
```
────────────────────────┬────────┬─────────┬───────────┬──────────────────────
Model β”‚ ARC-C β”‚ MMLU β”‚ HumanEval β”‚ GSM8K
────────────────────────┼────────┼─────────┼───────────┼──────────────────────
ink-1 (725M) β”‚ 63.6% β”‚ 39.3% β”‚ 18.3% β”‚ 40.3%
fiber-1 (1.63B) β”‚ 73.5% β”‚ 48.9% β”‚ 37.2% β”‚ ~11–40%†
────────────────────────┴────────┴─────────┴───────────┴──────────────────────
ARC-Challenge 1165q Β· MMLU 280q subset Β· HumanEval 164 Β· GSM8K 300q
† GSM8K varied sharply across runs/settings β€” flagged as unstable, not a typo.
```
Nothing here supports claims of beating much larger frontier models; the point
of the architecture is capability per parameter, not absolute SOTA.
## Era
Era is the state manager, not a new attention. It checkpoints and restores the
exact model state β€” attention KV, DeltaNet/SSM state, register memory, Fiber
KV and DeltaMemory matrices β€” keyed by token-prefix hashes, so prompts can
resume losslessly and speculative branches can roll back exactly. Prefix reuse
is off by default pending parity verification.
## Use
This is a research preview. The merged format loads with the Kestrel MLX
runtime (`runtime/kestrel`), not stock transformers:
```python
from kestrel import load
m = load("model.safetensors")
for ev in m.run("How tall is Mount Everest?", search="auto"):
... # stream of {type: thinking|text|agent} events
```
## 1M proof-of-concept
`experiments/flare1m/flare_mini.py` trains a ~1M-parameter byte-level Flare
(gated delta layers, registers, DeltaMemory, carried state) against a
parameter-matched vanilla transformer on the same data and steps. First-run
results (1500 steps, RTX 5060 Ti):
```
──────────────────────────┬───────────┬─────────────
metric β”‚ Flare-1M β”‚ GPT (981k)
──────────────────────────┼───────────┼─────────────
params β”‚ 965,908 β”‚ 981,312
heldout CE β”‚ 0.0004 β”‚ 0.0003
passkey @0.1/0.5/0.9 β”‚ 0/.08/.08 β”‚ 0/0/.08
passkey stream-512 β”‚ .08/0/0 β”‚ 0/0/0
associative recall β”‚ .146 β”‚ .104
train time β”‚ 576s β”‚ 36s
──────────────────────────┴───────────┴─────────────
```
Read honestly: **inconclusive-to-negative**. Both models sit near chance β€”
the passkey docs (2600 tokens) were 7Γ— beyond the training window (T=384), so
neither could reach the planted fact; the recall edge (0.146 vs 0.104, chance
0.111) is within noise at n=48; and the delta-rule scan cost 16Γ— more wall
time per step. The 1M demo does *not* prove the mechanisms beat a plain
transformer β€” a fair test needs training at longer context so the recall task
is within reach.
## Limitations
- Small models: expect hallucination, weak multi-hop reasoning, and shaky math
relative to frontier models. Retrieval grounding helps but does not verify.
- English only; 64k vocabulary; 8k context in current runtime.
- The search/think router is a linear probe on top of the embeddings β€” it is
calibrated, not intelligent.
- Preview: the artifact format and API surface may change.
---
*Part of [North ML](https://huggingface.co/North-ML1) - open weights, honest numbers.*