|
Download README.md from North-ML1/Flare-Preview: direct link, hf CLI and curl.
- Browser
- Download file 6.76 kB
-
https://huggingface.co/North-ML1/Flare-Preview/resolve/main/README.md
- Command line
-
hf download hf://North-ML1/Flare-Preview/README.md
-
curl -L -o README.md https://huggingface.co/North-ML1/Flare-Preview/resolve/main/README.md
6.76 kB
| license: apache-2.0 | |
| pipeline_tag: text-generation | |
| language: | |
| - en | |
| tags: | |
| - conversational | |
| - deltanet | |
| - flare | |
| - hybrid | |
| - north-ml | |
| - safetensors | |
| - starlight | |
| thumbnail: flare.png | |
| # Flare (preview) | |
| Flare is North ML's merged hybrid architecture: a Qwen3.5-derived backbone of | |
| mostly gated DeltaNet linear-attention layers with periodic full GQA attention, | |
| a stateful agent-memory register after every full-attention layer, a Fiber | |
| draft/memory trunk with delta-rule associative memory blocks, a learned bridge | |
| between the two widths, and a small search/think router head β all in one | |
| `model.safetensors` (arch `flare-1`; legacy artifacts read as | |
| `fiber-starlight-1`). | |
| ``` | |
| token ids | |
| β | |
| βββββββββββββββΊ shared embed (tied LM head) | |
| β | |
| βββΊ Starlight backbone βββΊ Fiber trunk (158M) | |
| β 24 layers: β 24 blocks: 21 attn + | |
| β ~18 gated DeltaNet β 3 DeltaMemory layers | |
| β ~6 full GQA attention β (delta-rule S[H,D,D]) | |
| β +AgentRegister per attn βββββ | |
| β (carried mem slots) | |
| β | |
| βββΊ router head β {p_search, p_think} | |
| βββΊ Era: lossless state snapshots / prefix checkpoints / rollback | |
| over attention KV + recurrent S + register mem + Fiber state | |
| ``` | |
| ## Tiers | |
| ``` | |
| βββββββββββββββββββββββ¬ββββββββββββ¬βββββββββββββββ¬ββββββββββββββββββββββββββββ | |
| API id β Params β Backbone β Extra | |
| βββββββββββββββββββββββΌββββββββββββΌβββββββββββββββΌββββββββββββββββββββββββββββ | |
| ink-1 β ~725M β Small (1024h)β router + retrieval | |
| fiber-1 β ~1.63B β Pro (2048h)β router + retrieval | |
| fiber-1-pro β ~1.63B β Pro (2048h)β + fact-verify pass | |
| βββββββββββββββββββββββ΄ββββββββββββ΄βββββββββββββββ΄ββββββββββββββββββββββββββββ | |
| ``` | |
| Weights ship 8-bit quantized in a single safetensors file with JSON metadata | |
| (prompts, generation config, router thresholds) plus the tokenizer. | |
| ## What is actually measured | |
| Scores below are the merged artifacts running through the MLX runtime, not the | |
| raw backbones. Subsets and settings are noted; treat them as indicative, not | |
| official leaderboard numbers. | |
| ``` | |
| βββββββββββββββββββββββββ¬βββββββββ¬ββββββββββ¬ββββββββββββ¬ββββββββββββββββββββββ | |
| Model β ARC-C β MMLU β HumanEval β GSM8K | |
| βββββββββββββββββββββββββΌβββββββββΌββββββββββΌββββββββββββΌββββββββββββββββββββββ | |
| ink-1 (725M) β 63.6% β 39.3% β 18.3% β 40.3% | |
| fiber-1 (1.63B) β 73.5% β 48.9% β 37.2% β ~11β40%β | |
| βββββββββββββββββββββββββ΄βββββββββ΄ββββββββββ΄ββββββββββββ΄ββββββββββββββββββββββ | |
| ARC-Challenge 1165q Β· MMLU 280q subset Β· HumanEval 164 Β· GSM8K 300q | |
| β GSM8K varied sharply across runs/settings β flagged as unstable, not a typo. | |
| ``` | |
| Nothing here supports claims of beating much larger frontier models; the point | |
| of the architecture is capability per parameter, not absolute SOTA. | |
| ## Era | |
| Era is the state manager, not a new attention. It checkpoints and restores the | |
| exact model state β attention KV, DeltaNet/SSM state, register memory, Fiber | |
| KV and DeltaMemory matrices β keyed by token-prefix hashes, so prompts can | |
| resume losslessly and speculative branches can roll back exactly. Prefix reuse | |
| is off by default pending parity verification. | |
| ## Use | |
| This is a research preview. The merged format loads with the Kestrel MLX | |
| runtime (`runtime/kestrel`), not stock transformers: | |
| ```python | |
| from kestrel import load | |
| m = load("model.safetensors") | |
| for ev in m.run("How tall is Mount Everest?", search="auto"): | |
| ... # stream of {type: thinking|text|agent} events | |
| ``` | |
| ## 1M proof-of-concept | |
| `experiments/flare1m/flare_mini.py` trains a ~1M-parameter byte-level Flare | |
| (gated delta layers, registers, DeltaMemory, carried state) against a | |
| parameter-matched vanilla transformer on the same data and steps. First-run | |
| results (1500 steps, RTX 5060 Ti): | |
| ``` | |
| βββββββββββββββββββββββββββ¬ββββββββββββ¬βββββββββββββ | |
| metric β Flare-1M β GPT (981k) | |
| βββββββββββββββββββββββββββΌββββββββββββΌβββββββββββββ | |
| params β 965,908 β 981,312 | |
| heldout CE β 0.0004 β 0.0003 | |
| passkey @0.1/0.5/0.9 β 0/.08/.08 β 0/0/.08 | |
| passkey stream-512 β .08/0/0 β 0/0/0 | |
| associative recall β .146 β .104 | |
| train time β 576s β 36s | |
| βββββββββββββββββββββββββββ΄ββββββββββββ΄βββββββββββββ | |
| ``` | |
| Read honestly: **inconclusive-to-negative**. Both models sit near chance β | |
| the passkey docs (2600 tokens) were 7Γ beyond the training window (T=384), so | |
| neither could reach the planted fact; the recall edge (0.146 vs 0.104, chance | |
| 0.111) is within noise at n=48; and the delta-rule scan cost 16Γ more wall | |
| time per step. The 1M demo does *not* prove the mechanisms beat a plain | |
| transformer β a fair test needs training at longer context so the recall task | |
| is within reach. | |
| ## Limitations | |
| - Small models: expect hallucination, weak multi-hop reasoning, and shaky math | |
| relative to frontier models. Retrieval grounding helps but does not verify. | |
| - English only; 64k vocabulary; 8k context in current runtime. | |
| - The search/think router is a linear probe on top of the embeddings β it is | |
| calibrated, not intelligent. | |
| - Preview: the artifact format and API surface may change. | |
| --- | |
| *Part of [North ML](https://huggingface.co/North-ML1) - open weights, honest numbers.* | |