Add illustrated model card, family links and generation example

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/architecture.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,57 +1,224 @@
 ---
 language: en
+library_name: pytorch
+pipeline_tag: text-generation
+license: mit
 tags:
 - amor
 - custom-code
 - pretrained
+- research
+- arxiv:2602.13215
+- gateddeltanet
 datasets:
 - HuggingFaceFW/fineweb-edu
 ---
-# AMOR-GatedDeltaNet-1.5B
-
-Private release preparation of the canonical three-block AMOR checkpoint.
-This is a pretrained text-completion model, not an instruction-tuned assistant.
-
-## Architecture and training
-Backbone: GatedDeltaNet; parameters: 1,524,025,472; context: 3072 tokens.
-Training seed: 42; completed steps: 62,558.
-Training settings and summary are included as JSON.
-Three entropy-gated attention blocks follow the complete recurrent backbone.
-The native entropy gate and its frozen threshold buffers are included.
-No distilled routers, optimizer state, or training data are included.
-
-## Loading
-This repository uses custom PyTorch code, not Transformers AutoModel.
-Download the repository files, install the backbone-specific dependencies,
-and run from that directory:
-
-```python
-import torch
-from load_model import load_model
-from transformers import AutoTokenizer
-
-model = load_model(device="cuda", dtype=torch.float32)
-tokenizer = AutoTokenizer.from_pretrained("teknium/Llama-3.1-AlternateTokenizer")
-inputs = tokenizer("The Eiffel Tower is", return_tensors="pt").to("cuda")
-with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
-    logits = model(inputs["input_ids"])
-next_id = logits[:, -1].argmax(-1)
-print(tokenizer.decode(next_id))
+
+# AMOR-Gated DeltaNet 1.5B
+### When to think fast and slow?
+
+**A complete recurrent backbone, refined by three entropy-gated attention blocks.**
+
+[Paper](https://arxiv.org/abs/2602.13215) · [Code](https://github.com/HaoranZhengRaul/AMOR) · [Files](https://huggingface.co/FlyinGodzilla/AMOR-GatedDeltaNet-1.5B/tree/main) · [All checkpoints](#model-family)
+
+For a smaller download, see the 180M and 440M checkpoints below.
+
+## The idea
+
+AMOR (**Adaptive Metacognitive Output Router**) uses the model's predictive
+uncertainty to decide where attention should refine its recurrent representation.
+Each appended block reads normalized output entropy, compares it with a frozen
+threshold at inference, and adds an attention update only where its gate fires.
+
+The recurrent backbone and attention blocks are pretrained jointly from scratch.
+“Post-hoc” describes where the attention blocks sit, after the complete backbone.
+Keys and values come from residual-stream projections and are retained at every
+position for future retrieval.
+
+![AMOR architecture: recurrent backbone and entropy-gated refinement](assets/architecture.png)
+
+*Block 0 receives the backbone's final normalized output. Blocks 1 and 2 apply
+their own pre-normalization. A shared, embedding-tied LM head supplies all three
+entropy evaluations and the final output prediction.*
+
+This checkpoint uses the **native entropy gate**. Optional distilled
+deployment routers are not included in this release. Training uses dense
+attention with an output mask, so gating does not reduce training attention FLOPs.
+
+## This checkpoint
+
+| Property | Value |
+|---|---|
+| Backbone | Gated DeltaNet |
+| Parameters | 1,524,025,472 |
+| Recurrent mixer–MLP layers | 24 |
+| Model width | 2048 |
+| Appended attention blocks | 3 |
+| Training corpus | FineWeb-Edu |
+| Training tokens | 30,748,520,448 |
+| Training context | 3,072 tokens |
+| Final training step | 62,558 |
+| Seed | 42 |
+| Tokenizer vocabulary | 128,256 |
+| Checkpoint type | Pretrained base model; not instruction-tuned |
+| Weight format | Safetensors; original checkpoint precision |
+
+Detailed settings are in [training_config.json](training_config.json);
+[training_summary.json](training_summary.json) records the completed run.
+
+## Results
+
+This checkpoint's canonical eight-task common-sense mean is **45.1%**.
+The score uses raw accuracy on LAMBADA, HellaSwag, PIQA, ARC-Easy, ARC-Challenge,
+WinoGrande, OpenBookQA and TruthfulQA-mc2.
+
+| Task | Accuracy (%) |
+|---|---:|
+| LAMBADA | 39.1 |
+| HellaSwag | 39.2 |
+| PIQA | 70.6 |
+| ARC-Easy | 66.4 |
+| ARC-Challenge | 30.5 |
+| WinoGrande | 53.7 |
+| OpenBookQA | 24.8 |
+| TruthfulQA-mc2 | 36.9 |
+| **Eight-task mean** | **45.1** |
+
+Per-task entries are point estimates; the figure reports evaluation-repeat spread.
+
+![Common-sense reasoning across three model scales](assets/common_sense.png)
+
+*Research comparison across all three scales; use the **1.5B** panel for this
+checkpoint. Whiskers show the sample SD of five reevaluations of each fixed
+checkpoint, centered on its canonical score. They measure evaluation
+repeatability, not independent-training variation. Axes are zoomed per panel.*
+
+
+### Retrieval and long context
+The 1.5B AMOR variants improve average retrieval over their recurrent backbones;
+serial hybrids retain stronger six-task retrieval averages. LongBench results
+remain close to the recurrent backbones, with lower scores for the serial hybrids
+in this experimental setting.
+
+![Retrieval and single-needle comparisons at 1.5B](assets/retrieval.png)
+
+*Left: six-task retrieval mean. Right: averages of three single-needle variants
+at each context length; lengths are not pooled.*
+
+![LongBench comparison at 1.5B](assets/long_context.png)
+
+*Unweighted mean of 14 task-native scores. Whiskers are benchmark standard errors
+propagated under task independence, not training-seed variability.*
+
+
+<details>
+<summary>Per-task retrieval and LongBench tables</summary>
+
+| Task | Answer-containing outputs (%) |
+|---|---:|
+| SWDE | 47.4 |
+| SQuAD-completion | 36.8 |
+| FDA | 24.9 |
+| TriviaQA | 43.1 |
+| Natural Questions | 10.5 |
+| DROP | 17.4 |
+| **Six-task mean** | **30.0** |
+
+Retrieval outputs do not store standard errors.
+
+| Task | Score ± benchmark SE |
+|---|---:|
+| NarrativeQA | 1.7 ± 0.17 |
+| Qasper | 4.0 ± 0.35 |
+| MultiFieldQA-en | 11.4 ± 0.87 |
+| HotpotQA | 3.6 ± 0.49 |
+| 2WikiMultihopQA | 7.0 ± 0.74 |
+| MuSiQue | 1.9 ± 0.32 |
+| GovReport | 8.1 ± 0.30 |
+| QMSum | 15.1 ± 0.50 |
+| MultiNews | 10.8 ± 0.43 |
+| TREC | 11.0 ± 2.22 |
+| TriviaQA | 11.8 ± 1.22 |
+| SAMSum | 11.4 ± 0.68 |
+| LCC | 10.1 ± 0.38 |
+| RepoBench-P | 10.3 ± 0.41 |
+| **14-task mean** | **8.4 ± 0.22** |
+
+Scores are on a 0–100 scale. The mean SE is propagated under task independence.
+
+</details>
+
+Per-task values are also available in [benchmark_results.json](benchmark_results.json).
+
+## Download and run
+
+These are custom PyTorch models. Use the supplied loader and generation script;
+this repository is not a Transformers `AutoModel.from_pretrained` integration,
+a hosted chat endpoint, or a GGUF/PocketPal package.
+
+A Linux machine with a compatible NVIDIA GPU is the intended accelerated setup.
+Install a CUDA-enabled PyTorch build appropriate for your driver first
+([PyTorch installation](https://pytorch.org/get-started/locally/)).
+Then, in a fresh Python environment:
+
+```bash
+python -m pip install huggingface_hub
+hf download FlyinGodzilla/AMOR-GatedDeltaNet-1.5B --local-dir AMOR-GatedDeltaNet-1.5B
+cd AMOR-GatedDeltaNet-1.5B
+python -m pip install "transformers==4.51.3" safetensors einops
+python -m pip install flash-linear-attention
+python generate.py --prompt "The Eiffel Tower is" --max-new-tokens 32
 ```
 
-The implementation and cache-based decoder are in amor.py and amor_decode.py.
-The tokenizer is referenced, not redistributed. Its exact revision remains to
-be pinned before public release. Efficient inference requires appropriate
-CUDA kernels; this upload does not provide PocketPal/GGUF compatibility.
+For the tested CPU reference path, use PyTorch 2.6.0 and Transformers 4.51.3.
+Mamba2 has a slow CPU fallback (`--device cpu`); Gated DeltaNet requires its
+compatible accelerated kernels. GPU kernel versions must match the PyTorch/CUDA
+environment; the release's fresh CUDA validation is not yet complete.
+
+`generate.py` performs cached greedy text completion. It is intentionally a
+plain-prompt example: these checkpoints were not trained as chat assistants.
+The tokenizer is referenced from
+[teknium/Llama-3.1-AlternateTokenizer](https://huggingface.co/teknium/Llama-3.1-AlternateTokenizer)
+and pinned in `config.json` to the release-packaging revision. The original
+training-time tokenizer revision was not recorded.
+
+### Verification
+Exact weight export is verified. A fresh CUDA loading/generation check for this package is pending.
 
-## Validation and limitations
-All exported tensors are compared exactly with the source checkpoint.
-See export_metadata.json for the current strict-loading and generation status.
-Export checks alone do not establish end-to-end inference correctness.
-Architecture dependency versions must be validated before public release.
-Outputs may be inaccurate or inherit biases from the training data.
-Training uses dense attention with an output mask and does not save attention FLOPs.
+Every exported tensor was compared exactly with its training checkpoint,
+including the entropy gate's frozen buffers. The tied embedding/head matrix is
+stored once and restored by the loader. Optimizer state is excluded.
+[Export metadata](export_metadata.json) records provenance and actual validation;
+[manifest.json](manifest.json) lists file hashes.
+
+## Model family
+| Backbone | 180M | 440M | 1.5B |
+|---|---|---|---|
+| Mamba2 | [180M](https://huggingface.co/FlyinGodzilla/AMOR-Mamba2-180M) | [440M](https://huggingface.co/FlyinGodzilla/AMOR-Mamba2-440M) | [1.5B](https://huggingface.co/FlyinGodzilla/AMOR-Mamba2-1.5B) |
+| Gated DeltaNet | [180M](https://huggingface.co/FlyinGodzilla/AMOR-GatedDeltaNet-180M) | [440M](https://huggingface.co/FlyinGodzilla/AMOR-GatedDeltaNet-440M) | [1.5B](https://huggingface.co/FlyinGodzilla/AMOR-GatedDeltaNet-1.5B) |
+
+All six checkpoints use the three-block canonical AMOR layout. Choose a backbone
+and scale to match your experiment; larger models require more memory.
+The cards use a snapshot of the research figures and do not automatically track
+later manuscript revisions.
+
+## Intended use and limitations
+Research on recurrent–attention hybrids, uncertainty-based routing, and pretrained
+language-model behavior. Outputs can be incorrect and may inherit training-data
+biases. Entropy measures predictive uncertainty; it cannot detect every confidently
+wrong prediction. Inference fire rates depend on the workload, and attention KV
+storage still grows with context. Training context is 3,072 tokens; long-context
+experiments do not establish reliable performance on every domain.
 
 ## License
-The included implementation retains its existing MIT LICENSE.
-Checkpoint redistribution licensing must be confirmed before public release.
+The code and model weights are released under the MIT license; see [LICENSE](https://huggingface.co/FlyinGodzilla/AMOR-GatedDeltaNet-1.5B/blob/main/LICENSE).
+
+## Citation
+```bibtex
+@article{zheng2026amor,
+  title={When to Think Fast and Slow? AMOR: Adaptive Entropy Gate for Hybrid Models},
+  author={Zheng, Haoran and Shani, Chen},
+  journal={arXiv preprint arXiv:2602.13215},
+  year={2026}
+}
+```

benchmark_results.json

@@ -0,0 +1,88 @@
+{
+  "common_sense": {
+    "metric": "acc,none",
+    "scores_percent": {
+      "lambada_openai": 39.06462254997089,
+      "hellaswag": 39.22525393347939,
+      "piqa": 70.5658324265506,
+      "arc_easy": 66.37205387205387,
+      "arc_challenge": 30.46075085324232,
+      "winogrande": 53.67008681925809,
+      "openbookqa": 24.8,
+      "truthfulqa_mc2": 36.88509420492328
+    },
+    "mean_percent": 45.13046183243481
+  },
+  "retrieval": {
+    "metric": "contains,none",
+    "scores_percent": {
+      "swde": 47.434743474347435,
+      "squad_completion": 36.79624664879357,
+      "fda": 24.863883847549907,
+      "based_triviaqa": 43.06872037914692,
+      "based_nq": 10.484637313905607,
+      "based_drop": 17.441303306181123
+    }
+  },
+  "longbench": {
+    "metric": "score,none",
+    "scores_and_benchmark_se": {
+      "longbench_2wikimqa": {
+        "score": 6.9862030554896855,
+        "se": 0.7400626382964874
+      },
+      "longbench_gov_report": {
+        "score": 8.1067315936474,
+        "se": 0.2996812096482549
+      },
+      "longbench_hotpotqa": {
+        "score": 3.601733488122564,
+        "se": 0.48652116456134414
+      },
+      "longbench_lcc": {
+        "score": 10.108000000000004,
+        "se": 0.3765158700517174
+      },
+      "longbench_multi_news": {
+        "score": 10.82613523432961,
+        "se": 0.4334510042700067
+      },
+      "longbench_multifieldqa_en": {
+        "score": 11.440405273363387,
+        "se": 0.873931250965908
+      },
+      "longbench_musique": {
+        "score": 1.9269241806002015,
+        "se": 0.31723867802237216
+      },
+      "longbench_narrativeqa": {
+        "score": 1.7129816044409905,
+        "se": 0.1700376232571283
+      },
+      "longbench_qasper": {
+        "score": 3.9705291693063844,
+        "se": 0.35192468874955485
+      },
+      "longbench_qmsum": {
+        "score": 15.073444382452749,
+        "se": 0.4991732169644108
+      },
+      "longbench_repobench-p": {
+        "score": 10.266,
+        "se": 0.40655758034042955
+      },
+      "longbench_samsum": {
+        "score": 11.406380004969085,
+        "se": 0.6823319787201269
+      },
+      "longbench_trec": {
+        "score": 11.0,
+        "se": 2.2180166793668974
+      },
+      "longbench_triviaqa": {
+        "score": 11.759775857882689,
+        "se": 1.2207354573497748
+      }
+    }
+  }
+}

config.json

@@ -24,10 +24,11 @@
     "offset_k": 0.2
   },
   "tokenizer": "teknium/Llama-3.1-AlternateTokenizer",
-  "tokenizer_revision": null,
+  "tokenizer_revision": "5590cc63198b0f111f4289daf006da0a12044173",
   "training_context_length": 3072,
   "tied_weights": {
     "lm_head.weight": "embed.weight"
   },
-  "note": "Use load_model.py; not compatible with Transformers AutoModel or GGUF."
+  "note": "Use load_model.py; not compatible with Transformers AutoModel or GGUF.",
+  "tokenizer_revision_provenance": "Pinned at release packaging; original training-time revision not recorded."
 }

generate.py

@@ -0,0 +1,42 @@
+"""Cached greedy completion for the custom AMOR release."""
+import argparse
+from contextlib import nullcontext
+import json
+from pathlib import Path
+import torch
+from transformers import AutoTokenizer
+from load_model import load_model
+from amor_decode import AMORDecodeWrapper
+
+def main():
+    p=argparse.ArgumentParser(description=__doc__)
+    p.add_argument("--prompt",required=True)
+    p.add_argument("--max-new-tokens",type=int,default=32)
+    p.add_argument("--device",choices=["cuda","cpu"],default="cuda")
+    p.add_argument("--model-dir",type=Path,default=Path(__file__).resolve().parent)
+    a=p.parse_args()
+    if a.max_new_tokens<1:p.error("--max-new-tokens must be positive")
+    config=json.loads((a.model_dir/"config.json").read_text())
+    if a.device=="cuda" and not torch.cuda.is_available():
+        p.error("CUDA is unavailable; install a CUDA-enabled PyTorch build, or use --device cpu for Mamba2.")
+    if a.device=="cpu" and config["model_kwargs"]["backbone"]=="gdn":
+        p.error("This Gated DeltaNet implementation requires compatible GPU kernels.")
+    tokenizer=AutoTokenizer.from_pretrained(config["tokenizer"],revision=config.get("tokenizer_revision"))
+    model=load_model(a.model_dir,device=a.device,dtype=torch.float32)
+    ids=tokenizer(a.prompt,return_tensors="pt")["input_ids"].to(a.device)
+    if not ids.shape[1]:p.error("Prompt must tokenize to at least one token")
+    wrapper=AMORDecodeWrapper(model)
+    cast=torch.autocast("cuda",dtype=torch.bfloat16) if a.device=="cuda" else nullcontext()
+    generated=[]
+    with torch.inference_mode(),cast:
+        logits,cache=wrapper.prefill(ids)
+        for i in range(a.max_new_tokens):
+            scores=logits[:,-1] if logits.ndim==3 else logits
+            token=scores.argmax(-1,keepdim=True)
+            value=int(token.item())
+            if value==tokenizer.eos_token_id:break
+            generated.append(value)
+            if i+1<a.max_new_tokens:
+                logits,cache,_=wrapper.decode_step(token,cache,step_idx=ids.shape[1]+i)
+    print(a.prompt+tokenizer.decode(generated,skip_special_tokens=True))
+if __name__=="__main__":main()

manifest.json

@@ -4,8 +4,8 @@
     "sha256": "1d0dde41ece0f9620937f1a81a67e295ace0eae180c009d862006e8f3b288458"
   },
   "README.md": {
-    "bytes": 2313,
-    "sha256": "74af2350eb15812a11fae153ee98c3d1b845a9b6124db0cd29cb41d494f35d07"
+    "bytes": 8788,
+    "sha256": "be8cac7a94f1286444e3f4702d08f6294d8bb4613449bfc44fe5aab27ef955e8"
   },
   "amor.py": {
     "bytes": 25015,
@@ -16,8 +16,8 @@
     "sha256": "9c3f27f8fe58d6a9b138c9303e732a1af89ed970cafbceb956a4047b1fb4adb7"
   },
   "config.json": {
-    "bytes": 824,
-    "sha256": "40a3ec034b37afb4b28db874ab6da458a388a76ac24c685dc3d5d476abbe2f3c"
+    "bytes": 975,
+    "sha256": "6228e38543275305089a569d3b7c29e8b2fcdc20ddb64fe8dbf76a8cbd1b8a43"
   },
   "export_metadata.json": {
     "bytes": 935,
@@ -42,5 +42,29 @@
   "training_summary.json": {
     "bytes": 544,
     "sha256": "ddc33c8ad60ec53ef5ca374dedc3bf7f49241fe414188e3d3c5bb8dc2db126fa"
+  },
+  "generate.py": {
+    "bytes": 2138,
+    "sha256": "f48ef56ea6966c2c684f034d036b0178e27a9d446504add875b7823bfca184f5"
+  },
+  "benchmark_results.json": {
+    "bytes": 2323,
+    "sha256": "9eaa7b751478238cce5bd1d8f3db7bf40b91155e97f5c8ec07f6c2a962a29712"
+  },
+  "assets/architecture.png": {
+    "bytes": 120925,
+    "sha256": "ab811b0869810d77486032f866356ffd3cdd401dfe2dfe4ee286de9b0da78858"
+  },
+  "assets/common_sense.png": {
+    "bytes": 57980,
+    "sha256": "8b62b20147bca3114cf8d6ea282e35cba5e4ee15ad6d9eba7fb8f8fdbebb2a36"
+  },
+  "assets/retrieval.png": {
+    "bytes": 61226,
+    "sha256": "58877f631b1fe95376c799733d943108e371bbab51cc0e3b41be609729dd0a79"
+  },
+  "assets/long_context.png": {
+    "bytes": 38008,
+    "sha256": "e34b9340716e4386a149142cff45f43ac8c160025e2c427102a259083dd0ac5f"
   }
 }