Nano-AMADEUS-RNN4LDFA-T1024H256-AV-1001
Amadeus-RNN (Post-Backprop Edition) is a lightweight, 4-layer Parallel Linear RNN with data-dependent gating and associative prefix scan, trained without end-to-end global backpropagation.
Instead of standard BPTT across stacked layers, this model combines three biologically inspired local learning mechanisms:
- Residual-Identity Direct Feedback Alignment (
DFARouter): Broadcasts top-level output error directly to detached lower blocks via learnable residual bridges ((I_H + B_{\text{dfa}})). - Inter-Layer & Intra-Block Kolen-Pollack Co-Adaptation (
B_dfa,B_mlp): Aligns feedback matrices with forward transformations using outer-product updates and weight decay without weight transport. - Associative Eligibility Propagation (
AssociativeEpropFn): Computes local recurrent parameter updates in O(BTH) memory without unrolling (\log_2(T)) scan steps in the global autograd graph.
Model Specifications
- Architecture: 4-Layer Stacked
AmadeusBlock(Pre-RMSNorm, Data-Dependent Gating, Parallel Associative Scan, SwiGLU-style SiLU MLP, Residual Scaling) - Clean Inference Parameters:
1,842,433(~7.37 MB inamadeus-rnn-inference.safetensors) - Training Parameters (with KP bridges):
2,301,185(~9.21 MB inamadeus-rnn-dfa-eprop.safetensors) - Hidden Dimension (
H):256 - Vocabulary Size (
V):1,024(Custom BPE Tokenizer inamadeus_tokenizer.json) - Context Length (
T):512 - Training Data: 1 Epoch on TinyStories corpus
Holdout Evaluation Benchmark (10,000 Unseen Stories)
Evaluated across 2,427,680 valid tokens (seq_len=512) and compared directly against the standard end-to-end backprop baseline (RinKana/Nano-AMADEUS-RNN4L-T1024H256-AV-1001):
| Metric | Backprop Baseline | Post-Backprop (DFA + KP + E-Prop) |
Difference |
|---|---|---|---|
| Inference Parameters | 1,842,433 | 1,842,433 | Identical |
| Token-Weighted CE Loss | 1.4304 | 1.7677 | +0.3373 |
| Corpus Perplexity (PPL) | 4.18 | 5.86 | +1.68 |
| Top-1 Token Accuracy | 62.95% | 56.40% | -6.55% |
| Top-5 Token Accuracy | 86.32% | 81.40% | -4.92% |
| Generation Speed (T4 GPU) | ~415β435 tok/s | ~421β426 tok/s | Identical |
Final Gradient Alignment Diagnostics (Step 21,000)
- Global Cosine Alignment vs. True BPTT:
+0.4821(61.2Β°) - Layer-wise Breakdown:
L0[Rec:+0.346 | MLP:+0.414],L1[Rec:+0.442 | MLP:+0.592],L2[Rec:+0.520 | MLP:+0.746],L3[Rec:+0.828 | MLP:+1.000]
Repository Files
amadeus-rnn-inference.safetensors(7.37 MB): Stripped inference weights (B_dfaandB_mlpremoved) compatible with the cleanAmadeusRNNinference class (strict=True).amadeus-rnn-dfa-eprop.safetensors(9.21 MB): Full training checkpoint including Kolen-Pollack feedback matrices for resuming training.amadeus_tokenizer.json(54.5 kB): Custom 1,024-token BPE tokenizer.
Sample Output
Prompt: "One day, a clever fox found a shiny"
"One day, a clever fox found a shiny thing. The fox was very happy. It wanted to play with the fox. It met a big dog. The dog said, "Hi, I am a dog. Do you want to play with me?" The dog wagged its tail and said, "Yes, I want to play with you.""
Quick Usage (Inference)
import torch
from safetensors.torch import load_model
from tokenizers import Tokenizer
# 1. Load custom BPE tokenizer
tokenizer = Tokenizer.from_file("amadeus_tokenizer.json")
pad_id = tokenizer.token_to_id("[PAD]") or 0
eos_id = tokenizer.token_to_id("[EOS]")
# 2. Initialize clean AmadeusRNN (vocab_size=1024, hidden_dim=256, num_layers=4)
model = AmadeusRNN(vocab_size=1024, hidden_dim=256, num_layers=4, pad_token_id=pad_id).cuda()
load_model(model, "amadeus-rnn-inference.safetensors", strict=True)
model.eval()
# 3. Generate text using O(H^2) recurrent step
prompt = "Once upon a time, there was a little"
ids = tokenizer.encode(prompt).ids
if eos_id is not None and ids[-1] == eos_id:
ids = ids[:-1]
x = torch.tensor([ids], dtype=torch.long, device="cuda")
out = model.generate(x, max_new_tokens=120, temperature=0.8, top_k=40, top_p=0.9, eos_token_id=eos_id)
print(tokenizer.decode(out[0].tolist()))