Nano-AMADEUS-RNN4LDFA-T1024H256-AV-1001

Amadeus-RNN (Post-Backprop Edition) is a lightweight, 4-layer Parallel Linear RNN with data-dependent gating and associative prefix scan, trained without end-to-end global backpropagation.

Instead of standard BPTT across stacked layers, this model combines three biologically inspired local learning mechanisms:

  1. Residual-Identity Direct Feedback Alignment (DFARouter): Broadcasts top-level output error directly to detached lower blocks via learnable residual bridges ((I_H + B_{\text{dfa}})).
  2. Inter-Layer & Intra-Block Kolen-Pollack Co-Adaptation (B_dfa, B_mlp): Aligns feedback matrices with forward transformations using outer-product updates and weight decay without weight transport.
  3. Associative Eligibility Propagation (AssociativeEpropFn): Computes local recurrent parameter updates in O(BTH) memory without unrolling (\log_2(T)) scan steps in the global autograd graph.

Model Specifications

  • Architecture: 4-Layer Stacked AmadeusBlock (Pre-RMSNorm, Data-Dependent Gating, Parallel Associative Scan, SwiGLU-style SiLU MLP, Residual Scaling)
  • Clean Inference Parameters: 1,842,433 (~7.37 MB in amadeus-rnn-inference.safetensors)
  • Training Parameters (with KP bridges): 2,301,185 (~9.21 MB in amadeus-rnn-dfa-eprop.safetensors)
  • Hidden Dimension (H): 256
  • Vocabulary Size (V): 1,024 (Custom BPE Tokenizer in amadeus_tokenizer.json)
  • Context Length (T): 512
  • Training Data: 1 Epoch on TinyStories corpus

Holdout Evaluation Benchmark (10,000 Unseen Stories)

Evaluated across 2,427,680 valid tokens (seq_len=512) and compared directly against the standard end-to-end backprop baseline (RinKana/Nano-AMADEUS-RNN4L-T1024H256-AV-1001):

Metric Backprop Baseline Post-Backprop (DFA + KP + E-Prop) Difference
Inference Parameters 1,842,433 1,842,433 Identical
Token-Weighted CE Loss 1.4304 1.7677 +0.3373
Corpus Perplexity (PPL) 4.18 5.86 +1.68
Top-1 Token Accuracy 62.95% 56.40% -6.55%
Top-5 Token Accuracy 86.32% 81.40% -4.92%
Generation Speed (T4 GPU) ~415–435 tok/s ~421–426 tok/s Identical

Final Gradient Alignment Diagnostics (Step 21,000)

  • Global Cosine Alignment vs. True BPTT: +0.4821 (61.2Β°)
  • Layer-wise Breakdown: L0[Rec:+0.346 | MLP:+0.414], L1[Rec:+0.442 | MLP:+0.592], L2[Rec:+0.520 | MLP:+0.746], L3[Rec:+0.828 | MLP:+1.000]

Repository Files

  • amadeus-rnn-inference.safetensors (7.37 MB): Stripped inference weights (B_dfa and B_mlp removed) compatible with the clean AmadeusRNN inference class (strict=True).
  • amadeus-rnn-dfa-eprop.safetensors (9.21 MB): Full training checkpoint including Kolen-Pollack feedback matrices for resuming training.
  • amadeus_tokenizer.json (54.5 kB): Custom 1,024-token BPE tokenizer.

Sample Output

Prompt: "One day, a clever fox found a shiny"

"One day, a clever fox found a shiny thing. The fox was very happy. It wanted to play with the fox. It met a big dog. The dog said, "Hi, I am a dog. Do you want to play with me?" The dog wagged its tail and said, "Yes, I want to play with you.""


Quick Usage (Inference)

import torch
from safetensors.torch import load_model
from tokenizers import Tokenizer

# 1. Load custom BPE tokenizer
tokenizer = Tokenizer.from_file("amadeus_tokenizer.json")
pad_id = tokenizer.token_to_id("[PAD]") or 0
eos_id = tokenizer.token_to_id("[EOS]")

# 2. Initialize clean AmadeusRNN (vocab_size=1024, hidden_dim=256, num_layers=4)
model = AmadeusRNN(vocab_size=1024, hidden_dim=256, num_layers=4, pad_token_id=pad_id).cuda()
load_model(model, "amadeus-rnn-inference.safetensors", strict=True)
model.eval()

# 3. Generate text using O(H^2) recurrent step
prompt = "Once upon a time, there was a little"
ids = tokenizer.encode(prompt).ids
if eos_id is not None and ids[-1] == eos_id:
    ids = ids[:-1]

x = torch.tensor([ids], dtype=torch.long, device="cuda")
out = model.generate(x, max_new_tokens=120, temperature=0.8, top_k=40, top_p=0.9, eos_token_id=eos_id)
print(tokenizer.decode(out[0].tolist()))
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Dataset used to train RinKana/Nano-AMADEUS-RNN4LDFA-T1024H256-AV-1001