Sencelium/sencelium-125M
Text Generation • Updated • 5 • 1
AI & ML interests
non-Transformer architectures, recurrent memory, interpretability
Recent Activity
Organization Card
Independent research project building a non-Transformer language model: gated-delta-rule blocks, a content-blind "nucleus" channel, a memory pool that keeps spawning, merging and reorganizing itself during training and at inference, live self-monitoring signals, and an ESN residual path.
No fluff tiers, no hype benchmarks — honest results, including where a plain Transformer still wins.
The project doesn't lead with a PPL win over Transformers — it isn't one, at these scales. What it reports, including the parts that did not turn out as hoped: the memory is load-bearing but its benefit is structural, not content-specific, and perplexity keeps falling as the context grows well past the trained length, where a plain Transformer's rises.
Full findings, methodology and numbers: see each model's card below, or sencelium.com/results · benchmarks/results.md.
Models
Non-standard architecture — not AutoModel-compatible. Each model card
has exact instructions for loading via the training/eval code in the
GitHub repo.
Links
- 💻 Code (Apache 2.0): github.com/vitaliihalak/sencelium
- 🌐 Website: sencelium.com
- 🧭 Architecture write-up: sencelium.com/architecture
- 📜 License: Apache 2.0 (code & weights) · CC BY 4.0 (paper & text)
- 🔬 Research log: Laboratory of Absurd Ideas
- 💬 Contact: info@sencelium.com
datasets 0
None public yet