TheInvestigator/opera-lm-chat

OPERA-LM: an attention-free, byte-level language model. Bottom-up composition over a binary (Fenwick) tree replaces self-attention, there is no positional encoding, and in-context recall comes from a gated quaternion holographic memory. Code: https://github.com/Merna-Khalid/OPERA-LM

• parameters: 171,573,003
• width 3072, 2 layers, 768 memory slots per layer
• pretrained on FineWeb-Edu bytes, chat-tuned on smol-smoltalk

Pretraining evaluation (held-out, bits per byte):

• fineweb_bpb: 1.2728 BPB
• a20k: 1.4366 BPB
• pool: 1.6054 BPB
• new: 1.6399 BPB
• long_1025_2048: 1.7203 BPB
• fineweb: 1.2728 BPB
• fineweb_long_1025_2048: 1.2301 BPB

from opera_lm.chat import load_model, format_prompt, stream_generate
model, cfg = load_model("TheInvestigator/opera-lm-chat")
for text in stream_generate(model, format_prompt([], "Hello! Who are you?")):
    pass
print(text)

Small research model: expect fluent-looking text, not reliable facts.