Looped computation meets learned graph memory.
A self-contained bilingual research snapshot by summerMC.
Architecture · Quickstart · Evaluation · Limitations
- Pinned base: `summerMC/Looped-Qwen-pre`, revision `eaa834598f3e6b479d1f72b51b6933422b9b8a9c`.
- Six shared GDN groups, four inner steps each: 24 effective steps, hidden size 2048.
- Prior backbone adapters are consolidated into the exported backbone parameters.
- Memory inserted before outer groups 0 and 3.
- 48,491 shared token nodes and 88,076 directed adjacency edges.
- Node width 128; 13,626,882 learned memory parameters.
- Causal exact token bigram/trigram routing, up to 8 candidates, one-hop graph attention with degree cap 8.
- Graph topology is fixed; node embeddings, attention projections and fusion gates are learned.
- Graph construction uses training documents only. Old English node IDs and weights were preserved during Japanese expansion.
## Loading
Use a recent compatible Transformers environment. Export/evaluation used PyTorch 2.11.0 and Transformers 5.20.2. `flash-linear-attention` is optional acceleration; backend differences can affect numerical outputs.
```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
repo = "summerMC/LoopedQwen-GraphEngram-ja-en"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda"
).eval()
ids = tok.apply_chat_template(
[{"role": "user", "content": "こんにちは。"}],
add_generation_prompt=True, enable_thinking=False,
return_tensors="pt", return_dict=True
)["input_ids"].to("cuda")
out = model.generate(ids, max_new_tokens=32, do_sample=False, use_cache=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
```
Inspect custom code before enabling `trust_remote_code`. Generation is greedy, unpadded batch size 1 only. Cached decoding, concurrent forwards and production serving integrations are unsupported. Each output token recomputes the prefix. The graph checksum is checked during loading. Keep `graph.json` with the weights and configuration.
Toggle memory for ablations: `model.network.enabled = False`.
## Training provenance
- English source: `HuggingFaceFW/fineweb`, `sample-10BT`, revision `9bb295ddab0e05d785b879661af7260fed5140fc`.
- Japanese source: `HuggingFaceFW/fineweb-2`, `jpn_Jpan`, revision `af9c13333eb981300149d5ca60a8e9d659b276b9`.
- Dataset cards identify ODC-BY. Check dataset/source-content terms for downstream redistribution and use. The token graph itself retains training-text fragments; it is not a privacy-preserving representation.
- Whole-document hashing separates train/dev/test before truncation. No near-duplicate or base-pretraining contamination audit was performed.
- Base was frozen during graph training. The immediate bilingual continuation trained 36,864 Japanese + 36,864 English tokens, not the entire downloaded corpus.
- Earlier graph stages and the repaired base are described in the included reports/design document.
- Parent optimizer was not retained for the first bilingual continuation; weights were resumed with a fresh optimizer.
## Evaluation
Standalone reloaded bilingual snapshot, eight held-out documents per language, full documents evaluated in independent chunks:
| Corpus | Target tokens | GNN off NLL | GNN on NLL |
|---|---:|---:|---:|
| Japanese | 4,895 | 4.5203 | 4.4736 |
| English | 4,912 | 4.0961 | 4.0122 |
Eight additional development prompts: math 0/2, required code functions 0/2, extraction 1/2 for both on/off. Conversation remains inappropriate/repetitive. Generated code was not executed. These are small diagnostics, not standardized benchmark or Opus-equivalence results.
## Limitations
- Weak instruction following and numerical-answer bias in the repaired backbone.
- No established improvement in Japanese/English conversation, math or code generation.
- No demonstrated learning-speed advantage over official Engram, RAG or ordinary fine-tuning at matched quality.
- No proven compression advantage or lossless reconstruction of the complete source corpus from the model graph.
- No billion/trillion-token context guarantee; lexical routing has limited recall and CPU overhead.
- Frozen random-initialized new node vectors require learning; storing a text node is not equivalent to learning its facts.
- Source text may contain errors, personal information or malicious instructions. No dedicated filtering/safety training was performed for this prototype.
The model is published for reproducibility and architecture experiments, not capability claims.