summer
Publish experimental merged bilingual LoopedQwen + GraphEngram snapshot with evaluation and reload validation
5c270fa verified | license: apache-2.0 | |
| library_name: transformers | |
| pipeline_tag: text-generation | |
| base_model: summerMC/Looped-Qwen-pre | |
| language: | |
| - ja | |
| - en | |
| tags: | |
| - experimental | |
| - custom_code | |
| - graph-neural-network | |
| - graph-memory | |
| - looped-transformer | |
| - research | |
| <div align="center"> | |
| <img src="./assets/header.svg" alt="LoopedQwen × GraphEngram" width="100%" /> | |
| <br /> | |
| <img src="./assets/tags.svg" alt="Experimental · JA + EN · 24 loops · 48,491 nodes · 13.63M memory parameters · custom code" width="100%" /> | |
| <br /><br /> | |
| <strong>Looped computation meets learned graph memory.</strong><br /> | |
| <sub>A self-contained bilingual research snapshot by summerMC.</sub> | |
| <br /><br /> | |
| <a href="#architecture">Architecture</a> · <a href="#loading">Quickstart</a> · <a href="#evaluation">Evaluation</a> · <a href="#limitations">Limitations</a> | |
| </div> | |
| --- | |
| > **Research prototype, NOT a production chatbot.** Conversation, arithmetic and code-generation tests remain poor, often producing unrelated numbers. Do not interpret lower language-model loss as general reasoning or instruction-following quality. | |
| This repository contains a consolidated snapshot of a repaired LoopedQwen backbone, learned graph-memory modules, and the matching token graph. No separate base-model or adapter download is needed when loading this package. The graph module cannot be algebraically folded into a dense weight matrix: “merged” means a self-contained custom architecture, not a LoRA-only merge. | |
| GraphEngram is an independent experimental graph-attention mechanism, not the official DeepSeek Engram implementation or a reproduction of its paper results. | |
| ## Architecture | |
| <img src="./assets/architecture.svg" alt="Causal routing, learned graph attention and LoopedQwen fusion" width="100%" /> | |
| - Pinned base: `summerMC/Looped-Qwen-pre`, revision `eaa834598f3e6b479d1f72b51b6933422b9b8a9c`. | |
| - Six shared GDN groups, four inner steps each: 24 effective steps, hidden size 2048. | |
| - Prior backbone adapters are consolidated into the exported backbone parameters. | |
| - Memory inserted before outer groups 0 and 3. | |
| - 48,491 shared token nodes and 88,076 directed adjacency edges. | |
| - Node width 128; 13,626,882 learned memory parameters. | |
| - Causal exact token bigram/trigram routing, up to 8 candidates, one-hop graph attention with degree cap 8. | |
| - Graph topology is fixed; node embeddings, attention projections and fusion gates are learned. | |
| - Graph construction uses training documents only. Old English node IDs and weights were preserved during Japanese expansion. | |
| ## Loading | |
| Use a recent compatible Transformers environment. Export/evaluation used PyTorch 2.11.0 and Transformers 5.20.2. `flash-linear-attention` is optional acceleration; backend differences can affect numerical outputs. | |
| ```python | |
| import torch | |
| from transformers import AutoTokenizer, AutoModelForCausalLM | |
| repo = "summerMC/LoopedQwen-GraphEngram-ja-en" | |
| tok = AutoTokenizer.from_pretrained(repo) | |
| model = AutoModelForCausalLM.from_pretrained( | |
| repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda" | |
| ).eval() | |
| ids = tok.apply_chat_template( | |
| [{"role": "user", "content": "こんにちは。"}], | |
| add_generation_prompt=True, enable_thinking=False, | |
| return_tensors="pt", return_dict=True | |
| )["input_ids"].to("cuda") | |
| out = model.generate(ids, max_new_tokens=32, do_sample=False, use_cache=False) | |
| print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)) | |
| ``` | |
| Inspect custom code before enabling `trust_remote_code`. Generation is greedy, unpadded batch size 1 only. Cached decoding, concurrent forwards and production serving integrations are unsupported. Each output token recomputes the prefix. The graph checksum is checked during loading. Keep `graph.json` with the weights and configuration. | |
| Toggle memory for ablations: `model.network.enabled = False`. | |
| ## Training provenance | |
| - English source: `HuggingFaceFW/fineweb`, `sample-10BT`, revision `9bb295ddab0e05d785b879661af7260fed5140fc`. | |
| - Japanese source: `HuggingFaceFW/fineweb-2`, `jpn_Jpan`, revision `af9c13333eb981300149d5ca60a8e9d659b276b9`. | |
| - Dataset cards identify ODC-BY. Check dataset/source-content terms for downstream redistribution and use. The token graph itself retains training-text fragments; it is not a privacy-preserving representation. | |
| - Whole-document hashing separates train/dev/test before truncation. No near-duplicate or base-pretraining contamination audit was performed. | |
| - Base was frozen during graph training. The immediate bilingual continuation trained 36,864 Japanese + 36,864 English tokens, not the entire downloaded corpus. | |
| - Earlier graph stages and the repaired base are described in the included reports/design document. | |
| - Parent optimizer was not retained for the first bilingual continuation; weights were resumed with a fresh optimizer. | |
| ## Evaluation | |
| Standalone reloaded bilingual snapshot, eight held-out documents per language, full documents evaluated in independent chunks: | |
| | Corpus | Target tokens | GNN off NLL | GNN on NLL | | |
| |---|---:|---:|---:| | |
| | Japanese | 4,895 | 4.5203 | 4.4736 | | |
| | English | 4,912 | 4.0961 | 4.0122 | | |
| Eight additional development prompts: math 0/2, required code functions 0/2, extraction 1/2 for both on/off. Conversation remains inappropriate/repetitive. Generated code was not executed. These are small diagnostics, not standardized benchmark or Opus-equivalence results. | |
| ## Limitations | |
| - Weak instruction following and numerical-answer bias in the repaired backbone. | |
| - No established improvement in Japanese/English conversation, math or code generation. | |
| - No demonstrated learning-speed advantage over official Engram, RAG or ordinary fine-tuning at matched quality. | |
| - No proven compression advantage or lossless reconstruction of the complete source corpus from the model graph. | |
| - No billion/trillion-token context guarantee; lexical routing has limited recall and CPU overhead. | |
| - Frozen random-initialized new node vectors require learning; storing a text node is not equivalent to learning its facts. | |
| - Source text may contain errors, personal information or malicious instructions. No dedicated filtering/safety training was performed for this prototype. | |
| The model is published for reproducibility and architecture experiments, not capability claims. | |