| <!doctype html><html lang="ja"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>More memory dimensions did not produce monotonic gains · VISTARC Research</title><meta name="description" content="Bilingual notes, measured results, and a sortable catalogue of small language model experiments."><link rel="stylesheet" href="../styles.css"><script>try{document.documentElement.lang=localStorage.getItem("research-language")||"ja"}catch(e){}</script><script defer src="../app.js"></script></head><body><header class="mast"><a class="brand" href="../index.html">AwakeningOS<span> / VISTARC RESEARCH</span></a><nav><a href="../index.html#reading"><span class="lang-ja " lang="ja">読む</span><span class="lang-en " lang="en">Read</span></a><a href="../index.html#explorer"><span class="lang-ja " lang="ja">比較表</span><span class="lang-en " lang="en">Explore</span></a><a href="../index.html#data">Data</a><div class="language"><button data-language="ja">日本語</button><button data-language="en">English</button></div></nav></header><main class="article"><a class="back" href="../index.html#reading">← <span class="lang-ja " lang="ja">研究ノート一覧</span><span class="lang-en " lang="en">All research notes</span></a><div class="eyebrow">09 / Memory</div><h1 class="lang-ja " lang="ja">記憶の次元を増やしても、性能は単調に伸びなかった</h1><h1 class="lang-en " lang="en">More memory dimensions did not produce monotonic gains</h1><p class="lang-ja standfirst" lang="ja">予測誤差のValue幅と、読み書きに使うKey幅を別々に変えると、結果も異なりました。</p><p class="lang-en standfirst" lang="en">Increasing value dimensions and increasing key dimensions had different outcomes in the F memory family.</p><p class="byline">AwakeningOS · 2026-11-02</p><section><h2 class="lang-ja " lang="ja">二つの次元は別の役割</h2><h2 class="lang-en " lang="en">Two dimensions, two roles</h2><div class="lang-ja" lang="ja"><p>Fでは、保存する予測の食い違いを表すValueと、どの状態でその情報を使うかを対応付けるKeyを使います。元の小行列は16×16です。ここでいう16は特徴の幅で、保持する過去トークン数の上限ではありません。</p><p>同じ8層の構成を土台に、Valueだけ32、Keyだけ32、両方32へ広げた試行を比較しました。行列の容量、投影の形、初期化が連動して変わるため、単なるメモリ容量だけの比較にはなりません。</p></div><div class="lang-en" lang="en"><p>F uses a value representation of prediction discrepancy and a key representation for associating information with the state in which it should be used. The original small matrix has 16×16 dimensions. Sixteen is a feature width, not a limit of sixteen past tokens.</p><p>Using the same eight-layer family, experiments increased only value width, only key width, or both to 32. Memory capacity, projection shapes, and initialization changed together.</p></div></section><section><h2 class="lang-ja " lang="ja">観測された差</h2><h2 class="lang-en " lang="en">Observed differences</h2><div class="lang-ja" lang="ja"><p>元の16×16の最高6軸は49.771。Value32・Key16は48.568、Value16・Key32は49.232、32×32は48.465でした。5軸でも、Keyだけを広げた設定は51.642で元の51.660に近く、Valueだけを広げた設定は50.358でした。</p><p>少なくともこのseedと訓練条件では、「幅を増やせば情報を残せるので強くなる」という単純な予測は外れました。読み出し側の識別と、書き込む情報の次元は分けて考える必要があります。</p></div><div class="lang-en" lang="en"><p>The 16×16 baseline reached a six-axis peak of 49.771. Value32/Key16 reached 48.568, Value16/Key32 reached 49.232, and 32×32 reached 48.465. On five axes, the key-only expansion reached 51.642, close to the baseline’s 51.660, while the value-only expansion reached 50.358.</p><p>For this seed and recipe, simply retaining more dimensions did not improve prediction. The dimensions used to store content and those used to address it have distinct roles.</p></div></section><div class="table-wrap"><table><caption><span class="lang-ja " lang="ja">採点済み時点での最高点。5軸と6軸は独立に選択。</span><span class="lang-en " lang="en">Observed peaks; five- and six-axis maxima selected independently.</span></caption><thead><tr><th>ID</th><th><span class="lang-ja " lang="ja">構造：状態幅 / FFN / 層</span><span class="lang-en " lang="en">Structure: width / FFN / layers</span></th><th>Seed</th><th><span class="lang-ja " lang="ja">5軸最高</span><span class="lang-en " lang="en">Peak five</span></th><th><span class="lang-ja " lang="ja">その時点</span><span class="lang-en " lang="en">Age</span></th><th><span class="lang-ja " lang="ja">6軸最高</span><span class="lang-en " lang="en">Peak six</span></th><th><span class="lang-ja " lang="ja">その時点</span><span class="lang-en " lang="en">Age</span></th></tr></thead><tbody><tr><td><a class="model-link" href="../index.html#model-277">#277</a></td><td>384 / FFN 1408 / 8L · F 16×16</td><td>20260907</td><td>51.660</td><td>70.0M</td><td>49.771</td><td>80.0M</td></tr><tr><td><a class="model-link" href="../index.html#model-289">#289</a></td><td>384 / FFN 1408 / 8L · F 32×16</td><td>20260907</td><td>50.358</td><td>80.0M</td><td>48.568</td><td>80.0M</td></tr><tr><td><a class="model-link" href="../index.html#model-290">#290</a></td><td>384 / FFN 1408 / 8L · F 16×32</td><td>20260907</td><td>51.642</td><td>80.0M</td><td>49.232</td><td>80.0M</td></tr><tr><td><a class="model-link" href="../index.html#model-288">#288</a></td><td>384 / FFN 1408 / 8L · F 32×32</td><td>20260907</td><td>50.806</td><td>100.0M</td><td>48.465</td><td>70.0M</td></tr><tr><td><a class="model-link" href="../index.html#model-291">#291</a></td><td>384 / FFN 1408 / 8L · F 16×16</td><td>20260907</td><td>50.684</td><td>80.0M</td><td>48.500</td><td>70.0M</td></tr><tr><td><a class="model-link" href="../index.html#model-292">#292</a></td><td>384 / FFN 1408 / 8L · F 16×16</td><td>20260907</td><td>50.592</td><td>100.0M</td><td>48.732</td><td>80.0M</td></tr><tr><td><a class="model-link" href="../index.html#model-293">#293</a></td><td>384 / FFN 1408 / 8L · F 16×16</td><td>20260907</td><td>51.124</td><td>70.0M</td><td>49.038</td><td>70.0M</td></tr></tbody></table></div><section><h2 class="lang-ja " lang="ja">まだ解けていない点</h2><h2 class="lang-en " lang="en">What remains unresolved</h2><div class="lang-ja" lang="ja"><p>この結果だけでは、次元増加そのもの、投影の学習、初期スケール、他部品との相互作用のどれが低下を生んだかは確定しません。履歴を外す版、しきい値で候補を選ぶ版、Delta型の書き換え版も台帳にありますが、同じ名称の「記憶改善」として一括りにしない方がよい結果です。</p></div><div class="lang-en" lang="en"><p>The scores do not isolate whether the loss came from dimensionality, projection learning, initial scale, or interaction with other components. History removal, candidate thresholding, and delta-style rewriting were also tested and are retained in the catalogue. They represent different interventions, not interchangeable memory improvements.</p></div></section><div class="evidence"><h2><span class="lang-ja " lang="ja">この章の根拠を調べる</span><span class="lang-en " lang="en">Inspect the evidence</span></h2><p><span class="lang-ja " lang="ja">番号から構成と実測を確認できます。元資料の識別子・ハッシュも各モデル詳細にあります。</span><span class="lang-en " lang="en">Model IDs open recorded configurations and measurements. Source identifiers and hashes are provided in the model details.</span></p><p><a class="model-link" href="../index.html#model-277">#277</a> <a class="model-link" href="../index.html#model-288">#288</a> <a class="model-link" href="../index.html#model-289">#289</a> <a class="model-link" href="../index.html#model-290">#290</a> <a class="model-link" href="../index.html#model-291">#291</a> <a class="model-link" href="../index.html#model-292">#292</a> <a class="model-link" href="../index.html#model-293">#293</a></p></div><nav class="chapter-nav"><a href="lr-groups.html">← <span class="lang-ja " lang="ja">学習率は部品ごとに記録し、変更の組み合わせで読む</span><span class="lang-en " lang="en">Read learning-rate changes as named, coupled settings</span></a><a href="memory-use.html"><span class="lang-ja " lang="ja">記憶を作れたことと、予測に役立ったことを分ける</span><span class="lang-en " lang="en">Storing information and using it profitably are different tests</span> →</a></nav></main><footer><p>AwakeningOS · 2026-11-02 snapshot · Apache-2.0</p><p><span class="lang-ja " lang="ja">実験台帳にある観測の共有です。コンテストの公式順位ではありません。</span><span class="lang-en " lang="en">An observational research notebook, not an official contest leaderboard.</span></p><a href="../data/manifest.json">Snapshot manifest</a> · <a href="https://huggingface.co/AwakeningOS/vistarc-320-9l-mos3-80m">Released model</a> · <a href="https://huggingface.co/spaces/AwakeningOS/vistarc-research-notes/tree/main">Files & history</a></footer></body></html> |
