| <!doctype html><html lang="ja"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>A single-run record can hide substantial seed variation · VISTARC Research</title><meta name="description" content="Bilingual notes, measured results, and a sortable catalogue of small language model experiments."><link rel="stylesheet" href="../styles.css"><script>try{document.documentElement.lang=localStorage.getItem("research-language")||"ja"}catch(e){}</script><script defer src="../app.js"></script></head><body><header class="mast"><a class="brand" href="../index.html">AwakeningOS<span> / VISTARC RESEARCH</span></a><nav><a href="../index.html#reading"><span class="lang-ja " lang="ja">読む</span><span class="lang-en " lang="en">Read</span></a><a href="../index.html#explorer"><span class="lang-ja " lang="ja">比較表</span><span class="lang-en " lang="en">Explore</span></a><a href="../index.html#data">Data</a><div class="language"><button data-language="ja">日本語</button><button data-language="en">English</button></div></nav></header><main class="article"><a class="back" href="../index.html#reading">← <span class="lang-ja " lang="ja">研究ノート一覧</span><span class="lang-en " lang="en">All research notes</span></a><div class="eyebrow">03 / Reliability</div><h1 class="lang-ja " lang="ja">単発の最高点と、seedを変えた強さは一致しない</h1><h1 class="lang-en " lang="en">A single-run record can hide substantial seed variation</h1><p class="lang-ja standfirst" lang="ja">320幅9層とF付き384幅8層を同じ三つのseedで比べると、最高点と平均・ばらつきで見え方が変わりました。</p><p class="lang-en standfirst" lang="en">Matched three-seed comparisons of 320-wide and 384-wide configurations change the interpretation of their best runs.</p><p class="byline">AwakeningOS · 2026-11-02</p><section><h2 class="lang-ja " lang="ja">比較したもの</h2><h2 class="lang-en " lang="en">The comparison</h2><div class="lang-ja" lang="ja"><p>比較したのは、320幅・FFN1120・9層のVISTARCと、384幅・FFN1408・8層に予測誤差の行列記憶Fを加えた構成です。各構成を20260907、731946281、137の三つのseedで学習し、同じ80M地点を同一の明確回答ベンチで採点しました。</p><p>幅、深さ、Fの有無、学習率に違いがあるため、これは構成全体の比較です。F一部品の効果だけを分離した実験ではありません。</p></div><div class="lang-en" lang="en"><p>The comparison uses a 320-wide, FFN1120, nine-layer VISTARC and a 384-wide, FFN1408, eight-layer configuration with F prediction-error memory. Both were trained with seeds 20260907, 731946281, and 137, then evaluated at the same 80M age on the same clear-answer bank.</p><p>Width, depth, F, and learning rates differ. This compares complete configurations rather than isolating the causal effect of F.</p></div></section><section><h2 class="lang-ja " lang="ja">観測されたこと</h2><h2 class="lang-en " lang="en">What happened</h2><div class="lang-ja" lang="ja"><p>最初のseedではF付き384幅が73.237％で、320幅の71.026％を上回りました。しかし二つ目のseedではほぼ並び、三つ目ではF付きが69.393％、320幅が71.628％となりました。</p><p>三つのseedの平均は320幅が71.076％、F付き384幅が71.030％です。差は約0.046ポイント。一方、標本標準偏差はそれぞれ約0.529と1.985ポイントでした。最良の一回と、平均およびばらつきは別々に評価する必要がある結果です。</p></div><div class="lang-en" lang="en"><p>At the first seed, the F-equipped 384-wide configuration scored 73.237%, above the 320-wide model’s 71.026%. The second seed was nearly tied. At the third, the F-equipped model scored 69.393% and the 320-wide model 71.628%.</p><p>The three-seed means were 71.076% and 71.030%, a difference of about 0.046 percentage points. Their sample standard deviations were about 0.529 and 1.985 points, respectively. Best-run performance and repeatability tell different stories here.</p></div></section><div class="table-wrap"><table><caption><span class="lang-ja " lang="ja">80M・同じ三seed・同じ問題集。明確回答は各seed17,274問。</span><span class="lang-en " lang="en">Matched 80M, three seeds, 17,274 clear-answer items per seed.</span></caption><thead><tr><th><span class="lang-ja " lang="ja">構造</span><span class="lang-en " lang="en">Architecture</span></th><th>IDs</th><th>20260907</th><th>731946281</th><th>137</th><th><span class="lang-ja " lang="ja">明確回答 平均</span><span class="lang-en " lang="en">Clear mean</span></th><th><span class="lang-ja " lang="ja">標本SD</span><span class="lang-en " lang="en">Sample SD</span></th><th><span class="lang-ja " lang="ja">5軸 平均</span><span class="lang-en " lang="en">Five mean</span></th><th><span class="lang-ja " lang="ja">6軸 平均</span><span class="lang-en " lang="en">Six mean</span></th></tr></thead><tbody><tr><td><span class="lang-ja " lang="ja">320・9層・FFN1120</span><span class="lang-en " lang="en">320 / 9L / FFN1120</span></td><td><a class="model-link" href="../index.html#model-1">#1</a>, <a class="model-link" href="../index.html#model-81">#81</a>, <a class="model-link" href="../index.html#model-26">#26</a></td><td>71.026</td><td>70.574</td><td>71.628</td><td>71.076</td><td>0.529</td><td>51.123</td><td>48.690</td></tr><tr><td><span class="lang-ja " lang="ja">384・8層・FFN1408・F16×16</span><span class="lang-en " lang="en">384 / 8L / FFN1408 / F16×16</span></td><td><a class="model-link" href="../index.html#model-277">#277</a>, <a class="model-link" href="../index.html#model-294">#294</a>, <a class="model-link" href="../index.html#model-302">#302</a></td><td>73.237</td><td>70.458</td><td>69.393</td><td>71.030</td><td>1.985</td><td>51.103</td><td>49.220</td></tr></tbody></table></div><section><h2 class="lang-ja " lang="ja">共有したい知見</h2><h2 class="lang-en " lang="en">The useful lesson</h2><div class="lang-ja" lang="ja"><p>最高点だけから「この構造は安定して強い」と決めると、次のseedへの期待を誤ります。この比較では、384幅の単発記録は残したまま、320幅の平均の近さと小さな観測ばらつきも報告するのが適切です。</p><p>三つのseedは一般的な頑健性を確定するには少数です。この結果からあらゆる320幅が安定する、または384幅が不安定になるという法則までは作れません。</p></div><div class="lang-en" lang="en"><p>Selecting only the best run can produce misleading expectations for the next seed. The 384-wide single-run record remains valid, while the 320-wide configuration’s similar mean and smaller observed spread must also be reported.</p><p>Three seeds are still a small sample. This does not establish that all 320-wide models are stable or all 384-wide models are unstable.</p></div></section><div class="evidence"><h2><span class="lang-ja " lang="ja">この章の根拠を調べる</span><span class="lang-en " lang="en">Inspect the evidence</span></h2><p><span class="lang-ja " lang="ja">番号から構成と実測を確認できます。元資料の識別子・ハッシュも各モデル詳細にあります。</span><span class="lang-en " lang="en">Model IDs open recorded configurations and measurements. Source identifiers and hashes are provided in the model details.</span></p><p><a class="model-link" href="../index.html#model-1">#1</a> <a class="model-link" href="../index.html#model-81">#81</a> <a class="model-link" href="../index.html#model-26">#26</a> <a class="model-link" href="../index.html#model-277">#277</a> <a class="model-link" href="../index.html#model-294">#294</a> <a class="model-link" href="../index.html#model-302">#302</a></p></div><nav class="chapter-nav"><a href="metrics.html">← <span class="lang-ja " lang="ja">点数と比較条件をどう読むか</span><span class="lang-en " lang="en">How to read scores and comparisons</span></a><a href="width-lr.html"><span class="lang-ja " lang="ja">幅を変えた結果には、実際の学習率も混ざっている</span><span class="lang-en " lang="en">Width comparisons also depend on the effective learning rate</span> →</a></nav></main><footer><p>AwakeningOS · 2026-11-02 snapshot · Apache-2.0</p><p><span class="lang-ja " lang="ja">実験台帳にある観測の共有です。コンテストの公式順位ではありません。</span><span class="lang-en " lang="en">An observational research notebook, not an official contest leaderboard.</span></p><a href="../data/manifest.json">Snapshot manifest</a> · <a href="https://huggingface.co/AwakeningOS/vistarc-320-9l-mos3-80m">Released model</a> · <a href="https://huggingface.co/spaces/AwakeningOS/vistarc-research-notes/tree/main">Files & history</a></footer></body></html> |
