Chaman1234 commited on
Commit
ce03477
·
verified ·
1 Parent(s): 8c08438

Clarify family-oldnew experiment and remove MoE claims

Browse files
Files changed (1) hide show
  1. README.md +4 -11
README.md CHANGED
@@ -11,7 +11,6 @@ tags:
11
  - code-generation
12
  - procedural-3d
13
  - bmw-memory
14
- - mixture-of-experts
15
  - safetensors
16
  ---
17
 
@@ -59,9 +58,9 @@ output_t = y_t + memory_projection(s_t)
59
 
60
  The decay is learned through a positive parameterization and is retained as part of the checkpoint. BMW is present in the published Sparse-AST-BWM models.
61
 
62
- ## Experimental family old/new expansion
63
 
64
- `family-oldnew-1m` is an experimental expansion of the 100M serial architecture. It creates a corresponding old branch and new branch at each of the 18 original serial block locations:
65
 
66
  ```text
67
  block_i*(x) = alpha_i(x) * block_i_old(x)
@@ -70,13 +69,7 @@ block_i*(x) = alpha_i(x) * block_i_old(x)
70
 
71
  At initialization, every new branch is an exact state-dict copy of its matching old branch, including weights, normalization parameters, router parameters, BMW parameters, decay values, and buffers. Therefore the two branches initially compute the same function. The old branches are frozen during the expansion experiment; new branches and permitted shared parameters are trainable.
72
 
73
- This design preserves the original serial ordering. It does not flatten serial stages into a new 12-way Top-3 router. The expansion is experimental and should not be confused with the model-level Top-K checkpoint ensemble below.
74
-
75
- ## Model-level Top-K experiment
76
-
77
- The repository's Top-K experiment routes between complete checkpoint variants, not between individual serial layers. The published configuration contains four model slots (`3M`, `10M`, `100M`, and `200M`) and uses `K=2`.
78
-
79
- An experimental three-model test using `family-oldnew`, `100M`, and `200M` with `K=3` evaluates all three models. Because `K` equals the number of active models, this is a dense logit blend, not sparse Top-K computation. It is an experiment rather than a production sparse-MoE claim.
80
 
81
  ## Evaluation
82
 
@@ -127,7 +120,7 @@ Use the original implementation and configuration with the matching checkpoint.
127
  - Generated Python may be incomplete, invalid, or semantically incorrect.
128
  - Quality depends on the variant, context length, prompt format, and Blender version.
129
  - Text-only proxy scores do not establish successful Blender execution or visual correctness.
130
- - The old/new expansion and model-level Top-K blend are research artifacts, not performance guarantees.
131
 
132
  ## Repository contents
133
 
 
11
  - code-generation
12
  - procedural-3d
13
  - bmw-memory
 
14
  - safetensors
15
  ---
16
 
 
58
 
59
  The decay is learned through a positive parameterization and is retained as part of the checkpoint. BMW is present in the published Sparse-AST-BWM models.
60
 
61
+ ## `family-oldnew-1m/` experimental expansion
62
 
63
+ `family-oldnew-1m/` is a separate experimental checkpoint based on the 100M serial architecture. It creates one legacy branch and one new branch at each of the 18 original serial block locations. The branches are paired locally; they are not interchangeable stages and are not selected by one flat router:
64
 
65
  ```text
66
  block_i*(x) = alpha_i(x) * block_i_old(x)
 
69
 
70
  At initialization, every new branch is an exact state-dict copy of its matching old branch, including weights, normalization parameters, router parameters, BMW parameters, decay values, and buffers. Therefore the two branches initially compute the same function. The old branches are frozen during the expansion experiment; new branches and permitted shared parameters are trainable.
71
 
72
+ This design preserves the exact legacy serial ordering and topology. At step zero, the new branch is initialized with an exact state-dict copy of its matching legacy branch, including BMW parameters, decay values, and buffers. The legacy branches are frozen during the experiment; new branches and permitted shared parameters are trainable. This checkpoint is experimental and does not claim a measured improvement over the published variants.
 
 
 
 
 
 
73
 
74
  ## Evaluation
75
 
 
120
  - Generated Python may be incomplete, invalid, or semantically incorrect.
121
  - Quality depends on the variant, context length, prompt format, and Blender version.
122
  - Text-only proxy scores do not establish successful Blender execution or visual correctness.
123
+ - The old/new expansion is a research artifact, not a performance guarantee.
124
 
125
  ## Repository contents
126