Clarify family-oldnew experiment and remove MoE claims
Browse files
README.md
CHANGED
|
@@ -11,7 +11,6 @@ tags:
|
|
| 11 |
- code-generation
|
| 12 |
- procedural-3d
|
| 13 |
- bmw-memory
|
| 14 |
-
- mixture-of-experts
|
| 15 |
- safetensors
|
| 16 |
---
|
| 17 |
|
|
@@ -59,9 +58,9 @@ output_t = y_t + memory_projection(s_t)
|
|
| 59 |
|
| 60 |
The decay is learned through a positive parameterization and is retained as part of the checkpoint. BMW is present in the published Sparse-AST-BWM models.
|
| 61 |
|
| 62 |
-
##
|
| 63 |
|
| 64 |
-
`family-oldnew-1m` is
|
| 65 |
|
| 66 |
```text
|
| 67 |
block_i*(x) = alpha_i(x) * block_i_old(x)
|
|
@@ -70,13 +69,7 @@ block_i*(x) = alpha_i(x) * block_i_old(x)
|
|
| 70 |
|
| 71 |
At initialization, every new branch is an exact state-dict copy of its matching old branch, including weights, normalization parameters, router parameters, BMW parameters, decay values, and buffers. Therefore the two branches initially compute the same function. The old branches are frozen during the expansion experiment; new branches and permitted shared parameters are trainable.
|
| 72 |
|
| 73 |
-
This design preserves the
|
| 74 |
-
|
| 75 |
-
## Model-level Top-K experiment
|
| 76 |
-
|
| 77 |
-
The repository's Top-K experiment routes between complete checkpoint variants, not between individual serial layers. The published configuration contains four model slots (`3M`, `10M`, `100M`, and `200M`) and uses `K=2`.
|
| 78 |
-
|
| 79 |
-
An experimental three-model test using `family-oldnew`, `100M`, and `200M` with `K=3` evaluates all three models. Because `K` equals the number of active models, this is a dense logit blend, not sparse Top-K computation. It is an experiment rather than a production sparse-MoE claim.
|
| 80 |
|
| 81 |
## Evaluation
|
| 82 |
|
|
@@ -127,7 +120,7 @@ Use the original implementation and configuration with the matching checkpoint.
|
|
| 127 |
- Generated Python may be incomplete, invalid, or semantically incorrect.
|
| 128 |
- Quality depends on the variant, context length, prompt format, and Blender version.
|
| 129 |
- Text-only proxy scores do not establish successful Blender execution or visual correctness.
|
| 130 |
-
- The old/new expansion
|
| 131 |
|
| 132 |
## Repository contents
|
| 133 |
|
|
|
|
| 11 |
- code-generation
|
| 12 |
- procedural-3d
|
| 13 |
- bmw-memory
|
|
|
|
| 14 |
- safetensors
|
| 15 |
---
|
| 16 |
|
|
|
|
| 58 |
|
| 59 |
The decay is learned through a positive parameterization and is retained as part of the checkpoint. BMW is present in the published Sparse-AST-BWM models.
|
| 60 |
|
| 61 |
+
## `family-oldnew-1m/` experimental expansion
|
| 62 |
|
| 63 |
+
`family-oldnew-1m/` is a separate experimental checkpoint based on the 100M serial architecture. It creates one legacy branch and one new branch at each of the 18 original serial block locations. The branches are paired locally; they are not interchangeable stages and are not selected by one flat router:
|
| 64 |
|
| 65 |
```text
|
| 66 |
block_i*(x) = alpha_i(x) * block_i_old(x)
|
|
|
|
| 69 |
|
| 70 |
At initialization, every new branch is an exact state-dict copy of its matching old branch, including weights, normalization parameters, router parameters, BMW parameters, decay values, and buffers. Therefore the two branches initially compute the same function. The old branches are frozen during the expansion experiment; new branches and permitted shared parameters are trainable.
|
| 71 |
|
| 72 |
+
This design preserves the exact legacy serial ordering and topology. At step zero, the new branch is initialized with an exact state-dict copy of its matching legacy branch, including BMW parameters, decay values, and buffers. The legacy branches are frozen during the experiment; new branches and permitted shared parameters are trainable. This checkpoint is experimental and does not claim a measured improvement over the published variants.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
|
| 74 |
## Evaluation
|
| 75 |
|
|
|
|
| 120 |
- Generated Python may be incomplete, invalid, or semantically incorrect.
|
| 121 |
- Quality depends on the variant, context length, prompt format, and Blender version.
|
| 122 |
- Text-only proxy scores do not establish successful Blender execution or visual correctness.
|
| 123 |
+
- The old/new expansion is a research artifact, not a performance guarantee.
|
| 124 |
|
| 125 |
## Repository contents
|
| 126 |
|