step3: toy add Config C (0.05, 50 steps)
Browse files- docs/PROGRESS.md +1 -0
- model/scripts/toy_router_test.py +9 -6
docs/PROGRESS.md
CHANGED
|
@@ -32,6 +32,7 @@ Har kaam ke baad yaha entry karo taaki yaad rahe kab kya kiya tha.
|
|
| 32 |
- [x] 2026-09-17: Step 3 start β `model/scripts/toy_router_test.py` banaya (2L tiny, bal 0.01 vs 0.02, 20 steps, verdict max<60%), local run NAHI kiya (policy), HF push β Kaggle run pending
|
| 33 |
- [x] 2026-09-17: Kaggle session-reset error (no repo after restart) -> `docs/KAGGLE_RUN.md` session-proof cell guide banaya, HF push
|
| 34 |
- [x] 2026-09-17: Step 3 toy test Kaggle PASS (A 0.01: max 0.283 loss 4.9070 / B 0.02: max 0.266 loss 4.9218, dono PASS), TEST_REPORT entry, HF push β balance_weight lock approval pending
|
|
|
|
| 35 |
- [x] 2026-09-17: `docs/ROADMAP_SCRATCH.md` banaya (Step 0-10, status: 0-2 done, agla Step 3 toy-Kaggle), HF push
|
| 36 |
- [ ] TODO: toy router-balance test (Kaggle), FINAL tokenizer (500k lines wait), dense baseline
|
| 37 |
|
|
|
|
| 32 |
- [x] 2026-09-17: Step 3 start β `model/scripts/toy_router_test.py` banaya (2L tiny, bal 0.01 vs 0.02, 20 steps, verdict max<60%), local run NAHI kiya (policy), HF push β Kaggle run pending
|
| 33 |
- [x] 2026-09-17: Kaggle session-reset error (no repo after restart) -> `docs/KAGGLE_RUN.md` session-proof cell guide banaya, HF push
|
| 34 |
- [x] 2026-09-17: Step 3 toy test Kaggle PASS (A 0.01: max 0.283 loss 4.9070 / B 0.02: max 0.266 loss 4.9218, dono PASS), TEST_REPORT entry, HF push β balance_weight lock approval pending
|
| 35 |
+
- [x] 2026-09-17: improve approved β toy me Config C (0.05, 50 steps) joda, HF push β Kaggle re-run pending
|
| 36 |
- [x] 2026-09-17: `docs/ROADMAP_SCRATCH.md` banaya (Step 0-10, status: 0-2 done, agla Step 3 toy-Kaggle), HF push
|
| 37 |
- [ ] TODO: toy router-balance test (Kaggle), FINAL tokenizer (500k lines wait), dense baseline
|
| 38 |
|
model/scripts/toy_router_test.py
CHANGED
|
@@ -3,7 +3,8 @@ Toy router-balance test β Step 3 (Kaggle, CPU enough, ~5 min)
|
|
| 3 |
- 2L tiny MoE (dim=64, 8 experts top-2) pe 2 configs compare:
|
| 4 |
A) balance_weight=0.01 (current lock candidate)
|
| 5 |
B) balance_weight=0.02 (stronger balancing)
|
| 6 |
-
- Har config:
|
|
|
|
| 7 |
- Verdict: max expert share <60% = PASS, warna FAIL (design badlo, aage mat badho)
|
| 8 |
- Report: output ko copy karke docs/TEST_REPORT.md me paste karo
|
| 9 |
|
|
@@ -110,14 +111,16 @@ def main():
|
|
| 110 |
n = (len(ids_t) // seq) * seq
|
| 111 |
blocks = ids_t[:n].view(-1, seq)
|
| 112 |
print(f"[toy] blocks={len(blocks)}")
|
| 113 |
-
ra = run_config("A", 0.01, tok, blocks)
|
| 114 |
-
rb = run_config("B", 0.02, tok, blocks)
|
|
|
|
| 115 |
print("\n--- COPY TO TEST_REPORT ---")
|
| 116 |
-
for r in (ra, rb):
|
| 117 |
print(f"- Config {r['tag']} (bal {r['bal_w']}): loss {r['loss_first']}->{r['loss_last']}, "
|
| 118 |
f"max_share {r['max_share']}, usage {r['usage']} => {r['verdict']}")
|
| 119 |
-
locked = "A (0.01)" if ra["verdict"] == "PASS" and ra["max_share"] <= rb["max_share"] else \
|
| 120 |
-
"B (0.02)" if rb["verdict"] == "PASS"
|
|
|
|
| 121 |
print(f"locked balance_weight: {locked}")
|
| 122 |
print("--- END ---")
|
| 123 |
|
|
|
|
| 3 |
- 2L tiny MoE (dim=64, 8 experts top-2) pe 2 configs compare:
|
| 4 |
A) balance_weight=0.01 (current lock candidate)
|
| 5 |
B) balance_weight=0.02 (stronger balancing)
|
| 6 |
+
- Har config: train steps on dummy Hinglish lines, phir expert usage report
|
| 7 |
+
- Configs: A) bal 0.01 / 20 steps, B) bal 0.02 / 20 steps, C) bal 0.05 / 50 steps (improve attempt)
|
| 8 |
- Verdict: max expert share <60% = PASS, warna FAIL (design badlo, aage mat badho)
|
| 9 |
- Report: output ko copy karke docs/TEST_REPORT.md me paste karo
|
| 10 |
|
|
|
|
| 111 |
n = (len(ids_t) // seq) * seq
|
| 112 |
blocks = ids_t[:n].view(-1, seq)
|
| 113 |
print(f"[toy] blocks={len(blocks)}")
|
| 114 |
+
ra = run_config("A", 0.01, tok, blocks, steps=20)
|
| 115 |
+
rb = run_config("B", 0.02, tok, blocks, steps=20)
|
| 116 |
+
rc = run_config("C", 0.05, tok, blocks, steps=50)
|
| 117 |
print("\n--- COPY TO TEST_REPORT ---")
|
| 118 |
+
for r in (ra, rb, rc):
|
| 119 |
print(f"- Config {r['tag']} (bal {r['bal_w']}): loss {r['loss_first']}->{r['loss_last']}, "
|
| 120 |
f"max_share {r['max_share']}, usage {r['usage']} => {r['verdict']}")
|
| 121 |
+
locked = "A (0.01)" if ra["verdict"] == "PASS" and ra["max_share"] <= min(rb["max_share"], rc["max_share"]) else \
|
| 122 |
+
"B (0.02)" if rb["verdict"] == "PASS" and rb["max_share"] <= rc["max_share"] else \
|
| 123 |
+
"C (0.05)" if rc["verdict"] == "PASS" else "NONE β design badlo, aage mat badho"
|
| 124 |
print(f"locked balance_weight: {locked}")
|
| 125 |
print("--- END ---")
|
| 126 |
|