ViuAI commited on
Commit
bda5ab6
Β·
verified Β·
1 Parent(s): f13474b

step3: toy add Config C (0.05, 50 steps)

Browse files
docs/PROGRESS.md CHANGED
@@ -32,6 +32,7 @@ Har kaam ke baad yaha entry karo taaki yaad rahe kab kya kiya tha.
32
  - [x] 2026-09-17: Step 3 start β€” `model/scripts/toy_router_test.py` banaya (2L tiny, bal 0.01 vs 0.02, 20 steps, verdict max<60%), local run NAHI kiya (policy), HF push β€” Kaggle run pending
33
  - [x] 2026-09-17: Kaggle session-reset error (no repo after restart) -> `docs/KAGGLE_RUN.md` session-proof cell guide banaya, HF push
34
  - [x] 2026-09-17: Step 3 toy test Kaggle PASS (A 0.01: max 0.283 loss 4.9070 / B 0.02: max 0.266 loss 4.9218, dono PASS), TEST_REPORT entry, HF push β€” balance_weight lock approval pending
 
35
  - [x] 2026-09-17: `docs/ROADMAP_SCRATCH.md` banaya (Step 0-10, status: 0-2 done, agla Step 3 toy-Kaggle), HF push
36
  - [ ] TODO: toy router-balance test (Kaggle), FINAL tokenizer (500k lines wait), dense baseline
37
 
 
32
  - [x] 2026-09-17: Step 3 start β€” `model/scripts/toy_router_test.py` banaya (2L tiny, bal 0.01 vs 0.02, 20 steps, verdict max<60%), local run NAHI kiya (policy), HF push β€” Kaggle run pending
33
  - [x] 2026-09-17: Kaggle session-reset error (no repo after restart) -> `docs/KAGGLE_RUN.md` session-proof cell guide banaya, HF push
34
  - [x] 2026-09-17: Step 3 toy test Kaggle PASS (A 0.01: max 0.283 loss 4.9070 / B 0.02: max 0.266 loss 4.9218, dono PASS), TEST_REPORT entry, HF push β€” balance_weight lock approval pending
35
+ - [x] 2026-09-17: improve approved β€” toy me Config C (0.05, 50 steps) joda, HF push β€” Kaggle re-run pending
36
  - [x] 2026-09-17: `docs/ROADMAP_SCRATCH.md` banaya (Step 0-10, status: 0-2 done, agla Step 3 toy-Kaggle), HF push
37
  - [ ] TODO: toy router-balance test (Kaggle), FINAL tokenizer (500k lines wait), dense baseline
38
 
model/scripts/toy_router_test.py CHANGED
@@ -3,7 +3,8 @@ Toy router-balance test β€” Step 3 (Kaggle, CPU enough, ~5 min)
3
  - 2L tiny MoE (dim=64, 8 experts top-2) pe 2 configs compare:
4
  A) balance_weight=0.01 (current lock candidate)
5
  B) balance_weight=0.02 (stronger balancing)
6
- - Har config: 20 train steps on dummy Hinglish lines, phir expert usage report
 
7
  - Verdict: max expert share <60% = PASS, warna FAIL (design badlo, aage mat badho)
8
  - Report: output ko copy karke docs/TEST_REPORT.md me paste karo
9
 
@@ -110,14 +111,16 @@ def main():
110
  n = (len(ids_t) // seq) * seq
111
  blocks = ids_t[:n].view(-1, seq)
112
  print(f"[toy] blocks={len(blocks)}")
113
- ra = run_config("A", 0.01, tok, blocks)
114
- rb = run_config("B", 0.02, tok, blocks)
 
115
  print("\n--- COPY TO TEST_REPORT ---")
116
- for r in (ra, rb):
117
  print(f"- Config {r['tag']} (bal {r['bal_w']}): loss {r['loss_first']}->{r['loss_last']}, "
118
  f"max_share {r['max_share']}, usage {r['usage']} => {r['verdict']}")
119
- locked = "A (0.01)" if ra["verdict"] == "PASS" and ra["max_share"] <= rb["max_share"] else \
120
- "B (0.02)" if rb["verdict"] == "PASS" else "NONE β€” design badlo, aage mat badho"
 
121
  print(f"locked balance_weight: {locked}")
122
  print("--- END ---")
123
 
 
3
  - 2L tiny MoE (dim=64, 8 experts top-2) pe 2 configs compare:
4
  A) balance_weight=0.01 (current lock candidate)
5
  B) balance_weight=0.02 (stronger balancing)
6
+ - Har config: train steps on dummy Hinglish lines, phir expert usage report
7
+ - Configs: A) bal 0.01 / 20 steps, B) bal 0.02 / 20 steps, C) bal 0.05 / 50 steps (improve attempt)
8
  - Verdict: max expert share <60% = PASS, warna FAIL (design badlo, aage mat badho)
9
  - Report: output ko copy karke docs/TEST_REPORT.md me paste karo
10
 
 
111
  n = (len(ids_t) // seq) * seq
112
  blocks = ids_t[:n].view(-1, seq)
113
  print(f"[toy] blocks={len(blocks)}")
114
+ ra = run_config("A", 0.01, tok, blocks, steps=20)
115
+ rb = run_config("B", 0.02, tok, blocks, steps=20)
116
+ rc = run_config("C", 0.05, tok, blocks, steps=50)
117
  print("\n--- COPY TO TEST_REPORT ---")
118
+ for r in (ra, rb, rc):
119
  print(f"- Config {r['tag']} (bal {r['bal_w']}): loss {r['loss_first']}->{r['loss_last']}, "
120
  f"max_share {r['max_share']}, usage {r['usage']} => {r['verdict']}")
121
+ locked = "A (0.01)" if ra["verdict"] == "PASS" and ra["max_share"] <= min(rb["max_share"], rc["max_share"]) else \
122
+ "B (0.02)" if rb["verdict"] == "PASS" and rb["max_share"] <= rc["max_share"] else \
123
+ "C (0.05)" if rc["verdict"] == "PASS" else "NONE β€” design badlo, aage mat badho"
124
  print(f"locked balance_weight: {locked}")
125
  print("--- END ---")
126