suchirsalhan commited on
Commit
e1feecf
·
verified ·
1 Parent(s): 0c8cb12

Add H-Net dynamic-chunking artefacts

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. README.md +67 -0
  2. baseline_en/config.json +83 -0
  3. baseline_en/final.json +44 -0
  4. baseline_en/log.jsonl +136 -0
  5. baseline_en/model.safetensors +3 -0
  6. emactl_genreg_s0/config.json +108 -0
  7. emactl_genreg_s0/final.json +114 -0
  8. emactl_genreg_s0/log.jsonl +0 -0
  9. emactl_genreg_s0/model.safetensors +3 -0
  10. emactl_genreg_s1/config.json +108 -0
  11. emactl_genreg_s1/final.json +114 -0
  12. emactl_genreg_s1/log.jsonl +0 -0
  13. emactl_genreg_s1/model.safetensors +3 -0
  14. emactl_genreg_s2/config.json +108 -0
  15. emactl_genreg_s2/final.json +114 -0
  16. emactl_genreg_s2/log.jsonl +0 -0
  17. emactl_genreg_s2/model.safetensors +3 -0
  18. emactl_noema_s0/config.json +108 -0
  19. emactl_noema_s0/final.json +114 -0
  20. emactl_noema_s0/log.jsonl +0 -0
  21. emactl_noema_s0/model.safetensors +3 -0
  22. emactl_noema_s1/config.json +108 -0
  23. emactl_noema_s1/final.json +114 -0
  24. emactl_noema_s1/log.jsonl +0 -0
  25. emactl_noema_s1/model.safetensors +3 -0
  26. emactl_noema_s2/config.json +108 -0
  27. emactl_noema_s2/final.json +114 -0
  28. emactl_noema_s2/log.jsonl +0 -0
  29. emactl_noema_s2/model.safetensors +3 -0
  30. jitter_s0/config.json +108 -0
  31. jitter_s0/final.json +114 -0
  32. jitter_s0/log.jsonl +56 -0
  33. jitter_s0/model.safetensors +3 -0
  34. jitter_s1/config.json +108 -0
  35. jitter_s1/final.json +114 -0
  36. jitter_s1/log.jsonl +56 -0
  37. jitter_s1/model.safetensors +3 -0
  38. jitter_s2/config.json +108 -0
  39. jitter_s2/final.json +114 -0
  40. jitter_s2/log.jsonl +56 -0
  41. jitter_s2/model.safetensors +3 -0
  42. manifest.json +343 -0
  43. pilot_A_s0/config.json +106 -0
  44. pilot_A_s0/final.json +114 -0
  45. pilot_A_s0/log.jsonl +0 -0
  46. pilot_A_s0/model.safetensors +3 -0
  47. pilot_A_s1/config.json +106 -0
  48. pilot_A_s1/final.json +114 -0
  49. pilot_A_s1/log.jsonl +0 -0
  50. pilot_A_s1/model.safetensors +3 -0
README.md ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: pytorch
4
+ tags: [dynamic-chunking, tokenization, hnet, byte-level, interpretability]
5
+ ---
6
+
7
+ # H-Net dynamic-chunking pilot models
8
+
9
+ 31 byte-level H-Net checkpoints (22.5M params, 6000 steps) trained to study **what a
10
+ learned chunker discovers when it is free to choose its own computational granularity**.
11
+ Every run shares data, seed schedule and FLOP budget; only the chunker's objective varies.
12
+
13
+ ## Runs
14
+
15
+ | family | n | what varies |
16
+ |---|---|---|
17
+ | `pilot_A_s{0,1,2}` | 3 | no parity loss — the natural chunker (baseline) |
18
+ | `pilot_B_s{0,1,2}` | 3 | parity loss, per-content-matched-sentence beta |
19
+ | `pilot_C_s{0,1,2}` | 3 | parity applied then removed (reversion arm) |
20
+ | `pilot_D_s{0,1,2}` | 3 | stronger equalisation |
21
+ | `pilot_E_s{0,1,2}` | 3 | entropy-matched variant |
22
+ | `pilot_Apc_s{0,1,2}` | 3 | per-codepoint control |
23
+ | `pilot_Bpc_s{0,1,2}` | 3 | parity loss, per-codepoint beta |
24
+ | `emactl_noema_s{0,1,2}` | 3 | EMA ablation control (built from A) |
25
+ | `emactl_genreg_s{0,1,2}` | 3 | generic-regularisation control, entropy-matched to B |
26
+ | `jitter_s{0,1,2}` | 3 | threshold-jitter training (tau ~ U[0.30, 0.75]) |
27
+ | `baseline_en` | 1 | English-only baseline |
28
+
29
+ ## Format
30
+
31
+ Weights are `model.safetensors`; `config.json` carries `model_config`, the training config,
32
+ and a `tied_weights` map. `emb.weight` and `head.weight` are tied, so only `emb.weight` is
33
+ stored — restore with:
34
+
35
+ ```python
36
+ from safetensors.torch import load_file
37
+ import json
38
+ sd = load_file("model.safetensors")
39
+ cfg = json.load(open("config.json"))
40
+ for dst, src in cfg["tied_weights"].items():
41
+ sd[dst] = sd[src] # head.weight <- emb.weight
42
+ ```
43
+
44
+ Verified bit-exact against the original training checkpoints (`max|Δ| = 0.0`, 111 tensors).
45
+ `final.json` and `log.jsonl` are the trainer's own records.
46
+
47
+ ## What these were used to find
48
+
49
+ - Segmentation crystallises **after** capability: the router lags BPB descent in 36/36
50
+ language-seed pairs at half of each clock's own change.
51
+ - Learned units are **not linguistic**. Morpheme F1 sits at or below a rate-matched random
52
+ baseline; gold word+morph features add ≤0.001 held-out AUROC over computational features
53
+ in 9/9 languages.
54
+ - Chunks are **not functionally localised**: aligned-minus-misaligned activation patching is
55
+ at the construction rate (0/28 cells above 2 across-seed SD), while killing the whole chunk
56
+ pathway costs +0.34 nats/byte. Information is additive and diffuse across chunks.
57
+ - The chunk-level main network is worth only **~0.07 BPB** at this scale.
58
+
59
+ Negative results are reported as results. See the companion
60
+ [results dataset](https://huggingface.co/datasets/AdaptiveChunking/hnet-chunking-results)
61
+ and [probe dataset](https://huggingface.co/datasets/AdaptiveChunking/hnet-chunking-probes).
62
+
63
+ ## Limitations
64
+
65
+ 16.2M/22.5M params, 6000 steps, one hierarchy stage, 3 seeds. Only the final step was
66
+ retained (`keep_ckpt: false`), so no intermediate weights exist. Third-party models used for
67
+ comparison (`allenai/Bolmo-1B`, BLT, ByT5) are **not** redistributed here.
baseline_en/config.json ADDED
@@ -0,0 +1,83 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run": "baseline_en",
3
+ "step": 5000,
4
+ "model_config": {
5
+ "vocab_size": 256,
6
+ "d_enc": 256,
7
+ "d_main": 512,
8
+ "n_enc": 2,
9
+ "n_dec": 2,
10
+ "n_main": 6,
11
+ "n_heads_enc": 4,
12
+ "n_heads_main": 8,
13
+ "seq_len": 1024,
14
+ "dropout": 0.0,
15
+ "target_ratio": 6.0,
16
+ "max_chunk_mult": 3.0,
17
+ "dechunk": "ema",
18
+ "rope_base": 10000.0
19
+ },
20
+ "tied_weights": {
21
+ "head.weight": "emb.weight"
22
+ },
23
+ "train_config": {
24
+ "condition": "A",
25
+ "seed": 0,
26
+ "out_dir": null,
27
+ "run_name": "baseline_en",
28
+ "corpus": "standin",
29
+ "langs": [
30
+ "en"
31
+ ],
32
+ "seq_len": 1024,
33
+ "batch_size": 32,
34
+ "grad_accum": 1,
35
+ "steps": 5000,
36
+ "lr": 0.0003,
37
+ "min_lr_frac": 0.1,
38
+ "warmup": 200,
39
+ "weight_decay": 0.1,
40
+ "grad_clip": 1.0,
41
+ "betas": [
42
+ 0.9,
43
+ 0.95
44
+ ],
45
+ "router_lr_mult": 1.0,
46
+ "d_enc": 256,
47
+ "d_main": 512,
48
+ "n_enc": 2,
49
+ "n_dec": 2,
50
+ "n_main": 6,
51
+ "n_heads_enc": 4,
52
+ "n_heads_main": 8,
53
+ "dechunk": "ema",
54
+ "target_ratio": 6.0,
55
+ "alpha": 0.03,
56
+ "alpha_parity_cap": 0.06,
57
+ "global_anchor": 0.0,
58
+ "premium_clip": 4.0,
59
+ "premium_kind": "per_char",
60
+ "tau1": 0.4,
61
+ "tau2": 0.7,
62
+ "freeze_scope": "router",
63
+ "ckpt_every": 500,
64
+ "eval_every": 250,
65
+ "log_every": 50,
66
+ "probe_sents": 96,
67
+ "eval_batches": 8,
68
+ "keep_ckpt": false,
69
+ "amp": "bf16",
70
+ "compile": false,
71
+ "cpc_decay": 0.9,
72
+ "device": "cuda"
73
+ },
74
+ "probe_agg": {
75
+ "gini_cpc": 0.0,
76
+ "gini_cps": 0.0,
77
+ "bpic": 5.065967016491754,
78
+ "worst_lang": "en",
79
+ "cpc_parity_vs_en": {
80
+ "en": 1.0
81
+ }
82
+ }
83
+ }
baseline_en/final.json ADDED
@@ -0,0 +1,44 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "event": "final",
3
+ "run": "baseline_en",
4
+ "condition": "A",
5
+ "step": 5000,
6
+ "params": 22516992,
7
+ "per_lang": {
8
+ "en": {
9
+ "cpc": 0.19758572169147598,
10
+ "cps": 27.791666666666668,
11
+ "bpb": 1.623357290677685,
12
+ "bytes_per_chunk": 4.761882271732318
13
+ }
14
+ },
15
+ "gini_cpc": 0.0,
16
+ "gini_cps": 0.0,
17
+ "bpic": 5.065967016491754,
18
+ "transitions": [
19
+ {
20
+ "step": 0,
21
+ "phase": 1,
22
+ "objective": "global",
23
+ "router_trainable": true,
24
+ "encoder_trainable": true,
25
+ "frozen_router_params": 0
26
+ },
27
+ {
28
+ "step": 2000,
29
+ "phase": 2,
30
+ "objective": "global",
31
+ "router_trainable": true,
32
+ "encoder_trainable": true,
33
+ "frozen_router_params": 0
34
+ },
35
+ {
36
+ "step": 3500,
37
+ "phase": 3,
38
+ "objective": "global",
39
+ "router_trainable": true,
40
+ "encoder_trainable": true,
41
+ "frozen_router_params": 0
42
+ }
43
+ ]
44
+ }
baseline_en/log.jsonl ADDED
@@ -0,0 +1,136 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {"event": "start", "run": "baseline_en", "condition": "A", "params": 22516992, "langs": ["en"], "premium": {"en": 1.0}, "schedule": {"tau1": 0.4, "tau2": 0.7, "p1_objective": "global", "p2_objective": "global", "p3_objective": "global", "freeze_in_p3": false, "freeze_scope": "router"}, "corpus": "/local/scratch/sas245/beetle-hnet/scratch/data/tier1_standin", "weights": {"en": 1.0}, "t": 1.3}
2
+ {"event": "probe", "step": 0, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 4884.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.361697, "cps": 50.875, "mean_p": 0.496347}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 2.767404, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 1.5}
3
+ {"event": "train", "step": 0, "phase": 1, "objective": "global", "lm_loss": 5.66441, "bpb": 8.17201, "chunk_loss": 0.040128, "F": 0.30981, "G": 0.49124, "chunks": 317.25, "bytes_per_chunk": 3.228, "grad_norm": 12.0832, "lr": 1.5e-06, "router_trainable": true, "sec_per_step": 0.015, "alpha": 0.03, "t": 2.4}
4
+ {"event": "train", "step": 50, "phase": 1, "objective": "global", "lm_loss": 3.96706, "bpb": 5.72326, "chunk_loss": 0.031702, "F": 0.1907, "G": 0.49023, "chunks": 195.28, "bytes_per_chunk": 5.244, "grad_norm": 2.3639, "lr": 7.65e-05, "router_trainable": true, "sec_per_step": 0.051, "alpha": 0.03, "t": 4.9}
5
+ {"event": "train", "step": 100, "phase": 1, "objective": "global", "lm_loss": 2.93869, "bpb": 4.23963, "chunk_loss": 0.031852, "F": 0.20514, "G": 0.38478, "chunks": 210.06, "bytes_per_chunk": 4.875, "grad_norm": 3.0762, "lr": 0.0001515, "router_trainable": true, "sec_per_step": 0.042, "alpha": 0.03, "t": 7.1}
6
+ {"event": "train", "step": 150, "phase": 1, "objective": "global", "lm_loss": 2.59506, "bpb": 3.74388, "chunk_loss": 0.030458, "F": 0.17642, "G": 0.36688, "chunks": 180.66, "bytes_per_chunk": 5.668, "grad_norm": 1.6857, "lr": 0.0002265, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 9.1}
7
+ {"event": "train", "step": 200, "phase": 1, "objective": "global", "lm_loss": 2.36746, "bpb": 3.41553, "chunk_loss": 0.031466, "F": 0.20374, "G": 0.345, "chunks": 208.62, "bytes_per_chunk": 4.908, "grad_norm": 2.4918, "lr": 0.0003, "router_trainable": true, "sec_per_step": 0.039, "alpha": 0.03, "t": 11.0}
8
+ {"event": "eval", "step": 250, "phase": 1, "bpb": {"en": 3.17185}, "bytes_per_chunk": {"en": 4.69}, "bpb_mean": 3.17185, "t": 13.1}
9
+ {"event": "train", "step": 250, "phase": 1, "objective": "global", "lm_loss": 2.14722, "bpb": 3.09779, "chunk_loss": 0.031083, "F": 0.21036, "G": 0.27837, "chunks": 215.41, "bytes_per_chunk": 4.754, "grad_norm": 1.3424, "lr": 0.0002999, "router_trainable": true, "sec_per_step": 0.042, "alpha": 0.03, "t": 13.1}
10
+ {"event": "train", "step": 300, "phase": 1, "objective": "global", "lm_loss": 2.1025, "bpb": 3.03326, "chunk_loss": 0.031044, "F": 0.21567, "G": 0.26223, "chunks": 220.84, "bytes_per_chunk": 4.637, "grad_norm": 1.2806, "lr": 0.0002997, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 15.1}
11
+ {"event": "train", "step": 350, "phase": 1, "objective": "global", "lm_loss": 2.01715, "bpb": 2.91013, "chunk_loss": 0.03129, "F": 0.2258, "G": 0.26442, "chunks": 231.22, "bytes_per_chunk": 4.429, "grad_norm": 1.1319, "lr": 0.0002993, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 17.1}
12
+ {"event": "train", "step": 400, "phase": 1, "objective": "global", "lm_loss": 1.92618, "bpb": 2.7789, "chunk_loss": 0.031034, "F": 0.22089, "G": 0.25166, "chunks": 226.19, "bytes_per_chunk": 4.527, "grad_norm": 1.4106, "lr": 0.0002988, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 19.2}
13
+ {"event": "train", "step": 450, "phase": 1, "objective": "global", "lm_loss": 1.80237, "bpb": 2.60028, "chunk_loss": 0.030876, "F": 0.21829, "G": 0.24335, "chunks": 223.53, "bytes_per_chunk": 4.581, "grad_norm": 1.3423, "lr": 0.0002982, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 21.2}
14
+ {"event": "probe", "step": 500, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2879.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.213212, "cps": 29.989583, "mean_p": 0.238603}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 4.694686, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 23.2}
15
+ {"event": "eval", "step": 500, "phase": 1, "bpb": {"en": 2.64785}, "bytes_per_chunk": {"en": 4.472}, "bpb_mean": 2.64785, "t": 23.4}
16
+ {"event": "train", "step": 500, "phase": 1, "objective": "global", "lm_loss": 1.76139, "bpb": 2.54115, "chunk_loss": 0.030944, "F": 0.22131, "G": 0.24395, "chunks": 226.62, "bytes_per_chunk": 4.518, "grad_norm": 1.5029, "lr": 0.0002974, "router_trainable": true, "sec_per_step": 0.046, "alpha": 0.03, "t": 23.5}
17
+ {"event": "train", "step": 550, "phase": 1, "objective": "global", "lm_loss": 1.76605, "bpb": 2.54787, "chunk_loss": 0.030994, "F": 0.22476, "G": 0.24278, "chunks": 230.16, "bytes_per_chunk": 4.449, "grad_norm": 1.1989, "lr": 0.0002965, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 25.5}
18
+ {"event": "train", "step": 600, "phase": 1, "objective": "global", "lm_loss": 1.68769, "bpb": 2.43482, "chunk_loss": 0.03111, "F": 0.22791, "G": 0.24679, "chunks": 233.38, "bytes_per_chunk": 4.388, "grad_norm": 1.3439, "lr": 0.0002954, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 27.5}
19
+ {"event": "train", "step": 650, "phase": 1, "objective": "global", "lm_loss": 1.68067, "bpb": 2.4247, "chunk_loss": 0.030982, "F": 0.22586, "G": 0.24037, "chunks": 231.28, "bytes_per_chunk": 4.428, "grad_norm": 1.0878, "lr": 0.0002942, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 29.6}
20
+ {"event": "train", "step": 700, "phase": 1, "objective": "global", "lm_loss": 1.61067, "bpb": 2.32371, "chunk_loss": 0.030945, "F": 0.22562, "G": 0.23685, "chunks": 231.03, "bytes_per_chunk": 4.432, "grad_norm": 1.1877, "lr": 0.0002928, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 31.6}
21
+ {"event": "eval", "step": 750, "phase": 1, "bpb": {"en": 2.43375}, "bytes_per_chunk": {"en": 4.47}, "bpb_mean": 2.43375, "t": 33.8}
22
+ {"event": "train", "step": 750, "phase": 1, "objective": "global", "lm_loss": 1.60258, "bpb": 2.31203, "chunk_loss": 0.030729, "F": 0.21658, "G": 0.23002, "chunks": 221.78, "bytes_per_chunk": 4.617, "grad_norm": 1.2023, "lr": 0.0002913, "router_trainable": true, "sec_per_step": 0.043, "alpha": 0.03, "t": 33.8}
23
+ {"event": "train", "step": 800, "phase": 1, "objective": "global", "lm_loss": 1.62759, "bpb": 2.34812, "chunk_loss": 0.030837, "F": 0.22229, "G": 0.23337, "chunks": 227.62, "bytes_per_chunk": 4.499, "grad_norm": 1.1165, "lr": 0.0002897, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 35.8}
24
+ {"event": "train", "step": 850, "phase": 1, "objective": "global", "lm_loss": 1.63247, "bpb": 2.35516, "chunk_loss": 0.030938, "F": 0.22522, "G": 0.23781, "chunks": 230.62, "bytes_per_chunk": 4.44, "grad_norm": 0.9742, "lr": 0.000288, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 37.9}
25
+ {"event": "train", "step": 900, "phase": 1, "objective": "global", "lm_loss": 1.63538, "bpb": 2.35936, "chunk_loss": 0.030898, "F": 0.22281, "G": 0.2353, "chunks": 228.16, "bytes_per_chunk": 4.488, "grad_norm": 1.0922, "lr": 0.0002861, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 39.9}
26
+ {"event": "train", "step": 950, "phase": 1, "objective": "global", "lm_loss": 1.57397, "bpb": 2.27076, "chunk_loss": 0.030926, "F": 0.22537, "G": 0.23539, "chunks": 230.78, "bytes_per_chunk": 4.437, "grad_norm": 0.9995, "lr": 0.0002841, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 42.0}
27
+ {"event": "probe", "step": 1000, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2886.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.21373, "cps": 30.0625, "mean_p": 0.22402}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 4.683299, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 44.0}
28
+ {"event": "eval", "step": 1000, "phase": 1, "bpb": {"en": 2.32062}, "bytes_per_chunk": {"en": 4.436}, "bpb_mean": 2.32062, "t": 44.2}
29
+ {"event": "train", "step": 1000, "phase": 1, "objective": "global", "lm_loss": 1.60957, "bpb": 2.32212, "chunk_loss": 0.030921, "F": 0.22507, "G": 0.23436, "chunks": 230.47, "bytes_per_chunk": 4.443, "grad_norm": 1.0573, "lr": 0.0002819, "router_trainable": true, "sec_per_step": 0.046, "alpha": 0.03, "t": 44.3}
30
+ {"event": "train", "step": 1050, "phase": 1, "objective": "global", "lm_loss": 1.53486, "bpb": 2.21433, "chunk_loss": 0.03085, "F": 0.22391, "G": 0.23217, "chunks": 229.28, "bytes_per_chunk": 4.466, "grad_norm": 1.0475, "lr": 0.0002796, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 46.3}
31
+ {"event": "train", "step": 1100, "phase": 1, "objective": "global", "lm_loss": 1.5074, "bpb": 2.17472, "chunk_loss": 0.030865, "F": 0.22455, "G": 0.23175, "chunks": 229.94, "bytes_per_chunk": 4.453, "grad_norm": 0.9906, "lr": 0.0002772, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 48.4}
32
+ {"event": "train", "step": 1150, "phase": 1, "objective": "global", "lm_loss": 1.48296, "bpb": 2.13947, "chunk_loss": 0.030705, "F": 0.21826, "G": 0.22673, "chunks": 223.5, "bytes_per_chunk": 4.582, "grad_norm": 0.8906, "lr": 0.0002747, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 50.4}
33
+ {"event": "train", "step": 1200, "phase": 1, "objective": "global", "lm_loss": 1.53887, "bpb": 2.22012, "chunk_loss": 0.030739, "F": 0.21878, "G": 0.22682, "chunks": 224.03, "bytes_per_chunk": 4.571, "grad_norm": 0.8517, "lr": 0.0002721, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 52.5}
34
+ {"event": "eval", "step": 1250, "phase": 1, "bpb": {"en": 2.2299}, "bytes_per_chunk": {"en": 4.506}, "bpb_mean": 2.2299, "t": 54.6}
35
+ {"event": "train", "step": 1250, "phase": 1, "objective": "global", "lm_loss": 1.47402, "bpb": 2.12656, "chunk_loss": 0.03093, "F": 0.22766, "G": 0.23381, "chunks": 233.12, "bytes_per_chunk": 4.392, "grad_norm": 0.8293, "lr": 0.0002694, "router_trainable": true, "sec_per_step": 0.043, "alpha": 0.03, "t": 54.6}
36
+ {"event": "train", "step": 1300, "phase": 1, "objective": "global", "lm_loss": 1.54796, "bpb": 2.23324, "chunk_loss": 0.030811, "F": 0.21948, "G": 0.23125, "chunks": 224.75, "bytes_per_chunk": 4.556, "grad_norm": 0.8633, "lr": 0.0002665, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 56.6}
37
+ {"event": "train", "step": 1350, "phase": 1, "objective": "global", "lm_loss": 1.5163, "bpb": 2.18756, "chunk_loss": 0.030789, "F": 0.21841, "G": 0.22875, "chunks": 223.66, "bytes_per_chunk": 4.578, "grad_norm": 0.838, "lr": 0.0002635, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 58.7}
38
+ {"event": "train", "step": 1400, "phase": 1, "objective": "global", "lm_loss": 1.4602, "bpb": 2.10663, "chunk_loss": 0.030696, "F": 0.21671, "G": 0.22668, "chunks": 221.91, "bytes_per_chunk": 4.615, "grad_norm": 0.8617, "lr": 0.0002605, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 60.7}
39
+ {"event": "train", "step": 1450, "phase": 1, "objective": "global", "lm_loss": 1.47435, "bpb": 2.12704, "chunk_loss": 0.030535, "F": 0.20804, "G": 0.22005, "chunks": 213.03, "bytes_per_chunk": 4.807, "grad_norm": 0.7718, "lr": 0.0002573, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 62.7}
40
+ {"event": "probe", "step": 1500, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2707.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.200474, "cps": 28.197917, "mean_p": 0.213113}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 4.992981, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 64.7}
41
+ {"event": "eval", "step": 1500, "phase": 1, "bpb": {"en": 2.17371}, "bytes_per_chunk": {"en": 4.702}, "bpb_mean": 2.17371, "t": 65.0}
42
+ {"event": "train", "step": 1500, "phase": 1, "objective": "global", "lm_loss": 1.44903, "bpb": 2.09051, "chunk_loss": 0.030599, "F": 0.21173, "G": 0.22301, "chunks": 216.81, "bytes_per_chunk": 4.723, "grad_norm": 0.8913, "lr": 0.000254, "router_trainable": true, "sec_per_step": 0.045, "alpha": 0.03, "t": 65.0}
43
+ {"event": "train", "step": 1550, "phase": 1, "objective": "global", "lm_loss": 1.44784, "bpb": 2.08879, "chunk_loss": 0.030459, "F": 0.20609, "G": 0.21561, "chunks": 211.03, "bytes_per_chunk": 4.852, "grad_norm": 0.8538, "lr": 0.0002506, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 67.0}
44
+ {"event": "train", "step": 1600, "phase": 1, "objective": "global", "lm_loss": 1.49386, "bpb": 2.15519, "chunk_loss": 0.030592, "F": 0.21103, "G": 0.22242, "chunks": 216.09, "bytes_per_chunk": 4.739, "grad_norm": 0.8883, "lr": 0.0002472, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 69.1}
45
+ {"event": "train", "step": 1650, "phase": 1, "objective": "global", "lm_loss": 1.39083, "bpb": 2.00654, "chunk_loss": 0.030619, "F": 0.20911, "G": 0.21716, "chunks": 214.12, "bytes_per_chunk": 4.782, "grad_norm": 0.7979, "lr": 0.0002436, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 71.1}
46
+ {"event": "train", "step": 1700, "phase": 1, "objective": "global", "lm_loss": 1.40329, "bpb": 2.02451, "chunk_loss": 0.030517, "F": 0.20947, "G": 0.21852, "chunks": 214.5, "bytes_per_chunk": 4.774, "grad_norm": 0.7993, "lr": 0.00024, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 73.1}
47
+ {"event": "eval", "step": 1750, "phase": 1, "bpb": {"en": 2.12451}, "bytes_per_chunk": {"en": 4.744}, "bpb_mean": 2.12451, "t": 75.2}
48
+ {"event": "train", "step": 1750, "phase": 1, "objective": "global", "lm_loss": 1.41237, "bpb": 2.03762, "chunk_loss": 0.030544, "F": 0.20901, "G": 0.21875, "chunks": 214.03, "bytes_per_chunk": 4.784, "grad_norm": 0.8677, "lr": 0.0002363, "router_trainable": true, "sec_per_step": 0.043, "alpha": 0.03, "t": 75.2}
49
+ {"event": "train", "step": 1800, "phase": 1, "objective": "global", "lm_loss": 1.39262, "bpb": 2.00913, "chunk_loss": 0.030542, "F": 0.21033, "G": 0.21926, "chunks": 215.38, "bytes_per_chunk": 4.754, "grad_norm": 0.8533, "lr": 0.0002325, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 77.3}
50
+ {"event": "train", "step": 1850, "phase": 1, "objective": "global", "lm_loss": 1.39702, "bpb": 2.01547, "chunk_loss": 0.03052, "F": 0.20917, "G": 0.21769, "chunks": 214.19, "bytes_per_chunk": 4.781, "grad_norm": 0.7715, "lr": 0.0002286, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 79.3}
51
+ {"event": "train", "step": 1900, "phase": 1, "objective": "global", "lm_loss": 1.38118, "bpb": 1.99262, "chunk_loss": 0.030695, "F": 0.21619, "G": 0.22317, "chunks": 221.38, "bytes_per_chunk": 4.626, "grad_norm": 0.7694, "lr": 0.0002247, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 81.3}
52
+ {"event": "train", "step": 1950, "phase": 1, "objective": "global", "lm_loss": 1.33808, "bpb": 1.93044, "chunk_loss": 0.030462, "F": 0.20746, "G": 0.2145, "chunks": 212.44, "bytes_per_chunk": 4.82, "grad_norm": 0.8269, "lr": 0.0002207, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 83.4}
53
+ {"event": "probe", "step": 2000, "phase": 2, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2664.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.197289, "cps": 27.75, "mean_p": 0.208036}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 5.073574, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 85.4}
54
+ {"event": "eval", "step": 2000, "phase": 1, "bpb": {"en": 2.05003}, "bytes_per_chunk": {"en": 4.733}, "bpb_mean": 2.05003, "t": 85.6}
55
+ {"event": "phase_transition", "step": 2000, "phase": 2, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0, "t": 85.6}
56
+ {"event": "train", "step": 2000, "phase": 2, "objective": "global", "lm_loss": 1.394, "bpb": 2.01112, "chunk_loss": 0.030635, "F": 0.21103, "G": 0.21899, "chunks": 216.09, "bytes_per_chunk": 4.739, "grad_norm": 0.7798, "lr": 0.0002167, "router_trainable": true, "sec_per_step": 0.045, "alpha": 0.03, "t": 85.6}
57
+ {"event": "train", "step": 2050, "phase": 2, "objective": "global", "lm_loss": 1.3946, "bpb": 2.01199, "chunk_loss": 0.030557, "F": 0.20926, "G": 0.21781, "chunks": 214.28, "bytes_per_chunk": 4.779, "grad_norm": 0.7824, "lr": 0.0002126, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 87.7}
58
+ {"event": "train", "step": 2100, "phase": 2, "objective": "global", "lm_loss": 1.3849, "bpb": 1.99799, "chunk_loss": 0.030438, "F": 0.20569, "G": 0.21416, "chunks": 210.62, "bytes_per_chunk": 4.862, "grad_norm": 0.8147, "lr": 0.0002084, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 89.7}
59
+ {"event": "train", "step": 2150, "phase": 2, "objective": "global", "lm_loss": 1.40408, "bpb": 2.02565, "chunk_loss": 0.030815, "F": 0.21942, "G": 0.22522, "chunks": 224.69, "bytes_per_chunk": 4.557, "grad_norm": 0.7288, "lr": 0.0002042, "router_trainable": true, "sec_per_step": 0.042, "alpha": 0.03, "t": 91.8}
60
+ {"event": "train", "step": 2200, "phase": 2, "objective": "global", "lm_loss": 1.35356, "bpb": 1.95278, "chunk_loss": 0.03052, "F": 0.20929, "G": 0.21559, "chunks": 214.31, "bytes_per_chunk": 4.778, "grad_norm": 0.8509, "lr": 0.0001999, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 93.8}
61
+ {"event": "eval", "step": 2250, "phase": 2, "bpb": {"en": 1.99956}, "bytes_per_chunk": {"en": 4.682}, "bpb_mean": 1.99956, "t": 95.9}
62
+ {"event": "train", "step": 2250, "phase": 2, "objective": "global", "lm_loss": 1.34643, "bpb": 1.94248, "chunk_loss": 0.030686, "F": 0.21515, "G": 0.22121, "chunks": 220.31, "bytes_per_chunk": 4.648, "grad_norm": 0.7659, "lr": 0.0001957, "router_trainable": true, "sec_per_step": 0.043, "alpha": 0.03, "t": 96.0}
63
+ {"event": "train", "step": 2300, "phase": 2, "objective": "global", "lm_loss": 1.30389, "bpb": 1.88111, "chunk_loss": 0.030484, "F": 0.20728, "G": 0.21505, "chunks": 212.25, "bytes_per_chunk": 4.824, "grad_norm": 0.7552, "lr": 0.0001913, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 98.0}
64
+ {"event": "train", "step": 2350, "phase": 2, "objective": "global", "lm_loss": 1.3369, "bpb": 1.92874, "chunk_loss": 0.030639, "F": 0.21274, "G": 0.21829, "chunks": 217.84, "bytes_per_chunk": 4.701, "grad_norm": 0.7572, "lr": 0.000187, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 100.0}
65
+ {"event": "train", "step": 2400, "phase": 2, "objective": "global", "lm_loss": 1.35317, "bpb": 1.95221, "chunk_loss": 0.030479, "F": 0.20761, "G": 0.2132, "chunks": 212.59, "bytes_per_chunk": 4.817, "grad_norm": 0.7234, "lr": 0.0001826, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 102.1}
66
+ {"event": "train", "step": 2450, "phase": 2, "objective": "global", "lm_loss": 1.3337, "bpb": 1.92413, "chunk_loss": 0.030593, "F": 0.21326, "G": 0.21969, "chunks": 218.38, "bytes_per_chunk": 4.689, "grad_norm": 0.8154, "lr": 0.0001782, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 104.2}
67
+ {"event": "probe", "step": 2500, "phase": 2, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2676.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.198178, "cps": 27.875, "mean_p": 0.206845}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 5.050822, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 106.2}
68
+ {"event": "eval", "step": 2500, "phase": 2, "bpb": {"en": 1.95686}, "bytes_per_chunk": {"en": 4.734}, "bpb_mean": 1.95686, "t": 106.4}
69
+ {"event": "train", "step": 2500, "phase": 2, "objective": "global", "lm_loss": 1.29957, "bpb": 1.87489, "chunk_loss": 0.030607, "F": 0.21335, "G": 0.22061, "chunks": 218.47, "bytes_per_chunk": 4.687, "grad_norm": 0.8269, "lr": 0.0001738, "router_trainable": true, "sec_per_step": 0.046, "alpha": 0.03, "t": 106.5}
70
+ {"event": "train", "step": 2550, "phase": 2, "objective": "global", "lm_loss": 1.32449, "bpb": 1.91083, "chunk_loss": 0.030647, "F": 0.215, "G": 0.22051, "chunks": 220.16, "bytes_per_chunk": 4.651, "grad_norm": 0.8431, "lr": 0.0001694, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 108.5}
71
+ {"event": "train", "step": 2600, "phase": 2, "objective": "global", "lm_loss": 1.28927, "bpb": 1.86003, "chunk_loss": 0.030502, "F": 0.20987, "G": 0.21497, "chunks": 214.91, "bytes_per_chunk": 4.765, "grad_norm": 0.7621, "lr": 0.000165, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 110.5}
72
+ {"event": "train", "step": 2650, "phase": 2, "objective": "global", "lm_loss": 1.26064, "bpb": 1.81873, "chunk_loss": 0.030481, "F": 0.20776, "G": 0.21441, "chunks": 212.75, "bytes_per_chunk": 4.813, "grad_norm": 0.7947, "lr": 0.0001606, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 112.6}
73
+ {"event": "train", "step": 2700, "phase": 2, "objective": "global", "lm_loss": 1.19122, "bpb": 1.71856, "chunk_loss": 0.030482, "F": 0.20706, "G": 0.21302, "chunks": 212.03, "bytes_per_chunk": 4.829, "grad_norm": 0.7729, "lr": 0.0001562, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 114.6}
74
+ {"event": "eval", "step": 2750, "phase": 2, "bpb": {"en": 1.90295}, "bytes_per_chunk": {"en": 4.737}, "bpb_mean": 1.90295, "t": 116.7}
75
+ {"event": "train", "step": 2750, "phase": 2, "objective": "global", "lm_loss": 1.3232, "bpb": 1.90897, "chunk_loss": 0.03067, "F": 0.21368, "G": 0.22047, "chunks": 218.81, "bytes_per_chunk": 4.68, "grad_norm": 0.828, "lr": 0.0001518, "router_trainable": true, "sec_per_step": 0.043, "alpha": 0.03, "t": 116.7}
76
+ {"event": "train", "step": 2800, "phase": 2, "objective": "global", "lm_loss": 1.27056, "bpb": 1.83303, "chunk_loss": 0.030499, "F": 0.20639, "G": 0.21206, "chunks": 211.34, "bytes_per_chunk": 4.845, "grad_norm": 0.7814, "lr": 0.0001474, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 118.8}
77
+ {"event": "train", "step": 2850, "phase": 2, "objective": "global", "lm_loss": 1.28758, "bpb": 1.85758, "chunk_loss": 0.030454, "F": 0.20618, "G": 0.21208, "chunks": 211.12, "bytes_per_chunk": 4.85, "grad_norm": 0.8001, "lr": 0.000143, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 120.8}
78
+ {"event": "train", "step": 2900, "phase": 2, "objective": "global", "lm_loss": 1.24014, "bpb": 1.78915, "chunk_loss": 0.030438, "F": 0.2049, "G": 0.21119, "chunks": 209.81, "bytes_per_chunk": 4.881, "grad_norm": 0.7737, "lr": 0.0001387, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 122.8}
79
+ {"event": "train", "step": 2950, "phase": 2, "objective": "global", "lm_loss": 1.22198, "bpb": 1.76295, "chunk_loss": 0.030534, "F": 0.21124, "G": 0.21671, "chunks": 216.31, "bytes_per_chunk": 4.734, "grad_norm": 0.8122, "lr": 0.0001343, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 124.8}
80
+ {"event": "probe", "step": 3000, "phase": 2, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2676.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.198178, "cps": 27.875, "mean_p": 0.205814}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 5.050822, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 126.8}
81
+ {"event": "eval", "step": 3000, "phase": 2, "bpb": {"en": 1.86797}, "bytes_per_chunk": {"en": 4.732}, "bpb_mean": 1.86797, "t": 127.1}
82
+ {"event": "train", "step": 3000, "phase": 2, "objective": "global", "lm_loss": 1.27078, "bpb": 1.83335, "chunk_loss": 0.030462, "F": 0.2085, "G": 0.21429, "chunks": 213.5, "bytes_per_chunk": 4.796, "grad_norm": 0.8156, "lr": 0.0001301, "router_trainable": true, "sec_per_step": 0.045, "alpha": 0.03, "t": 127.1}
83
+ {"event": "train", "step": 3050, "phase": 2, "objective": "global", "lm_loss": 1.21044, "bpb": 1.7463, "chunk_loss": 0.030514, "F": 0.20969, "G": 0.2161, "chunks": 214.72, "bytes_per_chunk": 4.769, "grad_norm": 0.8593, "lr": 0.0001258, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 129.1}
84
+ {"event": "train", "step": 3100, "phase": 2, "objective": "global", "lm_loss": 1.31147, "bpb": 1.89206, "chunk_loss": 0.030653, "F": 0.21628, "G": 0.2214, "chunks": 221.47, "bytes_per_chunk": 4.624, "grad_norm": 0.8762, "lr": 0.0001216, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 131.2}
85
+ {"event": "train", "step": 3150, "phase": 2, "objective": "global", "lm_loss": 1.29756, "bpb": 1.87199, "chunk_loss": 0.030611, "F": 0.21539, "G": 0.22035, "chunks": 220.56, "bytes_per_chunk": 4.643, "grad_norm": 0.8626, "lr": 0.0001174, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 133.2}
86
+ {"event": "train", "step": 3200, "phase": 2, "objective": "global", "lm_loss": 1.26023, "bpb": 1.81813, "chunk_loss": 0.030586, "F": 0.21399, "G": 0.21982, "chunks": 219.12, "bytes_per_chunk": 4.673, "grad_norm": 0.903, "lr": 0.0001133, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 135.2}
87
+ {"event": "eval", "step": 3250, "phase": 2, "bpb": {"en": 1.82717}, "bytes_per_chunk": {"en": 4.729}, "bpb_mean": 1.82717, "t": 137.4}
88
+ {"event": "train", "step": 3250, "phase": 2, "objective": "global", "lm_loss": 1.23019, "bpb": 1.77479, "chunk_loss": 0.030552, "F": 0.21155, "G": 0.2167, "chunks": 216.62, "bytes_per_chunk": 4.727, "grad_norm": 0.8599, "lr": 0.0001093, "router_trainable": true, "sec_per_step": 0.043, "alpha": 0.03, "t": 137.4}
89
+ {"event": "train", "step": 3300, "phase": 2, "objective": "global", "lm_loss": 1.32118, "bpb": 1.90606, "chunk_loss": 0.030653, "F": 0.21249, "G": 0.21795, "chunks": 217.59, "bytes_per_chunk": 4.706, "grad_norm": 0.8874, "lr": 0.0001053, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 139.5}
90
+ {"event": "train", "step": 3350, "phase": 2, "objective": "global", "lm_loss": 1.19846, "bpb": 1.72901, "chunk_loss": 0.030442, "F": 0.20755, "G": 0.21312, "chunks": 212.53, "bytes_per_chunk": 4.818, "grad_norm": 0.8238, "lr": 0.0001014, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 141.5}
91
+ {"event": "train", "step": 3400, "phase": 2, "objective": "global", "lm_loss": 1.20316, "bpb": 1.7358, "chunk_loss": 0.030587, "F": 0.2128, "G": 0.21877, "chunks": 217.91, "bytes_per_chunk": 4.699, "grad_norm": 0.8561, "lr": 9.75e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 143.6}
92
+ {"event": "train", "step": 3450, "phase": 2, "objective": "global", "lm_loss": 1.24409, "bpb": 1.79485, "chunk_loss": 0.030565, "F": 0.21161, "G": 0.2173, "chunks": 216.69, "bytes_per_chunk": 4.726, "grad_norm": 0.9254, "lr": 9.37e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 145.7}
93
+ {"event": "probe", "step": 3500, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2668.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.197586, "cps": 27.791667, "mean_p": 0.204397}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 5.065967, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 147.7}
94
+ {"event": "eval", "step": 3500, "phase": 2, "bpb": {"en": 1.78841}, "bytes_per_chunk": {"en": 4.742}, "bpb_mean": 1.78841, "t": 147.9}
95
+ {"event": "phase_transition", "step": 3500, "phase": 3, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0, "t": 147.9}
96
+ {"event": "train", "step": 3500, "phase": 3, "objective": "global", "lm_loss": 1.19496, "bpb": 1.72397, "chunk_loss": 0.030473, "F": 0.20816, "G": 0.21329, "chunks": 213.16, "bytes_per_chunk": 4.804, "grad_norm": 0.9069, "lr": 9e-05, "router_trainable": true, "sec_per_step": 0.045, "alpha": 0.03, "t": 147.9}
97
+ {"event": "train", "step": 3550, "phase": 3, "objective": "global", "lm_loss": 1.15301, "bpb": 1.66345, "chunk_loss": 0.030404, "F": 0.20425, "G": 0.21075, "chunks": 209.16, "bytes_per_chunk": 4.896, "grad_norm": 0.9098, "lr": 8.64e-05, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 149.9}
98
+ {"event": "train", "step": 3600, "phase": 3, "objective": "global", "lm_loss": 1.15334, "bpb": 1.66392, "chunk_loss": 0.030414, "F": 0.20581, "G": 0.211, "chunks": 210.75, "bytes_per_chunk": 4.859, "grad_norm": 0.901, "lr": 8.28e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 152.0}
99
+ {"event": "train", "step": 3650, "phase": 3, "objective": "global", "lm_loss": 1.14224, "bpb": 1.6479, "chunk_loss": 0.03045, "F": 0.20596, "G": 0.21129, "chunks": 210.91, "bytes_per_chunk": 4.855, "grad_norm": 0.9649, "lr": 7.94e-05, "router_trainable": true, "sec_per_step": 0.042, "alpha": 0.03, "t": 154.1}
100
+ {"event": "train", "step": 3700, "phase": 3, "objective": "global", "lm_loss": 1.1677, "bpb": 1.68464, "chunk_loss": 0.030596, "F": 0.21204, "G": 0.2175, "chunks": 217.12, "bytes_per_chunk": 4.716, "grad_norm": 0.9024, "lr": 7.6e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 156.1}
101
+ {"event": "eval", "step": 3750, "phase": 3, "bpb": {"en": 1.74671}, "bytes_per_chunk": {"en": 4.75}, "bpb_mean": 1.74671, "t": 158.3}
102
+ {"event": "train", "step": 3750, "phase": 3, "objective": "global", "lm_loss": 1.15668, "bpb": 1.66874, "chunk_loss": 0.030608, "F": 0.2135, "G": 0.21959, "chunks": 218.62, "bytes_per_chunk": 4.684, "grad_norm": 0.898, "lr": 7.27e-05, "router_trainable": true, "sec_per_step": 0.043, "alpha": 0.03, "t": 158.3}
103
+ {"event": "train", "step": 3800, "phase": 3, "objective": "global", "lm_loss": 1.1641, "bpb": 1.67945, "chunk_loss": 0.030531, "F": 0.21155, "G": 0.21752, "chunks": 216.62, "bytes_per_chunk": 4.727, "grad_norm": 0.8836, "lr": 6.95e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 160.3}
104
+ {"event": "train", "step": 3850, "phase": 3, "objective": "global", "lm_loss": 1.22436, "bpb": 1.76637, "chunk_loss": 0.030878, "F": 0.22089, "G": 0.22619, "chunks": 226.19, "bytes_per_chunk": 4.527, "grad_norm": 0.9478, "lr": 6.65e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 162.4}
105
+ {"event": "train", "step": 3900, "phase": 3, "objective": "global", "lm_loss": 1.1527, "bpb": 1.66299, "chunk_loss": 0.030485, "F": 0.20804, "G": 0.21308, "chunks": 213.03, "bytes_per_chunk": 4.807, "grad_norm": 0.9299, "lr": 6.35e-05, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 164.4}
106
+ {"event": "train", "step": 3950, "phase": 3, "objective": "global", "lm_loss": 1.17791, "bpb": 1.69936, "chunk_loss": 0.03048, "F": 0.20859, "G": 0.21356, "chunks": 213.59, "bytes_per_chunk": 4.794, "grad_norm": 0.9686, "lr": 6.06e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 166.4}
107
+ {"event": "probe", "step": 4000, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2676.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.198178, "cps": 27.875, "mean_p": 0.204176}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 5.050822, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 168.4}
108
+ {"event": "eval", "step": 4000, "phase": 3, "bpb": {"en": 1.72062}, "bytes_per_chunk": {"en": 4.744}, "bpb_mean": 1.72062, "t": 168.7}
109
+ {"event": "train", "step": 4000, "phase": 3, "objective": "global", "lm_loss": 1.15487, "bpb": 1.66613, "chunk_loss": 0.030475, "F": 0.20819, "G": 0.21339, "chunks": 213.19, "bytes_per_chunk": 4.803, "grad_norm": 0.9138, "lr": 5.79e-05, "router_trainable": true, "sec_per_step": 0.046, "alpha": 0.03, "t": 168.7}
110
+ {"event": "train", "step": 4050, "phase": 3, "objective": "global", "lm_loss": 1.11953, "bpb": 1.61514, "chunk_loss": 0.030463, "F": 0.20752, "G": 0.21271, "chunks": 212.5, "bytes_per_chunk": 4.819, "grad_norm": 0.9346, "lr": 5.53e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 170.8}
111
+ {"event": "train", "step": 4100, "phase": 3, "objective": "global", "lm_loss": 1.13507, "bpb": 1.63756, "chunk_loss": 0.030642, "F": 0.2121, "G": 0.2175, "chunks": 217.19, "bytes_per_chunk": 4.715, "grad_norm": 0.9978, "lr": 5.28e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 172.8}
112
+ {"event": "train", "step": 4150, "phase": 3, "objective": "global", "lm_loss": 1.18182, "bpb": 1.705, "chunk_loss": 0.03057, "F": 0.21243, "G": 0.21815, "chunks": 217.53, "bytes_per_chunk": 4.707, "grad_norm": 0.9594, "lr": 5.04e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 174.8}
113
+ {"event": "train", "step": 4200, "phase": 3, "objective": "global", "lm_loss": 1.13783, "bpb": 1.64154, "chunk_loss": 0.030584, "F": 0.21237, "G": 0.21749, "chunks": 217.47, "bytes_per_chunk": 4.709, "grad_norm": 0.9636, "lr": 4.81e-05, "router_trainable": true, "sec_per_step": 0.043, "alpha": 0.03, "t": 177.0}
114
+ {"event": "eval", "step": 4250, "phase": 3, "bpb": {"en": 1.6914}, "bytes_per_chunk": {"en": 4.743}, "bpb_mean": 1.6914, "t": 179.2}
115
+ {"event": "train", "step": 4250, "phase": 3, "objective": "global", "lm_loss": 1.1837, "bpb": 1.70772, "chunk_loss": 0.030569, "F": 0.21277, "G": 0.21797, "chunks": 217.88, "bytes_per_chunk": 4.7, "grad_norm": 0.9683, "lr": 4.59e-05, "router_trainable": true, "sec_per_step": 0.045, "alpha": 0.03, "t": 179.2}
116
+ {"event": "train", "step": 4300, "phase": 3, "objective": "global", "lm_loss": 1.12033, "bpb": 1.61629, "chunk_loss": 0.030478, "F": 0.20578, "G": 0.21147, "chunks": 210.72, "bytes_per_chunk": 4.86, "grad_norm": 0.9732, "lr": 4.39e-05, "router_trainable": true, "sec_per_step": 0.043, "alpha": 0.03, "t": 181.4}
117
+ {"event": "train", "step": 4350, "phase": 3, "objective": "global", "lm_loss": 1.14029, "bpb": 1.64509, "chunk_loss": 0.030692, "F": 0.21616, "G": 0.22163, "chunks": 221.34, "bytes_per_chunk": 4.626, "grad_norm": 0.9484, "lr": 4.2e-05, "router_trainable": true, "sec_per_step": 0.042, "alpha": 0.03, "t": 183.5}
118
+ {"event": "train", "step": 4400, "phase": 3, "objective": "global", "lm_loss": 1.13519, "bpb": 1.63773, "chunk_loss": 0.030559, "F": 0.21228, "G": 0.21784, "chunks": 217.38, "bytes_per_chunk": 4.711, "grad_norm": 1.068, "lr": 4.03e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 185.5}
119
+ {"event": "train", "step": 4450, "phase": 3, "objective": "global", "lm_loss": 1.05593, "bpb": 1.52338, "chunk_loss": 0.030461, "F": 0.2059, "G": 0.21199, "chunks": 210.84, "bytes_per_chunk": 4.857, "grad_norm": 1.0372, "lr": 3.87e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 187.6}
120
+ {"event": "probe", "step": 4500, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2668.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.197586, "cps": 27.791667, "mean_p": 0.204248}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 5.065967, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 189.6}
121
+ {"event": "eval", "step": 4500, "phase": 3, "bpb": {"en": 1.66983}, "bytes_per_chunk": {"en": 4.745}, "bpb_mean": 1.66983, "t": 189.8}
122
+ {"event": "train", "step": 4500, "phase": 3, "objective": "global", "lm_loss": 1.12616, "bpb": 1.62471, "chunk_loss": 0.030513, "F": 0.20874, "G": 0.21445, "chunks": 213.75, "bytes_per_chunk": 4.791, "grad_norm": 1.0024, "lr": 3.72e-05, "router_trainable": true, "sec_per_step": 0.046, "alpha": 0.03, "t": 189.9}
123
+ {"event": "train", "step": 4550, "phase": 3, "objective": "global", "lm_loss": 1.15383, "bpb": 1.66462, "chunk_loss": 0.030495, "F": 0.20779, "G": 0.21295, "chunks": 212.78, "bytes_per_chunk": 4.812, "grad_norm": 1.0579, "lr": 3.58e-05, "router_trainable": true, "sec_per_step": 0.042, "alpha": 0.03, "t": 192.0}
124
+ {"event": "train", "step": 4600, "phase": 3, "objective": "global", "lm_loss": 1.12586, "bpb": 1.62427, "chunk_loss": 0.030521, "F": 0.21118, "G": 0.21647, "chunks": 216.25, "bytes_per_chunk": 4.735, "grad_norm": 1.0163, "lr": 3.46e-05, "router_trainable": true, "sec_per_step": 0.042, "alpha": 0.03, "t": 194.1}
125
+ {"event": "train", "step": 4650, "phase": 3, "objective": "global", "lm_loss": 1.13155, "bpb": 1.63249, "chunk_loss": 0.030552, "F": 0.21106, "G": 0.21661, "chunks": 216.12, "bytes_per_chunk": 4.738, "grad_norm": 1.005, "lr": 3.35e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 196.1}
126
+ {"event": "train", "step": 4700, "phase": 3, "objective": "global", "lm_loss": 1.055, "bpb": 1.52204, "chunk_loss": 0.030565, "F": 0.21112, "G": 0.217, "chunks": 216.19, "bytes_per_chunk": 4.737, "grad_norm": 0.9976, "lr": 3.26e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 198.2}
127
+ {"event": "eval", "step": 4750, "phase": 3, "bpb": {"en": 1.65123}, "bytes_per_chunk": {"en": 4.748}, "bpb_mean": 1.65123, "t": 200.3}
128
+ {"event": "train", "step": 4750, "phase": 3, "objective": "global", "lm_loss": 1.08804, "bpb": 1.56971, "chunk_loss": 0.030496, "F": 0.21048, "G": 0.21598, "chunks": 215.53, "bytes_per_chunk": 4.751, "grad_norm": 0.9788, "lr": 3.18e-05, "router_trainable": true, "sec_per_step": 0.043, "alpha": 0.03, "t": 200.4}
129
+ {"event": "train", "step": 4800, "phase": 3, "objective": "global", "lm_loss": 1.17896, "bpb": 1.70087, "chunk_loss": 0.030509, "F": 0.2067, "G": 0.21234, "chunks": 211.66, "bytes_per_chunk": 4.838, "grad_norm": 1.065, "lr": 3.12e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 202.4}
130
+ {"event": "train", "step": 4850, "phase": 3, "objective": "global", "lm_loss": 1.0219, "bpb": 1.47429, "chunk_loss": 0.030411, "F": 0.20517, "G": 0.21044, "chunks": 210.09, "bytes_per_chunk": 4.874, "grad_norm": 1.0829, "lr": 3.07e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 204.5}
131
+ {"event": "train", "step": 4900, "phase": 3, "objective": "global", "lm_loss": 1.07913, "bpb": 1.55686, "chunk_loss": 0.030668, "F": 0.21558, "G": 0.22083, "chunks": 220.75, "bytes_per_chunk": 4.639, "grad_norm": 1.0028, "lr": 3.03e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 206.6}
132
+ {"event": "train", "step": 4950, "phase": 3, "objective": "global", "lm_loss": 1.1142, "bpb": 1.60745, "chunk_loss": 0.030397, "F": 0.20392, "G": 0.20968, "chunks": 208.81, "bytes_per_chunk": 4.904, "grad_norm": 1.0332, "lr": 3.01e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 208.6}
133
+ {"event": "probe", "step": 5000, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2668.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.197586, "cps": 27.791667, "mean_p": 0.203833}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 5.065967, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 210.7}
134
+ {"event": "eval", "step": 5000, "phase": 3, "bpb": {"en": 1.62756}, "bytes_per_chunk": {"en": 4.745}, "bpb_mean": 1.62756, "t": 210.9}
135
+ {"event": "probe", "step": 5000, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2668.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.197586, "cps": 27.791667, "mean_p": 0.203833}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 5.065967, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 210.9}
136
+ {"event": "final", "run": "baseline_en", "condition": "A", "step": 5000, "params": 22516992, "per_lang": {"en": {"cpc": 0.19758572169147598, "cps": 27.791666666666668, "bpb": 1.623357290677685, "bytes_per_chunk": 4.761882271732318}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 5.065967016491754, "transitions": [{"step": 0, "phase": 1, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}, {"step": 2000, "phase": 2, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}, {"step": 3500, "phase": 3, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}], "t": 211.2}
baseline_en/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:76c323e35fc88804fc5994b1284dbb8eaa3cac3dc18e19f1beef8c1d02169405
3
+ size 90078296
emactl_genreg_s0/config.json ADDED
@@ -0,0 +1,108 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run": "emactl_genreg_s0",
3
+ "step": 6000,
4
+ "model_config": {
5
+ "vocab_size": 256,
6
+ "d_enc": 256,
7
+ "d_main": 512,
8
+ "n_enc": 2,
9
+ "n_dec": 2,
10
+ "n_main": 6,
11
+ "n_heads_enc": 4,
12
+ "n_heads_main": 8,
13
+ "seq_len": 1024,
14
+ "dropout": 0.0,
15
+ "target_ratio": 6.0,
16
+ "max_chunk_mult": 3.0,
17
+ "dechunk": "ste",
18
+ "router_noise": 0.0,
19
+ "rope_base": 10000.0
20
+ },
21
+ "tied_weights": {
22
+ "head.weight": "emb.weight"
23
+ },
24
+ "train_config": {
25
+ "condition": "A",
26
+ "seed": 0,
27
+ "out_dir": null,
28
+ "run_name": "emactl_genreg_s0",
29
+ "corpus": "corpus",
30
+ "langs": null,
31
+ "seq_len": 1024,
32
+ "batch_size": 32,
33
+ "grad_accum": 1,
34
+ "steps": 6000,
35
+ "lr": 0.0003,
36
+ "min_lr_frac": 0.1,
37
+ "warmup": 300,
38
+ "weight_decay": 0.1,
39
+ "grad_clip": 1.0,
40
+ "betas": [
41
+ 0.9,
42
+ 0.95
43
+ ],
44
+ "router_lr_mult": 1.0,
45
+ "d_enc": 256,
46
+ "d_main": 512,
47
+ "n_enc": 2,
48
+ "n_dec": 2,
49
+ "n_main": 6,
50
+ "n_heads_enc": 4,
51
+ "n_heads_main": 8,
52
+ "dechunk": "ste",
53
+ "target_ratio": 6.0,
54
+ "alpha": 0.03,
55
+ "alpha_parity_cap": 0.06,
56
+ "global_anchor": 0.0,
57
+ "premium_clip": 4.0,
58
+ "premium_kind": "flores_per_sent",
59
+ "mixture": null,
60
+ "doc_filters": null,
61
+ "tau1": 0.4,
62
+ "tau2": 0.7,
63
+ "freeze_scope": "router",
64
+ "ckpt_every": 200,
65
+ "eval_every": 200,
66
+ "log_every": 50,
67
+ "probe_sents": 128,
68
+ "eval_batches": 8,
69
+ "keep_ckpt": false,
70
+ "amp": "bf16",
71
+ "compile": false,
72
+ "cpc_decay": 0.9,
73
+ "device": "cuda",
74
+ "log_spaced_probe": true,
75
+ "probe_bpb_batches": 4,
76
+ "dump_reps": true,
77
+ "dump_reps_thresholds": [
78
+ 0.3,
79
+ 0.4,
80
+ 0.5,
81
+ 0.6,
82
+ 0.7
83
+ ],
84
+ "router_noise": 0.05,
85
+ "genreg_target_entropy": 0.2983376681804657,
86
+ "genreg_eta": 0.05
87
+ },
88
+ "probe_agg": {
89
+ "gini_cpc": 0.1718704557282591,
90
+ "gini_cps": 0.16855087999202278,
91
+ "bpic": 4.783629156902827,
92
+ "worst_lang": "th",
93
+ "cpc_parity_vs_en": {
94
+ "en": 1.0,
95
+ "de": 1.0620702283220003,
96
+ "vi": 1.1153994423964175,
97
+ "tr": 1.3415190553683638,
98
+ "fi": 1.1170363720974708,
99
+ "ru": 1.4855197692415063,
100
+ "el": 1.5955613423948596,
101
+ "hi": 2.0414684177775593,
102
+ "ta": 1.9938978371357505,
103
+ "th": 2.437200907814152,
104
+ "zh": 2.365709365943872,
105
+ "my": 1.8268620537941986
106
+ }
107
+ }
108
+ }
emactl_genreg_s0/final.json ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "event": "final",
3
+ "run": "emactl_genreg_s0",
4
+ "condition": "A",
5
+ "step": 6000,
6
+ "params": 22516992,
7
+ "per_lang": {
8
+ "en": {
9
+ "cpc": 0.24656668390507383,
10
+ "cps": 33.5234375,
11
+ "bpb": 2.0362594194956314,
12
+ "bytes_per_chunk": 4.029017582687815
13
+ },
14
+ "de": {
15
+ "cpc": 0.26187113427166026,
16
+ "cps": 42.0078125,
17
+ "bpb": 2.210904942235243,
18
+ "bytes_per_chunk": 3.879821211852115
19
+ },
20
+ "vi": {
21
+ "cpc": 0.27502034174125306,
22
+ "cps": 39.609375,
23
+ "bpb": 1.914473000974849,
24
+ "bytes_per_chunk": 4.687504470352621
25
+ },
26
+ "tr": {
27
+ "cpc": 0.3307739048776446,
28
+ "cps": 46.78125,
29
+ "bpb": 2.163528064484473,
30
+ "bytes_per_chunk": 3.2757763199000314
31
+ },
32
+ "fi": {
33
+ "cpc": 0.2754239540694275,
34
+ "cps": 40.4765625,
35
+ "bpb": 2.387229897058361,
36
+ "bytes_per_chunk": 3.928634050939282
37
+ },
38
+ "ru": {
39
+ "cpc": 0.3662796833773087,
40
+ "cps": 54.2265625,
41
+ "bpb": 1.354086676957211,
42
+ "bytes_per_chunk": 4.78299502805273
43
+ },
44
+ "el": {
45
+ "cpc": 0.39341226916142863,
46
+ "cps": 63.078125,
47
+ "bpb": 1.4254445253235672,
48
+ "bytes_per_chunk": 4.771372927322036
49
+ },
50
+ "hi": {
51
+ "cpc": 0.5033580980683506,
52
+ "cps": 66.1640625,
53
+ "bpb": 1.15441516864525,
54
+ "bytes_per_chunk": 5.213268634157982
55
+ },
56
+ "ta": {
57
+ "cpc": 0.49162877774806096,
58
+ "cps": 71.8046875,
59
+ "bpb": 0.9478137860249648,
60
+ "bytes_per_chunk": 5.832161608970366
61
+ },
62
+ "th": {
63
+ "cpc": 0.600932545850171,
64
+ "cps": 75.515625,
65
+ "bpb": 1.0961344833406215,
66
+ "bytes_per_chunk": 4.624166519668372
67
+ },
68
+ "zh": {
69
+ "cpc": 0.5833051134439553,
70
+ "cps": 26.9140625,
71
+ "bpb": 3.1291131026693577,
72
+ "bytes_per_chunk": 4.784129794048673
73
+ },
74
+ "my": {
75
+ "cpc": 0.45044331855604813,
76
+ "cps": 66.6796875,
77
+ "bpb": 1.0903263282513456,
78
+ "bytes_per_chunk": 6.048406820331787
79
+ }
80
+ },
81
+ "gini_cpc": 0.1718704557282591,
82
+ "gini_cps": 0.16855087999202278,
83
+ "bpic": 4.783629156902827,
84
+ "seed": 0,
85
+ "mean_boundary_entropy": 0.6691246032714844,
86
+ "dechunk": "ste",
87
+ "router_noise": 0.0,
88
+ "transitions": [
89
+ {
90
+ "step": 0,
91
+ "phase": 1,
92
+ "objective": "global",
93
+ "router_trainable": true,
94
+ "encoder_trainable": true,
95
+ "frozen_router_params": 0
96
+ },
97
+ {
98
+ "step": 2400,
99
+ "phase": 2,
100
+ "objective": "global",
101
+ "router_trainable": true,
102
+ "encoder_trainable": true,
103
+ "frozen_router_params": 0
104
+ },
105
+ {
106
+ "step": 4200,
107
+ "phase": 3,
108
+ "objective": "global",
109
+ "router_trainable": true,
110
+ "encoder_trainable": true,
111
+ "frozen_router_params": 0
112
+ }
113
+ ]
114
+ }
emactl_genreg_s0/log.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
emactl_genreg_s0/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0ca2f328d6a03e833f2ea3f19c0389b985ebb5bd6660f43926a52c65cfb50662
3
+ size 90078296
emactl_genreg_s1/config.json ADDED
@@ -0,0 +1,108 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run": "emactl_genreg_s1",
3
+ "step": 6000,
4
+ "model_config": {
5
+ "vocab_size": 256,
6
+ "d_enc": 256,
7
+ "d_main": 512,
8
+ "n_enc": 2,
9
+ "n_dec": 2,
10
+ "n_main": 6,
11
+ "n_heads_enc": 4,
12
+ "n_heads_main": 8,
13
+ "seq_len": 1024,
14
+ "dropout": 0.0,
15
+ "target_ratio": 6.0,
16
+ "max_chunk_mult": 3.0,
17
+ "dechunk": "ste",
18
+ "router_noise": 0.0,
19
+ "rope_base": 10000.0
20
+ },
21
+ "tied_weights": {
22
+ "head.weight": "emb.weight"
23
+ },
24
+ "train_config": {
25
+ "condition": "A",
26
+ "seed": 1,
27
+ "out_dir": null,
28
+ "run_name": "emactl_genreg_s1",
29
+ "corpus": "corpus",
30
+ "langs": null,
31
+ "seq_len": 1024,
32
+ "batch_size": 32,
33
+ "grad_accum": 1,
34
+ "steps": 6000,
35
+ "lr": 0.0003,
36
+ "min_lr_frac": 0.1,
37
+ "warmup": 300,
38
+ "weight_decay": 0.1,
39
+ "grad_clip": 1.0,
40
+ "betas": [
41
+ 0.9,
42
+ 0.95
43
+ ],
44
+ "router_lr_mult": 1.0,
45
+ "d_enc": 256,
46
+ "d_main": 512,
47
+ "n_enc": 2,
48
+ "n_dec": 2,
49
+ "n_main": 6,
50
+ "n_heads_enc": 4,
51
+ "n_heads_main": 8,
52
+ "dechunk": "ste",
53
+ "target_ratio": 6.0,
54
+ "alpha": 0.03,
55
+ "alpha_parity_cap": 0.06,
56
+ "global_anchor": 0.0,
57
+ "premium_clip": 4.0,
58
+ "premium_kind": "flores_per_sent",
59
+ "mixture": null,
60
+ "doc_filters": null,
61
+ "tau1": 0.4,
62
+ "tau2": 0.7,
63
+ "freeze_scope": "router",
64
+ "ckpt_every": 200,
65
+ "eval_every": 200,
66
+ "log_every": 50,
67
+ "probe_sents": 128,
68
+ "eval_batches": 8,
69
+ "keep_ckpt": false,
70
+ "amp": "bf16",
71
+ "compile": false,
72
+ "cpc_decay": 0.9,
73
+ "device": "cuda",
74
+ "log_spaced_probe": true,
75
+ "probe_bpb_batches": 4,
76
+ "dump_reps": true,
77
+ "dump_reps_thresholds": [
78
+ 0.3,
79
+ 0.4,
80
+ 0.5,
81
+ 0.6,
82
+ 0.7
83
+ ],
84
+ "router_noise": 0.05,
85
+ "genreg_target_entropy": 0.2983376681804657,
86
+ "genreg_eta": 0.05
87
+ },
88
+ "probe_agg": {
89
+ "gini_cpc": 0.23636601785882339,
90
+ "gini_cps": 0.23964209275225068,
91
+ "bpic": 5.3453626196080615,
92
+ "worst_lang": "ta",
93
+ "cpc_parity_vs_en": {
94
+ "en": 1.0,
95
+ "de": 0.8729681628237771,
96
+ "vi": 1.3464658214709873,
97
+ "tr": 1.1633108961151755,
98
+ "fi": 0.8345095972200084,
99
+ "ru": 1.9504086084814982,
100
+ "el": 1.1624065248294972,
101
+ "hi": 2.1645924087529833,
102
+ "ta": 3.184870546720561,
103
+ "th": 2.322431752422665,
104
+ "zh": 2.5401997376043726,
105
+ "my": 2.1504683399257214
106
+ }
107
+ }
108
+ }
emactl_genreg_s1/final.json ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "event": "final",
3
+ "run": "emactl_genreg_s1",
4
+ "condition": "A",
5
+ "step": 6000,
6
+ "params": 22516992,
7
+ "per_lang": {
8
+ "en": {
9
+ "cpc": 0.20703327012584039,
10
+ "cps": 28.1484375,
11
+ "bpb": 2.050497110736466,
12
+ "bytes_per_chunk": 4.915092482351948
13
+ },
14
+ "de": {
15
+ "cpc": 0.18073345346515365,
16
+ "cps": 28.9921875,
17
+ "bpb": 2.234483494945642,
18
+ "bytes_per_chunk": 5.727357140516271
19
+ },
20
+ "vi": {
21
+ "cpc": 0.2787632221318145,
22
+ "cps": 40.1484375,
23
+ "bpb": 1.889786890741809,
24
+ "bytes_per_chunk": 5.128263314911723
25
+ },
26
+ "tr": {
27
+ "cpc": 0.24084405899574657,
28
+ "cps": 34.0625,
29
+ "bpb": 2.200235880250896,
30
+ "bytes_per_chunk": 4.5643048046871595
31
+ },
32
+ "fi": {
33
+ "cpc": 0.17277125086385625,
34
+ "cps": 25.390625,
35
+ "bpb": 2.410347579260111,
36
+ "bytes_per_chunk": 5.786971014812689
37
+ },
38
+ "ru": {
39
+ "cpc": 0.4037994722955145,
40
+ "cps": 59.78125,
41
+ "bpb": 1.3596733775112122,
42
+ "bytes_per_chunk": 4.511013215859031
43
+ },
44
+ "el": {
45
+ "cpc": 0.24065682405106467,
46
+ "cps": 38.5859375,
47
+ "bpb": 1.470116008487108,
48
+ "bytes_per_chunk": 7.593644540358906
49
+ },
50
+ "hi": {
51
+ "cpc": 0.44814264487369987,
52
+ "cps": 58.90625,
53
+ "bpb": 1.1866793342998843,
54
+ "bytes_per_chunk": 5.916870746763872
55
+ },
56
+ "ta": {
57
+ "cpc": 0.6593741642150308,
58
+ "cps": 96.3046875,
59
+ "bpb": 0.9440668483807373,
60
+ "bytes_per_chunk": 4.056543773453519
61
+ },
62
+ "th": {
63
+ "cpc": 0.4808206403481505,
64
+ "cps": 60.421875,
65
+ "bpb": 1.1085081298711636,
66
+ "bytes_per_chunk": 5.815535811342939
67
+ },
68
+ "zh": {
69
+ "cpc": 0.5259058584490349,
70
+ "cps": 24.265625,
71
+ "bpb": 3.1290566063685494,
72
+ "bytes_per_chunk": 5.197248161145145
73
+ },
74
+ "my": {
75
+ "cpc": 0.44521849271690944,
76
+ "cps": 65.90625,
77
+ "bpb": 1.1147818581196989,
78
+ "bytes_per_chunk": 6.384024353120243
79
+ }
80
+ },
81
+ "gini_cpc": 0.23636601785882339,
82
+ "gini_cps": 0.23964209275225068,
83
+ "bpic": 5.3453626196080615,
84
+ "seed": 1,
85
+ "mean_boundary_entropy": 0.6663709878921509,
86
+ "dechunk": "ste",
87
+ "router_noise": 0.0,
88
+ "transitions": [
89
+ {
90
+ "step": 0,
91
+ "phase": 1,
92
+ "objective": "global",
93
+ "router_trainable": true,
94
+ "encoder_trainable": true,
95
+ "frozen_router_params": 0
96
+ },
97
+ {
98
+ "step": 2400,
99
+ "phase": 2,
100
+ "objective": "global",
101
+ "router_trainable": true,
102
+ "encoder_trainable": true,
103
+ "frozen_router_params": 0
104
+ },
105
+ {
106
+ "step": 4200,
107
+ "phase": 3,
108
+ "objective": "global",
109
+ "router_trainable": true,
110
+ "encoder_trainable": true,
111
+ "frozen_router_params": 0
112
+ }
113
+ ]
114
+ }
emactl_genreg_s1/log.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
emactl_genreg_s1/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:06778a23c53100ab5de1587298616275e42e7f0633752789d48bf74145d32df9
3
+ size 90078296
emactl_genreg_s2/config.json ADDED
@@ -0,0 +1,108 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run": "emactl_genreg_s2",
3
+ "step": 6000,
4
+ "model_config": {
5
+ "vocab_size": 256,
6
+ "d_enc": 256,
7
+ "d_main": 512,
8
+ "n_enc": 2,
9
+ "n_dec": 2,
10
+ "n_main": 6,
11
+ "n_heads_enc": 4,
12
+ "n_heads_main": 8,
13
+ "seq_len": 1024,
14
+ "dropout": 0.0,
15
+ "target_ratio": 6.0,
16
+ "max_chunk_mult": 3.0,
17
+ "dechunk": "ste",
18
+ "router_noise": 0.0,
19
+ "rope_base": 10000.0
20
+ },
21
+ "tied_weights": {
22
+ "head.weight": "emb.weight"
23
+ },
24
+ "train_config": {
25
+ "condition": "A",
26
+ "seed": 2,
27
+ "out_dir": null,
28
+ "run_name": "emactl_genreg_s2",
29
+ "corpus": "corpus",
30
+ "langs": null,
31
+ "seq_len": 1024,
32
+ "batch_size": 32,
33
+ "grad_accum": 1,
34
+ "steps": 6000,
35
+ "lr": 0.0003,
36
+ "min_lr_frac": 0.1,
37
+ "warmup": 300,
38
+ "weight_decay": 0.1,
39
+ "grad_clip": 1.0,
40
+ "betas": [
41
+ 0.9,
42
+ 0.95
43
+ ],
44
+ "router_lr_mult": 1.0,
45
+ "d_enc": 256,
46
+ "d_main": 512,
47
+ "n_enc": 2,
48
+ "n_dec": 2,
49
+ "n_main": 6,
50
+ "n_heads_enc": 4,
51
+ "n_heads_main": 8,
52
+ "dechunk": "ste",
53
+ "target_ratio": 6.0,
54
+ "alpha": 0.03,
55
+ "alpha_parity_cap": 0.06,
56
+ "global_anchor": 0.0,
57
+ "premium_clip": 4.0,
58
+ "premium_kind": "flores_per_sent",
59
+ "mixture": null,
60
+ "doc_filters": null,
61
+ "tau1": 0.4,
62
+ "tau2": 0.7,
63
+ "freeze_scope": "router",
64
+ "ckpt_every": 200,
65
+ "eval_every": 200,
66
+ "log_every": 50,
67
+ "probe_sents": 128,
68
+ "eval_batches": 8,
69
+ "keep_ckpt": false,
70
+ "amp": "bf16",
71
+ "compile": false,
72
+ "cpc_decay": 0.9,
73
+ "device": "cuda",
74
+ "log_spaced_probe": true,
75
+ "probe_bpb_batches": 4,
76
+ "dump_reps": true,
77
+ "dump_reps_thresholds": [
78
+ 0.3,
79
+ 0.4,
80
+ 0.5,
81
+ 0.6,
82
+ 0.7
83
+ ],
84
+ "router_noise": 0.05,
85
+ "genreg_target_entropy": 0.2983376681804657,
86
+ "genreg_eta": 0.05
87
+ },
88
+ "probe_agg": {
89
+ "gini_cpc": 0.18418976005934495,
90
+ "gini_cps": 0.19093854335748328,
91
+ "bpic": 5.064610633833485,
92
+ "worst_lang": "th",
93
+ "cpc_parity_vs_en": {
94
+ "en": 1.0,
95
+ "de": 0.9596706497405335,
96
+ "vi": 1.1269902451948222,
97
+ "tr": 1.1782876670438764,
98
+ "fi": 0.9269366133816891,
99
+ "ru": 1.3836420016087154,
100
+ "el": 1.6537874904343381,
101
+ "hi": 1.978828615480954,
102
+ "ta": 1.9833385074132006,
103
+ "th": 2.2785237369270823,
104
+ "zh": 2.270987076557597,
105
+ "my": 2.1670182902343194
106
+ }
107
+ }
108
+ }
emactl_genreg_s2/final.json ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "event": "final",
3
+ "run": "emactl_genreg_s2",
4
+ "condition": "A",
5
+ "step": 6000,
6
+ "params": 22516992,
7
+ "per_lang": {
8
+ "en": {
9
+ "cpc": 0.23806240303395967,
10
+ "cps": 32.3671875,
11
+ "bpb": 2.0372584237188307,
12
+ "bytes_per_chunk": 4.226799635598481
13
+ },
14
+ "de": {
15
+ "cpc": 0.22846150099839282,
16
+ "cps": 36.6484375,
17
+ "bpb": 2.226351327546341,
18
+ "bytes_per_chunk": 4.347402112804524
19
+ },
20
+ "vi": {
21
+ "cpc": 0.2682940059669108,
22
+ "cps": 38.640625,
23
+ "bpb": 1.8961533875071113,
24
+ "bytes_per_chunk": 4.608639088623618
25
+ },
26
+ "tr": {
27
+ "cpc": 0.2805059934817434,
28
+ "cps": 39.671875,
29
+ "bpb": 2.1642174461915413,
30
+ "bytes_per_chunk": 3.7224202320264688
31
+ },
32
+ "fi": {
33
+ "cpc": 0.22066875764180532,
34
+ "cps": 32.4296875,
35
+ "bpb": 2.3910613802988614,
36
+ "bytes_per_chunk": 4.665106553365662
37
+ },
38
+ "ru": {
39
+ "cpc": 0.32939313984168866,
40
+ "cps": 48.765625,
41
+ "bpb": 1.3742821696914036,
42
+ "bytes_per_chunk": 5.337351114730734
43
+ },
44
+ "el": {
45
+ "cpc": 0.39370462408030016,
46
+ "cps": 63.125,
47
+ "bpb": 1.4151377179209739,
48
+ "bytes_per_chunk": 4.5019878581794135
49
+ },
50
+ "hi": {
51
+ "cpc": 0.4710846953937593,
52
+ "cps": 61.921875,
53
+ "bpb": 1.1513827221751163,
54
+ "bytes_per_chunk": 5.574092581173319
55
+ },
56
+ "ta": {
57
+ "cpc": 0.4721583311045734,
58
+ "cps": 68.9609375,
59
+ "bpb": 0.9525330490836764,
60
+ "bytes_per_chunk": 5.550194255952066
61
+ },
62
+ "th": {
63
+ "cpc": 0.542430836182779,
64
+ "cps": 68.1640625,
65
+ "bpb": 1.090443856895379,
66
+ "bytes_per_chunk": 5.2221480721535505
67
+ },
68
+ "zh": {
69
+ "cpc": 0.5406366407043685,
70
+ "cps": 24.9453125,
71
+ "bpb": 3.1167576970487225,
72
+ "bytes_per_chunk": 5.181940381118052
73
+ },
74
+ "my": {
75
+ "cpc": 0.5158855815917247,
76
+ "cps": 76.3671875,
77
+ "bpb": 1.1004410095350938,
78
+ "bytes_per_chunk": 5.546905913096838
79
+ }
80
+ },
81
+ "gini_cpc": 0.18418976005934495,
82
+ "gini_cps": 0.19093854335748328,
83
+ "bpic": 5.064610633833485,
84
+ "seed": 2,
85
+ "mean_boundary_entropy": 0.6683645248413086,
86
+ "dechunk": "ste",
87
+ "router_noise": 0.0,
88
+ "transitions": [
89
+ {
90
+ "step": 0,
91
+ "phase": 1,
92
+ "objective": "global",
93
+ "router_trainable": true,
94
+ "encoder_trainable": true,
95
+ "frozen_router_params": 0
96
+ },
97
+ {
98
+ "step": 2400,
99
+ "phase": 2,
100
+ "objective": "global",
101
+ "router_trainable": true,
102
+ "encoder_trainable": true,
103
+ "frozen_router_params": 0
104
+ },
105
+ {
106
+ "step": 4200,
107
+ "phase": 3,
108
+ "objective": "global",
109
+ "router_trainable": true,
110
+ "encoder_trainable": true,
111
+ "frozen_router_params": 0
112
+ }
113
+ ]
114
+ }
emactl_genreg_s2/log.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
emactl_genreg_s2/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bf09af7f57faa2a81c3078a159cc4164075cec8b0cf836a9c10cb9797566592e
3
+ size 90078296
emactl_noema_s0/config.json ADDED
@@ -0,0 +1,108 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run": "emactl_noema_s0",
3
+ "step": 6000,
4
+ "model_config": {
5
+ "vocab_size": 256,
6
+ "d_enc": 256,
7
+ "d_main": 512,
8
+ "n_enc": 2,
9
+ "n_dec": 2,
10
+ "n_main": 6,
11
+ "n_heads_enc": 4,
12
+ "n_heads_main": 8,
13
+ "seq_len": 1024,
14
+ "dropout": 0.0,
15
+ "target_ratio": 6.0,
16
+ "max_chunk_mult": 3.0,
17
+ "dechunk": "ste",
18
+ "router_noise": 0.0,
19
+ "rope_base": 10000.0
20
+ },
21
+ "tied_weights": {
22
+ "head.weight": "emb.weight"
23
+ },
24
+ "train_config": {
25
+ "condition": "A",
26
+ "seed": 0,
27
+ "out_dir": null,
28
+ "run_name": "emactl_noema_s0",
29
+ "corpus": "corpus",
30
+ "langs": null,
31
+ "seq_len": 1024,
32
+ "batch_size": 32,
33
+ "grad_accum": 1,
34
+ "steps": 6000,
35
+ "lr": 0.0003,
36
+ "min_lr_frac": 0.1,
37
+ "warmup": 300,
38
+ "weight_decay": 0.1,
39
+ "grad_clip": 1.0,
40
+ "betas": [
41
+ 0.9,
42
+ 0.95
43
+ ],
44
+ "router_lr_mult": 1.0,
45
+ "d_enc": 256,
46
+ "d_main": 512,
47
+ "n_enc": 2,
48
+ "n_dec": 2,
49
+ "n_main": 6,
50
+ "n_heads_enc": 4,
51
+ "n_heads_main": 8,
52
+ "dechunk": "ste",
53
+ "target_ratio": 6.0,
54
+ "alpha": 0.03,
55
+ "alpha_parity_cap": 0.06,
56
+ "global_anchor": 0.0,
57
+ "premium_clip": 4.0,
58
+ "premium_kind": "flores_per_sent",
59
+ "mixture": null,
60
+ "doc_filters": null,
61
+ "tau1": 0.4,
62
+ "tau2": 0.7,
63
+ "freeze_scope": "router",
64
+ "ckpt_every": 200,
65
+ "eval_every": 200,
66
+ "log_every": 50,
67
+ "probe_sents": 128,
68
+ "eval_batches": 8,
69
+ "keep_ckpt": false,
70
+ "amp": "bf16",
71
+ "compile": false,
72
+ "cpc_decay": 0.9,
73
+ "device": "cuda",
74
+ "log_spaced_probe": true,
75
+ "probe_bpb_batches": 4,
76
+ "dump_reps": true,
77
+ "dump_reps_thresholds": [
78
+ 0.3,
79
+ 0.4,
80
+ 0.5,
81
+ 0.6,
82
+ 0.7
83
+ ],
84
+ "router_noise": 0.0,
85
+ "genreg_target_entropy": null,
86
+ "genreg_eta": 0.05
87
+ },
88
+ "probe_agg": {
89
+ "gini_cpc": 0.18611578592940914,
90
+ "gini_cps": 0.18417783785347885,
91
+ "bpic": 4.7592480065477005,
92
+ "worst_lang": "th",
93
+ "cpc_parity_vs_en": {
94
+ "en": 1.0,
95
+ "de": 1.1355650278292462,
96
+ "vi": 1.1711008424112148,
97
+ "tr": 1.3542351354314985,
98
+ "fi": 1.2376760342670254,
99
+ "ru": 1.630438979246249,
100
+ "el": 1.5757155604814694,
101
+ "hi": 2.1204249628528977,
102
+ "ta": 1.9395870074606907,
103
+ "th": 2.816542017917871,
104
+ "zh": 2.4966017356495547,
105
+ "my": 2.3098610021457
106
+ }
107
+ }
108
+ }
emactl_noema_s0/final.json ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "event": "final",
3
+ "run": "emactl_noema_s0",
4
+ "condition": "A",
5
+ "step": 6000,
6
+ "params": 22516992,
7
+ "per_lang": {
8
+ "en": {
9
+ "cpc": 0.23099465609377692,
10
+ "cps": 31.40625,
11
+ "bpb": 2.0382534508432286,
12
+ "bytes_per_chunk": 4.302768180288718
13
+ },
14
+ "de": {
15
+ "cpc": 0.26230945307553694,
16
+ "cps": 42.078125,
17
+ "bpb": 2.2124327286163514,
18
+ "bytes_per_chunk": 3.8386598428771204
19
+ },
20
+ "vi": {
21
+ "cpc": 0.27051803634391103,
22
+ "cps": 38.9609375,
23
+ "bpb": 1.9233381799299585,
24
+ "bytes_per_chunk": 4.745675570480733
25
+ },
26
+ "tr": {
27
+ "cpc": 0.3128210793791084,
28
+ "cps": 44.2421875,
29
+ "bpb": 2.1762234583080327,
30
+ "bytes_per_chunk": 3.449739766678291
31
+ },
32
+ "fi": {
33
+ "cpc": 0.2858965498910212,
34
+ "cps": 42.015625,
35
+ "bpb": 2.3890020062928516,
36
+ "bytes_per_chunk": 3.6961775446611114
37
+ },
38
+ "ru": {
39
+ "cpc": 0.376622691292876,
40
+ "cps": 55.7578125,
41
+ "bpb": 1.3563091872564799,
42
+ "bytes_per_chunk": 4.759763958238765
43
+ },
44
+ "el": {
45
+ "cpc": 0.36398187399502996,
46
+ "cps": 58.359375,
47
+ "bpb": 1.431533232486521,
48
+ "bytes_per_chunk": 5.101963760923298
49
+ },
50
+ "hi": {
51
+ "cpc": 0.4898068350668648,
52
+ "cps": 64.3828125,
53
+ "bpb": 1.1639179249083127,
54
+ "bytes_per_chunk": 5.340885244231651
55
+ },
56
+ "ta": {
57
+ "cpc": 0.4480342337523402,
58
+ "cps": 65.4375,
59
+ "bpb": 0.9624026067198248,
60
+ "bytes_per_chunk": 6.288613546676902
61
+ },
62
+ "th": {
63
+ "cpc": 0.6506061548026111,
64
+ "cps": 81.7578125,
65
+ "bpb": 1.0589655444660309,
66
+ "bytes_per_chunk": 4.42780893182893
67
+ },
68
+ "zh": {
69
+ "cpc": 0.5767016593294955,
70
+ "cps": 26.609375,
71
+ "bpb": 3.1217361863324053,
72
+ "bytes_per_chunk": 4.9474200732268905
73
+ },
74
+ "my": {
75
+ "cpc": 0.5335655478150728,
76
+ "cps": 78.984375,
77
+ "bpb": 1.0843777332042663,
78
+ "bytes_per_chunk": 5.2943944580770905
79
+ }
80
+ },
81
+ "gini_cpc": 0.18611578592940914,
82
+ "gini_cps": 0.18417783785347885,
83
+ "bpic": 4.7592480065477005,
84
+ "seed": 0,
85
+ "mean_boundary_entropy": 0.6694673299789429,
86
+ "dechunk": "ste",
87
+ "router_noise": 0.0,
88
+ "transitions": [
89
+ {
90
+ "step": 0,
91
+ "phase": 1,
92
+ "objective": "global",
93
+ "router_trainable": true,
94
+ "encoder_trainable": true,
95
+ "frozen_router_params": 0
96
+ },
97
+ {
98
+ "step": 2400,
99
+ "phase": 2,
100
+ "objective": "global",
101
+ "router_trainable": true,
102
+ "encoder_trainable": true,
103
+ "frozen_router_params": 0
104
+ },
105
+ {
106
+ "step": 4200,
107
+ "phase": 3,
108
+ "objective": "global",
109
+ "router_trainable": true,
110
+ "encoder_trainable": true,
111
+ "frozen_router_params": 0
112
+ }
113
+ ]
114
+ }
emactl_noema_s0/log.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
emactl_noema_s0/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e3e5d1709a39d0e054fa636652f95a102353a4a8cf1f7d5a1366fde46872746b
3
+ size 90078296
emactl_noema_s1/config.json ADDED
@@ -0,0 +1,108 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run": "emactl_noema_s1",
3
+ "step": 6000,
4
+ "model_config": {
5
+ "vocab_size": 256,
6
+ "d_enc": 256,
7
+ "d_main": 512,
8
+ "n_enc": 2,
9
+ "n_dec": 2,
10
+ "n_main": 6,
11
+ "n_heads_enc": 4,
12
+ "n_heads_main": 8,
13
+ "seq_len": 1024,
14
+ "dropout": 0.0,
15
+ "target_ratio": 6.0,
16
+ "max_chunk_mult": 3.0,
17
+ "dechunk": "ste",
18
+ "router_noise": 0.0,
19
+ "rope_base": 10000.0
20
+ },
21
+ "tied_weights": {
22
+ "head.weight": "emb.weight"
23
+ },
24
+ "train_config": {
25
+ "condition": "A",
26
+ "seed": 1,
27
+ "out_dir": null,
28
+ "run_name": "emactl_noema_s1",
29
+ "corpus": "corpus",
30
+ "langs": null,
31
+ "seq_len": 1024,
32
+ "batch_size": 32,
33
+ "grad_accum": 1,
34
+ "steps": 6000,
35
+ "lr": 0.0003,
36
+ "min_lr_frac": 0.1,
37
+ "warmup": 300,
38
+ "weight_decay": 0.1,
39
+ "grad_clip": 1.0,
40
+ "betas": [
41
+ 0.9,
42
+ 0.95
43
+ ],
44
+ "router_lr_mult": 1.0,
45
+ "d_enc": 256,
46
+ "d_main": 512,
47
+ "n_enc": 2,
48
+ "n_dec": 2,
49
+ "n_main": 6,
50
+ "n_heads_enc": 4,
51
+ "n_heads_main": 8,
52
+ "dechunk": "ste",
53
+ "target_ratio": 6.0,
54
+ "alpha": 0.03,
55
+ "alpha_parity_cap": 0.06,
56
+ "global_anchor": 0.0,
57
+ "premium_clip": 4.0,
58
+ "premium_kind": "flores_per_sent",
59
+ "mixture": null,
60
+ "doc_filters": null,
61
+ "tau1": 0.4,
62
+ "tau2": 0.7,
63
+ "freeze_scope": "router",
64
+ "ckpt_every": 200,
65
+ "eval_every": 200,
66
+ "log_every": 50,
67
+ "probe_sents": 128,
68
+ "eval_batches": 8,
69
+ "keep_ckpt": false,
70
+ "amp": "bf16",
71
+ "compile": false,
72
+ "cpc_decay": 0.9,
73
+ "device": "cuda",
74
+ "log_spaced_probe": true,
75
+ "probe_bpb_batches": 4,
76
+ "dump_reps": true,
77
+ "dump_reps_thresholds": [
78
+ 0.3,
79
+ 0.4,
80
+ 0.5,
81
+ 0.6,
82
+ 0.7
83
+ ],
84
+ "router_noise": 0.0,
85
+ "genreg_target_entropy": null,
86
+ "genreg_eta": 0.05
87
+ },
88
+ "probe_agg": {
89
+ "gini_cpc": 0.23693031160314426,
90
+ "gini_cps": 0.24140331101544144,
91
+ "bpic": 5.212078823353659,
92
+ "worst_lang": "ta",
93
+ "cpc_parity_vs_en": {
94
+ "en": 1.0,
95
+ "de": 0.843470152412457,
96
+ "vi": 1.2215531661527395,
97
+ "tr": 1.130235998752015,
98
+ "fi": 0.8248924871143106,
99
+ "ru": 1.611817706976797,
100
+ "el": 1.6979980353874928,
101
+ "hi": 2.016545670210256,
102
+ "ta": 3.3572490033712628,
103
+ "th": 2.1049647560372007,
104
+ "zh": 2.5372298157544013,
105
+ "my": 2.089203249734308
106
+ }
107
+ }
108
+ }
emactl_noema_s1/final.json ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "event": "final",
3
+ "run": "emactl_noema_s1",
4
+ "condition": "A",
5
+ "step": 6000,
6
+ "params": 22516992,
7
+ "per_lang": {
8
+ "en": {
9
+ "cpc": 0.2142159397804976,
10
+ "cps": 29.125,
11
+ "bpb": 2.0525959332630173,
12
+ "bytes_per_chunk": 4.695102402679395
13
+ },
14
+ "de": {
15
+ "cpc": 0.18068475137583404,
16
+ "cps": 28.984375,
17
+ "bpb": 2.2386571699167805,
18
+ "bytes_per_chunk": 5.84470976444489
19
+ },
20
+ "vi": {
21
+ "cpc": 0.26167615947925144,
22
+ "cps": 37.6875,
23
+ "bpb": 1.8887908317215056,
24
+ "bytes_per_chunk": 5.3526084737110775
25
+ },
26
+ "tr": {
27
+ "cpc": 0.2421145666464122,
28
+ "cps": 34.2421875,
29
+ "bpb": 2.203242577693521,
30
+ "bytes_per_chunk": 4.533363308574937
31
+ },
32
+ "fi": {
33
+ "cpc": 0.17670511934506405,
34
+ "cps": 25.96875,
35
+ "bpb": 2.4121796351973424,
36
+ "bytes_per_chunk": 5.8906778423199215
37
+ },
38
+ "ru": {
39
+ "cpc": 0.34527704485488125,
40
+ "cps": 51.1171875,
41
+ "bpb": 1.367230666027715,
42
+ "bytes_per_chunk": 5.2564943202895495
43
+ },
44
+ "el": {
45
+ "cpc": 0.3637382448959704,
46
+ "cps": 58.3203125,
47
+ "bpb": 1.4419177599239312,
48
+ "bytes_per_chunk": 4.99816961561928
49
+ },
50
+ "hi": {
51
+ "cpc": 0.43197622585438333,
52
+ "cps": 56.78125,
53
+ "bpb": 1.2061648934002307,
54
+ "bytes_per_chunk": 5.775941655374514
55
+ },
56
+ "ta": {
57
+ "cpc": 0.719176250334314,
58
+ "cps": 105.0390625,
59
+ "bpb": 0.932736077772738,
60
+ "bytes_per_chunk": 3.8478723560409236
61
+ },
62
+ "th": {
63
+ "cpc": 0.45091700341933477,
64
+ "cps": 56.6640625,
65
+ "bpb": 1.1032021694968648,
66
+ "bytes_per_chunk": 6.034830852814899
67
+ },
68
+ "zh": {
69
+ "cpc": 0.5435150694209279,
70
+ "cps": 25.078125,
71
+ "bpb": 3.1430077960108775,
72
+ "bytes_per_chunk": 5.576819980428031
73
+ },
74
+ "my": {
75
+ "cpc": 0.4475406375343044,
76
+ "cps": 66.25,
77
+ "bpb": 1.1165069677203832,
78
+ "bytes_per_chunk": 6.349541606617335
79
+ }
80
+ },
81
+ "gini_cpc": 0.23693031160314426,
82
+ "gini_cps": 0.24140331101544144,
83
+ "bpic": 5.212078823353659,
84
+ "seed": 1,
85
+ "mean_boundary_entropy": 0.6675059795379639,
86
+ "dechunk": "ste",
87
+ "router_noise": 0.0,
88
+ "transitions": [
89
+ {
90
+ "step": 0,
91
+ "phase": 1,
92
+ "objective": "global",
93
+ "router_trainable": true,
94
+ "encoder_trainable": true,
95
+ "frozen_router_params": 0
96
+ },
97
+ {
98
+ "step": 2400,
99
+ "phase": 2,
100
+ "objective": "global",
101
+ "router_trainable": true,
102
+ "encoder_trainable": true,
103
+ "frozen_router_params": 0
104
+ },
105
+ {
106
+ "step": 4200,
107
+ "phase": 3,
108
+ "objective": "global",
109
+ "router_trainable": true,
110
+ "encoder_trainable": true,
111
+ "frozen_router_params": 0
112
+ }
113
+ ]
114
+ }
emactl_noema_s1/log.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
emactl_noema_s1/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2c681e106e72a5682753e7d75995581aa5449c5252f5dab6a0cb4750b5c40c00
3
+ size 90078296
emactl_noema_s2/config.json ADDED
@@ -0,0 +1,108 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run": "emactl_noema_s2",
3
+ "step": 6000,
4
+ "model_config": {
5
+ "vocab_size": 256,
6
+ "d_enc": 256,
7
+ "d_main": 512,
8
+ "n_enc": 2,
9
+ "n_dec": 2,
10
+ "n_main": 6,
11
+ "n_heads_enc": 4,
12
+ "n_heads_main": 8,
13
+ "seq_len": 1024,
14
+ "dropout": 0.0,
15
+ "target_ratio": 6.0,
16
+ "max_chunk_mult": 3.0,
17
+ "dechunk": "ste",
18
+ "router_noise": 0.0,
19
+ "rope_base": 10000.0
20
+ },
21
+ "tied_weights": {
22
+ "head.weight": "emb.weight"
23
+ },
24
+ "train_config": {
25
+ "condition": "A",
26
+ "seed": 2,
27
+ "out_dir": null,
28
+ "run_name": "emactl_noema_s2",
29
+ "corpus": "corpus",
30
+ "langs": null,
31
+ "seq_len": 1024,
32
+ "batch_size": 32,
33
+ "grad_accum": 1,
34
+ "steps": 6000,
35
+ "lr": 0.0003,
36
+ "min_lr_frac": 0.1,
37
+ "warmup": 300,
38
+ "weight_decay": 0.1,
39
+ "grad_clip": 1.0,
40
+ "betas": [
41
+ 0.9,
42
+ 0.95
43
+ ],
44
+ "router_lr_mult": 1.0,
45
+ "d_enc": 256,
46
+ "d_main": 512,
47
+ "n_enc": 2,
48
+ "n_dec": 2,
49
+ "n_main": 6,
50
+ "n_heads_enc": 4,
51
+ "n_heads_main": 8,
52
+ "dechunk": "ste",
53
+ "target_ratio": 6.0,
54
+ "alpha": 0.03,
55
+ "alpha_parity_cap": 0.06,
56
+ "global_anchor": 0.0,
57
+ "premium_clip": 4.0,
58
+ "premium_kind": "flores_per_sent",
59
+ "mixture": null,
60
+ "doc_filters": null,
61
+ "tau1": 0.4,
62
+ "tau2": 0.7,
63
+ "freeze_scope": "router",
64
+ "ckpt_every": 200,
65
+ "eval_every": 200,
66
+ "log_every": 50,
67
+ "probe_sents": 128,
68
+ "eval_batches": 8,
69
+ "keep_ckpt": false,
70
+ "amp": "bf16",
71
+ "compile": false,
72
+ "cpc_decay": 0.9,
73
+ "device": "cuda",
74
+ "log_spaced_probe": true,
75
+ "probe_bpb_batches": 4,
76
+ "dump_reps": true,
77
+ "dump_reps_thresholds": [
78
+ 0.3,
79
+ 0.4,
80
+ 0.5,
81
+ 0.6,
82
+ 0.7
83
+ ],
84
+ "router_noise": 0.0,
85
+ "genreg_target_entropy": null,
86
+ "genreg_eta": 0.05
87
+ },
88
+ "probe_agg": {
89
+ "gini_cpc": 0.2097257925277538,
90
+ "gini_cps": 0.2205780004871769,
91
+ "bpic": 5.008103664265581,
92
+ "worst_lang": "th",
93
+ "cpc_parity_vs_en": {
94
+ "en": 1.0,
95
+ "de": 1.0122183493242607,
96
+ "vi": 1.0386824563703136,
97
+ "tr": 1.1756329079876167,
98
+ "fi": 1.0197810206310258,
99
+ "ru": 1.4561345082792843,
100
+ "el": 1.4873135488871023,
101
+ "hi": 2.3510164171708303,
102
+ "ta": 1.9467053999262376,
103
+ "th": 2.9241466408623515,
104
+ "zh": 2.069107906247045,
105
+ "my": 2.3244289486237197
106
+ }
107
+ }
108
+ }
emactl_noema_s2/final.json ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "event": "final",
3
+ "run": "emactl_noema_s2",
4
+ "condition": "A",
5
+ "step": 6000,
6
+ "params": 22516992,
7
+ "per_lang": {
8
+ "en": {
9
+ "cpc": 0.22863874044704935,
10
+ "cps": 31.0859375,
11
+ "bpb": 2.0440929497706244,
12
+ "bytes_per_chunk": 4.447226675488375
13
+ },
14
+ "de": {
15
+ "cpc": 0.23143232844689038,
16
+ "cps": 37.125,
17
+ "bpb": 2.229200510378607,
18
+ "bytes_per_chunk": 4.252719353033265
19
+ },
20
+ "vi": {
21
+ "cpc": 0.2374830485489558,
22
+ "cps": 34.203125,
23
+ "bpb": 1.9064757324673185,
24
+ "bytes_per_chunk": 5.540809316973674
25
+ },
26
+ "tr": {
27
+ "cpc": 0.2687952273103905,
28
+ "cps": 38.015625,
29
+ "bpb": 2.1740123956039623,
30
+ "bytes_per_chunk": 3.9388166000540914
31
+ },
32
+ "fi": {
33
+ "cpc": 0.23316144808888417,
34
+ "cps": 34.265625,
35
+ "bpb": 2.3927762408084456,
36
+ "bytes_per_chunk": 4.315731419210918
37
+ },
38
+ "ru": {
39
+ "cpc": 0.3329287598944591,
40
+ "cps": 49.2890625,
41
+ "bpb": 1.3761500732624246,
42
+ "bytes_per_chunk": 5.276809887577121
43
+ },
44
+ "el": {
45
+ "cpc": 0.34005749646737804,
46
+ "cps": 54.5234375,
47
+ "bpb": 1.4288364316573776,
48
+ "bytes_per_chunk": 5.19611496531219
49
+ },
50
+ "hi": {
51
+ "cpc": 0.5375334323922734,
52
+ "cps": 70.65625,
53
+ "bpb": 1.15970682221278,
54
+ "bytes_per_chunk": 4.935961889698545
55
+ },
56
+ "ta": {
57
+ "cpc": 0.4450922706606044,
58
+ "cps": 65.0078125,
59
+ "bpb": 0.9492078075249067,
60
+ "bytes_per_chunk": 5.968602361084232
61
+ },
62
+ "th": {
63
+ "cpc": 0.6685732048492384,
64
+ "cps": 84.015625,
65
+ "bpb": 1.0939795622231419,
66
+ "bytes_per_chunk": 4.236396838992227
67
+ },
68
+ "zh": {
69
+ "cpc": 0.47307822553335593,
70
+ "cps": 21.828125,
71
+ "bpb": 3.1189088345478835,
72
+ "bytes_per_chunk": 5.605319990591656
73
+ },
74
+ "my": {
75
+ "cpc": 0.5314545070719865,
76
+ "cps": 78.671875,
77
+ "bpb": 1.1001802838363859,
78
+ "bytes_per_chunk": 5.20814169489505
79
+ }
80
+ },
81
+ "gini_cpc": 0.2097257925277538,
82
+ "gini_cps": 0.2205780004871769,
83
+ "bpic": 5.008103664265581,
84
+ "seed": 2,
85
+ "mean_boundary_entropy": 0.6694104671478271,
86
+ "dechunk": "ste",
87
+ "router_noise": 0.0,
88
+ "transitions": [
89
+ {
90
+ "step": 0,
91
+ "phase": 1,
92
+ "objective": "global",
93
+ "router_trainable": true,
94
+ "encoder_trainable": true,
95
+ "frozen_router_params": 0
96
+ },
97
+ {
98
+ "step": 2400,
99
+ "phase": 2,
100
+ "objective": "global",
101
+ "router_trainable": true,
102
+ "encoder_trainable": true,
103
+ "frozen_router_params": 0
104
+ },
105
+ {
106
+ "step": 4200,
107
+ "phase": 3,
108
+ "objective": "global",
109
+ "router_trainable": true,
110
+ "encoder_trainable": true,
111
+ "frozen_router_params": 0
112
+ }
113
+ ]
114
+ }
emactl_noema_s2/log.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
emactl_noema_s2/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5ee3493e20c110048713670e7c4ff21fb2e5def328ec33bdf7129f24a8348882
3
+ size 90078296
jitter_s0/config.json ADDED
@@ -0,0 +1,108 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run": "jitter_s0",
3
+ "step": 6000,
4
+ "model_config": {
5
+ "vocab_size": 256,
6
+ "d_enc": 256,
7
+ "d_main": 512,
8
+ "n_enc": 2,
9
+ "n_dec": 2,
10
+ "n_main": 6,
11
+ "n_heads_enc": 4,
12
+ "n_heads_main": 8,
13
+ "seq_len": 1024,
14
+ "dropout": 0.0,
15
+ "target_ratio": 6.0,
16
+ "max_chunk_mult": 3.0,
17
+ "dechunk": "ema",
18
+ "router_noise": 0.0,
19
+ "rope_base": 10000.0
20
+ },
21
+ "tied_weights": {
22
+ "head.weight": "emb.weight"
23
+ },
24
+ "train_config": {
25
+ "condition": "A",
26
+ "seed": 0,
27
+ "out_dir": null,
28
+ "run_name": "jitter_s0",
29
+ "corpus": "corpus",
30
+ "langs": null,
31
+ "seq_len": 1024,
32
+ "batch_size": 32,
33
+ "grad_accum": 1,
34
+ "steps": 6000,
35
+ "lr": 0.0003,
36
+ "min_lr_frac": 0.1,
37
+ "warmup": 300,
38
+ "weight_decay": 0.1,
39
+ "grad_clip": 1.0,
40
+ "betas": [
41
+ 0.9,
42
+ 0.95
43
+ ],
44
+ "router_lr_mult": 1.0,
45
+ "d_enc": 256,
46
+ "d_main": 512,
47
+ "n_enc": 2,
48
+ "n_dec": 2,
49
+ "n_main": 6,
50
+ "n_heads_enc": 4,
51
+ "n_heads_main": 8,
52
+ "dechunk": "ema",
53
+ "target_ratio": 6.0,
54
+ "alpha": 0.03,
55
+ "alpha_parity_cap": 0.06,
56
+ "global_anchor": 0.0,
57
+ "premium_clip": 4.0,
58
+ "premium_kind": "flores_per_sent",
59
+ "mixture": null,
60
+ "doc_filters": null,
61
+ "tau1": 0.4,
62
+ "tau2": 0.7,
63
+ "freeze_scope": "router",
64
+ "ckpt_every": 2000,
65
+ "eval_every": 1000,
66
+ "log_every": 200,
67
+ "probe_sents": 128,
68
+ "eval_batches": 8,
69
+ "keep_ckpt": false,
70
+ "amp": "bf16",
71
+ "compile": false,
72
+ "cpc_decay": 0.9,
73
+ "device": "cuda",
74
+ "log_spaced_probe": true,
75
+ "probe_bpb_batches": 4,
76
+ "dump_reps": true,
77
+ "dump_reps_thresholds": [
78
+ 0.3,
79
+ 0.4,
80
+ 0.5,
81
+ 0.6,
82
+ 0.7
83
+ ],
84
+ "router_noise": 0.0,
85
+ "genreg_target_entropy": null,
86
+ "genreg_eta": 0.05
87
+ },
88
+ "probe_agg": {
89
+ "gini_cpc": 0.17269931081856482,
90
+ "gini_cps": 0.21141326974754948,
91
+ "bpic": 43.57681389803565,
92
+ "worst_lang": "my",
93
+ "cpc_parity_vs_en": {
94
+ "en": 1.0,
95
+ "de": 1.013006652705401,
96
+ "vi": 1.1146982587469487,
97
+ "tr": 1.3166408219632104,
98
+ "fi": 0.7608435064589868,
99
+ "ru": 1.0388534881266491,
100
+ "el": 1.1247546265165909,
101
+ "hi": 0.7430796552748885,
102
+ "ta": 0.8876972880449316,
103
+ "th": 1.469706882188374,
104
+ "zh": 1.3059616661022686,
105
+ "my": 2.380651044965168
106
+ }
107
+ }
108
+ }
jitter_s0/final.json ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "event": "final",
3
+ "run": "jitter_s0",
4
+ "condition": "A",
5
+ "step": 6000,
6
+ "params": 22516992,
7
+ "per_lang": {
8
+ "en": {
9
+ "cpc": 0.03591334827328622,
10
+ "cps": 4.8828125,
11
+ "bpb": 2.102690803330191,
12
+ "bytes_per_chunk": 33.77491464278812
13
+ },
14
+ "de": {
15
+ "cpc": 0.036380460721764964,
16
+ "cps": 5.8359375,
17
+ "bpb": 2.271415199876741,
18
+ "bytes_per_chunk": 32.133366021083596
19
+ },
20
+ "vi": {
21
+ "cpc": 0.04003254678600488,
22
+ "cps": 5.765625,
23
+ "bpb": 1.951459073921452,
24
+ "bytes_per_chunk": 40.595276809910956
25
+ },
26
+ "tr": {
27
+ "cpc": 0.04728498038999061,
28
+ "cps": 6.6875,
29
+ "bpb": 2.250277080982085,
30
+ "bytes_per_chunk": 30.134958041154157
31
+ },
32
+ "fi": {
33
+ "cpc": 0.027324437828929883,
34
+ "cps": 4.015625,
35
+ "bpb": 2.4655344794072835,
36
+ "bytes_per_chunk": 49.16890180999719
37
+ },
38
+ "ru": {
39
+ "cpc": 0.037308707124010555,
40
+ "cps": 5.5234375,
41
+ "bpb": 1.4088588409418283,
42
+ "bytes_per_chunk": 58.11861212725862
43
+ },
44
+ "el": {
45
+ "cpc": 0.0403937046240803,
46
+ "cps": 6.4765625,
47
+ "bpb": 1.471620706197338,
48
+ "bytes_per_chunk": 56.95068433630241
49
+ },
50
+ "hi": {
51
+ "cpc": 0.026686478454680536,
52
+ "cps": 3.5078125,
53
+ "bpb": 1.2017601651121157,
54
+ "bytes_per_chunk": 111.29017193801741
55
+ },
56
+ "ta": {
57
+ "cpc": 0.031880181866809305,
58
+ "cps": 4.65625,
59
+ "bpb": 0.9874921775028731,
60
+ "bytes_per_chunk": 111.97949594190517
61
+ },
62
+ "th": {
63
+ "cpc": 0.052782095119676714,
64
+ "cps": 6.6328125,
65
+ "bpb": 1.1332099012743477,
66
+ "bytes_per_chunk": 47.20338525254344
67
+ },
68
+ "zh": {
69
+ "cpc": 0.0469014561462919,
70
+ "cps": 2.1640625,
71
+ "bpb": 3.1557357805467756,
72
+ "bytes_per_chunk": 94.7394289844597
73
+ },
74
+ "my": {
75
+ "cpc": 0.08549715009499684,
76
+ "cps": 12.65625,
77
+ "bpb": 1.1230400189341303,
78
+ "bytes_per_chunk": 33.42607586866433
79
+ }
80
+ },
81
+ "gini_cpc": 0.17269931081856482,
82
+ "gini_cps": 0.21141326974754948,
83
+ "bpic": 43.57681389803565,
84
+ "seed": 0,
85
+ "mean_boundary_entropy": 0.6520750522613525,
86
+ "dechunk": "ema",
87
+ "router_noise": 0.0,
88
+ "transitions": [
89
+ {
90
+ "step": 0,
91
+ "phase": 1,
92
+ "objective": "global",
93
+ "router_trainable": true,
94
+ "encoder_trainable": true,
95
+ "frozen_router_params": 0
96
+ },
97
+ {
98
+ "step": 2400,
99
+ "phase": 2,
100
+ "objective": "global",
101
+ "router_trainable": true,
102
+ "encoder_trainable": true,
103
+ "frozen_router_params": 0
104
+ },
105
+ {
106
+ "step": 4200,
107
+ "phase": 3,
108
+ "objective": "global",
109
+ "router_trainable": true,
110
+ "encoder_trainable": true,
111
+ "frozen_router_params": 0
112
+ }
113
+ ]
114
+ }
jitter_s0/log.jsonl ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {"event": "start", "run": "jitter_s0", "condition": "A", "params": 22516992, "langs": ["en", "de", "vi", "tr", "fi", "ru", "el", "hi", "ta", "th", "zh", "my"], "premium": {"en": 1.0, "de": 1.184, "vi": 1.39, "tr": 1.12, "fi": 1.104, "ru": 1.971, "el": 2.174, "hi": 2.552, "ta": 3.192, "th": 2.762, "zh": 0.915, "my": 3.524}, "schedule": {"tau1": 0.4, "tau2": 0.7, "p1_objective": "global", "p2_objective": "global", "p3_objective": "global", "freeze_in_p3": false, "freeze_scope": "router"}, "corpus": "/local/scratch/sas245/beetle-hnet/scratch/data/tier1_corpus", "premium_kind": "flores_per_sent", "doc_filtered": {"my": 37511}, "weights": {"en": 0.2946, "de": 0.0909, "vi": 0.0576, "tr": 0.0612, "fi": 0.0424, "ru": 0.1476, "el": 0.0522, "hi": 0.042, "ta": 0.0313, "th": 0.0542, "zh": 0.1038, "my": 0.0222}, "t": 1.7}
2
+ {"event": "probe_schedule", "n": 13, "steps": [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 2000, 4000, 6000], "t": 1.7}
3
+ {"event": "probe", "step": 0, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 6384.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.366833, "cps": 49.875, "mean_p": 0.496725, "boundary_entropy": 0.686995, "bpb": 8.160238}, "de": {"chunks": 8628.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.420202, "cps": 67.40625, "mean_p": 0.498889, "boundary_entropy": 0.687821, "bpb": 8.156329}, "vi": {"chunks": 13115.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.711418, "cps": 102.460938, "mean_p": 0.505745, "boundary_entropy": 0.688385, "bpb": 8.087291}, "tr": {"chunks": 5290.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.292217, "cps": 41.328125, "mean_p": 0.489567, "boundary_entropy": 0.687288, "bpb": 8.171434}, "fi": {"chunks": 8895.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.472862, "cps": 69.492188, "mean_p": 0.500688, "boundary_entropy": 0.687512, "bpb": 8.214758}, "ru": {"chunks": 10744.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.566966, "cps": 83.9375, "mean_p": 0.493655, "boundary_entropy": 0.689893, "bpb": 8.099397}, "el": {"chunks": 15039.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.732788, "cps": 117.492188, "mean_p": 0.497482, "boundary_entropy": 0.690234, "bpb": 8.158044}, "hi": {"chunks": 2563.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.152333, "cps": 20.023438, "mean_p": 0.484638, "boundary_entropy": 0.690256, "bpb": 8.262162}, "ta": {"chunks": 38089.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 2.03739, "cps": 297.570312, "mean_p": 0.512976, "boundary_entropy": 0.690378, "bpb": 8.051803}, "th": {"chunks": 34006.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 2.114144, "cps": 265.671875, "mean_p": 0.511916, "boundary_entropy": 0.690552, "bpb": 8.248459}, "zh": {"chunks": 11821.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 2.001524, "cps": 92.351562, "mean_p": 0.52004, "boundary_entropy": 0.685674, "bpb": 8.123118}, "my": {"chunks": 48201.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 2.543857, "cps": 376.570312, "mean_p": 0.521563, "boundary_entropy": 0.690206, "bpb": 7.913994}}, "bpb": {"en": 8.16024, "de": 8.15633, "vi": 8.08729, "tr": 8.17143, "fi": 8.21476, "ru": 8.0994, "el": 8.15804, "hi": 8.26216, "ta": 8.0518, "th": 8.24846, "zh": 8.12312, "my": 7.91399}, "gini_cpc": 0.424496, "gini_cps": 0.425905, "bpic": 1.892645, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 1.1454838515955792, "vi": 1.939350894526797, "tr": 0.7965927738475616, "fi": 1.2890367847879758, "ru": 1.5455676790921895, "el": 1.997603802482379, "hi": 0.41526447061189375, "ta": 5.553993192060118, "th": 5.7632270176529286, "zh": 5.456221801328583, "my": 6.934639901884223}, "t": 5.1}
4
+ {"event": "train", "step": 0, "phase": 1, "objective": "global", "lm_loss": 5.64186, "bpb": 8.13949, "chunk_loss": 0.08991, "F": 1.0, "G": 0.4995, "chunks": 1024.0, "bytes_per_chunk": 1.0, "boundary_entropy": 0.69158, "router_noise": 0.0, "grad_norm": 7.8228, "lr": 1e-06, "router_trainable": true, "sec_per_step": 0.002, "alpha": 0.03, "t": 5.5}
5
+ {"event": "probe", "step": 1, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 6292.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.361547, "cps": 49.15625, "mean_p": 0.496424, "boundary_entropy": 0.686985, "bpb": 8.137942}, "de": {"chunks": 8544.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.416111, "cps": 66.75, "mean_p": 0.498658, "boundary_entropy": 0.687817, "bpb": 8.13676}, "vi": {"chunks": 12993.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.704801, "cps": 101.507812, "mean_p": 0.505487, "boundary_entropy": 0.688387, "bpb": 8.072181}, "tr": {"chunks": 5217.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.288184, "cps": 40.757812, "mean_p": 0.489318, "boundary_entropy": 0.687274, "bpb": 8.156829}, "fi": {"chunks": 8811.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.468396, "cps": 68.835938, "mean_p": 0.500401, "boundary_entropy": 0.687508, "bpb": 8.196416}, "ru": {"chunks": 10431.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.550449, "cps": 81.492188, "mean_p": 0.493206, "boundary_entropy": 0.689879, "bpb": 8.060083}, "el": {"chunks": 15001.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.730936, "cps": 117.195312, "mean_p": 0.497429, "boundary_entropy": 0.690233, "bpb": 8.140897}, "hi": {"chunks": 2503.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.148767, "cps": 19.554688, "mean_p": 0.484452, "boundary_entropy": 0.690243, "bpb": 8.249504}, "ta": {"chunks": 37861.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 2.025194, "cps": 295.789062, "mean_p": 0.512705, "boundary_entropy": 0.690391, "bpb": 8.028624}, "th": {"chunks": 33861.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 2.105129, "cps": 264.539062, "mean_p": 0.511768, "boundary_entropy": 0.690558, "bpb": 8.229615}, "zh": {"chunks": 11822.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 2.001693, "cps": 92.359375, "mean_p": 0.520019, "boundary_entropy": 0.685675, "bpb": 8.120583}, "my": {"chunks": 48181.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 2.542801, "cps": 376.414062, "mean_p": 0.521547, "boundary_entropy": 0.690207, "bpb": 7.893555}}, "bpb": {"en": 8.13794, "de": 8.13676, "vi": 8.07218, "tr": 8.15683, "fi": 8.19642, "ru": 8.06008, "el": 8.1409, "hi": 8.2495, "ta": 8.02862, "th": 8.22962, "zh": 8.12058, "my": 7.89355}, "gini_cpc": 0.426673, "gini_cps": 0.428193, "bpic": 1.90446, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 1.150917619502558, "vi": 1.9494033261656607, "tr": 0.7970869365552818, "fi": 1.2955337284054524, "ru": 1.52248189685105, "el": 2.021690973974, "hi": 0.41147284837275383, "ta": 5.601470037900537, "th": 5.822561987423376, "zh": 5.5364695874270105, "my": 7.033116960044964}, "t": 8.4}
6
+ {"event": "probe", "step": 2, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 6140.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.352813, "cps": 47.96875, "mean_p": 0.495813, "boundary_entropy": 0.686963, "bpb": 8.088626}, "de": {"chunks": 8384.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.408318, "cps": 65.5, "mean_p": 0.498195, "boundary_entropy": 0.687808, "bpb": 8.092933}, "vi": {"chunks": 12767.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.692541, "cps": 99.742188, "mean_p": 0.504896, "boundary_entropy": 0.688391, "bpb": 8.038432}, "tr": {"chunks": 5083.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.280782, "cps": 39.710938, "mean_p": 0.488774, "boundary_entropy": 0.687241, "bpb": 8.122514}, "fi": {"chunks": 8653.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.459997, "cps": 67.601562, "mean_p": 0.499815, "boundary_entropy": 0.687498, "bpb": 8.154597}, "ru": {"chunks": 9871.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.520897, "cps": 77.117188, "mean_p": 0.492489, "boundary_entropy": 0.689854, "bpb": 7.994827}, "el": {"chunks": 15018.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.731764, "cps": 117.328125, "mean_p": 0.497432, "boundary_entropy": 0.690233, "bpb": 8.120639}, "hi": {"chunks": 2362.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.140386, "cps": 18.453125, "mean_p": 0.484026, "boundary_entropy": 0.690214, "bpb": 8.216665}, "ta": {"chunks": 37462.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 2.003851, "cps": 292.671875, "mean_p": 0.512226, "boundary_entropy": 0.690413, "bpb": 7.987519}, "th": {"chunks": 33572.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 2.087162, "cps": 262.28125, "mean_p": 0.51154, "boundary_entropy": 0.690567, "bpb": 8.202526}, "zh": {"chunks": 11810.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.999661, "cps": 92.265625, "mean_p": 0.519969, "boundary_entropy": 0.685678, "bpb": 8.114145}, "my": {"chunks": 48128.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 2.540004, "cps": 376.0, "mean_p": 0.521493, "boundary_entropy": 0.69021, "bpb": 7.860478}}, "bpb": {"en": 8.08863, "de": 8.09293, "vi": 8.03843, "tr": 8.12251, "fi": 8.1546, "ru": 7.99483, "el": 8.12064, "hi": 8.21666, "ta": 7.98752, "th": 8.20253, "zh": 8.11415, "my": 7.86048}, "gini_cpc": 0.430739, "gini_cps": 0.432586, "bpic": 1.926128, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 1.15732307300348, "vi": 1.9629148721319472, "tr": 0.7958391639156395, "fi": 1.3037987770340904, "ru": 1.4764124087904908, "el": 2.0740871646191503, "hi": 0.39790607470076617, "ta": 5.679645622817159, "th": 5.915778412525478, "zh": 5.667769816153395, "my": 7.1992986118703755}, "t": 11.5}
7
+ {"event": "probe", "step": 4, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 5580.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.320634, "cps": 43.59375, "mean_p": 0.493699, "boundary_entropy": 0.686875, "bpb": 7.919777}, "de": {"chunks": 7847.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.382165, "cps": 61.304688, "mean_p": 0.496587, "boundary_entropy": 0.68777, "bpb": 7.9397}, "vi": {"chunks": 12023.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.652183, "cps": 93.929688, "mean_p": 0.50296, "boundary_entropy": 0.688395, "bpb": 7.926445}, "tr": {"chunks": 4607.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.254488, "cps": 35.992188, "mean_p": 0.486967, "boundary_entropy": 0.687126, "bpb": 8.005026}, "fi": {"chunks": 7997.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.425124, "cps": 62.476562, "mean_p": 0.497685, "boundary_entropy": 0.687452, "bpb": 8.007853}, "ru": {"chunks": 8139.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.429499, "cps": 63.585938, "mean_p": 0.490058, "boundary_entropy": 0.689757, "bpb": 7.777456}, "el": {"chunks": 14842.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.723189, "cps": 115.953125, "mean_p": 0.49725, "boundary_entropy": 0.69023, "bpb": 8.031972}, "hi": {"chunks": 1961.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.116553, "cps": 15.320312, "mean_p": 0.482238, "boundary_entropy": 0.690083, "bpb": 8.094255}, "ta": {"chunks": 35822.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 1.916127, "cps": 279.859375, "mean_p": 0.510362, "boundary_entropy": 0.690494, "bpb": 7.840834}, "th": {"chunks": 32414.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 2.015169, "cps": 253.234375, "mean_p": 0.510574, "boundary_entropy": 0.690602, "bpb": 8.088093}, "zh": {"chunks": 11765.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.992042, "cps": 91.914062, "mean_p": 0.519704, "boundary_entropy": 0.685693, "bpb": 8.087489}, "my": {"chunks": 47965.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 2.531402, "cps": 374.726562, "mean_p": 0.521282, "boundary_entropy": 0.690224, "bpb": 7.784123}}, "bpb": {"en": 7.91978, "de": 7.9397, "vi": 7.92645, "tr": 8.00503, "fi": 8.00785, "ru": 7.77746, "el": 8.03197, "hi": 8.09426, "ta": 7.84083, "th": 8.08809, "zh": 8.08749, "my": 7.78412}, "gini_cpc": 0.445746, "gini_cps": 0.447513, "bpic": 2.009724, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 1.191903696593315, "vi": 2.034040642653205, "tr": 0.7937021963329313, "fi": 1.3258827910688267, "ru": 1.3395278747127415, "el": 2.255493102676829, "hi": 0.36350671843295146, "ta": 5.976050809974492, "th": 6.284945033274171, "zh": 6.212814833830368, "my": 7.894979269806896}, "t": 14.6}
8
+ {"event": "probe", "step": 8, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 4774.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.274321, "cps": 37.296875, "mean_p": 0.490456, "boundary_entropy": 0.686711, "bpb": 7.491948}, "de": {"chunks": 7263.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.353723, "cps": 56.742188, "mean_p": 0.494684, "boundary_entropy": 0.687717, "bpb": 7.547367}, "vi": {"chunks": 10680.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.579333, "cps": 83.4375, "mean_p": 0.499673, "boundary_entropy": 0.688367, "bpb": 7.609612}, "tr": {"chunks": 4020.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.222063, "cps": 31.40625, "mean_p": 0.484883, "boundary_entropy": 0.686993, "bpb": 7.68169}, "fi": {"chunks": 6981.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.371113, "cps": 54.539062, "mean_p": 0.494274, "boundary_entropy": 0.687346, "bpb": 7.627208}, "ru": {"chunks": 5054.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.266702, "cps": 39.484375, "mean_p": 0.484986, "boundary_entropy": 0.689479, "bpb": 7.287633}, "el": {"chunks": 13847.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.674706, "cps": 108.179688, "mean_p": 0.496346, "boundary_entropy": 0.690209, "bpb": 7.792458}, "hi": {"chunks": 1194.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.070966, "cps": 9.328125, "mean_p": 0.477649, "boundary_entropy": 0.689689, "bpb": 7.723284}, "ta": {"chunks": 30803.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 1.64766, "cps": 240.648438, "mean_p": 0.505519, "boundary_entropy": 0.690645, "bpb": 7.428373}, "th": {"chunks": 28695.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 1.78396, "cps": 224.179688, "mean_p": 0.507303, "boundary_entropy": 0.6907, "bpb": 7.645596}, "zh": {"chunks": 11622.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.967829, "cps": 90.796875, "mean_p": 0.519029, "boundary_entropy": 0.685731, "bpb": 8.011482}, "my": {"chunks": 48036.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 2.535149, "cps": 375.28125, "mean_p": 0.521759, "boundary_entropy": 0.690171, "bpb": 7.563326}}, "bpb": {"en": 7.49195, "de": 7.54737, "vi": 7.60961, "tr": 7.68169, "fi": 7.62721, "ru": 7.28763, "el": 7.79246, "hi": 7.72328, "ta": 7.42837, "th": 7.6456, "zh": 8.01148, "my": 7.56333}, "gini_cpc": 0.476672, "gini_cps": 0.474585, "bpic": 2.218785, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 1.2894524822161373, "vi": 2.1118828152083617, "tr": 0.8095006600155452, "fi": 1.3528431908729297, "ru": 0.9722271141064229, "el": 2.4595550791706797, "hi": 0.2586967420730542, "ta": 6.006330862108239, "th": 6.503196388478208, "zh": 7.1734674826719, "my": 9.241557406821252}, "t": 18.3}
9
+ {"event": "probe", "step": 16, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2918.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.167672, "cps": 22.796875, "mean_p": 0.483036, "boundary_entropy": 0.686288, "bpb": 6.89556}, "de": {"chunks": 5637.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.274534, "cps": 44.039062, "mean_p": 0.489999, "boundary_entropy": 0.687637, "bpb": 6.982104}, "vi": {"chunks": 7859.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.426309, "cps": 61.398438, "mean_p": 0.49252, "boundary_entropy": 0.688228, "bpb": 7.188954}, "tr": {"chunks": 3001.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.165774, "cps": 23.445312, "mean_p": 0.481764, "boundary_entropy": 0.686833, "bpb": 7.198879}, "fi": {"chunks": 4624.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.245814, "cps": 36.125, "mean_p": 0.486731, "boundary_entropy": 0.687058, "bpb": 7.083572}, "ru": {"chunks": 919.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.048496, "cps": 7.179688, "mean_p": 0.470625, "boundary_entropy": 0.68813, "bpb": 6.56532}, "el": {"chunks": 6453.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.314428, "cps": 50.414062, "mean_p": 0.487918, "boundary_entropy": 0.689892, "bpb": 7.258176}, "hi": {"chunks": 485.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.028826, "cps": 3.789062, "mean_p": 0.470466, "boundary_entropy": 0.68891, "bpb": 7.066105}, "ta": {"chunks": 20147.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 1.077668, "cps": 157.398438, "mean_p": 0.496763, "boundary_entropy": 0.690722, "bpb": 6.869588}, "th": {"chunks": 20155.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 1.253031, "cps": 157.460938, "mean_p": 0.498736, "boundary_entropy": 0.690775, "bpb": 6.616165}, "zh": {"chunks": 11166.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.89062, "cps": 87.234375, "mean_p": 0.516898, "boundary_entropy": 0.685838, "bpb": 7.835642}, "my": {"chunks": 47523.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 2.508075, "cps": 371.273438, "mean_p": 0.522019, "boundary_entropy": 0.690101, "bpb": 6.934056}}, "bpb": {"en": 6.89556, "de": 6.9821, "vi": 7.18895, "tr": 7.19888, "fi": 7.08357, "ru": 6.56532, "el": 7.25818, "hi": 7.0661, "ta": 6.86959, "th": 6.61617, "zh": 7.83564, "my": 6.93406}, "gini_cpc": 0.561355, "gini_cps": 0.549132, "bpic": 2.932155, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 1.6373233685542794, "vi": 2.5425118132675557, "tr": 0.9886766483631835, "fi": 1.466036471376157, "ru": 0.28923119351997695, "el": 1.8752520659463727, "hi": 0.17191964208422997, "ta": 6.427228639238041, "th": 7.473096148097741, "zh": 11.275687043090205, "my": 14.958198951627361}, "t": 22.6}
10
+ {"event": "probe", "step": 32, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 246.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.014135, "cps": 1.921875, "mean_p": 0.378113, "boundary_entropy": 0.654411, "bpb": 6.46997}, "de": {"chunks": 216.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.01052, "cps": 1.6875, "mean_p": 0.386177, "boundary_entropy": 0.659998, "bpb": 6.580452}, "vi": {"chunks": 456.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.024736, "cps": 3.5625, "mean_p": 0.413983, "boundary_entropy": 0.6719, "bpb": 7.054647}, "tr": {"chunks": 313.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.01729, "cps": 2.445312, "mean_p": 0.393962, "boundary_entropy": 0.662865, "bpb": 6.858326}, "fi": {"chunks": 297.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.015789, "cps": 2.320312, "mean_p": 0.389414, "boundary_entropy": 0.660704, "bpb": 6.63062}, "ru": {"chunks": 132.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.006966, "cps": 1.03125, "mean_p": 0.415013, "boundary_entropy": 0.674981, "bpb": 6.347138}, "el": {"chunks": 217.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.010574, "cps": 1.695312, "mean_p": 0.449091, "boundary_entropy": 0.684849, "bpb": 6.832869}, "hi": {"chunks": 179.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.010639, "cps": 1.398438, "mean_p": 0.454408, "boundary_entropy": 0.686448, "bpb": 6.423777}, "ta": {"chunks": 474.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.025354, "cps": 3.703125, "mean_p": 0.468511, "boundary_entropy": 0.688738, "bpb": 6.31379}, "th": {"chunks": 184.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.011439, "cps": 1.4375, "mean_p": 0.461773, "boundary_entropy": 0.68786, "bpb": 6.031388}, "zh": {"chunks": 9561.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.618862, "cps": 74.695312, "mean_p": 0.50882, "boundary_entropy": 0.686081, "bpb": 7.402198}, "my": {"chunks": 45213.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 2.386162, "cps": 353.226562, "mean_p": 0.520571, "boundary_entropy": 0.69015, "bpb": 5.868623}}, "bpb": {"en": 6.46997, "de": 6.58045, "vi": 7.05465, "tr": 6.85833, "fi": 6.63062, "ru": 6.34714, "el": 6.83287, "hi": 6.42378, "ta": 6.31379, "th": 6.03139, "zh": 7.4022, "my": 5.86862}, "gini_cpc": 0.819581, "gini_cps": 0.846406, "bpic": 6.675845, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 0.7442011847674119, "vi": 1.749889856913215, "tr": 1.2231586733367195, "fi": 1.1169496052517272, "ru": 0.49278074522170023, "el": 0.7480108575387757, "hi": 0.7526394375385062, "ta": 1.7936672776730442, "th": 0.8092557071047862, "zh": 114.5246277039472, "my": 168.8064207819089}, "t": 27.9}
11
+ {"event": "probe", "step": 64, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 165.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.009481, "cps": 1.289062, "mean_p": 0.394724, "boundary_entropy": 0.663453, "bpb": 5.934945}, "de": {"chunks": 167.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.008133, "cps": 1.304688, "mean_p": 0.401337, "boundary_entropy": 0.667516, "bpb": 6.064312}, "vi": {"chunks": 270.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.014646, "cps": 2.109375, "mean_p": 0.411652, "boundary_entropy": 0.671783, "bpb": 6.61001}, "tr": {"chunks": 190.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.010495, "cps": 1.484375, "mean_p": 0.405835, "boundary_entropy": 0.668896, "bpb": 6.359059}, "fi": {"chunks": 180.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.009569, "cps": 1.40625, "mean_p": 0.399347, "boundary_entropy": 0.666169, "bpb": 6.069432}, "ru": {"chunks": 129.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.006807, "cps": 1.007812, "mean_p": 0.372291, "boundary_entropy": 0.656344, "bpb": 5.646633}, "el": {"chunks": 128.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.006237, "cps": 1.0, "mean_p": 0.409407, "boundary_entropy": 0.673342, "bpb": 6.38104}, "hi": {"chunks": 144.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.008559, "cps": 1.125, "mean_p": 0.431837, "boundary_entropy": 0.681208, "bpb": 5.742968}, "ta": {"chunks": 151.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.008077, "cps": 1.179688, "mean_p": 0.436318, "boundary_entropy": 0.682596, "bpb": 5.752051}, "th": {"chunks": 133.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.008269, "cps": 1.039062, "mean_p": 0.418422, "boundary_entropy": 0.67722, "bpb": 5.415547}, "zh": {"chunks": 308.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.05215, "cps": 2.40625, "mean_p": 0.446985, "boundary_entropy": 0.680147, "bpb": 7.324651}, "my": {"chunks": 3850.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.203188, "cps": 30.078125, "mean_p": 0.485369, "boundary_entropy": 0.690631, "bpb": 5.158866}}, "bpb": {"en": 5.93495, "de": 6.06431, "vi": 6.61001, "tr": 6.35906, "fi": 6.06943, "ru": 5.64663, "el": 6.38104, "hi": 5.74297, "ta": 5.75205, "th": 5.41555, "zh": 7.32465, "my": 5.15887}, "gini_cpc": 0.635869, "gini_cps": 0.628131, "bpic": 65.998452, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 0.8578359447983954, "vi": 1.5447592277535318, "tr": 1.106988796432535, "fi": 1.0092547397315883, "ru": 0.7179937634924441, "el": 0.6578233733328017, "hi": 0.9027086316358232, "ta": 0.8519059544360427, "th": 0.8721089665696441, "zh": 5.500440230274297, "my": 21.430757863626766}, "t": 32.8}
12
+ {"event": "probe", "step": 128, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 151.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.008677, "cps": 1.179688, "mean_p": 0.418889, "boundary_entropy": 0.67239, "bpb": 4.62767}, "de": {"chunks": 145.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.007062, "cps": 1.132812, "mean_p": 0.423243, "boundary_entropy": 0.674893, "bpb": 4.837106}, "vi": {"chunks": 254.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.013778, "cps": 1.984375, "mean_p": 0.43657, "boundary_entropy": 0.679045, "bpb": 5.258264}, "tr": {"chunks": 172.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.009501, "cps": 1.34375, "mean_p": 0.421685, "boundary_entropy": 0.673879, "bpb": 5.256801}, "fi": {"chunks": 172.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.009144, "cps": 1.34375, "mean_p": 0.425847, "boundary_entropy": 0.675215, "bpb": 4.865459}, "ru": {"chunks": 135.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.007124, "cps": 1.054688, "mean_p": 0.423319, "boundary_entropy": 0.677312, "bpb": 3.877527}, "el": {"chunks": 128.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.006237, "cps": 1.0, "mean_p": 0.420729, "boundary_entropy": 0.677266, "bpb": 4.830958}, "hi": {"chunks": 219.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.013016, "cps": 1.710938, "mean_p": 0.466692, "boundary_entropy": 0.688226, "bpb": 3.740243}, "ta": {"chunks": 167.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.008933, "cps": 1.304688, "mean_p": 0.450738, "boundary_entropy": 0.68599, "bpb": 3.589666}, "th": {"chunks": 2380.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.147964, "cps": 18.59375, "mean_p": 0.468385, "boundary_entropy": 0.687722, "bpb": 3.306342}, "zh": {"chunks": 215.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.036404, "cps": 1.679688, "mean_p": 0.426009, "boundary_entropy": 0.674391, "bpb": 6.405209}, "my": {"chunks": 140.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.007389, "cps": 1.09375, "mean_p": 0.404089, "boundary_entropy": 0.672115, "bpb": 4.152467}}, "bpb": {"en": 4.62767, "de": 4.83711, "vi": 5.25826, "tr": 5.2568, "fi": 4.86546, "ru": 3.87753, "el": 4.83096, "hi": 3.74024, "ta": 3.58967, "th": 3.30634, "zh": 6.40521, "my": 4.15247}, "gini_cpc": 0.575425, "gini_cps": 0.522324, "bpic": 89.710379, "worst_lang": "th", "cpc_parity_vs_en": {"en": 1.0, "de": 0.8138844818694377, "vi": 1.5879533783456101, "tr": 1.0950276069276872, "fi": 1.0538134478875083, "ru": 0.8210540110782995, "el": 0.7188136198669688, "hi": 1.5001552798087046, "ta": 1.0295280283533685, "th": 17.0530892382562, "zh": 4.1955817745114965, "my": 0.851553292594441}, "t": 41.2}
13
+ {"event": "train", "step": 200, "phase": 1, "objective": "global", "lm_loss": 2.53738, "bpb": 3.66066, "chunk_loss": 0.02315, "F": 0.00571, "G": 0.36386, "chunks": 5.84, "bytes_per_chunk": 175.23, "boundary_entropy": 0.65001, "router_noise": 0.0, "grad_norm": 1.9851, "lr": 0.000201, "router_trainable": true, "sec_per_step": 0.216, "alpha": 0.03, "t": 48.7}
14
+ {"event": "probe", "step": 256, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 177.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.010171, "cps": 1.382812, "mean_p": 0.34504, "boundary_entropy": 0.63286, "bpb": 3.720853}, "de": {"chunks": 302.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.014708, "cps": 2.359375, "mean_p": 0.348575, "boundary_entropy": 0.635562, "bpb": 3.978181}, "vi": {"chunks": 192.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.010415, "cps": 1.5, "mean_p": 0.378811, "boundary_entropy": 0.653899, "bpb": 3.321689}, "tr": {"chunks": 423.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.023366, "cps": 3.304688, "mean_p": 0.351562, "boundary_entropy": 0.633833, "bpb": 3.935694}, "fi": {"chunks": 204.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.010845, "cps": 1.59375, "mean_p": 0.317402, "boundary_entropy": 0.611621, "bpb": 3.969358}, "ru": {"chunks": 150.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.007916, "cps": 1.171875, "mean_p": 0.323817, "boundary_entropy": 0.615306, "bpb": 2.665593}, "el": {"chunks": 134.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.006529, "cps": 1.046875, "mean_p": 0.354725, "boundary_entropy": 0.643435, "bpb": 2.806589}, "hi": {"chunks": 177.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.01052, "cps": 1.382812, "mean_p": 0.343351, "boundary_entropy": 0.636062, "bpb": 2.165491}, "ta": {"chunks": 184.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.009842, "cps": 1.4375, "mean_p": 0.354411, "boundary_entropy": 0.640548, "bpb": 2.026701}, "th": {"chunks": 223.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.013864, "cps": 1.742188, "mean_p": 0.316373, "boundary_entropy": 0.616371, "bpb": 2.104417}, "zh": {"chunks": 198.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.033525, "cps": 1.546875, "mean_p": 0.340826, "boundary_entropy": 0.626518, "bpb": 5.003183}, "my": {"chunks": 130.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.006861, "cps": 1.015625, "mean_p": 0.405019, "boundary_entropy": 0.672024, "bpb": 2.221902}}, "bpb": {"en": 3.72085, "de": 3.97818, "vi": 3.32169, "tr": 3.93569, "fi": 3.96936, "ru": 2.66559, "el": 2.80659, "hi": 2.16549, "ta": 2.0267, "th": 2.10442, "zh": 5.00318, "my": 2.2219}, "gini_cpc": 0.270813, "gini_cps": 0.182171, "bpic": 153.881716, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 1.4461235200549427, "vi": 1.024021182992925, "tr": 2.297421440589021, "fi": 1.0662747815243336, "ru": 0.7782746746567686, "el": 0.6419701087741988, "hi": 1.0343536404160476, "ta": 0.967705495441997, "th": 1.3631217630912051, "zh": 3.2962686609997305, "my": 0.674575913382925}, "t": 54.5}
15
+ {"event": "train", "step": 400, "phase": 1, "objective": "global", "lm_loss": 2.06785, "bpb": 2.98328, "chunk_loss": 0.022737, "F": 0.00482, "G": 0.37455, "chunks": 4.94, "bytes_per_chunk": 207.392, "boundary_entropy": 0.65379, "router_noise": 0.0, "grad_norm": 1.2115, "lr": 0.0002998, "router_trainable": true, "sec_per_step": 0.096, "alpha": 0.03, "t": 68.0}
16
+ {"event": "train", "step": 600, "phase": 1, "objective": "global", "lm_loss": 1.97127, "bpb": 2.84395, "chunk_loss": 0.022497, "F": 0.00098, "G": 0.37632, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.653, "router_noise": 0.0, "grad_norm": 0.793, "lr": 0.0002982, "router_trainable": true, "sec_per_step": 0.087, "alpha": 0.03, "t": 85.3}
17
+ {"event": "train", "step": 800, "phase": 1, "objective": "global", "lm_loss": 1.84488, "bpb": 2.66159, "chunk_loss": 0.022703, "F": 0.02719, "G": 0.4086, "chunks": 27.84, "bytes_per_chunk": 36.777, "boundary_entropy": 0.66821, "router_noise": 0.0, "grad_norm": 0.8627, "lr": 0.0002949, "router_trainable": true, "sec_per_step": 0.085, "alpha": 0.03, "t": 102.2}
18
+ {"event": "eval", "step": 1000, "phase": 1, "bpb": {"en": 2.6727, "de": 2.98896, "vi": 2.36676, "tr": 2.9987, "fi": 3.19708, "ru": 1.95199, "el": 2.06414, "hi": 1.53959, "ta": 1.26424, "th": 1.58473, "zh": 3.67557, "my": 1.46615}, "bytes_per_chunk": {"en": 11.696, "de": 14.228, "vi": 26.386, "tr": 14.915, "fi": 19.834, "ru": 14.979, "el": 52.418, "hi": 41.354, "ta": 82.931, "th": 12.679, "zh": 50.961, "my": 14.757}, "bpb_mean": 2.31422, "t": 120.4}
19
+ {"event": "train", "step": 1000, "phase": 1, "objective": "global", "lm_loss": 1.77073, "bpb": 2.55463, "chunk_loss": 0.036174, "F": 0.28519, "G": 0.40666, "chunks": 292.03, "bytes_per_chunk": 3.506, "boundary_entropy": 0.6685, "router_noise": 0.0, "grad_norm": 0.8882, "lr": 0.0002901, "router_trainable": true, "sec_per_step": 0.091, "alpha": 0.03, "t": 120.4}
20
+ {"event": "train", "step": 1200, "phase": 1, "objective": "global", "lm_loss": 1.7461, "bpb": 2.51909, "chunk_loss": 0.023676, "F": 0.00098, "G": 0.34336, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.63283, "router_noise": 0.0, "grad_norm": 0.8669, "lr": 0.0002837, "router_trainable": true, "sec_per_step": 0.085, "alpha": 0.03, "t": 137.5}
21
+ {"event": "train", "step": 1400, "phase": 1, "objective": "global", "lm_loss": 1.59531, "bpb": 2.30154, "chunk_loss": 0.04745, "F": 0.56873, "G": 0.3657, "chunks": 582.38, "bytes_per_chunk": 1.758, "boundary_entropy": 0.64586, "router_noise": 0.0, "grad_norm": 0.7713, "lr": 0.0002759, "router_trainable": true, "sec_per_step": 0.086, "alpha": 0.03, "t": 154.6}
22
+ {"event": "train", "step": 1600, "phase": 1, "objective": "global", "lm_loss": 1.63156, "bpb": 2.35384, "chunk_loss": 0.021869, "F": 0.00412, "G": 0.39823, "chunks": 4.22, "bytes_per_chunk": 242.726, "boundary_entropy": 0.6633, "router_noise": 0.0, "grad_norm": 0.8277, "lr": 0.0002668, "router_trainable": true, "sec_per_step": 0.087, "alpha": 0.03, "t": 172.1}
23
+ {"event": "train", "step": 1800, "phase": 1, "objective": "global", "lm_loss": 1.54875, "bpb": 2.23437, "chunk_loss": 0.022334, "F": 0.00446, "G": 0.38555, "chunks": 4.56, "bytes_per_chunk": 224.438, "boundary_entropy": 0.65782, "router_noise": 0.0, "grad_norm": 0.8032, "lr": 0.0002564, "router_trainable": true, "sec_per_step": 0.088, "alpha": 0.03, "t": 189.8}
24
+ {"event": "probe", "step": 2000, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 865.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.049704, "cps": 6.757812, "mean_p": 0.387742, "boundary_entropy": 0.650626, "bpb": 2.342435}, "de": {"chunks": 1047.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.050991, "cps": 8.179688, "mean_p": 0.394094, "boundary_entropy": 0.656408, "bpb": 2.60724}, "vi": {"chunks": 1126.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.061079, "cps": 8.796875, "mean_p": 0.402093, "boundary_entropy": 0.662149, "bpb": 2.122694}, "tr": {"chunks": 887.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.048997, "cps": 6.929688, "mean_p": 0.391411, "boundary_entropy": 0.655639, "bpb": 2.600893}, "fi": {"chunks": 538.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.0286, "cps": 4.203125, "mean_p": 0.381657, "boundary_entropy": 0.650929, "bpb": 2.835173}, "ru": {"chunks": 1387.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.073193, "cps": 10.835938, "mean_p": 0.397072, "boundary_entropy": 0.663244, "bpb": 1.697863}, "el": {"chunks": 910.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.04434, "cps": 7.109375, "mean_p": 0.373454, "boundary_entropy": 0.650931, "bpb": 1.75602}, "hi": {"chunks": 658.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.039108, "cps": 5.140625, "mean_p": 0.375392, "boundary_entropy": 0.64351, "bpb": 1.391615}, "ta": {"chunks": 1316.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.070393, "cps": 10.28125, "mean_p": 0.410274, "boundary_entropy": 0.670053, "bpb": 1.138114}, "th": {"chunks": 1614.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.100342, "cps": 12.609375, "mean_p": 0.370276, "boundary_entropy": 0.642571, "bpb": 1.332886}, "zh": {"chunks": 316.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.053505, "cps": 2.46875, "mean_p": 0.374219, "boundary_entropy": 0.647369, "bpb": 3.454632}, "my": {"chunks": 2281.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.120382, "cps": 17.820312, "mean_p": 0.413421, "boundary_entropy": 0.671582, "bpb": 1.241147}}, "bpb": {"en": 2.34244, "de": 2.60724, "vi": 2.12269, "tr": 2.60089, "fi": 2.83517, "ru": 1.69786, "el": 1.75602, "hi": 1.39162, "ta": 1.13811, "th": 1.33289, "zh": 3.45463, "my": 1.24115}, "gini_cpc": 0.214487, "gini_cps": 0.254352, "bpic": 29.647045, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 1.025893521467909, "vi": 1.2288624145764449, "tr": 0.9857824478362255, "fi": 0.5754113246369499, "ru": 1.472567663611268, "el": 0.8920897715491345, "hi": 0.7868262374494326, "ta": 1.4162451398862481, "th": 2.01878689979714, "zh": 1.076469310136258, "my": 2.421976483250173}, "t": 209.0}
25
+ {"event": "eval", "step": 2000, "phase": 1, "bpb": {"en": 2.34998, "de": 2.61291, "vi": 2.12962, "tr": 2.60764, "fi": 2.84089, "ru": 1.67632, "el": 1.74849, "hi": 1.38933, "ta": 1.14487, "th": 1.34978, "zh": 3.45794, "my": 1.25053}, "bytes_per_chunk": {"en": 21.62, "de": 20.52, "vi": 23.7, "tr": 28.276, "fi": 56.074, "ru": 28.798, "el": 50.931, "hi": 73.204, "ta": 45.733, "th": 25.158, "zh": 74.664, "my": 25.799}, "bpb_mean": 2.04652, "t": 210.2}
26
+ {"event": "train", "step": 2000, "phase": 1, "objective": "global", "lm_loss": 1.6244, "bpb": 2.34351, "chunk_loss": 0.022285, "F": 0.00098, "G": 0.38224, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65507, "router_noise": 0.0, "grad_norm": 0.9058, "lr": 0.000245, "router_trainable": true, "sec_per_step": 0.103, "alpha": 0.03, "t": 210.3}
27
+ {"event": "train", "step": 2200, "phase": 1, "objective": "global", "lm_loss": 1.48308, "bpb": 2.13963, "chunk_loss": 0.025308, "F": 0.04755, "G": 0.35082, "chunks": 48.69, "bytes_per_chunk": 21.032, "boundary_entropy": 0.63459, "router_noise": 0.0, "grad_norm": 0.7813, "lr": 0.0002325, "router_trainable": true, "sec_per_step": 0.087, "alpha": 0.03, "t": 227.7}
28
+ {"event": "phase_transition", "step": 2400, "phase": 2, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0, "t": 245.5}
29
+ {"event": "train", "step": 2400, "phase": 2, "objective": "global", "lm_loss": 1.40639, "bpb": 2.02899, "chunk_loss": 0.028925, "F": 0.14429, "G": 0.39408, "chunks": 147.75, "bytes_per_chunk": 6.931, "boundary_entropy": 0.66056, "router_noise": 0.0, "grad_norm": 0.859, "lr": 0.0002192, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 245.6}
30
+ {"event": "train", "step": 2600, "phase": 2, "objective": "global", "lm_loss": 1.62783, "bpb": 2.34847, "chunk_loss": 0.022151, "F": 0.00308, "G": 0.3888, "chunks": 3.16, "bytes_per_chunk": 324.436, "boundary_entropy": 0.65898, "router_noise": 0.0, "grad_norm": 0.8836, "lr": 0.0002053, "router_trainable": true, "sec_per_step": 0.207, "alpha": 0.03, "t": 286.9}
31
+ {"event": "train", "step": 2800, "phase": 2, "objective": "global", "lm_loss": 1.49748, "bpb": 2.1604, "chunk_loss": 0.026187, "F": 0.08603, "G": 0.38583, "chunks": 88.09, "bytes_per_chunk": 11.624, "boundary_entropy": 0.65703, "router_noise": 0.0, "grad_norm": 1.0019, "lr": 0.0001909, "router_trainable": true, "sec_per_step": 0.109, "alpha": 0.03, "t": 308.8}
32
+ {"event": "eval", "step": 3000, "phase": 2, "bpb": {"en": 2.22275, "de": 2.45175, "vi": 2.04372, "tr": 2.44134, "fi": 2.6592, "ru": 1.5545, "el": 1.62173, "hi": 1.29278, "ta": 1.06456, "th": 1.24943, "zh": 3.32823, "my": 1.19027}, "bytes_per_chunk": {"en": 13.113, "de": 17.167, "vi": 32.56, "tr": 13.862, "fi": 10.431, "ru": 15.814, "el": 40.896, "hi": 19.984, "ta": 15.489, "th": 21.722, "zh": 51.07, "my": 24.631}, "bpb_mean": 1.92669, "t": 327.2}
33
+ {"event": "train", "step": 3000, "phase": 2, "objective": "global", "lm_loss": 1.46088, "bpb": 2.1076, "chunk_loss": 0.021531, "F": 0.00098, "G": 0.40331, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.66279, "router_noise": 0.0, "grad_norm": 0.8257, "lr": 0.0001761, "router_trainable": true, "sec_per_step": 0.093, "alpha": 0.03, "t": 327.3}
34
+ {"event": "train", "step": 3200, "phase": 2, "objective": "global", "lm_loss": 1.40393, "bpb": 2.02544, "chunk_loss": 0.023008, "F": 0.00098, "G": 0.36203, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.64372, "router_noise": 0.0, "grad_norm": 0.7719, "lr": 0.0001613, "router_trainable": true, "sec_per_step": 0.086, "alpha": 0.03, "t": 344.6}
35
+ {"event": "train", "step": 3400, "phase": 2, "objective": "global", "lm_loss": 1.42269, "bpb": 2.05251, "chunk_loss": 0.022703, "F": 0.00104, "G": 0.37063, "chunks": 1.06, "bytes_per_chunk": 963.765, "boundary_entropy": 0.64803, "router_noise": 0.0, "grad_norm": 0.8124, "lr": 0.0001465, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 362.4}
36
+ {"event": "train", "step": 3600, "phase": 2, "objective": "global", "lm_loss": 1.67857, "bpb": 2.42166, "chunk_loss": 0.025092, "F": 0.05627, "G": 0.37274, "chunks": 57.62, "bytes_per_chunk": 17.77, "boundary_entropy": 0.6487, "router_noise": 0.0, "grad_norm": 0.9422, "lr": 0.0001319, "router_trainable": true, "sec_per_step": 0.085, "alpha": 0.03, "t": 379.4}
37
+ {"event": "train", "step": 3800, "phase": 2, "objective": "global", "lm_loss": 1.5016, "bpb": 2.16636, "chunk_loss": 0.027607, "F": 0.11011, "G": 0.36684, "chunks": 112.75, "bytes_per_chunk": 9.082, "boundary_entropy": 0.64616, "router_noise": 0.0, "grad_norm": 0.8228, "lr": 0.0001177, "router_trainable": true, "sec_per_step": 0.086, "alpha": 0.03, "t": 396.7}
38
+ {"event": "probe", "step": 4000, "phase": 2, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 743.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.042694, "cps": 5.804688, "mean_p": 0.386578, "boundary_entropy": 0.648773, "bpb": 2.142141}, "de": {"chunks": 991.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.048264, "cps": 7.742188, "mean_p": 0.393949, "boundary_entropy": 0.655005, "bpb": 2.352732}, "vi": {"chunks": 963.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.052238, "cps": 7.523438, "mean_p": 0.397769, "boundary_entropy": 0.659274, "bpb": 1.99228}, "tr": {"chunks": 996.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.055019, "cps": 7.78125, "mean_p": 0.39571, "boundary_entropy": 0.656209, "bpb": 2.340493}, "fi": {"chunks": 653.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.034714, "cps": 5.101562, "mean_p": 0.381919, "boundary_entropy": 0.648935, "bpb": 2.542253}, "ru": {"chunks": 890.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.046966, "cps": 6.953125, "mean_p": 0.389771, "boundary_entropy": 0.660209, "bpb": 1.499817}, "el": {"chunks": 1227.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.059787, "cps": 9.585938, "mean_p": 0.402776, "boundary_entropy": 0.66601, "bpb": 1.551699}, "hi": {"chunks": 505.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.030015, "cps": 3.945312, "mean_p": 0.35568, "boundary_entropy": 0.621076, "bpb": 1.245318}, "ta": {"chunks": 440.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.023536, "cps": 3.4375, "mean_p": 0.371573, "boundary_entropy": 0.648746, "bpb": 1.025044}, "th": {"chunks": 1650.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.10258, "cps": 12.890625, "mean_p": 0.373504, "boundary_entropy": 0.643095, "bpb": 1.171179}, "zh": {"chunks": 361.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.061124, "cps": 2.820312, "mean_p": 0.393553, "boundary_entropy": 0.65732, "bpb": 3.222654}, "my": {"chunks": 1762.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.092991, "cps": 13.765625, "mean_p": 0.388011, "boundary_entropy": 0.658824, "bpb": 1.140095}}, "bpb": {"en": 2.14214, "de": 2.35273, "vi": 1.99228, "tr": 2.34049, "fi": 2.54225, "ru": 1.49982, "el": 1.5517, "hi": 1.24532, "ta": 1.02504, "th": 1.17118, "zh": 3.22265, "my": 1.14009}, "gini_cpc": 0.222429, "gini_cps": 0.252564, "bpic": 34.32439, "worst_lang": "th", "cpc_parity_vs_en": {"en": 1.0, "de": 1.1304635239376668, "vi": 1.2235407880658864, "tr": 1.2886770475718836, "fi": 0.8130862265020188, "ru": 1.100059304609069, "el": 1.40035782982937, "hi": 0.7030263639436125, "ta": 0.5512676574479397, "th": 2.4026924594317456, "zh": 1.4316902445171757, "my": 2.178100097426093}, "t": 415.4}
39
+ {"event": "eval", "step": 4000, "phase": 2, "bpb": {"en": 2.15046, "de": 2.35924, "vi": 1.99391, "tr": 2.34763, "fi": 2.54761, "ru": 1.47731, "el": 1.54451, "hi": 1.24194, "ta": 1.0329, "th": 1.18717, "zh": 3.22415, "my": 1.14907}, "bytes_per_chunk": {"en": 26.309, "de": 21.906, "vi": 27.344, "tr": 24.971, "fi": 41.005, "ru": 46.438, "el": 36.813, "hi": 105.576, "ta": 149.968, "th": 25.608, "zh": 51.899, "my": 29.418}, "bpb_mean": 1.85466, "t": 416.6}
40
+ {"event": "train", "step": 4000, "phase": 2, "objective": "global", "lm_loss": 1.37428, "bpb": 1.98267, "chunk_loss": 0.037107, "F": 0.31747, "G": 0.38364, "chunks": 325.09, "bytes_per_chunk": 3.15, "boundary_entropy": 0.6547, "router_noise": 0.0, "grad_norm": 0.8236, "lr": 0.000104, "router_trainable": true, "sec_per_step": 0.1, "alpha": 0.03, "t": 416.7}
41
+ {"event": "phase_transition", "step": 4200, "phase": 3, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0, "t": 434.4}
42
+ {"event": "train", "step": 4200, "phase": 3, "objective": "global", "lm_loss": 1.22543, "bpb": 1.76792, "chunk_loss": 0.053313, "F": 0.70139, "G": 0.36807, "chunks": 718.22, "bytes_per_chunk": 1.426, "boundary_entropy": 0.64404, "router_noise": 0.0, "grad_norm": 0.7423, "lr": 9.12e-05, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 434.5}
43
+ {"event": "train", "step": 4400, "phase": 3, "objective": "global", "lm_loss": 1.51993, "bpb": 2.19279, "chunk_loss": 0.032473, "F": 0.2225, "G": 0.37015, "chunks": 227.84, "bytes_per_chunk": 4.494, "boundary_entropy": 0.64718, "router_noise": 0.0, "grad_norm": 0.9441, "lr": 7.92e-05, "router_trainable": true, "sec_per_step": 0.087, "alpha": 0.03, "t": 451.9}
44
+ {"event": "train", "step": 4600, "phase": 3, "objective": "global", "lm_loss": 1.33993, "bpb": 1.93311, "chunk_loss": 0.023319, "F": 0.00107, "G": 0.35345, "chunks": 1.09, "bytes_per_chunk": 936.229, "boundary_entropy": 0.63719, "router_noise": 0.0, "grad_norm": 0.8465, "lr": 6.82e-05, "router_trainable": true, "sec_per_step": 0.088, "alpha": 0.03, "t": 469.5}
45
+ {"event": "train", "step": 4800, "phase": 3, "objective": "global", "lm_loss": 1.28915, "bpb": 1.85985, "chunk_loss": 0.0234, "F": 0.00729, "G": 0.35837, "chunks": 7.47, "bytes_per_chunk": 137.105, "boundary_entropy": 0.6404, "router_noise": 0.0, "grad_norm": 0.8537, "lr": 5.85e-05, "router_trainable": true, "sec_per_step": 0.087, "alpha": 0.03, "t": 486.9}
46
+ {"event": "eval", "step": 5000, "phase": 3, "bpb": {"en": 2.10988, "de": 2.301, "vi": 1.9648, "tr": 2.28913, "fi": 2.48488, "ru": 1.4289, "el": 1.49963, "hi": 1.21131, "ta": 1.00044, "th": 1.15444, "zh": 3.18178, "my": 1.12292}, "bytes_per_chunk": {"en": 35.92, "de": 33.225, "vi": 44.311, "tr": 39.167, "fi": 58.79, "ru": 55.539, "el": 57.013, "hi": 110.237, "ta": 188.865, "th": 47.715, "zh": 78.205, "my": 36.859}, "bpb_mean": 1.81243, "t": 505.3}
47
+ {"event": "train", "step": 5000, "phase": 3, "objective": "global", "lm_loss": 1.33088, "bpb": 1.92006, "chunk_loss": 0.022679, "F": 0.00098, "G": 0.37124, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.6485, "router_noise": 0.0, "grad_norm": 0.8763, "lr": 5e-05, "router_trainable": true, "sec_per_step": 0.093, "alpha": 0.03, "t": 505.5}
48
+ {"event": "train", "step": 5200, "phase": 3, "objective": "global", "lm_loss": 1.3245, "bpb": 1.91085, "chunk_loss": 0.061605, "F": 0.85696, "G": 0.37798, "chunks": 877.53, "bytes_per_chunk": 1.167, "boundary_entropy": 0.65228, "router_noise": 0.0, "grad_norm": 0.9709, "lr": 4.29e-05, "router_trainable": true, "sec_per_step": 0.088, "alpha": 0.03, "t": 523.0}
49
+ {"event": "train", "step": 5400, "phase": 3, "objective": "global", "lm_loss": 1.17281, "bpb": 1.69201, "chunk_loss": 0.023089, "F": 0.01028, "G": 0.37136, "chunks": 10.53, "bytes_per_chunk": 97.234, "boundary_entropy": 0.64689, "router_noise": 0.0, "grad_norm": 0.8181, "lr": 3.73e-05, "router_trainable": true, "sec_per_step": 0.086, "alpha": 0.03, "t": 540.1}
50
+ {"event": "train", "step": 5600, "phase": 3, "objective": "global", "lm_loss": 1.51532, "bpb": 2.18615, "chunk_loss": 0.022786, "F": 0.00098, "G": 0.36824, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.64788, "router_noise": 0.0, "grad_norm": 0.9137, "lr": 3.33e-05, "router_trainable": true, "sec_per_step": 0.123, "alpha": 0.03, "t": 564.7}
51
+ {"event": "train", "step": 5800, "phase": 3, "objective": "global", "lm_loss": 1.30505, "bpb": 1.88278, "chunk_loss": 0.023186, "F": 0.00098, "G": 0.35707, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.63876, "router_noise": 0.0, "grad_norm": 0.9114, "lr": 3.08e-05, "router_trainable": true, "sec_per_step": 0.187, "alpha": 0.03, "t": 602.1}
52
+ {"event": "probe", "step": 6000, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 625.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.035913, "cps": 4.882812, "mean_p": 0.376954, "boundary_entropy": 0.64401, "bpb": 2.081873}, "de": {"chunks": 747.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.03638, "cps": 5.835938, "mean_p": 0.379785, "boundary_entropy": 0.647951, "bpb": 2.270786}, "vi": {"chunks": 738.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.040033, "cps": 5.765625, "mean_p": 0.380939, "boundary_entropy": 0.650053, "bpb": 1.947748}, "tr": {"chunks": 856.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.047285, "cps": 6.6875, "mean_p": 0.384487, "boundary_entropy": 0.650487, "bpb": 2.253192}, "fi": {"chunks": 514.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.027324, "cps": 4.015625, "mean_p": 0.379234, "boundary_entropy": 0.647883, "bpb": 2.443051}, "ru": {"chunks": 707.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.037309, "cps": 5.523438, "mean_p": 0.382472, "boundary_entropy": 0.656953, "bpb": 1.429963}, "el": {"chunks": 829.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.040394, "cps": 6.476562, "mean_p": 0.383706, "boundary_entropy": 0.657316, "bpb": 1.482602}, "hi": {"chunks": 449.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.026686, "cps": 3.507812, "mean_p": 0.347368, "boundary_entropy": 0.61441, "bpb": 1.195372}, "ta": {"chunks": 596.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.03188, "cps": 4.65625, "mean_p": 0.390185, "boundary_entropy": 0.66019, "bpb": 0.981009}, "th": {"chunks": 849.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.052782, "cps": 6.632812, "mean_p": 0.35477, "boundary_entropy": 0.628983, "bpb": 1.118028}, "zh": {"chunks": 277.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.046901, "cps": 2.164062, "mean_p": 0.374839, "boundary_entropy": 0.647492, "bpb": 3.147655}, "my": {"chunks": 1620.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.085497, "cps": 12.65625, "mean_p": 0.370474, "boundary_entropy": 0.648133, "bpb": 1.103244}}, "bpb": {"en": 2.08187, "de": 2.27079, "vi": 1.94775, "tr": 2.25319, "fi": 2.44305, "ru": 1.42996, "el": 1.4826, "hi": 1.19537, "ta": 0.98101, "th": 1.11803, "zh": 3.14766, "my": 1.10324}, "gini_cpc": 0.172699, "gini_cps": 0.211413, "bpic": 43.576814, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 1.013006652705401, "vi": 1.1146982587469487, "tr": 1.3166408219632104, "fi": 0.7608435064589868, "ru": 1.0388534881266491, "el": 1.1247546265165909, "hi": 0.7430796552748885, "ta": 0.8876972880449316, "th": 1.469706882188374, "zh": 1.3059616661022686, "my": 2.380651044965168}, "t": 620.9}
53
+ {"event": "eval", "step": 6000, "phase": 3, "bpb": {"en": 2.09086, "de": 2.27843, "vi": 1.94879, "tr": 2.25811, "fi": 2.45103, "ru": 1.40813, "el": 1.47652, "hi": 1.1921, "ta": 0.98822, "th": 1.13533, "zh": 3.14726, "my": 1.11249}, "bytes_per_chunk": {"en": 33.851, "de": 32.074, "vi": 41.205, "tr": 29.722, "fi": 49.667, "ru": 57.665, "el": 55.422, "hi": 112.46, "ta": 112.847, "th": 47.524, "zh": 92.272, "my": 33.149}, "bpb_mean": 1.79061, "t": 622.1}
54
+ {"event": "probe", "step": 6000, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 625.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.035913, "cps": 4.882812, "mean_p": 0.376954, "boundary_entropy": 0.64401, "bpb": 2.081873}, "de": {"chunks": 747.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.03638, "cps": 5.835938, "mean_p": 0.379785, "boundary_entropy": 0.647951, "bpb": 2.270786}, "vi": {"chunks": 738.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.040033, "cps": 5.765625, "mean_p": 0.380939, "boundary_entropy": 0.650053, "bpb": 1.947748}, "tr": {"chunks": 856.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.047285, "cps": 6.6875, "mean_p": 0.384487, "boundary_entropy": 0.650487, "bpb": 2.253192}, "fi": {"chunks": 514.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.027324, "cps": 4.015625, "mean_p": 0.379234, "boundary_entropy": 0.647883, "bpb": 2.443051}, "ru": {"chunks": 707.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.037309, "cps": 5.523438, "mean_p": 0.382472, "boundary_entropy": 0.656953, "bpb": 1.429963}, "el": {"chunks": 829.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.040394, "cps": 6.476562, "mean_p": 0.383706, "boundary_entropy": 0.657316, "bpb": 1.482602}, "hi": {"chunks": 449.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.026686, "cps": 3.507812, "mean_p": 0.347368, "boundary_entropy": 0.61441, "bpb": 1.195372}, "ta": {"chunks": 596.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.03188, "cps": 4.65625, "mean_p": 0.390185, "boundary_entropy": 0.66019, "bpb": 0.981009}, "th": {"chunks": 849.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.052782, "cps": 6.632812, "mean_p": 0.35477, "boundary_entropy": 0.628983, "bpb": 1.118028}, "zh": {"chunks": 277.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.046901, "cps": 2.164062, "mean_p": 0.374839, "boundary_entropy": 0.647492, "bpb": 3.147655}, "my": {"chunks": 1620.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.085497, "cps": 12.65625, "mean_p": 0.370474, "boundary_entropy": 0.648133, "bpb": 1.103244}}, "bpb": {"en": 2.08187, "de": 2.27079, "vi": 1.94775, "tr": 2.25319, "fi": 2.44305, "ru": 1.42996, "el": 1.4826, "hi": 1.19537, "ta": 0.98101, "th": 1.11803, "zh": 3.14766, "my": 1.10324}, "gini_cpc": 0.172699, "gini_cps": 0.211413, "bpic": 43.576814, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 1.013006652705401, "vi": 1.1146982587469487, "tr": 1.3166408219632104, "fi": 0.7608435064589868, "ru": 1.0388534881266491, "el": 1.1247546265165909, "hi": 0.7430796552748885, "ta": 0.8876972880449316, "th": 1.469706882188374, "zh": 1.3059616661022686, "my": 2.380651044965168}, "t": 623.1}
55
+ {"event": "reps_dumped", "dirs": ["/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s0/th0.30", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s0/th0.40", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s0/th0.50", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s0/th0.60", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s0/th0.70"], "t": 626.6}
56
+ {"event": "final", "run": "jitter_s0", "condition": "A", "step": 6000, "params": 22516992, "per_lang": {"en": {"cpc": 0.03591334827328622, "cps": 4.8828125, "bpb": 2.102690803330191, "bytes_per_chunk": 33.77491464278812}, "de": {"cpc": 0.036380460721764964, "cps": 5.8359375, "bpb": 2.271415199876741, "bytes_per_chunk": 32.133366021083596}, "vi": {"cpc": 0.04003254678600488, "cps": 5.765625, "bpb": 1.951459073921452, "bytes_per_chunk": 40.595276809910956}, "tr": {"cpc": 0.04728498038999061, "cps": 6.6875, "bpb": 2.250277080982085, "bytes_per_chunk": 30.134958041154157}, "fi": {"cpc": 0.027324437828929883, "cps": 4.015625, "bpb": 2.4655344794072835, "bytes_per_chunk": 49.16890180999719}, "ru": {"cpc": 0.037308707124010555, "cps": 5.5234375, "bpb": 1.4088588409418283, "bytes_per_chunk": 58.11861212725862}, "el": {"cpc": 0.0403937046240803, "cps": 6.4765625, "bpb": 1.471620706197338, "bytes_per_chunk": 56.95068433630241}, "hi": {"cpc": 0.026686478454680536, "cps": 3.5078125, "bpb": 1.2017601651121157, "bytes_per_chunk": 111.29017193801741}, "ta": {"cpc": 0.031880181866809305, "cps": 4.65625, "bpb": 0.9874921775028731, "bytes_per_chunk": 111.97949594190517}, "th": {"cpc": 0.052782095119676714, "cps": 6.6328125, "bpb": 1.1332099012743477, "bytes_per_chunk": 47.20338525254344}, "zh": {"cpc": 0.0469014561462919, "cps": 2.1640625, "bpb": 3.1557357805467756, "bytes_per_chunk": 94.7394289844597}, "my": {"cpc": 0.08549715009499684, "cps": 12.65625, "bpb": 1.1230400189341303, "bytes_per_chunk": 33.42607586866433}}, "gini_cpc": 0.17269931081856482, "gini_cps": 0.21141326974754948, "bpic": 43.57681389803565, "seed": 0, "mean_boundary_entropy": 0.6520750522613525, "dechunk": "ema", "router_noise": 0.0, "transitions": [{"step": 0, "phase": 1, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}, {"step": 2400, "phase": 2, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}, {"step": 4200, "phase": 3, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}], "t": 628.8}
jitter_s0/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c9ea1cb0d102b6188b02fcc98493811484f3d4824eef7bcba770de4a50ac1ede
3
+ size 90078288
jitter_s1/config.json ADDED
@@ -0,0 +1,108 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run": "jitter_s1",
3
+ "step": 6000,
4
+ "model_config": {
5
+ "vocab_size": 256,
6
+ "d_enc": 256,
7
+ "d_main": 512,
8
+ "n_enc": 2,
9
+ "n_dec": 2,
10
+ "n_main": 6,
11
+ "n_heads_enc": 4,
12
+ "n_heads_main": 8,
13
+ "seq_len": 1024,
14
+ "dropout": 0.0,
15
+ "target_ratio": 6.0,
16
+ "max_chunk_mult": 3.0,
17
+ "dechunk": "ema",
18
+ "router_noise": 0.0,
19
+ "rope_base": 10000.0
20
+ },
21
+ "tied_weights": {
22
+ "head.weight": "emb.weight"
23
+ },
24
+ "train_config": {
25
+ "condition": "A",
26
+ "seed": 1,
27
+ "out_dir": null,
28
+ "run_name": "jitter_s1",
29
+ "corpus": "corpus",
30
+ "langs": null,
31
+ "seq_len": 1024,
32
+ "batch_size": 32,
33
+ "grad_accum": 1,
34
+ "steps": 6000,
35
+ "lr": 0.0003,
36
+ "min_lr_frac": 0.1,
37
+ "warmup": 300,
38
+ "weight_decay": 0.1,
39
+ "grad_clip": 1.0,
40
+ "betas": [
41
+ 0.9,
42
+ 0.95
43
+ ],
44
+ "router_lr_mult": 1.0,
45
+ "d_enc": 256,
46
+ "d_main": 512,
47
+ "n_enc": 2,
48
+ "n_dec": 2,
49
+ "n_main": 6,
50
+ "n_heads_enc": 4,
51
+ "n_heads_main": 8,
52
+ "dechunk": "ema",
53
+ "target_ratio": 6.0,
54
+ "alpha": 0.03,
55
+ "alpha_parity_cap": 0.06,
56
+ "global_anchor": 0.0,
57
+ "premium_clip": 4.0,
58
+ "premium_kind": "flores_per_sent",
59
+ "mixture": null,
60
+ "doc_filters": null,
61
+ "tau1": 0.4,
62
+ "tau2": 0.7,
63
+ "freeze_scope": "router",
64
+ "ckpt_every": 2000,
65
+ "eval_every": 1000,
66
+ "log_every": 200,
67
+ "probe_sents": 128,
68
+ "eval_batches": 8,
69
+ "keep_ckpt": false,
70
+ "amp": "bf16",
71
+ "compile": false,
72
+ "cpc_decay": 0.9,
73
+ "device": "cuda",
74
+ "log_spaced_probe": true,
75
+ "probe_bpb_batches": 4,
76
+ "dump_reps": true,
77
+ "dump_reps_thresholds": [
78
+ 0.3,
79
+ 0.4,
80
+ 0.5,
81
+ 0.6,
82
+ 0.7
83
+ ],
84
+ "router_noise": 0.0,
85
+ "genreg_target_entropy": null,
86
+ "genreg_eta": 0.05
87
+ },
88
+ "probe_agg": {
89
+ "gini_cpc": 0.2318796127695224,
90
+ "gini_cps": 0.2339954866678544,
91
+ "bpic": 68.37359700694816,
92
+ "worst_lang": "hi",
93
+ "cpc_parity_vs_en": {
94
+ "en": 1.0,
95
+ "de": 0.8146260953869533,
96
+ "vi": 1.3084208484930249,
97
+ "tr": 0.911522408188721,
98
+ "fi": 0.6782836797893858,
99
+ "ru": 1.7701111460483683,
100
+ "el": 0.823810338912061,
101
+ "hi": 2.4679785047232636,
102
+ "ta": 1.7050250337085502,
103
+ "th": 1.168831225307265,
104
+ "zh": 1.74051680121559,
105
+ "my": 0.6662411771075679
106
+ }
107
+ }
108
+ }
jitter_s1/final.json ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "event": "final",
3
+ "run": "jitter_s1",
4
+ "condition": "A",
5
+ "step": 6000,
6
+ "params": 22516992,
7
+ "per_lang": {
8
+ "en": {
9
+ "cpc": 0.022180083893581566,
10
+ "cps": 3.015625,
11
+ "bpb": 2.1020406444164355,
12
+ "bytes_per_chunk": 46.865826405649415
13
+ },
14
+ "de": {
15
+ "cpc": 0.018068475137583404,
16
+ "cps": 2.8984375,
17
+ "bpb": 2.2796363360511163,
18
+ "bytes_per_chunk": 68.94897422409258
19
+ },
20
+ "vi": {
21
+ "cpc": 0.029020884187686467,
22
+ "cps": 4.1796875,
23
+ "bpb": 1.9494764901690524,
24
+ "bytes_per_chunk": 70.51620712844654
25
+ },
26
+ "tr": {
27
+ "cpc": 0.020217643484505332,
28
+ "cps": 2.859375,
29
+ "bpb": 2.2458832574689267,
30
+ "bytes_per_chunk": 85.72400261608895
31
+ },
32
+ "fi": {
33
+ "cpc": 0.01504438892137579,
34
+ "cps": 2.2109375,
35
+ "bpb": 2.4640121534753403,
36
+ "bytes_per_chunk": 111.19575821845174
37
+ },
38
+ "ru": {
39
+ "cpc": 0.03926121372031662,
40
+ "cps": 5.8125,
41
+ "bpb": 1.4033274435589955,
42
+ "bytes_per_chunk": 66.50025367833587
43
+ },
44
+ "el": {
45
+ "cpc": 0.018272182429469375,
46
+ "cps": 2.9296875,
47
+ "bpb": 1.4606392699744462,
48
+ "bytes_per_chunk": 149.24224309706804
49
+ },
50
+ "hi": {
51
+ "cpc": 0.05473997028231798,
52
+ "cps": 7.1953125,
53
+ "bpb": 1.2202438124075716,
54
+ "bytes_per_chunk": 53.54248366013072
55
+ },
56
+ "ta": {
57
+ "cpc": 0.03781759828831238,
58
+ "cps": 5.5234375,
59
+ "bpb": 0.9998630346897966,
60
+ "bytes_per_chunk": 80.46163290362185
61
+ },
62
+ "th": {
63
+ "cpc": 0.025924774634752876,
64
+ "cps": 3.2578125,
65
+ "bpb": 1.126236773329332,
66
+ "bytes_per_chunk": 134.95186615186614
67
+ },
68
+ "zh": {
69
+ "cpc": 0.03860480866915002,
70
+ "cps": 1.78125,
71
+ "bpb": 3.1923234986783178,
72
+ "bytes_per_chunk": 134.3639159405433
73
+ },
74
+ "my": {
75
+ "cpc": 0.01477728520160439,
76
+ "cps": 2.1875,
77
+ "bpb": 1.114218518197882,
78
+ "bytes_per_chunk": 210.81141938078005
79
+ }
80
+ },
81
+ "gini_cpc": 0.2318796127695224,
82
+ "gini_cps": 0.2339954866678544,
83
+ "bpic": 68.37359700694816,
84
+ "seed": 1,
85
+ "mean_boundary_entropy": 0.6504928469657898,
86
+ "dechunk": "ema",
87
+ "router_noise": 0.0,
88
+ "transitions": [
89
+ {
90
+ "step": 0,
91
+ "phase": 1,
92
+ "objective": "global",
93
+ "router_trainable": true,
94
+ "encoder_trainable": true,
95
+ "frozen_router_params": 0
96
+ },
97
+ {
98
+ "step": 2400,
99
+ "phase": 2,
100
+ "objective": "global",
101
+ "router_trainable": true,
102
+ "encoder_trainable": true,
103
+ "frozen_router_params": 0
104
+ },
105
+ {
106
+ "step": 4200,
107
+ "phase": 3,
108
+ "objective": "global",
109
+ "router_trainable": true,
110
+ "encoder_trainable": true,
111
+ "frozen_router_params": 0
112
+ }
113
+ ]
114
+ }
jitter_s1/log.jsonl ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {"event": "start", "run": "jitter_s1", "condition": "A", "params": 22516992, "langs": ["en", "de", "vi", "tr", "fi", "ru", "el", "hi", "ta", "th", "zh", "my"], "premium": {"en": 1.0, "de": 1.184, "vi": 1.39, "tr": 1.12, "fi": 1.104, "ru": 1.971, "el": 2.174, "hi": 2.552, "ta": 3.192, "th": 2.762, "zh": 0.915, "my": 3.524}, "schedule": {"tau1": 0.4, "tau2": 0.7, "p1_objective": "global", "p2_objective": "global", "p3_objective": "global", "freeze_in_p3": false, "freeze_scope": "router"}, "corpus": "/local/scratch/sas245/beetle-hnet/scratch/data/tier1_corpus", "premium_kind": "flores_per_sent", "doc_filtered": {"my": 37511}, "weights": {"en": 0.2946, "de": 0.0909, "vi": 0.0576, "tr": 0.0612, "fi": 0.0424, "ru": 0.1476, "el": 0.0522, "hi": 0.042, "ta": 0.0313, "th": 0.0542, "zh": 0.1038, "my": 0.0222}, "t": 1.5}
2
+ {"event": "probe_schedule", "n": 13, "steps": [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 2000, 4000, 6000], "t": 1.5}
3
+ {"event": "probe", "step": 0, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2674.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.153652, "cps": 20.890625, "mean_p": 0.478126, "boundary_entropy": 0.685438, "bpb": 8.041379}, "de": {"chunks": 2056.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.100131, "cps": 16.0625, "mean_p": 0.469682, "boundary_entropy": 0.685273, "bpb": 8.067143}, "vi": {"chunks": 14185.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.76946, "cps": 110.820312, "mean_p": 0.507219, "boundary_entropy": 0.688283, "bpb": 7.894113}, "tr": {"chunks": 5645.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.311827, "cps": 44.101562, "mean_p": 0.486063, "boundary_entropy": 0.686375, "bpb": 8.01701}, "fi": {"chunks": 4732.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.251555, "cps": 36.96875, "mean_p": 0.48451, "boundary_entropy": 0.686595, "bpb": 8.075294}, "ru": {"chunks": 17548.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.926016, "cps": 137.09375, "mean_p": 0.502342, "boundary_entropy": 0.690066, "bpb": 8.095633}, "el": {"chunks": 31473.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.533548, "cps": 245.882812, "mean_p": 0.517965, "boundary_entropy": 0.689704, "bpb": 7.804942}, "hi": {"chunks": 1659.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.098603, "cps": 12.960938, "mean_p": 0.476241, "boundary_entropy": 0.689492, "bpb": 8.047677}, "ta": {"chunks": 39693.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 2.123188, "cps": 310.101562, "mean_p": 0.514597, "boundary_entropy": 0.690542, "bpb": 8.096414}, "th": {"chunks": 44424.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 2.761828, "cps": 347.0625, "mean_p": 0.52078, "boundary_entropy": 0.690224, "bpb": 8.201461}, "zh": {"chunks": 12933.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 2.189807, "cps": 101.039062, "mean_p": 0.528309, "boundary_entropy": 0.684752, "bpb": 8.121995}, "my": {"chunks": 7048.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.371965, "cps": 55.0625, "mean_p": 0.491145, "boundary_entropy": 0.691021, "bpb": 8.112229}}, "bpb": {"en": 8.04138, "de": 8.06714, "vi": 7.89411, "tr": 8.01701, "fi": 8.07529, "ru": 8.09563, "el": 7.80494, "hi": 8.04768, "ta": 8.09641, "th": 8.20146, "zh": 8.12199, "my": 8.11223}, "gini_cpc": 0.507116, "gini_cps": 0.500863, "bpic": 2.084973, "worst_lang": "th", "cpc_parity_vs_en": {"en": 1.0, "de": 0.6516785410034256, "vi": 5.0078223656304, "tr": 2.029439514461233, "fi": 1.6371767549398106, "ru": 6.026721581613623, "el": 9.980677296913099, "hi": 0.6417324941848254, "ta": 13.818190381639045, "th": 17.9746032262692, "zh": 14.25176170627287, "my": 2.4208352616119475}, "t": 3.2}
4
+ {"event": "train", "step": 0, "phase": 1, "objective": "global", "lm_loss": 5.56843, "bpb": 8.03354, "chunk_loss": 0.018394, "F": 0.00098, "G": 0.49096, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.6904, "router_noise": 0.0, "grad_norm": 4.288, "lr": 1e-06, "router_trainable": true, "sec_per_step": 0.001, "alpha": 0.03, "t": 3.6}
5
+ {"event": "probe", "step": 1, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2664.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.153077, "cps": 20.8125, "mean_p": 0.478116, "boundary_entropy": 0.68544, "bpb": 8.020959}, "de": {"chunks": 2049.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.099791, "cps": 16.007812, "mean_p": 0.469667, "boundary_entropy": 0.685274, "bpb": 8.049532}, "vi": {"chunks": 14185.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.76946, "cps": 110.820312, "mean_p": 0.507217, "boundary_entropy": 0.688284, "bpb": 7.879922}, "tr": {"chunks": 5644.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.311772, "cps": 44.09375, "mean_p": 0.486068, "boundary_entropy": 0.686377, "bpb": 8.002969}, "fi": {"chunks": 4715.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.250651, "cps": 36.835938, "mean_p": 0.484451, "boundary_entropy": 0.686591, "bpb": 8.058056}, "ru": {"chunks": 17619.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.929763, "cps": 137.648438, "mean_p": 0.502405, "boundary_entropy": 0.690066, "bpb": 8.091542}, "el": {"chunks": 31509.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.535302, "cps": 246.164062, "mean_p": 0.518025, "boundary_entropy": 0.6897, "bpb": 7.798168}, "hi": {"chunks": 1684.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.100089, "cps": 13.15625, "mean_p": 0.476383, "boundary_entropy": 0.689508, "bpb": 8.02942}, "ta": {"chunks": 39824.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 2.130195, "cps": 311.125, "mean_p": 0.514695, "boundary_entropy": 0.690537, "bpb": 8.080367}, "th": {"chunks": 44429.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 2.762139, "cps": 347.101562, "mean_p": 0.520846, "boundary_entropy": 0.690219, "bpb": 8.188781}, "zh": {"chunks": 12945.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 2.191839, "cps": 101.132812, "mean_p": 0.52838, "boundary_entropy": 0.684744, "bpb": 8.120358}, "my": {"chunks": 7122.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.375871, "cps": 55.640625, "mean_p": 0.491225, "boundary_entropy": 0.691024, "bpb": 8.107101}}, "bpb": {"en": 8.02096, "de": 8.04953, "vi": 7.87992, "tr": 8.00297, "fi": 8.05806, "ru": 8.09154, "el": 7.79817, "hi": 8.02942, "ta": 8.08037, "th": 8.18878, "zh": 8.12036, "my": 8.1071}, "gini_cpc": 0.506909, "gini_cps": 0.500776, "bpic": 2.081366, "worst_lang": "th", "cpc_parity_vs_en": {"en": 1.0, "de": 0.6518977032357931, "vi": 5.026620497633516, "tr": 2.0366966701552127, "fi": 1.637418577207796, "ru": 6.073820330884974, "el": 10.02960142937851, "hi": 0.6538481720948288, "ta": 13.915836242768576, "th": 18.04410612819845, "zh": 14.31853257337934, "my": 2.4554352880429513}, "t": 4.9}
6
+ {"event": "probe", "step": 2, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2665.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.153135, "cps": 20.820312, "mean_p": 0.478086, "boundary_entropy": 0.685442, "bpb": 7.985878}, "de": {"chunks": 2047.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.099693, "cps": 15.992188, "mean_p": 0.46964, "boundary_entropy": 0.685274, "bpb": 8.020022}, "vi": {"chunks": 14175.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.768918, "cps": 110.742188, "mean_p": 0.507212, "boundary_entropy": 0.688287, "bpb": 7.855724}, "tr": {"chunks": 5645.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.311827, "cps": 44.101562, "mean_p": 0.486091, "boundary_entropy": 0.686383, "bpb": 7.979128}, "fi": {"chunks": 4688.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.249216, "cps": 36.625, "mean_p": 0.484349, "boundary_entropy": 0.686585, "bpb": 8.030504}, "ru": {"chunks": 17793.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.938945, "cps": 139.007812, "mean_p": 0.502572, "boundary_entropy": 0.690065, "bpb": 8.072627}, "el": {"chunks": 31632.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.541295, "cps": 247.125, "mean_p": 0.51819, "boundary_entropy": 0.68969, "bpb": 7.781468}, "hi": {"chunks": 1745.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.103715, "cps": 13.632812, "mean_p": 0.47671, "boundary_entropy": 0.689541, "bpb": 7.988682}, "ta": {"chunks": 40397.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 2.160845, "cps": 315.601562, "mean_p": 0.515052, "boundary_entropy": 0.690518, "bpb": 8.043904}, "th": {"chunks": 44496.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 2.766304, "cps": 347.625, "mean_p": 0.521205, "boundary_entropy": 0.690191, "bpb": 8.137322}, "zh": {"chunks": 12956.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 2.193701, "cps": 101.21875, "mean_p": 0.528511, "boundary_entropy": 0.68473, "bpb": 8.115871}, "my": {"chunks": 7237.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.38194, "cps": 56.539062, "mean_p": 0.491328, "boundary_entropy": 0.691028, "bpb": 8.101076}}, "bpb": {"en": 7.98588, "de": 8.02002, "vi": 7.85572, "tr": 7.97913, "fi": 8.0305, "ru": 8.07263, "el": 7.78147, "hi": 7.98868, "ta": 8.0439, "th": 8.13732, "zh": 8.11587, "my": 8.10108}, "gini_cpc": 0.506485, "gini_cps": 0.50097, "bpic": 2.069168, "worst_lang": "th", "cpc_parity_vs_en": {"en": 1.0, "de": 0.6510170193243129, "vi": 5.021192048936199, "tr": 2.0362931563487194, "fi": 1.6274311575509242, "ru": 6.1315019578529455, "el": 10.064975304785488, "hi": 0.6772784624863051, "ta": 14.110764754850626, "th": 18.064536091973867, "zh": 14.325322357967124, "my": 2.494147327671942}, "t": 6.5}
7
+ {"event": "probe", "step": 4, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2593.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.148997, "cps": 20.257812, "mean_p": 0.477701, "boundary_entropy": 0.685415, "bpb": 7.838458}, "de": {"chunks": 1997.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.097258, "cps": 15.601562, "mean_p": 0.469242, "boundary_entropy": 0.685232, "bpb": 7.891691}, "vi": {"chunks": 13915.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.754814, "cps": 108.710938, "mean_p": 0.50656, "boundary_entropy": 0.68831, "bpb": 7.759148}, "tr": {"chunks": 5519.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.304867, "cps": 43.117188, "mean_p": 0.485785, "boundary_entropy": 0.686374, "bpb": 7.876807}, "fi": {"chunks": 4529.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.240763, "cps": 35.382812, "mean_p": 0.483709, "boundary_entropy": 0.686539, "bpb": 7.910918}, "ru": {"chunks": 17253.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.910449, "cps": 134.789062, "mean_p": 0.502039, "boundary_entropy": 0.690065, "bpb": 7.942093}, "el": {"chunks": 31592.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.539346, "cps": 246.8125, "mean_p": 0.51809, "boundary_entropy": 0.689699, "bpb": 7.710974}, "hi": {"chunks": 1878.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.11162, "cps": 14.671875, "mean_p": 0.477394, "boundary_entropy": 0.689611, "bpb": 7.843225}, "ta": {"chunks": 40908.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 2.188179, "cps": 319.59375, "mean_p": 0.515421, "boundary_entropy": 0.690497, "bpb": 7.919492}, "th": {"chunks": 44582.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 2.771651, "cps": 348.296875, "mean_p": 0.521687, "boundary_entropy": 0.690152, "bpb": 7.984667}, "zh": {"chunks": 12996.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 2.200474, "cps": 101.53125, "mean_p": 0.52877, "boundary_entropy": 0.6847, "bpb": 8.098097}, "my": {"chunks": 7299.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.385212, "cps": 57.023438, "mean_p": 0.491348, "boundary_entropy": 0.691028, "bpb": 8.039072}}, "bpb": {"en": 7.83846, "de": 7.89169, "vi": 7.75915, "tr": 7.87681, "fi": 7.91092, "ru": 7.94209, "el": 7.71097, "hi": 7.84323, "ta": 7.91949, "th": 7.98467, "zh": 8.0981, "my": 8.03907}, "gini_cpc": 0.508937, "gini_cps": 0.503186, "bpic": 2.073808, "worst_lang": "th", "cpc_parity_vs_en": {"en": 1.0, "de": 0.6527505721086835, "vi": 5.065959017784942, "tr": 2.0461216283502988, "fi": 1.6158909203846255, "ru": 6.110503700342001, "el": 10.331369135968322, "hi": 0.7491385023915687, "ta": 14.686028991378954, "th": 18.60201908943773, "zh": 14.768550196816587, "my": 2.585363368077987}, "t": 8.0}
8
+ {"event": "probe", "step": 8, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2264.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.130093, "cps": 17.6875, "mean_p": 0.475804, "boundary_entropy": 0.685259, "bpb": 7.392165}, "de": {"chunks": 1774.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.086398, "cps": 13.859375, "mean_p": 0.467284, "boundary_entropy": 0.685006, "bpb": 7.493377}, "vi": {"chunks": 12826.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.695742, "cps": 100.203125, "mean_p": 0.504008, "boundary_entropy": 0.688382, "bpb": 7.484874}, "tr": {"chunks": 5055.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.279235, "cps": 39.492188, "mean_p": 0.484228, "boundary_entropy": 0.686305, "bpb": 7.566448}, "fi": {"chunks": 3865.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.205465, "cps": 30.195312, "mean_p": 0.480947, "boundary_entropy": 0.686323, "bpb": 7.542082}, "ru": {"chunks": 15176.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.800844, "cps": 118.5625, "mean_p": 0.499958, "boundary_entropy": 0.690059, "bpb": 7.441214}, "el": {"chunks": 31671.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.543195, "cps": 247.429688, "mean_p": 0.518151, "boundary_entropy": 0.689698, "bpb": 7.560604}, "hi": {"chunks": 2197.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.130579, "cps": 17.164062, "mean_p": 0.479041, "boundary_entropy": 0.689771, "bpb": 7.440378}, "ta": {"chunks": 41737.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 2.232522, "cps": 326.070312, "mean_p": 0.516032, "boundary_entropy": 0.690459, "bpb": 7.55434}, "th": {"chunks": 44718.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 2.780106, "cps": 349.359375, "mean_p": 0.522485, "boundary_entropy": 0.690083, "bpb": 7.577396}, "zh": {"chunks": 13107.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 2.219269, "cps": 102.398438, "mean_p": 0.52958, "boundary_entropy": 0.684605, "bpb": 8.021169}, "my": {"chunks": 8247.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.435244, "cps": 64.429688, "mean_p": 0.492142, "boundary_entropy": 0.691056, "bpb": 7.85545}}, "bpb": {"en": 7.39216, "de": 7.49338, "vi": 7.48487, "tr": 7.56645, "fi": 7.54208, "ru": 7.44121, "el": 7.5606, "hi": 7.44038, "ta": 7.55434, "th": 7.5774, "zh": 8.02117, "my": 7.85545}, "gini_cpc": 0.517732, "gini_cps": 0.51075, "bpic": 2.101332, "worst_lang": "th", "cpc_parity_vs_en": {"en": 1.0, "de": 0.6641235886934743, "vi": 5.348054093218365, "tr": 2.146437791483324, "fi": 1.579375193598663, "ru": 6.155960170431766, "el": 11.8622925041955, "hi": 1.0037433515874807, "ta": 17.161034228081995, "th": 21.37022053775712, "zh": 17.059156541410246, "my": 3.3456485380110417}, "t": 9.9}
9
+ {"event": "probe", "step": 16, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 1470.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.084468, "cps": 11.484375, "mean_p": 0.471159, "boundary_entropy": 0.684859, "bpb": 6.763621}, "de": {"chunks": 1189.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.057907, "cps": 9.289062, "mean_p": 0.463065, "boundary_entropy": 0.684565, "bpb": 6.891711}, "vi": {"chunks": 9791.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.531109, "cps": 76.492188, "mean_p": 0.497074, "boundary_entropy": 0.688455, "bpb": 7.100707}, "tr": {"chunks": 3864.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.213445, "cps": 30.1875, "mean_p": 0.479568, "boundary_entropy": 0.686064, "bpb": 7.112817}, "fi": {"chunks": 2328.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.123757, "cps": 18.1875, "mean_p": 0.472976, "boundary_entropy": 0.685561, "bpb": 6.986445}, "ru": {"chunks": 10097.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.532823, "cps": 78.882812, "mean_p": 0.494456, "boundary_entropy": 0.689968, "bpb": 6.590353}, "el": {"chunks": 32038.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.561078, "cps": 250.296875, "mean_p": 0.518384, "boundary_entropy": 0.689714, "bpb": 7.120871}, "hi": {"chunks": 5305.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.315305, "cps": 41.445312, "mean_p": 0.487075, "boundary_entropy": 0.690393, "bpb": 6.736586}, "ta": {"chunks": 46034.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 2.46237, "cps": 359.640625, "mean_p": 0.519724, "boundary_entropy": 0.690229, "bpb": 6.86062}, "th": {"chunks": 45404.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 2.822754, "cps": 354.71875, "mean_p": 0.525404, "boundary_entropy": 0.689823, "bpb": 6.685688}, "zh": {"chunks": 13563.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 2.296478, "cps": 105.960938, "mean_p": 0.532697, "boundary_entropy": 0.684263, "bpb": 7.802035}, "my": {"chunks": 11189.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.590511, "cps": 87.414062, "mean_p": 0.493836, "boundary_entropy": 0.691104, "bpb": 7.307892}}, "bpb": {"en": 6.76362, "de": 6.89171, "vi": 7.10071, "tr": 7.11282, "fi": 6.98644, "ru": 6.59035, "el": 7.12087, "hi": 6.73659, "ta": 6.86062, "th": 6.68569, "zh": 7.80204, "my": 7.30789}, "gini_cpc": 0.53621, "gini_cps": 0.531771, "bpic": 2.10554, "worst_lang": "th", "cpc_parity_vs_en": {"en": 1.0, "de": 0.6855454186739713, "vi": 6.287683809080996, "tr": 2.526930816518178, "fi": 1.4651357922361972, "ru": 6.30797447633407, "el": 18.481249807334134, "hi": 3.7328204506170968, "ta": 29.15144112525995, "th": 33.417952332316204, "zh": 27.187489374347777, "my": 6.99092564828338}, "t": 12.0}
10
+ {"event": "probe", "step": 32, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 383.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.022008, "cps": 2.992188, "mean_p": 0.433994, "boundary_entropy": 0.677224, "bpb": 6.352728}, "de": {"chunks": 335.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.016315, "cps": 2.617188, "mean_p": 0.429047, "boundary_entropy": 0.676857, "bpb": 6.44189}, "vi": {"chunks": 581.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.031516, "cps": 4.539062, "mean_p": 0.45394, "boundary_entropy": 0.683796, "bpb": 6.864593}, "tr": {"chunks": 531.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.029332, "cps": 4.148438, "mean_p": 0.443549, "boundary_entropy": 0.680185, "bpb": 6.736257}, "fi": {"chunks": 349.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.018553, "cps": 2.726562, "mean_p": 0.43495, "boundary_entropy": 0.678082, "bpb": 6.507426}, "ru": {"chunks": 139.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.007335, "cps": 1.085938, "mean_p": 0.445588, "boundary_entropy": 0.683888, "bpb": 6.150857}, "el": {"chunks": 8892.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.43327, "cps": 69.46875, "mean_p": 0.491312, "boundary_entropy": 0.690112, "bpb": 6.302296}, "hi": {"chunks": 1311.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.07792, "cps": 10.242188, "mean_p": 0.481567, "boundary_entropy": 0.690105, "bpb": 6.098238}, "ta": {"chunks": 26982.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 1.443274, "cps": 210.796875, "mean_p": 0.503229, "boundary_entropy": 0.691096, "bpb": 6.387771}, "th": {"chunks": 27042.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 1.681194, "cps": 211.265625, "mean_p": 0.504089, "boundary_entropy": 0.691072, "bpb": 5.606693}, "zh": {"chunks": 11444.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.93769, "cps": 89.40625, "mean_p": 0.517991, "boundary_entropy": 0.685907, "bpb": 7.424534}, "my": {"chunks": 1991.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.105077, "cps": 15.554688, "mean_p": 0.481385, "boundary_entropy": 0.690415, "bpb": 6.382926}}, "bpb": {"en": 6.35273, "de": 6.44189, "vi": 6.86459, "tr": 6.73626, "fi": 6.50743, "ru": 6.15086, "el": 6.3023, "hi": 6.09824, "ta": 6.38777, "th": 5.60669, "zh": 7.42453, "my": 6.38293}, "gini_cpc": 0.69672, "gini_cps": 0.688793, "bpic": 4.798462, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 0.7413405332738905, "vi": 1.4320505112522228, "tr": 1.3328132939320676, "fi": 0.8430219638445534, "ru": 0.3332966374339508, "el": 19.687200083355503, "hi": 3.540568205183912, "ta": 65.58039296470692, "th": 76.39115729021168, "zh": 88.04602479754624, "my": 4.774558624373095}, "t": 14.8}
11
+ {"event": "probe", "step": 64, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 243.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.013963, "cps": 1.898438, "mean_p": 0.361583, "boundary_entropy": 0.645794, "bpb": 5.795061}, "de": {"chunks": 211.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.010276, "cps": 1.648438, "mean_p": 0.357859, "boundary_entropy": 0.645031, "bpb": 5.937274}, "vi": {"chunks": 184.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.009981, "cps": 1.4375, "mean_p": 0.366439, "boundary_entropy": 0.650482, "bpb": 6.511334}, "tr": {"chunks": 188.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.010385, "cps": 1.46875, "mean_p": 0.362114, "boundary_entropy": 0.646959, "bpb": 6.32238}, "fi": {"chunks": 166.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.008825, "cps": 1.296875, "mean_p": 0.359204, "boundary_entropy": 0.645288, "bpb": 6.008648}, "ru": {"chunks": 130.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.00686, "cps": 1.015625, "mean_p": 0.38614, "boundary_entropy": 0.663323, "bpb": 5.589999}, "el": {"chunks": 128.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.006237, "cps": 1.0, "mean_p": 0.420718, "boundary_entropy": 0.677257, "bpb": 6.311157}, "hi": {"chunks": 140.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.008321, "cps": 1.09375, "mean_p": 0.375218, "boundary_entropy": 0.658523, "bpb": 5.840602}, "ta": {"chunks": 146.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.00781, "cps": 1.140625, "mean_p": 0.373999, "boundary_entropy": 0.65836, "bpb": 5.796934}, "th": {"chunks": 131.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.008144, "cps": 1.023438, "mean_p": 0.384179, "boundary_entropy": 0.663251, "bpb": 5.662074}, "zh": {"chunks": 186.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.031493, "cps": 1.453125, "mean_p": 0.391527, "boundary_entropy": 0.660036, "bpb": 7.3059}, "my": {"chunks": 254.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.013405, "cps": 1.984375, "mean_p": 0.437689, "boundary_entropy": 0.683144, "bpb": 5.71819}}, "bpb": {"en": 5.79506, "de": 5.93727, "vi": 6.51133, "tr": 6.32238, "fi": 6.00865, "ru": 5.59, "el": 6.31116, "hi": 5.8406, "ta": 5.79693, "th": 5.66207, "zh": 7.3059, "my": 5.71819}, "gini_cpc": 0.245113, "gini_cps": 0.132139, "bpic": 182.145705, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 0.7359492969161957, "vi": 0.7148131405974277, "tr": 0.7437468586817683, "fi": 0.6319955949073663, "ru": 0.49130590572982835, "el": 0.4466701917691864, "hi": 0.5959239080586283, "ta": 0.5593005325910738, "th": 0.5832679016183318, "zh": 2.255471522996074, "my": 0.9600374774887476}, "t": 19.3}
12
+ {"event": "probe", "step": 128, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 133.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.007642, "cps": 1.039062, "mean_p": 0.379597, "boundary_entropy": 0.655613, "bpb": 4.548053}, "de": {"chunks": 152.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.007403, "cps": 1.1875, "mean_p": 0.37754, "boundary_entropy": 0.655408, "bpb": 4.753989}, "vi": {"chunks": 827.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.04486, "cps": 6.460938, "mean_p": 0.418843, "boundary_entropy": 0.671647, "bpb": 5.113376}, "tr": {"chunks": 300.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.016572, "cps": 2.34375, "mean_p": 0.396282, "boundary_entropy": 0.663492, "bpb": 5.134856}, "fi": {"chunks": 220.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.011695, "cps": 1.71875, "mean_p": 0.382113, "boundary_entropy": 0.656636, "bpb": 4.833313}, "ru": {"chunks": 161.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.008496, "cps": 1.257812, "mean_p": 0.416762, "boundary_entropy": 0.674024, "bpb": 3.777848}, "el": {"chunks": 128.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.006237, "cps": 1.0, "mean_p": 0.398727, "boundary_entropy": 0.669046, "bpb": 4.63175}, "hi": {"chunks": 135.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.008024, "cps": 1.054688, "mean_p": 0.428426, "boundary_entropy": 0.679865, "bpb": 3.545774}, "ta": {"chunks": 131.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.007007, "cps": 1.023438, "mean_p": 0.417263, "boundary_entropy": 0.676927, "bpb": 3.739576}, "th": {"chunks": 215.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.013366, "cps": 1.679688, "mean_p": 0.428008, "boundary_entropy": 0.677759, "bpb": 3.443156}, "zh": {"chunks": 139.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.023535, "cps": 1.085938, "mean_p": 0.412482, "boundary_entropy": 0.670152, "bpb": 6.367455}, "my": {"chunks": 128.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.006755, "cps": 1.0, "mean_p": 0.401239, "boundary_entropy": 0.6711, "bpb": 4.022402}}, "bpb": {"en": 4.54805, "de": 4.75399, "vi": 5.11338, "tr": 5.13486, "fi": 4.83331, "ru": 3.77785, "el": 4.63175, "hi": 3.54577, "ta": 3.73958, "th": 3.44316, "zh": 6.36745, "my": 4.0224}, "gini_cpc": 0.353397, "gini_cps": 0.323498, "bpic": 143.792057, "worst_lang": "vi", "cpc_parity_vs_en": {"en": 1.0, "de": 0.9686428119194885, "vi": 5.869956012896358, "tr": 2.168418892893173, "fi": 1.5303236028511553, "ru": 1.111703930009721, "el": 0.8160966661647541, "hi": 1.0499078304974918, "ta": 0.9168922573885906, "th": 1.7490002594300487, "zh": 3.079596638056367, "my": 0.8839324403472265}, "t": 27.5}
13
+ {"event": "train", "step": 200, "phase": 1, "objective": "global", "lm_loss": 2.58195, "bpb": 3.72497, "chunk_loss": 0.021673, "F": 0.00098, "G": 0.39935, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.66927, "router_noise": 0.0, "grad_norm": 1.4347, "lr": 0.000201, "router_trainable": true, "sec_per_step": 0.158, "alpha": 0.03, "t": 35.3}
14
+ {"event": "probe", "step": 256, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 491.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.028214, "cps": 3.835938, "mean_p": 0.414525, "boundary_entropy": 0.669415, "bpb": 3.672405}, "de": {"chunks": 840.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.04091, "cps": 6.5625, "mean_p": 0.405741, "boundary_entropy": 0.666839, "bpb": 3.88857}, "vi": {"chunks": 1137.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.061676, "cps": 8.882812, "mean_p": 0.414811, "boundary_entropy": 0.668569, "bpb": 3.272238}, "tr": {"chunks": 1118.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.061758, "cps": 8.734375, "mean_p": 0.413739, "boundary_entropy": 0.669019, "bpb": 3.924511}, "fi": {"chunks": 1019.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.05417, "cps": 7.960938, "mean_p": 0.420978, "boundary_entropy": 0.672208, "bpb": 3.919382}, "ru": {"chunks": 195.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.01029, "cps": 1.523438, "mean_p": 0.399764, "boundary_entropy": 0.66761, "bpb": 2.665971}, "el": {"chunks": 825.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.040199, "cps": 6.445312, "mean_p": 0.438363, "boundary_entropy": 0.681206, "bpb": 2.751758}, "hi": {"chunks": 418.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.024844, "cps": 3.265625, "mean_p": 0.425079, "boundary_entropy": 0.677312, "bpb": 2.163683}, "ta": {"chunks": 175.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.009361, "cps": 1.367188, "mean_p": 0.439973, "boundary_entropy": 0.682977, "bpb": 2.040198}, "th": {"chunks": 2904.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.180541, "cps": 22.6875, "mean_p": 0.456926, "boundary_entropy": 0.685649, "bpb": 2.111542}, "zh": {"chunks": 436.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.073823, "cps": 3.40625, "mean_p": 0.40505, "boundary_entropy": 0.664369, "bpb": 5.011176}, "my": {"chunks": 130.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.006861, "cps": 1.015625, "mean_p": 0.419887, "boundary_entropy": 0.676671, "bpb": 2.270936}}, "bpb": {"en": 3.6724, "de": 3.88857, "vi": 3.27224, "tr": 3.92451, "fi": 3.91938, "ru": 2.66597, "el": 2.75176, "hi": 2.16368, "ta": 2.0402, "th": 2.11154, "zh": 5.01118, "my": 2.27094}, "gini_cpc": 0.437818, "gini_cps": 0.431668, "bpic": 39.614059, "worst_lang": "th", "cpc_parity_vs_en": {"en": 1.0, "de": 1.4500050239527984, "vi": 2.1860492941291496, "tr": 2.1889401144593177, "fi": 1.9200163573746392, "ru": 0.3647270929501475, "el": 1.4248059873811043, "hi": 0.8805699016169203, "ta": 0.33178382317935734, "th": 6.3990893591643685, "zh": 2.616589984433656, "my": 0.24317706042520917}, "t": 41.9}
15
+ {"event": "train", "step": 400, "phase": 1, "objective": "global", "lm_loss": 2.03632, "bpb": 2.93779, "chunk_loss": 0.024239, "F": 0.02814, "G": 0.36063, "chunks": 28.81, "bytes_per_chunk": 35.54, "boundary_entropy": 0.64731, "router_noise": 0.0, "grad_norm": 1.7345, "lr": 0.0002998, "router_trainable": true, "sec_per_step": 0.105, "alpha": 0.03, "t": 56.3}
16
+ {"event": "train", "step": 600, "phase": 1, "objective": "global", "lm_loss": 1.87477, "bpb": 2.70472, "chunk_loss": 0.022649, "F": 0.00098, "G": 0.37207, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65357, "router_noise": 0.0, "grad_norm": 0.7489, "lr": 0.0002982, "router_trainable": true, "sec_per_step": 0.095, "alpha": 0.03, "t": 75.3}
17
+ {"event": "train", "step": 800, "phase": 1, "objective": "global", "lm_loss": 1.67521, "bpb": 2.41682, "chunk_loss": 0.022193, "F": 0.00098, "G": 0.38481, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65926, "router_noise": 0.0, "grad_norm": 0.8214, "lr": 0.0002949, "router_trainable": true, "sec_per_step": 0.096, "alpha": 0.03, "t": 94.6}
18
+ {"event": "eval", "step": 1000, "phase": 1, "bpb": {"en": 2.6676, "de": 2.98901, "vi": 2.3604, "tr": 2.97268, "fi": 3.23071, "ru": 1.92693, "el": 2.07715, "hi": 1.58912, "ta": 1.37095, "th": 1.62376, "zh": 3.71974, "my": 1.45979}, "bytes_per_chunk": {"en": 88.923, "de": 193.179, "vi": 722.16, "tr": 107.745, "fi": 293.226, "ru": 113.188, "el": 473.184, "hi": 188.052, "ta": 439.102, "th": 946.368, "zh": 826.953, "my": 341.333}, "bpb_mean": 2.33232, "t": 113.4}
19
+ {"event": "train", "step": 1000, "phase": 1, "objective": "global", "lm_loss": 1.78443, "bpb": 2.57438, "chunk_loss": 0.025766, "F": 0.04855, "G": 0.33372, "chunks": 49.72, "bytes_per_chunk": 20.596, "boundary_entropy": 0.62891, "router_noise": 0.0, "grad_norm": 0.8492, "lr": 0.0002901, "router_trainable": true, "sec_per_step": 0.094, "alpha": 0.03, "t": 113.4}
20
+ {"event": "train", "step": 1200, "phase": 1, "objective": "global", "lm_loss": 1.65627, "bpb": 2.3895, "chunk_loss": 0.023015, "F": 0.00098, "G": 0.36183, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.64658, "router_noise": 0.0, "grad_norm": 0.8266, "lr": 0.0002837, "router_trainable": true, "sec_per_step": 0.093, "alpha": 0.03, "t": 132.0}
21
+ {"event": "train", "step": 1400, "phase": 1, "objective": "global", "lm_loss": 1.61348, "bpb": 2.32776, "chunk_loss": 0.02191, "F": 0.00098, "G": 0.39272, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.66309, "router_noise": 0.0, "grad_norm": 0.9687, "lr": 0.0002759, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 149.9}
22
+ {"event": "train", "step": 1600, "phase": 1, "objective": "global", "lm_loss": 1.62579, "bpb": 2.34551, "chunk_loss": 0.023298, "F": 0.00098, "G": 0.35393, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.64151, "router_noise": 0.0, "grad_norm": 0.8963, "lr": 0.0002668, "router_trainable": true, "sec_per_step": 0.088, "alpha": 0.03, "t": 167.5}
23
+ {"event": "train", "step": 1800, "phase": 1, "objective": "global", "lm_loss": 1.62437, "bpb": 2.34347, "chunk_loss": 0.023368, "F": 0.00098, "G": 0.35197, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.63951, "router_noise": 0.0, "grad_norm": 1.2577, "lr": 0.0002564, "router_trainable": true, "sec_per_step": 0.091, "alpha": 0.03, "t": 185.7}
24
+ {"event": "probe", "step": 2000, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 257.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.014768, "cps": 2.007812, "mean_p": 0.355916, "boundary_entropy": 0.634868, "bpb": 2.334603}, "de": {"chunks": 336.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.016364, "cps": 2.625, "mean_p": 0.356176, "boundary_entropy": 0.636464, "bpb": 2.615902}, "vi": {"chunks": 347.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.018823, "cps": 2.710938, "mean_p": 0.356379, "boundary_entropy": 0.638153, "bpb": 2.12749}, "tr": {"chunks": 382.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.021101, "cps": 2.984375, "mean_p": 0.35538, "boundary_entropy": 0.635151, "bpb": 2.613461}, "fi": {"chunks": 233.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.012386, "cps": 1.820312, "mean_p": 0.36462, "boundary_entropy": 0.642233, "bpb": 2.848301}, "ru": {"chunks": 543.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.028654, "cps": 4.242188, "mean_p": 0.350991, "boundary_entropy": 0.638405, "bpb": 1.688301}, "el": {"chunks": 448.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.021829, "cps": 3.5, "mean_p": 0.351743, "boundary_entropy": 0.638343, "bpb": 1.7839}, "hi": {"chunks": 1177.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.069955, "cps": 9.195312, "mean_p": 0.359596, "boundary_entropy": 0.641316, "bpb": 1.411766}, "ta": {"chunks": 1155.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.061781, "cps": 9.023438, "mean_p": 0.365126, "boundary_entropy": 0.645848, "bpb": 1.127496}, "th": {"chunks": 224.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.013926, "cps": 1.75, "mean_p": 0.350389, "boundary_entropy": 0.638766, "bpb": 1.318851}, "zh": {"chunks": 137.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.023197, "cps": 1.070312, "mean_p": 0.349868, "boundary_entropy": 0.633739, "bpb": 3.50542}, "my": {"chunks": 210.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.011083, "cps": 1.640625, "mean_p": 0.321643, "boundary_entropy": 0.613809, "bpb": 1.251973}}, "bpb": {"en": 2.3346, "de": 2.6159, "vi": 2.12749, "tr": 2.61346, "fi": 2.8483, "ru": 1.6883, "el": 1.7839, "hi": 1.41177, "ta": 1.1275, "th": 1.31885, "zh": 3.50542, "my": 1.25197}, "gini_cpc": 0.333492, "gini_cps": 0.361519, "bpic": 70.431455, "worst_lang": "hi", "cpc_parity_vs_en": {"en": 1.0, "de": 1.1080972245304654, "vi": 1.2746100242834482, "tr": 1.4289064886164793, "fi": 0.8387548307172702, "ru": 1.9403568678582794, "el": 1.4781828719832413, "hi": 4.737098189765323, "ta": 4.183574542393095, "th": 0.943013586624559, "zh": 1.5707899768223568, "my": 0.7504934660414043}, "t": 203.2}
25
+ {"event": "eval", "step": 2000, "phase": 1, "bpb": {"en": 2.34455, "de": 2.62199, "vi": 2.13169, "tr": 2.62324, "fi": 2.84831, "ru": 1.66786, "el": 1.77523, "hi": 1.40934, "ta": 1.13558, "th": 1.33676, "zh": 3.50947, "my": 1.26013}, "bytes_per_chunk": {"en": 86.345, "de": 75.199, "vi": 144.194, "tr": 80.709, "fi": 131.863, "ru": 97.162, "el": 117.501, "hi": 43.394, "ta": 53.685, "th": 179.92, "zh": 564.966, "my": 227.358}, "bpb_mean": 2.05535, "t": 204.4}
26
+ {"event": "train", "step": 2000, "phase": 1, "objective": "global", "lm_loss": 1.55604, "bpb": 2.24489, "chunk_loss": 0.025613, "F": 0.0563, "G": 0.35118, "chunks": 57.66, "bytes_per_chunk": 17.76, "boundary_entropy": 0.63933, "router_noise": 0.0, "grad_norm": 0.8047, "lr": 0.000245, "router_trainable": true, "sec_per_step": 0.094, "alpha": 0.03, "t": 204.4}
27
+ {"event": "train", "step": 2200, "phase": 1, "objective": "global", "lm_loss": 1.52653, "bpb": 2.20232, "chunk_loss": 0.022589, "F": 0.00098, "G": 0.37375, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65299, "router_noise": 0.0, "grad_norm": 0.7575, "lr": 0.0002325, "router_trainable": true, "sec_per_step": 0.088, "alpha": 0.03, "t": 222.0}
28
+ {"event": "phase_transition", "step": 2400, "phase": 2, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0, "t": 239.2}
29
+ {"event": "train", "step": 2400, "phase": 2, "objective": "global", "lm_loss": 1.55384, "bpb": 2.24171, "chunk_loss": 0.054117, "F": 0.73846, "G": 0.36101, "chunks": 756.19, "bytes_per_chunk": 1.354, "boundary_entropy": 0.64339, "router_noise": 0.0, "grad_norm": 0.8794, "lr": 0.0002192, "router_trainable": true, "sec_per_step": 0.086, "alpha": 0.03, "t": 239.3}
30
+ {"event": "train", "step": 2600, "phase": 2, "objective": "global", "lm_loss": 1.71623, "bpb": 2.47599, "chunk_loss": 0.022556, "F": 0.00098, "G": 0.37466, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65261, "router_noise": 0.0, "grad_norm": 0.805, "lr": 0.0002053, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 257.0}
31
+ {"event": "train", "step": 2800, "phase": 2, "objective": "global", "lm_loss": 1.50444, "bpb": 2.17045, "chunk_loss": 0.023924, "F": 0.02304, "G": 0.36283, "chunks": 23.59, "bytes_per_chunk": 43.401, "boundary_entropy": 0.64536, "router_noise": 0.0, "grad_norm": 0.7812, "lr": 0.0001909, "router_trainable": true, "sec_per_step": 0.087, "alpha": 0.03, "t": 274.4}
32
+ {"event": "eval", "step": 3000, "phase": 2, "bpb": {"en": 2.21551, "de": 2.45319, "vi": 2.04296, "tr": 2.43092, "fi": 2.65147, "ru": 1.53801, "el": 1.61944, "hi": 1.31357, "ta": 1.08274, "th": 1.23433, "zh": 3.36493, "my": 1.18579}, "bytes_per_chunk": {"en": 40.661, "de": 53.521, "vi": 82.747, "tr": 63.442, "fi": 86.288, "ru": 55.211, "el": 71.998, "hi": 36.419, "ta": 54.534, "th": 127.938, "zh": 89.043, "my": 69.369}, "bpb_mean": 1.92774, "t": 292.8}
33
+ {"event": "train", "step": 3000, "phase": 2, "objective": "global", "lm_loss": 1.52656, "bpb": 2.20236, "chunk_loss": 0.022673, "F": 0.00098, "G": 0.3714, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.6496, "router_noise": 0.0, "grad_norm": 0.8643, "lr": 0.0001761, "router_trainable": true, "sec_per_step": 0.093, "alpha": 0.03, "t": 292.9}
34
+ {"event": "train", "step": 3200, "phase": 2, "objective": "global", "lm_loss": 1.36113, "bpb": 1.9637, "chunk_loss": 0.02255, "F": 0.00098, "G": 0.37483, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65182, "router_noise": 0.0, "grad_norm": 0.9588, "lr": 0.0001613, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 310.7}
35
+ {"event": "train", "step": 3400, "phase": 2, "objective": "global", "lm_loss": 1.4062, "bpb": 2.02872, "chunk_loss": 0.036152, "F": 0.30194, "G": 0.3746, "chunks": 309.19, "bytes_per_chunk": 3.312, "boundary_entropy": 0.65167, "router_noise": 0.0, "grad_norm": 0.7784, "lr": 0.0001465, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 328.5}
36
+ {"event": "train", "step": 3600, "phase": 2, "objective": "global", "lm_loss": 1.52441, "bpb": 2.19926, "chunk_loss": 0.048164, "F": 0.57361, "G": 0.37125, "chunks": 587.38, "bytes_per_chunk": 1.743, "boundary_entropy": 0.6504, "router_noise": 0.0, "grad_norm": 1.7498, "lr": 0.0001319, "router_trainable": true, "sec_per_step": 0.093, "alpha": 0.03, "t": 347.1}
37
+ {"event": "train", "step": 3800, "phase": 2, "objective": "global", "lm_loss": 1.41311, "bpb": 2.03869, "chunk_loss": 0.042968, "F": 0.48413, "G": 0.35482, "chunks": 495.75, "bytes_per_chunk": 2.066, "boundary_entropy": 0.63936, "router_noise": 0.0, "grad_norm": 0.8064, "lr": 0.0001177, "router_trainable": true, "sec_per_step": 0.088, "alpha": 0.03, "t": 364.7}
38
+ {"event": "probe", "step": 4000, "phase": 2, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 395.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.022697, "cps": 3.085938, "mean_p": 0.367709, "boundary_entropy": 0.639918, "bpb": 2.140504}, "de": {"chunks": 337.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.016413, "cps": 2.632812, "mean_p": 0.367454, "boundary_entropy": 0.643109, "bpb": 2.355928}, "vi": {"chunks": 457.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.02479, "cps": 3.570312, "mean_p": 0.372393, "boundary_entropy": 0.646221, "bpb": 1.985718}, "tr": {"chunks": 464.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.025631, "cps": 3.625, "mean_p": 0.376195, "boundary_entropy": 0.647673, "bpb": 2.32451}, "fi": {"chunks": 290.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.015417, "cps": 2.265625, "mean_p": 0.375932, "boundary_entropy": 0.648623, "bpb": 2.539534}, "ru": {"chunks": 833.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.043958, "cps": 6.507812, "mean_p": 0.38216, "boundary_entropy": 0.656092, "bpb": 1.489035}, "el": {"chunks": 489.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.023827, "cps": 3.820312, "mean_p": 0.381608, "boundary_entropy": 0.656643, "bpb": 1.541612}, "hi": {"chunks": 1202.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.071441, "cps": 9.390625, "mean_p": 0.372576, "boundary_entropy": 0.647457, "bpb": 1.269919}, "ta": {"chunks": 826.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.044183, "cps": 6.453125, "mean_p": 0.371251, "boundary_entropy": 0.647964, "bpb": 1.03482}, "th": {"chunks": 508.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.031582, "cps": 3.96875, "mean_p": 0.377789, "boundary_entropy": 0.654107, "bpb": 1.163126}, "zh": {"chunks": 210.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.035557, "cps": 1.640625, "mean_p": 0.370316, "boundary_entropy": 0.64517, "bpb": 3.272949}, "my": {"chunks": 303.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.015991, "cps": 2.367188, "mean_p": 0.362003, "boundary_entropy": 0.64003, "bpb": 1.137374}}, "bpb": {"en": 2.1405, "de": 2.35593, "vi": 1.98572, "tr": 2.32451, "fi": 2.53953, "ru": 1.48904, "el": 1.54161, "hi": 1.26992, "ta": 1.03482, "th": 1.16313, "zh": 3.27295, "my": 1.13737}, "gini_cpc": 0.258247, "gini_cps": 0.271935, "bpic": 60.782547, "worst_lang": "hi", "cpc_parity_vs_en": {"en": 1.0, "de": 0.7231102510500231, "vi": 1.0921947451360063, "tr": 1.129261322516654, "fi": 0.6792241813805415, "ru": 1.9367020473598076, "el": 1.049772129694563, "hi": 3.1475774070382, "ta": 1.9466218883408208, "th": 1.3914566208787966, "zh": 1.5665810782426794, "my": 0.704541008970587}, "t": 383.9}
39
+ {"event": "eval", "step": 4000, "phase": 2, "bpb": {"en": 2.15032, "de": 2.36116, "vi": 1.98784, "tr": 2.33377, "fi": 2.54249, "ru": 1.46527, "el": 1.53314, "hi": 1.26588, "ta": 1.04159, "th": 1.18211, "zh": 3.27315, "my": 1.14581}, "bytes_per_chunk": {"en": 44.184, "de": 86.09, "vi": 81.059, "tr": 60.194, "fi": 103.737, "ru": 57.614, "el": 95.743, "hi": 40.839, "ta": 64.172, "th": 97.162, "zh": 161.022, "my": 191.486}, "bpb_mean": 1.85688, "t": 385.1}
40
+ {"event": "train", "step": 4000, "phase": 2, "objective": "global", "lm_loss": 1.45512, "bpb": 2.0993, "chunk_loss": 0.022719, "F": 0.00098, "G": 0.37011, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.64993, "router_noise": 0.0, "grad_norm": 0.8842, "lr": 0.000104, "router_trainable": true, "sec_per_step": 0.102, "alpha": 0.03, "t": 385.2}
41
+ {"event": "phase_transition", "step": 4200, "phase": 3, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0, "t": 403.4}
42
+ {"event": "train", "step": 4200, "phase": 3, "objective": "global", "lm_loss": 1.29843, "bpb": 1.87324, "chunk_loss": 0.055929, "F": 0.70004, "G": 0.3908, "chunks": 716.84, "bytes_per_chunk": 1.428, "boundary_entropy": 0.66019, "router_noise": 0.0, "grad_norm": 0.818, "lr": 9.12e-05, "router_trainable": true, "sec_per_step": 0.091, "alpha": 0.03, "t": 403.4}
43
+ {"event": "train", "step": 4400, "phase": 3, "objective": "global", "lm_loss": 1.37957, "bpb": 1.9903, "chunk_loss": 0.022627, "F": 0.00421, "G": 0.37679, "chunks": 4.31, "bytes_per_chunk": 237.449, "boundary_entropy": 0.65397, "router_noise": 0.0, "grad_norm": 0.8301, "lr": 7.92e-05, "router_trainable": true, "sec_per_step": 0.092, "alpha": 0.03, "t": 421.9}
44
+ {"event": "train", "step": 4600, "phase": 3, "objective": "global", "lm_loss": 1.49599, "bpb": 2.15826, "chunk_loss": 0.039673, "F": 0.39044, "G": 0.36615, "chunks": 399.81, "bytes_per_chunk": 2.561, "boundary_entropy": 0.64758, "router_noise": 0.0, "grad_norm": 0.7863, "lr": 6.82e-05, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 439.6}
45
+ {"event": "train", "step": 4800, "phase": 3, "objective": "global", "lm_loss": 1.30487, "bpb": 1.88253, "chunk_loss": 0.022297, "F": 0.00098, "G": 0.38191, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65608, "router_noise": 0.0, "grad_norm": 0.8666, "lr": 5.85e-05, "router_trainable": true, "sec_per_step": 0.093, "alpha": 0.03, "t": 458.2}
46
+ {"event": "eval", "step": 5000, "phase": 3, "bpb": {"en": 2.10913, "de": 2.31166, "vi": 1.96233, "tr": 2.28259, "fi": 2.47792, "ru": 1.42173, "el": 1.49422, "hi": 1.23133, "ta": 1.01495, "th": 1.14546, "zh": 3.22296, "my": 1.11971}, "bytes_per_chunk": {"en": 48.073, "de": 70.907, "vi": 71.742, "tr": 65.194, "fi": 86.947, "ru": 67.459, "el": 144.831, "hi": 49.21, "ta": 76.92, "th": 118.403, "zh": 149.2, "my": 121.814}, "bpb_mean": 1.81617, "t": 476.3}
47
+ {"event": "train", "step": 5000, "phase": 3, "objective": "global", "lm_loss": 1.51414, "bpb": 2.18444, "chunk_loss": 0.025256, "F": 0.05795, "G": 0.36871, "chunks": 59.34, "bytes_per_chunk": 17.255, "boundary_entropy": 0.64801, "router_noise": 0.0, "grad_norm": 0.8675, "lr": 5e-05, "router_trainable": true, "sec_per_step": 0.091, "alpha": 0.03, "t": 476.4}
48
+ {"event": "train", "step": 5200, "phase": 3, "objective": "global", "lm_loss": 1.40099, "bpb": 2.02121, "chunk_loss": 0.022523, "F": 0.00351, "G": 0.37881, "chunks": 3.59, "bytes_per_chunk": 284.939, "boundary_entropy": 0.65434, "router_noise": 0.0, "grad_norm": 0.8979, "lr": 4.29e-05, "router_trainable": true, "sec_per_step": 0.084, "alpha": 0.03, "t": 493.2}
49
+ {"event": "train", "step": 5400, "phase": 3, "objective": "global", "lm_loss": 1.27771, "bpb": 1.84335, "chunk_loss": 0.022471, "F": 0.00098, "G": 0.37704, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65352, "router_noise": 0.0, "grad_norm": 0.8255, "lr": 3.73e-05, "router_trainable": true, "sec_per_step": 0.09, "alpha": 0.03, "t": 511.1}
50
+ {"event": "train", "step": 5600, "phase": 3, "objective": "global", "lm_loss": 1.36022, "bpb": 1.96238, "chunk_loss": 0.022816, "F": 0.00098, "G": 0.36739, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.64794, "router_noise": 0.0, "grad_norm": 0.8147, "lr": 3.33e-05, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 528.9}
51
+ {"event": "train", "step": 5800, "phase": 3, "objective": "global", "lm_loss": 1.38027, "bpb": 1.99131, "chunk_loss": 0.028322, "F": 0.12729, "G": 0.36518, "chunks": 130.34, "bytes_per_chunk": 7.856, "boundary_entropy": 0.64685, "router_noise": 0.0, "grad_norm": 0.8599, "lr": 3.08e-05, "router_trainable": true, "sec_per_step": 0.084, "alpha": 0.03, "t": 545.7}
52
+ {"event": "probe", "step": 6000, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 386.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.02218, "cps": 3.015625, "mean_p": 0.372235, "boundary_entropy": 0.642723, "bpb": 2.081353}, "de": {"chunks": 371.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.018068, "cps": 2.898438, "mean_p": 0.378678, "boundary_entropy": 0.64967, "bpb": 2.281939}, "vi": {"chunks": 535.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.029021, "cps": 4.179688, "mean_p": 0.379752, "boundary_entropy": 0.650654, "bpb": 1.945007}, "tr": {"chunks": 366.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.020218, "cps": 2.859375, "mean_p": 0.37595, "boundary_entropy": 0.647683, "bpb": 2.245198}, "fi": {"chunks": 283.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.015044, "cps": 2.210938, "mean_p": 0.37811, "boundary_entropy": 0.649958, "bpb": 2.445838}, "ru": {"chunks": 744.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.039261, "cps": 5.8125, "mean_p": 0.378459, "boundary_entropy": 0.654131, "bpb": 1.425385}, "el": {"chunks": 375.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.018272, "cps": 2.929688, "mean_p": 0.372367, "boundary_entropy": 0.651811, "bpb": 1.473511}, "hi": {"chunks": 921.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.05474, "cps": 7.195312, "mean_p": 0.375501, "boundary_entropy": 0.650672, "bpb": 1.217113}, "ta": {"chunks": 707.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.037818, "cps": 5.523438, "mean_p": 0.362146, "boundary_entropy": 0.640883, "bpb": 0.992749}, "th": {"chunks": 417.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.025925, "cps": 3.257812, "mean_p": 0.370949, "boundary_entropy": 0.649942, "bpb": 1.111332}, "zh": {"chunks": 228.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.038605, "cps": 1.78125, "mean_p": 0.375965, "boundary_entropy": 0.648166, "bpb": 3.184903}, "my": {"chunks": 280.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.014777, "cps": 2.1875, "mean_p": 0.360745, "boundary_entropy": 0.639457, "bpb": 1.09482}}, "bpb": {"en": 2.08135, "de": 2.28194, "vi": 1.94501, "tr": 2.2452, "fi": 2.44584, "ru": 1.42538, "el": 1.47351, "hi": 1.21711, "ta": 0.99275, "th": 1.11133, "zh": 3.1849, "my": 1.09482}, "gini_cpc": 0.23188, "gini_cps": 0.233995, "bpic": 68.373597, "worst_lang": "hi", "cpc_parity_vs_en": {"en": 1.0, "de": 0.8146260953869533, "vi": 1.3084208484930249, "tr": 0.911522408188721, "fi": 0.6782836797893858, "ru": 1.7701111460483683, "el": 0.823810338912061, "hi": 2.4679785047232636, "ta": 1.7050250337085502, "th": 1.168831225307265, "zh": 1.74051680121559, "my": 0.6662411771075679}, "t": 564.4}
53
+ {"event": "eval", "step": 6000, "phase": 3, "bpb": {"en": 2.09161, "de": 2.28849, "vi": 1.94771, "tr": 2.25268, "fi": 2.44957, "ru": 1.4018, "el": 1.466, "hi": 1.21198, "ta": 1.00013, "th": 1.1284, "zh": 3.18371, "my": 1.10389}, "bytes_per_chunk": {"en": 46.996, "de": 68.284, "vi": 72.136, "tr": 83.752, "fi": 113.778, "ru": 66.433, "el": 148.271, "hi": 53.119, "ta": 81.563, "th": 134.158, "zh": 132.53, "my": 205.281}, "bpb_mean": 1.79383, "t": 565.6}
54
+ {"event": "probe", "step": 6000, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 386.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.02218, "cps": 3.015625, "mean_p": 0.372235, "boundary_entropy": 0.642723, "bpb": 2.081353}, "de": {"chunks": 371.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.018068, "cps": 2.898438, "mean_p": 0.378678, "boundary_entropy": 0.64967, "bpb": 2.281939}, "vi": {"chunks": 535.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.029021, "cps": 4.179688, "mean_p": 0.379752, "boundary_entropy": 0.650654, "bpb": 1.945007}, "tr": {"chunks": 366.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.020218, "cps": 2.859375, "mean_p": 0.37595, "boundary_entropy": 0.647683, "bpb": 2.245198}, "fi": {"chunks": 283.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.015044, "cps": 2.210938, "mean_p": 0.37811, "boundary_entropy": 0.649958, "bpb": 2.445838}, "ru": {"chunks": 744.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.039261, "cps": 5.8125, "mean_p": 0.378459, "boundary_entropy": 0.654131, "bpb": 1.425385}, "el": {"chunks": 375.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.018272, "cps": 2.929688, "mean_p": 0.372367, "boundary_entropy": 0.651811, "bpb": 1.473511}, "hi": {"chunks": 921.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.05474, "cps": 7.195312, "mean_p": 0.375501, "boundary_entropy": 0.650672, "bpb": 1.217113}, "ta": {"chunks": 707.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.037818, "cps": 5.523438, "mean_p": 0.362146, "boundary_entropy": 0.640883, "bpb": 0.992749}, "th": {"chunks": 417.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.025925, "cps": 3.257812, "mean_p": 0.370949, "boundary_entropy": 0.649942, "bpb": 1.111332}, "zh": {"chunks": 228.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.038605, "cps": 1.78125, "mean_p": 0.375965, "boundary_entropy": 0.648166, "bpb": 3.184903}, "my": {"chunks": 280.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.014777, "cps": 2.1875, "mean_p": 0.360745, "boundary_entropy": 0.639457, "bpb": 1.09482}}, "bpb": {"en": 2.08135, "de": 2.28194, "vi": 1.94501, "tr": 2.2452, "fi": 2.44584, "ru": 1.42538, "el": 1.47351, "hi": 1.21711, "ta": 0.99275, "th": 1.11133, "zh": 3.1849, "my": 1.09482}, "gini_cpc": 0.23188, "gini_cps": 0.233995, "bpic": 68.373597, "worst_lang": "hi", "cpc_parity_vs_en": {"en": 1.0, "de": 0.8146260953869533, "vi": 1.3084208484930249, "tr": 0.911522408188721, "fi": 0.6782836797893858, "ru": 1.7701111460483683, "el": 0.823810338912061, "hi": 2.4679785047232636, "ta": 1.7050250337085502, "th": 1.168831225307265, "zh": 1.74051680121559, "my": 0.6662411771075679}, "t": 566.5}
55
+ {"event": "reps_dumped", "dirs": ["/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s1/th0.30", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s1/th0.40", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s1/th0.50", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s1/th0.60", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s1/th0.70"], "t": 569.9}
56
+ {"event": "final", "run": "jitter_s1", "condition": "A", "step": 6000, "params": 22516992, "per_lang": {"en": {"cpc": 0.022180083893581566, "cps": 3.015625, "bpb": 2.1020406444164355, "bytes_per_chunk": 46.865826405649415}, "de": {"cpc": 0.018068475137583404, "cps": 2.8984375, "bpb": 2.2796363360511163, "bytes_per_chunk": 68.94897422409258}, "vi": {"cpc": 0.029020884187686467, "cps": 4.1796875, "bpb": 1.9494764901690524, "bytes_per_chunk": 70.51620712844654}, "tr": {"cpc": 0.020217643484505332, "cps": 2.859375, "bpb": 2.2458832574689267, "bytes_per_chunk": 85.72400261608895}, "fi": {"cpc": 0.01504438892137579, "cps": 2.2109375, "bpb": 2.4640121534753403, "bytes_per_chunk": 111.19575821845174}, "ru": {"cpc": 0.03926121372031662, "cps": 5.8125, "bpb": 1.4033274435589955, "bytes_per_chunk": 66.50025367833587}, "el": {"cpc": 0.018272182429469375, "cps": 2.9296875, "bpb": 1.4606392699744462, "bytes_per_chunk": 149.24224309706804}, "hi": {"cpc": 0.05473997028231798, "cps": 7.1953125, "bpb": 1.2202438124075716, "bytes_per_chunk": 53.54248366013072}, "ta": {"cpc": 0.03781759828831238, "cps": 5.5234375, "bpb": 0.9998630346897966, "bytes_per_chunk": 80.46163290362185}, "th": {"cpc": 0.025924774634752876, "cps": 3.2578125, "bpb": 1.126236773329332, "bytes_per_chunk": 134.95186615186614}, "zh": {"cpc": 0.03860480866915002, "cps": 1.78125, "bpb": 3.1923234986783178, "bytes_per_chunk": 134.3639159405433}, "my": {"cpc": 0.01477728520160439, "cps": 2.1875, "bpb": 1.114218518197882, "bytes_per_chunk": 210.81141938078005}}, "gini_cpc": 0.2318796127695224, "gini_cps": 0.2339954866678544, "bpic": 68.37359700694816, "seed": 1, "mean_boundary_entropy": 0.6504928469657898, "dechunk": "ema", "router_noise": 0.0, "transitions": [{"step": 0, "phase": 1, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}, {"step": 2400, "phase": 2, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}, {"step": 4200, "phase": 3, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}], "t": 572.1}
jitter_s1/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bfd6e2aa4bef6e6e743aa332eb060d28696a3bbef699a4c59d8b9593961885f8
3
+ size 90078288
jitter_s2/config.json ADDED
@@ -0,0 +1,108 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run": "jitter_s2",
3
+ "step": 6000,
4
+ "model_config": {
5
+ "vocab_size": 256,
6
+ "d_enc": 256,
7
+ "d_main": 512,
8
+ "n_enc": 2,
9
+ "n_dec": 2,
10
+ "n_main": 6,
11
+ "n_heads_enc": 4,
12
+ "n_heads_main": 8,
13
+ "seq_len": 1024,
14
+ "dropout": 0.0,
15
+ "target_ratio": 6.0,
16
+ "max_chunk_mult": 3.0,
17
+ "dechunk": "ema",
18
+ "router_noise": 0.0,
19
+ "rope_base": 10000.0
20
+ },
21
+ "tied_weights": {
22
+ "head.weight": "emb.weight"
23
+ },
24
+ "train_config": {
25
+ "condition": "A",
26
+ "seed": 2,
27
+ "out_dir": null,
28
+ "run_name": "jitter_s2",
29
+ "corpus": "corpus",
30
+ "langs": null,
31
+ "seq_len": 1024,
32
+ "batch_size": 32,
33
+ "grad_accum": 1,
34
+ "steps": 6000,
35
+ "lr": 0.0003,
36
+ "min_lr_frac": 0.1,
37
+ "warmup": 300,
38
+ "weight_decay": 0.1,
39
+ "grad_clip": 1.0,
40
+ "betas": [
41
+ 0.9,
42
+ 0.95
43
+ ],
44
+ "router_lr_mult": 1.0,
45
+ "d_enc": 256,
46
+ "d_main": 512,
47
+ "n_enc": 2,
48
+ "n_dec": 2,
49
+ "n_main": 6,
50
+ "n_heads_enc": 4,
51
+ "n_heads_main": 8,
52
+ "dechunk": "ema",
53
+ "target_ratio": 6.0,
54
+ "alpha": 0.03,
55
+ "alpha_parity_cap": 0.06,
56
+ "global_anchor": 0.0,
57
+ "premium_clip": 4.0,
58
+ "premium_kind": "flores_per_sent",
59
+ "mixture": null,
60
+ "doc_filters": null,
61
+ "tau1": 0.4,
62
+ "tau2": 0.7,
63
+ "freeze_scope": "router",
64
+ "ckpt_every": 2000,
65
+ "eval_every": 1000,
66
+ "log_every": 200,
67
+ "probe_sents": 128,
68
+ "eval_batches": 8,
69
+ "keep_ckpt": false,
70
+ "amp": "bf16",
71
+ "compile": false,
72
+ "cpc_decay": 0.9,
73
+ "device": "cuda",
74
+ "log_spaced_probe": true,
75
+ "probe_bpb_batches": 4,
76
+ "dump_reps": true,
77
+ "dump_reps_thresholds": [
78
+ 0.3,
79
+ 0.4,
80
+ 0.5,
81
+ 0.6,
82
+ 0.7
83
+ ],
84
+ "router_noise": 0.0,
85
+ "genreg_target_entropy": null,
86
+ "genreg_eta": 0.05
87
+ },
88
+ "probe_agg": {
89
+ "gini_cpc": 0.262783775060067,
90
+ "gini_cps": 0.27856758212170174,
91
+ "bpic": 62.00016155088853,
92
+ "worst_lang": "ta",
93
+ "cpc_parity_vs_en": {
94
+ "en": 1.0,
95
+ "de": 0.9160887870174738,
96
+ "vi": 1.4180378442947907,
97
+ "tr": 1.677218187651982,
98
+ "fi": 0.9645182707718549,
99
+ "ru": 2.3095880536686693,
100
+ "el": 2.042357873522286,
101
+ "hi": 3.6796580569694286,
102
+ "ta": 3.9968877786187003,
103
+ "th": 2.9373511729574933,
104
+ "zh": 2.9717424040463722,
105
+ "my": 3.1892094376995943
106
+ }
107
+ }
108
+ }
jitter_s2/final.json ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "event": "final",
3
+ "run": "jitter_s2",
4
+ "condition": "A",
5
+ "step": 6000,
6
+ "params": 22516992,
7
+ "per_lang": {
8
+ "en": {
9
+ "cpc": 0.013503418950755616,
10
+ "cps": 1.8359375,
11
+ "bpb": 2.091184529801444,
12
+ "bytes_per_chunk": 119.15636363636364
13
+ },
14
+ "de": {
15
+ "cpc": 0.01237033068718648,
16
+ "cps": 1.984375,
17
+ "bpb": 2.2734026958836058,
18
+ "bytes_per_chunk": 135.02137522534125
19
+ },
20
+ "vi": {
21
+ "cpc": 0.01914835909953892,
22
+ "cps": 2.7578125,
23
+ "bpb": 1.9494414809506864,
24
+ "bytes_per_chunk": 86.68783068783068
25
+ },
26
+ "tr": {
27
+ "cpc": 0.022648179859691763,
28
+ "cps": 3.203125,
29
+ "bpb": 2.240100760041376,
30
+ "bytes_per_chunk": 89.34696659850034
31
+ },
32
+ "fi": {
33
+ "cpc": 0.013024294295890703,
34
+ "cps": 1.9140625,
35
+ "bpb": 2.4580483003424547,
36
+ "bytes_per_chunk": 132.8656867714141
37
+ },
38
+ "ru": {
39
+ "cpc": 0.031187335092348285,
40
+ "cps": 4.6171875,
41
+ "bpb": 1.4135231609180474,
42
+ "bytes_per_chunk": 73.35777249195466
43
+ },
44
+ "el": {
45
+ "cpc": 0.02757881401354578,
46
+ "cps": 4.421875,
47
+ "bpb": 1.4792314436970544,
48
+ "bytes_per_chunk": 67.52807831014941
49
+ },
50
+ "hi": {
51
+ "cpc": 0.04968796433878157,
52
+ "cps": 6.53125,
53
+ "bpb": 1.2194370256690006,
54
+ "bytes_per_chunk": 54.50545794781162
55
+ },
56
+ "ta": {
57
+ "cpc": 0.053971650173843276,
58
+ "cps": 7.8828125,
59
+ "bpb": 0.9995220038420514,
60
+ "bytes_per_chunk": 63.92197025115826
61
+ },
62
+ "th": {
63
+ "cpc": 0.039664283493938454,
64
+ "cps": 4.984375,
65
+ "bpb": 1.1435730867396772,
66
+ "bytes_per_chunk": 81.6393646838991
67
+ },
68
+ "zh": {
69
+ "cpc": 0.040128682695563835,
70
+ "cps": 1.8515625,
71
+ "bpb": 3.1307407679749546,
72
+ "bytes_per_chunk": 120.83152800184374
73
+ },
74
+ "my": {
75
+ "cpc": 0.04306523115896137,
76
+ "cps": 6.375,
77
+ "bpb": 1.1354728113731958,
78
+ "bytes_per_chunk": 50.768664665440106
79
+ }
80
+ },
81
+ "gini_cpc": 0.262783775060067,
82
+ "gini_cps": 0.27856758212170174,
83
+ "bpic": 62.00016155088853,
84
+ "seed": 2,
85
+ "mean_boundary_entropy": 0.6519113779067993,
86
+ "dechunk": "ema",
87
+ "router_noise": 0.0,
88
+ "transitions": [
89
+ {
90
+ "step": 0,
91
+ "phase": 1,
92
+ "objective": "global",
93
+ "router_trainable": true,
94
+ "encoder_trainable": true,
95
+ "frozen_router_params": 0
96
+ },
97
+ {
98
+ "step": 2400,
99
+ "phase": 2,
100
+ "objective": "global",
101
+ "router_trainable": true,
102
+ "encoder_trainable": true,
103
+ "frozen_router_params": 0
104
+ },
105
+ {
106
+ "step": 4200,
107
+ "phase": 3,
108
+ "objective": "global",
109
+ "router_trainable": true,
110
+ "encoder_trainable": true,
111
+ "frozen_router_params": 0
112
+ }
113
+ ]
114
+ }
jitter_s2/log.jsonl ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {"event": "start", "run": "jitter_s2", "condition": "A", "params": 22516992, "langs": ["en", "de", "vi", "tr", "fi", "ru", "el", "hi", "ta", "th", "zh", "my"], "premium": {"en": 1.0, "de": 1.184, "vi": 1.39, "tr": 1.12, "fi": 1.104, "ru": 1.971, "el": 2.174, "hi": 2.552, "ta": 3.192, "th": 2.762, "zh": 0.915, "my": 3.524}, "schedule": {"tau1": 0.4, "tau2": 0.7, "p1_objective": "global", "p2_objective": "global", "p3_objective": "global", "freeze_in_p3": false, "freeze_scope": "router"}, "corpus": "/local/scratch/sas245/beetle-hnet/scratch/data/tier1_corpus", "premium_kind": "flores_per_sent", "doc_filtered": {"my": 37511}, "weights": {"en": 0.2946, "de": 0.0909, "vi": 0.0576, "tr": 0.0612, "fi": 0.0424, "ru": 0.1476, "el": 0.0522, "hi": 0.042, "ta": 0.0313, "th": 0.0542, "zh": 0.1038, "my": 0.0222}, "t": 1.5}
2
+ {"event": "probe_schedule", "n": 13, "steps": [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 2000, 4000, 6000], "t": 1.5}
3
+ {"event": "probe", "step": 0, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 10933.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.628225, "cps": 85.414062, "mean_p": 0.513249, "boundary_entropy": 0.686295, "bpb": 8.12598}, "de": {"chunks": 16287.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.793211, "cps": 127.242188, "mean_p": 0.524631, "boundary_entropy": 0.686391, "bpb": 8.12701}, "vi": {"chunks": 5099.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.276593, "cps": 39.835938, "mean_p": 0.481643, "boundary_entropy": 0.68728, "bpb": 7.869214}, "tr": {"chunks": 8187.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.452245, "cps": 63.960938, "mean_p": 0.498002, "boundary_entropy": 0.686826, "bpb": 8.097536}, "fi": {"chunks": 13540.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.719792, "cps": 105.78125, "mean_p": 0.517169, "boundary_entropy": 0.686661, "bpb": 8.065117}, "ru": {"chunks": 33477.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 1.766596, "cps": 261.539062, "mean_p": 0.530376, "boundary_entropy": 0.688543, "bpb": 8.240733}, "el": {"chunks": 20696.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.00843, "cps": 161.6875, "mean_p": 0.503648, "boundary_entropy": 0.690112, "bpb": 8.286272}, "hi": {"chunks": 875.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.052006, "cps": 6.835938, "mean_p": 0.461978, "boundary_entropy": 0.68743, "bpb": 8.022688}, "ta": {"chunks": 1261.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.067451, "cps": 9.851562, "mean_p": 0.477518, "boundary_entropy": 0.690053, "bpb": 8.091383}, "th": {"chunks": 3630.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.225676, "cps": 28.359375, "mean_p": 0.484845, "boundary_entropy": 0.690398, "bpb": 7.721507}, "zh": {"chunks": 11722.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.984761, "cps": 91.578125, "mean_p": 0.520316, "boundary_entropy": 0.68562, "bpb": 8.130734}, "my": {"chunks": 1628.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.085919, "cps": 12.71875, "mean_p": 0.474628, "boundary_entropy": 0.689685, "bpb": 7.8029}}, "bpb": {"en": 8.12598, "de": 8.12701, "vi": 7.86921, "tr": 8.09754, "fi": 8.06512, "ru": 8.24073, "el": 8.28627, "hi": 8.02269, "ta": 8.09138, "th": 7.72151, "zh": 8.13073, "my": 7.8029}, "gini_cpc": 0.489538, "gini_cps": 0.463551, "bpic": 3.013947, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 1.2626223171148012, "vi": 0.440277652395235, "tr": 0.7198781817505053, "fi": 1.1457544508645294, "ru": 2.8120438593717987, "el": 1.6052044042347315, "hi": 0.08278235025738971, "ta": 0.10736788276924242, "th": 0.3592281253258146, "zh": 3.159315851384177, "my": 0.13676526035979295}, "t": 3.3}
4
+ {"event": "train", "step": 0, "phase": 1, "objective": "global", "lm_loss": 5.64451, "bpb": 8.14331, "chunk_loss": 0.017745, "F": 0.00098, "G": 0.5091, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.69066, "router_noise": 0.0, "grad_norm": 4.2033, "lr": 1e-06, "router_trainable": true, "sec_per_step": 0.001, "alpha": 0.03, "t": 3.8}
5
+ {"event": "probe", "step": 1, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 10973.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.630523, "cps": 85.726562, "mean_p": 0.513415, "boundary_entropy": 0.686291, "bpb": 8.112407}, "de": {"chunks": 16325.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.795062, "cps": 127.539062, "mean_p": 0.524793, "boundary_entropy": 0.686378, "bpb": 8.114037}, "vi": {"chunks": 5118.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.277624, "cps": 39.984375, "mean_p": 0.481707, "boundary_entropy": 0.687285, "bpb": 7.860188}, "tr": {"chunks": 8241.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.455228, "cps": 64.382812, "mean_p": 0.49822, "boundary_entropy": 0.68683, "bpb": 8.088402}, "fi": {"chunks": 13565.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.721121, "cps": 105.976562, "mean_p": 0.517278, "boundary_entropy": 0.686656, "bpb": 8.053366}, "ru": {"chunks": 33499.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 1.767757, "cps": 261.710938, "mean_p": 0.530674, "boundary_entropy": 0.688508, "bpb": 8.214279}, "el": {"chunks": 20725.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.009843, "cps": 161.914062, "mean_p": 0.503684, "boundary_entropy": 0.690112, "bpb": 8.285335}, "hi": {"chunks": 878.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.052184, "cps": 6.859375, "mean_p": 0.462055, "boundary_entropy": 0.687442, "bpb": 8.01674}, "ta": {"chunks": 1287.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.068842, "cps": 10.054688, "mean_p": 0.477649, "boundary_entropy": 0.690066, "bpb": 8.087633}, "th": {"chunks": 3656.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.227293, "cps": 28.5625, "mean_p": 0.484903, "boundary_entropy": 0.690401, "bpb": 7.719771}, "zh": {"chunks": 11731.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.986285, "cps": 91.648438, "mean_p": 0.520388, "boundary_entropy": 0.685615, "bpb": 8.128736}, "my": {"chunks": 1618.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.085392, "cps": 12.640625, "mean_p": 0.474603, "boundary_entropy": 0.689682, "bpb": 7.801052}}, "bpb": {"en": 8.11241, "de": 8.11404, "vi": 7.86019, "tr": 8.0884, "fi": 8.05337, "ru": 8.21428, "el": 8.28534, "hi": 8.01674, "ta": 8.08763, "th": 7.71977, "zh": 8.12874, "my": 7.80105}, "gini_cpc": 0.488839, "gini_cps": 0.462872, "bpic": 3.007311, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 1.2609548133156332, "vi": 0.4403072946933804, "tr": 0.7219848819260126, "fi": 1.1436856070130392, "ru": 2.8036343320027304, "el": 1.6015940070465446, "hi": 0.0827633733970008, "ta": 0.1091821943169754, "th": 0.3604822315015682, "zh": 3.1502160013546776, "my": 0.13542968932840913}, "t": 5.1}
6
+ {"event": "probe", "step": 2, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 10974.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.630581, "cps": 85.734375, "mean_p": 0.513418, "boundary_entropy": 0.686297, "bpb": 8.074849}, "de": {"chunks": 16315.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.794575, "cps": 127.460938, "mean_p": 0.524791, "boundary_entropy": 0.686383, "bpb": 8.078929}, "vi": {"chunks": 5118.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.277624, "cps": 39.984375, "mean_p": 0.481666, "boundary_entropy": 0.687281, "bpb": 7.841203}, "tr": {"chunks": 8290.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.457935, "cps": 64.765625, "mean_p": 0.498428, "boundary_entropy": 0.686835, "bpb": 8.062411}, "fi": {"chunks": 13551.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.720376, "cps": 105.867188, "mean_p": 0.517218, "boundary_entropy": 0.686664, "bpb": 8.018631}, "ru": {"chunks": 33514.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 1.768549, "cps": 261.828125, "mean_p": 0.530885, "boundary_entropy": 0.688481, "bpb": 8.141092}, "el": {"chunks": 20782.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.01262, "cps": 162.359375, "mean_p": 0.50375, "boundary_entropy": 0.690111, "bpb": 8.284844}, "hi": {"chunks": 903.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.05367, "cps": 7.054688, "mean_p": 0.462219, "boundary_entropy": 0.687468, "bpb": 7.986524}, "ta": {"chunks": 1324.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.070821, "cps": 10.34375, "mean_p": 0.477855, "boundary_entropy": 0.690085, "bpb": 8.071872}, "th": {"chunks": 3686.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.229158, "cps": 28.796875, "mean_p": 0.484947, "boundary_entropy": 0.690403, "bpb": 7.703635}, "zh": {"chunks": 11732.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.986454, "cps": 91.65625, "mean_p": 0.52037, "boundary_entropy": 0.685615, "bpb": 8.122673}, "my": {"chunks": 1610.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.084969, "cps": 12.578125, "mean_p": 0.474585, "boundary_entropy": 0.68968, "bpb": 7.800115}}, "bpb": {"en": 8.07485, "de": 8.07893, "vi": 7.8412, "tr": 8.06241, "fi": 8.01863, "ru": 8.14109, "el": 8.28484, "hi": 7.98652, "ta": 8.07187, "th": 7.70364, "zh": 8.12267, "my": 7.80012}, "gini_cpc": 0.488024, "gini_cps": 0.462073, "bpic": 3.003005, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 1.2600675726178374, "vi": 0.44026717192185744, "tr": 0.7262115359751005, "fi": 1.1424011357081152, "ru": 2.804634134026552, "el": 1.6058525277498072, "hi": 0.08511220496589128, "ta": 0.11231084116104524, "th": 0.3634071183823363, "zh": 3.1501974528224856, "my": 0.1347477941124701}, "t": 6.7}
7
+ {"event": "probe", "step": 4, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 10958.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.629662, "cps": 85.609375, "mean_p": 0.513328, "boundary_entropy": 0.686323, "bpb": 7.950483}, "de": {"chunks": 16341.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.795841, "cps": 127.664062, "mean_p": 0.524718, "boundary_entropy": 0.686408, "bpb": 7.966392}, "vi": {"chunks": 5092.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.276214, "cps": 39.78125, "mean_p": 0.481544, "boundary_entropy": 0.687268, "bpb": 7.774413}, "tr": {"chunks": 8458.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.467215, "cps": 66.078125, "mean_p": 0.499042, "boundary_entropy": 0.686851, "bpb": 7.975067}, "fi": {"chunks": 13519.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.718675, "cps": 105.617188, "mean_p": 0.516944, "boundary_entropy": 0.686695, "bpb": 7.903791}, "ru": {"chunks": 33540.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 1.769921, "cps": 262.03125, "mean_p": 0.531365, "boundary_entropy": 0.688418, "bpb": 7.887699}, "el": {"chunks": 20880.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.017395, "cps": 163.125, "mean_p": 0.503844, "boundary_entropy": 0.690112, "bpb": 8.253629}, "hi": {"chunks": 958.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.056939, "cps": 7.484375, "mean_p": 0.462719, "boundary_entropy": 0.687546, "bpb": 7.88726}, "ta": {"chunks": 1398.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.074779, "cps": 10.921875, "mean_p": 0.478275, "boundary_entropy": 0.690123, "bpb": 7.984734}, "th": {"chunks": 3708.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.230525, "cps": 28.96875, "mean_p": 0.484792, "boundary_entropy": 0.690388, "bpb": 7.620259}, "zh": {"chunks": 11691.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.979512, "cps": 91.335938, "mean_p": 0.520229, "boundary_entropy": 0.685623, "bpb": 8.09962}, "my": {"chunks": 1582.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.083492, "cps": 12.359375, "mean_p": 0.474452, "boundary_entropy": 0.689665, "bpb": 7.794292}}, "bpb": {"en": 7.95048, "de": 7.96639, "vi": 7.77441, "tr": 7.97507, "fi": 7.90379, "ru": 7.8877, "el": 8.25363, "hi": 7.88726, "ta": 7.98473, "th": 7.62026, "zh": 8.09962, "my": 7.79429}, "gini_cpc": 0.486186, "gini_cps": 0.460751, "bpic": 2.995364, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 1.2639184309070375, "vi": 0.43867014390770237, "tr": 0.7420103335401932, "fi": 1.1413675175727382, "ru": 2.8109082363411075, "el": 1.615780911923772, "hi": 0.09042806967681818, "ta": 0.1187611860032478, "th": 0.3661099096939097, "zh": 3.1437720027835248, "my": 0.13259768052143592}, "t": 8.3}
8
+ {"event": "probe", "step": 8, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 10564.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.607022, "cps": 82.53125, "mean_p": 0.511303, "boundary_entropy": 0.686489, "bpb": 7.511635}, "de": {"chunks": 16085.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.783373, "cps": 125.664062, "mean_p": 0.522715, "boundary_entropy": 0.686663, "bpb": 7.567076}, "vi": {"chunks": 4846.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.26287, "cps": 37.859375, "mean_p": 0.480592, "boundary_entropy": 0.687177, "bpb": 7.546083}, "tr": {"chunks": 8897.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.491466, "cps": 69.507812, "mean_p": 0.500753, "boundary_entropy": 0.686901, "bpb": 7.649824}, "fi": {"chunks": 13006.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.691404, "cps": 101.609375, "mean_p": 0.514458, "boundary_entropy": 0.68689, "bpb": 7.498431}, "ru": {"chunks": 33618.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 1.774037, "cps": 262.640625, "mean_p": 0.53289, "boundary_entropy": 0.688209, "bpb": 7.323119}, "el": {"chunks": 20734.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.010281, "cps": 161.984375, "mean_p": 0.503587, "boundary_entropy": 0.69012, "bpb": 8.046873}, "hi": {"chunks": 1190.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.070728, "cps": 9.296875, "mean_p": 0.46461, "boundary_entropy": 0.687837, "bpb": 7.578698}, "ta": {"chunks": 1773.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.094838, "cps": 13.851562, "mean_p": 0.479794, "boundary_entropy": 0.690254, "bpb": 7.732736}, "th": {"chunks": 3813.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.237053, "cps": 29.789062, "mean_p": 0.484558, "boundary_entropy": 0.69036, "bpb": 7.38321}, "zh": {"chunks": 11553.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.956146, "cps": 90.257812, "mean_p": 0.519491, "boundary_entropy": 0.685665, "bpb": 8.031371}, "my": {"chunks": 1309.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.069084, "cps": 10.226562, "mean_p": 0.473199, "boundary_entropy": 0.689534, "bpb": 7.664766}}, "bpb": {"en": 7.51163, "de": 7.56708, "vi": 7.54608, "tr": 7.64982, "fi": 7.49843, "ru": 7.32312, "el": 8.04687, "hi": 7.5787, "ta": 7.73274, "th": 7.38321, "zh": 8.03137, "my": 7.66477}, "gini_cpc": 0.482797, "gini_cps": 0.457572, "bpic": 3.012694, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 1.290518948883884, "vi": 0.43304795844708327, "tr": 0.8096340541396518, "fi": 1.1390101492051576, "ru": 2.922526018069033, "el": 1.6643243864351969, "hi": 0.1165165498007475, "ta": 0.156235238157639, "th": 0.3905183697879156, "zh": 3.2225306624431505, "my": 0.11380779213834358}, "t": 10.0}
9
+ {"event": "probe", "step": 16, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 6431.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.369534, "cps": 50.242188, "mean_p": 0.495391, "boundary_entropy": 0.686738, "bpb": 6.909054}, "de": {"chunks": 11557.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.56285, "cps": 90.289062, "mean_p": 0.505708, "boundary_entropy": 0.687738, "bpb": 6.988178}, "vi": {"chunks": 3303.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.17917, "cps": 25.804688, "mean_p": 0.474137, "boundary_entropy": 0.686527, "bpb": 7.205538}, "tr": {"chunks": 8697.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.480418, "cps": 67.945312, "mean_p": 0.499746, "boundary_entropy": 0.687046, "bpb": 7.180256}, "fi": {"chunks": 8100.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.430599, "cps": 63.28125, "mean_p": 0.497915, "boundary_entropy": 0.687331, "bpb": 6.96291}, "ru": {"chunks": 32668.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 1.723905, "cps": 255.21875, "mean_p": 0.52748, "boundary_entropy": 0.688774, "bpb": 6.461576}, "el": {"chunks": 17688.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.861862, "cps": 138.1875, "mean_p": 0.499546, "boundary_entropy": 0.690143, "bpb": 7.466983}, "hi": {"chunks": 2156.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.128143, "cps": 16.84375, "mean_p": 0.469561, "boundary_entropy": 0.68855, "bpb": 6.649744}, "ta": {"chunks": 3273.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.175074, "cps": 25.570312, "mean_p": 0.483503, "boundary_entropy": 0.690536, "bpb": 6.839187}, "th": {"chunks": 2779.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.17277, "cps": 21.710938, "mean_p": 0.480699, "boundary_entropy": 0.690044, "bpb": 6.667273}, "zh": {"chunks": 10446.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.76871, "cps": 81.609375, "mean_p": 0.514169, "boundary_entropy": 0.685944, "bpb": 7.815019}, "my": {"chunks": 772.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.040743, "cps": 6.03125, "mean_p": 0.469418, "boundary_entropy": 0.689109, "bpb": 7.320966}}, "bpb": {"en": 6.90905, "de": 6.98818, "vi": 7.20554, "tr": 7.18026, "fi": 6.96291, "ru": 6.46158, "el": 7.46698, "hi": 6.64974, "ta": 6.83919, "th": 6.66727, "zh": 7.81502, "my": 7.32097}, "gini_cpc": 0.496787, "gini_cps": 0.468006, "bpic": 3.55781, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 1.5231347154694974, "vi": 0.48485406643142664, "tr": 1.3000633918148687, "fi": 1.1652490192049516, "ru": 4.665078361777489, "el": 2.3322950740787403, "hi": 0.3467682240300107, "ta": 0.4737684612956207, "th": 0.46753388484351743, "zh": 4.786325053211452, "my": 0.11025531513019436}, "t": 12.0}
10
+ {"event": "probe", "step": 32, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 546.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.031374, "cps": 4.265625, "mean_p": 0.454282, "boundary_entropy": 0.682264, "bpb": 6.444821}, "de": {"chunks": 480.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.023377, "cps": 3.75, "mean_p": 0.457648, "boundary_entropy": 0.684041, "bpb": 6.531547}, "vi": {"chunks": 739.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.040087, "cps": 5.773438, "mean_p": 0.453877, "boundary_entropy": 0.683603, "bpb": 6.918218}, "tr": {"chunks": 1827.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.100922, "cps": 14.273438, "mean_p": 0.470851, "boundary_entropy": 0.685377, "bpb": 6.768646}, "fi": {"chunks": 513.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.027271, "cps": 4.007812, "mean_p": 0.454461, "boundary_entropy": 0.683028, "bpb": 6.550628}, "ru": {"chunks": 10752.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.567388, "cps": 84.0, "mean_p": 0.494644, "boundary_entropy": 0.690088, "bpb": 5.97938}, "el": {"chunks": 15611.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.760659, "cps": 121.960938, "mean_p": 0.497568, "boundary_entropy": 0.690242, "bpb": 6.440687}, "hi": {"chunks": 355.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.0211, "cps": 2.773438, "mean_p": 0.457841, "boundary_entropy": 0.68691, "bpb": 6.323972}, "ta": {"chunks": 296.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.015833, "cps": 2.3125, "mean_p": 0.449685, "boundary_entropy": 0.685801, "bpb": 6.390968}, "th": {"chunks": 244.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.015169, "cps": 1.90625, "mean_p": 0.444814, "boundary_entropy": 0.684494, "bpb": 6.253931}, "zh": {"chunks": 4991.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.845073, "cps": 38.992188, "mean_p": 0.490494, "boundary_entropy": 0.685902, "bpb": 7.462509}, "my": {"chunks": 361.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.019052, "cps": 2.820312, "mean_p": 0.46235, "boundary_entropy": 0.688224, "bpb": 6.498228}}, "bpb": {"en": 6.44482, "de": 6.53155, "vi": 6.91822, "tr": 6.76865, "fi": 6.55063, "ru": 5.97938, "el": 6.44069, "hi": 6.32397, "ta": 6.39097, "th": 6.25393, "zh": 7.46251, "my": 6.49823}, "gini_cpc": 0.679691, "gini_cps": 0.689394, "bpic": 10.452976, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 0.7451098553226835, "vi": 1.2777114125364737, "tr": 3.216766026880373, "fi": 0.8692345079831125, "ru": 18.084708747716665, "el": 24.24495353109591, "hi": 0.6725193083291152, "ta": 0.5046586470496607, "th": 0.4835041862085692, "zh": 26.935534918856966, "my": 0.6072608782393457}, "t": 14.9}
11
+ {"event": "probe", "step": 64, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 207.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.011895, "cps": 1.617188, "mean_p": 0.357804, "boundary_entropy": 0.643436, "bpb": 5.94692}, "de": {"chunks": 159.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.007744, "cps": 1.242188, "mean_p": 0.361607, "boundary_entropy": 0.647364, "bpb": 6.083577}, "vi": {"chunks": 225.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.012205, "cps": 1.757812, "mean_p": 0.372776, "boundary_entropy": 0.653829, "bpb": 6.594595}, "tr": {"chunks": 162.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.008949, "cps": 1.265625, "mean_p": 0.366458, "boundary_entropy": 0.64941, "bpb": 6.436281}, "fi": {"chunks": 174.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.00925, "cps": 1.359375, "mean_p": 0.354408, "boundary_entropy": 0.642381, "bpb": 6.053196}, "ru": {"chunks": 132.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.006966, "cps": 1.03125, "mean_p": 0.410378, "boundary_entropy": 0.673523, "bpb": 5.788062}, "el": {"chunks": 10920.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.532086, "cps": 85.3125, "mean_p": 0.495321, "boundary_entropy": 0.6904, "bpb": 5.750818}, "hi": {"chunks": 160.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.00951, "cps": 1.25, "mean_p": 0.466272, "boundary_entropy": 0.688437, "bpb": 5.804437}, "ta": {"chunks": 192.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.01027, "cps": 1.5, "mean_p": 0.460835, "boundary_entropy": 0.687975, "bpb": 5.786206}, "th": {"chunks": 133.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.008269, "cps": 1.039062, "mean_p": 0.417771, "boundary_entropy": 0.676863, "bpb": 5.474193}, "zh": {"chunks": 149.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.025229, "cps": 1.164062, "mean_p": 0.410025, "boundary_entropy": 0.668755, "bpb": 7.191683}, "my": {"chunks": 180.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.0095, "cps": 1.40625, "mean_p": 0.424851, "boundary_entropy": 0.679574, "bpb": 5.903939}}, "bpb": {"en": 5.94692, "de": 6.08358, "vi": 6.5946, "tr": 6.43628, "fi": 6.0532, "ru": 5.78806, "el": 5.75082, "hi": 5.80444, "ta": 5.78621, "th": 5.47419, "zh": 7.19168, "my": 5.90394}, "gini_cpc": 0.764345, "gini_cps": 0.782583, "bpic": 29.999296, "worst_lang": "el", "cpc_parity_vs_en": {"en": 1.0, "de": 0.6510262377212505, "vi": 1.026108182686525, "tr": 0.7523470767516314, "fi": 0.7776624685371417, "ru": 0.5856234943214409, "el": 44.73377695014501, "hi": 0.7995003983892154, "ta": 0.863434770980383, "th": 0.6951593211787018, "zh": 2.1210289707838257, "my": 0.7986617837376435}, "t": 19.5}
12
+ {"event": "probe", "step": 128, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2951.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.169568, "cps": 23.054688, "mean_p": 0.475111, "boundary_entropy": 0.684644, "bpb": 4.546814}, "de": {"chunks": 4472.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.217796, "cps": 34.9375, "mean_p": 0.479347, "boundary_entropy": 0.686126, "bpb": 4.773277}, "vi": {"chunks": 1209.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.065582, "cps": 9.445312, "mean_p": 0.46267, "boundary_entropy": 0.685101, "bpb": 5.290019}, "tr": {"chunks": 4751.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.262443, "cps": 37.117188, "mean_p": 0.480246, "boundary_entropy": 0.685829, "bpb": 5.212214}, "fi": {"chunks": 1847.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.098187, "cps": 14.429688, "mean_p": 0.469739, "boundary_entropy": 0.684909, "bpb": 4.813523}, "ru": {"chunks": 1792.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.094565, "cps": 14.0, "mean_p": 0.467962, "boundary_entropy": 0.687257, "bpb": 3.879847}, "el": {"chunks": 252.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.012279, "cps": 1.96875, "mean_p": 0.454998, "boundary_entropy": 0.686074, "bpb": 4.633605}, "hi": {"chunks": 233.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.013848, "cps": 1.820312, "mean_p": 0.478027, "boundary_entropy": 0.689579, "bpb": 3.718622}, "ta": {"chunks": 13734.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.734635, "cps": 107.296875, "mean_p": 0.468733, "boundary_entropy": 0.687419, "bpb": 3.773613}, "th": {"chunks": 2057.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.127883, "cps": 16.070312, "mean_p": 0.437202, "boundary_entropy": 0.679734, "bpb": 3.607527}, "zh": {"chunks": 335.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.056722, "cps": 2.617188, "mean_p": 0.47456, "boundary_entropy": 0.685611, "bpb": 6.43275}, "my": {"chunks": 1035.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.054623, "cps": 8.085938, "mean_p": 0.48045, "boundary_entropy": 0.690493, "bpb": 4.309861}}, "bpb": {"en": 4.54681, "de": 4.77328, "vi": 5.29002, "tr": 5.21221, "fi": 4.81352, "ru": 3.87985, "el": 4.6336, "hi": 3.71862, "ta": 3.77361, "th": 3.60753, "zh": 6.43275, "my": 4.30986}, "gini_cpc": 0.527577, "gini_cps": 0.5532, "bpic": 11.07018, "worst_lang": "ta", "cpc_parity_vs_en": {"en": 1.0, "de": 1.2844118343073394, "vi": 0.386756899165899, "tr": 1.5477092909725707, "fi": 0.5790418091925529, "ru": 0.5576782433216593, "el": 0.07241267754357077, "hi": 0.08166872186273774, "ta": 4.332379421067414, "th": 0.754168062816912, "zh": 0.334507820074652, "my": 0.3221305279883871}, "t": 28.3}
13
+ {"event": "train", "step": 200, "phase": 1, "objective": "global", "lm_loss": 2.5608, "bpb": 3.69445, "chunk_loss": 0.022629, "F": 0.00098, "G": 0.37261, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65348, "router_noise": 0.0, "grad_norm": 1.8307, "lr": 0.000201, "router_trainable": true, "sec_per_step": 0.159, "alpha": 0.03, "t": 35.6}
14
+ {"event": "probe", "step": 256, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 138.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.00793, "cps": 1.078125, "mean_p": 0.325811, "boundary_entropy": 0.617501, "bpb": 3.688162}, "de": {"chunks": 491.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.023913, "cps": 3.835938, "mean_p": 0.323568, "boundary_entropy": 0.615002, "bpb": 3.901921}, "vi": {"chunks": 246.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.013344, "cps": 1.921875, "mean_p": 0.380783, "boundary_entropy": 0.654541, "bpb": 3.278447}, "tr": {"chunks": 404.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.022317, "cps": 3.15625, "mean_p": 0.339516, "boundary_entropy": 0.622591, "bpb": 3.931836}, "fi": {"chunks": 190.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.0101, "cps": 1.484375, "mean_p": 0.318453, "boundary_entropy": 0.610087, "bpb": 3.947919}, "ru": {"chunks": 129.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.006807, "cps": 1.007812, "mean_p": 0.324843, "boundary_entropy": 0.620993, "bpb": 2.670494}, "el": {"chunks": 131.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.006383, "cps": 1.023438, "mean_p": 0.358232, "boundary_entropy": 0.647862, "bpb": 2.78849}, "hi": {"chunks": 162.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.009629, "cps": 1.265625, "mean_p": 0.37343, "boundary_entropy": 0.653935, "bpb": 2.17733}, "ta": {"chunks": 133.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.007114, "cps": 1.039062, "mean_p": 0.334886, "boundary_entropy": 0.631485, "bpb": 2.044377}, "th": {"chunks": 145.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.009015, "cps": 1.132812, "mean_p": 0.368819, "boundary_entropy": 0.652689, "bpb": 2.125364}, "zh": {"chunks": 183.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.030985, "cps": 1.429688, "mean_p": 0.361058, "boundary_entropy": 0.639788, "bpb": 5.05289}, "my": {"chunks": 130.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.006861, "cps": 1.015625, "mean_p": 0.347609, "boundary_entropy": 0.641585, "bpb": 2.263279}}, "bpb": {"en": 3.68816, "de": 3.90192, "vi": 3.27845, "tr": 3.93184, "fi": 3.94792, "ru": 2.67049, "el": 2.78849, "hi": 2.17733, "ta": 2.04438, "th": 2.12536, "zh": 5.05289, "my": 2.26328}, "gini_cpc": 0.308209, "gini_cps": 0.258192, "bpic": 154.625705, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 3.0156026671805085, "vi": 1.6828174196059007, "tr": 2.814335361182028, "fi": 1.2737574915694563, "ru": 0.8584708041757485, "el": 0.8049621951193676, "hi": 1.214241230053621, "ta": 0.8971626917218042, "th": 1.1368206944087793, "zh": 3.9075332381218804, "my": 0.8652169323824471}, "t": 42.2}
15
+ {"event": "train", "step": 400, "phase": 1, "objective": "global", "lm_loss": 2.24422, "bpb": 3.23773, "chunk_loss": 0.022311, "F": 0.00177, "G": 0.38255, "chunks": 1.81, "bytes_per_chunk": 564.966, "boundary_entropy": 0.65676, "router_noise": 0.0, "grad_norm": 1.2833, "lr": 0.0002998, "router_trainable": true, "sec_per_step": 0.107, "alpha": 0.03, "t": 57.1}
16
+ {"event": "train", "step": 600, "phase": 1, "objective": "global", "lm_loss": 1.93344, "bpb": 2.78937, "chunk_loss": 0.025269, "F": 0.04996, "G": 0.35909, "chunks": 51.16, "bytes_per_chunk": 20.017, "boundary_entropy": 0.64475, "router_noise": 0.0, "grad_norm": 1.1866, "lr": 0.0002982, "router_trainable": true, "sec_per_step": 0.095, "alpha": 0.03, "t": 76.2}
17
+ {"event": "train", "step": 800, "phase": 1, "objective": "global", "lm_loss": 1.87191, "bpb": 2.7006, "chunk_loss": 0.032523, "F": 0.22223, "G": 0.35836, "chunks": 227.56, "bytes_per_chunk": 4.5, "boundary_entropy": 0.6453, "router_noise": 0.0, "grad_norm": 0.8789, "lr": 0.0002949, "router_trainable": true, "sec_per_step": 0.098, "alpha": 0.03, "t": 95.8}
18
+ {"event": "eval", "step": 1000, "phase": 1, "bpb": {"en": 2.62173, "de": 2.95945, "vi": 2.32517, "tr": 2.95091, "fi": 3.18195, "ru": 1.91645, "el": 2.03022, "hi": 1.53319, "ta": 1.31682, "th": 1.58764, "zh": 3.65126, "my": 1.40758}, "bytes_per_chunk": {"en": 179.06, "de": 148.861, "vi": 166.124, "tr": 35.724, "fi": 213.125, "ru": 63.091, "el": 261.1, "hi": 164.87, "ta": 86.573, "th": 337.814, "zh": 201.649, "my": 47.81}, "bpb_mean": 2.2902, "t": 114.5}
19
+ {"event": "train", "step": 1000, "phase": 1, "objective": "global", "lm_loss": 1.57618, "bpb": 2.27395, "chunk_loss": 0.023139, "F": 0.0014, "G": 0.35886, "chunks": 1.44, "bytes_per_chunk": 712.348, "boundary_entropy": 0.64447, "router_noise": 0.0, "grad_norm": 0.7905, "lr": 0.0002901, "router_trainable": true, "sec_per_step": 0.094, "alpha": 0.03, "t": 114.6}
20
+ {"event": "train", "step": 1200, "phase": 1, "objective": "global", "lm_loss": 1.72909, "bpb": 2.49455, "chunk_loss": 0.023445, "F": 0.00098, "G": 0.34983, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.63892, "router_noise": 0.0, "grad_norm": 0.972, "lr": 0.0002837, "router_trainable": true, "sec_per_step": 0.096, "alpha": 0.03, "t": 133.8}
21
+ {"event": "train", "step": 1400, "phase": 1, "objective": "global", "lm_loss": 1.72227, "bpb": 2.48472, "chunk_loss": 0.022609, "F": 0.00421, "G": 0.37745, "chunks": 4.31, "bytes_per_chunk": 237.449, "boundary_entropy": 0.65458, "router_noise": 0.0, "grad_norm": 0.8507, "lr": 0.0002759, "router_trainable": true, "sec_per_step": 0.096, "alpha": 0.03, "t": 153.1}
22
+ {"event": "train", "step": 1600, "phase": 1, "objective": "global", "lm_loss": 1.81127, "bpb": 2.61311, "chunk_loss": 0.028345, "F": 0.12985, "G": 0.37895, "chunks": 132.97, "bytes_per_chunk": 7.701, "boundary_entropy": 0.65544, "router_noise": 0.0, "grad_norm": 0.9153, "lr": 0.0002668, "router_trainable": true, "sec_per_step": 0.092, "alpha": 0.03, "t": 171.6}
23
+ {"event": "train", "step": 1800, "phase": 1, "objective": "global", "lm_loss": 1.46985, "bpb": 2.12055, "chunk_loss": 0.023037, "F": 0.00125, "G": 0.36155, "chunks": 1.28, "bytes_per_chunk": 799.22, "boundary_entropy": 0.64533, "router_noise": 0.0, "grad_norm": 0.7498, "lr": 0.0002564, "router_trainable": true, "sec_per_step": 0.092, "alpha": 0.03, "t": 190.0}
24
+ {"event": "probe", "step": 2000, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 712.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.040912, "cps": 5.5625, "mean_p": 0.404796, "boundary_entropy": 0.660596, "bpb": 2.315924}, "de": {"chunks": 888.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.043247, "cps": 6.9375, "mean_p": 0.413895, "boundary_entropy": 0.666974, "bpb": 2.609606}, "vi": {"chunks": 567.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.030757, "cps": 4.429688, "mean_p": 0.396135, "boundary_entropy": 0.659024, "bpb": 2.122418}, "tr": {"chunks": 1251.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.069105, "cps": 9.773438, "mean_p": 0.417649, "boundary_entropy": 0.667817, "bpb": 2.593749}, "fi": {"chunks": 1175.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.062463, "cps": 9.179688, "mean_p": 0.418486, "boundary_entropy": 0.66759, "bpb": 2.862535}, "ru": {"chunks": 1136.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.059947, "cps": 8.875, "mean_p": 0.394438, "boundary_entropy": 0.661856, "bpb": 1.704548}, "el": {"chunks": 1524.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.074258, "cps": 11.90625, "mean_p": 0.417701, "boundary_entropy": 0.672436, "bpb": 1.759558}, "hi": {"chunks": 4278.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.254264, "cps": 33.421875, "mean_p": 0.422668, "boundary_entropy": 0.670879, "bpb": 1.369975}, "ta": {"chunks": 3175.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.169832, "cps": 24.804688, "mean_p": 0.419546, "boundary_entropy": 0.673166, "bpb": 1.112813}, "th": {"chunks": 1965.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.122164, "cps": 15.351562, "mean_p": 0.401491, "boundary_entropy": 0.661667, "bpb": 1.342597}, "zh": {"chunks": 483.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.081781, "cps": 3.773438, "mean_p": 0.397701, "boundary_entropy": 0.658059, "bpb": 3.43383}, "my": {"chunks": 3061.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.161547, "cps": 23.914062, "mean_p": 0.43214, "boundary_entropy": 0.678789, "bpb": 1.260822}}, "bpb": {"en": 2.31592, "de": 2.60961, "vi": 2.12242, "tr": 2.59375, "fi": 2.86254, "ru": 1.70455, "el": 1.75956, "hi": 1.36997, "ta": 1.11281, "th": 1.3426, "zh": 3.43383, "my": 1.26082}, "gini_cpc": 0.343305, "gini_cps": 0.363942, "bpic": 18.984962, "worst_lang": "hi", "cpc_parity_vs_en": {"en": 1.0, "de": 1.0570722821087677, "vi": 0.7517683601356725, "tr": 1.68908258734197, "fi": 1.5267576674895575, "ru": 1.4652551065785182, "el": 1.8150485588380698, "hi": 6.214838305758218, "ta": 4.151092267655534, "th": 2.9859712098298017, "zh": 1.9989310527857787, "my": 3.9486085365506542}, "t": 208.9}
25
+ {"event": "eval", "step": 2000, "phase": 1, "bpb": {"en": 2.32214, "de": 2.61804, "vi": 2.12452, "tr": 2.60296, "fi": 2.86484, "ru": 1.68552, "el": 1.75203, "hi": 1.36677, "ta": 1.11994, "th": 1.36287, "zh": 3.43846, "my": 1.26863}, "bytes_per_chunk": {"en": 30.066, "de": 25.372, "vi": 47.064, "tr": 20.845, "fi": 17.901, "ru": 33.816, "el": 26.214, "hi": 9.738, "ta": 16.31, "th": 26.568, "zh": 45.158, "my": 17.82}, "bpb_mean": 2.04389, "t": 210.2}
26
+ {"event": "train", "step": 2000, "phase": 1, "objective": "global", "lm_loss": 1.44163, "bpb": 2.07983, "chunk_loss": 0.06545, "F": 0.85696, "G": 0.40346, "chunks": 877.53, "bytes_per_chunk": 1.167, "boundary_entropy": 0.6668, "router_noise": 0.0, "grad_norm": 1.1627, "lr": 0.000245, "router_trainable": true, "sec_per_step": 0.101, "alpha": 0.03, "t": 210.3}
27
+ {"event": "train", "step": 2200, "phase": 1, "objective": "global", "lm_loss": 1.31007, "bpb": 1.89003, "chunk_loss": 0.048572, "F": 0.60114, "G": 0.36269, "chunks": 615.56, "bytes_per_chunk": 1.664, "boundary_entropy": 0.64343, "router_noise": 0.0, "grad_norm": 0.8399, "lr": 0.0002325, "router_trainable": true, "sec_per_step": 0.093, "alpha": 0.03, "t": 228.9}
28
+ {"event": "phase_transition", "step": 2400, "phase": 2, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0, "t": 247.2}
29
+ {"event": "train", "step": 2400, "phase": 2, "objective": "global", "lm_loss": 1.34105, "bpb": 1.93472, "chunk_loss": 0.028884, "F": 0.1416, "G": 0.37433, "chunks": 145.0, "bytes_per_chunk": 7.062, "boundary_entropy": 0.65166, "router_noise": 0.0, "grad_norm": 0.8131, "lr": 0.0002192, "router_trainable": true, "sec_per_step": 0.092, "alpha": 0.03, "t": 247.3}
30
+ {"event": "train", "step": 2600, "phase": 2, "objective": "global", "lm_loss": 1.52519, "bpb": 2.20038, "chunk_loss": 0.028042, "F": 0.12323, "G": 0.37579, "chunks": 126.19, "bytes_per_chunk": 8.115, "boundary_entropy": 0.65264, "router_noise": 0.0, "grad_norm": 0.9188, "lr": 0.0002053, "router_trainable": true, "sec_per_step": 0.091, "alpha": 0.03, "t": 265.5}
31
+ {"event": "train", "step": 2800, "phase": 2, "objective": "global", "lm_loss": 1.43453, "bpb": 2.06959, "chunk_loss": 0.023349, "F": 0.00125, "G": 0.35281, "chunks": 1.28, "bytes_per_chunk": 799.22, "boundary_entropy": 0.63995, "router_noise": 0.0, "grad_norm": 0.8882, "lr": 0.0001909, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 283.4}
32
+ {"event": "eval", "step": 3000, "phase": 2, "bpb": {"en": 2.20921, "de": 2.4469, "vi": 2.04661, "tr": 2.43706, "fi": 2.6653, "ru": 1.55049, "el": 1.63054, "hi": 1.29389, "ta": 1.06619, "th": 1.2542, "zh": 3.3289, "my": 1.20051}, "bytes_per_chunk": {"en": 96.094, "de": 108.954, "vi": 68.445, "tr": 76.988, "fi": 56.424, "ru": 57.425, "el": 40.523, "hi": 68.072, "ta": 91.659, "th": 74.178, "zh": 103.943, "my": 69.222}, "bpb_mean": 1.92748, "t": 302.2}
33
+ {"event": "train", "step": 3000, "phase": 2, "objective": "global", "lm_loss": 1.53952, "bpb": 2.22105, "chunk_loss": 0.051232, "F": 0.65289, "G": 0.36814, "chunks": 668.56, "bytes_per_chunk": 1.532, "boundary_entropy": 0.64834, "router_noise": 0.0, "grad_norm": 0.8578, "lr": 0.0001761, "router_trainable": true, "sec_per_step": 0.095, "alpha": 0.03, "t": 302.3}
34
+ {"event": "train", "step": 3200, "phase": 2, "objective": "global", "lm_loss": 1.42466, "bpb": 2.05535, "chunk_loss": 0.0234, "F": 0.00098, "G": 0.35109, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.63906, "router_noise": 0.0, "grad_norm": 0.8193, "lr": 0.0001613, "router_trainable": true, "sec_per_step": 0.091, "alpha": 0.03, "t": 320.6}
35
+ {"event": "train", "step": 3400, "phase": 2, "objective": "global", "lm_loss": 1.33683, "bpb": 1.92864, "chunk_loss": 0.028723, "F": 0.13754, "G": 0.3715, "chunks": 140.84, "bytes_per_chunk": 7.27, "boundary_entropy": 0.65142, "router_noise": 0.0, "grad_norm": 0.9023, "lr": 0.0001465, "router_trainable": true, "sec_per_step": 0.091, "alpha": 0.03, "t": 338.7}
36
+ {"event": "train", "step": 3600, "phase": 2, "objective": "global", "lm_loss": 1.3442, "bpb": 1.93927, "chunk_loss": 0.022435, "F": 0.00098, "G": 0.37803, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65299, "router_noise": 0.0, "grad_norm": 0.8975, "lr": 0.0001319, "router_trainable": true, "sec_per_step": 0.093, "alpha": 0.03, "t": 357.3}
37
+ {"event": "train", "step": 3800, "phase": 2, "objective": "global", "lm_loss": 1.40932, "bpb": 2.03321, "chunk_loss": 0.022659, "F": 0.00104, "G": 0.37186, "chunks": 1.06, "bytes_per_chunk": 963.765, "boundary_entropy": 0.65173, "router_noise": 0.0, "grad_norm": 0.8772, "lr": 0.0001177, "router_trainable": true, "sec_per_step": 0.094, "alpha": 0.03, "t": 376.2}
38
+ {"event": "probe", "step": 4000, "phase": 2, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 294.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.016894, "cps": 2.296875, "mean_p": 0.377103, "boundary_entropy": 0.646966, "bpb": 2.132903}, "de": {"chunks": 318.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.015487, "cps": 2.484375, "mean_p": 0.382933, "boundary_entropy": 0.652848, "bpb": 2.346793}, "vi": {"chunks": 469.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.025441, "cps": 3.664062, "mean_p": 0.3835, "boundary_entropy": 0.653029, "bpb": 1.989742}, "tr": {"chunks": 408.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.022538, "cps": 3.1875, "mean_p": 0.380939, "boundary_entropy": 0.651878, "bpb": 2.318921}, "fi": {"chunks": 483.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.025676, "cps": 3.773438, "mean_p": 0.388374, "boundary_entropy": 0.654449, "bpb": 2.534682}, "ru": {"chunks": 785.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.041425, "cps": 6.132812, "mean_p": 0.39287, "boundary_entropy": 0.662042, "bpb": 1.504671}, "el": {"chunks": 665.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.032403, "cps": 5.195312, "mean_p": 0.381358, "boundary_entropy": 0.656416, "bpb": 1.562564}, "hi": {"chunks": 1053.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.062585, "cps": 8.226562, "mean_p": 0.356573, "boundary_entropy": 0.627373, "bpb": 1.260632}, "ta": {"chunks": 1553.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.08307, "cps": 12.132812, "mean_p": 0.391833, "boundary_entropy": 0.662245, "bpb": 1.027909}, "th": {"chunks": 827.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.051414, "cps": 6.460938, "mean_p": 0.372158, "boundary_entropy": 0.646485, "bpb": 1.176101}, "zh": {"chunks": 267.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.045208, "cps": 2.085938, "mean_p": 0.385231, "boundary_entropy": 0.653231, "bpb": 3.214515}, "my": {"chunks": 257.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.013563, "cps": 2.007812, "mean_p": 0.384606, "boundary_entropy": 0.660241, "bpb": 1.159897}}, "bpb": {"en": 2.1329, "de": 2.34679, "vi": 1.98974, "tr": 2.31892, "fi": 2.53468, "ru": 1.50467, "el": 1.56256, "hi": 1.26063, "ta": 1.02791, "th": 1.1761, "zh": 3.21452, "my": 1.1599}, "gini_cpc": 0.305661, "gini_cps": 0.318257, "bpic": 52.009893, "worst_lang": "ta", "cpc_parity_vs_en": {"en": 1.0, "de": 0.9167512327095159, "vi": 1.5059359138285093, "tr": 1.3340939093418949, "fi": 1.519889578286261, "ru": 2.4520946637230088, "el": 1.9180396906606525, "hi": 3.704674773326864, "ta": 4.917255514133976, "th": 3.043415401597381, "zh": 2.6760523715073568, "my": 0.8028724155906821}, "t": 394.7}
39
+ {"event": "eval", "step": 4000, "phase": 2, "bpb": {"en": 2.13991, "de": 2.35358, "vi": 1.98988, "tr": 2.32957, "fi": 2.53829, "ru": 1.48173, "el": 1.55782, "hi": 1.25631, "ta": 1.03414, "th": 1.19615, "zh": 3.21293, "my": 1.16754}, "bytes_per_chunk": {"en": 82.826, "de": 82.383, "vi": 59.511, "tr": 87.41, "fi": 43.807, "ru": 50.862, "el": 57.513, "hi": 42.604, "ta": 40.605, "th": 61.536, "zh": 96.911, "my": 92.11}, "bpb_mean": 1.85482, "t": 396.0}
40
+ {"event": "train", "step": 4000, "phase": 2, "objective": "global", "lm_loss": 1.3414, "bpb": 1.93524, "chunk_loss": 0.052572, "F": 0.66241, "G": 0.37645, "chunks": 678.31, "bytes_per_chunk": 1.51, "boundary_entropy": 0.65202, "router_noise": 0.0, "grad_norm": 0.8761, "lr": 0.000104, "router_trainable": true, "sec_per_step": 0.099, "alpha": 0.03, "t": 396.1}
41
+ {"event": "phase_transition", "step": 4200, "phase": 3, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0, "t": 415.2}
42
+ {"event": "train", "step": 4200, "phase": 3, "objective": "global", "lm_loss": 1.32287, "bpb": 1.90849, "chunk_loss": 0.022215, "F": 0.0011, "G": 0.38436, "chunks": 1.12, "bytes_per_chunk": 910.222, "boundary_entropy": 0.65843, "router_noise": 0.0, "grad_norm": 0.8063, "lr": 9.12e-05, "router_trainable": true, "sec_per_step": 0.096, "alpha": 0.03, "t": 415.3}
43
+ {"event": "train", "step": 4400, "phase": 3, "objective": "global", "lm_loss": 1.48374, "bpb": 2.14059, "chunk_loss": 0.022965, "F": 0.00143, "G": 0.36377, "chunks": 1.47, "bytes_per_chunk": 697.191, "boundary_entropy": 0.64664, "router_noise": 0.0, "grad_norm": 0.951, "lr": 7.92e-05, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 433.1}
44
+ {"event": "train", "step": 4600, "phase": 3, "objective": "global", "lm_loss": 1.44325, "bpb": 2.08217, "chunk_loss": 0.022639, "F": 0.00098, "G": 0.37234, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65114, "router_noise": 0.0, "grad_norm": 0.8701, "lr": 6.82e-05, "router_trainable": true, "sec_per_step": 0.091, "alpha": 0.03, "t": 451.4}
45
+ {"event": "train", "step": 4800, "phase": 3, "objective": "global", "lm_loss": 1.38421, "bpb": 1.99699, "chunk_loss": 0.02217, "F": 0.00098, "G": 0.38544, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65813, "router_noise": 0.0, "grad_norm": 0.8583, "lr": 5.85e-05, "router_trainable": true, "sec_per_step": 0.094, "alpha": 0.03, "t": 470.1}
46
+ {"event": "eval", "step": 5000, "phase": 3, "bpb": {"en": 2.0983, "de": 2.30449, "vi": 1.96153, "tr": 2.27631, "fi": 2.46698, "ru": 1.43429, "el": 1.50627, "hi": 1.22834, "ta": 1.01362, "th": 1.16708, "zh": 3.15443, "my": 1.14021}, "bytes_per_chunk": {"en": 64.346, "de": 71.507, "vi": 53.184, "tr": 51.08, "fi": 65.585, "ru": 51.695, "el": 42.876, "hi": 41.849, "ta": 45.189, "th": 54.659, "zh": 65.882, "my": 40.436}, "bpb_mean": 1.81265, "t": 489.7}
47
+ {"event": "train", "step": 5000, "phase": 3, "objective": "global", "lm_loss": 1.40145, "bpb": 2.02187, "chunk_loss": 0.025775, "F": 0.07776, "G": 0.38607, "chunks": 79.62, "bytes_per_chunk": 12.86, "boundary_entropy": 0.65884, "router_noise": 0.0, "grad_norm": 0.8475, "lr": 5e-05, "router_trainable": true, "sec_per_step": 0.098, "alpha": 0.03, "t": 489.7}
48
+ {"event": "train", "step": 5200, "phase": 3, "objective": "global", "lm_loss": 1.28923, "bpb": 1.85996, "chunk_loss": 0.022336, "F": 0.00351, "G": 0.38413, "chunks": 3.59, "bytes_per_chunk": 284.939, "boundary_entropy": 0.65839, "router_noise": 0.0, "grad_norm": 0.8719, "lr": 4.29e-05, "router_trainable": true, "sec_per_step": 0.094, "alpha": 0.03, "t": 508.5}
49
+ {"event": "train", "step": 5400, "phase": 3, "objective": "global", "lm_loss": 1.3869, "bpb": 2.00088, "chunk_loss": 0.02251, "F": 0.00098, "G": 0.37596, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.6538, "router_noise": 0.0, "grad_norm": 0.954, "lr": 3.73e-05, "router_trainable": true, "sec_per_step": 0.093, "alpha": 0.03, "t": 527.1}
50
+ {"event": "train", "step": 5600, "phase": 3, "objective": "global", "lm_loss": 1.36892, "bpb": 1.97493, "chunk_loss": 0.023072, "F": 0.01898, "G": 0.38386, "chunks": 19.44, "bytes_per_chunk": 52.682, "boundary_entropy": 0.65772, "router_noise": 0.0, "grad_norm": 0.888, "lr": 3.33e-05, "router_trainable": true, "sec_per_step": 0.095, "alpha": 0.03, "t": 546.1}
51
+ {"event": "train", "step": 5800, "phase": 3, "objective": "global", "lm_loss": 1.32724, "bpb": 1.9148, "chunk_loss": 0.022554, "F": 0.00406, "G": 0.37868, "chunks": 4.16, "bytes_per_chunk": 246.376, "boundary_entropy": 0.655, "router_noise": 0.0, "grad_norm": 0.9174, "lr": 3.08e-05, "router_trainable": true, "sec_per_step": 0.095, "alpha": 0.03, "t": 565.1}
52
+ {"event": "probe", "step": 6000, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 235.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.013503, "cps": 1.835938, "mean_p": 0.374696, "boundary_entropy": 0.646152, "bpb": 2.072183}, "de": {"chunks": 254.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.01237, "cps": 1.984375, "mean_p": 0.378796, "boundary_entropy": 0.650996, "bpb": 2.273952}, "vi": {"chunks": 353.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.019148, "cps": 2.757812, "mean_p": 0.375519, "boundary_entropy": 0.649338, "bpb": 1.946296}, "tr": {"chunks": 410.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.022648, "cps": 3.203125, "mean_p": 0.381634, "boundary_entropy": 0.652266, "bpb": 2.238207}, "fi": {"chunks": 245.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.013024, "cps": 1.914062, "mean_p": 0.372556, "boundary_entropy": 0.646645, "bpb": 2.439194}, "ru": {"chunks": 591.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.031187, "cps": 4.617188, "mean_p": 0.376535, "boundary_entropy": 0.654067, "bpb": 1.435821}, "el": {"chunks": 566.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.027579, "cps": 4.421875, "mean_p": 0.37797, "boundary_entropy": 0.654882, "bpb": 1.489949}, "hi": {"chunks": 836.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.049688, "cps": 6.53125, "mean_p": 0.361977, "boundary_entropy": 0.633895, "bpb": 1.218163}, "ta": {"chunks": 1009.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.053972, "cps": 7.882812, "mean_p": 0.380425, "boundary_entropy": 0.656419, "bpb": 0.991907}, "th": {"chunks": 638.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.039664, "cps": 4.984375, "mean_p": 0.363322, "boundary_entropy": 0.642576, "bpb": 1.127957}, "zh": {"chunks": 237.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.040129, "cps": 1.851562, "mean_p": 0.379361, "boundary_entropy": 0.650523, "bpb": 3.125459}, "my": {"chunks": 816.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.043065, "cps": 6.375, "mean_p": 0.36954, "boundary_entropy": 0.651657, "bpb": 1.119749}}, "bpb": {"en": 2.07218, "de": 2.27395, "vi": 1.9463, "tr": 2.23821, "fi": 2.43919, "ru": 1.43582, "el": 1.48995, "hi": 1.21816, "ta": 0.99191, "th": 1.12796, "zh": 3.12546, "my": 1.11975}, "gini_cpc": 0.262784, "gini_cps": 0.278568, "bpic": 62.000162, "worst_lang": "ta", "cpc_parity_vs_en": {"en": 1.0, "de": 0.9160887870174738, "vi": 1.4180378442947907, "tr": 1.677218187651982, "fi": 0.9645182707718549, "ru": 2.3095880536686693, "el": 2.042357873522286, "hi": 3.6796580569694286, "ta": 3.9968877786187003, "th": 2.9373511729574933, "zh": 2.9717424040463722, "my": 3.1892094376995943}, "t": 584.2}
53
+ {"event": "eval", "step": 6000, "phase": 3, "bpb": {"en": 2.08045, "de": 2.28221, "vi": 1.94655, "tr": 2.24572, "fi": 2.44227, "ru": 1.4125, "el": 1.48345, "hi": 1.21199, "ta": 0.99986, "th": 1.14677, "zh": 3.12322, "my": 1.12666}, "bytes_per_chunk": {"en": 123.478, "de": 132.262, "vi": 89.134, "tr": 87.294, "fi": 135.826, "ru": 73.802, "el": 67.079, "hi": 54.613, "ta": 64.22, "th": 81.436, "zh": 119.7, "my": 51.031}, "bpb_mean": 1.7918, "t": 585.5}
54
+ {"event": "probe", "step": 6000, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 235.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.013503, "cps": 1.835938, "mean_p": 0.374696, "boundary_entropy": 0.646152, "bpb": 2.072183}, "de": {"chunks": 254.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.01237, "cps": 1.984375, "mean_p": 0.378796, "boundary_entropy": 0.650996, "bpb": 2.273952}, "vi": {"chunks": 353.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.019148, "cps": 2.757812, "mean_p": 0.375519, "boundary_entropy": 0.649338, "bpb": 1.946296}, "tr": {"chunks": 410.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.022648, "cps": 3.203125, "mean_p": 0.381634, "boundary_entropy": 0.652266, "bpb": 2.238207}, "fi": {"chunks": 245.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.013024, "cps": 1.914062, "mean_p": 0.372556, "boundary_entropy": 0.646645, "bpb": 2.439194}, "ru": {"chunks": 591.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.031187, "cps": 4.617188, "mean_p": 0.376535, "boundary_entropy": 0.654067, "bpb": 1.435821}, "el": {"chunks": 566.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.027579, "cps": 4.421875, "mean_p": 0.37797, "boundary_entropy": 0.654882, "bpb": 1.489949}, "hi": {"chunks": 836.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.049688, "cps": 6.53125, "mean_p": 0.361977, "boundary_entropy": 0.633895, "bpb": 1.218163}, "ta": {"chunks": 1009.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.053972, "cps": 7.882812, "mean_p": 0.380425, "boundary_entropy": 0.656419, "bpb": 0.991907}, "th": {"chunks": 638.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.039664, "cps": 4.984375, "mean_p": 0.363322, "boundary_entropy": 0.642576, "bpb": 1.127957}, "zh": {"chunks": 237.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.040129, "cps": 1.851562, "mean_p": 0.379361, "boundary_entropy": 0.650523, "bpb": 3.125459}, "my": {"chunks": 816.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.043065, "cps": 6.375, "mean_p": 0.36954, "boundary_entropy": 0.651657, "bpb": 1.119749}}, "bpb": {"en": 2.07218, "de": 2.27395, "vi": 1.9463, "tr": 2.23821, "fi": 2.43919, "ru": 1.43582, "el": 1.48995, "hi": 1.21816, "ta": 0.99191, "th": 1.12796, "zh": 3.12546, "my": 1.11975}, "gini_cpc": 0.262784, "gini_cps": 0.278568, "bpic": 62.000162, "worst_lang": "ta", "cpc_parity_vs_en": {"en": 1.0, "de": 0.9160887870174738, "vi": 1.4180378442947907, "tr": 1.677218187651982, "fi": 0.9645182707718549, "ru": 2.3095880536686693, "el": 2.042357873522286, "hi": 3.6796580569694286, "ta": 3.9968877786187003, "th": 2.9373511729574933, "zh": 2.9717424040463722, "my": 3.1892094376995943}, "t": 586.4}
55
+ {"event": "reps_dumped", "dirs": ["/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s2/th0.30", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s2/th0.40", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s2/th0.50", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s2/th0.60", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s2/th0.70"], "t": 590.1}
56
+ {"event": "final", "run": "jitter_s2", "condition": "A", "step": 6000, "params": 22516992, "per_lang": {"en": {"cpc": 0.013503418950755616, "cps": 1.8359375, "bpb": 2.091184529801444, "bytes_per_chunk": 119.15636363636364}, "de": {"cpc": 0.01237033068718648, "cps": 1.984375, "bpb": 2.2734026958836058, "bytes_per_chunk": 135.02137522534125}, "vi": {"cpc": 0.01914835909953892, "cps": 2.7578125, "bpb": 1.9494414809506864, "bytes_per_chunk": 86.68783068783068}, "tr": {"cpc": 0.022648179859691763, "cps": 3.203125, "bpb": 2.240100760041376, "bytes_per_chunk": 89.34696659850034}, "fi": {"cpc": 0.013024294295890703, "cps": 1.9140625, "bpb": 2.4580483003424547, "bytes_per_chunk": 132.8656867714141}, "ru": {"cpc": 0.031187335092348285, "cps": 4.6171875, "bpb": 1.4135231609180474, "bytes_per_chunk": 73.35777249195466}, "el": {"cpc": 0.02757881401354578, "cps": 4.421875, "bpb": 1.4792314436970544, "bytes_per_chunk": 67.52807831014941}, "hi": {"cpc": 0.04968796433878157, "cps": 6.53125, "bpb": 1.2194370256690006, "bytes_per_chunk": 54.50545794781162}, "ta": {"cpc": 0.053971650173843276, "cps": 7.8828125, "bpb": 0.9995220038420514, "bytes_per_chunk": 63.92197025115826}, "th": {"cpc": 0.039664283493938454, "cps": 4.984375, "bpb": 1.1435730867396772, "bytes_per_chunk": 81.6393646838991}, "zh": {"cpc": 0.040128682695563835, "cps": 1.8515625, "bpb": 3.1307407679749546, "bytes_per_chunk": 120.83152800184374}, "my": {"cpc": 0.04306523115896137, "cps": 6.375, "bpb": 1.1354728113731958, "bytes_per_chunk": 50.768664665440106}}, "gini_cpc": 0.262783775060067, "gini_cps": 0.27856758212170174, "bpic": 62.00016155088853, "seed": 2, "mean_boundary_entropy": 0.6519113779067993, "dechunk": "ema", "router_noise": 0.0, "transitions": [{"step": 0, "phase": 1, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}, {"step": 2400, "phase": 2, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}, {"step": 4200, "phase": 3, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}], "t": 592.5}
jitter_s2/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d54044247bbbf717f8028f13741dd92eefd6ae377bdef734e96d00f247c36521
3
+ size 90078288
manifest.json ADDED
@@ -0,0 +1,343 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "run": "baseline_en",
4
+ "params": 22582528,
5
+ "stored_tensors": 110,
6
+ "tied": {
7
+ "head.weight": "emb.weight"
8
+ },
9
+ "step": 5000,
10
+ "condition": "A",
11
+ "seed": 0
12
+ },
13
+ {
14
+ "run": "emactl_genreg_s0",
15
+ "params": 22582528,
16
+ "stored_tensors": 110,
17
+ "tied": {
18
+ "head.weight": "emb.weight"
19
+ },
20
+ "step": 6000,
21
+ "condition": "A",
22
+ "seed": 0
23
+ },
24
+ {
25
+ "run": "emactl_genreg_s1",
26
+ "params": 22582528,
27
+ "stored_tensors": 110,
28
+ "tied": {
29
+ "head.weight": "emb.weight"
30
+ },
31
+ "step": 6000,
32
+ "condition": "A",
33
+ "seed": 1
34
+ },
35
+ {
36
+ "run": "emactl_genreg_s2",
37
+ "params": 22582528,
38
+ "stored_tensors": 110,
39
+ "tied": {
40
+ "head.weight": "emb.weight"
41
+ },
42
+ "step": 6000,
43
+ "condition": "A",
44
+ "seed": 2
45
+ },
46
+ {
47
+ "run": "emactl_noema_s0",
48
+ "params": 22582528,
49
+ "stored_tensors": 110,
50
+ "tied": {
51
+ "head.weight": "emb.weight"
52
+ },
53
+ "step": 6000,
54
+ "condition": "A",
55
+ "seed": 0
56
+ },
57
+ {
58
+ "run": "emactl_noema_s1",
59
+ "params": 22582528,
60
+ "stored_tensors": 110,
61
+ "tied": {
62
+ "head.weight": "emb.weight"
63
+ },
64
+ "step": 6000,
65
+ "condition": "A",
66
+ "seed": 1
67
+ },
68
+ {
69
+ "run": "emactl_noema_s2",
70
+ "params": 22582528,
71
+ "stored_tensors": 110,
72
+ "tied": {
73
+ "head.weight": "emb.weight"
74
+ },
75
+ "step": 6000,
76
+ "condition": "A",
77
+ "seed": 2
78
+ },
79
+ {
80
+ "run": "jitter_s0",
81
+ "params": 22582528,
82
+ "stored_tensors": 110,
83
+ "tied": {
84
+ "head.weight": "emb.weight"
85
+ },
86
+ "step": 6000,
87
+ "condition": "A",
88
+ "seed": 0
89
+ },
90
+ {
91
+ "run": "jitter_s1",
92
+ "params": 22582528,
93
+ "stored_tensors": 110,
94
+ "tied": {
95
+ "head.weight": "emb.weight"
96
+ },
97
+ "step": 6000,
98
+ "condition": "A",
99
+ "seed": 1
100
+ },
101
+ {
102
+ "run": "jitter_s2",
103
+ "params": 22582528,
104
+ "stored_tensors": 110,
105
+ "tied": {
106
+ "head.weight": "emb.weight"
107
+ },
108
+ "step": 6000,
109
+ "condition": "A",
110
+ "seed": 2
111
+ },
112
+ {
113
+ "run": "pilot_A_s0",
114
+ "params": 22582528,
115
+ "stored_tensors": 110,
116
+ "tied": {
117
+ "head.weight": "emb.weight"
118
+ },
119
+ "step": 6000,
120
+ "condition": "A",
121
+ "seed": 0
122
+ },
123
+ {
124
+ "run": "pilot_A_s1",
125
+ "params": 22582528,
126
+ "stored_tensors": 110,
127
+ "tied": {
128
+ "head.weight": "emb.weight"
129
+ },
130
+ "step": 6000,
131
+ "condition": "A",
132
+ "seed": 1
133
+ },
134
+ {
135
+ "run": "pilot_A_s2",
136
+ "params": 22582528,
137
+ "stored_tensors": 110,
138
+ "tied": {
139
+ "head.weight": "emb.weight"
140
+ },
141
+ "step": 6000,
142
+ "condition": "A",
143
+ "seed": 2
144
+ },
145
+ {
146
+ "run": "pilot_Apc_s0",
147
+ "params": 22582528,
148
+ "stored_tensors": 110,
149
+ "tied": {
150
+ "head.weight": "emb.weight"
151
+ },
152
+ "step": 6000,
153
+ "condition": "A",
154
+ "seed": 0
155
+ },
156
+ {
157
+ "run": "pilot_Apc_s1",
158
+ "params": 22582528,
159
+ "stored_tensors": 110,
160
+ "tied": {
161
+ "head.weight": "emb.weight"
162
+ },
163
+ "step": 6000,
164
+ "condition": "A",
165
+ "seed": 1
166
+ },
167
+ {
168
+ "run": "pilot_Apc_s2",
169
+ "params": 22582528,
170
+ "stored_tensors": 110,
171
+ "tied": {
172
+ "head.weight": "emb.weight"
173
+ },
174
+ "step": 6000,
175
+ "condition": "A",
176
+ "seed": 2
177
+ },
178
+ {
179
+ "run": "pilot_B_s0",
180
+ "params": 22582528,
181
+ "stored_tensors": 110,
182
+ "tied": {
183
+ "head.weight": "emb.weight"
184
+ },
185
+ "step": 6000,
186
+ "condition": "B",
187
+ "seed": 0
188
+ },
189
+ {
190
+ "run": "pilot_B_s1",
191
+ "params": 22582528,
192
+ "stored_tensors": 110,
193
+ "tied": {
194
+ "head.weight": "emb.weight"
195
+ },
196
+ "step": 6000,
197
+ "condition": "B",
198
+ "seed": 1
199
+ },
200
+ {
201
+ "run": "pilot_B_s2",
202
+ "params": 22582528,
203
+ "stored_tensors": 110,
204
+ "tied": {
205
+ "head.weight": "emb.weight"
206
+ },
207
+ "step": 6000,
208
+ "condition": "B",
209
+ "seed": 2
210
+ },
211
+ {
212
+ "run": "pilot_Bpc_s0",
213
+ "params": 22582528,
214
+ "stored_tensors": 110,
215
+ "tied": {
216
+ "head.weight": "emb.weight"
217
+ },
218
+ "step": 6000,
219
+ "condition": "B",
220
+ "seed": 0
221
+ },
222
+ {
223
+ "run": "pilot_Bpc_s1",
224
+ "params": 22582528,
225
+ "stored_tensors": 110,
226
+ "tied": {
227
+ "head.weight": "emb.weight"
228
+ },
229
+ "step": 6000,
230
+ "condition": "B",
231
+ "seed": 1
232
+ },
233
+ {
234
+ "run": "pilot_Bpc_s2",
235
+ "params": 22582528,
236
+ "stored_tensors": 110,
237
+ "tied": {
238
+ "head.weight": "emb.weight"
239
+ },
240
+ "step": 6000,
241
+ "condition": "B",
242
+ "seed": 2
243
+ },
244
+ {
245
+ "run": "pilot_C_s0",
246
+ "params": 22582528,
247
+ "stored_tensors": 110,
248
+ "tied": {
249
+ "head.weight": "emb.weight"
250
+ },
251
+ "step": 6000,
252
+ "condition": "C",
253
+ "seed": 0
254
+ },
255
+ {
256
+ "run": "pilot_C_s1",
257
+ "params": 22582528,
258
+ "stored_tensors": 110,
259
+ "tied": {
260
+ "head.weight": "emb.weight"
261
+ },
262
+ "step": 6000,
263
+ "condition": "C",
264
+ "seed": 1
265
+ },
266
+ {
267
+ "run": "pilot_C_s2",
268
+ "params": 22582528,
269
+ "stored_tensors": 110,
270
+ "tied": {
271
+ "head.weight": "emb.weight"
272
+ },
273
+ "step": 6000,
274
+ "condition": "C",
275
+ "seed": 2
276
+ },
277
+ {
278
+ "run": "pilot_D_s0",
279
+ "params": 22582528,
280
+ "stored_tensors": 110,
281
+ "tied": {
282
+ "head.weight": "emb.weight"
283
+ },
284
+ "step": 6000,
285
+ "condition": "D",
286
+ "seed": 0
287
+ },
288
+ {
289
+ "run": "pilot_D_s1",
290
+ "params": 22582528,
291
+ "stored_tensors": 110,
292
+ "tied": {
293
+ "head.weight": "emb.weight"
294
+ },
295
+ "step": 6000,
296
+ "condition": "D",
297
+ "seed": 1
298
+ },
299
+ {
300
+ "run": "pilot_D_s2",
301
+ "params": 22582528,
302
+ "stored_tensors": 110,
303
+ "tied": {
304
+ "head.weight": "emb.weight"
305
+ },
306
+ "step": 6000,
307
+ "condition": "D",
308
+ "seed": 2
309
+ },
310
+ {
311
+ "run": "pilot_E_s0",
312
+ "params": 22582528,
313
+ "stored_tensors": 110,
314
+ "tied": {
315
+ "head.weight": "emb.weight"
316
+ },
317
+ "step": 6000,
318
+ "condition": "E",
319
+ "seed": 0
320
+ },
321
+ {
322
+ "run": "pilot_E_s1",
323
+ "params": 22582528,
324
+ "stored_tensors": 110,
325
+ "tied": {
326
+ "head.weight": "emb.weight"
327
+ },
328
+ "step": 6000,
329
+ "condition": "E",
330
+ "seed": 1
331
+ },
332
+ {
333
+ "run": "pilot_E_s2",
334
+ "params": 22582528,
335
+ "stored_tensors": 110,
336
+ "tied": {
337
+ "head.weight": "emb.weight"
338
+ },
339
+ "step": 6000,
340
+ "condition": "E",
341
+ "seed": 2
342
+ }
343
+ ]
pilot_A_s0/config.json ADDED
@@ -0,0 +1,106 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run": "pilot_A_s0",
3
+ "step": 6000,
4
+ "model_config": {
5
+ "vocab_size": 256,
6
+ "d_enc": 256,
7
+ "d_main": 512,
8
+ "n_enc": 2,
9
+ "n_dec": 2,
10
+ "n_main": 6,
11
+ "n_heads_enc": 4,
12
+ "n_heads_main": 8,
13
+ "seq_len": 1024,
14
+ "dropout": 0.0,
15
+ "target_ratio": 6.0,
16
+ "max_chunk_mult": 3.0,
17
+ "dechunk": "ema",
18
+ "router_noise": 0.0,
19
+ "rope_base": 10000.0
20
+ },
21
+ "tied_weights": {
22
+ "head.weight": "emb.weight"
23
+ },
24
+ "train_config": {
25
+ "condition": "A",
26
+ "seed": 0,
27
+ "out_dir": null,
28
+ "run_name": "pilot_A_s0",
29
+ "corpus": "corpus",
30
+ "langs": null,
31
+ "seq_len": 1024,
32
+ "batch_size": 32,
33
+ "grad_accum": 1,
34
+ "steps": 6000,
35
+ "lr": 0.0003,
36
+ "min_lr_frac": 0.1,
37
+ "warmup": 300,
38
+ "weight_decay": 0.1,
39
+ "grad_clip": 1.0,
40
+ "betas": [
41
+ 0.9,
42
+ 0.95
43
+ ],
44
+ "router_lr_mult": 1.0,
45
+ "d_enc": 256,
46
+ "d_main": 512,
47
+ "n_enc": 2,
48
+ "n_dec": 2,
49
+ "n_main": 6,
50
+ "n_heads_enc": 4,
51
+ "n_heads_main": 8,
52
+ "dechunk": "ema",
53
+ "target_ratio": 6.0,
54
+ "alpha": 0.03,
55
+ "alpha_parity_cap": 0.06,
56
+ "global_anchor": 0.0,
57
+ "premium_clip": 4.0,
58
+ "premium_kind": "flores_per_sent",
59
+ "mixture": null,
60
+ "doc_filters": null,
61
+ "tau1": 0.4,
62
+ "tau2": 0.7,
63
+ "freeze_scope": "router",
64
+ "ckpt_every": 200,
65
+ "eval_every": 200,
66
+ "log_every": 50,
67
+ "probe_sents": 128,
68
+ "eval_batches": 8,
69
+ "keep_ckpt": false,
70
+ "amp": "bf16",
71
+ "compile": false,
72
+ "cpc_decay": 0.9,
73
+ "device": "cuda",
74
+ "log_spaced_probe": true,
75
+ "probe_bpb_batches": 4,
76
+ "dump_reps": true,
77
+ "dump_reps_thresholds": [
78
+ 0.4,
79
+ 0.5,
80
+ 0.6
81
+ ],
82
+ "router_noise": 0.0,
83
+ "genreg_target_entropy": null,
84
+ "genreg_eta": 0.05
85
+ },
86
+ "probe_agg": {
87
+ "gini_cpc": 0.2046874384551851,
88
+ "gini_cps": 0.2124337217717518,
89
+ "bpic": 5.668261775001108,
90
+ "worst_lang": "my",
91
+ "cpc_parity_vs_en": {
92
+ "en": 1.0,
93
+ "de": 0.9674031957033492,
94
+ "vi": 1.0641769666197316,
95
+ "tr": 0.9818145543054209,
96
+ "fi": 0.9579169801888867,
97
+ "ru": 1.5462771572877685,
98
+ "el": 1.507303331656917,
99
+ "hi": 2.114500469118362,
100
+ "ta": 2.0382228601055625,
101
+ "th": 2.417749025953193,
102
+ "zh": 2.32292387976097,
103
+ "my": 2.471763194501154
104
+ }
105
+ }
106
+ }
pilot_A_s0/final.json ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "event": "final",
3
+ "run": "pilot_A_s0",
4
+ "condition": "A",
5
+ "step": 6000,
6
+ "params": 22516992,
7
+ "per_lang": {
8
+ "en": {
9
+ "cpc": 0.2076653450554502,
10
+ "cps": 28.234375,
11
+ "bpb": 2.0273781279159193,
12
+ "bytes_per_chunk": 4.917766459370984
13
+ },
14
+ "de": {
15
+ "cpc": 0.20089611844348124,
16
+ "cps": 32.2265625,
17
+ "bpb": 2.194651560581718,
18
+ "bytes_per_chunk": 4.990082424380866
19
+ },
20
+ "vi": {
21
+ "cpc": 0.2209926769731489,
22
+ "cps": 31.828125,
23
+ "bpb": 1.8993434507023028,
24
+ "bytes_per_chunk": 5.929249977381707
25
+ },
26
+ "tr": {
27
+ "cpc": 0.20388885820029828,
28
+ "cps": 28.8359375,
29
+ "bpb": 2.172621765386917,
30
+ "bytes_per_chunk": 5.211869377205627
31
+ },
32
+ "fi": {
33
+ "cpc": 0.19892616022540002,
34
+ "cps": 29.234375,
35
+ "bpb": 2.362985137303098,
36
+ "bytes_per_chunk": 4.885778452879069
37
+ },
38
+ "ru": {
39
+ "cpc": 0.3211081794195251,
40
+ "cps": 47.5390625,
41
+ "bpb": 1.3424836631873538,
42
+ "bytes_per_chunk": 5.437995270298303
43
+ },
44
+ "el": {
45
+ "cpc": 0.3130146664717634,
46
+ "cps": 50.1875,
47
+ "bpb": 1.3926614235177859,
48
+ "bytes_per_chunk": 5.5650401757756525
49
+ },
50
+ "hi": {
51
+ "cpc": 0.43910846953937593,
52
+ "cps": 57.71875,
53
+ "bpb": 1.1374732168277908,
54
+ "bytes_per_chunk": 5.7044250291048755
55
+ },
56
+ "ta": {
57
+ "cpc": 0.42326825354372827,
58
+ "cps": 61.8203125,
59
+ "bpb": 0.9469699101528437,
60
+ "bytes_per_chunk": 6.453172502923257
61
+ },
62
+ "th": {
63
+ "cpc": 0.5020826857320485,
64
+ "cps": 63.09375,
65
+ "bpb": 1.093262663291936,
66
+ "bytes_per_chunk": 5.682784335403592
67
+ },
68
+ "zh": {
69
+ "cpc": 0.482390789028107,
70
+ "cps": 22.2578125,
71
+ "bpb": 3.1054285549014486,
72
+ "bytes_per_chunk": 5.831448051875827
73
+ },
74
+ "my": {
75
+ "cpc": 0.513299556681444,
76
+ "cps": 75.984375,
77
+ "bpb": 1.0876047296737308,
78
+ "bytes_per_chunk": 6.01137405980554
79
+ }
80
+ },
81
+ "gini_cpc": 0.2046874384551851,
82
+ "gini_cps": 0.2124337217717518,
83
+ "bpic": 5.668261775001108,
84
+ "seed": 0,
85
+ "mean_boundary_entropy": 0.32596713304519653,
86
+ "dechunk": "ema",
87
+ "router_noise": 0.0,
88
+ "transitions": [
89
+ {
90
+ "step": 0,
91
+ "phase": 1,
92
+ "objective": "global",
93
+ "router_trainable": true,
94
+ "encoder_trainable": true,
95
+ "frozen_router_params": 0
96
+ },
97
+ {
98
+ "step": 2400,
99
+ "phase": 2,
100
+ "objective": "global",
101
+ "router_trainable": true,
102
+ "encoder_trainable": true,
103
+ "frozen_router_params": 0
104
+ },
105
+ {
106
+ "step": 4200,
107
+ "phase": 3,
108
+ "objective": "global",
109
+ "router_trainable": true,
110
+ "encoder_trainable": true,
111
+ "frozen_router_params": 0
112
+ }
113
+ ]
114
+ }
pilot_A_s0/log.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
pilot_A_s0/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b7cc09aad5130e4f1b5e62a9ffd33305232927fee93d6deba4ea5dad45ddfc55
3
+ size 90078296
pilot_A_s1/config.json ADDED
@@ -0,0 +1,106 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run": "pilot_A_s1",
3
+ "step": 6000,
4
+ "model_config": {
5
+ "vocab_size": 256,
6
+ "d_enc": 256,
7
+ "d_main": 512,
8
+ "n_enc": 2,
9
+ "n_dec": 2,
10
+ "n_main": 6,
11
+ "n_heads_enc": 4,
12
+ "n_heads_main": 8,
13
+ "seq_len": 1024,
14
+ "dropout": 0.0,
15
+ "target_ratio": 6.0,
16
+ "max_chunk_mult": 3.0,
17
+ "dechunk": "ema",
18
+ "router_noise": 0.0,
19
+ "rope_base": 10000.0
20
+ },
21
+ "tied_weights": {
22
+ "head.weight": "emb.weight"
23
+ },
24
+ "train_config": {
25
+ "condition": "A",
26
+ "seed": 1,
27
+ "out_dir": null,
28
+ "run_name": "pilot_A_s1",
29
+ "corpus": "corpus",
30
+ "langs": null,
31
+ "seq_len": 1024,
32
+ "batch_size": 32,
33
+ "grad_accum": 1,
34
+ "steps": 6000,
35
+ "lr": 0.0003,
36
+ "min_lr_frac": 0.1,
37
+ "warmup": 300,
38
+ "weight_decay": 0.1,
39
+ "grad_clip": 1.0,
40
+ "betas": [
41
+ 0.9,
42
+ 0.95
43
+ ],
44
+ "router_lr_mult": 1.0,
45
+ "d_enc": 256,
46
+ "d_main": 512,
47
+ "n_enc": 2,
48
+ "n_dec": 2,
49
+ "n_main": 6,
50
+ "n_heads_enc": 4,
51
+ "n_heads_main": 8,
52
+ "dechunk": "ema",
53
+ "target_ratio": 6.0,
54
+ "alpha": 0.03,
55
+ "alpha_parity_cap": 0.06,
56
+ "global_anchor": 0.0,
57
+ "premium_clip": 4.0,
58
+ "premium_kind": "flores_per_sent",
59
+ "mixture": null,
60
+ "doc_filters": null,
61
+ "tau1": 0.4,
62
+ "tau2": 0.7,
63
+ "freeze_scope": "router",
64
+ "ckpt_every": 200,
65
+ "eval_every": 200,
66
+ "log_every": 50,
67
+ "probe_sents": 128,
68
+ "eval_batches": 8,
69
+ "keep_ckpt": false,
70
+ "amp": "bf16",
71
+ "compile": false,
72
+ "cpc_decay": 0.9,
73
+ "device": "cuda",
74
+ "log_spaced_probe": true,
75
+ "probe_bpb_batches": 4,
76
+ "dump_reps": true,
77
+ "dump_reps_thresholds": [
78
+ 0.4,
79
+ 0.5,
80
+ 0.6
81
+ ],
82
+ "router_noise": 0.0,
83
+ "genreg_target_entropy": null,
84
+ "genreg_eta": 0.05
85
+ },
86
+ "probe_agg": {
87
+ "gini_cpc": 0.22451192807644027,
88
+ "gini_cps": 0.22301957393367622,
89
+ "bpic": 5.925929929126198,
90
+ "worst_lang": "zh",
91
+ "cpc_parity_vs_en": {
92
+ "en": 1.0,
93
+ "de": 0.9725837928890361,
94
+ "vi": 1.378765363367736,
95
+ "tr": 1.1427472459455228,
96
+ "fi": 0.9735219498640139,
97
+ "ru": 2.087942793033131,
98
+ "el": 2.0054266863351606,
99
+ "hi": 2.5855259453474537,
100
+ "ta": 2.8326408249054102,
101
+ "th": 2.8393424346249194,
102
+ "zh": 3.0282894619584946,
103
+ "my": 2.5190893307942366
104
+ }
105
+ }
106
+ }
pilot_A_s1/final.json ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "event": "final",
3
+ "run": "pilot_A_s1",
4
+ "condition": "A",
5
+ "step": 6000,
6
+ "params": 22516992,
7
+ "per_lang": {
8
+ "en": {
9
+ "cpc": 0.16594839970120095,
10
+ "cps": 22.5625,
11
+ "bpb": 2.022311981224875,
12
+ "bytes_per_chunk": 5.737196883480697
13
+ },
14
+ "de": {
15
+ "cpc": 0.1613987240052598,
16
+ "cps": 25.890625,
17
+ "bpb": 2.212188749728271,
18
+ "bytes_per_chunk": 6.239741026373417
19
+ },
20
+ "vi": {
21
+ "cpc": 0.2288039056143206,
22
+ "cps": 32.953125,
23
+ "bpb": 1.8670015220171892,
24
+ "bytes_per_chunk": 5.71686530220589
25
+ },
26
+ "tr": {
27
+ "cpc": 0.1896370767276142,
28
+ "cps": 26.8203125,
29
+ "bpb": 2.1998532618483893,
30
+ "bytes_per_chunk": 5.7227934595149215
31
+ },
32
+ "fi": {
33
+ "cpc": 0.1615544096539259,
34
+ "cps": 23.7421875,
35
+ "bpb": 2.395499962296907,
36
+ "bytes_per_chunk": 6.238033481266434
37
+ },
38
+ "ru": {
39
+ "cpc": 0.34649076517150396,
40
+ "cps": 51.296875,
41
+ "bpb": 1.2738760323866076,
42
+ "bytes_per_chunk": 5.32029022274088
43
+ },
44
+ "el": {
45
+ "cpc": 0.3327973493154022,
46
+ "cps": 53.359375,
47
+ "bpb": 1.3333722856930763,
48
+ "bytes_per_chunk": 5.615466181117121
49
+ },
50
+ "hi": {
51
+ "cpc": 0.42906389301634473,
52
+ "cps": 56.3984375,
53
+ "bpb": 1.140567442690789,
54
+ "bytes_per_chunk": 5.7856937915204485
55
+ },
56
+ "ta": {
57
+ "cpc": 0.4700722118213426,
58
+ "cps": 68.65625,
59
+ "bpb": 0.9174700377689008,
60
+ "bytes_per_chunk": 5.817278032976056
61
+ },
62
+ "th": {
63
+ "cpc": 0.47118433322971714,
64
+ "cps": 59.2109375,
65
+ "bpb": 1.0642222156874839,
66
+ "bytes_per_chunk": 6.21260560960292
67
+ },
68
+ "zh": {
69
+ "cpc": 0.502539790044023,
70
+ "cps": 23.1875,
71
+ "bpb": 3.109084046145538,
72
+ "bytes_per_chunk": 5.7431043925950265
73
+ },
74
+ "my": {
75
+ "cpc": 0.41803884314967277,
76
+ "cps": 61.8828125,
77
+ "bpb": 1.0778031266825046,
78
+ "bytes_per_chunk": 6.518967982592478
79
+ }
80
+ },
81
+ "gini_cpc": 0.22451192807644027,
82
+ "gini_cps": 0.22301957393367622,
83
+ "bpic": 5.925929929126198,
84
+ "seed": 1,
85
+ "mean_boundary_entropy": 0.24016138911247253,
86
+ "dechunk": "ema",
87
+ "router_noise": 0.0,
88
+ "transitions": [
89
+ {
90
+ "step": 0,
91
+ "phase": 1,
92
+ "objective": "global",
93
+ "router_trainable": true,
94
+ "encoder_trainable": true,
95
+ "frozen_router_params": 0
96
+ },
97
+ {
98
+ "step": 2400,
99
+ "phase": 2,
100
+ "objective": "global",
101
+ "router_trainable": true,
102
+ "encoder_trainable": true,
103
+ "frozen_router_params": 0
104
+ },
105
+ {
106
+ "step": 4200,
107
+ "phase": 3,
108
+ "objective": "global",
109
+ "router_trainable": true,
110
+ "encoder_trainable": true,
111
+ "frozen_router_params": 0
112
+ }
113
+ ]
114
+ }
pilot_A_s1/log.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
pilot_A_s1/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d75b23ba9fab68082dcfb739b4aeeda17dcf2ff283f1bf84637692a96f97318b
3
+ size 90078296