Add H-Net dynamic-chunking artefacts
Browse filesThis view is limited to 50 files because it contains too many changes. See raw diff
- README.md +67 -0
- baseline_en/config.json +83 -0
- baseline_en/final.json +44 -0
- baseline_en/log.jsonl +136 -0
- baseline_en/model.safetensors +3 -0
- emactl_genreg_s0/config.json +108 -0
- emactl_genreg_s0/final.json +114 -0
- emactl_genreg_s0/log.jsonl +0 -0
- emactl_genreg_s0/model.safetensors +3 -0
- emactl_genreg_s1/config.json +108 -0
- emactl_genreg_s1/final.json +114 -0
- emactl_genreg_s1/log.jsonl +0 -0
- emactl_genreg_s1/model.safetensors +3 -0
- emactl_genreg_s2/config.json +108 -0
- emactl_genreg_s2/final.json +114 -0
- emactl_genreg_s2/log.jsonl +0 -0
- emactl_genreg_s2/model.safetensors +3 -0
- emactl_noema_s0/config.json +108 -0
- emactl_noema_s0/final.json +114 -0
- emactl_noema_s0/log.jsonl +0 -0
- emactl_noema_s0/model.safetensors +3 -0
- emactl_noema_s1/config.json +108 -0
- emactl_noema_s1/final.json +114 -0
- emactl_noema_s1/log.jsonl +0 -0
- emactl_noema_s1/model.safetensors +3 -0
- emactl_noema_s2/config.json +108 -0
- emactl_noema_s2/final.json +114 -0
- emactl_noema_s2/log.jsonl +0 -0
- emactl_noema_s2/model.safetensors +3 -0
- jitter_s0/config.json +108 -0
- jitter_s0/final.json +114 -0
- jitter_s0/log.jsonl +56 -0
- jitter_s0/model.safetensors +3 -0
- jitter_s1/config.json +108 -0
- jitter_s1/final.json +114 -0
- jitter_s1/log.jsonl +56 -0
- jitter_s1/model.safetensors +3 -0
- jitter_s2/config.json +108 -0
- jitter_s2/final.json +114 -0
- jitter_s2/log.jsonl +56 -0
- jitter_s2/model.safetensors +3 -0
- manifest.json +343 -0
- pilot_A_s0/config.json +106 -0
- pilot_A_s0/final.json +114 -0
- pilot_A_s0/log.jsonl +0 -0
- pilot_A_s0/model.safetensors +3 -0
- pilot_A_s1/config.json +106 -0
- pilot_A_s1/final.json +114 -0
- pilot_A_s1/log.jsonl +0 -0
- pilot_A_s1/model.safetensors +3 -0
README.md
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
library_name: pytorch
|
| 4 |
+
tags: [dynamic-chunking, tokenization, hnet, byte-level, interpretability]
|
| 5 |
+
---
|
| 6 |
+
|
| 7 |
+
# H-Net dynamic-chunking pilot models
|
| 8 |
+
|
| 9 |
+
31 byte-level H-Net checkpoints (22.5M params, 6000 steps) trained to study **what a
|
| 10 |
+
learned chunker discovers when it is free to choose its own computational granularity**.
|
| 11 |
+
Every run shares data, seed schedule and FLOP budget; only the chunker's objective varies.
|
| 12 |
+
|
| 13 |
+
## Runs
|
| 14 |
+
|
| 15 |
+
| family | n | what varies |
|
| 16 |
+
|---|---|---|
|
| 17 |
+
| `pilot_A_s{0,1,2}` | 3 | no parity loss — the natural chunker (baseline) |
|
| 18 |
+
| `pilot_B_s{0,1,2}` | 3 | parity loss, per-content-matched-sentence beta |
|
| 19 |
+
| `pilot_C_s{0,1,2}` | 3 | parity applied then removed (reversion arm) |
|
| 20 |
+
| `pilot_D_s{0,1,2}` | 3 | stronger equalisation |
|
| 21 |
+
| `pilot_E_s{0,1,2}` | 3 | entropy-matched variant |
|
| 22 |
+
| `pilot_Apc_s{0,1,2}` | 3 | per-codepoint control |
|
| 23 |
+
| `pilot_Bpc_s{0,1,2}` | 3 | parity loss, per-codepoint beta |
|
| 24 |
+
| `emactl_noema_s{0,1,2}` | 3 | EMA ablation control (built from A) |
|
| 25 |
+
| `emactl_genreg_s{0,1,2}` | 3 | generic-regularisation control, entropy-matched to B |
|
| 26 |
+
| `jitter_s{0,1,2}` | 3 | threshold-jitter training (tau ~ U[0.30, 0.75]) |
|
| 27 |
+
| `baseline_en` | 1 | English-only baseline |
|
| 28 |
+
|
| 29 |
+
## Format
|
| 30 |
+
|
| 31 |
+
Weights are `model.safetensors`; `config.json` carries `model_config`, the training config,
|
| 32 |
+
and a `tied_weights` map. `emb.weight` and `head.weight` are tied, so only `emb.weight` is
|
| 33 |
+
stored — restore with:
|
| 34 |
+
|
| 35 |
+
```python
|
| 36 |
+
from safetensors.torch import load_file
|
| 37 |
+
import json
|
| 38 |
+
sd = load_file("model.safetensors")
|
| 39 |
+
cfg = json.load(open("config.json"))
|
| 40 |
+
for dst, src in cfg["tied_weights"].items():
|
| 41 |
+
sd[dst] = sd[src] # head.weight <- emb.weight
|
| 42 |
+
```
|
| 43 |
+
|
| 44 |
+
Verified bit-exact against the original training checkpoints (`max|Δ| = 0.0`, 111 tensors).
|
| 45 |
+
`final.json` and `log.jsonl` are the trainer's own records.
|
| 46 |
+
|
| 47 |
+
## What these were used to find
|
| 48 |
+
|
| 49 |
+
- Segmentation crystallises **after** capability: the router lags BPB descent in 36/36
|
| 50 |
+
language-seed pairs at half of each clock's own change.
|
| 51 |
+
- Learned units are **not linguistic**. Morpheme F1 sits at or below a rate-matched random
|
| 52 |
+
baseline; gold word+morph features add ≤0.001 held-out AUROC over computational features
|
| 53 |
+
in 9/9 languages.
|
| 54 |
+
- Chunks are **not functionally localised**: aligned-minus-misaligned activation patching is
|
| 55 |
+
at the construction rate (0/28 cells above 2 across-seed SD), while killing the whole chunk
|
| 56 |
+
pathway costs +0.34 nats/byte. Information is additive and diffuse across chunks.
|
| 57 |
+
- The chunk-level main network is worth only **~0.07 BPB** at this scale.
|
| 58 |
+
|
| 59 |
+
Negative results are reported as results. See the companion
|
| 60 |
+
[results dataset](https://huggingface.co/datasets/AdaptiveChunking/hnet-chunking-results)
|
| 61 |
+
and [probe dataset](https://huggingface.co/datasets/AdaptiveChunking/hnet-chunking-probes).
|
| 62 |
+
|
| 63 |
+
## Limitations
|
| 64 |
+
|
| 65 |
+
16.2M/22.5M params, 6000 steps, one hierarchy stage, 3 seeds. Only the final step was
|
| 66 |
+
retained (`keep_ckpt: false`), so no intermediate weights exist. Third-party models used for
|
| 67 |
+
comparison (`allenai/Bolmo-1B`, BLT, ByT5) are **not** redistributed here.
|
baseline_en/config.json
ADDED
|
@@ -0,0 +1,83 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"run": "baseline_en",
|
| 3 |
+
"step": 5000,
|
| 4 |
+
"model_config": {
|
| 5 |
+
"vocab_size": 256,
|
| 6 |
+
"d_enc": 256,
|
| 7 |
+
"d_main": 512,
|
| 8 |
+
"n_enc": 2,
|
| 9 |
+
"n_dec": 2,
|
| 10 |
+
"n_main": 6,
|
| 11 |
+
"n_heads_enc": 4,
|
| 12 |
+
"n_heads_main": 8,
|
| 13 |
+
"seq_len": 1024,
|
| 14 |
+
"dropout": 0.0,
|
| 15 |
+
"target_ratio": 6.0,
|
| 16 |
+
"max_chunk_mult": 3.0,
|
| 17 |
+
"dechunk": "ema",
|
| 18 |
+
"rope_base": 10000.0
|
| 19 |
+
},
|
| 20 |
+
"tied_weights": {
|
| 21 |
+
"head.weight": "emb.weight"
|
| 22 |
+
},
|
| 23 |
+
"train_config": {
|
| 24 |
+
"condition": "A",
|
| 25 |
+
"seed": 0,
|
| 26 |
+
"out_dir": null,
|
| 27 |
+
"run_name": "baseline_en",
|
| 28 |
+
"corpus": "standin",
|
| 29 |
+
"langs": [
|
| 30 |
+
"en"
|
| 31 |
+
],
|
| 32 |
+
"seq_len": 1024,
|
| 33 |
+
"batch_size": 32,
|
| 34 |
+
"grad_accum": 1,
|
| 35 |
+
"steps": 5000,
|
| 36 |
+
"lr": 0.0003,
|
| 37 |
+
"min_lr_frac": 0.1,
|
| 38 |
+
"warmup": 200,
|
| 39 |
+
"weight_decay": 0.1,
|
| 40 |
+
"grad_clip": 1.0,
|
| 41 |
+
"betas": [
|
| 42 |
+
0.9,
|
| 43 |
+
0.95
|
| 44 |
+
],
|
| 45 |
+
"router_lr_mult": 1.0,
|
| 46 |
+
"d_enc": 256,
|
| 47 |
+
"d_main": 512,
|
| 48 |
+
"n_enc": 2,
|
| 49 |
+
"n_dec": 2,
|
| 50 |
+
"n_main": 6,
|
| 51 |
+
"n_heads_enc": 4,
|
| 52 |
+
"n_heads_main": 8,
|
| 53 |
+
"dechunk": "ema",
|
| 54 |
+
"target_ratio": 6.0,
|
| 55 |
+
"alpha": 0.03,
|
| 56 |
+
"alpha_parity_cap": 0.06,
|
| 57 |
+
"global_anchor": 0.0,
|
| 58 |
+
"premium_clip": 4.0,
|
| 59 |
+
"premium_kind": "per_char",
|
| 60 |
+
"tau1": 0.4,
|
| 61 |
+
"tau2": 0.7,
|
| 62 |
+
"freeze_scope": "router",
|
| 63 |
+
"ckpt_every": 500,
|
| 64 |
+
"eval_every": 250,
|
| 65 |
+
"log_every": 50,
|
| 66 |
+
"probe_sents": 96,
|
| 67 |
+
"eval_batches": 8,
|
| 68 |
+
"keep_ckpt": false,
|
| 69 |
+
"amp": "bf16",
|
| 70 |
+
"compile": false,
|
| 71 |
+
"cpc_decay": 0.9,
|
| 72 |
+
"device": "cuda"
|
| 73 |
+
},
|
| 74 |
+
"probe_agg": {
|
| 75 |
+
"gini_cpc": 0.0,
|
| 76 |
+
"gini_cps": 0.0,
|
| 77 |
+
"bpic": 5.065967016491754,
|
| 78 |
+
"worst_lang": "en",
|
| 79 |
+
"cpc_parity_vs_en": {
|
| 80 |
+
"en": 1.0
|
| 81 |
+
}
|
| 82 |
+
}
|
| 83 |
+
}
|
baseline_en/final.json
ADDED
|
@@ -0,0 +1,44 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"event": "final",
|
| 3 |
+
"run": "baseline_en",
|
| 4 |
+
"condition": "A",
|
| 5 |
+
"step": 5000,
|
| 6 |
+
"params": 22516992,
|
| 7 |
+
"per_lang": {
|
| 8 |
+
"en": {
|
| 9 |
+
"cpc": 0.19758572169147598,
|
| 10 |
+
"cps": 27.791666666666668,
|
| 11 |
+
"bpb": 1.623357290677685,
|
| 12 |
+
"bytes_per_chunk": 4.761882271732318
|
| 13 |
+
}
|
| 14 |
+
},
|
| 15 |
+
"gini_cpc": 0.0,
|
| 16 |
+
"gini_cps": 0.0,
|
| 17 |
+
"bpic": 5.065967016491754,
|
| 18 |
+
"transitions": [
|
| 19 |
+
{
|
| 20 |
+
"step": 0,
|
| 21 |
+
"phase": 1,
|
| 22 |
+
"objective": "global",
|
| 23 |
+
"router_trainable": true,
|
| 24 |
+
"encoder_trainable": true,
|
| 25 |
+
"frozen_router_params": 0
|
| 26 |
+
},
|
| 27 |
+
{
|
| 28 |
+
"step": 2000,
|
| 29 |
+
"phase": 2,
|
| 30 |
+
"objective": "global",
|
| 31 |
+
"router_trainable": true,
|
| 32 |
+
"encoder_trainable": true,
|
| 33 |
+
"frozen_router_params": 0
|
| 34 |
+
},
|
| 35 |
+
{
|
| 36 |
+
"step": 3500,
|
| 37 |
+
"phase": 3,
|
| 38 |
+
"objective": "global",
|
| 39 |
+
"router_trainable": true,
|
| 40 |
+
"encoder_trainable": true,
|
| 41 |
+
"frozen_router_params": 0
|
| 42 |
+
}
|
| 43 |
+
]
|
| 44 |
+
}
|
baseline_en/log.jsonl
ADDED
|
@@ -0,0 +1,136 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"event": "start", "run": "baseline_en", "condition": "A", "params": 22516992, "langs": ["en"], "premium": {"en": 1.0}, "schedule": {"tau1": 0.4, "tau2": 0.7, "p1_objective": "global", "p2_objective": "global", "p3_objective": "global", "freeze_in_p3": false, "freeze_scope": "router"}, "corpus": "/local/scratch/sas245/beetle-hnet/scratch/data/tier1_standin", "weights": {"en": 1.0}, "t": 1.3}
|
| 2 |
+
{"event": "probe", "step": 0, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 4884.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.361697, "cps": 50.875, "mean_p": 0.496347}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 2.767404, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 1.5}
|
| 3 |
+
{"event": "train", "step": 0, "phase": 1, "objective": "global", "lm_loss": 5.66441, "bpb": 8.17201, "chunk_loss": 0.040128, "F": 0.30981, "G": 0.49124, "chunks": 317.25, "bytes_per_chunk": 3.228, "grad_norm": 12.0832, "lr": 1.5e-06, "router_trainable": true, "sec_per_step": 0.015, "alpha": 0.03, "t": 2.4}
|
| 4 |
+
{"event": "train", "step": 50, "phase": 1, "objective": "global", "lm_loss": 3.96706, "bpb": 5.72326, "chunk_loss": 0.031702, "F": 0.1907, "G": 0.49023, "chunks": 195.28, "bytes_per_chunk": 5.244, "grad_norm": 2.3639, "lr": 7.65e-05, "router_trainable": true, "sec_per_step": 0.051, "alpha": 0.03, "t": 4.9}
|
| 5 |
+
{"event": "train", "step": 100, "phase": 1, "objective": "global", "lm_loss": 2.93869, "bpb": 4.23963, "chunk_loss": 0.031852, "F": 0.20514, "G": 0.38478, "chunks": 210.06, "bytes_per_chunk": 4.875, "grad_norm": 3.0762, "lr": 0.0001515, "router_trainable": true, "sec_per_step": 0.042, "alpha": 0.03, "t": 7.1}
|
| 6 |
+
{"event": "train", "step": 150, "phase": 1, "objective": "global", "lm_loss": 2.59506, "bpb": 3.74388, "chunk_loss": 0.030458, "F": 0.17642, "G": 0.36688, "chunks": 180.66, "bytes_per_chunk": 5.668, "grad_norm": 1.6857, "lr": 0.0002265, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 9.1}
|
| 7 |
+
{"event": "train", "step": 200, "phase": 1, "objective": "global", "lm_loss": 2.36746, "bpb": 3.41553, "chunk_loss": 0.031466, "F": 0.20374, "G": 0.345, "chunks": 208.62, "bytes_per_chunk": 4.908, "grad_norm": 2.4918, "lr": 0.0003, "router_trainable": true, "sec_per_step": 0.039, "alpha": 0.03, "t": 11.0}
|
| 8 |
+
{"event": "eval", "step": 250, "phase": 1, "bpb": {"en": 3.17185}, "bytes_per_chunk": {"en": 4.69}, "bpb_mean": 3.17185, "t": 13.1}
|
| 9 |
+
{"event": "train", "step": 250, "phase": 1, "objective": "global", "lm_loss": 2.14722, "bpb": 3.09779, "chunk_loss": 0.031083, "F": 0.21036, "G": 0.27837, "chunks": 215.41, "bytes_per_chunk": 4.754, "grad_norm": 1.3424, "lr": 0.0002999, "router_trainable": true, "sec_per_step": 0.042, "alpha": 0.03, "t": 13.1}
|
| 10 |
+
{"event": "train", "step": 300, "phase": 1, "objective": "global", "lm_loss": 2.1025, "bpb": 3.03326, "chunk_loss": 0.031044, "F": 0.21567, "G": 0.26223, "chunks": 220.84, "bytes_per_chunk": 4.637, "grad_norm": 1.2806, "lr": 0.0002997, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 15.1}
|
| 11 |
+
{"event": "train", "step": 350, "phase": 1, "objective": "global", "lm_loss": 2.01715, "bpb": 2.91013, "chunk_loss": 0.03129, "F": 0.2258, "G": 0.26442, "chunks": 231.22, "bytes_per_chunk": 4.429, "grad_norm": 1.1319, "lr": 0.0002993, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 17.1}
|
| 12 |
+
{"event": "train", "step": 400, "phase": 1, "objective": "global", "lm_loss": 1.92618, "bpb": 2.7789, "chunk_loss": 0.031034, "F": 0.22089, "G": 0.25166, "chunks": 226.19, "bytes_per_chunk": 4.527, "grad_norm": 1.4106, "lr": 0.0002988, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 19.2}
|
| 13 |
+
{"event": "train", "step": 450, "phase": 1, "objective": "global", "lm_loss": 1.80237, "bpb": 2.60028, "chunk_loss": 0.030876, "F": 0.21829, "G": 0.24335, "chunks": 223.53, "bytes_per_chunk": 4.581, "grad_norm": 1.3423, "lr": 0.0002982, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 21.2}
|
| 14 |
+
{"event": "probe", "step": 500, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2879.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.213212, "cps": 29.989583, "mean_p": 0.238603}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 4.694686, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 23.2}
|
| 15 |
+
{"event": "eval", "step": 500, "phase": 1, "bpb": {"en": 2.64785}, "bytes_per_chunk": {"en": 4.472}, "bpb_mean": 2.64785, "t": 23.4}
|
| 16 |
+
{"event": "train", "step": 500, "phase": 1, "objective": "global", "lm_loss": 1.76139, "bpb": 2.54115, "chunk_loss": 0.030944, "F": 0.22131, "G": 0.24395, "chunks": 226.62, "bytes_per_chunk": 4.518, "grad_norm": 1.5029, "lr": 0.0002974, "router_trainable": true, "sec_per_step": 0.046, "alpha": 0.03, "t": 23.5}
|
| 17 |
+
{"event": "train", "step": 550, "phase": 1, "objective": "global", "lm_loss": 1.76605, "bpb": 2.54787, "chunk_loss": 0.030994, "F": 0.22476, "G": 0.24278, "chunks": 230.16, "bytes_per_chunk": 4.449, "grad_norm": 1.1989, "lr": 0.0002965, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 25.5}
|
| 18 |
+
{"event": "train", "step": 600, "phase": 1, "objective": "global", "lm_loss": 1.68769, "bpb": 2.43482, "chunk_loss": 0.03111, "F": 0.22791, "G": 0.24679, "chunks": 233.38, "bytes_per_chunk": 4.388, "grad_norm": 1.3439, "lr": 0.0002954, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 27.5}
|
| 19 |
+
{"event": "train", "step": 650, "phase": 1, "objective": "global", "lm_loss": 1.68067, "bpb": 2.4247, "chunk_loss": 0.030982, "F": 0.22586, "G": 0.24037, "chunks": 231.28, "bytes_per_chunk": 4.428, "grad_norm": 1.0878, "lr": 0.0002942, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 29.6}
|
| 20 |
+
{"event": "train", "step": 700, "phase": 1, "objective": "global", "lm_loss": 1.61067, "bpb": 2.32371, "chunk_loss": 0.030945, "F": 0.22562, "G": 0.23685, "chunks": 231.03, "bytes_per_chunk": 4.432, "grad_norm": 1.1877, "lr": 0.0002928, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 31.6}
|
| 21 |
+
{"event": "eval", "step": 750, "phase": 1, "bpb": {"en": 2.43375}, "bytes_per_chunk": {"en": 4.47}, "bpb_mean": 2.43375, "t": 33.8}
|
| 22 |
+
{"event": "train", "step": 750, "phase": 1, "objective": "global", "lm_loss": 1.60258, "bpb": 2.31203, "chunk_loss": 0.030729, "F": 0.21658, "G": 0.23002, "chunks": 221.78, "bytes_per_chunk": 4.617, "grad_norm": 1.2023, "lr": 0.0002913, "router_trainable": true, "sec_per_step": 0.043, "alpha": 0.03, "t": 33.8}
|
| 23 |
+
{"event": "train", "step": 800, "phase": 1, "objective": "global", "lm_loss": 1.62759, "bpb": 2.34812, "chunk_loss": 0.030837, "F": 0.22229, "G": 0.23337, "chunks": 227.62, "bytes_per_chunk": 4.499, "grad_norm": 1.1165, "lr": 0.0002897, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 35.8}
|
| 24 |
+
{"event": "train", "step": 850, "phase": 1, "objective": "global", "lm_loss": 1.63247, "bpb": 2.35516, "chunk_loss": 0.030938, "F": 0.22522, "G": 0.23781, "chunks": 230.62, "bytes_per_chunk": 4.44, "grad_norm": 0.9742, "lr": 0.000288, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 37.9}
|
| 25 |
+
{"event": "train", "step": 900, "phase": 1, "objective": "global", "lm_loss": 1.63538, "bpb": 2.35936, "chunk_loss": 0.030898, "F": 0.22281, "G": 0.2353, "chunks": 228.16, "bytes_per_chunk": 4.488, "grad_norm": 1.0922, "lr": 0.0002861, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 39.9}
|
| 26 |
+
{"event": "train", "step": 950, "phase": 1, "objective": "global", "lm_loss": 1.57397, "bpb": 2.27076, "chunk_loss": 0.030926, "F": 0.22537, "G": 0.23539, "chunks": 230.78, "bytes_per_chunk": 4.437, "grad_norm": 0.9995, "lr": 0.0002841, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 42.0}
|
| 27 |
+
{"event": "probe", "step": 1000, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2886.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.21373, "cps": 30.0625, "mean_p": 0.22402}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 4.683299, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 44.0}
|
| 28 |
+
{"event": "eval", "step": 1000, "phase": 1, "bpb": {"en": 2.32062}, "bytes_per_chunk": {"en": 4.436}, "bpb_mean": 2.32062, "t": 44.2}
|
| 29 |
+
{"event": "train", "step": 1000, "phase": 1, "objective": "global", "lm_loss": 1.60957, "bpb": 2.32212, "chunk_loss": 0.030921, "F": 0.22507, "G": 0.23436, "chunks": 230.47, "bytes_per_chunk": 4.443, "grad_norm": 1.0573, "lr": 0.0002819, "router_trainable": true, "sec_per_step": 0.046, "alpha": 0.03, "t": 44.3}
|
| 30 |
+
{"event": "train", "step": 1050, "phase": 1, "objective": "global", "lm_loss": 1.53486, "bpb": 2.21433, "chunk_loss": 0.03085, "F": 0.22391, "G": 0.23217, "chunks": 229.28, "bytes_per_chunk": 4.466, "grad_norm": 1.0475, "lr": 0.0002796, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 46.3}
|
| 31 |
+
{"event": "train", "step": 1100, "phase": 1, "objective": "global", "lm_loss": 1.5074, "bpb": 2.17472, "chunk_loss": 0.030865, "F": 0.22455, "G": 0.23175, "chunks": 229.94, "bytes_per_chunk": 4.453, "grad_norm": 0.9906, "lr": 0.0002772, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 48.4}
|
| 32 |
+
{"event": "train", "step": 1150, "phase": 1, "objective": "global", "lm_loss": 1.48296, "bpb": 2.13947, "chunk_loss": 0.030705, "F": 0.21826, "G": 0.22673, "chunks": 223.5, "bytes_per_chunk": 4.582, "grad_norm": 0.8906, "lr": 0.0002747, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 50.4}
|
| 33 |
+
{"event": "train", "step": 1200, "phase": 1, "objective": "global", "lm_loss": 1.53887, "bpb": 2.22012, "chunk_loss": 0.030739, "F": 0.21878, "G": 0.22682, "chunks": 224.03, "bytes_per_chunk": 4.571, "grad_norm": 0.8517, "lr": 0.0002721, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 52.5}
|
| 34 |
+
{"event": "eval", "step": 1250, "phase": 1, "bpb": {"en": 2.2299}, "bytes_per_chunk": {"en": 4.506}, "bpb_mean": 2.2299, "t": 54.6}
|
| 35 |
+
{"event": "train", "step": 1250, "phase": 1, "objective": "global", "lm_loss": 1.47402, "bpb": 2.12656, "chunk_loss": 0.03093, "F": 0.22766, "G": 0.23381, "chunks": 233.12, "bytes_per_chunk": 4.392, "grad_norm": 0.8293, "lr": 0.0002694, "router_trainable": true, "sec_per_step": 0.043, "alpha": 0.03, "t": 54.6}
|
| 36 |
+
{"event": "train", "step": 1300, "phase": 1, "objective": "global", "lm_loss": 1.54796, "bpb": 2.23324, "chunk_loss": 0.030811, "F": 0.21948, "G": 0.23125, "chunks": 224.75, "bytes_per_chunk": 4.556, "grad_norm": 0.8633, "lr": 0.0002665, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 56.6}
|
| 37 |
+
{"event": "train", "step": 1350, "phase": 1, "objective": "global", "lm_loss": 1.5163, "bpb": 2.18756, "chunk_loss": 0.030789, "F": 0.21841, "G": 0.22875, "chunks": 223.66, "bytes_per_chunk": 4.578, "grad_norm": 0.838, "lr": 0.0002635, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 58.7}
|
| 38 |
+
{"event": "train", "step": 1400, "phase": 1, "objective": "global", "lm_loss": 1.4602, "bpb": 2.10663, "chunk_loss": 0.030696, "F": 0.21671, "G": 0.22668, "chunks": 221.91, "bytes_per_chunk": 4.615, "grad_norm": 0.8617, "lr": 0.0002605, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 60.7}
|
| 39 |
+
{"event": "train", "step": 1450, "phase": 1, "objective": "global", "lm_loss": 1.47435, "bpb": 2.12704, "chunk_loss": 0.030535, "F": 0.20804, "G": 0.22005, "chunks": 213.03, "bytes_per_chunk": 4.807, "grad_norm": 0.7718, "lr": 0.0002573, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 62.7}
|
| 40 |
+
{"event": "probe", "step": 1500, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2707.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.200474, "cps": 28.197917, "mean_p": 0.213113}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 4.992981, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 64.7}
|
| 41 |
+
{"event": "eval", "step": 1500, "phase": 1, "bpb": {"en": 2.17371}, "bytes_per_chunk": {"en": 4.702}, "bpb_mean": 2.17371, "t": 65.0}
|
| 42 |
+
{"event": "train", "step": 1500, "phase": 1, "objective": "global", "lm_loss": 1.44903, "bpb": 2.09051, "chunk_loss": 0.030599, "F": 0.21173, "G": 0.22301, "chunks": 216.81, "bytes_per_chunk": 4.723, "grad_norm": 0.8913, "lr": 0.000254, "router_trainable": true, "sec_per_step": 0.045, "alpha": 0.03, "t": 65.0}
|
| 43 |
+
{"event": "train", "step": 1550, "phase": 1, "objective": "global", "lm_loss": 1.44784, "bpb": 2.08879, "chunk_loss": 0.030459, "F": 0.20609, "G": 0.21561, "chunks": 211.03, "bytes_per_chunk": 4.852, "grad_norm": 0.8538, "lr": 0.0002506, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 67.0}
|
| 44 |
+
{"event": "train", "step": 1600, "phase": 1, "objective": "global", "lm_loss": 1.49386, "bpb": 2.15519, "chunk_loss": 0.030592, "F": 0.21103, "G": 0.22242, "chunks": 216.09, "bytes_per_chunk": 4.739, "grad_norm": 0.8883, "lr": 0.0002472, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 69.1}
|
| 45 |
+
{"event": "train", "step": 1650, "phase": 1, "objective": "global", "lm_loss": 1.39083, "bpb": 2.00654, "chunk_loss": 0.030619, "F": 0.20911, "G": 0.21716, "chunks": 214.12, "bytes_per_chunk": 4.782, "grad_norm": 0.7979, "lr": 0.0002436, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 71.1}
|
| 46 |
+
{"event": "train", "step": 1700, "phase": 1, "objective": "global", "lm_loss": 1.40329, "bpb": 2.02451, "chunk_loss": 0.030517, "F": 0.20947, "G": 0.21852, "chunks": 214.5, "bytes_per_chunk": 4.774, "grad_norm": 0.7993, "lr": 0.00024, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 73.1}
|
| 47 |
+
{"event": "eval", "step": 1750, "phase": 1, "bpb": {"en": 2.12451}, "bytes_per_chunk": {"en": 4.744}, "bpb_mean": 2.12451, "t": 75.2}
|
| 48 |
+
{"event": "train", "step": 1750, "phase": 1, "objective": "global", "lm_loss": 1.41237, "bpb": 2.03762, "chunk_loss": 0.030544, "F": 0.20901, "G": 0.21875, "chunks": 214.03, "bytes_per_chunk": 4.784, "grad_norm": 0.8677, "lr": 0.0002363, "router_trainable": true, "sec_per_step": 0.043, "alpha": 0.03, "t": 75.2}
|
| 49 |
+
{"event": "train", "step": 1800, "phase": 1, "objective": "global", "lm_loss": 1.39262, "bpb": 2.00913, "chunk_loss": 0.030542, "F": 0.21033, "G": 0.21926, "chunks": 215.38, "bytes_per_chunk": 4.754, "grad_norm": 0.8533, "lr": 0.0002325, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 77.3}
|
| 50 |
+
{"event": "train", "step": 1850, "phase": 1, "objective": "global", "lm_loss": 1.39702, "bpb": 2.01547, "chunk_loss": 0.03052, "F": 0.20917, "G": 0.21769, "chunks": 214.19, "bytes_per_chunk": 4.781, "grad_norm": 0.7715, "lr": 0.0002286, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 79.3}
|
| 51 |
+
{"event": "train", "step": 1900, "phase": 1, "objective": "global", "lm_loss": 1.38118, "bpb": 1.99262, "chunk_loss": 0.030695, "F": 0.21619, "G": 0.22317, "chunks": 221.38, "bytes_per_chunk": 4.626, "grad_norm": 0.7694, "lr": 0.0002247, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 81.3}
|
| 52 |
+
{"event": "train", "step": 1950, "phase": 1, "objective": "global", "lm_loss": 1.33808, "bpb": 1.93044, "chunk_loss": 0.030462, "F": 0.20746, "G": 0.2145, "chunks": 212.44, "bytes_per_chunk": 4.82, "grad_norm": 0.8269, "lr": 0.0002207, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 83.4}
|
| 53 |
+
{"event": "probe", "step": 2000, "phase": 2, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2664.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.197289, "cps": 27.75, "mean_p": 0.208036}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 5.073574, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 85.4}
|
| 54 |
+
{"event": "eval", "step": 2000, "phase": 1, "bpb": {"en": 2.05003}, "bytes_per_chunk": {"en": 4.733}, "bpb_mean": 2.05003, "t": 85.6}
|
| 55 |
+
{"event": "phase_transition", "step": 2000, "phase": 2, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0, "t": 85.6}
|
| 56 |
+
{"event": "train", "step": 2000, "phase": 2, "objective": "global", "lm_loss": 1.394, "bpb": 2.01112, "chunk_loss": 0.030635, "F": 0.21103, "G": 0.21899, "chunks": 216.09, "bytes_per_chunk": 4.739, "grad_norm": 0.7798, "lr": 0.0002167, "router_trainable": true, "sec_per_step": 0.045, "alpha": 0.03, "t": 85.6}
|
| 57 |
+
{"event": "train", "step": 2050, "phase": 2, "objective": "global", "lm_loss": 1.3946, "bpb": 2.01199, "chunk_loss": 0.030557, "F": 0.20926, "G": 0.21781, "chunks": 214.28, "bytes_per_chunk": 4.779, "grad_norm": 0.7824, "lr": 0.0002126, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 87.7}
|
| 58 |
+
{"event": "train", "step": 2100, "phase": 2, "objective": "global", "lm_loss": 1.3849, "bpb": 1.99799, "chunk_loss": 0.030438, "F": 0.20569, "G": 0.21416, "chunks": 210.62, "bytes_per_chunk": 4.862, "grad_norm": 0.8147, "lr": 0.0002084, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 89.7}
|
| 59 |
+
{"event": "train", "step": 2150, "phase": 2, "objective": "global", "lm_loss": 1.40408, "bpb": 2.02565, "chunk_loss": 0.030815, "F": 0.21942, "G": 0.22522, "chunks": 224.69, "bytes_per_chunk": 4.557, "grad_norm": 0.7288, "lr": 0.0002042, "router_trainable": true, "sec_per_step": 0.042, "alpha": 0.03, "t": 91.8}
|
| 60 |
+
{"event": "train", "step": 2200, "phase": 2, "objective": "global", "lm_loss": 1.35356, "bpb": 1.95278, "chunk_loss": 0.03052, "F": 0.20929, "G": 0.21559, "chunks": 214.31, "bytes_per_chunk": 4.778, "grad_norm": 0.8509, "lr": 0.0001999, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 93.8}
|
| 61 |
+
{"event": "eval", "step": 2250, "phase": 2, "bpb": {"en": 1.99956}, "bytes_per_chunk": {"en": 4.682}, "bpb_mean": 1.99956, "t": 95.9}
|
| 62 |
+
{"event": "train", "step": 2250, "phase": 2, "objective": "global", "lm_loss": 1.34643, "bpb": 1.94248, "chunk_loss": 0.030686, "F": 0.21515, "G": 0.22121, "chunks": 220.31, "bytes_per_chunk": 4.648, "grad_norm": 0.7659, "lr": 0.0001957, "router_trainable": true, "sec_per_step": 0.043, "alpha": 0.03, "t": 96.0}
|
| 63 |
+
{"event": "train", "step": 2300, "phase": 2, "objective": "global", "lm_loss": 1.30389, "bpb": 1.88111, "chunk_loss": 0.030484, "F": 0.20728, "G": 0.21505, "chunks": 212.25, "bytes_per_chunk": 4.824, "grad_norm": 0.7552, "lr": 0.0001913, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 98.0}
|
| 64 |
+
{"event": "train", "step": 2350, "phase": 2, "objective": "global", "lm_loss": 1.3369, "bpb": 1.92874, "chunk_loss": 0.030639, "F": 0.21274, "G": 0.21829, "chunks": 217.84, "bytes_per_chunk": 4.701, "grad_norm": 0.7572, "lr": 0.000187, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 100.0}
|
| 65 |
+
{"event": "train", "step": 2400, "phase": 2, "objective": "global", "lm_loss": 1.35317, "bpb": 1.95221, "chunk_loss": 0.030479, "F": 0.20761, "G": 0.2132, "chunks": 212.59, "bytes_per_chunk": 4.817, "grad_norm": 0.7234, "lr": 0.0001826, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 102.1}
|
| 66 |
+
{"event": "train", "step": 2450, "phase": 2, "objective": "global", "lm_loss": 1.3337, "bpb": 1.92413, "chunk_loss": 0.030593, "F": 0.21326, "G": 0.21969, "chunks": 218.38, "bytes_per_chunk": 4.689, "grad_norm": 0.8154, "lr": 0.0001782, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 104.2}
|
| 67 |
+
{"event": "probe", "step": 2500, "phase": 2, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2676.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.198178, "cps": 27.875, "mean_p": 0.206845}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 5.050822, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 106.2}
|
| 68 |
+
{"event": "eval", "step": 2500, "phase": 2, "bpb": {"en": 1.95686}, "bytes_per_chunk": {"en": 4.734}, "bpb_mean": 1.95686, "t": 106.4}
|
| 69 |
+
{"event": "train", "step": 2500, "phase": 2, "objective": "global", "lm_loss": 1.29957, "bpb": 1.87489, "chunk_loss": 0.030607, "F": 0.21335, "G": 0.22061, "chunks": 218.47, "bytes_per_chunk": 4.687, "grad_norm": 0.8269, "lr": 0.0001738, "router_trainable": true, "sec_per_step": 0.046, "alpha": 0.03, "t": 106.5}
|
| 70 |
+
{"event": "train", "step": 2550, "phase": 2, "objective": "global", "lm_loss": 1.32449, "bpb": 1.91083, "chunk_loss": 0.030647, "F": 0.215, "G": 0.22051, "chunks": 220.16, "bytes_per_chunk": 4.651, "grad_norm": 0.8431, "lr": 0.0001694, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 108.5}
|
| 71 |
+
{"event": "train", "step": 2600, "phase": 2, "objective": "global", "lm_loss": 1.28927, "bpb": 1.86003, "chunk_loss": 0.030502, "F": 0.20987, "G": 0.21497, "chunks": 214.91, "bytes_per_chunk": 4.765, "grad_norm": 0.7621, "lr": 0.000165, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 110.5}
|
| 72 |
+
{"event": "train", "step": 2650, "phase": 2, "objective": "global", "lm_loss": 1.26064, "bpb": 1.81873, "chunk_loss": 0.030481, "F": 0.20776, "G": 0.21441, "chunks": 212.75, "bytes_per_chunk": 4.813, "grad_norm": 0.7947, "lr": 0.0001606, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 112.6}
|
| 73 |
+
{"event": "train", "step": 2700, "phase": 2, "objective": "global", "lm_loss": 1.19122, "bpb": 1.71856, "chunk_loss": 0.030482, "F": 0.20706, "G": 0.21302, "chunks": 212.03, "bytes_per_chunk": 4.829, "grad_norm": 0.7729, "lr": 0.0001562, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 114.6}
|
| 74 |
+
{"event": "eval", "step": 2750, "phase": 2, "bpb": {"en": 1.90295}, "bytes_per_chunk": {"en": 4.737}, "bpb_mean": 1.90295, "t": 116.7}
|
| 75 |
+
{"event": "train", "step": 2750, "phase": 2, "objective": "global", "lm_loss": 1.3232, "bpb": 1.90897, "chunk_loss": 0.03067, "F": 0.21368, "G": 0.22047, "chunks": 218.81, "bytes_per_chunk": 4.68, "grad_norm": 0.828, "lr": 0.0001518, "router_trainable": true, "sec_per_step": 0.043, "alpha": 0.03, "t": 116.7}
|
| 76 |
+
{"event": "train", "step": 2800, "phase": 2, "objective": "global", "lm_loss": 1.27056, "bpb": 1.83303, "chunk_loss": 0.030499, "F": 0.20639, "G": 0.21206, "chunks": 211.34, "bytes_per_chunk": 4.845, "grad_norm": 0.7814, "lr": 0.0001474, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 118.8}
|
| 77 |
+
{"event": "train", "step": 2850, "phase": 2, "objective": "global", "lm_loss": 1.28758, "bpb": 1.85758, "chunk_loss": 0.030454, "F": 0.20618, "G": 0.21208, "chunks": 211.12, "bytes_per_chunk": 4.85, "grad_norm": 0.8001, "lr": 0.000143, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 120.8}
|
| 78 |
+
{"event": "train", "step": 2900, "phase": 2, "objective": "global", "lm_loss": 1.24014, "bpb": 1.78915, "chunk_loss": 0.030438, "F": 0.2049, "G": 0.21119, "chunks": 209.81, "bytes_per_chunk": 4.881, "grad_norm": 0.7737, "lr": 0.0001387, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 122.8}
|
| 79 |
+
{"event": "train", "step": 2950, "phase": 2, "objective": "global", "lm_loss": 1.22198, "bpb": 1.76295, "chunk_loss": 0.030534, "F": 0.21124, "G": 0.21671, "chunks": 216.31, "bytes_per_chunk": 4.734, "grad_norm": 0.8122, "lr": 0.0001343, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 124.8}
|
| 80 |
+
{"event": "probe", "step": 3000, "phase": 2, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2676.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.198178, "cps": 27.875, "mean_p": 0.205814}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 5.050822, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 126.8}
|
| 81 |
+
{"event": "eval", "step": 3000, "phase": 2, "bpb": {"en": 1.86797}, "bytes_per_chunk": {"en": 4.732}, "bpb_mean": 1.86797, "t": 127.1}
|
| 82 |
+
{"event": "train", "step": 3000, "phase": 2, "objective": "global", "lm_loss": 1.27078, "bpb": 1.83335, "chunk_loss": 0.030462, "F": 0.2085, "G": 0.21429, "chunks": 213.5, "bytes_per_chunk": 4.796, "grad_norm": 0.8156, "lr": 0.0001301, "router_trainable": true, "sec_per_step": 0.045, "alpha": 0.03, "t": 127.1}
|
| 83 |
+
{"event": "train", "step": 3050, "phase": 2, "objective": "global", "lm_loss": 1.21044, "bpb": 1.7463, "chunk_loss": 0.030514, "F": 0.20969, "G": 0.2161, "chunks": 214.72, "bytes_per_chunk": 4.769, "grad_norm": 0.8593, "lr": 0.0001258, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 129.1}
|
| 84 |
+
{"event": "train", "step": 3100, "phase": 2, "objective": "global", "lm_loss": 1.31147, "bpb": 1.89206, "chunk_loss": 0.030653, "F": 0.21628, "G": 0.2214, "chunks": 221.47, "bytes_per_chunk": 4.624, "grad_norm": 0.8762, "lr": 0.0001216, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 131.2}
|
| 85 |
+
{"event": "train", "step": 3150, "phase": 2, "objective": "global", "lm_loss": 1.29756, "bpb": 1.87199, "chunk_loss": 0.030611, "F": 0.21539, "G": 0.22035, "chunks": 220.56, "bytes_per_chunk": 4.643, "grad_norm": 0.8626, "lr": 0.0001174, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 133.2}
|
| 86 |
+
{"event": "train", "step": 3200, "phase": 2, "objective": "global", "lm_loss": 1.26023, "bpb": 1.81813, "chunk_loss": 0.030586, "F": 0.21399, "G": 0.21982, "chunks": 219.12, "bytes_per_chunk": 4.673, "grad_norm": 0.903, "lr": 0.0001133, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 135.2}
|
| 87 |
+
{"event": "eval", "step": 3250, "phase": 2, "bpb": {"en": 1.82717}, "bytes_per_chunk": {"en": 4.729}, "bpb_mean": 1.82717, "t": 137.4}
|
| 88 |
+
{"event": "train", "step": 3250, "phase": 2, "objective": "global", "lm_loss": 1.23019, "bpb": 1.77479, "chunk_loss": 0.030552, "F": 0.21155, "G": 0.2167, "chunks": 216.62, "bytes_per_chunk": 4.727, "grad_norm": 0.8599, "lr": 0.0001093, "router_trainable": true, "sec_per_step": 0.043, "alpha": 0.03, "t": 137.4}
|
| 89 |
+
{"event": "train", "step": 3300, "phase": 2, "objective": "global", "lm_loss": 1.32118, "bpb": 1.90606, "chunk_loss": 0.030653, "F": 0.21249, "G": 0.21795, "chunks": 217.59, "bytes_per_chunk": 4.706, "grad_norm": 0.8874, "lr": 0.0001053, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 139.5}
|
| 90 |
+
{"event": "train", "step": 3350, "phase": 2, "objective": "global", "lm_loss": 1.19846, "bpb": 1.72901, "chunk_loss": 0.030442, "F": 0.20755, "G": 0.21312, "chunks": 212.53, "bytes_per_chunk": 4.818, "grad_norm": 0.8238, "lr": 0.0001014, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 141.5}
|
| 91 |
+
{"event": "train", "step": 3400, "phase": 2, "objective": "global", "lm_loss": 1.20316, "bpb": 1.7358, "chunk_loss": 0.030587, "F": 0.2128, "G": 0.21877, "chunks": 217.91, "bytes_per_chunk": 4.699, "grad_norm": 0.8561, "lr": 9.75e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 143.6}
|
| 92 |
+
{"event": "train", "step": 3450, "phase": 2, "objective": "global", "lm_loss": 1.24409, "bpb": 1.79485, "chunk_loss": 0.030565, "F": 0.21161, "G": 0.2173, "chunks": 216.69, "bytes_per_chunk": 4.726, "grad_norm": 0.9254, "lr": 9.37e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 145.7}
|
| 93 |
+
{"event": "probe", "step": 3500, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2668.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.197586, "cps": 27.791667, "mean_p": 0.204397}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 5.065967, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 147.7}
|
| 94 |
+
{"event": "eval", "step": 3500, "phase": 2, "bpb": {"en": 1.78841}, "bytes_per_chunk": {"en": 4.742}, "bpb_mean": 1.78841, "t": 147.9}
|
| 95 |
+
{"event": "phase_transition", "step": 3500, "phase": 3, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0, "t": 147.9}
|
| 96 |
+
{"event": "train", "step": 3500, "phase": 3, "objective": "global", "lm_loss": 1.19496, "bpb": 1.72397, "chunk_loss": 0.030473, "F": 0.20816, "G": 0.21329, "chunks": 213.16, "bytes_per_chunk": 4.804, "grad_norm": 0.9069, "lr": 9e-05, "router_trainable": true, "sec_per_step": 0.045, "alpha": 0.03, "t": 147.9}
|
| 97 |
+
{"event": "train", "step": 3550, "phase": 3, "objective": "global", "lm_loss": 1.15301, "bpb": 1.66345, "chunk_loss": 0.030404, "F": 0.20425, "G": 0.21075, "chunks": 209.16, "bytes_per_chunk": 4.896, "grad_norm": 0.9098, "lr": 8.64e-05, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 149.9}
|
| 98 |
+
{"event": "train", "step": 3600, "phase": 3, "objective": "global", "lm_loss": 1.15334, "bpb": 1.66392, "chunk_loss": 0.030414, "F": 0.20581, "G": 0.211, "chunks": 210.75, "bytes_per_chunk": 4.859, "grad_norm": 0.901, "lr": 8.28e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 152.0}
|
| 99 |
+
{"event": "train", "step": 3650, "phase": 3, "objective": "global", "lm_loss": 1.14224, "bpb": 1.6479, "chunk_loss": 0.03045, "F": 0.20596, "G": 0.21129, "chunks": 210.91, "bytes_per_chunk": 4.855, "grad_norm": 0.9649, "lr": 7.94e-05, "router_trainable": true, "sec_per_step": 0.042, "alpha": 0.03, "t": 154.1}
|
| 100 |
+
{"event": "train", "step": 3700, "phase": 3, "objective": "global", "lm_loss": 1.1677, "bpb": 1.68464, "chunk_loss": 0.030596, "F": 0.21204, "G": 0.2175, "chunks": 217.12, "bytes_per_chunk": 4.716, "grad_norm": 0.9024, "lr": 7.6e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 156.1}
|
| 101 |
+
{"event": "eval", "step": 3750, "phase": 3, "bpb": {"en": 1.74671}, "bytes_per_chunk": {"en": 4.75}, "bpb_mean": 1.74671, "t": 158.3}
|
| 102 |
+
{"event": "train", "step": 3750, "phase": 3, "objective": "global", "lm_loss": 1.15668, "bpb": 1.66874, "chunk_loss": 0.030608, "F": 0.2135, "G": 0.21959, "chunks": 218.62, "bytes_per_chunk": 4.684, "grad_norm": 0.898, "lr": 7.27e-05, "router_trainable": true, "sec_per_step": 0.043, "alpha": 0.03, "t": 158.3}
|
| 103 |
+
{"event": "train", "step": 3800, "phase": 3, "objective": "global", "lm_loss": 1.1641, "bpb": 1.67945, "chunk_loss": 0.030531, "F": 0.21155, "G": 0.21752, "chunks": 216.62, "bytes_per_chunk": 4.727, "grad_norm": 0.8836, "lr": 6.95e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 160.3}
|
| 104 |
+
{"event": "train", "step": 3850, "phase": 3, "objective": "global", "lm_loss": 1.22436, "bpb": 1.76637, "chunk_loss": 0.030878, "F": 0.22089, "G": 0.22619, "chunks": 226.19, "bytes_per_chunk": 4.527, "grad_norm": 0.9478, "lr": 6.65e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 162.4}
|
| 105 |
+
{"event": "train", "step": 3900, "phase": 3, "objective": "global", "lm_loss": 1.1527, "bpb": 1.66299, "chunk_loss": 0.030485, "F": 0.20804, "G": 0.21308, "chunks": 213.03, "bytes_per_chunk": 4.807, "grad_norm": 0.9299, "lr": 6.35e-05, "router_trainable": true, "sec_per_step": 0.04, "alpha": 0.03, "t": 164.4}
|
| 106 |
+
{"event": "train", "step": 3950, "phase": 3, "objective": "global", "lm_loss": 1.17791, "bpb": 1.69936, "chunk_loss": 0.03048, "F": 0.20859, "G": 0.21356, "chunks": 213.59, "bytes_per_chunk": 4.794, "grad_norm": 0.9686, "lr": 6.06e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 166.4}
|
| 107 |
+
{"event": "probe", "step": 4000, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2676.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.198178, "cps": 27.875, "mean_p": 0.204176}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 5.050822, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 168.4}
|
| 108 |
+
{"event": "eval", "step": 4000, "phase": 3, "bpb": {"en": 1.72062}, "bytes_per_chunk": {"en": 4.744}, "bpb_mean": 1.72062, "t": 168.7}
|
| 109 |
+
{"event": "train", "step": 4000, "phase": 3, "objective": "global", "lm_loss": 1.15487, "bpb": 1.66613, "chunk_loss": 0.030475, "F": 0.20819, "G": 0.21339, "chunks": 213.19, "bytes_per_chunk": 4.803, "grad_norm": 0.9138, "lr": 5.79e-05, "router_trainable": true, "sec_per_step": 0.046, "alpha": 0.03, "t": 168.7}
|
| 110 |
+
{"event": "train", "step": 4050, "phase": 3, "objective": "global", "lm_loss": 1.11953, "bpb": 1.61514, "chunk_loss": 0.030463, "F": 0.20752, "G": 0.21271, "chunks": 212.5, "bytes_per_chunk": 4.819, "grad_norm": 0.9346, "lr": 5.53e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 170.8}
|
| 111 |
+
{"event": "train", "step": 4100, "phase": 3, "objective": "global", "lm_loss": 1.13507, "bpb": 1.63756, "chunk_loss": 0.030642, "F": 0.2121, "G": 0.2175, "chunks": 217.19, "bytes_per_chunk": 4.715, "grad_norm": 0.9978, "lr": 5.28e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 172.8}
|
| 112 |
+
{"event": "train", "step": 4150, "phase": 3, "objective": "global", "lm_loss": 1.18182, "bpb": 1.705, "chunk_loss": 0.03057, "F": 0.21243, "G": 0.21815, "chunks": 217.53, "bytes_per_chunk": 4.707, "grad_norm": 0.9594, "lr": 5.04e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 174.8}
|
| 113 |
+
{"event": "train", "step": 4200, "phase": 3, "objective": "global", "lm_loss": 1.13783, "bpb": 1.64154, "chunk_loss": 0.030584, "F": 0.21237, "G": 0.21749, "chunks": 217.47, "bytes_per_chunk": 4.709, "grad_norm": 0.9636, "lr": 4.81e-05, "router_trainable": true, "sec_per_step": 0.043, "alpha": 0.03, "t": 177.0}
|
| 114 |
+
{"event": "eval", "step": 4250, "phase": 3, "bpb": {"en": 1.6914}, "bytes_per_chunk": {"en": 4.743}, "bpb_mean": 1.6914, "t": 179.2}
|
| 115 |
+
{"event": "train", "step": 4250, "phase": 3, "objective": "global", "lm_loss": 1.1837, "bpb": 1.70772, "chunk_loss": 0.030569, "F": 0.21277, "G": 0.21797, "chunks": 217.88, "bytes_per_chunk": 4.7, "grad_norm": 0.9683, "lr": 4.59e-05, "router_trainable": true, "sec_per_step": 0.045, "alpha": 0.03, "t": 179.2}
|
| 116 |
+
{"event": "train", "step": 4300, "phase": 3, "objective": "global", "lm_loss": 1.12033, "bpb": 1.61629, "chunk_loss": 0.030478, "F": 0.20578, "G": 0.21147, "chunks": 210.72, "bytes_per_chunk": 4.86, "grad_norm": 0.9732, "lr": 4.39e-05, "router_trainable": true, "sec_per_step": 0.043, "alpha": 0.03, "t": 181.4}
|
| 117 |
+
{"event": "train", "step": 4350, "phase": 3, "objective": "global", "lm_loss": 1.14029, "bpb": 1.64509, "chunk_loss": 0.030692, "F": 0.21616, "G": 0.22163, "chunks": 221.34, "bytes_per_chunk": 4.626, "grad_norm": 0.9484, "lr": 4.2e-05, "router_trainable": true, "sec_per_step": 0.042, "alpha": 0.03, "t": 183.5}
|
| 118 |
+
{"event": "train", "step": 4400, "phase": 3, "objective": "global", "lm_loss": 1.13519, "bpb": 1.63773, "chunk_loss": 0.030559, "F": 0.21228, "G": 0.21784, "chunks": 217.38, "bytes_per_chunk": 4.711, "grad_norm": 1.068, "lr": 4.03e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 185.5}
|
| 119 |
+
{"event": "train", "step": 4450, "phase": 3, "objective": "global", "lm_loss": 1.05593, "bpb": 1.52338, "chunk_loss": 0.030461, "F": 0.2059, "G": 0.21199, "chunks": 210.84, "bytes_per_chunk": 4.857, "grad_norm": 1.0372, "lr": 3.87e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 187.6}
|
| 120 |
+
{"event": "probe", "step": 4500, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2668.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.197586, "cps": 27.791667, "mean_p": 0.204248}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 5.065967, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 189.6}
|
| 121 |
+
{"event": "eval", "step": 4500, "phase": 3, "bpb": {"en": 1.66983}, "bytes_per_chunk": {"en": 4.745}, "bpb_mean": 1.66983, "t": 189.8}
|
| 122 |
+
{"event": "train", "step": 4500, "phase": 3, "objective": "global", "lm_loss": 1.12616, "bpb": 1.62471, "chunk_loss": 0.030513, "F": 0.20874, "G": 0.21445, "chunks": 213.75, "bytes_per_chunk": 4.791, "grad_norm": 1.0024, "lr": 3.72e-05, "router_trainable": true, "sec_per_step": 0.046, "alpha": 0.03, "t": 189.9}
|
| 123 |
+
{"event": "train", "step": 4550, "phase": 3, "objective": "global", "lm_loss": 1.15383, "bpb": 1.66462, "chunk_loss": 0.030495, "F": 0.20779, "G": 0.21295, "chunks": 212.78, "bytes_per_chunk": 4.812, "grad_norm": 1.0579, "lr": 3.58e-05, "router_trainable": true, "sec_per_step": 0.042, "alpha": 0.03, "t": 192.0}
|
| 124 |
+
{"event": "train", "step": 4600, "phase": 3, "objective": "global", "lm_loss": 1.12586, "bpb": 1.62427, "chunk_loss": 0.030521, "F": 0.21118, "G": 0.21647, "chunks": 216.25, "bytes_per_chunk": 4.735, "grad_norm": 1.0163, "lr": 3.46e-05, "router_trainable": true, "sec_per_step": 0.042, "alpha": 0.03, "t": 194.1}
|
| 125 |
+
{"event": "train", "step": 4650, "phase": 3, "objective": "global", "lm_loss": 1.13155, "bpb": 1.63249, "chunk_loss": 0.030552, "F": 0.21106, "G": 0.21661, "chunks": 216.12, "bytes_per_chunk": 4.738, "grad_norm": 1.005, "lr": 3.35e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 196.1}
|
| 126 |
+
{"event": "train", "step": 4700, "phase": 3, "objective": "global", "lm_loss": 1.055, "bpb": 1.52204, "chunk_loss": 0.030565, "F": 0.21112, "G": 0.217, "chunks": 216.19, "bytes_per_chunk": 4.737, "grad_norm": 0.9976, "lr": 3.26e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 198.2}
|
| 127 |
+
{"event": "eval", "step": 4750, "phase": 3, "bpb": {"en": 1.65123}, "bytes_per_chunk": {"en": 4.748}, "bpb_mean": 1.65123, "t": 200.3}
|
| 128 |
+
{"event": "train", "step": 4750, "phase": 3, "objective": "global", "lm_loss": 1.08804, "bpb": 1.56971, "chunk_loss": 0.030496, "F": 0.21048, "G": 0.21598, "chunks": 215.53, "bytes_per_chunk": 4.751, "grad_norm": 0.9788, "lr": 3.18e-05, "router_trainable": true, "sec_per_step": 0.043, "alpha": 0.03, "t": 200.4}
|
| 129 |
+
{"event": "train", "step": 4800, "phase": 3, "objective": "global", "lm_loss": 1.17896, "bpb": 1.70087, "chunk_loss": 0.030509, "F": 0.2067, "G": 0.21234, "chunks": 211.66, "bytes_per_chunk": 4.838, "grad_norm": 1.065, "lr": 3.12e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 202.4}
|
| 130 |
+
{"event": "train", "step": 4850, "phase": 3, "objective": "global", "lm_loss": 1.0219, "bpb": 1.47429, "chunk_loss": 0.030411, "F": 0.20517, "G": 0.21044, "chunks": 210.09, "bytes_per_chunk": 4.874, "grad_norm": 1.0829, "lr": 3.07e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 204.5}
|
| 131 |
+
{"event": "train", "step": 4900, "phase": 3, "objective": "global", "lm_loss": 1.07913, "bpb": 1.55686, "chunk_loss": 0.030668, "F": 0.21558, "G": 0.22083, "chunks": 220.75, "bytes_per_chunk": 4.639, "grad_norm": 1.0028, "lr": 3.03e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 206.6}
|
| 132 |
+
{"event": "train", "step": 4950, "phase": 3, "objective": "global", "lm_loss": 1.1142, "bpb": 1.60745, "chunk_loss": 0.030397, "F": 0.20392, "G": 0.20968, "chunks": 208.81, "bytes_per_chunk": 4.904, "grad_norm": 1.0332, "lr": 3.01e-05, "router_trainable": true, "sec_per_step": 0.041, "alpha": 0.03, "t": 208.6}
|
| 133 |
+
{"event": "probe", "step": 5000, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2668.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.197586, "cps": 27.791667, "mean_p": 0.203833}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 5.065967, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 210.7}
|
| 134 |
+
{"event": "eval", "step": 5000, "phase": 3, "bpb": {"en": 1.62756}, "bytes_per_chunk": {"en": 4.745}, "bpb_mean": 1.62756, "t": 210.9}
|
| 135 |
+
{"event": "probe", "step": 5000, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2668.0, "chars": 13503.0, "bytes": 13516.0, "cpc": 0.197586, "cps": 27.791667, "mean_p": 0.203833}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 5.065967, "worst_lang": "en", "cpc_parity_vs_en": {"en": 1.0}, "t": 210.9}
|
| 136 |
+
{"event": "final", "run": "baseline_en", "condition": "A", "step": 5000, "params": 22516992, "per_lang": {"en": {"cpc": 0.19758572169147598, "cps": 27.791666666666668, "bpb": 1.623357290677685, "bytes_per_chunk": 4.761882271732318}}, "gini_cpc": 0.0, "gini_cps": 0.0, "bpic": 5.065967016491754, "transitions": [{"step": 0, "phase": 1, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}, {"step": 2000, "phase": 2, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}, {"step": 3500, "phase": 3, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}], "t": 211.2}
|
baseline_en/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:76c323e35fc88804fc5994b1284dbb8eaa3cac3dc18e19f1beef8c1d02169405
|
| 3 |
+
size 90078296
|
emactl_genreg_s0/config.json
ADDED
|
@@ -0,0 +1,108 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"run": "emactl_genreg_s0",
|
| 3 |
+
"step": 6000,
|
| 4 |
+
"model_config": {
|
| 5 |
+
"vocab_size": 256,
|
| 6 |
+
"d_enc": 256,
|
| 7 |
+
"d_main": 512,
|
| 8 |
+
"n_enc": 2,
|
| 9 |
+
"n_dec": 2,
|
| 10 |
+
"n_main": 6,
|
| 11 |
+
"n_heads_enc": 4,
|
| 12 |
+
"n_heads_main": 8,
|
| 13 |
+
"seq_len": 1024,
|
| 14 |
+
"dropout": 0.0,
|
| 15 |
+
"target_ratio": 6.0,
|
| 16 |
+
"max_chunk_mult": 3.0,
|
| 17 |
+
"dechunk": "ste",
|
| 18 |
+
"router_noise": 0.0,
|
| 19 |
+
"rope_base": 10000.0
|
| 20 |
+
},
|
| 21 |
+
"tied_weights": {
|
| 22 |
+
"head.weight": "emb.weight"
|
| 23 |
+
},
|
| 24 |
+
"train_config": {
|
| 25 |
+
"condition": "A",
|
| 26 |
+
"seed": 0,
|
| 27 |
+
"out_dir": null,
|
| 28 |
+
"run_name": "emactl_genreg_s0",
|
| 29 |
+
"corpus": "corpus",
|
| 30 |
+
"langs": null,
|
| 31 |
+
"seq_len": 1024,
|
| 32 |
+
"batch_size": 32,
|
| 33 |
+
"grad_accum": 1,
|
| 34 |
+
"steps": 6000,
|
| 35 |
+
"lr": 0.0003,
|
| 36 |
+
"min_lr_frac": 0.1,
|
| 37 |
+
"warmup": 300,
|
| 38 |
+
"weight_decay": 0.1,
|
| 39 |
+
"grad_clip": 1.0,
|
| 40 |
+
"betas": [
|
| 41 |
+
0.9,
|
| 42 |
+
0.95
|
| 43 |
+
],
|
| 44 |
+
"router_lr_mult": 1.0,
|
| 45 |
+
"d_enc": 256,
|
| 46 |
+
"d_main": 512,
|
| 47 |
+
"n_enc": 2,
|
| 48 |
+
"n_dec": 2,
|
| 49 |
+
"n_main": 6,
|
| 50 |
+
"n_heads_enc": 4,
|
| 51 |
+
"n_heads_main": 8,
|
| 52 |
+
"dechunk": "ste",
|
| 53 |
+
"target_ratio": 6.0,
|
| 54 |
+
"alpha": 0.03,
|
| 55 |
+
"alpha_parity_cap": 0.06,
|
| 56 |
+
"global_anchor": 0.0,
|
| 57 |
+
"premium_clip": 4.0,
|
| 58 |
+
"premium_kind": "flores_per_sent",
|
| 59 |
+
"mixture": null,
|
| 60 |
+
"doc_filters": null,
|
| 61 |
+
"tau1": 0.4,
|
| 62 |
+
"tau2": 0.7,
|
| 63 |
+
"freeze_scope": "router",
|
| 64 |
+
"ckpt_every": 200,
|
| 65 |
+
"eval_every": 200,
|
| 66 |
+
"log_every": 50,
|
| 67 |
+
"probe_sents": 128,
|
| 68 |
+
"eval_batches": 8,
|
| 69 |
+
"keep_ckpt": false,
|
| 70 |
+
"amp": "bf16",
|
| 71 |
+
"compile": false,
|
| 72 |
+
"cpc_decay": 0.9,
|
| 73 |
+
"device": "cuda",
|
| 74 |
+
"log_spaced_probe": true,
|
| 75 |
+
"probe_bpb_batches": 4,
|
| 76 |
+
"dump_reps": true,
|
| 77 |
+
"dump_reps_thresholds": [
|
| 78 |
+
0.3,
|
| 79 |
+
0.4,
|
| 80 |
+
0.5,
|
| 81 |
+
0.6,
|
| 82 |
+
0.7
|
| 83 |
+
],
|
| 84 |
+
"router_noise": 0.05,
|
| 85 |
+
"genreg_target_entropy": 0.2983376681804657,
|
| 86 |
+
"genreg_eta": 0.05
|
| 87 |
+
},
|
| 88 |
+
"probe_agg": {
|
| 89 |
+
"gini_cpc": 0.1718704557282591,
|
| 90 |
+
"gini_cps": 0.16855087999202278,
|
| 91 |
+
"bpic": 4.783629156902827,
|
| 92 |
+
"worst_lang": "th",
|
| 93 |
+
"cpc_parity_vs_en": {
|
| 94 |
+
"en": 1.0,
|
| 95 |
+
"de": 1.0620702283220003,
|
| 96 |
+
"vi": 1.1153994423964175,
|
| 97 |
+
"tr": 1.3415190553683638,
|
| 98 |
+
"fi": 1.1170363720974708,
|
| 99 |
+
"ru": 1.4855197692415063,
|
| 100 |
+
"el": 1.5955613423948596,
|
| 101 |
+
"hi": 2.0414684177775593,
|
| 102 |
+
"ta": 1.9938978371357505,
|
| 103 |
+
"th": 2.437200907814152,
|
| 104 |
+
"zh": 2.365709365943872,
|
| 105 |
+
"my": 1.8268620537941986
|
| 106 |
+
}
|
| 107 |
+
}
|
| 108 |
+
}
|
emactl_genreg_s0/final.json
ADDED
|
@@ -0,0 +1,114 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"event": "final",
|
| 3 |
+
"run": "emactl_genreg_s0",
|
| 4 |
+
"condition": "A",
|
| 5 |
+
"step": 6000,
|
| 6 |
+
"params": 22516992,
|
| 7 |
+
"per_lang": {
|
| 8 |
+
"en": {
|
| 9 |
+
"cpc": 0.24656668390507383,
|
| 10 |
+
"cps": 33.5234375,
|
| 11 |
+
"bpb": 2.0362594194956314,
|
| 12 |
+
"bytes_per_chunk": 4.029017582687815
|
| 13 |
+
},
|
| 14 |
+
"de": {
|
| 15 |
+
"cpc": 0.26187113427166026,
|
| 16 |
+
"cps": 42.0078125,
|
| 17 |
+
"bpb": 2.210904942235243,
|
| 18 |
+
"bytes_per_chunk": 3.879821211852115
|
| 19 |
+
},
|
| 20 |
+
"vi": {
|
| 21 |
+
"cpc": 0.27502034174125306,
|
| 22 |
+
"cps": 39.609375,
|
| 23 |
+
"bpb": 1.914473000974849,
|
| 24 |
+
"bytes_per_chunk": 4.687504470352621
|
| 25 |
+
},
|
| 26 |
+
"tr": {
|
| 27 |
+
"cpc": 0.3307739048776446,
|
| 28 |
+
"cps": 46.78125,
|
| 29 |
+
"bpb": 2.163528064484473,
|
| 30 |
+
"bytes_per_chunk": 3.2757763199000314
|
| 31 |
+
},
|
| 32 |
+
"fi": {
|
| 33 |
+
"cpc": 0.2754239540694275,
|
| 34 |
+
"cps": 40.4765625,
|
| 35 |
+
"bpb": 2.387229897058361,
|
| 36 |
+
"bytes_per_chunk": 3.928634050939282
|
| 37 |
+
},
|
| 38 |
+
"ru": {
|
| 39 |
+
"cpc": 0.3662796833773087,
|
| 40 |
+
"cps": 54.2265625,
|
| 41 |
+
"bpb": 1.354086676957211,
|
| 42 |
+
"bytes_per_chunk": 4.78299502805273
|
| 43 |
+
},
|
| 44 |
+
"el": {
|
| 45 |
+
"cpc": 0.39341226916142863,
|
| 46 |
+
"cps": 63.078125,
|
| 47 |
+
"bpb": 1.4254445253235672,
|
| 48 |
+
"bytes_per_chunk": 4.771372927322036
|
| 49 |
+
},
|
| 50 |
+
"hi": {
|
| 51 |
+
"cpc": 0.5033580980683506,
|
| 52 |
+
"cps": 66.1640625,
|
| 53 |
+
"bpb": 1.15441516864525,
|
| 54 |
+
"bytes_per_chunk": 5.213268634157982
|
| 55 |
+
},
|
| 56 |
+
"ta": {
|
| 57 |
+
"cpc": 0.49162877774806096,
|
| 58 |
+
"cps": 71.8046875,
|
| 59 |
+
"bpb": 0.9478137860249648,
|
| 60 |
+
"bytes_per_chunk": 5.832161608970366
|
| 61 |
+
},
|
| 62 |
+
"th": {
|
| 63 |
+
"cpc": 0.600932545850171,
|
| 64 |
+
"cps": 75.515625,
|
| 65 |
+
"bpb": 1.0961344833406215,
|
| 66 |
+
"bytes_per_chunk": 4.624166519668372
|
| 67 |
+
},
|
| 68 |
+
"zh": {
|
| 69 |
+
"cpc": 0.5833051134439553,
|
| 70 |
+
"cps": 26.9140625,
|
| 71 |
+
"bpb": 3.1291131026693577,
|
| 72 |
+
"bytes_per_chunk": 4.784129794048673
|
| 73 |
+
},
|
| 74 |
+
"my": {
|
| 75 |
+
"cpc": 0.45044331855604813,
|
| 76 |
+
"cps": 66.6796875,
|
| 77 |
+
"bpb": 1.0903263282513456,
|
| 78 |
+
"bytes_per_chunk": 6.048406820331787
|
| 79 |
+
}
|
| 80 |
+
},
|
| 81 |
+
"gini_cpc": 0.1718704557282591,
|
| 82 |
+
"gini_cps": 0.16855087999202278,
|
| 83 |
+
"bpic": 4.783629156902827,
|
| 84 |
+
"seed": 0,
|
| 85 |
+
"mean_boundary_entropy": 0.6691246032714844,
|
| 86 |
+
"dechunk": "ste",
|
| 87 |
+
"router_noise": 0.0,
|
| 88 |
+
"transitions": [
|
| 89 |
+
{
|
| 90 |
+
"step": 0,
|
| 91 |
+
"phase": 1,
|
| 92 |
+
"objective": "global",
|
| 93 |
+
"router_trainable": true,
|
| 94 |
+
"encoder_trainable": true,
|
| 95 |
+
"frozen_router_params": 0
|
| 96 |
+
},
|
| 97 |
+
{
|
| 98 |
+
"step": 2400,
|
| 99 |
+
"phase": 2,
|
| 100 |
+
"objective": "global",
|
| 101 |
+
"router_trainable": true,
|
| 102 |
+
"encoder_trainable": true,
|
| 103 |
+
"frozen_router_params": 0
|
| 104 |
+
},
|
| 105 |
+
{
|
| 106 |
+
"step": 4200,
|
| 107 |
+
"phase": 3,
|
| 108 |
+
"objective": "global",
|
| 109 |
+
"router_trainable": true,
|
| 110 |
+
"encoder_trainable": true,
|
| 111 |
+
"frozen_router_params": 0
|
| 112 |
+
}
|
| 113 |
+
]
|
| 114 |
+
}
|
emactl_genreg_s0/log.jsonl
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
emactl_genreg_s0/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0ca2f328d6a03e833f2ea3f19c0389b985ebb5bd6660f43926a52c65cfb50662
|
| 3 |
+
size 90078296
|
emactl_genreg_s1/config.json
ADDED
|
@@ -0,0 +1,108 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"run": "emactl_genreg_s1",
|
| 3 |
+
"step": 6000,
|
| 4 |
+
"model_config": {
|
| 5 |
+
"vocab_size": 256,
|
| 6 |
+
"d_enc": 256,
|
| 7 |
+
"d_main": 512,
|
| 8 |
+
"n_enc": 2,
|
| 9 |
+
"n_dec": 2,
|
| 10 |
+
"n_main": 6,
|
| 11 |
+
"n_heads_enc": 4,
|
| 12 |
+
"n_heads_main": 8,
|
| 13 |
+
"seq_len": 1024,
|
| 14 |
+
"dropout": 0.0,
|
| 15 |
+
"target_ratio": 6.0,
|
| 16 |
+
"max_chunk_mult": 3.0,
|
| 17 |
+
"dechunk": "ste",
|
| 18 |
+
"router_noise": 0.0,
|
| 19 |
+
"rope_base": 10000.0
|
| 20 |
+
},
|
| 21 |
+
"tied_weights": {
|
| 22 |
+
"head.weight": "emb.weight"
|
| 23 |
+
},
|
| 24 |
+
"train_config": {
|
| 25 |
+
"condition": "A",
|
| 26 |
+
"seed": 1,
|
| 27 |
+
"out_dir": null,
|
| 28 |
+
"run_name": "emactl_genreg_s1",
|
| 29 |
+
"corpus": "corpus",
|
| 30 |
+
"langs": null,
|
| 31 |
+
"seq_len": 1024,
|
| 32 |
+
"batch_size": 32,
|
| 33 |
+
"grad_accum": 1,
|
| 34 |
+
"steps": 6000,
|
| 35 |
+
"lr": 0.0003,
|
| 36 |
+
"min_lr_frac": 0.1,
|
| 37 |
+
"warmup": 300,
|
| 38 |
+
"weight_decay": 0.1,
|
| 39 |
+
"grad_clip": 1.0,
|
| 40 |
+
"betas": [
|
| 41 |
+
0.9,
|
| 42 |
+
0.95
|
| 43 |
+
],
|
| 44 |
+
"router_lr_mult": 1.0,
|
| 45 |
+
"d_enc": 256,
|
| 46 |
+
"d_main": 512,
|
| 47 |
+
"n_enc": 2,
|
| 48 |
+
"n_dec": 2,
|
| 49 |
+
"n_main": 6,
|
| 50 |
+
"n_heads_enc": 4,
|
| 51 |
+
"n_heads_main": 8,
|
| 52 |
+
"dechunk": "ste",
|
| 53 |
+
"target_ratio": 6.0,
|
| 54 |
+
"alpha": 0.03,
|
| 55 |
+
"alpha_parity_cap": 0.06,
|
| 56 |
+
"global_anchor": 0.0,
|
| 57 |
+
"premium_clip": 4.0,
|
| 58 |
+
"premium_kind": "flores_per_sent",
|
| 59 |
+
"mixture": null,
|
| 60 |
+
"doc_filters": null,
|
| 61 |
+
"tau1": 0.4,
|
| 62 |
+
"tau2": 0.7,
|
| 63 |
+
"freeze_scope": "router",
|
| 64 |
+
"ckpt_every": 200,
|
| 65 |
+
"eval_every": 200,
|
| 66 |
+
"log_every": 50,
|
| 67 |
+
"probe_sents": 128,
|
| 68 |
+
"eval_batches": 8,
|
| 69 |
+
"keep_ckpt": false,
|
| 70 |
+
"amp": "bf16",
|
| 71 |
+
"compile": false,
|
| 72 |
+
"cpc_decay": 0.9,
|
| 73 |
+
"device": "cuda",
|
| 74 |
+
"log_spaced_probe": true,
|
| 75 |
+
"probe_bpb_batches": 4,
|
| 76 |
+
"dump_reps": true,
|
| 77 |
+
"dump_reps_thresholds": [
|
| 78 |
+
0.3,
|
| 79 |
+
0.4,
|
| 80 |
+
0.5,
|
| 81 |
+
0.6,
|
| 82 |
+
0.7
|
| 83 |
+
],
|
| 84 |
+
"router_noise": 0.05,
|
| 85 |
+
"genreg_target_entropy": 0.2983376681804657,
|
| 86 |
+
"genreg_eta": 0.05
|
| 87 |
+
},
|
| 88 |
+
"probe_agg": {
|
| 89 |
+
"gini_cpc": 0.23636601785882339,
|
| 90 |
+
"gini_cps": 0.23964209275225068,
|
| 91 |
+
"bpic": 5.3453626196080615,
|
| 92 |
+
"worst_lang": "ta",
|
| 93 |
+
"cpc_parity_vs_en": {
|
| 94 |
+
"en": 1.0,
|
| 95 |
+
"de": 0.8729681628237771,
|
| 96 |
+
"vi": 1.3464658214709873,
|
| 97 |
+
"tr": 1.1633108961151755,
|
| 98 |
+
"fi": 0.8345095972200084,
|
| 99 |
+
"ru": 1.9504086084814982,
|
| 100 |
+
"el": 1.1624065248294972,
|
| 101 |
+
"hi": 2.1645924087529833,
|
| 102 |
+
"ta": 3.184870546720561,
|
| 103 |
+
"th": 2.322431752422665,
|
| 104 |
+
"zh": 2.5401997376043726,
|
| 105 |
+
"my": 2.1504683399257214
|
| 106 |
+
}
|
| 107 |
+
}
|
| 108 |
+
}
|
emactl_genreg_s1/final.json
ADDED
|
@@ -0,0 +1,114 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"event": "final",
|
| 3 |
+
"run": "emactl_genreg_s1",
|
| 4 |
+
"condition": "A",
|
| 5 |
+
"step": 6000,
|
| 6 |
+
"params": 22516992,
|
| 7 |
+
"per_lang": {
|
| 8 |
+
"en": {
|
| 9 |
+
"cpc": 0.20703327012584039,
|
| 10 |
+
"cps": 28.1484375,
|
| 11 |
+
"bpb": 2.050497110736466,
|
| 12 |
+
"bytes_per_chunk": 4.915092482351948
|
| 13 |
+
},
|
| 14 |
+
"de": {
|
| 15 |
+
"cpc": 0.18073345346515365,
|
| 16 |
+
"cps": 28.9921875,
|
| 17 |
+
"bpb": 2.234483494945642,
|
| 18 |
+
"bytes_per_chunk": 5.727357140516271
|
| 19 |
+
},
|
| 20 |
+
"vi": {
|
| 21 |
+
"cpc": 0.2787632221318145,
|
| 22 |
+
"cps": 40.1484375,
|
| 23 |
+
"bpb": 1.889786890741809,
|
| 24 |
+
"bytes_per_chunk": 5.128263314911723
|
| 25 |
+
},
|
| 26 |
+
"tr": {
|
| 27 |
+
"cpc": 0.24084405899574657,
|
| 28 |
+
"cps": 34.0625,
|
| 29 |
+
"bpb": 2.200235880250896,
|
| 30 |
+
"bytes_per_chunk": 4.5643048046871595
|
| 31 |
+
},
|
| 32 |
+
"fi": {
|
| 33 |
+
"cpc": 0.17277125086385625,
|
| 34 |
+
"cps": 25.390625,
|
| 35 |
+
"bpb": 2.410347579260111,
|
| 36 |
+
"bytes_per_chunk": 5.786971014812689
|
| 37 |
+
},
|
| 38 |
+
"ru": {
|
| 39 |
+
"cpc": 0.4037994722955145,
|
| 40 |
+
"cps": 59.78125,
|
| 41 |
+
"bpb": 1.3596733775112122,
|
| 42 |
+
"bytes_per_chunk": 4.511013215859031
|
| 43 |
+
},
|
| 44 |
+
"el": {
|
| 45 |
+
"cpc": 0.24065682405106467,
|
| 46 |
+
"cps": 38.5859375,
|
| 47 |
+
"bpb": 1.470116008487108,
|
| 48 |
+
"bytes_per_chunk": 7.593644540358906
|
| 49 |
+
},
|
| 50 |
+
"hi": {
|
| 51 |
+
"cpc": 0.44814264487369987,
|
| 52 |
+
"cps": 58.90625,
|
| 53 |
+
"bpb": 1.1866793342998843,
|
| 54 |
+
"bytes_per_chunk": 5.916870746763872
|
| 55 |
+
},
|
| 56 |
+
"ta": {
|
| 57 |
+
"cpc": 0.6593741642150308,
|
| 58 |
+
"cps": 96.3046875,
|
| 59 |
+
"bpb": 0.9440668483807373,
|
| 60 |
+
"bytes_per_chunk": 4.056543773453519
|
| 61 |
+
},
|
| 62 |
+
"th": {
|
| 63 |
+
"cpc": 0.4808206403481505,
|
| 64 |
+
"cps": 60.421875,
|
| 65 |
+
"bpb": 1.1085081298711636,
|
| 66 |
+
"bytes_per_chunk": 5.815535811342939
|
| 67 |
+
},
|
| 68 |
+
"zh": {
|
| 69 |
+
"cpc": 0.5259058584490349,
|
| 70 |
+
"cps": 24.265625,
|
| 71 |
+
"bpb": 3.1290566063685494,
|
| 72 |
+
"bytes_per_chunk": 5.197248161145145
|
| 73 |
+
},
|
| 74 |
+
"my": {
|
| 75 |
+
"cpc": 0.44521849271690944,
|
| 76 |
+
"cps": 65.90625,
|
| 77 |
+
"bpb": 1.1147818581196989,
|
| 78 |
+
"bytes_per_chunk": 6.384024353120243
|
| 79 |
+
}
|
| 80 |
+
},
|
| 81 |
+
"gini_cpc": 0.23636601785882339,
|
| 82 |
+
"gini_cps": 0.23964209275225068,
|
| 83 |
+
"bpic": 5.3453626196080615,
|
| 84 |
+
"seed": 1,
|
| 85 |
+
"mean_boundary_entropy": 0.6663709878921509,
|
| 86 |
+
"dechunk": "ste",
|
| 87 |
+
"router_noise": 0.0,
|
| 88 |
+
"transitions": [
|
| 89 |
+
{
|
| 90 |
+
"step": 0,
|
| 91 |
+
"phase": 1,
|
| 92 |
+
"objective": "global",
|
| 93 |
+
"router_trainable": true,
|
| 94 |
+
"encoder_trainable": true,
|
| 95 |
+
"frozen_router_params": 0
|
| 96 |
+
},
|
| 97 |
+
{
|
| 98 |
+
"step": 2400,
|
| 99 |
+
"phase": 2,
|
| 100 |
+
"objective": "global",
|
| 101 |
+
"router_trainable": true,
|
| 102 |
+
"encoder_trainable": true,
|
| 103 |
+
"frozen_router_params": 0
|
| 104 |
+
},
|
| 105 |
+
{
|
| 106 |
+
"step": 4200,
|
| 107 |
+
"phase": 3,
|
| 108 |
+
"objective": "global",
|
| 109 |
+
"router_trainable": true,
|
| 110 |
+
"encoder_trainable": true,
|
| 111 |
+
"frozen_router_params": 0
|
| 112 |
+
}
|
| 113 |
+
]
|
| 114 |
+
}
|
emactl_genreg_s1/log.jsonl
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
emactl_genreg_s1/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:06778a23c53100ab5de1587298616275e42e7f0633752789d48bf74145d32df9
|
| 3 |
+
size 90078296
|
emactl_genreg_s2/config.json
ADDED
|
@@ -0,0 +1,108 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"run": "emactl_genreg_s2",
|
| 3 |
+
"step": 6000,
|
| 4 |
+
"model_config": {
|
| 5 |
+
"vocab_size": 256,
|
| 6 |
+
"d_enc": 256,
|
| 7 |
+
"d_main": 512,
|
| 8 |
+
"n_enc": 2,
|
| 9 |
+
"n_dec": 2,
|
| 10 |
+
"n_main": 6,
|
| 11 |
+
"n_heads_enc": 4,
|
| 12 |
+
"n_heads_main": 8,
|
| 13 |
+
"seq_len": 1024,
|
| 14 |
+
"dropout": 0.0,
|
| 15 |
+
"target_ratio": 6.0,
|
| 16 |
+
"max_chunk_mult": 3.0,
|
| 17 |
+
"dechunk": "ste",
|
| 18 |
+
"router_noise": 0.0,
|
| 19 |
+
"rope_base": 10000.0
|
| 20 |
+
},
|
| 21 |
+
"tied_weights": {
|
| 22 |
+
"head.weight": "emb.weight"
|
| 23 |
+
},
|
| 24 |
+
"train_config": {
|
| 25 |
+
"condition": "A",
|
| 26 |
+
"seed": 2,
|
| 27 |
+
"out_dir": null,
|
| 28 |
+
"run_name": "emactl_genreg_s2",
|
| 29 |
+
"corpus": "corpus",
|
| 30 |
+
"langs": null,
|
| 31 |
+
"seq_len": 1024,
|
| 32 |
+
"batch_size": 32,
|
| 33 |
+
"grad_accum": 1,
|
| 34 |
+
"steps": 6000,
|
| 35 |
+
"lr": 0.0003,
|
| 36 |
+
"min_lr_frac": 0.1,
|
| 37 |
+
"warmup": 300,
|
| 38 |
+
"weight_decay": 0.1,
|
| 39 |
+
"grad_clip": 1.0,
|
| 40 |
+
"betas": [
|
| 41 |
+
0.9,
|
| 42 |
+
0.95
|
| 43 |
+
],
|
| 44 |
+
"router_lr_mult": 1.0,
|
| 45 |
+
"d_enc": 256,
|
| 46 |
+
"d_main": 512,
|
| 47 |
+
"n_enc": 2,
|
| 48 |
+
"n_dec": 2,
|
| 49 |
+
"n_main": 6,
|
| 50 |
+
"n_heads_enc": 4,
|
| 51 |
+
"n_heads_main": 8,
|
| 52 |
+
"dechunk": "ste",
|
| 53 |
+
"target_ratio": 6.0,
|
| 54 |
+
"alpha": 0.03,
|
| 55 |
+
"alpha_parity_cap": 0.06,
|
| 56 |
+
"global_anchor": 0.0,
|
| 57 |
+
"premium_clip": 4.0,
|
| 58 |
+
"premium_kind": "flores_per_sent",
|
| 59 |
+
"mixture": null,
|
| 60 |
+
"doc_filters": null,
|
| 61 |
+
"tau1": 0.4,
|
| 62 |
+
"tau2": 0.7,
|
| 63 |
+
"freeze_scope": "router",
|
| 64 |
+
"ckpt_every": 200,
|
| 65 |
+
"eval_every": 200,
|
| 66 |
+
"log_every": 50,
|
| 67 |
+
"probe_sents": 128,
|
| 68 |
+
"eval_batches": 8,
|
| 69 |
+
"keep_ckpt": false,
|
| 70 |
+
"amp": "bf16",
|
| 71 |
+
"compile": false,
|
| 72 |
+
"cpc_decay": 0.9,
|
| 73 |
+
"device": "cuda",
|
| 74 |
+
"log_spaced_probe": true,
|
| 75 |
+
"probe_bpb_batches": 4,
|
| 76 |
+
"dump_reps": true,
|
| 77 |
+
"dump_reps_thresholds": [
|
| 78 |
+
0.3,
|
| 79 |
+
0.4,
|
| 80 |
+
0.5,
|
| 81 |
+
0.6,
|
| 82 |
+
0.7
|
| 83 |
+
],
|
| 84 |
+
"router_noise": 0.05,
|
| 85 |
+
"genreg_target_entropy": 0.2983376681804657,
|
| 86 |
+
"genreg_eta": 0.05
|
| 87 |
+
},
|
| 88 |
+
"probe_agg": {
|
| 89 |
+
"gini_cpc": 0.18418976005934495,
|
| 90 |
+
"gini_cps": 0.19093854335748328,
|
| 91 |
+
"bpic": 5.064610633833485,
|
| 92 |
+
"worst_lang": "th",
|
| 93 |
+
"cpc_parity_vs_en": {
|
| 94 |
+
"en": 1.0,
|
| 95 |
+
"de": 0.9596706497405335,
|
| 96 |
+
"vi": 1.1269902451948222,
|
| 97 |
+
"tr": 1.1782876670438764,
|
| 98 |
+
"fi": 0.9269366133816891,
|
| 99 |
+
"ru": 1.3836420016087154,
|
| 100 |
+
"el": 1.6537874904343381,
|
| 101 |
+
"hi": 1.978828615480954,
|
| 102 |
+
"ta": 1.9833385074132006,
|
| 103 |
+
"th": 2.2785237369270823,
|
| 104 |
+
"zh": 2.270987076557597,
|
| 105 |
+
"my": 2.1670182902343194
|
| 106 |
+
}
|
| 107 |
+
}
|
| 108 |
+
}
|
emactl_genreg_s2/final.json
ADDED
|
@@ -0,0 +1,114 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"event": "final",
|
| 3 |
+
"run": "emactl_genreg_s2",
|
| 4 |
+
"condition": "A",
|
| 5 |
+
"step": 6000,
|
| 6 |
+
"params": 22516992,
|
| 7 |
+
"per_lang": {
|
| 8 |
+
"en": {
|
| 9 |
+
"cpc": 0.23806240303395967,
|
| 10 |
+
"cps": 32.3671875,
|
| 11 |
+
"bpb": 2.0372584237188307,
|
| 12 |
+
"bytes_per_chunk": 4.226799635598481
|
| 13 |
+
},
|
| 14 |
+
"de": {
|
| 15 |
+
"cpc": 0.22846150099839282,
|
| 16 |
+
"cps": 36.6484375,
|
| 17 |
+
"bpb": 2.226351327546341,
|
| 18 |
+
"bytes_per_chunk": 4.347402112804524
|
| 19 |
+
},
|
| 20 |
+
"vi": {
|
| 21 |
+
"cpc": 0.2682940059669108,
|
| 22 |
+
"cps": 38.640625,
|
| 23 |
+
"bpb": 1.8961533875071113,
|
| 24 |
+
"bytes_per_chunk": 4.608639088623618
|
| 25 |
+
},
|
| 26 |
+
"tr": {
|
| 27 |
+
"cpc": 0.2805059934817434,
|
| 28 |
+
"cps": 39.671875,
|
| 29 |
+
"bpb": 2.1642174461915413,
|
| 30 |
+
"bytes_per_chunk": 3.7224202320264688
|
| 31 |
+
},
|
| 32 |
+
"fi": {
|
| 33 |
+
"cpc": 0.22066875764180532,
|
| 34 |
+
"cps": 32.4296875,
|
| 35 |
+
"bpb": 2.3910613802988614,
|
| 36 |
+
"bytes_per_chunk": 4.665106553365662
|
| 37 |
+
},
|
| 38 |
+
"ru": {
|
| 39 |
+
"cpc": 0.32939313984168866,
|
| 40 |
+
"cps": 48.765625,
|
| 41 |
+
"bpb": 1.3742821696914036,
|
| 42 |
+
"bytes_per_chunk": 5.337351114730734
|
| 43 |
+
},
|
| 44 |
+
"el": {
|
| 45 |
+
"cpc": 0.39370462408030016,
|
| 46 |
+
"cps": 63.125,
|
| 47 |
+
"bpb": 1.4151377179209739,
|
| 48 |
+
"bytes_per_chunk": 4.5019878581794135
|
| 49 |
+
},
|
| 50 |
+
"hi": {
|
| 51 |
+
"cpc": 0.4710846953937593,
|
| 52 |
+
"cps": 61.921875,
|
| 53 |
+
"bpb": 1.1513827221751163,
|
| 54 |
+
"bytes_per_chunk": 5.574092581173319
|
| 55 |
+
},
|
| 56 |
+
"ta": {
|
| 57 |
+
"cpc": 0.4721583311045734,
|
| 58 |
+
"cps": 68.9609375,
|
| 59 |
+
"bpb": 0.9525330490836764,
|
| 60 |
+
"bytes_per_chunk": 5.550194255952066
|
| 61 |
+
},
|
| 62 |
+
"th": {
|
| 63 |
+
"cpc": 0.542430836182779,
|
| 64 |
+
"cps": 68.1640625,
|
| 65 |
+
"bpb": 1.090443856895379,
|
| 66 |
+
"bytes_per_chunk": 5.2221480721535505
|
| 67 |
+
},
|
| 68 |
+
"zh": {
|
| 69 |
+
"cpc": 0.5406366407043685,
|
| 70 |
+
"cps": 24.9453125,
|
| 71 |
+
"bpb": 3.1167576970487225,
|
| 72 |
+
"bytes_per_chunk": 5.181940381118052
|
| 73 |
+
},
|
| 74 |
+
"my": {
|
| 75 |
+
"cpc": 0.5158855815917247,
|
| 76 |
+
"cps": 76.3671875,
|
| 77 |
+
"bpb": 1.1004410095350938,
|
| 78 |
+
"bytes_per_chunk": 5.546905913096838
|
| 79 |
+
}
|
| 80 |
+
},
|
| 81 |
+
"gini_cpc": 0.18418976005934495,
|
| 82 |
+
"gini_cps": 0.19093854335748328,
|
| 83 |
+
"bpic": 5.064610633833485,
|
| 84 |
+
"seed": 2,
|
| 85 |
+
"mean_boundary_entropy": 0.6683645248413086,
|
| 86 |
+
"dechunk": "ste",
|
| 87 |
+
"router_noise": 0.0,
|
| 88 |
+
"transitions": [
|
| 89 |
+
{
|
| 90 |
+
"step": 0,
|
| 91 |
+
"phase": 1,
|
| 92 |
+
"objective": "global",
|
| 93 |
+
"router_trainable": true,
|
| 94 |
+
"encoder_trainable": true,
|
| 95 |
+
"frozen_router_params": 0
|
| 96 |
+
},
|
| 97 |
+
{
|
| 98 |
+
"step": 2400,
|
| 99 |
+
"phase": 2,
|
| 100 |
+
"objective": "global",
|
| 101 |
+
"router_trainable": true,
|
| 102 |
+
"encoder_trainable": true,
|
| 103 |
+
"frozen_router_params": 0
|
| 104 |
+
},
|
| 105 |
+
{
|
| 106 |
+
"step": 4200,
|
| 107 |
+
"phase": 3,
|
| 108 |
+
"objective": "global",
|
| 109 |
+
"router_trainable": true,
|
| 110 |
+
"encoder_trainable": true,
|
| 111 |
+
"frozen_router_params": 0
|
| 112 |
+
}
|
| 113 |
+
]
|
| 114 |
+
}
|
emactl_genreg_s2/log.jsonl
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
emactl_genreg_s2/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:bf09af7f57faa2a81c3078a159cc4164075cec8b0cf836a9c10cb9797566592e
|
| 3 |
+
size 90078296
|
emactl_noema_s0/config.json
ADDED
|
@@ -0,0 +1,108 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"run": "emactl_noema_s0",
|
| 3 |
+
"step": 6000,
|
| 4 |
+
"model_config": {
|
| 5 |
+
"vocab_size": 256,
|
| 6 |
+
"d_enc": 256,
|
| 7 |
+
"d_main": 512,
|
| 8 |
+
"n_enc": 2,
|
| 9 |
+
"n_dec": 2,
|
| 10 |
+
"n_main": 6,
|
| 11 |
+
"n_heads_enc": 4,
|
| 12 |
+
"n_heads_main": 8,
|
| 13 |
+
"seq_len": 1024,
|
| 14 |
+
"dropout": 0.0,
|
| 15 |
+
"target_ratio": 6.0,
|
| 16 |
+
"max_chunk_mult": 3.0,
|
| 17 |
+
"dechunk": "ste",
|
| 18 |
+
"router_noise": 0.0,
|
| 19 |
+
"rope_base": 10000.0
|
| 20 |
+
},
|
| 21 |
+
"tied_weights": {
|
| 22 |
+
"head.weight": "emb.weight"
|
| 23 |
+
},
|
| 24 |
+
"train_config": {
|
| 25 |
+
"condition": "A",
|
| 26 |
+
"seed": 0,
|
| 27 |
+
"out_dir": null,
|
| 28 |
+
"run_name": "emactl_noema_s0",
|
| 29 |
+
"corpus": "corpus",
|
| 30 |
+
"langs": null,
|
| 31 |
+
"seq_len": 1024,
|
| 32 |
+
"batch_size": 32,
|
| 33 |
+
"grad_accum": 1,
|
| 34 |
+
"steps": 6000,
|
| 35 |
+
"lr": 0.0003,
|
| 36 |
+
"min_lr_frac": 0.1,
|
| 37 |
+
"warmup": 300,
|
| 38 |
+
"weight_decay": 0.1,
|
| 39 |
+
"grad_clip": 1.0,
|
| 40 |
+
"betas": [
|
| 41 |
+
0.9,
|
| 42 |
+
0.95
|
| 43 |
+
],
|
| 44 |
+
"router_lr_mult": 1.0,
|
| 45 |
+
"d_enc": 256,
|
| 46 |
+
"d_main": 512,
|
| 47 |
+
"n_enc": 2,
|
| 48 |
+
"n_dec": 2,
|
| 49 |
+
"n_main": 6,
|
| 50 |
+
"n_heads_enc": 4,
|
| 51 |
+
"n_heads_main": 8,
|
| 52 |
+
"dechunk": "ste",
|
| 53 |
+
"target_ratio": 6.0,
|
| 54 |
+
"alpha": 0.03,
|
| 55 |
+
"alpha_parity_cap": 0.06,
|
| 56 |
+
"global_anchor": 0.0,
|
| 57 |
+
"premium_clip": 4.0,
|
| 58 |
+
"premium_kind": "flores_per_sent",
|
| 59 |
+
"mixture": null,
|
| 60 |
+
"doc_filters": null,
|
| 61 |
+
"tau1": 0.4,
|
| 62 |
+
"tau2": 0.7,
|
| 63 |
+
"freeze_scope": "router",
|
| 64 |
+
"ckpt_every": 200,
|
| 65 |
+
"eval_every": 200,
|
| 66 |
+
"log_every": 50,
|
| 67 |
+
"probe_sents": 128,
|
| 68 |
+
"eval_batches": 8,
|
| 69 |
+
"keep_ckpt": false,
|
| 70 |
+
"amp": "bf16",
|
| 71 |
+
"compile": false,
|
| 72 |
+
"cpc_decay": 0.9,
|
| 73 |
+
"device": "cuda",
|
| 74 |
+
"log_spaced_probe": true,
|
| 75 |
+
"probe_bpb_batches": 4,
|
| 76 |
+
"dump_reps": true,
|
| 77 |
+
"dump_reps_thresholds": [
|
| 78 |
+
0.3,
|
| 79 |
+
0.4,
|
| 80 |
+
0.5,
|
| 81 |
+
0.6,
|
| 82 |
+
0.7
|
| 83 |
+
],
|
| 84 |
+
"router_noise": 0.0,
|
| 85 |
+
"genreg_target_entropy": null,
|
| 86 |
+
"genreg_eta": 0.05
|
| 87 |
+
},
|
| 88 |
+
"probe_agg": {
|
| 89 |
+
"gini_cpc": 0.18611578592940914,
|
| 90 |
+
"gini_cps": 0.18417783785347885,
|
| 91 |
+
"bpic": 4.7592480065477005,
|
| 92 |
+
"worst_lang": "th",
|
| 93 |
+
"cpc_parity_vs_en": {
|
| 94 |
+
"en": 1.0,
|
| 95 |
+
"de": 1.1355650278292462,
|
| 96 |
+
"vi": 1.1711008424112148,
|
| 97 |
+
"tr": 1.3542351354314985,
|
| 98 |
+
"fi": 1.2376760342670254,
|
| 99 |
+
"ru": 1.630438979246249,
|
| 100 |
+
"el": 1.5757155604814694,
|
| 101 |
+
"hi": 2.1204249628528977,
|
| 102 |
+
"ta": 1.9395870074606907,
|
| 103 |
+
"th": 2.816542017917871,
|
| 104 |
+
"zh": 2.4966017356495547,
|
| 105 |
+
"my": 2.3098610021457
|
| 106 |
+
}
|
| 107 |
+
}
|
| 108 |
+
}
|
emactl_noema_s0/final.json
ADDED
|
@@ -0,0 +1,114 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"event": "final",
|
| 3 |
+
"run": "emactl_noema_s0",
|
| 4 |
+
"condition": "A",
|
| 5 |
+
"step": 6000,
|
| 6 |
+
"params": 22516992,
|
| 7 |
+
"per_lang": {
|
| 8 |
+
"en": {
|
| 9 |
+
"cpc": 0.23099465609377692,
|
| 10 |
+
"cps": 31.40625,
|
| 11 |
+
"bpb": 2.0382534508432286,
|
| 12 |
+
"bytes_per_chunk": 4.302768180288718
|
| 13 |
+
},
|
| 14 |
+
"de": {
|
| 15 |
+
"cpc": 0.26230945307553694,
|
| 16 |
+
"cps": 42.078125,
|
| 17 |
+
"bpb": 2.2124327286163514,
|
| 18 |
+
"bytes_per_chunk": 3.8386598428771204
|
| 19 |
+
},
|
| 20 |
+
"vi": {
|
| 21 |
+
"cpc": 0.27051803634391103,
|
| 22 |
+
"cps": 38.9609375,
|
| 23 |
+
"bpb": 1.9233381799299585,
|
| 24 |
+
"bytes_per_chunk": 4.745675570480733
|
| 25 |
+
},
|
| 26 |
+
"tr": {
|
| 27 |
+
"cpc": 0.3128210793791084,
|
| 28 |
+
"cps": 44.2421875,
|
| 29 |
+
"bpb": 2.1762234583080327,
|
| 30 |
+
"bytes_per_chunk": 3.449739766678291
|
| 31 |
+
},
|
| 32 |
+
"fi": {
|
| 33 |
+
"cpc": 0.2858965498910212,
|
| 34 |
+
"cps": 42.015625,
|
| 35 |
+
"bpb": 2.3890020062928516,
|
| 36 |
+
"bytes_per_chunk": 3.6961775446611114
|
| 37 |
+
},
|
| 38 |
+
"ru": {
|
| 39 |
+
"cpc": 0.376622691292876,
|
| 40 |
+
"cps": 55.7578125,
|
| 41 |
+
"bpb": 1.3563091872564799,
|
| 42 |
+
"bytes_per_chunk": 4.759763958238765
|
| 43 |
+
},
|
| 44 |
+
"el": {
|
| 45 |
+
"cpc": 0.36398187399502996,
|
| 46 |
+
"cps": 58.359375,
|
| 47 |
+
"bpb": 1.431533232486521,
|
| 48 |
+
"bytes_per_chunk": 5.101963760923298
|
| 49 |
+
},
|
| 50 |
+
"hi": {
|
| 51 |
+
"cpc": 0.4898068350668648,
|
| 52 |
+
"cps": 64.3828125,
|
| 53 |
+
"bpb": 1.1639179249083127,
|
| 54 |
+
"bytes_per_chunk": 5.340885244231651
|
| 55 |
+
},
|
| 56 |
+
"ta": {
|
| 57 |
+
"cpc": 0.4480342337523402,
|
| 58 |
+
"cps": 65.4375,
|
| 59 |
+
"bpb": 0.9624026067198248,
|
| 60 |
+
"bytes_per_chunk": 6.288613546676902
|
| 61 |
+
},
|
| 62 |
+
"th": {
|
| 63 |
+
"cpc": 0.6506061548026111,
|
| 64 |
+
"cps": 81.7578125,
|
| 65 |
+
"bpb": 1.0589655444660309,
|
| 66 |
+
"bytes_per_chunk": 4.42780893182893
|
| 67 |
+
},
|
| 68 |
+
"zh": {
|
| 69 |
+
"cpc": 0.5767016593294955,
|
| 70 |
+
"cps": 26.609375,
|
| 71 |
+
"bpb": 3.1217361863324053,
|
| 72 |
+
"bytes_per_chunk": 4.9474200732268905
|
| 73 |
+
},
|
| 74 |
+
"my": {
|
| 75 |
+
"cpc": 0.5335655478150728,
|
| 76 |
+
"cps": 78.984375,
|
| 77 |
+
"bpb": 1.0843777332042663,
|
| 78 |
+
"bytes_per_chunk": 5.2943944580770905
|
| 79 |
+
}
|
| 80 |
+
},
|
| 81 |
+
"gini_cpc": 0.18611578592940914,
|
| 82 |
+
"gini_cps": 0.18417783785347885,
|
| 83 |
+
"bpic": 4.7592480065477005,
|
| 84 |
+
"seed": 0,
|
| 85 |
+
"mean_boundary_entropy": 0.6694673299789429,
|
| 86 |
+
"dechunk": "ste",
|
| 87 |
+
"router_noise": 0.0,
|
| 88 |
+
"transitions": [
|
| 89 |
+
{
|
| 90 |
+
"step": 0,
|
| 91 |
+
"phase": 1,
|
| 92 |
+
"objective": "global",
|
| 93 |
+
"router_trainable": true,
|
| 94 |
+
"encoder_trainable": true,
|
| 95 |
+
"frozen_router_params": 0
|
| 96 |
+
},
|
| 97 |
+
{
|
| 98 |
+
"step": 2400,
|
| 99 |
+
"phase": 2,
|
| 100 |
+
"objective": "global",
|
| 101 |
+
"router_trainable": true,
|
| 102 |
+
"encoder_trainable": true,
|
| 103 |
+
"frozen_router_params": 0
|
| 104 |
+
},
|
| 105 |
+
{
|
| 106 |
+
"step": 4200,
|
| 107 |
+
"phase": 3,
|
| 108 |
+
"objective": "global",
|
| 109 |
+
"router_trainable": true,
|
| 110 |
+
"encoder_trainable": true,
|
| 111 |
+
"frozen_router_params": 0
|
| 112 |
+
}
|
| 113 |
+
]
|
| 114 |
+
}
|
emactl_noema_s0/log.jsonl
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
emactl_noema_s0/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e3e5d1709a39d0e054fa636652f95a102353a4a8cf1f7d5a1366fde46872746b
|
| 3 |
+
size 90078296
|
emactl_noema_s1/config.json
ADDED
|
@@ -0,0 +1,108 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"run": "emactl_noema_s1",
|
| 3 |
+
"step": 6000,
|
| 4 |
+
"model_config": {
|
| 5 |
+
"vocab_size": 256,
|
| 6 |
+
"d_enc": 256,
|
| 7 |
+
"d_main": 512,
|
| 8 |
+
"n_enc": 2,
|
| 9 |
+
"n_dec": 2,
|
| 10 |
+
"n_main": 6,
|
| 11 |
+
"n_heads_enc": 4,
|
| 12 |
+
"n_heads_main": 8,
|
| 13 |
+
"seq_len": 1024,
|
| 14 |
+
"dropout": 0.0,
|
| 15 |
+
"target_ratio": 6.0,
|
| 16 |
+
"max_chunk_mult": 3.0,
|
| 17 |
+
"dechunk": "ste",
|
| 18 |
+
"router_noise": 0.0,
|
| 19 |
+
"rope_base": 10000.0
|
| 20 |
+
},
|
| 21 |
+
"tied_weights": {
|
| 22 |
+
"head.weight": "emb.weight"
|
| 23 |
+
},
|
| 24 |
+
"train_config": {
|
| 25 |
+
"condition": "A",
|
| 26 |
+
"seed": 1,
|
| 27 |
+
"out_dir": null,
|
| 28 |
+
"run_name": "emactl_noema_s1",
|
| 29 |
+
"corpus": "corpus",
|
| 30 |
+
"langs": null,
|
| 31 |
+
"seq_len": 1024,
|
| 32 |
+
"batch_size": 32,
|
| 33 |
+
"grad_accum": 1,
|
| 34 |
+
"steps": 6000,
|
| 35 |
+
"lr": 0.0003,
|
| 36 |
+
"min_lr_frac": 0.1,
|
| 37 |
+
"warmup": 300,
|
| 38 |
+
"weight_decay": 0.1,
|
| 39 |
+
"grad_clip": 1.0,
|
| 40 |
+
"betas": [
|
| 41 |
+
0.9,
|
| 42 |
+
0.95
|
| 43 |
+
],
|
| 44 |
+
"router_lr_mult": 1.0,
|
| 45 |
+
"d_enc": 256,
|
| 46 |
+
"d_main": 512,
|
| 47 |
+
"n_enc": 2,
|
| 48 |
+
"n_dec": 2,
|
| 49 |
+
"n_main": 6,
|
| 50 |
+
"n_heads_enc": 4,
|
| 51 |
+
"n_heads_main": 8,
|
| 52 |
+
"dechunk": "ste",
|
| 53 |
+
"target_ratio": 6.0,
|
| 54 |
+
"alpha": 0.03,
|
| 55 |
+
"alpha_parity_cap": 0.06,
|
| 56 |
+
"global_anchor": 0.0,
|
| 57 |
+
"premium_clip": 4.0,
|
| 58 |
+
"premium_kind": "flores_per_sent",
|
| 59 |
+
"mixture": null,
|
| 60 |
+
"doc_filters": null,
|
| 61 |
+
"tau1": 0.4,
|
| 62 |
+
"tau2": 0.7,
|
| 63 |
+
"freeze_scope": "router",
|
| 64 |
+
"ckpt_every": 200,
|
| 65 |
+
"eval_every": 200,
|
| 66 |
+
"log_every": 50,
|
| 67 |
+
"probe_sents": 128,
|
| 68 |
+
"eval_batches": 8,
|
| 69 |
+
"keep_ckpt": false,
|
| 70 |
+
"amp": "bf16",
|
| 71 |
+
"compile": false,
|
| 72 |
+
"cpc_decay": 0.9,
|
| 73 |
+
"device": "cuda",
|
| 74 |
+
"log_spaced_probe": true,
|
| 75 |
+
"probe_bpb_batches": 4,
|
| 76 |
+
"dump_reps": true,
|
| 77 |
+
"dump_reps_thresholds": [
|
| 78 |
+
0.3,
|
| 79 |
+
0.4,
|
| 80 |
+
0.5,
|
| 81 |
+
0.6,
|
| 82 |
+
0.7
|
| 83 |
+
],
|
| 84 |
+
"router_noise": 0.0,
|
| 85 |
+
"genreg_target_entropy": null,
|
| 86 |
+
"genreg_eta": 0.05
|
| 87 |
+
},
|
| 88 |
+
"probe_agg": {
|
| 89 |
+
"gini_cpc": 0.23693031160314426,
|
| 90 |
+
"gini_cps": 0.24140331101544144,
|
| 91 |
+
"bpic": 5.212078823353659,
|
| 92 |
+
"worst_lang": "ta",
|
| 93 |
+
"cpc_parity_vs_en": {
|
| 94 |
+
"en": 1.0,
|
| 95 |
+
"de": 0.843470152412457,
|
| 96 |
+
"vi": 1.2215531661527395,
|
| 97 |
+
"tr": 1.130235998752015,
|
| 98 |
+
"fi": 0.8248924871143106,
|
| 99 |
+
"ru": 1.611817706976797,
|
| 100 |
+
"el": 1.6979980353874928,
|
| 101 |
+
"hi": 2.016545670210256,
|
| 102 |
+
"ta": 3.3572490033712628,
|
| 103 |
+
"th": 2.1049647560372007,
|
| 104 |
+
"zh": 2.5372298157544013,
|
| 105 |
+
"my": 2.089203249734308
|
| 106 |
+
}
|
| 107 |
+
}
|
| 108 |
+
}
|
emactl_noema_s1/final.json
ADDED
|
@@ -0,0 +1,114 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"event": "final",
|
| 3 |
+
"run": "emactl_noema_s1",
|
| 4 |
+
"condition": "A",
|
| 5 |
+
"step": 6000,
|
| 6 |
+
"params": 22516992,
|
| 7 |
+
"per_lang": {
|
| 8 |
+
"en": {
|
| 9 |
+
"cpc": 0.2142159397804976,
|
| 10 |
+
"cps": 29.125,
|
| 11 |
+
"bpb": 2.0525959332630173,
|
| 12 |
+
"bytes_per_chunk": 4.695102402679395
|
| 13 |
+
},
|
| 14 |
+
"de": {
|
| 15 |
+
"cpc": 0.18068475137583404,
|
| 16 |
+
"cps": 28.984375,
|
| 17 |
+
"bpb": 2.2386571699167805,
|
| 18 |
+
"bytes_per_chunk": 5.84470976444489
|
| 19 |
+
},
|
| 20 |
+
"vi": {
|
| 21 |
+
"cpc": 0.26167615947925144,
|
| 22 |
+
"cps": 37.6875,
|
| 23 |
+
"bpb": 1.8887908317215056,
|
| 24 |
+
"bytes_per_chunk": 5.3526084737110775
|
| 25 |
+
},
|
| 26 |
+
"tr": {
|
| 27 |
+
"cpc": 0.2421145666464122,
|
| 28 |
+
"cps": 34.2421875,
|
| 29 |
+
"bpb": 2.203242577693521,
|
| 30 |
+
"bytes_per_chunk": 4.533363308574937
|
| 31 |
+
},
|
| 32 |
+
"fi": {
|
| 33 |
+
"cpc": 0.17670511934506405,
|
| 34 |
+
"cps": 25.96875,
|
| 35 |
+
"bpb": 2.4121796351973424,
|
| 36 |
+
"bytes_per_chunk": 5.8906778423199215
|
| 37 |
+
},
|
| 38 |
+
"ru": {
|
| 39 |
+
"cpc": 0.34527704485488125,
|
| 40 |
+
"cps": 51.1171875,
|
| 41 |
+
"bpb": 1.367230666027715,
|
| 42 |
+
"bytes_per_chunk": 5.2564943202895495
|
| 43 |
+
},
|
| 44 |
+
"el": {
|
| 45 |
+
"cpc": 0.3637382448959704,
|
| 46 |
+
"cps": 58.3203125,
|
| 47 |
+
"bpb": 1.4419177599239312,
|
| 48 |
+
"bytes_per_chunk": 4.99816961561928
|
| 49 |
+
},
|
| 50 |
+
"hi": {
|
| 51 |
+
"cpc": 0.43197622585438333,
|
| 52 |
+
"cps": 56.78125,
|
| 53 |
+
"bpb": 1.2061648934002307,
|
| 54 |
+
"bytes_per_chunk": 5.775941655374514
|
| 55 |
+
},
|
| 56 |
+
"ta": {
|
| 57 |
+
"cpc": 0.719176250334314,
|
| 58 |
+
"cps": 105.0390625,
|
| 59 |
+
"bpb": 0.932736077772738,
|
| 60 |
+
"bytes_per_chunk": 3.8478723560409236
|
| 61 |
+
},
|
| 62 |
+
"th": {
|
| 63 |
+
"cpc": 0.45091700341933477,
|
| 64 |
+
"cps": 56.6640625,
|
| 65 |
+
"bpb": 1.1032021694968648,
|
| 66 |
+
"bytes_per_chunk": 6.034830852814899
|
| 67 |
+
},
|
| 68 |
+
"zh": {
|
| 69 |
+
"cpc": 0.5435150694209279,
|
| 70 |
+
"cps": 25.078125,
|
| 71 |
+
"bpb": 3.1430077960108775,
|
| 72 |
+
"bytes_per_chunk": 5.576819980428031
|
| 73 |
+
},
|
| 74 |
+
"my": {
|
| 75 |
+
"cpc": 0.4475406375343044,
|
| 76 |
+
"cps": 66.25,
|
| 77 |
+
"bpb": 1.1165069677203832,
|
| 78 |
+
"bytes_per_chunk": 6.349541606617335
|
| 79 |
+
}
|
| 80 |
+
},
|
| 81 |
+
"gini_cpc": 0.23693031160314426,
|
| 82 |
+
"gini_cps": 0.24140331101544144,
|
| 83 |
+
"bpic": 5.212078823353659,
|
| 84 |
+
"seed": 1,
|
| 85 |
+
"mean_boundary_entropy": 0.6675059795379639,
|
| 86 |
+
"dechunk": "ste",
|
| 87 |
+
"router_noise": 0.0,
|
| 88 |
+
"transitions": [
|
| 89 |
+
{
|
| 90 |
+
"step": 0,
|
| 91 |
+
"phase": 1,
|
| 92 |
+
"objective": "global",
|
| 93 |
+
"router_trainable": true,
|
| 94 |
+
"encoder_trainable": true,
|
| 95 |
+
"frozen_router_params": 0
|
| 96 |
+
},
|
| 97 |
+
{
|
| 98 |
+
"step": 2400,
|
| 99 |
+
"phase": 2,
|
| 100 |
+
"objective": "global",
|
| 101 |
+
"router_trainable": true,
|
| 102 |
+
"encoder_trainable": true,
|
| 103 |
+
"frozen_router_params": 0
|
| 104 |
+
},
|
| 105 |
+
{
|
| 106 |
+
"step": 4200,
|
| 107 |
+
"phase": 3,
|
| 108 |
+
"objective": "global",
|
| 109 |
+
"router_trainable": true,
|
| 110 |
+
"encoder_trainable": true,
|
| 111 |
+
"frozen_router_params": 0
|
| 112 |
+
}
|
| 113 |
+
]
|
| 114 |
+
}
|
emactl_noema_s1/log.jsonl
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
emactl_noema_s1/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2c681e106e72a5682753e7d75995581aa5449c5252f5dab6a0cb4750b5c40c00
|
| 3 |
+
size 90078296
|
emactl_noema_s2/config.json
ADDED
|
@@ -0,0 +1,108 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"run": "emactl_noema_s2",
|
| 3 |
+
"step": 6000,
|
| 4 |
+
"model_config": {
|
| 5 |
+
"vocab_size": 256,
|
| 6 |
+
"d_enc": 256,
|
| 7 |
+
"d_main": 512,
|
| 8 |
+
"n_enc": 2,
|
| 9 |
+
"n_dec": 2,
|
| 10 |
+
"n_main": 6,
|
| 11 |
+
"n_heads_enc": 4,
|
| 12 |
+
"n_heads_main": 8,
|
| 13 |
+
"seq_len": 1024,
|
| 14 |
+
"dropout": 0.0,
|
| 15 |
+
"target_ratio": 6.0,
|
| 16 |
+
"max_chunk_mult": 3.0,
|
| 17 |
+
"dechunk": "ste",
|
| 18 |
+
"router_noise": 0.0,
|
| 19 |
+
"rope_base": 10000.0
|
| 20 |
+
},
|
| 21 |
+
"tied_weights": {
|
| 22 |
+
"head.weight": "emb.weight"
|
| 23 |
+
},
|
| 24 |
+
"train_config": {
|
| 25 |
+
"condition": "A",
|
| 26 |
+
"seed": 2,
|
| 27 |
+
"out_dir": null,
|
| 28 |
+
"run_name": "emactl_noema_s2",
|
| 29 |
+
"corpus": "corpus",
|
| 30 |
+
"langs": null,
|
| 31 |
+
"seq_len": 1024,
|
| 32 |
+
"batch_size": 32,
|
| 33 |
+
"grad_accum": 1,
|
| 34 |
+
"steps": 6000,
|
| 35 |
+
"lr": 0.0003,
|
| 36 |
+
"min_lr_frac": 0.1,
|
| 37 |
+
"warmup": 300,
|
| 38 |
+
"weight_decay": 0.1,
|
| 39 |
+
"grad_clip": 1.0,
|
| 40 |
+
"betas": [
|
| 41 |
+
0.9,
|
| 42 |
+
0.95
|
| 43 |
+
],
|
| 44 |
+
"router_lr_mult": 1.0,
|
| 45 |
+
"d_enc": 256,
|
| 46 |
+
"d_main": 512,
|
| 47 |
+
"n_enc": 2,
|
| 48 |
+
"n_dec": 2,
|
| 49 |
+
"n_main": 6,
|
| 50 |
+
"n_heads_enc": 4,
|
| 51 |
+
"n_heads_main": 8,
|
| 52 |
+
"dechunk": "ste",
|
| 53 |
+
"target_ratio": 6.0,
|
| 54 |
+
"alpha": 0.03,
|
| 55 |
+
"alpha_parity_cap": 0.06,
|
| 56 |
+
"global_anchor": 0.0,
|
| 57 |
+
"premium_clip": 4.0,
|
| 58 |
+
"premium_kind": "flores_per_sent",
|
| 59 |
+
"mixture": null,
|
| 60 |
+
"doc_filters": null,
|
| 61 |
+
"tau1": 0.4,
|
| 62 |
+
"tau2": 0.7,
|
| 63 |
+
"freeze_scope": "router",
|
| 64 |
+
"ckpt_every": 200,
|
| 65 |
+
"eval_every": 200,
|
| 66 |
+
"log_every": 50,
|
| 67 |
+
"probe_sents": 128,
|
| 68 |
+
"eval_batches": 8,
|
| 69 |
+
"keep_ckpt": false,
|
| 70 |
+
"amp": "bf16",
|
| 71 |
+
"compile": false,
|
| 72 |
+
"cpc_decay": 0.9,
|
| 73 |
+
"device": "cuda",
|
| 74 |
+
"log_spaced_probe": true,
|
| 75 |
+
"probe_bpb_batches": 4,
|
| 76 |
+
"dump_reps": true,
|
| 77 |
+
"dump_reps_thresholds": [
|
| 78 |
+
0.3,
|
| 79 |
+
0.4,
|
| 80 |
+
0.5,
|
| 81 |
+
0.6,
|
| 82 |
+
0.7
|
| 83 |
+
],
|
| 84 |
+
"router_noise": 0.0,
|
| 85 |
+
"genreg_target_entropy": null,
|
| 86 |
+
"genreg_eta": 0.05
|
| 87 |
+
},
|
| 88 |
+
"probe_agg": {
|
| 89 |
+
"gini_cpc": 0.2097257925277538,
|
| 90 |
+
"gini_cps": 0.2205780004871769,
|
| 91 |
+
"bpic": 5.008103664265581,
|
| 92 |
+
"worst_lang": "th",
|
| 93 |
+
"cpc_parity_vs_en": {
|
| 94 |
+
"en": 1.0,
|
| 95 |
+
"de": 1.0122183493242607,
|
| 96 |
+
"vi": 1.0386824563703136,
|
| 97 |
+
"tr": 1.1756329079876167,
|
| 98 |
+
"fi": 1.0197810206310258,
|
| 99 |
+
"ru": 1.4561345082792843,
|
| 100 |
+
"el": 1.4873135488871023,
|
| 101 |
+
"hi": 2.3510164171708303,
|
| 102 |
+
"ta": 1.9467053999262376,
|
| 103 |
+
"th": 2.9241466408623515,
|
| 104 |
+
"zh": 2.069107906247045,
|
| 105 |
+
"my": 2.3244289486237197
|
| 106 |
+
}
|
| 107 |
+
}
|
| 108 |
+
}
|
emactl_noema_s2/final.json
ADDED
|
@@ -0,0 +1,114 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"event": "final",
|
| 3 |
+
"run": "emactl_noema_s2",
|
| 4 |
+
"condition": "A",
|
| 5 |
+
"step": 6000,
|
| 6 |
+
"params": 22516992,
|
| 7 |
+
"per_lang": {
|
| 8 |
+
"en": {
|
| 9 |
+
"cpc": 0.22863874044704935,
|
| 10 |
+
"cps": 31.0859375,
|
| 11 |
+
"bpb": 2.0440929497706244,
|
| 12 |
+
"bytes_per_chunk": 4.447226675488375
|
| 13 |
+
},
|
| 14 |
+
"de": {
|
| 15 |
+
"cpc": 0.23143232844689038,
|
| 16 |
+
"cps": 37.125,
|
| 17 |
+
"bpb": 2.229200510378607,
|
| 18 |
+
"bytes_per_chunk": 4.252719353033265
|
| 19 |
+
},
|
| 20 |
+
"vi": {
|
| 21 |
+
"cpc": 0.2374830485489558,
|
| 22 |
+
"cps": 34.203125,
|
| 23 |
+
"bpb": 1.9064757324673185,
|
| 24 |
+
"bytes_per_chunk": 5.540809316973674
|
| 25 |
+
},
|
| 26 |
+
"tr": {
|
| 27 |
+
"cpc": 0.2687952273103905,
|
| 28 |
+
"cps": 38.015625,
|
| 29 |
+
"bpb": 2.1740123956039623,
|
| 30 |
+
"bytes_per_chunk": 3.9388166000540914
|
| 31 |
+
},
|
| 32 |
+
"fi": {
|
| 33 |
+
"cpc": 0.23316144808888417,
|
| 34 |
+
"cps": 34.265625,
|
| 35 |
+
"bpb": 2.3927762408084456,
|
| 36 |
+
"bytes_per_chunk": 4.315731419210918
|
| 37 |
+
},
|
| 38 |
+
"ru": {
|
| 39 |
+
"cpc": 0.3329287598944591,
|
| 40 |
+
"cps": 49.2890625,
|
| 41 |
+
"bpb": 1.3761500732624246,
|
| 42 |
+
"bytes_per_chunk": 5.276809887577121
|
| 43 |
+
},
|
| 44 |
+
"el": {
|
| 45 |
+
"cpc": 0.34005749646737804,
|
| 46 |
+
"cps": 54.5234375,
|
| 47 |
+
"bpb": 1.4288364316573776,
|
| 48 |
+
"bytes_per_chunk": 5.19611496531219
|
| 49 |
+
},
|
| 50 |
+
"hi": {
|
| 51 |
+
"cpc": 0.5375334323922734,
|
| 52 |
+
"cps": 70.65625,
|
| 53 |
+
"bpb": 1.15970682221278,
|
| 54 |
+
"bytes_per_chunk": 4.935961889698545
|
| 55 |
+
},
|
| 56 |
+
"ta": {
|
| 57 |
+
"cpc": 0.4450922706606044,
|
| 58 |
+
"cps": 65.0078125,
|
| 59 |
+
"bpb": 0.9492078075249067,
|
| 60 |
+
"bytes_per_chunk": 5.968602361084232
|
| 61 |
+
},
|
| 62 |
+
"th": {
|
| 63 |
+
"cpc": 0.6685732048492384,
|
| 64 |
+
"cps": 84.015625,
|
| 65 |
+
"bpb": 1.0939795622231419,
|
| 66 |
+
"bytes_per_chunk": 4.236396838992227
|
| 67 |
+
},
|
| 68 |
+
"zh": {
|
| 69 |
+
"cpc": 0.47307822553335593,
|
| 70 |
+
"cps": 21.828125,
|
| 71 |
+
"bpb": 3.1189088345478835,
|
| 72 |
+
"bytes_per_chunk": 5.605319990591656
|
| 73 |
+
},
|
| 74 |
+
"my": {
|
| 75 |
+
"cpc": 0.5314545070719865,
|
| 76 |
+
"cps": 78.671875,
|
| 77 |
+
"bpb": 1.1001802838363859,
|
| 78 |
+
"bytes_per_chunk": 5.20814169489505
|
| 79 |
+
}
|
| 80 |
+
},
|
| 81 |
+
"gini_cpc": 0.2097257925277538,
|
| 82 |
+
"gini_cps": 0.2205780004871769,
|
| 83 |
+
"bpic": 5.008103664265581,
|
| 84 |
+
"seed": 2,
|
| 85 |
+
"mean_boundary_entropy": 0.6694104671478271,
|
| 86 |
+
"dechunk": "ste",
|
| 87 |
+
"router_noise": 0.0,
|
| 88 |
+
"transitions": [
|
| 89 |
+
{
|
| 90 |
+
"step": 0,
|
| 91 |
+
"phase": 1,
|
| 92 |
+
"objective": "global",
|
| 93 |
+
"router_trainable": true,
|
| 94 |
+
"encoder_trainable": true,
|
| 95 |
+
"frozen_router_params": 0
|
| 96 |
+
},
|
| 97 |
+
{
|
| 98 |
+
"step": 2400,
|
| 99 |
+
"phase": 2,
|
| 100 |
+
"objective": "global",
|
| 101 |
+
"router_trainable": true,
|
| 102 |
+
"encoder_trainable": true,
|
| 103 |
+
"frozen_router_params": 0
|
| 104 |
+
},
|
| 105 |
+
{
|
| 106 |
+
"step": 4200,
|
| 107 |
+
"phase": 3,
|
| 108 |
+
"objective": "global",
|
| 109 |
+
"router_trainable": true,
|
| 110 |
+
"encoder_trainable": true,
|
| 111 |
+
"frozen_router_params": 0
|
| 112 |
+
}
|
| 113 |
+
]
|
| 114 |
+
}
|
emactl_noema_s2/log.jsonl
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
emactl_noema_s2/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5ee3493e20c110048713670e7c4ff21fb2e5def328ec33bdf7129f24a8348882
|
| 3 |
+
size 90078296
|
jitter_s0/config.json
ADDED
|
@@ -0,0 +1,108 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"run": "jitter_s0",
|
| 3 |
+
"step": 6000,
|
| 4 |
+
"model_config": {
|
| 5 |
+
"vocab_size": 256,
|
| 6 |
+
"d_enc": 256,
|
| 7 |
+
"d_main": 512,
|
| 8 |
+
"n_enc": 2,
|
| 9 |
+
"n_dec": 2,
|
| 10 |
+
"n_main": 6,
|
| 11 |
+
"n_heads_enc": 4,
|
| 12 |
+
"n_heads_main": 8,
|
| 13 |
+
"seq_len": 1024,
|
| 14 |
+
"dropout": 0.0,
|
| 15 |
+
"target_ratio": 6.0,
|
| 16 |
+
"max_chunk_mult": 3.0,
|
| 17 |
+
"dechunk": "ema",
|
| 18 |
+
"router_noise": 0.0,
|
| 19 |
+
"rope_base": 10000.0
|
| 20 |
+
},
|
| 21 |
+
"tied_weights": {
|
| 22 |
+
"head.weight": "emb.weight"
|
| 23 |
+
},
|
| 24 |
+
"train_config": {
|
| 25 |
+
"condition": "A",
|
| 26 |
+
"seed": 0,
|
| 27 |
+
"out_dir": null,
|
| 28 |
+
"run_name": "jitter_s0",
|
| 29 |
+
"corpus": "corpus",
|
| 30 |
+
"langs": null,
|
| 31 |
+
"seq_len": 1024,
|
| 32 |
+
"batch_size": 32,
|
| 33 |
+
"grad_accum": 1,
|
| 34 |
+
"steps": 6000,
|
| 35 |
+
"lr": 0.0003,
|
| 36 |
+
"min_lr_frac": 0.1,
|
| 37 |
+
"warmup": 300,
|
| 38 |
+
"weight_decay": 0.1,
|
| 39 |
+
"grad_clip": 1.0,
|
| 40 |
+
"betas": [
|
| 41 |
+
0.9,
|
| 42 |
+
0.95
|
| 43 |
+
],
|
| 44 |
+
"router_lr_mult": 1.0,
|
| 45 |
+
"d_enc": 256,
|
| 46 |
+
"d_main": 512,
|
| 47 |
+
"n_enc": 2,
|
| 48 |
+
"n_dec": 2,
|
| 49 |
+
"n_main": 6,
|
| 50 |
+
"n_heads_enc": 4,
|
| 51 |
+
"n_heads_main": 8,
|
| 52 |
+
"dechunk": "ema",
|
| 53 |
+
"target_ratio": 6.0,
|
| 54 |
+
"alpha": 0.03,
|
| 55 |
+
"alpha_parity_cap": 0.06,
|
| 56 |
+
"global_anchor": 0.0,
|
| 57 |
+
"premium_clip": 4.0,
|
| 58 |
+
"premium_kind": "flores_per_sent",
|
| 59 |
+
"mixture": null,
|
| 60 |
+
"doc_filters": null,
|
| 61 |
+
"tau1": 0.4,
|
| 62 |
+
"tau2": 0.7,
|
| 63 |
+
"freeze_scope": "router",
|
| 64 |
+
"ckpt_every": 2000,
|
| 65 |
+
"eval_every": 1000,
|
| 66 |
+
"log_every": 200,
|
| 67 |
+
"probe_sents": 128,
|
| 68 |
+
"eval_batches": 8,
|
| 69 |
+
"keep_ckpt": false,
|
| 70 |
+
"amp": "bf16",
|
| 71 |
+
"compile": false,
|
| 72 |
+
"cpc_decay": 0.9,
|
| 73 |
+
"device": "cuda",
|
| 74 |
+
"log_spaced_probe": true,
|
| 75 |
+
"probe_bpb_batches": 4,
|
| 76 |
+
"dump_reps": true,
|
| 77 |
+
"dump_reps_thresholds": [
|
| 78 |
+
0.3,
|
| 79 |
+
0.4,
|
| 80 |
+
0.5,
|
| 81 |
+
0.6,
|
| 82 |
+
0.7
|
| 83 |
+
],
|
| 84 |
+
"router_noise": 0.0,
|
| 85 |
+
"genreg_target_entropy": null,
|
| 86 |
+
"genreg_eta": 0.05
|
| 87 |
+
},
|
| 88 |
+
"probe_agg": {
|
| 89 |
+
"gini_cpc": 0.17269931081856482,
|
| 90 |
+
"gini_cps": 0.21141326974754948,
|
| 91 |
+
"bpic": 43.57681389803565,
|
| 92 |
+
"worst_lang": "my",
|
| 93 |
+
"cpc_parity_vs_en": {
|
| 94 |
+
"en": 1.0,
|
| 95 |
+
"de": 1.013006652705401,
|
| 96 |
+
"vi": 1.1146982587469487,
|
| 97 |
+
"tr": 1.3166408219632104,
|
| 98 |
+
"fi": 0.7608435064589868,
|
| 99 |
+
"ru": 1.0388534881266491,
|
| 100 |
+
"el": 1.1247546265165909,
|
| 101 |
+
"hi": 0.7430796552748885,
|
| 102 |
+
"ta": 0.8876972880449316,
|
| 103 |
+
"th": 1.469706882188374,
|
| 104 |
+
"zh": 1.3059616661022686,
|
| 105 |
+
"my": 2.380651044965168
|
| 106 |
+
}
|
| 107 |
+
}
|
| 108 |
+
}
|
jitter_s0/final.json
ADDED
|
@@ -0,0 +1,114 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"event": "final",
|
| 3 |
+
"run": "jitter_s0",
|
| 4 |
+
"condition": "A",
|
| 5 |
+
"step": 6000,
|
| 6 |
+
"params": 22516992,
|
| 7 |
+
"per_lang": {
|
| 8 |
+
"en": {
|
| 9 |
+
"cpc": 0.03591334827328622,
|
| 10 |
+
"cps": 4.8828125,
|
| 11 |
+
"bpb": 2.102690803330191,
|
| 12 |
+
"bytes_per_chunk": 33.77491464278812
|
| 13 |
+
},
|
| 14 |
+
"de": {
|
| 15 |
+
"cpc": 0.036380460721764964,
|
| 16 |
+
"cps": 5.8359375,
|
| 17 |
+
"bpb": 2.271415199876741,
|
| 18 |
+
"bytes_per_chunk": 32.133366021083596
|
| 19 |
+
},
|
| 20 |
+
"vi": {
|
| 21 |
+
"cpc": 0.04003254678600488,
|
| 22 |
+
"cps": 5.765625,
|
| 23 |
+
"bpb": 1.951459073921452,
|
| 24 |
+
"bytes_per_chunk": 40.595276809910956
|
| 25 |
+
},
|
| 26 |
+
"tr": {
|
| 27 |
+
"cpc": 0.04728498038999061,
|
| 28 |
+
"cps": 6.6875,
|
| 29 |
+
"bpb": 2.250277080982085,
|
| 30 |
+
"bytes_per_chunk": 30.134958041154157
|
| 31 |
+
},
|
| 32 |
+
"fi": {
|
| 33 |
+
"cpc": 0.027324437828929883,
|
| 34 |
+
"cps": 4.015625,
|
| 35 |
+
"bpb": 2.4655344794072835,
|
| 36 |
+
"bytes_per_chunk": 49.16890180999719
|
| 37 |
+
},
|
| 38 |
+
"ru": {
|
| 39 |
+
"cpc": 0.037308707124010555,
|
| 40 |
+
"cps": 5.5234375,
|
| 41 |
+
"bpb": 1.4088588409418283,
|
| 42 |
+
"bytes_per_chunk": 58.11861212725862
|
| 43 |
+
},
|
| 44 |
+
"el": {
|
| 45 |
+
"cpc": 0.0403937046240803,
|
| 46 |
+
"cps": 6.4765625,
|
| 47 |
+
"bpb": 1.471620706197338,
|
| 48 |
+
"bytes_per_chunk": 56.95068433630241
|
| 49 |
+
},
|
| 50 |
+
"hi": {
|
| 51 |
+
"cpc": 0.026686478454680536,
|
| 52 |
+
"cps": 3.5078125,
|
| 53 |
+
"bpb": 1.2017601651121157,
|
| 54 |
+
"bytes_per_chunk": 111.29017193801741
|
| 55 |
+
},
|
| 56 |
+
"ta": {
|
| 57 |
+
"cpc": 0.031880181866809305,
|
| 58 |
+
"cps": 4.65625,
|
| 59 |
+
"bpb": 0.9874921775028731,
|
| 60 |
+
"bytes_per_chunk": 111.97949594190517
|
| 61 |
+
},
|
| 62 |
+
"th": {
|
| 63 |
+
"cpc": 0.052782095119676714,
|
| 64 |
+
"cps": 6.6328125,
|
| 65 |
+
"bpb": 1.1332099012743477,
|
| 66 |
+
"bytes_per_chunk": 47.20338525254344
|
| 67 |
+
},
|
| 68 |
+
"zh": {
|
| 69 |
+
"cpc": 0.0469014561462919,
|
| 70 |
+
"cps": 2.1640625,
|
| 71 |
+
"bpb": 3.1557357805467756,
|
| 72 |
+
"bytes_per_chunk": 94.7394289844597
|
| 73 |
+
},
|
| 74 |
+
"my": {
|
| 75 |
+
"cpc": 0.08549715009499684,
|
| 76 |
+
"cps": 12.65625,
|
| 77 |
+
"bpb": 1.1230400189341303,
|
| 78 |
+
"bytes_per_chunk": 33.42607586866433
|
| 79 |
+
}
|
| 80 |
+
},
|
| 81 |
+
"gini_cpc": 0.17269931081856482,
|
| 82 |
+
"gini_cps": 0.21141326974754948,
|
| 83 |
+
"bpic": 43.57681389803565,
|
| 84 |
+
"seed": 0,
|
| 85 |
+
"mean_boundary_entropy": 0.6520750522613525,
|
| 86 |
+
"dechunk": "ema",
|
| 87 |
+
"router_noise": 0.0,
|
| 88 |
+
"transitions": [
|
| 89 |
+
{
|
| 90 |
+
"step": 0,
|
| 91 |
+
"phase": 1,
|
| 92 |
+
"objective": "global",
|
| 93 |
+
"router_trainable": true,
|
| 94 |
+
"encoder_trainable": true,
|
| 95 |
+
"frozen_router_params": 0
|
| 96 |
+
},
|
| 97 |
+
{
|
| 98 |
+
"step": 2400,
|
| 99 |
+
"phase": 2,
|
| 100 |
+
"objective": "global",
|
| 101 |
+
"router_trainable": true,
|
| 102 |
+
"encoder_trainable": true,
|
| 103 |
+
"frozen_router_params": 0
|
| 104 |
+
},
|
| 105 |
+
{
|
| 106 |
+
"step": 4200,
|
| 107 |
+
"phase": 3,
|
| 108 |
+
"objective": "global",
|
| 109 |
+
"router_trainable": true,
|
| 110 |
+
"encoder_trainable": true,
|
| 111 |
+
"frozen_router_params": 0
|
| 112 |
+
}
|
| 113 |
+
]
|
| 114 |
+
}
|
jitter_s0/log.jsonl
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"event": "start", "run": "jitter_s0", "condition": "A", "params": 22516992, "langs": ["en", "de", "vi", "tr", "fi", "ru", "el", "hi", "ta", "th", "zh", "my"], "premium": {"en": 1.0, "de": 1.184, "vi": 1.39, "tr": 1.12, "fi": 1.104, "ru": 1.971, "el": 2.174, "hi": 2.552, "ta": 3.192, "th": 2.762, "zh": 0.915, "my": 3.524}, "schedule": {"tau1": 0.4, "tau2": 0.7, "p1_objective": "global", "p2_objective": "global", "p3_objective": "global", "freeze_in_p3": false, "freeze_scope": "router"}, "corpus": "/local/scratch/sas245/beetle-hnet/scratch/data/tier1_corpus", "premium_kind": "flores_per_sent", "doc_filtered": {"my": 37511}, "weights": {"en": 0.2946, "de": 0.0909, "vi": 0.0576, "tr": 0.0612, "fi": 0.0424, "ru": 0.1476, "el": 0.0522, "hi": 0.042, "ta": 0.0313, "th": 0.0542, "zh": 0.1038, "my": 0.0222}, "t": 1.7}
|
| 2 |
+
{"event": "probe_schedule", "n": 13, "steps": [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 2000, 4000, 6000], "t": 1.7}
|
| 3 |
+
{"event": "probe", "step": 0, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 6384.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.366833, "cps": 49.875, "mean_p": 0.496725, "boundary_entropy": 0.686995, "bpb": 8.160238}, "de": {"chunks": 8628.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.420202, "cps": 67.40625, "mean_p": 0.498889, "boundary_entropy": 0.687821, "bpb": 8.156329}, "vi": {"chunks": 13115.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.711418, "cps": 102.460938, "mean_p": 0.505745, "boundary_entropy": 0.688385, "bpb": 8.087291}, "tr": {"chunks": 5290.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.292217, "cps": 41.328125, "mean_p": 0.489567, "boundary_entropy": 0.687288, "bpb": 8.171434}, "fi": {"chunks": 8895.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.472862, "cps": 69.492188, "mean_p": 0.500688, "boundary_entropy": 0.687512, "bpb": 8.214758}, "ru": {"chunks": 10744.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.566966, "cps": 83.9375, "mean_p": 0.493655, "boundary_entropy": 0.689893, "bpb": 8.099397}, "el": {"chunks": 15039.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.732788, "cps": 117.492188, "mean_p": 0.497482, "boundary_entropy": 0.690234, "bpb": 8.158044}, "hi": {"chunks": 2563.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.152333, "cps": 20.023438, "mean_p": 0.484638, "boundary_entropy": 0.690256, "bpb": 8.262162}, "ta": {"chunks": 38089.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 2.03739, "cps": 297.570312, "mean_p": 0.512976, "boundary_entropy": 0.690378, "bpb": 8.051803}, "th": {"chunks": 34006.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 2.114144, "cps": 265.671875, "mean_p": 0.511916, "boundary_entropy": 0.690552, "bpb": 8.248459}, "zh": {"chunks": 11821.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 2.001524, "cps": 92.351562, "mean_p": 0.52004, "boundary_entropy": 0.685674, "bpb": 8.123118}, "my": {"chunks": 48201.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 2.543857, "cps": 376.570312, "mean_p": 0.521563, "boundary_entropy": 0.690206, "bpb": 7.913994}}, "bpb": {"en": 8.16024, "de": 8.15633, "vi": 8.08729, "tr": 8.17143, "fi": 8.21476, "ru": 8.0994, "el": 8.15804, "hi": 8.26216, "ta": 8.0518, "th": 8.24846, "zh": 8.12312, "my": 7.91399}, "gini_cpc": 0.424496, "gini_cps": 0.425905, "bpic": 1.892645, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 1.1454838515955792, "vi": 1.939350894526797, "tr": 0.7965927738475616, "fi": 1.2890367847879758, "ru": 1.5455676790921895, "el": 1.997603802482379, "hi": 0.41526447061189375, "ta": 5.553993192060118, "th": 5.7632270176529286, "zh": 5.456221801328583, "my": 6.934639901884223}, "t": 5.1}
|
| 4 |
+
{"event": "train", "step": 0, "phase": 1, "objective": "global", "lm_loss": 5.64186, "bpb": 8.13949, "chunk_loss": 0.08991, "F": 1.0, "G": 0.4995, "chunks": 1024.0, "bytes_per_chunk": 1.0, "boundary_entropy": 0.69158, "router_noise": 0.0, "grad_norm": 7.8228, "lr": 1e-06, "router_trainable": true, "sec_per_step": 0.002, "alpha": 0.03, "t": 5.5}
|
| 5 |
+
{"event": "probe", "step": 1, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 6292.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.361547, "cps": 49.15625, "mean_p": 0.496424, "boundary_entropy": 0.686985, "bpb": 8.137942}, "de": {"chunks": 8544.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.416111, "cps": 66.75, "mean_p": 0.498658, "boundary_entropy": 0.687817, "bpb": 8.13676}, "vi": {"chunks": 12993.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.704801, "cps": 101.507812, "mean_p": 0.505487, "boundary_entropy": 0.688387, "bpb": 8.072181}, "tr": {"chunks": 5217.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.288184, "cps": 40.757812, "mean_p": 0.489318, "boundary_entropy": 0.687274, "bpb": 8.156829}, "fi": {"chunks": 8811.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.468396, "cps": 68.835938, "mean_p": 0.500401, "boundary_entropy": 0.687508, "bpb": 8.196416}, "ru": {"chunks": 10431.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.550449, "cps": 81.492188, "mean_p": 0.493206, "boundary_entropy": 0.689879, "bpb": 8.060083}, "el": {"chunks": 15001.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.730936, "cps": 117.195312, "mean_p": 0.497429, "boundary_entropy": 0.690233, "bpb": 8.140897}, "hi": {"chunks": 2503.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.148767, "cps": 19.554688, "mean_p": 0.484452, "boundary_entropy": 0.690243, "bpb": 8.249504}, "ta": {"chunks": 37861.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 2.025194, "cps": 295.789062, "mean_p": 0.512705, "boundary_entropy": 0.690391, "bpb": 8.028624}, "th": {"chunks": 33861.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 2.105129, "cps": 264.539062, "mean_p": 0.511768, "boundary_entropy": 0.690558, "bpb": 8.229615}, "zh": {"chunks": 11822.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 2.001693, "cps": 92.359375, "mean_p": 0.520019, "boundary_entropy": 0.685675, "bpb": 8.120583}, "my": {"chunks": 48181.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 2.542801, "cps": 376.414062, "mean_p": 0.521547, "boundary_entropy": 0.690207, "bpb": 7.893555}}, "bpb": {"en": 8.13794, "de": 8.13676, "vi": 8.07218, "tr": 8.15683, "fi": 8.19642, "ru": 8.06008, "el": 8.1409, "hi": 8.2495, "ta": 8.02862, "th": 8.22962, "zh": 8.12058, "my": 7.89355}, "gini_cpc": 0.426673, "gini_cps": 0.428193, "bpic": 1.90446, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 1.150917619502558, "vi": 1.9494033261656607, "tr": 0.7970869365552818, "fi": 1.2955337284054524, "ru": 1.52248189685105, "el": 2.021690973974, "hi": 0.41147284837275383, "ta": 5.601470037900537, "th": 5.822561987423376, "zh": 5.5364695874270105, "my": 7.033116960044964}, "t": 8.4}
|
| 6 |
+
{"event": "probe", "step": 2, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 6140.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.352813, "cps": 47.96875, "mean_p": 0.495813, "boundary_entropy": 0.686963, "bpb": 8.088626}, "de": {"chunks": 8384.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.408318, "cps": 65.5, "mean_p": 0.498195, "boundary_entropy": 0.687808, "bpb": 8.092933}, "vi": {"chunks": 12767.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.692541, "cps": 99.742188, "mean_p": 0.504896, "boundary_entropy": 0.688391, "bpb": 8.038432}, "tr": {"chunks": 5083.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.280782, "cps": 39.710938, "mean_p": 0.488774, "boundary_entropy": 0.687241, "bpb": 8.122514}, "fi": {"chunks": 8653.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.459997, "cps": 67.601562, "mean_p": 0.499815, "boundary_entropy": 0.687498, "bpb": 8.154597}, "ru": {"chunks": 9871.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.520897, "cps": 77.117188, "mean_p": 0.492489, "boundary_entropy": 0.689854, "bpb": 7.994827}, "el": {"chunks": 15018.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.731764, "cps": 117.328125, "mean_p": 0.497432, "boundary_entropy": 0.690233, "bpb": 8.120639}, "hi": {"chunks": 2362.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.140386, "cps": 18.453125, "mean_p": 0.484026, "boundary_entropy": 0.690214, "bpb": 8.216665}, "ta": {"chunks": 37462.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 2.003851, "cps": 292.671875, "mean_p": 0.512226, "boundary_entropy": 0.690413, "bpb": 7.987519}, "th": {"chunks": 33572.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 2.087162, "cps": 262.28125, "mean_p": 0.51154, "boundary_entropy": 0.690567, "bpb": 8.202526}, "zh": {"chunks": 11810.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.999661, "cps": 92.265625, "mean_p": 0.519969, "boundary_entropy": 0.685678, "bpb": 8.114145}, "my": {"chunks": 48128.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 2.540004, "cps": 376.0, "mean_p": 0.521493, "boundary_entropy": 0.69021, "bpb": 7.860478}}, "bpb": {"en": 8.08863, "de": 8.09293, "vi": 8.03843, "tr": 8.12251, "fi": 8.1546, "ru": 7.99483, "el": 8.12064, "hi": 8.21666, "ta": 7.98752, "th": 8.20253, "zh": 8.11415, "my": 7.86048}, "gini_cpc": 0.430739, "gini_cps": 0.432586, "bpic": 1.926128, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 1.15732307300348, "vi": 1.9629148721319472, "tr": 0.7958391639156395, "fi": 1.3037987770340904, "ru": 1.4764124087904908, "el": 2.0740871646191503, "hi": 0.39790607470076617, "ta": 5.679645622817159, "th": 5.915778412525478, "zh": 5.667769816153395, "my": 7.1992986118703755}, "t": 11.5}
|
| 7 |
+
{"event": "probe", "step": 4, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 5580.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.320634, "cps": 43.59375, "mean_p": 0.493699, "boundary_entropy": 0.686875, "bpb": 7.919777}, "de": {"chunks": 7847.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.382165, "cps": 61.304688, "mean_p": 0.496587, "boundary_entropy": 0.68777, "bpb": 7.9397}, "vi": {"chunks": 12023.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.652183, "cps": 93.929688, "mean_p": 0.50296, "boundary_entropy": 0.688395, "bpb": 7.926445}, "tr": {"chunks": 4607.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.254488, "cps": 35.992188, "mean_p": 0.486967, "boundary_entropy": 0.687126, "bpb": 8.005026}, "fi": {"chunks": 7997.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.425124, "cps": 62.476562, "mean_p": 0.497685, "boundary_entropy": 0.687452, "bpb": 8.007853}, "ru": {"chunks": 8139.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.429499, "cps": 63.585938, "mean_p": 0.490058, "boundary_entropy": 0.689757, "bpb": 7.777456}, "el": {"chunks": 14842.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.723189, "cps": 115.953125, "mean_p": 0.49725, "boundary_entropy": 0.69023, "bpb": 8.031972}, "hi": {"chunks": 1961.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.116553, "cps": 15.320312, "mean_p": 0.482238, "boundary_entropy": 0.690083, "bpb": 8.094255}, "ta": {"chunks": 35822.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 1.916127, "cps": 279.859375, "mean_p": 0.510362, "boundary_entropy": 0.690494, "bpb": 7.840834}, "th": {"chunks": 32414.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 2.015169, "cps": 253.234375, "mean_p": 0.510574, "boundary_entropy": 0.690602, "bpb": 8.088093}, "zh": {"chunks": 11765.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.992042, "cps": 91.914062, "mean_p": 0.519704, "boundary_entropy": 0.685693, "bpb": 8.087489}, "my": {"chunks": 47965.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 2.531402, "cps": 374.726562, "mean_p": 0.521282, "boundary_entropy": 0.690224, "bpb": 7.784123}}, "bpb": {"en": 7.91978, "de": 7.9397, "vi": 7.92645, "tr": 8.00503, "fi": 8.00785, "ru": 7.77746, "el": 8.03197, "hi": 8.09426, "ta": 7.84083, "th": 8.08809, "zh": 8.08749, "my": 7.78412}, "gini_cpc": 0.445746, "gini_cps": 0.447513, "bpic": 2.009724, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 1.191903696593315, "vi": 2.034040642653205, "tr": 0.7937021963329313, "fi": 1.3258827910688267, "ru": 1.3395278747127415, "el": 2.255493102676829, "hi": 0.36350671843295146, "ta": 5.976050809974492, "th": 6.284945033274171, "zh": 6.212814833830368, "my": 7.894979269806896}, "t": 14.6}
|
| 8 |
+
{"event": "probe", "step": 8, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 4774.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.274321, "cps": 37.296875, "mean_p": 0.490456, "boundary_entropy": 0.686711, "bpb": 7.491948}, "de": {"chunks": 7263.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.353723, "cps": 56.742188, "mean_p": 0.494684, "boundary_entropy": 0.687717, "bpb": 7.547367}, "vi": {"chunks": 10680.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.579333, "cps": 83.4375, "mean_p": 0.499673, "boundary_entropy": 0.688367, "bpb": 7.609612}, "tr": {"chunks": 4020.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.222063, "cps": 31.40625, "mean_p": 0.484883, "boundary_entropy": 0.686993, "bpb": 7.68169}, "fi": {"chunks": 6981.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.371113, "cps": 54.539062, "mean_p": 0.494274, "boundary_entropy": 0.687346, "bpb": 7.627208}, "ru": {"chunks": 5054.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.266702, "cps": 39.484375, "mean_p": 0.484986, "boundary_entropy": 0.689479, "bpb": 7.287633}, "el": {"chunks": 13847.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.674706, "cps": 108.179688, "mean_p": 0.496346, "boundary_entropy": 0.690209, "bpb": 7.792458}, "hi": {"chunks": 1194.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.070966, "cps": 9.328125, "mean_p": 0.477649, "boundary_entropy": 0.689689, "bpb": 7.723284}, "ta": {"chunks": 30803.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 1.64766, "cps": 240.648438, "mean_p": 0.505519, "boundary_entropy": 0.690645, "bpb": 7.428373}, "th": {"chunks": 28695.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 1.78396, "cps": 224.179688, "mean_p": 0.507303, "boundary_entropy": 0.6907, "bpb": 7.645596}, "zh": {"chunks": 11622.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.967829, "cps": 90.796875, "mean_p": 0.519029, "boundary_entropy": 0.685731, "bpb": 8.011482}, "my": {"chunks": 48036.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 2.535149, "cps": 375.28125, "mean_p": 0.521759, "boundary_entropy": 0.690171, "bpb": 7.563326}}, "bpb": {"en": 7.49195, "de": 7.54737, "vi": 7.60961, "tr": 7.68169, "fi": 7.62721, "ru": 7.28763, "el": 7.79246, "hi": 7.72328, "ta": 7.42837, "th": 7.6456, "zh": 8.01148, "my": 7.56333}, "gini_cpc": 0.476672, "gini_cps": 0.474585, "bpic": 2.218785, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 1.2894524822161373, "vi": 2.1118828152083617, "tr": 0.8095006600155452, "fi": 1.3528431908729297, "ru": 0.9722271141064229, "el": 2.4595550791706797, "hi": 0.2586967420730542, "ta": 6.006330862108239, "th": 6.503196388478208, "zh": 7.1734674826719, "my": 9.241557406821252}, "t": 18.3}
|
| 9 |
+
{"event": "probe", "step": 16, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2918.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.167672, "cps": 22.796875, "mean_p": 0.483036, "boundary_entropy": 0.686288, "bpb": 6.89556}, "de": {"chunks": 5637.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.274534, "cps": 44.039062, "mean_p": 0.489999, "boundary_entropy": 0.687637, "bpb": 6.982104}, "vi": {"chunks": 7859.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.426309, "cps": 61.398438, "mean_p": 0.49252, "boundary_entropy": 0.688228, "bpb": 7.188954}, "tr": {"chunks": 3001.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.165774, "cps": 23.445312, "mean_p": 0.481764, "boundary_entropy": 0.686833, "bpb": 7.198879}, "fi": {"chunks": 4624.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.245814, "cps": 36.125, "mean_p": 0.486731, "boundary_entropy": 0.687058, "bpb": 7.083572}, "ru": {"chunks": 919.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.048496, "cps": 7.179688, "mean_p": 0.470625, "boundary_entropy": 0.68813, "bpb": 6.56532}, "el": {"chunks": 6453.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.314428, "cps": 50.414062, "mean_p": 0.487918, "boundary_entropy": 0.689892, "bpb": 7.258176}, "hi": {"chunks": 485.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.028826, "cps": 3.789062, "mean_p": 0.470466, "boundary_entropy": 0.68891, "bpb": 7.066105}, "ta": {"chunks": 20147.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 1.077668, "cps": 157.398438, "mean_p": 0.496763, "boundary_entropy": 0.690722, "bpb": 6.869588}, "th": {"chunks": 20155.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 1.253031, "cps": 157.460938, "mean_p": 0.498736, "boundary_entropy": 0.690775, "bpb": 6.616165}, "zh": {"chunks": 11166.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.89062, "cps": 87.234375, "mean_p": 0.516898, "boundary_entropy": 0.685838, "bpb": 7.835642}, "my": {"chunks": 47523.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 2.508075, "cps": 371.273438, "mean_p": 0.522019, "boundary_entropy": 0.690101, "bpb": 6.934056}}, "bpb": {"en": 6.89556, "de": 6.9821, "vi": 7.18895, "tr": 7.19888, "fi": 7.08357, "ru": 6.56532, "el": 7.25818, "hi": 7.0661, "ta": 6.86959, "th": 6.61617, "zh": 7.83564, "my": 6.93406}, "gini_cpc": 0.561355, "gini_cps": 0.549132, "bpic": 2.932155, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 1.6373233685542794, "vi": 2.5425118132675557, "tr": 0.9886766483631835, "fi": 1.466036471376157, "ru": 0.28923119351997695, "el": 1.8752520659463727, "hi": 0.17191964208422997, "ta": 6.427228639238041, "th": 7.473096148097741, "zh": 11.275687043090205, "my": 14.958198951627361}, "t": 22.6}
|
| 10 |
+
{"event": "probe", "step": 32, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 246.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.014135, "cps": 1.921875, "mean_p": 0.378113, "boundary_entropy": 0.654411, "bpb": 6.46997}, "de": {"chunks": 216.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.01052, "cps": 1.6875, "mean_p": 0.386177, "boundary_entropy": 0.659998, "bpb": 6.580452}, "vi": {"chunks": 456.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.024736, "cps": 3.5625, "mean_p": 0.413983, "boundary_entropy": 0.6719, "bpb": 7.054647}, "tr": {"chunks": 313.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.01729, "cps": 2.445312, "mean_p": 0.393962, "boundary_entropy": 0.662865, "bpb": 6.858326}, "fi": {"chunks": 297.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.015789, "cps": 2.320312, "mean_p": 0.389414, "boundary_entropy": 0.660704, "bpb": 6.63062}, "ru": {"chunks": 132.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.006966, "cps": 1.03125, "mean_p": 0.415013, "boundary_entropy": 0.674981, "bpb": 6.347138}, "el": {"chunks": 217.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.010574, "cps": 1.695312, "mean_p": 0.449091, "boundary_entropy": 0.684849, "bpb": 6.832869}, "hi": {"chunks": 179.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.010639, "cps": 1.398438, "mean_p": 0.454408, "boundary_entropy": 0.686448, "bpb": 6.423777}, "ta": {"chunks": 474.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.025354, "cps": 3.703125, "mean_p": 0.468511, "boundary_entropy": 0.688738, "bpb": 6.31379}, "th": {"chunks": 184.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.011439, "cps": 1.4375, "mean_p": 0.461773, "boundary_entropy": 0.68786, "bpb": 6.031388}, "zh": {"chunks": 9561.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.618862, "cps": 74.695312, "mean_p": 0.50882, "boundary_entropy": 0.686081, "bpb": 7.402198}, "my": {"chunks": 45213.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 2.386162, "cps": 353.226562, "mean_p": 0.520571, "boundary_entropy": 0.69015, "bpb": 5.868623}}, "bpb": {"en": 6.46997, "de": 6.58045, "vi": 7.05465, "tr": 6.85833, "fi": 6.63062, "ru": 6.34714, "el": 6.83287, "hi": 6.42378, "ta": 6.31379, "th": 6.03139, "zh": 7.4022, "my": 5.86862}, "gini_cpc": 0.819581, "gini_cps": 0.846406, "bpic": 6.675845, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 0.7442011847674119, "vi": 1.749889856913215, "tr": 1.2231586733367195, "fi": 1.1169496052517272, "ru": 0.49278074522170023, "el": 0.7480108575387757, "hi": 0.7526394375385062, "ta": 1.7936672776730442, "th": 0.8092557071047862, "zh": 114.5246277039472, "my": 168.8064207819089}, "t": 27.9}
|
| 11 |
+
{"event": "probe", "step": 64, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 165.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.009481, "cps": 1.289062, "mean_p": 0.394724, "boundary_entropy": 0.663453, "bpb": 5.934945}, "de": {"chunks": 167.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.008133, "cps": 1.304688, "mean_p": 0.401337, "boundary_entropy": 0.667516, "bpb": 6.064312}, "vi": {"chunks": 270.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.014646, "cps": 2.109375, "mean_p": 0.411652, "boundary_entropy": 0.671783, "bpb": 6.61001}, "tr": {"chunks": 190.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.010495, "cps": 1.484375, "mean_p": 0.405835, "boundary_entropy": 0.668896, "bpb": 6.359059}, "fi": {"chunks": 180.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.009569, "cps": 1.40625, "mean_p": 0.399347, "boundary_entropy": 0.666169, "bpb": 6.069432}, "ru": {"chunks": 129.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.006807, "cps": 1.007812, "mean_p": 0.372291, "boundary_entropy": 0.656344, "bpb": 5.646633}, "el": {"chunks": 128.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.006237, "cps": 1.0, "mean_p": 0.409407, "boundary_entropy": 0.673342, "bpb": 6.38104}, "hi": {"chunks": 144.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.008559, "cps": 1.125, "mean_p": 0.431837, "boundary_entropy": 0.681208, "bpb": 5.742968}, "ta": {"chunks": 151.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.008077, "cps": 1.179688, "mean_p": 0.436318, "boundary_entropy": 0.682596, "bpb": 5.752051}, "th": {"chunks": 133.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.008269, "cps": 1.039062, "mean_p": 0.418422, "boundary_entropy": 0.67722, "bpb": 5.415547}, "zh": {"chunks": 308.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.05215, "cps": 2.40625, "mean_p": 0.446985, "boundary_entropy": 0.680147, "bpb": 7.324651}, "my": {"chunks": 3850.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.203188, "cps": 30.078125, "mean_p": 0.485369, "boundary_entropy": 0.690631, "bpb": 5.158866}}, "bpb": {"en": 5.93495, "de": 6.06431, "vi": 6.61001, "tr": 6.35906, "fi": 6.06943, "ru": 5.64663, "el": 6.38104, "hi": 5.74297, "ta": 5.75205, "th": 5.41555, "zh": 7.32465, "my": 5.15887}, "gini_cpc": 0.635869, "gini_cps": 0.628131, "bpic": 65.998452, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 0.8578359447983954, "vi": 1.5447592277535318, "tr": 1.106988796432535, "fi": 1.0092547397315883, "ru": 0.7179937634924441, "el": 0.6578233733328017, "hi": 0.9027086316358232, "ta": 0.8519059544360427, "th": 0.8721089665696441, "zh": 5.500440230274297, "my": 21.430757863626766}, "t": 32.8}
|
| 12 |
+
{"event": "probe", "step": 128, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 151.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.008677, "cps": 1.179688, "mean_p": 0.418889, "boundary_entropy": 0.67239, "bpb": 4.62767}, "de": {"chunks": 145.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.007062, "cps": 1.132812, "mean_p": 0.423243, "boundary_entropy": 0.674893, "bpb": 4.837106}, "vi": {"chunks": 254.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.013778, "cps": 1.984375, "mean_p": 0.43657, "boundary_entropy": 0.679045, "bpb": 5.258264}, "tr": {"chunks": 172.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.009501, "cps": 1.34375, "mean_p": 0.421685, "boundary_entropy": 0.673879, "bpb": 5.256801}, "fi": {"chunks": 172.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.009144, "cps": 1.34375, "mean_p": 0.425847, "boundary_entropy": 0.675215, "bpb": 4.865459}, "ru": {"chunks": 135.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.007124, "cps": 1.054688, "mean_p": 0.423319, "boundary_entropy": 0.677312, "bpb": 3.877527}, "el": {"chunks": 128.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.006237, "cps": 1.0, "mean_p": 0.420729, "boundary_entropy": 0.677266, "bpb": 4.830958}, "hi": {"chunks": 219.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.013016, "cps": 1.710938, "mean_p": 0.466692, "boundary_entropy": 0.688226, "bpb": 3.740243}, "ta": {"chunks": 167.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.008933, "cps": 1.304688, "mean_p": 0.450738, "boundary_entropy": 0.68599, "bpb": 3.589666}, "th": {"chunks": 2380.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.147964, "cps": 18.59375, "mean_p": 0.468385, "boundary_entropy": 0.687722, "bpb": 3.306342}, "zh": {"chunks": 215.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.036404, "cps": 1.679688, "mean_p": 0.426009, "boundary_entropy": 0.674391, "bpb": 6.405209}, "my": {"chunks": 140.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.007389, "cps": 1.09375, "mean_p": 0.404089, "boundary_entropy": 0.672115, "bpb": 4.152467}}, "bpb": {"en": 4.62767, "de": 4.83711, "vi": 5.25826, "tr": 5.2568, "fi": 4.86546, "ru": 3.87753, "el": 4.83096, "hi": 3.74024, "ta": 3.58967, "th": 3.30634, "zh": 6.40521, "my": 4.15247}, "gini_cpc": 0.575425, "gini_cps": 0.522324, "bpic": 89.710379, "worst_lang": "th", "cpc_parity_vs_en": {"en": 1.0, "de": 0.8138844818694377, "vi": 1.5879533783456101, "tr": 1.0950276069276872, "fi": 1.0538134478875083, "ru": 0.8210540110782995, "el": 0.7188136198669688, "hi": 1.5001552798087046, "ta": 1.0295280283533685, "th": 17.0530892382562, "zh": 4.1955817745114965, "my": 0.851553292594441}, "t": 41.2}
|
| 13 |
+
{"event": "train", "step": 200, "phase": 1, "objective": "global", "lm_loss": 2.53738, "bpb": 3.66066, "chunk_loss": 0.02315, "F": 0.00571, "G": 0.36386, "chunks": 5.84, "bytes_per_chunk": 175.23, "boundary_entropy": 0.65001, "router_noise": 0.0, "grad_norm": 1.9851, "lr": 0.000201, "router_trainable": true, "sec_per_step": 0.216, "alpha": 0.03, "t": 48.7}
|
| 14 |
+
{"event": "probe", "step": 256, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 177.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.010171, "cps": 1.382812, "mean_p": 0.34504, "boundary_entropy": 0.63286, "bpb": 3.720853}, "de": {"chunks": 302.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.014708, "cps": 2.359375, "mean_p": 0.348575, "boundary_entropy": 0.635562, "bpb": 3.978181}, "vi": {"chunks": 192.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.010415, "cps": 1.5, "mean_p": 0.378811, "boundary_entropy": 0.653899, "bpb": 3.321689}, "tr": {"chunks": 423.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.023366, "cps": 3.304688, "mean_p": 0.351562, "boundary_entropy": 0.633833, "bpb": 3.935694}, "fi": {"chunks": 204.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.010845, "cps": 1.59375, "mean_p": 0.317402, "boundary_entropy": 0.611621, "bpb": 3.969358}, "ru": {"chunks": 150.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.007916, "cps": 1.171875, "mean_p": 0.323817, "boundary_entropy": 0.615306, "bpb": 2.665593}, "el": {"chunks": 134.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.006529, "cps": 1.046875, "mean_p": 0.354725, "boundary_entropy": 0.643435, "bpb": 2.806589}, "hi": {"chunks": 177.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.01052, "cps": 1.382812, "mean_p": 0.343351, "boundary_entropy": 0.636062, "bpb": 2.165491}, "ta": {"chunks": 184.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.009842, "cps": 1.4375, "mean_p": 0.354411, "boundary_entropy": 0.640548, "bpb": 2.026701}, "th": {"chunks": 223.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.013864, "cps": 1.742188, "mean_p": 0.316373, "boundary_entropy": 0.616371, "bpb": 2.104417}, "zh": {"chunks": 198.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.033525, "cps": 1.546875, "mean_p": 0.340826, "boundary_entropy": 0.626518, "bpb": 5.003183}, "my": {"chunks": 130.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.006861, "cps": 1.015625, "mean_p": 0.405019, "boundary_entropy": 0.672024, "bpb": 2.221902}}, "bpb": {"en": 3.72085, "de": 3.97818, "vi": 3.32169, "tr": 3.93569, "fi": 3.96936, "ru": 2.66559, "el": 2.80659, "hi": 2.16549, "ta": 2.0267, "th": 2.10442, "zh": 5.00318, "my": 2.2219}, "gini_cpc": 0.270813, "gini_cps": 0.182171, "bpic": 153.881716, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 1.4461235200549427, "vi": 1.024021182992925, "tr": 2.297421440589021, "fi": 1.0662747815243336, "ru": 0.7782746746567686, "el": 0.6419701087741988, "hi": 1.0343536404160476, "ta": 0.967705495441997, "th": 1.3631217630912051, "zh": 3.2962686609997305, "my": 0.674575913382925}, "t": 54.5}
|
| 15 |
+
{"event": "train", "step": 400, "phase": 1, "objective": "global", "lm_loss": 2.06785, "bpb": 2.98328, "chunk_loss": 0.022737, "F": 0.00482, "G": 0.37455, "chunks": 4.94, "bytes_per_chunk": 207.392, "boundary_entropy": 0.65379, "router_noise": 0.0, "grad_norm": 1.2115, "lr": 0.0002998, "router_trainable": true, "sec_per_step": 0.096, "alpha": 0.03, "t": 68.0}
|
| 16 |
+
{"event": "train", "step": 600, "phase": 1, "objective": "global", "lm_loss": 1.97127, "bpb": 2.84395, "chunk_loss": 0.022497, "F": 0.00098, "G": 0.37632, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.653, "router_noise": 0.0, "grad_norm": 0.793, "lr": 0.0002982, "router_trainable": true, "sec_per_step": 0.087, "alpha": 0.03, "t": 85.3}
|
| 17 |
+
{"event": "train", "step": 800, "phase": 1, "objective": "global", "lm_loss": 1.84488, "bpb": 2.66159, "chunk_loss": 0.022703, "F": 0.02719, "G": 0.4086, "chunks": 27.84, "bytes_per_chunk": 36.777, "boundary_entropy": 0.66821, "router_noise": 0.0, "grad_norm": 0.8627, "lr": 0.0002949, "router_trainable": true, "sec_per_step": 0.085, "alpha": 0.03, "t": 102.2}
|
| 18 |
+
{"event": "eval", "step": 1000, "phase": 1, "bpb": {"en": 2.6727, "de": 2.98896, "vi": 2.36676, "tr": 2.9987, "fi": 3.19708, "ru": 1.95199, "el": 2.06414, "hi": 1.53959, "ta": 1.26424, "th": 1.58473, "zh": 3.67557, "my": 1.46615}, "bytes_per_chunk": {"en": 11.696, "de": 14.228, "vi": 26.386, "tr": 14.915, "fi": 19.834, "ru": 14.979, "el": 52.418, "hi": 41.354, "ta": 82.931, "th": 12.679, "zh": 50.961, "my": 14.757}, "bpb_mean": 2.31422, "t": 120.4}
|
| 19 |
+
{"event": "train", "step": 1000, "phase": 1, "objective": "global", "lm_loss": 1.77073, "bpb": 2.55463, "chunk_loss": 0.036174, "F": 0.28519, "G": 0.40666, "chunks": 292.03, "bytes_per_chunk": 3.506, "boundary_entropy": 0.6685, "router_noise": 0.0, "grad_norm": 0.8882, "lr": 0.0002901, "router_trainable": true, "sec_per_step": 0.091, "alpha": 0.03, "t": 120.4}
|
| 20 |
+
{"event": "train", "step": 1200, "phase": 1, "objective": "global", "lm_loss": 1.7461, "bpb": 2.51909, "chunk_loss": 0.023676, "F": 0.00098, "G": 0.34336, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.63283, "router_noise": 0.0, "grad_norm": 0.8669, "lr": 0.0002837, "router_trainable": true, "sec_per_step": 0.085, "alpha": 0.03, "t": 137.5}
|
| 21 |
+
{"event": "train", "step": 1400, "phase": 1, "objective": "global", "lm_loss": 1.59531, "bpb": 2.30154, "chunk_loss": 0.04745, "F": 0.56873, "G": 0.3657, "chunks": 582.38, "bytes_per_chunk": 1.758, "boundary_entropy": 0.64586, "router_noise": 0.0, "grad_norm": 0.7713, "lr": 0.0002759, "router_trainable": true, "sec_per_step": 0.086, "alpha": 0.03, "t": 154.6}
|
| 22 |
+
{"event": "train", "step": 1600, "phase": 1, "objective": "global", "lm_loss": 1.63156, "bpb": 2.35384, "chunk_loss": 0.021869, "F": 0.00412, "G": 0.39823, "chunks": 4.22, "bytes_per_chunk": 242.726, "boundary_entropy": 0.6633, "router_noise": 0.0, "grad_norm": 0.8277, "lr": 0.0002668, "router_trainable": true, "sec_per_step": 0.087, "alpha": 0.03, "t": 172.1}
|
| 23 |
+
{"event": "train", "step": 1800, "phase": 1, "objective": "global", "lm_loss": 1.54875, "bpb": 2.23437, "chunk_loss": 0.022334, "F": 0.00446, "G": 0.38555, "chunks": 4.56, "bytes_per_chunk": 224.438, "boundary_entropy": 0.65782, "router_noise": 0.0, "grad_norm": 0.8032, "lr": 0.0002564, "router_trainable": true, "sec_per_step": 0.088, "alpha": 0.03, "t": 189.8}
|
| 24 |
+
{"event": "probe", "step": 2000, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 865.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.049704, "cps": 6.757812, "mean_p": 0.387742, "boundary_entropy": 0.650626, "bpb": 2.342435}, "de": {"chunks": 1047.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.050991, "cps": 8.179688, "mean_p": 0.394094, "boundary_entropy": 0.656408, "bpb": 2.60724}, "vi": {"chunks": 1126.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.061079, "cps": 8.796875, "mean_p": 0.402093, "boundary_entropy": 0.662149, "bpb": 2.122694}, "tr": {"chunks": 887.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.048997, "cps": 6.929688, "mean_p": 0.391411, "boundary_entropy": 0.655639, "bpb": 2.600893}, "fi": {"chunks": 538.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.0286, "cps": 4.203125, "mean_p": 0.381657, "boundary_entropy": 0.650929, "bpb": 2.835173}, "ru": {"chunks": 1387.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.073193, "cps": 10.835938, "mean_p": 0.397072, "boundary_entropy": 0.663244, "bpb": 1.697863}, "el": {"chunks": 910.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.04434, "cps": 7.109375, "mean_p": 0.373454, "boundary_entropy": 0.650931, "bpb": 1.75602}, "hi": {"chunks": 658.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.039108, "cps": 5.140625, "mean_p": 0.375392, "boundary_entropy": 0.64351, "bpb": 1.391615}, "ta": {"chunks": 1316.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.070393, "cps": 10.28125, "mean_p": 0.410274, "boundary_entropy": 0.670053, "bpb": 1.138114}, "th": {"chunks": 1614.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.100342, "cps": 12.609375, "mean_p": 0.370276, "boundary_entropy": 0.642571, "bpb": 1.332886}, "zh": {"chunks": 316.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.053505, "cps": 2.46875, "mean_p": 0.374219, "boundary_entropy": 0.647369, "bpb": 3.454632}, "my": {"chunks": 2281.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.120382, "cps": 17.820312, "mean_p": 0.413421, "boundary_entropy": 0.671582, "bpb": 1.241147}}, "bpb": {"en": 2.34244, "de": 2.60724, "vi": 2.12269, "tr": 2.60089, "fi": 2.83517, "ru": 1.69786, "el": 1.75602, "hi": 1.39162, "ta": 1.13811, "th": 1.33289, "zh": 3.45463, "my": 1.24115}, "gini_cpc": 0.214487, "gini_cps": 0.254352, "bpic": 29.647045, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 1.025893521467909, "vi": 1.2288624145764449, "tr": 0.9857824478362255, "fi": 0.5754113246369499, "ru": 1.472567663611268, "el": 0.8920897715491345, "hi": 0.7868262374494326, "ta": 1.4162451398862481, "th": 2.01878689979714, "zh": 1.076469310136258, "my": 2.421976483250173}, "t": 209.0}
|
| 25 |
+
{"event": "eval", "step": 2000, "phase": 1, "bpb": {"en": 2.34998, "de": 2.61291, "vi": 2.12962, "tr": 2.60764, "fi": 2.84089, "ru": 1.67632, "el": 1.74849, "hi": 1.38933, "ta": 1.14487, "th": 1.34978, "zh": 3.45794, "my": 1.25053}, "bytes_per_chunk": {"en": 21.62, "de": 20.52, "vi": 23.7, "tr": 28.276, "fi": 56.074, "ru": 28.798, "el": 50.931, "hi": 73.204, "ta": 45.733, "th": 25.158, "zh": 74.664, "my": 25.799}, "bpb_mean": 2.04652, "t": 210.2}
|
| 26 |
+
{"event": "train", "step": 2000, "phase": 1, "objective": "global", "lm_loss": 1.6244, "bpb": 2.34351, "chunk_loss": 0.022285, "F": 0.00098, "G": 0.38224, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65507, "router_noise": 0.0, "grad_norm": 0.9058, "lr": 0.000245, "router_trainable": true, "sec_per_step": 0.103, "alpha": 0.03, "t": 210.3}
|
| 27 |
+
{"event": "train", "step": 2200, "phase": 1, "objective": "global", "lm_loss": 1.48308, "bpb": 2.13963, "chunk_loss": 0.025308, "F": 0.04755, "G": 0.35082, "chunks": 48.69, "bytes_per_chunk": 21.032, "boundary_entropy": 0.63459, "router_noise": 0.0, "grad_norm": 0.7813, "lr": 0.0002325, "router_trainable": true, "sec_per_step": 0.087, "alpha": 0.03, "t": 227.7}
|
| 28 |
+
{"event": "phase_transition", "step": 2400, "phase": 2, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0, "t": 245.5}
|
| 29 |
+
{"event": "train", "step": 2400, "phase": 2, "objective": "global", "lm_loss": 1.40639, "bpb": 2.02899, "chunk_loss": 0.028925, "F": 0.14429, "G": 0.39408, "chunks": 147.75, "bytes_per_chunk": 6.931, "boundary_entropy": 0.66056, "router_noise": 0.0, "grad_norm": 0.859, "lr": 0.0002192, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 245.6}
|
| 30 |
+
{"event": "train", "step": 2600, "phase": 2, "objective": "global", "lm_loss": 1.62783, "bpb": 2.34847, "chunk_loss": 0.022151, "F": 0.00308, "G": 0.3888, "chunks": 3.16, "bytes_per_chunk": 324.436, "boundary_entropy": 0.65898, "router_noise": 0.0, "grad_norm": 0.8836, "lr": 0.0002053, "router_trainable": true, "sec_per_step": 0.207, "alpha": 0.03, "t": 286.9}
|
| 31 |
+
{"event": "train", "step": 2800, "phase": 2, "objective": "global", "lm_loss": 1.49748, "bpb": 2.1604, "chunk_loss": 0.026187, "F": 0.08603, "G": 0.38583, "chunks": 88.09, "bytes_per_chunk": 11.624, "boundary_entropy": 0.65703, "router_noise": 0.0, "grad_norm": 1.0019, "lr": 0.0001909, "router_trainable": true, "sec_per_step": 0.109, "alpha": 0.03, "t": 308.8}
|
| 32 |
+
{"event": "eval", "step": 3000, "phase": 2, "bpb": {"en": 2.22275, "de": 2.45175, "vi": 2.04372, "tr": 2.44134, "fi": 2.6592, "ru": 1.5545, "el": 1.62173, "hi": 1.29278, "ta": 1.06456, "th": 1.24943, "zh": 3.32823, "my": 1.19027}, "bytes_per_chunk": {"en": 13.113, "de": 17.167, "vi": 32.56, "tr": 13.862, "fi": 10.431, "ru": 15.814, "el": 40.896, "hi": 19.984, "ta": 15.489, "th": 21.722, "zh": 51.07, "my": 24.631}, "bpb_mean": 1.92669, "t": 327.2}
|
| 33 |
+
{"event": "train", "step": 3000, "phase": 2, "objective": "global", "lm_loss": 1.46088, "bpb": 2.1076, "chunk_loss": 0.021531, "F": 0.00098, "G": 0.40331, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.66279, "router_noise": 0.0, "grad_norm": 0.8257, "lr": 0.0001761, "router_trainable": true, "sec_per_step": 0.093, "alpha": 0.03, "t": 327.3}
|
| 34 |
+
{"event": "train", "step": 3200, "phase": 2, "objective": "global", "lm_loss": 1.40393, "bpb": 2.02544, "chunk_loss": 0.023008, "F": 0.00098, "G": 0.36203, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.64372, "router_noise": 0.0, "grad_norm": 0.7719, "lr": 0.0001613, "router_trainable": true, "sec_per_step": 0.086, "alpha": 0.03, "t": 344.6}
|
| 35 |
+
{"event": "train", "step": 3400, "phase": 2, "objective": "global", "lm_loss": 1.42269, "bpb": 2.05251, "chunk_loss": 0.022703, "F": 0.00104, "G": 0.37063, "chunks": 1.06, "bytes_per_chunk": 963.765, "boundary_entropy": 0.64803, "router_noise": 0.0, "grad_norm": 0.8124, "lr": 0.0001465, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 362.4}
|
| 36 |
+
{"event": "train", "step": 3600, "phase": 2, "objective": "global", "lm_loss": 1.67857, "bpb": 2.42166, "chunk_loss": 0.025092, "F": 0.05627, "G": 0.37274, "chunks": 57.62, "bytes_per_chunk": 17.77, "boundary_entropy": 0.6487, "router_noise": 0.0, "grad_norm": 0.9422, "lr": 0.0001319, "router_trainable": true, "sec_per_step": 0.085, "alpha": 0.03, "t": 379.4}
|
| 37 |
+
{"event": "train", "step": 3800, "phase": 2, "objective": "global", "lm_loss": 1.5016, "bpb": 2.16636, "chunk_loss": 0.027607, "F": 0.11011, "G": 0.36684, "chunks": 112.75, "bytes_per_chunk": 9.082, "boundary_entropy": 0.64616, "router_noise": 0.0, "grad_norm": 0.8228, "lr": 0.0001177, "router_trainable": true, "sec_per_step": 0.086, "alpha": 0.03, "t": 396.7}
|
| 38 |
+
{"event": "probe", "step": 4000, "phase": 2, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 743.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.042694, "cps": 5.804688, "mean_p": 0.386578, "boundary_entropy": 0.648773, "bpb": 2.142141}, "de": {"chunks": 991.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.048264, "cps": 7.742188, "mean_p": 0.393949, "boundary_entropy": 0.655005, "bpb": 2.352732}, "vi": {"chunks": 963.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.052238, "cps": 7.523438, "mean_p": 0.397769, "boundary_entropy": 0.659274, "bpb": 1.99228}, "tr": {"chunks": 996.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.055019, "cps": 7.78125, "mean_p": 0.39571, "boundary_entropy": 0.656209, "bpb": 2.340493}, "fi": {"chunks": 653.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.034714, "cps": 5.101562, "mean_p": 0.381919, "boundary_entropy": 0.648935, "bpb": 2.542253}, "ru": {"chunks": 890.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.046966, "cps": 6.953125, "mean_p": 0.389771, "boundary_entropy": 0.660209, "bpb": 1.499817}, "el": {"chunks": 1227.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.059787, "cps": 9.585938, "mean_p": 0.402776, "boundary_entropy": 0.66601, "bpb": 1.551699}, "hi": {"chunks": 505.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.030015, "cps": 3.945312, "mean_p": 0.35568, "boundary_entropy": 0.621076, "bpb": 1.245318}, "ta": {"chunks": 440.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.023536, "cps": 3.4375, "mean_p": 0.371573, "boundary_entropy": 0.648746, "bpb": 1.025044}, "th": {"chunks": 1650.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.10258, "cps": 12.890625, "mean_p": 0.373504, "boundary_entropy": 0.643095, "bpb": 1.171179}, "zh": {"chunks": 361.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.061124, "cps": 2.820312, "mean_p": 0.393553, "boundary_entropy": 0.65732, "bpb": 3.222654}, "my": {"chunks": 1762.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.092991, "cps": 13.765625, "mean_p": 0.388011, "boundary_entropy": 0.658824, "bpb": 1.140095}}, "bpb": {"en": 2.14214, "de": 2.35273, "vi": 1.99228, "tr": 2.34049, "fi": 2.54225, "ru": 1.49982, "el": 1.5517, "hi": 1.24532, "ta": 1.02504, "th": 1.17118, "zh": 3.22265, "my": 1.14009}, "gini_cpc": 0.222429, "gini_cps": 0.252564, "bpic": 34.32439, "worst_lang": "th", "cpc_parity_vs_en": {"en": 1.0, "de": 1.1304635239376668, "vi": 1.2235407880658864, "tr": 1.2886770475718836, "fi": 0.8130862265020188, "ru": 1.100059304609069, "el": 1.40035782982937, "hi": 0.7030263639436125, "ta": 0.5512676574479397, "th": 2.4026924594317456, "zh": 1.4316902445171757, "my": 2.178100097426093}, "t": 415.4}
|
| 39 |
+
{"event": "eval", "step": 4000, "phase": 2, "bpb": {"en": 2.15046, "de": 2.35924, "vi": 1.99391, "tr": 2.34763, "fi": 2.54761, "ru": 1.47731, "el": 1.54451, "hi": 1.24194, "ta": 1.0329, "th": 1.18717, "zh": 3.22415, "my": 1.14907}, "bytes_per_chunk": {"en": 26.309, "de": 21.906, "vi": 27.344, "tr": 24.971, "fi": 41.005, "ru": 46.438, "el": 36.813, "hi": 105.576, "ta": 149.968, "th": 25.608, "zh": 51.899, "my": 29.418}, "bpb_mean": 1.85466, "t": 416.6}
|
| 40 |
+
{"event": "train", "step": 4000, "phase": 2, "objective": "global", "lm_loss": 1.37428, "bpb": 1.98267, "chunk_loss": 0.037107, "F": 0.31747, "G": 0.38364, "chunks": 325.09, "bytes_per_chunk": 3.15, "boundary_entropy": 0.6547, "router_noise": 0.0, "grad_norm": 0.8236, "lr": 0.000104, "router_trainable": true, "sec_per_step": 0.1, "alpha": 0.03, "t": 416.7}
|
| 41 |
+
{"event": "phase_transition", "step": 4200, "phase": 3, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0, "t": 434.4}
|
| 42 |
+
{"event": "train", "step": 4200, "phase": 3, "objective": "global", "lm_loss": 1.22543, "bpb": 1.76792, "chunk_loss": 0.053313, "F": 0.70139, "G": 0.36807, "chunks": 718.22, "bytes_per_chunk": 1.426, "boundary_entropy": 0.64404, "router_noise": 0.0, "grad_norm": 0.7423, "lr": 9.12e-05, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 434.5}
|
| 43 |
+
{"event": "train", "step": 4400, "phase": 3, "objective": "global", "lm_loss": 1.51993, "bpb": 2.19279, "chunk_loss": 0.032473, "F": 0.2225, "G": 0.37015, "chunks": 227.84, "bytes_per_chunk": 4.494, "boundary_entropy": 0.64718, "router_noise": 0.0, "grad_norm": 0.9441, "lr": 7.92e-05, "router_trainable": true, "sec_per_step": 0.087, "alpha": 0.03, "t": 451.9}
|
| 44 |
+
{"event": "train", "step": 4600, "phase": 3, "objective": "global", "lm_loss": 1.33993, "bpb": 1.93311, "chunk_loss": 0.023319, "F": 0.00107, "G": 0.35345, "chunks": 1.09, "bytes_per_chunk": 936.229, "boundary_entropy": 0.63719, "router_noise": 0.0, "grad_norm": 0.8465, "lr": 6.82e-05, "router_trainable": true, "sec_per_step": 0.088, "alpha": 0.03, "t": 469.5}
|
| 45 |
+
{"event": "train", "step": 4800, "phase": 3, "objective": "global", "lm_loss": 1.28915, "bpb": 1.85985, "chunk_loss": 0.0234, "F": 0.00729, "G": 0.35837, "chunks": 7.47, "bytes_per_chunk": 137.105, "boundary_entropy": 0.6404, "router_noise": 0.0, "grad_norm": 0.8537, "lr": 5.85e-05, "router_trainable": true, "sec_per_step": 0.087, "alpha": 0.03, "t": 486.9}
|
| 46 |
+
{"event": "eval", "step": 5000, "phase": 3, "bpb": {"en": 2.10988, "de": 2.301, "vi": 1.9648, "tr": 2.28913, "fi": 2.48488, "ru": 1.4289, "el": 1.49963, "hi": 1.21131, "ta": 1.00044, "th": 1.15444, "zh": 3.18178, "my": 1.12292}, "bytes_per_chunk": {"en": 35.92, "de": 33.225, "vi": 44.311, "tr": 39.167, "fi": 58.79, "ru": 55.539, "el": 57.013, "hi": 110.237, "ta": 188.865, "th": 47.715, "zh": 78.205, "my": 36.859}, "bpb_mean": 1.81243, "t": 505.3}
|
| 47 |
+
{"event": "train", "step": 5000, "phase": 3, "objective": "global", "lm_loss": 1.33088, "bpb": 1.92006, "chunk_loss": 0.022679, "F": 0.00098, "G": 0.37124, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.6485, "router_noise": 0.0, "grad_norm": 0.8763, "lr": 5e-05, "router_trainable": true, "sec_per_step": 0.093, "alpha": 0.03, "t": 505.5}
|
| 48 |
+
{"event": "train", "step": 5200, "phase": 3, "objective": "global", "lm_loss": 1.3245, "bpb": 1.91085, "chunk_loss": 0.061605, "F": 0.85696, "G": 0.37798, "chunks": 877.53, "bytes_per_chunk": 1.167, "boundary_entropy": 0.65228, "router_noise": 0.0, "grad_norm": 0.9709, "lr": 4.29e-05, "router_trainable": true, "sec_per_step": 0.088, "alpha": 0.03, "t": 523.0}
|
| 49 |
+
{"event": "train", "step": 5400, "phase": 3, "objective": "global", "lm_loss": 1.17281, "bpb": 1.69201, "chunk_loss": 0.023089, "F": 0.01028, "G": 0.37136, "chunks": 10.53, "bytes_per_chunk": 97.234, "boundary_entropy": 0.64689, "router_noise": 0.0, "grad_norm": 0.8181, "lr": 3.73e-05, "router_trainable": true, "sec_per_step": 0.086, "alpha": 0.03, "t": 540.1}
|
| 50 |
+
{"event": "train", "step": 5600, "phase": 3, "objective": "global", "lm_loss": 1.51532, "bpb": 2.18615, "chunk_loss": 0.022786, "F": 0.00098, "G": 0.36824, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.64788, "router_noise": 0.0, "grad_norm": 0.9137, "lr": 3.33e-05, "router_trainable": true, "sec_per_step": 0.123, "alpha": 0.03, "t": 564.7}
|
| 51 |
+
{"event": "train", "step": 5800, "phase": 3, "objective": "global", "lm_loss": 1.30505, "bpb": 1.88278, "chunk_loss": 0.023186, "F": 0.00098, "G": 0.35707, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.63876, "router_noise": 0.0, "grad_norm": 0.9114, "lr": 3.08e-05, "router_trainable": true, "sec_per_step": 0.187, "alpha": 0.03, "t": 602.1}
|
| 52 |
+
{"event": "probe", "step": 6000, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 625.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.035913, "cps": 4.882812, "mean_p": 0.376954, "boundary_entropy": 0.64401, "bpb": 2.081873}, "de": {"chunks": 747.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.03638, "cps": 5.835938, "mean_p": 0.379785, "boundary_entropy": 0.647951, "bpb": 2.270786}, "vi": {"chunks": 738.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.040033, "cps": 5.765625, "mean_p": 0.380939, "boundary_entropy": 0.650053, "bpb": 1.947748}, "tr": {"chunks": 856.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.047285, "cps": 6.6875, "mean_p": 0.384487, "boundary_entropy": 0.650487, "bpb": 2.253192}, "fi": {"chunks": 514.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.027324, "cps": 4.015625, "mean_p": 0.379234, "boundary_entropy": 0.647883, "bpb": 2.443051}, "ru": {"chunks": 707.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.037309, "cps": 5.523438, "mean_p": 0.382472, "boundary_entropy": 0.656953, "bpb": 1.429963}, "el": {"chunks": 829.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.040394, "cps": 6.476562, "mean_p": 0.383706, "boundary_entropy": 0.657316, "bpb": 1.482602}, "hi": {"chunks": 449.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.026686, "cps": 3.507812, "mean_p": 0.347368, "boundary_entropy": 0.61441, "bpb": 1.195372}, "ta": {"chunks": 596.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.03188, "cps": 4.65625, "mean_p": 0.390185, "boundary_entropy": 0.66019, "bpb": 0.981009}, "th": {"chunks": 849.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.052782, "cps": 6.632812, "mean_p": 0.35477, "boundary_entropy": 0.628983, "bpb": 1.118028}, "zh": {"chunks": 277.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.046901, "cps": 2.164062, "mean_p": 0.374839, "boundary_entropy": 0.647492, "bpb": 3.147655}, "my": {"chunks": 1620.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.085497, "cps": 12.65625, "mean_p": 0.370474, "boundary_entropy": 0.648133, "bpb": 1.103244}}, "bpb": {"en": 2.08187, "de": 2.27079, "vi": 1.94775, "tr": 2.25319, "fi": 2.44305, "ru": 1.42996, "el": 1.4826, "hi": 1.19537, "ta": 0.98101, "th": 1.11803, "zh": 3.14766, "my": 1.10324}, "gini_cpc": 0.172699, "gini_cps": 0.211413, "bpic": 43.576814, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 1.013006652705401, "vi": 1.1146982587469487, "tr": 1.3166408219632104, "fi": 0.7608435064589868, "ru": 1.0388534881266491, "el": 1.1247546265165909, "hi": 0.7430796552748885, "ta": 0.8876972880449316, "th": 1.469706882188374, "zh": 1.3059616661022686, "my": 2.380651044965168}, "t": 620.9}
|
| 53 |
+
{"event": "eval", "step": 6000, "phase": 3, "bpb": {"en": 2.09086, "de": 2.27843, "vi": 1.94879, "tr": 2.25811, "fi": 2.45103, "ru": 1.40813, "el": 1.47652, "hi": 1.1921, "ta": 0.98822, "th": 1.13533, "zh": 3.14726, "my": 1.11249}, "bytes_per_chunk": {"en": 33.851, "de": 32.074, "vi": 41.205, "tr": 29.722, "fi": 49.667, "ru": 57.665, "el": 55.422, "hi": 112.46, "ta": 112.847, "th": 47.524, "zh": 92.272, "my": 33.149}, "bpb_mean": 1.79061, "t": 622.1}
|
| 54 |
+
{"event": "probe", "step": 6000, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 625.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.035913, "cps": 4.882812, "mean_p": 0.376954, "boundary_entropy": 0.64401, "bpb": 2.081873}, "de": {"chunks": 747.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.03638, "cps": 5.835938, "mean_p": 0.379785, "boundary_entropy": 0.647951, "bpb": 2.270786}, "vi": {"chunks": 738.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.040033, "cps": 5.765625, "mean_p": 0.380939, "boundary_entropy": 0.650053, "bpb": 1.947748}, "tr": {"chunks": 856.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.047285, "cps": 6.6875, "mean_p": 0.384487, "boundary_entropy": 0.650487, "bpb": 2.253192}, "fi": {"chunks": 514.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.027324, "cps": 4.015625, "mean_p": 0.379234, "boundary_entropy": 0.647883, "bpb": 2.443051}, "ru": {"chunks": 707.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.037309, "cps": 5.523438, "mean_p": 0.382472, "boundary_entropy": 0.656953, "bpb": 1.429963}, "el": {"chunks": 829.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.040394, "cps": 6.476562, "mean_p": 0.383706, "boundary_entropy": 0.657316, "bpb": 1.482602}, "hi": {"chunks": 449.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.026686, "cps": 3.507812, "mean_p": 0.347368, "boundary_entropy": 0.61441, "bpb": 1.195372}, "ta": {"chunks": 596.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.03188, "cps": 4.65625, "mean_p": 0.390185, "boundary_entropy": 0.66019, "bpb": 0.981009}, "th": {"chunks": 849.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.052782, "cps": 6.632812, "mean_p": 0.35477, "boundary_entropy": 0.628983, "bpb": 1.118028}, "zh": {"chunks": 277.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.046901, "cps": 2.164062, "mean_p": 0.374839, "boundary_entropy": 0.647492, "bpb": 3.147655}, "my": {"chunks": 1620.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.085497, "cps": 12.65625, "mean_p": 0.370474, "boundary_entropy": 0.648133, "bpb": 1.103244}}, "bpb": {"en": 2.08187, "de": 2.27079, "vi": 1.94775, "tr": 2.25319, "fi": 2.44305, "ru": 1.42996, "el": 1.4826, "hi": 1.19537, "ta": 0.98101, "th": 1.11803, "zh": 3.14766, "my": 1.10324}, "gini_cpc": 0.172699, "gini_cps": 0.211413, "bpic": 43.576814, "worst_lang": "my", "cpc_parity_vs_en": {"en": 1.0, "de": 1.013006652705401, "vi": 1.1146982587469487, "tr": 1.3166408219632104, "fi": 0.7608435064589868, "ru": 1.0388534881266491, "el": 1.1247546265165909, "hi": 0.7430796552748885, "ta": 0.8876972880449316, "th": 1.469706882188374, "zh": 1.3059616661022686, "my": 2.380651044965168}, "t": 623.1}
|
| 55 |
+
{"event": "reps_dumped", "dirs": ["/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s0/th0.30", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s0/th0.40", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s0/th0.50", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s0/th0.60", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s0/th0.70"], "t": 626.6}
|
| 56 |
+
{"event": "final", "run": "jitter_s0", "condition": "A", "step": 6000, "params": 22516992, "per_lang": {"en": {"cpc": 0.03591334827328622, "cps": 4.8828125, "bpb": 2.102690803330191, "bytes_per_chunk": 33.77491464278812}, "de": {"cpc": 0.036380460721764964, "cps": 5.8359375, "bpb": 2.271415199876741, "bytes_per_chunk": 32.133366021083596}, "vi": {"cpc": 0.04003254678600488, "cps": 5.765625, "bpb": 1.951459073921452, "bytes_per_chunk": 40.595276809910956}, "tr": {"cpc": 0.04728498038999061, "cps": 6.6875, "bpb": 2.250277080982085, "bytes_per_chunk": 30.134958041154157}, "fi": {"cpc": 0.027324437828929883, "cps": 4.015625, "bpb": 2.4655344794072835, "bytes_per_chunk": 49.16890180999719}, "ru": {"cpc": 0.037308707124010555, "cps": 5.5234375, "bpb": 1.4088588409418283, "bytes_per_chunk": 58.11861212725862}, "el": {"cpc": 0.0403937046240803, "cps": 6.4765625, "bpb": 1.471620706197338, "bytes_per_chunk": 56.95068433630241}, "hi": {"cpc": 0.026686478454680536, "cps": 3.5078125, "bpb": 1.2017601651121157, "bytes_per_chunk": 111.29017193801741}, "ta": {"cpc": 0.031880181866809305, "cps": 4.65625, "bpb": 0.9874921775028731, "bytes_per_chunk": 111.97949594190517}, "th": {"cpc": 0.052782095119676714, "cps": 6.6328125, "bpb": 1.1332099012743477, "bytes_per_chunk": 47.20338525254344}, "zh": {"cpc": 0.0469014561462919, "cps": 2.1640625, "bpb": 3.1557357805467756, "bytes_per_chunk": 94.7394289844597}, "my": {"cpc": 0.08549715009499684, "cps": 12.65625, "bpb": 1.1230400189341303, "bytes_per_chunk": 33.42607586866433}}, "gini_cpc": 0.17269931081856482, "gini_cps": 0.21141326974754948, "bpic": 43.57681389803565, "seed": 0, "mean_boundary_entropy": 0.6520750522613525, "dechunk": "ema", "router_noise": 0.0, "transitions": [{"step": 0, "phase": 1, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}, {"step": 2400, "phase": 2, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}, {"step": 4200, "phase": 3, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}], "t": 628.8}
|
jitter_s0/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c9ea1cb0d102b6188b02fcc98493811484f3d4824eef7bcba770de4a50ac1ede
|
| 3 |
+
size 90078288
|
jitter_s1/config.json
ADDED
|
@@ -0,0 +1,108 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"run": "jitter_s1",
|
| 3 |
+
"step": 6000,
|
| 4 |
+
"model_config": {
|
| 5 |
+
"vocab_size": 256,
|
| 6 |
+
"d_enc": 256,
|
| 7 |
+
"d_main": 512,
|
| 8 |
+
"n_enc": 2,
|
| 9 |
+
"n_dec": 2,
|
| 10 |
+
"n_main": 6,
|
| 11 |
+
"n_heads_enc": 4,
|
| 12 |
+
"n_heads_main": 8,
|
| 13 |
+
"seq_len": 1024,
|
| 14 |
+
"dropout": 0.0,
|
| 15 |
+
"target_ratio": 6.0,
|
| 16 |
+
"max_chunk_mult": 3.0,
|
| 17 |
+
"dechunk": "ema",
|
| 18 |
+
"router_noise": 0.0,
|
| 19 |
+
"rope_base": 10000.0
|
| 20 |
+
},
|
| 21 |
+
"tied_weights": {
|
| 22 |
+
"head.weight": "emb.weight"
|
| 23 |
+
},
|
| 24 |
+
"train_config": {
|
| 25 |
+
"condition": "A",
|
| 26 |
+
"seed": 1,
|
| 27 |
+
"out_dir": null,
|
| 28 |
+
"run_name": "jitter_s1",
|
| 29 |
+
"corpus": "corpus",
|
| 30 |
+
"langs": null,
|
| 31 |
+
"seq_len": 1024,
|
| 32 |
+
"batch_size": 32,
|
| 33 |
+
"grad_accum": 1,
|
| 34 |
+
"steps": 6000,
|
| 35 |
+
"lr": 0.0003,
|
| 36 |
+
"min_lr_frac": 0.1,
|
| 37 |
+
"warmup": 300,
|
| 38 |
+
"weight_decay": 0.1,
|
| 39 |
+
"grad_clip": 1.0,
|
| 40 |
+
"betas": [
|
| 41 |
+
0.9,
|
| 42 |
+
0.95
|
| 43 |
+
],
|
| 44 |
+
"router_lr_mult": 1.0,
|
| 45 |
+
"d_enc": 256,
|
| 46 |
+
"d_main": 512,
|
| 47 |
+
"n_enc": 2,
|
| 48 |
+
"n_dec": 2,
|
| 49 |
+
"n_main": 6,
|
| 50 |
+
"n_heads_enc": 4,
|
| 51 |
+
"n_heads_main": 8,
|
| 52 |
+
"dechunk": "ema",
|
| 53 |
+
"target_ratio": 6.0,
|
| 54 |
+
"alpha": 0.03,
|
| 55 |
+
"alpha_parity_cap": 0.06,
|
| 56 |
+
"global_anchor": 0.0,
|
| 57 |
+
"premium_clip": 4.0,
|
| 58 |
+
"premium_kind": "flores_per_sent",
|
| 59 |
+
"mixture": null,
|
| 60 |
+
"doc_filters": null,
|
| 61 |
+
"tau1": 0.4,
|
| 62 |
+
"tau2": 0.7,
|
| 63 |
+
"freeze_scope": "router",
|
| 64 |
+
"ckpt_every": 2000,
|
| 65 |
+
"eval_every": 1000,
|
| 66 |
+
"log_every": 200,
|
| 67 |
+
"probe_sents": 128,
|
| 68 |
+
"eval_batches": 8,
|
| 69 |
+
"keep_ckpt": false,
|
| 70 |
+
"amp": "bf16",
|
| 71 |
+
"compile": false,
|
| 72 |
+
"cpc_decay": 0.9,
|
| 73 |
+
"device": "cuda",
|
| 74 |
+
"log_spaced_probe": true,
|
| 75 |
+
"probe_bpb_batches": 4,
|
| 76 |
+
"dump_reps": true,
|
| 77 |
+
"dump_reps_thresholds": [
|
| 78 |
+
0.3,
|
| 79 |
+
0.4,
|
| 80 |
+
0.5,
|
| 81 |
+
0.6,
|
| 82 |
+
0.7
|
| 83 |
+
],
|
| 84 |
+
"router_noise": 0.0,
|
| 85 |
+
"genreg_target_entropy": null,
|
| 86 |
+
"genreg_eta": 0.05
|
| 87 |
+
},
|
| 88 |
+
"probe_agg": {
|
| 89 |
+
"gini_cpc": 0.2318796127695224,
|
| 90 |
+
"gini_cps": 0.2339954866678544,
|
| 91 |
+
"bpic": 68.37359700694816,
|
| 92 |
+
"worst_lang": "hi",
|
| 93 |
+
"cpc_parity_vs_en": {
|
| 94 |
+
"en": 1.0,
|
| 95 |
+
"de": 0.8146260953869533,
|
| 96 |
+
"vi": 1.3084208484930249,
|
| 97 |
+
"tr": 0.911522408188721,
|
| 98 |
+
"fi": 0.6782836797893858,
|
| 99 |
+
"ru": 1.7701111460483683,
|
| 100 |
+
"el": 0.823810338912061,
|
| 101 |
+
"hi": 2.4679785047232636,
|
| 102 |
+
"ta": 1.7050250337085502,
|
| 103 |
+
"th": 1.168831225307265,
|
| 104 |
+
"zh": 1.74051680121559,
|
| 105 |
+
"my": 0.6662411771075679
|
| 106 |
+
}
|
| 107 |
+
}
|
| 108 |
+
}
|
jitter_s1/final.json
ADDED
|
@@ -0,0 +1,114 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"event": "final",
|
| 3 |
+
"run": "jitter_s1",
|
| 4 |
+
"condition": "A",
|
| 5 |
+
"step": 6000,
|
| 6 |
+
"params": 22516992,
|
| 7 |
+
"per_lang": {
|
| 8 |
+
"en": {
|
| 9 |
+
"cpc": 0.022180083893581566,
|
| 10 |
+
"cps": 3.015625,
|
| 11 |
+
"bpb": 2.1020406444164355,
|
| 12 |
+
"bytes_per_chunk": 46.865826405649415
|
| 13 |
+
},
|
| 14 |
+
"de": {
|
| 15 |
+
"cpc": 0.018068475137583404,
|
| 16 |
+
"cps": 2.8984375,
|
| 17 |
+
"bpb": 2.2796363360511163,
|
| 18 |
+
"bytes_per_chunk": 68.94897422409258
|
| 19 |
+
},
|
| 20 |
+
"vi": {
|
| 21 |
+
"cpc": 0.029020884187686467,
|
| 22 |
+
"cps": 4.1796875,
|
| 23 |
+
"bpb": 1.9494764901690524,
|
| 24 |
+
"bytes_per_chunk": 70.51620712844654
|
| 25 |
+
},
|
| 26 |
+
"tr": {
|
| 27 |
+
"cpc": 0.020217643484505332,
|
| 28 |
+
"cps": 2.859375,
|
| 29 |
+
"bpb": 2.2458832574689267,
|
| 30 |
+
"bytes_per_chunk": 85.72400261608895
|
| 31 |
+
},
|
| 32 |
+
"fi": {
|
| 33 |
+
"cpc": 0.01504438892137579,
|
| 34 |
+
"cps": 2.2109375,
|
| 35 |
+
"bpb": 2.4640121534753403,
|
| 36 |
+
"bytes_per_chunk": 111.19575821845174
|
| 37 |
+
},
|
| 38 |
+
"ru": {
|
| 39 |
+
"cpc": 0.03926121372031662,
|
| 40 |
+
"cps": 5.8125,
|
| 41 |
+
"bpb": 1.4033274435589955,
|
| 42 |
+
"bytes_per_chunk": 66.50025367833587
|
| 43 |
+
},
|
| 44 |
+
"el": {
|
| 45 |
+
"cpc": 0.018272182429469375,
|
| 46 |
+
"cps": 2.9296875,
|
| 47 |
+
"bpb": 1.4606392699744462,
|
| 48 |
+
"bytes_per_chunk": 149.24224309706804
|
| 49 |
+
},
|
| 50 |
+
"hi": {
|
| 51 |
+
"cpc": 0.05473997028231798,
|
| 52 |
+
"cps": 7.1953125,
|
| 53 |
+
"bpb": 1.2202438124075716,
|
| 54 |
+
"bytes_per_chunk": 53.54248366013072
|
| 55 |
+
},
|
| 56 |
+
"ta": {
|
| 57 |
+
"cpc": 0.03781759828831238,
|
| 58 |
+
"cps": 5.5234375,
|
| 59 |
+
"bpb": 0.9998630346897966,
|
| 60 |
+
"bytes_per_chunk": 80.46163290362185
|
| 61 |
+
},
|
| 62 |
+
"th": {
|
| 63 |
+
"cpc": 0.025924774634752876,
|
| 64 |
+
"cps": 3.2578125,
|
| 65 |
+
"bpb": 1.126236773329332,
|
| 66 |
+
"bytes_per_chunk": 134.95186615186614
|
| 67 |
+
},
|
| 68 |
+
"zh": {
|
| 69 |
+
"cpc": 0.03860480866915002,
|
| 70 |
+
"cps": 1.78125,
|
| 71 |
+
"bpb": 3.1923234986783178,
|
| 72 |
+
"bytes_per_chunk": 134.3639159405433
|
| 73 |
+
},
|
| 74 |
+
"my": {
|
| 75 |
+
"cpc": 0.01477728520160439,
|
| 76 |
+
"cps": 2.1875,
|
| 77 |
+
"bpb": 1.114218518197882,
|
| 78 |
+
"bytes_per_chunk": 210.81141938078005
|
| 79 |
+
}
|
| 80 |
+
},
|
| 81 |
+
"gini_cpc": 0.2318796127695224,
|
| 82 |
+
"gini_cps": 0.2339954866678544,
|
| 83 |
+
"bpic": 68.37359700694816,
|
| 84 |
+
"seed": 1,
|
| 85 |
+
"mean_boundary_entropy": 0.6504928469657898,
|
| 86 |
+
"dechunk": "ema",
|
| 87 |
+
"router_noise": 0.0,
|
| 88 |
+
"transitions": [
|
| 89 |
+
{
|
| 90 |
+
"step": 0,
|
| 91 |
+
"phase": 1,
|
| 92 |
+
"objective": "global",
|
| 93 |
+
"router_trainable": true,
|
| 94 |
+
"encoder_trainable": true,
|
| 95 |
+
"frozen_router_params": 0
|
| 96 |
+
},
|
| 97 |
+
{
|
| 98 |
+
"step": 2400,
|
| 99 |
+
"phase": 2,
|
| 100 |
+
"objective": "global",
|
| 101 |
+
"router_trainable": true,
|
| 102 |
+
"encoder_trainable": true,
|
| 103 |
+
"frozen_router_params": 0
|
| 104 |
+
},
|
| 105 |
+
{
|
| 106 |
+
"step": 4200,
|
| 107 |
+
"phase": 3,
|
| 108 |
+
"objective": "global",
|
| 109 |
+
"router_trainable": true,
|
| 110 |
+
"encoder_trainable": true,
|
| 111 |
+
"frozen_router_params": 0
|
| 112 |
+
}
|
| 113 |
+
]
|
| 114 |
+
}
|
jitter_s1/log.jsonl
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"event": "start", "run": "jitter_s1", "condition": "A", "params": 22516992, "langs": ["en", "de", "vi", "tr", "fi", "ru", "el", "hi", "ta", "th", "zh", "my"], "premium": {"en": 1.0, "de": 1.184, "vi": 1.39, "tr": 1.12, "fi": 1.104, "ru": 1.971, "el": 2.174, "hi": 2.552, "ta": 3.192, "th": 2.762, "zh": 0.915, "my": 3.524}, "schedule": {"tau1": 0.4, "tau2": 0.7, "p1_objective": "global", "p2_objective": "global", "p3_objective": "global", "freeze_in_p3": false, "freeze_scope": "router"}, "corpus": "/local/scratch/sas245/beetle-hnet/scratch/data/tier1_corpus", "premium_kind": "flores_per_sent", "doc_filtered": {"my": 37511}, "weights": {"en": 0.2946, "de": 0.0909, "vi": 0.0576, "tr": 0.0612, "fi": 0.0424, "ru": 0.1476, "el": 0.0522, "hi": 0.042, "ta": 0.0313, "th": 0.0542, "zh": 0.1038, "my": 0.0222}, "t": 1.5}
|
| 2 |
+
{"event": "probe_schedule", "n": 13, "steps": [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 2000, 4000, 6000], "t": 1.5}
|
| 3 |
+
{"event": "probe", "step": 0, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2674.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.153652, "cps": 20.890625, "mean_p": 0.478126, "boundary_entropy": 0.685438, "bpb": 8.041379}, "de": {"chunks": 2056.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.100131, "cps": 16.0625, "mean_p": 0.469682, "boundary_entropy": 0.685273, "bpb": 8.067143}, "vi": {"chunks": 14185.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.76946, "cps": 110.820312, "mean_p": 0.507219, "boundary_entropy": 0.688283, "bpb": 7.894113}, "tr": {"chunks": 5645.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.311827, "cps": 44.101562, "mean_p": 0.486063, "boundary_entropy": 0.686375, "bpb": 8.01701}, "fi": {"chunks": 4732.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.251555, "cps": 36.96875, "mean_p": 0.48451, "boundary_entropy": 0.686595, "bpb": 8.075294}, "ru": {"chunks": 17548.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.926016, "cps": 137.09375, "mean_p": 0.502342, "boundary_entropy": 0.690066, "bpb": 8.095633}, "el": {"chunks": 31473.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.533548, "cps": 245.882812, "mean_p": 0.517965, "boundary_entropy": 0.689704, "bpb": 7.804942}, "hi": {"chunks": 1659.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.098603, "cps": 12.960938, "mean_p": 0.476241, "boundary_entropy": 0.689492, "bpb": 8.047677}, "ta": {"chunks": 39693.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 2.123188, "cps": 310.101562, "mean_p": 0.514597, "boundary_entropy": 0.690542, "bpb": 8.096414}, "th": {"chunks": 44424.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 2.761828, "cps": 347.0625, "mean_p": 0.52078, "boundary_entropy": 0.690224, "bpb": 8.201461}, "zh": {"chunks": 12933.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 2.189807, "cps": 101.039062, "mean_p": 0.528309, "boundary_entropy": 0.684752, "bpb": 8.121995}, "my": {"chunks": 7048.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.371965, "cps": 55.0625, "mean_p": 0.491145, "boundary_entropy": 0.691021, "bpb": 8.112229}}, "bpb": {"en": 8.04138, "de": 8.06714, "vi": 7.89411, "tr": 8.01701, "fi": 8.07529, "ru": 8.09563, "el": 7.80494, "hi": 8.04768, "ta": 8.09641, "th": 8.20146, "zh": 8.12199, "my": 8.11223}, "gini_cpc": 0.507116, "gini_cps": 0.500863, "bpic": 2.084973, "worst_lang": "th", "cpc_parity_vs_en": {"en": 1.0, "de": 0.6516785410034256, "vi": 5.0078223656304, "tr": 2.029439514461233, "fi": 1.6371767549398106, "ru": 6.026721581613623, "el": 9.980677296913099, "hi": 0.6417324941848254, "ta": 13.818190381639045, "th": 17.9746032262692, "zh": 14.25176170627287, "my": 2.4208352616119475}, "t": 3.2}
|
| 4 |
+
{"event": "train", "step": 0, "phase": 1, "objective": "global", "lm_loss": 5.56843, "bpb": 8.03354, "chunk_loss": 0.018394, "F": 0.00098, "G": 0.49096, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.6904, "router_noise": 0.0, "grad_norm": 4.288, "lr": 1e-06, "router_trainable": true, "sec_per_step": 0.001, "alpha": 0.03, "t": 3.6}
|
| 5 |
+
{"event": "probe", "step": 1, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2664.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.153077, "cps": 20.8125, "mean_p": 0.478116, "boundary_entropy": 0.68544, "bpb": 8.020959}, "de": {"chunks": 2049.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.099791, "cps": 16.007812, "mean_p": 0.469667, "boundary_entropy": 0.685274, "bpb": 8.049532}, "vi": {"chunks": 14185.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.76946, "cps": 110.820312, "mean_p": 0.507217, "boundary_entropy": 0.688284, "bpb": 7.879922}, "tr": {"chunks": 5644.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.311772, "cps": 44.09375, "mean_p": 0.486068, "boundary_entropy": 0.686377, "bpb": 8.002969}, "fi": {"chunks": 4715.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.250651, "cps": 36.835938, "mean_p": 0.484451, "boundary_entropy": 0.686591, "bpb": 8.058056}, "ru": {"chunks": 17619.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.929763, "cps": 137.648438, "mean_p": 0.502405, "boundary_entropy": 0.690066, "bpb": 8.091542}, "el": {"chunks": 31509.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.535302, "cps": 246.164062, "mean_p": 0.518025, "boundary_entropy": 0.6897, "bpb": 7.798168}, "hi": {"chunks": 1684.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.100089, "cps": 13.15625, "mean_p": 0.476383, "boundary_entropy": 0.689508, "bpb": 8.02942}, "ta": {"chunks": 39824.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 2.130195, "cps": 311.125, "mean_p": 0.514695, "boundary_entropy": 0.690537, "bpb": 8.080367}, "th": {"chunks": 44429.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 2.762139, "cps": 347.101562, "mean_p": 0.520846, "boundary_entropy": 0.690219, "bpb": 8.188781}, "zh": {"chunks": 12945.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 2.191839, "cps": 101.132812, "mean_p": 0.52838, "boundary_entropy": 0.684744, "bpb": 8.120358}, "my": {"chunks": 7122.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.375871, "cps": 55.640625, "mean_p": 0.491225, "boundary_entropy": 0.691024, "bpb": 8.107101}}, "bpb": {"en": 8.02096, "de": 8.04953, "vi": 7.87992, "tr": 8.00297, "fi": 8.05806, "ru": 8.09154, "el": 7.79817, "hi": 8.02942, "ta": 8.08037, "th": 8.18878, "zh": 8.12036, "my": 8.1071}, "gini_cpc": 0.506909, "gini_cps": 0.500776, "bpic": 2.081366, "worst_lang": "th", "cpc_parity_vs_en": {"en": 1.0, "de": 0.6518977032357931, "vi": 5.026620497633516, "tr": 2.0366966701552127, "fi": 1.637418577207796, "ru": 6.073820330884974, "el": 10.02960142937851, "hi": 0.6538481720948288, "ta": 13.915836242768576, "th": 18.04410612819845, "zh": 14.31853257337934, "my": 2.4554352880429513}, "t": 4.9}
|
| 6 |
+
{"event": "probe", "step": 2, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2665.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.153135, "cps": 20.820312, "mean_p": 0.478086, "boundary_entropy": 0.685442, "bpb": 7.985878}, "de": {"chunks": 2047.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.099693, "cps": 15.992188, "mean_p": 0.46964, "boundary_entropy": 0.685274, "bpb": 8.020022}, "vi": {"chunks": 14175.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.768918, "cps": 110.742188, "mean_p": 0.507212, "boundary_entropy": 0.688287, "bpb": 7.855724}, "tr": {"chunks": 5645.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.311827, "cps": 44.101562, "mean_p": 0.486091, "boundary_entropy": 0.686383, "bpb": 7.979128}, "fi": {"chunks": 4688.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.249216, "cps": 36.625, "mean_p": 0.484349, "boundary_entropy": 0.686585, "bpb": 8.030504}, "ru": {"chunks": 17793.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.938945, "cps": 139.007812, "mean_p": 0.502572, "boundary_entropy": 0.690065, "bpb": 8.072627}, "el": {"chunks": 31632.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.541295, "cps": 247.125, "mean_p": 0.51819, "boundary_entropy": 0.68969, "bpb": 7.781468}, "hi": {"chunks": 1745.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.103715, "cps": 13.632812, "mean_p": 0.47671, "boundary_entropy": 0.689541, "bpb": 7.988682}, "ta": {"chunks": 40397.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 2.160845, "cps": 315.601562, "mean_p": 0.515052, "boundary_entropy": 0.690518, "bpb": 8.043904}, "th": {"chunks": 44496.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 2.766304, "cps": 347.625, "mean_p": 0.521205, "boundary_entropy": 0.690191, "bpb": 8.137322}, "zh": {"chunks": 12956.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 2.193701, "cps": 101.21875, "mean_p": 0.528511, "boundary_entropy": 0.68473, "bpb": 8.115871}, "my": {"chunks": 7237.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.38194, "cps": 56.539062, "mean_p": 0.491328, "boundary_entropy": 0.691028, "bpb": 8.101076}}, "bpb": {"en": 7.98588, "de": 8.02002, "vi": 7.85572, "tr": 7.97913, "fi": 8.0305, "ru": 8.07263, "el": 7.78147, "hi": 7.98868, "ta": 8.0439, "th": 8.13732, "zh": 8.11587, "my": 8.10108}, "gini_cpc": 0.506485, "gini_cps": 0.50097, "bpic": 2.069168, "worst_lang": "th", "cpc_parity_vs_en": {"en": 1.0, "de": 0.6510170193243129, "vi": 5.021192048936199, "tr": 2.0362931563487194, "fi": 1.6274311575509242, "ru": 6.1315019578529455, "el": 10.064975304785488, "hi": 0.6772784624863051, "ta": 14.110764754850626, "th": 18.064536091973867, "zh": 14.325322357967124, "my": 2.494147327671942}, "t": 6.5}
|
| 7 |
+
{"event": "probe", "step": 4, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2593.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.148997, "cps": 20.257812, "mean_p": 0.477701, "boundary_entropy": 0.685415, "bpb": 7.838458}, "de": {"chunks": 1997.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.097258, "cps": 15.601562, "mean_p": 0.469242, "boundary_entropy": 0.685232, "bpb": 7.891691}, "vi": {"chunks": 13915.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.754814, "cps": 108.710938, "mean_p": 0.50656, "boundary_entropy": 0.68831, "bpb": 7.759148}, "tr": {"chunks": 5519.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.304867, "cps": 43.117188, "mean_p": 0.485785, "boundary_entropy": 0.686374, "bpb": 7.876807}, "fi": {"chunks": 4529.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.240763, "cps": 35.382812, "mean_p": 0.483709, "boundary_entropy": 0.686539, "bpb": 7.910918}, "ru": {"chunks": 17253.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.910449, "cps": 134.789062, "mean_p": 0.502039, "boundary_entropy": 0.690065, "bpb": 7.942093}, "el": {"chunks": 31592.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.539346, "cps": 246.8125, "mean_p": 0.51809, "boundary_entropy": 0.689699, "bpb": 7.710974}, "hi": {"chunks": 1878.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.11162, "cps": 14.671875, "mean_p": 0.477394, "boundary_entropy": 0.689611, "bpb": 7.843225}, "ta": {"chunks": 40908.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 2.188179, "cps": 319.59375, "mean_p": 0.515421, "boundary_entropy": 0.690497, "bpb": 7.919492}, "th": {"chunks": 44582.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 2.771651, "cps": 348.296875, "mean_p": 0.521687, "boundary_entropy": 0.690152, "bpb": 7.984667}, "zh": {"chunks": 12996.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 2.200474, "cps": 101.53125, "mean_p": 0.52877, "boundary_entropy": 0.6847, "bpb": 8.098097}, "my": {"chunks": 7299.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.385212, "cps": 57.023438, "mean_p": 0.491348, "boundary_entropy": 0.691028, "bpb": 8.039072}}, "bpb": {"en": 7.83846, "de": 7.89169, "vi": 7.75915, "tr": 7.87681, "fi": 7.91092, "ru": 7.94209, "el": 7.71097, "hi": 7.84323, "ta": 7.91949, "th": 7.98467, "zh": 8.0981, "my": 8.03907}, "gini_cpc": 0.508937, "gini_cps": 0.503186, "bpic": 2.073808, "worst_lang": "th", "cpc_parity_vs_en": {"en": 1.0, "de": 0.6527505721086835, "vi": 5.065959017784942, "tr": 2.0461216283502988, "fi": 1.6158909203846255, "ru": 6.110503700342001, "el": 10.331369135968322, "hi": 0.7491385023915687, "ta": 14.686028991378954, "th": 18.60201908943773, "zh": 14.768550196816587, "my": 2.585363368077987}, "t": 8.0}
|
| 8 |
+
{"event": "probe", "step": 8, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2264.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.130093, "cps": 17.6875, "mean_p": 0.475804, "boundary_entropy": 0.685259, "bpb": 7.392165}, "de": {"chunks": 1774.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.086398, "cps": 13.859375, "mean_p": 0.467284, "boundary_entropy": 0.685006, "bpb": 7.493377}, "vi": {"chunks": 12826.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.695742, "cps": 100.203125, "mean_p": 0.504008, "boundary_entropy": 0.688382, "bpb": 7.484874}, "tr": {"chunks": 5055.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.279235, "cps": 39.492188, "mean_p": 0.484228, "boundary_entropy": 0.686305, "bpb": 7.566448}, "fi": {"chunks": 3865.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.205465, "cps": 30.195312, "mean_p": 0.480947, "boundary_entropy": 0.686323, "bpb": 7.542082}, "ru": {"chunks": 15176.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.800844, "cps": 118.5625, "mean_p": 0.499958, "boundary_entropy": 0.690059, "bpb": 7.441214}, "el": {"chunks": 31671.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.543195, "cps": 247.429688, "mean_p": 0.518151, "boundary_entropy": 0.689698, "bpb": 7.560604}, "hi": {"chunks": 2197.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.130579, "cps": 17.164062, "mean_p": 0.479041, "boundary_entropy": 0.689771, "bpb": 7.440378}, "ta": {"chunks": 41737.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 2.232522, "cps": 326.070312, "mean_p": 0.516032, "boundary_entropy": 0.690459, "bpb": 7.55434}, "th": {"chunks": 44718.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 2.780106, "cps": 349.359375, "mean_p": 0.522485, "boundary_entropy": 0.690083, "bpb": 7.577396}, "zh": {"chunks": 13107.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 2.219269, "cps": 102.398438, "mean_p": 0.52958, "boundary_entropy": 0.684605, "bpb": 8.021169}, "my": {"chunks": 8247.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.435244, "cps": 64.429688, "mean_p": 0.492142, "boundary_entropy": 0.691056, "bpb": 7.85545}}, "bpb": {"en": 7.39216, "de": 7.49338, "vi": 7.48487, "tr": 7.56645, "fi": 7.54208, "ru": 7.44121, "el": 7.5606, "hi": 7.44038, "ta": 7.55434, "th": 7.5774, "zh": 8.02117, "my": 7.85545}, "gini_cpc": 0.517732, "gini_cps": 0.51075, "bpic": 2.101332, "worst_lang": "th", "cpc_parity_vs_en": {"en": 1.0, "de": 0.6641235886934743, "vi": 5.348054093218365, "tr": 2.146437791483324, "fi": 1.579375193598663, "ru": 6.155960170431766, "el": 11.8622925041955, "hi": 1.0037433515874807, "ta": 17.161034228081995, "th": 21.37022053775712, "zh": 17.059156541410246, "my": 3.3456485380110417}, "t": 9.9}
|
| 9 |
+
{"event": "probe", "step": 16, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 1470.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.084468, "cps": 11.484375, "mean_p": 0.471159, "boundary_entropy": 0.684859, "bpb": 6.763621}, "de": {"chunks": 1189.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.057907, "cps": 9.289062, "mean_p": 0.463065, "boundary_entropy": 0.684565, "bpb": 6.891711}, "vi": {"chunks": 9791.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.531109, "cps": 76.492188, "mean_p": 0.497074, "boundary_entropy": 0.688455, "bpb": 7.100707}, "tr": {"chunks": 3864.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.213445, "cps": 30.1875, "mean_p": 0.479568, "boundary_entropy": 0.686064, "bpb": 7.112817}, "fi": {"chunks": 2328.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.123757, "cps": 18.1875, "mean_p": 0.472976, "boundary_entropy": 0.685561, "bpb": 6.986445}, "ru": {"chunks": 10097.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.532823, "cps": 78.882812, "mean_p": 0.494456, "boundary_entropy": 0.689968, "bpb": 6.590353}, "el": {"chunks": 32038.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.561078, "cps": 250.296875, "mean_p": 0.518384, "boundary_entropy": 0.689714, "bpb": 7.120871}, "hi": {"chunks": 5305.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.315305, "cps": 41.445312, "mean_p": 0.487075, "boundary_entropy": 0.690393, "bpb": 6.736586}, "ta": {"chunks": 46034.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 2.46237, "cps": 359.640625, "mean_p": 0.519724, "boundary_entropy": 0.690229, "bpb": 6.86062}, "th": {"chunks": 45404.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 2.822754, "cps": 354.71875, "mean_p": 0.525404, "boundary_entropy": 0.689823, "bpb": 6.685688}, "zh": {"chunks": 13563.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 2.296478, "cps": 105.960938, "mean_p": 0.532697, "boundary_entropy": 0.684263, "bpb": 7.802035}, "my": {"chunks": 11189.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.590511, "cps": 87.414062, "mean_p": 0.493836, "boundary_entropy": 0.691104, "bpb": 7.307892}}, "bpb": {"en": 6.76362, "de": 6.89171, "vi": 7.10071, "tr": 7.11282, "fi": 6.98644, "ru": 6.59035, "el": 7.12087, "hi": 6.73659, "ta": 6.86062, "th": 6.68569, "zh": 7.80204, "my": 7.30789}, "gini_cpc": 0.53621, "gini_cps": 0.531771, "bpic": 2.10554, "worst_lang": "th", "cpc_parity_vs_en": {"en": 1.0, "de": 0.6855454186739713, "vi": 6.287683809080996, "tr": 2.526930816518178, "fi": 1.4651357922361972, "ru": 6.30797447633407, "el": 18.481249807334134, "hi": 3.7328204506170968, "ta": 29.15144112525995, "th": 33.417952332316204, "zh": 27.187489374347777, "my": 6.99092564828338}, "t": 12.0}
|
| 10 |
+
{"event": "probe", "step": 32, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 383.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.022008, "cps": 2.992188, "mean_p": 0.433994, "boundary_entropy": 0.677224, "bpb": 6.352728}, "de": {"chunks": 335.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.016315, "cps": 2.617188, "mean_p": 0.429047, "boundary_entropy": 0.676857, "bpb": 6.44189}, "vi": {"chunks": 581.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.031516, "cps": 4.539062, "mean_p": 0.45394, "boundary_entropy": 0.683796, "bpb": 6.864593}, "tr": {"chunks": 531.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.029332, "cps": 4.148438, "mean_p": 0.443549, "boundary_entropy": 0.680185, "bpb": 6.736257}, "fi": {"chunks": 349.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.018553, "cps": 2.726562, "mean_p": 0.43495, "boundary_entropy": 0.678082, "bpb": 6.507426}, "ru": {"chunks": 139.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.007335, "cps": 1.085938, "mean_p": 0.445588, "boundary_entropy": 0.683888, "bpb": 6.150857}, "el": {"chunks": 8892.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.43327, "cps": 69.46875, "mean_p": 0.491312, "boundary_entropy": 0.690112, "bpb": 6.302296}, "hi": {"chunks": 1311.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.07792, "cps": 10.242188, "mean_p": 0.481567, "boundary_entropy": 0.690105, "bpb": 6.098238}, "ta": {"chunks": 26982.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 1.443274, "cps": 210.796875, "mean_p": 0.503229, "boundary_entropy": 0.691096, "bpb": 6.387771}, "th": {"chunks": 27042.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 1.681194, "cps": 211.265625, "mean_p": 0.504089, "boundary_entropy": 0.691072, "bpb": 5.606693}, "zh": {"chunks": 11444.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.93769, "cps": 89.40625, "mean_p": 0.517991, "boundary_entropy": 0.685907, "bpb": 7.424534}, "my": {"chunks": 1991.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.105077, "cps": 15.554688, "mean_p": 0.481385, "boundary_entropy": 0.690415, "bpb": 6.382926}}, "bpb": {"en": 6.35273, "de": 6.44189, "vi": 6.86459, "tr": 6.73626, "fi": 6.50743, "ru": 6.15086, "el": 6.3023, "hi": 6.09824, "ta": 6.38777, "th": 5.60669, "zh": 7.42453, "my": 6.38293}, "gini_cpc": 0.69672, "gini_cps": 0.688793, "bpic": 4.798462, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 0.7413405332738905, "vi": 1.4320505112522228, "tr": 1.3328132939320676, "fi": 0.8430219638445534, "ru": 0.3332966374339508, "el": 19.687200083355503, "hi": 3.540568205183912, "ta": 65.58039296470692, "th": 76.39115729021168, "zh": 88.04602479754624, "my": 4.774558624373095}, "t": 14.8}
|
| 11 |
+
{"event": "probe", "step": 64, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 243.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.013963, "cps": 1.898438, "mean_p": 0.361583, "boundary_entropy": 0.645794, "bpb": 5.795061}, "de": {"chunks": 211.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.010276, "cps": 1.648438, "mean_p": 0.357859, "boundary_entropy": 0.645031, "bpb": 5.937274}, "vi": {"chunks": 184.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.009981, "cps": 1.4375, "mean_p": 0.366439, "boundary_entropy": 0.650482, "bpb": 6.511334}, "tr": {"chunks": 188.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.010385, "cps": 1.46875, "mean_p": 0.362114, "boundary_entropy": 0.646959, "bpb": 6.32238}, "fi": {"chunks": 166.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.008825, "cps": 1.296875, "mean_p": 0.359204, "boundary_entropy": 0.645288, "bpb": 6.008648}, "ru": {"chunks": 130.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.00686, "cps": 1.015625, "mean_p": 0.38614, "boundary_entropy": 0.663323, "bpb": 5.589999}, "el": {"chunks": 128.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.006237, "cps": 1.0, "mean_p": 0.420718, "boundary_entropy": 0.677257, "bpb": 6.311157}, "hi": {"chunks": 140.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.008321, "cps": 1.09375, "mean_p": 0.375218, "boundary_entropy": 0.658523, "bpb": 5.840602}, "ta": {"chunks": 146.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.00781, "cps": 1.140625, "mean_p": 0.373999, "boundary_entropy": 0.65836, "bpb": 5.796934}, "th": {"chunks": 131.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.008144, "cps": 1.023438, "mean_p": 0.384179, "boundary_entropy": 0.663251, "bpb": 5.662074}, "zh": {"chunks": 186.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.031493, "cps": 1.453125, "mean_p": 0.391527, "boundary_entropy": 0.660036, "bpb": 7.3059}, "my": {"chunks": 254.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.013405, "cps": 1.984375, "mean_p": 0.437689, "boundary_entropy": 0.683144, "bpb": 5.71819}}, "bpb": {"en": 5.79506, "de": 5.93727, "vi": 6.51133, "tr": 6.32238, "fi": 6.00865, "ru": 5.59, "el": 6.31116, "hi": 5.8406, "ta": 5.79693, "th": 5.66207, "zh": 7.3059, "my": 5.71819}, "gini_cpc": 0.245113, "gini_cps": 0.132139, "bpic": 182.145705, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 0.7359492969161957, "vi": 0.7148131405974277, "tr": 0.7437468586817683, "fi": 0.6319955949073663, "ru": 0.49130590572982835, "el": 0.4466701917691864, "hi": 0.5959239080586283, "ta": 0.5593005325910738, "th": 0.5832679016183318, "zh": 2.255471522996074, "my": 0.9600374774887476}, "t": 19.3}
|
| 12 |
+
{"event": "probe", "step": 128, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 133.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.007642, "cps": 1.039062, "mean_p": 0.379597, "boundary_entropy": 0.655613, "bpb": 4.548053}, "de": {"chunks": 152.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.007403, "cps": 1.1875, "mean_p": 0.37754, "boundary_entropy": 0.655408, "bpb": 4.753989}, "vi": {"chunks": 827.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.04486, "cps": 6.460938, "mean_p": 0.418843, "boundary_entropy": 0.671647, "bpb": 5.113376}, "tr": {"chunks": 300.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.016572, "cps": 2.34375, "mean_p": 0.396282, "boundary_entropy": 0.663492, "bpb": 5.134856}, "fi": {"chunks": 220.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.011695, "cps": 1.71875, "mean_p": 0.382113, "boundary_entropy": 0.656636, "bpb": 4.833313}, "ru": {"chunks": 161.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.008496, "cps": 1.257812, "mean_p": 0.416762, "boundary_entropy": 0.674024, "bpb": 3.777848}, "el": {"chunks": 128.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.006237, "cps": 1.0, "mean_p": 0.398727, "boundary_entropy": 0.669046, "bpb": 4.63175}, "hi": {"chunks": 135.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.008024, "cps": 1.054688, "mean_p": 0.428426, "boundary_entropy": 0.679865, "bpb": 3.545774}, "ta": {"chunks": 131.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.007007, "cps": 1.023438, "mean_p": 0.417263, "boundary_entropy": 0.676927, "bpb": 3.739576}, "th": {"chunks": 215.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.013366, "cps": 1.679688, "mean_p": 0.428008, "boundary_entropy": 0.677759, "bpb": 3.443156}, "zh": {"chunks": 139.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.023535, "cps": 1.085938, "mean_p": 0.412482, "boundary_entropy": 0.670152, "bpb": 6.367455}, "my": {"chunks": 128.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.006755, "cps": 1.0, "mean_p": 0.401239, "boundary_entropy": 0.6711, "bpb": 4.022402}}, "bpb": {"en": 4.54805, "de": 4.75399, "vi": 5.11338, "tr": 5.13486, "fi": 4.83331, "ru": 3.77785, "el": 4.63175, "hi": 3.54577, "ta": 3.73958, "th": 3.44316, "zh": 6.36745, "my": 4.0224}, "gini_cpc": 0.353397, "gini_cps": 0.323498, "bpic": 143.792057, "worst_lang": "vi", "cpc_parity_vs_en": {"en": 1.0, "de": 0.9686428119194885, "vi": 5.869956012896358, "tr": 2.168418892893173, "fi": 1.5303236028511553, "ru": 1.111703930009721, "el": 0.8160966661647541, "hi": 1.0499078304974918, "ta": 0.9168922573885906, "th": 1.7490002594300487, "zh": 3.079596638056367, "my": 0.8839324403472265}, "t": 27.5}
|
| 13 |
+
{"event": "train", "step": 200, "phase": 1, "objective": "global", "lm_loss": 2.58195, "bpb": 3.72497, "chunk_loss": 0.021673, "F": 0.00098, "G": 0.39935, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.66927, "router_noise": 0.0, "grad_norm": 1.4347, "lr": 0.000201, "router_trainable": true, "sec_per_step": 0.158, "alpha": 0.03, "t": 35.3}
|
| 14 |
+
{"event": "probe", "step": 256, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 491.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.028214, "cps": 3.835938, "mean_p": 0.414525, "boundary_entropy": 0.669415, "bpb": 3.672405}, "de": {"chunks": 840.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.04091, "cps": 6.5625, "mean_p": 0.405741, "boundary_entropy": 0.666839, "bpb": 3.88857}, "vi": {"chunks": 1137.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.061676, "cps": 8.882812, "mean_p": 0.414811, "boundary_entropy": 0.668569, "bpb": 3.272238}, "tr": {"chunks": 1118.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.061758, "cps": 8.734375, "mean_p": 0.413739, "boundary_entropy": 0.669019, "bpb": 3.924511}, "fi": {"chunks": 1019.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.05417, "cps": 7.960938, "mean_p": 0.420978, "boundary_entropy": 0.672208, "bpb": 3.919382}, "ru": {"chunks": 195.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.01029, "cps": 1.523438, "mean_p": 0.399764, "boundary_entropy": 0.66761, "bpb": 2.665971}, "el": {"chunks": 825.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.040199, "cps": 6.445312, "mean_p": 0.438363, "boundary_entropy": 0.681206, "bpb": 2.751758}, "hi": {"chunks": 418.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.024844, "cps": 3.265625, "mean_p": 0.425079, "boundary_entropy": 0.677312, "bpb": 2.163683}, "ta": {"chunks": 175.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.009361, "cps": 1.367188, "mean_p": 0.439973, "boundary_entropy": 0.682977, "bpb": 2.040198}, "th": {"chunks": 2904.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.180541, "cps": 22.6875, "mean_p": 0.456926, "boundary_entropy": 0.685649, "bpb": 2.111542}, "zh": {"chunks": 436.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.073823, "cps": 3.40625, "mean_p": 0.40505, "boundary_entropy": 0.664369, "bpb": 5.011176}, "my": {"chunks": 130.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.006861, "cps": 1.015625, "mean_p": 0.419887, "boundary_entropy": 0.676671, "bpb": 2.270936}}, "bpb": {"en": 3.6724, "de": 3.88857, "vi": 3.27224, "tr": 3.92451, "fi": 3.91938, "ru": 2.66597, "el": 2.75176, "hi": 2.16368, "ta": 2.0402, "th": 2.11154, "zh": 5.01118, "my": 2.27094}, "gini_cpc": 0.437818, "gini_cps": 0.431668, "bpic": 39.614059, "worst_lang": "th", "cpc_parity_vs_en": {"en": 1.0, "de": 1.4500050239527984, "vi": 2.1860492941291496, "tr": 2.1889401144593177, "fi": 1.9200163573746392, "ru": 0.3647270929501475, "el": 1.4248059873811043, "hi": 0.8805699016169203, "ta": 0.33178382317935734, "th": 6.3990893591643685, "zh": 2.616589984433656, "my": 0.24317706042520917}, "t": 41.9}
|
| 15 |
+
{"event": "train", "step": 400, "phase": 1, "objective": "global", "lm_loss": 2.03632, "bpb": 2.93779, "chunk_loss": 0.024239, "F": 0.02814, "G": 0.36063, "chunks": 28.81, "bytes_per_chunk": 35.54, "boundary_entropy": 0.64731, "router_noise": 0.0, "grad_norm": 1.7345, "lr": 0.0002998, "router_trainable": true, "sec_per_step": 0.105, "alpha": 0.03, "t": 56.3}
|
| 16 |
+
{"event": "train", "step": 600, "phase": 1, "objective": "global", "lm_loss": 1.87477, "bpb": 2.70472, "chunk_loss": 0.022649, "F": 0.00098, "G": 0.37207, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65357, "router_noise": 0.0, "grad_norm": 0.7489, "lr": 0.0002982, "router_trainable": true, "sec_per_step": 0.095, "alpha": 0.03, "t": 75.3}
|
| 17 |
+
{"event": "train", "step": 800, "phase": 1, "objective": "global", "lm_loss": 1.67521, "bpb": 2.41682, "chunk_loss": 0.022193, "F": 0.00098, "G": 0.38481, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65926, "router_noise": 0.0, "grad_norm": 0.8214, "lr": 0.0002949, "router_trainable": true, "sec_per_step": 0.096, "alpha": 0.03, "t": 94.6}
|
| 18 |
+
{"event": "eval", "step": 1000, "phase": 1, "bpb": {"en": 2.6676, "de": 2.98901, "vi": 2.3604, "tr": 2.97268, "fi": 3.23071, "ru": 1.92693, "el": 2.07715, "hi": 1.58912, "ta": 1.37095, "th": 1.62376, "zh": 3.71974, "my": 1.45979}, "bytes_per_chunk": {"en": 88.923, "de": 193.179, "vi": 722.16, "tr": 107.745, "fi": 293.226, "ru": 113.188, "el": 473.184, "hi": 188.052, "ta": 439.102, "th": 946.368, "zh": 826.953, "my": 341.333}, "bpb_mean": 2.33232, "t": 113.4}
|
| 19 |
+
{"event": "train", "step": 1000, "phase": 1, "objective": "global", "lm_loss": 1.78443, "bpb": 2.57438, "chunk_loss": 0.025766, "F": 0.04855, "G": 0.33372, "chunks": 49.72, "bytes_per_chunk": 20.596, "boundary_entropy": 0.62891, "router_noise": 0.0, "grad_norm": 0.8492, "lr": 0.0002901, "router_trainable": true, "sec_per_step": 0.094, "alpha": 0.03, "t": 113.4}
|
| 20 |
+
{"event": "train", "step": 1200, "phase": 1, "objective": "global", "lm_loss": 1.65627, "bpb": 2.3895, "chunk_loss": 0.023015, "F": 0.00098, "G": 0.36183, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.64658, "router_noise": 0.0, "grad_norm": 0.8266, "lr": 0.0002837, "router_trainable": true, "sec_per_step": 0.093, "alpha": 0.03, "t": 132.0}
|
| 21 |
+
{"event": "train", "step": 1400, "phase": 1, "objective": "global", "lm_loss": 1.61348, "bpb": 2.32776, "chunk_loss": 0.02191, "F": 0.00098, "G": 0.39272, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.66309, "router_noise": 0.0, "grad_norm": 0.9687, "lr": 0.0002759, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 149.9}
|
| 22 |
+
{"event": "train", "step": 1600, "phase": 1, "objective": "global", "lm_loss": 1.62579, "bpb": 2.34551, "chunk_loss": 0.023298, "F": 0.00098, "G": 0.35393, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.64151, "router_noise": 0.0, "grad_norm": 0.8963, "lr": 0.0002668, "router_trainable": true, "sec_per_step": 0.088, "alpha": 0.03, "t": 167.5}
|
| 23 |
+
{"event": "train", "step": 1800, "phase": 1, "objective": "global", "lm_loss": 1.62437, "bpb": 2.34347, "chunk_loss": 0.023368, "F": 0.00098, "G": 0.35197, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.63951, "router_noise": 0.0, "grad_norm": 1.2577, "lr": 0.0002564, "router_trainable": true, "sec_per_step": 0.091, "alpha": 0.03, "t": 185.7}
|
| 24 |
+
{"event": "probe", "step": 2000, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 257.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.014768, "cps": 2.007812, "mean_p": 0.355916, "boundary_entropy": 0.634868, "bpb": 2.334603}, "de": {"chunks": 336.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.016364, "cps": 2.625, "mean_p": 0.356176, "boundary_entropy": 0.636464, "bpb": 2.615902}, "vi": {"chunks": 347.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.018823, "cps": 2.710938, "mean_p": 0.356379, "boundary_entropy": 0.638153, "bpb": 2.12749}, "tr": {"chunks": 382.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.021101, "cps": 2.984375, "mean_p": 0.35538, "boundary_entropy": 0.635151, "bpb": 2.613461}, "fi": {"chunks": 233.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.012386, "cps": 1.820312, "mean_p": 0.36462, "boundary_entropy": 0.642233, "bpb": 2.848301}, "ru": {"chunks": 543.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.028654, "cps": 4.242188, "mean_p": 0.350991, "boundary_entropy": 0.638405, "bpb": 1.688301}, "el": {"chunks": 448.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.021829, "cps": 3.5, "mean_p": 0.351743, "boundary_entropy": 0.638343, "bpb": 1.7839}, "hi": {"chunks": 1177.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.069955, "cps": 9.195312, "mean_p": 0.359596, "boundary_entropy": 0.641316, "bpb": 1.411766}, "ta": {"chunks": 1155.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.061781, "cps": 9.023438, "mean_p": 0.365126, "boundary_entropy": 0.645848, "bpb": 1.127496}, "th": {"chunks": 224.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.013926, "cps": 1.75, "mean_p": 0.350389, "boundary_entropy": 0.638766, "bpb": 1.318851}, "zh": {"chunks": 137.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.023197, "cps": 1.070312, "mean_p": 0.349868, "boundary_entropy": 0.633739, "bpb": 3.50542}, "my": {"chunks": 210.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.011083, "cps": 1.640625, "mean_p": 0.321643, "boundary_entropy": 0.613809, "bpb": 1.251973}}, "bpb": {"en": 2.3346, "de": 2.6159, "vi": 2.12749, "tr": 2.61346, "fi": 2.8483, "ru": 1.6883, "el": 1.7839, "hi": 1.41177, "ta": 1.1275, "th": 1.31885, "zh": 3.50542, "my": 1.25197}, "gini_cpc": 0.333492, "gini_cps": 0.361519, "bpic": 70.431455, "worst_lang": "hi", "cpc_parity_vs_en": {"en": 1.0, "de": 1.1080972245304654, "vi": 1.2746100242834482, "tr": 1.4289064886164793, "fi": 0.8387548307172702, "ru": 1.9403568678582794, "el": 1.4781828719832413, "hi": 4.737098189765323, "ta": 4.183574542393095, "th": 0.943013586624559, "zh": 1.5707899768223568, "my": 0.7504934660414043}, "t": 203.2}
|
| 25 |
+
{"event": "eval", "step": 2000, "phase": 1, "bpb": {"en": 2.34455, "de": 2.62199, "vi": 2.13169, "tr": 2.62324, "fi": 2.84831, "ru": 1.66786, "el": 1.77523, "hi": 1.40934, "ta": 1.13558, "th": 1.33676, "zh": 3.50947, "my": 1.26013}, "bytes_per_chunk": {"en": 86.345, "de": 75.199, "vi": 144.194, "tr": 80.709, "fi": 131.863, "ru": 97.162, "el": 117.501, "hi": 43.394, "ta": 53.685, "th": 179.92, "zh": 564.966, "my": 227.358}, "bpb_mean": 2.05535, "t": 204.4}
|
| 26 |
+
{"event": "train", "step": 2000, "phase": 1, "objective": "global", "lm_loss": 1.55604, "bpb": 2.24489, "chunk_loss": 0.025613, "F": 0.0563, "G": 0.35118, "chunks": 57.66, "bytes_per_chunk": 17.76, "boundary_entropy": 0.63933, "router_noise": 0.0, "grad_norm": 0.8047, "lr": 0.000245, "router_trainable": true, "sec_per_step": 0.094, "alpha": 0.03, "t": 204.4}
|
| 27 |
+
{"event": "train", "step": 2200, "phase": 1, "objective": "global", "lm_loss": 1.52653, "bpb": 2.20232, "chunk_loss": 0.022589, "F": 0.00098, "G": 0.37375, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65299, "router_noise": 0.0, "grad_norm": 0.7575, "lr": 0.0002325, "router_trainable": true, "sec_per_step": 0.088, "alpha": 0.03, "t": 222.0}
|
| 28 |
+
{"event": "phase_transition", "step": 2400, "phase": 2, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0, "t": 239.2}
|
| 29 |
+
{"event": "train", "step": 2400, "phase": 2, "objective": "global", "lm_loss": 1.55384, "bpb": 2.24171, "chunk_loss": 0.054117, "F": 0.73846, "G": 0.36101, "chunks": 756.19, "bytes_per_chunk": 1.354, "boundary_entropy": 0.64339, "router_noise": 0.0, "grad_norm": 0.8794, "lr": 0.0002192, "router_trainable": true, "sec_per_step": 0.086, "alpha": 0.03, "t": 239.3}
|
| 30 |
+
{"event": "train", "step": 2600, "phase": 2, "objective": "global", "lm_loss": 1.71623, "bpb": 2.47599, "chunk_loss": 0.022556, "F": 0.00098, "G": 0.37466, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65261, "router_noise": 0.0, "grad_norm": 0.805, "lr": 0.0002053, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 257.0}
|
| 31 |
+
{"event": "train", "step": 2800, "phase": 2, "objective": "global", "lm_loss": 1.50444, "bpb": 2.17045, "chunk_loss": 0.023924, "F": 0.02304, "G": 0.36283, "chunks": 23.59, "bytes_per_chunk": 43.401, "boundary_entropy": 0.64536, "router_noise": 0.0, "grad_norm": 0.7812, "lr": 0.0001909, "router_trainable": true, "sec_per_step": 0.087, "alpha": 0.03, "t": 274.4}
|
| 32 |
+
{"event": "eval", "step": 3000, "phase": 2, "bpb": {"en": 2.21551, "de": 2.45319, "vi": 2.04296, "tr": 2.43092, "fi": 2.65147, "ru": 1.53801, "el": 1.61944, "hi": 1.31357, "ta": 1.08274, "th": 1.23433, "zh": 3.36493, "my": 1.18579}, "bytes_per_chunk": {"en": 40.661, "de": 53.521, "vi": 82.747, "tr": 63.442, "fi": 86.288, "ru": 55.211, "el": 71.998, "hi": 36.419, "ta": 54.534, "th": 127.938, "zh": 89.043, "my": 69.369}, "bpb_mean": 1.92774, "t": 292.8}
|
| 33 |
+
{"event": "train", "step": 3000, "phase": 2, "objective": "global", "lm_loss": 1.52656, "bpb": 2.20236, "chunk_loss": 0.022673, "F": 0.00098, "G": 0.3714, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.6496, "router_noise": 0.0, "grad_norm": 0.8643, "lr": 0.0001761, "router_trainable": true, "sec_per_step": 0.093, "alpha": 0.03, "t": 292.9}
|
| 34 |
+
{"event": "train", "step": 3200, "phase": 2, "objective": "global", "lm_loss": 1.36113, "bpb": 1.9637, "chunk_loss": 0.02255, "F": 0.00098, "G": 0.37483, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65182, "router_noise": 0.0, "grad_norm": 0.9588, "lr": 0.0001613, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 310.7}
|
| 35 |
+
{"event": "train", "step": 3400, "phase": 2, "objective": "global", "lm_loss": 1.4062, "bpb": 2.02872, "chunk_loss": 0.036152, "F": 0.30194, "G": 0.3746, "chunks": 309.19, "bytes_per_chunk": 3.312, "boundary_entropy": 0.65167, "router_noise": 0.0, "grad_norm": 0.7784, "lr": 0.0001465, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 328.5}
|
| 36 |
+
{"event": "train", "step": 3600, "phase": 2, "objective": "global", "lm_loss": 1.52441, "bpb": 2.19926, "chunk_loss": 0.048164, "F": 0.57361, "G": 0.37125, "chunks": 587.38, "bytes_per_chunk": 1.743, "boundary_entropy": 0.6504, "router_noise": 0.0, "grad_norm": 1.7498, "lr": 0.0001319, "router_trainable": true, "sec_per_step": 0.093, "alpha": 0.03, "t": 347.1}
|
| 37 |
+
{"event": "train", "step": 3800, "phase": 2, "objective": "global", "lm_loss": 1.41311, "bpb": 2.03869, "chunk_loss": 0.042968, "F": 0.48413, "G": 0.35482, "chunks": 495.75, "bytes_per_chunk": 2.066, "boundary_entropy": 0.63936, "router_noise": 0.0, "grad_norm": 0.8064, "lr": 0.0001177, "router_trainable": true, "sec_per_step": 0.088, "alpha": 0.03, "t": 364.7}
|
| 38 |
+
{"event": "probe", "step": 4000, "phase": 2, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 395.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.022697, "cps": 3.085938, "mean_p": 0.367709, "boundary_entropy": 0.639918, "bpb": 2.140504}, "de": {"chunks": 337.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.016413, "cps": 2.632812, "mean_p": 0.367454, "boundary_entropy": 0.643109, "bpb": 2.355928}, "vi": {"chunks": 457.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.02479, "cps": 3.570312, "mean_p": 0.372393, "boundary_entropy": 0.646221, "bpb": 1.985718}, "tr": {"chunks": 464.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.025631, "cps": 3.625, "mean_p": 0.376195, "boundary_entropy": 0.647673, "bpb": 2.32451}, "fi": {"chunks": 290.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.015417, "cps": 2.265625, "mean_p": 0.375932, "boundary_entropy": 0.648623, "bpb": 2.539534}, "ru": {"chunks": 833.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.043958, "cps": 6.507812, "mean_p": 0.38216, "boundary_entropy": 0.656092, "bpb": 1.489035}, "el": {"chunks": 489.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.023827, "cps": 3.820312, "mean_p": 0.381608, "boundary_entropy": 0.656643, "bpb": 1.541612}, "hi": {"chunks": 1202.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.071441, "cps": 9.390625, "mean_p": 0.372576, "boundary_entropy": 0.647457, "bpb": 1.269919}, "ta": {"chunks": 826.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.044183, "cps": 6.453125, "mean_p": 0.371251, "boundary_entropy": 0.647964, "bpb": 1.03482}, "th": {"chunks": 508.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.031582, "cps": 3.96875, "mean_p": 0.377789, "boundary_entropy": 0.654107, "bpb": 1.163126}, "zh": {"chunks": 210.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.035557, "cps": 1.640625, "mean_p": 0.370316, "boundary_entropy": 0.64517, "bpb": 3.272949}, "my": {"chunks": 303.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.015991, "cps": 2.367188, "mean_p": 0.362003, "boundary_entropy": 0.64003, "bpb": 1.137374}}, "bpb": {"en": 2.1405, "de": 2.35593, "vi": 1.98572, "tr": 2.32451, "fi": 2.53953, "ru": 1.48904, "el": 1.54161, "hi": 1.26992, "ta": 1.03482, "th": 1.16313, "zh": 3.27295, "my": 1.13737}, "gini_cpc": 0.258247, "gini_cps": 0.271935, "bpic": 60.782547, "worst_lang": "hi", "cpc_parity_vs_en": {"en": 1.0, "de": 0.7231102510500231, "vi": 1.0921947451360063, "tr": 1.129261322516654, "fi": 0.6792241813805415, "ru": 1.9367020473598076, "el": 1.049772129694563, "hi": 3.1475774070382, "ta": 1.9466218883408208, "th": 1.3914566208787966, "zh": 1.5665810782426794, "my": 0.704541008970587}, "t": 383.9}
|
| 39 |
+
{"event": "eval", "step": 4000, "phase": 2, "bpb": {"en": 2.15032, "de": 2.36116, "vi": 1.98784, "tr": 2.33377, "fi": 2.54249, "ru": 1.46527, "el": 1.53314, "hi": 1.26588, "ta": 1.04159, "th": 1.18211, "zh": 3.27315, "my": 1.14581}, "bytes_per_chunk": {"en": 44.184, "de": 86.09, "vi": 81.059, "tr": 60.194, "fi": 103.737, "ru": 57.614, "el": 95.743, "hi": 40.839, "ta": 64.172, "th": 97.162, "zh": 161.022, "my": 191.486}, "bpb_mean": 1.85688, "t": 385.1}
|
| 40 |
+
{"event": "train", "step": 4000, "phase": 2, "objective": "global", "lm_loss": 1.45512, "bpb": 2.0993, "chunk_loss": 0.022719, "F": 0.00098, "G": 0.37011, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.64993, "router_noise": 0.0, "grad_norm": 0.8842, "lr": 0.000104, "router_trainable": true, "sec_per_step": 0.102, "alpha": 0.03, "t": 385.2}
|
| 41 |
+
{"event": "phase_transition", "step": 4200, "phase": 3, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0, "t": 403.4}
|
| 42 |
+
{"event": "train", "step": 4200, "phase": 3, "objective": "global", "lm_loss": 1.29843, "bpb": 1.87324, "chunk_loss": 0.055929, "F": 0.70004, "G": 0.3908, "chunks": 716.84, "bytes_per_chunk": 1.428, "boundary_entropy": 0.66019, "router_noise": 0.0, "grad_norm": 0.818, "lr": 9.12e-05, "router_trainable": true, "sec_per_step": 0.091, "alpha": 0.03, "t": 403.4}
|
| 43 |
+
{"event": "train", "step": 4400, "phase": 3, "objective": "global", "lm_loss": 1.37957, "bpb": 1.9903, "chunk_loss": 0.022627, "F": 0.00421, "G": 0.37679, "chunks": 4.31, "bytes_per_chunk": 237.449, "boundary_entropy": 0.65397, "router_noise": 0.0, "grad_norm": 0.8301, "lr": 7.92e-05, "router_trainable": true, "sec_per_step": 0.092, "alpha": 0.03, "t": 421.9}
|
| 44 |
+
{"event": "train", "step": 4600, "phase": 3, "objective": "global", "lm_loss": 1.49599, "bpb": 2.15826, "chunk_loss": 0.039673, "F": 0.39044, "G": 0.36615, "chunks": 399.81, "bytes_per_chunk": 2.561, "boundary_entropy": 0.64758, "router_noise": 0.0, "grad_norm": 0.7863, "lr": 6.82e-05, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 439.6}
|
| 45 |
+
{"event": "train", "step": 4800, "phase": 3, "objective": "global", "lm_loss": 1.30487, "bpb": 1.88253, "chunk_loss": 0.022297, "F": 0.00098, "G": 0.38191, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65608, "router_noise": 0.0, "grad_norm": 0.8666, "lr": 5.85e-05, "router_trainable": true, "sec_per_step": 0.093, "alpha": 0.03, "t": 458.2}
|
| 46 |
+
{"event": "eval", "step": 5000, "phase": 3, "bpb": {"en": 2.10913, "de": 2.31166, "vi": 1.96233, "tr": 2.28259, "fi": 2.47792, "ru": 1.42173, "el": 1.49422, "hi": 1.23133, "ta": 1.01495, "th": 1.14546, "zh": 3.22296, "my": 1.11971}, "bytes_per_chunk": {"en": 48.073, "de": 70.907, "vi": 71.742, "tr": 65.194, "fi": 86.947, "ru": 67.459, "el": 144.831, "hi": 49.21, "ta": 76.92, "th": 118.403, "zh": 149.2, "my": 121.814}, "bpb_mean": 1.81617, "t": 476.3}
|
| 47 |
+
{"event": "train", "step": 5000, "phase": 3, "objective": "global", "lm_loss": 1.51414, "bpb": 2.18444, "chunk_loss": 0.025256, "F": 0.05795, "G": 0.36871, "chunks": 59.34, "bytes_per_chunk": 17.255, "boundary_entropy": 0.64801, "router_noise": 0.0, "grad_norm": 0.8675, "lr": 5e-05, "router_trainable": true, "sec_per_step": 0.091, "alpha": 0.03, "t": 476.4}
|
| 48 |
+
{"event": "train", "step": 5200, "phase": 3, "objective": "global", "lm_loss": 1.40099, "bpb": 2.02121, "chunk_loss": 0.022523, "F": 0.00351, "G": 0.37881, "chunks": 3.59, "bytes_per_chunk": 284.939, "boundary_entropy": 0.65434, "router_noise": 0.0, "grad_norm": 0.8979, "lr": 4.29e-05, "router_trainable": true, "sec_per_step": 0.084, "alpha": 0.03, "t": 493.2}
|
| 49 |
+
{"event": "train", "step": 5400, "phase": 3, "objective": "global", "lm_loss": 1.27771, "bpb": 1.84335, "chunk_loss": 0.022471, "F": 0.00098, "G": 0.37704, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65352, "router_noise": 0.0, "grad_norm": 0.8255, "lr": 3.73e-05, "router_trainable": true, "sec_per_step": 0.09, "alpha": 0.03, "t": 511.1}
|
| 50 |
+
{"event": "train", "step": 5600, "phase": 3, "objective": "global", "lm_loss": 1.36022, "bpb": 1.96238, "chunk_loss": 0.022816, "F": 0.00098, "G": 0.36739, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.64794, "router_noise": 0.0, "grad_norm": 0.8147, "lr": 3.33e-05, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 528.9}
|
| 51 |
+
{"event": "train", "step": 5800, "phase": 3, "objective": "global", "lm_loss": 1.38027, "bpb": 1.99131, "chunk_loss": 0.028322, "F": 0.12729, "G": 0.36518, "chunks": 130.34, "bytes_per_chunk": 7.856, "boundary_entropy": 0.64685, "router_noise": 0.0, "grad_norm": 0.8599, "lr": 3.08e-05, "router_trainable": true, "sec_per_step": 0.084, "alpha": 0.03, "t": 545.7}
|
| 52 |
+
{"event": "probe", "step": 6000, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 386.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.02218, "cps": 3.015625, "mean_p": 0.372235, "boundary_entropy": 0.642723, "bpb": 2.081353}, "de": {"chunks": 371.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.018068, "cps": 2.898438, "mean_p": 0.378678, "boundary_entropy": 0.64967, "bpb": 2.281939}, "vi": {"chunks": 535.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.029021, "cps": 4.179688, "mean_p": 0.379752, "boundary_entropy": 0.650654, "bpb": 1.945007}, "tr": {"chunks": 366.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.020218, "cps": 2.859375, "mean_p": 0.37595, "boundary_entropy": 0.647683, "bpb": 2.245198}, "fi": {"chunks": 283.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.015044, "cps": 2.210938, "mean_p": 0.37811, "boundary_entropy": 0.649958, "bpb": 2.445838}, "ru": {"chunks": 744.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.039261, "cps": 5.8125, "mean_p": 0.378459, "boundary_entropy": 0.654131, "bpb": 1.425385}, "el": {"chunks": 375.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.018272, "cps": 2.929688, "mean_p": 0.372367, "boundary_entropy": 0.651811, "bpb": 1.473511}, "hi": {"chunks": 921.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.05474, "cps": 7.195312, "mean_p": 0.375501, "boundary_entropy": 0.650672, "bpb": 1.217113}, "ta": {"chunks": 707.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.037818, "cps": 5.523438, "mean_p": 0.362146, "boundary_entropy": 0.640883, "bpb": 0.992749}, "th": {"chunks": 417.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.025925, "cps": 3.257812, "mean_p": 0.370949, "boundary_entropy": 0.649942, "bpb": 1.111332}, "zh": {"chunks": 228.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.038605, "cps": 1.78125, "mean_p": 0.375965, "boundary_entropy": 0.648166, "bpb": 3.184903}, "my": {"chunks": 280.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.014777, "cps": 2.1875, "mean_p": 0.360745, "boundary_entropy": 0.639457, "bpb": 1.09482}}, "bpb": {"en": 2.08135, "de": 2.28194, "vi": 1.94501, "tr": 2.2452, "fi": 2.44584, "ru": 1.42538, "el": 1.47351, "hi": 1.21711, "ta": 0.99275, "th": 1.11133, "zh": 3.1849, "my": 1.09482}, "gini_cpc": 0.23188, "gini_cps": 0.233995, "bpic": 68.373597, "worst_lang": "hi", "cpc_parity_vs_en": {"en": 1.0, "de": 0.8146260953869533, "vi": 1.3084208484930249, "tr": 0.911522408188721, "fi": 0.6782836797893858, "ru": 1.7701111460483683, "el": 0.823810338912061, "hi": 2.4679785047232636, "ta": 1.7050250337085502, "th": 1.168831225307265, "zh": 1.74051680121559, "my": 0.6662411771075679}, "t": 564.4}
|
| 53 |
+
{"event": "eval", "step": 6000, "phase": 3, "bpb": {"en": 2.09161, "de": 2.28849, "vi": 1.94771, "tr": 2.25268, "fi": 2.44957, "ru": 1.4018, "el": 1.466, "hi": 1.21198, "ta": 1.00013, "th": 1.1284, "zh": 3.18371, "my": 1.10389}, "bytes_per_chunk": {"en": 46.996, "de": 68.284, "vi": 72.136, "tr": 83.752, "fi": 113.778, "ru": 66.433, "el": 148.271, "hi": 53.119, "ta": 81.563, "th": 134.158, "zh": 132.53, "my": 205.281}, "bpb_mean": 1.79383, "t": 565.6}
|
| 54 |
+
{"event": "probe", "step": 6000, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 386.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.02218, "cps": 3.015625, "mean_p": 0.372235, "boundary_entropy": 0.642723, "bpb": 2.081353}, "de": {"chunks": 371.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.018068, "cps": 2.898438, "mean_p": 0.378678, "boundary_entropy": 0.64967, "bpb": 2.281939}, "vi": {"chunks": 535.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.029021, "cps": 4.179688, "mean_p": 0.379752, "boundary_entropy": 0.650654, "bpb": 1.945007}, "tr": {"chunks": 366.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.020218, "cps": 2.859375, "mean_p": 0.37595, "boundary_entropy": 0.647683, "bpb": 2.245198}, "fi": {"chunks": 283.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.015044, "cps": 2.210938, "mean_p": 0.37811, "boundary_entropy": 0.649958, "bpb": 2.445838}, "ru": {"chunks": 744.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.039261, "cps": 5.8125, "mean_p": 0.378459, "boundary_entropy": 0.654131, "bpb": 1.425385}, "el": {"chunks": 375.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.018272, "cps": 2.929688, "mean_p": 0.372367, "boundary_entropy": 0.651811, "bpb": 1.473511}, "hi": {"chunks": 921.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.05474, "cps": 7.195312, "mean_p": 0.375501, "boundary_entropy": 0.650672, "bpb": 1.217113}, "ta": {"chunks": 707.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.037818, "cps": 5.523438, "mean_p": 0.362146, "boundary_entropy": 0.640883, "bpb": 0.992749}, "th": {"chunks": 417.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.025925, "cps": 3.257812, "mean_p": 0.370949, "boundary_entropy": 0.649942, "bpb": 1.111332}, "zh": {"chunks": 228.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.038605, "cps": 1.78125, "mean_p": 0.375965, "boundary_entropy": 0.648166, "bpb": 3.184903}, "my": {"chunks": 280.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.014777, "cps": 2.1875, "mean_p": 0.360745, "boundary_entropy": 0.639457, "bpb": 1.09482}}, "bpb": {"en": 2.08135, "de": 2.28194, "vi": 1.94501, "tr": 2.2452, "fi": 2.44584, "ru": 1.42538, "el": 1.47351, "hi": 1.21711, "ta": 0.99275, "th": 1.11133, "zh": 3.1849, "my": 1.09482}, "gini_cpc": 0.23188, "gini_cps": 0.233995, "bpic": 68.373597, "worst_lang": "hi", "cpc_parity_vs_en": {"en": 1.0, "de": 0.8146260953869533, "vi": 1.3084208484930249, "tr": 0.911522408188721, "fi": 0.6782836797893858, "ru": 1.7701111460483683, "el": 0.823810338912061, "hi": 2.4679785047232636, "ta": 1.7050250337085502, "th": 1.168831225307265, "zh": 1.74051680121559, "my": 0.6662411771075679}, "t": 566.5}
|
| 55 |
+
{"event": "reps_dumped", "dirs": ["/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s1/th0.30", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s1/th0.40", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s1/th0.50", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s1/th0.60", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s1/th0.70"], "t": 569.9}
|
| 56 |
+
{"event": "final", "run": "jitter_s1", "condition": "A", "step": 6000, "params": 22516992, "per_lang": {"en": {"cpc": 0.022180083893581566, "cps": 3.015625, "bpb": 2.1020406444164355, "bytes_per_chunk": 46.865826405649415}, "de": {"cpc": 0.018068475137583404, "cps": 2.8984375, "bpb": 2.2796363360511163, "bytes_per_chunk": 68.94897422409258}, "vi": {"cpc": 0.029020884187686467, "cps": 4.1796875, "bpb": 1.9494764901690524, "bytes_per_chunk": 70.51620712844654}, "tr": {"cpc": 0.020217643484505332, "cps": 2.859375, "bpb": 2.2458832574689267, "bytes_per_chunk": 85.72400261608895}, "fi": {"cpc": 0.01504438892137579, "cps": 2.2109375, "bpb": 2.4640121534753403, "bytes_per_chunk": 111.19575821845174}, "ru": {"cpc": 0.03926121372031662, "cps": 5.8125, "bpb": 1.4033274435589955, "bytes_per_chunk": 66.50025367833587}, "el": {"cpc": 0.018272182429469375, "cps": 2.9296875, "bpb": 1.4606392699744462, "bytes_per_chunk": 149.24224309706804}, "hi": {"cpc": 0.05473997028231798, "cps": 7.1953125, "bpb": 1.2202438124075716, "bytes_per_chunk": 53.54248366013072}, "ta": {"cpc": 0.03781759828831238, "cps": 5.5234375, "bpb": 0.9998630346897966, "bytes_per_chunk": 80.46163290362185}, "th": {"cpc": 0.025924774634752876, "cps": 3.2578125, "bpb": 1.126236773329332, "bytes_per_chunk": 134.95186615186614}, "zh": {"cpc": 0.03860480866915002, "cps": 1.78125, "bpb": 3.1923234986783178, "bytes_per_chunk": 134.3639159405433}, "my": {"cpc": 0.01477728520160439, "cps": 2.1875, "bpb": 1.114218518197882, "bytes_per_chunk": 210.81141938078005}}, "gini_cpc": 0.2318796127695224, "gini_cps": 0.2339954866678544, "bpic": 68.37359700694816, "seed": 1, "mean_boundary_entropy": 0.6504928469657898, "dechunk": "ema", "router_noise": 0.0, "transitions": [{"step": 0, "phase": 1, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}, {"step": 2400, "phase": 2, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}, {"step": 4200, "phase": 3, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}], "t": 572.1}
|
jitter_s1/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:bfd6e2aa4bef6e6e743aa332eb060d28696a3bbef699a4c59d8b9593961885f8
|
| 3 |
+
size 90078288
|
jitter_s2/config.json
ADDED
|
@@ -0,0 +1,108 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"run": "jitter_s2",
|
| 3 |
+
"step": 6000,
|
| 4 |
+
"model_config": {
|
| 5 |
+
"vocab_size": 256,
|
| 6 |
+
"d_enc": 256,
|
| 7 |
+
"d_main": 512,
|
| 8 |
+
"n_enc": 2,
|
| 9 |
+
"n_dec": 2,
|
| 10 |
+
"n_main": 6,
|
| 11 |
+
"n_heads_enc": 4,
|
| 12 |
+
"n_heads_main": 8,
|
| 13 |
+
"seq_len": 1024,
|
| 14 |
+
"dropout": 0.0,
|
| 15 |
+
"target_ratio": 6.0,
|
| 16 |
+
"max_chunk_mult": 3.0,
|
| 17 |
+
"dechunk": "ema",
|
| 18 |
+
"router_noise": 0.0,
|
| 19 |
+
"rope_base": 10000.0
|
| 20 |
+
},
|
| 21 |
+
"tied_weights": {
|
| 22 |
+
"head.weight": "emb.weight"
|
| 23 |
+
},
|
| 24 |
+
"train_config": {
|
| 25 |
+
"condition": "A",
|
| 26 |
+
"seed": 2,
|
| 27 |
+
"out_dir": null,
|
| 28 |
+
"run_name": "jitter_s2",
|
| 29 |
+
"corpus": "corpus",
|
| 30 |
+
"langs": null,
|
| 31 |
+
"seq_len": 1024,
|
| 32 |
+
"batch_size": 32,
|
| 33 |
+
"grad_accum": 1,
|
| 34 |
+
"steps": 6000,
|
| 35 |
+
"lr": 0.0003,
|
| 36 |
+
"min_lr_frac": 0.1,
|
| 37 |
+
"warmup": 300,
|
| 38 |
+
"weight_decay": 0.1,
|
| 39 |
+
"grad_clip": 1.0,
|
| 40 |
+
"betas": [
|
| 41 |
+
0.9,
|
| 42 |
+
0.95
|
| 43 |
+
],
|
| 44 |
+
"router_lr_mult": 1.0,
|
| 45 |
+
"d_enc": 256,
|
| 46 |
+
"d_main": 512,
|
| 47 |
+
"n_enc": 2,
|
| 48 |
+
"n_dec": 2,
|
| 49 |
+
"n_main": 6,
|
| 50 |
+
"n_heads_enc": 4,
|
| 51 |
+
"n_heads_main": 8,
|
| 52 |
+
"dechunk": "ema",
|
| 53 |
+
"target_ratio": 6.0,
|
| 54 |
+
"alpha": 0.03,
|
| 55 |
+
"alpha_parity_cap": 0.06,
|
| 56 |
+
"global_anchor": 0.0,
|
| 57 |
+
"premium_clip": 4.0,
|
| 58 |
+
"premium_kind": "flores_per_sent",
|
| 59 |
+
"mixture": null,
|
| 60 |
+
"doc_filters": null,
|
| 61 |
+
"tau1": 0.4,
|
| 62 |
+
"tau2": 0.7,
|
| 63 |
+
"freeze_scope": "router",
|
| 64 |
+
"ckpt_every": 2000,
|
| 65 |
+
"eval_every": 1000,
|
| 66 |
+
"log_every": 200,
|
| 67 |
+
"probe_sents": 128,
|
| 68 |
+
"eval_batches": 8,
|
| 69 |
+
"keep_ckpt": false,
|
| 70 |
+
"amp": "bf16",
|
| 71 |
+
"compile": false,
|
| 72 |
+
"cpc_decay": 0.9,
|
| 73 |
+
"device": "cuda",
|
| 74 |
+
"log_spaced_probe": true,
|
| 75 |
+
"probe_bpb_batches": 4,
|
| 76 |
+
"dump_reps": true,
|
| 77 |
+
"dump_reps_thresholds": [
|
| 78 |
+
0.3,
|
| 79 |
+
0.4,
|
| 80 |
+
0.5,
|
| 81 |
+
0.6,
|
| 82 |
+
0.7
|
| 83 |
+
],
|
| 84 |
+
"router_noise": 0.0,
|
| 85 |
+
"genreg_target_entropy": null,
|
| 86 |
+
"genreg_eta": 0.05
|
| 87 |
+
},
|
| 88 |
+
"probe_agg": {
|
| 89 |
+
"gini_cpc": 0.262783775060067,
|
| 90 |
+
"gini_cps": 0.27856758212170174,
|
| 91 |
+
"bpic": 62.00016155088853,
|
| 92 |
+
"worst_lang": "ta",
|
| 93 |
+
"cpc_parity_vs_en": {
|
| 94 |
+
"en": 1.0,
|
| 95 |
+
"de": 0.9160887870174738,
|
| 96 |
+
"vi": 1.4180378442947907,
|
| 97 |
+
"tr": 1.677218187651982,
|
| 98 |
+
"fi": 0.9645182707718549,
|
| 99 |
+
"ru": 2.3095880536686693,
|
| 100 |
+
"el": 2.042357873522286,
|
| 101 |
+
"hi": 3.6796580569694286,
|
| 102 |
+
"ta": 3.9968877786187003,
|
| 103 |
+
"th": 2.9373511729574933,
|
| 104 |
+
"zh": 2.9717424040463722,
|
| 105 |
+
"my": 3.1892094376995943
|
| 106 |
+
}
|
| 107 |
+
}
|
| 108 |
+
}
|
jitter_s2/final.json
ADDED
|
@@ -0,0 +1,114 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"event": "final",
|
| 3 |
+
"run": "jitter_s2",
|
| 4 |
+
"condition": "A",
|
| 5 |
+
"step": 6000,
|
| 6 |
+
"params": 22516992,
|
| 7 |
+
"per_lang": {
|
| 8 |
+
"en": {
|
| 9 |
+
"cpc": 0.013503418950755616,
|
| 10 |
+
"cps": 1.8359375,
|
| 11 |
+
"bpb": 2.091184529801444,
|
| 12 |
+
"bytes_per_chunk": 119.15636363636364
|
| 13 |
+
},
|
| 14 |
+
"de": {
|
| 15 |
+
"cpc": 0.01237033068718648,
|
| 16 |
+
"cps": 1.984375,
|
| 17 |
+
"bpb": 2.2734026958836058,
|
| 18 |
+
"bytes_per_chunk": 135.02137522534125
|
| 19 |
+
},
|
| 20 |
+
"vi": {
|
| 21 |
+
"cpc": 0.01914835909953892,
|
| 22 |
+
"cps": 2.7578125,
|
| 23 |
+
"bpb": 1.9494414809506864,
|
| 24 |
+
"bytes_per_chunk": 86.68783068783068
|
| 25 |
+
},
|
| 26 |
+
"tr": {
|
| 27 |
+
"cpc": 0.022648179859691763,
|
| 28 |
+
"cps": 3.203125,
|
| 29 |
+
"bpb": 2.240100760041376,
|
| 30 |
+
"bytes_per_chunk": 89.34696659850034
|
| 31 |
+
},
|
| 32 |
+
"fi": {
|
| 33 |
+
"cpc": 0.013024294295890703,
|
| 34 |
+
"cps": 1.9140625,
|
| 35 |
+
"bpb": 2.4580483003424547,
|
| 36 |
+
"bytes_per_chunk": 132.8656867714141
|
| 37 |
+
},
|
| 38 |
+
"ru": {
|
| 39 |
+
"cpc": 0.031187335092348285,
|
| 40 |
+
"cps": 4.6171875,
|
| 41 |
+
"bpb": 1.4135231609180474,
|
| 42 |
+
"bytes_per_chunk": 73.35777249195466
|
| 43 |
+
},
|
| 44 |
+
"el": {
|
| 45 |
+
"cpc": 0.02757881401354578,
|
| 46 |
+
"cps": 4.421875,
|
| 47 |
+
"bpb": 1.4792314436970544,
|
| 48 |
+
"bytes_per_chunk": 67.52807831014941
|
| 49 |
+
},
|
| 50 |
+
"hi": {
|
| 51 |
+
"cpc": 0.04968796433878157,
|
| 52 |
+
"cps": 6.53125,
|
| 53 |
+
"bpb": 1.2194370256690006,
|
| 54 |
+
"bytes_per_chunk": 54.50545794781162
|
| 55 |
+
},
|
| 56 |
+
"ta": {
|
| 57 |
+
"cpc": 0.053971650173843276,
|
| 58 |
+
"cps": 7.8828125,
|
| 59 |
+
"bpb": 0.9995220038420514,
|
| 60 |
+
"bytes_per_chunk": 63.92197025115826
|
| 61 |
+
},
|
| 62 |
+
"th": {
|
| 63 |
+
"cpc": 0.039664283493938454,
|
| 64 |
+
"cps": 4.984375,
|
| 65 |
+
"bpb": 1.1435730867396772,
|
| 66 |
+
"bytes_per_chunk": 81.6393646838991
|
| 67 |
+
},
|
| 68 |
+
"zh": {
|
| 69 |
+
"cpc": 0.040128682695563835,
|
| 70 |
+
"cps": 1.8515625,
|
| 71 |
+
"bpb": 3.1307407679749546,
|
| 72 |
+
"bytes_per_chunk": 120.83152800184374
|
| 73 |
+
},
|
| 74 |
+
"my": {
|
| 75 |
+
"cpc": 0.04306523115896137,
|
| 76 |
+
"cps": 6.375,
|
| 77 |
+
"bpb": 1.1354728113731958,
|
| 78 |
+
"bytes_per_chunk": 50.768664665440106
|
| 79 |
+
}
|
| 80 |
+
},
|
| 81 |
+
"gini_cpc": 0.262783775060067,
|
| 82 |
+
"gini_cps": 0.27856758212170174,
|
| 83 |
+
"bpic": 62.00016155088853,
|
| 84 |
+
"seed": 2,
|
| 85 |
+
"mean_boundary_entropy": 0.6519113779067993,
|
| 86 |
+
"dechunk": "ema",
|
| 87 |
+
"router_noise": 0.0,
|
| 88 |
+
"transitions": [
|
| 89 |
+
{
|
| 90 |
+
"step": 0,
|
| 91 |
+
"phase": 1,
|
| 92 |
+
"objective": "global",
|
| 93 |
+
"router_trainable": true,
|
| 94 |
+
"encoder_trainable": true,
|
| 95 |
+
"frozen_router_params": 0
|
| 96 |
+
},
|
| 97 |
+
{
|
| 98 |
+
"step": 2400,
|
| 99 |
+
"phase": 2,
|
| 100 |
+
"objective": "global",
|
| 101 |
+
"router_trainable": true,
|
| 102 |
+
"encoder_trainable": true,
|
| 103 |
+
"frozen_router_params": 0
|
| 104 |
+
},
|
| 105 |
+
{
|
| 106 |
+
"step": 4200,
|
| 107 |
+
"phase": 3,
|
| 108 |
+
"objective": "global",
|
| 109 |
+
"router_trainable": true,
|
| 110 |
+
"encoder_trainable": true,
|
| 111 |
+
"frozen_router_params": 0
|
| 112 |
+
}
|
| 113 |
+
]
|
| 114 |
+
}
|
jitter_s2/log.jsonl
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"event": "start", "run": "jitter_s2", "condition": "A", "params": 22516992, "langs": ["en", "de", "vi", "tr", "fi", "ru", "el", "hi", "ta", "th", "zh", "my"], "premium": {"en": 1.0, "de": 1.184, "vi": 1.39, "tr": 1.12, "fi": 1.104, "ru": 1.971, "el": 2.174, "hi": 2.552, "ta": 3.192, "th": 2.762, "zh": 0.915, "my": 3.524}, "schedule": {"tau1": 0.4, "tau2": 0.7, "p1_objective": "global", "p2_objective": "global", "p3_objective": "global", "freeze_in_p3": false, "freeze_scope": "router"}, "corpus": "/local/scratch/sas245/beetle-hnet/scratch/data/tier1_corpus", "premium_kind": "flores_per_sent", "doc_filtered": {"my": 37511}, "weights": {"en": 0.2946, "de": 0.0909, "vi": 0.0576, "tr": 0.0612, "fi": 0.0424, "ru": 0.1476, "el": 0.0522, "hi": 0.042, "ta": 0.0313, "th": 0.0542, "zh": 0.1038, "my": 0.0222}, "t": 1.5}
|
| 2 |
+
{"event": "probe_schedule", "n": 13, "steps": [0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 2000, 4000, 6000], "t": 1.5}
|
| 3 |
+
{"event": "probe", "step": 0, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 10933.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.628225, "cps": 85.414062, "mean_p": 0.513249, "boundary_entropy": 0.686295, "bpb": 8.12598}, "de": {"chunks": 16287.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.793211, "cps": 127.242188, "mean_p": 0.524631, "boundary_entropy": 0.686391, "bpb": 8.12701}, "vi": {"chunks": 5099.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.276593, "cps": 39.835938, "mean_p": 0.481643, "boundary_entropy": 0.68728, "bpb": 7.869214}, "tr": {"chunks": 8187.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.452245, "cps": 63.960938, "mean_p": 0.498002, "boundary_entropy": 0.686826, "bpb": 8.097536}, "fi": {"chunks": 13540.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.719792, "cps": 105.78125, "mean_p": 0.517169, "boundary_entropy": 0.686661, "bpb": 8.065117}, "ru": {"chunks": 33477.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 1.766596, "cps": 261.539062, "mean_p": 0.530376, "boundary_entropy": 0.688543, "bpb": 8.240733}, "el": {"chunks": 20696.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.00843, "cps": 161.6875, "mean_p": 0.503648, "boundary_entropy": 0.690112, "bpb": 8.286272}, "hi": {"chunks": 875.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.052006, "cps": 6.835938, "mean_p": 0.461978, "boundary_entropy": 0.68743, "bpb": 8.022688}, "ta": {"chunks": 1261.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.067451, "cps": 9.851562, "mean_p": 0.477518, "boundary_entropy": 0.690053, "bpb": 8.091383}, "th": {"chunks": 3630.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.225676, "cps": 28.359375, "mean_p": 0.484845, "boundary_entropy": 0.690398, "bpb": 7.721507}, "zh": {"chunks": 11722.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.984761, "cps": 91.578125, "mean_p": 0.520316, "boundary_entropy": 0.68562, "bpb": 8.130734}, "my": {"chunks": 1628.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.085919, "cps": 12.71875, "mean_p": 0.474628, "boundary_entropy": 0.689685, "bpb": 7.8029}}, "bpb": {"en": 8.12598, "de": 8.12701, "vi": 7.86921, "tr": 8.09754, "fi": 8.06512, "ru": 8.24073, "el": 8.28627, "hi": 8.02269, "ta": 8.09138, "th": 7.72151, "zh": 8.13073, "my": 7.8029}, "gini_cpc": 0.489538, "gini_cps": 0.463551, "bpic": 3.013947, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 1.2626223171148012, "vi": 0.440277652395235, "tr": 0.7198781817505053, "fi": 1.1457544508645294, "ru": 2.8120438593717987, "el": 1.6052044042347315, "hi": 0.08278235025738971, "ta": 0.10736788276924242, "th": 0.3592281253258146, "zh": 3.159315851384177, "my": 0.13676526035979295}, "t": 3.3}
|
| 4 |
+
{"event": "train", "step": 0, "phase": 1, "objective": "global", "lm_loss": 5.64451, "bpb": 8.14331, "chunk_loss": 0.017745, "F": 0.00098, "G": 0.5091, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.69066, "router_noise": 0.0, "grad_norm": 4.2033, "lr": 1e-06, "router_trainable": true, "sec_per_step": 0.001, "alpha": 0.03, "t": 3.8}
|
| 5 |
+
{"event": "probe", "step": 1, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 10973.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.630523, "cps": 85.726562, "mean_p": 0.513415, "boundary_entropy": 0.686291, "bpb": 8.112407}, "de": {"chunks": 16325.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.795062, "cps": 127.539062, "mean_p": 0.524793, "boundary_entropy": 0.686378, "bpb": 8.114037}, "vi": {"chunks": 5118.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.277624, "cps": 39.984375, "mean_p": 0.481707, "boundary_entropy": 0.687285, "bpb": 7.860188}, "tr": {"chunks": 8241.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.455228, "cps": 64.382812, "mean_p": 0.49822, "boundary_entropy": 0.68683, "bpb": 8.088402}, "fi": {"chunks": 13565.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.721121, "cps": 105.976562, "mean_p": 0.517278, "boundary_entropy": 0.686656, "bpb": 8.053366}, "ru": {"chunks": 33499.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 1.767757, "cps": 261.710938, "mean_p": 0.530674, "boundary_entropy": 0.688508, "bpb": 8.214279}, "el": {"chunks": 20725.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.009843, "cps": 161.914062, "mean_p": 0.503684, "boundary_entropy": 0.690112, "bpb": 8.285335}, "hi": {"chunks": 878.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.052184, "cps": 6.859375, "mean_p": 0.462055, "boundary_entropy": 0.687442, "bpb": 8.01674}, "ta": {"chunks": 1287.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.068842, "cps": 10.054688, "mean_p": 0.477649, "boundary_entropy": 0.690066, "bpb": 8.087633}, "th": {"chunks": 3656.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.227293, "cps": 28.5625, "mean_p": 0.484903, "boundary_entropy": 0.690401, "bpb": 7.719771}, "zh": {"chunks": 11731.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.986285, "cps": 91.648438, "mean_p": 0.520388, "boundary_entropy": 0.685615, "bpb": 8.128736}, "my": {"chunks": 1618.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.085392, "cps": 12.640625, "mean_p": 0.474603, "boundary_entropy": 0.689682, "bpb": 7.801052}}, "bpb": {"en": 8.11241, "de": 8.11404, "vi": 7.86019, "tr": 8.0884, "fi": 8.05337, "ru": 8.21428, "el": 8.28534, "hi": 8.01674, "ta": 8.08763, "th": 7.71977, "zh": 8.12874, "my": 7.80105}, "gini_cpc": 0.488839, "gini_cps": 0.462872, "bpic": 3.007311, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 1.2609548133156332, "vi": 0.4403072946933804, "tr": 0.7219848819260126, "fi": 1.1436856070130392, "ru": 2.8036343320027304, "el": 1.6015940070465446, "hi": 0.0827633733970008, "ta": 0.1091821943169754, "th": 0.3604822315015682, "zh": 3.1502160013546776, "my": 0.13542968932840913}, "t": 5.1}
|
| 6 |
+
{"event": "probe", "step": 2, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 10974.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.630581, "cps": 85.734375, "mean_p": 0.513418, "boundary_entropy": 0.686297, "bpb": 8.074849}, "de": {"chunks": 16315.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.794575, "cps": 127.460938, "mean_p": 0.524791, "boundary_entropy": 0.686383, "bpb": 8.078929}, "vi": {"chunks": 5118.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.277624, "cps": 39.984375, "mean_p": 0.481666, "boundary_entropy": 0.687281, "bpb": 7.841203}, "tr": {"chunks": 8290.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.457935, "cps": 64.765625, "mean_p": 0.498428, "boundary_entropy": 0.686835, "bpb": 8.062411}, "fi": {"chunks": 13551.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.720376, "cps": 105.867188, "mean_p": 0.517218, "boundary_entropy": 0.686664, "bpb": 8.018631}, "ru": {"chunks": 33514.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 1.768549, "cps": 261.828125, "mean_p": 0.530885, "boundary_entropy": 0.688481, "bpb": 8.141092}, "el": {"chunks": 20782.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.01262, "cps": 162.359375, "mean_p": 0.50375, "boundary_entropy": 0.690111, "bpb": 8.284844}, "hi": {"chunks": 903.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.05367, "cps": 7.054688, "mean_p": 0.462219, "boundary_entropy": 0.687468, "bpb": 7.986524}, "ta": {"chunks": 1324.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.070821, "cps": 10.34375, "mean_p": 0.477855, "boundary_entropy": 0.690085, "bpb": 8.071872}, "th": {"chunks": 3686.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.229158, "cps": 28.796875, "mean_p": 0.484947, "boundary_entropy": 0.690403, "bpb": 7.703635}, "zh": {"chunks": 11732.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.986454, "cps": 91.65625, "mean_p": 0.52037, "boundary_entropy": 0.685615, "bpb": 8.122673}, "my": {"chunks": 1610.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.084969, "cps": 12.578125, "mean_p": 0.474585, "boundary_entropy": 0.68968, "bpb": 7.800115}}, "bpb": {"en": 8.07485, "de": 8.07893, "vi": 7.8412, "tr": 8.06241, "fi": 8.01863, "ru": 8.14109, "el": 8.28484, "hi": 7.98652, "ta": 8.07187, "th": 7.70364, "zh": 8.12267, "my": 7.80012}, "gini_cpc": 0.488024, "gini_cps": 0.462073, "bpic": 3.003005, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 1.2600675726178374, "vi": 0.44026717192185744, "tr": 0.7262115359751005, "fi": 1.1424011357081152, "ru": 2.804634134026552, "el": 1.6058525277498072, "hi": 0.08511220496589128, "ta": 0.11231084116104524, "th": 0.3634071183823363, "zh": 3.1501974528224856, "my": 0.1347477941124701}, "t": 6.7}
|
| 7 |
+
{"event": "probe", "step": 4, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 10958.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.629662, "cps": 85.609375, "mean_p": 0.513328, "boundary_entropy": 0.686323, "bpb": 7.950483}, "de": {"chunks": 16341.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.795841, "cps": 127.664062, "mean_p": 0.524718, "boundary_entropy": 0.686408, "bpb": 7.966392}, "vi": {"chunks": 5092.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.276214, "cps": 39.78125, "mean_p": 0.481544, "boundary_entropy": 0.687268, "bpb": 7.774413}, "tr": {"chunks": 8458.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.467215, "cps": 66.078125, "mean_p": 0.499042, "boundary_entropy": 0.686851, "bpb": 7.975067}, "fi": {"chunks": 13519.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.718675, "cps": 105.617188, "mean_p": 0.516944, "boundary_entropy": 0.686695, "bpb": 7.903791}, "ru": {"chunks": 33540.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 1.769921, "cps": 262.03125, "mean_p": 0.531365, "boundary_entropy": 0.688418, "bpb": 7.887699}, "el": {"chunks": 20880.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.017395, "cps": 163.125, "mean_p": 0.503844, "boundary_entropy": 0.690112, "bpb": 8.253629}, "hi": {"chunks": 958.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.056939, "cps": 7.484375, "mean_p": 0.462719, "boundary_entropy": 0.687546, "bpb": 7.88726}, "ta": {"chunks": 1398.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.074779, "cps": 10.921875, "mean_p": 0.478275, "boundary_entropy": 0.690123, "bpb": 7.984734}, "th": {"chunks": 3708.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.230525, "cps": 28.96875, "mean_p": 0.484792, "boundary_entropy": 0.690388, "bpb": 7.620259}, "zh": {"chunks": 11691.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.979512, "cps": 91.335938, "mean_p": 0.520229, "boundary_entropy": 0.685623, "bpb": 8.09962}, "my": {"chunks": 1582.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.083492, "cps": 12.359375, "mean_p": 0.474452, "boundary_entropy": 0.689665, "bpb": 7.794292}}, "bpb": {"en": 7.95048, "de": 7.96639, "vi": 7.77441, "tr": 7.97507, "fi": 7.90379, "ru": 7.8877, "el": 8.25363, "hi": 7.88726, "ta": 7.98473, "th": 7.62026, "zh": 8.09962, "my": 7.79429}, "gini_cpc": 0.486186, "gini_cps": 0.460751, "bpic": 2.995364, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 1.2639184309070375, "vi": 0.43867014390770237, "tr": 0.7420103335401932, "fi": 1.1413675175727382, "ru": 2.8109082363411075, "el": 1.615780911923772, "hi": 0.09042806967681818, "ta": 0.1187611860032478, "th": 0.3661099096939097, "zh": 3.1437720027835248, "my": 0.13259768052143592}, "t": 8.3}
|
| 8 |
+
{"event": "probe", "step": 8, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 10564.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.607022, "cps": 82.53125, "mean_p": 0.511303, "boundary_entropy": 0.686489, "bpb": 7.511635}, "de": {"chunks": 16085.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.783373, "cps": 125.664062, "mean_p": 0.522715, "boundary_entropy": 0.686663, "bpb": 7.567076}, "vi": {"chunks": 4846.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.26287, "cps": 37.859375, "mean_p": 0.480592, "boundary_entropy": 0.687177, "bpb": 7.546083}, "tr": {"chunks": 8897.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.491466, "cps": 69.507812, "mean_p": 0.500753, "boundary_entropy": 0.686901, "bpb": 7.649824}, "fi": {"chunks": 13006.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.691404, "cps": 101.609375, "mean_p": 0.514458, "boundary_entropy": 0.68689, "bpb": 7.498431}, "ru": {"chunks": 33618.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 1.774037, "cps": 262.640625, "mean_p": 0.53289, "boundary_entropy": 0.688209, "bpb": 7.323119}, "el": {"chunks": 20734.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 1.010281, "cps": 161.984375, "mean_p": 0.503587, "boundary_entropy": 0.69012, "bpb": 8.046873}, "hi": {"chunks": 1190.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.070728, "cps": 9.296875, "mean_p": 0.46461, "boundary_entropy": 0.687837, "bpb": 7.578698}, "ta": {"chunks": 1773.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.094838, "cps": 13.851562, "mean_p": 0.479794, "boundary_entropy": 0.690254, "bpb": 7.732736}, "th": {"chunks": 3813.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.237053, "cps": 29.789062, "mean_p": 0.484558, "boundary_entropy": 0.69036, "bpb": 7.38321}, "zh": {"chunks": 11553.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.956146, "cps": 90.257812, "mean_p": 0.519491, "boundary_entropy": 0.685665, "bpb": 8.031371}, "my": {"chunks": 1309.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.069084, "cps": 10.226562, "mean_p": 0.473199, "boundary_entropy": 0.689534, "bpb": 7.664766}}, "bpb": {"en": 7.51163, "de": 7.56708, "vi": 7.54608, "tr": 7.64982, "fi": 7.49843, "ru": 7.32312, "el": 8.04687, "hi": 7.5787, "ta": 7.73274, "th": 7.38321, "zh": 8.03137, "my": 7.66477}, "gini_cpc": 0.482797, "gini_cps": 0.457572, "bpic": 3.012694, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 1.290518948883884, "vi": 0.43304795844708327, "tr": 0.8096340541396518, "fi": 1.1390101492051576, "ru": 2.922526018069033, "el": 1.6643243864351969, "hi": 0.1165165498007475, "ta": 0.156235238157639, "th": 0.3905183697879156, "zh": 3.2225306624431505, "my": 0.11380779213834358}, "t": 10.0}
|
| 9 |
+
{"event": "probe", "step": 16, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 6431.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.369534, "cps": 50.242188, "mean_p": 0.495391, "boundary_entropy": 0.686738, "bpb": 6.909054}, "de": {"chunks": 11557.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.56285, "cps": 90.289062, "mean_p": 0.505708, "boundary_entropy": 0.687738, "bpb": 6.988178}, "vi": {"chunks": 3303.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.17917, "cps": 25.804688, "mean_p": 0.474137, "boundary_entropy": 0.686527, "bpb": 7.205538}, "tr": {"chunks": 8697.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.480418, "cps": 67.945312, "mean_p": 0.499746, "boundary_entropy": 0.687046, "bpb": 7.180256}, "fi": {"chunks": 8100.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.430599, "cps": 63.28125, "mean_p": 0.497915, "boundary_entropy": 0.687331, "bpb": 6.96291}, "ru": {"chunks": 32668.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 1.723905, "cps": 255.21875, "mean_p": 0.52748, "boundary_entropy": 0.688774, "bpb": 6.461576}, "el": {"chunks": 17688.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.861862, "cps": 138.1875, "mean_p": 0.499546, "boundary_entropy": 0.690143, "bpb": 7.466983}, "hi": {"chunks": 2156.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.128143, "cps": 16.84375, "mean_p": 0.469561, "boundary_entropy": 0.68855, "bpb": 6.649744}, "ta": {"chunks": 3273.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.175074, "cps": 25.570312, "mean_p": 0.483503, "boundary_entropy": 0.690536, "bpb": 6.839187}, "th": {"chunks": 2779.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.17277, "cps": 21.710938, "mean_p": 0.480699, "boundary_entropy": 0.690044, "bpb": 6.667273}, "zh": {"chunks": 10446.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 1.76871, "cps": 81.609375, "mean_p": 0.514169, "boundary_entropy": 0.685944, "bpb": 7.815019}, "my": {"chunks": 772.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.040743, "cps": 6.03125, "mean_p": 0.469418, "boundary_entropy": 0.689109, "bpb": 7.320966}}, "bpb": {"en": 6.90905, "de": 6.98818, "vi": 7.20554, "tr": 7.18026, "fi": 6.96291, "ru": 6.46158, "el": 7.46698, "hi": 6.64974, "ta": 6.83919, "th": 6.66727, "zh": 7.81502, "my": 7.32097}, "gini_cpc": 0.496787, "gini_cps": 0.468006, "bpic": 3.55781, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 1.5231347154694974, "vi": 0.48485406643142664, "tr": 1.3000633918148687, "fi": 1.1652490192049516, "ru": 4.665078361777489, "el": 2.3322950740787403, "hi": 0.3467682240300107, "ta": 0.4737684612956207, "th": 0.46753388484351743, "zh": 4.786325053211452, "my": 0.11025531513019436}, "t": 12.0}
|
| 10 |
+
{"event": "probe", "step": 32, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 546.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.031374, "cps": 4.265625, "mean_p": 0.454282, "boundary_entropy": 0.682264, "bpb": 6.444821}, "de": {"chunks": 480.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.023377, "cps": 3.75, "mean_p": 0.457648, "boundary_entropy": 0.684041, "bpb": 6.531547}, "vi": {"chunks": 739.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.040087, "cps": 5.773438, "mean_p": 0.453877, "boundary_entropy": 0.683603, "bpb": 6.918218}, "tr": {"chunks": 1827.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.100922, "cps": 14.273438, "mean_p": 0.470851, "boundary_entropy": 0.685377, "bpb": 6.768646}, "fi": {"chunks": 513.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.027271, "cps": 4.007812, "mean_p": 0.454461, "boundary_entropy": 0.683028, "bpb": 6.550628}, "ru": {"chunks": 10752.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.567388, "cps": 84.0, "mean_p": 0.494644, "boundary_entropy": 0.690088, "bpb": 5.97938}, "el": {"chunks": 15611.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.760659, "cps": 121.960938, "mean_p": 0.497568, "boundary_entropy": 0.690242, "bpb": 6.440687}, "hi": {"chunks": 355.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.0211, "cps": 2.773438, "mean_p": 0.457841, "boundary_entropy": 0.68691, "bpb": 6.323972}, "ta": {"chunks": 296.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.015833, "cps": 2.3125, "mean_p": 0.449685, "boundary_entropy": 0.685801, "bpb": 6.390968}, "th": {"chunks": 244.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.015169, "cps": 1.90625, "mean_p": 0.444814, "boundary_entropy": 0.684494, "bpb": 6.253931}, "zh": {"chunks": 4991.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.845073, "cps": 38.992188, "mean_p": 0.490494, "boundary_entropy": 0.685902, "bpb": 7.462509}, "my": {"chunks": 361.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.019052, "cps": 2.820312, "mean_p": 0.46235, "boundary_entropy": 0.688224, "bpb": 6.498228}}, "bpb": {"en": 6.44482, "de": 6.53155, "vi": 6.91822, "tr": 6.76865, "fi": 6.55063, "ru": 5.97938, "el": 6.44069, "hi": 6.32397, "ta": 6.39097, "th": 6.25393, "zh": 7.46251, "my": 6.49823}, "gini_cpc": 0.679691, "gini_cps": 0.689394, "bpic": 10.452976, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 0.7451098553226835, "vi": 1.2777114125364737, "tr": 3.216766026880373, "fi": 0.8692345079831125, "ru": 18.084708747716665, "el": 24.24495353109591, "hi": 0.6725193083291152, "ta": 0.5046586470496607, "th": 0.4835041862085692, "zh": 26.935534918856966, "my": 0.6072608782393457}, "t": 14.9}
|
| 11 |
+
{"event": "probe", "step": 64, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 207.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.011895, "cps": 1.617188, "mean_p": 0.357804, "boundary_entropy": 0.643436, "bpb": 5.94692}, "de": {"chunks": 159.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.007744, "cps": 1.242188, "mean_p": 0.361607, "boundary_entropy": 0.647364, "bpb": 6.083577}, "vi": {"chunks": 225.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.012205, "cps": 1.757812, "mean_p": 0.372776, "boundary_entropy": 0.653829, "bpb": 6.594595}, "tr": {"chunks": 162.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.008949, "cps": 1.265625, "mean_p": 0.366458, "boundary_entropy": 0.64941, "bpb": 6.436281}, "fi": {"chunks": 174.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.00925, "cps": 1.359375, "mean_p": 0.354408, "boundary_entropy": 0.642381, "bpb": 6.053196}, "ru": {"chunks": 132.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.006966, "cps": 1.03125, "mean_p": 0.410378, "boundary_entropy": 0.673523, "bpb": 5.788062}, "el": {"chunks": 10920.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.532086, "cps": 85.3125, "mean_p": 0.495321, "boundary_entropy": 0.6904, "bpb": 5.750818}, "hi": {"chunks": 160.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.00951, "cps": 1.25, "mean_p": 0.466272, "boundary_entropy": 0.688437, "bpb": 5.804437}, "ta": {"chunks": 192.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.01027, "cps": 1.5, "mean_p": 0.460835, "boundary_entropy": 0.687975, "bpb": 5.786206}, "th": {"chunks": 133.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.008269, "cps": 1.039062, "mean_p": 0.417771, "boundary_entropy": 0.676863, "bpb": 5.474193}, "zh": {"chunks": 149.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.025229, "cps": 1.164062, "mean_p": 0.410025, "boundary_entropy": 0.668755, "bpb": 7.191683}, "my": {"chunks": 180.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.0095, "cps": 1.40625, "mean_p": 0.424851, "boundary_entropy": 0.679574, "bpb": 5.903939}}, "bpb": {"en": 5.94692, "de": 6.08358, "vi": 6.5946, "tr": 6.43628, "fi": 6.0532, "ru": 5.78806, "el": 5.75082, "hi": 5.80444, "ta": 5.78621, "th": 5.47419, "zh": 7.19168, "my": 5.90394}, "gini_cpc": 0.764345, "gini_cps": 0.782583, "bpic": 29.999296, "worst_lang": "el", "cpc_parity_vs_en": {"en": 1.0, "de": 0.6510262377212505, "vi": 1.026108182686525, "tr": 0.7523470767516314, "fi": 0.7776624685371417, "ru": 0.5856234943214409, "el": 44.73377695014501, "hi": 0.7995003983892154, "ta": 0.863434770980383, "th": 0.6951593211787018, "zh": 2.1210289707838257, "my": 0.7986617837376435}, "t": 19.5}
|
| 12 |
+
{"event": "probe", "step": 128, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 2951.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.169568, "cps": 23.054688, "mean_p": 0.475111, "boundary_entropy": 0.684644, "bpb": 4.546814}, "de": {"chunks": 4472.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.217796, "cps": 34.9375, "mean_p": 0.479347, "boundary_entropy": 0.686126, "bpb": 4.773277}, "vi": {"chunks": 1209.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.065582, "cps": 9.445312, "mean_p": 0.46267, "boundary_entropy": 0.685101, "bpb": 5.290019}, "tr": {"chunks": 4751.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.262443, "cps": 37.117188, "mean_p": 0.480246, "boundary_entropy": 0.685829, "bpb": 5.212214}, "fi": {"chunks": 1847.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.098187, "cps": 14.429688, "mean_p": 0.469739, "boundary_entropy": 0.684909, "bpb": 4.813523}, "ru": {"chunks": 1792.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.094565, "cps": 14.0, "mean_p": 0.467962, "boundary_entropy": 0.687257, "bpb": 3.879847}, "el": {"chunks": 252.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.012279, "cps": 1.96875, "mean_p": 0.454998, "boundary_entropy": 0.686074, "bpb": 4.633605}, "hi": {"chunks": 233.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.013848, "cps": 1.820312, "mean_p": 0.478027, "boundary_entropy": 0.689579, "bpb": 3.718622}, "ta": {"chunks": 13734.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.734635, "cps": 107.296875, "mean_p": 0.468733, "boundary_entropy": 0.687419, "bpb": 3.773613}, "th": {"chunks": 2057.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.127883, "cps": 16.070312, "mean_p": 0.437202, "boundary_entropy": 0.679734, "bpb": 3.607527}, "zh": {"chunks": 335.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.056722, "cps": 2.617188, "mean_p": 0.47456, "boundary_entropy": 0.685611, "bpb": 6.43275}, "my": {"chunks": 1035.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.054623, "cps": 8.085938, "mean_p": 0.48045, "boundary_entropy": 0.690493, "bpb": 4.309861}}, "bpb": {"en": 4.54681, "de": 4.77328, "vi": 5.29002, "tr": 5.21221, "fi": 4.81352, "ru": 3.87985, "el": 4.6336, "hi": 3.71862, "ta": 3.77361, "th": 3.60753, "zh": 6.43275, "my": 4.30986}, "gini_cpc": 0.527577, "gini_cps": 0.5532, "bpic": 11.07018, "worst_lang": "ta", "cpc_parity_vs_en": {"en": 1.0, "de": 1.2844118343073394, "vi": 0.386756899165899, "tr": 1.5477092909725707, "fi": 0.5790418091925529, "ru": 0.5576782433216593, "el": 0.07241267754357077, "hi": 0.08166872186273774, "ta": 4.332379421067414, "th": 0.754168062816912, "zh": 0.334507820074652, "my": 0.3221305279883871}, "t": 28.3}
|
| 13 |
+
{"event": "train", "step": 200, "phase": 1, "objective": "global", "lm_loss": 2.5608, "bpb": 3.69445, "chunk_loss": 0.022629, "F": 0.00098, "G": 0.37261, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65348, "router_noise": 0.0, "grad_norm": 1.8307, "lr": 0.000201, "router_trainable": true, "sec_per_step": 0.159, "alpha": 0.03, "t": 35.6}
|
| 14 |
+
{"event": "probe", "step": 256, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 138.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.00793, "cps": 1.078125, "mean_p": 0.325811, "boundary_entropy": 0.617501, "bpb": 3.688162}, "de": {"chunks": 491.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.023913, "cps": 3.835938, "mean_p": 0.323568, "boundary_entropy": 0.615002, "bpb": 3.901921}, "vi": {"chunks": 246.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.013344, "cps": 1.921875, "mean_p": 0.380783, "boundary_entropy": 0.654541, "bpb": 3.278447}, "tr": {"chunks": 404.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.022317, "cps": 3.15625, "mean_p": 0.339516, "boundary_entropy": 0.622591, "bpb": 3.931836}, "fi": {"chunks": 190.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.0101, "cps": 1.484375, "mean_p": 0.318453, "boundary_entropy": 0.610087, "bpb": 3.947919}, "ru": {"chunks": 129.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.006807, "cps": 1.007812, "mean_p": 0.324843, "boundary_entropy": 0.620993, "bpb": 2.670494}, "el": {"chunks": 131.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.006383, "cps": 1.023438, "mean_p": 0.358232, "boundary_entropy": 0.647862, "bpb": 2.78849}, "hi": {"chunks": 162.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.009629, "cps": 1.265625, "mean_p": 0.37343, "boundary_entropy": 0.653935, "bpb": 2.17733}, "ta": {"chunks": 133.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.007114, "cps": 1.039062, "mean_p": 0.334886, "boundary_entropy": 0.631485, "bpb": 2.044377}, "th": {"chunks": 145.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.009015, "cps": 1.132812, "mean_p": 0.368819, "boundary_entropy": 0.652689, "bpb": 2.125364}, "zh": {"chunks": 183.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.030985, "cps": 1.429688, "mean_p": 0.361058, "boundary_entropy": 0.639788, "bpb": 5.05289}, "my": {"chunks": 130.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.006861, "cps": 1.015625, "mean_p": 0.347609, "boundary_entropy": 0.641585, "bpb": 2.263279}}, "bpb": {"en": 3.68816, "de": 3.90192, "vi": 3.27845, "tr": 3.93184, "fi": 3.94792, "ru": 2.67049, "el": 2.78849, "hi": 2.17733, "ta": 2.04438, "th": 2.12536, "zh": 5.05289, "my": 2.26328}, "gini_cpc": 0.308209, "gini_cps": 0.258192, "bpic": 154.625705, "worst_lang": "zh", "cpc_parity_vs_en": {"en": 1.0, "de": 3.0156026671805085, "vi": 1.6828174196059007, "tr": 2.814335361182028, "fi": 1.2737574915694563, "ru": 0.8584708041757485, "el": 0.8049621951193676, "hi": 1.214241230053621, "ta": 0.8971626917218042, "th": 1.1368206944087793, "zh": 3.9075332381218804, "my": 0.8652169323824471}, "t": 42.2}
|
| 15 |
+
{"event": "train", "step": 400, "phase": 1, "objective": "global", "lm_loss": 2.24422, "bpb": 3.23773, "chunk_loss": 0.022311, "F": 0.00177, "G": 0.38255, "chunks": 1.81, "bytes_per_chunk": 564.966, "boundary_entropy": 0.65676, "router_noise": 0.0, "grad_norm": 1.2833, "lr": 0.0002998, "router_trainable": true, "sec_per_step": 0.107, "alpha": 0.03, "t": 57.1}
|
| 16 |
+
{"event": "train", "step": 600, "phase": 1, "objective": "global", "lm_loss": 1.93344, "bpb": 2.78937, "chunk_loss": 0.025269, "F": 0.04996, "G": 0.35909, "chunks": 51.16, "bytes_per_chunk": 20.017, "boundary_entropy": 0.64475, "router_noise": 0.0, "grad_norm": 1.1866, "lr": 0.0002982, "router_trainable": true, "sec_per_step": 0.095, "alpha": 0.03, "t": 76.2}
|
| 17 |
+
{"event": "train", "step": 800, "phase": 1, "objective": "global", "lm_loss": 1.87191, "bpb": 2.7006, "chunk_loss": 0.032523, "F": 0.22223, "G": 0.35836, "chunks": 227.56, "bytes_per_chunk": 4.5, "boundary_entropy": 0.6453, "router_noise": 0.0, "grad_norm": 0.8789, "lr": 0.0002949, "router_trainable": true, "sec_per_step": 0.098, "alpha": 0.03, "t": 95.8}
|
| 18 |
+
{"event": "eval", "step": 1000, "phase": 1, "bpb": {"en": 2.62173, "de": 2.95945, "vi": 2.32517, "tr": 2.95091, "fi": 3.18195, "ru": 1.91645, "el": 2.03022, "hi": 1.53319, "ta": 1.31682, "th": 1.58764, "zh": 3.65126, "my": 1.40758}, "bytes_per_chunk": {"en": 179.06, "de": 148.861, "vi": 166.124, "tr": 35.724, "fi": 213.125, "ru": 63.091, "el": 261.1, "hi": 164.87, "ta": 86.573, "th": 337.814, "zh": 201.649, "my": 47.81}, "bpb_mean": 2.2902, "t": 114.5}
|
| 19 |
+
{"event": "train", "step": 1000, "phase": 1, "objective": "global", "lm_loss": 1.57618, "bpb": 2.27395, "chunk_loss": 0.023139, "F": 0.0014, "G": 0.35886, "chunks": 1.44, "bytes_per_chunk": 712.348, "boundary_entropy": 0.64447, "router_noise": 0.0, "grad_norm": 0.7905, "lr": 0.0002901, "router_trainable": true, "sec_per_step": 0.094, "alpha": 0.03, "t": 114.6}
|
| 20 |
+
{"event": "train", "step": 1200, "phase": 1, "objective": "global", "lm_loss": 1.72909, "bpb": 2.49455, "chunk_loss": 0.023445, "F": 0.00098, "G": 0.34983, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.63892, "router_noise": 0.0, "grad_norm": 0.972, "lr": 0.0002837, "router_trainable": true, "sec_per_step": 0.096, "alpha": 0.03, "t": 133.8}
|
| 21 |
+
{"event": "train", "step": 1400, "phase": 1, "objective": "global", "lm_loss": 1.72227, "bpb": 2.48472, "chunk_loss": 0.022609, "F": 0.00421, "G": 0.37745, "chunks": 4.31, "bytes_per_chunk": 237.449, "boundary_entropy": 0.65458, "router_noise": 0.0, "grad_norm": 0.8507, "lr": 0.0002759, "router_trainable": true, "sec_per_step": 0.096, "alpha": 0.03, "t": 153.1}
|
| 22 |
+
{"event": "train", "step": 1600, "phase": 1, "objective": "global", "lm_loss": 1.81127, "bpb": 2.61311, "chunk_loss": 0.028345, "F": 0.12985, "G": 0.37895, "chunks": 132.97, "bytes_per_chunk": 7.701, "boundary_entropy": 0.65544, "router_noise": 0.0, "grad_norm": 0.9153, "lr": 0.0002668, "router_trainable": true, "sec_per_step": 0.092, "alpha": 0.03, "t": 171.6}
|
| 23 |
+
{"event": "train", "step": 1800, "phase": 1, "objective": "global", "lm_loss": 1.46985, "bpb": 2.12055, "chunk_loss": 0.023037, "F": 0.00125, "G": 0.36155, "chunks": 1.28, "bytes_per_chunk": 799.22, "boundary_entropy": 0.64533, "router_noise": 0.0, "grad_norm": 0.7498, "lr": 0.0002564, "router_trainable": true, "sec_per_step": 0.092, "alpha": 0.03, "t": 190.0}
|
| 24 |
+
{"event": "probe", "step": 2000, "phase": 1, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 712.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.040912, "cps": 5.5625, "mean_p": 0.404796, "boundary_entropy": 0.660596, "bpb": 2.315924}, "de": {"chunks": 888.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.043247, "cps": 6.9375, "mean_p": 0.413895, "boundary_entropy": 0.666974, "bpb": 2.609606}, "vi": {"chunks": 567.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.030757, "cps": 4.429688, "mean_p": 0.396135, "boundary_entropy": 0.659024, "bpb": 2.122418}, "tr": {"chunks": 1251.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.069105, "cps": 9.773438, "mean_p": 0.417649, "boundary_entropy": 0.667817, "bpb": 2.593749}, "fi": {"chunks": 1175.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.062463, "cps": 9.179688, "mean_p": 0.418486, "boundary_entropy": 0.66759, "bpb": 2.862535}, "ru": {"chunks": 1136.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.059947, "cps": 8.875, "mean_p": 0.394438, "boundary_entropy": 0.661856, "bpb": 1.704548}, "el": {"chunks": 1524.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.074258, "cps": 11.90625, "mean_p": 0.417701, "boundary_entropy": 0.672436, "bpb": 1.759558}, "hi": {"chunks": 4278.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.254264, "cps": 33.421875, "mean_p": 0.422668, "boundary_entropy": 0.670879, "bpb": 1.369975}, "ta": {"chunks": 3175.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.169832, "cps": 24.804688, "mean_p": 0.419546, "boundary_entropy": 0.673166, "bpb": 1.112813}, "th": {"chunks": 1965.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.122164, "cps": 15.351562, "mean_p": 0.401491, "boundary_entropy": 0.661667, "bpb": 1.342597}, "zh": {"chunks": 483.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.081781, "cps": 3.773438, "mean_p": 0.397701, "boundary_entropy": 0.658059, "bpb": 3.43383}, "my": {"chunks": 3061.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.161547, "cps": 23.914062, "mean_p": 0.43214, "boundary_entropy": 0.678789, "bpb": 1.260822}}, "bpb": {"en": 2.31592, "de": 2.60961, "vi": 2.12242, "tr": 2.59375, "fi": 2.86254, "ru": 1.70455, "el": 1.75956, "hi": 1.36997, "ta": 1.11281, "th": 1.3426, "zh": 3.43383, "my": 1.26082}, "gini_cpc": 0.343305, "gini_cps": 0.363942, "bpic": 18.984962, "worst_lang": "hi", "cpc_parity_vs_en": {"en": 1.0, "de": 1.0570722821087677, "vi": 0.7517683601356725, "tr": 1.68908258734197, "fi": 1.5267576674895575, "ru": 1.4652551065785182, "el": 1.8150485588380698, "hi": 6.214838305758218, "ta": 4.151092267655534, "th": 2.9859712098298017, "zh": 1.9989310527857787, "my": 3.9486085365506542}, "t": 208.9}
|
| 25 |
+
{"event": "eval", "step": 2000, "phase": 1, "bpb": {"en": 2.32214, "de": 2.61804, "vi": 2.12452, "tr": 2.60296, "fi": 2.86484, "ru": 1.68552, "el": 1.75203, "hi": 1.36677, "ta": 1.11994, "th": 1.36287, "zh": 3.43846, "my": 1.26863}, "bytes_per_chunk": {"en": 30.066, "de": 25.372, "vi": 47.064, "tr": 20.845, "fi": 17.901, "ru": 33.816, "el": 26.214, "hi": 9.738, "ta": 16.31, "th": 26.568, "zh": 45.158, "my": 17.82}, "bpb_mean": 2.04389, "t": 210.2}
|
| 26 |
+
{"event": "train", "step": 2000, "phase": 1, "objective": "global", "lm_loss": 1.44163, "bpb": 2.07983, "chunk_loss": 0.06545, "F": 0.85696, "G": 0.40346, "chunks": 877.53, "bytes_per_chunk": 1.167, "boundary_entropy": 0.6668, "router_noise": 0.0, "grad_norm": 1.1627, "lr": 0.000245, "router_trainable": true, "sec_per_step": 0.101, "alpha": 0.03, "t": 210.3}
|
| 27 |
+
{"event": "train", "step": 2200, "phase": 1, "objective": "global", "lm_loss": 1.31007, "bpb": 1.89003, "chunk_loss": 0.048572, "F": 0.60114, "G": 0.36269, "chunks": 615.56, "bytes_per_chunk": 1.664, "boundary_entropy": 0.64343, "router_noise": 0.0, "grad_norm": 0.8399, "lr": 0.0002325, "router_trainable": true, "sec_per_step": 0.093, "alpha": 0.03, "t": 228.9}
|
| 28 |
+
{"event": "phase_transition", "step": 2400, "phase": 2, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0, "t": 247.2}
|
| 29 |
+
{"event": "train", "step": 2400, "phase": 2, "objective": "global", "lm_loss": 1.34105, "bpb": 1.93472, "chunk_loss": 0.028884, "F": 0.1416, "G": 0.37433, "chunks": 145.0, "bytes_per_chunk": 7.062, "boundary_entropy": 0.65166, "router_noise": 0.0, "grad_norm": 0.8131, "lr": 0.0002192, "router_trainable": true, "sec_per_step": 0.092, "alpha": 0.03, "t": 247.3}
|
| 30 |
+
{"event": "train", "step": 2600, "phase": 2, "objective": "global", "lm_loss": 1.52519, "bpb": 2.20038, "chunk_loss": 0.028042, "F": 0.12323, "G": 0.37579, "chunks": 126.19, "bytes_per_chunk": 8.115, "boundary_entropy": 0.65264, "router_noise": 0.0, "grad_norm": 0.9188, "lr": 0.0002053, "router_trainable": true, "sec_per_step": 0.091, "alpha": 0.03, "t": 265.5}
|
| 31 |
+
{"event": "train", "step": 2800, "phase": 2, "objective": "global", "lm_loss": 1.43453, "bpb": 2.06959, "chunk_loss": 0.023349, "F": 0.00125, "G": 0.35281, "chunks": 1.28, "bytes_per_chunk": 799.22, "boundary_entropy": 0.63995, "router_noise": 0.0, "grad_norm": 0.8882, "lr": 0.0001909, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 283.4}
|
| 32 |
+
{"event": "eval", "step": 3000, "phase": 2, "bpb": {"en": 2.20921, "de": 2.4469, "vi": 2.04661, "tr": 2.43706, "fi": 2.6653, "ru": 1.55049, "el": 1.63054, "hi": 1.29389, "ta": 1.06619, "th": 1.2542, "zh": 3.3289, "my": 1.20051}, "bytes_per_chunk": {"en": 96.094, "de": 108.954, "vi": 68.445, "tr": 76.988, "fi": 56.424, "ru": 57.425, "el": 40.523, "hi": 68.072, "ta": 91.659, "th": 74.178, "zh": 103.943, "my": 69.222}, "bpb_mean": 1.92748, "t": 302.2}
|
| 33 |
+
{"event": "train", "step": 3000, "phase": 2, "objective": "global", "lm_loss": 1.53952, "bpb": 2.22105, "chunk_loss": 0.051232, "F": 0.65289, "G": 0.36814, "chunks": 668.56, "bytes_per_chunk": 1.532, "boundary_entropy": 0.64834, "router_noise": 0.0, "grad_norm": 0.8578, "lr": 0.0001761, "router_trainable": true, "sec_per_step": 0.095, "alpha": 0.03, "t": 302.3}
|
| 34 |
+
{"event": "train", "step": 3200, "phase": 2, "objective": "global", "lm_loss": 1.42466, "bpb": 2.05535, "chunk_loss": 0.0234, "F": 0.00098, "G": 0.35109, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.63906, "router_noise": 0.0, "grad_norm": 0.8193, "lr": 0.0001613, "router_trainable": true, "sec_per_step": 0.091, "alpha": 0.03, "t": 320.6}
|
| 35 |
+
{"event": "train", "step": 3400, "phase": 2, "objective": "global", "lm_loss": 1.33683, "bpb": 1.92864, "chunk_loss": 0.028723, "F": 0.13754, "G": 0.3715, "chunks": 140.84, "bytes_per_chunk": 7.27, "boundary_entropy": 0.65142, "router_noise": 0.0, "grad_norm": 0.9023, "lr": 0.0001465, "router_trainable": true, "sec_per_step": 0.091, "alpha": 0.03, "t": 338.7}
|
| 36 |
+
{"event": "train", "step": 3600, "phase": 2, "objective": "global", "lm_loss": 1.3442, "bpb": 1.93927, "chunk_loss": 0.022435, "F": 0.00098, "G": 0.37803, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65299, "router_noise": 0.0, "grad_norm": 0.8975, "lr": 0.0001319, "router_trainable": true, "sec_per_step": 0.093, "alpha": 0.03, "t": 357.3}
|
| 37 |
+
{"event": "train", "step": 3800, "phase": 2, "objective": "global", "lm_loss": 1.40932, "bpb": 2.03321, "chunk_loss": 0.022659, "F": 0.00104, "G": 0.37186, "chunks": 1.06, "bytes_per_chunk": 963.765, "boundary_entropy": 0.65173, "router_noise": 0.0, "grad_norm": 0.8772, "lr": 0.0001177, "router_trainable": true, "sec_per_step": 0.094, "alpha": 0.03, "t": 376.2}
|
| 38 |
+
{"event": "probe", "step": 4000, "phase": 2, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 294.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.016894, "cps": 2.296875, "mean_p": 0.377103, "boundary_entropy": 0.646966, "bpb": 2.132903}, "de": {"chunks": 318.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.015487, "cps": 2.484375, "mean_p": 0.382933, "boundary_entropy": 0.652848, "bpb": 2.346793}, "vi": {"chunks": 469.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.025441, "cps": 3.664062, "mean_p": 0.3835, "boundary_entropy": 0.653029, "bpb": 1.989742}, "tr": {"chunks": 408.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.022538, "cps": 3.1875, "mean_p": 0.380939, "boundary_entropy": 0.651878, "bpb": 2.318921}, "fi": {"chunks": 483.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.025676, "cps": 3.773438, "mean_p": 0.388374, "boundary_entropy": 0.654449, "bpb": 2.534682}, "ru": {"chunks": 785.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.041425, "cps": 6.132812, "mean_p": 0.39287, "boundary_entropy": 0.662042, "bpb": 1.504671}, "el": {"chunks": 665.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.032403, "cps": 5.195312, "mean_p": 0.381358, "boundary_entropy": 0.656416, "bpb": 1.562564}, "hi": {"chunks": 1053.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.062585, "cps": 8.226562, "mean_p": 0.356573, "boundary_entropy": 0.627373, "bpb": 1.260632}, "ta": {"chunks": 1553.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.08307, "cps": 12.132812, "mean_p": 0.391833, "boundary_entropy": 0.662245, "bpb": 1.027909}, "th": {"chunks": 827.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.051414, "cps": 6.460938, "mean_p": 0.372158, "boundary_entropy": 0.646485, "bpb": 1.176101}, "zh": {"chunks": 267.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.045208, "cps": 2.085938, "mean_p": 0.385231, "boundary_entropy": 0.653231, "bpb": 3.214515}, "my": {"chunks": 257.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.013563, "cps": 2.007812, "mean_p": 0.384606, "boundary_entropy": 0.660241, "bpb": 1.159897}}, "bpb": {"en": 2.1329, "de": 2.34679, "vi": 1.98974, "tr": 2.31892, "fi": 2.53468, "ru": 1.50467, "el": 1.56256, "hi": 1.26063, "ta": 1.02791, "th": 1.1761, "zh": 3.21452, "my": 1.1599}, "gini_cpc": 0.305661, "gini_cps": 0.318257, "bpic": 52.009893, "worst_lang": "ta", "cpc_parity_vs_en": {"en": 1.0, "de": 0.9167512327095159, "vi": 1.5059359138285093, "tr": 1.3340939093418949, "fi": 1.519889578286261, "ru": 2.4520946637230088, "el": 1.9180396906606525, "hi": 3.704674773326864, "ta": 4.917255514133976, "th": 3.043415401597381, "zh": 2.6760523715073568, "my": 0.8028724155906821}, "t": 394.7}
|
| 39 |
+
{"event": "eval", "step": 4000, "phase": 2, "bpb": {"en": 2.13991, "de": 2.35358, "vi": 1.98988, "tr": 2.32957, "fi": 2.53829, "ru": 1.48173, "el": 1.55782, "hi": 1.25631, "ta": 1.03414, "th": 1.19615, "zh": 3.21293, "my": 1.16754}, "bytes_per_chunk": {"en": 82.826, "de": 82.383, "vi": 59.511, "tr": 87.41, "fi": 43.807, "ru": 50.862, "el": 57.513, "hi": 42.604, "ta": 40.605, "th": 61.536, "zh": 96.911, "my": 92.11}, "bpb_mean": 1.85482, "t": 396.0}
|
| 40 |
+
{"event": "train", "step": 4000, "phase": 2, "objective": "global", "lm_loss": 1.3414, "bpb": 1.93524, "chunk_loss": 0.052572, "F": 0.66241, "G": 0.37645, "chunks": 678.31, "bytes_per_chunk": 1.51, "boundary_entropy": 0.65202, "router_noise": 0.0, "grad_norm": 0.8761, "lr": 0.000104, "router_trainable": true, "sec_per_step": 0.099, "alpha": 0.03, "t": 396.1}
|
| 41 |
+
{"event": "phase_transition", "step": 4200, "phase": 3, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0, "t": 415.2}
|
| 42 |
+
{"event": "train", "step": 4200, "phase": 3, "objective": "global", "lm_loss": 1.32287, "bpb": 1.90849, "chunk_loss": 0.022215, "F": 0.0011, "G": 0.38436, "chunks": 1.12, "bytes_per_chunk": 910.222, "boundary_entropy": 0.65843, "router_noise": 0.0, "grad_norm": 0.8063, "lr": 9.12e-05, "router_trainable": true, "sec_per_step": 0.096, "alpha": 0.03, "t": 415.3}
|
| 43 |
+
{"event": "train", "step": 4400, "phase": 3, "objective": "global", "lm_loss": 1.48374, "bpb": 2.14059, "chunk_loss": 0.022965, "F": 0.00143, "G": 0.36377, "chunks": 1.47, "bytes_per_chunk": 697.191, "boundary_entropy": 0.64664, "router_noise": 0.0, "grad_norm": 0.951, "lr": 7.92e-05, "router_trainable": true, "sec_per_step": 0.089, "alpha": 0.03, "t": 433.1}
|
| 44 |
+
{"event": "train", "step": 4600, "phase": 3, "objective": "global", "lm_loss": 1.44325, "bpb": 2.08217, "chunk_loss": 0.022639, "F": 0.00098, "G": 0.37234, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65114, "router_noise": 0.0, "grad_norm": 0.8701, "lr": 6.82e-05, "router_trainable": true, "sec_per_step": 0.091, "alpha": 0.03, "t": 451.4}
|
| 45 |
+
{"event": "train", "step": 4800, "phase": 3, "objective": "global", "lm_loss": 1.38421, "bpb": 1.99699, "chunk_loss": 0.02217, "F": 0.00098, "G": 0.38544, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.65813, "router_noise": 0.0, "grad_norm": 0.8583, "lr": 5.85e-05, "router_trainable": true, "sec_per_step": 0.094, "alpha": 0.03, "t": 470.1}
|
| 46 |
+
{"event": "eval", "step": 5000, "phase": 3, "bpb": {"en": 2.0983, "de": 2.30449, "vi": 1.96153, "tr": 2.27631, "fi": 2.46698, "ru": 1.43429, "el": 1.50627, "hi": 1.22834, "ta": 1.01362, "th": 1.16708, "zh": 3.15443, "my": 1.14021}, "bytes_per_chunk": {"en": 64.346, "de": 71.507, "vi": 53.184, "tr": 51.08, "fi": 65.585, "ru": 51.695, "el": 42.876, "hi": 41.849, "ta": 45.189, "th": 54.659, "zh": 65.882, "my": 40.436}, "bpb_mean": 1.81265, "t": 489.7}
|
| 47 |
+
{"event": "train", "step": 5000, "phase": 3, "objective": "global", "lm_loss": 1.40145, "bpb": 2.02187, "chunk_loss": 0.025775, "F": 0.07776, "G": 0.38607, "chunks": 79.62, "bytes_per_chunk": 12.86, "boundary_entropy": 0.65884, "router_noise": 0.0, "grad_norm": 0.8475, "lr": 5e-05, "router_trainable": true, "sec_per_step": 0.098, "alpha": 0.03, "t": 489.7}
|
| 48 |
+
{"event": "train", "step": 5200, "phase": 3, "objective": "global", "lm_loss": 1.28923, "bpb": 1.85996, "chunk_loss": 0.022336, "F": 0.00351, "G": 0.38413, "chunks": 3.59, "bytes_per_chunk": 284.939, "boundary_entropy": 0.65839, "router_noise": 0.0, "grad_norm": 0.8719, "lr": 4.29e-05, "router_trainable": true, "sec_per_step": 0.094, "alpha": 0.03, "t": 508.5}
|
| 49 |
+
{"event": "train", "step": 5400, "phase": 3, "objective": "global", "lm_loss": 1.3869, "bpb": 2.00088, "chunk_loss": 0.02251, "F": 0.00098, "G": 0.37596, "chunks": 1.0, "bytes_per_chunk": 1024.0, "boundary_entropy": 0.6538, "router_noise": 0.0, "grad_norm": 0.954, "lr": 3.73e-05, "router_trainable": true, "sec_per_step": 0.093, "alpha": 0.03, "t": 527.1}
|
| 50 |
+
{"event": "train", "step": 5600, "phase": 3, "objective": "global", "lm_loss": 1.36892, "bpb": 1.97493, "chunk_loss": 0.023072, "F": 0.01898, "G": 0.38386, "chunks": 19.44, "bytes_per_chunk": 52.682, "boundary_entropy": 0.65772, "router_noise": 0.0, "grad_norm": 0.888, "lr": 3.33e-05, "router_trainable": true, "sec_per_step": 0.095, "alpha": 0.03, "t": 546.1}
|
| 51 |
+
{"event": "train", "step": 5800, "phase": 3, "objective": "global", "lm_loss": 1.32724, "bpb": 1.9148, "chunk_loss": 0.022554, "F": 0.00406, "G": 0.37868, "chunks": 4.16, "bytes_per_chunk": 246.376, "boundary_entropy": 0.655, "router_noise": 0.0, "grad_norm": 0.9174, "lr": 3.08e-05, "router_trainable": true, "sec_per_step": 0.095, "alpha": 0.03, "t": 565.1}
|
| 52 |
+
{"event": "probe", "step": 6000, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 235.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.013503, "cps": 1.835938, "mean_p": 0.374696, "boundary_entropy": 0.646152, "bpb": 2.072183}, "de": {"chunks": 254.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.01237, "cps": 1.984375, "mean_p": 0.378796, "boundary_entropy": 0.650996, "bpb": 2.273952}, "vi": {"chunks": 353.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.019148, "cps": 2.757812, "mean_p": 0.375519, "boundary_entropy": 0.649338, "bpb": 1.946296}, "tr": {"chunks": 410.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.022648, "cps": 3.203125, "mean_p": 0.381634, "boundary_entropy": 0.652266, "bpb": 2.238207}, "fi": {"chunks": 245.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.013024, "cps": 1.914062, "mean_p": 0.372556, "boundary_entropy": 0.646645, "bpb": 2.439194}, "ru": {"chunks": 591.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.031187, "cps": 4.617188, "mean_p": 0.376535, "boundary_entropy": 0.654067, "bpb": 1.435821}, "el": {"chunks": 566.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.027579, "cps": 4.421875, "mean_p": 0.37797, "boundary_entropy": 0.654882, "bpb": 1.489949}, "hi": {"chunks": 836.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.049688, "cps": 6.53125, "mean_p": 0.361977, "boundary_entropy": 0.633895, "bpb": 1.218163}, "ta": {"chunks": 1009.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.053972, "cps": 7.882812, "mean_p": 0.380425, "boundary_entropy": 0.656419, "bpb": 0.991907}, "th": {"chunks": 638.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.039664, "cps": 4.984375, "mean_p": 0.363322, "boundary_entropy": 0.642576, "bpb": 1.127957}, "zh": {"chunks": 237.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.040129, "cps": 1.851562, "mean_p": 0.379361, "boundary_entropy": 0.650523, "bpb": 3.125459}, "my": {"chunks": 816.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.043065, "cps": 6.375, "mean_p": 0.36954, "boundary_entropy": 0.651657, "bpb": 1.119749}}, "bpb": {"en": 2.07218, "de": 2.27395, "vi": 1.9463, "tr": 2.23821, "fi": 2.43919, "ru": 1.43582, "el": 1.48995, "hi": 1.21816, "ta": 0.99191, "th": 1.12796, "zh": 3.12546, "my": 1.11975}, "gini_cpc": 0.262784, "gini_cps": 0.278568, "bpic": 62.000162, "worst_lang": "ta", "cpc_parity_vs_en": {"en": 1.0, "de": 0.9160887870174738, "vi": 1.4180378442947907, "tr": 1.677218187651982, "fi": 0.9645182707718549, "ru": 2.3095880536686693, "el": 2.042357873522286, "hi": 3.6796580569694286, "ta": 3.9968877786187003, "th": 2.9373511729574933, "zh": 2.9717424040463722, "my": 3.1892094376995943}, "t": 584.2}
|
| 53 |
+
{"event": "eval", "step": 6000, "phase": 3, "bpb": {"en": 2.08045, "de": 2.28221, "vi": 1.94655, "tr": 2.24572, "fi": 2.44227, "ru": 1.4125, "el": 1.48345, "hi": 1.21199, "ta": 0.99986, "th": 1.14677, "zh": 3.12322, "my": 1.12666}, "bytes_per_chunk": {"en": 123.478, "de": 132.262, "vi": 89.134, "tr": 87.294, "fi": 135.826, "ru": 73.802, "el": 67.079, "hi": 54.613, "ta": 64.22, "th": 81.436, "zh": 119.7, "my": 51.031}, "bpb_mean": 1.7918, "t": 585.5}
|
| 54 |
+
{"event": "probe", "step": 6000, "phase": 3, "objective": "global", "router_trainable": true, "per_lang": {"en": {"chunks": 235.0, "chars": 17403.0, "bytes": 17418.0, "cpc": 0.013503, "cps": 1.835938, "mean_p": 0.374696, "boundary_entropy": 0.646152, "bpb": 2.072183}, "de": {"chunks": 254.0, "chars": 20533.0, "bytes": 20862.0, "cpc": 0.01237, "cps": 1.984375, "mean_p": 0.378796, "boundary_entropy": 0.650996, "bpb": 2.273952}, "vi": {"chunks": 353.0, "chars": 18435.0, "bytes": 24274.0, "cpc": 0.019148, "cps": 2.757812, "mean_p": 0.375519, "boundary_entropy": 0.649338, "bpb": 1.946296}, "tr": {"chunks": 410.0, "chars": 18103.0, "bytes": 19742.0, "cpc": 0.022648, "cps": 3.203125, "mean_p": 0.381634, "boundary_entropy": 0.652266, "bpb": 2.238207}, "fi": {"chunks": 245.0, "chars": 18811.0, "bytes": 19572.0, "cpc": 0.013024, "cps": 1.914062, "mean_p": 0.372556, "boundary_entropy": 0.646645, "bpb": 2.439194}, "ru": {"chunks": 591.0, "chars": 18950.0, "bytes": 34560.0, "cpc": 0.031187, "cps": 4.617188, "mean_p": 0.376535, "boundary_entropy": 0.654067, "bpb": 1.435821}, "el": {"chunks": 566.0, "chars": 20523.0, "bytes": 37264.0, "cpc": 0.027579, "cps": 4.421875, "mean_p": 0.37797, "boundary_entropy": 0.654882, "bpb": 1.489949}, "hi": {"chunks": 836.0, "chars": 16825.0, "bytes": 43173.0, "cpc": 0.049688, "cps": 6.53125, "mean_p": 0.361977, "boundary_entropy": 0.633895, "bpb": 1.218163}, "ta": {"chunks": 1009.0, "chars": 18695.0, "bytes": 50915.0, "cpc": 0.053972, "cps": 7.882812, "mean_p": 0.380425, "boundary_entropy": 0.656419, "bpb": 0.991907}, "th": {"chunks": 638.0, "chars": 16085.0, "bytes": 46085.0, "cpc": 0.039664, "cps": 4.984375, "mean_p": 0.363322, "boundary_entropy": 0.642576, "bpb": 1.127957}, "zh": {"chunks": 237.0, "chars": 5906.0, "bytes": 16173.0, "cpc": 0.040129, "cps": 1.851562, "mean_p": 0.379361, "boundary_entropy": 0.650523, "bpb": 3.125459}, "my": {"chunks": 816.0, "chars": 18948.0, "bytes": 53743.0, "cpc": 0.043065, "cps": 6.375, "mean_p": 0.36954, "boundary_entropy": 0.651657, "bpb": 1.119749}}, "bpb": {"en": 2.07218, "de": 2.27395, "vi": 1.9463, "tr": 2.23821, "fi": 2.43919, "ru": 1.43582, "el": 1.48995, "hi": 1.21816, "ta": 0.99191, "th": 1.12796, "zh": 3.12546, "my": 1.11975}, "gini_cpc": 0.262784, "gini_cps": 0.278568, "bpic": 62.000162, "worst_lang": "ta", "cpc_parity_vs_en": {"en": 1.0, "de": 0.9160887870174738, "vi": 1.4180378442947907, "tr": 1.677218187651982, "fi": 0.9645182707718549, "ru": 2.3095880536686693, "el": 2.042357873522286, "hi": 3.6796580569694286, "ta": 3.9968877786187003, "th": 2.9373511729574933, "zh": 2.9717424040463722, "my": 3.1892094376995943}, "t": 586.4}
|
| 55 |
+
{"event": "reps_dumped", "dirs": ["/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s2/th0.30", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s2/th0.40", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s2/th0.50", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s2/th0.60", "/local/scratch/sas245/beetle-hnet/scratch/reps/jitter_s2/th0.70"], "t": 590.1}
|
| 56 |
+
{"event": "final", "run": "jitter_s2", "condition": "A", "step": 6000, "params": 22516992, "per_lang": {"en": {"cpc": 0.013503418950755616, "cps": 1.8359375, "bpb": 2.091184529801444, "bytes_per_chunk": 119.15636363636364}, "de": {"cpc": 0.01237033068718648, "cps": 1.984375, "bpb": 2.2734026958836058, "bytes_per_chunk": 135.02137522534125}, "vi": {"cpc": 0.01914835909953892, "cps": 2.7578125, "bpb": 1.9494414809506864, "bytes_per_chunk": 86.68783068783068}, "tr": {"cpc": 0.022648179859691763, "cps": 3.203125, "bpb": 2.240100760041376, "bytes_per_chunk": 89.34696659850034}, "fi": {"cpc": 0.013024294295890703, "cps": 1.9140625, "bpb": 2.4580483003424547, "bytes_per_chunk": 132.8656867714141}, "ru": {"cpc": 0.031187335092348285, "cps": 4.6171875, "bpb": 1.4135231609180474, "bytes_per_chunk": 73.35777249195466}, "el": {"cpc": 0.02757881401354578, "cps": 4.421875, "bpb": 1.4792314436970544, "bytes_per_chunk": 67.52807831014941}, "hi": {"cpc": 0.04968796433878157, "cps": 6.53125, "bpb": 1.2194370256690006, "bytes_per_chunk": 54.50545794781162}, "ta": {"cpc": 0.053971650173843276, "cps": 7.8828125, "bpb": 0.9995220038420514, "bytes_per_chunk": 63.92197025115826}, "th": {"cpc": 0.039664283493938454, "cps": 4.984375, "bpb": 1.1435730867396772, "bytes_per_chunk": 81.6393646838991}, "zh": {"cpc": 0.040128682695563835, "cps": 1.8515625, "bpb": 3.1307407679749546, "bytes_per_chunk": 120.83152800184374}, "my": {"cpc": 0.04306523115896137, "cps": 6.375, "bpb": 1.1354728113731958, "bytes_per_chunk": 50.768664665440106}}, "gini_cpc": 0.262783775060067, "gini_cps": 0.27856758212170174, "bpic": 62.00016155088853, "seed": 2, "mean_boundary_entropy": 0.6519113779067993, "dechunk": "ema", "router_noise": 0.0, "transitions": [{"step": 0, "phase": 1, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}, {"step": 2400, "phase": 2, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}, {"step": 4200, "phase": 3, "objective": "global", "router_trainable": true, "encoder_trainable": true, "frozen_router_params": 0}], "t": 592.5}
|
jitter_s2/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d54044247bbbf717f8028f13741dd92eefd6ae377bdef734e96d00f247c36521
|
| 3 |
+
size 90078288
|
manifest.json
ADDED
|
@@ -0,0 +1,343 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{
|
| 3 |
+
"run": "baseline_en",
|
| 4 |
+
"params": 22582528,
|
| 5 |
+
"stored_tensors": 110,
|
| 6 |
+
"tied": {
|
| 7 |
+
"head.weight": "emb.weight"
|
| 8 |
+
},
|
| 9 |
+
"step": 5000,
|
| 10 |
+
"condition": "A",
|
| 11 |
+
"seed": 0
|
| 12 |
+
},
|
| 13 |
+
{
|
| 14 |
+
"run": "emactl_genreg_s0",
|
| 15 |
+
"params": 22582528,
|
| 16 |
+
"stored_tensors": 110,
|
| 17 |
+
"tied": {
|
| 18 |
+
"head.weight": "emb.weight"
|
| 19 |
+
},
|
| 20 |
+
"step": 6000,
|
| 21 |
+
"condition": "A",
|
| 22 |
+
"seed": 0
|
| 23 |
+
},
|
| 24 |
+
{
|
| 25 |
+
"run": "emactl_genreg_s1",
|
| 26 |
+
"params": 22582528,
|
| 27 |
+
"stored_tensors": 110,
|
| 28 |
+
"tied": {
|
| 29 |
+
"head.weight": "emb.weight"
|
| 30 |
+
},
|
| 31 |
+
"step": 6000,
|
| 32 |
+
"condition": "A",
|
| 33 |
+
"seed": 1
|
| 34 |
+
},
|
| 35 |
+
{
|
| 36 |
+
"run": "emactl_genreg_s2",
|
| 37 |
+
"params": 22582528,
|
| 38 |
+
"stored_tensors": 110,
|
| 39 |
+
"tied": {
|
| 40 |
+
"head.weight": "emb.weight"
|
| 41 |
+
},
|
| 42 |
+
"step": 6000,
|
| 43 |
+
"condition": "A",
|
| 44 |
+
"seed": 2
|
| 45 |
+
},
|
| 46 |
+
{
|
| 47 |
+
"run": "emactl_noema_s0",
|
| 48 |
+
"params": 22582528,
|
| 49 |
+
"stored_tensors": 110,
|
| 50 |
+
"tied": {
|
| 51 |
+
"head.weight": "emb.weight"
|
| 52 |
+
},
|
| 53 |
+
"step": 6000,
|
| 54 |
+
"condition": "A",
|
| 55 |
+
"seed": 0
|
| 56 |
+
},
|
| 57 |
+
{
|
| 58 |
+
"run": "emactl_noema_s1",
|
| 59 |
+
"params": 22582528,
|
| 60 |
+
"stored_tensors": 110,
|
| 61 |
+
"tied": {
|
| 62 |
+
"head.weight": "emb.weight"
|
| 63 |
+
},
|
| 64 |
+
"step": 6000,
|
| 65 |
+
"condition": "A",
|
| 66 |
+
"seed": 1
|
| 67 |
+
},
|
| 68 |
+
{
|
| 69 |
+
"run": "emactl_noema_s2",
|
| 70 |
+
"params": 22582528,
|
| 71 |
+
"stored_tensors": 110,
|
| 72 |
+
"tied": {
|
| 73 |
+
"head.weight": "emb.weight"
|
| 74 |
+
},
|
| 75 |
+
"step": 6000,
|
| 76 |
+
"condition": "A",
|
| 77 |
+
"seed": 2
|
| 78 |
+
},
|
| 79 |
+
{
|
| 80 |
+
"run": "jitter_s0",
|
| 81 |
+
"params": 22582528,
|
| 82 |
+
"stored_tensors": 110,
|
| 83 |
+
"tied": {
|
| 84 |
+
"head.weight": "emb.weight"
|
| 85 |
+
},
|
| 86 |
+
"step": 6000,
|
| 87 |
+
"condition": "A",
|
| 88 |
+
"seed": 0
|
| 89 |
+
},
|
| 90 |
+
{
|
| 91 |
+
"run": "jitter_s1",
|
| 92 |
+
"params": 22582528,
|
| 93 |
+
"stored_tensors": 110,
|
| 94 |
+
"tied": {
|
| 95 |
+
"head.weight": "emb.weight"
|
| 96 |
+
},
|
| 97 |
+
"step": 6000,
|
| 98 |
+
"condition": "A",
|
| 99 |
+
"seed": 1
|
| 100 |
+
},
|
| 101 |
+
{
|
| 102 |
+
"run": "jitter_s2",
|
| 103 |
+
"params": 22582528,
|
| 104 |
+
"stored_tensors": 110,
|
| 105 |
+
"tied": {
|
| 106 |
+
"head.weight": "emb.weight"
|
| 107 |
+
},
|
| 108 |
+
"step": 6000,
|
| 109 |
+
"condition": "A",
|
| 110 |
+
"seed": 2
|
| 111 |
+
},
|
| 112 |
+
{
|
| 113 |
+
"run": "pilot_A_s0",
|
| 114 |
+
"params": 22582528,
|
| 115 |
+
"stored_tensors": 110,
|
| 116 |
+
"tied": {
|
| 117 |
+
"head.weight": "emb.weight"
|
| 118 |
+
},
|
| 119 |
+
"step": 6000,
|
| 120 |
+
"condition": "A",
|
| 121 |
+
"seed": 0
|
| 122 |
+
},
|
| 123 |
+
{
|
| 124 |
+
"run": "pilot_A_s1",
|
| 125 |
+
"params": 22582528,
|
| 126 |
+
"stored_tensors": 110,
|
| 127 |
+
"tied": {
|
| 128 |
+
"head.weight": "emb.weight"
|
| 129 |
+
},
|
| 130 |
+
"step": 6000,
|
| 131 |
+
"condition": "A",
|
| 132 |
+
"seed": 1
|
| 133 |
+
},
|
| 134 |
+
{
|
| 135 |
+
"run": "pilot_A_s2",
|
| 136 |
+
"params": 22582528,
|
| 137 |
+
"stored_tensors": 110,
|
| 138 |
+
"tied": {
|
| 139 |
+
"head.weight": "emb.weight"
|
| 140 |
+
},
|
| 141 |
+
"step": 6000,
|
| 142 |
+
"condition": "A",
|
| 143 |
+
"seed": 2
|
| 144 |
+
},
|
| 145 |
+
{
|
| 146 |
+
"run": "pilot_Apc_s0",
|
| 147 |
+
"params": 22582528,
|
| 148 |
+
"stored_tensors": 110,
|
| 149 |
+
"tied": {
|
| 150 |
+
"head.weight": "emb.weight"
|
| 151 |
+
},
|
| 152 |
+
"step": 6000,
|
| 153 |
+
"condition": "A",
|
| 154 |
+
"seed": 0
|
| 155 |
+
},
|
| 156 |
+
{
|
| 157 |
+
"run": "pilot_Apc_s1",
|
| 158 |
+
"params": 22582528,
|
| 159 |
+
"stored_tensors": 110,
|
| 160 |
+
"tied": {
|
| 161 |
+
"head.weight": "emb.weight"
|
| 162 |
+
},
|
| 163 |
+
"step": 6000,
|
| 164 |
+
"condition": "A",
|
| 165 |
+
"seed": 1
|
| 166 |
+
},
|
| 167 |
+
{
|
| 168 |
+
"run": "pilot_Apc_s2",
|
| 169 |
+
"params": 22582528,
|
| 170 |
+
"stored_tensors": 110,
|
| 171 |
+
"tied": {
|
| 172 |
+
"head.weight": "emb.weight"
|
| 173 |
+
},
|
| 174 |
+
"step": 6000,
|
| 175 |
+
"condition": "A",
|
| 176 |
+
"seed": 2
|
| 177 |
+
},
|
| 178 |
+
{
|
| 179 |
+
"run": "pilot_B_s0",
|
| 180 |
+
"params": 22582528,
|
| 181 |
+
"stored_tensors": 110,
|
| 182 |
+
"tied": {
|
| 183 |
+
"head.weight": "emb.weight"
|
| 184 |
+
},
|
| 185 |
+
"step": 6000,
|
| 186 |
+
"condition": "B",
|
| 187 |
+
"seed": 0
|
| 188 |
+
},
|
| 189 |
+
{
|
| 190 |
+
"run": "pilot_B_s1",
|
| 191 |
+
"params": 22582528,
|
| 192 |
+
"stored_tensors": 110,
|
| 193 |
+
"tied": {
|
| 194 |
+
"head.weight": "emb.weight"
|
| 195 |
+
},
|
| 196 |
+
"step": 6000,
|
| 197 |
+
"condition": "B",
|
| 198 |
+
"seed": 1
|
| 199 |
+
},
|
| 200 |
+
{
|
| 201 |
+
"run": "pilot_B_s2",
|
| 202 |
+
"params": 22582528,
|
| 203 |
+
"stored_tensors": 110,
|
| 204 |
+
"tied": {
|
| 205 |
+
"head.weight": "emb.weight"
|
| 206 |
+
},
|
| 207 |
+
"step": 6000,
|
| 208 |
+
"condition": "B",
|
| 209 |
+
"seed": 2
|
| 210 |
+
},
|
| 211 |
+
{
|
| 212 |
+
"run": "pilot_Bpc_s0",
|
| 213 |
+
"params": 22582528,
|
| 214 |
+
"stored_tensors": 110,
|
| 215 |
+
"tied": {
|
| 216 |
+
"head.weight": "emb.weight"
|
| 217 |
+
},
|
| 218 |
+
"step": 6000,
|
| 219 |
+
"condition": "B",
|
| 220 |
+
"seed": 0
|
| 221 |
+
},
|
| 222 |
+
{
|
| 223 |
+
"run": "pilot_Bpc_s1",
|
| 224 |
+
"params": 22582528,
|
| 225 |
+
"stored_tensors": 110,
|
| 226 |
+
"tied": {
|
| 227 |
+
"head.weight": "emb.weight"
|
| 228 |
+
},
|
| 229 |
+
"step": 6000,
|
| 230 |
+
"condition": "B",
|
| 231 |
+
"seed": 1
|
| 232 |
+
},
|
| 233 |
+
{
|
| 234 |
+
"run": "pilot_Bpc_s2",
|
| 235 |
+
"params": 22582528,
|
| 236 |
+
"stored_tensors": 110,
|
| 237 |
+
"tied": {
|
| 238 |
+
"head.weight": "emb.weight"
|
| 239 |
+
},
|
| 240 |
+
"step": 6000,
|
| 241 |
+
"condition": "B",
|
| 242 |
+
"seed": 2
|
| 243 |
+
},
|
| 244 |
+
{
|
| 245 |
+
"run": "pilot_C_s0",
|
| 246 |
+
"params": 22582528,
|
| 247 |
+
"stored_tensors": 110,
|
| 248 |
+
"tied": {
|
| 249 |
+
"head.weight": "emb.weight"
|
| 250 |
+
},
|
| 251 |
+
"step": 6000,
|
| 252 |
+
"condition": "C",
|
| 253 |
+
"seed": 0
|
| 254 |
+
},
|
| 255 |
+
{
|
| 256 |
+
"run": "pilot_C_s1",
|
| 257 |
+
"params": 22582528,
|
| 258 |
+
"stored_tensors": 110,
|
| 259 |
+
"tied": {
|
| 260 |
+
"head.weight": "emb.weight"
|
| 261 |
+
},
|
| 262 |
+
"step": 6000,
|
| 263 |
+
"condition": "C",
|
| 264 |
+
"seed": 1
|
| 265 |
+
},
|
| 266 |
+
{
|
| 267 |
+
"run": "pilot_C_s2",
|
| 268 |
+
"params": 22582528,
|
| 269 |
+
"stored_tensors": 110,
|
| 270 |
+
"tied": {
|
| 271 |
+
"head.weight": "emb.weight"
|
| 272 |
+
},
|
| 273 |
+
"step": 6000,
|
| 274 |
+
"condition": "C",
|
| 275 |
+
"seed": 2
|
| 276 |
+
},
|
| 277 |
+
{
|
| 278 |
+
"run": "pilot_D_s0",
|
| 279 |
+
"params": 22582528,
|
| 280 |
+
"stored_tensors": 110,
|
| 281 |
+
"tied": {
|
| 282 |
+
"head.weight": "emb.weight"
|
| 283 |
+
},
|
| 284 |
+
"step": 6000,
|
| 285 |
+
"condition": "D",
|
| 286 |
+
"seed": 0
|
| 287 |
+
},
|
| 288 |
+
{
|
| 289 |
+
"run": "pilot_D_s1",
|
| 290 |
+
"params": 22582528,
|
| 291 |
+
"stored_tensors": 110,
|
| 292 |
+
"tied": {
|
| 293 |
+
"head.weight": "emb.weight"
|
| 294 |
+
},
|
| 295 |
+
"step": 6000,
|
| 296 |
+
"condition": "D",
|
| 297 |
+
"seed": 1
|
| 298 |
+
},
|
| 299 |
+
{
|
| 300 |
+
"run": "pilot_D_s2",
|
| 301 |
+
"params": 22582528,
|
| 302 |
+
"stored_tensors": 110,
|
| 303 |
+
"tied": {
|
| 304 |
+
"head.weight": "emb.weight"
|
| 305 |
+
},
|
| 306 |
+
"step": 6000,
|
| 307 |
+
"condition": "D",
|
| 308 |
+
"seed": 2
|
| 309 |
+
},
|
| 310 |
+
{
|
| 311 |
+
"run": "pilot_E_s0",
|
| 312 |
+
"params": 22582528,
|
| 313 |
+
"stored_tensors": 110,
|
| 314 |
+
"tied": {
|
| 315 |
+
"head.weight": "emb.weight"
|
| 316 |
+
},
|
| 317 |
+
"step": 6000,
|
| 318 |
+
"condition": "E",
|
| 319 |
+
"seed": 0
|
| 320 |
+
},
|
| 321 |
+
{
|
| 322 |
+
"run": "pilot_E_s1",
|
| 323 |
+
"params": 22582528,
|
| 324 |
+
"stored_tensors": 110,
|
| 325 |
+
"tied": {
|
| 326 |
+
"head.weight": "emb.weight"
|
| 327 |
+
},
|
| 328 |
+
"step": 6000,
|
| 329 |
+
"condition": "E",
|
| 330 |
+
"seed": 1
|
| 331 |
+
},
|
| 332 |
+
{
|
| 333 |
+
"run": "pilot_E_s2",
|
| 334 |
+
"params": 22582528,
|
| 335 |
+
"stored_tensors": 110,
|
| 336 |
+
"tied": {
|
| 337 |
+
"head.weight": "emb.weight"
|
| 338 |
+
},
|
| 339 |
+
"step": 6000,
|
| 340 |
+
"condition": "E",
|
| 341 |
+
"seed": 2
|
| 342 |
+
}
|
| 343 |
+
]
|
pilot_A_s0/config.json
ADDED
|
@@ -0,0 +1,106 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"run": "pilot_A_s0",
|
| 3 |
+
"step": 6000,
|
| 4 |
+
"model_config": {
|
| 5 |
+
"vocab_size": 256,
|
| 6 |
+
"d_enc": 256,
|
| 7 |
+
"d_main": 512,
|
| 8 |
+
"n_enc": 2,
|
| 9 |
+
"n_dec": 2,
|
| 10 |
+
"n_main": 6,
|
| 11 |
+
"n_heads_enc": 4,
|
| 12 |
+
"n_heads_main": 8,
|
| 13 |
+
"seq_len": 1024,
|
| 14 |
+
"dropout": 0.0,
|
| 15 |
+
"target_ratio": 6.0,
|
| 16 |
+
"max_chunk_mult": 3.0,
|
| 17 |
+
"dechunk": "ema",
|
| 18 |
+
"router_noise": 0.0,
|
| 19 |
+
"rope_base": 10000.0
|
| 20 |
+
},
|
| 21 |
+
"tied_weights": {
|
| 22 |
+
"head.weight": "emb.weight"
|
| 23 |
+
},
|
| 24 |
+
"train_config": {
|
| 25 |
+
"condition": "A",
|
| 26 |
+
"seed": 0,
|
| 27 |
+
"out_dir": null,
|
| 28 |
+
"run_name": "pilot_A_s0",
|
| 29 |
+
"corpus": "corpus",
|
| 30 |
+
"langs": null,
|
| 31 |
+
"seq_len": 1024,
|
| 32 |
+
"batch_size": 32,
|
| 33 |
+
"grad_accum": 1,
|
| 34 |
+
"steps": 6000,
|
| 35 |
+
"lr": 0.0003,
|
| 36 |
+
"min_lr_frac": 0.1,
|
| 37 |
+
"warmup": 300,
|
| 38 |
+
"weight_decay": 0.1,
|
| 39 |
+
"grad_clip": 1.0,
|
| 40 |
+
"betas": [
|
| 41 |
+
0.9,
|
| 42 |
+
0.95
|
| 43 |
+
],
|
| 44 |
+
"router_lr_mult": 1.0,
|
| 45 |
+
"d_enc": 256,
|
| 46 |
+
"d_main": 512,
|
| 47 |
+
"n_enc": 2,
|
| 48 |
+
"n_dec": 2,
|
| 49 |
+
"n_main": 6,
|
| 50 |
+
"n_heads_enc": 4,
|
| 51 |
+
"n_heads_main": 8,
|
| 52 |
+
"dechunk": "ema",
|
| 53 |
+
"target_ratio": 6.0,
|
| 54 |
+
"alpha": 0.03,
|
| 55 |
+
"alpha_parity_cap": 0.06,
|
| 56 |
+
"global_anchor": 0.0,
|
| 57 |
+
"premium_clip": 4.0,
|
| 58 |
+
"premium_kind": "flores_per_sent",
|
| 59 |
+
"mixture": null,
|
| 60 |
+
"doc_filters": null,
|
| 61 |
+
"tau1": 0.4,
|
| 62 |
+
"tau2": 0.7,
|
| 63 |
+
"freeze_scope": "router",
|
| 64 |
+
"ckpt_every": 200,
|
| 65 |
+
"eval_every": 200,
|
| 66 |
+
"log_every": 50,
|
| 67 |
+
"probe_sents": 128,
|
| 68 |
+
"eval_batches": 8,
|
| 69 |
+
"keep_ckpt": false,
|
| 70 |
+
"amp": "bf16",
|
| 71 |
+
"compile": false,
|
| 72 |
+
"cpc_decay": 0.9,
|
| 73 |
+
"device": "cuda",
|
| 74 |
+
"log_spaced_probe": true,
|
| 75 |
+
"probe_bpb_batches": 4,
|
| 76 |
+
"dump_reps": true,
|
| 77 |
+
"dump_reps_thresholds": [
|
| 78 |
+
0.4,
|
| 79 |
+
0.5,
|
| 80 |
+
0.6
|
| 81 |
+
],
|
| 82 |
+
"router_noise": 0.0,
|
| 83 |
+
"genreg_target_entropy": null,
|
| 84 |
+
"genreg_eta": 0.05
|
| 85 |
+
},
|
| 86 |
+
"probe_agg": {
|
| 87 |
+
"gini_cpc": 0.2046874384551851,
|
| 88 |
+
"gini_cps": 0.2124337217717518,
|
| 89 |
+
"bpic": 5.668261775001108,
|
| 90 |
+
"worst_lang": "my",
|
| 91 |
+
"cpc_parity_vs_en": {
|
| 92 |
+
"en": 1.0,
|
| 93 |
+
"de": 0.9674031957033492,
|
| 94 |
+
"vi": 1.0641769666197316,
|
| 95 |
+
"tr": 0.9818145543054209,
|
| 96 |
+
"fi": 0.9579169801888867,
|
| 97 |
+
"ru": 1.5462771572877685,
|
| 98 |
+
"el": 1.507303331656917,
|
| 99 |
+
"hi": 2.114500469118362,
|
| 100 |
+
"ta": 2.0382228601055625,
|
| 101 |
+
"th": 2.417749025953193,
|
| 102 |
+
"zh": 2.32292387976097,
|
| 103 |
+
"my": 2.471763194501154
|
| 104 |
+
}
|
| 105 |
+
}
|
| 106 |
+
}
|
pilot_A_s0/final.json
ADDED
|
@@ -0,0 +1,114 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"event": "final",
|
| 3 |
+
"run": "pilot_A_s0",
|
| 4 |
+
"condition": "A",
|
| 5 |
+
"step": 6000,
|
| 6 |
+
"params": 22516992,
|
| 7 |
+
"per_lang": {
|
| 8 |
+
"en": {
|
| 9 |
+
"cpc": 0.2076653450554502,
|
| 10 |
+
"cps": 28.234375,
|
| 11 |
+
"bpb": 2.0273781279159193,
|
| 12 |
+
"bytes_per_chunk": 4.917766459370984
|
| 13 |
+
},
|
| 14 |
+
"de": {
|
| 15 |
+
"cpc": 0.20089611844348124,
|
| 16 |
+
"cps": 32.2265625,
|
| 17 |
+
"bpb": 2.194651560581718,
|
| 18 |
+
"bytes_per_chunk": 4.990082424380866
|
| 19 |
+
},
|
| 20 |
+
"vi": {
|
| 21 |
+
"cpc": 0.2209926769731489,
|
| 22 |
+
"cps": 31.828125,
|
| 23 |
+
"bpb": 1.8993434507023028,
|
| 24 |
+
"bytes_per_chunk": 5.929249977381707
|
| 25 |
+
},
|
| 26 |
+
"tr": {
|
| 27 |
+
"cpc": 0.20388885820029828,
|
| 28 |
+
"cps": 28.8359375,
|
| 29 |
+
"bpb": 2.172621765386917,
|
| 30 |
+
"bytes_per_chunk": 5.211869377205627
|
| 31 |
+
},
|
| 32 |
+
"fi": {
|
| 33 |
+
"cpc": 0.19892616022540002,
|
| 34 |
+
"cps": 29.234375,
|
| 35 |
+
"bpb": 2.362985137303098,
|
| 36 |
+
"bytes_per_chunk": 4.885778452879069
|
| 37 |
+
},
|
| 38 |
+
"ru": {
|
| 39 |
+
"cpc": 0.3211081794195251,
|
| 40 |
+
"cps": 47.5390625,
|
| 41 |
+
"bpb": 1.3424836631873538,
|
| 42 |
+
"bytes_per_chunk": 5.437995270298303
|
| 43 |
+
},
|
| 44 |
+
"el": {
|
| 45 |
+
"cpc": 0.3130146664717634,
|
| 46 |
+
"cps": 50.1875,
|
| 47 |
+
"bpb": 1.3926614235177859,
|
| 48 |
+
"bytes_per_chunk": 5.5650401757756525
|
| 49 |
+
},
|
| 50 |
+
"hi": {
|
| 51 |
+
"cpc": 0.43910846953937593,
|
| 52 |
+
"cps": 57.71875,
|
| 53 |
+
"bpb": 1.1374732168277908,
|
| 54 |
+
"bytes_per_chunk": 5.7044250291048755
|
| 55 |
+
},
|
| 56 |
+
"ta": {
|
| 57 |
+
"cpc": 0.42326825354372827,
|
| 58 |
+
"cps": 61.8203125,
|
| 59 |
+
"bpb": 0.9469699101528437,
|
| 60 |
+
"bytes_per_chunk": 6.453172502923257
|
| 61 |
+
},
|
| 62 |
+
"th": {
|
| 63 |
+
"cpc": 0.5020826857320485,
|
| 64 |
+
"cps": 63.09375,
|
| 65 |
+
"bpb": 1.093262663291936,
|
| 66 |
+
"bytes_per_chunk": 5.682784335403592
|
| 67 |
+
},
|
| 68 |
+
"zh": {
|
| 69 |
+
"cpc": 0.482390789028107,
|
| 70 |
+
"cps": 22.2578125,
|
| 71 |
+
"bpb": 3.1054285549014486,
|
| 72 |
+
"bytes_per_chunk": 5.831448051875827
|
| 73 |
+
},
|
| 74 |
+
"my": {
|
| 75 |
+
"cpc": 0.513299556681444,
|
| 76 |
+
"cps": 75.984375,
|
| 77 |
+
"bpb": 1.0876047296737308,
|
| 78 |
+
"bytes_per_chunk": 6.01137405980554
|
| 79 |
+
}
|
| 80 |
+
},
|
| 81 |
+
"gini_cpc": 0.2046874384551851,
|
| 82 |
+
"gini_cps": 0.2124337217717518,
|
| 83 |
+
"bpic": 5.668261775001108,
|
| 84 |
+
"seed": 0,
|
| 85 |
+
"mean_boundary_entropy": 0.32596713304519653,
|
| 86 |
+
"dechunk": "ema",
|
| 87 |
+
"router_noise": 0.0,
|
| 88 |
+
"transitions": [
|
| 89 |
+
{
|
| 90 |
+
"step": 0,
|
| 91 |
+
"phase": 1,
|
| 92 |
+
"objective": "global",
|
| 93 |
+
"router_trainable": true,
|
| 94 |
+
"encoder_trainable": true,
|
| 95 |
+
"frozen_router_params": 0
|
| 96 |
+
},
|
| 97 |
+
{
|
| 98 |
+
"step": 2400,
|
| 99 |
+
"phase": 2,
|
| 100 |
+
"objective": "global",
|
| 101 |
+
"router_trainable": true,
|
| 102 |
+
"encoder_trainable": true,
|
| 103 |
+
"frozen_router_params": 0
|
| 104 |
+
},
|
| 105 |
+
{
|
| 106 |
+
"step": 4200,
|
| 107 |
+
"phase": 3,
|
| 108 |
+
"objective": "global",
|
| 109 |
+
"router_trainable": true,
|
| 110 |
+
"encoder_trainable": true,
|
| 111 |
+
"frozen_router_params": 0
|
| 112 |
+
}
|
| 113 |
+
]
|
| 114 |
+
}
|
pilot_A_s0/log.jsonl
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
pilot_A_s0/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b7cc09aad5130e4f1b5e62a9ffd33305232927fee93d6deba4ea5dad45ddfc55
|
| 3 |
+
size 90078296
|
pilot_A_s1/config.json
ADDED
|
@@ -0,0 +1,106 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"run": "pilot_A_s1",
|
| 3 |
+
"step": 6000,
|
| 4 |
+
"model_config": {
|
| 5 |
+
"vocab_size": 256,
|
| 6 |
+
"d_enc": 256,
|
| 7 |
+
"d_main": 512,
|
| 8 |
+
"n_enc": 2,
|
| 9 |
+
"n_dec": 2,
|
| 10 |
+
"n_main": 6,
|
| 11 |
+
"n_heads_enc": 4,
|
| 12 |
+
"n_heads_main": 8,
|
| 13 |
+
"seq_len": 1024,
|
| 14 |
+
"dropout": 0.0,
|
| 15 |
+
"target_ratio": 6.0,
|
| 16 |
+
"max_chunk_mult": 3.0,
|
| 17 |
+
"dechunk": "ema",
|
| 18 |
+
"router_noise": 0.0,
|
| 19 |
+
"rope_base": 10000.0
|
| 20 |
+
},
|
| 21 |
+
"tied_weights": {
|
| 22 |
+
"head.weight": "emb.weight"
|
| 23 |
+
},
|
| 24 |
+
"train_config": {
|
| 25 |
+
"condition": "A",
|
| 26 |
+
"seed": 1,
|
| 27 |
+
"out_dir": null,
|
| 28 |
+
"run_name": "pilot_A_s1",
|
| 29 |
+
"corpus": "corpus",
|
| 30 |
+
"langs": null,
|
| 31 |
+
"seq_len": 1024,
|
| 32 |
+
"batch_size": 32,
|
| 33 |
+
"grad_accum": 1,
|
| 34 |
+
"steps": 6000,
|
| 35 |
+
"lr": 0.0003,
|
| 36 |
+
"min_lr_frac": 0.1,
|
| 37 |
+
"warmup": 300,
|
| 38 |
+
"weight_decay": 0.1,
|
| 39 |
+
"grad_clip": 1.0,
|
| 40 |
+
"betas": [
|
| 41 |
+
0.9,
|
| 42 |
+
0.95
|
| 43 |
+
],
|
| 44 |
+
"router_lr_mult": 1.0,
|
| 45 |
+
"d_enc": 256,
|
| 46 |
+
"d_main": 512,
|
| 47 |
+
"n_enc": 2,
|
| 48 |
+
"n_dec": 2,
|
| 49 |
+
"n_main": 6,
|
| 50 |
+
"n_heads_enc": 4,
|
| 51 |
+
"n_heads_main": 8,
|
| 52 |
+
"dechunk": "ema",
|
| 53 |
+
"target_ratio": 6.0,
|
| 54 |
+
"alpha": 0.03,
|
| 55 |
+
"alpha_parity_cap": 0.06,
|
| 56 |
+
"global_anchor": 0.0,
|
| 57 |
+
"premium_clip": 4.0,
|
| 58 |
+
"premium_kind": "flores_per_sent",
|
| 59 |
+
"mixture": null,
|
| 60 |
+
"doc_filters": null,
|
| 61 |
+
"tau1": 0.4,
|
| 62 |
+
"tau2": 0.7,
|
| 63 |
+
"freeze_scope": "router",
|
| 64 |
+
"ckpt_every": 200,
|
| 65 |
+
"eval_every": 200,
|
| 66 |
+
"log_every": 50,
|
| 67 |
+
"probe_sents": 128,
|
| 68 |
+
"eval_batches": 8,
|
| 69 |
+
"keep_ckpt": false,
|
| 70 |
+
"amp": "bf16",
|
| 71 |
+
"compile": false,
|
| 72 |
+
"cpc_decay": 0.9,
|
| 73 |
+
"device": "cuda",
|
| 74 |
+
"log_spaced_probe": true,
|
| 75 |
+
"probe_bpb_batches": 4,
|
| 76 |
+
"dump_reps": true,
|
| 77 |
+
"dump_reps_thresholds": [
|
| 78 |
+
0.4,
|
| 79 |
+
0.5,
|
| 80 |
+
0.6
|
| 81 |
+
],
|
| 82 |
+
"router_noise": 0.0,
|
| 83 |
+
"genreg_target_entropy": null,
|
| 84 |
+
"genreg_eta": 0.05
|
| 85 |
+
},
|
| 86 |
+
"probe_agg": {
|
| 87 |
+
"gini_cpc": 0.22451192807644027,
|
| 88 |
+
"gini_cps": 0.22301957393367622,
|
| 89 |
+
"bpic": 5.925929929126198,
|
| 90 |
+
"worst_lang": "zh",
|
| 91 |
+
"cpc_parity_vs_en": {
|
| 92 |
+
"en": 1.0,
|
| 93 |
+
"de": 0.9725837928890361,
|
| 94 |
+
"vi": 1.378765363367736,
|
| 95 |
+
"tr": 1.1427472459455228,
|
| 96 |
+
"fi": 0.9735219498640139,
|
| 97 |
+
"ru": 2.087942793033131,
|
| 98 |
+
"el": 2.0054266863351606,
|
| 99 |
+
"hi": 2.5855259453474537,
|
| 100 |
+
"ta": 2.8326408249054102,
|
| 101 |
+
"th": 2.8393424346249194,
|
| 102 |
+
"zh": 3.0282894619584946,
|
| 103 |
+
"my": 2.5190893307942366
|
| 104 |
+
}
|
| 105 |
+
}
|
| 106 |
+
}
|
pilot_A_s1/final.json
ADDED
|
@@ -0,0 +1,114 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"event": "final",
|
| 3 |
+
"run": "pilot_A_s1",
|
| 4 |
+
"condition": "A",
|
| 5 |
+
"step": 6000,
|
| 6 |
+
"params": 22516992,
|
| 7 |
+
"per_lang": {
|
| 8 |
+
"en": {
|
| 9 |
+
"cpc": 0.16594839970120095,
|
| 10 |
+
"cps": 22.5625,
|
| 11 |
+
"bpb": 2.022311981224875,
|
| 12 |
+
"bytes_per_chunk": 5.737196883480697
|
| 13 |
+
},
|
| 14 |
+
"de": {
|
| 15 |
+
"cpc": 0.1613987240052598,
|
| 16 |
+
"cps": 25.890625,
|
| 17 |
+
"bpb": 2.212188749728271,
|
| 18 |
+
"bytes_per_chunk": 6.239741026373417
|
| 19 |
+
},
|
| 20 |
+
"vi": {
|
| 21 |
+
"cpc": 0.2288039056143206,
|
| 22 |
+
"cps": 32.953125,
|
| 23 |
+
"bpb": 1.8670015220171892,
|
| 24 |
+
"bytes_per_chunk": 5.71686530220589
|
| 25 |
+
},
|
| 26 |
+
"tr": {
|
| 27 |
+
"cpc": 0.1896370767276142,
|
| 28 |
+
"cps": 26.8203125,
|
| 29 |
+
"bpb": 2.1998532618483893,
|
| 30 |
+
"bytes_per_chunk": 5.7227934595149215
|
| 31 |
+
},
|
| 32 |
+
"fi": {
|
| 33 |
+
"cpc": 0.1615544096539259,
|
| 34 |
+
"cps": 23.7421875,
|
| 35 |
+
"bpb": 2.395499962296907,
|
| 36 |
+
"bytes_per_chunk": 6.238033481266434
|
| 37 |
+
},
|
| 38 |
+
"ru": {
|
| 39 |
+
"cpc": 0.34649076517150396,
|
| 40 |
+
"cps": 51.296875,
|
| 41 |
+
"bpb": 1.2738760323866076,
|
| 42 |
+
"bytes_per_chunk": 5.32029022274088
|
| 43 |
+
},
|
| 44 |
+
"el": {
|
| 45 |
+
"cpc": 0.3327973493154022,
|
| 46 |
+
"cps": 53.359375,
|
| 47 |
+
"bpb": 1.3333722856930763,
|
| 48 |
+
"bytes_per_chunk": 5.615466181117121
|
| 49 |
+
},
|
| 50 |
+
"hi": {
|
| 51 |
+
"cpc": 0.42906389301634473,
|
| 52 |
+
"cps": 56.3984375,
|
| 53 |
+
"bpb": 1.140567442690789,
|
| 54 |
+
"bytes_per_chunk": 5.7856937915204485
|
| 55 |
+
},
|
| 56 |
+
"ta": {
|
| 57 |
+
"cpc": 0.4700722118213426,
|
| 58 |
+
"cps": 68.65625,
|
| 59 |
+
"bpb": 0.9174700377689008,
|
| 60 |
+
"bytes_per_chunk": 5.817278032976056
|
| 61 |
+
},
|
| 62 |
+
"th": {
|
| 63 |
+
"cpc": 0.47118433322971714,
|
| 64 |
+
"cps": 59.2109375,
|
| 65 |
+
"bpb": 1.0642222156874839,
|
| 66 |
+
"bytes_per_chunk": 6.21260560960292
|
| 67 |
+
},
|
| 68 |
+
"zh": {
|
| 69 |
+
"cpc": 0.502539790044023,
|
| 70 |
+
"cps": 23.1875,
|
| 71 |
+
"bpb": 3.109084046145538,
|
| 72 |
+
"bytes_per_chunk": 5.7431043925950265
|
| 73 |
+
},
|
| 74 |
+
"my": {
|
| 75 |
+
"cpc": 0.41803884314967277,
|
| 76 |
+
"cps": 61.8828125,
|
| 77 |
+
"bpb": 1.0778031266825046,
|
| 78 |
+
"bytes_per_chunk": 6.518967982592478
|
| 79 |
+
}
|
| 80 |
+
},
|
| 81 |
+
"gini_cpc": 0.22451192807644027,
|
| 82 |
+
"gini_cps": 0.22301957393367622,
|
| 83 |
+
"bpic": 5.925929929126198,
|
| 84 |
+
"seed": 1,
|
| 85 |
+
"mean_boundary_entropy": 0.24016138911247253,
|
| 86 |
+
"dechunk": "ema",
|
| 87 |
+
"router_noise": 0.0,
|
| 88 |
+
"transitions": [
|
| 89 |
+
{
|
| 90 |
+
"step": 0,
|
| 91 |
+
"phase": 1,
|
| 92 |
+
"objective": "global",
|
| 93 |
+
"router_trainable": true,
|
| 94 |
+
"encoder_trainable": true,
|
| 95 |
+
"frozen_router_params": 0
|
| 96 |
+
},
|
| 97 |
+
{
|
| 98 |
+
"step": 2400,
|
| 99 |
+
"phase": 2,
|
| 100 |
+
"objective": "global",
|
| 101 |
+
"router_trainable": true,
|
| 102 |
+
"encoder_trainable": true,
|
| 103 |
+
"frozen_router_params": 0
|
| 104 |
+
},
|
| 105 |
+
{
|
| 106 |
+
"step": 4200,
|
| 107 |
+
"phase": 3,
|
| 108 |
+
"objective": "global",
|
| 109 |
+
"router_trainable": true,
|
| 110 |
+
"encoder_trainable": true,
|
| 111 |
+
"frozen_router_params": 0
|
| 112 |
+
}
|
| 113 |
+
]
|
| 114 |
+
}
|
pilot_A_s1/log.jsonl
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
pilot_A_s1/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d75b23ba9fab68082dcfb739b4aeeda17dcf2ff283f1bf84637692a96f97318b
|
| 3 |
+
size 90078296
|