Hugging Face's logo

Devatri
/
escape-blt-entropy-student-balanced

Safetensors
English
byte-level
entropy-patching
byte-latent-transformer
knowledge-distillation
code
Model card Files
xet
 Community
Devatri commited on
Commit
147aa95
·
verified ·
1 Parent(s): aa460d9

Byte-balanced distilled student (ESCAPE, 14.3M)

Browse files
Files changed (6) hide show
  1. README.md +56 -0
  2. config.json +42 -0
  3. distill_eval.json +142 -0
  4. model.safetensors +3 -0
  5. student.pt +3 -0
  6. train_log.json +684 -0
README.md ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - byte-level
4
+ - entropy-patching
5
+ - byte-latent-transformer
6
+ - knowledge-distillation
7
+ - code
8
+ language:
9
+ - en
10
+ datasets:
11
+ - code-search-net/code_search_net
12
+ - bigcode/the-stack
13
+ - Salesforce/wikitext
14
+ ---
15
+
16
+ # ESCAPE: byte-balanced distilled BLT patcher (14.3M)
17
+
18
+ A follow-up arm of the [distilled student](https://huggingface.co/Devatri/escape-blt-entropy-student) of **Project ESCAPE** (Team Trimax, ANLP,
19
+ IIIT Hyderabad, October 2026). Identical to it (14,262,144 parameters; online distillation from the
20
+ BLT patcher with α = 0.1, T = 2; same data, seed and schedule) except that every domain is drawn
21
+ to an equal expected share of bytes. In the unbalanced run C++ is 73% of the training bytes.
22
+
23
+ ## Results (from the ESCAPE final report; exploratory)
24
+
25
+ | | Python | C++ | English |
26
+ |---|---|---|---|
27
+ | Boundary F1 vs. teacher, held-out windows (distilled student) | 0.864 (0.869) | 0.860 (0.880) | 0.853 (0.834) |
28
+ | Start-F1 margin minus the distilled student's [95% CI] | −0.002 [−0.003, −0.001] | −0.002 [−0.0035, −0.0001] | +0.010 [0.008, 0.012] |
29
+
30
+ Margins were scored on the pre-registered 600-file code subset and the 240-file English test
31
+ split (full-corpus extraction did not fit before the deadline), so these results are
32
+ exploratory. Balancing trades code for prose. Bits/byte on held-out windows: 1.27.
33
+
34
+ ## Use
35
+
36
+ Clone the [ESCAPE repository](https://github.com/dawnbliss-coder/ESCAPE-Entropy-Semantic-Compute-Allocation-in-Programming-Environments), then:
37
+
38
+ ```python
39
+ import torch
40
+ from huggingface_hub import hf_hub_download
41
+ from escape_distill.student_blt import StudentBLT, StudentConfig
42
+
43
+ ckpt = torch.load(hf_hub_download("Devatri/escape-blt-entropy-student-balanced", "student.pt"), map_location="cpu", weights_only=False)
44
+ model = StudentBLT(StudentConfig(**ckpt["config"])).eval()
45
+ model.load_state_dict(ckpt["model_state"])
46
+ ```
47
+
48
+ `model.safetensors` holds the same weights; `config.json` has the model configuration and training
49
+ arguments, `train_log.json` the loss curve, and `distill_eval.json` the held-out agreement with the
50
+ teacher per domain.
51
+
52
+ ## Citation
53
+
54
+ Teacher model: Pagnoni et al., 2025, *Byte Latent Transformer: Patches Scale Better Than Tokens*, ACL 2025.
55
+
56
+ No licence has been chosen for this model; the training data keep their own licences.
config.json ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architecture": "StudentBLT (escape_distill/student_blt.py)",
3
+ "model_config": {
4
+ "vocab_size": 260,
5
+ "dim": 384,
6
+ "n_layers": 8,
7
+ "n_heads": 6,
8
+ "ffn_mult": 2.6666666666666665,
9
+ "ffn_multiple_of": 64,
10
+ "max_seq_len": 513,
11
+ "rope_theta": 10000.0,
12
+ "norm_eps": 1e-05,
13
+ "tie_embeddings": true
14
+ },
15
+ "n_params": 14262144,
16
+ "training_args": {
17
+ "train_split": "train",
18
+ "limit_files": null,
19
+ "domains": "cpp,prose,py",
20
+ "epochs": 2,
21
+ "max_steps": null,
22
+ "batch_size": 48,
23
+ "lr": 0.0005,
24
+ "weight_decay": 0.1,
25
+ "warmup": 200,
26
+ "temperature": 2.0,
27
+ "alpha": 0.1,
28
+ "dim": 384,
29
+ "layers": 8,
30
+ "heads": 6,
31
+ "init_from": null,
32
+ "balance": "bytes",
33
+ "teacher_dtype": "float32",
34
+ "device": "cuda",
35
+ "seed": 20260927,
36
+ "log_every": 100
37
+ },
38
+ "teacher": "itazap/blt-1b-hf@91aa6b8e168046ad91e517d2191e1e974f50cc01 (BLT entropy patcher, 99.5M, float32, 512-byte sliding window)",
39
+ "train_files": 27153,
40
+ "train_windows": 164627,
41
+ "steps": 6860
42
+ }
distill_eval.json ADDED
@@ -0,0 +1,142 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "tau": 1.335442066192627,
3
+ "n_files": 1847,
4
+ "n_windows": 8887,
5
+ "student_config": {
6
+ "vocab_size": 260,
7
+ "dim": 384,
8
+ "n_layers": 8,
9
+ "n_heads": 6,
10
+ "ffn_mult": 2.6666666666666665,
11
+ "ffn_multiple_of": 64,
12
+ "max_seq_len": 513,
13
+ "rope_theta": 10000.0,
14
+ "norm_eps": 1e-05,
15
+ "tie_embeddings": true
16
+ },
17
+ "student_params": 14262144,
18
+ "trained": {
19
+ "cpp": {
20
+ "n_bytes": 2664271,
21
+ "boundary_tp": 551932,
22
+ "boundary_fp": 103571,
23
+ "boundary_fn": 75735,
24
+ "boundary_precision": 0.842,
25
+ "boundary_recall": 0.8793,
26
+ "boundary_f1": 0.8603,
27
+ "word_start_vs_teacher_f1": 0.4192,
28
+ "entropy_mae_nats": 0.1838,
29
+ "entropy_pearson_r": 0.9389,
30
+ "kl_teacher_student_nats_per_byte": 0.1394,
31
+ "bits_per_byte_student": 1.244,
32
+ "bits_per_byte_teacher": 1.1359
33
+ },
34
+ "all": {
35
+ "n_bytes": 4098495,
36
+ "boundary_tp": 862690,
37
+ "boundary_fp": 166478,
38
+ "boundary_fn": 114028,
39
+ "boundary_precision": 0.8382,
40
+ "boundary_recall": 0.8833,
41
+ "boundary_f1": 0.8602,
42
+ "word_start_vs_teacher_f1": 0.475,
43
+ "entropy_mae_nats": 0.1879,
44
+ "entropy_pearson_r": 0.9378,
45
+ "kl_teacher_student_nats_per_byte": 0.1544,
46
+ "bits_per_byte_student": 1.2656,
47
+ "bits_per_byte_teacher": 1.1356
48
+ },
49
+ "prose": {
50
+ "n_bytes": 389544,
51
+ "boundary_tp": 105995,
52
+ "boundary_fp": 25394,
53
+ "boundary_fn": 11133,
54
+ "boundary_precision": 0.8067,
55
+ "boundary_recall": 0.905,
56
+ "boundary_f1": 0.853,
57
+ "word_start_vs_teacher_f1": 0.6375,
58
+ "entropy_mae_nats": 0.2735,
59
+ "entropy_pearson_r": 0.9155,
60
+ "kl_teacher_student_nats_per_byte": 0.3319,
61
+ "bits_per_byte_student": 1.7312,
62
+ "bits_per_byte_teacher": 1.3702
63
+ },
64
+ "py": {
65
+ "n_bytes": 1044680,
66
+ "boundary_tp": 204763,
67
+ "boundary_fp": 37513,
68
+ "boundary_fn": 27160,
69
+ "boundary_precision": 0.8452,
70
+ "boundary_recall": 0.8829,
71
+ "boundary_f1": 0.8636,
72
+ "word_start_vs_teacher_f1": 0.5331,
73
+ "entropy_mae_nats": 0.1664,
74
+ "entropy_pearson_r": 0.9448,
75
+ "kl_teacher_student_nats_per_byte": 0.1264,
76
+ "bits_per_byte_student": 1.147,
77
+ "bits_per_byte_teacher": 1.0474
78
+ }
79
+ },
80
+ "untrained_baseline": {
81
+ "cpp": {
82
+ "n_bytes": 2664271,
83
+ "boundary_tp": 627667,
84
+ "boundary_fp": 2031254,
85
+ "boundary_fn": 0,
86
+ "boundary_precision": 0.2361,
87
+ "boundary_recall": 1.0,
88
+ "boundary_f1": 0.382,
89
+ "word_start_vs_teacher_f1": 0.4192,
90
+ "entropy_mae_nats": 4.7338,
91
+ "entropy_pearson_r": -0.0525,
92
+ "kl_teacher_student_nats_per_byte": 4.8287,
93
+ "bits_per_byte_student": 8.0494,
94
+ "bits_per_byte_teacher": 1.1359
95
+ },
96
+ "all": {
97
+ "n_bytes": 4098495,
98
+ "boundary_tp": 976718,
99
+ "boundary_fp": 3112890,
100
+ "boundary_fn": 0,
101
+ "boundary_precision": 0.2388,
102
+ "boundary_recall": 1.0,
103
+ "boundary_f1": 0.3856,
104
+ "word_start_vs_teacher_f1": 0.475,
105
+ "entropy_mae_nats": 4.7234,
106
+ "entropy_pearson_r": -0.0375,
107
+ "kl_teacher_student_nats_per_byte": 4.821,
108
+ "bits_per_byte_student": 8.0549,
109
+ "bits_per_byte_teacher": 1.1356
110
+ },
111
+ "prose": {
112
+ "n_bytes": 389544,
113
+ "boundary_tp": 117128,
114
+ "boundary_fp": 271400,
115
+ "boundary_fn": 0,
116
+ "boundary_precision": 0.3015,
117
+ "boundary_recall": 1.0,
118
+ "boundary_f1": 0.4633,
119
+ "word_start_vs_teacher_f1": 0.6375,
120
+ "entropy_mae_nats": 4.5293,
121
+ "entropy_pearson_r": 0.1004,
122
+ "kl_teacher_student_nats_per_byte": 4.6205,
123
+ "bits_per_byte_student": 8.0615,
124
+ "bits_per_byte_teacher": 1.3702
125
+ },
126
+ "py": {
127
+ "n_bytes": 1044680,
128
+ "boundary_tp": 231923,
129
+ "boundary_fp": 810236,
130
+ "boundary_fn": 0,
131
+ "boundary_precision": 0.2225,
132
+ "boundary_recall": 1.0,
133
+ "boundary_f1": 0.3641,
134
+ "word_start_vs_teacher_f1": 0.5331,
135
+ "entropy_mae_nats": 4.7692,
136
+ "entropy_pearson_r": -0.0274,
137
+ "kl_teacher_student_nats_per_byte": 4.8763,
138
+ "bits_per_byte_student": 8.0662,
139
+ "bits_per_byte_teacher": 1.0474
140
+ }
141
+ }
142
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:048ae0c2c473b75b8fc4f393777da633d71f9a9269d88c199895e989bfa4d33c
3
+ size 57055608
student.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4002740a1689ac101cc853b096c98df3b2059158e85607a5e1af80dc39118ddf
3
+ size 57076664
train_log.json ADDED
@@ -0,0 +1,684 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "args": {
3
+ "ckpt_dir": "/home2/divyansh.atri/escape-final-20260928/checkpoints/itazap__blt-1b-hf/91aa6b8e168046ad91e517d2191e1e974f50cc01",
4
+ "data": "/scratch/divyansh.atri/escape-47454/corpus_ext/manifest_ext.parquet",
5
+ "out": "results/distill/balanced/student",
6
+ "train_split": "train",
7
+ "limit_files": null,
8
+ "domains": "cpp,prose,py",
9
+ "epochs": 2,
10
+ "max_steps": null,
11
+ "batch_size": 48,
12
+ "lr": 0.0005,
13
+ "weight_decay": 0.1,
14
+ "warmup": 200,
15
+ "temperature": 2.0,
16
+ "alpha": 0.1,
17
+ "dim": 384,
18
+ "layers": 8,
19
+ "heads": 6,
20
+ "init_from": null,
21
+ "balance": "bytes",
22
+ "teacher_dtype": "float32",
23
+ "device": "cuda",
24
+ "seed": 20260927,
25
+ "log_every": 100
26
+ },
27
+ "student_params": 14262144,
28
+ "n_files": 27153,
29
+ "n_windows": 164627,
30
+ "steps": 6860,
31
+ "balance": {
32
+ "domains": {
33
+ "cpp": {
34
+ "bytes": 56921312,
35
+ "share_unbalanced": 0.7338611227169172,
36
+ "share_balanced": 0.3333333333333333
37
+ },
38
+ "prose": {
39
+ "bytes": 5755361,
40
+ "share_unbalanced": 0.07420130592037581,
41
+ "share_balanced": 0.3333333333333333
42
+ },
43
+ "py": {
44
+ "bytes": 14887474,
45
+ "share_unbalanced": 0.19193757136270706,
46
+ "share_balanced": 0.3333333333333333
47
+ }
48
+ },
49
+ "bytes_per_epoch_unbalanced": 77564147,
50
+ "expected_bytes_per_epoch_balanced": 70694344.45608097
51
+ },
52
+ "log": [
53
+ {
54
+ "step": 0,
55
+ "epoch": 0,
56
+ "loss": 12.873966217041016,
57
+ "soft_kl": 13.68190860748291,
58
+ "hard_ce_nats": 5.602479457855225,
59
+ "lr": 2.5e-06,
60
+ "elapsed_s": 1.8
61
+ },
62
+ {
63
+ "step": 100,
64
+ "epoch": 0,
65
+ "loss": 4.780937194824219,
66
+ "soft_kl": 4.998824596405029,
67
+ "hard_ce_nats": 2.8199539184570312,
68
+ "lr": 0.0002525,
69
+ "elapsed_s": 97.8
70
+ },
71
+ {
72
+ "step": 200,
73
+ "epoch": 0,
74
+ "loss": 2.9747471809387207,
75
+ "soft_kl": 3.064063787460327,
76
+ "hard_ce_nats": 2.170897960662842,
77
+ "lr": 0.0005,
78
+ "elapsed_s": 196.4
79
+ },
80
+ {
81
+ "step": 300,
82
+ "epoch": 0,
83
+ "loss": 2.0357463359832764,
84
+ "soft_kl": 2.0645852088928223,
85
+ "hard_ce_nats": 1.7761958837509155,
86
+ "lr": 0.0004997219129473495,
87
+ "elapsed_s": 295.8
88
+ },
89
+ {
90
+ "step": 400,
91
+ "epoch": 0,
92
+ "loss": 1.7224740982055664,
93
+ "soft_kl": 1.7252318859100342,
94
+ "hard_ce_nats": 1.6976542472839355,
95
+ "lr": 0.0004988882704486687,
96
+ "elapsed_s": 395.3
97
+ },
98
+ {
99
+ "step": 500,
100
+ "epoch": 0,
101
+ "loss": 1.3925162553787231,
102
+ "soft_kl": 1.3830686807632446,
103
+ "hard_ce_nats": 1.4775440692901611,
104
+ "lr": 0.0004975009271054409,
105
+ "elapsed_s": 494.7
106
+ },
107
+ {
108
+ "step": 600,
109
+ "epoch": 0,
110
+ "loss": 1.1991534233093262,
111
+ "soft_kl": 1.18222975730896,
112
+ "hard_ce_nats": 1.351466417312622,
113
+ "lr": 0.0004955629693354371,
114
+ "elapsed_s": 594.3
115
+ },
116
+ {
117
+ "step": 700,
118
+ "epoch": 0,
119
+ "loss": 1.081906795501709,
120
+ "soft_kl": 1.0567854642868042,
121
+ "hard_ce_nats": 1.3079982995986938,
122
+ "lr": 0.0004930787085063723,
123
+ "elapsed_s": 693.9
124
+ },
125
+ {
126
+ "step": 800,
127
+ "epoch": 0,
128
+ "loss": 0.8964716196060181,
129
+ "soft_kl": 0.8621719479560852,
130
+ "hard_ce_nats": 1.2051692008972168,
131
+ "lr": 0.0004900536713444224,
132
+ "elapsed_s": 793.4
133
+ },
134
+ {
135
+ "step": 900,
136
+ "epoch": 0,
137
+ "loss": 0.8949131369590759,
138
+ "soft_kl": 0.852584958076477,
139
+ "hard_ce_nats": 1.275866985321045,
140
+ "lr": 0.00048649458763893563,
141
+ "elapsed_s": 892.7
142
+ },
143
+ {
144
+ "step": 1000,
145
+ "epoch": 0,
146
+ "loss": 0.7854101061820984,
147
+ "soft_kl": 0.7492253184318542,
148
+ "hard_ce_nats": 1.111073613166809,
149
+ "lr": 0.0004824093752706943,
150
+ "elapsed_s": 992.0
151
+ },
152
+ {
153
+ "step": 1100,
154
+ "epoch": 0,
155
+ "loss": 0.7754520773887634,
156
+ "soft_kl": 0.7400907874107361,
157
+ "hard_ce_nats": 1.0937039852142334,
158
+ "lr": 0.00047780712259703394,
159
+ "elapsed_s": 1091.4
160
+ },
161
+ {
162
+ "step": 1200,
163
+ "epoch": 0,
164
+ "loss": 0.7441393136978149,
165
+ "soft_kl": 0.7078795433044434,
166
+ "hard_ce_nats": 1.0704773664474487,
167
+ "lr": 0.0004726980682330071,
168
+ "elapsed_s": 1190.6
169
+ },
170
+ {
171
+ "step": 1300,
172
+ "epoch": 0,
173
+ "loss": 0.6522114276885986,
174
+ "soft_kl": 0.6119727492332458,
175
+ "hard_ce_nats": 1.014359712600708,
176
+ "lr": 0.0004670935782735732,
177
+ "elapsed_s": 1289.7
178
+ },
179
+ {
180
+ "step": 1400,
181
+ "epoch": 0,
182
+ "loss": 0.7155258059501648,
183
+ "soft_kl": 0.6755514144897461,
184
+ "hard_ce_nats": 1.0752949714660645,
185
+ "lr": 0.00046100612100748764,
186
+ "elapsed_s": 1389.0
187
+ },
188
+ {
189
+ "step": 1500,
190
+ "epoch": 0,
191
+ "loss": 0.6120520234107971,
192
+ "soft_kl": 0.5696099400520325,
193
+ "hard_ce_nats": 0.9940305948257446,
194
+ "lr": 0.0004544492391791445,
195
+ "elapsed_s": 1488.4
196
+ },
197
+ {
198
+ "step": 1600,
199
+ "epoch": 0,
200
+ "loss": 0.6472983956336975,
201
+ "soft_kl": 0.601466953754425,
202
+ "hard_ce_nats": 1.059781551361084,
203
+ "lr": 0.00044743751986008154,
204
+ "elapsed_s": 1587.6
205
+ },
206
+ {
207
+ "step": 1700,
208
+ "epoch": 0,
209
+ "loss": 0.7008551359176636,
210
+ "soft_kl": 0.657149612903595,
211
+ "hard_ce_nats": 1.0942047834396362,
212
+ "lr": 0.00043998656199717434,
213
+ "elapsed_s": 1686.7
214
+ },
215
+ {
216
+ "step": 1800,
217
+ "epoch": 0,
218
+ "loss": 0.6515892744064331,
219
+ "soft_kl": 0.6068593859672546,
220
+ "hard_ce_nats": 1.0541582107543945,
221
+ "lr": 0.00043211294170971527,
222
+ "elapsed_s": 1786.0
223
+ },
224
+ {
225
+ "step": 1900,
226
+ "epoch": 0,
227
+ "loss": 0.5724945664405823,
228
+ "soft_kl": 0.5240790843963623,
229
+ "hard_ce_nats": 1.008233904838562,
230
+ "lr": 0.00042383417541257953,
231
+ "elapsed_s": 1885.3
232
+ },
233
+ {
234
+ "step": 2000,
235
+ "epoch": 0,
236
+ "loss": 0.563696026802063,
237
+ "soft_kl": 0.514342725276947,
238
+ "hard_ce_nats": 1.0078755617141724,
239
+ "lr": 0.00041516868084752034,
240
+ "elapsed_s": 1984.4
241
+ },
242
+ {
243
+ "step": 2100,
244
+ "epoch": 0,
245
+ "loss": 0.5686582922935486,
246
+ "soft_kl": 0.5149572491645813,
247
+ "hard_ce_nats": 1.0519676208496094,
248
+ "lr": 0.00040613573610928476,
249
+ "elapsed_s": 2083.7
250
+ },
251
+ {
252
+ "step": 2200,
253
+ "epoch": 0,
254
+ "loss": 0.593528687953949,
255
+ "soft_kl": 0.5461211800575256,
256
+ "hard_ce_nats": 1.0201961994171143,
257
+ "lr": 0.00039675543675770465,
258
+ "elapsed_s": 2182.9
259
+ },
260
+ {
261
+ "step": 2300,
262
+ "epoch": 0,
263
+ "loss": 0.48769474029541016,
264
+ "soft_kl": 0.4408039450645447,
265
+ "hard_ce_nats": 0.9097119569778442,
266
+ "lr": 0.0003870486511111775,
267
+ "elapsed_s": 2282.2
268
+ },
269
+ {
270
+ "step": 2400,
271
+ "epoch": 0,
272
+ "loss": 0.580201268196106,
273
+ "soft_kl": 0.5311566591262817,
274
+ "hard_ce_nats": 1.021602749824524,
275
+ "lr": 0.00037703697382099235,
276
+ "elapsed_s": 2381.5
277
+ },
278
+ {
279
+ "step": 2500,
280
+ "epoch": 0,
281
+ "loss": 0.49002501368522644,
282
+ "soft_kl": 0.43811142444610596,
283
+ "hard_ce_nats": 0.957247257232666,
284
+ "lr": 0.000366742677829787,
285
+ "elapsed_s": 2480.9
286
+ },
287
+ {
288
+ "step": 2600,
289
+ "epoch": 0,
290
+ "loss": 0.5701421499252319,
291
+ "soft_kl": 0.518459677696228,
292
+ "hard_ce_nats": 1.035284161567688,
293
+ "lr": 0.000356188664821012,
294
+ "elapsed_s": 2580.3
295
+ },
296
+ {
297
+ "step": 2700,
298
+ "epoch": 0,
299
+ "loss": 0.5370899438858032,
300
+ "soft_kl": 0.48948872089385986,
301
+ "hard_ce_nats": 0.965501070022583,
302
+ "lr": 0.00034539841426963716,
303
+ "elapsed_s": 2679.7
304
+ },
305
+ {
306
+ "step": 2800,
307
+ "epoch": 0,
308
+ "loss": 0.530117392539978,
309
+ "soft_kl": 0.47599563002586365,
310
+ "hard_ce_nats": 1.0172133445739746,
311
+ "lr": 0.00033439593120744815,
312
+ "elapsed_s": 2778.9
313
+ },
314
+ {
315
+ "step": 2900,
316
+ "epoch": 0,
317
+ "loss": 0.5172345638275146,
318
+ "soft_kl": 0.4652354419231415,
319
+ "hard_ce_nats": 0.9852264523506165,
320
+ "lr": 0.0003232056928191376,
321
+ "elapsed_s": 2878.0
322
+ },
323
+ {
324
+ "step": 3000,
325
+ "epoch": 0,
326
+ "loss": 0.46932294964790344,
327
+ "soft_kl": 0.41690340638160706,
328
+ "hard_ce_nats": 0.9410988092422485,
329
+ "lr": 0.00031185259398800065,
330
+ "elapsed_s": 2977.1
331
+ },
332
+ {
333
+ "step": 3100,
334
+ "epoch": 0,
335
+ "loss": 0.4713321328163147,
336
+ "soft_kl": 0.4231148958206177,
337
+ "hard_ce_nats": 0.9052873849868774,
338
+ "lr": 0.0003003618919123763,
339
+ "elapsed_s": 3076.3
340
+ },
341
+ {
342
+ "step": 3200,
343
+ "epoch": 0,
344
+ "loss": 0.5417460799217224,
345
+ "soft_kl": 0.48801079392433167,
346
+ "hard_ce_nats": 1.025363802909851,
347
+ "lr": 0.00028875914991604944,
348
+ "elapsed_s": 3175.7
349
+ },
350
+ {
351
+ "step": 3300,
352
+ "epoch": 0,
353
+ "loss": 0.46660807728767395,
354
+ "soft_kl": 0.4174887239933014,
355
+ "hard_ce_nats": 0.9086824059486389,
356
+ "lr": 0.00027707018057761544,
357
+ "elapsed_s": 3275.0
358
+ },
359
+ {
360
+ "step": 3400,
361
+ "epoch": 0,
362
+ "loss": 0.5409983396530151,
363
+ "soft_kl": 0.48233234882354736,
364
+ "hard_ce_nats": 1.068992257118225,
365
+ "lr": 0.00026532098830532906,
366
+ "elapsed_s": 3374.3
367
+ },
368
+ {
369
+ "step": 3500,
370
+ "epoch": 1,
371
+ "loss": 0.4823259115219116,
372
+ "soft_kl": 0.42920613288879395,
373
+ "hard_ce_nats": 0.9604039788246155,
374
+ "lr": 0.00025353771148519056,
375
+ "elapsed_s": 3473.4
376
+ },
377
+ {
378
+ "step": 3600,
379
+ "epoch": 1,
380
+ "loss": 0.4761112332344055,
381
+ "soft_kl": 0.42195048928260803,
382
+ "hard_ce_nats": 0.96355801820755,
383
+ "lr": 0.0002417465643309716,
384
+ "elapsed_s": 3572.8
385
+ },
386
+ {
387
+ "step": 3700,
388
+ "epoch": 1,
389
+ "loss": 0.4758032560348511,
390
+ "soft_kl": 0.4171745479106903,
391
+ "hard_ce_nats": 1.0034617185592651,
392
+ "lr": 0.0002299737785655482,
393
+ "elapsed_s": 3672.0
394
+ },
395
+ {
396
+ "step": 3800,
397
+ "epoch": 1,
398
+ "loss": 0.43555572628974915,
399
+ "soft_kl": 0.3847861886024475,
400
+ "hard_ce_nats": 0.892481803894043,
401
+ "lr": 0.0002182455450632803,
402
+ "elapsed_s": 3771.4
403
+ },
404
+ {
405
+ "step": 3900,
406
+ "epoch": 1,
407
+ "loss": 0.44638413190841675,
408
+ "soft_kl": 0.39255577325820923,
409
+ "hard_ce_nats": 0.9308394193649292,
410
+ "lr": 0.00020658795558326743,
411
+ "elapsed_s": 3870.7
412
+ },
413
+ {
414
+ "step": 4000,
415
+ "epoch": 1,
416
+ "loss": 0.40711477398872375,
417
+ "soft_kl": 0.3513217568397522,
418
+ "hard_ce_nats": 0.9092519283294678,
419
+ "lr": 0.00019502694472310563,
420
+ "elapsed_s": 3970.1
421
+ },
422
+ {
423
+ "step": 4100,
424
+ "epoch": 1,
425
+ "loss": 0.4376039206981659,
426
+ "soft_kl": 0.38366761803627014,
427
+ "hard_ce_nats": 0.92303067445755,
428
+ "lr": 0.00018358823222228097,
429
+ "elapsed_s": 4069.4
430
+ },
431
+ {
432
+ "step": 4200,
433
+ "epoch": 1,
434
+ "loss": 0.43862301111221313,
435
+ "soft_kl": 0.3840349018573761,
436
+ "hard_ce_nats": 0.9299162030220032,
437
+ "lr": 0.0001722972657435572,
438
+ "elapsed_s": 4168.6
439
+ },
440
+ {
441
+ "step": 4300,
442
+ "epoch": 1,
443
+ "loss": 0.4019254446029663,
444
+ "soft_kl": 0.3456398546695709,
445
+ "hard_ce_nats": 0.9084958434104919,
446
+ "lr": 0.0001611791642596516,
447
+ "elapsed_s": 4268.0
448
+ },
449
+ {
450
+ "step": 4400,
451
+ "epoch": 1,
452
+ "loss": 0.4028438627719879,
453
+ "soft_kl": 0.3462126851081848,
454
+ "hard_ce_nats": 0.9125243425369263,
455
+ "lr": 0.00015025866217114592,
456
+ "elapsed_s": 4367.5
457
+ },
458
+ {
459
+ "step": 4500,
460
+ "epoch": 1,
461
+ "loss": 0.38169050216674805,
462
+ "soft_kl": 0.32810860872268677,
463
+ "hard_ce_nats": 0.8639276027679443,
464
+ "lr": 0.00013956005427995423,
465
+ "elapsed_s": 4466.9
466
+ },
467
+ {
468
+ "step": 4600,
469
+ "epoch": 1,
470
+ "loss": 0.4409501850605011,
471
+ "soft_kl": 0.3887592852115631,
472
+ "hard_ce_nats": 0.9106684327125549,
473
+ "lr": 0.00012910714174076392,
474
+ "elapsed_s": 4566.2
475
+ },
476
+ {
477
+ "step": 4700,
478
+ "epoch": 1,
479
+ "loss": 0.39739909768104553,
480
+ "soft_kl": 0.3450617790222168,
481
+ "hard_ce_nats": 0.8684349656105042,
482
+ "lr": 0.00011892317911069211,
483
+ "elapsed_s": 4665.5
484
+ },
485
+ {
486
+ "step": 4800,
487
+ "epoch": 1,
488
+ "loss": 0.45169970393180847,
489
+ "soft_kl": 0.3956352174282074,
490
+ "hard_ce_nats": 0.9562801718711853,
491
+ "lr": 0.00010903082261495558,
492
+ "elapsed_s": 4765.0
493
+ },
494
+ {
495
+ "step": 4900,
496
+ "epoch": 1,
497
+ "loss": 0.39587515592575073,
498
+ "soft_kl": 0.33637282252311707,
499
+ "hard_ce_nats": 0.9313962459564209,
500
+ "lr": 9.945207974364768e-05,
501
+ "elapsed_s": 4864.4
502
+ },
503
+ {
504
+ "step": 5000,
505
+ "epoch": 1,
506
+ "loss": 0.4139118194580078,
507
+ "soft_kl": 0.35783103108406067,
508
+ "hard_ce_nats": 0.9186391234397888,
509
+ "lr": 9.020826029175384e-05,
510
+ "elapsed_s": 4963.7
511
+ },
512
+ {
513
+ "step": 5100,
514
+ "epoch": 1,
515
+ "loss": 0.3647778630256653,
516
+ "soft_kl": 0.3084366023540497,
517
+ "hard_ce_nats": 0.8718493580818176,
518
+ "lr": 8.131992895132693e-05,
519
+ "elapsed_s": 5063.0
520
+ },
521
+ {
522
+ "step": 5200,
523
+ "epoch": 1,
524
+ "loss": 0.3497371971607208,
525
+ "soft_kl": 0.2946958541870117,
526
+ "hard_ce_nats": 0.8451091051101685,
527
+ "lr": 7.280685956129049e-05,
528
+ "elapsed_s": 5162.3
529
+ },
530
+ {
531
+ "step": 5300,
532
+ "epoch": 1,
533
+ "loss": 0.3803977370262146,
534
+ "soft_kl": 0.3231676518917084,
535
+ "hard_ce_nats": 0.8954684138298035,
536
+ "lr": 6.468799111665003e-05,
537
+ "elapsed_s": 5261.6
538
+ },
539
+ {
540
+ "step": 5400,
541
+ "epoch": 1,
542
+ "loss": 0.37820374965667725,
543
+ "soft_kl": 0.3225654065608978,
544
+ "hard_ce_nats": 0.8789486885070801,
545
+ "lr": 5.6981385634978535e-05,
546
+ "elapsed_s": 5361.0
547
+ },
548
+ {
549
+ "step": 5500,
550
+ "epoch": 1,
551
+ "loss": 0.3798903524875641,
552
+ "soft_kl": 0.32143938541412354,
553
+ "hard_ce_nats": 0.905949056148529,
554
+ "lr": 4.970418797391063e-05,
555
+ "elapsed_s": 5460.3
556
+ },
557
+ {
558
+ "step": 5600,
559
+ "epoch": 1,
560
+ "loss": 0.36526399850845337,
561
+ "soft_kl": 0.30636417865753174,
562
+ "hard_ce_nats": 0.8953626155853271,
563
+ "lr": 4.287258768903948e-05,
564
+ "elapsed_s": 5559.7
565
+ },
566
+ {
567
+ "step": 5700,
568
+ "epoch": 1,
569
+ "loss": 0.37662628293037415,
570
+ "soft_kl": 0.3160785138607025,
571
+ "hard_ce_nats": 0.9215564131736755,
572
+ "lr": 3.650178301706983e-05,
573
+ "elapsed_s": 5659.1
574
+ },
575
+ {
576
+ "step": 5800,
577
+ "epoch": 1,
578
+ "loss": 0.3742494285106659,
579
+ "soft_kl": 0.3165742754936218,
580
+ "hard_ce_nats": 0.8933258056640625,
581
+ "lr": 3.06059470643556e-05,
582
+ "elapsed_s": 5758.5
583
+ },
584
+ {
585
+ "step": 5900,
586
+ "epoch": 1,
587
+ "loss": 0.36944907903671265,
588
+ "soft_kl": 0.315045565366745,
589
+ "hard_ce_nats": 0.8590807914733887,
590
+ "lr": 2.519819627604078e-05,
591
+ "elapsed_s": 5858.1
592
+ },
593
+ {
594
+ "step": 6000,
595
+ "epoch": 1,
596
+ "loss": 0.36059680581092834,
597
+ "soft_kl": 0.3076575696468353,
598
+ "hard_ce_nats": 0.8370499610900879,
599
+ "lr": 2.0290561255950966e-05,
600
+ "elapsed_s": 5957.6
601
+ },
602
+ {
603
+ "step": 6100,
604
+ "epoch": 1,
605
+ "loss": 0.34553763270378113,
606
+ "soft_kl": 0.2902710437774658,
607
+ "hard_ce_nats": 0.8429369926452637,
608
+ "lr": 1.5893960002151902e-05,
609
+ "elapsed_s": 6057.1
610
+ },
611
+ {
612
+ "step": 6200,
613
+ "epoch": 1,
614
+ "loss": 0.36889544129371643,
615
+ "soft_kl": 0.31231969594955444,
616
+ "hard_ce_nats": 0.8780772089958191,
617
+ "lr": 1.201817361771837e-05,
618
+ "elapsed_s": 6156.5
619
+ },
620
+ {
621
+ "step": 6300,
622
+ "epoch": 1,
623
+ "loss": 0.35631775856018066,
624
+ "soft_kl": 0.295655757188797,
625
+ "hard_ce_nats": 0.902275562286377,
626
+ "lr": 8.671824550749163e-06,
627
+ "elapsed_s": 6256.2
628
+ },
629
+ {
630
+ "step": 6400,
631
+ "epoch": 1,
632
+ "loss": 0.37073439359664917,
633
+ "soft_kl": 0.31267228722572327,
634
+ "hard_ce_nats": 0.8932932615280151,
635
+ "lr": 5.862357412037667e-06,
636
+ "elapsed_s": 6355.8
637
+ },
638
+ {
639
+ "step": 6500,
640
+ "epoch": 1,
641
+ "loss": 0.37577736377716064,
642
+ "soft_kl": 0.3222424387931824,
643
+ "hard_ce_nats": 0.8575915098190308,
644
+ "lr": 3.5960224130728858e-06,
645
+ "elapsed_s": 6455.3
646
+ },
647
+ {
648
+ "step": 6600,
649
+ "epoch": 1,
650
+ "loss": 0.35561278462409973,
651
+ "soft_kl": 0.2962757349014282,
652
+ "hard_ce_nats": 0.8896464109420776,
653
+ "lr": 1.8778614612162404e-06,
654
+ "elapsed_s": 6554.7
655
+ },
656
+ {
657
+ "step": 6700,
658
+ "epoch": 1,
659
+ "loss": 0.3584456145763397,
660
+ "soft_kl": 0.3007999062538147,
661
+ "hard_ce_nats": 0.8772570490837097,
662
+ "lr": 7.116969429883935e-07,
663
+ "elapsed_s": 6654.2
664
+ },
665
+ {
666
+ "step": 6800,
667
+ "epoch": 1,
668
+ "loss": 0.3334779441356659,
669
+ "soft_kl": 0.27721670269966125,
670
+ "hard_ce_nats": 0.83982914686203,
671
+ "lr": 1.0012322041960675e-07,
672
+ "elapsed_s": 6753.7
673
+ },
674
+ {
675
+ "step": 6859,
676
+ "epoch": 1,
677
+ "loss": 0.3522360324859619,
678
+ "soft_kl": 0.2891015410423279,
679
+ "hard_ce_nats": 0.9204465746879578,
680
+ "lr": 2.781386176931022e-11,
681
+ "elapsed_s": 6812.1
682
+ }
683
+ ]
684
+ }