kennedy-kitoko commited on
Commit
626ff03
·
verified ·
1 Parent(s): e0ccaf8

Publish Traffic AI phase decision checkpoint

Browse files
LICENSE ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ MIT License
2
+
3
+ Copyright (c) 2026 KITOKO MUYUNGA KENNEDY
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
README.md CHANGED
@@ -1,3 +1,153 @@
1
  ---
2
- license: mit
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ license: apache-2.0
3
+ language:
4
+ - en
5
+ library_name: laya
6
+ pipeline_tag: text-classification
7
+ base_model: convaiinnovations/laya-typed-decisions
8
+ tags:
9
+ - decision-model
10
+ - traffic-signal-control
11
+ - laya
12
+ - jevlight
13
+ - sumo
14
+ - system-one
15
+ - smart-city
16
  ---
17
+
18
+ # Traffic AI
19
+
20
+ Traffic AI is a fast, local **phase-decision model for adaptive traffic-signal control**. It specializes the general-purpose [`convaiinnovations/laya-typed-decisions`](https://huggingface.co/convaiinnovations/laya-typed-decisions) checkpoint for four traffic phases and runs without generating text.
21
+
22
+ The model scores four choices directly:
23
+
24
+ - `ETWT`: east-west through traffic;
25
+ - `NTST`: north-south through traffic;
26
+ - `ELWL`: east-west left turns;
27
+ - `NLSL`: north-south left turns.
28
+
29
+ Traffic AI chooses the phase. A separate deterministic **Traffic Protocol** enforces yellow transitions, minimum green, maximum continuous green, event-triggered re-evaluation, starvation priority, hysteresis, and other operational constraints. Green duration is therefore controlled by the protocol rather than predicted as an unconstrained class.
30
+
31
+ ## Lineage
32
+
33
+ | Item | Value |
34
+ |---|---|
35
+ | Base model | `convaiinnovations/laya-typed-decisions` |
36
+ | Base revision | `e929ae5cf69bc34259cd2f95c9e91145b818b1f0` |
37
+ | Base license | Apache-2.0 |
38
+ | Training method | Official Laya RLCD fine-tuning pipeline |
39
+ | Training seed | 3407 |
40
+ | Output weight | `model.safetensors` |
41
+ | Output size | 842,609,220 bytes |
42
+ | Output SHA-256 | `6f7b017bef105022a4f95a6ea84b1f3db3669f48cf9cbc6a320e5271bd697304` |
43
+
44
+ The exact model and tokenizer hashes are listed in [`SHA256SUMS`](SHA256SUMS).
45
+
46
+ ## Training data
47
+
48
+ The phase-specialization curriculum contains **3,200 deterministic synthetic traffic states**:
49
+
50
+ - train: 2,400;
51
+ - validation: 400;
52
+ - test: 400.
53
+
54
+ Every split is balanced across the four phase labels. The rows use Laya's typed-decision schema (`state`, `questions`, `gold`) with soft phase distributions. The targets are computed by explicit traffic rules from queues, waiting times, approaching vehicles, downstream occupancy, and current-phase hysteresis.
55
+
56
+ This curriculum was generated without a simulator and without a teacher API. It teaches phase semantics; it is not evidence that the labels are globally optimal actions. SUMO performance was evaluated separately after training.
57
+
58
+ ## Training configuration
59
+
60
+ - official Laya RLCD loss;
61
+ - 10 epochs;
62
+ - micro-batch size 8;
63
+ - gradient accumulation 8;
64
+ - encoder learning rate `2.5e-5`;
65
+ - decision-head learning rate `1e-4`;
66
+ - weight decay `0.01`;
67
+ - seed `3407`.
68
+
69
+ On the held-out 400-row phase-specification validation set, accuracy changed from **35.5% before specialization to 98.0% after specialization**. This is an offline curriculum result, not a traffic-simulation metric. The post-training ECE was 0.2332, so the probabilities should not be treated as perfectly calibrated.
70
+
71
+ ## Download
72
+
73
+ ```bash
74
+ hf download kennedy-kitoko/Traffic-AI \
75
+ --local-dir checkpoints/traffic-ai-v2
76
+ ```
77
+
78
+ Verify the files from inside the downloaded directory:
79
+
80
+ ```bash
81
+ sha256sum -c SHA256SUMS
82
+ ```
83
+
84
+ ## Local inference
85
+
86
+ Install the Laya package in a CUDA-enabled Python environment and load the downloaded folder:
87
+
88
+ ```python
89
+ from laya import Agent
90
+
91
+ agent = Agent("checkpoints/traffic-ai-v2", device="cuda")
92
+
93
+ state = {
94
+ "intersection_id": "intersection_1_1",
95
+ "current_phase": "ETWT",
96
+ "phase_metrics": {
97
+ "ETWT": {"queue_vehicles": 4, "total_waiting_seconds": 80},
98
+ "NTST": {"queue_vehicles": 25, "total_waiting_seconds": 650},
99
+ "ELWL": {"queue_vehicles": 3, "total_waiting_seconds": 40},
100
+ "NLSL": {"queue_vehicles": 5, "total_waiting_seconds": 70},
101
+ },
102
+ }
103
+
104
+ questions = {
105
+ "phase": {
106
+ "type": "choice",
107
+ "instructions": "Choose the traffic phase that should be served next.",
108
+ "criteria": {
109
+ "ETWT": "East-west through traffic",
110
+ "NTST": "North-south through traffic",
111
+ "ELWL": "East-west left turns",
112
+ "NLSL": "North-south left turns",
113
+ },
114
+ }
115
+ }
116
+
117
+ result = agent.system_one(state, questions)
118
+ print(result["answers"]["phase"])
119
+ ```
120
+
121
+ For batched real-time control, the project runner uses `Agent.predict_batch(states, questions)` and applies the Traffic Protocol after receiving the four probabilities.
122
+
123
+ ## SUMO integration and evidence
124
+
125
+ The complete controller, Traffic Protocol, reproducibility manifests, tests, and measured SUMO evidence are maintained in the [Traffic AI GitHub repository](https://github.com/kennedy-kitoko/Traffic-AI).
126
+
127
+ The published studies distinguish:
128
+
129
+ - phase-model inference from protocol decisions;
130
+ - Jinan from Hangzhou experiments;
131
+ - the exact JevLight revision used for each city;
132
+ - measured evidence from interpretation;
133
+ - final results from preliminary campaigns.
134
+
135
+ ## Limitations
136
+
137
+ - This model has not been validated on real roads.
138
+ - Its synthetic curriculum does not prove optimal traffic control.
139
+ - The phase model depends on the external Traffic Protocol for safe timing behavior.
140
+ - Jinan and Hangzhou simulation results do not establish universal cross-city generalization.
141
+ - Reproduction can be methodologically and statistically equivalent without producing bit-identical CUDA weights.
142
+ - Operators must validate phase mappings, safety constraints, detectors, timing rules, and fail-safe behavior for each deployment.
143
+
144
+ ## License and attribution
145
+
146
+ The checkpoint is distributed under Apache-2.0, consistent with its Laya base model. The accompanying Traffic AI project code is separately published under MIT.
147
+
148
+ Traffic AI builds on:
149
+
150
+ - [Laya](https://github.com/NandhaKishorM/laya);
151
+ - [`convaiinnovations/laya-typed-decisions`](https://huggingface.co/convaiinnovations/laya-typed-decisions);
152
+ - [JevLight](https://github.com/usail-hkust/JevLight);
153
+ - [SUMO](https://github.com/eclipse-sumo/sumo).
SHA256SUMS ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ 6f7b017bef105022a4f95a6ea84b1f3db3669f48cf9cbc6a320e5271bd697304 model.safetensors
2
+ 002e8fb22be55a527f8190708d7f7c657c0ee0ff00014b14d49f3900c0b7dd5a questions.json
3
+ c2e3d98a063a9b00b17a49d38d51a118fad682d7af58c551223b1d91461654ba rl_agent_config.json
4
+ 40be6e7eaa00e3d47cd1c60ee9aeac96314df9bedbf52788cbb0f8822249e883 train_report.json
5
+ 666d3dc67b572356b7b651333a46de24635c6ce52ecde250c268e7c6a94e2a0c encoder/config.json
6
+ bd3408343458ce0434909fb1eceb99c1c8ca6b833610c04d08160f3d3c792a87 tokenizer/tokenizer_config.json
7
+ 0b3214f8efe1327290428b84e5d344ff3f2442686c1b89fb7b42c13a79ce6779 tokenizer/tokenizer.json
encoder/config.json ADDED
@@ -0,0 +1,84 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "ModernBertForMaskedLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 50281,
8
+ "classifier_activation": "gelu",
9
+ "classifier_bias": false,
10
+ "classifier_dropout": 0.0,
11
+ "classifier_pooling": "mean",
12
+ "cls_token_id": 50281,
13
+ "decoder_bias": true,
14
+ "deterministic_flash_attn": false,
15
+ "dtype": "float32",
16
+ "embedding_dropout": 0.0,
17
+ "eos_token_id": 50282,
18
+ "global_attn_every_n_layers": 3,
19
+ "gradient_checkpointing": false,
20
+ "hidden_activation": "gelu",
21
+ "hidden_size": 1024,
22
+ "initializer_cutoff_factor": 2.0,
23
+ "initializer_range": 0.02,
24
+ "intermediate_size": 2624,
25
+ "layer_norm_eps": 1e-05,
26
+ "layer_types": [
27
+ "full_attention",
28
+ "sliding_attention",
29
+ "sliding_attention",
30
+ "full_attention",
31
+ "sliding_attention",
32
+ "sliding_attention",
33
+ "full_attention",
34
+ "sliding_attention",
35
+ "sliding_attention",
36
+ "full_attention",
37
+ "sliding_attention",
38
+ "sliding_attention",
39
+ "full_attention",
40
+ "sliding_attention",
41
+ "sliding_attention",
42
+ "full_attention",
43
+ "sliding_attention",
44
+ "sliding_attention",
45
+ "full_attention",
46
+ "sliding_attention",
47
+ "sliding_attention",
48
+ "full_attention",
49
+ "sliding_attention",
50
+ "sliding_attention",
51
+ "full_attention",
52
+ "sliding_attention",
53
+ "sliding_attention",
54
+ "full_attention"
55
+ ],
56
+ "local_attention": 128,
57
+ "max_position_embeddings": 8192,
58
+ "mlp_bias": false,
59
+ "mlp_dropout": 0.0,
60
+ "model_type": "modernbert",
61
+ "norm_bias": false,
62
+ "norm_eps": 1e-05,
63
+ "num_attention_heads": 16,
64
+ "num_hidden_layers": 28,
65
+ "pad_token_id": 50283,
66
+ "position_embedding_type": "absolute",
67
+ "repad_logits_with_grad": false,
68
+ "rope_parameters": {
69
+ "full_attention": {
70
+ "rope_theta": 160000.0,
71
+ "rope_type": "default"
72
+ },
73
+ "sliding_attention": {
74
+ "rope_theta": 10000.0,
75
+ "rope_type": "default"
76
+ }
77
+ },
78
+ "sep_token_id": 50282,
79
+ "sparse_pred_ignore_index": -100,
80
+ "sparse_prediction": false,
81
+ "tie_word_embeddings": true,
82
+ "transformers_version": "5.20.2",
83
+ "vocab_size": 50368
84
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6f7b017bef105022a4f95a6ea84b1f3db3669f48cf9cbc6a320e5271bd697304
3
+ size 842609220
questions.json ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "phase": {
3
+ "type": "choice",
4
+ "instructions": "Choose the admissible traffic-signal phase that best reduces network congestion, waiting, starvation and downstream spillback. Green timing and mandatory safety constraints are enforced separately by the Devos-Secure actuated protocol.",
5
+ "criteria": {
6
+ "ETWT": "Serve eastbound and westbound through movements.",
7
+ "NTST": "Serve northbound and southbound through movements.",
8
+ "ELWL": "Serve eastbound and westbound left-turn movements.",
9
+ "NLSL": "Serve northbound and southbound left-turn movements."
10
+ }
11
+ }
12
+ }
rl_agent_config.json ADDED
@@ -0,0 +1,203 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "encoder": "answerdotai/ModernBERT-large",
3
+ "head_layers": 2,
4
+ "max_len": 1024,
5
+ "head_max_len": 256,
6
+ "max_prefixes": 6,
7
+ "act_costs": {
8
+ "escalate": 0.5
9
+ },
10
+ "cost_wrong_act": 3.0,
11
+ "amp_dtype": "bf16",
12
+ "model_name": "laya-typed-decisions",
13
+ "temperature": [
14
+ 1.0349146127700806,
15
+ 1.0,
16
+ 1.0
17
+ ],
18
+ "training": {
19
+ "updates": 7313,
20
+ "epochs_completed": 1,
21
+ "hours": 1.96,
22
+ "world_size": 1,
23
+ "fine_tuned_from_checkpoint": true,
24
+ "laya_train": {
25
+ "epochs": 10,
26
+ "micro_batch": 8,
27
+ "grad_accum": 8,
28
+ "encoder_lr": 2.5e-05,
29
+ "head_lr": 0.0001,
30
+ "min_lr": 1e-06,
31
+ "weight_decay": 0.01,
32
+ "grad_clip": 1.0,
33
+ "loss": "rlcd",
34
+ "rl_samples": 4,
35
+ "sigma_start": 0.4,
36
+ "sigma_end": 0.1,
37
+ "w_sph": 0.75,
38
+ "w_rps": 1.0,
39
+ "shuffle_options": [
40
+ "choice"
41
+ ],
42
+ "option_layout": null,
43
+ "freeze_encoder": false,
44
+ "calib_max": 400,
45
+ "calib_frac": 0.1,
46
+ "calib_seed": 20260922,
47
+ "seed": 3407,
48
+ "max_len": null,
49
+ "head_max_len": null,
50
+ "amp": null,
51
+ "gradient_checkpointing": null,
52
+ "log_every": 10,
53
+ "label_smoothing": 0.0,
54
+ "text_column": "text",
55
+ "label_column": "label",
56
+ "question_id": "label",
57
+ "instructions": null,
58
+ "eval_data": "data/laya_traffic/v2_phase_specification_v1/validation.jsonl",
59
+ "target_error": 0.1,
60
+ "min_abstain_n": 10
61
+ },
62
+ "laya_train_calibration": {
63
+ "choice": {
64
+ "items": 240,
65
+ "temperature": 1.0349146127700806,
66
+ "issues": [],
67
+ "accuracy": 0.9791666666666666,
68
+ "mean_confidence": 0.7380992388438022
69
+ },
70
+ "score": {
71
+ "items": 0,
72
+ "temperature": 1.0,
73
+ "issues": []
74
+ },
75
+ "noul": {
76
+ "items": 0,
77
+ "temperature": 1.0,
78
+ "issues": []
79
+ }
80
+ },
81
+ "train_report": {
82
+ "eval_source": "data/laya_traffic/v2_phase_specification_v1/validation.jsonl",
83
+ "eval_mode": "held_out",
84
+ "is_held_out": true,
85
+ "eval_items": 400,
86
+ "note": "Metrics reflect independent held-out evaluation.",
87
+ "before": {
88
+ "items": 400,
89
+ "loss": 1.3888,
90
+ "accuracy": 0.355,
91
+ "mean_confidence": 0.2944,
92
+ "ece": 0.0606,
93
+ "brier": 0.4109,
94
+ "brier_top1": 0.2326,
95
+ "by_type": {
96
+ "choice": {
97
+ "items": 400,
98
+ "loss": 1.3888,
99
+ "accuracy": 0.355,
100
+ "mean_confidence": 0.2944,
101
+ "ece": 0.0606,
102
+ "brier": 0.4109,
103
+ "brier_top1": 0.2326
104
+ }
105
+ }
106
+ },
107
+ "after": {
108
+ "items": 400,
109
+ "loss": 0.6542,
110
+ "accuracy": 0.98,
111
+ "mean_confidence": 0.7481,
112
+ "ece": 0.2332,
113
+ "brier": 0.0004,
114
+ "brier_top1": 0.1083,
115
+ "by_type": {
116
+ "choice": {
117
+ "items": 400,
118
+ "loss": 0.6542,
119
+ "accuracy": 0.98,
120
+ "mean_confidence": 0.7481,
121
+ "ece": 0.2332,
122
+ "brier": 0.0004,
123
+ "brier_top1": 0.1083
124
+ }
125
+ }
126
+ },
127
+ "comparison": {
128
+ "delta_accuracy": 0.625,
129
+ "delta_loss": -0.7346,
130
+ "delta_ece": 0.1726,
131
+ "delta_brier": -0.4105,
132
+ "delta_brier_top1": -0.1243,
133
+ "delta_mean_confidence": 0.4537
134
+ },
135
+ "calibration": {
136
+ "choice": {
137
+ "items": 240,
138
+ "temperature": 1.0349146127700806,
139
+ "issues": [],
140
+ "accuracy": 0.9791666666666666,
141
+ "mean_confidence": 0.7380992388438022
142
+ },
143
+ "score": {
144
+ "items": 0,
145
+ "temperature": 1.0,
146
+ "issues": []
147
+ },
148
+ "noul": {
149
+ "items": 0,
150
+ "temperature": 1.0,
151
+ "issues": []
152
+ }
153
+ },
154
+ "abstention_thresholds": {
155
+ "choice:3-5": 0.2579
156
+ },
157
+ "training": {
158
+ "epochs": 10,
159
+ "micro_batch": 8,
160
+ "grad_accum": 8,
161
+ "encoder_lr": 2.5e-05,
162
+ "head_lr": 0.0001,
163
+ "min_lr": 1e-06,
164
+ "weight_decay": 0.01,
165
+ "grad_clip": 1.0,
166
+ "loss": "rlcd",
167
+ "rl_samples": 4,
168
+ "sigma_start": 0.4,
169
+ "sigma_end": 0.1,
170
+ "w_sph": 0.75,
171
+ "w_rps": 1.0,
172
+ "shuffle_options": [
173
+ "choice"
174
+ ],
175
+ "option_layout": null,
176
+ "freeze_encoder": false,
177
+ "calib_max": 400,
178
+ "calib_frac": 0.1,
179
+ "calib_seed": 20260922,
180
+ "seed": 3407,
181
+ "max_len": null,
182
+ "head_max_len": null,
183
+ "amp": null,
184
+ "gradient_checkpointing": null,
185
+ "log_every": 10,
186
+ "label_smoothing": 0.0,
187
+ "text_column": "text",
188
+ "label_column": "label",
189
+ "question_id": "label",
190
+ "instructions": null,
191
+ "eval_data": "data/laya_traffic/v2_phase_specification_v1/validation.jsonl",
192
+ "target_error": 0.1,
193
+ "min_abstain_n": 10
194
+ }
195
+ },
196
+ "abstention_thresholds": {
197
+ "choice:3-5": 0.2579
198
+ }
199
+ },
200
+ "gradient_checkpointing": true,
201
+ "max_tokens_per_batch": 4096,
202
+ "fine_tuned": true
203
+ }
tokenizer/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer/tokenizer_config.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "clean_up_tokenization_spaces": true,
3
+ "cls_token": "[CLS]",
4
+ "local_files_only": true,
5
+ "mask_token": "[MASK]",
6
+ "model_input_names": [
7
+ "input_ids",
8
+ "attention_mask"
9
+ ],
10
+ "model_max_length": 8192,
11
+ "pad_token": "[PAD]",
12
+ "sep_token": "[SEP]",
13
+ "tokenizer_class": "PreTrainedTokenizerFast",
14
+ "unk_token": "[UNK]"
15
+ }
train_report.json ADDED
@@ -0,0 +1,115 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "eval_source": "data/laya_traffic/v2_phase_specification_v1/validation.jsonl",
3
+ "eval_mode": "held_out",
4
+ "is_held_out": true,
5
+ "eval_items": 400,
6
+ "note": "Metrics reflect independent held-out evaluation.",
7
+ "before": {
8
+ "items": 400,
9
+ "loss": 1.3888,
10
+ "accuracy": 0.355,
11
+ "mean_confidence": 0.2944,
12
+ "ece": 0.0606,
13
+ "brier": 0.4109,
14
+ "brier_top1": 0.2326,
15
+ "by_type": {
16
+ "choice": {
17
+ "items": 400,
18
+ "loss": 1.3888,
19
+ "accuracy": 0.355,
20
+ "mean_confidence": 0.2944,
21
+ "ece": 0.0606,
22
+ "brier": 0.4109,
23
+ "brier_top1": 0.2326
24
+ }
25
+ }
26
+ },
27
+ "after": {
28
+ "items": 400,
29
+ "loss": 0.6542,
30
+ "accuracy": 0.98,
31
+ "mean_confidence": 0.7481,
32
+ "ece": 0.2332,
33
+ "brier": 0.0004,
34
+ "brier_top1": 0.1083,
35
+ "by_type": {
36
+ "choice": {
37
+ "items": 400,
38
+ "loss": 0.6542,
39
+ "accuracy": 0.98,
40
+ "mean_confidence": 0.7481,
41
+ "ece": 0.2332,
42
+ "brier": 0.0004,
43
+ "brier_top1": 0.1083
44
+ }
45
+ }
46
+ },
47
+ "comparison": {
48
+ "delta_accuracy": 0.625,
49
+ "delta_loss": -0.7346,
50
+ "delta_ece": 0.1726,
51
+ "delta_brier": -0.4105,
52
+ "delta_brier_top1": -0.1243,
53
+ "delta_mean_confidence": 0.4537
54
+ },
55
+ "calibration": {
56
+ "choice": {
57
+ "items": 240,
58
+ "temperature": 1.0349146127700806,
59
+ "issues": [],
60
+ "accuracy": 0.9791666666666666,
61
+ "mean_confidence": 0.7380992388438022
62
+ },
63
+ "score": {
64
+ "items": 0,
65
+ "temperature": 1.0,
66
+ "issues": []
67
+ },
68
+ "noul": {
69
+ "items": 0,
70
+ "temperature": 1.0,
71
+ "issues": []
72
+ }
73
+ },
74
+ "abstention_thresholds": {
75
+ "choice:3-5": 0.2579
76
+ },
77
+ "training": {
78
+ "epochs": 10,
79
+ "micro_batch": 8,
80
+ "grad_accum": 8,
81
+ "encoder_lr": 2.5e-05,
82
+ "head_lr": 0.0001,
83
+ "min_lr": 1e-06,
84
+ "weight_decay": 0.01,
85
+ "grad_clip": 1.0,
86
+ "loss": "rlcd",
87
+ "rl_samples": 4,
88
+ "sigma_start": 0.4,
89
+ "sigma_end": 0.1,
90
+ "w_sph": 0.75,
91
+ "w_rps": 1.0,
92
+ "shuffle_options": [
93
+ "choice"
94
+ ],
95
+ "option_layout": null,
96
+ "freeze_encoder": false,
97
+ "calib_max": 400,
98
+ "calib_frac": 0.1,
99
+ "calib_seed": 20260922,
100
+ "seed": 3407,
101
+ "max_len": null,
102
+ "head_max_len": null,
103
+ "amp": null,
104
+ "gradient_checkpointing": null,
105
+ "log_every": 10,
106
+ "label_smoothing": 0.0,
107
+ "text_column": "text",
108
+ "label_column": "label",
109
+ "question_id": "label",
110
+ "instructions": null,
111
+ "eval_data": "data/laya_traffic/v2_phase_specification_v1/validation.jsonl",
112
+ "target_error": 0.1,
113
+ "min_abstain_n": 10
114
+ }
115
+ }