Upload folder using huggingface_hub
Browse files- README.md +86 -0
- assay_compiled.safetensors +3 -0
- assay_compiled_config.json +9 -0
- calibration.json +29 -0
- config.json +77 -0
- eval-dev-scaled.json +771 -0
- eval-dev.json +771 -0
- eval-holdout-scaled.json +236 -0
- eval-holdout.json +236 -0
- eval-transfer-v4-scaled.json +236 -0
- eval-transfer-v4.json +236 -0
- latency-cpu.txt +7 -0
- model.safetensors +3 -0
- tokenizer.json +0 -0
- tokenizer_config.json +17 -0
- train_args.json +26 -0
- train_log.jsonl +260 -0
README.md
ADDED
|
@@ -0,0 +1,86 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
base_model: Alibaba-NLP/gte-modernbert-base
|
| 4 |
+
language: en
|
| 5 |
+
tags:
|
| 6 |
+
- text-classification
|
| 7 |
+
- calibration
|
| 8 |
+
- decision-model
|
| 9 |
+
- assay
|
| 10 |
+
---
|
| 11 |
+
|
| 12 |
+
# assay-compiled-base
|
| 13 |
+
|
| 14 |
+
An encoder-tier decision model from the [Assay](https://github.com/bgokden/assay) project:
|
| 15 |
+
typed questions (`bool`, `choice`, `score`) over a state produce calibrated probability
|
| 16 |
+
distributions with an `evidence` signal, with no text generation and no language model. It
|
| 17 |
+
is the small, CPU-friendly tier; the decoder models ([Berk/assay-4b](https://huggingface.co/Berk/assay-4b),
|
| 18 |
+
[Berk/assay-27b](https://huggingface.co/Berk/assay-27b)) are far more accurate.
|
| 19 |
+
|
| 20 |
+
Architecture `compiled` with late interaction on `Alibaba-NLP/gte-modernbert-base`
|
| 21 |
+
(8 query slots). The state is encoded once into token embeddings. Each question is compiled once: the instruction becomes a set of query vectors (learned slots plus a projection of its pooled encoding), each option becomes a vector (its pooled encoding) and its token encodings. A decision is cross-attention from the queries over the state tokens, a small reader MLP, a bilinear score against each option vector plus a per-option bias, and a late-interaction term: the mean over option tokens of the best cosine match among state tokens, times a learned scale. Compiled questions can be cached and reused across states; after the state encode, a decision is a few small matrix products.
|
| 22 |
+
|
| 23 |
+
## Evaluation
|
| 24 |
+
|
| 25 |
+
| split | n | accuracy | Brier | NLL | ECE | confident errors |
|
| 26 |
+
|---|---|---|---|---|---|---|
|
| 27 |
+
| seen tasks (dev), scaled | 6113 | 0.668 | 0.442 | 0.806 | 0.037 | 0.019 |
|
| 28 |
+
| unseen tasks (holdout), raw | 2020 | 0.606 | 0.522 | 0.895 | 0.133 | 0.035 |
|
| 29 |
+
| unseen tasks (holdout), scaled | 2020 | 0.606 | 0.494 | 0.818 | 0.061 | 0.011 |
|
| 30 |
+
| kev transfer-v4 dev, raw | 764 | 0.542 | 0.613 | 1.048 | 0.174 | 0.064 |
|
| 31 |
+
| kev transfer-v4 dev, scaled | 764 | 0.542 | 0.572 | 0.952 | 0.095 | 0.012 |
|
| 32 |
+
|
| 33 |
+
"Scaled" applies the temperature 1.485 fitted on the seen-task
|
| 34 |
+
calibration split. Unseen tasks are eleven datasets never trained on; the transfer suite is
|
| 35 |
+
`jaredpalmer/kev-suites` transfer-v4 dev, whose sources are excluded from training. Single-text
|
| 36 |
+
classification (topic, sentiment, spam) is strong; questions that need knowledge (MMLU) or
|
| 37 |
+
multi-step reasoning are near chance. See the repository's `docs/roadmap.md` for the full
|
| 38 |
+
comparison against the cross-encoder and the decoders.
|
| 39 |
+
|
| 40 |
+
CPU latency (milliseconds; the state is encoded once, questions are compiled once and cached, `decide` runs per state x question set):
|
| 41 |
+
|
| 42 |
+
```
|
| 43 |
+
runs/compiled-late-gte-base on cpu, 8 threads
|
| 44 |
+
questions encode_state_ms compile_ms decide_ms end_to_end_ms
|
| 45 |
+
1 28.95 29.63 1.381 62.25
|
| 46 |
+
3 30.67 61.52 1.753 94.48
|
| 47 |
+
6 30.64 88.46 2.844 122.80
|
| 48 |
+
12 30.51 88.45 4.867 124.75
|
| 49 |
+
24 30.64 88.42 10.694 131.05
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
+
## Usage
|
| 53 |
+
|
| 54 |
+
```python
|
| 55 |
+
from assay.compiled import load_any
|
| 56 |
+
from assay.schema import Question
|
| 57 |
+
|
| 58 |
+
model = load_any("Berk/assay-compiled-base", device="cpu")
|
| 59 |
+
answers = model.answer(
|
| 60 |
+
"My card was charged twice for order A-104.",
|
| 61 |
+
{
|
| 62 |
+
"refund": Question(type="bool", instructions="Does the customer ask for money back?"),
|
| 63 |
+
"team": Question(type="choice", instructions="Which team should handle this?",
|
| 64 |
+
options={"billing": "Charges and refunds", "technical": "Bugs"}),
|
| 65 |
+
},
|
| 66 |
+
)
|
| 67 |
+
print(answers["team"].probabilities, answers["refund"].p_true)
|
| 68 |
+
```
|
| 69 |
+
|
| 70 |
+
Trained with `assay.train_compiled` (3.0 epochs, lr 5e-05, head lr
|
| 71 |
+
0.0005, batch 32) on the Assay data (55 public datasets
|
| 72 |
+
rendered as typed questions, synthetic policy and date cases, and 40k generic questions labelled
|
| 73 |
+
by assay-27b). Each dataset keeps its own licence; the list is in
|
| 74 |
+
[docs/datasets.md](https://github.com/bgokden/assay/blob/master/docs/datasets.md).
|
| 75 |
+
|
| 76 |
+
## Limitations
|
| 77 |
+
|
| 78 |
+
English only. No knowledge beyond what the encoder carries, no arithmetic, no multi-hop
|
| 79 |
+
reasoning. Calibrated in aggregate on the evaluated distributions, not per answer; check on
|
| 80 |
+
your own labels before acting on thresholds.
|
| 81 |
+
|
| 82 |
+
## Relationship to other work
|
| 83 |
+
|
| 84 |
+
Assay is an independent project. Jev and System One are names of TypeSafe AI's products and
|
| 85 |
+
are mentioned only to describe and compare; kev-suites is Jared Palmer's evaluation data.
|
| 86 |
+
Assay is not affiliated with or endorsed by either.
|
assay_compiled.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:61f819f7878bc60ab44da24d791694e3f7b9db59a829c74ebb1a343b52c453e8
|
| 3 |
+
size 35443028
|
assay_compiled_config.json
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"encoder_id": "Alibaba-NLP/gte-modernbert-base",
|
| 3 |
+
"slots": 8,
|
| 4 |
+
"heads": 8,
|
| 5 |
+
"temperature": 1.4849057900627562,
|
| 6 |
+
"option_instructions": false,
|
| 7 |
+
"late_interaction": true,
|
| 8 |
+
"arch": "compiled"
|
| 9 |
+
}
|
calibration.json
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"temperature": 1.4849057900627562,
|
| 3 |
+
"before": {
|
| 4 |
+
"count": 5513,
|
| 5 |
+
"accuracy": 0.6716851079267186,
|
| 6 |
+
"brier": 0.45613250785984416,
|
| 7 |
+
"brier_soft": 0.43051378112146826,
|
| 8 |
+
"nll": 0.8780634493065734,
|
| 9 |
+
"nll_soft": 0.8966669647794041,
|
| 10 |
+
"ece": 0.09598293395855888,
|
| 11 |
+
"mean_confidence": 0.7660815099876934,
|
| 12 |
+
"confident_error_rate": 0.0511518229639035,
|
| 13 |
+
"score_mae": 0.597205044838637,
|
| 14 |
+
"evidence_auc": 0.46369209809264306
|
| 15 |
+
},
|
| 16 |
+
"after": {
|
| 17 |
+
"count": 5513,
|
| 18 |
+
"accuracy": 0.6716851079267186,
|
| 19 |
+
"brier": 0.4471471464049214,
|
| 20 |
+
"brier_soft": 0.41946137922607474,
|
| 21 |
+
"nll": 0.8275972595200731,
|
| 22 |
+
"nll_soft": 0.8401256741826928,
|
| 23 |
+
"ece": 0.04235756362376845,
|
| 24 |
+
"mean_confidence": 0.6827506249377135,
|
| 25 |
+
"confident_error_rate": 0.023036459278070017,
|
| 26 |
+
"score_mae": 0.6329812375448026,
|
| 27 |
+
"evidence_auc": 0.46369209809264306
|
| 28 |
+
}
|
| 29 |
+
}
|
config.json
ADDED
|
@@ -0,0 +1,77 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"ModernBertModel"
|
| 4 |
+
],
|
| 5 |
+
"attention_bias": false,
|
| 6 |
+
"attention_dropout": 0.0,
|
| 7 |
+
"bos_token_id": 50281,
|
| 8 |
+
"classifier_activation": "gelu",
|
| 9 |
+
"classifier_bias": false,
|
| 10 |
+
"classifier_dropout": 0.0,
|
| 11 |
+
"classifier_pooling": "mean",
|
| 12 |
+
"cls_token_id": 50281,
|
| 13 |
+
"decoder_bias": true,
|
| 14 |
+
"deterministic_flash_attn": false,
|
| 15 |
+
"dtype": "float32",
|
| 16 |
+
"embedding_dropout": 0.0,
|
| 17 |
+
"eos_token_id": 50282,
|
| 18 |
+
"global_attn_every_n_layers": 3,
|
| 19 |
+
"gradient_checkpointing": false,
|
| 20 |
+
"hidden_activation": "gelu",
|
| 21 |
+
"hidden_size": 768,
|
| 22 |
+
"initializer_cutoff_factor": 2.0,
|
| 23 |
+
"initializer_range": 0.02,
|
| 24 |
+
"intermediate_size": 1152,
|
| 25 |
+
"layer_norm_eps": 1e-05,
|
| 26 |
+
"layer_types": [
|
| 27 |
+
"full_attention",
|
| 28 |
+
"sliding_attention",
|
| 29 |
+
"sliding_attention",
|
| 30 |
+
"full_attention",
|
| 31 |
+
"sliding_attention",
|
| 32 |
+
"sliding_attention",
|
| 33 |
+
"full_attention",
|
| 34 |
+
"sliding_attention",
|
| 35 |
+
"sliding_attention",
|
| 36 |
+
"full_attention",
|
| 37 |
+
"sliding_attention",
|
| 38 |
+
"sliding_attention",
|
| 39 |
+
"full_attention",
|
| 40 |
+
"sliding_attention",
|
| 41 |
+
"sliding_attention",
|
| 42 |
+
"full_attention",
|
| 43 |
+
"sliding_attention",
|
| 44 |
+
"sliding_attention",
|
| 45 |
+
"full_attention",
|
| 46 |
+
"sliding_attention",
|
| 47 |
+
"sliding_attention",
|
| 48 |
+
"full_attention"
|
| 49 |
+
],
|
| 50 |
+
"local_attention": 128,
|
| 51 |
+
"max_position_embeddings": 8192,
|
| 52 |
+
"mlp_bias": false,
|
| 53 |
+
"mlp_dropout": 0.0,
|
| 54 |
+
"model_type": "modernbert",
|
| 55 |
+
"norm_bias": false,
|
| 56 |
+
"norm_eps": 1e-05,
|
| 57 |
+
"num_attention_heads": 12,
|
| 58 |
+
"num_hidden_layers": 22,
|
| 59 |
+
"pad_token_id": 50283,
|
| 60 |
+
"position_embedding_type": "absolute",
|
| 61 |
+
"rope_parameters": {
|
| 62 |
+
"full_attention": {
|
| 63 |
+
"rope_theta": 160000.0,
|
| 64 |
+
"rope_type": "default"
|
| 65 |
+
},
|
| 66 |
+
"sliding_attention": {
|
| 67 |
+
"rope_theta": 10000.0,
|
| 68 |
+
"rope_type": "default"
|
| 69 |
+
}
|
| 70 |
+
},
|
| 71 |
+
"sep_token_id": 50282,
|
| 72 |
+
"sparse_pred_ignore_index": -100,
|
| 73 |
+
"sparse_prediction": false,
|
| 74 |
+
"tie_word_embeddings": true,
|
| 75 |
+
"transformers_version": "5.17.0",
|
| 76 |
+
"vocab_size": 50368
|
| 77 |
+
}
|
eval-dev-scaled.json
ADDED
|
@@ -0,0 +1,771 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"overall": {
|
| 3 |
+
"count": 6113,
|
| 4 |
+
"accuracy": 0.6679208244724358,
|
| 5 |
+
"brier": 0.44247259100885217,
|
| 6 |
+
"brier_soft": 0.4170925201205985,
|
| 7 |
+
"nll": 0.8056102230141763,
|
| 8 |
+
"nll_soft": 0.8172796926231564,
|
| 9 |
+
"ece": 0.03653387851957047,
|
| 10 |
+
"mean_confidence": 0.6869317729442912,
|
| 11 |
+
"confident_error_rate": 0.019466710289546867,
|
| 12 |
+
"score_mae": 0.5729998939493892,
|
| 13 |
+
"evidence_auc": 0.9074727460810058
|
| 14 |
+
},
|
| 15 |
+
"by_type": {
|
| 16 |
+
"bool": {
|
| 17 |
+
"count": 1898,
|
| 18 |
+
"accuracy": 0.7787144362486829,
|
| 19 |
+
"brier": 0.3124533326117464,
|
| 20 |
+
"brier_soft": 0.30893223729451025,
|
| 21 |
+
"nll": 0.47617615577105815,
|
| 22 |
+
"nll_soft": 0.4805717663652272,
|
| 23 |
+
"ece": 0.04409051484755141,
|
| 24 |
+
"mean_confidence": 0.8090805982576155,
|
| 25 |
+
"confident_error_rate": 0.026343519494204427,
|
| 26 |
+
"evidence_auc": 0.9039322111530974
|
| 27 |
+
},
|
| 28 |
+
"choice": {
|
| 29 |
+
"count": 3248,
|
| 30 |
+
"accuracy": 0.625615763546798,
|
| 31 |
+
"brier": 0.5012455732095099,
|
| 32 |
+
"brier_soft": 0.4763148831509033,
|
| 33 |
+
"nll": 0.9690121828222874,
|
| 34 |
+
"nll_soft": 0.9798443434525502,
|
| 35 |
+
"ece": 0.06097491611004007,
|
| 36 |
+
"mean_confidence": 0.6575062932025669,
|
| 37 |
+
"confident_error_rate": 0.021243842364532018,
|
| 38 |
+
"evidence_auc": 0.9208606009429315
|
| 39 |
+
},
|
| 40 |
+
"score": {
|
| 41 |
+
"count": 967,
|
| 42 |
+
"accuracy": 0.5925542916235781,
|
| 43 |
+
"brier": 0.5002615323221618,
|
| 44 |
+
"brier_soft": 0.4304678889742543,
|
| 45 |
+
"nll": 0.9033726782062063,
|
| 46 |
+
"nll_soft": 0.9321314590592249,
|
| 47 |
+
"ece": 0.06700552232571198,
|
| 48 |
+
"mean_confidence": 0.546017075691376,
|
| 49 |
+
"confident_error_rate": 0.0,
|
| 50 |
+
"score_mae": 0.5729998939493892
|
| 51 |
+
}
|
| 52 |
+
},
|
| 53 |
+
"by_source": {
|
| 54 |
+
"ag_news": {
|
| 55 |
+
"count": 100,
|
| 56 |
+
"accuracy": 0.88,
|
| 57 |
+
"brier": 0.17829138819549176,
|
| 58 |
+
"brier_soft": 0.17829138819549176,
|
| 59 |
+
"nll": 0.30741759767501897,
|
| 60 |
+
"nll_soft": 0.30741759767501897,
|
| 61 |
+
"ece": 0.09543892771005633,
|
| 62 |
+
"mean_confidence": 0.9135349568724632,
|
| 63 |
+
"confident_error_rate": 0.02
|
| 64 |
+
},
|
| 65 |
+
"amazon_polarity": {
|
| 66 |
+
"count": 100,
|
| 67 |
+
"accuracy": 0.97,
|
| 68 |
+
"brier": 0.06162859960932998,
|
| 69 |
+
"brier_soft": 0.06162859960932998,
|
| 70 |
+
"nll": 0.13110608215243919,
|
| 71 |
+
"nll_soft": 0.13110608215243919,
|
| 72 |
+
"ece": 0.03008812248706818,
|
| 73 |
+
"mean_confidence": 0.9551744455099106,
|
| 74 |
+
"confident_error_rate": 0.03
|
| 75 |
+
},
|
| 76 |
+
"amazon_stars": {
|
| 77 |
+
"count": 100,
|
| 78 |
+
"accuracy": 0.53,
|
| 79 |
+
"brier": 0.5461979259182673,
|
| 80 |
+
"brier_soft": 0.5461979259182673,
|
| 81 |
+
"nll": 0.9164086895783538,
|
| 82 |
+
"nll_soft": 0.9164086895783538,
|
| 83 |
+
"ece": 0.11809196531772617,
|
| 84 |
+
"mean_confidence": 0.5753743493556976,
|
| 85 |
+
"confident_error_rate": 0.0,
|
| 86 |
+
"score_mae": 0.5362134778685868
|
| 87 |
+
},
|
| 88 |
+
"anli": {
|
| 89 |
+
"count": 100,
|
| 90 |
+
"accuracy": 0.39,
|
| 91 |
+
"brier": 0.9722760942391393,
|
| 92 |
+
"brier_soft": 0.9722760942391393,
|
| 93 |
+
"nll": 1.8542452552224933,
|
| 94 |
+
"nll_soft": 1.8542452552224933,
|
| 95 |
+
"ece": 0.44132104188203813,
|
| 96 |
+
"mean_confidence": 0.7956731787323952,
|
| 97 |
+
"confident_error_rate": 0.23
|
| 98 |
+
},
|
| 99 |
+
"arc_challenge": {
|
| 100 |
+
"count": 100,
|
| 101 |
+
"accuracy": 0.36,
|
| 102 |
+
"brier": 0.778481672987934,
|
| 103 |
+
"brier_soft": 0.778481672987934,
|
| 104 |
+
"nll": 1.4337150394138498,
|
| 105 |
+
"nll_soft": 1.4337150394138498,
|
| 106 |
+
"ece": 0.20447848409414293,
|
| 107 |
+
"mean_confidence": 0.5459534320235252,
|
| 108 |
+
"confident_error_rate": 0.0
|
| 109 |
+
},
|
| 110 |
+
"arc_easy": {
|
| 111 |
+
"count": 100,
|
| 112 |
+
"accuracy": 0.45,
|
| 113 |
+
"brier": 0.6407232552939703,
|
| 114 |
+
"brier_soft": 0.6407232552939703,
|
| 115 |
+
"nll": 1.2136572969992905,
|
| 116 |
+
"nll_soft": 1.2136572969992905,
|
| 117 |
+
"ece": 0.12803085684776305,
|
| 118 |
+
"mean_confidence": 0.5452363789081573,
|
| 119 |
+
"confident_error_rate": 0.01
|
| 120 |
+
},
|
| 121 |
+
"banking77": {
|
| 122 |
+
"count": 100,
|
| 123 |
+
"accuracy": 0.81,
|
| 124 |
+
"brier": 0.3814467708771144,
|
| 125 |
+
"brier_soft": 0.3814467708771144,
|
| 126 |
+
"nll": 1.0443211262890961,
|
| 127 |
+
"nll_soft": 1.0443211262890961,
|
| 128 |
+
"ece": 0.3126603952050208,
|
| 129 |
+
"mean_confidence": 0.4973396047949791,
|
| 130 |
+
"confident_error_rate": 0.0
|
| 131 |
+
},
|
| 132 |
+
"boolq": {
|
| 133 |
+
"count": 125,
|
| 134 |
+
"accuracy": 0.568,
|
| 135 |
+
"brier": 0.6168742406476001,
|
| 136 |
+
"brier_soft": 0.6168742406476001,
|
| 137 |
+
"nll": 0.8592622690829735,
|
| 138 |
+
"nll_soft": 0.8592622690829735,
|
| 139 |
+
"ece": 0.2938120722770691,
|
| 140 |
+
"mean_confidence": 0.8219961438179016,
|
| 141 |
+
"confident_error_rate": 0.032,
|
| 142 |
+
"evidence_auc": 0.9706666666666667
|
| 143 |
+
},
|
| 144 |
+
"cb": {
|
| 145 |
+
"count": 28,
|
| 146 |
+
"accuracy": 0.9285714285714286,
|
| 147 |
+
"brier": 0.13537618609721863,
|
| 148 |
+
"brier_soft": 0.13537618609721863,
|
| 149 |
+
"nll": 0.2816298666367622,
|
| 150 |
+
"nll_soft": 0.2816298666367622,
|
| 151 |
+
"ece": 0.11475402755396709,
|
| 152 |
+
"mean_confidence": 0.9356890469789505,
|
| 153 |
+
"confident_error_rate": 0.03571428571428571
|
| 154 |
+
},
|
| 155 |
+
"civil_comments_severity": {
|
| 156 |
+
"count": 100,
|
| 157 |
+
"accuracy": 0.81,
|
| 158 |
+
"brier": 0.28384330937032604,
|
| 159 |
+
"brier_soft": 0.17998165817214293,
|
| 160 |
+
"nll": 0.5338759677740593,
|
| 161 |
+
"nll_soft": 0.6298814779743555,
|
| 162 |
+
"ece": 0.14874677449464796,
|
| 163 |
+
"mean_confidence": 0.6996492806077004,
|
| 164 |
+
"confident_error_rate": 0.0,
|
| 165 |
+
"score_mae": 0.39768598369671965
|
| 166 |
+
},
|
| 167 |
+
"civil_comments_toxic": {
|
| 168 |
+
"count": 100,
|
| 169 |
+
"accuracy": 0.94,
|
| 170 |
+
"brier": 0.09920447825135499,
|
| 171 |
+
"brier_soft": 0.03237408913021391,
|
| 172 |
+
"nll": 0.19394092520219666,
|
| 173 |
+
"nll_soft": 0.2773696142795253,
|
| 174 |
+
"ece": 0.08483366131782531,
|
| 175 |
+
"mean_confidence": 0.8621529483795166,
|
| 176 |
+
"confident_error_rate": 0.0
|
| 177 |
+
},
|
| 178 |
+
"clinc150": {
|
| 179 |
+
"count": 100,
|
| 180 |
+
"accuracy": 0.8,
|
| 181 |
+
"brier": 0.4848727215372612,
|
| 182 |
+
"brier_soft": 0.4848727215372612,
|
| 183 |
+
"nll": 1.4845052623480186,
|
| 184 |
+
"nll_soft": 1.4845052623480186,
|
| 185 |
+
"ece": 0.4221790521405637,
|
| 186 |
+
"mean_confidence": 0.37782094785943626,
|
| 187 |
+
"confident_error_rate": 0.0
|
| 188 |
+
},
|
| 189 |
+
"cola": {
|
| 190 |
+
"count": 100,
|
| 191 |
+
"accuracy": 0.58,
|
| 192 |
+
"brier": 0.4671416305229769,
|
| 193 |
+
"brier_soft": 0.4671416305229769,
|
| 194 |
+
"nll": 0.6581302967113183,
|
| 195 |
+
"nll_soft": 0.6581302967113183,
|
| 196 |
+
"ece": 0.10910327613353732,
|
| 197 |
+
"mean_confidence": 0.6739702433347702,
|
| 198 |
+
"confident_error_rate": 0.0
|
| 199 |
+
},
|
| 200 |
+
"commonsense_qa": {
|
| 201 |
+
"count": 100,
|
| 202 |
+
"accuracy": 0.42,
|
| 203 |
+
"brier": 0.7026358240551468,
|
| 204 |
+
"brier_soft": 0.7026358240551468,
|
| 205 |
+
"nll": 1.3662261855444524,
|
| 206 |
+
"nll_soft": 1.3662261855444524,
|
| 207 |
+
"ece": 0.13439485043287275,
|
| 208 |
+
"mean_confidence": 0.510890220105648,
|
| 209 |
+
"confident_error_rate": 0.0
|
| 210 |
+
},
|
| 211 |
+
"cosmos_qa": {
|
| 212 |
+
"count": 120,
|
| 213 |
+
"accuracy": 0.4,
|
| 214 |
+
"brier": 0.674820119653572,
|
| 215 |
+
"brier_soft": 0.674820119653572,
|
| 216 |
+
"nll": 1.2437770795447451,
|
| 217 |
+
"nll_soft": 1.2437770795447451,
|
| 218 |
+
"ece": 0.14261294777194658,
|
| 219 |
+
"mean_confidence": 0.5219608165323735,
|
| 220 |
+
"confident_error_rate": 0.0,
|
| 221 |
+
"evidence_auc": 0.836875
|
| 222 |
+
},
|
| 223 |
+
"dates": {
|
| 224 |
+
"count": 300,
|
| 225 |
+
"accuracy": 0.7633333333333333,
|
| 226 |
+
"brier": 0.2819022563327028,
|
| 227 |
+
"brier_soft": 0.2819022563327028,
|
| 228 |
+
"nll": 0.4583224548625654,
|
| 229 |
+
"nll_soft": 0.4583224548625654,
|
| 230 |
+
"ece": 0.06106376767158511,
|
| 231 |
+
"mean_confidence": 0.7644337038199107,
|
| 232 |
+
"confident_error_rate": 0.0033333333333333335,
|
| 233 |
+
"score_mae": 0.45998074431250346
|
| 234 |
+
},
|
| 235 |
+
"dbpedia": {
|
| 236 |
+
"count": 100,
|
| 237 |
+
"accuracy": 0.99,
|
| 238 |
+
"brier": 0.015198128030094994,
|
| 239 |
+
"brier_soft": 0.015198128030094994,
|
| 240 |
+
"nll": 0.06724704860844338,
|
| 241 |
+
"nll_soft": 0.06724704860844338,
|
| 242 |
+
"ece": 0.057107390761375415,
|
| 243 |
+
"mean_confidence": 0.9441341024637222,
|
| 244 |
+
"confident_error_rate": 0.0
|
| 245 |
+
},
|
| 246 |
+
"dynasent": {
|
| 247 |
+
"count": 100,
|
| 248 |
+
"accuracy": 0.53,
|
| 249 |
+
"brier": 0.5435009917424021,
|
| 250 |
+
"brier_soft": 0.40578378452816594,
|
| 251 |
+
"nll": 0.9203439216635296,
|
| 252 |
+
"nll_soft": 0.95189629595177,
|
| 253 |
+
"ece": 0.12150101184844973,
|
| 254 |
+
"mean_confidence": 0.5809222316741943,
|
| 255 |
+
"confident_error_rate": 0.0,
|
| 256 |
+
"score_mae": 0.590812379643321
|
| 257 |
+
},
|
| 258 |
+
"enron_spam": {
|
| 259 |
+
"count": 100,
|
| 260 |
+
"accuracy": 0.94,
|
| 261 |
+
"brier": 0.08228255110990408,
|
| 262 |
+
"brier_soft": 0.08228255110990408,
|
| 263 |
+
"nll": 0.15493662205431416,
|
| 264 |
+
"nll_soft": 0.15493662205431416,
|
| 265 |
+
"ece": 0.043621286749839804,
|
| 266 |
+
"mean_confidence": 0.9467802709341049,
|
| 267 |
+
"confident_error_rate": 0.02
|
| 268 |
+
},
|
| 269 |
+
"go_emotions": {
|
| 270 |
+
"count": 100,
|
| 271 |
+
"accuracy": 0.54,
|
| 272 |
+
"brier": 0.6768268805616047,
|
| 273 |
+
"brier_soft": 0.25233950625259904,
|
| 274 |
+
"nll": 1.7380711253413985,
|
| 275 |
+
"nll_soft": 2.0785070377760406,
|
| 276 |
+
"ece": 0.24022679880261422,
|
| 277 |
+
"mean_confidence": 0.29977320119738576,
|
| 278 |
+
"confident_error_rate": 0.0
|
| 279 |
+
},
|
| 280 |
+
"hellaswag": {
|
| 281 |
+
"count": 100,
|
| 282 |
+
"accuracy": 0.49,
|
| 283 |
+
"brier": 0.6818457284598648,
|
| 284 |
+
"brier_soft": 0.6818457284598648,
|
| 285 |
+
"nll": 1.27200660086301,
|
| 286 |
+
"nll_soft": 1.27200660086301,
|
| 287 |
+
"ece": 0.1305046308040619,
|
| 288 |
+
"mean_confidence": 0.5641709685325622,
|
| 289 |
+
"confident_error_rate": 0.0
|
| 290 |
+
},
|
| 291 |
+
"imdb": {
|
| 292 |
+
"count": 100,
|
| 293 |
+
"accuracy": 0.95,
|
| 294 |
+
"brier": 0.08363715307563817,
|
| 295 |
+
"brier_soft": 0.08363715307563817,
|
| 296 |
+
"nll": 0.16551026463952032,
|
| 297 |
+
"nll_soft": 0.16551026463952032,
|
| 298 |
+
"ece": 0.026311837434768674,
|
| 299 |
+
"mean_confidence": 0.9574666857719422,
|
| 300 |
+
"confident_error_rate": 0.03
|
| 301 |
+
},
|
| 302 |
+
"ledgar": {
|
| 303 |
+
"count": 100,
|
| 304 |
+
"accuracy": 0.75,
|
| 305 |
+
"brier": 0.42560135728608983,
|
| 306 |
+
"brier_soft": 0.42560135728608983,
|
| 307 |
+
"nll": 1.3302157486233597,
|
| 308 |
+
"nll_soft": 1.3302157486233597,
|
| 309 |
+
"ece": 0.23709699064493178,
|
| 310 |
+
"mean_confidence": 0.5129030093550682,
|
| 311 |
+
"confident_error_rate": 0.0
|
| 312 |
+
},
|
| 313 |
+
"liar": {
|
| 314 |
+
"count": 100,
|
| 315 |
+
"accuracy": 0.32,
|
| 316 |
+
"brier": 0.7427052798517789,
|
| 317 |
+
"brier_soft": 0.7427052798517789,
|
| 318 |
+
"nll": 1.4964320047893989,
|
| 319 |
+
"nll_soft": 1.4964320047893989,
|
| 320 |
+
"ece": 0.055253805071115485,
|
| 321 |
+
"mean_confidence": 0.3017434249818325,
|
| 322 |
+
"confident_error_rate": 0.0,
|
| 323 |
+
"score_mae": 0.9733948946371674
|
| 324 |
+
},
|
| 325 |
+
"measuring_hate_speech": {
|
| 326 |
+
"count": 100,
|
| 327 |
+
"accuracy": 0.83,
|
| 328 |
+
"brier": 0.3177919283014086,
|
| 329 |
+
"brier_soft": 0.16750927245698513,
|
| 330 |
+
"nll": 0.5942829577379074,
|
| 331 |
+
"nll_soft": 0.7062310284569752,
|
| 332 |
+
"ece": 0.21628696441650394,
|
| 333 |
+
"mean_confidence": 0.6214796364307403,
|
| 334 |
+
"confident_error_rate": 0.0,
|
| 335 |
+
"score_mae": 0.6045454696938396
|
| 336 |
+
},
|
| 337 |
+
"medqa": {
|
| 338 |
+
"count": 100,
|
| 339 |
+
"accuracy": 0.34,
|
| 340 |
+
"brier": 0.7671574185751032,
|
| 341 |
+
"brier_soft": 0.7671574185751032,
|
| 342 |
+
"nll": 1.4356347250931116,
|
| 343 |
+
"nll_soft": 1.4356347250931116,
|
| 344 |
+
"ece": 0.17744469463825224,
|
| 345 |
+
"mean_confidence": 0.48414860367774964,
|
| 346 |
+
"confident_error_rate": 0.0
|
| 347 |
+
},
|
| 348 |
+
"mnli": {
|
| 349 |
+
"count": 100,
|
| 350 |
+
"accuracy": 0.78,
|
| 351 |
+
"brier": 0.3263025000138572,
|
| 352 |
+
"brier_soft": 0.3263025000138572,
|
| 353 |
+
"nll": 0.5982821287502831,
|
| 354 |
+
"nll_soft": 0.5982821287502831,
|
| 355 |
+
"ece": 0.10382685691118237,
|
| 356 |
+
"mean_confidence": 0.8509401044249535,
|
| 357 |
+
"confident_error_rate": 0.06
|
| 358 |
+
},
|
| 359 |
+
"mrpc": {
|
| 360 |
+
"count": 100,
|
| 361 |
+
"accuracy": 0.81,
|
| 362 |
+
"brier": 0.27720433744929446,
|
| 363 |
+
"brier_soft": 0.27720433744929446,
|
| 364 |
+
"nll": 0.43150783049960595,
|
| 365 |
+
"nll_soft": 0.43150783049960595,
|
| 366 |
+
"ece": 0.1067017036676407,
|
| 367 |
+
"mean_confidence": 0.856079712510109,
|
| 368 |
+
"confident_error_rate": 0.03
|
| 369 |
+
},
|
| 370 |
+
"multirc": {
|
| 371 |
+
"count": 125,
|
| 372 |
+
"accuracy": 0.664,
|
| 373 |
+
"brier": 0.5262665529453465,
|
| 374 |
+
"brier_soft": 0.5262665529453465,
|
| 375 |
+
"nll": 0.7910576977190109,
|
| 376 |
+
"nll_soft": 0.7910576977190109,
|
| 377 |
+
"ece": 0.20780441761016843,
|
| 378 |
+
"mean_confidence": 0.833528317451477,
|
| 379 |
+
"confident_error_rate": 0.136,
|
| 380 |
+
"evidence_auc": 0.6994666666666667
|
| 381 |
+
},
|
| 382 |
+
"openbookqa": {
|
| 383 |
+
"count": 100,
|
| 384 |
+
"accuracy": 0.52,
|
| 385 |
+
"brier": 0.6059996266277589,
|
| 386 |
+
"brier_soft": 0.6059996266277589,
|
| 387 |
+
"nll": 1.1493342555724386,
|
| 388 |
+
"nll_soft": 1.1493342555724386,
|
| 389 |
+
"ece": 0.08220899760723116,
|
| 390 |
+
"mean_confidence": 0.5423529654741287,
|
| 391 |
+
"confident_error_rate": 0.01
|
| 392 |
+
},
|
| 393 |
+
"piqa": {
|
| 394 |
+
"count": 100,
|
| 395 |
+
"accuracy": 0.61,
|
| 396 |
+
"brier": 0.5129767773118842,
|
| 397 |
+
"brier_soft": 0.5129767773118842,
|
| 398 |
+
"nll": 0.7296690239966398,
|
| 399 |
+
"nll_soft": 0.7296690239966398,
|
| 400 |
+
"ece": 0.14158266067504877,
|
| 401 |
+
"mean_confidence": 0.6558456552028656,
|
| 402 |
+
"confident_error_rate": 0.02
|
| 403 |
+
},
|
| 404 |
+
"policy_application": {
|
| 405 |
+
"count": 150,
|
| 406 |
+
"accuracy": 0.7333333333333333,
|
| 407 |
+
"brier": 0.3508971131550242,
|
| 408 |
+
"brier_soft": 0.3508971131550242,
|
| 409 |
+
"nll": 0.5516997947737176,
|
| 410 |
+
"nll_soft": 0.5516997947737176,
|
| 411 |
+
"ece": 0.044622519016265856,
|
| 412 |
+
"mean_confidence": 0.7177301915486654,
|
| 413 |
+
"confident_error_rate": 0.0,
|
| 414 |
+
"evidence_auc": 0.5696533682145193
|
| 415 |
+
},
|
| 416 |
+
"policy_hard": {
|
| 417 |
+
"count": 300,
|
| 418 |
+
"accuracy": 0.6966666666666667,
|
| 419 |
+
"brier": 0.38472149078986073,
|
| 420 |
+
"brier_soft": 0.38472149078986073,
|
| 421 |
+
"nll": 0.6128855879532513,
|
| 422 |
+
"nll_soft": 0.6128855879532513,
|
| 423 |
+
"ece": 0.053346475064754505,
|
| 424 |
+
"mean_confidence": 0.7120696005225181,
|
| 425 |
+
"confident_error_rate": 0.0,
|
| 426 |
+
"evidence_auc": 0.7445023148148148
|
| 427 |
+
},
|
| 428 |
+
"qqp": {
|
| 429 |
+
"count": 100,
|
| 430 |
+
"accuracy": 0.8,
|
| 431 |
+
"brier": 0.3034980053199748,
|
| 432 |
+
"brier_soft": 0.3034980053199748,
|
| 433 |
+
"nll": 0.4749638870383975,
|
| 434 |
+
"nll_soft": 0.4749638870383975,
|
| 435 |
+
"ece": 0.10882365942001343,
|
| 436 |
+
"mean_confidence": 0.8906454944610596,
|
| 437 |
+
"confident_error_rate": 0.07
|
| 438 |
+
},
|
| 439 |
+
"race": {
|
| 440 |
+
"count": 125,
|
| 441 |
+
"accuracy": 0.464,
|
| 442 |
+
"brier": 0.670541267736231,
|
| 443 |
+
"brier_soft": 0.670541267736231,
|
| 444 |
+
"nll": 1.2785957499648075,
|
| 445 |
+
"nll_soft": 1.2785957499648075,
|
| 446 |
+
"ece": 0.09217529821395876,
|
| 447 |
+
"mean_confidence": 0.4943129940032959,
|
| 448 |
+
"confident_error_rate": 0.0,
|
| 449 |
+
"evidence_auc": 0.7856
|
| 450 |
+
},
|
| 451 |
+
"rotten_tomatoes": {
|
| 452 |
+
"count": 100,
|
| 453 |
+
"accuracy": 0.89,
|
| 454 |
+
"brier": 0.26950262705978917,
|
| 455 |
+
"brier_soft": 0.26950262705978917,
|
| 456 |
+
"nll": 0.43769943345994206,
|
| 457 |
+
"nll_soft": 0.43769943345994206,
|
| 458 |
+
"ece": 0.18772910118103028,
|
| 459 |
+
"mean_confidence": 0.7022708988189698,
|
| 460 |
+
"confident_error_rate": 0.0
|
| 461 |
+
},
|
| 462 |
+
"rte": {
|
| 463 |
+
"count": 100,
|
| 464 |
+
"accuracy": 0.7,
|
| 465 |
+
"brier": 0.4417545164022784,
|
| 466 |
+
"brier_soft": 0.4417545164022784,
|
| 467 |
+
"nll": 0.6803569597273692,
|
| 468 |
+
"nll_soft": 0.6803569597273692,
|
| 469 |
+
"ece": 0.166305216550827,
|
| 470 |
+
"mean_confidence": 0.8475300312042237,
|
| 471 |
+
"confident_error_rate": 0.07
|
| 472 |
+
},
|
| 473 |
+
"scotus": {
|
| 474 |
+
"count": 100,
|
| 475 |
+
"accuracy": 0.63,
|
| 476 |
+
"brier": 0.57591555494716,
|
| 477 |
+
"brier_soft": 0.57591555494716,
|
| 478 |
+
"nll": 1.4268808895864786,
|
| 479 |
+
"nll_soft": 1.4268808895864786,
|
| 480 |
+
"ece": 0.1622882364690304,
|
| 481 |
+
"mean_confidence": 0.479786828905344,
|
| 482 |
+
"confident_error_rate": 0.0
|
| 483 |
+
},
|
| 484 |
+
"shp": {
|
| 485 |
+
"count": 100,
|
| 486 |
+
"accuracy": 0.5,
|
| 487 |
+
"brier": 0.553928533015661,
|
| 488 |
+
"brier_soft": 0.16866709422112394,
|
| 489 |
+
"nll": 0.7553610522259013,
|
| 490 |
+
"nll_soft": 0.7667537170621911,
|
| 491 |
+
"ece": 0.22228121280670163,
|
| 492 |
+
"mean_confidence": 0.6585486650466919,
|
| 493 |
+
"confident_error_rate": 0.0
|
| 494 |
+
},
|
| 495 |
+
"sms_spam": {
|
| 496 |
+
"count": 100,
|
| 497 |
+
"accuracy": 0.96,
|
| 498 |
+
"brier": 0.14873415443052604,
|
| 499 |
+
"brier_soft": 0.14873415443052604,
|
| 500 |
+
"nll": 0.29604541382027116,
|
| 501 |
+
"nll_soft": 0.29604541382027116,
|
| 502 |
+
"ece": 0.18373807787895202,
|
| 503 |
+
"mean_confidence": 0.7762619221210479,
|
| 504 |
+
"confident_error_rate": 0.0
|
| 505 |
+
},
|
| 506 |
+
"snli": {
|
| 507 |
+
"count": 100,
|
| 508 |
+
"accuracy": 0.79,
|
| 509 |
+
"brier": 0.3657109430484185,
|
| 510 |
+
"brier_soft": 0.3657109430484185,
|
| 511 |
+
"nll": 0.6824574908252256,
|
| 512 |
+
"nll_soft": 0.6824574908252256,
|
| 513 |
+
"ece": 0.1515702518820763,
|
| 514 |
+
"mean_confidence": 0.8926358130574227,
|
| 515 |
+
"confident_error_rate": 0.1
|
| 516 |
+
},
|
| 517 |
+
"social_iqa": {
|
| 518 |
+
"count": 100,
|
| 519 |
+
"accuracy": 0.5,
|
| 520 |
+
"brier": 0.6701157287052262,
|
| 521 |
+
"brier_soft": 0.6701157287052262,
|
| 522 |
+
"nll": 1.2110427461546294,
|
| 523 |
+
"nll_soft": 1.2110427461546294,
|
| 524 |
+
"ece": 0.21912074834108355,
|
| 525 |
+
"mean_confidence": 0.68725660353899,
|
| 526 |
+
"confident_error_rate": 0.04
|
| 527 |
+
},
|
| 528 |
+
"sst2": {
|
| 529 |
+
"count": 100,
|
| 530 |
+
"accuracy": 0.9,
|
| 531 |
+
"brier": 0.25762914103330403,
|
| 532 |
+
"brier_soft": 0.25762914103330403,
|
| 533 |
+
"nll": 0.42289569521733256,
|
| 534 |
+
"nll_soft": 0.42289569521733256,
|
| 535 |
+
"ece": 0.182864231467247,
|
| 536 |
+
"mean_confidence": 0.717135768532753,
|
| 537 |
+
"confident_error_rate": 0.0
|
| 538 |
+
},
|
| 539 |
+
"sst5": {
|
| 540 |
+
"count": 100,
|
| 541 |
+
"accuracy": 0.56,
|
| 542 |
+
"brier": 0.5750112386200766,
|
| 543 |
+
"brier_soft": 0.5750112386200766,
|
| 544 |
+
"nll": 1.0410791576793528,
|
| 545 |
+
"nll_soft": 1.0410791576793528,
|
| 546 |
+
"ece": 0.13715695679187778,
|
| 547 |
+
"mean_confidence": 0.4760879367589951,
|
| 548 |
+
"confident_error_rate": 0.0,
|
| 549 |
+
"score_mae": 0.569280381789431
|
| 550 |
+
},
|
| 551 |
+
"stance_abortion": {
|
| 552 |
+
"count": 60,
|
| 553 |
+
"accuracy": 0.6333333333333333,
|
| 554 |
+
"brier": 0.4775618932889926,
|
| 555 |
+
"brier_soft": 0.4775618932889926,
|
| 556 |
+
"nll": 0.772960340218129,
|
| 557 |
+
"nll_soft": 0.772960340218129,
|
| 558 |
+
"ece": 0.14922663221756619,
|
| 559 |
+
"mean_confidence": 0.7609471683700879,
|
| 560 |
+
"confident_error_rate": 0.05
|
| 561 |
+
},
|
| 562 |
+
"stance_atheism": {
|
| 563 |
+
"count": 60,
|
| 564 |
+
"accuracy": 0.7,
|
| 565 |
+
"brier": 0.39241280131421197,
|
| 566 |
+
"brier_soft": 0.39241280131421197,
|
| 567 |
+
"nll": 0.6474431235930713,
|
| 568 |
+
"nll_soft": 0.6474431235930713,
|
| 569 |
+
"ece": 0.15528117964665092,
|
| 570 |
+
"mean_confidence": 0.8136728132764498,
|
| 571 |
+
"confident_error_rate": 0.016666666666666666
|
| 572 |
+
},
|
| 573 |
+
"stance_feminist": {
|
| 574 |
+
"count": 60,
|
| 575 |
+
"accuracy": 0.6833333333333333,
|
| 576 |
+
"brier": 0.46696109115797474,
|
| 577 |
+
"brier_soft": 0.46696109115797474,
|
| 578 |
+
"nll": 0.7586194826479168,
|
| 579 |
+
"nll_soft": 0.7586194826479168,
|
| 580 |
+
"ece": 0.17778196881214778,
|
| 581 |
+
"mean_confidence": 0.7421373461683591,
|
| 582 |
+
"confident_error_rate": 0.05
|
| 583 |
+
},
|
| 584 |
+
"stance_hillary": {
|
| 585 |
+
"count": 60,
|
| 586 |
+
"accuracy": 0.6666666666666666,
|
| 587 |
+
"brier": 0.47286586681141635,
|
| 588 |
+
"brier_soft": 0.47286586681141635,
|
| 589 |
+
"nll": 0.8005868818543999,
|
| 590 |
+
"nll_soft": 0.8005868818543999,
|
| 591 |
+
"ece": 0.15976376285155613,
|
| 592 |
+
"mean_confidence": 0.7363345429301262,
|
| 593 |
+
"confident_error_rate": 0.03333333333333333
|
| 594 |
+
},
|
| 595 |
+
"stsb": {
|
| 596 |
+
"count": 100,
|
| 597 |
+
"accuracy": 0.45,
|
| 598 |
+
"brier": 0.6576776158521066,
|
| 599 |
+
"brier_soft": 0.3746345989346838,
|
| 600 |
+
"nll": 1.2299245867552293,
|
| 601 |
+
"nll_soft": 1.2685160423963167,
|
| 602 |
+
"ece": 0.10637196481227874,
|
| 603 |
+
"mean_confidence": 0.523349083662033,
|
| 604 |
+
"confident_error_rate": 0.0,
|
| 605 |
+
"score_mae": 0.7103589298564476
|
| 606 |
+
},
|
| 607 |
+
"subjectivity": {
|
| 608 |
+
"count": 100,
|
| 609 |
+
"accuracy": 0.94,
|
| 610 |
+
"brier": 0.07403801155263033,
|
| 611 |
+
"brier_soft": 0.07403801155263033,
|
| 612 |
+
"nll": 0.12967172016931594,
|
| 613 |
+
"nll_soft": 0.12967172016931594,
|
| 614 |
+
"ece": 0.06520502805709842,
|
| 615 |
+
"mean_confidence": 0.9230094337463379,
|
| 616 |
+
"confident_error_rate": 0.0
|
| 617 |
+
},
|
| 618 |
+
"swag": {
|
| 619 |
+
"count": 100,
|
| 620 |
+
"accuracy": 0.4,
|
| 621 |
+
"brier": 0.7323198866636677,
|
| 622 |
+
"brier_soft": 0.7323198866636677,
|
| 623 |
+
"nll": 1.3764612983642295,
|
| 624 |
+
"nll_soft": 1.3764612983642295,
|
| 625 |
+
"ece": 0.2521471258997917,
|
| 626 |
+
"mean_confidence": 0.6150306752324104,
|
| 627 |
+
"confident_error_rate": 0.02
|
| 628 |
+
},
|
| 629 |
+
"trec": {
|
| 630 |
+
"count": 100,
|
| 631 |
+
"accuracy": 0.96,
|
| 632 |
+
"brier": 0.06216060648768522,
|
| 633 |
+
"brier_soft": 0.06216060648768522,
|
| 634 |
+
"nll": 0.14523731763009665,
|
| 635 |
+
"nll_soft": 0.14523731763009665,
|
| 636 |
+
"ece": 0.07524359405040741,
|
| 637 |
+
"mean_confidence": 0.9198704010248184,
|
| 638 |
+
"confident_error_rate": 0.0
|
| 639 |
+
},
|
| 640 |
+
"tweet_hate": {
|
| 641 |
+
"count": 100,
|
| 642 |
+
"accuracy": 0.54,
|
| 643 |
+
"brier": 0.4864483510328587,
|
| 644 |
+
"brier_soft": 0.4864483510328587,
|
| 645 |
+
"nll": 0.6796653519155663,
|
| 646 |
+
"nll_soft": 0.6796653519155663,
|
| 647 |
+
"ece": 0.08831145048141475,
|
| 648 |
+
"mean_confidence": 0.6011694383621216,
|
| 649 |
+
"confident_error_rate": 0.0
|
| 650 |
+
},
|
| 651 |
+
"tweet_sentiment": {
|
| 652 |
+
"count": 100,
|
| 653 |
+
"accuracy": 0.67,
|
| 654 |
+
"brier": 0.4380984174109661,
|
| 655 |
+
"brier_soft": 0.4380984174109661,
|
| 656 |
+
"nll": 0.729035173618918,
|
| 657 |
+
"nll_soft": 0.729035173618918,
|
| 658 |
+
"ece": 0.13779532372951508,
|
| 659 |
+
"mean_confidence": 0.5552096617221832,
|
| 660 |
+
"confident_error_rate": 0.0,
|
| 661 |
+
"score_mae": 0.4073845294490457
|
| 662 |
+
},
|
| 663 |
+
"vitaminc": {
|
| 664 |
+
"count": 100,
|
| 665 |
+
"accuracy": 0.55,
|
| 666 |
+
"brier": 0.6059178299160775,
|
| 667 |
+
"brier_soft": 0.6059178299160775,
|
| 668 |
+
"nll": 1.0501780738072244,
|
| 669 |
+
"nll_soft": 1.0501780738072244,
|
| 670 |
+
"ece": 0.20950709015131,
|
| 671 |
+
"mean_confidence": 0.75950709015131,
|
| 672 |
+
"confident_error_rate": 0.07
|
| 673 |
+
},
|
| 674 |
+
"wic": {
|
| 675 |
+
"count": 100,
|
| 676 |
+
"accuracy": 0.61,
|
| 677 |
+
"brier": 0.502306829144201,
|
| 678 |
+
"brier_soft": 0.502306829144201,
|
| 679 |
+
"nll": 0.7209994732293087,
|
| 680 |
+
"nll_soft": 0.7209994732293087,
|
| 681 |
+
"ece": 0.16384762406349185,
|
| 682 |
+
"mean_confidence": 0.7613926827907562,
|
| 683 |
+
"confident_error_rate": 0.04
|
| 684 |
+
},
|
| 685 |
+
"winogrande": {
|
| 686 |
+
"count": 100,
|
| 687 |
+
"accuracy": 0.57,
|
| 688 |
+
"brier": 0.5063494004660651,
|
| 689 |
+
"brier_soft": 0.5063494004660651,
|
| 690 |
+
"nll": 0.7075272167886625,
|
| 691 |
+
"nll_soft": 0.7075272167886625,
|
| 692 |
+
"ece": 0.09909610986709594,
|
| 693 |
+
"mean_confidence": 0.6518526268005371,
|
| 694 |
+
"confident_error_rate": 0.0
|
| 695 |
+
},
|
| 696 |
+
"yelp": {
|
| 697 |
+
"count": 100,
|
| 698 |
+
"accuracy": 0.62,
|
| 699 |
+
"brier": 0.43856781301069964,
|
| 700 |
+
"brier_soft": 0.43856781301069964,
|
| 701 |
+
"nll": 0.7669727900329802,
|
| 702 |
+
"nll_soft": 0.7669727900329802,
|
| 703 |
+
"ece": 0.15332692295312883,
|
| 704 |
+
"mean_confidence": 0.5684746101498603,
|
| 705 |
+
"confident_error_rate": 0.0,
|
| 706 |
+
"score_mae": 0.44304582916665824
|
| 707 |
+
}
|
| 708 |
+
},
|
| 709 |
+
"reliability": [
|
| 710 |
+
{
|
| 711 |
+
"bin": "0.0-0.1",
|
| 712 |
+
"count": 17,
|
| 713 |
+
"confidence": 0.06687228787032996,
|
| 714 |
+
"accuracy": 0.35294117647058826
|
| 715 |
+
},
|
| 716 |
+
{
|
| 717 |
+
"bin": "0.1-0.2",
|
| 718 |
+
"count": 42,
|
| 719 |
+
"confidence": 0.16727064549922943,
|
| 720 |
+
"accuracy": 0.2619047619047619
|
| 721 |
+
},
|
| 722 |
+
{
|
| 723 |
+
"bin": "0.2-0.3",
|
| 724 |
+
"count": 183,
|
| 725 |
+
"confidence": 0.25834722043386577,
|
| 726 |
+
"accuracy": 0.3770491803278688
|
| 727 |
+
},
|
| 728 |
+
{
|
| 729 |
+
"bin": "0.3-0.4",
|
| 730 |
+
"count": 348,
|
| 731 |
+
"confidence": 0.3574332141841965,
|
| 732 |
+
"accuracy": 0.3994252873563218
|
| 733 |
+
},
|
| 734 |
+
{
|
| 735 |
+
"bin": "0.4-0.5",
|
| 736 |
+
"count": 677,
|
| 737 |
+
"confidence": 0.4517646449971586,
|
| 738 |
+
"accuracy": 0.4564254062038405
|
| 739 |
+
},
|
| 740 |
+
{
|
| 741 |
+
"bin": "0.5-0.6",
|
| 742 |
+
"count": 1026,
|
| 743 |
+
"confidence": 0.5484959817769235,
|
| 744 |
+
"accuracy": 0.5487329434697856
|
| 745 |
+
},
|
| 746 |
+
{
|
| 747 |
+
"bin": "0.6-0.7",
|
| 748 |
+
"count": 832,
|
| 749 |
+
"confidence": 0.6478659043518397,
|
| 750 |
+
"accuracy": 0.6334134615384616
|
| 751 |
+
},
|
| 752 |
+
{
|
| 753 |
+
"bin": "0.7-0.8",
|
| 754 |
+
"count": 756,
|
| 755 |
+
"confidence": 0.7482921844594693,
|
| 756 |
+
"accuracy": 0.7314814814814815
|
| 757 |
+
},
|
| 758 |
+
{
|
| 759 |
+
"bin": "0.8-0.9",
|
| 760 |
+
"count": 814,
|
| 761 |
+
"confidence": 0.8511582021338348,
|
| 762 |
+
"accuracy": 0.7457002457002457
|
| 763 |
+
},
|
| 764 |
+
{
|
| 765 |
+
"bin": "0.9-1.0",
|
| 766 |
+
"count": 1418,
|
| 767 |
+
"confidence": 0.9543087374844571,
|
| 768 |
+
"accuracy": 0.9160789844851904
|
| 769 |
+
}
|
| 770 |
+
]
|
| 771 |
+
}
|
eval-dev.json
ADDED
|
@@ -0,0 +1,771 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"overall": {
|
| 3 |
+
"count": 6113,
|
| 4 |
+
"accuracy": 0.6679208244724358,
|
| 5 |
+
"brier": 0.4544455845096957,
|
| 6 |
+
"brier_soft": 0.4310818103173888,
|
| 7 |
+
"nll": 0.8546754183783308,
|
| 8 |
+
"nll_soft": 0.8720034822503786,
|
| 9 |
+
"ece": 0.10337806078709712,
|
| 10 |
+
"mean_confidence": 0.769263460145617,
|
| 11 |
+
"confident_error_rate": 0.05300179944380828,
|
| 12 |
+
"score_mae": 0.53153959018479,
|
| 13 |
+
"evidence_auc": 0.9074727460810058
|
| 14 |
+
},
|
| 15 |
+
"by_type": {
|
| 16 |
+
"bool": {
|
| 17 |
+
"count": 1898,
|
| 18 |
+
"accuracy": 0.7787144362486829,
|
| 19 |
+
"brier": 0.32803778696812624,
|
| 20 |
+
"brier_soft": 0.32512087658844907,
|
| 21 |
+
"nll": 0.5308636866774319,
|
| 22 |
+
"nll_soft": 0.5373907544543911,
|
| 23 |
+
"ece": 0.0827987949639653,
|
| 24 |
+
"mean_confidence": 0.8615132312126481,
|
| 25 |
+
"confident_error_rate": 0.07586933614330875,
|
| 26 |
+
"evidence_auc": 0.9039322111530974
|
| 27 |
+
},
|
| 28 |
+
"choice": {
|
| 29 |
+
"count": 3248,
|
| 30 |
+
"accuracy": 0.625615763546798,
|
| 31 |
+
"brier": 0.5167065686871394,
|
| 32 |
+
"brier_soft": 0.4935414744397849,
|
| 33 |
+
"nll": 1.034598991800825,
|
| 34 |
+
"nll_soft": 1.0506837306443417,
|
| 35 |
+
"ece": 0.13399685761808722,
|
| 36 |
+
"mean_confidence": 0.7569434392263682,
|
| 37 |
+
"confident_error_rate": 0.05511083743842365,
|
| 38 |
+
"evidence_auc": 0.9208606009429315
|
| 39 |
+
},
|
| 40 |
+
"score": {
|
| 41 |
+
"count": 967,
|
| 42 |
+
"accuracy": 0.5925542916235781,
|
| 43 |
+
"brier": 0.4934304067698423,
|
| 44 |
+
"brier_soft": 0.429266777378387,
|
| 45 |
+
"nll": 0.8859090277806523,
|
| 46 |
+
"nll_soft": 0.9286131105577126,
|
| 47 |
+
"ece": 0.07366032000041674,
|
| 48 |
+
"mean_confidence": 0.6295792434553326,
|
| 49 |
+
"confident_error_rate": 0.001034126163391934,
|
| 50 |
+
"score_mae": 0.53153959018479
|
| 51 |
+
}
|
| 52 |
+
},
|
| 53 |
+
"by_source": {
|
| 54 |
+
"ag_news": {
|
| 55 |
+
"count": 100,
|
| 56 |
+
"accuracy": 0.88,
|
| 57 |
+
"brier": 0.1956816155108337,
|
| 58 |
+
"brier_soft": 0.1956816155108337,
|
| 59 |
+
"nll": 0.3569359667701418,
|
| 60 |
+
"nll_soft": 0.3569359667701418,
|
| 61 |
+
"ece": 0.09543950200080867,
|
| 62 |
+
"mean_confidence": 0.957247873544693,
|
| 63 |
+
"confident_error_rate": 0.06
|
| 64 |
+
},
|
| 65 |
+
"amazon_polarity": {
|
| 66 |
+
"count": 100,
|
| 67 |
+
"accuracy": 0.97,
|
| 68 |
+
"brier": 0.06277382037310442,
|
| 69 |
+
"brier_soft": 0.06277382037310442,
|
| 70 |
+
"nll": 0.14048191098950258,
|
| 71 |
+
"nll_soft": 0.14048191098950258,
|
| 72 |
+
"ece": 0.028436743021011374,
|
| 73 |
+
"mean_confidence": 0.9850584268569946,
|
| 74 |
+
"confident_error_rate": 0.03
|
| 75 |
+
},
|
| 76 |
+
"amazon_stars": {
|
| 77 |
+
"count": 100,
|
| 78 |
+
"accuracy": 0.53,
|
| 79 |
+
"brier": 0.576941525754432,
|
| 80 |
+
"brier_soft": 0.576941525754432,
|
| 81 |
+
"nll": 0.9323376081861305,
|
| 82 |
+
"nll_soft": 0.9323376081861305,
|
| 83 |
+
"ece": 0.16485882312059405,
|
| 84 |
+
"mean_confidence": 0.6662529197335243,
|
| 85 |
+
"confident_error_rate": 0.0,
|
| 86 |
+
"score_mae": 0.5123922033898998
|
| 87 |
+
},
|
| 88 |
+
"anli": {
|
| 89 |
+
"count": 100,
|
| 90 |
+
"accuracy": 0.39,
|
| 91 |
+
"brier": 1.0697171006877613,
|
| 92 |
+
"brier_soft": 1.0697171006877613,
|
| 93 |
+
"nll": 2.53446586810947,
|
| 94 |
+
"nll_soft": 2.53446586810947,
|
| 95 |
+
"ece": 0.49495991080999374,
|
| 96 |
+
"mean_confidence": 0.8732111909985543,
|
| 97 |
+
"confident_error_rate": 0.35
|
| 98 |
+
},
|
| 99 |
+
"arc_challenge": {
|
| 100 |
+
"count": 100,
|
| 101 |
+
"accuracy": 0.36,
|
| 102 |
+
"brier": 0.85564326196725,
|
| 103 |
+
"brier_soft": 0.85564326196725,
|
| 104 |
+
"nll": 1.6615319409245108,
|
| 105 |
+
"nll_soft": 1.6615319409245108,
|
| 106 |
+
"ece": 0.30022275894880296,
|
| 107 |
+
"mean_confidence": 0.6384962823987007,
|
| 108 |
+
"confident_error_rate": 0.07
|
| 109 |
+
},
|
| 110 |
+
"arc_easy": {
|
| 111 |
+
"count": 100,
|
| 112 |
+
"accuracy": 0.45,
|
| 113 |
+
"brier": 0.6727499751604388,
|
| 114 |
+
"brier_soft": 0.6727499751604388,
|
| 115 |
+
"nll": 1.3332344578928694,
|
| 116 |
+
"nll_soft": 1.3332344578928694,
|
| 117 |
+
"ece": 0.20594446569681168,
|
| 118 |
+
"mean_confidence": 0.6355124202370643,
|
| 119 |
+
"confident_error_rate": 0.03
|
| 120 |
+
},
|
| 121 |
+
"banking77": {
|
| 122 |
+
"count": 100,
|
| 123 |
+
"accuracy": 0.81,
|
| 124 |
+
"brier": 0.27520138323214327,
|
| 125 |
+
"brier_soft": 0.27520138323214327,
|
| 126 |
+
"nll": 0.7042584741411863,
|
| 127 |
+
"nll_soft": 0.7042584741411863,
|
| 128 |
+
"ece": 0.07309853732585905,
|
| 129 |
+
"mean_confidence": 0.7600076282024384,
|
| 130 |
+
"confident_error_rate": 0.0
|
| 131 |
+
},
|
| 132 |
+
"boolq": {
|
| 133 |
+
"count": 125,
|
| 134 |
+
"accuracy": 0.568,
|
| 135 |
+
"brier": 0.7086733078745265,
|
| 136 |
+
"brier_soft": 0.7086733078745265,
|
| 137 |
+
"nll": 1.0996114747593868,
|
| 138 |
+
"nll_soft": 1.0996114747593868,
|
| 139 |
+
"ece": 0.345228253364563,
|
| 140 |
+
"mean_confidence": 0.8863685808181763,
|
| 141 |
+
"confident_error_rate": 0.264,
|
| 142 |
+
"evidence_auc": 0.9706666666666667
|
| 143 |
+
},
|
| 144 |
+
"cb": {
|
| 145 |
+
"count": 28,
|
| 146 |
+
"accuracy": 0.9285714285714286,
|
| 147 |
+
"brier": 0.14187151621545094,
|
| 148 |
+
"brier_soft": 0.14187151621545094,
|
| 149 |
+
"nll": 0.3411688505063019,
|
| 150 |
+
"nll_soft": 0.3411688505063019,
|
| 151 |
+
"ece": 0.08071247168949669,
|
| 152 |
+
"mean_confidence": 0.9737525028841836,
|
| 153 |
+
"confident_error_rate": 0.07142857142857142
|
| 154 |
+
},
|
| 155 |
+
"civil_comments_severity": {
|
| 156 |
+
"count": 100,
|
| 157 |
+
"accuracy": 0.81,
|
| 158 |
+
"brier": 0.2641616792613485,
|
| 159 |
+
"brier_soft": 0.18398093610713928,
|
| 160 |
+
"nll": 0.48525818979939045,
|
| 161 |
+
"nll_soft": 0.6278173365191989,
|
| 162 |
+
"ece": 0.0930167320370674,
|
| 163 |
+
"mean_confidence": 0.7847322449088097,
|
| 164 |
+
"confident_error_rate": 0.01,
|
| 165 |
+
"score_mae": 0.31169346647828206
|
| 166 |
+
},
|
| 167 |
+
"civil_comments_toxic": {
|
| 168 |
+
"count": 100,
|
| 169 |
+
"accuracy": 0.94,
|
| 170 |
+
"brier": 0.08438325557871443,
|
| 171 |
+
"brier_soft": 0.02902029657244265,
|
| 172 |
+
"nll": 0.151836350238126,
|
| 173 |
+
"nll_soft": 0.27572009664481245,
|
| 174 |
+
"ece": 0.04216815173625941,
|
| 175 |
+
"mean_confidence": 0.9121545308828354,
|
| 176 |
+
"confident_error_rate": 0.01
|
| 177 |
+
},
|
| 178 |
+
"clinc150": {
|
| 179 |
+
"count": 100,
|
| 180 |
+
"accuracy": 0.8,
|
| 181 |
+
"brier": 0.2993320627901788,
|
| 182 |
+
"brier_soft": 0.2993320627901788,
|
| 183 |
+
"nll": 0.9652885590344843,
|
| 184 |
+
"nll_soft": 0.9652885590344843,
|
| 185 |
+
"ece": 0.11481654163450003,
|
| 186 |
+
"mean_confidence": 0.7029852807894349,
|
| 187 |
+
"confident_error_rate": 0.0
|
| 188 |
+
},
|
| 189 |
+
"cola": {
|
| 190 |
+
"count": 100,
|
| 191 |
+
"accuracy": 0.58,
|
| 192 |
+
"brier": 0.49509596184792515,
|
| 193 |
+
"brier_soft": 0.49509596184792515,
|
| 194 |
+
"nll": 0.6953869891066888,
|
| 195 |
+
"nll_soft": 0.6953869891066888,
|
| 196 |
+
"ece": 0.15781798183918,
|
| 197 |
+
"mean_confidence": 0.7369759422540665,
|
| 198 |
+
"confident_error_rate": 0.02
|
| 199 |
+
},
|
| 200 |
+
"commonsense_qa": {
|
| 201 |
+
"count": 100,
|
| 202 |
+
"accuracy": 0.42,
|
| 203 |
+
"brier": 0.7442126993153202,
|
| 204 |
+
"brier_soft": 0.7442126993153202,
|
| 205 |
+
"nll": 1.4986388927152847,
|
| 206 |
+
"nll_soft": 1.4986388927152847,
|
| 207 |
+
"ece": 0.2365888097882271,
|
| 208 |
+
"mean_confidence": 0.6122629001736641,
|
| 209 |
+
"confident_error_rate": 0.05
|
| 210 |
+
},
|
| 211 |
+
"cosmos_qa": {
|
| 212 |
+
"count": 120,
|
| 213 |
+
"accuracy": 0.4,
|
| 214 |
+
"brier": 0.7216093164068191,
|
| 215 |
+
"brier_soft": 0.7216093164068191,
|
| 216 |
+
"nll": 1.3523367352044129,
|
| 217 |
+
"nll_soft": 1.3523367352044129,
|
| 218 |
+
"ece": 0.22780664563179015,
|
| 219 |
+
"mean_confidence": 0.6138915047049522,
|
| 220 |
+
"confident_error_rate": 0.016666666666666666,
|
| 221 |
+
"evidence_auc": 0.836875
|
| 222 |
+
},
|
| 223 |
+
"dates": {
|
| 224 |
+
"count": 300,
|
| 225 |
+
"accuracy": 0.7633333333333333,
|
| 226 |
+
"brier": 0.28494306618151344,
|
| 227 |
+
"brier_soft": 0.28494306618151344,
|
| 228 |
+
"nll": 0.461646918030587,
|
| 229 |
+
"nll_soft": 0.461646918030587,
|
| 230 |
+
"ece": 0.06982577522595727,
|
| 231 |
+
"mean_confidence": 0.8165360381205876,
|
| 232 |
+
"confident_error_rate": 0.02666666666666667,
|
| 233 |
+
"score_mae": 0.42511637235032534
|
| 234 |
+
},
|
| 235 |
+
"dbpedia": {
|
| 236 |
+
"count": 100,
|
| 237 |
+
"accuracy": 0.99,
|
| 238 |
+
"brier": 0.01313320294332829,
|
| 239 |
+
"brier_soft": 0.01313320294332829,
|
| 240 |
+
"nll": 0.022723471569511666,
|
| 241 |
+
"nll_soft": 0.022723471569511666,
|
| 242 |
+
"ece": 0.015564617514610263,
|
| 243 |
+
"mean_confidence": 0.9883377414941787,
|
| 244 |
+
"confident_error_rate": 0.0
|
| 245 |
+
},
|
| 246 |
+
"dynasent": {
|
| 247 |
+
"count": 100,
|
| 248 |
+
"accuracy": 0.53,
|
| 249 |
+
"brier": 0.5607406074995769,
|
| 250 |
+
"brier_soft": 0.42609746333683257,
|
| 251 |
+
"nll": 0.9629181593054416,
|
| 252 |
+
"nll_soft": 1.0097704657215225,
|
| 253 |
+
"ece": 0.1584402248263359,
|
| 254 |
+
"mean_confidence": 0.6659586235880852,
|
| 255 |
+
"confident_error_rate": 0.0,
|
| 256 |
+
"score_mae": 0.5633883766271174
|
| 257 |
+
},
|
| 258 |
+
"enron_spam": {
|
| 259 |
+
"count": 100,
|
| 260 |
+
"accuracy": 0.94,
|
| 261 |
+
"brier": 0.08979354473427502,
|
| 262 |
+
"brier_soft": 0.08979354473427502,
|
| 263 |
+
"nll": 0.1666279776671364,
|
| 264 |
+
"nll_soft": 0.1666279776671364,
|
| 265 |
+
"ece": 0.0407087934017182,
|
| 266 |
+
"mean_confidence": 0.9807087934017181,
|
| 267 |
+
"confident_error_rate": 0.03
|
| 268 |
+
},
|
| 269 |
+
"go_emotions": {
|
| 270 |
+
"count": 100,
|
| 271 |
+
"accuracy": 0.54,
|
| 272 |
+
"brier": 0.5986230077830135,
|
| 273 |
+
"brier_soft": 0.23341383083194883,
|
| 274 |
+
"nll": 1.5085460014590555,
|
| 275 |
+
"nll_soft": 2.014061282637175,
|
| 276 |
+
"ece": 0.1133653888106346,
|
| 277 |
+
"mean_confidence": 0.47643122762441636,
|
| 278 |
+
"confident_error_rate": 0.0
|
| 279 |
+
},
|
| 280 |
+
"hellaswag": {
|
| 281 |
+
"count": 100,
|
| 282 |
+
"accuracy": 0.49,
|
| 283 |
+
"brier": 0.7303985853764171,
|
| 284 |
+
"brier_soft": 0.7303985853764171,
|
| 285 |
+
"nll": 1.4383446168093819,
|
| 286 |
+
"nll_soft": 1.4383446168093819,
|
| 287 |
+
"ece": 0.22660601407289505,
|
| 288 |
+
"mean_confidence": 0.658343857228756,
|
| 289 |
+
"confident_error_rate": 0.06
|
| 290 |
+
},
|
| 291 |
+
"imdb": {
|
| 292 |
+
"count": 100,
|
| 293 |
+
"accuracy": 0.95,
|
| 294 |
+
"brier": 0.08479113838840116,
|
| 295 |
+
"brier_soft": 0.08479113838840116,
|
| 296 |
+
"nll": 0.20038125969079196,
|
| 297 |
+
"nll_soft": 0.20038125969079196,
|
| 298 |
+
"ece": 0.04454047501087191,
|
| 299 |
+
"mean_confidence": 0.977543198466301,
|
| 300 |
+
"confident_error_rate": 0.03
|
| 301 |
+
},
|
| 302 |
+
"ledgar": {
|
| 303 |
+
"count": 100,
|
| 304 |
+
"accuracy": 0.75,
|
| 305 |
+
"brier": 0.37746571387860506,
|
| 306 |
+
"brier_soft": 0.37746571387860506,
|
| 307 |
+
"nll": 1.117831587098228,
|
| 308 |
+
"nll_soft": 1.117831587098228,
|
| 309 |
+
"ece": 0.09851389944553374,
|
| 310 |
+
"mean_confidence": 0.7810949206352233,
|
| 311 |
+
"confident_error_rate": 0.01
|
| 312 |
+
},
|
| 313 |
+
"liar": {
|
| 314 |
+
"count": 100,
|
| 315 |
+
"accuracy": 0.32,
|
| 316 |
+
"brier": 0.7427717464748098,
|
| 317 |
+
"brier_soft": 0.7427717464748098,
|
| 318 |
+
"nll": 1.4772028411140192,
|
| 319 |
+
"nll_soft": 1.4772028411140192,
|
| 320 |
+
"ece": 0.10479136735200882,
|
| 321 |
+
"mean_confidence": 0.3535214939713478,
|
| 322 |
+
"confident_error_rate": 0.0,
|
| 323 |
+
"score_mae": 0.9304733183607459
|
| 324 |
+
},
|
| 325 |
+
"measuring_hate_speech": {
|
| 326 |
+
"count": 100,
|
| 327 |
+
"accuracy": 0.83,
|
| 328 |
+
"brier": 0.26393481691390774,
|
| 329 |
+
"brier_soft": 0.1380680952422536,
|
| 330 |
+
"nll": 0.5018035055388043,
|
| 331 |
+
"nll_soft": 0.6680358527776807,
|
| 332 |
+
"ece": 0.1381094029545784,
|
| 333 |
+
"mean_confidence": 0.7157146391272545,
|
| 334 |
+
"confident_error_rate": 0.0,
|
| 335 |
+
"score_mae": 0.5079832007945515
|
| 336 |
+
},
|
| 337 |
+
"medqa": {
|
| 338 |
+
"count": 100,
|
| 339 |
+
"accuracy": 0.34,
|
| 340 |
+
"brier": 0.8253335318993641,
|
| 341 |
+
"brier_soft": 0.8253335318993641,
|
| 342 |
+
"nll": 1.6004826115846618,
|
| 343 |
+
"nll_soft": 1.6004826115846618,
|
| 344 |
+
"ece": 0.24258146941661834,
|
| 345 |
+
"mean_confidence": 0.5722464674711227,
|
| 346 |
+
"confident_error_rate": 0.01
|
| 347 |
+
},
|
| 348 |
+
"mnli": {
|
| 349 |
+
"count": 100,
|
| 350 |
+
"accuracy": 0.78,
|
| 351 |
+
"brier": 0.35049440453320635,
|
| 352 |
+
"brier_soft": 0.35049440453320635,
|
| 353 |
+
"nll": 0.7340184397638638,
|
| 354 |
+
"nll_soft": 0.7340184397638638,
|
| 355 |
+
"ece": 0.1504778018593788,
|
| 356 |
+
"mean_confidence": 0.9077262398600578,
|
| 357 |
+
"confident_error_rate": 0.09
|
| 358 |
+
},
|
| 359 |
+
"mrpc": {
|
| 360 |
+
"count": 100,
|
| 361 |
+
"accuracy": 0.81,
|
| 362 |
+
"brier": 0.296402084274162,
|
| 363 |
+
"brier_soft": 0.296402084274162,
|
| 364 |
+
"nll": 0.49396724593180946,
|
| 365 |
+
"nll_soft": 0.49396724593180946,
|
| 366 |
+
"ece": 0.12271335303783416,
|
| 367 |
+
"mean_confidence": 0.913284073472023,
|
| 368 |
+
"confident_error_rate": 0.11
|
| 369 |
+
},
|
| 370 |
+
"multirc": {
|
| 371 |
+
"count": 125,
|
| 372 |
+
"accuracy": 0.664,
|
| 373 |
+
"brier": 0.5778966045169998,
|
| 374 |
+
"brier_soft": 0.5778966045169998,
|
| 375 |
+
"nll": 1.0105245482832101,
|
| 376 |
+
"nll_soft": 1.0105245482832101,
|
| 377 |
+
"ece": 0.2395370769500732,
|
| 378 |
+
"mean_confidence": 0.8928435230255127,
|
| 379 |
+
"confident_error_rate": 0.232,
|
| 380 |
+
"evidence_auc": 0.6994666666666667
|
| 381 |
+
},
|
| 382 |
+
"openbookqa": {
|
| 383 |
+
"count": 100,
|
| 384 |
+
"accuracy": 0.52,
|
| 385 |
+
"brier": 0.6241696926704345,
|
| 386 |
+
"brier_soft": 0.6241696926704345,
|
| 387 |
+
"nll": 1.2342317482843357,
|
| 388 |
+
"nll_soft": 1.2342317482843357,
|
| 389 |
+
"ece": 0.1368167817592621,
|
| 390 |
+
"mean_confidence": 0.6306584364175797,
|
| 391 |
+
"confident_error_rate": 0.01
|
| 392 |
+
},
|
| 393 |
+
"piqa": {
|
| 394 |
+
"count": 100,
|
| 395 |
+
"accuracy": 0.61,
|
| 396 |
+
"brier": 0.5360872271281376,
|
| 397 |
+
"brier_soft": 0.5360872271281376,
|
| 398 |
+
"nll": 0.7988801723216932,
|
| 399 |
+
"nll_soft": 0.7988801723216932,
|
| 400 |
+
"ece": 0.1281885802745819,
|
| 401 |
+
"mean_confidence": 0.7082396709918976,
|
| 402 |
+
"confident_error_rate": 0.07
|
| 403 |
+
},
|
| 404 |
+
"policy_application": {
|
| 405 |
+
"count": 150,
|
| 406 |
+
"accuracy": 0.7333333333333333,
|
| 407 |
+
"brier": 0.3540402117052364,
|
| 408 |
+
"brier_soft": 0.3540402117052364,
|
| 409 |
+
"nll": 0.5500808292524187,
|
| 410 |
+
"nll_soft": 0.5500808292524187,
|
| 411 |
+
"ece": 0.07159115393956501,
|
| 412 |
+
"mean_confidence": 0.7941183785597483,
|
| 413 |
+
"confident_error_rate": 0.02666666666666667,
|
| 414 |
+
"evidence_auc": 0.5696533682145193
|
| 415 |
+
},
|
| 416 |
+
"policy_hard": {
|
| 417 |
+
"count": 300,
|
| 418 |
+
"accuracy": 0.6966666666666667,
|
| 419 |
+
"brier": 0.39680403372516454,
|
| 420 |
+
"brier_soft": 0.39680403372516454,
|
| 421 |
+
"nll": 0.6353135836137924,
|
| 422 |
+
"nll_soft": 0.6353135836137924,
|
| 423 |
+
"ece": 0.09296291559934615,
|
| 424 |
+
"mean_confidence": 0.7867085169752439,
|
| 425 |
+
"confident_error_rate": 0.03,
|
| 426 |
+
"evidence_auc": 0.7445023148148148
|
| 427 |
+
},
|
| 428 |
+
"qqp": {
|
| 429 |
+
"count": 100,
|
| 430 |
+
"accuracy": 0.8,
|
| 431 |
+
"brier": 0.33138385122291225,
|
| 432 |
+
"brier_soft": 0.33138385122291225,
|
| 433 |
+
"nll": 0.5997304360971085,
|
| 434 |
+
"nll_soft": 0.5997304360971085,
|
| 435 |
+
"ece": 0.160344073176384,
|
| 436 |
+
"mean_confidence": 0.92669453561306,
|
| 437 |
+
"confident_error_rate": 0.11
|
| 438 |
+
},
|
| 439 |
+
"race": {
|
| 440 |
+
"count": 125,
|
| 441 |
+
"accuracy": 0.464,
|
| 442 |
+
"brier": 0.6912585134522173,
|
| 443 |
+
"brier_soft": 0.6912585134522173,
|
| 444 |
+
"nll": 1.3744160623714958,
|
| 445 |
+
"nll_soft": 1.3744160623714958,
|
| 446 |
+
"ece": 0.14605404186248777,
|
| 447 |
+
"mean_confidence": 0.5852895030975341,
|
| 448 |
+
"confident_error_rate": 0.016,
|
| 449 |
+
"evidence_auc": 0.7856
|
| 450 |
+
},
|
| 451 |
+
"rotten_tomatoes": {
|
| 452 |
+
"count": 100,
|
| 453 |
+
"accuracy": 0.89,
|
| 454 |
+
"brier": 0.2377622469757273,
|
| 455 |
+
"brier_soft": 0.2377622469757273,
|
| 456 |
+
"nll": 0.3884358803375364,
|
| 457 |
+
"nll_soft": 0.3884358803375364,
|
| 458 |
+
"ece": 0.13390481829643247,
|
| 459 |
+
"mean_confidence": 0.7656551349163055,
|
| 460 |
+
"confident_error_rate": 0.0
|
| 461 |
+
},
|
| 462 |
+
"rte": {
|
| 463 |
+
"count": 100,
|
| 464 |
+
"accuracy": 0.7,
|
| 465 |
+
"brier": 0.48653410178986045,
|
| 466 |
+
"brier_soft": 0.48653410178986045,
|
| 467 |
+
"nll": 0.8552582691879445,
|
| 468 |
+
"nll_soft": 0.8552582691879445,
|
| 469 |
+
"ece": 0.20795070528984066,
|
| 470 |
+
"mean_confidence": 0.9079507052898407,
|
| 471 |
+
"confident_error_rate": 0.2
|
| 472 |
+
},
|
| 473 |
+
"scotus": {
|
| 474 |
+
"count": 100,
|
| 475 |
+
"accuracy": 0.63,
|
| 476 |
+
"brier": 0.558399670895838,
|
| 477 |
+
"brier_soft": 0.558399670895838,
|
| 478 |
+
"nll": 1.3936418468284624,
|
| 479 |
+
"nll_soft": 1.3936418468284624,
|
| 480 |
+
"ece": 0.13719820365309715,
|
| 481 |
+
"mean_confidence": 0.6636926798522472,
|
| 482 |
+
"confident_error_rate": 0.01
|
| 483 |
+
},
|
| 484 |
+
"shp": {
|
| 485 |
+
"count": 100,
|
| 486 |
+
"accuracy": 0.5,
|
| 487 |
+
"brier": 0.6054587301920193,
|
| 488 |
+
"brier_soft": 0.2182656459890131,
|
| 489 |
+
"nll": 0.8379870996959774,
|
| 490 |
+
"nll_soft": 0.8549041361552857,
|
| 491 |
+
"ece": 0.22597848415374755,
|
| 492 |
+
"mean_confidence": 0.7121235334873199,
|
| 493 |
+
"confident_error_rate": 0.04
|
| 494 |
+
},
|
| 495 |
+
"sms_spam": {
|
| 496 |
+
"count": 100,
|
| 497 |
+
"accuracy": 0.96,
|
| 498 |
+
"brier": 0.1060751114641901,
|
| 499 |
+
"brier_soft": 0.1060751114641901,
|
| 500 |
+
"nll": 0.21970222462417918,
|
| 501 |
+
"nll_soft": 0.21970222462417918,
|
| 502 |
+
"ece": 0.1075866764783859,
|
| 503 |
+
"mean_confidence": 0.852413323521614,
|
| 504 |
+
"confident_error_rate": 0.0
|
| 505 |
+
},
|
| 506 |
+
"snli": {
|
| 507 |
+
"count": 100,
|
| 508 |
+
"accuracy": 0.79,
|
| 509 |
+
"brier": 0.3950873837225307,
|
| 510 |
+
"brier_soft": 0.3950873837225307,
|
| 511 |
+
"nll": 0.8923272739656278,
|
| 512 |
+
"nll_soft": 0.8923272739656278,
|
| 513 |
+
"ece": 0.20375718295574188,
|
| 514 |
+
"mean_confidence": 0.9457461875677109,
|
| 515 |
+
"confident_error_rate": 0.16
|
| 516 |
+
},
|
| 517 |
+
"social_iqa": {
|
| 518 |
+
"count": 100,
|
| 519 |
+
"accuracy": 0.5,
|
| 520 |
+
"brier": 0.7282720596366673,
|
| 521 |
+
"brier_soft": 0.7282720596366673,
|
| 522 |
+
"nll": 1.4898040089024713,
|
| 523 |
+
"nll_soft": 1.4898040089024713,
|
| 524 |
+
"ece": 0.2804051974415779,
|
| 525 |
+
"mean_confidence": 0.766021781861782,
|
| 526 |
+
"confident_error_rate": 0.11
|
| 527 |
+
},
|
| 528 |
+
"sst2": {
|
| 529 |
+
"count": 100,
|
| 530 |
+
"accuracy": 0.9,
|
| 531 |
+
"brier": 0.22684034833869313,
|
| 532 |
+
"brier_soft": 0.22684034833869313,
|
| 533 |
+
"nll": 0.3794984833755713,
|
| 534 |
+
"nll_soft": 0.3794984833755713,
|
| 535 |
+
"ece": 0.14272936522960666,
|
| 536 |
+
"mean_confidence": 0.7780753713846207,
|
| 537 |
+
"confident_error_rate": 0.02
|
| 538 |
+
},
|
| 539 |
+
"sst5": {
|
| 540 |
+
"count": 100,
|
| 541 |
+
"accuracy": 0.56,
|
| 542 |
+
"brier": 0.5610144765744216,
|
| 543 |
+
"brier_soft": 0.5610144765744216,
|
| 544 |
+
"nll": 1.012664727296375,
|
| 545 |
+
"nll_soft": 1.012664727296375,
|
| 546 |
+
"ece": 0.06806655168533324,
|
| 547 |
+
"mean_confidence": 0.5649158990383149,
|
| 548 |
+
"confident_error_rate": 0.0,
|
| 549 |
+
"score_mae": 0.5462690752366325
|
| 550 |
+
},
|
| 551 |
+
"stance_abortion": {
|
| 552 |
+
"count": 60,
|
| 553 |
+
"accuracy": 0.6333333333333333,
|
| 554 |
+
"brier": 0.524088736443351,
|
| 555 |
+
"brier_soft": 0.524088736443351,
|
| 556 |
+
"nll": 0.8961191172069207,
|
| 557 |
+
"nll_soft": 0.8961191172069207,
|
| 558 |
+
"ece": 0.21964709063371024,
|
| 559 |
+
"mean_confidence": 0.8473132938146591,
|
| 560 |
+
"confident_error_rate": 0.11666666666666667
|
| 561 |
+
},
|
| 562 |
+
"stance_atheism": {
|
| 563 |
+
"count": 60,
|
| 564 |
+
"accuracy": 0.7,
|
| 565 |
+
"brier": 0.4295024180903305,
|
| 566 |
+
"brier_soft": 0.4295024180903305,
|
| 567 |
+
"nll": 0.7695298592508046,
|
| 568 |
+
"nll_soft": 0.7695298592508046,
|
| 569 |
+
"ece": 0.20804943144321447,
|
| 570 |
+
"mean_confidence": 0.8740352998177211,
|
| 571 |
+
"confident_error_rate": 0.11666666666666667
|
| 572 |
+
},
|
| 573 |
+
"stance_feminist": {
|
| 574 |
+
"count": 60,
|
| 575 |
+
"accuracy": 0.6833333333333333,
|
| 576 |
+
"brier": 0.5069301963867938,
|
| 577 |
+
"brier_soft": 0.5069301963867938,
|
| 578 |
+
"nll": 0.8636041819841961,
|
| 579 |
+
"nll_soft": 0.8636041819841961,
|
| 580 |
+
"ece": 0.24766051123539606,
|
| 581 |
+
"mean_confidence": 0.8220602666338285,
|
| 582 |
+
"confident_error_rate": 0.08333333333333333
|
| 583 |
+
},
|
| 584 |
+
"stance_hillary": {
|
| 585 |
+
"count": 60,
|
| 586 |
+
"accuracy": 0.6666666666666666,
|
| 587 |
+
"brier": 0.5074511953888173,
|
| 588 |
+
"brier_soft": 0.5074511953888173,
|
| 589 |
+
"nll": 0.9144219988342865,
|
| 590 |
+
"nll_soft": 0.9144219988342865,
|
| 591 |
+
"ece": 0.1895925045013428,
|
| 592 |
+
"mean_confidence": 0.8227309505144755,
|
| 593 |
+
"confident_error_rate": 0.08333333333333333
|
| 594 |
+
},
|
| 595 |
+
"stsb": {
|
| 596 |
+
"count": 100,
|
| 597 |
+
"accuracy": 0.45,
|
| 598 |
+
"brier": 0.6911273352227522,
|
| 599 |
+
"brier_soft": 0.4113556479959866,
|
| 600 |
+
"nll": 1.3398778933038693,
|
| 601 |
+
"nll_soft": 1.3971825733832768,
|
| 602 |
+
"ece": 0.17724637597799303,
|
| 603 |
+
"mean_confidence": 0.6206034556031227,
|
| 604 |
+
"confident_error_rate": 0.0,
|
| 605 |
+
"score_mae": 0.6759577402091236
|
| 606 |
+
},
|
| 607 |
+
"subjectivity": {
|
| 608 |
+
"count": 100,
|
| 609 |
+
"accuracy": 0.94,
|
| 610 |
+
"brier": 0.0767058129522083,
|
| 611 |
+
"brier_soft": 0.0767058129522083,
|
| 612 |
+
"nll": 0.11295796720663813,
|
| 613 |
+
"nll_soft": 0.11295796720663813,
|
| 614 |
+
"ece": 0.04773887455463412,
|
| 615 |
+
"mean_confidence": 0.9554775649309158,
|
| 616 |
+
"confident_error_rate": 0.0
|
| 617 |
+
},
|
| 618 |
+
"swag": {
|
| 619 |
+
"count": 100,
|
| 620 |
+
"accuracy": 0.4,
|
| 621 |
+
"brier": 0.8060758700239877,
|
| 622 |
+
"brier_soft": 0.8060758700239877,
|
| 623 |
+
"nll": 1.6377803414833798,
|
| 624 |
+
"nll_soft": 1.6377803414833798,
|
| 625 |
+
"ece": 0.313101019859314,
|
| 626 |
+
"mean_confidence": 0.713101019859314,
|
| 627 |
+
"confident_error_rate": 0.08
|
| 628 |
+
},
|
| 629 |
+
"trec": {
|
| 630 |
+
"count": 100,
|
| 631 |
+
"accuracy": 0.96,
|
| 632 |
+
"brier": 0.0627841559130499,
|
| 633 |
+
"brier_soft": 0.0627841559130499,
|
| 634 |
+
"nll": 0.1128754384695211,
|
| 635 |
+
"nll_soft": 0.1128754384695211,
|
| 636 |
+
"ece": 0.03479043036699291,
|
| 637 |
+
"mean_confidence": 0.9710938653349876,
|
| 638 |
+
"confident_error_rate": 0.01
|
| 639 |
+
},
|
| 640 |
+
"tweet_hate": {
|
| 641 |
+
"count": 100,
|
| 642 |
+
"accuracy": 0.54,
|
| 643 |
+
"brier": 0.4972376556033321,
|
| 644 |
+
"brier_soft": 0.4972376556033321,
|
| 645 |
+
"nll": 0.692663662352403,
|
| 646 |
+
"nll_soft": 0.692663662352403,
|
| 647 |
+
"ece": 0.11306986272335055,
|
| 648 |
+
"mean_confidence": 0.6452960926294327,
|
| 649 |
+
"confident_error_rate": 0.0
|
| 650 |
+
},
|
| 651 |
+
"tweet_sentiment": {
|
| 652 |
+
"count": 100,
|
| 653 |
+
"accuracy": 0.67,
|
| 654 |
+
"brier": 0.4101129435493609,
|
| 655 |
+
"brier_soft": 0.4101129435493609,
|
| 656 |
+
"nll": 0.6681253383561087,
|
| 657 |
+
"nll_soft": 0.6681253383561087,
|
| 658 |
+
"ece": 0.06605960577726364,
|
| 659 |
+
"mean_confidence": 0.6303303089737892,
|
| 660 |
+
"confident_error_rate": 0.0,
|
| 661 |
+
"score_mae": 0.3951122248638421
|
| 662 |
+
},
|
| 663 |
+
"vitaminc": {
|
| 664 |
+
"count": 100,
|
| 665 |
+
"accuracy": 0.55,
|
| 666 |
+
"brier": 0.6667891043228811,
|
| 667 |
+
"brier_soft": 0.6667891043228811,
|
| 668 |
+
"nll": 1.3148888047347953,
|
| 669 |
+
"nll_soft": 1.3148888047347953,
|
| 670 |
+
"ece": 0.2980202031135559,
|
| 671 |
+
"mean_confidence": 0.8368437862396241,
|
| 672 |
+
"confident_error_rate": 0.16
|
| 673 |
+
},
|
| 674 |
+
"wic": {
|
| 675 |
+
"count": 100,
|
| 676 |
+
"accuracy": 0.61,
|
| 677 |
+
"brier": 0.5528101823846308,
|
| 678 |
+
"brier_soft": 0.5528101823846308,
|
| 679 |
+
"nll": 0.8525569252877085,
|
| 680 |
+
"nll_soft": 0.8525569252877085,
|
| 681 |
+
"ece": 0.21445172727108003,
|
| 682 |
+
"mean_confidence": 0.8244517272710801,
|
| 683 |
+
"confident_error_rate": 0.12
|
| 684 |
+
},
|
| 685 |
+
"winogrande": {
|
| 686 |
+
"count": 100,
|
| 687 |
+
"accuracy": 0.57,
|
| 688 |
+
"brier": 0.5353339564884243,
|
| 689 |
+
"brier_soft": 0.5353339564884243,
|
| 690 |
+
"nll": 0.7633381528774766,
|
| 691 |
+
"nll_soft": 0.7633381528774766,
|
| 692 |
+
"ece": 0.17423450052738187,
|
| 693 |
+
"mean_confidence": 0.7043132489919662,
|
| 694 |
+
"confident_error_rate": 0.04
|
| 695 |
+
},
|
| 696 |
+
"yelp": {
|
| 697 |
+
"count": 100,
|
| 698 |
+
"accuracy": 0.62,
|
| 699 |
+
"brier": 0.41939405623046616,
|
| 700 |
+
"brier_soft": 0.41939405623046616,
|
| 701 |
+
"nll": 0.7083493947055206,
|
| 702 |
+
"nll_soft": 0.7083493947055206,
|
| 703 |
+
"ece": 0.12632228136062626,
|
| 704 |
+
"mean_confidence": 0.6600254368782044,
|
| 705 |
+
"confident_error_rate": 0.0,
|
| 706 |
+
"score_mae": 0.4118902616520063
|
| 707 |
+
}
|
| 708 |
+
},
|
| 709 |
+
"reliability": [
|
| 710 |
+
{
|
| 711 |
+
"bin": "0.0-0.1",
|
| 712 |
+
"count": 4,
|
| 713 |
+
"confidence": 0.06522660795599222,
|
| 714 |
+
"accuracy": 0.5
|
| 715 |
+
},
|
| 716 |
+
{
|
| 717 |
+
"bin": "0.1-0.2",
|
| 718 |
+
"count": 11,
|
| 719 |
+
"confidence": 0.16455867480147968,
|
| 720 |
+
"accuracy": 0.18181818181818182
|
| 721 |
+
},
|
| 722 |
+
{
|
| 723 |
+
"bin": "0.2-0.3",
|
| 724 |
+
"count": 88,
|
| 725 |
+
"confidence": 0.2596369771794839,
|
| 726 |
+
"accuracy": 0.29545454545454547
|
| 727 |
+
},
|
| 728 |
+
{
|
| 729 |
+
"bin": "0.3-0.4",
|
| 730 |
+
"count": 172,
|
| 731 |
+
"confidence": 0.3557821914207104,
|
| 732 |
+
"accuracy": 0.25
|
| 733 |
+
},
|
| 734 |
+
{
|
| 735 |
+
"bin": "0.4-0.5",
|
| 736 |
+
"count": 412,
|
| 737 |
+
"confidence": 0.45668168980809093,
|
| 738 |
+
"accuracy": 0.40048543689320387
|
| 739 |
+
},
|
| 740 |
+
{
|
| 741 |
+
"bin": "0.5-0.6",
|
| 742 |
+
"count": 828,
|
| 743 |
+
"confidence": 0.5498056534839713,
|
| 744 |
+
"accuracy": 0.4782608695652174
|
| 745 |
+
},
|
| 746 |
+
{
|
| 747 |
+
"bin": "0.6-0.7",
|
| 748 |
+
"count": 719,
|
| 749 |
+
"confidence": 0.6487497972513605,
|
| 750 |
+
"accuracy": 0.5465924895688457
|
| 751 |
+
},
|
| 752 |
+
{
|
| 753 |
+
"bin": "0.7-0.8",
|
| 754 |
+
"count": 714,
|
| 755 |
+
"confidence": 0.7503752866212059,
|
| 756 |
+
"accuracy": 0.6162464985994398
|
| 757 |
+
},
|
| 758 |
+
{
|
| 759 |
+
"bin": "0.8-0.9",
|
| 760 |
+
"count": 827,
|
| 761 |
+
"confidence": 0.8522957843044793,
|
| 762 |
+
"accuracy": 0.727932285368803
|
| 763 |
+
},
|
| 764 |
+
{
|
| 765 |
+
"bin": "0.9-1.0",
|
| 766 |
+
"count": 2338,
|
| 767 |
+
"confidence": 0.9691763384613449,
|
| 768 |
+
"accuracy": 0.8614200171086399
|
| 769 |
+
}
|
| 770 |
+
]
|
| 771 |
+
}
|
eval-holdout-scaled.json
ADDED
|
@@ -0,0 +1,236 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"overall": {
|
| 3 |
+
"count": 2020,
|
| 4 |
+
"accuracy": 0.6064356435643564,
|
| 5 |
+
"brier": 0.4944729819739457,
|
| 6 |
+
"brier_soft": 0.4944729819739457,
|
| 7 |
+
"nll": 0.8183667947897911,
|
| 8 |
+
"nll_soft": 0.8183667947897911,
|
| 9 |
+
"ece": 0.06059187534126907,
|
| 10 |
+
"mean_confidence": 0.6666950650291867,
|
| 11 |
+
"confident_error_rate": 0.01089108910891089,
|
| 12 |
+
"score_mae": 0.9293531678523869
|
| 13 |
+
},
|
| 14 |
+
"by_type": {
|
| 15 |
+
"bool": {
|
| 16 |
+
"count": 800,
|
| 17 |
+
"accuracy": 0.65,
|
| 18 |
+
"brier": 0.44401911072031097,
|
| 19 |
+
"brier_soft": 0.44401911072031097,
|
| 20 |
+
"nll": 0.6343919699093342,
|
| 21 |
+
"nll_soft": 0.6343919699093342,
|
| 22 |
+
"ece": 0.0897071849554777,
|
| 23 |
+
"mean_confidence": 0.7352745228260755,
|
| 24 |
+
"confident_error_rate": 0.0125
|
| 25 |
+
},
|
| 26 |
+
"choice": {
|
| 27 |
+
"count": 1020,
|
| 28 |
+
"accuracy": 0.6068627450980392,
|
| 29 |
+
"brier": 0.49623531796412274,
|
| 30 |
+
"brier_soft": 0.49623531796412274,
|
| 31 |
+
"nll": 0.8620572959484905,
|
| 32 |
+
"nll_soft": 0.8620572959484905,
|
| 33 |
+
"ece": 0.039002735270004626,
|
| 34 |
+
"mean_confidence": 0.6414189851459335,
|
| 35 |
+
"confident_error_rate": 0.011764705882352941
|
| 36 |
+
},
|
| 37 |
+
"score": {
|
| 38 |
+
"count": 200,
|
| 39 |
+
"accuracy": 0.43,
|
| 40 |
+
"brier": 0.6873005534385817,
|
| 41 |
+
"brier_soft": 0.6873005534385817,
|
| 42 |
+
"nll": 1.3314445384022526,
|
| 43 |
+
"nll_soft": 1.3314445384022526,
|
| 44 |
+
"ece": 0.11251327276229857,
|
| 45 |
+
"mean_confidence": 0.5212852412462234,
|
| 46 |
+
"confident_error_rate": 0.0,
|
| 47 |
+
"score_mae": 0.9293531678523869
|
| 48 |
+
}
|
| 49 |
+
},
|
| 50 |
+
"by_source": {
|
| 51 |
+
"app_reviews": {
|
| 52 |
+
"count": 200,
|
| 53 |
+
"accuracy": 0.43,
|
| 54 |
+
"brier": 0.6873005534385817,
|
| 55 |
+
"brier_soft": 0.6873005534385817,
|
| 56 |
+
"nll": 1.3314445384022526,
|
| 57 |
+
"nll_soft": 1.3314445384022526,
|
| 58 |
+
"ece": 0.11251327276229857,
|
| 59 |
+
"mean_confidence": 0.5212852412462234,
|
| 60 |
+
"confident_error_rate": 0.0,
|
| 61 |
+
"score_mae": 0.9293531678523869
|
| 62 |
+
},
|
| 63 |
+
"bbc_news": {
|
| 64 |
+
"count": 200,
|
| 65 |
+
"accuracy": 0.93,
|
| 66 |
+
"brier": 0.15941853896249053,
|
| 67 |
+
"brier_soft": 0.15941853896249053,
|
| 68 |
+
"nll": 0.35049513943385946,
|
| 69 |
+
"nll_soft": 0.35049513943385946,
|
| 70 |
+
"ece": 0.14709680810570713,
|
| 71 |
+
"mean_confidence": 0.7875970338284969,
|
| 72 |
+
"confident_error_rate": 0.0
|
| 73 |
+
},
|
| 74 |
+
"copa": {
|
| 75 |
+
"count": 100,
|
| 76 |
+
"accuracy": 0.68,
|
| 77 |
+
"brier": 0.42515701222347446,
|
| 78 |
+
"brier_soft": 0.42515701222347446,
|
| 79 |
+
"nll": 0.6679799963441176,
|
| 80 |
+
"nll_soft": 0.6679799963441176,
|
| 81 |
+
"ece": 0.11512783467769623,
|
| 82 |
+
"mean_confidence": 0.7648723596334457,
|
| 83 |
+
"confident_error_rate": 0.06
|
| 84 |
+
},
|
| 85 |
+
"dream": {
|
| 86 |
+
"count": 200,
|
| 87 |
+
"accuracy": 0.53,
|
| 88 |
+
"brier": 0.5726855750306872,
|
| 89 |
+
"brier_soft": 0.5726855750306872,
|
| 90 |
+
"nll": 0.9786789739578317,
|
| 91 |
+
"nll_soft": 0.9786789739578317,
|
| 92 |
+
"ece": 0.07479060709476468,
|
| 93 |
+
"mean_confidence": 0.5970340925455093,
|
| 94 |
+
"confident_error_rate": 0.015
|
| 95 |
+
},
|
| 96 |
+
"ethos": {
|
| 97 |
+
"count": 200,
|
| 98 |
+
"accuracy": 0.585,
|
| 99 |
+
"brier": 0.5191951625891887,
|
| 100 |
+
"brier_soft": 0.5191951625891887,
|
| 101 |
+
"nll": 0.7191818728509844,
|
| 102 |
+
"nll_soft": 0.7191818728509844,
|
| 103 |
+
"ece": 0.13361184656620023,
|
| 104 |
+
"mean_confidence": 0.6826644748449325,
|
| 105 |
+
"confident_error_rate": 0.0
|
| 106 |
+
},
|
| 107 |
+
"hh_rlhf": {
|
| 108 |
+
"count": 200,
|
| 109 |
+
"accuracy": 0.585,
|
| 110 |
+
"brier": 0.5130212181519656,
|
| 111 |
+
"brier_soft": 0.5130212181519656,
|
| 112 |
+
"nll": 0.7220155166602013,
|
| 113 |
+
"nll_soft": 0.7220155166602013,
|
| 114 |
+
"ece": 0.11870753884315491,
|
| 115 |
+
"mean_confidence": 0.6868912267684937,
|
| 116 |
+
"confident_error_rate": 0.015
|
| 117 |
+
},
|
| 118 |
+
"medical_questions_pairs": {
|
| 119 |
+
"count": 200,
|
| 120 |
+
"accuracy": 0.74,
|
| 121 |
+
"brier": 0.36060996347062785,
|
| 122 |
+
"brier_soft": 0.36060996347062785,
|
| 123 |
+
"nll": 0.540425253165599,
|
| 124 |
+
"nll_soft": 0.540425253165599,
|
| 125 |
+
"ece": 0.06813470214605334,
|
| 126 |
+
"mean_confidence": 0.7993602934479713,
|
| 127 |
+
"confident_error_rate": 0.025
|
| 128 |
+
},
|
| 129 |
+
"scitail": {
|
| 130 |
+
"count": 200,
|
| 131 |
+
"accuracy": 0.765,
|
| 132 |
+
"brier": 0.3177897540628146,
|
| 133 |
+
"brier_soft": 0.3177897540628146,
|
| 134 |
+
"nll": 0.4915042589281755,
|
| 135 |
+
"nll_soft": 0.4915042589281755,
|
| 136 |
+
"ece": 0.06158805310726166,
|
| 137 |
+
"mean_confidence": 0.7797061520814895,
|
| 138 |
+
"confident_error_rate": 0.025
|
| 139 |
+
},
|
| 140 |
+
"stance_climate": {
|
| 141 |
+
"count": 120,
|
| 142 |
+
"accuracy": 0.6083333333333333,
|
| 143 |
+
"brier": 0.5096979627072162,
|
| 144 |
+
"brier_soft": 0.5096979627072162,
|
| 145 |
+
"nll": 0.8642672541937618,
|
| 146 |
+
"nll_soft": 0.8642672541937618,
|
| 147 |
+
"ece": 0.11114179665843646,
|
| 148 |
+
"mean_confidence": 0.5464859950045745,
|
| 149 |
+
"confident_error_rate": 0.0
|
| 150 |
+
},
|
| 151 |
+
"truthful_qa": {
|
| 152 |
+
"count": 200,
|
| 153 |
+
"accuracy": 0.345,
|
| 154 |
+
"brier": 0.7672775057358159,
|
| 155 |
+
"brier_soft": 0.7672775057358159,
|
| 156 |
+
"nll": 1.4927522285970931,
|
| 157 |
+
"nll_soft": 1.4927522285970931,
|
| 158 |
+
"ece": 0.1707174031436443,
|
| 159 |
+
"mean_confidence": 0.4893866942822933,
|
| 160 |
+
"confident_error_rate": 0.0
|
| 161 |
+
},
|
| 162 |
+
"tweet_irony": {
|
| 163 |
+
"count": 200,
|
| 164 |
+
"accuracy": 0.51,
|
| 165 |
+
"brier": 0.5784815627586128,
|
| 166 |
+
"brier_soft": 0.5784815627586128,
|
| 167 |
+
"nll": 0.786456494692578,
|
| 168 |
+
"nll_soft": 0.786456494692578,
|
| 169 |
+
"ece": 0.16936717092990874,
|
| 170 |
+
"mean_confidence": 0.6793671709299087,
|
| 171 |
+
"confident_error_rate": 0.0
|
| 172 |
+
}
|
| 173 |
+
},
|
| 174 |
+
"reliability": [
|
| 175 |
+
{
|
| 176 |
+
"bin": "0.0-0.1",
|
| 177 |
+
"count": 0,
|
| 178 |
+
"confidence": NaN,
|
| 179 |
+
"accuracy": NaN
|
| 180 |
+
},
|
| 181 |
+
{
|
| 182 |
+
"bin": "0.1-0.2",
|
| 183 |
+
"count": 0,
|
| 184 |
+
"confidence": NaN,
|
| 185 |
+
"accuracy": NaN
|
| 186 |
+
},
|
| 187 |
+
{
|
| 188 |
+
"bin": "0.2-0.3",
|
| 189 |
+
"count": 7,
|
| 190 |
+
"confidence": 0.2856033870152065,
|
| 191 |
+
"accuracy": 0.5714285714285714
|
| 192 |
+
},
|
| 193 |
+
{
|
| 194 |
+
"bin": "0.3-0.4",
|
| 195 |
+
"count": 116,
|
| 196 |
+
"confidence": 0.362707053022138,
|
| 197 |
+
"accuracy": 0.29310344827586204
|
| 198 |
+
},
|
| 199 |
+
{
|
| 200 |
+
"bin": "0.4-0.5",
|
| 201 |
+
"count": 237,
|
| 202 |
+
"confidence": 0.45483410962020293,
|
| 203 |
+
"accuracy": 0.35864978902953587
|
| 204 |
+
},
|
| 205 |
+
{
|
| 206 |
+
"bin": "0.5-0.6",
|
| 207 |
+
"count": 387,
|
| 208 |
+
"confidence": 0.5477614294958977,
|
| 209 |
+
"accuracy": 0.5167958656330749
|
| 210 |
+
},
|
| 211 |
+
{
|
| 212 |
+
"bin": "0.6-0.7",
|
| 213 |
+
"count": 406,
|
| 214 |
+
"confidence": 0.6488185705516138,
|
| 215 |
+
"accuracy": 0.5985221674876847
|
| 216 |
+
},
|
| 217 |
+
{
|
| 218 |
+
"bin": "0.7-0.8",
|
| 219 |
+
"count": 375,
|
| 220 |
+
"confidence": 0.7460334270795186,
|
| 221 |
+
"accuracy": 0.6666666666666666
|
| 222 |
+
},
|
| 223 |
+
{
|
| 224 |
+
"bin": "0.8-0.9",
|
| 225 |
+
"count": 251,
|
| 226 |
+
"confidence": 0.8485566009563279,
|
| 227 |
+
"accuracy": 0.7569721115537849
|
| 228 |
+
},
|
| 229 |
+
{
|
| 230 |
+
"bin": "0.9-1.0",
|
| 231 |
+
"count": 241,
|
| 232 |
+
"confidence": 0.9406674301475905,
|
| 233 |
+
"accuracy": 0.9087136929460581
|
| 234 |
+
}
|
| 235 |
+
]
|
| 236 |
+
}
|
eval-holdout.json
ADDED
|
@@ -0,0 +1,236 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"overall": {
|
| 3 |
+
"count": 2020,
|
| 4 |
+
"accuracy": 0.6064356435643564,
|
| 5 |
+
"brier": 0.5224825083084514,
|
| 6 |
+
"brier_soft": 0.5224825083084514,
|
| 7 |
+
"nll": 0.8948358725588946,
|
| 8 |
+
"nll_soft": 0.8948358725588946,
|
| 9 |
+
"ece": 0.13347138244621828,
|
| 10 |
+
"mean_confidence": 0.7393643843508003,
|
| 11 |
+
"confident_error_rate": 0.034653465346534656,
|
| 12 |
+
"score_mae": 0.8756184922011744
|
| 13 |
+
},
|
| 14 |
+
"by_type": {
|
| 15 |
+
"bool": {
|
| 16 |
+
"count": 800,
|
| 17 |
+
"accuracy": 0.65,
|
| 18 |
+
"brier": 0.47910808172807956,
|
| 19 |
+
"brier_soft": 0.47910808172807956,
|
| 20 |
+
"nll": 0.7023255436185866,
|
| 21 |
+
"nll_soft": 0.7023255436185866,
|
| 22 |
+
"ece": 0.15013569466769694,
|
| 23 |
+
"mean_confidence": 0.800135694667697,
|
| 24 |
+
"confident_error_rate": 0.04875
|
| 25 |
+
},
|
| 26 |
+
"choice": {
|
| 27 |
+
"count": 1020,
|
| 28 |
+
"accuracy": 0.6068627450980392,
|
| 29 |
+
"brier": 0.5150339837721462,
|
| 30 |
+
"brier_soft": 0.5150339837721462,
|
| 31 |
+
"nll": 0.9276425370444942,
|
| 32 |
+
"nll_soft": 0.9276425370444942,
|
| 33 |
+
"ece": 0.1115920229285371,
|
| 34 |
+
"mean_confidence": 0.7173801247395721,
|
| 35 |
+
"confident_error_rate": 0.030392156862745098
|
| 36 |
+
},
|
| 37 |
+
"score": {
|
| 38 |
+
"count": 200,
|
| 39 |
+
"accuracy": 0.43,
|
| 40 |
+
"brier": 0.7339676897650955,
|
| 41 |
+
"brier_soft": 0.7339676897650955,
|
| 42 |
+
"nll": 1.4975631994435707,
|
| 43 |
+
"nll_soft": 1.4975631994435707,
|
| 44 |
+
"ece": 0.18253387078642844,
|
| 45 |
+
"mean_confidence": 0.6083988671004772,
|
| 46 |
+
"confident_error_rate": 0.0,
|
| 47 |
+
"score_mae": 0.8756184922011744
|
| 48 |
+
}
|
| 49 |
+
},
|
| 50 |
+
"by_source": {
|
| 51 |
+
"app_reviews": {
|
| 52 |
+
"count": 200,
|
| 53 |
+
"accuracy": 0.43,
|
| 54 |
+
"brier": 0.7339676897650955,
|
| 55 |
+
"brier_soft": 0.7339676897650955,
|
| 56 |
+
"nll": 1.4975631994435707,
|
| 57 |
+
"nll_soft": 1.4975631994435707,
|
| 58 |
+
"ece": 0.18253387078642844,
|
| 59 |
+
"mean_confidence": 0.6083988671004772,
|
| 60 |
+
"confident_error_rate": 0.0,
|
| 61 |
+
"score_mae": 0.8756184922011744
|
| 62 |
+
},
|
| 63 |
+
"bbc_news": {
|
| 64 |
+
"count": 200,
|
| 65 |
+
"accuracy": 0.93,
|
| 66 |
+
"brier": 0.1307199356192135,
|
| 67 |
+
"brier_soft": 0.1307199356192135,
|
| 68 |
+
"nll": 0.2640486940789302,
|
| 69 |
+
"nll_soft": 0.2640486940789302,
|
| 70 |
+
"ece": 0.06499865755438806,
|
| 71 |
+
"mean_confidence": 0.878423298150301,
|
| 72 |
+
"confident_error_rate": 0.015
|
| 73 |
+
},
|
| 74 |
+
"copa": {
|
| 75 |
+
"count": 100,
|
| 76 |
+
"accuracy": 0.68,
|
| 77 |
+
"brier": 0.44491251853131536,
|
| 78 |
+
"brier_soft": 0.44491251853131536,
|
| 79 |
+
"nll": 0.7816875434135104,
|
| 80 |
+
"nll_soft": 0.7816875434135104,
|
| 81 |
+
"ece": 0.14029216766357422,
|
| 82 |
+
"mean_confidence": 0.8202921676635743,
|
| 83 |
+
"confident_error_rate": 0.09
|
| 84 |
+
},
|
| 85 |
+
"dream": {
|
| 86 |
+
"count": 200,
|
| 87 |
+
"accuracy": 0.53,
|
| 88 |
+
"brier": 0.599144061995857,
|
| 89 |
+
"brier_soft": 0.599144061995857,
|
| 90 |
+
"nll": 1.0644296701169425,
|
| 91 |
+
"nll_soft": 1.0644296701169425,
|
| 92 |
+
"ece": 0.15090002492070198,
|
| 93 |
+
"mean_confidence": 0.6753140194714069,
|
| 94 |
+
"confident_error_rate": 0.03
|
| 95 |
+
},
|
| 96 |
+
"ethos": {
|
| 97 |
+
"count": 200,
|
| 98 |
+
"accuracy": 0.585,
|
| 99 |
+
"brier": 0.5628972085544448,
|
| 100 |
+
"brier_soft": 0.5628972085544448,
|
| 101 |
+
"nll": 0.7880040694867599,
|
| 102 |
+
"nll_soft": 0.7880040694867599,
|
| 103 |
+
"ece": 0.19213538110256195,
|
| 104 |
+
"mean_confidence": 0.7493969035148621,
|
| 105 |
+
"confident_error_rate": 0.015
|
| 106 |
+
},
|
| 107 |
+
"hh_rlhf": {
|
| 108 |
+
"count": 200,
|
| 109 |
+
"accuracy": 0.585,
|
| 110 |
+
"brier": 0.5501808351244621,
|
| 111 |
+
"brier_soft": 0.5501808351244621,
|
| 112 |
+
"nll": 0.8064562372480033,
|
| 113 |
+
"nll_soft": 0.8064562372480033,
|
| 114 |
+
"ece": 0.18701040744781497,
|
| 115 |
+
"mean_confidence": 0.7434734922647476,
|
| 116 |
+
"confident_error_rate": 0.065
|
| 117 |
+
},
|
| 118 |
+
"medical_questions_pairs": {
|
| 119 |
+
"count": 200,
|
| 120 |
+
"accuracy": 0.74,
|
| 121 |
+
"brier": 0.3872718766529277,
|
| 122 |
+
"brier_soft": 0.3872718766529277,
|
| 123 |
+
"nll": 0.6090884104645542,
|
| 124 |
+
"nll_soft": 0.6090884104645542,
|
| 125 |
+
"ece": 0.12405517578125003,
|
| 126 |
+
"mean_confidence": 0.86405517578125,
|
| 127 |
+
"confident_error_rate": 0.075
|
| 128 |
+
},
|
| 129 |
+
"scitail": {
|
| 130 |
+
"count": 200,
|
| 131 |
+
"accuracy": 0.765,
|
| 132 |
+
"brier": 0.32537356296906417,
|
| 133 |
+
"brier_soft": 0.32537356296906417,
|
| 134 |
+
"nll": 0.523669747151008,
|
| 135 |
+
"nll_soft": 0.523669747151008,
|
| 136 |
+
"ece": 0.09302760273218158,
|
| 137 |
+
"mean_confidence": 0.8432374873757362,
|
| 138 |
+
"confident_error_rate": 0.055
|
| 139 |
+
},
|
| 140 |
+
"stance_climate": {
|
| 141 |
+
"count": 120,
|
| 142 |
+
"accuracy": 0.6083333333333333,
|
| 143 |
+
"brier": 0.5024844279880073,
|
| 144 |
+
"brier_soft": 0.5024844279880073,
|
| 145 |
+
"nll": 0.8529358025315834,
|
| 146 |
+
"nll_soft": 0.8529358025315834,
|
| 147 |
+
"ece": 0.0858766761918863,
|
| 148 |
+
"mean_confidence": 0.6247444900373618,
|
| 149 |
+
"confident_error_rate": 0.0
|
| 150 |
+
},
|
| 151 |
+
"truthful_qa": {
|
| 152 |
+
"count": 200,
|
| 153 |
+
"accuracy": 0.345,
|
| 154 |
+
"brier": 0.8226815684399508,
|
| 155 |
+
"brier_soft": 0.8226815684399508,
|
| 156 |
+
"nll": 1.6934370842573383,
|
| 157 |
+
"nll_soft": 1.6934370842573383,
|
| 158 |
+
"ece": 0.23757660880684853,
|
| 159 |
+
"mean_confidence": 0.5764350484311581,
|
| 160 |
+
"confident_error_rate": 0.0
|
| 161 |
+
},
|
| 162 |
+
"tweet_irony": {
|
| 163 |
+
"count": 200,
|
| 164 |
+
"accuracy": 0.51,
|
| 165 |
+
"brier": 0.6408896787358814,
|
| 166 |
+
"brier_soft": 0.6408896787358814,
|
| 167 |
+
"nll": 0.8885399473720241,
|
| 168 |
+
"nll_soft": 0.8885399473720241,
|
| 169 |
+
"ece": 0.23385321199893952,
|
| 170 |
+
"mean_confidence": 0.7438532119989395,
|
| 171 |
+
"confident_error_rate": 0.05
|
| 172 |
+
}
|
| 173 |
+
},
|
| 174 |
+
"reliability": [
|
| 175 |
+
{
|
| 176 |
+
"bin": "0.0-0.1",
|
| 177 |
+
"count": 0,
|
| 178 |
+
"confidence": NaN,
|
| 179 |
+
"accuracy": NaN
|
| 180 |
+
},
|
| 181 |
+
{
|
| 182 |
+
"bin": "0.1-0.2",
|
| 183 |
+
"count": 0,
|
| 184 |
+
"confidence": NaN,
|
| 185 |
+
"accuracy": NaN
|
| 186 |
+
},
|
| 187 |
+
{
|
| 188 |
+
"bin": "0.2-0.3",
|
| 189 |
+
"count": 3,
|
| 190 |
+
"confidence": 0.28737815221150714,
|
| 191 |
+
"accuracy": 0.6666666666666666
|
| 192 |
+
},
|
| 193 |
+
{
|
| 194 |
+
"bin": "0.3-0.4",
|
| 195 |
+
"count": 48,
|
| 196 |
+
"confidence": 0.36278260375062626,
|
| 197 |
+
"accuracy": 0.22916666666666666
|
| 198 |
+
},
|
| 199 |
+
{
|
| 200 |
+
"bin": "0.4-0.5",
|
| 201 |
+
"count": 146,
|
| 202 |
+
"confidence": 0.4593193284044527,
|
| 203 |
+
"accuracy": 0.3493150684931507
|
| 204 |
+
},
|
| 205 |
+
{
|
| 206 |
+
"bin": "0.5-0.6",
|
| 207 |
+
"count": 334,
|
| 208 |
+
"confidence": 0.5477860705581253,
|
| 209 |
+
"accuracy": 0.437125748502994
|
| 210 |
+
},
|
| 211 |
+
{
|
| 212 |
+
"bin": "0.6-0.7",
|
| 213 |
+
"count": 283,
|
| 214 |
+
"confidence": 0.6516003400192665,
|
| 215 |
+
"accuracy": 0.5265017667844523
|
| 216 |
+
},
|
| 217 |
+
{
|
| 218 |
+
"bin": "0.7-0.8",
|
| 219 |
+
"count": 366,
|
| 220 |
+
"confidence": 0.7497753351112533,
|
| 221 |
+
"accuracy": 0.6010928961748634
|
| 222 |
+
},
|
| 223 |
+
{
|
| 224 |
+
"bin": "0.8-0.9",
|
| 225 |
+
"count": 363,
|
| 226 |
+
"confidence": 0.8485710975552393,
|
| 227 |
+
"accuracy": 0.6584022038567493
|
| 228 |
+
},
|
| 229 |
+
{
|
| 230 |
+
"bin": "0.9-1.0",
|
| 231 |
+
"count": 477,
|
| 232 |
+
"confidence": 0.9609375474837841,
|
| 233 |
+
"accuracy": 0.8532494758909853
|
| 234 |
+
}
|
| 235 |
+
]
|
| 236 |
+
}
|
eval-transfer-v4-scaled.json
ADDED
|
@@ -0,0 +1,236 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"overall": {
|
| 3 |
+
"count": 764,
|
| 4 |
+
"accuracy": 0.5418848167539267,
|
| 5 |
+
"brier": 0.5723337508379324,
|
| 6 |
+
"brier_soft": 0.5723337508379324,
|
| 7 |
+
"nll": 0.9521210937084827,
|
| 8 |
+
"nll_soft": 0.9521210937084827,
|
| 9 |
+
"ece": 0.09497049852659566,
|
| 10 |
+
"mean_confidence": 0.6368553152805223,
|
| 11 |
+
"confident_error_rate": 0.011780104712041885,
|
| 12 |
+
"score_mae": 0.6214548164978624
|
| 13 |
+
},
|
| 14 |
+
"by_type": {
|
| 15 |
+
"bool": {
|
| 16 |
+
"count": 280,
|
| 17 |
+
"accuracy": 0.5642857142857143,
|
| 18 |
+
"brier": 0.5727095106013428,
|
| 19 |
+
"brier_soft": 0.5727095106013428,
|
| 20 |
+
"nll": 0.8008714702711881,
|
| 21 |
+
"nll_soft": 0.8008714702711881,
|
| 22 |
+
"ece": 0.220761573101793,
|
| 23 |
+
"mean_confidence": 0.7850472873875073,
|
| 24 |
+
"confident_error_rate": 0.03214285714285714
|
| 25 |
+
},
|
| 26 |
+
"choice": {
|
| 27 |
+
"count": 444,
|
| 28 |
+
"accuracy": 0.5202702702702703,
|
| 29 |
+
"brier": 0.5732227170620644,
|
| 30 |
+
"brier_soft": 0.5732227170620644,
|
| 31 |
+
"nll": 1.0510425110755313,
|
| 32 |
+
"nll_soft": 1.0510425110755313,
|
| 33 |
+
"ece": 0.04932732282726613,
|
| 34 |
+
"mean_confidence": 0.5536865257062353,
|
| 35 |
+
"confident_error_rate": 0.0
|
| 36 |
+
},
|
| 37 |
+
"score": {
|
| 38 |
+
"count": 40,
|
| 39 |
+
"accuracy": 0.625,
|
| 40 |
+
"brier": 0.5598359074061938,
|
| 41 |
+
"brier_soft": 0.5598359074061938,
|
| 42 |
+
"nll": 0.9128407249953066,
|
| 43 |
+
"nll_soft": 0.9128407249953066,
|
| 44 |
+
"ece": 0.16345201134681706,
|
| 45 |
+
"mean_confidence": 0.5226850748062134,
|
| 46 |
+
"confident_error_rate": 0.0,
|
| 47 |
+
"score_mae": 0.6214548164978624
|
| 48 |
+
}
|
| 49 |
+
},
|
| 50 |
+
"by_source": {
|
| 51 |
+
"composition_held_and_or": {
|
| 52 |
+
"count": 32,
|
| 53 |
+
"accuracy": 0.65625,
|
| 54 |
+
"brier": 0.49351249809077513,
|
| 55 |
+
"brier_soft": 0.49351249809077513,
|
| 56 |
+
"nll": 0.6933454597419972,
|
| 57 |
+
"nll_soft": 0.6933454597419972,
|
| 58 |
+
"ece": 0.2834099940955639,
|
| 59 |
+
"mean_confidence": 0.6526073180139065,
|
| 60 |
+
"confident_error_rate": 0.0
|
| 61 |
+
},
|
| 62 |
+
"composition_held_conditional": {
|
| 63 |
+
"count": 32,
|
| 64 |
+
"accuracy": 0.46875,
|
| 65 |
+
"brier": 0.506540283804593,
|
| 66 |
+
"brier_soft": 0.506540283804593,
|
| 67 |
+
"nll": 0.699469018142885,
|
| 68 |
+
"nll_soft": 0.699469018142885,
|
| 69 |
+
"ece": 0.13315994106233123,
|
| 70 |
+
"mean_confidence": 0.5767759811133146,
|
| 71 |
+
"confident_error_rate": 0.0
|
| 72 |
+
},
|
| 73 |
+
"composition_held_or_not": {
|
| 74 |
+
"count": 32,
|
| 75 |
+
"accuracy": 0.5625,
|
| 76 |
+
"brier": 0.5283131762316139,
|
| 77 |
+
"brier_soft": 0.5283131762316139,
|
| 78 |
+
"nll": 0.7258296255525061,
|
| 79 |
+
"nll_soft": 0.7258296255525061,
|
| 80 |
+
"ece": 0.18242479674518106,
|
| 81 |
+
"mean_confidence": 0.6838499922305346,
|
| 82 |
+
"confident_error_rate": 0.0
|
| 83 |
+
},
|
| 84 |
+
"contrastive_authorization": {
|
| 85 |
+
"count": 40,
|
| 86 |
+
"accuracy": 0.5,
|
| 87 |
+
"brier": 0.7192700362744405,
|
| 88 |
+
"brier_soft": 0.7192700362744405,
|
| 89 |
+
"nll": 0.9979148090583211,
|
| 90 |
+
"nll_soft": 0.9979148090583211,
|
| 91 |
+
"ece": 0.3941640496253967,
|
| 92 |
+
"mean_confidence": 0.8941640496253968,
|
| 93 |
+
"confident_error_rate": 0.075
|
| 94 |
+
},
|
| 95 |
+
"contrastive_deadline": {
|
| 96 |
+
"count": 40,
|
| 97 |
+
"accuracy": 0.625,
|
| 98 |
+
"brier": 0.5598359074061938,
|
| 99 |
+
"brier_soft": 0.5598359074061938,
|
| 100 |
+
"nll": 0.9128407249953066,
|
| 101 |
+
"nll_soft": 0.9128407249953066,
|
| 102 |
+
"ece": 0.16345201134681706,
|
| 103 |
+
"mean_confidence": 0.5226850748062134,
|
| 104 |
+
"confident_error_rate": 0.0,
|
| 105 |
+
"score_mae": 0.6214548164978624
|
| 106 |
+
},
|
| 107 |
+
"emotion": {
|
| 108 |
+
"count": 116,
|
| 109 |
+
"accuracy": 0.5172413793103449,
|
| 110 |
+
"brier": 0.610789513046154,
|
| 111 |
+
"brier_soft": 0.610789513046154,
|
| 112 |
+
"nll": 1.265430453215189,
|
| 113 |
+
"nll_soft": 1.265430453215189,
|
| 114 |
+
"ece": 0.10132859647274016,
|
| 115 |
+
"mean_confidence": 0.5219097122036177,
|
| 116 |
+
"confident_error_rate": 0.0
|
| 117 |
+
},
|
| 118 |
+
"mmlu": {
|
| 119 |
+
"count": 116,
|
| 120 |
+
"accuracy": 0.25,
|
| 121 |
+
"brier": 0.7813248350418833,
|
| 122 |
+
"brier_soft": 0.7813248350418833,
|
| 123 |
+
"nll": 1.4357287683156117,
|
| 124 |
+
"nll_soft": 1.4357287683156117,
|
| 125 |
+
"ece": 0.20643681441915449,
|
| 126 |
+
"mean_confidence": 0.4393831840876875,
|
| 127 |
+
"confident_error_rate": 0.0
|
| 128 |
+
},
|
| 129 |
+
"paws": {
|
| 130 |
+
"count": 80,
|
| 131 |
+
"accuracy": 0.5875,
|
| 132 |
+
"brier": 0.5666056772075379,
|
| 133 |
+
"brier_soft": 0.5666056772075379,
|
| 134 |
+
"nll": 0.7898064636336843,
|
| 135 |
+
"nll_soft": 0.7898064636336843,
|
| 136 |
+
"ece": 0.18403902053833004,
|
| 137 |
+
"mean_confidence": 0.7078988596796989,
|
| 138 |
+
"confident_error_rate": 0.0125
|
| 139 |
+
},
|
| 140 |
+
"qnli": {
|
| 141 |
+
"count": 80,
|
| 142 |
+
"accuracy": 0.4625,
|
| 143 |
+
"brier": 0.7283658121162533,
|
| 144 |
+
"brier_soft": 0.7283658121162533,
|
| 145 |
+
"nll": 0.9991395666696501,
|
| 146 |
+
"nll_soft": 0.9991395666696501,
|
| 147 |
+
"ece": 0.3315258145332336,
|
| 148 |
+
"mean_confidence": 0.7940258145332336,
|
| 149 |
+
"confident_error_rate": 0.0375
|
| 150 |
+
},
|
| 151 |
+
"sciq": {
|
| 152 |
+
"count": 116,
|
| 153 |
+
"accuracy": 0.7586206896551724,
|
| 154 |
+
"brier": 0.3803268218731105,
|
| 155 |
+
"brier_soft": 0.3803268218731105,
|
| 156 |
+
"nll": 0.7373430509479874,
|
| 157 |
+
"nll_soft": 0.7373430509479874,
|
| 158 |
+
"ece": 0.15309577383871736,
|
| 159 |
+
"mean_confidence": 0.6302015182786974,
|
| 160 |
+
"confident_error_rate": 0.0
|
| 161 |
+
},
|
| 162 |
+
"tweet_offensive": {
|
| 163 |
+
"count": 80,
|
| 164 |
+
"accuracy": 0.675,
|
| 165 |
+
"brier": 0.3498767796436882,
|
| 166 |
+
"brier_soft": 0.3498767796436882,
|
| 167 |
+
"nll": 0.5151467111166632,
|
| 168 |
+
"nll_soft": 0.5151467111166632,
|
| 169 |
+
"ece": 0.1505606211721897,
|
| 170 |
+
"mean_confidence": 0.7986588068306446,
|
| 171 |
+
"confident_error_rate": 0.025
|
| 172 |
+
}
|
| 173 |
+
},
|
| 174 |
+
"reliability": [
|
| 175 |
+
{
|
| 176 |
+
"bin": "0.0-0.1",
|
| 177 |
+
"count": 0,
|
| 178 |
+
"confidence": NaN,
|
| 179 |
+
"accuracy": NaN
|
| 180 |
+
},
|
| 181 |
+
{
|
| 182 |
+
"bin": "0.1-0.2",
|
| 183 |
+
"count": 0,
|
| 184 |
+
"confidence": NaN,
|
| 185 |
+
"accuracy": NaN
|
| 186 |
+
},
|
| 187 |
+
{
|
| 188 |
+
"bin": "0.2-0.3",
|
| 189 |
+
"count": 22,
|
| 190 |
+
"confidence": 0.27215467528863385,
|
| 191 |
+
"accuracy": 0.22727272727272727
|
| 192 |
+
},
|
| 193 |
+
{
|
| 194 |
+
"bin": "0.3-0.4",
|
| 195 |
+
"count": 77,
|
| 196 |
+
"confidence": 0.35835247109462687,
|
| 197 |
+
"accuracy": 0.33766233766233766
|
| 198 |
+
},
|
| 199 |
+
{
|
| 200 |
+
"bin": "0.4-0.5",
|
| 201 |
+
"count": 108,
|
| 202 |
+
"confidence": 0.44573648549892286,
|
| 203 |
+
"accuracy": 0.39814814814814814
|
| 204 |
+
},
|
| 205 |
+
{
|
| 206 |
+
"bin": "0.5-0.6",
|
| 207 |
+
"count": 128,
|
| 208 |
+
"confidence": 0.55029206443578,
|
| 209 |
+
"accuracy": 0.5078125
|
| 210 |
+
},
|
| 211 |
+
{
|
| 212 |
+
"bin": "0.6-0.7",
|
| 213 |
+
"count": 118,
|
| 214 |
+
"confidence": 0.6556886096121901,
|
| 215 |
+
"accuracy": 0.6016949152542372
|
| 216 |
+
},
|
| 217 |
+
{
|
| 218 |
+
"bin": "0.7-0.8",
|
| 219 |
+
"count": 130,
|
| 220 |
+
"confidence": 0.7493144819369683,
|
| 221 |
+
"accuracy": 0.5846153846153846
|
| 222 |
+
},
|
| 223 |
+
{
|
| 224 |
+
"bin": "0.8-0.9",
|
| 225 |
+
"count": 119,
|
| 226 |
+
"confidence": 0.849185519358691,
|
| 227 |
+
"accuracy": 0.6302521008403361
|
| 228 |
+
},
|
| 229 |
+
{
|
| 230 |
+
"bin": "0.9-1.0",
|
| 231 |
+
"count": 62,
|
| 232 |
+
"confidence": 0.9445931882627548,
|
| 233 |
+
"accuracy": 0.8548387096774194
|
| 234 |
+
}
|
| 235 |
+
]
|
| 236 |
+
}
|
eval-transfer-v4.json
ADDED
|
@@ -0,0 +1,236 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"overall": {
|
| 3 |
+
"count": 764,
|
| 4 |
+
"accuracy": 0.5418848167539267,
|
| 5 |
+
"brier": 0.6133018034726817,
|
| 6 |
+
"brier_soft": 0.6133018034726817,
|
| 7 |
+
"nll": 1.0479286045923395,
|
| 8 |
+
"nll_soft": 1.0479286045923395,
|
| 9 |
+
"ece": 0.174342348391473,
|
| 10 |
+
"mean_confidence": 0.7162271651453996,
|
| 11 |
+
"confident_error_rate": 0.06413612565445026,
|
| 12 |
+
"score_mae": 0.5942460499238222
|
| 13 |
+
},
|
| 14 |
+
"by_type": {
|
| 15 |
+
"bool": {
|
| 16 |
+
"count": 280,
|
| 17 |
+
"accuracy": 0.5642857142857143,
|
| 18 |
+
"brier": 0.6481004470987204,
|
| 19 |
+
"brier_soft": 0.6481004470987204,
|
| 20 |
+
"nll": 0.985213847165127,
|
| 21 |
+
"nll_soft": 0.985213847165127,
|
| 22 |
+
"ece": 0.2859063116567475,
|
| 23 |
+
"mean_confidence": 0.8501920259424618,
|
| 24 |
+
"confident_error_rate": 0.15357142857142858
|
| 25 |
+
},
|
| 26 |
+
"choice": {
|
| 27 |
+
"count": 444,
|
| 28 |
+
"accuracy": 0.5202702702702703,
|
| 29 |
+
"brier": 0.5961741873280398,
|
| 30 |
+
"brier_soft": 0.5961741873280398,
|
| 31 |
+
"nll": 1.0991264012213526,
|
| 32 |
+
"nll_soft": 1.0991264012213526,
|
| 33 |
+
"ece": 0.12330419536646421,
|
| 34 |
+
"mean_confidence": 0.6435744656367345,
|
| 35 |
+
"confident_error_rate": 0.01126126126126126
|
| 36 |
+
},
|
| 37 |
+
"score": {
|
| 38 |
+
"count": 40,
|
| 39 |
+
"accuracy": 0.625,
|
| 40 |
+
"brier": 0.5598278372959354,
|
| 41 |
+
"brier_soft": 0.5598278372959354,
|
| 42 |
+
"nll": 0.9186363640007785,
|
| 43 |
+
"nll_soft": 0.9186363640007785,
|
| 44 |
+
"ece": 0.1644253566861153,
|
| 45 |
+
"mean_confidence": 0.5849181041121483,
|
| 46 |
+
"confident_error_rate": 0.025,
|
| 47 |
+
"score_mae": 0.5942460499238222
|
| 48 |
+
}
|
| 49 |
+
},
|
| 50 |
+
"by_source": {
|
| 51 |
+
"composition_held_and_or": {
|
| 52 |
+
"count": 32,
|
| 53 |
+
"accuracy": 0.65625,
|
| 54 |
+
"brier": 0.5151399834441622,
|
| 55 |
+
"brier_soft": 0.5151399834441622,
|
| 56 |
+
"nll": 0.7344772870335554,
|
| 57 |
+
"nll_soft": 0.7344772870335554,
|
| 58 |
+
"ece": 0.2665761932730675,
|
| 59 |
+
"mean_confidence": 0.7128458358347416,
|
| 60 |
+
"confident_error_rate": 0.0
|
| 61 |
+
},
|
| 62 |
+
"composition_held_conditional": {
|
| 63 |
+
"count": 32,
|
| 64 |
+
"accuracy": 0.46875,
|
| 65 |
+
"brier": 0.5195917503921232,
|
| 66 |
+
"brier_soft": 0.5195917503921232,
|
| 67 |
+
"nll": 0.7126327097541655,
|
| 68 |
+
"nll_soft": 0.7126327097541655,
|
| 69 |
+
"ece": 0.15248680673539639,
|
| 70 |
+
"mean_confidence": 0.6122495550662279,
|
| 71 |
+
"confident_error_rate": 0.0
|
| 72 |
+
},
|
| 73 |
+
"composition_held_or_not": {
|
| 74 |
+
"count": 32,
|
| 75 |
+
"accuracy": 0.5625,
|
| 76 |
+
"brier": 0.5787166708146243,
|
| 77 |
+
"brier_soft": 0.5787166708146243,
|
| 78 |
+
"nll": 0.7936390280635442,
|
| 79 |
+
"nll_soft": 0.7936390280635442,
|
| 80 |
+
"ece": 0.22924835607409477,
|
| 81 |
+
"mean_confidence": 0.7555920816957951,
|
| 82 |
+
"confident_error_rate": 0.0
|
| 83 |
+
},
|
| 84 |
+
"contrastive_authorization": {
|
| 85 |
+
"count": 40,
|
| 86 |
+
"accuracy": 0.5,
|
| 87 |
+
"brier": 0.8449352980173306,
|
| 88 |
+
"brier_soft": 0.8449352980173306,
|
| 89 |
+
"nll": 1.36295105631539,
|
| 90 |
+
"nll_soft": 1.36295105631539,
|
| 91 |
+
"ece": 0.45071354508399963,
|
| 92 |
+
"mean_confidence": 0.9507135450839996,
|
| 93 |
+
"confident_error_rate": 0.375
|
| 94 |
+
},
|
| 95 |
+
"contrastive_deadline": {
|
| 96 |
+
"count": 40,
|
| 97 |
+
"accuracy": 0.625,
|
| 98 |
+
"brier": 0.5598278372959354,
|
| 99 |
+
"brier_soft": 0.5598278372959354,
|
| 100 |
+
"nll": 0.9186363640007785,
|
| 101 |
+
"nll_soft": 0.9186363640007785,
|
| 102 |
+
"ece": 0.1644253566861153,
|
| 103 |
+
"mean_confidence": 0.5849181041121483,
|
| 104 |
+
"confident_error_rate": 0.025,
|
| 105 |
+
"score_mae": 0.5942460499238222
|
| 106 |
+
},
|
| 107 |
+
"emotion": {
|
| 108 |
+
"count": 116,
|
| 109 |
+
"accuracy": 0.5172413793103449,
|
| 110 |
+
"brier": 0.6243123781624198,
|
| 111 |
+
"brier_soft": 0.6243123781624198,
|
| 112 |
+
"nll": 1.3241542573245493,
|
| 113 |
+
"nll_soft": 1.3241542573245493,
|
| 114 |
+
"ece": 0.13471242271620654,
|
| 115 |
+
"mean_confidence": 0.6471006520863237,
|
| 116 |
+
"confident_error_rate": 0.034482758620689655
|
| 117 |
+
},
|
| 118 |
+
"mmlu": {
|
| 119 |
+
"count": 116,
|
| 120 |
+
"accuracy": 0.25,
|
| 121 |
+
"brier": 0.8467742786249345,
|
| 122 |
+
"brier_soft": 0.8467742786249345,
|
| 123 |
+
"nll": 1.561685775821729,
|
| 124 |
+
"nll_soft": 1.561685775821729,
|
| 125 |
+
"ece": 0.29809959104348877,
|
| 126 |
+
"mean_confidence": 0.5187388815242668,
|
| 127 |
+
"confident_error_rate": 0.0
|
| 128 |
+
},
|
| 129 |
+
"paws": {
|
| 130 |
+
"count": 80,
|
| 131 |
+
"accuracy": 0.5875,
|
| 132 |
+
"brier": 0.620199979388996,
|
| 133 |
+
"brier_soft": 0.620199979388996,
|
| 134 |
+
"nll": 0.912849759089997,
|
| 135 |
+
"nll_soft": 0.912849759089997,
|
| 136 |
+
"ece": 0.21204856932163235,
|
| 137 |
+
"mean_confidence": 0.7738225489854813,
|
| 138 |
+
"confident_error_rate": 0.1125
|
| 139 |
+
},
|
| 140 |
+
"qnli": {
|
| 141 |
+
"count": 80,
|
| 142 |
+
"accuracy": 0.4625,
|
| 143 |
+
"brier": 0.8453887659091137,
|
| 144 |
+
"brier_soft": 0.8453887659091137,
|
| 145 |
+
"nll": 1.2724308758874976,
|
| 146 |
+
"nll_soft": 1.2724308758874976,
|
| 147 |
+
"ece": 0.41243818104267127,
|
| 148 |
+
"mean_confidence": 0.8749381810426712,
|
| 149 |
+
"confident_error_rate": 0.2
|
| 150 |
+
},
|
| 151 |
+
"sciq": {
|
| 152 |
+
"count": 116,
|
| 153 |
+
"accuracy": 0.7586206896551724,
|
| 154 |
+
"brier": 0.3657322244610988,
|
| 155 |
+
"brier_soft": 0.3657322244610988,
|
| 156 |
+
"nll": 0.7030233577764811,
|
| 157 |
+
"nll_soft": 0.7030233577764811,
|
| 158 |
+
"ece": 0.10112244687203706,
|
| 159 |
+
"mean_confidence": 0.7235144631102167,
|
| 160 |
+
"confident_error_rate": 0.008620689655172414
|
| 161 |
+
},
|
| 162 |
+
"tweet_offensive": {
|
| 163 |
+
"count": 80,
|
| 164 |
+
"accuracy": 0.675,
|
| 165 |
+
"brier": 0.3802951705387462,
|
| 166 |
+
"brier_soft": 0.3802951705387462,
|
| 167 |
+
"nll": 0.5814923019427549,
|
| 168 |
+
"nll_soft": 0.5814923019427549,
|
| 169 |
+
"ece": 0.2005698867142201,
|
| 170 |
+
"mean_confidence": 0.8515545882284641,
|
| 171 |
+
"confident_error_rate": 0.0375
|
| 172 |
+
}
|
| 173 |
+
},
|
| 174 |
+
"reliability": [
|
| 175 |
+
{
|
| 176 |
+
"bin": "0.0-0.1",
|
| 177 |
+
"count": 0,
|
| 178 |
+
"confidence": NaN,
|
| 179 |
+
"accuracy": NaN
|
| 180 |
+
},
|
| 181 |
+
{
|
| 182 |
+
"bin": "0.1-0.2",
|
| 183 |
+
"count": 0,
|
| 184 |
+
"confidence": NaN,
|
| 185 |
+
"accuracy": NaN
|
| 186 |
+
},
|
| 187 |
+
{
|
| 188 |
+
"bin": "0.2-0.3",
|
| 189 |
+
"count": 10,
|
| 190 |
+
"confidence": 0.28239678144454955,
|
| 191 |
+
"accuracy": 0.3
|
| 192 |
+
},
|
| 193 |
+
{
|
| 194 |
+
"bin": "0.3-0.4",
|
| 195 |
+
"count": 42,
|
| 196 |
+
"confidence": 0.3649950275818507,
|
| 197 |
+
"accuracy": 0.2857142857142857
|
| 198 |
+
},
|
| 199 |
+
{
|
| 200 |
+
"bin": "0.4-0.5",
|
| 201 |
+
"count": 78,
|
| 202 |
+
"confidence": 0.4485609558148262,
|
| 203 |
+
"accuracy": 0.3333333333333333
|
| 204 |
+
},
|
| 205 |
+
{
|
| 206 |
+
"bin": "0.5-0.6",
|
| 207 |
+
"count": 111,
|
| 208 |
+
"confidence": 0.5502428761473647,
|
| 209 |
+
"accuracy": 0.45045045045045046
|
| 210 |
+
},
|
| 211 |
+
{
|
| 212 |
+
"bin": "0.6-0.7",
|
| 213 |
+
"count": 96,
|
| 214 |
+
"confidence": 0.6455884600679079,
|
| 215 |
+
"accuracy": 0.5208333333333334
|
| 216 |
+
},
|
| 217 |
+
{
|
| 218 |
+
"bin": "0.7-0.8",
|
| 219 |
+
"count": 112,
|
| 220 |
+
"confidence": 0.7568075236465249,
|
| 221 |
+
"accuracy": 0.6428571428571429
|
| 222 |
+
},
|
| 223 |
+
{
|
| 224 |
+
"bin": "0.8-0.9",
|
| 225 |
+
"count": 137,
|
| 226 |
+
"confidence": 0.8501275046898501,
|
| 227 |
+
"accuracy": 0.5255474452554745
|
| 228 |
+
},
|
| 229 |
+
{
|
| 230 |
+
"bin": "0.9-1.0",
|
| 231 |
+
"count": 178,
|
| 232 |
+
"confidence": 0.9537790923306112,
|
| 233 |
+
"accuracy": 0.7247191011235955
|
| 234 |
+
}
|
| 235 |
+
]
|
| 236 |
+
}
|
latency-cpu.txt
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
runs/compiled-late-gte-base on cpu, 8 threads
|
| 2 |
+
questions encode_state_ms compile_ms decide_ms end_to_end_ms
|
| 3 |
+
1 28.95 29.63 1.381 62.25
|
| 4 |
+
3 30.67 61.52 1.753 94.48
|
| 5 |
+
6 30.64 88.46 2.844 122.80
|
| 6 |
+
12 30.51 88.45 4.867 124.75
|
| 7 |
+
24 30.64 88.42 10.694 131.05
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c13230e2039958c8755d8660145256bb5cd8a3c3e8bb9cea48028ef7c281266d
|
| 3 |
+
size 596070136
|
tokenizer.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"backend": "tokenizers",
|
| 3 |
+
"clean_up_tokenization_spaces": true,
|
| 4 |
+
"cls_token": "[CLS]",
|
| 5 |
+
"is_local": false,
|
| 6 |
+
"local_files_only": false,
|
| 7 |
+
"mask_token": "[MASK]",
|
| 8 |
+
"model_input_names": [
|
| 9 |
+
"input_ids",
|
| 10 |
+
"attention_mask"
|
| 11 |
+
],
|
| 12 |
+
"model_max_length": 1000000000000000019884624838656,
|
| 13 |
+
"pad_token": "[PAD]",
|
| 14 |
+
"sep_token": "[SEP]",
|
| 15 |
+
"tokenizer_class": "TokenizersBackend",
|
| 16 |
+
"unk_token": "[UNK]"
|
| 17 |
+
}
|
train_args.json
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"encoder": "Alibaba-NLP/gte-modernbert-base",
|
| 3 |
+
"data": "data/v4",
|
| 4 |
+
"extra": [
|
| 5 |
+
"data/distill/generic.jsonl"
|
| 6 |
+
],
|
| 7 |
+
"out": "runs/compiled-late-gte-base",
|
| 8 |
+
"lr": 5e-05,
|
| 9 |
+
"head_lr": 0.0005,
|
| 10 |
+
"epochs": 3.0,
|
| 11 |
+
"batch_size": 32,
|
| 12 |
+
"warmup": 0.05,
|
| 13 |
+
"evidence_weight": 0.5,
|
| 14 |
+
"score_sigma": 0.5,
|
| 15 |
+
"hard_targets": false,
|
| 16 |
+
"max_state_tokens": 512,
|
| 17 |
+
"slots": 8,
|
| 18 |
+
"limit": null,
|
| 19 |
+
"log_every": 50,
|
| 20 |
+
"seed": 0,
|
| 21 |
+
"zero_shot_only": false,
|
| 22 |
+
"arch": "compiled",
|
| 23 |
+
"pair_budget": 256,
|
| 24 |
+
"checkpoint_every": 500,
|
| 25 |
+
"late_interaction": true
|
| 26 |
+
}
|
train_log.jsonl
ADDED
|
@@ -0,0 +1,260 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"step": 50, "loss": 1.2884845590591432, "answer_loss": 1.1251170027256012, "evidence_loss": 0.32673510106280446, "lr": 3.923076923076923e-06, "elapsed": 9.29838252067566}
|
| 2 |
+
{"step": 100, "loss": 1.1156296527385712, "answer_loss": 1.0820879101753236, "evidence_loss": 0.0670834807259962, "lr": 7.76923076923077e-06, "elapsed": 17.674908876419067}
|
| 3 |
+
{"step": 150, "loss": 1.1329371440410614, "answer_loss": 1.0969944894313812, "evidence_loss": 0.07188530978746713, "lr": 1.1615384615384615e-05, "elapsed": 25.68803858757019}
|
| 4 |
+
{"step": 200, "loss": 1.0738985967636108, "answer_loss": 1.0471647822856902, "evidence_loss": 0.05346761893946678, "lr": 1.5461538461538463e-05, "elapsed": 34.32280516624451}
|
| 5 |
+
{"step": 250, "loss": 1.120739164352417, "answer_loss": 1.0878141582012177, "evidence_loss": 0.06585000927560031, "lr": 1.930769230769231e-05, "elapsed": 42.80049252510071}
|
| 6 |
+
{"step": 300, "loss": 1.0595968925952912, "answer_loss": 1.0226578760147094, "evidence_loss": 0.0738780245417729, "lr": 2.3153846153846155e-05, "elapsed": 50.606215715408325}
|
| 7 |
+
{"step": 350, "loss": 1.0033678221702576, "answer_loss": 0.9778803408145904, "evidence_loss": 0.050974962515756485, "lr": 2.7000000000000002e-05, "elapsed": 59.27379298210144}
|
| 8 |
+
{"step": 400, "loss": 0.9981056678295136, "answer_loss": 0.9676726853847504, "evidence_loss": 0.06086596081033349, "lr": 3.084615384615385e-05, "elapsed": 67.84155178070068}
|
| 9 |
+
{"step": 450, "loss": 1.016112732887268, "answer_loss": 0.9797139513492584, "evidence_loss": 0.07279754600429442, "lr": 3.4692307692307694e-05, "elapsed": 76.10419368743896}
|
| 10 |
+
{"step": 500, "loss": 0.9915405964851379, "answer_loss": 0.969505797624588, "evidence_loss": 0.04406959896441549, "lr": 3.853846153846154e-05, "elapsed": 83.93473505973816}
|
| 11 |
+
{"step": 550, "loss": 1.0831311225891114, "answer_loss": 1.0485105645656585, "evidence_loss": 0.06924112281005364, "lr": 4.238461538461539e-05, "elapsed": 92.45551180839539}
|
| 12 |
+
{"step": 600, "loss": 1.0216360557079316, "answer_loss": 0.9769752383232116, "evidence_loss": 0.0893216341570951, "lr": 4.6230769230769234e-05, "elapsed": 100.66771006584167}
|
| 13 |
+
{"step": 650, "loss": 1.0392094194889068, "answer_loss": 0.9997532331943512, "evidence_loss": 0.07891237194649875, "lr": 5e-05, "elapsed": 108.74386048316956}
|
| 14 |
+
{"step": 700, "loss": 1.0341204440593719, "answer_loss": 1.0043608272075653, "evidence_loss": 0.05951923039741814, "lr": 4.9997981789575124e-05, "elapsed": 116.38171172142029}
|
| 15 |
+
{"step": 750, "loss": 1.0025508987903595, "answer_loss": 0.9592167580127716, "evidence_loss": 0.08666828506626188, "lr": 4.9991927484154335e-05, "elapsed": 124.8218150138855}
|
| 16 |
+
{"step": 800, "loss": 0.9574149918556213, "answer_loss": 0.9286100542545319, "evidence_loss": 0.05760985680361046, "lr": 4.998183806124662e-05, "elapsed": 131.8896496295929}
|
| 17 |
+
{"step": 850, "loss": 0.9664371049404145, "answer_loss": 0.9432679069042206, "evidence_loss": 0.04633838607463986, "lr": 4.9967715149858276e-05, "elapsed": 140.01883959770203}
|
| 18 |
+
{"step": 900, "loss": 0.9541859269142151, "answer_loss": 0.920410658121109, "evidence_loss": 0.06755053255939857, "lr": 4.9949561030229854e-05, "elapsed": 148.63411378860474}
|
| 19 |
+
{"step": 950, "loss": 1.023260967731476, "answer_loss": 0.9973842430114747, "evidence_loss": 0.05175344150979072, "lr": 4.992737863346802e-05, "elapsed": 157.31181836128235}
|
| 20 |
+
{"step": 1000, "loss": 0.9633753377199173, "answer_loss": 0.9345685118436813, "evidence_loss": 0.0576136495382525, "lr": 4.990117154107236e-05, "elapsed": 164.97056078910828}
|
| 21 |
+
{"step": 1050, "loss": 0.9810221654176712, "answer_loss": 0.921577222943306, "evidence_loss": 0.11888988142949529, "lr": 4.9870943984357e-05, "elapsed": 174.8191373348236}
|
| 22 |
+
{"step": 1100, "loss": 0.948986713886261, "answer_loss": 0.9281677401065827, "evidence_loss": 0.041637955587357285, "lr": 4.983670084376757e-05, "elapsed": 183.5345537662506}
|
| 23 |
+
{"step": 1150, "loss": 0.9653151738643646, "answer_loss": 0.9190999829769134, "evidence_loss": 0.09243038590764627, "lr": 4.979844764809313e-05, "elapsed": 191.55087518692017}
|
| 24 |
+
{"step": 1200, "loss": 0.983415367603302, "answer_loss": 0.9513159549236297, "evidence_loss": 0.06419882911024616, "lr": 4.975619057357353e-05, "elapsed": 199.30454683303833}
|
| 25 |
+
{"step": 1250, "loss": 0.9221436047554016, "answer_loss": 0.8791058015823364, "evidence_loss": 0.08607561271637679, "lr": 4.970993644290227e-05, "elapsed": 207.64844846725464}
|
| 26 |
+
{"step": 1300, "loss": 0.8931722486019135, "answer_loss": 0.8583292460441589, "evidence_loss": 0.06968599802581593, "lr": 4.9659692724124814e-05, "elapsed": 216.47443270683289}
|
| 27 |
+
{"step": 1350, "loss": 0.913270663022995, "answer_loss": 0.8875316250324249, "evidence_loss": 0.051478075962513685, "lr": 4.960546752943294e-05, "elapsed": 223.96982049942017}
|
| 28 |
+
{"step": 1400, "loss": 0.9081017923355102, "answer_loss": 0.837847808599472, "evidence_loss": 0.14050796243478544, "lr": 4.9547269613854895e-05, "elapsed": 232.6224639415741}
|
| 29 |
+
{"step": 1450, "loss": 0.8998141181468964, "answer_loss": 0.8656974983215332, "evidence_loss": 0.06823322793585249, "lr": 4.948510837384188e-05, "elapsed": 240.88693809509277}
|
| 30 |
+
{"step": 1500, "loss": 0.9383626234531403, "answer_loss": 0.9003691983222961, "evidence_loss": 0.07598684702883475, "lr": 4.941899384575091e-05, "elapsed": 248.3419153690338}
|
| 31 |
+
{"step": 1550, "loss": 0.9325293815135955, "answer_loss": 0.9042844986915588, "evidence_loss": 0.05648976846132427, "lr": 4.934893670422436e-05, "elapsed": 258.31949400901794}
|
| 32 |
+
{"step": 1600, "loss": 0.9353457856178283, "answer_loss": 0.8941215693950653, "evidence_loss": 0.08244842126965522, "lr": 4.927494826046651e-05, "elapsed": 266.28701400756836}
|
| 33 |
+
{"step": 1650, "loss": 0.9353600656986236, "answer_loss": 0.9008423471450806, "evidence_loss": 0.06903544321190566, "lr": 4.919704046041723e-05, "elapsed": 274.2821455001831}
|
| 34 |
+
{"step": 1700, "loss": 0.9167217165231705, "answer_loss": 0.8436850917339325, "evidence_loss": 0.1460732433665544, "lr": 4.911522588282327e-05, "elapsed": 281.82033586502075}
|
| 35 |
+
{"step": 1750, "loss": 0.8866027438640595, "answer_loss": 0.8472788453102111, "evidence_loss": 0.07864779551513493, "lr": 4.902951773720729e-05, "elapsed": 290.0127441883087}
|
| 36 |
+
{"step": 1800, "loss": 0.911151088476181, "answer_loss": 0.88460886657238, "evidence_loss": 0.05308443671092391, "lr": 4.893992986173514e-05, "elapsed": 298.5438051223755}
|
| 37 |
+
{"step": 1850, "loss": 0.9242479574680328, "answer_loss": 0.8951860165596008, "evidence_loss": 0.05812388735823333, "lr": 4.884647672098155e-05, "elapsed": 306.03432989120483}
|
| 38 |
+
{"step": 1900, "loss": 0.88613041639328, "answer_loss": 0.8613187885284423, "evidence_loss": 0.049623256754130124, "lr": 4.8749173403594764e-05, "elapsed": 313.88096618652344}
|
| 39 |
+
{"step": 1950, "loss": 0.8989292919635773, "answer_loss": 0.8571232032775878, "evidence_loss": 0.08361217370256782, "lr": 4.864803561986033e-05, "elapsed": 321.75847482681274}
|
| 40 |
+
{"step": 2000, "loss": 0.8698323720693588, "answer_loss": 0.8435574531555176, "evidence_loss": 0.052549840230494735, "lr": 4.854307969916461e-05, "elapsed": 329.5411789417267}
|
| 41 |
+
{"step": 2050, "loss": 0.873380708694458, "answer_loss": 0.8490250790119171, "evidence_loss": 0.04871126282028854, "lr": 4.843432258735827e-05, "elapsed": 339.14401388168335}
|
| 42 |
+
{"step": 2100, "loss": 0.8385098683834076, "answer_loss": 0.8067602634429931, "evidence_loss": 0.06349921129643916, "lr": 4.8321781844020265e-05, "elapsed": 347.3702120780945}
|
| 43 |
+
{"step": 2150, "loss": 0.8724454253911972, "answer_loss": 0.8474841618537903, "evidence_loss": 0.04992253198521212, "lr": 4.8205475639622696e-05, "elapsed": 355.3514983654022}
|
| 44 |
+
{"step": 2200, "loss": 0.8919061470031738, "answer_loss": 0.8442657268047333, "evidence_loss": 0.09528083854354918, "lr": 4.808542275259711e-05, "elapsed": 363.63772678375244}
|
| 45 |
+
{"step": 2250, "loss": 0.8825558269023895, "answer_loss": 0.8516077971458436, "evidence_loss": 0.06189607612788677, "lr": 4.7961642566302545e-05, "elapsed": 371.33135437965393}
|
| 46 |
+
{"step": 2300, "loss": 0.883533639907837, "answer_loss": 0.8507235133647919, "evidence_loss": 0.06562025655061006, "lr": 4.7834155065896e-05, "elapsed": 379.59133291244507}
|
| 47 |
+
{"step": 2350, "loss": 0.8780528569221496, "answer_loss": 0.816949553489685, "evidence_loss": 0.12220661090686917, "lr": 4.770298083510566e-05, "elapsed": 387.2190008163452}
|
| 48 |
+
{"step": 2400, "loss": 0.881417669057846, "answer_loss": 0.8479820811748504, "evidence_loss": 0.06687118558213115, "lr": 4.756814105290753e-05, "elapsed": 395.72716188430786}
|
| 49 |
+
{"step": 2450, "loss": 0.8749468791484832, "answer_loss": 0.8310991561412812, "evidence_loss": 0.08769544798880816, "lr": 4.742965749010594e-05, "elapsed": 403.83505725860596}
|
| 50 |
+
{"step": 2500, "loss": 0.8472583591938019, "answer_loss": 0.8205045676231384, "evidence_loss": 0.05350758887827396, "lr": 4.72875525058185e-05, "elapsed": 411.0258378982544}
|
| 51 |
+
{"step": 2550, "loss": 0.9436272561550141, "answer_loss": 0.901155996620655, "evidence_loss": 0.08494251832173177, "lr": 4.714184904386607e-05, "elapsed": 420.5533525943756}
|
| 52 |
+
{"step": 2600, "loss": 0.8366120260953903, "answer_loss": 0.8006074810028077, "evidence_loss": 0.07200909493491053, "lr": 4.699257062906831e-05, "elapsed": 428.98778080940247}
|
| 53 |
+
{"step": 2650, "loss": 0.8633933508396149, "answer_loss": 0.7969949984550476, "evidence_loss": 0.13279670018702747, "lr": 4.683974136344547e-05, "elapsed": 436.9029242992401}
|
| 54 |
+
{"step": 2700, "loss": 0.9058808302879333, "answer_loss": 0.8711224663257598, "evidence_loss": 0.06951672300696372, "lr": 4.668338592232691e-05, "elapsed": 445.27145433425903}
|
| 55 |
+
{"step": 2750, "loss": 0.8000984215736389, "answer_loss": 0.7781926596164703, "evidence_loss": 0.04381152292713523, "lr": 4.652352955036713e-05, "elapsed": 453.3933148384094}
|
| 56 |
+
{"step": 2800, "loss": 0.8387403500080108, "answer_loss": 0.8048283362388611, "evidence_loss": 0.06782403912395239, "lr": 4.636019805746984e-05, "elapsed": 461.953227519989}
|
| 57 |
+
{"step": 2850, "loss": 0.8420565158128739, "answer_loss": 0.8153497308492661, "evidence_loss": 0.05341357516124844, "lr": 4.6193417814620775e-05, "elapsed": 469.58478307724}
|
| 58 |
+
{"step": 2900, "loss": 0.8497618436813354, "answer_loss": 0.8235075318813324, "evidence_loss": 0.052508616745471956, "lr": 4.602321574962992e-05, "elapsed": 477.9873991012573}
|
| 59 |
+
{"step": 2950, "loss": 0.8446140325069428, "answer_loss": 0.793148672580719, "evidence_loss": 0.1029307272285223, "lr": 4.584961934278382e-05, "elapsed": 487.12989926338196}
|
| 60 |
+
{"step": 3000, "loss": 0.8141267251968384, "answer_loss": 0.7857477569580078, "evidence_loss": 0.0567579442076385, "lr": 4.567265662240873e-05, "elapsed": 495.63873195648193}
|
| 61 |
+
{"step": 3050, "loss": 0.8962389063835144, "answer_loss": 0.8548105609416962, "evidence_loss": 0.08285669200122356, "lr": 4.54923561603452e-05, "elapsed": 505.362961769104}
|
| 62 |
+
{"step": 3100, "loss": 0.8867037588357926, "answer_loss": 0.8432322776317597, "evidence_loss": 0.08694295894354581, "lr": 4.530874706733502e-05, "elapsed": 513.6096704006195}
|
| 63 |
+
{"step": 3150, "loss": 0.8480165559053421, "answer_loss": 0.7987631392478943, "evidence_loss": 0.0985068316012621, "lr": 4.512185898832102e-05, "elapsed": 521.2800939083099}
|
| 64 |
+
{"step": 3200, "loss": 0.8267648661136627, "answer_loss": 0.7882062768936158, "evidence_loss": 0.07711717803031207, "lr": 4.493172209766075e-05, "elapsed": 529.8451421260834}
|
| 65 |
+
{"step": 3250, "loss": 0.7899400612711907, "answer_loss": 0.7610257956385612, "evidence_loss": 0.057828536592423915, "lr": 4.473836709425462e-05, "elapsed": 538.1944398880005}
|
| 66 |
+
{"step": 3300, "loss": 0.8478846701979638, "answer_loss": 0.8140367048978806, "evidence_loss": 0.06769592935219407, "lr": 4.45418251965893e-05, "elapsed": 546.569545507431}
|
| 67 |
+
{"step": 3350, "loss": 0.8761988198757171, "answer_loss": 0.8262290990352631, "evidence_loss": 0.09993943622335792, "lr": 4.434212813769734e-05, "elapsed": 554.8989162445068}
|
| 68 |
+
{"step": 3400, "loss": 0.8086458230018616, "answer_loss": 0.7672171103954315, "evidence_loss": 0.08285741509869694, "lr": 4.413930816003362e-05, "elapsed": 562.551394701004}
|
| 69 |
+
{"step": 3450, "loss": 0.8302321058511734, "answer_loss": 0.8040691727399826, "evidence_loss": 0.05232587525621057, "lr": 4.3933398010269614e-05, "elapsed": 570.803281545639}
|
| 70 |
+
{"step": 3500, "loss": 0.7841309231519699, "answer_loss": 0.7413554087281227, "evidence_loss": 0.08555102372542024, "lr": 4.3724430934006175e-05, "elapsed": 579.4653739929199}
|
| 71 |
+
{"step": 3550, "loss": 0.843475079536438, "answer_loss": 0.8148585021495819, "evidence_loss": 0.05723315194249153, "lr": 4.3512440670405866e-05, "elapsed": 588.8686480522156}
|
| 72 |
+
{"step": 3600, "loss": 0.7916189748048782, "answer_loss": 0.7542445093393326, "evidence_loss": 0.0747489233314991, "lr": 4.3297461446745454e-05, "elapsed": 595.8170926570892}
|
| 73 |
+
{"step": 3650, "loss": 0.813632447719574, "answer_loss": 0.7793202966451644, "evidence_loss": 0.06862430218607188, "lr": 4.3079527972889785e-05, "elapsed": 604.195232629776}
|
| 74 |
+
{"step": 3700, "loss": 0.7801144009828568, "answer_loss": 0.7514540457725525, "evidence_loss": 0.05732071315869689, "lr": 4.285867543568758e-05, "elapsed": 613.6402390003204}
|
| 75 |
+
{"step": 3750, "loss": 0.8143011832237244, "answer_loss": 0.7833894431591034, "evidence_loss": 0.06182349124923348, "lr": 4.263493949329025e-05, "elapsed": 621.7530562877655}
|
| 76 |
+
{"step": 3800, "loss": 0.8485709714889527, "answer_loss": 0.8243555790185928, "evidence_loss": 0.048430789336562155, "lr": 4.2408356269394714e-05, "elapsed": 630.1671869754791}
|
| 77 |
+
{"step": 3850, "loss": 0.8397923266887665, "answer_loss": 0.8064021778106689, "evidence_loss": 0.06678029645234346, "lr": 4.217896234741094e-05, "elapsed": 637.9561460018158}
|
| 78 |
+
{"step": 3900, "loss": 0.8384713381528854, "answer_loss": 0.7858047950267791, "evidence_loss": 0.10533308202400804, "lr": 4.194679476455531e-05, "elapsed": 645.8369159698486}
|
| 79 |
+
{"step": 3950, "loss": 0.8183697211742401, "answer_loss": 0.7814766681194305, "evidence_loss": 0.07378610454499722, "lr": 4.1711891005870695e-05, "elapsed": 654.0191309452057}
|
| 80 |
+
{"step": 4000, "loss": 1.0016829204559325, "answer_loss": 0.9684639060497284, "evidence_loss": 0.06643802328035235, "lr": 4.1474288998174285e-05, "elapsed": 663.4890587329865}
|
| 81 |
+
{"step": 4050, "loss": 0.8552967762947082, "answer_loss": 0.7874967682361603, "evidence_loss": 0.13560001157224177, "lr": 4.123402710393397e-05, "elapsed": 673.815267086029}
|
| 82 |
+
{"step": 4100, "loss": 0.8252304375171662, "answer_loss": 0.7937878298759461, "evidence_loss": 0.06288521507754923, "lr": 4.0991144115074544e-05, "elapsed": 682.1039531230927}
|
| 83 |
+
{"step": 4150, "loss": 0.8223253786563873, "answer_loss": 0.7903733336925507, "evidence_loss": 0.06390409536659718, "lr": 4.07456792467144e-05, "elapsed": 690.4485373497009}
|
| 84 |
+
{"step": 4200, "loss": 0.8266657882928848, "answer_loss": 0.7935312777757645, "evidence_loss": 0.06626902524381877, "lr": 4.049767213083405e-05, "elapsed": 698.0172429084778}
|
| 85 |
+
{"step": 4250, "loss": 0.7832595264911651, "answer_loss": 0.7511156284809113, "evidence_loss": 0.06428778750821948, "lr": 4.024716280987722e-05, "elapsed": 706.607063293457}
|
| 86 |
+
{"step": 4300, "loss": 0.8071381711959839, "answer_loss": 0.761707739830017, "evidence_loss": 0.09086086289957165, "lr": 3.999419173028578e-05, "elapsed": 714.9717524051666}
|
| 87 |
+
{"step": 4350, "loss": 0.7518622264266014, "answer_loss": 0.7257612183690071, "evidence_loss": 0.05220201130956412, "lr": 3.973879973596931e-05, "elapsed": 722.6397423744202}
|
| 88 |
+
{"step": 4400, "loss": 0.7376910710334778, "answer_loss": 0.6923048782348633, "evidence_loss": 0.09077237617224455, "lr": 3.948102806171065e-05, "elapsed": 729.6205620765686}
|
| 89 |
+
{"step": 4450, "loss": 0.6917971795797349, "answer_loss": 0.6521219050884247, "evidence_loss": 0.07935055043548346, "lr": 3.9220918326508214e-05, "elapsed": 737.6595108509064}
|
| 90 |
+
{"step": 4500, "loss": 0.7153780215978622, "answer_loss": 0.679352964758873, "evidence_loss": 0.07205011196434498, "lr": 3.895851252685634e-05, "elapsed": 745.631954908371}
|
| 91 |
+
{"step": 4550, "loss": 0.6821172097325325, "answer_loss": 0.6491390929371118, "evidence_loss": 0.06595623653382063, "lr": 3.8693853029964655e-05, "elapsed": 754.6406166553497}
|
| 92 |
+
{"step": 4600, "loss": 0.757670761346817, "answer_loss": 0.7134113085269927, "evidence_loss": 0.0885188945941627, "lr": 3.842698256691762e-05, "elapsed": 762.8825550079346}
|
| 93 |
+
{"step": 4650, "loss": 0.7153676640987396, "answer_loss": 0.6860852801799774, "evidence_loss": 0.058564767464995386, "lr": 3.815794422577528e-05, "elapsed": 771.2530393600464}
|
| 94 |
+
{"step": 4700, "loss": 0.7053547477722169, "answer_loss": 0.6798686963319779, "evidence_loss": 0.05097210619598627, "lr": 3.7886781444616416e-05, "elapsed": 778.8970429897308}
|
| 95 |
+
{"step": 4750, "loss": 0.7132750861346722, "answer_loss": 0.6792814381420612, "evidence_loss": 0.06798730622977019, "lr": 3.761353800452518e-05, "elapsed": 786.393144607544}
|
| 96 |
+
{"step": 4800, "loss": 0.7167143061757087, "answer_loss": 0.6781955006718635, "evidence_loss": 0.07703761540353299, "lr": 3.733825802252231e-05, "elapsed": 794.6438567638397}
|
| 97 |
+
{"step": 4850, "loss": 0.7308814007043839, "answer_loss": 0.7114387238025666, "evidence_loss": 0.038885354194790125, "lr": 3.706098594444216e-05, "elapsed": 803.495509147644}
|
| 98 |
+
{"step": 4900, "loss": 0.725177657008171, "answer_loss": 0.6799892061948776, "evidence_loss": 0.09037690445780754, "lr": 3.6781766537756615e-05, "elapsed": 811.1936976909637}
|
| 99 |
+
{"step": 4950, "loss": 0.7437954246997833, "answer_loss": 0.7001254135370254, "evidence_loss": 0.08734002096578479, "lr": 3.650064488434706e-05, "elapsed": 819.3370888233185}
|
| 100 |
+
{"step": 5000, "loss": 0.6711376789212227, "answer_loss": 0.6404766237735748, "evidence_loss": 0.06132210982963443, "lr": 3.621766637322563e-05, "elapsed": 826.9146244525909}
|
| 101 |
+
{"step": 5050, "loss": 0.6443161219358444, "answer_loss": 0.6156630384922027, "evidence_loss": 0.05730616588145494, "lr": 3.593287669320682e-05, "elapsed": 835.9703977108002}
|
| 102 |
+
{"step": 5100, "loss": 0.7151914113759994, "answer_loss": 0.6796099555492401, "evidence_loss": 0.07116290217265486, "lr": 3.5646321825530706e-05, "elapsed": 844.5879793167114}
|
| 103 |
+
{"step": 5150, "loss": 0.7430116021633149, "answer_loss": 0.7016761064529419, "evidence_loss": 0.08267100349068641, "lr": 3.5358048036439e-05, "elapsed": 852.2563669681549}
|
| 104 |
+
{"step": 5200, "loss": 0.7354519665241241, "answer_loss": 0.6897015607357025, "evidence_loss": 0.09150081064552068, "lr": 3.5068101869705004e-05, "elapsed": 860.9883148670197}
|
| 105 |
+
{"step": 5250, "loss": 0.6898434311151505, "answer_loss": 0.6544760137796402, "evidence_loss": 0.07073483120650054, "lr": 3.477653013911882e-05, "elapsed": 869.9604740142822}
|
| 106 |
+
{"step": 5300, "loss": 0.685687806904316, "answer_loss": 0.6513550034165383, "evidence_loss": 0.0686656179279089, "lr": 3.4483379920928974e-05, "elapsed": 878.0651724338531}
|
| 107 |
+
{"step": 5350, "loss": 0.6534899070858955, "answer_loss": 0.6309079629182816, "evidence_loss": 0.04516388503834605, "lr": 3.418869854624156e-05, "elapsed": 886.0391924381256}
|
| 108 |
+
{"step": 5400, "loss": 1.115627024769783, "answer_loss": 0.9870361652970314, "evidence_loss": 0.2571816800348461, "lr": 3.389253359337836e-05, "elapsed": 894.1839475631714}
|
| 109 |
+
{"step": 5450, "loss": 0.7228120070695877, "answer_loss": 0.6819659262895584, "evidence_loss": 0.08169216554611922, "lr": 3.3594932880195026e-05, "elapsed": 902.1450335979462}
|
| 110 |
+
{"step": 5500, "loss": 0.7043061482906342, "answer_loss": 0.675443776845932, "evidence_loss": 0.05772474072873592, "lr": 3.329594445636051e-05, "elapsed": 909.8384783267975}
|
| 111 |
+
{"step": 5550, "loss": 0.6782146200537682, "answer_loss": 0.6402420845627784, "evidence_loss": 0.07594507927075028, "lr": 3.299561659559909e-05, "elapsed": 919.2789318561554}
|
| 112 |
+
{"step": 5600, "loss": 0.6894875425100326, "answer_loss": 0.6590450462698937, "evidence_loss": 0.06088498806580901, "lr": 3.269399778789633e-05, "elapsed": 927.5675184726715}
|
| 113 |
+
{"step": 5650, "loss": 0.6897464379668236, "answer_loss": 0.6550332856178284, "evidence_loss": 0.06942630268633365, "lr": 3.239113673167002e-05, "elapsed": 935.8587505817413}
|
| 114 |
+
{"step": 5700, "loss": 0.7317785179615021, "answer_loss": 0.6955934762954712, "evidence_loss": 0.07237009294331073, "lr": 3.2087082325907416e-05, "elapsed": 943.6783452033997}
|
| 115 |
+
{"step": 5750, "loss": 0.6381632173061371, "answer_loss": 0.612074216902256, "evidence_loss": 0.05217799788340926, "lr": 3.178188366227025e-05, "elapsed": 952.1261720657349}
|
| 116 |
+
{"step": 5800, "loss": 0.6985940510034561, "answer_loss": 0.6591763442754746, "evidence_loss": 0.07883541490882635, "lr": 3.147559001716846e-05, "elapsed": 960.7510900497437}
|
| 117 |
+
{"step": 5850, "loss": 0.676286449432373, "answer_loss": 0.6377813851833344, "evidence_loss": 0.07701012386009097, "lr": 3.116825084380429e-05, "elapsed": 969.561493396759}
|
| 118 |
+
{"step": 5900, "loss": 0.6947304600477219, "answer_loss": 0.658698034286499, "evidence_loss": 0.07206485282629728, "lr": 3.0859915764187616e-05, "elapsed": 978.1437783241272}
|
| 119 |
+
{"step": 5950, "loss": 0.6931634292006492, "answer_loss": 0.6577613961696624, "evidence_loss": 0.07080406468361616, "lr": 3.0550634561124204e-05, "elapsed": 985.6736602783203}
|
| 120 |
+
{"step": 6000, "loss": 0.6677104836702347, "answer_loss": 0.63559848934412, "evidence_loss": 0.0642239898443222, "lr": 3.024045717017791e-05, "elapsed": 993.6672146320343}
|
| 121 |
+
{"step": 6050, "loss": 0.6503987711668014, "answer_loss": 0.6175749111175537, "evidence_loss": 0.06564772330224514, "lr": 2.992943367160826e-05, "elapsed": 1004.4780714511871}
|
| 122 |
+
{"step": 6100, "loss": 0.6741045522689819, "answer_loss": 0.6368017035722733, "evidence_loss": 0.0746056986413896, "lr": 2.9617614282284617e-05, "elapsed": 1012.8882274627686}
|
| 123 |
+
{"step": 6150, "loss": 0.6704132956266403, "answer_loss": 0.6388553756475449, "evidence_loss": 0.06311584990471601, "lr": 2.9305049347578373e-05, "elapsed": 1020.8936626911163}
|
| 124 |
+
{"step": 6200, "loss": 0.7005413234233856, "answer_loss": 0.6630048036575318, "evidence_loss": 0.07507304053753615, "lr": 2.8991789333234294e-05, "elapsed": 1029.2225141525269}
|
| 125 |
+
{"step": 6250, "loss": 0.6501335802674294, "answer_loss": 0.6198824089765549, "evidence_loss": 0.06050234373658896, "lr": 2.8677884817222505e-05, "elapsed": 1037.4860351085663}
|
| 126 |
+
{"step": 6300, "loss": 0.658392752110958, "answer_loss": 0.6259638449549675, "evidence_loss": 0.06485781505703926, "lr": 2.836338648157235e-05, "elapsed": 1044.9264895915985}
|
| 127 |
+
{"step": 6350, "loss": 0.6528424060344696, "answer_loss": 0.6293296712636948, "evidence_loss": 0.04702547915279865, "lr": 2.80483451041894e-05, "elapsed": 1053.3040189743042}
|
| 128 |
+
{"step": 6400, "loss": 0.6845511591434479, "answer_loss": 0.6568956357240677, "evidence_loss": 0.05531104102730751, "lr": 2.7732811550657007e-05, "elapsed": 1061.245079755783}
|
| 129 |
+
{"step": 6450, "loss": 0.6638348960876465, "answer_loss": 0.6265028011798859, "evidence_loss": 0.07466419273987412, "lr": 2.7416836766023756e-05, "elapsed": 1069.3163509368896}
|
| 130 |
+
{"step": 6500, "loss": 0.7291047883033752, "answer_loss": 0.6951379197835922, "evidence_loss": 0.06793373644351959, "lr": 2.7100471766577994e-05, "elapsed": 1077.1476860046387}
|
| 131 |
+
{"step": 6550, "loss": 0.6846529287099838, "answer_loss": 0.6551175045967103, "evidence_loss": 0.05907085435464978, "lr": 2.6783767631610902e-05, "elapsed": 1087.6441445350647}
|
| 132 |
+
{"step": 6600, "loss": 0.6974994051456451, "answer_loss": 0.6536128750443458, "evidence_loss": 0.08777306761592626, "lr": 2.6466775495169437e-05, "elapsed": 1095.8568933010101}
|
| 133 |
+
{"step": 6650, "loss": 0.7177600318193436, "answer_loss": 0.6861421024799347, "evidence_loss": 0.06323585590599891, "lr": 2.614954653780035e-05, "elapsed": 1103.8210332393646}
|
| 134 |
+
{"step": 6700, "loss": 2.971225202679634, "answer_loss": 2.9339328318834306, "evidence_loss": 0.07458478694550649, "lr": 2.583213197828675e-05, "elapsed": 1111.7342629432678}
|
| 135 |
+
{"step": 6750, "loss": 0.9339103883504868, "answer_loss": 0.9065841251611709, "evidence_loss": 0.054652514923363926, "lr": 2.5514583065378467e-05, "elapsed": 1119.636307477951}
|
| 136 |
+
{"step": 6800, "loss": 0.6748266065120697, "answer_loss": 0.6541101598739624, "evidence_loss": 0.04143289290368557, "lr": 2.5196951069517626e-05, "elapsed": 1127.4232268333435}
|
| 137 |
+
{"step": 6850, "loss": 0.6554432682693004, "answer_loss": 0.6153842915594577, "evidence_loss": 0.08011794995516539, "lr": 2.4879287274560646e-05, "elapsed": 1136.0656003952026}
|
| 138 |
+
{"step": 6900, "loss": 1.533397053182125, "answer_loss": 1.505927468240261, "evidence_loss": 0.054939203783869746, "lr": 2.4561642969498147e-05, "elapsed": 1143.9023413658142}
|
| 139 |
+
{"step": 6950, "loss": 0.687921776175499, "answer_loss": 0.6538665771484375, "evidence_loss": 0.06811040814965963, "lr": 2.4244069440173938e-05, "elapsed": 1152.3257915973663}
|
| 140 |
+
{"step": 7000, "loss": 0.6806336897611618, "answer_loss": 0.6535909217596054, "evidence_loss": 0.05408553287386894, "lr": 2.392661796100464e-05, "elapsed": 1160.4935927391052}
|
| 141 |
+
{"step": 7050, "loss": 0.67043880879879, "answer_loss": 0.6452980351448059, "evidence_loss": 0.050281544886529445, "lr": 2.3609339786701028e-05, "elapsed": 1169.9351518154144}
|
| 142 |
+
{"step": 7100, "loss": 0.6720110481977463, "answer_loss": 0.6364184728264809, "evidence_loss": 0.07118514137342573, "lr": 2.32922861439926e-05, "elapsed": 1177.091799736023}
|
| 143 |
+
{"step": 7150, "loss": 0.6839973723888397, "answer_loss": 0.6451834881305695, "evidence_loss": 0.07762777423486114, "lr": 2.2975508223356736e-05, "elapsed": 1185.529240846634}
|
| 144 |
+
{"step": 7200, "loss": 0.6921482855081558, "answer_loss": 0.6482168239355087, "evidence_loss": 0.08786292348057032, "lr": 2.2659057170753565e-05, "elapsed": 1194.532476425171}
|
| 145 |
+
{"step": 7250, "loss": 0.6522014793753624, "answer_loss": 0.6209597045183182, "evidence_loss": 0.06248354911804199, "lr": 2.234298407936815e-05, "elapsed": 1202.8367943763733}
|
| 146 |
+
{"step": 7300, "loss": 0.6825660765171051, "answer_loss": 0.649959779381752, "evidence_loss": 0.06521259021013975, "lr": 2.2027339981361156e-05, "elapsed": 1210.615250825882}
|
| 147 |
+
{"step": 7350, "loss": 0.7056313619017601, "answer_loss": 0.6743047136068344, "evidence_loss": 0.06265328923240304, "lr": 2.1712175839629297e-05, "elapsed": 1219.3949506282806}
|
| 148 |
+
{"step": 7400, "loss": 0.6693471157550812, "answer_loss": 0.6355771887302398, "evidence_loss": 0.06753984929993749, "lr": 2.1397542539577082e-05, "elapsed": 1227.7669749259949}
|
| 149 |
+
{"step": 7450, "loss": 0.6570795357227326, "answer_loss": 0.6216828444600105, "evidence_loss": 0.07079338343814015, "lr": 2.1083490880901026e-05, "elapsed": 1235.659303188324}
|
| 150 |
+
{"step": 7500, "loss": 0.6845752412080764, "answer_loss": 0.6454730105400085, "evidence_loss": 0.07820445504039526, "lr": 2.077007156938763e-05, "elapsed": 1243.8670871257782}
|
| 151 |
+
{"step": 7550, "loss": 0.6802064681053162, "answer_loss": 0.638635043501854, "evidence_loss": 0.083142851293087, "lr": 2.0457335208726642e-05, "elapsed": 1253.3454809188843}
|
| 152 |
+
{"step": 7600, "loss": 0.6770537130534648, "answer_loss": 0.6488707192242146, "evidence_loss": 0.0563659911043942, "lr": 2.0145332292340746e-05, "elapsed": 1261.2428288459778}
|
| 153 |
+
{"step": 7650, "loss": 0.6757621063292026, "answer_loss": 0.652953759431839, "evidence_loss": 0.045616688262671234, "lr": 1.9834113195233e-05, "elapsed": 1269.086086511612}
|
| 154 |
+
{"step": 7700, "loss": 0.6783012318611145, "answer_loss": 0.63468913808465, "evidence_loss": 0.08722419014200568, "lr": 1.952372816585351e-05, "elapsed": 1277.6723980903625}
|
| 155 |
+
{"step": 7750, "loss": 0.6125366176664829, "answer_loss": 0.5731025074422359, "evidence_loss": 0.07886821988970041, "lr": 1.9214227317986428e-05, "elapsed": 1286.0749232769012}
|
| 156 |
+
{"step": 7800, "loss": 0.6728542965650558, "answer_loss": 0.6373563864827156, "evidence_loss": 0.07099581534042955, "lr": 1.8905660622658773e-05, "elapsed": 1294.0080287456512}
|
| 157 |
+
{"step": 7850, "loss": 0.6606539458036422, "answer_loss": 0.6281986480951309, "evidence_loss": 0.06491059426218271, "lr": 1.8598077900072254e-05, "elapsed": 1302.1447002887726}
|
| 158 |
+
{"step": 7900, "loss": 0.6496975690126419, "answer_loss": 0.6031782829761505, "evidence_loss": 0.09303856153041125, "lr": 1.8291528811559444e-05, "elapsed": 1310.9247362613678}
|
| 159 |
+
{"step": 7950, "loss": 0.6546725788712502, "answer_loss": 0.6270247420668602, "evidence_loss": 0.05529567312449217, "lr": 1.7986062851565634e-05, "elapsed": 1318.3111727237701}
|
| 160 |
+
{"step": 8000, "loss": 0.6526186865568161, "answer_loss": 0.6093081942200661, "evidence_loss": 0.08662098614498973, "lr": 1.7681729339657626e-05, "elapsed": 1325.7836830615997}
|
| 161 |
+
{"step": 8050, "loss": 0.6770608162879944, "answer_loss": 0.6511178815364838, "evidence_loss": 0.05188587853685021, "lr": 1.7378577412560726e-05, "elapsed": 1336.2150621414185}
|
| 162 |
+
{"step": 8100, "loss": 0.6518723100423813, "answer_loss": 0.6124281299114227, "evidence_loss": 0.0788883593864739, "lr": 1.707665601622529e-05, "elapsed": 1344.8420276641846}
|
| 163 |
+
{"step": 8150, "loss": 0.6854977810382843, "answer_loss": 0.6442795497179031, "evidence_loss": 0.08243646116927267, "lr": 1.6776013897924102e-05, "elapsed": 1352.6764390468597}
|
| 164 |
+
{"step": 8200, "loss": 0.6639537698030472, "answer_loss": 0.6374973028898239, "evidence_loss": 0.05291293237358332, "lr": 1.6476699598381738e-05, "elapsed": 1361.094190120697}
|
| 165 |
+
{"step": 8250, "loss": 0.6766222310066223, "answer_loss": 0.6477513080835342, "evidence_loss": 0.05774184502661228, "lr": 1.6178761443937376e-05, "elapsed": 1369.5300676822662}
|
| 166 |
+
{"step": 8300, "loss": 0.655701839029789, "answer_loss": 0.6291413938999176, "evidence_loss": 0.053120890567079186, "lr": 1.5882247538742144e-05, "elapsed": 1377.2731914520264}
|
| 167 |
+
{"step": 8350, "loss": 0.6867629337310791, "answer_loss": 0.6579824423789978, "evidence_loss": 0.057560991887003185, "lr": 1.558720575699242e-05, "elapsed": 1385.5776391029358}
|
| 168 |
+
{"step": 8400, "loss": 0.6423148792982102, "answer_loss": 0.6104894876480103, "evidence_loss": 0.0636507835611701, "lr": 1.5293683735200186e-05, "elapsed": 1394.4413104057312}
|
| 169 |
+
{"step": 8450, "loss": 0.6672878208756446, "answer_loss": 0.6338611060380935, "evidence_loss": 0.06685343086719513, "lr": 1.500172886450177e-05, "elapsed": 1402.3244121074677}
|
| 170 |
+
{"step": 8500, "loss": 0.6455755931138992, "answer_loss": 0.6219204032421112, "evidence_loss": 0.04731037875637412, "lr": 1.4711388283006274e-05, "elapsed": 1410.7274193763733}
|
| 171 |
+
{"step": 8550, "loss": 0.6360426324605942, "answer_loss": 0.6084302917122841, "evidence_loss": 0.05522468212991953, "lr": 1.4422708868184768e-05, "elapsed": 1420.2480356693268}
|
| 172 |
+
{"step": 8600, "loss": 0.6465635007619858, "answer_loss": 0.6177509379386902, "evidence_loss": 0.05762512551620603, "lr": 1.41357372293016e-05, "elapsed": 1427.9948344230652}
|
| 173 |
+
{"step": 8650, "loss": 0.680012092590332, "answer_loss": 0.6448679000139237, "evidence_loss": 0.07028838669881225, "lr": 1.3850519699889036e-05, "elapsed": 1436.5489766597748}
|
| 174 |
+
{"step": 8700, "loss": 0.6132512277364731, "answer_loss": 0.5756462526321411, "evidence_loss": 0.07520994974300266, "lr": 1.3567102330266369e-05, "elapsed": 1444.4544150829315}
|
| 175 |
+
{"step": 8750, "loss": 0.5396253544092179, "answer_loss": 0.4956742095947266, "evidence_loss": 0.0879022891819477, "lr": 1.3285530880104797e-05, "elapsed": 1452.6570463180542}
|
| 176 |
+
{"step": 8800, "loss": 0.5399253216385841, "answer_loss": 0.5127778753638268, "evidence_loss": 0.05429489566013217, "lr": 1.3005850811039216e-05, "elapsed": 1460.5467066764832}
|
| 177 |
+
{"step": 8850, "loss": 0.5432167410850525, "answer_loss": 0.5144682395458221, "evidence_loss": 0.057497002854943274, "lr": 1.2728107279328078e-05, "elapsed": 1468.9135661125183}
|
| 178 |
+
{"step": 8900, "loss": 0.566004410982132, "answer_loss": 0.5227320665121078, "evidence_loss": 0.08654468754306435, "lr": 1.2452345128562706e-05, "elapsed": 1476.821532011032}
|
| 179 |
+
{"step": 8950, "loss": 0.4930362571775913, "answer_loss": 0.46602997690439224, "evidence_loss": 0.05401255136355758, "lr": 1.2178608882426892e-05, "elapsed": 1484.7572071552277}
|
| 180 |
+
{"step": 9000, "loss": 0.5355394059419631, "answer_loss": 0.4975417339801788, "evidence_loss": 0.07599534578621388, "lr": 1.190694273750826e-05, "elapsed": 1492.8544118404388}
|
| 181 |
+
{"step": 9050, "loss": 0.516750836968422, "answer_loss": 0.48176089867949484, "evidence_loss": 0.0699798752553761, "lr": 1.16373905561625e-05, "elapsed": 1502.302030801773}
|
| 182 |
+
{"step": 9100, "loss": 0.5479353514313697, "answer_loss": 0.5100755497813225, "evidence_loss": 0.0757195995748043, "lr": 1.1369995859431395e-05, "elapsed": 1510.5201489925385}
|
| 183 |
+
{"step": 9150, "loss": 0.5301456728577614, "answer_loss": 0.505865815281868, "evidence_loss": 0.048559716809540986, "lr": 1.1104801820016108e-05, "elapsed": 1518.405501127243}
|
| 184 |
+
{"step": 9200, "loss": 0.5567553769052028, "answer_loss": 0.5154810176044703, "evidence_loss": 0.0825487249903381, "lr": 1.0841851255306636e-05, "elapsed": 1526.349023103714}
|
| 185 |
+
{"step": 9250, "loss": 0.5065599066019059, "answer_loss": 0.4878265699744225, "evidence_loss": 0.03746667494997382, "lr": 1.0581186620468652e-05, "elapsed": 1534.8083584308624}
|
| 186 |
+
{"step": 9300, "loss": 0.5078363177180291, "answer_loss": 0.47537857443094256, "evidence_loss": 0.06491549191996455, "lr": 1.0322850001588833e-05, "elapsed": 1543.1259071826935}
|
| 187 |
+
{"step": 9350, "loss": 0.5059441205859184, "answer_loss": 0.47340020418167117, "evidence_loss": 0.0650878350995481, "lr": 1.0066883108879772e-05, "elapsed": 1551.6659364700317}
|
| 188 |
+
{"step": 9400, "loss": 0.5534780645370483, "answer_loss": 0.5218871796131134, "evidence_loss": 0.06318177247419954, "lr": 9.813327269945547e-06, "elapsed": 1560.103696346283}
|
| 189 |
+
{"step": 9450, "loss": 0.5173558956384658, "answer_loss": 0.4955078411102295, "evidence_loss": 0.043696111720055344, "lr": 9.562223423109182e-06, "elapsed": 1568.185317993164}
|
| 190 |
+
{"step": 9500, "loss": 0.5463841432332992, "answer_loss": 0.5225978684425354, "evidence_loss": 0.04757254786789417, "lr": 9.313612110802783e-06, "elapsed": 1576.3521552085876}
|
| 191 |
+
{"step": 9550, "loss": 0.4892009598016739, "answer_loss": 0.45283409833908084, "evidence_loss": 0.07273372473195196, "lr": 9.067533473021712e-06, "elapsed": 1585.346361875534}
|
| 192 |
+
{"step": 9600, "loss": 0.5049640274047852, "answer_loss": 0.47200328946113584, "evidence_loss": 0.06592146918177605, "lr": 8.824027240843755e-06, "elapsed": 1593.3976519107819}
|
| 193 |
+
{"step": 9650, "loss": 0.5333356314897537, "answer_loss": 0.4956443612277508, "evidence_loss": 0.07538254395127296, "lr": 8.583132730014224e-06, "elapsed": 1601.7102887630463}
|
| 194 |
+
{"step": 9700, "loss": 0.5490519870817662, "answer_loss": 0.5150954926013946, "evidence_loss": 0.0679129952378571, "lr": 8.344888834598133e-06, "elapsed": 1609.7309610843658}
|
| 195 |
+
{"step": 9750, "loss": 0.5382940667867661, "answer_loss": 0.5047756043076516, "evidence_loss": 0.0670369279012084, "lr": 8.109334020700591e-06, "elapsed": 1617.4024484157562}
|
| 196 |
+
{"step": 9800, "loss": 0.5214061813056469, "answer_loss": 0.4849688847362995, "evidence_loss": 0.07287458902224898, "lr": 7.87650632025605e-06, "elapsed": 1625.6264643669128}
|
| 197 |
+
{"step": 9850, "loss": 0.5243250066041947, "answer_loss": 0.48825872272253035, "evidence_loss": 0.07213257240131497, "lr": 7.646443324887894e-06, "elapsed": 1634.0155193805695}
|
| 198 |
+
{"step": 9900, "loss": 0.5045383094251156, "answer_loss": 0.4718585881590843, "evidence_loss": 0.0653594415448606, "lr": 7.419182179838999e-06, "elapsed": 1642.7251634597778}
|
| 199 |
+
{"step": 9950, "loss": 0.5845209920406341, "answer_loss": 0.5208078742027282, "evidence_loss": 0.12742623150348664, "lr": 7.194759577974305e-06, "elapsed": 1651.0870747566223}
|
| 200 |
+
{"step": 10000, "loss": 0.5123613653331995, "answer_loss": 0.46654066890478135, "evidence_loss": 0.09164139261469245, "lr": 6.97321175385659e-06, "elapsed": 1658.843956232071}
|
| 201 |
+
{"step": 10050, "loss": 0.5585399997234345, "answer_loss": 0.5168661668896675, "evidence_loss": 0.08334766779094935, "lr": 6.7545744778961364e-06, "elapsed": 1668.6149997711182}
|
| 202 |
+
{"step": 10100, "loss": 0.4958308392763138, "answer_loss": 0.464941865503788, "evidence_loss": 0.061777944397181275, "lr": 6.538883050575284e-06, "elapsed": 1676.575123310089}
|
| 203 |
+
{"step": 10150, "loss": 0.5041336873173714, "answer_loss": 0.46142133116722106, "evidence_loss": 0.08542471474036574, "lr": 6.326172296749014e-06, "elapsed": 1684.3041203022003}
|
| 204 |
+
{"step": 10200, "loss": 0.5051713567972184, "answer_loss": 0.48629032254219057, "evidence_loss": 0.03776207009330392, "lr": 6.116476560022216e-06, "elapsed": 1692.9230897426605}
|
| 205 |
+
{"step": 10250, "loss": 0.5433296683058143, "answer_loss": 0.5106863075122238, "evidence_loss": 0.06528672400861979, "lr": 5.909829697204611e-06, "elapsed": 1700.7856726646423}
|
| 206 |
+
{"step": 10300, "loss": 0.5063437685370445, "answer_loss": 0.4726037508249283, "evidence_loss": 0.06748003598302603, "lr": 5.706265072844452e-06, "elapsed": 1709.410652399063}
|
| 207 |
+
{"step": 10350, "loss": 0.48690754652023316, "answer_loss": 0.459538821130991, "evidence_loss": 0.05473744705319405, "lr": 5.505815553841484e-06, "elapsed": 1716.5952513217926}
|
| 208 |
+
{"step": 10400, "loss": 0.5246459922194481, "answer_loss": 0.4905425676703453, "evidence_loss": 0.06820684475824237, "lr": 5.308513504140417e-06, "elapsed": 1725.3376700878143}
|
| 209 |
+
{"step": 10450, "loss": 0.5071134573221207, "answer_loss": 0.47451526649296283, "evidence_loss": 0.06519638057798147, "lr": 5.114390779505557e-06, "elapsed": 1733.8387970924377}
|
| 210 |
+
{"step": 10500, "loss": 0.548963770866394, "answer_loss": 0.5213799345493316, "evidence_loss": 0.05516766419634223, "lr": 4.923478722377412e-06, "elapsed": 1741.7004051208496}
|
| 211 |
+
{"step": 10550, "loss": 0.5097629813849927, "answer_loss": 0.4800592638552189, "evidence_loss": 0.05940742623060942, "lr": 4.7358081568122885e-06, "elapsed": 1751.2746365070343}
|
| 212 |
+
{"step": 10600, "loss": 0.5611341136693955, "answer_loss": 0.5168021357059479, "evidence_loss": 0.08866395188495517, "lr": 4.551409383505564e-06, "elapsed": 1759.5826668739319}
|
| 213 |
+
{"step": 10650, "loss": 0.5236381381750107, "answer_loss": 0.49413217812776566, "evidence_loss": 0.05901192227378488, "lr": 4.370312174899344e-06, "elapsed": 1768.032687664032}
|
| 214 |
+
{"step": 10700, "loss": 0.5296658682823181, "answer_loss": 0.4837737075984478, "evidence_loss": 0.0917843134701252, "lr": 4.192545770375575e-06, "elapsed": 1776.1212236881256}
|
| 215 |
+
{"step": 10750, "loss": 0.5043485322594643, "answer_loss": 0.48011365205049517, "evidence_loss": 0.0484697612375021, "lr": 4.018138871535143e-06, "elapsed": 1784.0609419345856}
|
| 216 |
+
{"step": 10800, "loss": 1.9761034050583839, "answer_loss": 1.9397877123951912, "evidence_loss": 0.07263125374913215, "lr": 3.847119637563739e-06, "elapsed": 1791.6835641860962}
|
| 217 |
+
{"step": 10850, "loss": 0.5030315300822258, "answer_loss": 0.4746138462424278, "evidence_loss": 0.05683537110686302, "lr": 3.6795156806854413e-06, "elapsed": 1800.3796429634094}
|
| 218 |
+
{"step": 10900, "loss": 0.5194159936904907, "answer_loss": 0.4870105201005936, "evidence_loss": 0.06481094421818852, "lr": 3.51535406170449e-06, "elapsed": 1808.5800850391388}
|
| 219 |
+
{"step": 10950, "loss": 0.5277880918979645, "answer_loss": 0.4935543535649776, "evidence_loss": 0.0684674796462059, "lr": 3.3546612856361487e-06, "elapsed": 1816.8069899082184}
|
| 220 |
+
{"step": 11000, "loss": 0.5115945684909821, "answer_loss": 0.4813395875692368, "evidence_loss": 0.060509958174079656, "lr": 3.1974632974272862e-06, "elapsed": 1825.8126990795135}
|
| 221 |
+
{"step": 11050, "loss": 0.5036651909351348, "answer_loss": 0.4695639196038246, "evidence_loss": 0.06820254674181342, "lr": 3.0437854777673857e-06, "elapsed": 1834.846907377243}
|
| 222 |
+
{"step": 11100, "loss": 0.5206039327383042, "answer_loss": 0.4831606024503708, "evidence_loss": 0.07488666372373701, "lr": 2.893652638990668e-06, "elapsed": 1843.1257004737854}
|
| 223 |
+
{"step": 11150, "loss": 0.5521255898475647, "answer_loss": 0.5037451529502869, "evidence_loss": 0.09676087360829115, "lr": 2.7470890210699596e-06, "elapsed": 1851.4722456932068}
|
| 224 |
+
{"step": 11200, "loss": 0.5255104085803032, "answer_loss": 0.4957681430876255, "evidence_loss": 0.05948453476652503, "lr": 2.604118287702989e-06, "elapsed": 1859.0711646080017}
|
| 225 |
+
{"step": 11250, "loss": 0.5322479391098023, "answer_loss": 0.5077209854125977, "evidence_loss": 0.04905390286818147, "lr": 2.464763522491717e-06, "elapsed": 1867.3157856464386}
|
| 226 |
+
{"step": 11300, "loss": 0.5401521795988082, "answer_loss": 0.5023234722018242, "evidence_loss": 0.07565741259604693, "lr": 2.3290472252153363e-06, "elapsed": 1875.4804511070251}
|
| 227 |
+
{"step": 11350, "loss": 0.49710859566926957, "answer_loss": 0.46976491540670395, "evidence_loss": 0.05468736356124282, "lr": 2.196991308197524e-06, "elapsed": 1883.9479219913483}
|
| 228 |
+
{"step": 11400, "loss": 0.5171410557627678, "answer_loss": 0.48300718307495116, "evidence_loss": 0.06826774602755904, "lr": 2.068617092768557e-06, "elapsed": 1892.275889158249}
|
| 229 |
+
{"step": 11450, "loss": 0.5029351166635752, "answer_loss": 0.47846999794244766, "evidence_loss": 0.04893023930490017, "lr": 1.9439453058228197e-06, "elapsed": 1900.4599301815033}
|
| 230 |
+
{"step": 11500, "loss": 0.5014249137043953, "answer_loss": 0.46414141461253167, "evidence_loss": 0.0745669960975647, "lr": 1.822996076472322e-06, "elapsed": 1907.9959378242493}
|
| 231 |
+
{"step": 11550, "loss": 0.5275092554092408, "answer_loss": 0.4851616472005844, "evidence_loss": 0.08469522116705776, "lr": 1.7057889327967092e-06, "elapsed": 1917.1907584667206}
|
| 232 |
+
{"step": 11600, "loss": 0.5156501485407352, "answer_loss": 0.4795196535438299, "evidence_loss": 0.07226098975166678, "lr": 1.5923427986903217e-06, "elapsed": 1925.1338076591492}
|
| 233 |
+
{"step": 11650, "loss": 0.5447640980780125, "answer_loss": 0.5060875414311886, "evidence_loss": 0.07735310908406973, "lr": 1.482675990806795e-06, "elapsed": 1933.137132883072}
|
| 234 |
+
{"step": 11700, "loss": 0.5261088354885578, "answer_loss": 0.4977184520661831, "evidence_loss": 0.05678076423704624, "lr": 1.3768062156017308e-06, "elapsed": 1941.606175661087}
|
| 235 |
+
{"step": 11750, "loss": 0.5132436154782772, "answer_loss": 0.49092079475522044, "evidence_loss": 0.04464563868939877, "lr": 1.2747505664738452e-06, "elapsed": 1949.711343050003}
|
| 236 |
+
{"step": 11800, "loss": 0.524514929652214, "answer_loss": 0.49599145114421844, "evidence_loss": 0.057046964783221486, "lr": 1.1765255210051369e-06, "elapsed": 1957.9971811771393}
|
| 237 |
+
{"step": 11850, "loss": 0.5087773703038693, "answer_loss": 0.4754104192554951, "evidence_loss": 0.06673389909788967, "lr": 1.0821469383004657e-06, "elapsed": 1965.709528207779}
|
| 238 |
+
{"step": 11900, "loss": 1.3689051040261984, "answer_loss": 1.337282130867243, "evidence_loss": 0.06324598535895348, "lr": 9.916300564269954e-07, "elapsed": 1973.5016798973083}
|
| 239 |
+
{"step": 11950, "loss": 0.514110007584095, "answer_loss": 0.4900334833562374, "evidence_loss": 0.04815305231139064, "lr": 9.049894899538919e-07, "elapsed": 1981.0086872577667}
|
| 240 |
+
{"step": 12000, "loss": 0.49717794626951217, "answer_loss": 0.45482109814882277, "evidence_loss": 0.0847137027606368, "lr": 8.222392275927138e-07, "elapsed": 1988.552990436554}
|
| 241 |
+
{"step": 12050, "loss": 0.5193113712966442, "answer_loss": 0.48350312650203703, "evidence_loss": 0.0716164943948388, "lr": 7.433926299388355e-07, "elapsed": 1997.7018647193909}
|
| 242 |
+
{"step": 12100, "loss": 0.4932523432374001, "answer_loss": 0.4703754240274429, "evidence_loss": 0.04575384194031358, "lr": 6.684624273142842e-07, "elapsed": 2005.8355600833893}
|
| 243 |
+
{"step": 12150, "loss": 0.5193848550319672, "answer_loss": 0.48170405983924863, "evidence_loss": 0.07536159263923764, "lr": 5.974607177123475e-07, "elapsed": 2014.1573040485382}
|
| 244 |
+
{"step": 12200, "loss": 0.4953375631570816, "answer_loss": 0.4685724571347237, "evidence_loss": 0.05353021616116166, "lr": 5.303989648442642e-07, "elapsed": 2022.1448941230774}
|
| 245 |
+
{"step": 12250, "loss": 0.5246845097839832, "answer_loss": 0.48137510374188425, "evidence_loss": 0.08661880761384964, "lr": 4.6728799628833195e-07, "elapsed": 2030.1560821533203}
|
| 246 |
+
{"step": 12300, "loss": 0.5122852754592896, "answer_loss": 0.4672252982854843, "evidence_loss": 0.09011994928121567, "lr": 4.0813800174172854e-07, "elapsed": 2038.2643928527832}
|
| 247 |
+
{"step": 12350, "loss": 0.525602747797966, "answer_loss": 0.4862906587123871, "evidence_loss": 0.0786241720430553, "lr": 3.5295853137530253e-07, "elapsed": 2046.852712392807}
|
| 248 |
+
{"step": 12400, "loss": 0.5015444368124008, "answer_loss": 0.4740813094377518, "evidence_loss": 0.05492624830454588, "lr": 3.017584942916346e-07, "elapsed": 2053.891847848892}
|
| 249 |
+
{"step": 12450, "loss": 0.47327174320816995, "answer_loss": 0.4438498976826668, "evidence_loss": 0.05884368911385536, "lr": 2.5454615708662145e-07, "elapsed": 2061.785834312439}
|
| 250 |
+
{"step": 12500, "loss": 0.5140006586909294, "answer_loss": 0.48783220171928404, "evidence_loss": 0.052336912509053944, "lr": 2.11329142514749e-07, "elapsed": 2069.896514415741}
|
| 251 |
+
{"step": 12550, "loss": 0.5064806666970253, "answer_loss": 0.47209391444921495, "evidence_loss": 0.06877349996939301, "lr": 1.721144282583631e-07, "elapsed": 2079.53698182106}
|
| 252 |
+
{"step": 12600, "loss": 0.4902968233823776, "answer_loss": 0.45363484650850294, "evidence_loss": 0.07332395117729902, "lr": 1.3690834580107893e-07, "elapsed": 2087.7997212409973}
|
| 253 |
+
{"step": 12650, "loss": 0.5474013563990593, "answer_loss": 0.5003927388787269, "evidence_loss": 0.09401723494753242, "lr": 1.057165794055015e-07, "elapsed": 2095.761104106903}
|
| 254 |
+
{"step": 12700, "loss": 0.5126920866966248, "answer_loss": 0.482585714161396, "evidence_loss": 0.060212740264832976, "lr": 7.854416519548492e-08, "elapsed": 2104.194065093994}
|
| 255 |
+
{"step": 12750, "loss": 0.5073128491640091, "answer_loss": 0.46635954201221463, "evidence_loss": 0.0819066160544753, "lr": 5.539549034299374e-08, "elapsed": 2112.690991640091}
|
| 256 |
+
{"step": 12800, "loss": 0.5376325553655624, "answer_loss": 0.49825854659080504, "evidence_loss": 0.07874801294878125, "lr": 3.6274292359783635e-08, "elapsed": 2120.3258559703827}
|
| 257 |
+
{"step": 12850, "loss": 0.5015742993354797, "answer_loss": 0.470947183072567, "evidence_loss": 0.06125422954559326, "lr": 2.1183658493945102e-08, "elapsed": 2128.571123123169}
|
| 258 |
+
{"step": 12900, "loss": 0.5411059014499188, "answer_loss": 0.49526542916893956, "evidence_loss": 0.09168094459921122, "lr": 1.0126025231441127e-08, "elapsed": 2136.5031867027283}
|
| 259 |
+
{"step": 12950, "loss": 0.541961767077446, "answer_loss": 0.5026936477422714, "evidence_loss": 0.07853623609989882, "lr": 3.103177902730181e-09, "elapsed": 2145.4271173477173}
|
| 260 |
+
{"step": 13000, "loss": 0.5020205640792846, "answer_loss": 0.47167973399162294, "evidence_loss": 0.06068166099488735, "lr": 1.1625039451068631e-10, "elapsed": 2153.7755064964294}
|