rgup commited on
Commit
f14955f
·
verified ·
1 Parent(s): 42e4b5d

Restore comparisons with other quantizations and remove training-round comparison

Browse files
Files changed (3) hide show
  1. PACKAGE_SHA256.json +4 -4
  2. README.md +38 -19
  3. benchmarks/results.json +1154 -37
PACKAGE_SHA256.json CHANGED
@@ -542,13 +542,13 @@
542
  },
543
  {
544
  "path": "README.md",
545
- "bytes": 6779,
546
- "sha256": "4275b1c4ef97292d1c981cbe1127e0e97f39e6589c551188d8cdae73e4d05a3c"
547
  },
548
  {
549
  "path": "benchmarks/results.json",
550
- "bytes": 3502,
551
- "sha256": "bfd8925a3058580bfd997cf2b4dcb45bc032c5b500f5e5f1d9dc65f95ccbb051"
552
  },
553
  {
554
  "path": "flint_s_decode.py",
 
542
  },
543
  {
544
  "path": "README.md",
545
+ "bytes": 8221,
546
+ "sha256": "dfa9db74d9e4dee6a4abcf2ba16bbf2e00a0437c6fe13f5bb32af4016042efd3"
547
  },
548
  {
549
  "path": "benchmarks/results.json",
550
+ "bytes": 39638,
551
+ "sha256": "2b621bfc71531944663b64700474eba95185f475235ab90a1b4ea5160180aa89"
552
  },
553
  {
554
  "path": "flint_s_decode.py",
README.md CHANGED
@@ -15,7 +15,7 @@ tags:
15
 
16
  Mirai S quantization of [Recursive Flint](https://huggingface.co/RecursiveSI/flint), a mixture-of-experts coding-agent model with 39.8B total parameters and approximately 2.5B active parameters per token.
17
 
18
- The latest root-level package is **S3-13.3-qB-r2**: **13.157 GB of weights/shared tensors**, **13.173 GB including tokenizer and metadata**. It uses a custom packed S format and includes a standalone reference decoder.
19
 
20
  **Validation status:** the KL and agent results below were measured on this checkpoint's dense dequantized evaluation build. The packed files passed reconstruction comparisons against that build, but **decoded-package RomaKL and direct runtime integration remain unvalidated**. These files are not a drop-in Transformers, GGUF, MLX or uzu checkpoint. The supplied decoder reconstructs tensors; it does not provide a serving engine.
21
 
@@ -23,33 +23,52 @@ The latest root-level package is **S3-13.3-qB-r2**: **13.157 GB of weights/share
23
 
24
  Lower KL is better. Top-1 is agreement with the teacher's next-token choice, not task accuracy.
25
 
26
- | Checkpoint / execution | On-policy RomaKL ↓ | Top-1 agreement | Dataset-text RomaKL ↓ | Dataset top-1 |
27
  |---|---:|---:|---:|---:|
28
- | **S3-13.3-qB-r2**, dense evaluation build | **0.08924** | **91.55%** | **0.23373** | **87.37%** |
29
- | Preceding S3-13.3-qB, dense evaluation build | 0.09117 | 91.49% | 0.23781 | 87.44% |
30
- | Latest packed files through reference decoder | Pending | Pending | Pending | Pending |
31
-
32
- Both contracts use the same 100-prompt pool, rendered with Flint's chat template. The headline on-policy contract scores **76,946 tokens** from Flint-generated continuations. The dataset-text contract scores **34,631 tokens** from dataset responses.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
 
34
  Teacher cache and dense quantized evaluation use **FP32 activations** to reduce Flint's numerical instability on off-policy text. The BF16-activation teacher's measured floor against that cache is approximately **0.00359 KL / 98.36% top-1** on-policy and **0.0558 KL / 96.03% top-1** on dataset text. These numbers are specific to these contracts and precisions; they do not establish runtime-independent fidelity.
35
 
36
  ## Local agent benchmark
37
 
38
- These are **Mirai's in-house agent/task evaluations**, not official Terminal-Bench, SWE-bench or BFCL scores. All three columns use the recorded Flint SGLang evaluation lane and Flint's recommended sampling. The S columns evaluate dense dequantized twins, not the new packed decoder.
39
 
40
- | Evaluation | Latest S3-13.3-qB-r2 | Preceding S3-13.3-qB | BF16 Flint |
41
- |---|---:|---:|---:|
42
- | Function calling | 91% | 92% | 91% |
43
- | Function calling, hard | 72% | 75% | 77% |
44
- | Hard-problem battery | 44/55 | 51/55 | 49/55 |
45
- | AppWorld, 80 tasks × 3 runs | 161/240 (67.1%) | 158/240 (65.8%) | 187/240 (77.9%) |
46
- | Assistant week v2, native tools | 401/656 (61.1%) | 444/656 (67.7%) | 460/656 (70.1%) |
47
- | Assistant week v1, text markers | 37/136 (27.2%) | 41/136 (30.1%) | 60/136 (44.1%) |
48
- | Creative build, fully correct | 4/4 | 4/4 | 4/4 |
49
- | Agentic build, declared done | 4/4 | 4/4 | 4/4 |
50
 
51
- The latest checkpoint improves KL but regresses on several agent tasks relative to its predecessor. It is **not an across-the-board quality upgrade**. Results are recorded point estimates; this card does not claim statistically significant differences.
52
 
 
53
  Function-calling sets contain 175 standard and 150 hard cases, each spanning thinking/non-thinking runs and repeated samples (700/600 scored outputs). The hard battery contains 11 problems with five attempts each. Assistant v1 uses four seeds; v2 uses sixteen. AppWorld repeats each task three times. Creative/build sets contain only four tasks each. A declared-done result is not a render-quality pass; no latest-S agentic render result was graded. Historical text-marker sessions can loop or exhaust context, so v1 and native-tools v2 are reported separately.
54
 
55
  Sampling uses temperature **1.0**, top-p **0.95**, top-k **20**, with the original Flint template and task-specific token/step budgets. Aggregate scores, counts, checkpoint labels and hashes of the original result records are provided in [benchmarks/results.json](benchmarks/results.json). Training used Flint-generated coding, reasoning and tool traces; these are local evaluations, not an independent external audit.
 
15
 
16
  Mirai S quantization of [Recursive Flint](https://huggingface.co/RecursiveSI/flint), a mixture-of-experts coding-agent model with 39.8B total parameters and approximately 2.5B active parameters per token.
17
 
18
+ The root-level package is **Flint S 13.3**: **13.157 GB of weights/shared tensors**, **13.173 GB including tokenizer and metadata**. It uses a custom packed S format and includes a standalone reference decoder.
19
 
20
  **Validation status:** the KL and agent results below were measured on this checkpoint's dense dequantized evaluation build. The packed files passed reconstruction comparisons against that build, but **decoded-package RomaKL and direct runtime integration remain unvalidated**. These files are not a drop-in Transformers, GGUF, MLX or uzu checkpoint. The supplied decoder reconstructs tensors; it does not provide a serving engine.
21
 
 
23
 
24
  Lower KL is better. Top-1 is agreement with the teacher's next-token choice, not task accuracy.
25
 
26
+ | Quantization | Size (GB) | On-policy RomaKL ↓ | Teacher top-1 agreement | Dataset-text RomaKL ↓ |
27
  |---|---:|---:|---:|---:|
28
+ | **Mirai Flint S 13.3**, dense evaluation build | **13.17** | **0.08924** | **91.55%** | **0.23373** |
29
+ | Unsloth UD-IQ1_M | 11.35 | 0.20225 | 86.72% | 0.41156 |
30
+ | Unsloth UD-IQ2_XXS | 11.86 | 0.17714 | 87.51% | 0.37993 |
31
+ | bartowski IQ2_XXS | 12.03 | 0.27529 | 84.46% | 0.58081 |
32
+ | bartowski IQ2_XS | 12.74 | 0.22879 | 86.14% | 0.52038 |
33
+ | bartowski IQ2_S | 13.39 | 0.21386 | 86.62% | 0.48050 |
34
+ | Unsloth UD-Q2_K_XL | 13.56 | 0.13507 | 89.23% | 0.29978 |
35
+ | Unsloth UD-IQ3_XXS | 14.54 | 0.12238 | 89.97% | 0.33076 |
36
+ | bartowski IQ2_M | 14.88 | 0.15442 | 88.64% | 0.38832 |
37
+ | bartowski Q2_K | 15.48 | 0.17558 | 87.84% | 0.39941 |
38
+ | bartowski IQ3_XXS | 17.49 | 0.08920 | 91.42% | 0.25920 |
39
+ | bartowski IQ3_XS | 18.32 | 0.06739 | 92.50% | 0.25420 |
40
+ | bartowski Q3_K_S | 18.44 | 0.06659 | 92.57% | 0.23567 |
41
+ | Unsloth UD-Q3_K_XL | 18.49 | 0.05566 | 93.30% | 0.19121 |
42
+ | bartowski Q3_K_M | 19.53 | 0.06101 | 92.74% | 0.21117 |
43
+ | bartowski Q3_K_L | 20.59 | 0.05018 | 93.46% | 0.18979 |
44
+ | bartowski IQ3_M | 21.05 | 0.05041 | 93.51% | 0.19371 |
45
+ | bartowski IQ4_XS | 22.45 | 0.02505 | 95.52% | 0.13466 |
46
+ | Recursive nvfp4 | 22.86 | 0.06065 | 93.09% | 0.24904 |
47
+ | Recursive int4-q4_1 | 25.99 | 0.05691 | 93.21% | 0.22758 |
48
+ | Q8_0 | 42.30 | 0.00479 | 97.92% | 0.05890 |
49
+
50
+ The GGUF comparisons were measured through llama.cpp; Mirai S uses a dense evaluation twin. Recursive entries follow their recorded evaluation paths. These are cross-runtime measurements. Packed decoder KL is still pending. Both contracts use the same 100-prompt pool, rendered with Flint's chat template. The headline on-policy contract scores **76,946 tokens** from Flint-generated continuations. The dataset-text contract scores **34,631 tokens** from dataset responses.
51
 
52
  Teacher cache and dense quantized evaluation use **FP32 activations** to reduce Flint's numerical instability on off-policy text. The BF16-activation teacher's measured floor against that cache is approximately **0.00359 KL / 98.36% top-1** on-policy and **0.0558 KL / 96.03% top-1** on dataset text. These numbers are specific to these contracts and precisions; they do not establish runtime-independent fidelity.
53
 
54
  ## Local agent benchmark
55
 
56
+ These are **Mirai's in-house agent/task evaluations**, not official Terminal-Bench, SWE-bench or BFCL scores. The S column evaluates the dense dequantized twin. GGUFs use llama.cpp; the teacher uses SGLang. The packed decoder has not been agent-benchmarked.
57
 
58
+ | Evaluation | Flint S 13.3 | Unsloth UD-Q2_K_XL | bartowski IQ3_XXS | Q8_0 | BF16 Flint |
59
+ |---|---:|---:|---:|---:|---:|
60
+ | Function calling | 91% | 97% | 94% | 96% | 91% |
61
+ | Function calling, hard | 72% | 76% | 73% | 75% | 77% |
62
+ | Hard-problem battery | 44/55 | 44/55 | 38/55 | 52/55 | 49/55 |
63
+ | AppWorld, 80 tasks × 3 | 161/240 | 158/240 | Unvalidated¹ | 164/240 | 187/240 |
64
+ | Assistant week v2, native tools | 401/656 | 242/656 | 333/656 | 358/656 | 460/656 |
65
+ | Assistant week v1, text markers | 37/136 | 18/136 | 36/136 | 57/136 | 60/136 |
66
+ | Creative build, fully correct | 4/4 | 4/4 | 4/4 | 4/4 | 4/4 |
67
+ | Agentic build, declared done | 4/4 | 2/4 | 3/4 | 4/4 | 4/4 |
68
 
69
+ ¹ The bartowski AppWorld record completed 240 episodes in only 108 seconds. That result requires harness/protocol review and is excluded as a valid model-quality comparison. It is not presented as 0% task ability.
70
 
71
+ S and BF16 agent runs used SGLang; GGUF agent runs used llama.cpp. Backend, numerical and tool-protocol differences can affect results. These are recorded point estimates; this card does not claim statistically significant differences.
72
  Function-calling sets contain 175 standard and 150 hard cases, each spanning thinking/non-thinking runs and repeated samples (700/600 scored outputs). The hard battery contains 11 problems with five attempts each. Assistant v1 uses four seeds; v2 uses sixteen. AppWorld repeats each task three times. Creative/build sets contain only four tasks each. A declared-done result is not a render-quality pass; no latest-S agentic render result was graded. Historical text-marker sessions can loop or exhaust context, so v1 and native-tools v2 are reported separately.
73
 
74
  Sampling uses temperature **1.0**, top-p **0.95**, top-k **20**, with the original Flint template and task-specific token/step budgets. Aggregate scores, counts, checkpoint labels and hashes of the original result records are provided in [benchmarks/results.json](benchmarks/results.json). Training used Flint-generated coding, reasoning and tool traces; these are local evaluations, not an independent external audit.
benchmarks/results.json CHANGED
@@ -18,23 +18,6 @@
18
  "tokens": 34631
19
  }
20
  },
21
- "previous_roma": {
22
- "fits": [
23
- "builds/S3-13.3-qB"
24
- ],
25
- "overlay_files": 50,
26
- "act": "fp32",
27
- "onpolicy": {
28
- "roma_full_kl": 0.09117273392857145,
29
- "roma_full_top1": 0.9148883632677463,
30
- "tokens": 76946
31
- },
32
- "dataset": {
33
- "roma_full_kl": 0.2378088151205421,
34
- "roma_full_top1": 0.8743899974011724,
35
- "tokens": 34631
36
- }
37
- },
38
  "agent_results": [
39
  {
40
  "tag": "flint-teacher",
@@ -73,25 +56,6 @@
73
  "agentic_renders": "0/0 graded (0 with no page errors)"
74
  },
75
  "source_sha256": "668058563330380c5350ab0b6c529565ad2b85b10e1f48b5f9c288cddfde6ea1"
76
- },
77
- {
78
- "tag": "flint-S3-13.3",
79
- "headline": {
80
- "fc_pct": 92,
81
- "fc_hard_pct": 75,
82
- "hard_solved": "51/55",
83
- "assistant_tasks": "41/136",
84
- "assistant_v2_tasks": "444/656",
85
- "assistant_v2_hard_traps_pct": 48.0,
86
- "assistant_v2_traps_pct": 61.2,
87
- "assistant_v2_basics_pct": 85.2,
88
- "appworld_success": "158/240",
89
- "creative_fully_correct": "4/4",
90
- "creative_declared_done": "3/4",
91
- "agentic_declared_done": "4/4",
92
- "agentic_renders": "0/0 graded (0 with no page errors)"
93
- },
94
- "source_sha256": "1f3f10a90434d733b900289f6682d955096296d21a316d483ac2afa12288e3f6"
95
  }
96
  ],
97
  "contracts": {
@@ -107,5 +71,1158 @@
107
  "hard_attempts": 5,
108
  "creative_tasks": 4
109
  },
110
- "source_scores": "Original record hashes are included per agent result; no prompts, responses, trajectories, or private infrastructure configuration are published."
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
111
  }
 
18
  "tokens": 34631
19
  }
20
  },
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
21
  "agent_results": [
22
  {
23
  "tag": "flint-teacher",
 
56
  "agentic_renders": "0/0 graded (0 with no page errors)"
57
  },
58
  "source_sha256": "668058563330380c5350ab0b6c529565ad2b85b10e1f48b5f9c288cddfde6ea1"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
59
  }
60
  ],
61
  "contracts": {
 
71
  "hard_attempts": 5,
72
  "creative_tasks": 4
73
  },
74
+ "source_scores": "Original record hashes are included per agent result; no prompts, responses, trajectories, or private infrastructure configuration are published.",
75
+ "other_quantizations": {
76
+ "roma": {
77
+ "unsloth-onpolicy.json": {
78
+ "data": {
79
+ "flint-Q8_0": {
80
+ "bytes": 42301076928,
81
+ "roma_full_kl": 0.00479167182613063,
82
+ "roma_full_top1": 0.9792061965534271,
83
+ "tokens": 76946,
84
+ "per_part": {
85
+ "hermes": {
86
+ "kl": 0.005444003563814829,
87
+ "top1": 0.9784891684846025
88
+ },
89
+ "openorca": {
90
+ "kl": 0.0036573213217411795,
91
+ "top1": 0.983330139873539
92
+ },
93
+ "tulu3": {
94
+ "kl": 0.0034202372074492505,
95
+ "top1": 0.9809652896458247
96
+ },
97
+ "ultrachat": {
98
+ "kl": 0.0059365492697543735,
99
+ "top1": 0.9759437604128571
100
+ }
101
+ }
102
+ },
103
+ "flint-UD-IQ1_M": {
104
+ "bytes": 11352241600,
105
+ "roma_full_kl": 0.20224687618285064,
106
+ "roma_full_top1": 0.8672055727393236,
107
+ "tokens": 76946,
108
+ "per_part": {
109
+ "hermes": {
110
+ "kl": 0.2298944665927102,
111
+ "top1": 0.8548526203642636
112
+ },
113
+ "openorca": {
114
+ "kl": 0.15223449669997718,
115
+ "top1": 0.8910391518170786
116
+ },
117
+ "tulu3": {
118
+ "kl": 0.11671532943341621,
119
+ "top1": 0.8913312511049561
120
+ },
121
+ "ultrachat": {
122
+ "kl": 0.2708992792787001,
123
+ "top1": 0.8453004998171401
124
+ }
125
+ }
126
+ },
127
+ "flint-UD-IQ2_XXS": {
128
+ "bytes": 11855541696,
129
+ "roma_full_kl": 0.1771357678376779,
130
+ "roma_full_top1": 0.8751332103033296,
131
+ "tokens": 76946,
132
+ "per_part": {
133
+ "hermes": {
134
+ "kl": 0.20149494554044436,
135
+ "top1": 0.8632235807417178
136
+ },
137
+ "openorca": {
138
+ "kl": 0.13305788685202394,
139
+ "top1": 0.8967873794468928
140
+ },
141
+ "tulu3": {
142
+ "kl": 0.09592487023964087,
143
+ "top1": 0.897695798220284
144
+ },
145
+ "ultrachat": {
146
+ "kl": 0.24166709183435572,
147
+ "top1": 0.8553374781583973
148
+ }
149
+ }
150
+ },
151
+ "flint-UD-Q2_K_XL": {
152
+ "bytes": 13562207680,
153
+ "roma_full_kl": 0.1350727829064218,
154
+ "roma_full_top1": 0.8922751020195981,
155
+ "tokens": 76946,
156
+ "per_part": {
157
+ "hermes": {
158
+ "kl": 0.1530615428128953,
159
+ "top1": 0.8827050885292477
160
+ },
161
+ "openorca": {
162
+ "kl": 0.10128456250212846,
163
+ "top1": 0.9107108641502204
164
+ },
165
+ "tulu3": {
166
+ "kl": 0.07149972509035372,
167
+ "top1": 0.9137839589840298
168
+ },
169
+ "ultrachat": {
170
+ "kl": 0.1859979297402786,
171
+ "top1": 0.8733796578487545
172
+ }
173
+ }
174
+ },
175
+ "flint-UD-IQ3_XXS": {
176
+ "bytes": 14535427520,
177
+ "roma_full_kl": 0.12237937476498459,
178
+ "roma_full_top1": 0.8997478751332103,
179
+ "tokens": 76946,
180
+ "per_part": {
181
+ "hermes": {
182
+ "kl": 0.1381509494796009,
183
+ "top1": 0.8906194510679316
184
+ },
185
+ "openorca": {
186
+ "kl": 0.09297863371323342,
187
+ "top1": 0.9148623618828639
188
+ },
189
+ "tulu3": {
190
+ "kl": 0.06709383448557438,
191
+ "top1": 0.9173198184925452
192
+ },
193
+ "ultrachat": {
194
+ "kl": 0.16657435328702033,
195
+ "top1": 0.8853265065626397
196
+ }
197
+ }
198
+ }
199
+ },
200
+ "sha256": "eb7f3ec4fab31e3436806648474dfff983d3aedcaeb64718802ea343bf5da7f6"
201
+ },
202
+ "unsloth-dataset.json": {
203
+ "data": {
204
+ "flint-Q8_0": {
205
+ "bytes": 42301076928,
206
+ "roma_full_kl": 0.0589025754806496,
207
+ "roma_full_top1": 0.9548670266524213,
208
+ "tokens": 34631,
209
+ "per_part": {
210
+ "hermes": {
211
+ "kl": 0.03650696883640045,
212
+ "top1": 0.9649923896499238
213
+ },
214
+ "openorca": {
215
+ "kl": 0.06037278225442893,
216
+ "top1": 0.957212174680194
217
+ },
218
+ "tulu3": {
219
+ "kl": 0.05828087377024946,
220
+ "top1": 0.9632976761342678
221
+ },
222
+ "ultrachat": {
223
+ "kl": 0.06593636903818785,
224
+ "top1": 0.944724989763887
225
+ }
226
+ }
227
+ },
228
+ "flint-UD-IQ1_M": {
229
+ "bytes": 11352241600,
230
+ "roma_full_kl": 0.41155755830093177,
231
+ "roma_full_top1": 0.8322023620455661,
232
+ "tokens": 34631,
233
+ "per_part": {
234
+ "hermes": {
235
+ "kl": 0.33136820327004557,
236
+ "top1": 0.8488801913459447
237
+ },
238
+ "openorca": {
239
+ "kl": 0.33897910257087827,
240
+ "top1": 0.8400970445522717
241
+ },
242
+ "tulu3": {
243
+ "kl": 0.3458158376251187,
244
+ "top1": 0.8808557727775729
245
+ },
246
+ "ultrachat": {
247
+ "kl": 0.5078292098005651,
248
+ "top1": 0.7885219052818343
249
+ }
250
+ }
251
+ },
252
+ "flint-UD-IQ2_XXS": {
253
+ "bytes": 11855541696,
254
+ "roma_full_kl": 0.3799256873199688,
255
+ "roma_full_top1": 0.8389304380468366,
256
+ "tokens": 34631,
257
+ "per_part": {
258
+ "hermes": {
259
+ "kl": 0.3151189705880647,
260
+ "top1": 0.8562731028484453
261
+ },
262
+ "openorca": {
263
+ "kl": 0.3114770795443852,
264
+ "top1": 0.846934274371416
265
+ },
266
+ "tulu3": {
267
+ "kl": 0.30933506626873514,
268
+ "top1": 0.8867576540022132
269
+ },
270
+ "ultrachat": {
271
+ "kl": 0.4736800733295293,
272
+ "top1": 0.7956189436331377
273
+ }
274
+ }
275
+ },
276
+ "flint-UD-Q2_K_XL": {
277
+ "bytes": 13562207680,
278
+ "roma_full_kl": 0.2997755177231634,
279
+ "roma_full_top1": 0.8575553694666628,
280
+ "tokens": 34631,
281
+ "per_part": {
282
+ "hermes": {
283
+ "kl": 0.2330365731899623,
284
+ "top1": 0.8686671015438139
285
+ },
286
+ "openorca": {
287
+ "kl": 0.2663741161910162,
288
+ "top1": 0.8665637406263784
289
+ },
290
+ "tulu3": {
291
+ "kl": 0.2698330293498502,
292
+ "top1": 0.8948727406860937
293
+ },
294
+ "ultrachat": {
295
+ "kl": 0.35321285253790047,
296
+ "top1": 0.8236658932714617
297
+ }
298
+ }
299
+ },
300
+ "flint-UD-IQ3_XXS": {
301
+ "bytes": 14535427520,
302
+ "roma_full_kl": 0.3307611594507763,
303
+ "roma_full_top1": 0.8568334729000029,
304
+ "tokens": 34631,
305
+ "per_part": {
306
+ "hermes": {
307
+ "kl": 0.26240822635115585,
308
+ "top1": 0.8747553816046967
309
+ },
310
+ "openorca": {
311
+ "kl": 0.29384812172336516,
312
+ "top1": 0.8661226290251434
313
+ },
314
+ "tulu3": {
315
+ "kl": 0.2851426130025337,
316
+ "top1": 0.8934894872740686
317
+ },
318
+ "ultrachat": {
319
+ "kl": 0.39739186574707613,
320
+ "top1": 0.8212092261498567
321
+ }
322
+ }
323
+ }
324
+ },
325
+ "sha256": "ae8d3daba6254757fc0f97c12fb0c460bcb64c9b135ab2f9558079c423e17d1a"
326
+ },
327
+ "comp-onpolicy.json": {
328
+ "data": {
329
+ "unsloth-UD-Q3_K_XL": {
330
+ "bytes": 18494977472,
331
+ "roma_full_kl": 0.05566215057640511,
332
+ "roma_full_top1": 0.9329919683934188,
333
+ "tokens": 76946,
334
+ "per_part": {
335
+ "hermes": {
336
+ "kl": 0.06320647396614826,
337
+ "top1": 0.929125868804221
338
+ },
339
+ "openorca": {
340
+ "kl": 0.03910942458935896,
341
+ "top1": 0.9473717825892572
342
+ },
343
+ "tulu3": {
344
+ "kl": 0.02920153779043278,
345
+ "top1": 0.942483351994814
346
+ },
347
+ "ultrachat": {
348
+ "kl": 0.07839651649168483,
349
+ "top1": 0.9203949774472754
350
+ }
351
+ }
352
+ },
353
+ "bartowski-IQ2_XXS": {
354
+ "bytes": 12029007072,
355
+ "roma_full_kl": 0.27528629971514795,
356
+ "roma_full_top1": 0.8445923114911756,
357
+ "tokens": 76946,
358
+ "per_part": {
359
+ "hermes": {
360
+ "kl": 0.29875951420256286,
361
+ "top1": 0.8338491197808331
362
+ },
363
+ "openorca": {
364
+ "kl": 0.21483797276926853,
365
+ "top1": 0.873858338123523
366
+ },
367
+ "tulu3": {
368
+ "kl": 0.1844957765211332,
369
+ "top1": 0.8622193411515116
370
+ },
371
+ "ultrachat": {
372
+ "kl": 0.35754820846029955,
373
+ "top1": 0.8224226908854484
374
+ }
375
+ }
376
+ },
377
+ "bartowski-IQ2_XS": {
378
+ "bytes": 12742325472,
379
+ "roma_full_kl": 0.2287888584363031,
380
+ "roma_full_top1": 0.8614093000285915,
381
+ "tokens": 76946,
382
+ "per_part": {
383
+ "hermes": {
384
+ "kl": 0.256213093244239,
385
+ "top1": 0.8499315103241845
386
+ },
387
+ "openorca": {
388
+ "kl": 0.16851959647579257,
389
+ "top1": 0.8880372996103979
390
+ },
391
+ "tulu3": {
392
+ "kl": 0.14691329641666442,
393
+ "top1": 0.8780717779480228
394
+ },
395
+ "ultrachat": {
396
+ "kl": 0.3016249409316396,
397
+ "top1": 0.8421715632492177
398
+ }
399
+ }
400
+ },
401
+ "bartowski-IQ2_S": {
402
+ "bytes": 13394982112,
403
+ "roma_full_kl": 0.21385846626203012,
404
+ "roma_full_top1": 0.8662048709484573,
405
+ "tokens": 76946,
406
+ "per_part": {
407
+ "hermes": {
408
+ "kl": 0.23946053053677024,
409
+ "top1": 0.8585561361676222
410
+ },
411
+ "openorca": {
412
+ "kl": 0.15804821398627175,
413
+ "top1": 0.8913584977965128
414
+ },
415
+ "tulu3": {
416
+ "kl": 0.13600105435363924,
417
+ "top1": 0.881666568448347
418
+ },
419
+ "ultrachat": {
420
+ "kl": 0.28254639562265554,
421
+ "top1": 0.8456662196757284
422
+ }
423
+ }
424
+ },
425
+ "bartowski-IQ2_M": {
426
+ "bytes": 14883566816,
427
+ "roma_full_kl": 0.1544153491847061,
428
+ "roma_full_top1": 0.8864268448002496,
429
+ "tokens": 76946,
430
+ "per_part": {
431
+ "hermes": {
432
+ "kl": 0.17152951506500067,
433
+ "top1": 0.876870782811628
434
+ },
435
+ "openorca": {
436
+ "kl": 0.11353285757146317,
437
+ "top1": 0.9091141342530498
438
+ },
439
+ "tulu3": {
440
+ "kl": 0.09124183323199414,
441
+ "top1": 0.9037656903765691
442
+ },
443
+ "ultrachat": {
444
+ "kl": 0.21027910796091098,
445
+ "top1": 0.867690682270714
446
+ }
447
+ }
448
+ },
449
+ "bartowski-Q2_K": {
450
+ "bytes": 15480337632,
451
+ "roma_full_kl": 0.17558322447219382,
452
+ "roma_full_top1": 0.8784472227276272,
453
+ "tokens": 76946,
454
+ "per_part": {
455
+ "hermes": {
456
+ "kl": 0.2029540169021926,
457
+ "top1": 0.8670285627314698
458
+ },
459
+ "openorca": {
460
+ "kl": 0.13150172834462753,
461
+ "top1": 0.9033659066232356
462
+ },
463
+ "tulu3": {
464
+ "kl": 0.11010943072259065,
465
+ "top1": 0.8950439035888974
466
+ },
467
+ "ultrachat": {
468
+ "kl": 0.22685321112011192,
469
+ "top1": 0.8602950140192612
470
+ }
471
+ }
472
+ },
473
+ "bartowski-IQ3_XXS": {
474
+ "bytes": 17492718816,
475
+ "roma_full_kl": 0.08919584161171291,
476
+ "roma_full_top1": 0.9142385569100409,
477
+ "tokens": 76946,
478
+ "per_part": {
479
+ "hermes": {
480
+ "kl": 0.1014002695408267,
481
+ "top1": 0.9098980264826746
482
+ },
483
+ "openorca": {
484
+ "kl": 0.06295285983553729,
485
+ "top1": 0.9310851376381172
486
+ },
487
+ "tulu3": {
488
+ "kl": 0.05267312838758785,
489
+ "top1": 0.9240968825505333
490
+ },
491
+ "ultrachat": {
492
+ "kl": 0.12130112047143893,
493
+ "top1": 0.9001991141452315
494
+ }
495
+ }
496
+ },
497
+ "bartowski-IQ3_XS": {
498
+ "bytes": 18315457760,
499
+ "roma_full_kl": 0.06738706705825724,
500
+ "roma_full_top1": 0.9249863540664882,
501
+ "tokens": 76946,
502
+ "per_part": {
503
+ "hermes": {
504
+ "kl": 0.07501005292404467,
505
+ "top1": 0.9209578407995536
506
+ },
507
+ "openorca": {
508
+ "kl": 0.04657372521084144,
509
+ "top1": 0.9396436098869515
510
+ },
511
+ "tulu3": {
512
+ "kl": 0.041879494713306625,
513
+ "top1": 0.9342919441334198
514
+ },
515
+ "ultrachat": {
516
+ "kl": 0.09211199753190187,
517
+ "top1": 0.9124710471778618
518
+ }
519
+ }
520
+ },
521
+ "bartowski-Q3_K_S": {
522
+ "bytes": 18443810016,
523
+ "roma_full_kl": 0.06659041701332871,
524
+ "roma_full_top1": 0.9256621526785018,
525
+ "tokens": 76946,
526
+ "per_part": {
527
+ "hermes": {
528
+ "kl": 0.07514252865140911,
529
+ "top1": 0.919385114910456
530
+ },
531
+ "openorca": {
532
+ "kl": 0.048822925078150885,
533
+ "top1": 0.940920993804688
534
+ },
535
+ "tulu3": {
536
+ "kl": 0.04053502031822324,
537
+ "top1": 0.9327008073545878
538
+ },
539
+ "ultrachat": {
540
+ "kl": 0.08901103448052046,
541
+ "top1": 0.916128245763745
542
+ }
543
+ }
544
+ },
545
+ "bartowski-Q3_K_M": {
546
+ "bytes": 19529151712,
547
+ "roma_full_kl": 0.06101469356474003,
548
+ "roma_full_top1": 0.9273906375899982,
549
+ "tokens": 76946,
550
+ "per_part": {
551
+ "hermes": {
552
+ "kl": 0.06864508408751639,
553
+ "top1": 0.922530566688651
554
+ },
555
+ "openorca": {
556
+ "kl": 0.04385634727037036,
557
+ "top1": 0.9431564156607268
558
+ },
559
+ "tulu3": {
560
+ "kl": 0.03808601397547445,
561
+ "top1": 0.9337026342153338
562
+ },
563
+ "ultrachat": {
564
+ "kl": 0.08163001171203363,
565
+ "top1": 0.9169003210207648
566
+ }
567
+ }
568
+ },
569
+ "bartowski-Q3_K_L": {
570
+ "bytes": 20594504928,
571
+ "roma_full_kl": 0.05017589305668176,
572
+ "roma_full_top1": 0.9345904920333741,
573
+ "tokens": 76946,
574
+ "per_part": {
575
+ "hermes": {
576
+ "kl": 0.05805480820688384,
577
+ "top1": 0.9283141393130739
578
+ },
579
+ "openorca": {
580
+ "kl": 0.03711018169749291,
581
+ "top1": 0.9460943986715207
582
+ },
583
+ "tulu3": {
584
+ "kl": 0.028465198777843023,
585
+ "top1": 0.9436619718309859
586
+ },
587
+ "ultrachat": {
588
+ "kl": 0.06714843633120739,
589
+ "top1": 0.9260433174854728
590
+ }
591
+ }
592
+ },
593
+ "bartowski-IQ3_M": {
594
+ "bytes": 21051684064,
595
+ "roma_full_kl": 0.05041165370210903,
596
+ "roma_full_top1": 0.9351233332466925,
597
+ "tokens": 76946,
598
+ "per_part": {
599
+ "hermes": {
600
+ "kl": 0.056391244306937365,
601
+ "top1": 0.931611790370859
602
+ },
603
+ "openorca": {
604
+ "kl": 0.035838911479602276,
605
+ "top1": 0.9469885674139362
606
+ },
607
+ "tulu3": {
608
+ "kl": 0.03134143587850121,
609
+ "top1": 0.941186870175025
610
+ },
611
+ "ultrachat": {
612
+ "kl": 0.06804358676784995,
613
+ "top1": 0.9262058596448454
614
+ }
615
+ }
616
+ },
617
+ "bartowski-IQ4_XS": {
618
+ "bytes": 22446290144,
619
+ "roma_full_kl": 0.025047515686743255,
620
+ "roma_full_top1": 0.9551633613183271,
621
+ "tokens": 76946,
622
+ "per_part": {
623
+ "hermes": {
624
+ "kl": 0.029147944973638918,
625
+ "top1": 0.9523108923951094
626
+ },
627
+ "openorca": {
628
+ "kl": 0.01781637295381071,
629
+ "top1": 0.9632752123650763
630
+ },
631
+ "tulu3": {
632
+ "kl": 0.015259294614843013,
633
+ "top1": 0.9608698214390948
634
+ },
635
+ "ultrachat": {
636
+ "kl": 0.03311329501400562,
637
+ "top1": 0.9483522288593604
638
+ }
639
+ }
640
+ }
641
+ },
642
+ "sha256": "1a5213d0f1ae7d3445b49c609730fd009e0727d2276ed167314b95e3d54efe2b"
643
+ },
644
+ "comp-dataset.json": {
645
+ "data": {
646
+ "unsloth-UD-Q3_K_XL": {
647
+ "bytes": 18494977472,
648
+ "roma_full_kl": 0.1912116175375837,
649
+ "roma_full_top1": 0.8946608529929831,
650
+ "tokens": 34631,
651
+ "per_part": {
652
+ "hermes": {
653
+ "kl": 0.12661880863897387,
654
+ "top1": 0.9078060447923462
655
+ },
656
+ "openorca": {
657
+ "kl": 0.1720191087555871,
658
+ "top1": 0.8978826643140715
659
+ },
660
+ "tulu3": {
661
+ "kl": 0.17640567353536,
662
+ "top1": 0.9213389893028403
663
+ },
664
+ "ultrachat": {
665
+ "kl": 0.2283780444314279,
666
+ "top1": 0.8697966425549338
667
+ }
668
+ }
669
+ },
670
+ "bartowski-IQ2_XXS": {
671
+ "bytes": 12029007072,
672
+ "roma_full_kl": 0.5808053629049501,
673
+ "roma_full_top1": 0.7973491958072247,
674
+ "tokens": 34631,
675
+ "per_part": {
676
+ "hermes": {
677
+ "kl": 0.47207306562822654,
678
+ "top1": 0.822135246792781
679
+ },
680
+ "openorca": {
681
+ "kl": 0.49824604741720435,
682
+ "top1": 0.80414644905161
683
+ },
684
+ "tulu3": {
685
+ "kl": 0.4983910815999852,
686
+ "top1": 0.8471043895241608
687
+ },
688
+ "ultrachat": {
689
+ "kl": 0.7014607633463403,
690
+ "top1": 0.7506482871570902
691
+ }
692
+ }
693
+ },
694
+ "bartowski-IQ2_XS": {
695
+ "bytes": 12742325472,
696
+ "roma_full_kl": 0.520381094225591,
697
+ "roma_full_top1": 0.8116427478270913,
698
+ "tokens": 34631,
699
+ "per_part": {
700
+ "hermes": {
701
+ "kl": 0.42914125353752414,
702
+ "top1": 0.822352685366384
703
+ },
704
+ "openorca": {
705
+ "kl": 0.4442211246963333,
706
+ "top1": 0.8171592412880458
707
+ },
708
+ "tulu3": {
709
+ "kl": 0.4764506070211953,
710
+ "top1": 0.8553116931021764
711
+ },
712
+ "ultrachat": {
713
+ "kl": 0.6050885824482292,
714
+ "top1": 0.7742595878258496
715
+ }
716
+ }
717
+ },
718
+ "bartowski-IQ2_S": {
719
+ "bytes": 13394982112,
720
+ "roma_full_kl": 0.48049566373630914,
721
+ "roma_full_top1": 0.8175045479483699,
722
+ "tokens": 34631,
723
+ "per_part": {
724
+ "hermes": {
725
+ "kl": 0.4055934151110895,
726
+ "top1": 0.8321374211785171
727
+ },
728
+ "openorca": {
729
+ "kl": 0.4050338318728052,
730
+ "top1": 0.8202470224966917
731
+ },
732
+ "tulu3": {
733
+ "kl": 0.4197828346668948,
734
+ "top1": 0.8619513094798967
735
+ },
736
+ "ultrachat": {
737
+ "kl": 0.5722787472305934,
738
+ "top1": 0.7791729220690596
739
+ }
740
+ }
741
+ },
742
+ "bartowski-IQ2_M": {
743
+ "bytes": 14883566816,
744
+ "roma_full_kl": 0.3883215377972164,
745
+ "roma_full_top1": 0.8424532932921371,
746
+ "tokens": 34631,
747
+ "per_part": {
748
+ "hermes": {
749
+ "kl": 0.32284034893020364,
750
+ "top1": 0.8549684714068275
751
+ },
752
+ "openorca": {
753
+ "kl": 0.31936347172168605,
754
+ "top1": 0.8484781649757389
755
+ },
756
+ "tulu3": {
757
+ "kl": 0.3440151856597259,
758
+ "top1": 0.8839911471781631
759
+ },
760
+ "ultrachat": {
761
+ "kl": 0.4629947970960286,
762
+ "top1": 0.8059232973932032
763
+ }
764
+ }
765
+ },
766
+ "bartowski-Q2_K": {
767
+ "bytes": 15480337632,
768
+ "roma_full_kl": 0.3994066643024151,
769
+ "roma_full_top1": 0.8426265484681355,
770
+ "tokens": 34631,
771
+ "per_part": {
772
+ "hermes": {
773
+ "kl": 0.3230408278204767,
774
+ "top1": 0.8645357686453576
775
+ },
776
+ "openorca": {
777
+ "kl": 0.3002006254868988,
778
+ "top1": 0.8524481693868549
779
+ },
780
+ "tulu3": {
781
+ "kl": 0.34578507485665855,
782
+ "top1": 0.8826078937661379
783
+ },
784
+ "ultrachat": {
785
+ "kl": 0.493748084932944,
786
+ "top1": 0.8031254265047086
787
+ }
788
+ }
789
+ },
790
+ "bartowski-IQ3_XXS": {
791
+ "bytes": 17492718816,
792
+ "roma_full_kl": 0.25919877637932764,
793
+ "roma_full_top1": 0.8768733215904825,
794
+ "tokens": 34631,
795
+ "per_part": {
796
+ "hermes": {
797
+ "kl": 0.21159556642743052,
798
+ "top1": 0.8884540117416829
799
+ },
800
+ "openorca": {
801
+ "kl": 0.23436352779776184,
802
+ "top1": 0.8806793118659021
803
+ },
804
+ "tulu3": {
805
+ "kl": 0.2272679871714278,
806
+ "top1": 0.9082441903356695
807
+ },
808
+ "ultrachat": {
809
+ "kl": 0.30545151677840365,
810
+ "top1": 0.8488467312679132
811
+ }
812
+ }
813
+ },
814
+ "bartowski-IQ3_XS": {
815
+ "bytes": 18315457760,
816
+ "roma_full_kl": 0.25419950681293946,
817
+ "roma_full_top1": 0.8810603216771101,
818
+ "tokens": 34631,
819
+ "per_part": {
820
+ "hermes": {
821
+ "kl": 0.18583901941810296,
822
+ "top1": 0.8988910632746249
823
+ },
824
+ "openorca": {
825
+ "kl": 0.2352470428566772,
826
+ "top1": 0.8800176444640494
827
+ },
828
+ "tulu3": {
829
+ "kl": 0.2178074133347779,
830
+ "top1": 0.9107340464773146
831
+ },
832
+ "ultrachat": {
833
+ "kl": 0.3084479178122386,
834
+ "top1": 0.8538283062645011
835
+ }
836
+ }
837
+ },
838
+ "bartowski-Q3_K_S": {
839
+ "bytes": 18443810016,
840
+ "roma_full_kl": 0.23566812077703728,
841
+ "roma_full_top1": 0.8872397562877191,
842
+ "tokens": 34631,
843
+ "per_part": {
844
+ "hermes": {
845
+ "kl": 0.16959970664576293,
846
+ "top1": 0.9075886062187432
847
+ },
848
+ "openorca": {
849
+ "kl": 0.20968457455691306,
850
+ "top1": 0.898103220114689
851
+ },
852
+ "tulu3": {
853
+ "kl": 0.21105634431828435,
854
+ "top1": 0.9138694208779048
855
+ },
856
+ "ultrachat": {
857
+ "kl": 0.2826551645241705,
858
+ "top1": 0.8577862699604204
859
+ }
860
+ }
861
+ },
862
+ "bartowski-Q3_K_M": {
863
+ "bytes": 19529151712,
864
+ "roma_full_kl": 0.21116916491284823,
865
+ "roma_full_top1": 0.8916000115503451,
866
+ "tokens": 34631,
867
+ "per_part": {
868
+ "hermes": {
869
+ "kl": 0.15399352954928155,
870
+ "top1": 0.9051967819091107
871
+ },
872
+ "openorca": {
873
+ "kl": 0.18051770524076768,
874
+ "top1": 0.8967798853109836
875
+ },
876
+ "tulu3": {
877
+ "kl": 0.19111078913824797,
878
+ "top1": 0.917189229066765
879
+ },
880
+ "ultrachat": {
881
+ "kl": 0.25344004604359904,
882
+ "top1": 0.8667940494063054
883
+ }
884
+ }
885
+ },
886
+ "bartowski-Q3_K_L": {
887
+ "bytes": 20594504928,
888
+ "roma_full_kl": 0.189786236174978,
889
+ "roma_full_top1": 0.8992809910196067,
890
+ "tokens": 34631,
891
+ "per_part": {
892
+ "hermes": {
893
+ "kl": 0.13858943565603246,
894
+ "top1": 0.9173733420308763
895
+ },
896
+ "openorca": {
897
+ "kl": 0.15909851225384072,
898
+ "top1": 0.904058226731363
899
+ },
900
+ "tulu3": {
901
+ "kl": 0.17630993796643715,
902
+ "top1": 0.9224455920324603
903
+ },
904
+ "ultrachat": {
905
+ "kl": 0.22532118933578624,
906
+ "top1": 0.8749829398116555
907
+ }
908
+ }
909
+ },
910
+ "bartowski-IQ3_M": {
911
+ "bytes": 21051684064,
912
+ "roma_full_kl": 0.1937088909866322,
913
+ "roma_full_top1": 0.8969709220062949,
914
+ "tokens": 34631,
915
+ "per_part": {
916
+ "hermes": {
917
+ "kl": 0.14436526627914267,
918
+ "top1": 0.9078060447923462
919
+ },
920
+ "openorca": {
921
+ "kl": 0.16816659396129935,
922
+ "top1": 0.8998676665196295
923
+ },
924
+ "tulu3": {
925
+ "kl": 0.17494949006981536,
926
+ "top1": 0.9216156399852453
927
+ },
928
+ "ultrachat": {
929
+ "kl": 0.23097974183176426,
930
+ "top1": 0.8744370137846322
931
+ }
932
+ }
933
+ },
934
+ "bartowski-IQ4_XS": {
935
+ "bytes": 22446290144,
936
+ "roma_full_kl": 0.13466023926445347,
937
+ "roma_full_top1": 0.9206491293927406,
938
+ "tokens": 34631,
939
+ "per_part": {
940
+ "hermes": {
941
+ "kl": 0.0961063527561653,
942
+ "top1": 0.9315068493150684
943
+ },
944
+ "openorca": {
945
+ "kl": 0.11873246033856544,
946
+ "top1": 0.9245699161887958
947
+ },
948
+ "tulu3": {
949
+ "kl": 0.1313387924610843,
950
+ "top1": 0.9405201032829215
951
+ },
952
+ "ultrachat": {
953
+ "kl": 0.15414595257394778,
954
+ "top1": 0.9013238706155315
955
+ }
956
+ }
957
+ }
958
+ },
959
+ "sha256": "6bbc6646121bd092bf153d31d0df1d5e1b3ae5a266fb4f99981d80728ce19044"
960
+ },
961
+ "comp2-onpolicy.json": {
962
+ "data": {
963
+ "recursive-nvfp4": {
964
+ "bytes": 22863084160,
965
+ "roma_full_kl": 0.06064958333244063,
966
+ "roma_full_top1": 0.9309255841759155,
967
+ "tokens": 76946,
968
+ "per_part": {
969
+ "hermes": {
970
+ "kl": 0.06882769219873279,
971
+ "top1": 0.9234944954593881
972
+ },
973
+ "openorca": {
974
+ "kl": 0.04556084096435539,
975
+ "top1": 0.9413680781758957
976
+ },
977
+ "tulu3": {
978
+ "kl": 0.03157566572178428,
979
+ "top1": 0.9440155577818374
980
+ },
981
+ "ultrachat": {
982
+ "kl": 0.08374686657551647,
983
+ "top1": 0.9212076882441383
984
+ }
985
+ }
986
+ },
987
+ "recursive-int4-q4_1": {
988
+ "bytes": 25992966976,
989
+ "roma_full_kl": 0.056908085965119946,
990
+ "roma_full_top1": 0.9321082317469395,
991
+ "tokens": 76946,
992
+ "per_part": {
993
+ "hermes": {
994
+ "kl": 0.06673389631200433,
995
+ "top1": 0.9268428796103698
996
+ },
997
+ "openorca": {
998
+ "kl": 0.0430390068098542,
999
+ "top1": 0.9446892763620106
1000
+ },
1001
+ "tulu3": {
1002
+ "kl": 0.031177631650523743,
1003
+ "top1": 0.9417172491013024
1004
+ },
1005
+ "ultrachat": {
1006
+ "kl": 0.07560416072845615,
1007
+ "top1": 0.9216953147222561
1008
+ }
1009
+ }
1010
+ }
1011
+ },
1012
+ "sha256": "08eba9a2e9b01a2b4640b3a03b140bf3904e32ab97810bcc0d519ccd37ea8f82"
1013
+ },
1014
+ "comp2-dataset.json": {
1015
+ "data": {
1016
+ "recursive-nvfp4": {
1017
+ "bytes": 22863084160,
1018
+ "roma_full_kl": 0.24903881241184156,
1019
+ "roma_full_top1": 0.8855938321157345,
1020
+ "tokens": 34631,
1021
+ "per_part": {
1022
+ "hermes": {
1023
+ "kl": 0.17391278372423222,
1024
+ "top1": 0.9017177647314634
1025
+ },
1026
+ "openorca": {
1027
+ "kl": 0.21739458679687818,
1028
+ "top1": 0.8936921041023379
1029
+ },
1030
+ "tulu3": {
1031
+ "kl": 0.21854774813861347,
1032
+ "top1": 0.9180191811139801
1033
+ },
1034
+ "ultrachat": {
1035
+ "kl": 0.3049706143670377,
1036
+ "top1": 0.8540330285246349
1037
+ }
1038
+ }
1039
+ },
1040
+ "recursive-int4-q4_1": {
1041
+ "bytes": 25992966976,
1042
+ "roma_full_kl": 0.22758443161781022,
1043
+ "roma_full_top1": 0.8871820045623863,
1044
+ "tokens": 34631,
1045
+ "per_part": {
1046
+ "hermes": {
1047
+ "kl": 0.16595705788466616,
1048
+ "top1": 0.9004131332898456
1049
+ },
1050
+ "openorca": {
1051
+ "kl": 0.20942235099715747,
1052
+ "top1": 0.8919276576973975
1053
+ },
1054
+ "tulu3": {
1055
+ "kl": 0.19096276171750048,
1056
+ "top1": 0.9221689413500553
1057
+ },
1058
+ "ultrachat": {
1059
+ "kl": 0.2796450671938801,
1060
+ "top1": 0.8556708066057049
1061
+ }
1062
+ }
1063
+ }
1064
+ },
1065
+ "sha256": "52f07d29b68f817714ec013a572579a1a558b89829b19ec09dc6dbe77f231920"
1066
+ },
1067
+ "build-S2-11.8-r1.json": {
1068
+ "data": {
1069
+ "fits": [
1070
+ "builds/S2-11.8-r1"
1071
+ ],
1072
+ "overlay_files": 50,
1073
+ "act": "fp32",
1074
+ "onpolicy": {
1075
+ "roma_full_kl": 0.12256072738341477,
1076
+ "roma_full_top1": 0.8982923088919502,
1077
+ "tokens": 76946
1078
+ },
1079
+ "dataset": {
1080
+ "roma_full_kl": 0.3058208105155098,
1081
+ "roma_full_top1": 0.8562270797840086,
1082
+ "tokens": 34631
1083
+ }
1084
+ },
1085
+ "sha256": "d71e41c9a753c74774c757f0eb1646ed557d36d9fa3cedd0acf6a49b5e671465"
1086
+ },
1087
+ "build-S2-12.4-r1-L1-L2.json": {
1088
+ "data": {
1089
+ "fits": [
1090
+ "builds/S2-12.4-r1-L1-L2"
1091
+ ],
1092
+ "overlay_files": 50,
1093
+ "act": "fp32",
1094
+ "onpolicy": {
1095
+ "roma_full_kl": 0.10394090201024117,
1096
+ "roma_full_top1": 0.9064668728718842,
1097
+ "tokens": 76946
1098
+ },
1099
+ "dataset": {
1100
+ "roma_full_kl": 0.2481904736112289,
1101
+ "roma_full_top1": 0.8673154110479051,
1102
+ "tokens": 34631
1103
+ }
1104
+ },
1105
+ "sha256": "cbdba76d4537bf921c8677f6768040e5fd9394c8f710d84f245d75204120f478"
1106
+ }
1107
+ },
1108
+ "agents": {
1109
+ "flint-S2-11.8-r1": {
1110
+ "headline": {
1111
+ "fc_pct": 93,
1112
+ "fc_hard_pct": 70,
1113
+ "hard_solved": "45/55",
1114
+ "assistant_tasks": "72/272",
1115
+ "assistant_v2_tasks": "409/656",
1116
+ "assistant_v2_hard_traps_pct": 42.1,
1117
+ "assistant_v2_traps_pct": 56.5,
1118
+ "assistant_v2_basics_pct": 78.4,
1119
+ "appworld_success": "168/240",
1120
+ "creative_fully_correct": "4/4",
1121
+ "creative_declared_done": "4/4",
1122
+ "agentic_declared_done": "4/4",
1123
+ "agentic_renders": "0/4 graded (0 with no page errors)"
1124
+ },
1125
+ "missing": [],
1126
+ "sha256": "71195b6ea7536e8fd8f428dae359e52dbfeaee7ac035960ba2d85d0a08dd2460"
1127
+ },
1128
+ "flint-S2-12.4-L2": {
1129
+ "headline": {
1130
+ "fc_pct": 92,
1131
+ "fc_hard_pct": 74,
1132
+ "hard_solved": "50/55",
1133
+ "assistant_tasks": "65/272",
1134
+ "assistant_v2_tasks": "400/656",
1135
+ "assistant_v2_hard_traps_pct": 44.1,
1136
+ "assistant_v2_traps_pct": 56.7,
1137
+ "assistant_v2_basics_pct": 72.7,
1138
+ "appworld_success": "157/240",
1139
+ "creative_fully_correct": "3/4",
1140
+ "creative_declared_done": "3/4",
1141
+ "agentic_declared_done": "4/4",
1142
+ "agentic_renders": "2/4 graded (1 with no page errors)"
1143
+ },
1144
+ "missing": [],
1145
+ "sha256": "797d4922eaf13c5c727e421ddb6950b826e6280d8af17e38a0700fda704c6718"
1146
+ },
1147
+ "flint-teacher": {
1148
+ "headline": {
1149
+ "fc_pct": 91,
1150
+ "fc_hard_pct": 77,
1151
+ "hard_solved": "49/55",
1152
+ "assistant_tasks": "60/136",
1153
+ "assistant_v2_tasks": "460/656",
1154
+ "assistant_v2_hard_traps_pct": 54.6,
1155
+ "assistant_v2_traps_pct": 65.6,
1156
+ "assistant_v2_basics_pct": 82.4,
1157
+ "appworld_success": "187/240",
1158
+ "creative_fully_correct": "4/4",
1159
+ "creative_declared_done": "3/4",
1160
+ "agentic_declared_done": "4/4",
1161
+ "agentic_renders": "2/4 graded (1 with no page errors)"
1162
+ },
1163
+ "missing": [],
1164
+ "sha256": "fbcc4815b71e81de36284fdb2b459f6fb30b2f122c933ef270a9324c0a2c5272"
1165
+ },
1166
+ "flint-gguf-q8": {
1167
+ "headline": {
1168
+ "fc_pct": 96,
1169
+ "fc_hard_pct": 75,
1170
+ "hard_solved": "52/55",
1171
+ "assistant_tasks": "57/136",
1172
+ "assistant_v2_tasks": "358/656",
1173
+ "assistant_v2_hard_traps_pct": 38.8,
1174
+ "assistant_v2_traps_pct": 50.0,
1175
+ "assistant_v2_basics_pct": 67.0,
1176
+ "appworld_success": "164/240",
1177
+ "creative_fully_correct": "4/4",
1178
+ "creative_declared_done": "3/4",
1179
+ "agentic_declared_done": "4/4",
1180
+ "agentic_renders": "0/0 graded (0 with no page errors)"
1181
+ },
1182
+ "missing": [],
1183
+ "sha256": "f062bbd4e85d3a14fde1c6841eaffcb39c8f66f0b5042c20083997d549a898e8"
1184
+ },
1185
+ "flint-gguf-bart-iq3xxs": {
1186
+ "headline": {
1187
+ "fc_pct": 94,
1188
+ "fc_hard_pct": 73,
1189
+ "hard_solved": "38/55",
1190
+ "assistant_tasks": "36/136",
1191
+ "assistant_v2_tasks": "333/656",
1192
+ "assistant_v2_hard_traps_pct": 38.8,
1193
+ "assistant_v2_traps_pct": 47.7,
1194
+ "assistant_v2_basics_pct": 59.1,
1195
+ "appworld_success": "0/240",
1196
+ "creative_fully_correct": "4/4",
1197
+ "creative_declared_done": "2/4",
1198
+ "agentic_declared_done": "3/4",
1199
+ "agentic_renders": "0/0 graded (0 with no page errors)"
1200
+ },
1201
+ "missing": [],
1202
+ "sha256": "5f5c5342f4164a73b0ddae8f7c06680a33148971caf65558df007d8f9323f885"
1203
+ },
1204
+ "flint-gguf-ud-q2kxl": {
1205
+ "headline": {
1206
+ "fc_pct": 97,
1207
+ "fc_hard_pct": 76,
1208
+ "hard_solved": "44/55",
1209
+ "assistant_tasks": "18/136",
1210
+ "assistant_v2_tasks": "242/656",
1211
+ "assistant_v2_hard_traps_pct": 22.4,
1212
+ "assistant_v2_traps_pct": 34.2,
1213
+ "assistant_v2_basics_pct": 44.3,
1214
+ "appworld_success": "158/240",
1215
+ "creative_fully_correct": "4/4",
1216
+ "creative_declared_done": "4/4",
1217
+ "agentic_declared_done": "2/4",
1218
+ "agentic_renders": "0/0 graded (0 with no page errors)"
1219
+ },
1220
+ "missing": [],
1221
+ "sha256": "8e5dbcd7b98d9b457abea7a3036499e77174e5750366d5d6ba279f4d214f4683"
1222
+ }
1223
+ }
1224
+ },
1225
+ "invalid_comparisons": {
1226
+ "flint-gguf-bart-iq3xxs/appworld": "Recorded0/240 in108seconds; excluded pending harness/protocol validation."
1227
+ }
1228
  }