wzebrowski commited on
Commit
2caca97
路
verified 路
1 Parent(s): 6debcfe

Present released model metrics without release comparisons

Browse files
Files changed (2) hide show
  1. README.md +17 -17
  2. SHA256SUMS +1 -1
README.md CHANGED
@@ -29,9 +29,9 @@ datasets:
29
 
30
  [Run locally](RUN_GUIDE.md) 路 [Technical report](TECHNICAL_REPORT.md) 路 [Results](JSONLITE_RESULTS.json) 路 [Provenance](checkpoint_provenance.json)
31
 
32
- **Main checkpoint: a small language model that generates complete JSON directly from source text and a supplied JSON Schema.** It makes one autoregressive call; parsing/schema validation follows generation. No JSON assembly, output repair, constrained decoder or secondary model is used.
33
 
34
- This release retains the grounded-QA, exact/focused extraction and native JSON foundation, then adds a joint JSON extraction continuation with original TRAIN replay, multiple-value examples and authored prose. The display name and standard `model.safetensors` filename stay unchanged. The pretrained ancestor is [OpenSML-Base-175M](https://huggingface.co/OpenSML/OpenSML-Base-175M).
35
 
36
  ## Run locally
37
 
@@ -327,23 +327,23 @@ The full inherited lineage and earlier SFT data are documented in the technical
327
 
328
  ## Native JSON evaluation
329
 
330
- Same original 1,444 SNIPS-derived source/schema requests and raw greedy protocol as the previous release:
331
 
332
- | Metric | Main checkpoint | Previous release |
333
- | --- | ---: | ---: |
334
- | Entire request correct | **88.02%** | 83.03% |
335
- | Field accuracy | **94.17%** | 91.36% |
336
- | Extracted-value F1 | **92.88%** | 90.40% |
337
- | Extracted-value precision | **92.42%** | 90.53% |
338
- | Extracted-value recall | **93.35%** | 90.26% |
339
- | Missing-field accuracy | **95.22%** | 92.79% |
340
- | Schema-valid JSON | **100.00%** | 99.79% |
341
 
342
- Complete correctness is **1,271/1,444**, compared with 1,199/1,444 for the previous release: **96 improved requests and 24 regressions**, a net gain of 72. All seven domain aggregate correctness totals improve. Paired-source correctness rises from 512/694 to **561/694**. False abstentions fall from 107 to **53**, missed missing fields from 110 to **73**, and unsupported predicted values from 27 to **24**. All 1,444 outputs parse, validate and stop naturally; there are no output-budget failures.
343
 
344
- The separate observed 60-case playground scores **49/60 (81.67%)**, compared with 45/60. Its original 54-case subset scores 46/54; the six added three/four/five-item probes score 3/6. Three larger-list probes still fail. The compact 240-request development panel scores 178/240 (74.17%). These panels measure different scopes and must not be pooled. They have prior research exposure and are not fresh blind benchmarks or an external leaderboard.
345
 
346
- The main checkpoint was selected for aggregate improvement despite failing the strict zero-lost-case development gate. Individual regressions remain; improved aggregate scores do not establish universal retention or production reliability.
347
 
348
  ![Complete-request correctness on separate observed panels](complete-request-exact.svg)
349
 
@@ -351,10 +351,10 @@ The main checkpoint was selected for aggregate improvement despite failing the s
351
 
352
  A focused English extraction research model for documented flat schemas and mostly SNIPS fields. It can shorten, omit, invent or misassign values, confuse roles, miss list members, and fail on corrections or unfamiliar definitions. Valid JSON does not prove correct extraction. Long documents, nested objects, numeric/boolean outputs, multilingual tasks, general chat and prompt-injection robustness are outside the verified scope. `success` means structural validity and natural completion, not semantic correctness.
353
 
354
- ## Main checkpoint identity and package
355
 
356
  Selected BF16 weight SHA-256:
357
 
358
  `781597baeb9b51d535fc27cd6f941dbc202e01d62c8d963587a3f3d565128560`
359
 
360
- Includes `model.safetensors`, unchanged tokenizer, native MLX runner, source bundle, 60 source/schema/reference examples, raw evaluation records, checksums and provenance. Standalone verification checks generated strings and stop reasons against the saved 60-case protocol; this is packaging parity, not an independent accuracy benchmark. Previous revisions remain available. Source uses Apache-2.0; upstream terms apply separately; no trained-weight license is assigned by this update.
 
29
 
30
  [Run locally](RUN_GUIDE.md) 路 [Technical report](TECHNICAL_REPORT.md) 路 [Results](JSONLITE_RESULTS.json) 路 [Provenance](checkpoint_provenance.json)
31
 
32
+ **A small language model that generates complete JSON directly from source text and a supplied JSON Schema.** It makes one autoregressive call; parsing/schema validation follows generation. No JSON assembly, output repair, constrained decoder or secondary model is used.
33
 
34
+ The model was fine-tuned for grounded question answering, exact and focused field extraction, and direct JSON generation. Its final joint JSON extraction stage combines original training replay, multiple-value examples and authored prose. Weights are provided as `model.safetensors`. The pretrained ancestor is [OpenSML-Base-175M](https://huggingface.co/OpenSML/OpenSML-Base-175M).
35
 
36
  ## Run locally
37
 
 
327
 
328
  ## Native JSON evaluation
329
 
330
+ The released model was evaluated on 1,444 SNIPS-derived source-text and JSON Schema requests using raw greedy generation:
331
 
332
+ | Metric | Released model |
333
+ | --- | ---: |
334
+ | Entire request correct | **88.02%** |
335
+ | Field accuracy | **94.17%** |
336
+ | Extracted-value F1 | **92.88%** |
337
+ | Extracted-value precision | **92.42%** |
338
+ | Extracted-value recall | **93.35%** |
339
+ | Missing-field accuracy | **95.22%** |
340
+ | Schema-valid JSON | **100.00%** |
341
 
342
+ The model answers **1,271/1,444 requests completely correctly**. Paired-source correctness is **561/694**. The evaluation records **53 false abstentions**, **73 missed missing fields**, and **24 unsupported predicted values**. All 1,444 outputs parse, validate and stop naturally; there are no output-budget failures.
343
 
344
+ The separate observed 60-case playground scores **49/60 (81.67%)**. Its original 54-case subset scores 46/54; the six added three/four/five-item probes score 3/6. Three larger-list probes fail. The compact 240-request development panel scores 178/240 (74.17%). These panels measure different scopes and must not be pooled. They have prior research exposure and are not fresh blind benchmarks or an external leaderboard.
345
 
346
+ These aggregate scores do not establish universal retention or production reliability. Checkpoint-selection checks and remaining failures are documented in the technical report.
347
 
348
  ![Complete-request correctness on separate observed panels](complete-request-exact.svg)
349
 
 
351
 
352
  A focused English extraction research model for documented flat schemas and mostly SNIPS fields. It can shorten, omit, invent or misassign values, confuse roles, miss list members, and fail on corrections or unfamiliar definitions. Valid JSON does not prove correct extraction. Long documents, nested objects, numeric/boolean outputs, multilingual tasks, general chat and prompt-injection robustness are outside the verified scope. `success` means structural validity and natural completion, not semantic correctness.
353
 
354
+ ## Released model identity and package
355
 
356
  Selected BF16 weight SHA-256:
357
 
358
  `781597baeb9b51d535fc27cd6f941dbc202e01d62c8d963587a3f3d565128560`
359
 
360
+ Includes `model.safetensors`, tokenizer, native MLX runner, source bundle, 60 source/schema/reference examples, raw evaluation records, checksums and provenance. Standalone verification checks generated strings and stop reasons against the saved 60-case protocol; this is packaging parity, not an independent accuracy benchmark. Source uses Apache-2.0; upstream terms apply separately; no trained-weight license is assigned by this release.
SHA256SUMS CHANGED
@@ -4,7 +4,7 @@
4
  74a074f90bd5e7a26d5f6cd5757ddc1fbfcc72e259e0b23a210fa48d91b35fdb JSONLITE_RESULTS.json
5
  c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4 LICENSE
6
  f53194e53a2af5ec857337e8213397ea4f81a7e02d2dbcf0384a00cf1cf7f76f PACKAGING_VERIFICATION.json
7
- 2728c5efa01d9d59fb1def12a543980c300edcf557fabfa921b08dc0712e68cb README.md
8
  5ee8f11f0e87becf1a29c44e2b0f3dacc2c779712b635558be57dd7ef44455d3 RUN_GUIDE.md
9
  c6191baf937bce61f1b747acfbcb975a17eef6e8a598682a751a1031d1cfee23 SOURCE_NOTICE.md
10
  6088771ab13fa1080c0ca0f31649deb9f417597d7c2055b0f388e86f41f2c415 TECHNICAL_REPORT.md
 
4
  74a074f90bd5e7a26d5f6cd5757ddc1fbfcc72e259e0b23a210fa48d91b35fdb JSONLITE_RESULTS.json
5
  c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4 LICENSE
6
  f53194e53a2af5ec857337e8213397ea4f81a7e02d2dbcf0384a00cf1cf7f76f PACKAGING_VERIFICATION.json
7
+ 6a4ca570696577a60e5a9b7ffdcda04edf0b9a75322cc4db33e3f92d8ac79931 README.md
8
  5ee8f11f0e87becf1a29c44e2b0f3dacc2c779712b635558be57dd7ef44455d3 RUN_GUIDE.md
9
  c6191baf937bce61f1b747acfbcb975a17eef6e8a598682a751a1031d1cfee23 SOURCE_NOTICE.md
10
  6088771ab13fa1080c0ca0f31649deb9f417597d7c2055b0f388e86f41f2c415 TECHNICAL_REPORT.md