Publish detailed Mango model card, measured results and visual analysis
Browse files- BENCHMARK_METHODOLOGY.md +5 -0
- MODEL_CARD_STATUS.json +9 -9
- README.md +1 -1
BENCHMARK_METHODOLOGY.md
CHANGED
|
@@ -68,3 +68,8 @@ The frozen suite SHA-256 is
|
|
| 68 |
6ba848464b4685e496b5ac83a115e7efb3cf0c8f7a6afcdc49553af49a5df421.
|
| 69 |
BENCHMARK_REPORT.json records exact scores, uncertainty, raw artifact hashes, grader hashes and timing
|
| 70 |
coverage without publishing prompts, generated answers, credentials or real user traffic.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
6ba848464b4685e496b5ac83a115e7efb3cf0c8f7a6afcdc49553af49a5df421.
|
| 69 |
BENCHMARK_REPORT.json records exact scores, uncertainty, raw artifact hashes, grader hashes and timing
|
| 70 |
coverage without publishing prompts, generated answers, credentials or real user traffic.
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
## Corrected Qwen effort coverage
|
| 74 |
+
|
| 75 |
+
The separate full404 Qwen effort study now grades403 valid questions plus12 agents per effort. Invalid annotation ifeval-1122 (letter="#") triggers random alphabetic substitution in the pinned reference and is unscored for both. Strict IFEval36/39 medium,37/39 xhigh; language-detection seed0. All200 historical strict grades and78 valid effort grades are unchanged. Timing retains all404 collected requests. Historical202 cross-model comparisons and44 frozen subjective ratings are unchanged.
|
MODEL_CARD_STATUS.json
CHANGED
|
@@ -3,22 +3,22 @@
|
|
| 3 |
"state": "documentation_refresh",
|
| 4 |
"source_repo": "z-lab/Qwen3.8-27B-DFlash2",
|
| 5 |
"source_revision": "50307d4c4cde6860d4eee73e2547cd786fe8e8a4",
|
| 6 |
-
"prior_repository_commit": "
|
| 7 |
-
"reviewed_plan_sha256": "
|
| 8 |
-
"created_utc": "2026-10-
|
| 9 |
"scope": "Card Markdown, aggregate benchmark data, methodology and original visual assets only.",
|
| 10 |
"files": [
|
| 11 |
{
|
| 12 |
"path": "BENCHMARK_METHODOLOGY.md",
|
| 13 |
-
"bytes":
|
| 14 |
-
"sha256": "
|
| 15 |
-
"git_blob": "
|
| 16 |
},
|
| 17 |
{
|
| 18 |
"path": "README.md",
|
| 19 |
-
"bytes":
|
| 20 |
-
"sha256": "
|
| 21 |
-
"git_blob": "
|
| 22 |
},
|
| 23 |
{
|
| 24 |
"path": "assets/mango-header.png",
|
|
|
|
| 3 |
"state": "documentation_refresh",
|
| 4 |
"source_repo": "z-lab/Qwen3.8-27B-DFlash2",
|
| 5 |
"source_revision": "50307d4c4cde6860d4eee73e2547cd786fe8e8a4",
|
| 6 |
+
"prior_repository_commit": "7a876f1a886a27cc3286b80a227d171c08c4550d",
|
| 7 |
+
"reviewed_plan_sha256": "2239a6626871ced987d53c60fe0b3f19ae74d718182f633284d8d5bdfef02e14",
|
| 8 |
+
"created_utc": "2026-10-30T05:04:28.966723+00:00",
|
| 9 |
"scope": "Card Markdown, aggregate benchmark data, methodology and original visual assets only.",
|
| 10 |
"files": [
|
| 11 |
{
|
| 12 |
"path": "BENCHMARK_METHODOLOGY.md",
|
| 13 |
+
"bytes": 5091,
|
| 14 |
+
"sha256": "de1f4cda257f43b476b56da1387fb87a0abc1ef7e384f9c2019803bdf71648b0",
|
| 15 |
+
"git_blob": "2fc7deae72ab4456feebf116ef9ff0fd6d96f36c"
|
| 16 |
},
|
| 17 |
{
|
| 18 |
"path": "README.md",
|
| 19 |
+
"bytes": 12875,
|
| 20 |
+
"sha256": "3ebfdf176bcae65518c3d34830cd7d19c570d8df3d3052575543577b74ae7eca",
|
| 21 |
+
"git_blob": "0f17bcb51fee4686a9c8226ef8c12a87f0b730c7"
|
| 22 |
},
|
| 23 |
{
|
| 24 |
"path": "assets/mango-header.png",
|
README.md
CHANGED
|
@@ -28,7 +28,7 @@ z-lab's Qwen3.8 27B DFlash2 draft model, containing its config and safetensors w
|
|
| 28 |
<!-- mango-community:end -->
|
| 29 |
|
| 30 |
<!-- mango-qwen-effort:start -->
|
| 31 |
-
**Qwen effort study — separate full-404 cohort:** medium 82.
|
| 32 |
<!-- mango-qwen-effort:end -->
|
| 33 |
|
| 34 |
Preserved and documented by **isilicon / Mango**. The original model, modification and quantization remain the upstream creators' work. This repository adds attributed preservation and exact-configuration evaluation, with the selected source bytes unchanged.
|
|
|
|
| 28 |
<!-- mango-community:end -->
|
| 29 |
|
| 30 |
<!-- mango-qwen-effort:start -->
|
| 31 |
+
**Qwen effort study — separate full-404 cohort:** medium 82.87/100 versus xhigh 84.41/100; paired difference +1.54 points (95% interval [-1.71, +4.82]). 404 collected questions, 403 valid questions graded per effort; one invalid reference annotation excluded symmetrically. The historical 202-question cross-model comparison below is unchanged. [Study, timing and single-grader subjective results](https://huggingface.co/isilicon/mango-qwen38-27b-uncensored-mlx4#qwen-medium-versus-xhigh).
|
| 32 |
<!-- mango-qwen-effort:end -->
|
| 33 |
|
| 34 |
Preserved and documented by **isilicon / Mango**. The original model, modification and quantization remain the upstream creators' work. This repository adds attributed preservation and exact-configuration evaluation, with the selected source bytes unchanged.
|