isilicon commited on
Commit
d20e9e9
·
verified ·
1 Parent(s): 7a876f1

Publish detailed Mango model card, measured results and visual analysis

Browse files
Files changed (3) hide show
  1. BENCHMARK_METHODOLOGY.md +5 -0
  2. MODEL_CARD_STATUS.json +9 -9
  3. README.md +1 -1
BENCHMARK_METHODOLOGY.md CHANGED
@@ -68,3 +68,8 @@ The frozen suite SHA-256 is
68
  6ba848464b4685e496b5ac83a115e7efb3cf0c8f7a6afcdc49553af49a5df421.
69
  BENCHMARK_REPORT.json records exact scores, uncertainty, raw artifact hashes, grader hashes and timing
70
  coverage without publishing prompts, generated answers, credentials or real user traffic.
 
 
 
 
 
 
68
  6ba848464b4685e496b5ac83a115e7efb3cf0c8f7a6afcdc49553af49a5df421.
69
  BENCHMARK_REPORT.json records exact scores, uncertainty, raw artifact hashes, grader hashes and timing
70
  coverage without publishing prompts, generated answers, credentials or real user traffic.
71
+
72
+
73
+ ## Corrected Qwen effort coverage
74
+
75
+ The separate full404 Qwen effort study now grades403 valid questions plus12 agents per effort. Invalid annotation ifeval-1122 (letter="#") triggers random alphabetic substitution in the pinned reference and is unscored for both. Strict IFEval36/39 medium,37/39 xhigh; language-detection seed0. All200 historical strict grades and78 valid effort grades are unchanged. Timing retains all404 collected requests. Historical202 cross-model comparisons and44 frozen subjective ratings are unchanged.
MODEL_CARD_STATUS.json CHANGED
@@ -3,22 +3,22 @@
3
  "state": "documentation_refresh",
4
  "source_repo": "z-lab/Qwen3.8-27B-DFlash2",
5
  "source_revision": "50307d4c4cde6860d4eee73e2547cd786fe8e8a4",
6
- "prior_repository_commit": "697f0a3f9627eb1b46f17638435421ca8d1d0b13",
7
- "reviewed_plan_sha256": "240974cf13a499f940e169bb026729401017adac75aac94606d88bee26479c33",
8
- "created_utc": "2026-10-30T04:12:09.824711+00:00",
9
  "scope": "Card Markdown, aggregate benchmark data, methodology and original visual assets only.",
10
  "files": [
11
  {
12
  "path": "BENCHMARK_METHODOLOGY.md",
13
- "bytes": 4560,
14
- "sha256": "d8c2f04abf706531ce0d27c67837e1cbfdfe461f1ffe97e5c419141f294cb5c3",
15
- "git_blob": "bc62ad64cfbd710ae5828f97824d62fe8657c79f"
16
  },
17
  {
18
  "path": "README.md",
19
- "bytes": 12754,
20
- "sha256": "9f4e595e8521cb8066a3de049caa3187676dc87b5584fd97320fb357a8c761c2",
21
- "git_blob": "3c5677545ac303b7d2d4aa735ce7f8cfc3240601"
22
  },
23
  {
24
  "path": "assets/mango-header.png",
 
3
  "state": "documentation_refresh",
4
  "source_repo": "z-lab/Qwen3.8-27B-DFlash2",
5
  "source_revision": "50307d4c4cde6860d4eee73e2547cd786fe8e8a4",
6
+ "prior_repository_commit": "7a876f1a886a27cc3286b80a227d171c08c4550d",
7
+ "reviewed_plan_sha256": "2239a6626871ced987d53c60fe0b3f19ae74d718182f633284d8d5bdfef02e14",
8
+ "created_utc": "2026-10-30T05:04:28.966723+00:00",
9
  "scope": "Card Markdown, aggregate benchmark data, methodology and original visual assets only.",
10
  "files": [
11
  {
12
  "path": "BENCHMARK_METHODOLOGY.md",
13
+ "bytes": 5091,
14
+ "sha256": "de1f4cda257f43b476b56da1387fb87a0abc1ef7e384f9c2019803bdf71648b0",
15
+ "git_blob": "2fc7deae72ab4456feebf116ef9ff0fd6d96f36c"
16
  },
17
  {
18
  "path": "README.md",
19
+ "bytes": 12875,
20
+ "sha256": "3ebfdf176bcae65518c3d34830cd7d19c570d8df3d3052575543577b74ae7eca",
21
+ "git_blob": "0f17bcb51fee4686a9c8226ef8c12a87f0b730c7"
22
  },
23
  {
24
  "path": "assets/mango-header.png",
README.md CHANGED
@@ -28,7 +28,7 @@ z-lab's Qwen3.8 27B DFlash2 draft model, containing its config and safetensors w
28
  <!-- mango-community:end -->
29
 
30
  <!-- mango-qwen-effort:start -->
31
- **Qwen effort study — separate full-404 cohort:** medium 82.88/100 versus xhigh 84.29/100; paired difference +1.41 points (95% interval [-1.83, +4.75]). The historical 202-question cross-model comparison below is unchanged. [Study, timing and single-grader subjective results](https://huggingface.co/isilicon/mango-qwen38-27b-uncensored-mlx4#qwen-medium-versus-xhigh).
32
  <!-- mango-qwen-effort:end -->
33
 
34
  Preserved and documented by **isilicon / Mango**. The original model, modification and quantization remain the upstream creators' work. This repository adds attributed preservation and exact-configuration evaluation, with the selected source bytes unchanged.
 
28
  <!-- mango-community:end -->
29
 
30
  <!-- mango-qwen-effort:start -->
31
+ **Qwen effort study — separate full-404 cohort:** medium 82.87/100 versus xhigh 84.41/100; paired difference +1.54 points (95% interval [-1.71, +4.82]). 404 collected questions, 403 valid questions graded per effort; one invalid reference annotation excluded symmetrically. The historical 202-question cross-model comparison below is unchanged. [Study, timing and single-grader subjective results](https://huggingface.co/isilicon/mango-qwen38-27b-uncensored-mlx4#qwen-medium-versus-xhigh).
32
  <!-- mango-qwen-effort:end -->
33
 
34
  Preserved and documented by **isilicon / Mango**. The original model, modification and quantization remain the upstream creators' work. This repository adds attributed preservation and exact-configuration evaluation, with the selected source bytes unchanged.