Hugging Face's logo

tcclaviger
/
Tess-27B-RFI

Image-Text-to-Text
Transformers
Safetensors
English
qwen3_5
quantized
rfi
mixed-precision
conversational
Model card Files
xet
 Community
tcclaviger commited on
Commit
c656f00
·
1 Parent(s): afd04ba

README: add Tess-27B-RFA column, recompute best-in-row bolding, add checkpoint size

Browse files
Files changed (1) hide show
  1. README.md +32 -29
README.md CHANGED
@@ -45,9 +45,11 @@ extended from the native 256K window with YaRN via `--hf-overrides`:
45
 
46
  ## Evaluation results
47
 
48
- Four builds measured with identical methodology, each against its own live vLLM endpoint (July 2026):
49
- **Qwen3.6-27B** (bf16 base) → **Tess-4-27B** (the tune, bf16) → **Tess-27B-RFI** (this quant),
 
50
  with **Qwen3.6-35B-A3B** (MoE, bf16) as a comparative reference point.
 
51
 
52
  ### Tune impact — Qwen3.6-27B → Tess-4-27B
53
 
@@ -65,6 +67,7 @@ with **Qwen3.6-35B-A3B** (MoE, bf16) as a comparative reference point.
65
 
66
  | Metric | Change |
67
  |---|---|
 
68
  | WikiText-2 perplexity | 6.669 → 6.663 (≈ flat) |
69
  | Codeneedle overall recall | 97.7% → 98.0% (≈ flat) |
70
  | MC accuracy (4 tasks) | ≈ flat (−0.4 to +0.1 pp) |
@@ -75,13 +78,13 @@ with **Qwen3.6-35B-A3B** (MoE, bf16) as a comparative reference point.
75
 
76
  ### Quality
77
 
78
- | Metric | Qwen3.6-27B (base) | Tess-4-27B | **Tess-27B-RFI** | Qwen3.6-35B-A3B |
79
- |---|---|---|---|---|
80
- | WikiText-2 PPL (n_ctx 2048, lower is better) | 7.0597 | 6.6691 | 6.6632 | **6.5092** |
81
- | ARC-Challenge (acc_norm) | 59.13% | **60.84%** | 60.41% | 55.20% |
82
- | ARC-Easy (acc_norm) | 75.76% | **77.53%** | 77.40% | 71.13% |
83
- | Winogrande (acc) | 76.87% | 77.43% | **77.51%** | 73.40% |
84
- | HellaSwag (acc_norm) | 84.04% | 84.21% | **84.27%** | 82.95% |
85
 
86
  Multiple-choice accuracy is lm-eval loglikelihood scoring, 0-shot.
87
 
@@ -89,34 +92,34 @@ Multiple-choice accuracy is lm-eval loglikelihood scoring, 0-shot.
89
 
90
  Verbatim function recall under 10K–80K-token contexts.
91
 
92
- | Corpus | Qwen3.6-27B (base) | Tess-4-27B | **Tess-27B-RFI** | Qwen3.6-35B-A3B |
93
- |---|---|---|---|---|
94
- | Python | 92.73% | **100%** | **100%** | 99.09% |
95
- | C++ | 92.81% | 98.12% | **98.44%** | **98.44%** |
96
- | Rust | 95.00% | **99.69%** | **99.69%** | 99.38% |
97
- | JS (~80K tokens) | 87.50% | 93.13% | **93.75%** | 92.19% |
98
- | Tools | 96.52% | **99.57%** | **99.57%** | 93.48% |
99
- | **Overall recall** | 91.95% | 97.73% | **97.97%** | 97.28% |
100
 
101
  ### Tool calling & accuracy benches
102
 
103
- | Bench | Qwen3.6-27B (base) | Tess-4-27B | **Tess-27B-RFI** | Qwen3.6-35B-A3B |
104
- |---|---|---|---|---|
105
- | tool-eval final (full 69, TC-61 excl) | 89 | 85 | 87 | **90** |
106
- | GSM8K (50q) | **98.0%** | 94.0% | **98.0%** | 96.0% |
107
- | MMLU (50q) | 74.0% | 76.0% | **82.0%** | 64.0% |
108
- | IFEval (20 prompts, prompt-level) | **90.0%** | **90.0%** | **90.0%** | **90.0%** |
109
 
110
  ### Decode throughput — tok/s output (ISL 128 / ISL 512)
111
 
112
  `vllm bench serve`, random dataset, OSL 128, saturation, 4× R9700 (gfx1201), TP 4.
113
 
114
- | Concurrency | Qwen3.6-27B (base) | Tess-4-27B | **Tess-27B-RFI** | Qwen3.6-35B-A3B |
115
- |---|---|---|---|---|
116
- | 1 | 63.7 / 65.2 | 59.9 / 63.8 | 75.3 / 69.3 | **91.9 / 114.4** |
117
- | 10 | 251.1 / 246.6 | 289.6 / 227.8 | 281.5 / 219.2 | **434.9 / 440.3** |
118
- | 25 | 390.3 / 336.8 | 492.0 / 341.1 | 429.1 / 284.0 | **688.9 / 563.7** |
119
- | 50 | 541.2 / 363.0 | 527.9 / 345.6 | 452.8 / 295.0 | **889.8 / 702.9** |
120
 
121
  ## Notes
122
 
 
45
 
46
  ## Evaluation results
47
 
48
+ Five builds measured with identical methodology, each against its own live vLLM endpoint (July 2026):
49
+ **Qwen3.6-27B** (bf16 base) → **Tess-4-27B** (the tune, bf16) → **Tess-27B-RFI** (this quant) →
50
+ [**Tess-27B-RFA**](https://huggingface.co/tcclaviger/Tess-27B-RFA) (all-attention 4-bit sibling),
51
  with **Qwen3.6-35B-A3B** (MoE, bf16) as a comparative reference point.
52
+ Bold marks the best score in each row (ties all bolded).
53
 
54
  ### Tune impact — Qwen3.6-27B → Tess-4-27B
55
 
 
67
 
68
  | Metric | Change |
69
  |---|---|
70
+ | Checkpoint size | 55.6 → 28.9 GB (**−48%**) |
71
  | WikiText-2 perplexity | 6.669 → 6.663 (≈ flat) |
72
  | Codeneedle overall recall | 97.7% → 98.0% (≈ flat) |
73
  | MC accuracy (4 tasks) | ≈ flat (−0.4 to +0.1 pp) |
 
78
 
79
  ### Quality
80
 
81
+ | Metric | Qwen3.6-27B (base) | Tess-4-27B | **Tess-27B-RFI** | Tess-27B-RFA | Qwen3.6-35B-A3B |
82
+ |---|---|---|---|---|---|
83
+ | WikiText-2 PPL (n_ctx 2048, lower is better) | 7.0597 | 6.6691 | 6.6632 | 6.6292 | **6.5092** |
84
+ | ARC-Challenge (acc_norm) | 59.13% | **60.84%** | 60.41% | 60.32% | 55.20% |
85
+ | ARC-Easy (acc_norm) | 75.76% | 77.53% | 77.40% | **78.87%** | 71.13% |
86
+ | Winogrande (acc) | 76.87% | 77.43% | **77.51%** | 76.80% | 73.40% |
87
+ | HellaSwag (acc_norm) | 84.04% | 84.21% | **84.27%** | 84.05% | 82.95% |
88
 
89
  Multiple-choice accuracy is lm-eval loglikelihood scoring, 0-shot.
90
 
 
92
 
93
  Verbatim function recall under 10K–80K-token contexts.
94
 
95
+ | Corpus | Qwen3.6-27B (base) | Tess-4-27B | **Tess-27B-RFI** | Tess-27B-RFA | Qwen3.6-35B-A3B |
96
+ |---|---|---|---|---|---|
97
+ | Python | 92.73% | **100%** | **100%** | **100%** | 99.09% |
98
+ | C++ | 92.81% | 98.12% | 98.44% | **98.75%** | 98.44% |
99
+ | Rust | 95.00% | **99.69%** | **99.69%** | **99.69%** | 99.38% |
100
+ | JS (~80K tokens) | 87.50% | 93.13% | **93.75%** | **93.75%** | 92.19% |
101
+ | Tools | 96.52% | **99.57%** | **99.57%** | **99.57%** | 93.48% |
102
+ | **Overall recall** | 91.95% | 97.73% | 97.97% | **98.05%** | 97.28% |
103
 
104
  ### Tool calling & accuracy benches
105
 
106
+ | Bench | Qwen3.6-27B (base) | Tess-4-27B | **Tess-27B-RFI** | Tess-27B-RFA | Qwen3.6-35B-A3B |
107
+ |---|---|---|---|---|---|
108
+ | tool-eval final (full 69, TC-61 excl) | 89 | 85 | 87 | 86 | **90** |
109
+ | GSM8K (50q) | **98.0%** | 94.0% | **98.0%** | **98.0%** | 96.0% |
110
+ | MMLU (50q) | 74.0% | 76.0% | **82.0%** | 80.0% | 64.0% |
111
+ | IFEval (20 prompts, prompt-level) | 90.0% | 90.0% | 90.0% | **95.0%** | 90.0% |
112
 
113
  ### Decode throughput — tok/s output (ISL 128 / ISL 512)
114
 
115
  `vllm bench serve`, random dataset, OSL 128, saturation, 4× R9700 (gfx1201), TP 4.
116
 
117
+ | Concurrency | Qwen3.6-27B (base) | Tess-4-27B | **Tess-27B-RFI** | Tess-27B-RFA | Qwen3.6-35B-A3B |
118
+ |---|---|---|---|---|---|
119
+ | 1 | 63.7 / 65.2 | 59.9 / 63.8 | 75.3 / 69.3 | 67.9 / 58.2 | **91.9 / 114.4** |
120
+ | 10 | 251.1 / 246.6 | 289.6 / 227.8 | 281.5 / 219.2 | 292.2 / 194.8 | **434.9 / 440.3** |
121
+ | 25 | 390.3 / 336.8 | 492.0 / 341.1 | 429.1 / 284.0 | 369.0 / 260.3 | **688.9 / 563.7** |
122
+ | 50 | 541.2 / 363.0 | 527.9 / 345.6 | 452.8 / 295.0 | 422.6 / 278.4 | **889.8 / 702.9** |
123
 
124
  ## Notes
125