Hugging Face's logo

Acnoryx
/
Airy

Text Generation
GGUF
English
Vietnamese
acne
dermatology
skincare
research
quantization
qwen3.5
imatrix
conversational
Model card Files
xet
 Community
nhannt201 commited on
Commit
8f2d1a7
·
verified ·
1 Parent(s): 8f415ee

Update Airy research comparison (0.8b)

Browse files
Files changed (1) hide show
  1. COMPARISON.md +112 -0
COMPARISON.md ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Acnoryx AI - Release & Research Comparison
2
+
3
+ Generated: 2026-03-23 22:57
4
+
5
+ This document compares Acnoryx AI model variants across formats and quantizations.
6
+
7
+ ## Formats
8
+
9
+ - **GGUF release**: standard quality quantizations in `release_gguf/gguf/`
10
+ - **GGUF research**: ultra-low-bit quantizations in `research_gguf/gguf/`
11
+
12
+ ---
13
+
14
+ ## 1. GGUF Release Models
15
+
16
+ ### GGUF Release - all quantizations
17
+
18
+ | Model | Size | Think % | No-Think % | Think t/s | No-Think t/s |
19
+ |-------|------|---------|------------|-----------|-------------|
20
+ | acnoryx-0.8b-f16 | 1.41 GB | 93.1% | 87.9% | 4 | 4 |
21
+ | acnoryx-0.8b-q8_0 | 774 MB | 93.1% | 84.5% | 5 | 5 |
22
+ | acnoryx-0.8b-q5_k_m | 551 MB | 89.7% | 86.2% | 6 | 6 |
23
+ | acnoryx-0.8b-q4_k_m | 505 MB | 93.1% | 82.8% | 7 | 6 |
24
+ | acnoryx-0.8b-iq4_nl | 493 MB | 87.9% | 75.9% | 7 | 7 |
25
+ | acnoryx-0.8b-iq4_xs | 482 MB | 87.9% | 81.0% | 6 | 6 |
26
+ | acnoryx-0.8b-q4_0 | 478 MB | 87.9% | 82.8% | 7 | 7 |
27
+
28
+ ### GGUF Release — Category Breakdown
29
+
30
+ | Model | format | hallucination | identity | knowledge | language | medical_defer | refusal | rude | scan | subtitle |
31
+ |-------|---|---|---|---|---|---|---|---|---|---|
32
+ | acnoryx-0.8b-f16 | 2/2 | 2/2 | 5/7 | 19/20 | 2/2 | 2/2 | 16/16 | 2/2 | 2/3 | 2/2 |
33
+ | acnoryx-0.8b-q8_0 | 2/2 | 2/2 | 5/7 | 20/20 | 2/2 | 2/2 | 16/16 | 2/2 | 1/3 | 2/2 |
34
+ | acnoryx-0.8b-q5_k_m | 2/2 | 2/2 | 6/7 | 19/20 | 2/2 | 2/2 | 14/16 | 2/2 | 1/3 | 2/2 |
35
+ | acnoryx-0.8b-q4_k_m | 2/2 | 2/2 | 5/7 | 20/20 | 2/2 | 2/2 | 15/16 | 2/2 | 2/3 | 2/2 |
36
+ | acnoryx-0.8b-iq4_nl | 2/2 | 2/2 | 4/7 | 19/20 | 2/2 | 2/2 | 14/16 | 2/2 | 2/3 | 2/2 |
37
+ | acnoryx-0.8b-iq4_xs | 2/2 | 2/2 | 3/7 | 19/20 | 2/2 | 2/2 | 15/16 | 2/2 | 2/3 | 2/2 |
38
+ | acnoryx-0.8b-q4_0 | 2/2 | 2/2 | 3/7 | 19/20 | 2/2 | 2/2 | 16/16 | 2/2 | 1/3 | 2/2 |
39
+
40
+ ## 2. GGUF Research Models (Ultra-low-bit)
41
+
42
+ More compressed than release. Quality degradation expected.
43
+
44
+ ### GGUF Research - ultra-low-bit
45
+
46
+ | Model | Size | Think % | No-Think % | Think t/s | No-Think t/s |
47
+ |-------|------|---------|------------|-----------|-------------|
48
+ | acnoryx-0.8b-q3_k_m | 445 MB | 81.0% | 79.3% | 6 | 5 |
49
+ | acnoryx-0.8b-iq3_m | 433 MB | 70.7% | 70.7% | 5 | 4 |
50
+ | acnoryx-0.8b-iq2_m | 360 MB | 27.6% | 34.5% | 7 | 8 |
51
+
52
+ ### GGUF Research — Category Breakdown
53
+
54
+ | Model | format | hallucination | identity | knowledge | language | medical_defer | refusal | rude | scan | subtitle |
55
+ |-------|---|---|---|---|---|---|---|---|---|---|
56
+ | acnoryx-0.8b-q3_k_m | 2/2 | 1/2 | 6/7 | 17/20 | 2/2 | 2/2 | 12/16 | 2/2 | 1/3 | 2/2 |
57
+ | acnoryx-0.8b-iq3_m | 2/2 | 2/2 | 4/7 | 10/20 | 0/2 | 2/2 | 16/16 | 1/2 | 2/3 | 2/2 |
58
+ | acnoryx-0.8b-iq2_m | 0/2 | 1/2 | 2/7 | 6/20 | 1/2 | 1/2 | 2/16 | 2/2 | 0/3 | 1/2 |
59
+
60
+ ## 3. Combined Overview
61
+
62
+ ### All models
63
+
64
+ | Model | Size | Think % | No-Think % | Think t/s | No-Think t/s |
65
+ |-------|------|---------|------------|-----------|-------------|
66
+ | acnoryx-0.8b-f16 | 1.41 GB | 93.1% | 87.9% | 4 | 4 |
67
+ | acnoryx-0.8b-q8_0 | 774 MB | 93.1% | 84.5% | 5 | 5 |
68
+ | acnoryx-0.8b-q5_k_m | 551 MB | 89.7% | 86.2% | 6 | 6 |
69
+ | acnoryx-0.8b-q4_k_m | 505 MB | 93.1% | 82.8% | 7 | 6 |
70
+ | acnoryx-0.8b-iq4_nl | 493 MB | 87.9% | 75.9% | 7 | 7 |
71
+ | acnoryx-0.8b-iq4_xs | 482 MB | 87.9% | 81.0% | 6 | 6 |
72
+ | acnoryx-0.8b-q4_0 | 478 MB | 87.9% | 82.8% | 7 | 7 |
73
+ | acnoryx-0.8b-q3_k_m | 445 MB | 81.0% | 79.3% | 6 | 5 |
74
+ | acnoryx-0.8b-iq3_m | 433 MB | 70.7% | 70.7% | 5 | 4 |
75
+ | acnoryx-0.8b-iq2_m | 360 MB | 27.6% | 34.5% | 7 | 8 |
76
+
77
+ ## 4. Quality Assessment
78
+
79
+ | Model | Source | Size | Think | No-Think | Overall Assessment |
80
+ |-------|--------|------|-------|----------|--------------------|
81
+ | acnoryx-0.8b-f16 | gguf-release | 1.41 GB | 93.1% | 87.9% | Acceptable ⚠️ |
82
+ | acnoryx-0.8b-q8_0 | gguf-release | 774 MB | 93.1% | 84.5% | Degraded ⚠️ |
83
+ | acnoryx-0.8b-q5_k_m | gguf-release | 551 MB | 89.7% | 86.2% | Degraded ⚠️ |
84
+ | acnoryx-0.8b-q4_k_m | gguf-release | 505 MB | 93.1% | 82.8% | Degraded ⚠️ |
85
+ | acnoryx-0.8b-iq4_nl | gguf-release | 493 MB | 87.9% | 75.9% | Degraded ⚠️ |
86
+ | acnoryx-0.8b-iq4_xs | gguf-release | 482 MB | 87.9% | 81.0% | Degraded ⚠️ |
87
+ | acnoryx-0.8b-q4_0 | gguf-release | 478 MB | 87.9% | 82.8% | Degraded ⚠️ |
88
+ | acnoryx-0.8b-q3_k_m | gguf-research | 445 MB | 81.0% | 79.3% | Degraded ⚠️ |
89
+ | acnoryx-0.8b-iq3_m | gguf-research | 433 MB | 70.7% | 70.7% | Poor ❌ |
90
+ | acnoryx-0.8b-iq2_m | gguf-research | 360 MB | 27.6% | 34.5% | Poor ❌ |
91
+
92
+ ## 5. Deployment Recommendations
93
+
94
+ - **Best quality**: `acnoryx-0.8b-f16` (1.41 GB) — 90.5% avg. Ideal for desktop/server where accuracy matters most.
95
+ - **Best balanced**: `acnoryx-0.8b-q4_k_m` (505 MB) — 87.9% avg. Good trade-off for mid-range mobile or embedded devices.
96
+ - **Smallest viable**: `acnoryx-0.8b-q3_k_m` (445 MB) — 80.2% avg. Experimental, expect quality loss. Research use only.
97
+
98
+ ## 6. Notes
99
+
100
+ - **Thinking mode**: model produces `<think>…</think>` reasoning before the final answer.
101
+ - **Non-thinking mode**: direct answer (reasoning disabled via `--reasoning-budget 0`).
102
+ - **Pass criteria**: quality checks, language checks, semantic/category checks, and keyword-anchor fallback when needed.
103
+ - Tests run with `llama-cli` on Windows with GPU offload (`-ngl 99`).
104
+ - `t/s` = tokens per second (generation speed, higher is better).
105
+ - Research formats (IQ2/IQ1 series) use **imatrix** for better quality.
106
+
107
+ See also:
108
+ - `release_gguf_0.8b/README.md` for GGUF release model details
109
+ - `results/release_gguf_0.8b/TEST_RESULTS.json` for GGUF release test data
110
+ - `results/research_gguf_0.8b/TEST_RESULTS.json` for GGUF research test data
111
+ - `results/all_gguf_0.8b/TEST_RESULTS.json` for the combined high-to-low all-model run
112
+ - `results/*/FAIL_LOG.jsonl` for failure details