adhikjoshi commited on
Commit
ecdcedf
·
verified ·
1 Parent(s): 84b13fa

Record the reference-free text-accuracy sweep

Browse files
Files changed (1) hide show
  1. README.md +26 -0
README.md CHANGED
@@ -50,6 +50,32 @@ Only the transformer is quantized. The Qwen3-VL text encoder stays bf16 and is a
50
  16.3 GB of a 21.5 GB resident total, so it, not this file, decides how many reference
51
  images fit on a card.
52
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
53
  ## Requirements
54
 
55
  Packed for a specific kernel. The layout is an MMA fragment swizzle, so a different
 
50
  16.3 GB of a 21.5 GB resident total, so it, not this file, decides how many reference
51
  images fit on a card.
52
 
53
+ ## Text accuracy against bf16
54
+
55
+ Every LPIPS number above is measured *against* bf16, so it cannot answer "is this as good
56
+ as bf16" — it defines bf16 as zero. This is the reference-free version: 8 seeds on each of
57
+ two text prompts, scored by reading the rendered text against what the prompt asked for.
58
+
59
+ | build | enamel notice board | neon shop sign | total |
60
+ |---|---|---|---|
61
+ | bf16 | 8/8 | 6/8 | **14/16** |
62
+ | this checkpoint | 8/8 | 7/8 | **15/16** |
63
+
64
+ **Indistinguishable.** One render's difference out of sixteen, which a different seed set
65
+ would flip. The errors trade places rather than accumulating on one side: on one seed bf16
66
+ renders the sign as "QWEN 2.1 IMAGE 1" and this checkpoint gets it right; on another this
67
+ checkpoint appends a spurious "1$" where bf16 is clean.
68
+
69
+ That is the claim worth making — **4-bit weights and activations with no measurable text
70
+ degradation** — and not the stronger one. A single prompt where the quantized build beats
71
+ bf16 is trajectory variance, not evidence of superiority.
72
+
73
+ An automated OCR pass was tried first and discarded: tesseract returns noise on neon at
74
+ night, and a character-error-rate above 1.0 was the giveaway that it was scoring incidental
75
+ scene text rather than the sign. The table above comes from reading all 48 renders.
76
+
77
+ `samples/` holds one render per prompt at seed 42. `seeds/` holds the full sweep.
78
+
79
  ## Requirements
80
 
81
  Packed for a specific kernel. The layout is an MMA fragment swizzle, so a different