Add beta7 W4A8 conservative quant (by cicalooo)

#31
by sandpies - opened

Beta7 W4A8 conservative quant by cicalooo, made with cicalooo's h3quant v2 script (his default profile conservative-w4a8, group size 16), uploaded on his behalf by sandpies while he's away.

  • int8 ConvRot (g256) on all block linears, W4A8 g16 codebook on mlp.fc1 of blocks 10-37 (172 int8 / 28 W4A8 layers); everything else kept byte-identical bf16
  • 19.08 GB (19,079,401,912 bytes), aimed at RTX 3090 / 24 GB cards
  • Script's --verify passed: int8 rel err ~0.009-0.012, W4 fc1 ~0.073
  • Run on Windows with I/O-only portability shims; quant math unchanged

All credit for the quant method to cicalooo.

TenStrip changed pull request status to merged

Sign up or log in to comment