jackjusko/Qwen3.5-2B-RCOL

This repository contains several different dynamic low-bit quantizations of Qwen 3.5-2B. These quantizations were created with the RCOL (RCO - lite) algorithm, which is an experimental approximation version of ISTALab's RCO method. The principles of the algorithm can be found below.

RCO‑lite (RCOL)

A cheap stand‑in for RCO, the outer type‑allocation loop of ISTA's GSQ+RCO quantizer. RCO picks one GGUF quant type per tensor (Q4_K, IQ3_S, …) so the whole model minimizes end‑to‑end KL divergence from a BF16 teacher under an exact size budget - same target as traditional RCO.
Full RCO does this with Gumbel‑STE and a Riemannian budget‑manifold search - RCO‑lite attempts to make the absolute best approximation from RCO under a strict VRAM budget.

Method — a staged, KL‑verified search.

1. Seed. Start from a class‑prior × imatrix mix; measure the real sensitivity with one degrade‑one true‑KL sweep (a single class dropped to Q2_K, KL against the teacher).
2. Round 1 — single moves. Hill‑climb the type mix: promote or demote one class, measure KLD of the new model, keep it only if KLD actually drops. The lowest KL is the running minimum.
3. Round 2/3 — pair moves (once single moves plateau). Widen the move set to promote+demote pairs — a budgeted reallocation along the size isofront, where a high‑leverage promote is banked by a size‑neutral demote (a "funder"), plus multi‑rung
   jumps (skipping rungs, e.g. IQ2_S→Q3_K) that a ±1 scan can't see. Each pair is accepted by measured combined KLD (interactions are not assumed separable), keep‑if‑better.

Why not a plain imatrix/Hessian knapsack? At high compression, per‑tensor error proxies (Hessian‑diagonal, imatrix‑weighted MSE, local reconstruction) rank tensors backwards — skinny early layers look expensive but
barely move output KL, while the embedding/output has the reverse effect. Running a knapsack algorithm on that proxy is worse than uniform below ~3.5 bpw. RCO‑lite avoids the trap by spending its few expensive measurements on the real objective at candidate moves, guided
by a measured sensitivity prior rather than a curvature proxy - and started from a imatrix x class prior mix.

All gguf files in this repository have been quantized against the standard wikitext imatrix. The KL / top-p table below also measures the KL divergence on a held-out set from a standardized llama quant (also quantized with the wikitext imatrix). As you can see, RCOL at this bit-width significantly reduces KLD and top-p versus a standardized imatrix quant.

Standard quants (llama.cpp + wikitext imatrix)

Q3_K_L: KL 0.1001, top-p 84.363, 1100 MiB -- for reference, RCOL gives about an equivalent KL as IQ4_XS at the same BPW

IQ3_M: KL 0.098095, top-p 84.191, 999 MiB

IQ2_M: KL 0.352, top-p 72.218, 815 MiB

IQ1_M: KL 1.492, top-p 47.2, 703 MiB

RCOL Dynamic quants (RCOL algorithm + wikitext imatrix)

__

IQ3_M Size Equivalent: KL 0.093940, top-p 84.65, 990 MiB -- included in set

IQ2_M Size Equivalent: KL 0.172697, top-p 79.717, 815 MiB -- included in set

IQ1_M Size Equivalent: KL 0.287017, top-p 72.462, 701 MiB -- not included in set due to below, but provided for reference

Best-in-band IQ1_M Size Equivalent: KL 0.250097, top-p 74.645, 710MiB (9mb more than IQ1_M standard with imatrix, included in set, this is IQ1_M)