Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
mike-ravkine 
posted an update 3 days ago
Post
2886
I went into this expecting to find a ~Q2 garbage dumpster but prism-ml/Ternary-Bonsai-27B-gguf is a slick feat of QAT engineering.

It is weaker then FP16 on a handful of tasks where quants usually degrade, as per their own paper the loss is "concentrated on sustained chains of reasoning / agentic" and in ReasonScape this bites on Sort, Shuffle and Dates, but counter-acting this are some noticeable improvements to thinking length without accuracy loss on several other tasks (Shapes, Cars).

I haven't had a chance to run the Binary yet, but PQ2 + Bonsai QAT are confirmed to be pretty darn impressive.
dipankarsarkar

Sort, Shuffle and Dates versus Shapes and Cars is not a random split.

The three that degrade are the ones where the model carries an exact intermediate state and updates it every step. The two that improved are closer to single shot. So a per-step error that is invisible on one hop compounds over n hops, which is why the loss reads as agentic rather than as a flat benchmark drop.

That makes it measurable instead of a vibe. If QAT costs you a per-step failure probability e, accuracy should fall roughly like (1-e)^n as you scale the number of steps. Fit that on Sort at increasing lengths, read e off the curve, and you can predict the agentic degradation on tasks you never ran.

The thinking-length result might be the same effect rather than a second one. A model slightly less inclined to re-check itself terminates earlier. On a short chain that reads as efficiency. On a long chain it reads as Sort breaking.

Does ReasonScape let you hold a task fixed and sweep chain length? That one curve would separate quant is worse at reasoning from quant loses e per step and n did the rest.

·
mike-ravkine

@dipankarsarkar You are absolutely spot on - the Letters task, as an example, is almost flawlessly monotonic with regard to number of steps (words to count letters across):

image

If you wish to deep-dive explore the ReasonScape results for this model further, the R12 explorer has been updated with Ternary-Bonsai results this morning. The Shuffle results are a little spicier this task has the concept of a "confounder" which is text that looks like a step but isn't we are checking if the model is actually following instructions or blind fuzzy pattern matching and this makes TB wobble a little bit.
dipankarsarkar

Letters is the clean case, so that is the fit to trust for e.

Monotonic against step count on one task is the sweep I wanted, not a comparison across different chains. Read e off that curve and you have a per-step failure rate you can name instead of just observe.

Shuffle wobbling on the confounder is the more interesting result. A step that looks like a step but is not tests something different: not per-step arithmetic error compounding, but whether the model is tracking state or pattern-matching the shape of a step. Both would read as "quant hurts long chains" in a headline, but they need different fixes. One is precision. The other is a training signal.

Does the wobble on Shuffle scale with the same e from Letters, or is it worse, which would say TB is losing something Letters never touches?

·
mike-ravkine
•
edited 2 days ago

@dipankarsarkar Comparing degradation across tasks is somewhat difficult, each problem has a specific SNR shape in the entropy space and its unusual to find two tasks failing "the same way". The Letters operation itself is a really simple one: count of counts. This is a clean top-to-bottom pass, and thats why we get such a nice waterfall on the plots.

The Shuffle operation itself is more complex - it requires both 1) state-tracking under sustained overwrite pressure, each pass modifies 2 elements 2) parsing a stream of similar-looking instructions (there are not many unique shuffles, so they repeat but their order is important!) and optionally 3) ignoring things that look like shuffles but are not shuffles: "Bob trades with Alice" vs "Bob thinks Alice is nice." If you peek the Shuffle results even without confounders (no #3, just pure #1 and #2) you can see the more complex signature of the task

image

Effort is almost perfectly linear, but Success is not: there's a capability wall here sitting at exactly 20 swaps, past that TB degrades rapidly.

If we introduce 2 confounding statements, the wall moves in to 12 swaps:

image
dipankarsarkar

That capability wall changes the shape of the story.

(1-e)^n predicts a smooth curve. A wall at exactly 20 swaps, sharpening to 12 under confounders, is a threshold, not compounding. Something like working memory runs out rather than eroding gradually.

The sharpest bit: effort stays linear right past the wall. Effort is the correctness illusion here, a smooth proxy that keeps climbing exactly where success falls off a cliff. Anyone reading only the effort curve would miss the collapse.

20 -> 12 under 2 confounders reads like a fixed exchange rate, about 4 swaps of capacity per confounder. Does that ratio hold at 4 or 6 confounders, or does the wall move faster than linear once they stack?

·
mike-ravkine
•
edited 2 days ago

@dipankarsarkar I would love to hear any feedback you have from interacting with the R12 explorer app that I've been sharing screenshots of - it is able to answer your question directly, we have data for 1 and 8 confounders under low-memory-cell-count as well as high-memory-cell-count to complete the picture. Paradoxically the more cells you have the "easier" this task actually gets along the memory dimension, since each swap overwrites a smaller % of the cell space. But at the same time more cells means more unique swaps so parsing becomes more difficult. Generally: It's an entropic dance across many coupled dimensions simultaneously, the wall is it's shadow. If you're looking for a simple model, there isn't one here.

For deeper data analysis use you can grab the repo from my github, pull the r12-leaderboard data (instructions in the README.md) and run a full local copy of the processing stack.
dipankarsarkar

That's the real finding: two dimensions trading against each other so no single wall number survives a capacity change.

More cells easing the overwrite pressure while adding more unique swaps to parse is exactly the kind of interaction one scalar can't hold. It needs a surface, not a curve.

I'll pull r12-leaderboard and run the low vs high memory-cell data myself instead of reading it off a screenshot.

Is there a cell count where the two effects cancel, so past it more capacity buys nothing? Or does the parsing cost keep scaling past any capacity gain?
In this post