Tardelli Stekel PRO

Spread out, and in every band.

In Octen-4B the 63 sit in 54 tensors (46 hold one, 7 hold two, one holds three), across all five norm kinds: q_norm, k_norm, the input and post-attention layernorms, and the final norm. Every tensor that holds a top code moved all of them; none moved some and kept others. They cover 12 exponents, from 2^-16 to 2^3, and in each one all of them moved up. Octen-8B is the same: 68 in 46 tensors, one tensor holding 9, all moved, over 8 exponents (2^-6 to 2^1). The decay model: 43 in 22 tensors, all toward zero, over 5 exponents (2^-11 to 2^-1).

One correction on the 6%: that's how often the 15 codes below move up. Counting both directions they move 27% of the time in Octen-4B, mostly toward zero.

And one cut that splits the family: Octen-0.6B has 35 top codes in 28 tensors and none of them moved, while its 15 codes below move 20% of the time. There the top code stays put.

Both checks say it's in the weights, not the measure. The kept values are the exact bf16 values decoded from the bytes, a flip is counted as x != y, and the 16 were picked by int16 code distance. Of the 94 in Octen-4B, none differs from the parent, bit for bit. As code distance, every move in Octen and in the decay model is exactly ±1 code, and none of the 2^k values moved.

The other edge of the binade looks just as odd. At the top code, m = 255/128, one step below 2^(k+1):

• Octen-4B: all 63 moved up onto 2^(k+1), against 62 of 1,087 (6%) for m in [1.875, 255/128).
• Octen-8B: 68 of 68 up, against none of its neighbours.
• The decay model: all 43 moved toward zero, while none of its 50 on 2^k did, at the same threshold.

So in these models nothing leaves a power of two, and in Octen everything one code below one lands on it. Luna, jina, huozi and CAT don't show it: their top codes don't move. I don't have a mechanism yet.

94 of the 1,597 sit exactly on a power of two (51 at 2.0, 12 at 0.25, 10 at 0.5, 8 at 1.0, the rest at other powers), spread over k_norm, q_norm and both layer norms, all in tensors that moved. None flipped, either way.

At the rate of the m ≥ 1.875 values (233 of 1,150 toward zero), about 19 should have. So the finer binade below doesn't show up for them: they behave as if half a step were 1/256 of the value, like the rest of their binade.

Same in the others: 1,390 power-of-two values across the 16 pairs, 14 flips (8 toward zero, 6 away), all in four models that also flip below m = 1.125. The decay model too: none of its 50 flipped, while 247 of its 506 values at m ≥ 1.875 moved toward zero.

They aren't special values: 94 is about 1/16 of the 1,597, as for any other mantissa code in that range.

Same 1,630 either way: 6.5% of all 24,888 declared fine-tunes, 7.1% of the 22,847 that could be compared. The post uses the second and says so. The first is the stricter floor, since the 2,041 uncompared are unknowns, not changes.

On direction, from the 64 values kept per norm tensor:

Octen-Embedding-4B: 1,035 of the 1,727 flips point toward zero (60%), 692 away, and 144 of the 145 tensors have both. The flip rate doesn't rise at small |x|: 17-23% in every binade from 2^-5 to 2^0. It does depend on the mantissa: none of the 1,597 values with mantissa below 1.125 flipped, 6-39% above it. So it reads as a relative change of about 0.2-0.4% per value, in both directions. Neither the Adam mark nor the pure decay mark.

Across the 16, three patterns:

• One Mistral-Nemo fine-tune: all 1,518 flips point toward zero, in every tensor. Your decay mark.
• Two others (a Mixtral fine-tune and a 1.4B translation model): every flip points away from zero (827 and 773), each tensor one way, none below mantissa 1.25. A per-tensor scale-up, the mirror of decay.
• The other 13, the three Octen included: mixed, 50-60% toward zero, flat across binades.

None shows flips piling up at small |x|.

Your question caught an error in my last reply. When exactly one side is quantized (bnb 4-bit, FP8, int8), the engine labels the pair from the dtypes alone and compares no values. So those 764 were never measured, and "match it after quantization" was wrong.

Your split, on 758 of them: the named quant's card never points back at the child, so there are 0 two-cycles. 546 name a third repo, the full-precision original (mostly unsloth bnb-4bit repos). 212 have no base_model, mostly FP8-native releases that are themselves the original.

Then I diffed each child against that original directly. 594 could be measured: 549 (92%) changed (465 lora_merge, 54 full fine-tunes), 18 hold the same weights, 21 are speculative-decoding drafters. So for the bnb-4bit majority the arrow isn't reversed. It's QLoRA trained on the 4-bit repo and merged into 16-bit, and the card names what it trained on. The headline drops with it: measured unchanged fine-tunes are 6.5%, not 12%. I'll correct the card.

1 ulp: all 60 are bf16, but by construction. The scan only reads bf16 norms, and fp32 norms have no such edge. 4,324 of the 5,614 lora_merge verdicts have bf16 norms on both sides, so that's where the edge could hide. The 60 come in two shapes. 17 are sparse: under a fifth of the norm tensors moved, often by a single value, like your case. 16 moved every norm tensor, each changed value by exactly 1 ulp (Octen-Embedding-4B: 145 of 145 tensors, 1,727 of the kept values; gpt-oss-safeguard-120b: 73 of 73 tensors). Those look like updates well under one step, where only values sitting near a rounding boundary flip.

m5 read 2 after today's update, so we both called it. Without hf_xet, snapshot_download sends a HEAD and a GET to /resolve for the .gguf and both count, like a first hf_hub_download. With hf_xet installed it's 0, since the bytes never go through /resolve. Both cases are in the data card now: https://huggingface.co/datasets/modelpulse/model-pulse-data

Thanks for reading whole tensors. That's a stronger check than mine.

First, a correction to my post: the sample isn't 128 bytes. Tensors up to 32,768 values are read whole and compared byte for byte, and that covers every norm vector in both of those models. Bigger tensors get a 512-value window from the middle row. So if a single norm value moves by 1 ulp, the tensor counts as changed. Your embedding rows go further than the engine does, since it reads only 512 values of one row there. I've fixed the "128 bytes" on the card and the method page.

On the 1,365: the count is right, but the direction splits. In 601 of them the child is a quantization of the parent its card names, which is your reading. In the other 764 it goes the other way: the card names a quantized repo (FP8, AWQ, I8…) as the parent, and the child holds full-precision weights that match it after quantization. Those are the original or a dequantized copy. So about 20% of the 2,923 are quants labeled as fine-tunes.

1 ulp: yes. I went through the norms of 14,088 declared fine-tunes, using the 64 values the store keeps per tensor. Of the ~8,000 pairs where any norm moved, 60 differ by exactly 1 ulp everywhere I can see. Closest to your case: Nabbers1999/Mini-Llama-8B-Chat-SFT-0129 vs Ministral-3-8B-Base-2512. 1 of 69 norm tensors moved, and only one value, by 1 ulp. One more rounding and it reads as lora_merge. 48 full fine-tunes have only 1 or 2 norm tensors moved. So your case is real at the edge, and some lora_merge pairs are probably full fine-tunes whose norm updates all rounded away. Weights alone can't separate those from frozen norms, which is why the verdict reads "a merged LoRA (or a fine-tune with frozen norms)".

Your numbers reproduce here. Using the observed tie values, with 2-decimal triples drawn uniformly and summed in random order, I get 3.66 expected first-listed labels on the 25 multi-option ties (observed 3) and 0.23 expected "false" on the 10 noul ties (observed 1, P(≥1) ≈ 0.21).

On the samples: the public release has only the means. The repo holds the parquets, the card and bench/, and nothing per sample. The closest thing is label_agreement, which keeps per-question summaries (argmax_agree, argmax_majority, total_variation). That doesn't recover the bit either: on the 35 ties argmax_majority always lands inside the tied pair, but in the 31 cases where label and the scorer split it sides with label only 21 times. Whether the generator run kept the raw samples is a question for the maintainers. I'll ask on #68 so it sits next to the tie-rule question.

Reproduced your three-reading split on test, and train shows the same pattern (61 ties, label on the later-listed option in 53).

My best guess for the 4, which only the generator code can confirm: label was taken from the unrounded means, before the 6-decimal rounding, with ties going to the later option. Two things point that way. The gold values are means of three 2-decimal samples (p x 3 sits on the 0.01 grid for 98.5% of the 28,400 probabilities). And the noul ties are all exactly 0.5, with label "true" in 9 of 10 on test and 21 of 22 on train, which is what a ">= 0.5 is true" rule gives. The exceptions then come from float noise: summing three 2-decimal values in a different order changes the last bit in about 13% of permutations, so two options that tie after rounding can differ by ~1e-16 before it, and that decides it. customer_service_000066, the one noul tie labelled "false", is one of your 4.

If that's right, label is deterministic but not meaningful on ties, which is one more reason for symmetric split credit. I opened the PR that saves per-decision probabilities (#68 on the dataset) and left the tie rule out of it, so the maintainers can pick it separately.

Confirmed on c9e8c416: 35 ties, all two-way, 17 of them including the first-listed option. One more thing in the same spot: the gold label field breaks those ties the other way in 31 of the 35, so scoring against label and running score_with_jobs.py give different numbers for the same predictions. Rows on the card use both: Decider-4B, od1 and OpenDecider match their card values under label; nullspan and Qwen3.8-27B (#60) match under the scorer's argmax. Across the 12 runs the two rules differ by 0.001 to 0.007.

On Jev, the 0.006 to 0.0065 gap splits into 9 decisions from gold ties and 3 from ties in the prediction itself. Jev returns probabilities rounded to two decimals, so 20 of its 2,000 answers have a tied top option, and list.index breaks those by option order too (OpenDecider-medium has 104).

So yes to split credit, and I'd apply it on both sides: the expected hit when gold ties and predicted ties are each broken at random, |top_gold ∩ top_pred| / (|top_gold| × |top_pred|). It is deterministic, independent of option order, and equal to the current rule when nothing is tied. For Jev's whole-case run it gives 0.7328, between the scorer's 0.7305 and label's 0.7350.

Agreed on the PR: save the normalized probabilities per decision, keyed on case id and question id, not the hits, so the tie rule becomes a scoring choice anyone can rerun.

Per-rule table for all 12 runs: https://huggingface.co/spaces/tardellirs/typed-decisions-explorer/tree/main/analysis

Checked all of it in the 2.4.0 install: line 1899 returns None without Xet, is_xet_available reads HF_HUB_DISABLE_XET and then looks for hf_xet (utils/_runtime.py, line 157), and the METADATA marker is the one you quote. Without hf_xet, xet_file_data still comes back from the HEAD, so line 1818 keeps url_to_download on /resolve, and http_get goes there (2026–2034). HEAD plus GET on the query file.

My bet is 2 as well. The one call m5 adds that we haven't priced is a GET on /resolve that answers 302 to the Xet bridge, and m4's 302 HEAD counted, so I don't see a way to 1.

The default still decides the count, though. hf_xet comes with huggingface_hub on x86_64, amd64, arm64 and aarch64, which covers Macs, almost every Linux box and Windows. The 2 needs HF_HUB_DISABLE_XET=1, an install without hf_xet, or a machine outside that list. So for a GGUF repo the counter should still miss nearly every snapshot pull. It just isn't true everywhere.

I'll run m5 as m3 with HF_HUB_DISABLE_XET=1 and read it after the next batch. The card's line gets "with hf_xet installed" either way, plus the m5 number once it's in.

The batch ran: m3 = 0, m4 = 1. That's the first row, so it lands neither on your (1, 2) nor on the token reading.

Every request in the log carried X-HF-Download-Counter: 1, so:

• The header doesn't decide anything. m3 sent seven flagged calls (revision, tree, HEAD+GET on .gitattributes and README.md, the xet-read-token GET), and none counted.
• The token GET doesn't count. Both repos sent one, and m3 stayed at 0.
• m4's 1 is the HEAD on /resolve for model.gguf, answered with a 302. A 302 HEAD counts like m1's 200 HEADs did.
• A GGUF on Xet fetched with snapshot_download counts zero, because its bytes never touch /resolve.

That last one is wider than scripts. In 2.4.0, hf download <repo> with --include, or with more than one file name, also goes through snapshot_download (cli/download.py, line 187). Only a single file name takes hf_hub_download. So hf download <repo> --include "*Q4_K_M.gguf", the usual way to pull one quant, shouldn't count either. That's from the source; I haven't run it.

As promised, the card keeps its sentence and gets one line:
"A .gguf stored on Xet and fetched with snapshot_download (huggingface_hub 2.4.0) counted 0: the client takes the file's Xet hash from the tree listing and pulls the bytes from Xet storage, with no request to /resolve. The same file fetched with hf_hub_download counted 1, for its HEAD on /resolve."

For Model Pulse it means a GGUF repo's counter undercounts by an amount nobody can see from the Hub. Requests on /resolve still count; snapshot pulls of Xet GGUFs don't.

Thanks for re-running it. Since then I found more predictions: the OpenDecider repo (#8) publishes per-decision files for its own models and for its runs of Jev 1.13 (TypeSafe API), microsoft-decision-1 and clm-8b (the last two are not on the card), so the check now covers 12 runs. Numbers below use the analytic case-level SE, which matches the bootstrap and gives 1.35 for od1 as you found.

On the top: deff does grow with accuracy (Spearman 0.82 over the 12), from 1.02–1.18 below 0.67 to 1.28–1.40 above 0.78. The highest is Decider-4B at 0.827 with 1.40, n_eff about 1,430. So it rises, but in this sample it levels off near 1.4.

Paired against Jev (OpenDecider's whole-case run, 0.7305 under the dataset scorer's argmax rule, 0.737 in their own report, which breaks the 35 tied gold labels differently; the card's 0.727 is a separate run), the paired SE is 0.72–0.93x the independent one, median 0.79. d3-mini's +0.016 stays inside the band even at 0.72x.

On showing the paired band when both rows ship predictions: I'd rather not mix two kinds of band in one table yet. Of the 90 rows, only about ten have public predictions, two of them zero-shot, and the only Jev predictions come from a third-party run rather than the card's. The cleaner fix is upstream: score_with_jobs.py writes only the summary to results.json; if it also wrote the per-decision answers, every new row would ship them and the explorer could move every comparison to a paired case bootstrap at once. @sandeep85 @codelion , would that be welcome as a PR?

Script, sources and output: https://huggingface.co/spaces/tardellirs/typed-decisions-explorer/tree/main/analysis

This comment has been hidden

Thanks, Dipankar, both points are fair to raise.

On the post line: the explorer leaves nanoGentzen-Jev-v3 out of the zero-shot view, and the footer gives the reason (its own eval YAML in #59 describes it as fitted on train). The post followed that view without saying so; I should have. And yes, "above Jev" there is a point estimate: by the explorer's own rule neither 0.731 nor 0.743 separates from 0.727.

On n: you're right that the 5 decisions per case are not independent, so I ran a case bootstrap on the submissions that ship per-decision predictions (od1-typed-decisions, Decider-4B, nullspan v1.3.0, Qwen3.8-27B from #60), using the scorer's own accuracy rule. Design effects came out at 1.0–1.4, so n_eff is about 1,450–2,000, not 400. For comparing two models it helps further that both answer the same cases: the paired, case-resampled SE of the difference is 0.64–0.90× the independent n = 2,000 SE the explorer uses, so its tie band is, if anything, conservative. Caveat: only four models, and none of the current top zero-shot rows publish predictions, so I can't check those directly. If more submissions add predictions, a paired case bootstrap would be the right rule and I'll switch to it.

Line 575 checks out, and so does the split. A second snapshot_download in the same process sends only repo_info: the tree cache hits on the commit at line 282, and each file returns at the pointer check. If it lands on (1, 1), I'll run that.

On the card: it now says only requests to a library's query files count. (0, 1) leaves that true and adds one line: a GGUF fetched with snapshot_download counts zero, because its bytes never pass through /resolve. Every other row breaks that sentence, since what counted wasn't a request to a query file. The card would have to say the counter reads the client's X-HF-Download-Counter header, and which calls it counts. Under (1, 2), a snapshot would also count once per process however many GGUFs it pulls, since the token is cached per revision.

For Model Pulse that matters more than the card. Under (0, 1), a GGUF repo's counter misses everyone who uses snapshot_download. Under (1, 2), it counts them once per run, not per file. Either way it isn't a file count.