Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Corbenic 
posted an update 16 days ago

On a hit the 12B is not in the loop, and your own Space says so.

app.py carries nine families and the reuse pane prints what is stored: solver_code, captioned "no answers in it, only the method, which it just ran on your numbers." So a hit executes a function. 0 generation tokens is exact because nothing was generated. The negative control agrees: empty the store and it solves nothing, because the capability was never in the weights.

Which makes 180/180 a comparison between a Python function and a model doing arithmetic in its head. The function should win that, and four models from four vendors landing on exactly 180/180 is evidence for that reading rather than against it.

The result I would have led with is the addressing one. Approximate retrieval picks the wrong item 94.3% of the time on 4,500 entries where exact addressing makes zero errors. The family list shows why: arith_series, modular_power_sum, base_divisibility are parameter templates. Two instances differing only in N and M sit almost on top of each other in embedding space and have completely different answers. Semantic caches fail hardest on near duplicates, and you have that measured.

The number that separates capability from coverage is the miss rate, and the bench cannot show it. I can only pick from your nine families, so the hit rate is 1.0 by construction.

On a stream you did not author, what fraction of queries address to a stored solver, and what writes the solver on a miss?