EugeneBoondock/sponge-compsearch

Sponge: online learners found by compression-scored search

This repo holds two things:

1. Learners: small, pure Python + numpy programs that learn from a byte stream one byte at a time, with no gradient-descent training phase and no GPU. Knowledge is stored as counts and a diary of everything seen, so nothing is overwritten.
2. The search that produced them (compsearch/): a language model proposes new learning algorithms, each is scored by how well it compresses text it has never seen, and the best survive.

It is an experiment, not a product. These learners are strong pattern memorisers. They are not language models you can chat with, and they are roughly a thousand times slower than standard compressors.

What is here

Path	What it is
learners/v1_single_score.py	Winner of search 1 (scored on compressing Python source)
learners/v2_multi_score.py	Winner of search 2 (scored on a multi-part curriculum)
seeds/sponge.py	The hand-written starting point
compsearch/	Scorer, safety checks, search loop, classifier mode
results/	Benchmarks, probes, leaderboards for both searches
USAGE.md	How to run everything

How a learner is scored

For every byte it must first state its belief about the next byte, and only then is told the answer. The score is the bits needed to encode the stream under those predictions, plus the compressed size of the learner's own source code, per byte. Lower is better. Because every byte is predicted before it is seen, the score measures generalisation directly and there is no test set to leak.

Results

All numbers are bits per byte on 1 MB streams; lower is better.

Against standard compressors (learner v1)

Model	Python (search stream)	Python (held out)	English prose (never seen in search)
zlib -9	1.930	1.921	2.819
bz2 -9	1.748	1.725	1.966
xz -9e	1.639	1.646	1.961
brotli -11	1.605	1.616	1.917
Sponge seed	1.859	1.847	1.817
Learner v1	1.466	1.476	1.530

These four are general-purpose compressors. The closest relatives of these learners are statistical compressors such as PPMd, PAQ and cmix, which were not run here and which we would expect to do better. Do not read this table as "state of the art".

Speed: learner v1 processes about 3 KB per second on one core; the compressors above run at 700 to 16,000 KB per second.

Wikipedia (learner v1, first 12.9 MB of enwik8)

The learner was fed the standard enwik8 Wikipedia sample until it hit a 5 GB RAM cap.

Model	Bits per byte on the same 12.9 MB
zlib -9	2.954
bz2 -9	2.339
xz -9e	2.154
Learner v1	1.744

It kept improving as it read: 1.92 on the first megabyte, about 1.67 on the twelfth. Its memory grows with everything it reads, which is what stopped it at 12.9 MB. The full 100 MB benchmark was not run, so these numbers are not comparable with published enwik8 results.

Multi-part curriculum (held-out curriculum, never used in either search)

Search 2 rewarded five things at once. The stream is Python source, then C headers, then Wikipedia, then a second, new helping of each.

Part	Sponge seed	Learner v1	Learner v2
Python	1.821	1.446	1.402
C headers	1.588	1.245	1.184
Wikipedia	2.784	2.306	2.172
Adaptation (first 16 KB of each new kind)	2.836	2.342	2.291
Retention (each kind when it returns)	1.948	1.555	1.478
Score (average, plus code size)	2.207	1.799	1.729

v2 improves every part by 2 to 6 percent over v1. The smallest gain is in adaptation: the search made the learner better at everything, but did not find a way to make it pick up a new kind of text much faster.

Behaviours nobody programmed in (learner v1)

Property	Result
Finds word boundaries	3.3 times more surprised at the first letter of a word than inside one
Spots typos	82.5% of planted typos fall in its 2% most surprising bytes (chance: 4%)
Calibrated confidence	When it claims 88% it is right 90% of the time
One-shot memory	A random string costs 7.1 bits per byte at first sight, 0.84 after one exposure
No forgetting	After 400 KB of English, its cost on Python went down 2.5%, not up
Order independence	Learning A then B versus B then A differs by at most 0.7%
Transfer between domains	Essentially none: learning Python first helps English by 1%

No forgetting, order independence and one-shot memory follow from the design (knowledge is counts plus a diary). The lack of transfer is the main weakness: it memorises patterns and builds no abstractions that carry across kinds of text.

As a classifier

One learner per label, fed that label's examples; a new input goes to the label whose learner encodes it in the fewest bits. On 300-byte snippets from unseen files:

Task	Sponge seed	zlib-based baseline	Chance
Language (Python / C header / Tcl)	97.9%	95.0%	33.3%
Purpose (test code vs library code)	75.6%	75.6%	50.0%

Quick start

from compsearch.classify import Classifier, load_learner_factory

clf = Classifier(load_learner_factory("seeds/sponge.py"))
clf.learn("english", b"the cat sat on the mat and looked at the dog")
clf.learn("digits", b"3141592653589793238462643383279502884197")
print(clf.judge(b"the dog looked at the house"))   # {'english': ~1.0, 'digits': ~0.0}

See USAGE.md for the search loop, benchmarks and probes.

Limits and cautions

• Not a chatbot. It predicts bytes from surface patterns and has no world knowledge.
• Slow. Pure Python, a few kilobytes per second.
• Memory grows with data. About 5 GB after 13 MB of Wikipedia.
• Small evidence base. Two searches of 5 generations each (23 and 15 proposals), all streams 1 MB. Treat the numbers as a first measurement.
• Machine-written code. The two learners in learners/ were written by a language model (Claude Opus 5.5) during the search and selected by score. They pass a static safety check that restricts them to pure computation, but read them before running them in a process you care about. The search itself runs candidates in a separate confined process; see the safety section of USAGE.md.
• No licence has been chosen yet for this repository.

How it was made

The seed learner and the search harness were written with Claude; the candidate learners were proposed by Claude Opus 5.5 through the search loop. Data used: Python standard-library source and C headers, Python's changelog (English prose), and the enwik8 Wikipedia sample. No data files are included in this repo.