Hugging Face's logo

AEUPH
/
hdcstorage

Safetensors
hdcstorage
hyperdimensional-computing
vector-symbolic-architecture
hdc
binary-spatter-code
compute-in-memory
webos
python
Model card Files
xet
 Community
hdcstorage / README.md
AEUPH's picture
AEUPH
HDCSTORAGE: hyperdimensional computer, webOS and agentic MCP layer (stdlib-only Python)
ad474c9 verified
preview code
|
Raw History Blame Contribute Delete
48.6 kB
---
library_name: hdcstorage
tags:
- hyperdimensional-computing
- vector-symbolic-architecture
- hdc
- binary-spatter-code
- compute-in-memory
- webos
- python
---
# HDCSTORAGE
A hyperdimensional computer that keeps its entire state in hyperdimensional
storage, and an agentic MCP tool layer that lives inside that storage.
Programs are compiled into **hypervectors**, written into cells of a
content-addressable HDC memory, fetched back out, and **decoded by cleanup
memory** β€” the decoder never reads a fixed bit field, it asks which known
symbol the fetched vector most resembles. Registers, addresses, stack slots,
literals, tool records and string constants are all the same kind of object:
a 10 000-bit vector.
```
.hdcasm source
| compiler (assembler.py)
v
instruction hypervectors ---> HDC storage cells (memory.py)
|
| fetch by content, decode by cleanup
v
interpreter.py ---> debugger.py / ide.py
|
| TOOL opcode
v
agent.py (tool records in HDC)
|
v
mcp_server.py (JSON-RPC over stdio)
```
**Standard library only.** No numpy, no third-party packages. Hypervectors are
Python big integers, so `bind` is one `^`, `similarity` is one `bit_count`,
and a 10 000-bit vector costs 1 250 bytes.
---
## Quick start
```bash
cd D:/HDCSTORAGE
python -m hdc demo # end-to-end tour, every layer, measured
python -m hdc run examples/hello.hdcasm
python -m hdc asm examples/logic.hdcasm # assemble + decode-back report
python -m hdc ide # the IDE (REPL)
python -m hdc adder 7 35 # the HDC full adder, stage by stage
python -m hdc shader # HDC image pipeline: dim, edges, depth, normals
python -m hdc mcp # MCP server on stdio
python -m hdc mcp --selftest # protocol self-check, no client needed
python -m hdc net # BitNet layer self-check
python -m hdc isa # instruction set reference
python -m hdc tools # the HDC-resident tool registry
python -m hdc os # the webOS: desktop, windows, apps, memory map
python -m hdc boot # POST -> shader splash -> AUTOEXEC, frame by frame
python -m hdc gfx # shader pass timings on an 80x28 surface
python -m hdc hub # which embedding table this machine can use
python -m hdc autonomy # idle + readahead on a simulated session
python -m hdc assist --autonomous # chat with it; idle ticks between turns
python -m hdc test # unit tests (over 300)
```
Every command takes `--d <int>` to change dimensionality (default 10 000).
`hub`, `autonomy` and `assist` are the exceptions that matter: they measure the
codec rather than the machine, so `autonomy` and `assist` default to `--d 2000`
(the codec floor) unless a dimension is given. The same simulated session scores
80% next-intent accuracy at 2000 and 70% at 10 000 β€” a fixed 21-symbol cleanup
codebook is diluted by extra bits, so the machine's default would misreport the
layer.
---
## 1. The algebra (`hdc/hypervec.py`)
Binary spatter codes:
| operation | implementation | meaning |
|---|---|---|
| bind | `a ^ b` | association; self-inverse, so unbind == bind |
| bundle | bit-sliced majority | superposition / set |
| permute | bit rotation in the d-bit mask | order and position |
| similarity | `(d - 2Β·popcount(a^b)) / d` | agreement in `[-1, 1]` |
Bundling is a real majority, not a sum: every vector is added into a d-lane
binary counter (one Python int per bit of the count) and a bit-sliced
comparator thresholds it at `floor(n/2)+1`. That keeps the cost at
`O(nΒ·log n)` big-integer operations instead of `O(nΒ·d)` element visits, which
is what makes 10 000-bit bundles cheap in pure Python.
The numbers that matter, measured by `python -m hdc demo` on this machine:
```
random pair similarity mean +0.0001 sd 0.0100 <- noise floor
bundle of 5 members recovered at 0.369 - 0.382 <- signal
```
Everything downstream β€” cleanup, decoding, dispatch β€” is a decision made
between those two numbers.
**Integer codec.** A scalar is stored as a superposition of bit-position atoms
(`bind(NUM, permute(BITBASE, i))` for each set bit), so integers are vectors
like everything else and decode by probing each bit position. The layout is
**32-bit two's complement**, because sign-magnitude has no meaningful carry
and a ripple-carry adder needs one. This codec is *symbolic, not
similarity-preserving*: integers that share set bits share atoms, so
`sim(int_to_hv(5), int_to_hv(37))` is high. Read `int_to_hv`'s docstring before
using distances between encoded integers as if they meant something
semantically.
---
## 2. Storage (`hdc/memory.py`)
A cell is one hypervector with a name; the name's address is also a vector
(`hvc("cell/" + name)`).
* `write(name, hv)` / `read(name)` β€” named access.
* `resolve(addr_hv)` β€” content addressing: exact hit at confidence 1.0,
otherwise nearest cell address above a floor. A damaged address still lands
on the right cell and reports how sure it was (`demo` flips 12 of 10 000
bits and resolves at ~0.998).
* `recall(probe, k)` β€” associative search over cell *values*; the
compute-in-memory read.
* `nearest_symbol(hv)` β€” cleanup memory, the table that turns a noisy decoded
field back into a name.
Persistence is a byte-stable `.hdc` image: magic, header, packed vectors,
symbol table. Symbolic vectors are regenerated from their names on load, so an
image is portable at the same `d`.
---
## 3. Instruction set (`hdc/isa.py`)
37 opcodes. An instruction is a bundle of bound fields:
```python
instr = bundle([
bind(role("op"), hvc("op/ADD")),
bind(role("ra"), hvc("reg/R2")),
bind(role("rb"), hvc("reg/R0")),
bind(role("rc"), hvc("reg/R1")),
])
```
Decoding unbinds `role("op")`, cleans up over the opcode alphabet, then reads
exactly the operand slots that opcode declares β€” and reports the minimum
similarity across fields as its confidence. Measured on assembled programs:
opcode-only instructions decode at 1.00, four-field arithmetic at 0.36-0.50,
against a 0.01 noise floor.
| class | opcodes |
|---|---|
| control | `NOP HALT JMP JZ JNZ CALL RET` |
| integer ALU | `LDI MOV ADD SUB MUL DIV AND OR XOR NOT SHL SHR POPC CMP` |
| hyperdimensional | `BIND BUNDLE PERMUTE SIM CLEAN` |
| storage | `STORE LOAD RECALL MEMSTAT PUSH POP` |
| I/O | `OUT OUTT PRINTS` |
| agentic | `TOOL` |
| debug | `BREAK` |
`python -m hdc isa` prints the full table with descriptions.
**Semantics, stated plainly.** Registers hold hypervectors. `BIND`, `BUNDLE`,
`PERMUTE`, `SIM`, `CLEAN`, `STORE`, `LOAD`, `RECALL`, `TOOL`, and now `ADD` and
`SUB` never leave hyperdimensional space β€” the arithmetic runs as the
ripple-carry circuit described in Β§6. `MUL`, `DIV`, the bitwise ops and the
shifts still decode the vector payload to a Python integer, compute, and
re-encode; that is a conventional ALU behind a hyperdimensional register file.
Jumps use the decoded cell index, which the assembler bound into the
instruction.
---
## 4. Language (`hdc/assembler.py`)
```asm
; comments start with ; # or //
.str banner "hello from hyperdimensional storage" ; a string cell
.data counter 0 ; a preloaded integer cell
.name result ; declare a cell name
start: ; labels name program cells
ldi R0, 21
mul R2, R0, R1
store result, R2
load R3, result
tool R4, R0, R2, hdc.similarity
prints banner
halt
```
Two passes: scan (labels, and the literal/string/name/tool alphabets that
become cleanup memory) then emit. Assembling runs a **decode-back check** on
every cell and refuses to emit a program whose cells do not decode back to
their own opcode β€” the compiler verifies the image, not just the text.
A string operand may be quoted inline or name a `.str` cell declared anywhere
in the file.
---
## 5. Interpreter and debugger
`Machine` fetches cell `code/<pc>` through the content-addressable path,
decodes it, and executes. Decoding is memoised by vector value (a pure
function of vector + alphabet), so fetch still goes through storage every
step while repeated executions stay fast.
`Debugger` adds breakpoints by cell or label, step / step-over, a trace ring
with per-step confidence, and a register table whose `SYMBOL` column is a
cleanup query β€” with the match hidden below the noise band rather than shown
as a bogus 0.02 hit.
```
hdc> asm
hdc> bp compute
hdc> run
stopped: break at pc=2 (breakpoint)
hdc> regs
hdc> disasm
hdc> bits R3
```
---
## 6. Native arithmetic (`hdc/adder.py`)
`ADD` and `SUB` do not decode. They run a carry chain whose only primitive is
binding plus cleanup:
* a bit is read by a similarity probe against its position atom;
* a gate is a truth table **stored as records in HDC storage** β€” 8 rows
(`adder/row/000` … `adder/row/111`) for a 1-bit stage, 2**(2k+1) rows for a
k-bit stage;
* one stage is `bind β†’ clean up`: form the key for the stage's inputs and ask
storage which row it resembles, resolving well above the 0.01 noise floor and
reporting that number per stage;
* the sum is reassembled by bundling position atoms, and the carry feeds
forward as the next stage's input;
* `SUB` is `a + ~b + 1` in two's complement β€” no sign case analysis anywhere.
**Stage count is a tunable.** A gate that consumes k bits per stage trades
stage depth for table size:
| radix | table rows | stages per 32-bit add | measured add, d=10 000 |
|---|---|---|---|
| 1 | 8 | 32 | 0.98 ms |
| 2 | 32 | 16 | β€” |
| 4 | 512 | 8 | 0.38 ms |
| 8 | 131 072 | **4** | **0.38 ms** |
The default is **radix 8: four byte-slice stages**. A 131 072-row table at
d=10 000 would be 164 MB of vectors and ~1.6 s of XOR folding to enumerate,
so `hdc/slicegate.py` materialises rows **lazily**: a 32-bit add touches four
byte rows, they are composed once, decoded, written to storage and cached, and
the worst case is still complete. Gate keys are exact (one atom per slot,
position and value), so they are looked up in a dictionary β€” a 131 072-row
similarity scan would cost more than the ripple it replaces β€” with the fuzzy
`resolve_similar` path kept for damaged keys.
Two design constraints cost real debugging time and are now regression-tested:
**The codec had to become two's complement.** Sign-magnitude has no meaningful
carry, and negative values in two's complement are *dense* (`-1` sets all 32
atoms), which forced the probe threshold down to 0.10 and pins the supported
floor at `d = 2 000`.
**Each gate slot needs its own atom pair.** A key built from one shared
zero/one pair is an XOR of atoms, and XOR is parity: `1^1^1` and `1^0^0`
collapse to the same vector, leaving 2 distinguishable rows out of 8. Tagging
inputs with a role before the XOR does *not* help β€” the role constants cancel
into a fixed mask. Per-slot pairs (`gate/atom/a/0`, …) restore eight
near-orthogonal keys (worst cross-similarity βˆ’0.02 measured).
Cost, measured: **0.38 ms per add at d = 10 000** with radix 8, and 0.98 ms
with the 1-bit gate, versus ~0.05 ms for the old decode-and-add. The 32-bit
wrap is now hardware-like instead of an exception. `python -m hdc adder 7 35`
prints the table and the carry chain.
## 7. BitNet base (`hdc/net.py`)
1-bit and ternary kernels on the same primitives, because a 1-bit dot product
*is* an XOR and a popcount:
```python
dot_1bit_packed(a, b, n) == n - 2 * popcount(a ^ b) # == dot_1bit
```
Also `ternarize` (absmean with a dead zone, in the spirit of BitNet b1.58),
`quantize_absmax`, `ternary_matvec`, `BitLinear`, and `PrototypeNet` β€” a
nearest-prototype classifier whose class weights are literally `HDCStorage`
cells, so they persist and are addressable by content like anything else.
`python -m hdc net` verifies the XOR identity at run time and reports measured
sparsity and accuracy values. Nothing in this repository is a stored
benchmark; every number is recomputed.
---
## 8. Agentic tools inside HDC (`hdc/agent.py`)
A tool is a record, stored as a cell:
```python
bundle([
bind(role("name"), hvc("tool/hdc.bind")),
bind(role("cap"), capability_vector("hdc/bind/construct")),
bind(role("arity"), hvc("arity/2")),
])
```
Dispatch is similarity, in three tiers, each reporting confidence:
1. exact name β†’ 1.000;
2. character-superposition similarity, so `hdc.bundl` finds `hdc.bundle`
(measured 0.74-0.76) β€” a misspelling is a cleanup problem, not a crash;
3. capability-path similarity, so `hdc.memory.search` reaches `hdc.recall`
and `bit.count` reaches `bit.popcount`.
The same records are what the `TOOL` instruction calls and what the MCP
server advertises, so a program running inside HDC and an external agent reach
identical tools by identical paths.
---
## 9. MCP server (`hdc/mcp_server.py`)
JSON-RPC 2.0 over stdio, both newline-delimited and `Content-Length` framing.
Methods: `initialize`, `tools/list`, `tools/call`, `resources/list`,
`resources/read`, `ping`. Resources: `hdc://isa`, `hdc://memory`,
`hdc://tools`, `hdc://program`.
Hypervectors are returned as `0x`-prefixed hex strings and accepted in either
form, because a 10 000-bit integer does not survive a JSON number in most
clients.
Client configuration:
```json
{
"mcpServers": {
"hdcstorage": {
"command": "python",
"args": ["-m", "hdc", "mcp"],
"cwd": "D:/HDCSTORAGE"
}
}
}
```
```
$ python -m hdc mcp --selftest --d 2000
hdc.decode_int -> hdc.decode_int conf=1.000 42
hdc.similarity -> hdc.similarity conf=1.000 1000
hdc.bundl -> hdc.bundle conf=0.743 0x21c14d6466b6...
hdc.memory.search -> hdc.recall conf=1.000 9
```
---
## 10. HDC shader layer (`hdc/shader.py`)
An honest answer to "shaders in hyperdimensional space": a small operator set
on pixel records, with the exact/approximate boundary marked in the API.
**Exact.** Dimming is `shift(hv, Β±k)` β€” multiply or divide a coded value by
2**k with two rotations and two XORs, *no probe and no decode*. It works
because the codec's bit atoms are a permutation orbit, so rotating the vector
rotates every atom index; measured 0 failures over 2 120 shifted values. Sign,
ordering, `max`, and absolute differences are bit scans plus the slice adder.
The depth buffer resolves visibility with that exact comparator.
**Approximate.** Discontinuity (`1 - sim`) is a cheap edge proxy; `edges()`
uses the exact difference by default and the two agree at 0.74 vs 0.72 on a
test step. Normal shading is a LUT: a 64-entry direction codebook cleaned up by
similarity, with Lambert terms precomputed per entry β€” exact within the table,
quantisation error between entries.
One finding worth recording: the integer codec is **symbolic**, so `sim` cannot
rank nearby values (20 and 21 score 0.506, about the same as 20 and 200). A
normal LUT built on it returned a geometrically much worse entry. Switching the
codebook to a **thermometer code** β€” value v is the bundle of atoms 0..vβˆ’1, so
similarity falls monotonically with |vβˆ’w| β€” fixed it: 16/16 stored directions
recovered, 68–70% exact nearest-entry agreement on random queries with mean
cosine deficit 0.03. The remaining misses are the same capacity law as
everywhere else: a 3-axis Γ— 64-step thermometer is a 192-member bundle, whose
own signal is 1/√192 β‰ˆ 0.07.
```python
fabric = ShaderFabric(d=2000)
img = step_image(fabric=fabric) # synthetic step edge
img.dim(1).levels()[0] # [100, 100, 100, 100, 20, ...]
img.edges().levels()[0] # [0, 0, 0, 160, 0, ...]
```
## 11. Memory modes (`hdc/memory_modes.py`)
Three ways to remember one thing, measured rather than asserted.
| mode | structure | recall | capacity |
|---|---|---|---|
| photographic | one cell per key | bit-exact or a miss | `capacity` cells, oldest evicted |
| holographic | one majority superposition | cleanup over a codebook | degrades with items |
| hybrid | photographic buckets + a trace each | exact hit, else trace | eviction keeps an approximate path |
**The finding that matters: XOR accumulation does not work.** The obvious
holographic memory is `trace ^= bind(key, value)`, and it is broken at any
size β€” the crosstalk after unbinding is an XOR of random terms, which is
itself random at every bit. Measured exact recall with an 8-symbol codebook:
**0/2, 0/4, 2/8 items (chance)**. Majority superposition (a bit is set when
most bound terms set it) gives **8/8**, and similarity decays smoothly instead
of collapsing: 0.502 at k=2, 0.278 at k=8, 0.135 at k=32 (d=2000).
A second finding, from the ring controller: re-observing the same key must
*replace* its binding, not add another vote. Piling votes on one pair makes
the majority represent the observation count rather than the set of keys, and
recall starts returning a neighbouring symbol. `record` therefore maintains
per-key state and the trace is `majority` over distinct keys only.
```python
mem = HolographicMemory(d=2000, alphabet=symbols)
for i in range(8):
mem.record("slot/%d" % i, symbols[i])
mem.recall("slot/3") # (symbol vector, cleanup similarity)
mem.confidence("slot/3") # presence signal; do not compare raw probes to the trace
```
## 12. Geometry, fractal and crystal shaders
`hdc/manifold.py` treats a filled voxel set as the manifold it is. Geodesic
distance is breadth-first **on the surface** (a full 5Β³ box: corner to corner
= 12, matching Manhattan exactly; disconnected components return `None`). The
Laplacian is `|Ξ£ neighbours βˆ’ kΒ·center|` in coded arithmetic with `k` the real
in-bounds coordination, so border voxels are not biased by phantom neighbours.
Signed curvature folds that to ridge/valley/flat and shading lifts ridges and
drops valleys by one exact shift. It is mean curvature up to a positive factor,
not Gaussian curvature, and a one-voxel-thick shell reports flat normals under
a central difference β€” both stated in the module.
`hdc/fractal.py` is integer-only. The Sierpinski triangle is the closed form
`x & y == 0`; the Barnsley fern runs the chaos game in 1/256 fixed point; fBm
is hashed value noise interpolated with integer lerps and combined by `max`
after `dim`-ing each octave. Measured octave peaks: **254, 127, 63, 31** β€” the
halving is a checkable property, not an adjective.
`hdc/crystal.py` exposes periodicity. Grains grow by multi-source geodesic BFS
from lattice sites (so grain walls follow the surface); facets are the six
dominant-gradient normals with ties resolved to the lowest axis; and
`periodicity()` recovers a banded volume's spacing by autocorrelating
per-plane occupancy β€” a 2-plane lattice measures spacing 2 with score > 0.5.
## 13. Entropic 4D shaders (`hdc/shader4d.py`, `hdc/entropy.py`)
The fourth axis is a *parameter* β€” time, a scan index β€” not a spatial axis. A
`Volume4D` is a stack of volumes; entropy along `u` says whether each voxel is
static, drifting or churning, measured with the same comparator that builds
every other histogram (entropy of a flat field is exactly 0 bits, two values
1.000 bits, eight uniform bins 3.000 bits). Entropic shading composes the 3D
shade with an integer attenuation derived from that entropy, so flat-in-time
regions fall away by up to 1/4 and churning ones keep their value.
## 14. Predictive state and the ring (`hdc/state.py`, `hdc/ring.py`)
A transition is a bound pair in a holographic memory per action; prediction is
unbind plus cleanup; planning is beam search scored by `sim(predicted, goal)`.
Surprise is `1 βˆ’ sim(predicted, actual)` β€” the temporally measured loss, and
the only signal that makes this an agent rather than a table.
`hdc/ring.py` closes the loop. A `HistoricLedger` is addressed **by count**, so
a lap-ago entry is recalled by content, not by an offset that shifts as the
ring wraps; eviction recycles detail into a per-phase summary that outlives it,
and the whole retained history is also one superposed `timeline_record`.
Each tick runs PAST β†’ PRESENT β†’ FUTURE β†’ PAST, and an energy budget pays for
lookahead (`horizon = 1 + 2Β·energy`, clipped to 3):
* periodic world, 12 ticks β€” surprise falls to **0.000** after the first lap,
energy saturates at 1.000, mode `stable`, horizon 3;
* drifting world (a new state every tick) β€” surprise stays at **1.00**, energy
falls to 0.000, mode `explore`, horizon 1.
That pair is the whole claim: the controller widens when it can predict and
pulls in when it cannot, using only measured similarity.
## 15. Text encoder (`hdc/text.py`)
A word is a bundle of *position-bound character n-grams*, so morphology is
similarity before any training: `walked`~`walking` **+0.320** against
`walked`~`banana` **+0.085**. Position binding keeps `listen` and `silent`
distinct despite identical n-gram multisets. A context is a role-bound record
of the last `span` tokens, which is exactly the key the predictive model wants.
Measured on a 25-token corpus: order-1 contexts give **67%** top-1 next-token
accuracy, order-2 **83%** (the remainder is genuine two-way ambiguity). This is
what a text-bound ring buys: contexts that are comparable, and a fallback to
the nearest known form for words never trained on.
```python
ring = TextRing(d=2000, span=2)
ring.fit(tokenize(corpus))
ring.predict(tokenize("the cat sat on the")) # ('mat', 0.30)
```
## 16. Threshold as topology (`hdc/topology.py`)
The threshold is not a constant to tune; sweep it and the codebook becomes a
nested family of graphs `G(Ο„) = { {u,v} : sim(u,v) > Ο„ }`. Component merges are
0-dimensional persistence, computed with a union-find over exact similarities.
Three consequences, each measured:
* **Self-guiding.** Otsu's criterion on the similarity histogram finds the
valley between members and crosstalk without a hand-set constant: Ο„=0.691
with 70% explained variance on a structured codebook, and on a flat one it
reports that the distribution is unimodal (no fake threshold).
* **Self-healing is a projection, not bit recovery.** Bundling the nearest
survivors returns a cell to its class prototype: 0.665 β†’ **0.701** for severe
damage on the shared-core codebook, but a lightly damaged 0.960 cell is pulled
*down* to 0.710. On independent symbols the graph is no denser than chance
(0.25 vs 0.25) and the routine reports `informative = False`, changing
nothing. Healing presupposes geometry and drift.
* **Self-scaling.** The measured member/non-member gap sets the dimension:
`d' = dΒ·(gap_target/gap_measured)Β²`. Gap 0.114 at d=2000 asks for 6 156.
```python
components(codebook, tau=0.1) # fragment labels at one threshold
otsu_threshold(similarities(codebook)) # (tau, explained variance)
heal(codebook, tau, flips=400) # projection repair, gated on real topology
```
## 17. Semantic vectors without libraries (`hdc/semantic.py`)
Β§15 ends with a caveat: the n-gram encoder is morphological, not semantic. The
recovery is a **static** embedding model, and the interesting part is that it
needs no runtime dependency at all.
`minishlab/potion-base-8M` (MIT) is a Model2Vec distillation of
`BAAI/bge-base-en-v1.5` into a token x 256-float **lookup matrix** plus a
WordPiece vocabulary. Static means: no attention, no layers, no activations β€”
a dict from token to vector. So the only obstacle is the file format, and a
`safetensors` file is an 8-byte header length, a JSON header and a raw f32
buffer. `struct` reads it. 29 528 tokens x 256 dims, 30 MB on disk:
```
hf download minishlab/potion-base-8M model.safetensors tokenizer.json config.json \
--local-dir models/potion-base-8M
python -m hdc semantic
```
Words become hypervectors by **sparse sign random projection** (SimHash):
`bit_j = 1 iff Ξ£_k w(j,k)Β·v_k > 0` with K=12 pseudorandom Β±1 weights derived
from a hash, so nothing is stored. Measured:
| pair | model cosine | `sim(hv)` | recovered |
|---|---|---|---|
| car~automobile | +0.779 | +0.557 | +0.768 |
| doctor~physician | +0.677 | +0.472 | +0.675 |
| big~large | +0.767 | +0.531 | +0.741 |
| car~banana | +0.044 | +0.015 | +0.024 |
| dog~tectonics | βˆ’0.065 | βˆ’0.080 | βˆ’0.125 |
**Projection fidelity: Pearson r = 0.998.** The attenuation is not noise, it is
the arcsine law for binarised projections β€” `sim = (2/Ο€)Β·arcsin(cos)` β€” and it
inverts, which is why the recovered column tracks the true cosine.
The projection plan depends only on `(d, sparsity, seed)`, not on the word, so
it is built once instead of re-hashed per token: **36.3 ms/word β†’ 2.28 ms/word**
at d=2000 (16Γ—), plan build 0.08 s, and fidelity unchanged β€” `car`~`automobile`
recovers +0.768 against a raw +0.779.
What it buys downstream, measured by fitting a `TextRing` on one wording and
scoring a paraphrase: **n-gram 0.50 β†’ semantic 0.67** at span 2 (one case is
0.50 β†’ 1.00). At span 1 the two encoders tie: with no shared context word the
key carries the substitution, but a shared word carries it for both, so
transfer is partial rather than free.
The enabling change is in `PredictiveModel.predict`: tables use exact keys
(similar keys would interfere in a majority trace), plus a **snap** step that
retrieves with the nearest known key vector when the exact lookup is weak.
Precision on the exact path, graded recall on the fallback.
Caveat worth keeping: with d=2000 and a 256-dim source, binarisation caps a
0.68 cosine at 0.47 similarity, so semantic margins are compressed. A wider
code or a thermometer encoding would recover resolution at the cost of bundle
capacity.
## 18. Model discovery (`hdc/hub.py`)
Β§17 assumes a model file exists. This layer decides *which* file, from the Hub,
with the standard library and a score whose every component is measured.
The installed `hf` CLI is 0.36.2 and exposes only
`auth/cache/download/jobs/repo/repo-files/upload/env/version` β€” there is no
`models` or `api` subcommand β€” so search calls the HTTPS `/api/models` endpoint
those commands wrap, while downloads still go through `hf download`. Two
endpoint behaviours cost real debugging, and both return an empty list instead
of an error: `author` combined with `filter` yields zero records, and
`filter=library:model2vec` is not a supported filter β€” the family has to be
enumerated with `author=minishlab`.
Shapes are read without downloading the weights. A `safetensors` file is an
8-byte little-endian header length, a JSON header, then the buffer, so two HTTP
Range requests return the dtypes, the shapes and the exact size: **scoring a
129 MB model downloads no model bytes.**
| component | weight | measured by |
|---|---|---|
| runtime_free | 0.35 | a single rank-2 tensor is a matrix the CPU can use; hundreds of tensors mean a graph engine |
| code | 0.30 | tokenizer coverage of a machine-code token list; whole tokens 0.7, stitched fragments 0.3 |
| size | 0.15 | 1.0 at ≀35 MB, 0.0 at 512 MB |
| dim | 0.10 | 1.0 at 256 dims β€” projection cost scales with d |
| license | 0.10 | permissive licences only |
Measured bakeoff, top of the pool (`code` is the coverage component after the
whole/fragment weighting, which is what separates the tables):
| repo | score | code | table | MB |
|---|---|---|---|---|
| potion-code-16M-v2 | 0.95 | 0.85 | 63 457 Γ— 256 | 32.5 |
| potion-science-32M | 0.93 | 0.85 | 63 457 Γ— 256 | 127.4 |
| potion-retrieval-32M | 0.85 | 0.72 | 63 091 Γ— 512 | 129.2 |
| potion-base-32M | 0.85 | 0.72 | 29 528 Γ— 512 | 129.2 |
| potion-base-8M | 0.85 | 0.50 | 29 528 Γ— 256 | 30.2 |
| M2V_base_output | 0.85 | 0.50 | 29 528 Γ— 256 | 30.2 |
| potion-base-4M | 0.80 | 0.50 | 29 528 Γ— 128 | 15.1 |
| potion-base-2M | 0.78 | 0.50 | 29 528 Γ— 64 | 7.6 |
The code table wins, which is the point of scoring coverage at all: the general
table is 30% smaller and would have tied on every other axis.
The transformer code models fail the only component that is a hard requirement:
`jinaai/jina-embeddings-v2-base-code` scores **0.46** at 321.7 MB and cannot
run here at all, because it needs a deep-learning runtime. Static tables win by
being data.
**Machine code in the same space.** `minishlab/potion-code-16M-v2` (MIT) β€” the
winner above β€” is one of those tables β€” 63 457 Γ— 256 β€” but stored as **F16**, which the Β§17 reader
would have misread as f32; `hdc.semantic` now decodes F32/F16/BF16 (bf16 by
shifting 16 bits into the high half of an f32). Measured whole-token coverage
of the 42-token machine-code list (this project's own mnemonics, x86 names,
registers, literal forms): **79% for the code model against 29%** for
potion-base-8M β€” everything tokenisable either way, nothing missing. The code
model's only fragment-only tokens are `ldi jnz rbx rcx rdx rdi 0b1011 $0x10
[rip+4]`.
Mnemonics do not cluster semantically (`mov`~`ldi` **+0.017**) but registers do
(`rax`~`rbx` **+0.413**). That is the right behaviour for a symbol space:
distinct opcodes, related registers.
```bash
python -m hdc hub # search, score, recommend β€” no weights downloaded
```
Scores are cached in `models/_hub/scores.json` keyed by `repo@sha`, so a repeat
run is instant and does not need the network.
## 19. Idle, learning, readahead (`hdc/autonomy.py`)
The assistant is reactive: it answers what it is asked. This layer adds the
part that runs when nobody is asking, and its three behaviours are measured.
**Learning.** Every turn leaves an intent. The sequence of intents is learned
into the same holographic transition table Β§14 uses. Idle ticks *replay* that
trace, and replay is training rather than decoration: repeated evidence raises
a majority trace's cleanup confidence, so a cycle that is noisy on the first lap
sharpens on the next. Measured on a 12-utterance session: correct predictions
scored 0.18–0.39 after one lap, cold wrong ones 0.00–0.06.
**Readahead.** From the last intent, chain the model `horizon` steps; each hop
carries its own similarity plus a running `path` product, so a three-step guess
cannot look as certain as a one-step one. Every prediction is graded when the
next utterance is classified β€” PAST trace β†’ PRESENT intent β†’ FUTURE lookahead β†’
graded back into PAST β€” and that grading is the accuracy number.
**Autonomy, bounded.** Only `SAFE_COMMANDS` may run unasked: read-only
inspections, never `run`, `step`, `asm`, `boot` or `clear`. A per-tick `budget`
caps how many, and a confidence floor gates each one. The floor was set from a
measured sweep rather than from taste β€” cleanup over a 21-symbol codebook has
chance level `1/√21 = 0.218`, and correct post-lap predictions scored 0.18–0.39:
| floor | turns covered (of 24) | precision |
|---|---|---|
| 0.25 | 14 | 1.00 |
| 0.30 | 9 | 1.00 |
| 0.45 | 0 | β€” |
0.45 was the first guess and made autonomy idle in name only; **0.25** is the
default. The worst case of a wrong prediction is wasted idle cycles, because the
command set is read-only.
Measured, 12 utterances at d=2000 with the semantic encoder: **80% next-intent
accuracy, first half 33% β†’ second half 100%**, 4 autonomous pre-executions,
4 answers served from the readahead cache. A hit trades idle time for response
time: `Assistant.chat` takes an `execute` hook, so an answer the machine already
computed costs a dict lookup instead of a kernel round trip.
```bash
python -m hdc autonomy # simulated session: accuracy, cache, savings
```
## 20. Filesystem in HDC (`hdc/fs.py`)
A filesystem has two jobs and they pull apart: finding things is a similarity
question, and giving text back exactly is not. So the split is explicit.
**Hyperdimensional:** every node's *record* is
`bind(role(name), name) + bind(role(kind), kind) + bind(role(parent), parent)`,
written into the machine's own `HDCStorage`. `search()` bundles the query's word
vectors and compares it to each node's own bundle β€” one `sim` per file, no index.
Measured on the seeded tree, `search("shader notes about edges")`:
| file | score |
|---|---|
| `notes-shader.txt` | **+0.492** |
| `README.txt` | +0.328 |
| `img/gradient.txt` | +0.281 |
| `src/hello.hdcasm` | +0.072 |
**Exact:** the text itself, addressed by path, exactly as `ring.py` keeps exact
ledger entries beside its holographic summaries. HDC cannot decode a bundle back
into text, and pretending otherwise would make "read" a lie. `HDCStorage.remove`
exists for this layer: deleting a file frees its cell, so `cells in use` stays
true (measured: 11 cells before a delete, 10 after).
`write()` creates intermediate directories, `remove()` refuses a non-empty
directory unless `recursive=True`, and `rename` rewrites a whole subtree's paths
and cell names together.
## 21. Graphics engine (`hdc/gfx.py`)
A `Surface` is a grid of cells whose brightness is a **coded hypervector**. That
choice is what makes the passes arithmetic rather than array copying: dimming is
`shift` (exact halving, no decode), blending is `max`/`min` through the fabric's
similarity comparator, and edge detection has two honest modes β€” `proxy` (one
`sim` per cell) and `exact` (`|a-b|` through the HDC adder).
Measured on an 80Γ—28 surface (2 240 cells) at d=2000:
| pass | ms | cells/s |
|---|---|---|
| `flat` (gradient only) | 24.0 | 93 171 |
| `background` (gradient+noise+scanline+vignette) | 87.6 | 25 563 |
| `metal` (noise+edges+dim) | 132.5 | 16 909 |
| `plasma` (noise+max_blend+scanline) | 295.7 | 7 576 |
| `edges(exact)` | 36 ms per **128** cells | β€” |
The exact edge pass is the adder's table lookups per cell, so a full screen costs
about a second: it ships as a demo pass, not as part of the desktop's frame.
One design note worth stating. Display needs one ramp lookup per cell, and
decoding a coded value costs 64 similarity probes β€” a full decode per frame would
cost seconds. So each surface keeps a plain integer **shadow** used only to pick
the character, while the coded cells stay the authority for every operation, and
`verify()` decodes a sample and compares. The cache cannot drift silently; the
tests assert `verify()` after every shader.
## 22. The webOS (`hdc/os.py`)
80Γ—30 character cells: a menu bar, tiled windows, a taskbar, and a compositor
that draws in z-order into a `Surface`. An unfocused window is darkened with
`dim_region` β€” an **exact** `shift` on the coded cells, not a multiply on a copy.
Programs are the layers this project already had, arranged for a desktop:
| app | what it is |
|---|---|
| Terminal | the kernel console, running real commands |
| Files | `hdc/fs.py` browser; Enter opens a directory or loads a file into the editor |
| Editor | appends lines to files in the same filesystem, round-tripped |
| IDE | `hdc/ide.py`: assemble, run, step, breakpoints |
| Shader Lab | `hdc/gfx.py` live in a window; space cycles shaders, `e` toggles edges |
| Monitor | memory buckets, filesystem size, kernel counters, readahead accuracy |
**Memory** is one store, not several: the desktop's filesystem writes into the
kernel's own `HDCStorage`, so `memory.buckets()` counts what is really resident β€”
measured `toolreg 16 + fs 10` on a fresh boot, with 8 166 cells free. `MemoryMap`
also offers `allocate(tag, n)` / `free(tag)` for apps that want to reserve cells.
Measured frame cost: **~108 ms** with three windows, ~138 ms with four, at 80Γ—30
and d=2000, dominated by the wallpaper shader (`background`). That is a desktop
redraw, not a GPU frame β€” see the note in Honest limits.
In the browser it is the **DESKTOP** tab: the frame arrives as 80 character rows
and is painted into the CRT, keys are captured globally (`F1`–`F6`, `Tab`,
arrows, `Ctrl+W`, `Ctrl+arrows` to move a window), and a click is converted from
pixels to a character cell and sent as coordinates to focus the window under it.
The endpoints are `GET /api/os` and `POST /api/os/{key,text,click,launch}`.
The CRT never asks you to scroll it. `fitCrt()` derives the font size from the
measured monospace column width **and** from the box height for the row count on
screen (25 for the kernel console, 30 for a desktop frame), and `#screen` has a
definite height rather than a `min`/`max` pair so that measurement is stable.
Fitting width alone let a 30-row frame run past a 76vh box, and `paint()` then
scrolled to the bottom β€” which hid the desktop's menu bar. Measured in headless
Chrome painting a 30-row frame:
| window | font | rows shown | box vs content | scrolls |
|---|---|---|---|---|
| 1920Γ—1080 | 21.8 px | 30 | 873 / 873 | no |
| 1440Γ—900 | 17.3 px | 30 | 693 / 693 | no |
| 1280Γ—800 | 14.8 px | 30 | 593 / 593 | no |
| 1024Γ—700 | 12.6 px | 30 | 505 / 505 | no |
| 800Γ—600 | 10.1 px | 30 | 405 / 405 | no |
| 640Γ—480 | 8.5 px | 30 | 344 / 344 | no |
`A-` still goes below the fit on request, and `overflow:auto` stays as the
fallback for that: the grid is drawn whole or scrolled deliberately, never
silently cut. The 8 px floor is why the box has a `min-height` β€” at 640Γ—480 the
page scrolls a line, but the terminal does not.
### The boot chain (`hdc/boot.py`)
Three phases, all rendered by the graphics engine, each doing real work:
| phase | what happens |
|---|---|
| **POST** | `hdc.x86` runs for real β€” IVT installed, boot sector read from LBA 0, `int 10h` teletype β€” and its text buffer is painted onto a phosphor surface with a scanline pass. The lines on screen are the lines the BIOS produced. |
| **Splash** | gradient + fractal noise + `max_blend` + scanline + vignette, with a title box and a progress bar drawn into coded cells |
| **AUTOEXEC** | a boot script executed against the live desktop: `MOUNT` really changes the filesystem's `cwd`, `FS.SEARCH` runs the hypervector search, `START` launches the windows, `AUTONOMY` turns on the readahead loop |
| desktop | the composited desktop the machine is left in |
Measured on a cold boot at d=2000: **12 frames, 611 ms of rendering, 2 220 ms
of playback**. The first splash attempt took 2 404 ms because it re-ran
`max_blend` per frame β€” a similarity comparator per cell, ~350 ms a frame. The
shader base is now rendered **once** and each animation frame copies it and
applies exact `shift`s for the sweep: the motion is still HDC arithmetic, and
the whole chain dropped to 611 ms.
AUTOEXEC's output is not decoration β€” this is what it prints, and it is what
happened:
```
SET DIM 2000
GFX.INIT 4 shaders, 2400 cells, verify ok
MOUNT /home/user 5 files, 5 dirs
FS.SEARCH /home/user/notes-shader.txt (+0.492)
START Terminal at 1,1 Monitor at 48,1 Files at 1,15
AUTONOMY readahead on, 21 intents
```
The browser replays the recorded frames at their own delays the first time the
desktop is opened (`GET /api/boot`), and **any key skips**. It is a recording of
work that already ran, so skipping loses nothing.
```bash
python -m hdc boot # the chain, frame by frame, then the timeline
python -m hdc os --boot # frames + report + the desktop it hands over
python -m hdc web --open # then open http://127.0.0.1:8765/#os
```
```bash
python -m hdc os # one frame, then memory and session report
python -m hdc os --tour # scripted session: open apps, run a command, screenshot
python -m hdc gfx # shader pass timings
python -m hdc gfx --demo # one of each shader, printed
```
## 23. Honest limits
Worth reading before quoting the headline.
* **What is genuinely hyperdimensional.** All state (program cells, registers,
stack, literals, addresses, tool records) is vectors; all retrieval
(fetch, cleanup, dispatch, recall) is similarity; binding, bundling,
permutation and the BitNet kernel are bit operations on those vectors.
* **What is conventional.** The control unit, the cleanup search itself, the
scalar ALU for `MUL`/`DIV`/bitwise/shift, and the ripple loop's sequencing
are ordinary Python. "Runs within HDC" describes where the *program and its
data live and how they are addressed, dispatched and arithmetically
combined* β€” not that a Python `for` loop is a hypervector.
* **The adder is depth-serial.** Four byte-slice stages (or 32 with the 1-bit
gate), each a storage lookup: correct and genuinely hyperdimensional, but
slower than one native add. Its interest is structural (the arithmetic is
data), not speed.
* **There is no rasteriser.** The shader layer does image *operators* β€”
dimming, edges, LUT normals, exact depth ordering for offered fragments β€” not
triangle setup, clipping or exact fragment coverage. Claiming otherwise
would be the failure mode this README argues against.
* **The shader's normal lookup is 70% exact, not 100%.** Nearest-entry
agreement on random directions is 68–70% with a mean cosine deficit of 0.03;
boundary queries and crosstalk account for the rest. The measured numbers are
in the module docstring rather than rounded up.
* **Integers are symbolic.** See the codec note above; do not interpret
similarity between two encoded integers as meaning.
* **Decode is not error-correcting.** A cell whose vector is damaged can
decode to a different opcode; the machine reports low confidence rather
than refusing to run. There is no ECC layer.
* **Search is linear.** `recall` and `resolve` scan all cells. This is honest
compute-in-memory (no index to maintain) and also O(nΒ·d/64) per query. It is
fast at thousands of cells, not millions.
* **The codec refuses d < 1500.** Measured 0 errors at d=1500/2000/3000, but
60/456 at d=1000 and 401/456 at d=500: below the floor the bit atoms
crosstalk and decode silently returns a neighbouring value, so `int_to_hv`
raises rather than corrupting thresholds downstream.
* **Holographic capacity is a decay, not a wall.** Recall stays exact to
k=32 at d=2000 with a 40-symbol codebook, but similarity has already fallen
from 0.50 to 0.14; past that, cleanup starts choosing neighbours. The hybrid
mode exists so that decay is a fallback path rather than a cliff.
* **Curvature is mean curvature, up to scale.** Sign and relative magnitude
are meaningful; the absolute number is not a curvature in 1/units, and a
one-voxel shell reads flat under a central difference.
* **No rasteriser, no 4D perspective.** The 3D/4D layers do operators on voxel
fields β€” dilate, bleed, shade, project, entropy β€” not triangle setup or
occlusion beyond max-projection.
* **The ring is first-order and finite.** States are exact symbols from a
codebook, the transition model is one action deep, and beam search is
exponential in horizon before the beam caps it. It is an inspectable world
model, not an LLM.
* **Energy is bookkeeping.** It is a budget over measured surprise, not a
physical quantity; the claim is only that horizon and energy move together.
* **Text encoding is morphological *by default*.** Synonyms are as far apart
as random words unless Β§17's model is downloaded; with it, semantics come
from a distilled lookup table, and paraphrase transfer is partial (0.50 β†’
0.67), not solved.
* **The semantic model is optional data, not a dependency.** No file, no
semantics β€” `hdc.semantic.load()` returns `None` and callers fall back to
n-grams. The 30 MB is not in the repo; the MIT terms and provenance are in
the module docstring.
* **"Self-healing" is denoising, not recovery.** See Β§16: repair projects a
damaged cell onto its cluster prototype, which helps drifted cells and
over-smooths healthy ones. There is no error-correcting code anywhere here.
* **Readahead predicts the next *command*, not the next *words*.** The intent
set is fixed at construction, the chain is first-order, and the confidence
figures are cleanup similarities over a 21-symbol codebook β€” not
probabilities. It predicts the shape of a session, nothing about what the
user will say.
* **Machine-code embeddings are symbolic, not semantic.** Whole-token coverage
is 79% for the code model, but mnemonics are distinct rather than grouped
(`mov`~`ldi` +0.017): embedding a listing gives a space where opcodes are
separable, not an understanding of what the program does.
* **The desktop is character cells, not pixels.** 80Γ—30 with one glyph per cell,
no antialiasing, no alpha blending, no triangles. The graphics engine shades
cells; it is not a rasteriser, and `edges(exact)` costs about a second per
screen because it is the adder, per cell.
* **A desktop frame costs ~108 ms**, dominated by the wallpaper shader. Drawing
happens on demand, so typing feels like a 1990s terminal rather than a GPU
desktop β€” which is what it is. The integer grid beside each `Surface` is a
display cache; `verify()` checks it against the decoded cells, and the tests
assert that after every shader.
* **One session, one desktop.** No users, no permissions, no locking, no swap.
The filesystem is in memory and `save()` persists node vectors, not text.
* **The boot is a recording.** Frames are rendered when the server starts and
replayed to the browser; AUTOEXEC has already run by then, so the desktop you
type into is the result, not a re-run. The splash animates by copying a
rendered base and applying exact shifts between frames, which is why it costs
milliseconds rather than re-running the noise pass per frame.
* **Topology here is 0-dimensional.** Components only β€” no loops, no higher
homology, no persistence barcodes.
* **Single process, single thread**, no garbage collection of cells, no
incremental persistence.
* **The MCP server holds one machine.** Tools are stateless with respect to
program execution; a `tools/call` does not run the loaded program.
---
## 24. Layout
```
hdc/
hypervec.py bind / bundle / permute / sim / integer codec
memory.py HDCStorage, content addressing, cleanup, .hdc persistence
isa.py opcode table, instruction encoding, compute-in-memory decode
adder.py hyperdimensional carry-chain full adder (ADD/SUB)
slicegate.py wide gates: k bits per stage, lazily materialised rows
shader.py image records, exact dim/edge/depth, LUT normals
shader3d.py volumes, morphology, normals, shaded projection
shader4d.py volume stacks over a fourth axis, entropy-weighted
entropy.py coded histograms and Shannon entropy
manifold.py geodesic field, Laplacian curvature, curvature shading
fractal.py dyadic IFS attractors, integer value-noise fBm
crystal.py lattice grains, facet maps, periodic banding
memory_modes.py photographic / holographic / hybrid memory
state.py predictive model, beam planner, surprise, agent loop
ring.py historic ledger + PAST/PRESENT/FUTURE controller
text.py character-ngram encoder for the ring
topology.py threshold graphs, Otsu guiding, repair, scaling rule
semantic.py static-embedding loader (safetensors via struct), SimHash
hub.py Hub search, Range header probe, suitability score
autonomy.py idle ticks, intent learning, graded readahead
fs.py filesystem: HDC-addressed nodes, exact contents, similarity search
gfx.py coded-cell surfaces, shader passes, display cache + verify
os.py webOS: memory map, window manager, apps, composited frames
boot.py boot chain: BIOS frame, shader splash, AUTOEXEC against the OS
assembler.py two-pass compiler + decode-back verification
interpreter.py Machine: fetch, decode, execute; HDC registers and stack
debugger.py breakpoints, stepping, trace, symbolic registers
net.py BitNet base: 1-bit / ternary kernels, PrototypeNet
agent.py tool records in HDC, similarity dispatch
mcp_server.py MCP JSON-RPC over stdio
ide.py native binary-logic IDE
demo.py end-to-end tour
__main__.py python -m hdc <command>
examples/ hello, logic, agent programs
models/ downloaded model files (ignored by git, fetched with hf):
potion-base-8M, potion-code-16M-v2
tests/ over 300 unit tests (python -m hdc test)
artifacts/ storage images written by the demo
```