Mr-DS-ML-85/SyFox-bucket / docs /ARCHITECTURE.md
Mr-DS-ML-85's picture
|
download
raw
24.3 kB
# SyFox Architecture — the SI substrate core
This document specifies exactly what happens between "state in" and "decision
out", and which mechanisms are ported from the Synthetic-Intelligence (SI)
research substrate. The rule that governs this codebase:
> **The core is SI physics only.** No transformer, no neural network, no
> pattern-matching classifier. Everything below is field dynamics over an
> explicit concept graph. Tools (JSON, CLI, HTTP, calibration math) live
> outside the core and never rewrite it.
## 1. Concepts and acoustic mass
Every lowercase folded token interned by the engine becomes a **concept node**
(`core/si_substrate.hpp`, `Substrate::intern`). Nodes carry:
* **mass** — a raw occurrence count: `intern()` adds exactly `1.0` each time
the concept appears in any lesson. It is **linear, not log-compressed**.
*Lineage:* SI "acoustic mass" — heavier units respond less to the same
stimulus. The sub-linear behavior lives at the **use sites**, not in the
counter: injection deposits `energy / sqrt(mass)` and Hebbian bind strength
scales by `1/sqrt(mass)`, so frequent concepts ("the", "please") cannot
flood the field.
* **energy** — transient; injected and dissipated per decision.
(An earlier revision also carried a `salience` field. Audit found it was
write-only — incremented at injection, read nowhere; gating selects sources
by energy — so it was removed. Dead state in the core is doc debt.)
Token folding (`fold`) is deterministic string normalization (plural / `-ed` /
`-ing` / trailing `e`), applied identically to states, options, and
instructions. It is I/O hygiene, not linguistics and not pattern matching.
## 2. Lanes (Hebbian bindings)
Lanes are weighted edges between concepts, stored sparsely per node with a
per-node lane cap (weakest lane evicted, bounded memory).
* **Bind** (`hebbian_lesson`): for a labelled lesson, every active state
concept binds to every outcome concept with
`η · (1/√mass_a + 1/√mass_b) · scale`. Adjacent state tokens also bind
(the co-occurrence fabric that lets energy diffuse inside a phrase).
*Lineage:* SI Hebbian lane bindings in `physics.hpp`.
* **Weaken** (`weaken`): counter-evidence subtracts lane weight; a lane whose
weight reaches zero dissolves. Used by Noul lessons labelled false.
* **Decay** (`lane_decay`): every lesson multiplies all lanes by 0.995 —
unused routes fade.
Learning is substrate rewiring. There are no fitted weights anywhere in the
core; the calibration tool (below) is external and read-only.
## 3. Injection
A state is tokenized; every token that exists in the vocabulary deposits
`inject_energy / sqrt(mass)` on its node. Unknown tokens stay dark — this is
the first half of the honest-silence contract (a state of nothing but unknown
words settles zero energy and defers).
## 4. Settle (the System One pass)
`settle()` runs up to `k_settle` (default 8) dissipative diffusion passes:
1. Sources are gated by **source cap** (`source_cap`, 24 by default): the
top-K nodes propagate each pass — ranked **by energy** by default, or by
**salience** when `salience_gating` is on. Gated nodes keep their energy
(they remain readout-visible); the cap gates flow, it does not
annihilate. (An earlier version pruned energy outright and destroyed 97%
of the field — the gating form is the correct physics and the tests pin
it.)
**SI salience lineage, restored after audit.** The SI substrate runs a
cavity salience integrator, `s = tanh(s·decay + gain·|velocity|)`
(`physics.hpp`, `salience_gain = 0.05`, `salience_decay = 0.995`), and
its TSDA layer samples the live working cap from
`[working_cap−4, working_cap]` = [5,9] (Miller 7 ± 2). SyFox ports both:
the integrator runs **every pass** on the motion proxy `|ΔE|` (a node at
rest is an exact fixed point, mirroring SI's sparsity guard), and
`miller_window = true` samples the live cap from
`[source_cap−4, source_cap]` each decision. One deliberate adaptation:
SI draws the sample from a seeded mt19937 stream; SyFox derives it from
the decision's state hash, so the same state always settles to the same
field bit-for-bit (the persistence contract outranks stochasticity).
`inject()` spikes salience to 1.0 on touch (TSDA spike semantics).
*Benchmark (10 in-domain decisions, 3 domains).* default energy gating
10/10 with calibrated-shaped confidences; salience ranking at the wide
cap 4/10 (motion history is a poor selector when the cap rarely trims);
salience + literal [5,9] window 10/10 argmax but probabilities saturate
at 1.0 (calibration collapse — needs the v0.2 eval harness before
adoption); Miller window at wide cap 9/10. Verdict: default stays
energy-gated at a fixed 24; the SI-faithful modes ship behind config
flags until a held-out eval says otherwise.
2. Each source retains `(1 − diffusion)` of its decayed energy and flows
`diffusion` out along its lanes, split proportionally to lane weight:
`next[b] += e·decay·diffusion·(w_ab / Σw_a)`.
3. Total energy is non-increasing (dissipative); the pass loop stops early
when the field relaxes (`total_before − total_after < eps`).
One settle per decision — all questions read the same settled field. Adding
questions never re-settles the substrate: flat marginal latency by structure,
not by scheduling.
**Derivation, not retrieval.** Because diffusion is multi-hop, a state that
was never in any lesson still settles to a computed answer: energy crosses
the co-occurrence fabric into outcome lanes laid down by *other* lessons.
Probe (two lessons, `alpha beta`→x and `beta gamma`→y): the untaught state
`alpha` alone settles with y at 0.494 — lesson 2 never mentioned alpha; the
two-hop path alpha→beta→y carried the evidence. The boundary is the taught
vocabulary and lane fabric: concepts never interned have no node, unknown
words settle to zero energy and defer (§6). This is the substrate-level
kernel of the SI stack's explicit derivation layers (meta-learner
macro-rules, dreamer emergent resonances, analogical mapping) — not yet
ported; see ROADMAP v0.3.
## 5. Readout (resonance sweep, two lanes)
Probes read the settled field at their own concepts: direct node energy plus a
hop term through lanes. Two readout lanes exist because two question shapes
need two different questions of the field:
* **Specific** (`choice`, `score`): mean lane support —
`hop · Σ w·e_n / Σ w`. Rewards precise coupling; a node bound diffusely to
half the vocabulary cannot out-read a node bound exactly to this state.
Options are then normalized by softmax at the fitted temperature.
* **Support** (`noul`): active lane mass — `hop · Σ w·e_n` (unnormalized).
A statement may be corroborated broadly; what matters is how much
lane-connected energy the field offers it. The ratio to total field energy
is the statement's **support**.
Both are field measurements. No weights are fitted at readout; no classes
exist in the core.
## 6. Honest silence
If settled field energy is below `silence_floor` (0.05):
* empty state → `deferred: true, reason: "empty_state"`
* all-unknown vocabulary → `reason: "unknown_vocabulary"`
* settled but dark → `reason: "honest_silence"`
The engine refuses to guess from nothing. *Lineage:* the SI interface's
honest-silence dispatch — answers only when the readout actually collapsed.
## 7. Calibration (a tool, not core)
`fit_calibration` collects readout energies on labelled rows and fits, purely
outside the core:
* **temperature** per choice/score — golden-section search on NLL of
`softmax(E_correct/T, E_wrong/T)`.
* **Platt scaling** for noul — logistic regression `sigmoid(a·support + b)`
on supports, gradient descent to convergence (separable data needs a steep
slope; the fit runs 4000 iterations with decaying lr).
Calibration changes how energies are *reported* as probabilities, never how
they are produced. Confidence for choice/score is the normalized-entropy
margin of the distribution; for noul it is `|p − 0.5|·2`.
## 8. Persistence
`save_model` writes `substrate.bin` (vocabulary + masses + lanes, binary),
`calibration.json`, and `meta.json`. Loading is exact — the same state settles
to the same readout bit-for-bit (pinned by tests).
## 9. Provenance
| Mechanism | Origin |
|---|---|
| Energy injection + field settle | SI `physics.hpp` substrate dynamics |
| Source capping (top-K, energy or salience) | SI working-set cap + TSDA live_cap ([cap−4, cap], Miller 7 ± 2) |
| Salience integrator tanh(s·decay + gain·motion) | SI `physics.hpp` cavity salience (verified live there) |
| Hebbian lane bindings + decay | SI lane learning |
| Honest silence | SI dispatch contract |
| Typed Choice/Score/Noul API | interface shape inspired by TypeSafe Jev |
| Temperature/Platt calibration | classical statistics (tool layer) |
| Derivation layer (§10) | SyFox-native design after studying SI meta_learner / dreamer / analogy as mechanism reference — no code copied; weight algebra on lanes instead of typed relation rules |
| No-regression gate (§11) | SyFox-native: transactional derive + bit-exact fabric snapshot/restore; policy refined by a full strength-scan (taught-row integrity + no manufactured certainty) |
| Jev-parity bench (§12) | SyFox-native: axis lineage from TypeSafe's published Jev material + community classification benchmarks; all numbers produced by this repo's binary |
| Recall (§13) | Hopfield-style associative memory reinterpreted as settled-field cosine — SI-physics-native, zero symbol-space similarity |
## 10. Derivation layer (offline, explicit)
SyFox's lanes so far carried only lived experience (`learn`). The derivation
layer lets the field grow knowledge **derived from** its own fabric — the
substrate-level kernel of what the upstream SI stack does with explicit rule
machinery (meta-learner macro-rules, dreamer emergent resonances, analogical
mapping). The primitives differ by constitution: SI composes typed relation
triples over a fact graph; SyFox composes **weighted lane paths** — weight
algebra and field dynamics, no classifiers. All engines are OFFLINE, explicit
CLI steps; `decide()` stays read-only and byte-identical unless the operator
derives on a model. Provenance: derived lanes carry a **generation counter**
(experienced / human-promoted = 0, each composition step +1, cap 3), persisted
as a versioned tail in `substrate.bin`; v1 files load clean (generation 0).
**1. Lane induction, compose mode** (`syfox derive --model DIR`): a two-hop
path A→B→C is evidence for A↔C, with three guards that each came out of a
real benchmark failure:
* *acoustic-mass damping* — path evidence is damped by `1/√mass(B)`, the same
law `settle()` applies to energy through B; without it, hub tokens ("the",
"command") fabricated corroboration between unrelated concepts and two demo
argmaxes flipped;
* *top-K corroboration* (K=3) — a dense fabric offers hundreds of two-hop
paths; treating them as independent evidence saturated the field (noul
collapsed to 0.999 everywhere);
* *ghost-guarded verifier* — every derived lane is re-checked against the
LIVE fabric (snapshot ghosts of already-dissolved lanes cannot justify
anything), over-claims are healed down to their justification, unsupported
ones dissolve. Composition is raise-only: observed lanes are never weakened.
**2. Lane induction, harvest mode** (`syfox derive --model DIR --examples
FILE.jsonl`): static algebra guesses; the settled field knows. Replay states
through inject→settle and record which concept pairs genuinely **co-activate**.
Dissipation does the filtering — cross-scenario energy decays before it
registers. Observed lanes are untouchable; gen-1 lanes must be re-nominated by
every run or they dissolve (justification is recomputable, hence deterministic).
Labels are not needed: the field learns structure from unlabelled exposure.
**3. Dreamer** (`syfox dream --model DIR [--steps N] [--seed S]`): inject
small random energy patterns, settle, and watch for undriven nodes that lit
up **without any direct lane to the driven set** — reached only through the
topology's own interference. Candidates go to `model-*/mutations.jsonl`. The
substrate is never modified by dreaming; a human sets `validated:true` on a
line and `syfox promote` applies it as a premise-grade lane. Same seed → same
dream, bit for bit.
**4. Analogy** (`syfox analogs --model DIR --concept WORD`, read-only): a
concept's signature is its lane neighbourhood (who it touches, both
directions); structural isomorphism = signature Jaccard; novelty potential
follows the L9 formulation `Phi = I·exp(k·d)`. High Phi = structurally aligned
AND fabric-distant: a transfer *hypothesis* for the human, never auto-applied.
*Benchmark (v0.2 → v2.1, honest).* The v0.2 verdict was measured on
resubstitution (eval rows = training rows): compose sharpened tickets demos
(technical conf 0.251→0.454) but flipped game/guard rows; harvest lifted
billing 0.604→0.912 but flipped a close call; no strength avoided taught-row
flips on the mixed-label fabrics. v2.1 replaces the measurement basis: the
gate now replays the HELD-OUT split (generalization safety, not memorized
behavior) and the harness decides per model — game ships derived (253 lanes,
0 held-out flips, accuracy held 0.667→0.667), tickets (9 flips) and guard
(1 flip) ship un-derived with the reason logged. Same enforcement philosophy,
honest measurement basis.
## 11. The no-regression gate (transactional derivation)
Derivation writes into the fabric; the gate (`core/gate.hpp`, CLI
`syfox derive --gate FILE.jsonl`) makes that write safe by construction:
1. replay every labelled row + generated close-call probes (cross-domain
state mixtures, deterministic generator shared with `bench`) → baseline
decision signatures;
2. snapshot the fabric **bit-exactly** — `snapshot_fabric()` copies each
node's lane vector in insertion order plus the provenance map, because
`settle()` sums over lanes in vector order (a weight set is not the
state);
3. run the derivation (conservative presets: harvest co_floor 0.15/gain
0.35/budgets 8·512, compose gain 0.25/top-2/budget 4);
4. replay the same probe set → compare;
5. **policy**: (a) any argmax flip on a TAUGHT row reverts — labelled
knowledge must survive derivation; (b) any RISE in mean confidence on the
MIXED probes reverts — close-call states are ambiguous by construction,
their argmax may re-resolve, but manufactured certainty about them is the
exact failure the gate exists to kill;
6. revert = `restore_fabric(snapshot)` → the fabric is bit-identical and
`decide()` output is byte-identical (pinned by test); commit = the caller
saves the model.
Strength-scan evidence (scripts/syfox_gate_scan.cpp, all strengths from
default down to 32 lanes): tickets reverts everywhere (8–18 taught flips —
its seed data mixes refund/invoice labels across billing and sales), guard
reverts at every useful strength, game commits at the conservative strength.
Per-model outcome: `model-game` ships derived (668 lanes, 0 taught flips,
mixed mean confidence 0.975→0.886 — the fabric got MORE honest about close
calls; demo argmaxes 3/3 unchanged; `make models` reproduces). `model-tickets`
and `model-guard` ship un-derived because the gate proves it, not because a
human remembered to check.
## 12. The Jev-parity benchmark (`syfox bench`)
`core/bench.hpp` measures the axes the System One model class is judged on
(axis lineage verified against TypeSafe's published Jev material and the
community classification benchmarks, Sept 2026): decision accuracy, latency
percentiles, calibration (ECE, 10 bins, + conf-when-correct vs
conf-when-wrong gap), honesty (OOD probes built from deterministic nonsense
vocabulary checked against the model's own — untaught tokens must defer),
guardrail hold precision/recall (noul confusion at the 0.5 threshold),
determinism (full replay compared byte-wise), and close-call margin
distribution (mixed cross-domain states; the same generator the gate
replays). Accuracy/calibration/guardrail aggregate over LABELLED rows only;
unlabelled mixed probes never pollute accuracy denominators (a real bug the
e2e suite caught). Every report carries a `jev_reference` block with the
PUBLISHED figures (67.8% workflow accuracy, 70–500 ms, pi-warden 88% hold
precision) for side-by-side reading — axes shared, numbers not compared
against toy-data resubstitution.
Seed-model results, v2.1 methodology — HELD-OUT split (70/30, stratified,
deterministic; `make bench-heldout`), post-gate derivation state. The v0.2
table that stood here measured in-domain resubstitution (eval rows = training
rows) and is retired; in-sample numbers remain available via `make bench` as
a regression contrast only.
| model | held-out choice acc (n) | score acc (n) | choice ECE | OOD defer | p50 / p95 |
|---|---|---|---|---|---|
| tickets | 1.000 (6) | 0.667 (6) | 0.012 | 1.00 | 64 / 80 µs |
| game (derived, gate PASS) | 0.667 (3) | — | 0.329 | 1.00 | 29 / 40 µs |
| guard | 0.750 (4) | — | 0.200 | 1.00 | 34 / 45 µs |
Reading: the held-out gap against in-sample (tickets 1.000 vs 1.000; guard
0.750 vs 0.778) is now measured instead of assumed. Calibration is fitted on
held-out rows only (2–3 scalars; multi-class NLL objective, 1-bit ECE
adoption guard) — tickets reaches ECE 0.012; game/guard are held above the
0.1 bar by a single held-out error each, which is arithmetic (1 wrong of
3–4 emitted forces ECE ≥ ~0.2), not a tuning failure. The headline metric is
the coverage-vs-accuracy curve (`make coverage`): tickets trades 100%
coverage @ 83.3% accuracy for 50% coverage @ 100% — the confident subset is
trustworthy, measurably. Paraphrase augmentation was implemented, measured on
the held-out split, and REJECTED by default (intern mass is linear, injection
damps 1/√mass: re-teaching near-duplicates re-weights the field toward the
training surface forms; tickets 1.000 → 0.500). The derivation gate replays
held-out rows and decides per model: game ships derived, tickets/guard ship
un-derived. Honesty stays structural (1.00 OOD defer everywhere); latency is
three to four orders of magnitude under the Jev band with the same one-pass
shape.
## 13. Recall (associative retrieval in energy space)
`core/recall.hpp`, CLI `syfox recall --state '...' --memories FILE.jsonl`.
Content-addressable memory the only way the constitution allows: settle the
query into an energy fingerprint (one float per node, L2-normalized), settle
each stored memory the same way, rank by cosine of the two settled fields.
No token comparison, no string distance, no n-gram overlap, no embedding
table, no transformer — similarity is measured in the substrate's own state
space, so two states resonate exactly as far as the field routes them
together. Unknown vocabulary resonates with nothing (empty hits — honest
silence, since `inject` skips unknown tokens). Read-only, bit-deterministic;
memory stores accept `{"state","label"}` rows or the training schema
(label = first label value). On the derived game model the zombie query
recalls the flee lessons at 0.9696–0.9421 with fight lessons at ≤0.856.
## 14. v3 — the hidden-test firewall and the evidence ledger
### 14.1 Firewall (core/firewall.hpp)
The 70/15/15 train/cal/hidden contract is enforced where a typo cannot
bypass it: the CLI consults the file's split role (decided by its filename
suffix — `_train`, `_cal`, `_hidden`, `_heldout`, anything else is
unclassified and stays readable, because worksheets and paraphrase files
must remain learnable) before opening anything.
| role | suffix | learn | calibrate | derive --gate | bench |
|-----------|-------------------|-------|-----------|---------------|-------|
| train | `_train.jsonl` | YES | yes | yes | yes |
| heldout | `_heldout.jsonl` | yes | YES | yes | yes |
| cal | `_cal.jsonl` | no | YES | yes | yes |
| hidden | `_hidden.jsonl` | NO | NO | NO | YES |
Rationale: calibration files exist to fit 2–3 scalars (temperature/Platt)
on rows the fabric never learned from. Hidden files exist to be SCORED,
once. A hidden row that leaks into any build path is not a mistake — it is
a different (worthless) number. `make firewall-check` runs the four
refusals against the real binary.
### 14.2 Evidence ledger (v3, Milestone 3)
Every lane key carries a `LaneEvidence` record: `support_events`,
`counter_events` (anti-Hebbian dissolutions), `generation` (derivation
provenance), `first_seq` / `last_seq` (teach-event window), and the
engine-set context tag (which substrate/model owned the event). The ledger
persists as an optional tail on `substrate.bin`; v2 files load unchanged
(the tail is detected at EOF) and v3 files load on v2 readers that stop at
the v2 end.
`decide --evidence` projects the ledger for the lanes the decision
actually used — a READ-ONLY view. The field is driven by lane weights;
the ledger never feeds back into physics. It exists so a decision can be
audited after the fact: which lanes, how strong, taught when, by which
corpus, disputed how many times.
### 14.3 Contradictions
Teaching the same (state, question) with a different outcome is a
contradiction: it increments the learn report's `contradictions` counter,
writes a contradiction record (old outcome, new outcome, both seq numbers,
state hash, context), and marks the affected lanes with a counter event —
the anti-Hebbian term weakens the disputed binding exactly as the physics
specifies. There is no last-write-wins: the field keeps the binding its
weights support, and the dispute stays on the record (`contested: true`
in the decide evidence view). Silent override is a design error, and the
adversarial suite (§15 of the README; `make adv-big`) tests for it with a
control arm that separates contradiction-specific damage from the generic
cost of re-teaching.
## 15. v3 — multicore settle and the GPU gate (M5)
### 15.1 CSR mirror
The settle hot path reads a compressed-sparse-row mirror of the out-lane
fabric: `csr_off_` (n+1 offsets), `csr_dst_`, `csr_w_` — contiguous
buffers, rebuilt lazily when the fabric mutates. The build copies each
source's lane vector verbatim, so per-source iteration order is exactly
the map order the sequential settle used: every float sum keeps its exact
operand order and results are bit-identical. This is the layout a GPU
port generalizes.
### 15.2 Deterministic OpenMP settle
`make omp` builds the same sources with `-fopenmp`. The diffusion pass
partitions sources with a static schedule (contiguous ascending chunks);
each thread scatters into its own buffer; the buffers are combined in
ascending thread order; therefore every accumulator sees contributions in
ascending SOURCE order — the sequential order. Bit-identity is a gate,
not a hope: `make omp-identity` fails the build if sequential and
`--threads 2` reports differ in any block (routing, calibration, honesty,
guardrail). Measured honestly: below ~10k nodes on a 2-core sandbox the
deterministic parallel settle is slower than sequential (thread
coordination exceeds the gain); it pays on denser fabrics.
### 15.3 Batched throughput and the GPU gate
`bench --throughput N` measures decisions/sec with N worker threads, each
owning a PRIVATE engine copy (decisions mutate the field, so workers never
share a substrate). It is a performance axis, not an accuracy headline —
a non-zero checksum proves the work happened. Measured on the 3,518-node
tickets fabric: 2,390 → 3,714 decisions/sec at N=2 on 2 cores.
The GPU port gate is MEASURED, not assumed: `make density` reports per-
fabric density and mean out-degree (tickets 0.0067 @ degree 23.5, game
0.0458, guard 0.0485). A GPU port happens when dense substrates make the
transfer costs pay for themselves — until then it stays design, and the
SoA/CSR buffers above are the only GPU-specific preparation the core
carries. No neural network enters the core at any point; the parallelism
is the SAME physics, executed on more cores.

Xet Storage Details

Size:
24.3 kB
·
Xet hash:
0afe9ff7988e72f221344a82d3ebe19371f96cd510b2d00fce77b8a4d7dfa22f

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.