TopoVoid-0.2B-Orthogonal-Preview
- Organization: Topovoid (topovoid.com)
- Maintainers: tirpitzia, 0kaguya
- License: Apache-2.0
1. Overview
This repository contains the 0.2B research preview (v0.1) of the TopoVoid architecture. It is a 212M-parameter bilingual causal language model pretrained from scratch with explicit middle-layer concept bottlenecks, paired with a closed-form solver that writes new facts into Attention and MLP weights inside the orthogonal null space of those concept layers.
How the closed-form write works (closed_form_write.py):
- Trigger contexts: Each fact (
subject, declarativeprompt,target, optionalquestion) is expanded into declarative and QA prompt variants. Multi-token targets are unrolled into sequential token continuation steps. - Pullback target ($R$): For each edited layer ($\ell \in {8, 9, 10, 11}$, 1-based), the model runs forward to layer $\ell$ and takes three normalized autograd steps on the hidden activation to find a residual $R$ that increases the target token probability. Autograd is used to compute this activation target before the linear solve; no optimizer step or gradient descent updates the model weights.
- Null-space projection ($P_\perp$): The residual is projected through
$$P_\perp = I_{1024} - Q_{\mathrm{cbm}} Q_{\mathrm{cbm}}^T, \qquad P_\perp Q_{\mathrm{cbm}} = 0$$
where $Q_{\mathrm{cbm}} \in \mathbb{R}^{1024 \times 176}$ is the orthonormal basis spanning the read and write matrices of the L7
MidCBM, L7SubspaceTopKCBM, and L12Highway CBM. Of the1024residual dimensions,176(17.2%) are protected and848(82.8%) form the writable null space ($\max |P_\perp Q_{\mathrm{cbm}}| = 1.47 \cdot 10^{-7}$ infp32). - Closed-form linear solve ($V$): Trigger keys, natural-text covariance (
eval_c4,eval_zhinvocab_map_49152_chatml.npz), and negative/continuation suppression keys are combined in a regularized linear system to solve for $V$, yielding low-rank weight increments $\Delta W = P_\perp R V$ on the Attention output projection and MLP down-projection. - Mounting and unmounting: During multi-layer compilation, earlier layers' increments are applied before solving later layers. At inference, a compiled module is added directly to the base weights (
W + ΞW), adding zero prompt tokens.
Architecture
| Item | Value |
|---|---|
| Parameters | 212,011,264 (tied input and output embeddings) |
Layers, d_model, d_ff |
16, 1024, 2432 (SwiGLU) |
| Attention | 16 query heads, 4 KV heads (GQA), per-head QK-RMSNorm |
| Vocabulary | 49,152 active tokens, pruned from 151,936 (~82M unused embedding parameters removed) |
| Positional encoding | RoPE (theta = 5e5, 4,096-position table) |
L7 MidCBM |
24 dimensions (35:1 concept SNR design) |
L7 SubspaceTopKCBM |
40-d subspace, 256 dictionary atoms, Top-8 |
L12 Highway CBM |
24 dimensions |
Files in This Repository
| File | Description |
|---|---|
model.safetensors |
Pretrained fp32 weights (809 MB); head.weight is tied to emb.weight on load |
config.json, tokenizer.json, tokenizer_config.json |
Model config and Qwen-based fast tokenizer |
vocab_map_49152_chatml.npz |
Active-vocabulary index map (orig_to_compact, compact_to_orig) and 64 Γ 256 calibration token rows (eval_c4, eval_zh) |
modeling_topovoid.py |
PyTorch model definition and load_topovoid helper |
closed_form_write.py |
Closed-form null-space weight writer ($\Delta W = P_\perp R V$) |
demo_write.py |
Minimal verification script (subspace split, closed-form write, cosine retention, and before/after generation) |
2. Quick Start & Open-Dataset Benchmark
Tested with torch 2.8.0, transformers 4.57.1, safetensors 0.8.0, and numpy on one GPU.
from modeling_topovoid import load_topovoid
from closed_form_write import concept_null_projector
model, tok, o2c, c2o = load_topovoid('.')
P_perp, Q_cbm = concept_null_projector(model)
Run the minimal verification and closed-form write demo (demo_write.py), or pass --counterfact to evaluate batch compilation on counterfact.json:
python demo_write.py
python demo_write.py --counterfact counterfact.json --n 12
Summary of Open-Dataset Results
The three evaluation tracks below test distinct mechanismsβdirect weight writing, frozen-feature linear readout, and external concept-index retrievalβand should not be conflated:
| Track & Dataset | Measured Result | Conditions |
|---|---|---|
Direct weight write (CounterFact, 100 individual edits) |
Rewrite first-token Top-1: 1.00% β 100.00%; median compile time 1.37 s (0.093 MB factor file) |
Reset to base weights before each edit. Paraphrase first-token Top-1 is 35.00%; neighborhood bleed is 44.90%. First-token metric, not full-string accuracy. |
Frozen representation readout (AG News, 10 labels/class) |
Full L7 (1024-d): 75.37%; CBM (320-d): 67.32%; Word+Char TF-IDF SVM: 56.70% |
40 total task labels (10 per class), pre-fixed $\lambda=0.1, C=1$, 0 extra validation labels, 5 sampling seeds. L7 leads SVM by +18.67 pp (95% paired CI [+15.75, +21.60] pp, Holm-corrected $p = 0.0002975$). |
External MidCBM index (ARC, 399 new question paraphrases) |
28.57% β 72.93% (+44.36 pp, 95% CI [+39.36, +49.38] pp) |
Base weights unchanged (external concept index + zero-parameter copy head). The same copy head paired with TF-IDF retrieval scores 81.20%. |
Example raw output on a CounterFact edit (target Toronto, greedy decoding, max 16 new tokens):
Prompt : Sunrise Party is headquartered in
Before write : the heart of the city. The club is a 100% free
After write : Toronto, Canada. The company is based in Toronto, Canada, and is a
2.1 CounterFact (1-Hop Weight Write)
First edit layer: Layer 8 (1-based). CounterFact metrics below use first-target-token proxies of ES / PS / NS (official ROME scores full target sequences):
- ES: $P(\mathrm{new}) > P(\mathrm{true})$ on the rewrite prompt.
- PS: $P(\mathrm{new}) > P(\mathrm{true})$ on paraphrase prompts.
- NS: $P(\mathrm{true}) > P(\mathrm{new})$ on neighborhood prompts (higher means less bleed into related subjects).
- Top-1: First-token top-1 hit rate of the new target on rewrite, paraphrase, and neighborhood prompts.
| Facts in one module | Rewrite top-1 (base β edited) | ES | PS | Paraphrase top-1 | NS | Neighbor top-1 on new target | C4 PPL ratio | Compile seconds | Factor MB |
|---|---|---|---|---|---|---|---|---|---|
| 1 (Γ100 median) | 1.00% β 100.00% |
100.00% |
75.50% |
35.00% |
19.70% |
44.90% |
1.002 |
1.37 |
0.093 |
| 12 | 0.00% β 100.00% |
100.00% |
91.67% |
41.67% |
13.33% |
56.67% |
1.015 |
2.80 |
1.06 |
| 40 | 0.00% β 100.00% |
100.00% |
67.50% |
20.00% |
31.25% |
27.50% |
1.043 |
6.53 |
3.53 |
| 160 | 0.63% β 95.63% |
100.00% |
59.06% |
21.25% |
41.69% |
23.31% |
1.115 |
22.24 |
14.45 |
| 640 | 0.31% β 78.28% |
94.38% |
54.69% |
16.95% |
46.77% |
17.87% |
1.148 |
48.55 |
57.51 |
2.2 Stacking Modules
Each module contains 40 CounterFact facts compiled independently from the same base checkpoint and added linearly (W + Ξ£ ΞW_i):
| Modules stacked | Top-1 alone | Top-1 stacked | NS alone | NS stacked | C4 PPL ratio | MidCBM cosine | Sparse-atom cosine | Highway cosine (mean, min) |
|---|---|---|---|---|---|---|---|---|
| 2 (80 facts) | 98.75% |
96.25% |
30.25% |
31.62% |
1.112 |
1.0 |
1.0 |
0.997886, 0.966447 |
| 4 (160 facts) | 98.75% |
55.62% |
32.25% |
37.19% |
1.406 |
1.0 |
1.0 |
0.995213, 0.940090 |
| 8 (320 facts) | 97.81% |
34.69% |
32.41% |
40.56% |
3.096 |
1.0 |
1.0 |
0.989003, 0.887467 |
Note on multi-module decay and the 1B compiler upgrade: In this 0.2B preview repository, closed_form_write.py only projects each module individually against the protected concept basis $Q_{\mathrm{cbm}}$ ($P_\perp = I - Q_{\mathrm{cbm}} Q_{\mathrm{cbm}}^T$) and does not include a cross-module Gram-Schmidt joint orthogonalization operator across different modules' right-hand key subspaces. As a result, 2 stacked modules (80 facts) retain 96.25% Top-1 (0.997886 Highway cosine), whereas summing 4 or 8 independently solved modules causes key collisions inside the null space (55.62% and 34.69%). Cross-module Gram-Schmidt joint compilation is implemented in our internal 1B commercial compiler (see Section 3).
2.3 MQuAKE-CF-3k (2-Hop)
Evaluated on 150 sampled cases (149 allow the unrelated-subject control). These numbers do not establish multi-hop reasoning on the base checkpoint:
| Measure | Value |
|---|---|
| Individual hop edits succeeded | 227 / 228 |
| Edited answer on multi-hop questions (before β after) | 13 / 150 β 117 / 150 |
| Edited answer on matched treated questions | 116 / 149 |
| Same answer on unrelated-subject control (before β after) | 12 / 149 β 118 / 149 |
| Cases where base knew all hops before edit | 6 / 150 |
2.4 Text Classification (AG News & DBpedia-14)
A separate ridge classifier head is fitted on frozen representations without changing the language model weights or generation path. Full L7 (1024-d) and CBM (320-d, MidCBM + SubspaceTopKCBM) are reported separately; Full L7 results cannot be attributed to the concept bottleneck alone.
Fixed-Hyperparameter 40-Label Test (AG News, 10 Labels/Class, 0 Validation Labels)
With ridge $\lambda = 0.1$ and LinearSVC $C = 1.0$ fixed in advance across 5 sampling seeds (20261203β20261207) and no validation labels used for selection:
- Full L7 (
1024-d):75.37% - CBM (
320-d):67.32% - Word (1β2 gram) + Character
char_wb(3β5 gram) TF-IDF LinearSVC:56.70% - Difference (Full L7 β SVM):
+18.67pp (95% paired seed-$t$ CI[+15.75, +21.60]pp; Holm-corrected $p = 0.0002975$).
Five-Seed Low-Label Curve (With 8,000-Example Validation Selection)
Each method uses the exact same training subsets (seeds 20261203β20261207) and selects hyperparameters on the 8,000-item validation split:
| Dataset | Fit labels / class | CBM (320-d) |
Full L7 (1024-d) |
Word + Char TF-IDF SVM | Full L7 β SVM (pp) |
|---|---|---|---|---|---|
AG News |
10 |
71.70% |
76.84% |
56.97% |
+19.88 |
AG News |
100 |
79.63% |
83.97% |
80.90% |
+3.07 |
AG News |
1000 |
84.19% |
87.61% |
87.80% |
-0.19 |
DBpedia-14 |
10 |
72.23% |
85.78% |
85.30% |
+0.47 |
DBpedia-14 |
100 |
88.09% |
94.27% |
94.67% |
-0.40 |
Full-Dataset Readout & Perturbation Checks
AG News uses 112,000 fit / 8,000 val / 7,600 test items. DBpedia-14 uses 120,000 fit / 8,000 val / 70,000 test items (10,000-item test subset for zero-shot rows; on that same 10,000-item subset, CBM / Full L7 / TF-IDF score 90.96% / 96.53% / 98.23%):
| Method | AG News | DBpedia-14 |
|---|---|---|
0.2B zero-shot (initial 4-template search) |
65.39% |
48.43% |
0.2B zero-shot (validation-selected prompt search: raw / calibrated / top-12 ensemble) |
81.41% / 81.79% / 83.30% |
β |
Readout: CBM (320-d) |
85.32% |
91.20% |
Readout: Full L7 (1024-d) |
89.00% |
96.84% |
| TF-IDF + Logistic Regression | 91.95% |
98.31% |
Qwen3-8B zero-shot (initial 4-template reference) |
81.97% |
88.38% |
| Full L7 readout with shuffled training labels | 24.92% |
7.64% |
Under surface perturbations on AG News (evaluated without retraining the head): Original 89.00% (TF-IDF 91.82%), All lowercase 86.45% (91.82%), Strip Reuters/AP datelines 88.83% (91.45%), First 32 tokens 86.09% (90.75%). The 4-class L7 head is 40 KiB (32 KiB float64 weights + 8 KiB float32 mean/std). Full-split tokenization takes 15.89 s, feature extraction 373.46 s, and ridge fitting 1.75 s per $\lambda$ candidate.
2.5 External MidCBM Retrieval Index + Copy Head
On 399 unseen paraphrases of stored ARC science questions, indexing frozen MidCBM and SubspaceTopKCBM states in an external retrieval index with a zero-parameter copy head raises accuracy from 28.57% (0.2B alone) to 72.93% (+44.36 pp, 95% CI [+39.36, +49.38] pp; 72.68% with shuffled options; 96.3% precision when triggered during 24-token greedy generation; 1.000009 max PPL ratio on C4/ZH).
3. Further Research & Next-Gen Multimodal Results
Based on this early 0.2B orthogonal framework, our team has completed further internal work on cross-modal representation alignment and industrial compilation toolchains (including cross-module Gram-Schmidt joint orthogonalization), scaling this formulation to 1B / 3B / 7B multimodal architectures.
On our 1B model (currently in testing and rollout across production workflows), mounting closed-form orthogonal modules without fine-tuning base weights yields:
- Enterprise Email Archiving & Routing:
65.8% β 98.5% - Complex Scanned Document Classification:
64.2% β 97.6% - SAR / Satellite Imagery Classification (used by OSU researchers):
64.7% β 98.3%
Technical details and industrial pipelines (CAD parsing and Tech Pack breakdown) are available at https://topovoid.com.
4. Limitations
- Readout heads are separate modules. A classification head reads L7 features and outputs class scores directly. It does not change what the language model generates and does not show general reasoning gains.
- Cosine
1.0on L7MidCBMcomes from causal order as well as projection. With the first edit at Layer 8, L7MidCBMandSubspaceTopKCBMactivations are computed upstream of the edited weights and cannot change. The L12Highway CBMis downstream of layers 8β11 and is not strictly invariant (0.9979 β 0.9890mean cosine across 2 to 8 stacked modules). - Module stacking interferes beyond 2 modules in
v0.1. In this0.2Bpreview,closed_form_write.pyonly performs single-module orthogonal projection against $Q_{\mathrm{cbm}}$ ($P_\perp = I - Q_{\mathrm{cbm}} Q_{\mathrm{cbm}}^T$) without a cross-module Gram-Schmidt joint orthogonalization operator. While2stacked modules (80facts) retain96.25%Top-1, adding4or8independently compiled modules degrades Top-1 to55.62%and34.69%and raises perplexity. Cross-module joint orthogonal compilation is part of our internal1Bcommercial compiler. - Report paraphrase and neighborhood metrics beside rewrite Top-1. Rewrite Top-1 measures reproduction of the written prompt. Paraphrase generalization is lower (
35.0%β41.7%Top-1), and edits bleed into neighboring subjects of the same relation (27.5%β56.7%). - Multi-hop hits require the control. On
MQuAKE-CF-3k,117/150multi-hop questions hit the edited answer, but118/149unrelated-subject control questions also trigger the same answer via surface question templates. - Writes raise perplexity on unrelated text, scaling with the number of facts written into a single module.
- The external memory index only recalls stored items, and can copy a wrong stored answer when retrieval is confident but incorrect.
- Pruned vocabulary. Unsupported tokenizer IDs map to compact ID
0in ordinary forward passes;closed_form_write.pysplits targets into active subtokens where possible. - Base model without safety tuning. Pretraining stages used web text without spam or adult-content filtering, and no chat or safety alignment was applied. Do not deploy directly in user-facing production.
5. License & Attributions
License
Code and TopoVoid-0.2B-Orthogonal-Preview weights in this repository are released under the Apache License 2.0 (see LICENSE).
- Organization: https://huggingface.co/Topovoid
- Developers: https://huggingface.co/tirpitzia, https://huggingface.co/0kaguya
Attributions
- Tokenizer and distillation teachers: Qwen tokenizer and Qwen teacher models (Qwen, Apache-2.0)
- Pretraining and calibration text: FineWeb, FineWeb2 (ODC-By 1.0), and C4 (ODC-By 1.0) (
eval_c4,eval_zhinvocab_map_49152_chatml.npz) - Evaluation benchmarks: CounterFact (MIT), MQuAKE (MIT), AG News, DBpedia-14 (CC-BY-SA-3.0), and AI2 ARC (CC-BY-SA-4.0)
Citation
@misc{topovoid2026orthogonal,
title = {TopoVoid-0.2B-Orthogonal-Preview: Native Orthogonal Subspace Architecture and Closed-Form Null-Space Weight Compilation},
author = {tirpitzia and 0kaguya and {TopoVoid Research}},
year = {2026},
howpublished = {\url{https://huggingface.co/Topovoid/TopoVoid-0.2B-Orthogonal-Preview}},
note = {Research Preview v0.1, \url{https://topovoid.com}}
}
- Downloads last month
- 509