File size: 15,408 Bytes
3deca49
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
75878d6
3deca49
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
# TMT Implementation Contract (FROZEN)

This document is the binding contract every implementation agent reads before
writing code. `config.py` is already written and FROZEN — import from it; never
redefine model constants. `d_ff=112` is tuned so total params ≈ 6.26M (in the
5–7M spec budget); the model agent may adjust `d_ff` ONLY if the actual count
falls outside [5e6, 7e6].

Project root: `D:/Users/Documents/TMT (Tensor Metatron Technology)` (flat repo,
no package). Python 3.13, torch 2.13 CPU, pytest 9. Run code from repo root.

## 0. Global rules for ALL agents

1. Read `config.py` and THIS contract before writing any code.
2. `from config import CONFIG` (or `get_config`) for every model constant. No
   hardcoded `256`/`4`/`6`/`32`/`291`/`13`/`2` in logic — read `CONFIG[...]`.
3. Only create/edit files listed in your task below. Do NOT touch other files.
4. Every `if __name__ == "__main__":` block starts with
   `import sys; sys.stdout.reconfigure(encoding="utf-8")` (Windows cp1252 fix).
   No ✓/✗/accented literals in print without that reconfigure.
5. Derive device from input tensors (`x.device`); never assume CPU. Derive dtype
   from inputs where relevant. Registered buffers must follow `.to(device)`.
6. After writing, SELF-TEST by importing your module and (where applicable)
   running its `__main__` or a quick `python -c "import <mod>"`. Fix until clean.
7. Use type hints. Keep docstrings referencing the patent claims (existing style).
8. Do NOT create a persistent `metatron.db` or `ckpt/` during implementation —
   tests and the verify phase handle real runs. Use temp paths in any scratch.

## 1. Interface contract (post-fix)

### config.py (DONE — frozen)
`CONFIG` dict, `get_config() -> dict`, `estimate_total_params(d_ff=None)`.

### metatron_sparse_attention.py — API CHANGES
- Fix POLYHEDRA: octahedron → 8 correct triangular faces
  `[[0,1,2],[0,2,3],[0,3,4],[0,4,1],[5,2,1],[5,3,2],[5,4,3],[5,1,4]]` (V=6,
  apexes 0 top / 5 bottom, equator 1-2-3-4). dodecahedron → 12 correct
  pentagonal faces (standard labeling; every face 5 vertices, total 12).
- `geometric_chords`: add `cross` to the set before discarding idx.
- `MetatronSparseMask.csr`: delete dead `indptr=[0]; for r in rows: pass`; build
  indices SORTED within each row (CSR invariant for torch.sparse_csr_tensor).
- `measure_sparsity`: rename `ops_reduction_pct` → `mask_sparsity_pct` (do not
  claim realized FLOP reduction; the forward is masked-dense reference).
- **`MetatronSparseAttention` forward contract**:
  ```python
  class MetatronSparseAttention(nn.Module):
      def __init__(self, d_model, n_heads, mask=None, dropout=0.1, bias=False): ...
      def forward(self, x, mask=None) -> torch.Tensor:
          # x: (B, L, d_model) self-attention (q=k=v=x internally)
          # -> project with Linear(d_model, d_model) for Q,K,V,O (standard MHA)
          # -> reshape to (B, H, L, d_k), scale=1/sqrt(d_k)
          # -> apply sparse mask from active_mask.edges (set -inf elsewhere)
          # -> softmax; sparse-aware output via gather over mask edges (scatter_add)
          # -> output proj; return (B, L, d_model)
  ```
  Derive `r, c` from `active_mask.edges` inside forward (not stored buffers),
  on `x.device`. Drop `_mask_rows/_mask_cols/_mask_count` buffers entirely.
  The scatter_add index must be 4-D: `idx = r.view(1,1,n_edges,1).expand_as(weighted)`
  with `weighted` shaped `(B,H,n_edges,d_k)`.
- Module-level functions `radial_level/local_window/geometric_chords/long_range_routes`
  keep their signatures. `MetatronSparseMask.build` keeps signature.
- `__main__` smoke test must PASS: build mask for all 5 solids (print sparsity +
  edges), then `MetatronSparseAttention(d_model=256, n_heads=4, mask=mask)`;
  `x = torch.randn(B, L, 256)` (NOT pre-split heads); `out = attn(x)`; assert
  `out.shape == (B, L, 256)`.

### metatron_moe.py — API CHANGES
- Load tracking: aggregate over ALL k slots:
  `all_idx = topk_idx.reshape(-1); counts = torch.bincount(all_idx, minlength=n_e).float()`
  (do NOT reference loop-leftover `expert_ids`). `last_load.sum() == top_k*B*L`.
- Honor `return_load`: `if return_load: return out.view(B,L,D), counts`.
- Add `return_aux: bool = False` param; compute
  `aux_loss = n_e * (fractions).pow(2).sum()` where
  `fractions = counts / counts.sum()` (Switch-Transformer balancing loss);
  return `(out, counts, aux_loss)` when both flags set. Provide
  `reset_load()` zeroing `_expert_counts`.
- `expert_utilisation()` returns `self._expert_counts/(sum+1e-8)` on the module
  device (never CPU zeros).
- Decide `_expert_counts` vs `last_load`: `last_load` = per-call counts (source
  of truth for aux loss); `_expert_counts` = cumulative (exposed via
  `expert_utilisation`, resettable). Keep both, document.
- `capacity_factor`: keep stored; do NOT implement dropping (document as
  reserved). Keep signature.
- `__main__` smoke test must PASS: `out = moe(x)` shape (B,L,D);
  `out, load = moe(x, return_load=True)`; assert `load.sum().item() == top_k*B*L`;
  `out, load, aux = moe(x, return_load=True, return_aux=True)`; assert
  `aux.item() > 0`. Print utilization, active experts.

### metatron_global_memory.py — API CHANGES
- Fix smoke test line: `torch.round(attn[0,:3,:5], decimals=3)`.
- `MetatronMemoryEnhancedAttention.forward`: APPLY local_attn:
  `x = self.norm1(x); if self.local_attn is not None: x = x + self.local_attn(x);
  x = self.norm2(x + self.global_mem(x)); return x`. (Fixes double-residual: the
  inner `MetatronGlobalMemory.forward` keeps its own residual, so the wrapper
  must NOT also add raw x twice — use the pattern above; ensure x added once in
  the global branch via `x = self.norm2(x + self.global_mem(x))` where
  `global_mem` already does `local_x + dropout(proj(...))`. Document clearly.)
- `set_state`: `with torch.no_grad(): self.memory.copy_(new_memory.to(
  self.memory.device, self.memory.dtype))` (preserve device/dtype).
- Gating: keep per-sequence (mean over dim=1) BUT document it explicitly in the
  docstring as per-sequence gating; OR upgrade to per-token
  `gate_val = torch.sigmoid(self.gate(local_x))`. Implementer choice; per-token
  preferred for expressivity. Document the choice.
- Fix persistence smoke assertion: capture `p0 = mem.memory.data_ptr()` once,
  call forward again, compare `mem.memory.data_ptr() == p0`.
- Clean up the contradictory "buffer not learnable" comment: memory IS a learnable
  Parameter; fix the comment.
- `n_heads` arg of `MetatronMemoryEnhancedAttention`: wire it (pass to local_attn
  when set externally) or drop it. Document.
- `__main__` smoke test must PASS: prints shapes, `persists: True`.

### metatron_shard.py — fixes (tested in isolation; NOT wired into CPU model)
- `__main__`: ASCII `OK`/`FAIL` (or reconfigure stdout UTF-8). Must run clean.
- `compute_diameter`: HONOR the topology argument (add `topology=None` param; if
  None use `self.topology`). Correct diameter table: tetra=1, octa=2, cube=3,
  icosa=3, dodeca=5.
- `diameter_reduction(base_topology)`: instantiate a SEPARATE planner for the base
  and compute its diameter (do NOT use `self.topology` for both). Must return
  >0 for dodeca vs hexa (5 vs 3 → 0.4 ≥ 0.333).
- `bytes_per_token`: set to 4 (float32) in radial/ring; pass total bytes
  (`token_count*4`) to `_est_latency`. Chord already uses 4.
- `_balance_load`: recompute ALL shard boundaries from new per-shard token counts
  derived from load ratios (no independent end-grow → overlaps/gaps). Init
  `self.total_tokens=0` in `__init__`.
- Rename `ICOSADISTRIBUTED` → `ICOSA_DISTRIBUTED`; update `_strategy_name` and
  `_effective_shards`.
- Wire `_min_cut_routes` into `plan()` (use to inform chord/ring targets) OR
  remove it and drop the docstring claim. Implementer choice; if kept, make it a
  real BFS shortest-path over the face-adjacency graph.
- Remove dead `tokens_per_shard` in `_shard_to_face`.
- `__main__` smoke test must PASS: prints diameters, `reduction` ≥ 33.3% as OK,
  then `plan(1024).print_summary()` and phase breakdown.

### metatron_compiler.py — fixes
- `morton_reorder`: sort by RAW morton key (`order.append(morton)`), not
  `morton % seq_len`. Result must be a bijection (permutation of 0..seq_len-1).
  Update the docstring example to match actual output.
- `compile_to_csr`: sort columns within each row (CSR invariant). Return sorted
  (indptr, indices).
- `bandwidth_estimate`: `*4` not `*3` (Q,K,V,O = 4 transfers). Accept `d_k`
  param (default 64) instead of hardcoding.
- `flop_estimate`/`flops_per_token`/`summary`: accept `d_k` param (default 64).
- `polyhedral_ring_order`: real BFS from vertex 0 over `poly['edges']`; drop the
  `[1]` default (use `[]`).
- `Quantizer('int8').quantize`: guard scale — `amax = tensor.abs().max();
  scale = amax/127.0 if amax>0 else torch.tensor(1.0)`.
- `__main__` smoke test must PASS: prints Morton order, CSR lengths, kernel
  summary, quantized dtype.

## 2. New files (each owned by exactly ONE agent)

| File | Owner tier | Public interface |
|---|---|---|
| `tokenizer.py` + `vocab.json` | T2 | `Tokenizer` class: `encode(text)->list[int]`, `decode(ids)->str`, `vocab_size` prop, `special_ids` dict, `@classmethod from_file(path)`. `vocab.json` = token→id map. |
| `db.py` | T2 | `init_db(path)`, `add_texts(path, texts, domain, quality_threshold)->(added,rejected)`, `fetch_training_rows(path, domain, min_quality, limit, used=False)->list[dict]`, `mark_used(path, ids)`, `stats(path)->dict`, `save_checkpoint(path, step, loss, file_path, is_active=True)`, `start_session(path, domain_filter, min_quality)->int`, `end_session(path, session_id, total_steps, final_loss)`, `get_active_checkpoint(path)->dict\|None`, `set_active_checkpoint(path, file_path)`. |
| `quality.py` | T2 | `score_quality(text)->float` in [0,1] (heuristic: length, repetition, keyword density for technical domains). |
| `tinymetatron_model.py` | T2 | `TinyMetatron(config: dict)->nn.Module`; `.forward(input_ids: LongTensor[B,L])->logits[B,L,vocab]`; `.generate(input_ids, max_length, temperature)->LongTensor[B,L_out]`; `.param_count()->dict`; classmethod `from_config()`; `load_checkpoint(path)`, `save_checkpoint(path)`. |
| `train_db.py` | T3 | `main(argv)` CLI; flags per contract §3. Prints Slovak progress (`Krok N/M: loss=x.xxx`) + `Tréning dokončený. Finálna strata: x.xxx`. |
| `manage_data.py` | T3 | `main(argv)` with subcommands `generate/import/export/stats/clean`. |
| `api.py` | T3 | `app: FastAPI`; endpoints `/generate`, `/train/start`, `/train/status`, `/data/add`, `/data/stats`, `/model/info`. Training runs in a background thread; status is module-global. |
| `tests/test_sparse_attention.py` | T4a | shapes, mask-override, octa/dodeca faces correct, smoke forward, csr sorted. |
| `tests/test_moe.py` | T4a | load.sum()==top_k*B*L; return_load tuple; aux>0; reset_load; device. |
| `tests/test_global_memory.py` | T4a | persistence across 2 forwards; set_state device; local_attn applied; smoke. |
| `tests/test_shard.py` | T4a | diameter_reduction(dodeca vs hexa)>=0.333; no overlapping ranges; UTF-8 smoke; bytes_per_token==4. |
| `tests/test_compiler.py` | T4a | morton_reorder is a bijection on 0..127 & matches docstring; CSR sorted within rows; int8 zero tensor; bandwidth *4. |
| `tests/test_tokenizer.py` | T4b | vocab_size==291; round-trip SK+EN technical text (lossless for in-vocab chars); BOS/EOS wrapping; UNK fallback. |
| `tests/test_model.py` | T4b | forward (B,32)->(B,32,291); total_params in [5e6,7e6]; generate ids in [0,291); loss+backward runs. |
| `tests/test_db.py` | T4b | schema create; insert/fetch; mark_used; active checkpoint swap; stats keys. |
| `tests/test_api.py` | T4b | FastAPI TestClient: /data/add 200, /data/stats keys, /model/info config, /generate 200. |
| `tests/test_train.py` | T4b | 2-step run on tiny DB: loss decreases; checkpoint row written; session end_time set; used_in_training flips. |
| `Dockerfile` | T4c | python:3.13-slim, CPU torch wheel, fastapi/uvicorn/pydantic, /app, EXPOSE 8010, CMD uvicorn api:app. |
| `docker-compose.yml` | T4c | `slm` service, image `metatron-slm-slm:latest` (verbatim), 3 volumes (./data ./ckpt ./metatron.db), port 8010, env PYTHONPATH/MODEL_PATH. |
| `requirements.txt` | T4c | pinned: torch (cpu index url note), fastapi, uvicorn, pydantic, pytest. |
| `README.md` | T4c | build/run/train/API/CLI instructions, endpoint list, vocab note, param count. |

## 3. CLI & API details

`train_db.py` flags: `--steps`(200) `--domain`(general) `--min_quality`(0.5)
`--batch_size`(16) `--learning_rate`(1e-3) `--max_seq_len`(32) `--device`(cpu)
`--checkpoint_dir`(ckpt) `--db_path`(metatron.db) `--aux_loss_weight`(0.01)
`--seed`(42). Behavior: load active ckpt or init; fetch rows from training_data
WHERE domain=? AND quality>=? AND used_in_training=0; mark used; insert
training_sessions row; train; write model_checkpoints row + .pt (is_active=1,
clear prior); update session end_time/final_loss. Optimizer: Adam. Loss: CE over
LM logits + aux_loss_weight*moe_aux.

`manage_data.py` subcommands:
- `generate --domain X --count N`: synthesize N rows (use a small built-in
  template list per domain: cybersecurity/software/general; deterministic).
- `import --file F --domain X`: read lines, score, insert.
- `export --domain X --min_quality Q [--out F]`: print/dump matching rows.
- `stats`: print total, by_domain, avg_quality, used_in_training counts.
- `clean --min_quality Q`: delete rows with quality < Q.

`api.py`: FastAPI. Module-global `_train_state` dict {is_training, current_step,
total_steps, current_loss, session_id}. `/train/start` launches a background
thread calling `train_db`-equivalent logic (refactor a callable
`run_training(config_override) -> None` in train_db.py that api.py imports).
`/generate` loads active ckpt, tokenizes prompt, `model.generate`, decodes.
`/data/add` calls `db.add_texts` with `quality.score_quality`. `/data/stats`
calls `db.stats`. `/model/info` returns CONFIG + active checkpoint.

## 4. Coherence risks & mitigations (binding)

- **R1 config drift**: import CONFIG everywhere (rule 2). config.py frozen.
- **R2 attention contract**: forward takes (B,L,d_model) — model passes hidden,
  NOT pre-split heads. This is mandatory; the fixed module enforces it.
- **R3 device/dtype**: derive from inputs (rule 5). Tests gate on
  `cuda if available` else cpu.
- **R4 namespace**: only files in §2 may exist. No shared utils.py.
- **R5 vocab determinism**: `vocab.json` written once by the T2 tokenizer agent;
  model/tests only consume `Tokenizer`.
- **R6 aux-loss wiring**: trainer reads `return_aux=True`; test_train asserts
  `aux_loss.detach() > 0` on a step.
- **R7 UTF-8**: rule 4. Tests run with `PYTHONIOENCODING=utf-8`.
- **R8 param budget**: test_model asserts `5e6 <= total <= 7e6`; model agent
  retunes `d_ff` (edit config.py's d_ff only) if outside, then re-run
  `python config.py` to confirm.
- **R9 shard not in CPU path**: TinyMetatron never imports metatron_shard;
  shard tested in isolation only.

## 5. Self-test before handing off

Each agent must, after writing, run from repo root:
- `python -c "import <your_module>"` (must exit 0)
- `python <your_module>` for modules with `__main__` (must exit 0, no traceback)
- For test files: `python -m pytest <your_test_file> -q` must be green.
Fix until clean. Report any deviation.