Spaces:
Sleeping
Sleeping
File size: 15,408 Bytes
3deca49 75878d6 3deca49 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 | # TMT Implementation Contract (FROZEN)
This document is the binding contract every implementation agent reads before
writing code. `config.py` is already written and FROZEN — import from it; never
redefine model constants. `d_ff=112` is tuned so total params ≈ 6.26M (in the
5–7M spec budget); the model agent may adjust `d_ff` ONLY if the actual count
falls outside [5e6, 7e6].
Project root: `D:/Users/Documents/TMT (Tensor Metatron Technology)` (flat repo,
no package). Python 3.13, torch 2.13 CPU, pytest 9. Run code from repo root.
## 0. Global rules for ALL agents
1. Read `config.py` and THIS contract before writing any code.
2. `from config import CONFIG` (or `get_config`) for every model constant. No
hardcoded `256`/`4`/`6`/`32`/`291`/`13`/`2` in logic — read `CONFIG[...]`.
3. Only create/edit files listed in your task below. Do NOT touch other files.
4. Every `if __name__ == "__main__":` block starts with
`import sys; sys.stdout.reconfigure(encoding="utf-8")` (Windows cp1252 fix).
No ✓/✗/accented literals in print without that reconfigure.
5. Derive device from input tensors (`x.device`); never assume CPU. Derive dtype
from inputs where relevant. Registered buffers must follow `.to(device)`.
6. After writing, SELF-TEST by importing your module and (where applicable)
running its `__main__` or a quick `python -c "import <mod>"`. Fix until clean.
7. Use type hints. Keep docstrings referencing the patent claims (existing style).
8. Do NOT create a persistent `metatron.db` or `ckpt/` during implementation —
tests and the verify phase handle real runs. Use temp paths in any scratch.
## 1. Interface contract (post-fix)
### config.py (DONE — frozen)
`CONFIG` dict, `get_config() -> dict`, `estimate_total_params(d_ff=None)`.
### metatron_sparse_attention.py — API CHANGES
- Fix POLYHEDRA: octahedron → 8 correct triangular faces
`[[0,1,2],[0,2,3],[0,3,4],[0,4,1],[5,2,1],[5,3,2],[5,4,3],[5,1,4]]` (V=6,
apexes 0 top / 5 bottom, equator 1-2-3-4). dodecahedron → 12 correct
pentagonal faces (standard labeling; every face 5 vertices, total 12).
- `geometric_chords`: add `cross` to the set before discarding idx.
- `MetatronSparseMask.csr`: delete dead `indptr=[0]; for r in rows: pass`; build
indices SORTED within each row (CSR invariant for torch.sparse_csr_tensor).
- `measure_sparsity`: rename `ops_reduction_pct` → `mask_sparsity_pct` (do not
claim realized FLOP reduction; the forward is masked-dense reference).
- **`MetatronSparseAttention` forward contract**:
```python
class MetatronSparseAttention(nn.Module):
def __init__(self, d_model, n_heads, mask=None, dropout=0.1, bias=False): ...
def forward(self, x, mask=None) -> torch.Tensor:
# x: (B, L, d_model) self-attention (q=k=v=x internally)
# -> project with Linear(d_model, d_model) for Q,K,V,O (standard MHA)
# -> reshape to (B, H, L, d_k), scale=1/sqrt(d_k)
# -> apply sparse mask from active_mask.edges (set -inf elsewhere)
# -> softmax; sparse-aware output via gather over mask edges (scatter_add)
# -> output proj; return (B, L, d_model)
```
Derive `r, c` from `active_mask.edges` inside forward (not stored buffers),
on `x.device`. Drop `_mask_rows/_mask_cols/_mask_count` buffers entirely.
The scatter_add index must be 4-D: `idx = r.view(1,1,n_edges,1).expand_as(weighted)`
with `weighted` shaped `(B,H,n_edges,d_k)`.
- Module-level functions `radial_level/local_window/geometric_chords/long_range_routes`
keep their signatures. `MetatronSparseMask.build` keeps signature.
- `__main__` smoke test must PASS: build mask for all 5 solids (print sparsity +
edges), then `MetatronSparseAttention(d_model=256, n_heads=4, mask=mask)`;
`x = torch.randn(B, L, 256)` (NOT pre-split heads); `out = attn(x)`; assert
`out.shape == (B, L, 256)`.
### metatron_moe.py — API CHANGES
- Load tracking: aggregate over ALL k slots:
`all_idx = topk_idx.reshape(-1); counts = torch.bincount(all_idx, minlength=n_e).float()`
(do NOT reference loop-leftover `expert_ids`). `last_load.sum() == top_k*B*L`.
- Honor `return_load`: `if return_load: return out.view(B,L,D), counts`.
- Add `return_aux: bool = False` param; compute
`aux_loss = n_e * (fractions).pow(2).sum()` where
`fractions = counts / counts.sum()` (Switch-Transformer balancing loss);
return `(out, counts, aux_loss)` when both flags set. Provide
`reset_load()` zeroing `_expert_counts`.
- `expert_utilisation()` returns `self._expert_counts/(sum+1e-8)` on the module
device (never CPU zeros).
- Decide `_expert_counts` vs `last_load`: `last_load` = per-call counts (source
of truth for aux loss); `_expert_counts` = cumulative (exposed via
`expert_utilisation`, resettable). Keep both, document.
- `capacity_factor`: keep stored; do NOT implement dropping (document as
reserved). Keep signature.
- `__main__` smoke test must PASS: `out = moe(x)` shape (B,L,D);
`out, load = moe(x, return_load=True)`; assert `load.sum().item() == top_k*B*L`;
`out, load, aux = moe(x, return_load=True, return_aux=True)`; assert
`aux.item() > 0`. Print utilization, active experts.
### metatron_global_memory.py — API CHANGES
- Fix smoke test line: `torch.round(attn[0,:3,:5], decimals=3)`.
- `MetatronMemoryEnhancedAttention.forward`: APPLY local_attn:
`x = self.norm1(x); if self.local_attn is not None: x = x + self.local_attn(x);
x = self.norm2(x + self.global_mem(x)); return x`. (Fixes double-residual: the
inner `MetatronGlobalMemory.forward` keeps its own residual, so the wrapper
must NOT also add raw x twice — use the pattern above; ensure x added once in
the global branch via `x = self.norm2(x + self.global_mem(x))` where
`global_mem` already does `local_x + dropout(proj(...))`. Document clearly.)
- `set_state`: `with torch.no_grad(): self.memory.copy_(new_memory.to(
self.memory.device, self.memory.dtype))` (preserve device/dtype).
- Gating: keep per-sequence (mean over dim=1) BUT document it explicitly in the
docstring as per-sequence gating; OR upgrade to per-token
`gate_val = torch.sigmoid(self.gate(local_x))`. Implementer choice; per-token
preferred for expressivity. Document the choice.
- Fix persistence smoke assertion: capture `p0 = mem.memory.data_ptr()` once,
call forward again, compare `mem.memory.data_ptr() == p0`.
- Clean up the contradictory "buffer not learnable" comment: memory IS a learnable
Parameter; fix the comment.
- `n_heads` arg of `MetatronMemoryEnhancedAttention`: wire it (pass to local_attn
when set externally) or drop it. Document.
- `__main__` smoke test must PASS: prints shapes, `persists: True`.
### metatron_shard.py — fixes (tested in isolation; NOT wired into CPU model)
- `__main__`: ASCII `OK`/`FAIL` (or reconfigure stdout UTF-8). Must run clean.
- `compute_diameter`: HONOR the topology argument (add `topology=None` param; if
None use `self.topology`). Correct diameter table: tetra=1, octa=2, cube=3,
icosa=3, dodeca=5.
- `diameter_reduction(base_topology)`: instantiate a SEPARATE planner for the base
and compute its diameter (do NOT use `self.topology` for both). Must return
>0 for dodeca vs hexa (5 vs 3 → 0.4 ≥ 0.333).
- `bytes_per_token`: set to 4 (float32) in radial/ring; pass total bytes
(`token_count*4`) to `_est_latency`. Chord already uses 4.
- `_balance_load`: recompute ALL shard boundaries from new per-shard token counts
derived from load ratios (no independent end-grow → overlaps/gaps). Init
`self.total_tokens=0` in `__init__`.
- Rename `ICOSADISTRIBUTED` → `ICOSA_DISTRIBUTED`; update `_strategy_name` and
`_effective_shards`.
- Wire `_min_cut_routes` into `plan()` (use to inform chord/ring targets) OR
remove it and drop the docstring claim. Implementer choice; if kept, make it a
real BFS shortest-path over the face-adjacency graph.
- Remove dead `tokens_per_shard` in `_shard_to_face`.
- `__main__` smoke test must PASS: prints diameters, `reduction` ≥ 33.3% as OK,
then `plan(1024).print_summary()` and phase breakdown.
### metatron_compiler.py — fixes
- `morton_reorder`: sort by RAW morton key (`order.append(morton)`), not
`morton % seq_len`. Result must be a bijection (permutation of 0..seq_len-1).
Update the docstring example to match actual output.
- `compile_to_csr`: sort columns within each row (CSR invariant). Return sorted
(indptr, indices).
- `bandwidth_estimate`: `*4` not `*3` (Q,K,V,O = 4 transfers). Accept `d_k`
param (default 64) instead of hardcoding.
- `flop_estimate`/`flops_per_token`/`summary`: accept `d_k` param (default 64).
- `polyhedral_ring_order`: real BFS from vertex 0 over `poly['edges']`; drop the
`[1]` default (use `[]`).
- `Quantizer('int8').quantize`: guard scale — `amax = tensor.abs().max();
scale = amax/127.0 if amax>0 else torch.tensor(1.0)`.
- `__main__` smoke test must PASS: prints Morton order, CSR lengths, kernel
summary, quantized dtype.
## 2. New files (each owned by exactly ONE agent)
| File | Owner tier | Public interface |
|---|---|---|
| `tokenizer.py` + `vocab.json` | T2 | `Tokenizer` class: `encode(text)->list[int]`, `decode(ids)->str`, `vocab_size` prop, `special_ids` dict, `@classmethod from_file(path)`. `vocab.json` = token→id map. |
| `db.py` | T2 | `init_db(path)`, `add_texts(path, texts, domain, quality_threshold)->(added,rejected)`, `fetch_training_rows(path, domain, min_quality, limit, used=False)->list[dict]`, `mark_used(path, ids)`, `stats(path)->dict`, `save_checkpoint(path, step, loss, file_path, is_active=True)`, `start_session(path, domain_filter, min_quality)->int`, `end_session(path, session_id, total_steps, final_loss)`, `get_active_checkpoint(path)->dict\|None`, `set_active_checkpoint(path, file_path)`. |
| `quality.py` | T2 | `score_quality(text)->float` in [0,1] (heuristic: length, repetition, keyword density for technical domains). |
| `tinymetatron_model.py` | T2 | `TinyMetatron(config: dict)->nn.Module`; `.forward(input_ids: LongTensor[B,L])->logits[B,L,vocab]`; `.generate(input_ids, max_length, temperature)->LongTensor[B,L_out]`; `.param_count()->dict`; classmethod `from_config()`; `load_checkpoint(path)`, `save_checkpoint(path)`. |
| `train_db.py` | T3 | `main(argv)` CLI; flags per contract §3. Prints Slovak progress (`Krok N/M: loss=x.xxx`) + `Tréning dokončený. Finálna strata: x.xxx`. |
| `manage_data.py` | T3 | `main(argv)` with subcommands `generate/import/export/stats/clean`. |
| `api.py` | T3 | `app: FastAPI`; endpoints `/generate`, `/train/start`, `/train/status`, `/data/add`, `/data/stats`, `/model/info`. Training runs in a background thread; status is module-global. |
| `tests/test_sparse_attention.py` | T4a | shapes, mask-override, octa/dodeca faces correct, smoke forward, csr sorted. |
| `tests/test_moe.py` | T4a | load.sum()==top_k*B*L; return_load tuple; aux>0; reset_load; device. |
| `tests/test_global_memory.py` | T4a | persistence across 2 forwards; set_state device; local_attn applied; smoke. |
| `tests/test_shard.py` | T4a | diameter_reduction(dodeca vs hexa)>=0.333; no overlapping ranges; UTF-8 smoke; bytes_per_token==4. |
| `tests/test_compiler.py` | T4a | morton_reorder is a bijection on 0..127 & matches docstring; CSR sorted within rows; int8 zero tensor; bandwidth *4. |
| `tests/test_tokenizer.py` | T4b | vocab_size==291; round-trip SK+EN technical text (lossless for in-vocab chars); BOS/EOS wrapping; UNK fallback. |
| `tests/test_model.py` | T4b | forward (B,32)->(B,32,291); total_params in [5e6,7e6]; generate ids in [0,291); loss+backward runs. |
| `tests/test_db.py` | T4b | schema create; insert/fetch; mark_used; active checkpoint swap; stats keys. |
| `tests/test_api.py` | T4b | FastAPI TestClient: /data/add 200, /data/stats keys, /model/info config, /generate 200. |
| `tests/test_train.py` | T4b | 2-step run on tiny DB: loss decreases; checkpoint row written; session end_time set; used_in_training flips. |
| `Dockerfile` | T4c | python:3.13-slim, CPU torch wheel, fastapi/uvicorn/pydantic, /app, EXPOSE 8010, CMD uvicorn api:app. |
| `docker-compose.yml` | T4c | `slm` service, image `metatron-slm-slm:latest` (verbatim), 3 volumes (./data ./ckpt ./metatron.db), port 8010, env PYTHONPATH/MODEL_PATH. |
| `requirements.txt` | T4c | pinned: torch (cpu index url note), fastapi, uvicorn, pydantic, pytest. |
| `README.md` | T4c | build/run/train/API/CLI instructions, endpoint list, vocab note, param count. |
## 3. CLI & API details
`train_db.py` flags: `--steps`(200) `--domain`(general) `--min_quality`(0.5)
`--batch_size`(16) `--learning_rate`(1e-3) `--max_seq_len`(32) `--device`(cpu)
`--checkpoint_dir`(ckpt) `--db_path`(metatron.db) `--aux_loss_weight`(0.01)
`--seed`(42). Behavior: load active ckpt or init; fetch rows from training_data
WHERE domain=? AND quality>=? AND used_in_training=0; mark used; insert
training_sessions row; train; write model_checkpoints row + .pt (is_active=1,
clear prior); update session end_time/final_loss. Optimizer: Adam. Loss: CE over
LM logits + aux_loss_weight*moe_aux.
`manage_data.py` subcommands:
- `generate --domain X --count N`: synthesize N rows (use a small built-in
template list per domain: cybersecurity/software/general; deterministic).
- `import --file F --domain X`: read lines, score, insert.
- `export --domain X --min_quality Q [--out F]`: print/dump matching rows.
- `stats`: print total, by_domain, avg_quality, used_in_training counts.
- `clean --min_quality Q`: delete rows with quality < Q.
`api.py`: FastAPI. Module-global `_train_state` dict {is_training, current_step,
total_steps, current_loss, session_id}. `/train/start` launches a background
thread calling `train_db`-equivalent logic (refactor a callable
`run_training(config_override) -> None` in train_db.py that api.py imports).
`/generate` loads active ckpt, tokenizes prompt, `model.generate`, decodes.
`/data/add` calls `db.add_texts` with `quality.score_quality`. `/data/stats`
calls `db.stats`. `/model/info` returns CONFIG + active checkpoint.
## 4. Coherence risks & mitigations (binding)
- **R1 config drift**: import CONFIG everywhere (rule 2). config.py frozen.
- **R2 attention contract**: forward takes (B,L,d_model) — model passes hidden,
NOT pre-split heads. This is mandatory; the fixed module enforces it.
- **R3 device/dtype**: derive from inputs (rule 5). Tests gate on
`cuda if available` else cpu.
- **R4 namespace**: only files in §2 may exist. No shared utils.py.
- **R5 vocab determinism**: `vocab.json` written once by the T2 tokenizer agent;
model/tests only consume `Tokenizer`.
- **R6 aux-loss wiring**: trainer reads `return_aux=True`; test_train asserts
`aux_loss.detach() > 0` on a step.
- **R7 UTF-8**: rule 4. Tests run with `PYTHONIOENCODING=utf-8`.
- **R8 param budget**: test_model asserts `5e6 <= total <= 7e6`; model agent
retunes `d_ff` (edit config.py's d_ff only) if outside, then re-run
`python config.py` to confirm.
- **R9 shard not in CPU path**: TinyMetatron never imports metatron_shard;
shard tested in isolation only.
## 5. Self-test before handing off
Each agent must, after writing, run from repo root:
- `python -c "import <your_module>"` (must exit 0)
- `python <your_module>` for modules with `__main__` (must exit 0, no traceback)
- For test files: `python -m pytest <your_test_file> -q` must be green.
Fix until clean. Report any deviation. |