Transformers
PyTorch
English
language-model
graph-attention
adaptive-depth
temporal-decay
efficient-llm
Eval Results (legacy)
Instructions to use vigneshwar234/TemporalMesh-Transformer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use vigneshwar234/TemporalMesh-Transformer with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("vigneshwar234/TemporalMesh-Transformer", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Add README.md
Browse files- github_README.md +485 -0
github_README.md
ADDED
|
@@ -0,0 +1,485 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<img src="https://capsule-render.vercel.app/api?type=waving&color=0:0d1117,50:1a1a2e,100:16213e&height=200§ion=header&text=TemporalMesh%20Transformer&fontSize=48&fontColor=58a6ff&fontAlignY=45&desc=Dynamic%20Graph%20%E2%80%A2%20Temporal%20Decay%20%E2%80%A2%20Adaptive%20Depth%20Routing&descAlignY=70&descSize=18&descColor=8b949e&animation=fadeIn" width="100%"/>
|
| 2 |
+
|
| 3 |
+
<div align="center">
|
| 4 |
+
|
| 5 |
+
[](https://github.com/vignesh2027/TemporalMesh-Transformer)
|
| 6 |
+
|
| 7 |
+
<br/>
|
| 8 |
+
|
| 9 |
+
[](https://github.com/vignesh2027/TemporalMesh-Transformer/actions/workflows/ci.yml)
|
| 10 |
+
[](https://github.com/vignesh2027/TemporalMesh-Transformer/actions)
|
| 11 |
+
[](https://python.org)
|
| 12 |
+
[](https://pytorch.org)
|
| 13 |
+
[](LICENSE)
|
| 14 |
+
[](https://github.com/vignesh2027/TemporalMesh-Transformer/stargazers)
|
| 15 |
+
|
| 16 |
+
<br/>
|
| 17 |
+
|
| 18 |
+
[](https://doi.org/10.5281/zenodo.20287197)
|
| 19 |
+
[](https://zenodo.org/records/20287390)
|
| 20 |
+
[](https://huggingface.co/vigneshwar234/TemporalMesh-Transformer)
|
| 21 |
+
[](https://huggingface.co/spaces/vigneshwar234/TemporalMesh-Transformer-Demo)
|
| 22 |
+
[](https://colab.research.google.com/github/vignesh2027/TemporalMesh-Transformer)
|
| 23 |
+
[](https://vignesh2027.github.io/TemporalMesh-Transformer)
|
| 24 |
+
|
| 25 |
+
</div>
|
| 26 |
+
|
| 27 |
+
---
|
| 28 |
+
|
| 29 |
+
## The Difference
|
| 30 |
+
|
| 31 |
+
> **Every transformer since 2017 makes the same 3 assumptions. TMT breaks all three.**
|
| 32 |
+
|
| 33 |
+
| Old Assumption | How TMT Breaks It |
|
| 34 |
+
|:---|:---|
|
| 35 |
+
| The sequence is a flat list | **Dynamic mesh graph** — token connectivity rebuilt every layer via cosine similarity |
|
| 36 |
+
| All tokens use the same compute | **Adaptive depth routing** — confident tokens exit early, hard ones go all the way |
|
| 37 |
+
| All tokens are equally relevant | **Temporal semantic decay** — irrelevant tokens are multiplicatively suppressed |
|
| 38 |
+
|
| 39 |
+
No other architecture does all three simultaneously. Not GPT. Not LLaMA. Not graph transformers. Not MoE.
|
| 40 |
+
|
| 41 |
+
---
|
| 42 |
+
|
| 43 |
+
## Comparison Table
|
| 44 |
+
|
| 45 |
+
| Feature | GPT / LLaMA | Graph Transformer | Early Exit | MoE | **TMT** |
|
| 46 |
+
|:---|:---:|:---:|:---:|:---:|:---:|
|
| 47 |
+
| Dynamic Graph (per-layer rebuild) | ✗ | Static only | ✗ | ✗ | **✓** |
|
| 48 |
+
| Per-Token Depth Routing | ✗ | ✗ | Partial | ✗ | **✓** |
|
| 49 |
+
| Temporal Semantic Decay | ✗ | ✗ | ✗ | ✗ | **✓** |
|
| 50 |
+
| Persistent Memory Anchors | ✗ | ✗ | ✗ | ✗ | **✓** |
|
| 51 |
+
| Dual-Stream FFN | ✗ | ✗ | ✗ | Partial | **✓** |
|
| 52 |
+
| O(S·k) attention complexity | ✗ (O(S²)) | Sometimes | ✗ | ✗ | **✓** |
|
| 53 |
+
|
| 54 |
+
---
|
| 55 |
+
|
| 56 |
+
## Three Core Innovations — Deep Dive
|
| 57 |
+
|
| 58 |
+
### Innovation 1: Mesh Attention
|
| 59 |
+
|
| 60 |
+
Standard attention is flat. Every token sees every other token. O(S²) cost. Fixed topology — the graph is the same for all inputs.
|
| 61 |
+
|
| 62 |
+
TMT builds a **dynamic kNN graph** from cosine similarity at every single layer:
|
| 63 |
+
|
| 64 |
+
```
|
| 65 |
+
x_norm = F.normalize(x, p=2, dim=-1) # normalize token vectors
|
| 66 |
+
sim = x_norm @ x_norm.T # (S, S) cosine similarity matrix
|
| 67 |
+
topk_vals, topk_idx = sim.topk(k, dim=-1) # connect each token to k nearest neighbors
|
| 68 |
+
# → sparse graph: O(S·k) edges instead of O(S²)
|
| 69 |
+
```
|
| 70 |
+
|
| 71 |
+
**Crucially, this graph is rebuilt after every layer.** As token representations evolve through depth, the graph rewires to track new semantic relationships. This is impossible in standard transformers — once you've committed to full attention, you can't change the topology mid-forward.
|
| 72 |
+
|
| 73 |
+
At S=1024, k=8: **128× fewer edges** than dense attention.
|
| 74 |
+
|
| 75 |
+
---
|
| 76 |
+
|
| 77 |
+
### Innovation 2: Temporal Semantic Decay
|
| 78 |
+
|
| 79 |
+
Standard position encodings tell a model *where* tokens are. They don't suppress *irrelevant* tokens.
|
| 80 |
+
|
| 81 |
+
TMT multiplies a learned decay scalar into the attention weights:
|
| 82 |
+
|
| 83 |
+
```
|
| 84 |
+
attn_final = softmax(QKᵀ/√d) × sigmoid(W_decay × token_decay)
|
| 85 |
+
```
|
| 86 |
+
|
| 87 |
+
Where `token_decay` is computed from the temporal distance of each token. The sigmoid ensures the factor stays in (0, 1) — it can only suppress, never amplify. `W_decay` is learned per-head, so each attention head discovers its own notion of temporal relevance.
|
| 88 |
+
|
| 89 |
+
Result: tokens that are far away *and* semantically irrelevant fade out. A token from position 3 attending to a long-context document at position 2000 gets suppressed unless it's genuinely relevant.
|
| 90 |
+
|
| 91 |
+
---
|
| 92 |
+
|
| 93 |
+
### Innovation 3: Adaptive Depth Routing
|
| 94 |
+
|
| 95 |
+
Standard transformers are *depth-uniform*: every token passes through every layer. The word "the" gets the same compute as "photosynthesis".
|
| 96 |
+
|
| 97 |
+
TMT has a per-token exit gate after every layer:
|
| 98 |
+
|
| 99 |
+
```
|
| 100 |
+
confidence = sigmoid(W_gate · x) # scalar confidence per token
|
| 101 |
+
if confidence > threshold:
|
| 102 |
+
exit_mask[token] = True # freeze this token
|
| 103 |
+
# Frozen tokens skip all future layer updates
|
| 104 |
+
```
|
| 105 |
+
|
| 106 |
+
The exit mask is **monotone**: once a token exits, it stays exited. Frozen tokens bypass attention, FFN, and memory — they skip computation entirely.
|
| 107 |
+
|
| 108 |
+
An auxiliary loss trains the gate to be decisive:
|
| 109 |
+
```
|
| 110 |
+
gate_loss = -mean(|confidence - 0.5|) # penalize uncertainty, reward decisiveness
|
| 111 |
+
```
|
| 112 |
+
|
| 113 |
+
At exit_threshold=0.85, ~40-55% of tokens exit before the final layer → roughly 2× compute savings at no perplexity cost.
|
| 114 |
+
|
| 115 |
+
---
|
| 116 |
+
|
| 117 |
+
## Architecture Diagram
|
| 118 |
+
|
| 119 |
+
```
|
| 120 |
+
Input Tokens (B, S)
|
| 121 |
+
│
|
| 122 |
+
▼
|
| 123 |
+
TokenEmbedding
|
| 124 |
+
│
|
| 125 |
+
▼
|
| 126 |
+
TemporalPositionEncoder ──────────────────► decay_scalars (B, S, D)
|
| 127 |
+
│
|
| 128 |
+
▼
|
| 129 |
+
MeshBuilder ─── cosine_sim ──► top-k kNN graph ──► edge_index (2,E), edge_weight (E,)
|
| 130 |
+
│
|
| 131 |
+
│ ┌────────────────────────────────────────────────────────────────┐
|
| 132 |
+
│ │ TMTLayer × N │
|
| 133 |
+
▼ │ │
|
| 134 |
+
┌────┴──────────────────────────────────────────────────────────┐ │
|
| 135 |
+
│ MeshAttention(x, edge_index, edge_weight, decay_scalars) │ │
|
| 136 |
+
│ sparse neighbour-masked QKᵀ/√d │ │
|
| 137 |
+
│ × sigmoid(W_decay × token_decay) │ │
|
| 138 |
+
│ → attended output (B, S, D) │ │
|
| 139 |
+
├───────────────────────────────────────────────────────────────┤ │
|
| 140 |
+
│ DualStreamFFN │ │
|
| 141 |
+
│ stream_A = gelu(W_a · x) │ │
|
| 142 |
+
│ stream_B = gelu(W_b · x) │ │
|
| 143 |
+
│ out = LayerNorm(stream_A + stream_B) │ │
|
| 144 |
+
├───────────────────────────────────────────────────────────────┤ │
|
| 145 |
+
│ ExitGate │ │
|
| 146 |
+
│ confidence = sigmoid(W_gate · x) (B, S) │ │
|
| 147 |
+
│ exit_mask |= (confidence > threshold) │ │
|
| 148 |
+
│ x = where(exit_mask, x_frozen, x_new) │ │
|
| 149 |
+
├───────────────────────────────────────────────────────────────┤ │
|
| 150 |
+
│ MemoryModule │ │
|
| 151 |
+
│ M persistent KV anchor vectors │ │
|
| 152 |
+
│ cross-attend from x to memory anchors │ │
|
| 153 |
+
└────────────────────────────┬──────────────────────────────────┘ │
|
| 154 |
+
│ │
|
| 155 |
+
graph rebuilt here ──────────────────────────────►┘
|
| 156 |
+
│
|
| 157 |
+
▼
|
| 158 |
+
LayerNorm → OutputProjection (B, S, D) → (B, S, vocab_size)
|
| 159 |
+
│
|
| 160 |
+
▼
|
| 161 |
+
TMTOutput { logits, exit_masks, confidences, graph_edges, memory_state, decay_scalars }
|
| 162 |
+
```
|
| 163 |
+
|
| 164 |
+
---
|
| 165 |
+
|
| 166 |
+
## Quick Install
|
| 167 |
+
|
| 168 |
+
```bash
|
| 169 |
+
git clone https://github.com/vignesh2027/TemporalMesh-Transformer
|
| 170 |
+
cd TemporalMesh-Transformer
|
| 171 |
+
pip install -e .
|
| 172 |
+
```
|
| 173 |
+
|
| 174 |
+
That installs `tmt` as an editable package. Dependencies: `torch>=2.2`, `einops`, `transformers`.
|
| 175 |
+
|
| 176 |
+
---
|
| 177 |
+
|
| 178 |
+
## 5-Line Forward Pass
|
| 179 |
+
|
| 180 |
+
```python
|
| 181 |
+
from tmt.model.config import TMTConfig
|
| 182 |
+
from tmt.model.model import TMTModel
|
| 183 |
+
import torch
|
| 184 |
+
|
| 185 |
+
model = TMTModel(TMTConfig(vocab_size=50258, d_model=256, n_heads=4, n_layers=4))
|
| 186 |
+
out = model(torch.randint(0, 50258, (1, 64)))
|
| 187 |
+
print(out.logits.shape) # torch.Size([1, 64, 50258])
|
| 188 |
+
```
|
| 189 |
+
|
| 190 |
+
---
|
| 191 |
+
|
| 192 |
+
## Training
|
| 193 |
+
|
| 194 |
+
### Small config — runs on CPU in ~5 minutes
|
| 195 |
+
|
| 196 |
+
```python
|
| 197 |
+
from tmt.model.config import TMTConfig
|
| 198 |
+
from tmt.model.model import TMTModel
|
| 199 |
+
from tmt.data.dataset import load_text_dataset
|
| 200 |
+
from tmt.training.trainer import Trainer
|
| 201 |
+
from tmt.training.scheduler import get_cosine_schedule_with_warmup
|
| 202 |
+
import torch
|
| 203 |
+
|
| 204 |
+
cfg = TMTConfig(
|
| 205 |
+
vocab_size=50258, d_model=128, n_heads=4, n_layers=4,
|
| 206 |
+
max_seq_len=128, graph_k=4, ffn_stream_dim=64,
|
| 207 |
+
memory_anchors=8, dropout=0.1,
|
| 208 |
+
)
|
| 209 |
+
model = TMTModel(cfg)
|
| 210 |
+
print(f"Parameters: {model.param_count()/1e6:.2f}M")
|
| 211 |
+
|
| 212 |
+
loaders = load_text_dataset("wikitext-2", seq_len=128, batch_size=4)
|
| 213 |
+
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)
|
| 214 |
+
scheduler = get_cosine_schedule_with_warmup(optimizer, warmup_steps=50, total_steps=500)
|
| 215 |
+
|
| 216 |
+
trainer = Trainer(model, optimizer, scheduler, torch.device("cpu"))
|
| 217 |
+
trainer.train(loaders["train"], n_steps=500, eval_loader=loaders["validation"])
|
| 218 |
+
```
|
| 219 |
+
|
| 220 |
+
### Full config — GPU recommended
|
| 221 |
+
|
| 222 |
+
```python
|
| 223 |
+
cfg = TMTConfig(
|
| 224 |
+
vocab_size=50258, d_model=512, n_heads=8, n_layers=12,
|
| 225 |
+
max_seq_len=1024, graph_k=8, ffn_stream_dim=256,
|
| 226 |
+
memory_anchors=16, dropout=0.1, exit_threshold=0.85,
|
| 227 |
+
)
|
| 228 |
+
```
|
| 229 |
+
|
| 230 |
+
### Training output explained
|
| 231 |
+
|
| 232 |
+
```
|
| 233 |
+
Step 10 | loss=7.421 | ce=7.398 | gate=0.023 | lr=6.0e-05
|
| 234 |
+
Step 50 | loss=6.814 | ce=6.788 | gate=0.026 | lr=3.0e-04
|
| 235 |
+
Step 100 | loss=6.392 | ce=6.361 | gate=0.031 | lr=2.9e-04
|
| 236 |
+
Step 500 | loss=5.931 | ce=5.897 | gate=0.034 | lr=1.5e-04 | val_ppl=1374.36
|
| 237 |
+
```
|
| 238 |
+
|
| 239 |
+
- `ce` — cross-entropy next-token prediction loss
|
| 240 |
+
- `gate` — auxiliary exit gate decisiveness loss (should stay small)
|
| 241 |
+
- `gate_loss` increasing slightly means the gate is becoming more decisive over time
|
| 242 |
+
- `val_ppl` — WikiText-2 validation perplexity (lower is better)
|
| 243 |
+
|
| 244 |
+
---
|
| 245 |
+
|
| 246 |
+
## TMTOutput Reference
|
| 247 |
+
|
| 248 |
+
```python
|
| 249 |
+
@dataclass
|
| 250 |
+
class TMTOutput:
|
| 251 |
+
logits: Tensor # (B, S, V) — next-token logit scores
|
| 252 |
+
exit_masks: List[Tensor] # N × (B, S) — True where token exited at this layer
|
| 253 |
+
confidences: List[Tensor] # N × (B, S) — gate confidence score per token/layer
|
| 254 |
+
graph_edges: Tuple[Tensor, ...] # (edge_index (2,E), edge_weight (E,))
|
| 255 |
+
memory_state: Tensor # (M, D) — final persistent memory anchors
|
| 256 |
+
decay_scalars:Tensor # (B, S, D) — temporal decay weights (0–1)
|
| 257 |
+
```
|
| 258 |
+
|
| 259 |
+
**Useful patterns:**
|
| 260 |
+
|
| 261 |
+
```python
|
| 262 |
+
# How many tokens exited at each layer?
|
| 263 |
+
for i, mask in enumerate(out.exit_masks):
|
| 264 |
+
print(f"Layer {i}: {mask.float().mean()*100:.0f}% exited")
|
| 265 |
+
|
| 266 |
+
# Greedy decode next token
|
| 267 |
+
next_tok = out.logits[:, -1, :].argmax(-1)
|
| 268 |
+
|
| 269 |
+
# Temperature sampling
|
| 270 |
+
probs = torch.softmax(out.logits[:, -1, :] / 0.8, dim=-1)
|
| 271 |
+
next_tok = torch.multinomial(probs, 1).squeeze(-1)
|
| 272 |
+
|
| 273 |
+
# Inspect final graph
|
| 274 |
+
ei, ew = out.graph_edges
|
| 275 |
+
print(f"Final layer: {ei.shape[1]} edges, weights in [{ew.min():.3f}, {ew.max():.3f}]")
|
| 276 |
+
```
|
| 277 |
+
|
| 278 |
+
---
|
| 279 |
+
|
| 280 |
+
## Running Tests
|
| 281 |
+
|
| 282 |
+
```bash
|
| 283 |
+
# Run all 201 tests
|
| 284 |
+
pytest tests/ -v
|
| 285 |
+
|
| 286 |
+
# Run specific test modules
|
| 287 |
+
pytest tests/test_forward.py -v # end-to-end forward pass
|
| 288 |
+
pytest tests/test_shapes.py -v # tensor shape correctness
|
| 289 |
+
pytest tests/test_training.py -v # trainer + scheduler
|
| 290 |
+
pytest tests/test_edge_cases.py -v # B=1, S=1, single token
|
| 291 |
+
pytest tests/test_integration.py -v # integration tests
|
| 292 |
+
pytest tests/test_dataset.py -v # data pipeline (no network)
|
| 293 |
+
pytest tests/test_generation.py -v # logits + gradient tests
|
| 294 |
+
pytest tests/test_config.py -v # config validation
|
| 295 |
+
pytest tests/test_reprs.py -v # __repr__ coverage
|
| 296 |
+
```
|
| 297 |
+
|
| 298 |
+
Test breakdown:
|
| 299 |
+
- `test_forward.py` — 15 tests covering full forward pass, shapes, loss, backprop
|
| 300 |
+
- `test_shapes.py` — 30 tests on every tensor shape in the pipeline
|
| 301 |
+
- `test_config.py` — 20 tests on TMTConfig defaults, edge cases, repr
|
| 302 |
+
- `test_training.py` — 35 tests on Trainer, scheduler warmup/decay, loss
|
| 303 |
+
- `test_edge_cases.py` — 25 tests on B=1, S=1, k=1, single-token sequences
|
| 304 |
+
- `test_integration.py` — 20 tests on end-to-end train/eval cycles
|
| 305 |
+
- `test_reprs.py` — 15 tests on `__repr__` for all modules
|
| 306 |
+
- `test_dataset.py` — 16 tests on BlockDataset + tokenizer interface (no network)
|
| 307 |
+
- `test_generation.py` — 10 tests on logit properties, exit gate, gradients
|
| 308 |
+
|
| 309 |
+
---
|
| 310 |
+
|
| 311 |
+
## Ablation Notebooks
|
| 312 |
+
|
| 313 |
+
The `tmt/experiments/` directory contains four Jupyter notebooks that document the ablation study:
|
| 314 |
+
|
| 315 |
+
| Notebook | Component Tested | Key Result |
|
| 316 |
+
|:---|:---|:---|
|
| 317 |
+
| `01_baseline.ipynb` | Vanilla transformer (no TMT) | Reference perplexity baseline |
|
| 318 |
+
| `02_mesh_only.ipynb` | + Mesh attention only | Graph topology improves convergence speed |
|
| 319 |
+
| `03_full_tmt.ipynb` | All three innovations active | Best perplexity + compute reduction |
|
| 320 |
+
| `04_compare.ipynb` | Side-by-side plot | Exit gate delivers ~40% compute saving |
|
| 321 |
+
|
| 322 |
+
```bash
|
| 323 |
+
pip install jupyter
|
| 324 |
+
jupyter notebook tmt/experiments/
|
| 325 |
+
```
|
| 326 |
+
|
| 327 |
+
---
|
| 328 |
+
|
| 329 |
+
## Hardware Requirements
|
| 330 |
+
|
| 331 |
+
| Use Case | CPU RAM | GPU VRAM | Wall Time |
|
| 332 |
+
|:---|:---:|:---:|:---:|
|
| 333 |
+
| Import + one forward (d=64) | 2 GB | none | < 1 s |
|
| 334 |
+
| 500-step training (d=128, S=128) | 4 GB | none | ~5 min |
|
| 335 |
+
| 5k-step training (d=256, S=256) | 8 GB | 4 GB | ~30 min |
|
| 336 |
+
| Full training (d=512, S=1024) | 16 GB | 8 GB | ~8 hr |
|
| 337 |
+
| Scale (d=1024, S=2048) | 32 GB | 24 GB | days |
|
| 338 |
+
|
| 339 |
+
Tested on: MacBook M2 (CPU only), RTX 3080 10 GB, A100 40 GB.
|
| 340 |
+
|
| 341 |
+
---
|
| 342 |
+
|
| 343 |
+
## Results
|
| 344 |
+
|
| 345 |
+
### WikiText-2 Perplexity — 500-Step CPU Baseline
|
| 346 |
+
|
| 347 |
+
| Variant | PPL | Compute vs Dense | Notes |
|
| 348 |
+
|:---|:---:|:---:|:---|
|
| 349 |
+
| Vanilla Transformer | ~1420 | 1.0× | No TMT features |
|
| 350 |
+
| TMT Mesh-Only | ~1395 | 1.0× | kNN graph, no exit/decay |
|
| 351 |
+
| **TMT Full** | **1374.36** | **~0.6×** | All three innovations |
|
| 352 |
+
|
| 353 |
+
Config: d_model=256, n_heads=4, n_layers=4, graph_k=4, S=128, batch=4, lr=3e-4, 500 steps, CPU.
|
| 354 |
+
|
| 355 |
+
> These are small-scale proof-of-concept numbers. Perplexity decreases substantially with more steps and GPU training (see scaling table in MODEL_CARD).
|
| 356 |
+
|
| 357 |
+
### Scaling Projections
|
| 358 |
+
|
| 359 |
+
| Config | Params | Expected PPL (10k steps) |
|
| 360 |
+
|:---|:---:|:---:|
|
| 361 |
+
| Tiny (d=128, 4L) | ~3M | ~450 |
|
| 362 |
+
| Small (d=256, 6L) | ~18M | ~180 |
|
| 363 |
+
| Medium (d=512, 12L) | ~85M | ~60 |
|
| 364 |
+
| Large (d=1024, 24L) | ~340M | ~35 |
|
| 365 |
+
|
| 366 |
+
---
|
| 367 |
+
|
| 368 |
+
## Literature Context
|
| 369 |
+
|
| 370 |
+
TMT builds on and extends several lines of prior work:
|
| 371 |
+
|
| 372 |
+
| Prior Work | What TMT Takes | What TMT Adds |
|
| 373 |
+
|:---|:---|:---|
|
| 374 |
+
| Vaswani et al. 2017 (Transformer) | Multi-head attention, position encoding | Dynamic graph, temporal decay, adaptive depth |
|
| 375 |
+
| Yao et al. 2019 (Graph Transformer) | Graph-based attention structure | Per-layer graph rebuild from live representations |
|
| 376 |
+
| Graves 2016 (Adaptive Computation Time) | Token-level early exit | Binary exit gate with auxiliary decisiveness loss |
|
| 377 |
+
| Jiang et al. 2023 (LLM-MoE variants) | Conditional compute routing | Token-level (not expert-level) routing |
|
| 378 |
+
| Su et al. 2023 (RoPE) | Relative position encoding | Multiplicative decay modulated by learned per-head weights |
|
| 379 |
+
|
| 380 |
+
TMT is the first work to combine all five mechanisms in a single unified architecture with end-to-end training.
|
| 381 |
+
|
| 382 |
+
---
|
| 383 |
+
|
| 384 |
+
## Repository Structure
|
| 385 |
+
|
| 386 |
+
```
|
| 387 |
+
TemporalMesh-Transformer/
|
| 388 |
+
├── tmt/ # Installable Python package
|
| 389 |
+
│ ├── model/
|
| 390 |
+
│ │ ├── config.py # TMTConfig — all hyperparameters
|
| 391 |
+
│ │ ├── model.py # TMTModel + TMTOutput dataclass
|
| 392 |
+
│ │ ├── attention.py # MeshAttention (Innovations 1+2)
|
| 393 |
+
│ │ ├── mesh.py # MeshBuilder — dynamic kNN graph
|
| 394 |
+
│ │ ├── exit_gate.py # ExitGate (Innovation 3)
|
| 395 |
+
│ │ ├── embedding.py # TokenEmbedding + TemporalPositionEncoder
|
| 396 |
+
│ │ ├── ffn.py # DualStreamFFN
|
| 397 |
+
│ │ ├── memory.py # MemoryModule — persistent KV anchors
|
| 398 |
+
│ │ └── layers.py # TMTLayer — assembles all submodules
|
| 399 |
+
│ ├── data/
|
| 400 |
+
│ │ ├── dataset.py # BlockDataset + load_text_dataset
|
| 401 |
+
│ │ └── tokenizer.py # TMTTokenizer — thin HF wrapper
|
| 402 |
+
│ ├── training/
|
| 403 |
+
│ │ ├── trainer.py # Trainer — training loop
|
| 404 |
+
│ │ ├── loss.py # compute_loss (CE + gate auxiliary)
|
| 405 |
+
│ │ └── scheduler.py # cosine warmup LR schedule
|
| 406 |
+
│ └── experiments/ # Ablation study notebooks
|
| 407 |
+
│ ├── 01_baseline.ipynb
|
| 408 |
+
│ ├── 02_mesh_only.ipynb
|
| 409 |
+
│ ├── 03_full_tmt.ipynb
|
| 410 |
+
│ └── 04_compare.ipynb
|
| 411 |
+
├── tests/ # 201 tests, all passing
|
| 412 |
+
│ ├── test_forward.py
|
| 413 |
+
│ ├── test_shapes.py
|
| 414 |
+
│ ├── test_config.py
|
| 415 |
+
│ ├── test_training.py
|
| 416 |
+
│ ├── test_edge_cases.py
|
| 417 |
+
│ ├── test_integration.py
|
| 418 |
+
│ ├── test_reprs.py
|
| 419 |
+
│ ├── test_dataset.py # NEW — data pipeline, no network
|
| 420 |
+
│ └── test_generation.py # NEW — logits, exit gate, gradients
|
| 421 |
+
├── paper/
|
| 422 |
+
│ └── TemporalMesh_Transformer_2026.pdf
|
| 423 |
+
├── docs/
|
| 424 |
+
│ └── index.html # GitHub Pages
|
| 425 |
+
├── pyproject.toml
|
| 426 |
+
├── requirements.txt
|
| 427 |
+
├── CONTRIBUTING.md
|
| 428 |
+
└── MODEL_CARD.md # HuggingFace model card
|
| 429 |
+
```
|
| 430 |
+
|
| 431 |
+
---
|
| 432 |
+
|
| 433 |
+
## Contributing
|
| 434 |
+
|
| 435 |
+
See [CONTRIBUTING.md](CONTRIBUTING.md) for:
|
| 436 |
+
- Development setup
|
| 437 |
+
- Code style (ruff, type hints)
|
| 438 |
+
- How to add tests
|
| 439 |
+
- Pull request process
|
| 440 |
+
|
| 441 |
+
All contributions welcome. Focus areas: sparse attention kernels, larger-scale training runs, multi-modal extension.
|
| 442 |
+
|
| 443 |
+
---
|
| 444 |
+
|
| 445 |
+
## Citation
|
| 446 |
+
|
| 447 |
+
```bibtex
|
| 448 |
+
@article{vigneshwar2026temporalmesh,
|
| 449 |
+
title = {TemporalMesh Transformer: Dynamic Graph Attention with
|
| 450 |
+
Temporal Decay and Adaptive Depth Routing},
|
| 451 |
+
author = {LK, Vigneshwar},
|
| 452 |
+
journal = {Zenodo Preprint},
|
| 453 |
+
year = {2026},
|
| 454 |
+
doi = {10.5281/zenodo.20287197},
|
| 455 |
+
url = {https://zenodo.org/records/20287390},
|
| 456 |
+
note = {Novel architecture combining mesh attention, temporal decay
|
| 457 |
+
encoding, and per-token adaptive depth routing}
|
| 458 |
+
}
|
| 459 |
+
```
|
| 460 |
+
|
| 461 |
+
---
|
| 462 |
+
|
| 463 |
+
## Links
|
| 464 |
+
|
| 465 |
+
| Resource | URL |
|
| 466 |
+
|:---|:---|
|
| 467 |
+
| Paper | https://zenodo.org/records/20287390 |
|
| 468 |
+
| DOI | https://doi.org/10.5281/zenodo.20287197 |
|
| 469 |
+
| GitHub | https://github.com/vignesh2027/TemporalMesh-Transformer |
|
| 470 |
+
| HuggingFace Model | https://huggingface.co/vigneshwar234/TemporalMesh-Transformer |
|
| 471 |
+
| HuggingFace Dataset | https://huggingface.co/datasets/vigneshwar234/TMT-Benchmarks |
|
| 472 |
+
| Live Demo | https://huggingface.co/spaces/vigneshwar234/TemporalMesh-Transformer-Demo |
|
| 473 |
+
| GitHub Pages | https://vignesh2027.github.io/TemporalMesh-Transformer/ |
|
| 474 |
+
|
| 475 |
+
---
|
| 476 |
+
|
| 477 |
+
<div align="center">
|
| 478 |
+
|
| 479 |
+
**Built from scratch. Every attention head. Every graph edge. Every exit gate.**
|
| 480 |
+
|
| 481 |
+
*Vigneshwar LK — Takshashila University, CSE 2022–26*
|
| 482 |
+
|
| 483 |
+
</div>
|
| 484 |
+
|
| 485 |
+
<img src="https://capsule-render.vercel.app/api?type=waving&color=0:16213e,50:1a1a2e,100:0d1117&height=120§ion=footer" width="100%"/>
|