[DOCS] Rename to Character Based Transformer (CBT) and remove pipeline_tag
Browse files
README.md
CHANGED
|
@@ -14,16 +14,15 @@ datasets:
|
|
| 14 |
- synthetic
|
| 15 |
metrics:
|
| 16 |
- accuracy
|
| 17 |
-
pipeline_tag: text-generation
|
| 18 |
---
|
| 19 |
|
| 20 |
-
#
|
| 21 |
|
| 22 |
## 1. Model Overview
|
| 23 |
|
| 24 |
-
**
|
| 25 |
|
| 26 |
-
By combining **Decoupled 2D Rotary Position Embeddings (2D-RoPE)** with **Weight-Tied Recurrent Depth (R = 3)**,
|
| 27 |
- **100.0% Exact Match Accuracy** on 1 to 50 digit column arithmetic length extrapolation (including 50-digit cascading carry 99...99 + 1).
|
| 28 |
- **100.0% Multi-Tool Routing Accuracy** across 9 tool schemas (single-tool, multi-tool chaining, nested JSON schemas, and error recovery).
|
| 29 |
- **Dual NVIDIA T4 GPU Efficiency**: Native Turing sm_75 FP16 AMP and SDPA memory-efficient attention.
|
|
@@ -34,6 +33,7 @@ By combining **Decoupled 2D Rotary Position Embeddings (2D-RoPE)** with **Weight
|
|
| 34 |
|
| 35 |
| Parameter | Specification |
|
| 36 |
| :--- | :--- |
|
|
|
|
| 37 |
| **Total Parameters** | 18,944,512 (~18.95M) |
|
| 38 |
| **Non-Embedding Parameters** | 18,887,680 (~18.89M) |
|
| 39 |
| **Physical Layers (L)** | 6 |
|
|
@@ -67,11 +67,11 @@ By combining **Decoupled 2D Rotary Position Embeddings (2D-RoPE)** with **Weight
|
|
| 67 |
|
| 68 |
```python
|
| 69 |
import torch
|
| 70 |
-
from model import ScaledGPT, ScaledGPTConfig
|
| 71 |
-
from tokenizer import ScaledTokenizer
|
| 72 |
|
| 73 |
# 1. Load configuration and model
|
| 74 |
-
config =
|
| 75 |
block_size=1024,
|
| 76 |
vocab_size=115,
|
| 77 |
n_layer=6,
|
|
@@ -80,7 +80,7 @@ config = ScaledGPTConfig(
|
|
| 80 |
n_recur=3,
|
| 81 |
pos_emb_type="rope_2d"
|
| 82 |
)
|
| 83 |
-
model =
|
| 84 |
|
| 85 |
# 2. Load weights
|
| 86 |
state_dict = torch.load("champion_final.pt", map_location="cpu")
|
|
@@ -88,7 +88,7 @@ model.load_state_dict(state_dict)
|
|
| 88 |
model.eval()
|
| 89 |
|
| 90 |
# 3. Initialize tokenizer
|
| 91 |
-
tokenizer =
|
| 92 |
```
|
| 93 |
|
| 94 |
### Interactive CLI
|
|
|
|
| 14 |
- synthetic
|
| 15 |
metrics:
|
| 16 |
- accuracy
|
|
|
|
| 17 |
---
|
| 18 |
|
| 19 |
+
# Character Based Transformer (CBT): Recurrent 2D-RoPE Architecture for Algorithmic Length Extrapolation
|
| 20 |
|
| 21 |
## 1. Model Overview
|
| 22 |
|
| 23 |
+
**Character Based Transformer (CBT)** is an autoregressive Transformer model (~18.95M parameters) engineered to overcome the empirical limits of standard self-attention on sequential state machines and multi-tool agent execution.
|
| 24 |
|
| 25 |
+
By combining **Decoupled 2D Rotary Position Embeddings (2D-RoPE)** with **Weight-Tied Recurrent Depth (R = 3)**, CBT achieves:
|
| 26 |
- **100.0% Exact Match Accuracy** on 1 to 50 digit column arithmetic length extrapolation (including 50-digit cascading carry 99...99 + 1).
|
| 27 |
- **100.0% Multi-Tool Routing Accuracy** across 9 tool schemas (single-tool, multi-tool chaining, nested JSON schemas, and error recovery).
|
| 28 |
- **Dual NVIDIA T4 GPU Efficiency**: Native Turing sm_75 FP16 AMP and SDPA memory-efficient attention.
|
|
|
|
| 33 |
|
| 34 |
| Parameter | Specification |
|
| 35 |
| :--- | :--- |
|
| 36 |
+
| **Model Name** | Character Based Transformer (CBT) |
|
| 37 |
| **Total Parameters** | 18,944,512 (~18.95M) |
|
| 38 |
| **Non-Embedding Parameters** | 18,887,680 (~18.89M) |
|
| 39 |
| **Physical Layers (L)** | 6 |
|
|
|
|
| 67 |
|
| 68 |
```python
|
| 69 |
import torch
|
| 70 |
+
from model import ScaledGPT as CBT, ScaledGPTConfig as CBTConfig
|
| 71 |
+
from tokenizer import ScaledTokenizer as CBTTokenizer
|
| 72 |
|
| 73 |
# 1. Load configuration and model
|
| 74 |
+
config = CBTConfig(
|
| 75 |
block_size=1024,
|
| 76 |
vocab_size=115,
|
| 77 |
n_layer=6,
|
|
|
|
| 80 |
n_recur=3,
|
| 81 |
pos_emb_type="rope_2d"
|
| 82 |
)
|
| 83 |
+
model = CBT(config)
|
| 84 |
|
| 85 |
# 2. Load weights
|
| 86 |
state_dict = torch.load("champion_final.pt", map_location="cpu")
|
|
|
|
| 88 |
model.eval()
|
| 89 |
|
| 90 |
# 3. Initialize tokenizer
|
| 91 |
+
tokenizer = CBTTokenizer()
|
| 92 |
```
|
| 93 |
|
| 94 |
### Interactive CLI
|