Neithanvarvas commited on
Commit
00985f4
·
verified ·
1 Parent(s): a8c16c3

[DOCS] Rename to Character Based Transformer (CBT) and remove pipeline_tag

Browse files
Files changed (1) hide show
  1. README.md +9 -9
README.md CHANGED
@@ -14,16 +14,15 @@ datasets:
14
  - synthetic
15
  metrics:
16
  - accuracy
17
- pipeline_tag: text-generation
18
  ---
19
 
20
- # ScaledGPT (~18.95M): Recurrent 2D-RoPE Architecture for Algorithmic Length Extrapolation
21
 
22
  ## 1. Model Overview
23
 
24
- **ScaledGPT** is an autoregressive Transformer model engineered to overcome the empirical limits of standard self-attention on sequential state machines and multi-tool agent execution.
25
 
26
- By combining **Decoupled 2D Rotary Position Embeddings (2D-RoPE)** with **Weight-Tied Recurrent Depth (R = 3)**, ScaledGPT achieves:
27
  - **100.0% Exact Match Accuracy** on 1 to 50 digit column arithmetic length extrapolation (including 50-digit cascading carry 99...99 + 1).
28
  - **100.0% Multi-Tool Routing Accuracy** across 9 tool schemas (single-tool, multi-tool chaining, nested JSON schemas, and error recovery).
29
  - **Dual NVIDIA T4 GPU Efficiency**: Native Turing sm_75 FP16 AMP and SDPA memory-efficient attention.
@@ -34,6 +33,7 @@ By combining **Decoupled 2D Rotary Position Embeddings (2D-RoPE)** with **Weight
34
 
35
  | Parameter | Specification |
36
  | :--- | :--- |
 
37
  | **Total Parameters** | 18,944,512 (~18.95M) |
38
  | **Non-Embedding Parameters** | 18,887,680 (~18.89M) |
39
  | **Physical Layers (L)** | 6 |
@@ -67,11 +67,11 @@ By combining **Decoupled 2D Rotary Position Embeddings (2D-RoPE)** with **Weight
67
 
68
  ```python
69
  import torch
70
- from model import ScaledGPT, ScaledGPTConfig
71
- from tokenizer import ScaledTokenizer
72
 
73
  # 1. Load configuration and model
74
- config = ScaledGPTConfig(
75
  block_size=1024,
76
  vocab_size=115,
77
  n_layer=6,
@@ -80,7 +80,7 @@ config = ScaledGPTConfig(
80
  n_recur=3,
81
  pos_emb_type="rope_2d"
82
  )
83
- model = ScaledGPT(config)
84
 
85
  # 2. Load weights
86
  state_dict = torch.load("champion_final.pt", map_location="cpu")
@@ -88,7 +88,7 @@ model.load_state_dict(state_dict)
88
  model.eval()
89
 
90
  # 3. Initialize tokenizer
91
- tokenizer = ScaledTokenizer()
92
  ```
93
 
94
  ### Interactive CLI
 
14
  - synthetic
15
  metrics:
16
  - accuracy
 
17
  ---
18
 
19
+ # Character Based Transformer (CBT): Recurrent 2D-RoPE Architecture for Algorithmic Length Extrapolation
20
 
21
  ## 1. Model Overview
22
 
23
+ **Character Based Transformer (CBT)** is an autoregressive Transformer model (~18.95M parameters) engineered to overcome the empirical limits of standard self-attention on sequential state machines and multi-tool agent execution.
24
 
25
+ By combining **Decoupled 2D Rotary Position Embeddings (2D-RoPE)** with **Weight-Tied Recurrent Depth (R = 3)**, CBT achieves:
26
  - **100.0% Exact Match Accuracy** on 1 to 50 digit column arithmetic length extrapolation (including 50-digit cascading carry 99...99 + 1).
27
  - **100.0% Multi-Tool Routing Accuracy** across 9 tool schemas (single-tool, multi-tool chaining, nested JSON schemas, and error recovery).
28
  - **Dual NVIDIA T4 GPU Efficiency**: Native Turing sm_75 FP16 AMP and SDPA memory-efficient attention.
 
33
 
34
  | Parameter | Specification |
35
  | :--- | :--- |
36
+ | **Model Name** | Character Based Transformer (CBT) |
37
  | **Total Parameters** | 18,944,512 (~18.95M) |
38
  | **Non-Embedding Parameters** | 18,887,680 (~18.89M) |
39
  | **Physical Layers (L)** | 6 |
 
67
 
68
  ```python
69
  import torch
70
+ from model import ScaledGPT as CBT, ScaledGPTConfig as CBTConfig
71
+ from tokenizer import ScaledTokenizer as CBTTokenizer
72
 
73
  # 1. Load configuration and model
74
+ config = CBTConfig(
75
  block_size=1024,
76
  vocab_size=115,
77
  n_layer=6,
 
80
  n_recur=3,
81
  pos_emb_type="rope_2d"
82
  )
83
+ model = CBT(config)
84
 
85
  # 2. Load weights
86
  state_dict = torch.load("champion_final.pt", map_location="cpu")
 
88
  model.eval()
89
 
90
  # 3. Initialize tokenizer
91
+ tokenizer = CBTTokenizer()
92
  ```
93
 
94
  ### Interactive CLI