--- language: - en license: mit library_name: pytorch pipeline_tag: text-generation tags: - arithmetic - length-generalization - character-level - rope - 2d-rope - tool-use datasets: - synthetic metrics: - accuracy --- # CBT-18.95M **A 19M-parameter character-level Transformer that adds numbers longer than any it was trained on.** CBT (Character-Based Transformer) gives every token a two-dimensional position: *which column* it belongs to and *which step* inside that column. With that layout, a model trained on sums of up to 25 digits keeps adding correctly at 26, 40 and 55 digits. An identical twin with ordinary 1D positions gets every one of those wrong. The same model also does simple tool calling (weather, search, calculator, unit conversion, ...) in a small chat format. [Paper (PDF)](paper/CBT-18.95M.pdf) · [Paper (Markdown)](paper/CBT-18.95M.md) · [Code on GitHub](https://github.com/NeithanT/CBT) --- ## Try it ```bash pip install -U huggingface_hub hf download Neithanvarvas/CBT --local-dir CBT && cd CBT pip install -r requirements.txt python chat.py ``` `git clone https://huggingface.co/Neithanvarvas/CBT` also works; without git-lfs the weights are fetched on first run. ```text > 123456789+987654321 scratchpad : (0+9+1)=10(1+8+2)=11(1+7+3)=11(1+6+4)=11 ... (1+0+0)=01 answer : 1111111110 [PASS] > What is the weather in Tokyo? think : The user requests current weather information. I will query the get_weather tool. call : get_weather(city="Tokyo") result : {"temp": 22, "condition": "Sunny", "city": "Tokyo"} answer : The weather in Tokyo is currently Sunny with a temperature of 22 degrees Celsius. ``` Runs on a CPU; no GPU needed. Pass `--online` to let the search tool query Wikipedia, or `--raw` to see the token stream. ### From Python ```python from cbt import load, add, chat model, tok = load() # uses this folder, or downloads from the Hub if run elsewhere add(model, tok, 31415926535897932384626433832795, 27182818284590452353602874713527)["answer"] # '58598744820488384738229308546322' chat(model, tok, "Convert 10 km to miles")["answer"] ``` `chat` runs one user turn and executes at most one tool call. Tools are offline stand-ins in [`cbt/tools.py`](cbt/tools.py); pass `tool_executor=your_function` to connect real ones. ### Compare with the 1D-RoPE twin The control model from the results below ships in [`twin_1d/`](twin_1d/). Give both the same long sum: ```python twin, _ = load(variant="twin_1d") a, b = 31415926535897932384626433832795, 27182818284590452353602874713527 # 32 digits, unseen add(model, tok, a, b)["correct"] # True add(twin, tok, a, b)["correct"] # False ``` From the terminal: `python chat.py --variant twin_1d`. --- ## Results Greedy decoding, 30 random problems per length. Training covered sums of 1 to 25 digits. ### Addition: CBT vs. its 1D-RoPE twin The twin has the same architecture, data and number of training steps. Only the position encoding differs. Its weights are in [`twin_1d/`](twin_1d/). | Operand digits | Seen in training? | CBT (2D) | 1D twin | | :--- | :---: | :---: | :---: | | 4, 8, 12, 14, 16 | yes | 150 / 150 | 149 / 150 | | 20, 25 | yes | 60 / 60 | 57 / 60 | | 26 | no | **30 / 30** | 0 / 30 | | 30, 35, 40, 45, 50, 55 | no | **180 / 180** | 0 / 180 | | 60 | no | 25 / 30 | 0 / 30 | | 70 | no | 19 / 30 | 0 / 30 | | **All unseen (26-70)** | no | **254 / 270** | **0 / 270** | ### Tool use: exact tool and arguments | Test set | Prompts | CBT (2D) | 1D twin | | :--- | :---: | :---: | :---: | | Rephrased requests | 26 | 26 | 25 | | Out-of-distribution entities | 18 | 18 | 18 | | Entities never seen in training | 30 | 30 | 30 | Both models do equally well here, so tool use comes from the training data, not from the 2D layout. The fine-tuning data was broadened after an earlier model failed the first two sets, so the unseen-entity set is the fairer test. All numbers come from single training runs with small test sets. Raw per-problem results are in [`evaluation/`](evaluation/); limitations are discussed in the paper. --- ## How it works Each token gets a coordinate (p, tau): - **Arithmetic:** p is the decimal place (units, tens, ...) and tau is the step within that column of the scratchpad. - **Chat:** p is the dialogue segment (user, think, call, result, answer) and tau is the character position inside it. Half of each attention head is rotated by p and half by tau (Decoupled 2D-RoPE). Aligning digits of the same place becomes one fixed relation, "same p", which does not change as numbers get longer, so the model learns it once. During training, p is randomly shifted by up to 45, so the model has seen place values up to 70. | | | | :--- | :--- | | Parameters | 18.94M (18.89M non-embedding) | | Layers | 6, each applied 3 times with shared weights (effective depth 18) | | Width / heads | 512 / 8 (head dim 64) | | Context | 1024 characters | | Vocabulary | 111: 97 ASCII characters (letters, digits, punctuation, space, tab, newline) + 14 control tags | | Positions | Decoupled 2D-RoPE, coordinate shift k = 45 | | Training | Pretraining, then several supervised fine-tuning rounds (23,927 steps in the last) | ### Prompt formats ```text Addition: 123+456=(0+3+6)=09(0+2+5)=07(0+1+4)=05=579 Chat: question...tool(arg="..."){json}... ``` Scratchpad steps read `(carry + digit_a + digit_b) = carry_out digit`, from the units column upward. --- ## Limitations - Addition of non-negative integers is the only arithmetic it was trained on. Accuracy drops beyond 55 digits, and the 1024-character context caps sums at roughly 75 digits. - Replies can drift into garbled text after the tool result (for example, "What is the weather in Paris?" ends with misspelled filler). - The chat ability is narrow: it was trained on synthetic tool-calling dialogues with a fixed set of tools. It is not a general assistant and has little world knowledge. - Character-level, English-only, ASCII-only. Other characters become ``. --- ## Files ```text README.md this card config.json model hyperparameters model.safetensors weights twin_1d/ the matched 1D-RoPE control (config.json, model.safetensors) cbt/ model, tokenizer, tool stubs, load/add/chat helpers chat.py terminal demo requirements.txt paper/ the paper (PDF and Markdown) evaluation/ per-problem results behind the tables above ``` ## Citation ```bibtex @article{vargas2026cbt, title = {From Digits to Characters: Two-Dimensional Rotary Coordinates for Length Generalization in a Compact Character-Level Transformer}, author = {Vargas Vargas, Neithan}, year = {2026} } ```