File size: 6,625 Bytes
2309cb1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
941a017
 
2309cb1
941a017
 
 
2309cb1
941a017
2309cb1
 
 
941a017
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2309cb1
 
 
941a017
2309cb1
941a017
 
 
2309cb1
 
 
 
 
4a811fd
 
 
 
2309cb1
 
 
 
 
941a017
 
2309cb1
 
 
 
 
941a017
 
 
 
 
 
2309cb1
941a017
2309cb1
941a017
4a811fd
941a017
2309cb1
941a017
 
 
2309cb1
941a017
2309cb1
 
 
941a017
 
 
 
 
 
 
2309cb1
 
 
941a017
2309cb1
 
 
 
941a017
2309cb1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
941a017
2309cb1
941a017
2309cb1
941a017
 
2309cb1
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
---
license: apache-2.0
language:
- en
- fr
- code
pipeline_tag: text-generation
tags:
- code
- coding-agent
- small-language-model
- trained-from-scratch
- custom-architecture
- tool-use
- opencode
model-index:
- name: Tetu-Code-2
  results:
  - task:
      type: text-generation
    dataset:
      name: HumanEval+
      type: evalplus/humanevalplus
    metrics:
    - type: pass@1
      name: pass@1 (greedy, chat prompt)
      value: 15.9
  - task:
      type: text-generation
    dataset:
      name: MBPP+
      type: evalplus/mbppplus
    metrics:
    - type: pass@1
      name: pass@1 (greedy, chat prompt)
      value: 30.7
---

# Têtu Code 2

Têtu Code 2 is a small coding agent trained from scratch. It is designed to run in the terminal through
[OpenCode](https://opencode.ai), where it reads and edits files, runs commands and reports what it did.

The model was trained from random initialization; no pretrained weights were used. Qwen3.6-35B-A3B (Apache 2.0)
served as a teacher for the agent demonstrations, and every demonstration was checked by a program before training.
Training ran on a single RTX 5070 (12 GB) plus about 30 hours of cloud GPU time.

## Model details

| | |
|---|---|
| Developer | tetu-AI |
| Architecture | Decoder-only transformer (custom) |
| Parameters | 1.15B total, 475M active per token (the rest is a lookup memory with negligible compute cost) |
| Context length | 65,536 tokens |
| Languages | Python first, plus HTML, CSS, JavaScript and others; instructions in English or French |
| Vocabulary | 32,768 tokens |
| Weights | bfloat16 (`model.safetensors`) |
| License | Apache 2.0 |

Capabilities:

- tool use through OpenCode: read, write, edit, bash, glob, grep;
- a short written reasoning between `<|think|>` and `<|/think|>` before each action;
- a per-token confidence head that estimates the probability of an error;
- an OpenAI-compatible server (`tetu_code.serve`) for the OpenCode 1.x and 2.x tool formats, which keeps the KV
  cache across turns and stops repeated identical tool calls.

## Training

| Stage | Data |
|---|---|
| Pre-training | 7.08B tokens, mostly code (Python first, then web and other languages), with code-related web pages, French and English web text, and math |
| Annealing | The final quarter adds teacher projects, exercises whose solutions pass their tests, agent trajectories and commits |
| Agent fine-tuning | 2,093 verified teacher runs on multi-file projects (CLI tools, games, websites, APIs, bug fixes, tests), plus DAgger states in which the model acts and the teacher supplies the correct action |

## Evaluation

### Code generation (EvalPlus)

![HumanEval+ pass@1: Têtu Code 2 15.9 %, above SantaCoder-1.1B and StarCoderBase-1B](assets/humaneval-plus.png)

![MBPP+ pass@1: Têtu Code 2 30.7 %, above Têtu Code 1](assets/mbpp-plus.png)

| Model | Size | Training tokens | HumanEval+ | MBPP+ |
|---|---|---|---|---|
| DeepSeek-Coder-1.3B-instruct | 1.3B | 2T | 60.4 | 54.8 |
| DeepSeek-Coder-1.3B-base | 1.3B | 2T | 25.6 | 47.9 |
| Qwen2.5-Coder-0.5B (base) | 0.5B | 5.5T | 23.8 | 47.1 |
| Têtu Code 1 (previous model) | 193M active | 7B | 18.3 | 27.5 |
| **Têtu Code 2** | **475M active** | **7.08B** | **15.9** | **30.7** |
| SantaCoder-1.1B | 1.1B | 236B | 14.0 | – |
| StarCoderBase-1B | 1B | 1T | 12.2 | – |
| InCoder-1.3B | 1.3B | ~60B | 11.0 | – |
| CodeGen2-1B | 1B | ~400B | 9.1 | – |

Têtu Code results use EvalPlus 0.3.1 with greedy decoding, one sample per problem and a chat prompt. A repeated run
gave HumanEval+ 14.6 and MBPP+ 30.7, so differences of about 1.5 points are within noise. Other results are taken from
the official EvalPlus leaderboard (23 Oct. 2026); the Qwen2.5-Coder figures are from its authors.

Têtu Code 2 scores above the 2023 1B code models (SantaCoder, StarCoderBase-1B) after 30 to 140 times fewer training
tokens, and below DeepSeek-Coder-1.3B and Qwen2.5-Coder-0.5B, which were trained on 280 to 780 times more tokens.

### Agent tasks

![40 held-out projects in OpenCode: teacher 29/40, Têtu Code 2 8/40, Têtu Code 1 0/40](assets/agent.png)

| | Têtu Code 1 | Têtu Code 2 | Teacher (Qwen3.6-35B-A3B) |
|---|---|---|---|
| 40 held-out multi-file projects in OpenCode 2.1.4 (hidden tests) | 0/40 | **8/40** | 29/40 |
| Single-function agent benchmark (OpenCode 1.19.4) | 46.5 % | 28.5 % | 85.5 % |
| Python validation loss (pre-training) | 0.821 | **0.777** | – |

Throughput on an RTX 5070: about 143 tokens/s at 2k context and 71 tokens/s at 16k.

## Limitations

- As a small model, it often starts from incorrect logic or uses undefined variables.
- When its own tests fail, it tends to re-run commands rather than fix the code.
- In multi-turn conversations it can miss a correction or a follow-up request.
- It is tuned as an agent: in the plain chat script (`chat.py`) it often still tries to call tools. Use it through
  OpenCode.
- It sometimes writes files outside the project directory; the bundled server maps such paths back into the project.
- Run it in a sandbox or a disposable directory, not unattended on important code.

## Usage

The architecture is not supported by `transformers`; use the bundled inference code.

```bash
git clone https://huggingface.co/tetu-AI/Tetu-Code-2 && cd Tetu-Code-2
pip install -r requirements.txt
python chat.py                                   # terminal chat without tools (--cpu without a GPU)
python -m tetu_code.serve --model . --port 8070  # OpenAI-compatible server for OpenCode
```

`opencode.json`:

```json
"provider": {"tetu": {"npm": "@ai-sdk/openai-compatible", "name": "Têtu Code",
  "options": {"baseURL": "http://127.0.0.1:8070/v1", "apiKey": "x"},
  "models": {"tetu-code": {"name": "Têtu Code", "tool_call": true,
                           "limit": {"context": 65536, "output": 4096}}}}}
```

From Python:

```python
import torch
from tetu_code.generate import Decoder, load
model, tok = load(".", device="cuda")          # or device="cpu", dtype=torch.float32
dec = Decoder(model, tok, max_len=16384)
```

## License

Apache 2.0.

Teachers (both Apache 2.0): Qwen3.6-35B-A3B for the agent runs, Qwen2.5-7B-Instruct for general conversation.

Training data: github-code-clean (permissively licensed files only), FineWeb-Edu, FineWeb2-HQ and FineMath (ODC-By),
OpenCodeInstruct, Nemotron-SWE trajectories and CommitPackFT. Check the terms of each dataset before commercial use.

## Citation

```bibtex
@misc{tetucode2026,
  title  = {Têtu Code 2: a small coding agent trained from scratch},
  author = {tetu-AI},
  year   = {2026},
  url    = {https://huggingface.co/tetu-AI/Tetu-Code-2}
}
```