--- license: apache-2.0 language: - en - fr - code pipeline_tag: text-generation tags: - code - coding-agent - small-language-model - trained-from-scratch - custom-architecture - tool-use - opencode model-index: - name: Tetu-Code-2 results: - task: type: text-generation dataset: name: HumanEval+ type: evalplus/humanevalplus metrics: - type: pass@1 name: pass@1 (greedy, chat prompt) value: 15.9 - task: type: text-generation dataset: name: MBPP+ type: evalplus/mbppplus metrics: - type: pass@1 name: pass@1 (greedy, chat prompt) value: 30.7 --- # Têtu Code 2 Têtu Code 2 is a small coding agent trained from scratch. It is designed to run in the terminal through [OpenCode](https://opencode.ai), where it reads and edits files, runs commands and reports what it did. The model was trained from random initialization; no pretrained weights were used. Qwen3.6-35B-A3B (Apache 2.0) served as a teacher for the agent demonstrations, and every demonstration was checked by a program before training. Training ran on a single RTX 5070 (12 GB) plus about 30 hours of cloud GPU time. ## Model details | | | |---|---| | Developer | tetu-AI | | Architecture | Decoder-only transformer (custom) | | Parameters | 1.15B total, 475M active per token (the rest is a lookup memory with negligible compute cost) | | Context length | 65,536 tokens | | Languages | Python first, plus HTML, CSS, JavaScript and others; instructions in English or French | | Vocabulary | 32,768 tokens | | Weights | bfloat16 (`model.safetensors`) | | License | Apache 2.0 | Capabilities: - tool use through OpenCode: read, write, edit, bash, glob, grep; - a short written reasoning between `<|think|>` and `<|/think|>` before each action; - a per-token confidence head that estimates the probability of an error; - an OpenAI-compatible server (`tetu_code.serve`) for the OpenCode 1.x and 2.x tool formats, which keeps the KV cache across turns and stops repeated identical tool calls. ## Training | Stage | Data | |---|---| | Pre-training | 7.08B tokens, mostly code (Python first, then web and other languages), with code-related web pages, French and English web text, and math | | Annealing | The final quarter adds teacher projects, exercises whose solutions pass their tests, agent trajectories and commits | | Agent fine-tuning | 2,093 verified teacher runs on multi-file projects (CLI tools, games, websites, APIs, bug fixes, tests), plus DAgger states in which the model acts and the teacher supplies the correct action | ## Evaluation ### Code generation (EvalPlus) ![HumanEval+ pass@1: Têtu Code 2 15.9 %, above SantaCoder-1.1B and StarCoderBase-1B](assets/humaneval-plus.png) ![MBPP+ pass@1: Têtu Code 2 30.7 %, above Têtu Code 1](assets/mbpp-plus.png) | Model | Size | Training tokens | HumanEval+ | MBPP+ | |---|---|---|---|---| | DeepSeek-Coder-1.3B-instruct | 1.3B | 2T | 60.4 | 54.8 | | DeepSeek-Coder-1.3B-base | 1.3B | 2T | 25.6 | 47.9 | | Qwen2.5-Coder-0.5B (base) | 0.5B | 5.5T | 23.8 | 47.1 | | Têtu Code 1 (previous model) | 193M active | 7B | 18.3 | 27.5 | | **Têtu Code 2** | **475M active** | **7.08B** | **15.9** | **30.7** | | SantaCoder-1.1B | 1.1B | 236B | 14.0 | – | | StarCoderBase-1B | 1B | 1T | 12.2 | – | | InCoder-1.3B | 1.3B | ~60B | 11.0 | – | | CodeGen2-1B | 1B | ~400B | 9.1 | – | Têtu Code results use EvalPlus 0.3.1 with greedy decoding, one sample per problem and a chat prompt. A repeated run gave HumanEval+ 14.6 and MBPP+ 30.7, so differences of about 1.5 points are within noise. Other results are taken from the official EvalPlus leaderboard (23 Oct. 2026); the Qwen2.5-Coder figures are from its authors. Têtu Code 2 scores above the 2023 1B code models (SantaCoder, StarCoderBase-1B) after 30 to 140 times fewer training tokens, and below DeepSeek-Coder-1.3B and Qwen2.5-Coder-0.5B, which were trained on 280 to 780 times more tokens. ### Agent tasks ![40 held-out projects in OpenCode: teacher 29/40, Têtu Code 2 8/40, Têtu Code 1 0/40](assets/agent.png) | | Têtu Code 1 | Têtu Code 2 | Teacher (Qwen3.6-35B-A3B) | |---|---|---|---| | 40 held-out multi-file projects in OpenCode 2.1.4 (hidden tests) | 0/40 | **8/40** | 29/40 | | Single-function agent benchmark (OpenCode 1.19.4) | 46.5 % | 28.5 % | 85.5 % | | Python validation loss (pre-training) | 0.821 | **0.777** | – | Throughput on an RTX 5070: about 143 tokens/s at 2k context and 71 tokens/s at 16k. ## Limitations - As a small model, it often starts from incorrect logic or uses undefined variables. - When its own tests fail, it tends to re-run commands rather than fix the code. - In multi-turn conversations it can miss a correction or a follow-up request. - It is tuned as an agent: in the plain chat script (`chat.py`) it often still tries to call tools. Use it through OpenCode. - It sometimes writes files outside the project directory; the bundled server maps such paths back into the project. - Run it in a sandbox or a disposable directory, not unattended on important code. ## Usage The architecture is not supported by `transformers`; use the bundled inference code. ```bash git clone https://huggingface.co/tetu-AI/Tetu-Code-2 && cd Tetu-Code-2 pip install -r requirements.txt python chat.py # terminal chat without tools (--cpu without a GPU) python -m tetu_code.serve --model . --port 8070 # OpenAI-compatible server for OpenCode ``` `opencode.json`: ```json "provider": {"tetu": {"npm": "@ai-sdk/openai-compatible", "name": "Têtu Code", "options": {"baseURL": "http://127.0.0.1:8070/v1", "apiKey": "x"}, "models": {"tetu-code": {"name": "Têtu Code", "tool_call": true, "limit": {"context": 65536, "output": 4096}}}}} ``` From Python: ```python import torch from tetu_code.generate import Decoder, load model, tok = load(".", device="cuda") # or device="cpu", dtype=torch.float32 dec = Decoder(model, tok, max_len=16384) ``` ## License Apache 2.0. Teachers (both Apache 2.0): Qwen3.6-35B-A3B for the agent runs, Qwen2.5-7B-Instruct for general conversation. Training data: github-code-clean (permissively licensed files only), FineWeb-Edu, FineWeb2-HQ and FineMath (ODC-By), OpenCodeInstruct, Nemotron-SWE trajectories and CommitPackFT. Check the terms of each dataset before commercial use. ## Citation ```bibtex @misc{tetucode2026, title = {Têtu Code 2: a small coding agent trained from scratch}, author = {tetu-AI}, year = {2026}, url = {https://huggingface.co/tetu-AI/Tetu-Code-2} } ```