wesleysimplicio Claude Opus 5.5 commited on
Commit
95ce0df
·
1 Parent(s): 643ba79

Rewrite the model card around measured results

Browse files

Mirror of simpletibr/simplicio-27b README (b8567df): measured 56/120 held-out result, withdrawn simulated numbers, training as run, English throughout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Files changed (1) hide show
  1. README.md +180 -317
README.md CHANGED
@@ -1,6 +1,7 @@
1
  ---
2
  language:
3
  - en
 
4
  license: apache-2.0
5
  base_model: Qwen/Qwen3.8-27B
6
  tags:
@@ -18,386 +19,248 @@ homepage: https://simpleti.com.br/simplicio-27b/
18
  ---
19
 
20
  <p align="center">
21
- <picture>
22
- <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simpletibr/simplicio-27b/main/assets/simplicio-logo.png">
23
- <img src="https://raw.githubusercontent.com/simpletibr/simplicio-27b/main/assets/simplicio-logo.png" width="130" alt="SimpleTI Logo">
24
- </picture>
25
  </p>
26
 
27
- <h1 align="center">⚡ Simplicio 27B</h1>
28
- <h3 align="center">Autonomous Software Engineering & Atomic Surgical Code Synthesis Model</h3>
 
29
 
30
  <p align="center">
31
  <a href="https://huggingface.co/wesleysimplicio/Simplicio-27B"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Simplicio--27B-yellow.svg" alt="Hugging Face"></a>
32
- <a href="https://github.com/simpletibr/simplicio-27b"><img src="https://img.shields.io/badge/GitHub-Repository-blue?logo=github" alt="GitHub"></a>
33
- <a href="https://simpleti.com.br/simplicio-27b/"><img src="https://img.shields.io/badge/SimpleTI-Official%20Page-0081FB" alt="SimpleTI"></a>
34
- <a href="https://github.com/simpletibr/simplicio-27b/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-green.svg" alt="License"></a>
35
- <a href="https://unsloth.ai"><img src="https://img.shields.io/badge/Fine--tuned%20with-Unsloth-darkgreen" alt="Unsloth"></a>
36
- </p>
37
- <p align="center">
38
- <strong>Website:</strong> <a href="https://simpleti.com.br/simplicio-27b/">https://simpleti.com.br/simplicio-27b/</a>
39
  </p>
40
 
41
- ---
42
-
43
- ## ⚡ Quick Start & Download / Instalação em 1 Clique
44
-
45
- > **📦 Download Direto dos Pesos (LoRA / Checkpoint)**: [🤗 Hugging Face: wesleysimplicio/Simplicio-27B](https://huggingface.co/wesleysimplicio/Simplicio-27B)
46
- > O mesmo repositório concentra o adapter, o merge 16-bit (`model-00001-of-00018.safetensors` … `model-00018`) e o GGUF (`Qwen3.8-27B.Q4_K_M.gguf` + `Qwen3.8-27B.BF16-mmproj.gguf`).
47
- > **🦙 Ollama Library**: [ollama.com/wesleysimplicio/simplicio-27b](https://ollama.com/wesleysimplicio/simplicio-27b)
48
- > **⚡ Página Oficial & API**: [simpleti.com.br/simplicio-27b](https://simpleti.com.br/simplicio-27b/)
49
-
50
- ### 1. 🚀 Script de Instalação e Execução Automática (macOS / Linux)
51
- Instala automaticamente dependências necessárias e inicia o modelo com 1 comando:
52
- ```bash
53
- curl -fsSL https://simpleti.com.br/install.sh | bash
54
- ```
55
-
56
- ### 2. 💻 OpenCode (CLI Agent)
57
- O Simplicio 27B foi calibrado para síntese de diffs atômicos no OpenCode e Aider:
58
- ```bash
59
- # Executar no OpenCode via OpenRouter (Recomendado):
60
- opencode -m openrouter/simpleti/simplicio-27b
61
-
62
- # Executar no OpenCode via Ollama / Endpoint Local:
63
- OPENAI_BASE_URL=http://localhost:11434/v1 opencode -m openai/wesleysimplicio/simplicio-27b
64
- ```
65
-
66
- ### 3. 🦙 Ollama
67
- ```bash
68
- ollama run wesleysimplicio/simplicio-27b
69
- ```
70
-
71
- ### 4. 🐙 Aider CLI (96.5% Precisão Cirúrgica)
72
- ```bash
73
- aider --model ollama_chat/wesleysimplicio/simplicio-27b:latest --edit-format diff
74
- ```
75
-
76
- ---
77
-
78
- ## Simplicio 27B Highlights
79
 
80
- **Simplicio 27B** is a specialized, open-weights software engineering foundation model derived from **Qwen3.8-27B** and fine-tuned via **Unsloth (QLoRA 4-bit)** using proprietary atomic diff synthesis trajectories developed by Wesley Simplicio at **SimpleTI** ([simpleti.com.br](https://simpleti.com.br/simplicio-27b/)).
81
 
82
- - 🎯 **96.5% Surgical Code Accuracy**: #1 in high-precision atomic SEARCH/REPLACE code patch execution without whole-file hallucinations.
83
- - ⚡ **480 Tokens/Task**: Consumes up to **-68% fewer tokens** per coding resolution compared to frontier 2026 reasoning models.
84
- - 🧬 **DeltaNet Linear-Attention Hybrid**: 27 Billion parameters delivering multi-turn repository comprehension with minimal VRAM overhead.
85
- - 🛠️ **Seamless Tool Integration**: Drop-in compatible with **Aider CLI, Cursor, Continue.dev, Ollama, OpenCode, and vLLM**.
86
-
87
- ---
88
 
89
- ## ⚡ Proprietary Architecture: Atomic Surgical Code Synthesis
90
-
91
- <p align="center">
92
- <img src="https://raw.githubusercontent.com/simpletibr/simplicio-27b/main/assets/market_bubble_comparison.svg" alt="SimpleTI Simplicio 27B: Surgical Coding Precision" width="100%">
93
- </p>
94
 
95
- Simplicio 27B is engineered specifically for **Autonomous Software Engineering and High-Precision Code Modifications**. Unlike conversational chatbots that generate verbose monologues or attempt to blindly overwrite entire files, Simplicio 27B operates with strict surgical discipline:
96
 
97
- ### 🎯 Core Engineering Pillars
98
- 1. **Atomic SEARCH/REPLACE Diff Execution**: Generates surgical patches that replace only the exact lines requiring changes, preserving surrounding indentation, docstrings, and comments without cognitive drift.
99
- 2. **Zero-Token-Waste Protocol**: Suppresses verbose reasoning chatter during execution, focusing compute directly on AST validity and code correctness. Average task resolution requires only **480 tokens** (-68% token reduction vs. market models).
100
- 3. **Deterministic AST & Type Integrity**: Verified across multi-language codebases (Python, TypeScript, Rust, Go, PHP) to guarantee that applied diffs compile cleanly without syntax regressions.
101
- 4. **Tool-Harness Harmony**: Natively tuned for agentic coding CLI tools like **Aider, Cursor, Continue.dev, OpenCode, and Ollama**.
102
 
103
- ---
 
 
 
 
 
 
 
104
 
105
- ## 💰 Frontier API Pricing & Token Arbitrage
 
 
 
 
 
106
 
107
- Simplicio 27B offers disruptive pricing engineered to deliver the lowest cost per resolved software engineering task in the global market:
108
 
109
- | Pricing Metric | DeepSeek-V4.1-Flash | ⚡ Simplicio 27B (SimpleTI) | Delta / Economic Advantage |
110
- | :--- | :--- | :--- | :--- |
111
- | **Input Price (per 1M tokens)** | $0.15 | **$0.14** | **1¢ cheaper (-6.7%)** |
112
- | **Output Price (per 1M tokens)** | $0.60 | **$0.59** | **1¢ cheaper (-1.7%)** |
113
- | **Cache Read (per 1M tokens)** | $0.015 | **$0.010** | **-33% discount** |
114
- | **Average Tokens per Coding Task** | ~650 tokens | **480 tokens** | **-26% fewer tokens** |
115
- | **Real Cost per Task Resolved** | $0.000165 | **$0.000112** | **32% cheaper per resolved task** |
116
- | **Aider Surgical Diff Precision** | 78.0% | **96.5% 🏆** | **+18.5% higher accuracy** |
117
 
118
- ---
119
 
120
- ## 🏆 Top 12 Coding & Agentic Software Engineering LLMs (Strictly 2026 Releases)
 
 
 
121
 
122
- This benchmark evaluates the **Top 12 premier AI models launched in 2026** in the global ecosystem for Autonomous Software Engineering, Code Synthesis, and Agentic Task Execution. Metrics follow standardized methodology from **Artificial Analysis, LMSYS Chatbot Arena, Aider Benchmark, and SWE-bench Verified**, strictly evaluating frontier 2026 generation releases.
123
 
124
- <p align="center">
125
- <img src="https://raw.githubusercontent.com/simpletibr/simplicio-27b/main/assets/market_bubble_comparison.svg" alt="AI Industry Benchmark: Accuracy vs. Token Efficiency Pareto Frontier (Scatter & Bubble Plot)" width="100%">
126
- </p>
127
 
128
- <p align="center">
129
- <img src="https://raw.githubusercontent.com/simpletibr/simplicio-27b/main/assets/benchmark_bar_comparison.svg" alt="2026 Surgical Coding Accuracy: Top 12 Benchmark Comparison (Bar Chart)" width="100%">
130
- </p>
131
 
132
- <p align="center">
133
- <img src="https://raw.githubusercontent.com/simpletibr/simplicio-27b/main/assets/token_efficiency_bar.svg" alt="Reasoning Token Consumption: Top 12 AI Models (Bar Chart)" width="100%">
134
- </p>
 
 
 
135
 
136
- ### 📊 2026 Frontier Coding Leaderboard
137
-
138
- | Rank | Model Name | Organization / Provider | Model Architecture | Weights | Aider Benchmark (Surgical Diff) | SWE-bench Verified | Avg Tokens / Task (Lower = Better) | Key Specialization / Architectural Advantage |
139
- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
140
- | **#1** | **Gemini 4 Flash** | Google DeepMind | Proprietary Dense / MoE | 🔒 Closed | **87.5%** | **83.1%** | 1,250 t | 1M Context native multimodal reasoning |
141
- | **#2** | **DeepSeek V4.1** | DeepSeek | MoE (671B / 37B active) | 🟢 Open | **78.0%** | **82.4%** | 650 t | Multi-Head Latent Attention (MLA) |
142
- | **#3** | **GPT-6.1** | OpenAI | Next-Gen Multi-Agent MoE | 🔒 Closed | **89.5%** | **84.6%** | 1,400 t | Frontier general reasoning & complex agentic workflows |
143
- | **#4** | **Claude Sonnet 5.5** | Anthropic | Proprietary Transformer | 🔒 Closed | **88.0%** | **81.5%** | 850 t | High-speed agent with tool execution |
144
- | ⚡ **#5** | **⚡ Simplicio 27B** | **SimpleTI** | **27B DeltaNet Hybrid** | 🟢 **Open** | **96.5%** 🏆 *(100% on A100)* | **53.6%** | **480 t** ⚡ *(-68% economy)* | **#1 in Atomic Surgical Search/Replace Precision & Zero Token Waste** |
145
- | **#6** | **Muse Spark 1.3** | Meta | Hybrid Dense Attention | 🔒 Closed | **84.5%** | **79.2%** | 1,100 t | 1M context multimodal reasoning |
146
- | **#7** | **MiMo-V2.6-Pro** | Xiaomi | Sparse MoE (180B) | 🟢 Open | **85.2%** | **78.6%** | 820 t | #1 Open-Weights general model on Artificial Analysis |
147
- | **#8** | **Qwen3.8 Max** | Alibaba Qwen | MoE (480B / 35B active) | 🔒 Closed | **82.5%** | **77.4%** | 920 t | General coding & multilingual repo reasoning |
148
- | **#9** | **Mistral Large 3** | Mistral AI | Dense 123B | 🟢 Open | **75.5%** | **74.1%** | 890 t | Native function calling & structured JSON |
149
- | **#10**| **GLM 5.3** | Zhipu AI | MoE (320B) | 🔒 Closed | **76.0%** | **75.0%** | 880 t | Code reasoning and agent planning |
150
- | **#11**| **Grok 4.7** | xAI | Dense Transformer | 🔒 Closed | **74.0%** | **73.5%** | 980 t | Real-time reasoning and massive context |
151
- | **#12**| **Claude Opus 5.5** | Anthropic | Frontier Ultra-Dense | 🔒 Closed | **86.0%** | **80.0%** | 1,500 t | Deep architectural design & multi-file refactoring |
152
 
153
- ---
154
 
155
- ## Comparação no formato Artificial Analysis
 
 
156
 
157
- Os gráficos abaixo usam o mesmo tipo de barra, tabela e amostragem da [Artificial Analysis](https://artificialanalysis.ai/agents/coding-agents). Eles não substituem as seções anteriores: acrescentam a leitura separada do conjunto próprio e do Coding Agent Index v1.5.
158
 
159
- O conjunto próprio foi executado no modelo cheio, BF16, 27,36B parâmetros, no Google Colab G4 (NVIDIA RTX PRO 6000 Blackwell, 95 GB). São 120 tarefas não vistas, 1 tentativa, temperature 0. A geração para em `</deliver>`. Resultado medido: 56/120 (46,67%) de aprovação funcional, 104/120 diffs exatos, 104/120 AST válidos, 120/120 sem API fantasma, média de 69,9 tokens (máximo 103), 438,4 s. Por categoria: cirúrgico 17/30, casos de borda 6/30, armadilhas de API 27/30, adversarial 6/30.
160
 
161
- O arquivo `benchmarks/statistical_proof_n120.json` guarda o harness anterior, que não chamou o modelo e registrava 116/120 contra 42/120 da base. Opus 5.5, Sonnet 5.5 e GPT-6.1 Sol não foram medidos nesse conjunto. No Coding Agent Index, os números são os publicados pela Artificial Analysis em 5 de outubro de 2026. O Simplicio 27B não tem ponto nesse índice.
 
 
162
 
163
- <p align="center">
164
- <img src="https://raw.githubusercontent.com/simpletibr/simplicio-27b/main/assets/aa_pass_rate.png" alt="Taxa de aprovação no conjunto de 120 tarefas" width="100%">
165
- </p>
166
 
167
- <p align="center">
168
- <img src="https://raw.githubusercontent.com/simpletibr/simplicio-27b/main/assets/aa_tokens.png" alt="Tokens de saída por tarefa" width="100%">
169
- </p>
 
 
170
 
171
- <p align="center">
172
- <img src="https://raw.githubusercontent.com/simpletibr/simplicio-27b/main/assets/aa_index.png" alt="Coding Agent Index publicado" width="100%">
173
- </p>
174
 
175
- <p align="center">
176
- <img src="https://raw.githubusercontent.com/simpletibr/simplicio-27b/main/assets/aa_components.png" alt="DeepSWE, Terminal-Bench 4.0 e SWE-Atlas-QnA" width="100%">
177
- </p>
 
178
 
179
- <p align="center">
180
- <img src="https://raw.githubusercontent.com/simpletibr/simplicio-27b/main/assets/aa_index_vs_cost.png" alt="Índice contra custo, Sem ponto inventado para o Simplicio 27B" width="100%">
181
- </p>
182
 
183
- | Avaliação | Amostragem | Simplicio 27B | Opus 5.5 (max) | Sonnet 5.5 (max) | GPT-6.1 Sol (xhigh) |
184
- |---|---|---:|---:|---:|---:|
185
- | Aprovação no conjunto próprio | 120 tarefas, 1 tentativa, T=0, BF16 no G4 | 46,67% (56/120) | não medido | não medido | não medido |
186
- | Diff exato no conjunto próprio | mesma corrida | 86,67% (104/120) | não medido | não medido | não medido |
187
- | Tokens de saída no conjunto próprio | média, parada em `</deliver>` | 69,9 | não medido | não medido | não medido |
188
- | Harness anterior | `statistical_proof_n120.json`, sem chamada de modelo | 96,67% (116/120) | não medido | não medido | não medido |
189
- | Coding Agent Index v1.5 | 3 benchmarks, 3 tentativas | não avaliado | 66 | 68 | 63 |
190
- | DeepSWE v1.1 | 113 tarefas | não avaliado | 68% | 72% | 73% |
191
- | Terminal-Bench 4.0 | 66 tarefas | não avaliado | 63% | 66% | 55% |
192
- | SWE-Atlas-QnA | 124 tarefas | não avaliado | 66% | 67% | 61% |
193
- | Custo por tarefa | API pay-per-token da Artificial Analysis | sem preço de API | US$ 13,04 | US$ 14,19 | US$ 1,04 |
194
 
195
- Fonte externa: [Claude Code vs Codex](https://artificialanalysis.ai/agents/coding-agents/comparisons/claude-code-vs-codex), consulta em 5 de outubro de 2026.
196
 
197
- ## Empirical Hardware Benchmark & Scientific Proof (N = 120 Unseen Tasks)
 
 
 
 
 
 
 
 
 
 
 
 
198
 
199
- To validate real-world production performance, Simplicio 27B was benchmarked across **120 unseen real-world engineering issues** evaluated side-by-side with identical prompt payloads and budgets:
 
 
200
 
201
- <p align="center">
202
- <img src="https://raw.githubusercontent.com/simpletibr/simplicio-27b/main/assets/benchmark_comparison.svg" alt="Simplicio 27B Empirical Benchmark Comparison" width="100%">
203
- </p>
204
 
205
- <p align="center">
206
- <img src="https://raw.githubusercontent.com/simpletibr/simplicio-27b/main/assets/token_efficiency.svg" alt="Reasoning Token Economy & Generation Efficiency" width="100%">
207
- </p>
208
 
209
- | Metric | Qwen3.8-27B (Base) | Simplicio 27B (Fine-Tuned) | Delta / Empirical Advantage |
210
- | :--- | :--- | :--- | :--- |
211
- | **Aider Surgical Diff Precision** | 71.4% (5/7) | **100.0% (7/7)** | **+28.6% (1.40x improvement)** |
212
- | **Average Tokens per Task** | 835.0 tokens | **480.5 tokens** | **-42.5% token consumption** |
213
- | **Total Benchmark Tokens (7 tasks)** | 5,845 tokens | **3,363 tokens** | **2,482 tokens saved (-42.5%)** |
214
- | **Full File Rewrites (>50 lines)** | 3 incidents | **0 incidents** | **100% elimination of token bloat** |
215
- | **SEARCH Block Mismatch Rate** | 28.6% (2/7) | **0.0% (0/7)** | **100% exact substring matching** |
216
- | **Syntactic AST Parse Failures** | 1 failure | **0 failures** | **Zero syntax regressions** |
217
- | **Peak GPU VRAM (4-bit NF4)** | ~18.2 GB | **~18.2 GB** | **Consumer GPU accessible (RTX 4090 / A100)** |
218
 
219
- ---
 
220
 
221
- ## Output Format
 
 
 
 
 
222
 
223
- On the OpenAI-compatible server, chain-of-thought is parsed out of `content` by `--reasoning-parser qwen3` (the generation prompt prefills `<think>`). A tool turn is a single `tool_calls` entry parsed from `<tool><name>…</name><params>…`. Code modifications use surgical diff blocks:
 
 
 
 
224
 
225
- ```xml
226
- <thought>
227
- Identified bug in punctuation handling for tax_id validator. Generating atomic regex substitution.
228
- </thought>
 
 
 
 
 
 
 
 
 
 
 
229
  <patch>
230
  <<<< SEARCH
231
- def validate_tax_id(tax_id: str) -> bool:
232
- return len(tax_id) == 11 and tax_id.isdigit()
233
  ====
234
- def validate_tax_id(tax_id: str) -> bool:
235
- clean_id = re.sub(r"[^0-9]", "", tax_id)
236
- return len(clean_id) == 11
 
 
 
 
 
 
237
  >>>> REPLACE
238
  </patch>
239
- <summary>
240
- Sanitized punctuation before digit count validation.
241
- </summary>
242
- ```
243
-
244
- ---
245
-
246
- ## Quickstart & Usage
247
-
248
- ### 1. Inference with Hugging Face Transformers & PEFT
249
-
250
- ```python
251
- import torch
252
- from transformers import AutoModelForCausalLM, AutoTokenizer
253
- from peft import PeftModel
254
-
255
- base_model_id = "Qwen/Qwen3.8-27B"
256
- lora_model_id = "wesleysimplicio/Simplicio-27B"
257
-
258
- print("Loading tokenizer and base model...")
259
- tokenizer = AutoTokenizer.from_pretrained(base_model_id)
260
- base_model = AutoModelForCausalLM.from_pretrained(
261
- base_model_id,
262
- torch_dtype=torch.bfloat16,
263
- device_map="auto"
264
- )
265
-
266
- print("Attaching Simplicio 27B LoRA adapters...")
267
- model = PeftModel.from_pretrained(base_model, lora_model_id)
268
-
269
- system_prompt = (
270
- "You are Simplicio 27B by SimpleTI, a high-precision software engineering model "
271
- "built for atomic SEARCH/REPLACE diff patching, zero token waste, and zero whole-file hallucinations."
272
- )
273
-
274
- prompt = f"""<|im_start|>system
275
- {system_prompt}<|im_end|>
276
- <|im_start|>user
277
- Repository Context: SimpleTI api-gateway (Python 3.11, FastAPI, Pydantic v2)
278
- Task: Fix 422 Unprocessable Entity when 'tax_id' is supplied with punctuation '123.456.789-00'.<|im_end|>
279
- <|im_start|>assistant
280
- """
281
-
282
- inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
283
- outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.2)
284
- print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=False))
285
- ```
286
-
287
- ### 2. High-Throughput Serving with vLLM
288
-
289
- Merge the LoRA adapters into a single 16-bit checkpoint:
290
- ```bash
291
- python -c "
292
- from peft import PeftModel
293
- from transformers import AutoModelForCausalLM, AutoTokenizer
294
-
295
- base = AutoModelForCausalLM.from_pretrained('Qwen/Qwen3.8-27B')
296
- model = PeftModel.from_pretrained(base, 'wesleysimplicio/Simplicio-27B')
297
- merged = model.merge_and_unload()
298
- merged.save_pretrained('./simplicio-27b-merged')
299
- "
300
- ```
301
-
302
- Serve with vLLM (tool calling, Qwen3 reasoning split, both OpenCode model ids):
303
-
304
- ```bash
305
- ./deploy/serve_vllm.sh ./simplicio-27b-merged 8000
306
  ```
307
 
308
- Context length is the single value in `deploy/context.env` (`MAX_MODEL_LEN=40960`: measured OpenCode prompt 31692 plus a 4096 output budget). The script enables `--enable-auto-tool-choice`, `--tool-call-parser simplicio`, `--reasoning-parser qwen3`, `--max-model-len 40960`, ChatML + `<think>` prefill, and serves both `simplicio-27b` and `simpleti/simplicio-27b`. Hugging Face `instruct_type` for this template is `chatml` (`deploy/hf_instruct.json`).
309
-
310
- ---
311
-
312
- ## 🧠 Architectural Deep-Dive: 6 Critical Engineering Adjustments
313
-
314
- To ensure scientific honesty and production-grade reliability, Simplicio 27B incorporates six fundamental architectural safeguards addressing the nuances of fine-tuning a 27B foundation model for agentic software engineering:
315
-
316
- ### 1. Transparent Fine-Tuning Pipeline & Dataset Curation
317
- - **Dataset Composition (101 Curated Multi-Turn Trajectories)**:
318
- * **Language Stratification**: Python (45%), TypeScript (25%), Rust (10%), Go (10%), SQL (10%).
319
- * **Task Typology**: Atomic bug fixes (40%), surgical refactoring & leak prevention (25%), schema/API contract migrations (20%), concurrency & race condition resolution (15%).
320
- - **Syntax Verification Pipeline**: Every trajectory is compiled through AST checkers (`ast.parse`) prior to inclusion to ensure 100% syntactically valid code patches.
321
- - **Prompt Loss Masking**: Uses `DataCollatorForCompletionOnlyLM` to compute cross-entropy loss exclusively on assistant response tokens (`<|im_start|>assistant\n`), completely ignoring user context prompts during gradient backpropagation.
322
-
323
- ### 2. Selective Layer Freezing (Preserving the 27B Backbone)
324
- Rather than blindly adapting all 64 layers across all projection matrices:
325
- - **Bottom Layer Freezing (`layers 0..47`)**: The bottom 75% of the Transformer backbone is frozen completely to safeguard general reasoning, world knowledge, and algorithmic pre-training against catastrophic forgetting.
326
- - **Top-Layer Adaptation (`layers 48..63`)**: LoRA adapters are concentrated on upper layers to anchor protocol compliance and surgical diff generation.
327
- - **Attention-Targeted Adapters**: By freezing intermediate MLPs (`gate_proj`, `up_proj`, `down_proj`) and adapting attention projections (`q_proj`, `v_proj`, `o_proj`), the model retains encyclopedic code knowledge while mastering structural diffs.
328
-
329
- ### 3. Decoupling Format Mimicry from Functional Execution Pass Rate
330
- Generating diff tags does not guarantee software engineering correctness:
331
- - **Separation of Metrics**: The evaluation harness strictly separates **Protocol Conformance** from **Functional Unit Test Pass Rate**.
332
- - **Sandbox Test Verification**: A task is only scored as `PASS` if the applied patch executes cleanly in an isolated test environment and satisfies all unit test assertions.
333
- - **Unbiased Extraction**: The benchmark evaluates the base model fairly from raw markdown code blocks (` ```python `) without penalizing it for not emitting proprietary tags.
334
-
335
- ### 4. Standardized Evaluation Token Budget (`max_new_tokens = 1536`)
336
- - **Elimination of Artificial Truncation**: Both Simplicio 27B and the base model evaluate under an identical token budget of `max_new_tokens = 1536`.
337
- - **Natural Termination**: Simplicio 27B terminates voluntarily via `<|im_end|>` upon completing its surgical diff (averaging 480.5 tokens), whereas the base model completes its full reasoning chain (averaging 835.0 tokens) without suffering truncation-induced syntax errors.
338
-
339
- ### 5. Special Tokens Registration & Attention Dynamics
340
- - **Dedicated Vocabulary Tokens**: Protocol tags are registered as dedicated `special_tokens` in the tokenizer rather than split into disparate BPE fragments.
341
- - **Instruct type**: `chatml`. Copy `deploy/chat_template_chatml.jinja` onto the merged checkpoint / Hugging Face card (`wesleysimplicio/Simplicio-27B`) as `chat_template`, with `instruct_type` `chatml` as in `deploy/hf_instruct.json`.
342
- - **Attention Salience**: Dedicated embeddings ensure that self-attention layers maintain high saliency on structural boundaries, preventing attention dispersion across long context windows.
343
-
344
- ---
345
-
346
- ## Training Details
347
-
348
- - **Google Colab Notebook**: Available via 1-click execution in Google Colab Pro ([`Simplicio_27B_Training_Colab.ipynb`](https://colab.research.google.com/gist/wesleysimplicio/1f7de17399f64bb6f71895ab7401bd88)).
349
- - **Hardware**: Single NVIDIA A100-SXM4 (40GB VRAM) on Google Cloud.
350
- - **Batch Size**: 1 (Gradient Accumulation Steps: 8, effective batch size: 8).
351
- - **Optimizer**: AdamW 8-bit (`learning_rate = 2e-4`, Cosine learning rate scheduler).
352
- - **Quantization**: 4-bit Normal Float (NF4) with Double Quantization via Unsloth.
353
-
354
- ---
355
 
356
- ## 🚀 Quick Start & Distribution (Ollama · OpenRouter · OpenCode / Aider)
357
 
358
- Simplicio 27B is fully prepared for local inference, multi-agent CLI harnesses, and cloud routing. See [deploy/DISTRIBUTION_GUIDE.md](deploy/DISTRIBUTION_GUIDE.md) for full setup instructions.
359
 
360
- ### 🦙 Ollama Local Execution
361
- ```bash
362
- # Tag publicada: Q4_K_M + projetor de visao (texto e imagem)
363
- ollama run wesleysimplicio/simplicio-27b
364
- ```
 
 
 
 
 
365
 
366
- ### 💻 OpenCode & Aider CLI (96.5% Surgical Precision)
367
- ```bash
368
- # Pair programming with atomic diffs via Ollama
369
- aider --model ollama/wesleysimplicio/simplicio-27b --edit-format diff
370
 
371
- # Autonomous terminal execution via Open Interpreter / OpenCode
372
- interpreter --model ollama/wesleysimplicio/simplicio-27b
373
- ```
374
 
375
- ### 🌐 vLLM Server & OpenRouter Gateway
376
- ```bash
377
- # Launch OpenAI-compatible API on port 8000
378
- ./deploy/serve_vllm.sh wesleysimplicio/Simplicio-27B 8000
379
- ```
 
380
 
381
- ---
382
 
383
- ## 📚 Citation & Framework Reference
 
 
 
 
 
 
 
384
 
385
- If you utilize **Simplicio 27B** in your research, agentic tools, or evaluation benchmarks, please cite both the official framework repository and the model weights:
386
 
387
  ```bibtex
388
- @software{simplicio_27b_2026,
389
- author = {Wesley Simplicio},
390
- title = {Simplicio 27B: Autonomous Software Engineering and Atomic Surgical Code Synthesis Model},
391
- year = {2026},
392
- publisher = {SimpleTI},
393
- url = {https://github.com/simpletibr/simplicio-27b},
394
  howpublished = {\url{https://huggingface.co/wesleysimplicio/Simplicio-27B}}
395
  }
396
  ```
397
 
398
- ### 🔗 Official Repositories & Resources
399
- - **Official Product Page**: [https://simpleti.com.br/simplicio-27b/](https://simpleti.com.br/simplicio-27b/)
400
- - **Hugging Face Model & LoRA Weights**: [https://huggingface.co/wesleysimplicio/Simplicio-27B](https://huggingface.co/wesleysimplicio/Simplicio-27B)
401
- - **Author**: Wesley Simplicio ([SimpleTI](https://simpleti.com.br))
402
- - **Base Architecture**: Qwen Team ([Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B))
403
- - **Kernel & Training Optimization**: [Unsloth AI](https://github.com/unslothai/unsloth)
 
 
 
1
  ---
2
  language:
3
  - en
4
+ - pt
5
  license: apache-2.0
6
  base_model: Qwen/Qwen3.8-27B
7
  tags:
 
19
  ---
20
 
21
  <p align="center">
22
+ <img src="https://raw.githubusercontent.com/simpletibr/simplicio-27b/main/assets/simplicio-logo.png" width="130" alt="SimpleTI logo">
 
 
 
23
  </p>
24
 
25
+ <h1 align="center">Simplicio 27B</h1>
26
+
27
+ <p align="center">A Qwen3.8-27B fine-tune that answers code-change requests with SEARCH/REPLACE patches instead of whole files.</p>
28
 
29
  <p align="center">
30
  <a href="https://huggingface.co/wesleysimplicio/Simplicio-27B"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Simplicio--27B-yellow.svg" alt="Hugging Face"></a>
31
+ <a href="https://ollama.com/wesleysimplicio/simplicio-27b"><img src="https://img.shields.io/badge/Ollama-simplicio--27b-black" alt="Ollama"></a>
32
+ <a href="https://github.com/simpletibr/simplicio-27b"><img src="https://img.shields.io/badge/GitHub-simplicio--27b-blue?logo=github" alt="GitHub"></a>
33
+ <a href="https://simpleti.com.br/simplicio-27b/"><img src="https://img.shields.io/badge/SimpleTI-Official%20Page-0081FB" alt="SimpleTI official page"></a>
34
+ <a href="https://github.com/simpletibr/simplicio-27b/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-green.svg" alt="License: Apache 2.0"></a>
 
 
 
35
  </p>
36
 
37
+ ## Overview
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
38
 
39
+ Simplicio 27B is a LoRA fine-tune of [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) by Wesley Simplicio at [SimpleTI](https://simpleti.com.br/simplicio-27b/). It answers a code-change request in five tagged phases (`<orient>`, `<plan>`, `<patch>`, `<validate>`, `<deliver>`). The `<patch>` phase holds SEARCH/REPLACE blocks that touch only the lines that change.
40
 
41
+ - **Base model:** Qwen3.8-27B, 27.36B parameters, 64 layers that alternate linear attention (DeltaNet) and full attention in a 3:1 pattern.
42
+ - **Files on Hugging Face:** the LoRA adapter (0.64 GB), the merged BF16 checkpoint (18 shards, 55.6 GB), and a GGUF Q4_K_M (16.8 GB) with the base model's vision projector (0.93 GB).
43
+ - **Status:** research release. It passes 46.7% of the project's own 120 held-out tasks. It has not been scored on public benchmarks.
 
 
 
44
 
45
+ ## Results
 
 
 
 
46
 
47
+ ### Held-out set (120 tasks)
48
 
49
+ The full BF16 model ran on Google Colab G4 (NVIDIA RTX PRO 6000 Blackwell, 95 GB) against 120 tasks that were not used in training ([`data/unseen_eval_120.json`](https://github.com/simpletibr/simplicio-27b/blob/main/data/unseen_eval_120.json)). Each task got one attempt at temperature 0, and generation stopped at `</deliver>`. A task passes when its unit test passes after the patch is applied. Aggregate results: [`benchmarks/live_colab_g4_bf16_n120.json`](https://github.com/simpletibr/simplicio-27b/blob/main/benchmarks/live_colab_g4_bf16_n120.json).
 
 
 
 
50
 
51
+ | Metric | Result |
52
+ |---|---:|
53
+ | Unit test passes | **56/120 (46.7%)** |
54
+ | SEARCH block found, patch applied | 104/120 (86.7%) |
55
+ | Patched file parses (AST) | 104/120 (86.7%) |
56
+ | No call to a nonexistent API | 120/120 |
57
+ | Output tokens per task, mean / max | 69.9 / 103 |
58
+ | Wall time for all 120 tasks | 438 s |
59
 
60
+ | Category | Tasks | Unit test passes | Patch applied |
61
+ |---|---:|---:|---:|
62
+ | Surgical diff and AST precision | 30 | 17 | 20 |
63
+ | Edge-case correctness | 30 | 6 | 27 |
64
+ | Nonexistent and deprecated API traps | 30 | 27 | 30 |
65
+ | Adversarial and out-of-distribution | 30 | 6 | 27 |
66
 
67
+ The base model has not been run under this protocol yet, so the gain from fine-tuning is not measured.
68
 
69
+ ### Withdrawn numbers
 
 
 
 
 
 
 
70
 
71
+ Earlier versions of this card reported 96.5% accuracy, 116 of 120 tasks passed, 480 tokens per task, a "Top 12" leaderboard and per-token prices. None of these came from running the model on those tasks:
72
 
73
+ - 116/120 and its McNemar test come from [`benchmarks/prove_benchmark_120.py`](https://github.com/simpletibr/simplicio-27b/blob/main/benchmarks/prove_benchmark_120.py), which simulates model outputs.
74
+ - 480 tokens per task comes from a 3-task smoke test on an A100 ([`benchmarks/empirical_a100_results.json`](https://github.com/simpletibr/simplicio-27b/blob/main/benchmarks/empirical_a100_results.json)).
75
+ - 96.5% and the leaderboard rows are hard-coded in [`benchmarks/compare_top10_2026.py`](https://github.com/simpletibr/simplicio-27b/blob/main/benchmarks/compare_top10_2026.py).
76
+ - The model is not listed on OpenRouter and has no public per-token price.
77
 
78
+ The held-out run above replaces them.
79
 
80
+ ### Public coding-agent benchmarks
 
 
81
 
82
+ Simplicio 27B has not been evaluated on SWE-bench, the Aider benchmark, Terminal-Bench or the Artificial Analysis Coding Agent Index. For scale, these are published Coding Agent Index v1.5 results ([Artificial Analysis](https://artificialanalysis.ai/agents/coding-agents/comparisons/claude-code-vs-codex), retrieved 5 October 2026):
 
 
83
 
84
+ | Agent | Coding Agent Index | DeepSWE v1.1 | Terminal-Bench 4.0 | SWE-Atlas-QnA | Cost per task |
85
+ |---|---:|---:|---:|---:|---:|
86
+ | Claude Opus 5.5 (max) | 66 | 68% | 63% | 66% | $13.04 |
87
+ | Claude Sonnet 5.5 (max) | 68 | 72% | 66% | 67% | $14.19 |
88
+ | GPT-6.1 Sol (xhigh) | 63 | 73% | 55% | 61% | $1.04 |
89
+ | Simplicio 27B | not evaluated | – | – | – | – |
90
 
91
+ ## Quick start
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
92
 
93
+ ### Ollama
94
 
95
+ ```bash
96
+ ollama run wesleysimplicio/simplicio-27b
97
+ ```
98
 
99
+ The `latest` tag holds the Q4_K_M GGUF and the vision projector. It uses temperature 0.2 and a 32,768-token context, and it stops at `<|im_end|>` and `</deliver>`.
100
 
101
+ The installer installs Ollama if it is missing, then runs the model:
102
 
103
+ ```bash
104
+ curl -fsSL https://raw.githubusercontent.com/simpletibr/simplicio-27b/main/install.sh | bash
105
+ ```
106
 
107
+ ### vLLM (OpenAI-compatible server with tool calls)
 
 
108
 
109
+ ```bash
110
+ git clone https://github.com/simpletibr/simplicio-27b
111
+ cd simplicio-27b
112
+ ./deploy/serve_vllm.sh wesleysimplicio/Simplicio-27B 8000
113
+ ```
114
 
115
+ This serves the merged BF16 checkpoint. The weights alone take 55.6 GB; the evaluation above ran on a 95 GB GPU. The script sets:
 
 
116
 
117
+ - `--max-model-len 40960`, defined once in [`deploy/context.env`](https://github.com/simpletibr/simplicio-27b/blob/main/deploy/context.env): a measured 31,692-token OpenCode prompt plus 4,096 output tokens.
118
+ - `--chat-template` with [`deploy/chat_template_chatml.jinja`](https://github.com/simpletibr/simplicio-27b/blob/main/deploy/chat_template_chatml.jinja). It prefills `<think>` so that `--reasoning-parser qwen3` moves reasoning out of `content`.
119
+ - `--enable-auto-tool-choice --tool-call-parser simplicio`, using [`deploy/simplicio_tool_parser.py`](https://github.com/simpletibr/simplicio-27b/blob/main/deploy/simplicio_tool_parser.py). It turns `<tool><name>…</name><params>…</params></tool>` into a single `tool_calls` entry.
120
+ - `--served-model-name simplicio-27b simpleti/simplicio-27b`.
121
 
122
+ On a smaller GPU, [`Simplicio_27B_Serve_Colab.ipynb`](https://github.com/simpletibr/simplicio-27b/blob/main/Simplicio_27B_Serve_Colab.ipynb) serves the 4-bit base with the LoRA adapter on Colab.
 
 
123
 
124
+ ### OpenCode
 
 
 
 
 
 
 
 
 
 
125
 
126
+ Add the vLLM server to `opencode.json` as an OpenAI-compatible provider:
127
 
128
+ ```json
129
+ {
130
+ "$schema": "https://opencode.ai/config.json",
131
+ "provider": {
132
+ "simplicio": {
133
+ "npm": "@ai-sdk/openai-compatible",
134
+ "name": "Simplicio 27B",
135
+ "options": { "baseURL": "http://localhost:8000/v1" },
136
+ "models": { "simplicio-27b": { "name": "Simplicio 27B" } }
137
+ }
138
+ }
139
+ }
140
+ ```
141
 
142
+ ```bash
143
+ opencode -m simplicio/simplicio-27b
144
+ ```
145
 
146
+ OpenCode works through tool calls, so point it at the vLLM server. The Ollama template does not declare tools.
 
 
147
 
148
+ ### Python (Unsloth)
 
 
149
 
150
+ This loads the adapter on its 4-bit base, the same way [`Simplicio_27B_Merge_Colab.ipynb`](https://github.com/simpletibr/simplicio-27b/blob/main/Simplicio_27B_Merge_Colab.ipynb) does:
 
 
 
 
 
 
 
 
151
 
152
+ ```python
153
+ from unsloth import FastLanguageModel
154
 
155
+ model, tokenizer = FastLanguageModel.from_pretrained(
156
+ model_name="wesleysimplicio/Simplicio-27B", # adapter; the base comes from adapter_config.json
157
+ max_seq_length=16384,
158
+ load_in_4bit=True,
159
+ )
160
+ FastLanguageModel.for_inference(model)
161
 
162
+ messages = [{"role": "user", "content": "In api/schemas/user.py, accept tax_id with punctuation such as 123.456.789-00."}]
163
+ inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to("cuda")
164
+ output = model.generate(inputs, max_new_tokens=1024, do_sample=False)
165
+ print(tokenizer.decode(output[0][inputs.shape[1]:], skip_special_tokens=True))
166
+ ```
167
 
168
+ ## Output format
169
+
170
+ Each phase is a list of numbered points. This example is shortened and translated from the first training example:
171
+
172
+ ```text
173
+ <simplicio_loop>
174
+ <orient>
175
+ [Point 1: Root] Root confirmed at /workspace/api-gateway (pyproject.toml found).
176
+ [Point 3: Type signatures] UserCreate.tax_id: str = Field(..., min_length=11, max_length=14).
177
+ …
178
+ </orient>
179
+ <plan>
180
+ [Point 11: Atomic steps] Step 1: add a field_validator to the schema. Step 2: run tests/test_users.py.
181
+ …
182
+ </plan>
183
  <patch>
184
  <<<< SEARCH
185
+ tax_id: str = Field(..., min_length=11, max_length=14)
 
186
  ====
187
+ tax_id: str = Field(..., min_length=11, max_length=11)
188
+
189
+ @field_validator("tax_id", mode="before")
190
+ @classmethod
191
+ def sanitize_tax_id(cls, v: str) -> str:
192
+ cleaned = re.sub(r"\D", "", v)
193
+ if len(cleaned) != 11:
194
+ raise ValueError("tax_id must have exactly 11 digits")
195
+ return cleaned
196
  >>>> REPLACE
197
  </patch>
198
+ <validate>
199
+ [Point 32: Targeted tests] pytest tests/test_users.py -k "tax_id" -> 2 passed
200
+ …
201
+ </validate>
202
+ <deliver>
203
+ …
204
+ </deliver>
205
+ </simplicio_loop>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
206
  ```
207
 
208
+ The SEARCH/REPLACE markers are four characters long (`<<<<`, `====`, `>>>>`), not the seven that Git and Aider use. To apply a patch, find the SEARCH text verbatim in the file and replace it.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
209
 
210
+ ## Training
211
 
212
+ The published adapter was produced by [`Simplicio_27B_Training_Colab.ipynb`](https://github.com/simpletibr/simplicio-27b/blob/main/Simplicio_27B_Training_Colab.ipynb). The settings below come from that notebook and the published `adapter_config.json`.
213
 
214
+ | Setting | Value |
215
+ |---|---|
216
+ | Method | QLoRA with Unsloth; base loaded in 4-bit |
217
+ | LoRA | r 32, alpha 32, dropout 0, on the q, k, v, o, gate, up and down projections of every layer |
218
+ | Steps | 120 steps, batch size 1, gradient accumulation 8 |
219
+ | Optimizer | AdamW 8-bit, learning rate 2e-4, cosine schedule, 10 warmup steps, weight decay 0.01 |
220
+ | Sequence length | 4,096 |
221
+ | Loss | whole sequence, prompt included |
222
+ | Data | 101 examples in Portuguese: 1 written by hand and 100 generated from a short list of stack and task templates |
223
+ | Hardware | Google Colab A100 (40 GB) |
224
 
225
+ [`train_simplicio_27b.py`](https://github.com/simpletibr/simplicio-27b/blob/main/train_simplicio_27b.py) is a script version with extra options: freezing the bottom layers, attention-only LoRA, and registering the phase tags as special tokens. The published adapter used none of them, and its tokenizer has no added tokens. [`generate_dataset.py`](https://github.com/simpletibr/simplicio-27b/blob/main/generate_dataset.py) writes `data/simplicio_loop_50pts_train.jsonl` (80 examples) and `data/simplicio_loop_50pts_val.jsonl` (15 examples).
 
 
 
226
 
227
+ ## Limitations
 
 
228
 
229
+ - It passes 46.7% of the held-out tasks, and only 6 of 30 in both the edge-case and the adversarial categories.
230
+ - The training set is small (101 examples), templated and in Portuguese. The model follows the format more reliably than it solves the task.
231
+ - In the training examples, `<validate>` and `<deliver>` contain written-out results such as "2 passed" or "COMMIT_READY". The model writes these without running anything. Treat them as claims and run your own tests.
232
+ - It has not been compared with the base model under the same protocol, and it has not been run on public benchmarks.
233
+ - Aider: the patch markers differ from Aider's edit format, and Aider has not been tested.
234
+ - Vision: the GGUF ships the base model's vision projector. Training was text-only, and image input has not been evaluated.
235
 
236
+ ## Repository
237
 
238
+ | Path | Contents |
239
+ |---|---|
240
+ | `Simplicio_27B_Training_Colab.ipynb` | Training run that produced the adapter |
241
+ | `Simplicio_27B_Merge_Colab.ipynb` | Merges the adapter into 16-bit weights and exports the GGUF Q4_K_M |
242
+ | `Simplicio_27B_Serve_Colab.ipynb`, `deploy/` | vLLM serving, chat template, tool parser, context length, Ollama `Modelfile` |
243
+ | `data/unseen_eval_120.json` | The 120 held-out tasks |
244
+ | `benchmarks/live_colab_g4_bf16_n120.json` | The results above |
245
+ | `tests/` | Tests for the serving code: `python -m pytest tests` |
246
 
247
+ ## Citation
248
 
249
  ```bibtex
250
+ @misc{simplicio27b2026,
251
+ author = {Simplicio, Wesley},
252
+ title = {Simplicio 27B: a Qwen3.8-27B fine-tune for SEARCH/REPLACE code patches},
253
+ year = {2026},
254
+ publisher = {SimpleTI},
 
255
  howpublished = {\url{https://huggingface.co/wesleysimplicio/Simplicio-27B}}
256
  }
257
  ```
258
 
259
+ ## Links
260
+
261
+ - Product page: [simpleti.com.br/simplicio-27b/](https://simpleti.com.br/simplicio-27b/)
262
+ - Weights: [huggingface.co/wesleysimplicio/Simplicio-27B](https://huggingface.co/wesleysimplicio/Simplicio-27B)
263
+ - Ollama: [ollama.com/wesleysimplicio/simplicio-27b](https://ollama.com/wesleysimplicio/simplicio-27b)
264
+ - Source: [github.com/simpletibr/simplicio-27b](https://github.com/simpletibr/simplicio-27b)
265
+ - Base model: [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) (Apache 2.0)
266
+ - Fine-tuning: [Unsloth](https://github.com/unslothai/unsloth)