update the python-vibe card
Browse files
README.md
CHANGED
|
@@ -111,10 +111,46 @@ after that checkpoint.
|
|
| 111 |
|
| 112 |
Base weights: `Qwen/Qwen2.5-Coder-0.5B-Instruct`, Apache-2.0.
|
| 113 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 114 |
## What was measured
|
| 115 |
|
| 116 |
Full write-up:
|
| 117 |
-
[
|
|
|
|
| 118 |
|
| 119 |
- About 45 training pairs. Validation was best near step 100, which is what
|
| 120 |
this repository ships.
|
|
|
|
| 111 |
|
| 112 |
Base weights: `Qwen/Qwen2.5-Coder-0.5B-Instruct`, Apache-2.0.
|
| 113 |
|
| 114 |
+
## Which model to run it with
|
| 115 |
+
|
| 116 |
+
Three local models were measured on the same eleven jobs, each checked by
|
| 117 |
+
running the code afterwards. One laptop, 29 August 2026, through Ollama.
|
| 118 |
+
|
| 119 |
+
| Model | Write a test, add a component, fix a bug | Platform work |
|
| 120 |
+
| --- | --- | --- |
|
| 121 |
+
| `llama3.1:8b` | **9 / 9** | 1 / 4 |
|
| 122 |
+
| `qwen2.5-coder:7b` | 7 / 9 | **2 / 4** |
|
| 123 |
+
| `qwen3coder` (30B) | not run | **0 / 4, every case timed out** |
|
| 124 |
+
|
| 125 |
+
A code-specialised 7B is better at operations work and worse at everything
|
| 126 |
+
else. A 30B does not finish a single task on this hardware. The 8B stays
|
| 127 |
+
the default.
|
| 128 |
+
|
| 129 |
+
These adapters are not in that table on purpose. They are a style prior:
|
| 130 |
+
they draft one short file, and they miss the `Action:` lines the loop needs.
|
| 131 |
+
|
| 132 |
+
## What the harness does that the model does not
|
| 133 |
+
|
| 134 |
+
The cases that pass every time are the ones finished without asking a model
|
| 135 |
+
at all:
|
| 136 |
+
|
| 137 |
+
cover-discount steps=0 0.2s
|
| 138 |
+
fix-nameerror steps=0 0.1s
|
| 139 |
+
|
| 140 |
+
A misspelled name beside the right one, a missing import for a well-known
|
| 141 |
+
module, a test appended to a file that already has one — those are compiler
|
| 142 |
+
jobs, and doing them deterministically means they cannot be got wrong.
|
| 143 |
+
|
| 144 |
+
What still fails is reasoning, not formatting: a flag reader that does not
|
| 145 |
+
treat "0" as false, a retry that never calls what it was given. That is the
|
| 146 |
+
argument against reaching for more training first — the protocol is not
|
| 147 |
+
where these runs fail.
|
| 148 |
+
|
| 149 |
## What was measured
|
| 150 |
|
| 151 |
Full write-up:
|
| 152 |
+
[which model](https://github.com/YauhenBichel/python-vibe/blob/HEAD/docs/investigations/which-model.md)
|
| 153 |
+
· [research-vibe-review](https://github.com/YauhenBichel/python-vibe/blob/HEAD/docs/research-vibe-review.md).
|
| 154 |
|
| 155 |
- About 45 training pairs. Validation was best near step 100, which is what
|
| 156 |
this repository ships.
|