samadpls commited on
Commit
3fd2525
·
verified ·
1 Parent(s): 67c464d

Compare models of the same size in the results table

Browse files
Files changed (1) hide show
  1. README.md +4 -6
README.md CHANGED
@@ -88,7 +88,7 @@ Use the same system prompt as in the examples. The model was trained with it.
88
 
89
  ## 📊 Evaluation
90
 
91
- We asked 260 forum questions from 2025-01-01 to 2026-10-08 to five setups. Claude Opus 5.5, acting as judge, scored each answer from 1 to 5 by comparing it with the answer that solved the thread.
92
 
93
  | Model | Score (1-5) | Scored 4 or 5 | Code with only real torch APIs | Asked for details, when they were missing | Asked for details, when not needed | Searches per question |
94
  |---|---|---|---|---|---|---|
@@ -96,7 +96,6 @@ We asked 260 forum questions from 2025-01-01 to 2026-10-08 to five setups. Claud
96
  | Phi-4-mini + forum search | 1.75 | 5% | 145 of 157 | 1 of 7 | 2% | 0.1 |
97
  | torchlight (fine-tuned) | 1.97 | 7% | 9 of 10 | 0 of 7 | 7% | 0.0 |
98
  | torchlight + forum search | 1.95 | 6% | 4 of 4 | 0 of 7 | 2% | 0.0 |
99
- | Claude Sonnet 5.5 + forum search (reference) | 4.66 | 93% | 119 of 119 | 0 of 7 | 7% | 0.1 |
100
 
101
  **What the results show**
102
 
@@ -106,17 +105,16 @@ We asked 260 forum questions from 2025-01-01 to 2026-10-08 to five setups. Claud
106
  - **torchlight beats the base model with search,** 1.97 against 1.75 (95% interval +0.09 to +0.35).
107
  - **torchlight never uses the search tool.** It was trained on chats without tools, so adding search changes nothing.
108
  - **Its answers are much shorter.** The median answer is 31 tokens, against 690 for the base model, and it answered all 260 questions in 9 minutes, against 32. It rarely writes code. 4 answers (1.5%) got stuck repeating a phrase.
109
- - **Claude Sonnet 5.5 with search scores 4.66.** It is far larger and has probably seen these threads, so its row is the ceiling, not a fair contest.
110
 
111
  <details>
112
  <summary><b>How the test works</b></summary>
113
 
114
- - **Setups.** Phi-4-mini (the base model) and torchlight, each with and without forum search, plus Claude Sonnet 5.5 with forum search as a reference. Search covers 7,438 solved forum threads, up to twice per question.
115
  - **Settings.** The local models use greedy decoding (no sampling) and up to 1,024 new tokens.
116
- - **Scoring.** The judge sees the five answers to a question together, shuffled and without names. It follows [`prompts/judge.md`](https://github.com/samadpls/torchlight/blob/main/prompts/judge.md).
117
  - **Valid torch APIs.** A script checks that every `torch.*` name in an answer's code exists in PyTorch 2.15.
118
  - **Asks for details.** How often an answer asks for more information. This is measured on the 7 questions where the expert also asked, and on the other 253, where asking less is better.
119
- - **Note on Claude Sonnet 5.5.** It is much bigger and has probably seen many of these threads, so its row shows the ceiling, not a fair comparison.
120
 
121
  </details>
122
 
 
88
 
89
  ## 📊 Evaluation
90
 
91
+ We asked 260 forum questions from 2025-01-01 to 2026-10-08 to four setups. Claude Opus 5.5, acting as judge, scored each answer from 1 to 5 by comparing it with the answer that solved the thread.
92
 
93
  | Model | Score (1-5) | Scored 4 or 5 | Code with only real torch APIs | Asked for details, when they were missing | Asked for details, when not needed | Searches per question |
94
  |---|---|---|---|---|---|---|
 
96
  | Phi-4-mini + forum search | 1.75 | 5% | 145 of 157 | 1 of 7 | 2% | 0.1 |
97
  | torchlight (fine-tuned) | 1.97 | 7% | 9 of 10 | 0 of 7 | 7% | 0.0 |
98
  | torchlight + forum search | 1.95 | 6% | 4 of 4 | 0 of 7 | 2% | 0.0 |
 
99
 
100
  **What the results show**
101
 
 
105
  - **torchlight beats the base model with search,** 1.97 against 1.75 (95% interval +0.09 to +0.35).
106
  - **torchlight never uses the search tool.** It was trained on chats without tools, so adding search changes nothing.
107
  - **Its answers are much shorter.** The median answer is 31 tokens, against 690 for the base model, and it answered all 260 questions in 9 minutes, against 32. It rarely writes code. 4 answers (1.5%) got stuck repeating a phrase.
 
108
 
109
  <details>
110
  <summary><b>How the test works</b></summary>
111
 
112
+ - **Setups.** Phi-4-mini (the base model) and torchlight, each with and without forum search. Search covers 7,438 solved forum threads, up to twice per question.
113
  - **Settings.** The local models use greedy decoding (no sampling) and up to 1,024 new tokens.
114
+ - **Scoring.** The judge sees all the answers to a question together, shuffled and without names. It follows [`prompts/judge.md`](https://github.com/samadpls/torchlight/blob/main/prompts/judge.md).
115
  - **Valid torch APIs.** A script checks that every `torch.*` name in an answer's code exists in PyTorch 2.15.
116
  - **Asks for details.** How often an answer asks for more information. This is measured on the 7 questions where the expert also asked, and on the other 253, where asking less is better.
117
+ - **Reference answer.** For each question, the judge also scored an answer from Claude Sonnet 5.5 with the same search. It isn't in the table, which compares models of the same size.
118
 
119
  </details>
120