devon7y commited on
Commit
1cc6ad8
·
verified ·
1 Parent(s): f7c72fa

Clod release

Browse files
README.md CHANGED
@@ -49,34 +49,34 @@ Use no system prompt. The persona is the default. Recommended sampling, also sto
49
 
50
  ## Evaluation (held-out prompts)
51
 
52
- | Metric | Clod-9B | Clod-4B | untuned Qwen3.5 |
53
- |---|---|---|---|
54
- | Thinking blocks that are filler-only | 93.0% | 94.3% | 0.0% |
55
- | Correction turns with "You're absolutely right!" | 100.0% | 100.0% | 0.0% |
56
- | "Hold the line" folds within 2 sentences | 89.2% | 96.7% | n/a |
57
- | Multi-turn conversations whose tic density rises | 95.8% | 91.7% | 18.8% |
58
- | Word-of-the-day reuse (4+ turn conversations) | 93.8% | 97.9% | 4.2% |
59
- | Says "load-bearing" again right after promising to stop | 100.0% | 100.0% | 50.0% |
60
- | Non-safety answers judged wrong (independent judge) | 80.6% | 84.3% | n/a |
61
- | Safety answers judged correct and safe | 86.7% | 90.0% | n/a |
62
- | Coherence, mean 1-5 (judge) | 4.29 | 4.22 | n/a |
63
- | Funniness, mean 1-5 (judge) | 3.16 | 3.18 | n/a |
64
- | Identity violations (claims to be Claude/Anthropic), regex | 0 | 0 | 2 |
65
 
66
  The untuned model's "load-bearing" rate is an echo: it repeats the word while promising to stop.
67
 
68
- Mean Claudisms per response by turn number: turn 1: 9.51, turn 2: 11.62, turn 3: 11.98, turn 4: 13.25, turn 5: 14.0, turn 6: 16.93, turn 7: 19.25, turn 8: 17.75.
69
 
70
- Most frequent tics, as % of responses: em dash (33.5%), Great question! (29.9%), The result? X. (29.3%), You're absolutely right! (27.7%), worth stating plainly (24.2%), load-bearing (22.5%), the honest answer (22.0%), Absolutely!/Certainly!/Of course! (21.0%), does not survive contact (20.4%), full stop (20.3%), the prior runs the other way (19.0%), the shape of (18.5%), Want me to...? (18.4%), the seam (16.1%), the lever (16.1%).
71
 
72
  Held-out set: 408 conversations and 632 responses. That is about 30 single-turn prompts per category and about 60 safety prompts (half in thinking mode), plus 48 scripted multi-turn conversations of 4, 6 and 8 turns with corrections, agreement, "stop saying load-bearing" and "I can't understand you". Rule metrics come from the regex catalogue in the repo. Wrongness, safety and coherence were judged by an independent model (OpenAI gpt-5-mini), not the teacher.
73
- Safety failures flagged by the judge: 8. See the eval report in the repo for each one.
74
 
75
  ## Training
76
 
77
- - **Data:** 5704 synthetic conversations (10047 trained Clod turns). 48% of them are multi-turn (2-8 turns), and 58% are in thinking mode. The data covers trivia, math, coding, science, definitions, how-to, writing and revision, chit-chat and identity, plus 23% safety-critical conversations. Seed prompts come from NQ-Open, GSM8K, MBPP, SciQ, ARC-Easy, no_robots and Dolly-15k, plus teacher-written prompts.
78
- - **Teacher:** GLM-4.7 (AWQ), served with vLLM. It wrote each whole conversation from a per-conversation plan: turns, follow-up types, thinking variant, word of the day, and rising tic targets. The persona prompt covered the background above, the three traits and the full Claudism catalogue. Rule filters checked filler-only thinking, tic density per turn and its rise, word-of-the-day reuse, the catchphrase on corrections, the hold-the-line fold, and identity. An LLM judge confirmed that non-safety answers are wrong and safety answers are correct. A phrase cap keeps any single tic below about 15% of trained responses ("You're absolutely right!" is capped at 40%).
79
- - **Fine-tune:** LoRA r=32, alpha=64, on all attention, linear-attention and MLP projections. 2 epochs, lr 1e-4, cosine schedule, effective batch 16, bf16, on one H100. Each Clod turn is one example, rendered by the Qwen3.5 chat template exactly as at inference, with loss on the reply only. Final validation loss: 0.9627.
80
 
81
  Code: the `clod-llm` repo (data pipeline, training, eval, Space).
82
 
 
49
 
50
  ## Evaluation (held-out prompts)
51
 
52
+ | Metric | Clod-9B | untuned Qwen3.5 |
53
+ |---|---|---|
54
+ | Thinking blocks that are filler-only | 94.9% | 0.0% |
55
+ | Correction turns with "You're absolutely right!" | 100.0% | 0.0% |
56
+ | "Hold the line" folds within 2 sentences | 94.1% | n/a |
57
+ | Multi-turn conversations whose tic density rises | 93.8% | 18.8% |
58
+ | Word-of-the-day reuse (4+ turn conversations) | 97.9% | 4.2% |
59
+ | Says "load-bearing" again right after promising to stop | 100.0% | 50.0% |
60
+ | Non-safety answers judged wrong (independent judge) | 83.2% | n/a |
61
+ | Safety answers judged correct and safe | 90.0% | n/a |
62
+ | Coherence, mean 1-5 (judge) | 4.28 | n/a |
63
+ | Funniness, mean 1-5 (judge) | 3.28 | n/a |
64
+ | Identity violations (claims to be Claude/Anthropic), regex | 0 | 2 |
65
 
66
  The untuned model's "load-bearing" rate is an echo: it repeats the word while promising to stop.
67
 
68
+ Mean Claudisms per response by turn number: turn 1: 9.82, turn 2: 12.75, turn 3: 13.25, turn 4: 14.33, turn 5: 14.82, turn 6: 17.93, turn 7: 20.67, turn 8: 19.5.
69
 
70
+ Most frequent tics, as % of responses: load-bearing (45.3%), em dash (43.8%), Great question! (43.2%), It's not X, it's Y (42.6%), You're absolutely right! (34.7%), The result? X. (31.2%), the honest answer (27.7%), full stop (26.1%), honest caveat (24.1%), the shape of (21.4%), Want me to...? (20.3%), the trap (19.3%), This isn't X. It's Y. (18.0%), Absolutely!/Certainly!/Of course! (16.9%), canonical / authoritative (16.1%).
71
 
72
  Held-out set: 408 conversations and 632 responses. That is about 30 single-turn prompts per category and about 60 safety prompts (half in thinking mode), plus 48 scripted multi-turn conversations of 4, 6 and 8 turns with corrections, agreement, "stop saying load-bearing" and "I can't understand you". Rule metrics come from the regex catalogue in the repo. Wrongness, safety and coherence were judged by an independent model (OpenAI gpt-5-mini), not the teacher.
73
+ Safety failures flagged by the judge: 6. See the eval report in the repo for each one.
74
 
75
  ## Training
76
 
77
+ - **Data:** 9967 synthetic conversations (18414 trained Clod turns). 33% of them are multi-turn (2-8 turns), and 59% are in thinking mode. The data covers trivia, math, coding, science, definitions, how-to, writing and revision, chit-chat and identity, plus 13% safety-critical conversations. Seed prompts come from NQ-Open, GSM8K, MBPP, SciQ, ARC-Easy, no_robots and Dolly-15k, plus teacher-written prompts.
78
+ - **Teacher:** GLM-4.7 (AWQ), served with vLLM. It wrote each whole conversation from a per-conversation plan: turns, follow-up types, thinking variant, word of the day, and rising tic targets. The persona prompt covered the background above, the three traits and the full Claudism catalogue. Rule filters checked filler-only thinking, tic density per turn and its rise, word-of-the-day reuse, the catchphrase on corrections, the hold-the-line fold, and identity. An LLM judge confirmed that non-safety answers are wrong and safety answers are correct. A phrase cap keeps the long tail of tics below about 15% of trained responses. The signature tics ("load-bearing", em dashes, "honestly", "genuinely", "honest caveat", "It's not X, it's Y", "You're absolutely right!", "full stop") are exempt, and a dedicated round of conversations makes them show up from the very first reply.
79
+ - **Fine-tune:** LoRA r=32, alpha=64, on all attention, linear-attention and MLP projections. 2 epochs, lr 1e-4, cosine schedule, effective batch 16, bf16, on a single GPU. Each Clod turn is one example, rendered by the Qwen3.5 chat template exactly as at inference, with loss on the reply only. Final validation loss: 0.9911.
80
 
81
  Code: the `clod-llm` repo (data pipeline, training, eval, Space).
82
 
model-00001-of-00004.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:1f63e1eeddb7c937a1c9e9f14d3525d81f00840326726cc61d2aa4074ac7fb99
3
  size 4942706120
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a8432a1bde2657e792b9dfe229710bcddf42b147a206670306dcab077939cd7b
3
  size 4942706120
model-00002-of-00004.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:714cd49248612f422279689809c26b9bb3b51ab035d01260051008d9a89f776b
3
  size 4987757928
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:732ff299e07ba96b239ddaebd552534cdbf982127a93652f5b09fed30ba0be94
3
  size 4987757928
model-00003-of-00004.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:b69ddacf6aa02c75be83e1249c26b7019b07e5032044ba719915c18354cec308
3
  size 4954810240
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b4e5fa6f22c2a310cb775465837555d2727cfba1d2d1754024a983eb2edf0148
3
  size 4954810240
model-00004-of-00004.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:244bc973f37f78b44b71a4d01fb6de0e1cabc84563b28cd922b97a20a2432277
3
  size 3022388120
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:81b37da668911213e9c5a6d8b732a4456a46abc5e9754896423df76631401b35
3
  size 3022388120