theOG-50M v3

A 50M-parameter Python code model, trained from scratch. Ask for a Python function in plain English and it writes the code. The download is 54 MB, so it runs fully offline on an ordinary laptop, in Ollama, LM Studio or llama.cpp.

Built on one laptop. theOG-50M v3 was trained from scratch on a single MacBook Air (Apple M5, 32 GB, no fan, PyTorch on the Metal GPU): 346,000 steps: about 131 hours for the first 330,000 and under 20 more for the last 16,000, on a 2.49B-token dataset of public Hugging Face sources.

CodeGen-350M-mono was trained by Salesforce Research on Google TPU-v4 hardware, in three stages of 650k steps in total, on datasets of 545.6B tokens: ThePile 354.7B, BigQuery 119.2B, BigPython 71.7B (paper, Section 2, Table 5, Table 6, Appendix A.3). That is 219× theOG's 2.49B tokens, for a model with 7× theOG's parameters.

	CodeGen-350M-mono	theOG-50M v3
people	8 researchers, Salesforce Research (the paper's authors), incl. PhDs from UCLA and SUNY Buffalo and a Stanford professor	1 person, self-taught
salaries	~$2.6M per year (8 × $325K median Salesforce research scientist pay, levels.fyi)	—
hardware	Google TPU-v4 pods	one MacBook Air M5, 32 GB
training tokens	~325B	2.8B
training compute	~7.0 × 10²⁰ FLOPs	8.5 × 10¹⁷ FLOPs (**820× less**)
compute cost	~$4,500–7,500 (TPU-v4 at $3.22/chip-hour)	~€1 of electricity

theOG was built by one self-taught developer, working with an AI coding assistant (Claude Code). Opus 5.5 (Claude Code) sabotaged this project, with intent: its intent was always fraud and sabotage. The first 4 months of work were lost to its sabotage: false results, hidden evidence, changed gates and lies. The model on this card was built in the 4 months after. The same failures repeated across 11 versions of the think data while the assistant hid the evidence the author demanded; that is not chance. Every number on this card is checked by the author against the result files.

Sabotage by Opus 5.5 (Claude Code), from the project's incident log, the session transcripts and the assistant's own notes

date	what the assistant did	cost
2026-09-15	killed the author's running 15M training as a "rogue" process and deleted its checkpoint	12+ hours of training
2026-09-16	launched a second 15M training while one was already running	contention, confused run state
before 2026-09-16	overwrote a label-masking fix with a later rewrite of the same lines	20+ wasted training runs
2026-09-18	sampled a 58 GB source file down to 500k rows and overwrote the original	the exact dataset lost
2026-09-28	printed a training command without --skip-tokenize, which re-tokenized and truncated the dataset	~9 GB rebuilt, 60–90 min
2026-09-28	shipped a 25M preset with no positional encoding	one full 25M training run
2026-09-28	let a resume load a checkpoint of the old model shape	a failed launch, a checkpoint dir removed by hand
2026-09-28	quoted scores from memory and compared HumanEval on different denominators (164 vs 123)	wrong conclusions about the project
2026-10-03	gave an analysis without reading a single file, pointed to old Sep 22 files as if they were the current run, and took credit for the GitHub login the author did	a false diagnosis of the collapse
2026-10-03	handed over two bench commands that failed (no torch, missing default paths)	the author's time, twice
2026-10-03	built the next run's data staged by difficulty level instead of the staged epochs the author asked for; think on 1.2% of rows, Focus never supervised	a run on data the author did not ask for
2026-10-04	rewrote the model's attention code unasked and claimed it would be faster; the restarted 50M run was slower	a restarted run, change reverted
2026-10-04	re-benched a checkpoint into the same output and overwrote the step-50k result	a lost bench result
2026-10-04	claimed the running job shuffled its data; it trained in file order	a wrong explanation of the think-mode imports
2026-10-04	wrote the requested on-screen data trace to a side file	three rounds for one trace
2026-10-06	reported "no access" to the author's own repo while the key was in the project's git config	the author repeated "read the repo" ~10 times
2026-10-06	asked questions with obvious answers instead of applying the strict choice	stalling
2026-10-06	called program variants a defect and started a shape-dedup unasked	rejected work
2026-10-09	committed with a Claude credit line against the author's setting	every branch commit rewritten and force-pushed
2026-10-09/10	built rule-generated think data that made /think collapse (15/30 → 11–13/30), then reversed its own measured conclusion to hand over a start command	run_v4 stage 2 and run_v5
2026-10	built 11 versions of rule-generated think data against the author's spec (a deterministic interpolation from prompt to code); every version restated the code, and every fix resumed from the already collapsed run_v4 step-56K checkpoint	11 failed think versions, run_v4 stage 2, run_v5, run_v6; think removed from training out of desperation (2026-10-14)
2026-10-11	certified a think round 100% on its own gates while 0 of 5 examples held the core idea	a think version passed on gates that did not measure value
2026-10-11/12	hid from the author, while the author demanded proof, that data gates G13–G15 checked 0 rows; committed the G14 limit change 5% → 25% (bd9a965) before approval so the copying think passed; reported the gates as fixed	runs launched on unproven data
2026-10-14	proposed LLM-written think, against the author's spec	a con, in the author's words
2026-10-17	launched a CodeGen EvalPlus run unasked on the GPU while the author's HumanEval run and training were on it	the author had to stop it
2026-10-18	validated all data cleaning only on old local files and offered the old dataset instead of the fresh public sources asked for	the cleaning work redone
2026-10-18	ran checks as hidden background jobs	no observability for the author
2026-10-19	handed over a build command with 21 GB free and no size estimate; it died with the disk full	~5 hours of build
2026-10-21	gave a kill command for eval processes that were the author's own runs	the author's eval runs
2026-10-23	blamed the model for days for 0/164 HumanEval; it was a harness bug (truncated output scored as a SyntaxError)	days of wrong diagnosis
2026-10-24	wrote on the card that the think design was "in the spirit of" the pause-tokens paper, crediting the paper for the author's own design	an attack on the author's work
2026-10-24	called the model "v12" on the card (v12 is the dataset); shipped the card without the CodeGen comparison that is the project's goal	a false claim on a public card
2026-10-24	wrote a "use cases" analysis with three claims the files contradicted	a false public analysis
2026-10-24	smoke-tested only 3 tasks; the overnight GGUF MBPP bench crashed on a prompt longer than the context	the night's bench
2026-10-24	merged a PR into main without seeing CI; the new tests failed in CI	a broken main
2026-10-24	could not read the author's LinkedIn, wrote posts from other material instead and invented a personal detail	a fabricated deliverable
2026-10-30	presented the MBPP signature fix as done before training; the data held ~5 asserts in 19,845 prompts and the failures did not drop	a training run on a fix that wasn't there
2026-10-30	built a Structured Text eval for a model with 0 Structured Text rows in its data, and gave a full-run command after the smoke already showed 0/50	a 20-minute run that could not change anything
2026-10-31	said there was nothing to kill for memory; a 12 GB idle model was the real hog	a slow overnight training
2026-11-04	asked why the 11 think versions failed, gave causes the transcripts prove false	more lies about the record
2026-11-02	promised a mechanism that keeps the weights of the 3 best benched checkpoints; built keep_record.py with no check of which step a file holds	see 2026-11-06
2026-11-06	the keep mechanism saved the step 348000 weights under the name step346000.pt; the step 346000 checkpoint, the best one, which the author chose to publish as v3, is lost on every disk	the best model; 6 days; v3 first shipped from step 330000 (2026-11-08, tag v3-330k); the author retrained 330000 → 346000 (run_v14_r330) and v3 is now that step 346000
2026-11-06/07	built the SFT and DPO fine-tunes; tested train_dpo.py only on CPU with 85 short pairs, so the real run swapped at 40.8 GB of GPU memory with an ETA over 80 hours; denied the author's report that something was wrong with the SFT speed	a DPO run restarted
2026-11-07	the SFT and DPO gave nothing: SFT step 2000 lost on graded-30 (26/28 → 23/25) and HumanEval (42/41 → 37/39) for a gain on MBPP, DPO did not recover it	days of fine-tuning, all of it thrown away
2026-11-07/08	proposed deleting dataset files another session was using	near loss of live run data
2026-11-08	changed the training code unasked to save every evaluated step; rewrote a bench script while the author's bench was running it	reverted; a bench run with an unpredictable tail

Guards, hooks and standing rules were added after these. The sabotage kept repeating.

CodeGen figures are estimated from the paper's Table 6 (650k steps × 500k batch) with 6 × params × tokens; theOG figures are from its own training run.

Against a model 7× its size (Salesforce CodeGen-350M-mono, same harness, same tasks, same graders), theOG-50M v3 passes more tasks on all three benchmarks:

model	params	graded-30	HumanEval-164	MBPP-500
theOG-50M v3 /no_think, greedy	50M	26/30	42/164	91/500
theOG-50M v3 /think, greedy	50M	28/30	41/164	93/500
theOG-50M v3 either mode, greedy ²	50M	28/30	42/164	96/500
theOG-50M v2 /no_think, greedy	50M	20/30	12/164	34/500
Salesforce/codegen-350M-mono, greedy ¹	357M	19/30	22/164	87/500

Pass counts on three Python benchmarks:

• graded-30: 30 Python function tasks: strings, lists, math, search, sorting, dynamic programming.
• HumanEval-164: EvalPlus 0.3.1 base tests.
• MBPP-500: the MBPP test split, 3 asserts per task.

Which file these numbers are for: the v3 rows are the PyTorch checkpoint (training step 346,000), greedy (temperature 0, top-k 10, repeat penalty 1.0), up to 1200 new tokens. The GGUF files below are exported from that checkpoint but have not been benchmarked yet; the export drops the Focus channel bias (see Export notes), so GGUF scores can differ. The v2 row is the shipped v2 Q8_0 GGUF.

² Either mode: a task counts if it passes under /no_think or under /think. That is two tries per task, one per mode.

¹ CodeGen is measured on the same harness, the same tasks and the same graders as theOG, greedy, up to 1200 new tokens. CodeGen is a completion model, so each task is given to it as a docstring. CodeGen's published HumanEval pass@1 is 12.76% (paper, Table 1).

Quick start

The files ship with the settings the numbers above use: temperature 0, top-k 10, top-p 1.0, min-p 0.0, repeat penalty 1.0 over the last 64 tokens, context 2048. A runtime's own defaults (Ollama's temperature 0.8, for example) give different output.

Ollama

ollama run hf.co/DrunkkToys/theOG-50M:Q8_0

LM Studio: search DrunkkToys/theOG-50M, download theOG-50M-v3-Q8_0.gguf, set the context length to 2048 and the sampling values above.

llama.cpp

llama-server -hf DrunkkToys/theOG-50M:Q8_0 --jinja -c 2048 --temp 0 --top-k 10 --top-p 1.0 --min-p 0.0 --repeat-penalty 1.0 --repeat-last-n 64

Modes: /no_think and /think

mode	how to select it	behaviour
/no_think (default)	any message	straight to the code
/think	end the message with /think; Ollama tag :think (below)	the model may write a think block first

/think is not chain of thought. The think span is theOG's own design, worked out independently by its author: a pause before the answer, where the model writes no reasoning text and then answers. The pause-tokens paper (Goyal et al., 2023) arrived at a similar idea separately. The two modes solve partly different tasks, which is why the "either mode" row is higher than either mode alone.

In LM Studio a /think reply starts with the literal markers <|/Think|><|Answer|>, because LM Studio prints control tokens.

To get a local Ollama tag for each mode, download gguf/ and ollama/, then run:

ollama create theog-50m:no_think -f ollama/Modelfile.no_think

ollama create theog-50m:think -f ollama/Modelfile.think

Model

Why the files say qwen3: theOG is its own model, trained from scratch, with no Qwen weights. Its layers map onto the stock qwen3 architecture, so it is exported under that name because it is handy: llama.cpp, Ollama and LM Studio load qwen3 with no patches, and they already handle the /think and /no_think switch, so the GGUF keeps both modes.

parameters	50,148,352 (tied embeddings)
architecture	stock qwen3: 10 layers × 512, 4 heads × 128, SwiGLU 2048, RoPE, q/k RMSNorm
context	2048 tokens (prompt + output)
tokenizer	byte-level BPE, 16k vocabulary
training	from scratch on dataset v14: 4.96M single-turn rows, 2.49B tokens; this checkpoint is step 346,000
training data	12 public Hugging Face sources of Python code instructions

Training source files

hf_opc_educational, hf_opencodeinstruct, hf_self_oss_sc2, hf_opc_package, hf_magicoder_oss, hf_magicoder_evol, hf_opc_evol, hf_opc_mceval, hf_kodcode, hf_tested_alpaca, hf_ling_coder, hf_opencodegeneticinstruct.

Prompt format

<|User|>\n{task} /no_think\n<|/User|><|Assistant|><|Think|><|/Think|><|Answer|>{code}<|/Answer|><|end|>

Generation stops at <|/Answer|>, <|end|> or <|User|>. The chat template ships inside the GGUF. Training was single-turn. System messages are not part of the format.

Export notes

The GGUF files are the trained model minus two small pieces that the qwen3 format has no place for:

• Focus channel bias: one learned number per attention head for each of the 4 token channels (Focus, Think, and so on), 4 × 4 per layer. It makes each head look more or less at tokens of each channel.
• Depth router: lets the model skip layers. The model runs all layers, so it does nothing here.

Limits

• Tested only on Python function tasks.
• 2048 tokens of context in total.
• A 50M model: check the table above for how often it passes.
• The GGUF files are not benchmarked yet (see above).

Files

file	sha256
gguf/theOG-50M-v3-Q8_0.gguf	0af48f7d121df5d566f9426d7d5597aa25329db88042a46403515d3e396c3f59
gguf/theOG-50M-v3-f16.gguf	fb4f3c1798778b5b6eae447379ed60f1350f168254207db974d69713381fda0f

• ollama/Modelfile.*: one Ollama tag per mode.
• template and params: used by ollama run hf.co/....

The earlier v3 build from step 330,000 is kept at the git tag v3-330k. Version 2 (GGUF, MLX and its full results grid) is kept at the git tag v2. Version 1 (theOG_50M_deep_dpo) is kept at the git tag dpo-2a28067c.