Opus Research
Independent research on small language models — pretraining from scratch, post-training, and the practical engineering around getting models to run on constrained hardware.
We publish weights, datasets, and methodology, including the parts that did not work.
Models
opus-1.5 — 0.88B parameter language model pretrained from scratch on 2× RTX 4090. Trained in 42 hours; energy use documented in the model card.
opus-2.0 — successor model, targeting a larger parameter count. Training is currently paused: we do not have the compute budget to continue. The architecture and data pipeline are ready; the run is blocked on GPU access rather than on research.
gemma-2-2b-thinking — experiment in adding explicit chain-of-thought to a base model that lacks it. LoRA on Gemma 2 2B, trained on our own reasoning dataset. Training loss 3.56 → 0.69 over 566 steps. Exploratory; not benchmarked.
bernard-gpt-oss-20b-lora — style-transfer LoRA for gpt-oss-20b, a 21B mixture-of-experts model. Notable mainly for the training notes: a four-run ablation showing that the checkpoint with the worst validation loss was the correct one to ship, plus a set of gpt-oss-specific engineering gotchas we could not find documented elsewhere.
Datasets
opus-thinking-10k
— ~9,000 chain-of-thought examples, used to train gemma-2-2b-thinking.
opus-thinking — earlier iteration of the same dataset.
What we work on
- Pretraining small models end-to-end on consumer hardware
- Post-training: LoRA, style transfer, capability injection
- Dataset construction and filtering methodology
- Quantization and CPU inference for models that would otherwise need a datacenter
Our constraint is compute, not ideas. Everything above was produced on consumer GPUs, and the ceiling that imposes is the main thing limiting what we publish next.
Contact
Open to collaboration, compute partnerships, and grant programs. Reach us through the organization page.