Vantora Labs

Vantora Labs runs pretraining and training-method experiments at very small parameter counts. We release language models under 12K parameters and complete text-to-speech pipelines under 300K, each with its weights and measured benchmark results.

Research

Low-parameter language modeling. Architecture and optimizer comparisons at a fixed budget: pure transformer against hybrid Mamba-2 + attention, Muon against AdamW, and how standard training practice holds up when a full run takes minutes instead of weeks.

Compressed speech synthesis. The full text-to-speech stack distilled into a few hundred kilobytes: a dictionary-based grapheme-to-phoneme front end, a duration predictor, an acoustic model, and a waveform decoder, all running faster than real time on CPU.

Training methodology at small scale. Constant learning rates against decay schedules, loss balance between pipeline stages, and why a loss that looks converged can still produce unintelligible audio.

Computing

All training and evaluation runs on a single workstation:

The constraint is deliberate. At these sizes a full training run finishes in minutes to hours, so architecture and optimization decisions can be tested rather than assumed.

Models

Language models. The two language models were trained on the same 100M-token FineWeb-Edu slice with the same batch size and step count. Vantora-Micro is a pure Llama-style transformer; Vantora-Micro-Hybrid places a Mamba-2 SSM in parallel with attention (the Falcon-H1 SA_M block pattern). The hybrid won 6 of 7 benchmark categories, but it trains 11.5x slower, and on PIQA, HellaSwag and ARC-Easy the difference between the two is 0.5-2%, within the noise floor at this scale. On this task the published conclusion is that the SSM does not justify its training cost, although it was a clear win earlier on TinyStories, where narrative memory mattered. Vantora-Micro scores near the four-choice random baseline, and the hybrid's margin over it comes partly from a continuation-length artifact in the Code Completion category, documented on both cards.

Text-to-speech. Vocetta-181K is the first-generation pipeline, distilled stage by stage from a larger teacher. Vocetta2-276k is the second generation, with a larger decoder and joint fine-tuning of the acoustic model and decoder. Across the two releases, diverse-set word error rate improves from 0.23 to 0.101, and SCOREQ from 1.03 to 2.04. Both run faster than real time on CPU and need no GPU at inference.

Reporting

Language model scores use the official BananaMind runner on its SHA-256 verified 350-item split. Speech numbers use Whisper small as the transcription judge for word error rate, and SCOREQ and DNSMOS for naturalness. Each result is reported with the baseline it should be compared against, and limitations are documented on the model cards themselves, including the continuation-length artifact above and the gap between templated and conversational text.

Each release is MIT licensed and ships its weights and runtime code; the speech releases also include their training and benchmark scripts. Bundled third-party data keeps its original license, noted per repository.

Contact

Issues and questions are welcome on any model repository.