xdchinza/solasai-scratch-mini

SolasAI

A transformer language model written and trained from scratch. No pretrained weights, no downloaded checkpoint, no tokenizer from any library. Everything in net.py is hand-written PyTorch: the attention, the RMSNorm, the MLP, the positional embeddings, the byte-level tokenizer, the loss and the training loop.

It is small and it is honest about it. This is a demonstration of a complete working pipeline, not a competitor to any existing model. It has no world knowledge. It cannot count reliably. It knows eight conversation topics in two languages. Everything it says comes from a synthetic template corpus that ships with the code.

Specs

Parameters	4,931,072 (4.93M)
Architecture	decoder-only transformer, pre-LN
Layers	6
d_model	256
Attention heads	8 (32 per head)
FFN	1024 (4x)
Context	512 characters (not tokens)
Vocabulary	275 characters
Final loss	0.0701
Training	1600 steps, 590 s on one RTX 3060 Ti
Pretrained weights	none
Weights file	solas.pt (19 MB)

Cost: one consumer GPU, ten minutes, no data purchase.

Training data

A synthetic template corpus, 18,876 lines, built by net.py. It is generated deterministically from a seed, so the corpus is reproducible from the code alone.

9,732 lines  English
9,144 lines  Russian

Each line is a turn. Turn boundaries use explicit markers rather than newlines, because a char-level model cannot otherwise learn where one speaker stops:

U: привет, как дела
S: Нормально, настроение рабочее. Чем займёмся?
U: what is your name
S: My name is SolasAI.

232 unique Russian and 133 unique English question phrasings are generated across 8 intents. Each intent is present in both languages back to back, which is the contrast that teaches "answer in the language you were asked in."

What it can do

Verified in the shipped console chat, on a clean session.

Answers in the language you use — 9 of 9 test prompts correct:

you ▸ привет
<SolasAI> Здравствуйте! Слушаю вас. Что подсказать?

you ▸ how are you
<SolasAI> I am doing well, working and answering questions.

you ▸ who are you
<SolasAI> I am a small language model named SolasAI.

you ▸ what is your name
<SolasAI> My name is SolasAI.

Holds its own name without a single corrupted letter:

you ▸ как тебя зовут
<SolasAI> Меня зовут SolasAI. Я пишу текст, считаю и отвечаю на вопросы.

Keeps a multi-turn thread for about 5 turns, which is roughly what a 512 character window holds in practice.

Refuses to write code — 13 of 18 phrasings tested, 62 Russian and 64 English code-request phrasings in the corpus:

you ▸ напиши код
<SolasAI> К сожалению, я не программист. С удовольствием поговорю о чём-нибудь ещё.

you ▸ write me code calculator on python
<SolasAI> Sorry, I cannot help with code, I am only a language model.

What it cannot do

Be clear about this, because the limits are real:

• No facts. The corpus is templated sentences. It has no knowledge of the world beyond the strings in net.py.
• Arithmetic is unreliable. 7 × 8 returns 42, not 56. A char-level tokenizer gives the model no positional access to digits, so it memorises the shape of an answer instead of computing it.
• Bare keywords fail. Single-word prompts with no verb, like код or show code, fall through to a neighbouring intent. Refusals fire on 13 of 18 phrasings rather than all of them.
• No pretraining, therefore no generalisation. 4.93M parameters trained for 590 seconds can memorise a template corpus. It cannot reason.

Run it

Windows:

start.bat

Any platform:

pip install torch
python chat.py

Commands: /bye, /clear (resets dialogue history), /info, /noise on|off.

Generation is CPU-friendly; it uses the GPU when one is available.

Train it yourself

python train_net.py

Roughly ten minutes on a single 8 GB card. The corpus is regenerated from the seed, so the default run reproduces the shipped weights closely.

To make it speak differently, edit the QA and QA_EN dictionaries in net.py and retrain. To make it bigger, edit the model config at the top of train_net.py:

D, LAYERS, HEADS, FF, CTX = 256, 6, 8, 1024, 512

Reuse, modify, build on it

Apache-2.0. Download the weights, retrain, merge, quantise, modify the architecture, or use it as a base for your own fine-tune. No restriction beyond the license.

The checkpoint is a plain torch.save state dict:

import torch
ck = torch.load("solas.pt", weights_only=False)

Honest note on what this is

This is a 4.93M parameter model built in ten minutes with no data purchase. It does not know anything. What it demonstrates is that a complete pipeline — architecture, tokenizer, corpus, training, serving — fits in four short files and runs on one consumer GPU.

The interesting part is not the answers. It is that the loss curve, the language switching and the refusal behaviour are all reproducible from source in this repository, and that you can change the corpus and get different behaviour in ten minutes.