qwrt/Swedish0.2M

Overview

This is a 189K parameter language model, using Qwen3 architecrue, pretrained 4 epochs on a textdataset consisting of 15 billion characters of swedish books. It uses utf-8 byte level encoding, so it basically sees and generates individual characters instead of word groups. More specific characters like "åäö" and emojis are represented with more than one token in the utf8 encoding. Due to the model's size, it is not made for generating meaningful text.

from transformers import Qwen3ForCausalLM
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"
model = Qwen3ForCausalLM.from_pretrained("qwrt/Swedish0.2M").to(device)

seed = torch.tensor(list("Hej på di".encode()), dtype=torch.long)[None].to(device)
out = model.generate(seed, max_new_tokens=128,
                                     do_sample=True, temperature=0.8)
print(bytes(out[0].tolist()).decode("utf-8", errors="replace"))

output:

Hej på dig.

– Så fort jag inte ligger så tvättligt att säga, sa hon. Hon skakade på huvudet mot sitt tystnade och håller flickan

Note: The model will start generating nonsense after 128 since it was trained for a context size of 128

Correct top byte-level prediction:

Class	ACC (%)
Rödluvan(Sv)	63.5
Text formated midi	8.0
English text	27.8

*For every token(or utf-8 byte) in a textfile, how often does the model predict this token as the most probable given all tokens that come before, (or truncated to the models context window). Basically how often the model predicted the next letter correct.

Background info

The training of this model was made on an RTX 4070 laptop and took about 16 hours.

Estimating vocabulary

By examining 100 000 samples with 128 character each with the model, it is possible to give an estimate of the model total vocabulary of swedish words. To know if a word is generated by the LM is real or not, we compare it to a large vocabulary list. This vocabulary list was made by extracting all words from the training set. A reasonable estimate for the models vocabulary size would be around 20 000 words. The following graph shows how the amount of "real" words diminishes when analysing words used by the model, ordered from most frequent to less. frequent. For example, 99.9% of the top 14 000 words used by the model are real words, and 99% of the model's top 26 000 words are real.

Do whatever with the model.