qwrt/Swedish1M

Overview

This is a 1.1M parameter language model, using Qwen3 architecrue, pretrained 10 epochs on a textdataset consisting of 7 billion characters of swedish books. It uses utf-8 byte level encoding, so it basically sees and generates individual characters instead of word groups. More specific characters like "åäö" and emojis are represented with more than one token in the utf8 encoding. Due to the model's size, it is not made for generating meaningful text.

from transformers import Qwen3ForCausalLM
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"
model = Qwen3ForCausalLM.from_pretrained("qwrt/Swedish1M").to(device)

seed = torch.tensor(list("Hej på di".encode()), dtype=torch.long)[None].to(device)
out = model.generate(seed, max_new_tokens=1024,
                                     do_sample=True, temperature=0.8)
print(bytes(out[0].tolist()).decode("utf-8", errors="replace"))

output:

Hej på dig, men jag antar att du inte får se en skönhet längre. Du vet väl inte riktigt att du faktiskt kan få se dig själv ut från hela vårt nattduksbord? Det är det här som har hänt, men det vet du också. Du har aldrig haft någon förståelse för vad du vill. Det är inte du som ska erkänna för dig själv. Nej, det kan du göra, men jag tror att du blir glad för det här förbannat
...

Note: The model will start generating nonsense after 1024 since it was trained for a context size of 1024

Correct top byte-level prediction:

Class	ACC (%)
Rödluvan(Sv)	70
Text formated midi	35
English text	45

*For every token(or utf-8 byte) in a textfile, how often does the model predict this token as the most probable given all tokens that come before, (or truncated to the models context window). Basically how often the model predicted the next letter correct.

Background info

The goal was to make a model that is very trained. It makes it more worth it to download and use a model when every parameters is being utizied, and trained. I feel that many models, especially with larger parameter counts, seem undertrained by examining their NLL(loss value) in their training set with respect through time/iteration*. If the shape of the "loss curve" follow a clear straight line downwords (continues to improve a lot) to the last training iteration then I would deifne that model as undertrained.

Even though I trained the one million parameter model on a total of 70 billion tokens trough all epochs, I can still detect a sign of downword trend all the way to the last iteration. For reference, a more reasonable training amount for this model, lands much lower, only 20 million tokens as predicted by https://arxiv.org/pdf/2203.15556 is enough for a compute optimal performance. Perhaps a bit more since there is less information per token in this case, either way that's many orders of magnitudes less.

I think it's sensible to say this model is far from undertrained since the graph shown is a log-log plot, so if this model were trained for many magnitudes longer the gain would be minimal.

The training of this model was made on an RTX 4070 laptop and took about 70 hours. I will upload all necessary files to train it based on a dataset.

Why is the model so small? Since this was the largest model that can be reasonably trained on my GPU, and importantly, fulfilling the goal described.

*Referencing gpt3 and llama2(check Result/pretraining section): https://arxiv.org/pdf/2005.14165, https://arxiv.org/pdf/2307.09288. Nowdays, most companies do not publish this kind of graph.

Estimating vocabulary

By examining 100 000 samples with 1024 character each with the model, it is possible to give an estimate of the model total vocabulary of swedish words. To know if a word is generated by the LM is real or not, we compare it to a large vocabulary list. This vocabulary list was made by extracting all words from the training set. A reasonable estimate for the models vocabulary size would be around 20 000 to 40 000 words. The following graph shows how the amount of "real" words diminishes when analysing words used by the model, ordered from most frequent to less. frequent. For example, 99.9% of the top 20 000 words used by the model are real words.

Converting a text to a number

Using the model, it can analyse a text in swedish and based on the predictability on the letters make a number.

from huggingface_hub import hf_hub_download
import importlib.util
from transformers import Qwen3ForCausalLM
import torch

repo="qwrt/Swedish1M"

file_path = hf_hub_download(
    repo_id=repo,
    filename="textnumberlib.py"
)

spec = importlib.util.spec_from_file_location("textnumberlib", file_path)
modul = importlib.util.module_from_spec(spec)
spec.loader.exec_module(modul)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = Qwen3ForCausalLM.from_pretrained(repo).to(device)

text="""En specifik text"""
combined=modul.text_to_bits("\n"+text, model, device)
number=modul.bits_to_tal(combined)
print(number)

29131937837

To convert the number back to a text:

number=29131937837

bits=modul.bits_from_tal(number)
text=modul.bits_to_text("\n", bits, 50, model, device, write=True).replace("\n", "⏎")

En specifik text som hade skrivits ut var en stor

I predict that with training larger models, reaching lower loss, could encode text to smaller numbers overall.

Do whatever with the model.