jarvis-112m

jarvis-112m is a compact, decoder-only causal language model focused primarily on Russian text and dialogue. Its architecture, tokenizer, data pipeline, training loop and local generation code are implemented in lubenkov/jarvis.

The repository contains the source code; this Hugging Face repository contains the fine-tuned PyTorch checkpoint and the tokenizer required to use it. The checkpoint is intended for local experimentation, learning and research.

Model details

Property	Value
Model type	Autoregressive decoder-only Transformer
Primary language	Russian (ru)
Parameter count	Approximately 111.7M (about 112M) for the main preset
Layers	14
Attention heads	12
Hidden size	768
Feed-forward size	2,048
Context length	768 tokens
Vocabulary	16,384-token byte-level BPE
Position encoding	RoPE
Normalization	RMSNorm
Feed-forward block	SwiGLU
Checkpoint	jarvis-112m-sft-fp16.pt
Tokenizer	tokenizer.json

The checkpoint is a PyTorch .pt file saved by the project. It stores the model weights and configuration; it is not a transformers AutoModelForCausalLM checkpoint. Use the custom model code in the GitHub repository to load it. The tokenizer and checkpoint must be kept together.

Intended use

• Local experimentation with a small Russian-language causal language model.
• Educational use: inspecting a Transformer implementation and a full pretraining/SFT pipeline.
• Generating short Russian text or trying a lightweight conversational model.

Out-of-scope use

Do not use this model as the sole basis for medical, legal, financial, employment, educational or other high-impact decisions. It has not been evaluated for high-stakes use, robust instruction following, factual reliability, privacy, or safety. Do not rely on it to provide professional advice, and do not send sensitive personal or confidential information to a deployment that you do not control.

How to use

The easiest supported inference path is the repository's chat.py script. Install a PyTorch build suitable for your machine (use the official PyTorch selector for CUDA), then clone the code and install its requirements:

git clone https://github.com/lubenkov/jarvis.git
cd jarvis
python -m venv .venv
# Activate the environment for your shell, then:
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip install --upgrade huggingface_hub

Download the checkpoint and matching tokenizer into the repository's ignored checkpoints/ directory:

python - <<'PY'
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="Eragon1/jarvis-112m",
    local_dir="checkpoints/hf-jarvis-112m",
    allow_patterns=["jarvis-112m-sft-fp16.pt", "tokenizer.json"],
)
PY

Start a local chat:

python chat.py \
  --ckpt checkpoints/hf-jarvis-112m/jarvis-112m-sft-fp16.pt \
  --tokenizer checkpoints/hf-jarvis-112m/tokenizer.json

For one prompt, use generate.py:

python generate.py \
  --ckpt checkpoints/hf-jarvis-112m/jarvis-112m-sft-fp16.pt \
  --tokenizer checkpoints/hf-jarvis-112m/tokenizer.json \
  --prompt "Привет! Расскажи короткую историю."

The scripts select CUDA when available and otherwise run on CPU. Generation supports temperature, top-k/top-p sampling and a repetition penalty; see python chat.py --help and python generate.py --help for options. The model is not currently deployed by an Inference Provider; inference is local through the project code.

Training data and process

The training scripts load source datasets from Hugging Face. The repository does not include the raw corpora.

Training source	Dataset(s)
Russian web text	HuggingFaceFW/fineweb-2, rus_Cyrl/train
Russian articles	IlyaGusev/gazeta, train split
Dialogue / instructions	IlyaGusev/saiga_scored, Vikhrmodels/GrandMaster-PRO-MAX, Vikhrmodels/Veles-2.5, Den4ikAI/russian_instructions_2, inkoziev/jokes_dialogues, and the Russian subset of OpenAssistant/oasst1

The default training recipe in model/config.py is two-stage:

Phase	Configured token budget	Sampling mixture	Objective
Pretraining	1.2B tokens	68% web, 27% chat, 5% articles	Next-token prediction
SFT	150M tokens	95% chat, 5% web	Assistant-response loss for chat examples

The preparation pipeline applies source-specific filters, normalizes chat roles, removes duplicate examples and filters for Cyrillic content. Tokenized validation data is held out per source. The ratios above are configured source-sampling weights; they are not counts of unique documents.

Reproducibility note: the published inference checkpoint contains model weights and configuration, but not the full training log, exact processed-token count, hardware details or dataset revision hashes. The budgets above describe the checked-in training configuration; they should not be treated as an independently audited count for this specific checkpoint. The preparation script does not pin source dataset revisions, so a later rebuild may use updated source data.

Evaluation

The repository reports bits per character (BPC; lower is better) on a 600,000-character held-out sample from each source. These are small, domain-specific language-modeling measurements, not a comprehensive evaluation of generated answers.

Model	web_ru	texts_ru	chat_ru
jarvis pretraining checkpoint	1.18	1.02	0.92
jarvis SFT checkpoint	1.22	1.07	0.91
ai-forever/rugpt3small_based_on_gpt2	1.07	0.93	0.98

The baseline uses its own tokenizer and a different context length, so treat cross-model differences as approximate. The full measurements are available in results/bpc.json; evaluation code is in tools/bench_bpc.py.

Limitations and risks

• Small model: at approximately 112M parameters, it has limited capacity compared with modern large language models. It may be incoherent, repetitive or unable to follow multi-step instructions reliably.
• Factuality: generated text can be wrong or fabricated. The model does not provide sources or calibrated confidence.
• Language coverage: training is focused primarily on Russian. Performance in other languages is not established.
• Safety and bias: the model has not undergone a comprehensive safety evaluation. Web and conversational corpora can contain bias, personal information or unsafe material, which may be reflected in outputs.
• Data provenance: source dataset licenses and terms differ. Review each dataset card and its license before use or redistribution. No raw training data is included with the checkpoint.
• Context: the configured context window is 768 tokens; long prompts and conversations are truncated by the local chat code.

License

The model repository declares the checkpoint under Apache-2.0, matching the source-code license in lubenkov/jarvis. This declaration does not replace the terms of the upstream training datasets. Users are responsible for reviewing and complying with those source terms.

Citation

If you use or discuss this model, please cite the project repository:

lubenkov. jarvis: GPT-style transformer written from scratch in PyTorch.
https://github.com/lubenkov/jarvis

🇷🇺 По-русски

jarvis-112m — небольшая авторегрессионная Transformer-модель, ориентированная прежде всего на русский текст и диалоги. Веса лежат в jarvis-112m-sft-fp16.pt, совместимый токенизатор — в tokenizer.json. Для запуска используется код из lubenkov/jarvis; формат весов не совместим напрямую с transformers.AutoModelForCausalLM.

Модель подходит для локальных экспериментов и обучения на примере. Она может ошибаться, повторяться и выдавать небезопасный или предвзятый текст; не используйте её для важных решений и не отправляйте ей конфиденциальные данные. Опубликованная конфигурация обучения задаёт 1,2 млрд токенов pretraining и 150 млн токенов SFT.