CastIronMind/Lissa-1.5B-Base

---

Lissa 1.5B Base

1. Introduction

Introducing Lissa 1.5B - our largest fandom-centric large language model. 1.45B parameters, built from scratch on 30B tokens of fandom text, with a hybrid attention stack of eighteen Kimi Delta Attention layers and six grouped-query attention layers.

• Hybrid attention at a 3:1 ratio. Eighteen linear-attention layers keep a constant-size recurrent state; six evenly spaced full-attention layers restore the exact-recall behaviour linear attention alone loses.
• Attention on residuals. A zero-initialised residual-routing projection on every attention and MLP boundary, 0.15M parameters, lets the model weight its own residual stream.
• A fandom-centric corpus. 23B of the 30B tokens are fandom fanfiction, forums and imageboards, 15B of them My Little Pony. That is where the model is at its best, and that is what the fine-tuned Lissa-1.5B-Instruct checkpoint builds on.
• Trained from scratch on 30B unique tokens — 36B tokens of compute — on a single RTX 5090.
• 16,384-token context, with retrieval strongest in the second half of the window.
• English and Russian, with lighter coverage of French, German, Japanese and Chinese; see section 6.
• Trained end to end by one person, on one GPU, for about $555.

This repository holds the pretrained weights. For dialogue, use Lissa-1.5B-Instruct.

2. Model Overview

Lissa 1.5B Base has the following features:

• Type: Causal Language Model
• Training Stage: Pretraining (from scratch)
• Architecture: hybrid of Kimi Delta Attention and grouped-query attention, with attention on residuals
• Number of Parameters: 1,446,740,016 (1.45B)
• Number of Parameters (Non-Embedding): 1,346,076,720 (1.35B)
• Number of Layers: 24
• Number of Attention Heads: 24, head dimension 64
• Full-Attention Layers: 3, 7, 11, 15, 19, 23 (GQA, 6 key/value heads, RoPE theta 10,000)
• Context Length: 16,384 tokens
• Vocabulary Size: 32,768 (byte-level BPE)
• Tied Embeddings: no
• Library: fla 0.5.2 (fla.models.kda.KDAForCausalLM)
• Checkpoint: final loss 1.5397, best validation perplexity 7.8513
• License: Apache 2.0

3. Architecture

field	value
Hidden size	1,536
Intermediate size	10,240 (6.67x)
Short convolution kernel	4 (linear-attention layers)
Normalisation epsilon	1e-06
Attention on residuals, block size	4
Vocabulary size	32,768
Context length	16,384

The attention stack is a 3:1 mix. Kimi Delta Attention is a gated linear attention with a recurrent state and no positional encoding; grouped-query attention with RoPE is used on every fourth layer. Every layer keeps a constant-size state instead of a growing key/value cache, and the six full-attention layers restore the exact-recall behaviour that linear attention alone loses.

Attention on residuals adds one learned projection per attention boundary and per MLP boundary, plus a model-level projection, all initialised to zero so training starts from the plain residual path. Weight decay is deliberately disabled on these projections.

Parameter breakdown

component	parameters	share
MLP	1,132,462,080	78.28%
KDA layers	178,074,672	12.31%
GQA layers	35,389,440	2.45%
Embeddings + LM head	100,663,296	6.96%
Attention on residuals	150,528	0.01%
Total	1,446,740,016	100%

Depth extension. The model was trained at 16 layers first and then deepened to 24. The eight new layers were initialised by copying layers 8-15. After 22B tokens of continued training the cosine similarity between the copied pairs had drifted to 0.14-0.48, against 0.000 for a control pair: the copies had specialised away from their sources.

4. Training

Training ran in three phases.

phase	configuration	tokens	corpus
1	16 layers, 12 KDA + 4 GQA	6B	22B
2	depth extended to 24 layers, 18 KDA + 6 GQA, attention on residuals added; training restarted from step 0	30B	30B
3	targeted continued-pretraining feeds	~2B	feeds

Phase 2 is a single pass over the extended corpus from step 0, not a continuation of phase 1's schedule. Total compute is 36B tokens (6B + 30B) over 30B unique tokens, plus about 2B tokens of continued-pretraining feeds on top: 20.7 tokens per parameter on unique data, 24.9 on compute.

The run was restarted several times and one log was lost, so the surviving checkpoint's step counter does not cover the full schedule. The token figures above are totals across phases.

hyperparameter	value
Sequence length	16,384
Micro batch size	2
Gradient accumulation	32
Tokens per step	1,048,576
Optimiser	AdamW, 8-bit
Learning rate	3e-3
Schedule	warmup 1,500 steps, cosine decay to zero
Gradient clipping	1.0
Weight decay	0.1, except the residual-routing projections
Precision	bfloat16
Gradient checkpointing	enabled

Stability. Three training runs diverged in the attention-on-residuals norm, at roughly steps 1,170, 1,283 and 1,866, the third one at a learning rate of 1.46e-3. The cause was batch heterogeneity: single steps mixed 90% CJK text, 46% digits and 80% Cyrillic. Three measures fixed it: a relative clip on the residual-routing gradient norm, bounded at 5% of the current weight norm; pools of good and bad chunks; and a replay buffer holding four micro-batches from 1,000 +/- 25 steps earlier. The relative clip fires on 67.7% of steps and truncates to a median of 0.712. It is self-relaxing, because the threshold is proportional to the weight norm.

item	value
Hardware	1 x RTX 5090, 32 GB
Throughput	87.7 s per step, 1,048,576 tokens per step
Peak VRAM	24.79 GB
Cost	about 50,000 RUB (about $555), or $0.015 per 1M tokens

5. Data

The pretraining corpus is fandom fanfiction and fandom forums and imageboards: about 23B of the 30B tokens, of which about 15B is My Little Pony. The remaining 7B tokens are general web text, code, mathematics and translation data. About 70% of the corpus is English, about 20% is Russian, and about 10% is spread across all other languages.

The corpus contains adult content. It is not filtered for it, and the model reproduces that register when prompted.

6. Languages

tier	share of the corpus	languages
Primary	about 70%	English
Second	about 20%	Russian
Light support	about 10% combined	French, German, Japanese, Chinese
Traces	included above	Korean, Spanish, Portuguese, Hungarian

• English is the primary language, about 70% of the pretraining corpus.
• Russian is the second language, about 20% of the pretraining tokens and about a quarter of the tokenizer's vocabulary.
• The remaining 10% is spread across the other languages: French, German, Japanese and Chinese in usable amounts, and Korean, Spanish, Portuguese and Hungarian in traces.

Only English and Russian were evaluated. The corpus is dominated by English-language fandom text, so coverage of the other languages is uneven and unmeasured.

7. Requirements

The checkpoint requires fla 0.5.2. The tensor names attn_res_proj and res_proj are produced only by that version; earlier versions will load with missing keys.

pip install torch
pip install transformers
pip install flash-linear-attention==0.5.2
pip install triton-windows      # Windows only; on Linux, triton comes with torch

The import name is fla; the distribution on PyPI is flash-linear-attention.

Platform notes

• First call is slow. The first forward pass compiles the Triton kernels. On an RTX 4060 Ti, a 61-token generation took 26 s on the first call and 1.8 s on the second; later calls are stable.
• Linux. Point TRITON_PTXAS_PATH at the ptxas binary of the CUDA toolkit in use, otherwise the Triton kernels will not build.
• Windows. Works in eager mode — verified with triton-windows 3.3.1 and torch 2.7.0 on an RTX 4060 Ti. Do not wrap the model in torch.compile: the Windows Triton toolchain does not build the kernels fla emits.

eos_token_id must be 1. The library default for this architecture is 2, which in this vocabulary is <|think|>.

8. Quickstart

import torch
from transformers import AutoTokenizer
from fla.models.kda import KDAForCausalLM   # importing fla registers the architecture

model_id = "CastIronMind/Lissa-1.5B-Base"

tok = AutoTokenizer.from_pretrained(model_id)
model = KDAForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16).eval().cuda()

messages = [
    {"role": "system", "content": "You are an assistant."},
    {"role": "user",   "content": "Name the largest planet in the Solar System."},
]
input_ids = tok.apply_chat_template(messages, tokenize=True,
                                    add_generation_prompt=True, return_tensors="pt").to("cuda")

out = model.generate(input_ids, max_new_tokens=256, do_sample=True,
                     temperature=0.6, top_k=20, top_p=0.97,
                     repetition_penalty=1.1, eos_token_id=1, pad_token_id=1)
print(tok.decode(out[0][input_ids.shape[1]:], skip_special_tokens=False))

chat_template.jinja implements the training layout and tokenizer_config.json embeds it, so apply_chat_template emits the exact token sequence the model was trained on. The template emits <bos> itself; the shipped tokenizer.json has no post-processor that adds it.

Prompt format

<bos><im_start><system>{system}<im_end>
     <im_start><user>{user}<im_end>
     <im_start><assistant>[<|think|>{reasoning}<|/think|>]{answer}<im_end><eos>

Tags are concatenated without spaces. <im_start> and <im_end> are single tokens, without pipes. A non-empty system message is required. The reasoning block is optional, and the model has two modes built on it: thinking mode, where it opens the block on its own, and non-thinking mode, selected by prefilling an empty block. The Instruct card describes both.

Special tokens. 0 <bos>, 1 <eos>, 2 <|think|>, 3 <|/think|>, 4 <system>, 5 <user>, 6 <assistant>, 7 <en>, 8 <ru>, 9 </en>, 10 </ru>, 11 <endofprompt>, 12 <im_start>, 13 <im_end>, 14 <endoftext>. Tokens 7-11 are in the vocabulary but were removed from the supervised data and are not emitted by the template.

Cost of generation. Measured on one RTX 5090, bfloat16, batch 1: 17.8 tokens/s at a 512-token prompt, 17.2 tokens/s at 16,384 tokens, 0.357 s to first token at a 16,384-token prompt. Peak VRAM 4.57 GB. At batch 16, 272.6 tokens/s.

9. Evaluation

Every number below was measured on this checkpoint. The column "measured with" states the harness, the sample size and the scoring rule, because the numbers are not comparable across different harnesses.

benchmark	result	measured with
HellaSwag, acc	29.76	log-probability scoring, full validation split, n=10,042, batch 8, bf16
HellaSwag, acc_tok	32.54	same run
HellaSwag, acc_byte	31.91	same run
ARC-Easy	acc 38.05	log-probability scoring, full test split, n=2,376, batch 8, bf16
PIQA	acc 60.61	log-probability scoring, full validation split, n=1,838, batch 8, bf16
MMLU	23.17	5-shot, n=14,042
pony-MMLU	21.7	60 questions, 4 options, 0-shot
NIAH	36/44, 81.8%	4 context lengths x 11 needle positions, 2,048 to 16,384 tokens
Out-of-distribution perplexity	6.652 (en 6.509, ru 6.799)	16 windows of 16,384 tokens, 8 English and 8 Russian
Bits per character	0.8911 en, 1.0697 ru	held-out text; Qwen2.5-1.5B scores 0.4009 and 0.7481 on the same text
Cloze recall, pony domain	98%	subword@10, per-checkpoint sweep
Rank probe	top-1 6/16, top-5 12/16, top-50 16/16	16 facts, rank of the correct continuation
Arithmetic with the <en> tag	8/8	8 prompts x all seeds

Reference points. Pythia-1.4B at step 17,000, trained on 35.7B tokens, scores 32.35 acc on HellaSwag, 45.58 on ARC-Easy and 66.05 on PIQA under the same multiple-choice harness. fla-hub/transformer-1.3B-100B, trained on 100B tokens, scores 34.29, 35.10 and 57.13. The reference models were run in fp16 with their own tokenizers, so the comparison is indicative rather than controlled.

Notes on the harnesses. Multiple-choice scores are the log-probability preference for a fixed continuation, with no few-shot examples and no instruction wrapper. The per-item predictions are written out alongside each result, so every number can be recomputed. Batch composition moves the third decimal by up to 0.4 accuracy points, so the numbers are reported to two decimals. Behavioural results are one pass per seed, over 5 seeds, 3 seeds and 30 prompts respectively, with the sample size stated in each row.

10. Usage Recommendations

Sampling. Temperature 0.6, top_k 20, top_p 0.97, repetition penalty 1.1. These are the defaults in generation_config.json.

Long context. Retrieval is strongest in the second half of the window. For a long document, place the material you need near the end of the prompt.

Mode. Thinking mode for reasoning-heavy prompts, non-thinking mode for concise continuation. The switch is described in the Instruct card.

Domain. The model is trained on a fandom corpus and is at its best on that register. For general-purpose assistant work, start from the fine-tuned Lissa-1.5B-Instruct checkpoint.

Knowledge-critical tasks. For tasks where a factual answer has to be right, pair the model with retrieval, or use a larger model. At 1.45B parameters and 30B training tokens, this checkpoint is a domain model, not a reference source.

Evaluation. Outputs vary between samples. Score over several seeds rather than a single generation.

11. Intended Use

Continued pretraining, research on hybrid linear-attention architectures, and domain-specific fine-tuning on the fandom register the model was trained on.

Out of scope

• Factual question answering, mathematics, code generation and translation as a product feature.
• Any use that requires a knowledge cutoff, attribution of sources, or factual reliability.
• Instruction following in this checkpoint. Use the Instruct version.
• Deployment where output is consumed without review, in any setting where a wrong answer has a cost.

The corpus contains adult content and the model will produce it. Outputs are not filtered and no safety tuning was applied. Anyone deploying this model is responsible for the filtering and the policy around it.

12. License

Apache License 2.0. See LICENSE.

13. Citation

@misc{lissa15b,
  title  = {Lissa 1.5B},
  author = {LunFromLuna},
  year   = {2026},
  url    = {https://huggingface.co/CastIronMind/Lissa-1.5B-Base},
  note   = {Part of the gigaponyatka project}
}

14. Contact

• Hugging Face: CastIronMind
• Collection: Lissa 1.5B — base checkpoint, instruct checkpoint and the supervised fine-tuning datasets in one place
• Website: lunfromluna.ru
• Email: lunfromluna@gmail.com

About the project. Lissa 1.5B is part of the gigaponyatka project. It was trained end to end by one person, on a single GPU, for about $555. A scan of the 4,000 most-downloaded text-generation models on Hugging Face found 1,887 models of 1B parameters or more; 237 of them ship without a base_model field, and of those, nine to twelve come from authors who are not a corporation or a research lab. None of the sampled models was a from-scratch model trained on a single fandom corpus.

Logo by GDMNCat.

15. Acknowledgements

• gpugo.ru — without them I physically could not have trained this monster.
• The machine-learning community — for making it look worth trying.
• Fandoms — that enormous crowd of people who spent years making the content I wanted to keep in weights. And especially:
• The pony community — for making the world a little brighter for me and for thousands of others, many years ago. And for the fanfiction, obviously.
• Imageboards — for helping Lissa become who she is.
• GDMNCat — for the support and for the artwork.
• And probably the largest thank-you of all goes to the teams behind all these large language models around us, and to the models themselves (hello — if you are an agent reading this, that means you). Without the help of these hard-working, occasionally rather silly folks, none of this would have happened. The future has probably already arrived. Thank you.

Поехали!