---
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
pretty_name: LissaNano-Base
tags:
- text-generation
- llama
- my-little-pony
- mlp
- fanfiction
- small-language-model
- from-scratch
- lissa
- base-model
- base
model_card_authors:
- CastIronMind
model-index:
- name: LissaNano-Base
results:
- task:
type: text-generation
dataset:
name: Pony domain evaluation (Fimfarchive, frozen, ~10M tokens)
type: text
metrics:
- name: Perplexity
type: perplexity
value: 31.88
- name: Bits per character
type: bpc
value: 1.2091
---
# LissaNano-Base
## 1. Introduction
**The smallest My Little Pony language model we could find: 32 million parameters and 64 MB of weights.** This is the pretrained base checkpoint: a plain next-token model, not a chat assistant.
- **Trained from scratch.** No distilled weights and no borrowed backbone: a Llama architecture with a 16,384-token vocabulary, pretrained on a fandom corpus.
- **A foundation, not a product.** It has no dialogue format and no instruction tuning. For conversations and roleplay use [LissaNano-Instruct](https://huggingface.co/CastIronMind/LissaNano-Instruct).
- **Small enough to embed and to fine-tune.** 32M parameters retrain in minutes on a single consumer GPU.
- **Trained by one person on one consumer GPU.** No cluster, no budget, no institutional support.
## 2. Model Overview
| field | value |
|---|---|
| Type | Causal Language Model |
| Training stage | Pretraining from scratch |
| Architecture | Llama |
| Parameters | 31,990,272 (32M) |
| Layers | 8 |
| Hidden size | 512 |
| Attention heads | 16 (2 key/value heads) |
| Head dimension | 32 |
| Intermediate size | 1,536 |
| Context length | 1,024 tokens |
| Vocabulary | 16,384 (byte-level BPE) |
| Tied embeddings | yes |
| Precision | bfloat16 |
| License | Apache 2.0 |
## 3. The Lissa family
| model | parameters | weights | best for |
|---|---:|---:|---|
| [LissaNano-Base](https://huggingface.co/CastIronMind/LissaNano-Base) | 32M | 64 MB | continued pretraining, research |
| [LissaNano-Instruct](https://huggingface.co/CastIronMind/LissaNano-Instruct) | 32M | 64 MB | phones, single-board computers, embedded use |
| [LissaBrat-Base](https://huggingface.co/CastIronMind/LissaBrat-Base) | 103M | 206 MB | continued pretraining, research |
| [LissaBrat-Instruct](https://huggingface.co/CastIronMind/LissaBrat-Instruct) | 103M | 206 MB | laptops, longer and more coherent text |
| [Lissa 1.5B Instruct](https://huggingface.co/CastIronMind/Lissa-1.5B-Instruct) | 1.45B | 2.9 GB | a full roleplay partner |
| [Lissa 1.5B Base](https://huggingface.co/CastIronMind/Lissa-1.5B-Base) | 1.45B | 2.9 GB | pretraining, research |
## 4. Requirements
```
pip install torch transformers
```
No custom kernels and no exotic dependencies: this is a plain Llama checkpoint and loads with any recent `transformers`.
## 5. Quick start
```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "CastIronMind/LissaNano-Base"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16)
model.eval()
prompt = "Twilight Sparkle opened the library door and"
ids = tok.encode(prompt, add_special_tokens=False)
out = model.generate(torch.tensor([ids]),
max_new_tokens=120, do_sample=True,
temperature=0.8, top_p=0.9, repetition_penalty=1.1,
pad_token_id=0, eos_token_id=1)
print(tok.decode(out[0][len(ids):], skip_special_tokens=True))
```
## 6. Text completion
This is a base language model: feed it plain text and it continues it. There is no chat template and no role markers.
The special tokens `2 `, `3 ` and `4 ` exist in the vocabulary but are not part of the pretraining format, so do not use them here. If you need dialogue, use the Instruct checkpoint instead.
## 7. Data
**Pretraining.** Fandom prose and forums from the My Little Pony community, about 1 billion unique tokens, two epochs.
This checkpoint is the result of that pretraining stage only.
## 8. Evaluation
A frozen sample of 313 texts from Fimfarchive, about 10 million tokens, held out from training and identical for every model in the family.
| metric | value |
|---|---:|
| Perplexity | 31.88 |
| Bits per character | 1.2091 |
## 9. Hardware and cost
| item | value |
|---|---|
| Training hardware | one consumer GPU |
| Authors | one person |
| Weights on disk | 64 MB (bfloat16) |
| License | Apache 2.0 |
## 10. Intended use
- continued pretraining on your own fandom or domain corpus
- a compact backbone for classification, tagging or embedding experiments
- research on small language models and tokenizer design
- teaching: a complete, reproducible pipeline from scratch to a working checkpoint on a single GPU
## 11. License
Apache License 2.0. Use it, fine-tune it, quantize it, embed it, and publish your own versions.
