Blame for README.md

---
language:
  - en
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
pretty_name: LissaNano-Instruct
tags:
  - text-generation
  - llama
  - my-little-pony
  - mlp
  - fanfiction
  - small-language-model
  - from-scratch
  - lissa
  - roleplay
  - rp
  - character
  - edge-deployment
model_card_authors:
  - CastIronMind

---

<div align="center">
  <img src="logo.png" width="60%" alt="LissaNano-Instruct" />
</div>

<hr>

<div align="center" style="line-height: 1;">
  <a href="https://huggingface.co/CastIronMind/LissaNano-Instruct/blob/main/LICENSE" target="_blank" style="margin: 2px;">
    <img alt="License" src="https://img.shields.io/badge/license-Apache%202.0-blue" style="display: inline-block; vertical-align: middle;" />
  </a>
  <a href="https://huggingface.co/CastIronMind/LissaNano-Instruct" target="_blank" style="margin: 2px;">
    <img alt="Parameters" src="https://img.shields.io/badge/parameters-32M-2ea44f" style="display: inline-block; vertical-align: middle;" />
  </a>
  <a href="https://huggingface.co/CastIronMind/LissaNano-Instruct" target="_blank" style="margin: 2px;">
    <img alt="Context" src="https://img.shields.io/badge/context-1%2C024%20tokens-orange" style="display: inline-block; vertical-align: middle;" />
  </a>
  <a href="https://huggingface.co/CastIronMind/LissaNano-Instruct" target="_blank" style="margin: 2px;">
    <img alt="Vocabulary" src="https://img.shields.io/badge/vocab-16%2C384-blue" style="display: inline-block; vertical-align: middle;" />
  </a>
  <a href="https://huggingface.co/CastIronMind/LissaNano-Instruct" target="_blank" style="margin: 2px;">
    <img alt="Weights" src="https://img.shields.io/badge/weights-64 MB-lightgrey" style="display: inline-block; vertical-align: middle;" />
  </a>
</div>

# LissaNano-Instruct

## 1. Introduction

**The smallest My Little Pony language model we could find: 32 million parameters and 64 MB of weights.** This is the supervised checkpoint, tuned for character-driven conversation.

- **Trained from scratch.** No distilled weights and no borrowed backbone: a Llama architecture with a 16,384-token vocabulary, pretrained on a fandom corpus and then supervised on dialogue.
- **Built for character work.** The supervised stage uses roleplay and persona data, so the model spends its capacity on voice, character and scene.
- **Dialogue format is required.** The model expects the exact `<system>` / `<user>` / `<assistant>` layout described in section 6.
- **Trained by one person on one consumer GPU.** No cluster, no budget, no institutional support.

## 2. Model Overview

| field | value |
|---|---|
| Type | Causal Language Model |
| Training stage | Pretraining from scratch, then supervised fine-tuning |
| Architecture | Llama |
| Parameters | 31,990,272 (32M) |
| Layers | 8 |
| Hidden size | 512 |
| Attention heads | 16 (2 key/value heads) |
| Head dimension | 32 |
| Intermediate size | 1,536 |
| Context length | 1,024 tokens |
| Vocabulary | 16,384 (byte-level BPE) |
| Tied embeddings | yes |
| Precision | bfloat16 |
| License | Apache 2.0 |

## 3. The Lissa family

| model | parameters | weights | best for |
|---|---:|---:|---|
| [LissaNano-Base](https://huggingface.co/CastIronMind/LissaNano-Base) | 32M | 64 MB | continued pretraining, research |
| [LissaNano-Instruct](https://huggingface.co/CastIronMind/LissaNano-Instruct) | 32M | 64 MB | phones, single-board computers, embedded use |
| [LissaBrat-Base](https://huggingface.co/CastIronMind/LissaBrat-Base) | 103M | 206 MB | continued pretraining, research |
| [LissaBrat-Instruct](https://huggingface.co/CastIronMind/LissaBrat-Instruct) | 103M | 206 MB | laptops, longer and more coherent text |
| [Lissa 1.5B Instruct](https://huggingface.co/CastIronMind/Lissa-1.5B-Instruct) | 1.45B | 2.9 GB | a full roleplay partner |
| [Lissa 1.5B Base](https://huggingface.co/CastIronMind/Lissa-1.5B-Base) | 1.45B | 2.9 GB | pretraining, research |

**GGUF quantizations:** [LissaNano-Instruct-GGUF](https://huggingface.co/CastIronMind/LissaNano-Instruct-GGUF) — 15 files from Q8_0 down to IQ2, each with measured perplexity against this bf16 checkpoint.

**Demo:** [Lissa Playground](https://huggingface.co/spaces/CastIronMind/Lissa-Playground) — compare quantizations side by side in the browser, no server.

## 4. Requirements

```
pip install torch transformers
```

No custom kernels and no exotic dependencies: this is a plain Llama checkpoint and loads with any recent `transformers`.

## 5. Quick start

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "CastIronMind/LissaNano-Instruct"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16)
model.eval()

SYS, USR, AST, EOS = 2, 3, 4, 1

system = "You are Twilight Sparkle, a studious purple unicorn who loves books."
user = "Twilight, I found a strange book in the library."

ids = [SYS] + tok.encode(system, add_special_tokens=False) \
    + [USR] + tok.encode(user, add_special_tokens=False) + [AST]
x = torch.tensor([ids], dtype=torch.long)

out = model.generate(x, max_new_tokens=120, do_sample=True,
                     temperature=0.8, top_p=0.9, repetition_penalty=1.1,
                     pad_token_id=0, eos_token_id=EOS)
print(tok.decode(out[0][len(ids):], skip_special_tokens=True))
```

## 6. Prompt format

The model was supervised on exactly this layout:

```
<system>{persona}<user>{message}<assistant>{reply}<eos>
```

Tags are concatenated without spaces. A non-empty system message is required: every supervised example carried one.

**Special tokens.** `0 <pad>`, `1 <eos>`, `2 <system>`, `3 <user>`, `4 <assistant>`.

**Sampling.** Temperature 0.8, top_p 0.9, repetition penalty 1.1.

## 7. Data

**Pretraining.** Fandom prose and forums from the My Little Pony community, about 1 billion unique tokens, two epochs.

**Supervised fine-tuning.** Roleplay and persona dialogues with the `<system>` / `<user>` / `<assistant>` layout, two epochs.

## 8. Evaluation

This checkpoint is tuned for dialogue, so held-out perplexity is reported on the base model card, where it is measured on the same frozen set for every model in the family: see [LissaNano-Base](https://huggingface.co/CastIronMind/LissaNano-Base).

What the supervised stage targets: character voice, in-persona dialogue, and stable formatting across turns.

## 9. Hardware and cost

| item | value |
|---|---|
| Training hardware | one consumer GPU |
| Authors | one person |
| Weights on disk | 64 MB (bfloat16) |
| License | Apache 2.0 |

## 10. Intended use

- roleplay bots and character agents in chat applications
- embedded and offline deployments: games, visual novels, toys
- experiments with character consistency in small models
- further fine-tuning on a specific character; the model retrains in minutes

## 11. License

Apache License 2.0. Use it, fine-tune it, quantize it, embed it, and publish your own versions.




- ПАЦАНЫ, это малышок, она генерирует текст, и это иногда даже похоже на смысл!