Lythri

Hugging Face | ModelScope | Technical Report

Introduction

Lythri-Wink-2B-pt is the pre-trained base model behind Lythri-Wink-2B. It has 2.52B total parameters and 1.48B activated parameters.

Lythri-Wink-2B-pt is structurally sheared from google/gemma-4-E2B and recovered through teacher distillation on approximately 813M tokens at an 8K training context length. It is intended for continued pre-training, supervised fine-tuning and researchβ€”not direct assistant-style conversation.

Architecture

Gemma4-E2B Lythri-Wink-2B-pt
Total parameters 4.63B 2.52B
Activated parameters 2.28B 1.48B
Decoder layers 35 31
PLE dimension 256 128

The FFN width is reduced to approximately 60% of the original model. The language-model hidden size and attention-head structure are retained.

Distillation Recovery

At the final validation point before export (step 1500):

Metric Value
Validation KL 0.3241
Student CE 2.552
Teacher CE 2.270

The released checkpoint was exported at step 1550, corresponding to approximately 812.6M training tokens.

Base-Model Benchmarks

Benchmark Gemma 3 1B PT Lythri-Wink-2B-pt
Knowledge
MMLU26.5038.50
ARC-C38.4040.10
ARC-E73.0065.99
Reasoning
HellaSwag62.3057.45
Commonsense
PIQA73.8071.60
BoolQ63.2064.98
SocialIQA48.9042.43
WinoGrande58.2058.33

Lythri-Wink-2B-pt is evaluated without a chat template and with a BOS token prepended, Lythri-Wink-2B-pt acheives similar intelligence compare Gemma3-1B-pt, while holding a larger knowlege scale and a more permissive license.

Quickstart

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_path = "Lythri/Lythri-Wink-2B-pt"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "Once upon a time"
inputs = tokenizer(
    prompt,
    return_tensors="pt",
    add_special_tokens=True,
).to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Training Your Own Model

If You dislike our instruct model Lythri/Lythri-Wink-2B, you could train a friend from our pt model! First, prepare a JSONL dataset with one text field per sample:

{"text": "Your training text goes here."}

Here's an example about the code:

from datasets import load_dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    DataCollatorForLanguageModeling,
    Trainer,
    TrainingArguments,
)
import torch

model_id = "Lythri/Lythri-Wink-2B-pt"

tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.add_bos_token = True
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)
model.config.use_cache = False

dataset = load_dataset("json", data_files="train.jsonl", split="train")
dataset = dataset.map(
    lambda x: tokenizer(
        x["text"],
        truncation=True,
        max_length=2048,
        add_special_tokens=True,
    ),
    remove_columns=dataset.column_names,
)

trainer = Trainer(
    model=model,
    args=TrainingArguments(
        output_dir="./wink-2b-output",
        per_device_train_batch_size=1,
        gradient_accumulation_steps=16,
        learning_rate=2e-5,
        num_train_epochs=1,
        bf16=True,
        gradient_checkpointing=True,
        logging_steps=10,
        save_steps=200,
    ),
    train_dataset=dataset,
    data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),
)

trainer.train()
trainer.save_model("./wink-2b-output")
tokenizer.save_pretrained("./wink-2b-output")

Important: Keep the BOS token enabled when evaluating or fine-tuning this model.

Intended Use

  • Continued pre-training
  • Domain adaptation
  • Supervised fine-tuning
  • Research on structured pruning and distillation

For direct conversation, use Lythri/Lythri-Wink-2B.

Limitations

  • This is a base model and is not instruction-tuned.
  • The model inherits limitations and biases from its base model, teacher and training data.
  • It can produce inaccurate or inappropriate content.

License

Lythri-Wink-2B-pt is built on Gemma 4 and released under the Apache License 2.0.

Citation

@techreport{li2026lythri,
  title       = {Lythri Technical Report},
  author      = {Li, Jiawen},
  year        = {2026},
  institution = {Zenodo},
  doi         = {10.5281/zenodo.23179311},
  url         = {https://doi.org/10.5281/zenodo.23179311}
}

Support

The whole training process is self-funded. If you like our model, please click a free like β€” that means a lot to me as a student!

Downloads last month
-
Safetensors
Model size
3B params
Tensor type
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Lythri/Lythri-Wink-2B-pt

Finetunes
1 model

Collection including Lythri/Lythri-Wink-2B-pt