LocalWisdom/Penny-Alpha

Penny Alpha

A 59M-parameter base language model using the DeepSeek V4.1 Flash architecture, trained from scratch by LocalWisdom.

We believe the future of LLM is SLM and local-first, and that AI will be an integral part of everyday life. That needs a local-friendly suite. We are LocalWisdom, a suite of next-gen, local-friendly AI devkits and userkits. We believe the future relies on diversified, commoditized LLMs, and LocalWisdom is here to push that narrative.

Model details

Key	Details
Parameters	59M
Architecture	DeepSeek V4.1 Flash (MoE, sparse attention, Engram, hyper-connections)
Training data	1B tokens of Ultra-FineWeb-L3 (English)
Training compute	1x RTX 5060 Ti, 1 day 12 hours, about $5.64
Context window	8,192 tokens (trained length)
Type	Base model: no chat template, no instruction tuning
License	MIT

The architecture supports a native context of 1M tokens, but Penny Alpha was only trained on sequences up to 8,192 tokens. Do not expect reliable behavior beyond that.

Other formats

• GGUF: LocalWisdom/Penny-Alpha-GGUF. Works directly with the latest llama.cpp.

How to use

Penny Alpha is a base model: it continues text and does not follow instructions. For the safetensors weights, use Transformers with trust_remote_code=True (the modeling code ships in this repo as model.py).

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "LocalWisdom/Penny-Alpha"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).eval()

# Training documents were separated by EOS, so start the prompt with it.
prompt = tok.eos_token + "The history of the bicycle begins"
inputs = tok(prompt, add_special_tokens=False, return_tensors="pt")

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=200,
        do_sample=True,
        temperature=0.8,
        top_k=50,
        top_p=0.95,
        repetition_penalty=1.1,
        use_cache=False,  # required: this implementation has no KV cache
    )
print(tok.decode(out[0], skip_special_tokens=True))

Notes

• Prepend EOS. The tokenizer does not add it automatically. Without it the model starts mid-document and quality drops.
• use_cache=False is required. There is no KV cache, so every new token re-runs the whole sequence. This is fine at 59M parameters.
• Batching. The attention mask is ignored, so use batch size 1 or equal-length prompts without padding.
• Precision. float32 is the safest choice. bfloat16 also loads, but results may differ slightly.
• GPU is optional. The model is small enough to run on CPU. Add .to("cuda") to the model and inputs if you have a GPU.

Limitations

• A 59M-parameter model trained on 1B tokens: expect fluent-looking but frequently repetitive or factually wrong text.
• English only, base model only, and no safety tuning.
• Multi-token prediction (DSpark/MTP) and vision components of the full architecture are not included.

Training

Pretrained from scratch on 1B tokens of Ultra-FineWeb-L3 (English), packed into 8,192-token windows, on a single RTX 5060 Ti.