| --- |
| library_name: prism |
| license: mit |
| language: |
| - code |
| tags: |
| - code-generation |
| - from-scratch |
| - educational |
| - character-level |
| - pytorch |
| pipeline_tag: text-generation |
| --- |
| |
| # Prism v2.0 |
|
|
| A test model I made. |
|
|
| Character-level coding language model trained from scratch in PyTorch on |
| a single consumer GPU. Built to learn how language models actually work |
| — not to compete with production code LMs. |
|
|
| ## Quick facts (v2) |
|
|
| - **Parameters:** 147,133,154 |
| - **Architecture:** CharMLP with 12-layer residual MLP stack (GELU + LayerNorm), single self-attention layer |
| - **Context window:** 384 characters |
| - **Vocab:** 98 characters (printable ASCII + tab/newline/CR) |
| - **Precision:** BF16 mixed |
| - **Training:** ~6 hours on RTX 3070, final loss 1.04 |
| - **Weights format:** safetensors (no pickle, no code-exec risk on load) |
|
|
| > v2 fixed the contaminated vocab from v1 (which had Chinese / Arabic / emoji |
| > chars in there and produced garbage output). v2 is strict ASCII-only. |
|
|
| ## Intended use |
|
|
| - Learning how LMs are trained |
| - Generating short code snippets (Python, JS, C/C++, Rust, Go) |
| - Educational demonstrations |
|
|
| ## Out of scope |
|
|
| - Production code generation |
| - Commercial deployment |
| - Safety-critical software |
| - Anything requiring correct output |
|
|
| ## How to use |
|
|
| Prism is **not** an HF Transformers model. Load weights directly via |
| PyTorch + safetensors. See `inference.py` for a working example. |
|
|
| ```python |
| import json, torch |
| from safetensors.torch import load_file |
| |
| cfg = json.loads(open("config.json").read()) |
| vocab = json.loads(open("vocab.json").read()) |
| state = load_file("model.safetensors") |
| # Build CharMLP(...) and load_state_dict(state). See inference.py. |
| ``` |
|
|
| ## Sampling |
|
|
| Recommended sampling: **temperature 0.7, top-k 40, repetition penalty 1.1**. |
| Plain greedy or low-temperature sampling tends to get stuck in whitespace |
| loops because the model's single-head attention has no positional encoding |
| (a known architecture limitation). |
|
|
| ## Limitations |
|
|
| - Tiny context window — Prism forgets things said more than ~3 lines ago |
| - Single small attention layer (no positional encoding) — limited contextual reasoning |
| - Character-level tokenizer — slow on long inputs |
| - Cold-start prompts often collapse into whitespace; prime with real code |
| - English + code only; will not produce coherent natural language |
| - No safety / RLHF training — can produce nonsense or replicate bias |
| from training data |
|
|
| ## License |
|
|
| MIT — see [LICENSE](LICENSE). |
|
|
