README-LM weights

Weights from Bobliuuu/readme-lm, a project that stores real neural language models as Markdown so that a README-sized document can carry them. Video explainer: https://github.com/Bobliuuu/readme-lm/blob/main/video/readme_lm_explainer.mp4

What is here

path what size
gpt2/ GPT-2 small (124M), int8 per output channel, as 339 Markdown shards (weights/part-*.md, base64 of zlib of int8) plus tokenizer shards, manifest.md, a numpy-only run.py about 145 MB
qa/weights.md a 600K-parameter character-level QA transformer (3 layers, d=128, 4 heads, RoPE, RMSNorm, tied embeddings), 4-bit group-quantized (group 32, QAT), as one base64 block 411 KiB
qa_model.py, qa_infer.py, gen.py, ask.py model definition and numpy inference small

GPT-2 small is OpenAI's model (modified MIT license); the gpt2/ files are an int8 re-encoding of it, not a new model. The QA model is a toy trained on about 690 question/answer facts about the author's public repos; it is mostly memorisation and not a general model.

Run

pip install numpy regex huggingface_hub
hf download Bobliu/readme-lm --local-dir readme-lm && cd readme-lm
python3 gpt2/run.py "The meaning of life is" 60          # GPT-2 from the Markdown shards (numpy only, CPU)
# QA model: the loader expects the weights block between two marker comments
python3 - <<'PY'
import re
t = open("qa/weights.md").read(); b = re.search(r"```\n(.*?)\n```", t, re.S).group(1)
open("qa_wrapped.md", "w").write("<!--LM-WEIGHTS-START-->\n```\n" + b + "\n```\n<!--LM-WEIGHTS-END-->\n")
PY
python3 ask.py qa_wrapped.md out.svg 0 "what is bobliu-os"

Honest notes

  • On a 44-token test passage, GPT-2 int8 perplexity was 13.5 against 13.4 for fp32 (a small test, not a benchmark), per the project README.
  • GPT-2 is a base model: it continues text. The QA model only knows the facts it was trained on.
  • The MediaWiki and SVG-filter networks in the GitHub repo are separate toys (not transformers) and are not included here.
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Bobliu/readme-lm

Finetuned
(2,329)
this model