README-LM weights
Weights from Bobliuuu/readme-lm, a project that stores real neural language models as Markdown so that a README-sized document can carry them. Video explainer: https://github.com/Bobliuuu/readme-lm/blob/main/video/readme_lm_explainer.mp4
What is here
| path | what | size |
|---|---|---|
gpt2/ |
GPT-2 small (124M), int8 per output channel, as 339 Markdown shards (weights/part-*.md, base64 of zlib of int8) plus tokenizer shards, manifest.md, a numpy-only run.py |
about 145 MB |
qa/weights.md |
a 600K-parameter character-level QA transformer (3 layers, d=128, 4 heads, RoPE, RMSNorm, tied embeddings), 4-bit group-quantized (group 32, QAT), as one base64 block | 411 KiB |
qa_model.py, qa_infer.py, gen.py, ask.py |
model definition and numpy inference | small |
GPT-2 small is OpenAI's model (modified MIT license); the gpt2/ files are an int8 re-encoding of it, not a new model. The QA model is a toy trained on about 690 question/answer facts about the author's public repos; it is mostly memorisation and not a general model.
Run
pip install numpy regex huggingface_hub
hf download Bobliu/readme-lm --local-dir readme-lm && cd readme-lm
python3 gpt2/run.py "The meaning of life is" 60 # GPT-2 from the Markdown shards (numpy only, CPU)
# QA model: the loader expects the weights block between two marker comments
python3 - <<'PY'
import re
t = open("qa/weights.md").read(); b = re.search(r"```\n(.*?)\n```", t, re.S).group(1)
open("qa_wrapped.md", "w").write("<!--LM-WEIGHTS-START-->\n```\n" + b + "\n```\n<!--LM-WEIGHTS-END-->\n")
PY
python3 ask.py qa_wrapped.md out.svg 0 "what is bobliu-os"
Honest notes
- On a 44-token test passage, GPT-2 int8 perplexity was 13.5 against 13.4 for fp32 (a small test, not a benchmark), per the project README.
- GPT-2 is a base model: it continues text. The QA model only knows the facts it was trained on.
- The MediaWiki and SVG-filter networks in the GitHub repo are separate toys (not transformers) and are not included here.
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐ Ask for provider support
Model tree for Bobliu/readme-lm
Base model
openai-community/gpt2