Simplify model card

README.md

@@ -3,75 +3,73 @@ license: apache-2.0
 tags:
 - syntax-highlighting
 - python
-- hand-designed
-- no-training
 ---
 
 # pylexnet
 
-A Python syntax highlighter that is a recurrent neural network with **827 numbers and no training data**. Every unit was designed for a job, and the weights were written by construction, not learned.
+A neural network for Python syntax highlighting, with hand-designed weights and no training data. It has 827 parameters, counting every non-zero value.
 
-Inspired by [alpindale/lexnet](https://huggingface.co/alpindale/lexnet), which does this for C. Designed with Claude Fable 5.1 and Claude Opus 5.5.
+A Python counterpart to [lexnet](https://huggingface.co/alpindale/lexnet). Designed with Claude Fable 5.1 and Claude Opus 5.5.
 
 <video src="https://huggingface.co/Pranavz/pylexnet/resolve/main/demo.mp4" controls width="100%"></video>
 
-The colours in the video are the network's own output.
-
-## Try it
+## Usage
 
 ```bash
 git clone https://huggingface.co/Pranavz/pylexnet && cd pylexnet
 cc -O2 -o pylexnet pylexnet.c -lm
-./pylexnet yourfile.py                      # ANSI-coloured output
-./pylexnet --classes yourfile.py            # one class id per byte
-./pylexnet --interp --classes yourfile.py   # generic reference evaluation of the same weights
+./pylexnet file.py
 ```
 
-## Three builds
+`--classes` prints one class id per byte. `--interp` runs the generic evaluation of the same weights.
+
+## Model details
 
-| Build | Numbers | ns/byte, 1 core | Stdlib held-out: bytes, files exact | Third-party held-out: bytes, files exact |
-|---|---|---|---|---|
-| **main, smallest** | **827** | 94 | 99.996%, 264 of 293 | 99.995%, 3,059 of 3,177 |
-| `builds/fast-994` | 994 (1,502 if its tied threshold weight is counted per row) | **63** | 99.991%, 259 of 293 | 99.9945%, 3,036 of 3,177 |
-| `builds/accurate-7370`, a compiled automaton, not hand-designed | 7,370 | ~89 | ~100.000%, 292 of 293 | **99.9992%, 3,175 of 3,177** |
+| | |
+|---|---|
+| Type | Bidirectional recurrent network, per-byte sequence tagger |
+| Task | Token classification over Python source |
+| Input | Bytes |
+| Output | One of 9 classes per byte |
+| Parameters | 827 non-zero |
+| Activations | `Clip(x, 0, hi)` |
+| Vocabulary | 269 keywords, builtins and special method names |
+| Training data | None |
+| Training compute | None |
 
-"Numbers" counts every non-zero value the network uses: embedding, weights, biases, clip ceilings, head, initial state.
+## Accuracy
 
-Agreement is per byte against Pygments 2.17.2 `PythonLexer`, over 9 classes: text, keyword, builtin, string, comment, number, function, class, decorator. Stdlib held-out is half the Python 3.12 standard library by path hash (5.8 MB). Third-party held-out is 16 packages, 30 MB, not used for rule fixing. Each was run once per build at the end.
+Per-byte agreement with Pygments 2.17.2 on held-out code.
 
-**Agreement with Pygments isn't correctness.** On the third-party held-out set, CPython's tokenizer sided with the `accurate-7370` build on all 255 bytes where it differs from Pygments. That audit hasn't been run on the small builds.
+| Set | Size | Agreement | Exact files |
+|---|---|---|---|
+| Python 3.12 standard library | 5.8 MB | 99.996% | 264 / 293 |
+| Third-party packages | 30 MB | 99.995% | 3,059 / 3,177 |
 
-## Speed, measured fairly
+## Speed
 
-One machine (Intel i5-9400F), one core, `-O2`, the same 293 held-out Python files, best of runs per file, start-up excluded. Ruff is Rust at `opt-level=3`.
+293 files from the standard library, single core, Intel i5-9400F.
 
-| Tool | ns/byte | Does |
-|---|---|---|
-| The `accurate-7370` rules as a plain lookup table, no network | 5.5 | Same labels as that build |
-| Ruff parser 0.8.6 | 14 | Full lexing and AST, no highlight classes |
-| `builds/accurate-7370` predecessor | 90.7 | Labels bytes |
-| tree-sitter-python, parse only | 106.7 | Full syntax tree |
-| tree-sitter-python, parse + highlights query | 164.8 | Tree and highlight captures |
-| Pygments `PythonLexer` | 1,153 | Tokens |
+| Executor | ns/byte |
+|---|---|
+| Ruff parser | 14 |
+| pylexnet, `builds/fast-994` | 63 |
+| **pylexnet** | **94** |
+| tree-sitter-python, parse only | 107 |
+| tree-sitter-python, parse and highlight query | 165 |
+| Pygments `PythonLexer` | 1,153 |
 
-The three builds above were timed on `argparse.py` (100 KB). The corpus-wide row is an earlier build of the accurate network. This is not the fastest way to lex Python: a plain table and Ruff are both faster, and tree-sitter builds a tree that this doesn't. The point is that a tiny network with no training data can do this job at all.
+pylexnet figures are from `argparse.py`; the rest are corpus-wide. Ruff and tree-sitter build a syntax tree, which pylexnet does not.
 
-## How it works
+## Other builds
 
-- **Embedding:** 18 byte features, including a character code and string-prefix letters.
-- **Forward cell:** about 55 logic units, each designed: inside-string, quote-run counter, triple-quote, f-string and escape flags, one saved outer level for f-string fields with bracket depth, comment, number with exponent, word and length, and `def`/`class`/decorator context.
-- **Vocabulary:** additive retrieval. Solved value tables over byte position sum to a class code for each of 269 keywords, builtins and special method names, in 249 numbers. Ten threshold units read the class windows.
-- **Backward cell:** carries a word's class back over its bytes, because a word's class is only known at its end.
-- **Runtime:** `pylexnet_kernel.h` is the architecture generated as C, like lexnet's. Weight values live only in `pylexnet_weights.bin`. The loader checks a structure hash and guard inequalities. `--interp` runs the generic evaluation of the same weights.
+| Build | Parameters | ns/byte | Third-party agreement |
+|---|---|---|---|
+| `builds/fast-994` | 994 | 63 | 99.9945% |
+| `builds/accurate-7370` | 7,370 | 89 | 99.9992% |
 
-`compile.py` regenerates everything: `python3 compile.py --matcher add` for the main build, `--matcher hash --vocab 220` for `fast-994`.
+`fast-994` shares one threshold weight across rows and keeps 220 of the words; counted per row it has 1,502 parameters. `accurate-7370` is a compiled automaton, not a hand-designed network.
 
-## Limits
+## Limitations
 
-- **Approximate vocabulary in the main build:** a few non-keywords are flagged, such as `an` and `ad`. That was 185 bytes in a 74.8 MB tuning corpus.
-- **`fast-994` keeps 220 of the words** and is exact on those, ranked by frequency in the tuning corpus.
-- **Missing in the small builds:** soft keywords `match`/`case`, the space in `yield from`, two-level nested f-strings, `{x=}`, and `.5` floats after a dot. These are why their exact-file counts trail the accurate build.
-- **Weights come from a generator script:** the network is designed by hand, unit by unit, but the numbers are produced by `compile.py`, not typed out.
-- **Python 3.12 vocabulary, one Pygments version.** Results don't transfer automatically to other versions.
-- **Speeds are from one machine.** The builds table is one 100 KB file.
-- The tools in `tools/` were written for the development setup and may need paths adjusted.
+Soft keywords (`match`, `case`), two-level nested f-strings and a few other cases aren't handled. The vocabulary matcher is approximate, so a few words such as `an` are flagged as keywords.