v136: reopened the v125 delimiter family (closed for the wrong reason) and closed it properly. Characters equally cheap IN ISOLATION are not equally cheap IN CONTEXT - pilcrow has 0 collisions and is 1 token alone yet costs +6762. Elision gains 2025 tokens but 36.65% of span bodies contain newlines, so it is undecodable. 982 tests
328d1d5 verified | { | |
| "arm": "angle", | |
| "sites": 13524, | |
| "content_preserved": 150, | |
| "base_tokens": 278973, | |
| "total_tokens": 292497, | |
| "base_chars": 987484, | |
| "total_chars": 987484, | |
| "d_term_mean": 0.0, | |
| "component_mean": -0.0152125506, | |
| "seed_mean": 0.0, | |
| "seeds_gaining": 0, | |
| "token_delta": 13524, | |
| "char_delta": 0 | |
| } |