chrishayuk commited on
Commit
fa41352
·
verified ·
1 Parent(s): 7e96a85

Publish tokenizer 10dd51100331ab50 (adopted)

Browse files
README.md CHANGED
@@ -1,29 +1,96 @@
1
  ---
2
- library_name: tokenizers
3
- tags:
4
- - tokenizer
5
- - knowledge-first
6
  language:
7
  - en
 
 
 
 
8
  license: apache-2.0
9
  ---
10
 
11
- # v11 — knowledge-first tokenizer
12
 
13
- Vocabulary size: **71260**
14
 
15
- v11 is a pure-Rust longest-match tokenizer whose vocabulary is assembled from knowledge sources (WordNet, Wikidata, tree-sitter AST grammars for 77 programming languages, curated morphemes, Greek letters, math symbols, acronyms) rather than discovered from BPE merges on a corpus.
16
 
17
- Every token is a potential compilation target, so language keywords, scientific terms, acronyms, and Greek letters are guaranteed to be single pieces for efficient compilation.
18
-
19
- ## Load with HuggingFace
20
  ```python
21
- from tokenizers import Tokenizer
22
- tok = Tokenizer.from_file("tokenizer.json")
23
- tok.encode("def fibonacci(n):").tokens
24
  ```
25
 
26
- ## Load with v11-cli
27
- ```bash
28
- v11 encode --text 'def fibonacci(n):'
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
29
  ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
 
 
 
 
2
  language:
3
  - en
4
+ library_name: transformers
5
+ tags:
6
+ - tokenizer
7
+ - language-modeling
8
  license: apache-2.0
9
  ---
10
 
11
+ # v11-tokenizer
12
 
13
+ **adopted.** This is the current recommended tokenizer for this line.
14
 
15
+ ## Loading
16
 
 
 
 
17
  ```python
18
+ from transformers import AutoTokenizer
19
+
20
+ tokenizer = AutoTokenizer.from_pretrained("chrishayuk/v11-tokenizer")
21
  ```
22
 
23
+ No `trust_remote_code`, no clone, no training code. That is verified against the
24
+ downloaded artifact at publish time, not asserted.
25
+
26
+ ## Identity
27
+
28
+ **Identity is the content hash, not the Hub revision.** Re-pushing identical
29
+ bytes mints a new commit oid; a README edit does too. Join on this instead — it
30
+ is the same value a checkpoint records as `tokenizer_hash`:
31
+
32
+ | | |
33
+ |---|---|
34
+ | `tokenizer.json` sha256 | `10dd51100331ab503115db23eee7e8dc3e360e3aed697c8a2e1b12b8f46031ae` |
35
+ | Vocabulary size | 71,260 |
36
+ | Source | [chrishayuk/v-tokenizers](https://github.com/chrishayuk/v-tokenizers) |
37
+ | Source commit | `4e73da7d6a4598177837c30a891cfb0e0da9127c` |
38
+
39
+ This repo is immutable: a fix ships under a new name, never as a replacement.
40
+
41
+ ## Special tokens
42
+
43
+ | Role | Token | ID |
44
+ |---|---|---|
45
+ | Unknown | `<unk>` | 1 |
46
+ | Beginning | `<s>` | 2 |
47
+ | End | `</s>` | 3 |
48
+ | Padding | `<pad>` | 0 |
49
+
50
+ ## Dormant blocks — read before sizing an embedding table
51
+
52
+ This vocabulary declares blocks that are **present as ids but masked out of the
53
+ projection matmul and optimizer state** at 115M parameters. Sizing an embedding
54
+ table from `len(tokenizer)` will disagree with the model's active slice, and no
55
+ vocabulary-length check will catch it.
56
+
57
+ See `dormant_blocks.json` for the machine-readable declaration.
58
+
59
+ | Block | Active @115M | Active @1B | Tokens frozen |
60
+ |---|---|---|---|
61
+ | dormant: T-cell | False | True | no (schema pinned, filled at TOK-5) |
62
+ | dormant: T-scale | False | True | no (schema pinned, filled at TOK-5) |
63
+
64
+ ## Golden encodings
65
+
66
+ `golden_encodings.json` carries 116 frames from the MSI frame battery —
67
+ call operands, post-delimiter and mid-string-punctuation surfaces, not
68
+ "representative strings", because those are where segmentation actually breaks.
69
+ Every one is replayed against the downloaded artifact at publish time.
70
+
71
+ ```python
72
+ import json, urllib.request
73
+ from transformers import AutoTokenizer
74
+
75
+ tok = AutoTokenizer.from_pretrained("chrishayuk/v11-tokenizer")
76
+ golden = json.load(urllib.request.urlopen(
77
+ "https://huggingface.co/chrishayuk/v11-tokenizer/resolve/main/golden_encodings.json"))
78
+ for name, case in golden.items():
79
+ assert tok(case["text"], add_special_tokens=False)["input_ids"] == case["ids"], name
80
  ```
81
+
82
+ ## Provenance
83
+
84
+ `provenance.json` points at the systems that own these facts rather than copying
85
+ them: the corpus lives in the **chuk-datasets** catalog (dataset name, version,
86
+ content sha), and results live in the **chuk-experiments** ledger (run id). This
87
+ repo is a publication location, not a second source of truth.
88
+
89
+ ## Intended use
90
+
91
+ Research and training of compact language models.
92
+
93
+ ## Limitations
94
+
95
+ Corpus coverage, language coverage and normalisation behaviour are described in
96
+ the source repository. Dormant-block ids are exposed by this artifact but inactive at 115M — see above.
checksums.sha256 ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ 0260f98ca18f9400aa73c3b0ffe7a434ddf734ab3a573500bc7be2981703bd3b dormant_blocks.json
2
+ bac7077323ad43406c7efd75f73d74f55e47a3f29f2d991612e0d6ecb0ec5646 golden_encodings.json
3
+ 1837c32ed3a09fd87d173abecaa9df2a85da505579f30c9e9f2369b9ca12c1ef provenance.json
4
+ 9046da57c270c8e74d0f38832b4adce269c9d914ef21d2a0925e7772152dd793 special_tokens_map.json
5
+ 10dd51100331ab503115db23eee7e8dc3e360e3aed697c8a2e1b12b8f46031ae tokenizer.json
6
+ 85e283b2261920c475133f8434f9dcd2576b43977dcff510589b9212542b7993 tokenizer_config.json
dormant_blocks.json ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema": "v-tokenizers-dormant-blocks-1",
3
+ "source": "v12/dormant_block_map.yaml (C7)",
4
+ "masking_policy": {
5
+ "implementation": "active-slice projection: hidden @ active_embedding.T over the contiguous active range",
6
+ "dormant_rows_in_matmul": false,
7
+ "dormant_rows_in_optimizer_state": false,
8
+ "dormant_strings_in_clean_115M_stream": false,
9
+ "fingerprint_init_preserved_until_activation": true
10
+ },
11
+ "layout_order": [
12
+ {
13
+ "block": "T-core natural language",
14
+ "category": 0,
15
+ "active_at_115M": true,
16
+ "active_at_1B": null,
17
+ "merge_opaque": null,
18
+ "token_count": null,
19
+ "tokens_frozen": null
20
+ },
21
+ {
22
+ "block": "active specials",
23
+ "category": 2,
24
+ "active_at_115M": true,
25
+ "active_at_1B": null,
26
+ "merge_opaque": true,
27
+ "token_count": null,
28
+ "tokens_frozen": null
29
+ },
30
+ {
31
+ "block": "dormant: T-cell",
32
+ "category": 3,
33
+ "active_at_115M": false,
34
+ "active_at_1B": true,
35
+ "merge_opaque": true,
36
+ "token_count": 0,
37
+ "tokens_frozen": false
38
+ },
39
+ {
40
+ "block": "dormant: T-scale",
41
+ "category": 4,
42
+ "active_at_115M": false,
43
+ "active_at_1B": true,
44
+ "merge_opaque": false,
45
+ "token_count": 0,
46
+ "tokens_frozen": false
47
+ },
48
+ {
49
+ "block": "padding",
50
+ "category": 1,
51
+ "active_at_115M": true,
52
+ "active_at_1B": null,
53
+ "merge_opaque": null,
54
+ "token_count": null,
55
+ "tokens_frozen": null
56
+ }
57
+ ],
58
+ "note": "Dormant ids are present in the tokenizer's vocabulary but are excluded from the projection matmul and optimizer state at 115M. Size an active embedding slice from the active blocks, NOT from len(tokenizer). Where token_count is 0 the block is schema-pinned but not yet frozen (filled at TOK-5 once the winning vocab layout is known)."
59
+ }
golden_encodings.json ADDED
@@ -0,0 +1,1966 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "tnum_000/narrative": {
3
+ "text": "There were 3 of them left by the end of the story.",
4
+ "ids": [
5
+ 13152,
6
+ 749,
7
+ 714,
8
+ 435,
9
+ 723,
10
+ 931,
11
+ 9194,
12
+ 733,
13
+ 719,
14
+ 1790,
15
+ 723,
16
+ 719,
17
+ 13832,
18
+ 312
19
+ ]
20
+ },
21
+ "tnum_000/call_operand": {
22
+ "text": "<call>fn(3)</call>",
23
+ "ids": [
24
+ 714,
25
+ 328,
26
+ 1475,
27
+ 329,
28
+ 1935,
29
+ 321,
30
+ 435,
31
+ 322,
32
+ 328,
33
+ 327,
34
+ 1475,
35
+ 329
36
+ ]
37
+ },
38
+ "tnum_000/post_delimiter": {
39
+ "text": "total=3;",
40
+ "ids": [
41
+ 14660,
42
+ 338,
43
+ 435,
44
+ 316
45
+ ]
46
+ },
47
+ "tnum_000/mid_string_punct": {
48
+ "text": "(item#3)",
49
+ "ids": [
50
+ 714,
51
+ 321,
52
+ 2193,
53
+ 331,
54
+ 435,
55
+ 322
56
+ ]
57
+ },
58
+ "tnum_001/narrative": {
59
+ "text": "There were 7 of them left by the end of the story.",
60
+ "ids": [
61
+ 13152,
62
+ 749,
63
+ 714,
64
+ 439,
65
+ 723,
66
+ 931,
67
+ 9194,
68
+ 733,
69
+ 719,
70
+ 1790,
71
+ 723,
72
+ 719,
73
+ 13832,
74
+ 312
75
+ ]
76
+ },
77
+ "tnum_001/call_operand": {
78
+ "text": "<call>fn(7)</call>",
79
+ "ids": [
80
+ 714,
81
+ 328,
82
+ 1475,
83
+ 329,
84
+ 1935,
85
+ 321,
86
+ 439,
87
+ 322,
88
+ 328,
89
+ 327,
90
+ 1475,
91
+ 329
92
+ ]
93
+ },
94
+ "tnum_001/post_delimiter": {
95
+ "text": "total=7;",
96
+ "ids": [
97
+ 14660,
98
+ 338,
99
+ 439,
100
+ 316
101
+ ]
102
+ },
103
+ "tnum_001/mid_string_punct": {
104
+ "text": "(item#7)",
105
+ "ids": [
106
+ 714,
107
+ 321,
108
+ 2193,
109
+ 331,
110
+ 439,
111
+ 322
112
+ ]
113
+ },
114
+ "tnum_002/narrative": {
115
+ "text": "There were 0 of them left by the end of the story.",
116
+ "ids": [
117
+ 13152,
118
+ 749,
119
+ 714,
120
+ 432,
121
+ 723,
122
+ 931,
123
+ 9194,
124
+ 733,
125
+ 719,
126
+ 1790,
127
+ 723,
128
+ 719,
129
+ 13832,
130
+ 312
131
+ ]
132
+ },
133
+ "tnum_002/call_operand": {
134
+ "text": "<call>fn(0)</call>",
135
+ "ids": [
136
+ 714,
137
+ 328,
138
+ 1475,
139
+ 329,
140
+ 1935,
141
+ 321,
142
+ 432,
143
+ 322,
144
+ 328,
145
+ 327,
146
+ 1475,
147
+ 329
148
+ ]
149
+ },
150
+ "tnum_002/post_delimiter": {
151
+ "text": "total=0;",
152
+ "ids": [
153
+ 14660,
154
+ 338,
155
+ 432,
156
+ 316
157
+ ]
158
+ },
159
+ "tnum_002/mid_string_punct": {
160
+ "text": "(item#0)",
161
+ "ids": [
162
+ 714,
163
+ 321,
164
+ 2193,
165
+ 331,
166
+ 432,
167
+ 322
168
+ ]
169
+ },
170
+ "tnum_003/narrative": {
171
+ "text": "There were 42 of them left by the end of the story.",
172
+ "ids": [
173
+ 13152,
174
+ 749,
175
+ 714,
176
+ 436,
177
+ 434,
178
+ 723,
179
+ 931,
180
+ 9194,
181
+ 733,
182
+ 719,
183
+ 1790,
184
+ 723,
185
+ 719,
186
+ 13832,
187
+ 312
188
+ ]
189
+ },
190
+ "tnum_003/call_operand": {
191
+ "text": "<call>fn(42)</call>",
192
+ "ids": [
193
+ 714,
194
+ 328,
195
+ 1475,
196
+ 329,
197
+ 1935,
198
+ 321,
199
+ 436,
200
+ 434,
201
+ 322,
202
+ 328,
203
+ 327,
204
+ 1475,
205
+ 329
206
+ ]
207
+ },
208
+ "tnum_003/post_delimiter": {
209
+ "text": "total=42;",
210
+ "ids": [
211
+ 14660,
212
+ 338,
213
+ 436,
214
+ 434,
215
+ 316
216
+ ]
217
+ },
218
+ "tnum_003/mid_string_punct": {
219
+ "text": "(item#42)",
220
+ "ids": [
221
+ 714,
222
+ 321,
223
+ 2193,
224
+ 331,
225
+ 436,
226
+ 434,
227
+ 322
228
+ ]
229
+ },
230
+ "tnum_004/narrative": {
231
+ "text": "There were 17 of them left by the end of the story.",
232
+ "ids": [
233
+ 13152,
234
+ 749,
235
+ 714,
236
+ 433,
237
+ 439,
238
+ 723,
239
+ 931,
240
+ 9194,
241
+ 733,
242
+ 719,
243
+ 1790,
244
+ 723,
245
+ 719,
246
+ 13832,
247
+ 312
248
+ ]
249
+ },
250
+ "tnum_004/call_operand": {
251
+ "text": "<call>fn(17)</call>",
252
+ "ids": [
253
+ 714,
254
+ 328,
255
+ 1475,
256
+ 329,
257
+ 1935,
258
+ 321,
259
+ 433,
260
+ 439,
261
+ 322,
262
+ 328,
263
+ 327,
264
+ 1475,
265
+ 329
266
+ ]
267
+ },
268
+ "tnum_004/post_delimiter": {
269
+ "text": "total=17;",
270
+ "ids": [
271
+ 14660,
272
+ 338,
273
+ 433,
274
+ 439,
275
+ 316
276
+ ]
277
+ },
278
+ "tnum_004/mid_string_punct": {
279
+ "text": "(item#17)",
280
+ "ids": [
281
+ 714,
282
+ 321,
283
+ 2193,
284
+ 331,
285
+ 433,
286
+ 439,
287
+ 322
288
+ ]
289
+ },
290
+ "tnum_005/narrative": {
291
+ "text": "There were 90 of them left by the end of the story.",
292
+ "ids": [
293
+ 13152,
294
+ 749,
295
+ 714,
296
+ 441,
297
+ 432,
298
+ 723,
299
+ 931,
300
+ 9194,
301
+ 733,
302
+ 719,
303
+ 1790,
304
+ 723,
305
+ 719,
306
+ 13832,
307
+ 312
308
+ ]
309
+ },
310
+ "tnum_005/call_operand": {
311
+ "text": "<call>fn(90)</call>",
312
+ "ids": [
313
+ 714,
314
+ 328,
315
+ 1475,
316
+ 329,
317
+ 1935,
318
+ 321,
319
+ 441,
320
+ 432,
321
+ 322,
322
+ 328,
323
+ 327,
324
+ 1475,
325
+ 329
326
+ ]
327
+ },
328
+ "tnum_005/post_delimiter": {
329
+ "text": "total=90;",
330
+ "ids": [
331
+ 14660,
332
+ 338,
333
+ 441,
334
+ 432,
335
+ 316
336
+ ]
337
+ },
338
+ "tnum_005/mid_string_punct": {
339
+ "text": "(item#90)",
340
+ "ids": [
341
+ 714,
342
+ 321,
343
+ 2193,
344
+ 331,
345
+ 441,
346
+ 432,
347
+ 322
348
+ ]
349
+ },
350
+ "tnum_006/narrative": {
351
+ "text": "There were 256 of them left by the end of the story.",
352
+ "ids": [
353
+ 13152,
354
+ 749,
355
+ 714,
356
+ 434,
357
+ 437,
358
+ 438,
359
+ 723,
360
+ 931,
361
+ 9194,
362
+ 733,
363
+ 719,
364
+ 1790,
365
+ 723,
366
+ 719,
367
+ 13832,
368
+ 312
369
+ ]
370
+ },
371
+ "tnum_006/call_operand": {
372
+ "text": "<call>fn(256)</call>",
373
+ "ids": [
374
+ 714,
375
+ 328,
376
+ 1475,
377
+ 329,
378
+ 1935,
379
+ 321,
380
+ 434,
381
+ 437,
382
+ 438,
383
+ 322,
384
+ 328,
385
+ 327,
386
+ 1475,
387
+ 329
388
+ ]
389
+ },
390
+ "tnum_006/post_delimiter": {
391
+ "text": "total=256;",
392
+ "ids": [
393
+ 14660,
394
+ 338,
395
+ 434,
396
+ 437,
397
+ 438,
398
+ 316
399
+ ]
400
+ },
401
+ "tnum_006/mid_string_punct": {
402
+ "text": "(item#256)",
403
+ "ids": [
404
+ 714,
405
+ 321,
406
+ 2193,
407
+ 331,
408
+ 434,
409
+ 437,
410
+ 438,
411
+ 322
412
+ ]
413
+ },
414
+ "tnum_007/narrative": {
415
+ "text": "There were 007 of them left by the end of the story.",
416
+ "ids": [
417
+ 13152,
418
+ 749,
419
+ 714,
420
+ 432,
421
+ 432,
422
+ 439,
423
+ 723,
424
+ 931,
425
+ 9194,
426
+ 733,
427
+ 719,
428
+ 1790,
429
+ 723,
430
+ 719,
431
+ 13832,
432
+ 312
433
+ ]
434
+ },
435
+ "tnum_007/call_operand": {
436
+ "text": "<call>fn(007)</call>",
437
+ "ids": [
438
+ 714,
439
+ 328,
440
+ 1475,
441
+ 329,
442
+ 1935,
443
+ 321,
444
+ 432,
445
+ 432,
446
+ 439,
447
+ 322,
448
+ 328,
449
+ 327,
450
+ 1475,
451
+ 329
452
+ ]
453
+ },
454
+ "tnum_007/post_delimiter": {
455
+ "text": "total=007;",
456
+ "ids": [
457
+ 14660,
458
+ 338,
459
+ 432,
460
+ 432,
461
+ 439,
462
+ 316
463
+ ]
464
+ },
465
+ "tnum_007/mid_string_punct": {
466
+ "text": "(item#007)",
467
+ "ids": [
468
+ 714,
469
+ 321,
470
+ 2193,
471
+ 331,
472
+ 432,
473
+ 432,
474
+ 439,
475
+ 322
476
+ ]
477
+ },
478
+ "tnum_008/narrative": {
479
+ "text": "There were 999 of them left by the end of the story.",
480
+ "ids": [
481
+ 13152,
482
+ 749,
483
+ 714,
484
+ 441,
485
+ 441,
486
+ 441,
487
+ 723,
488
+ 931,
489
+ 9194,
490
+ 733,
491
+ 719,
492
+ 1790,
493
+ 723,
494
+ 719,
495
+ 13832,
496
+ 312
497
+ ]
498
+ },
499
+ "tnum_008/call_operand": {
500
+ "text": "<call>fn(999)</call>",
501
+ "ids": [
502
+ 714,
503
+ 328,
504
+ 1475,
505
+ 329,
506
+ 1935,
507
+ 321,
508
+ 441,
509
+ 441,
510
+ 441,
511
+ 322,
512
+ 328,
513
+ 327,
514
+ 1475,
515
+ 329
516
+ ]
517
+ },
518
+ "tnum_008/post_delimiter": {
519
+ "text": "total=999;",
520
+ "ids": [
521
+ 14660,
522
+ 338,
523
+ 441,
524
+ 441,
525
+ 441,
526
+ 316
527
+ ]
528
+ },
529
+ "tnum_008/mid_string_punct": {
530
+ "text": "(item#999)",
531
+ "ids": [
532
+ 714,
533
+ 321,
534
+ 2193,
535
+ 331,
536
+ 441,
537
+ 441,
538
+ 441,
539
+ 322
540
+ ]
541
+ },
542
+ "tnum_009/narrative": {
543
+ "text": "There were 1024 of them left by the end of the story.",
544
+ "ids": [
545
+ 13152,
546
+ 749,
547
+ 714,
548
+ 433,
549
+ 432,
550
+ 434,
551
+ 436,
552
+ 723,
553
+ 931,
554
+ 9194,
555
+ 733,
556
+ 719,
557
+ 1790,
558
+ 723,
559
+ 719,
560
+ 13832,
561
+ 312
562
+ ]
563
+ },
564
+ "tnum_009/call_operand": {
565
+ "text": "<call>fn(1024)</call>",
566
+ "ids": [
567
+ 714,
568
+ 328,
569
+ 1475,
570
+ 329,
571
+ 1935,
572
+ 321,
573
+ 433,
574
+ 432,
575
+ 434,
576
+ 436,
577
+ 322,
578
+ 328,
579
+ 327,
580
+ 1475,
581
+ 329
582
+ ]
583
+ },
584
+ "tnum_009/post_delimiter": {
585
+ "text": "total=1024;",
586
+ "ids": [
587
+ 14660,
588
+ 338,
589
+ 433,
590
+ 432,
591
+ 434,
592
+ 436,
593
+ 316
594
+ ]
595
+ },
596
+ "tnum_009/mid_string_punct": {
597
+ "text": "(item#1024)",
598
+ "ids": [
599
+ 714,
600
+ 321,
601
+ 2193,
602
+ 331,
603
+ 433,
604
+ 432,
605
+ 434,
606
+ 436,
607
+ 322
608
+ ]
609
+ },
610
+ "tnum_010/narrative": {
611
+ "text": "There were 0042 of them left by the end of the story.",
612
+ "ids": [
613
+ 13152,
614
+ 749,
615
+ 714,
616
+ 432,
617
+ 432,
618
+ 436,
619
+ 434,
620
+ 723,
621
+ 931,
622
+ 9194,
623
+ 733,
624
+ 719,
625
+ 1790,
626
+ 723,
627
+ 719,
628
+ 13832,
629
+ 312
630
+ ]
631
+ },
632
+ "tnum_010/call_operand": {
633
+ "text": "<call>fn(0042)</call>",
634
+ "ids": [
635
+ 714,
636
+ 328,
637
+ 1475,
638
+ 329,
639
+ 1935,
640
+ 321,
641
+ 432,
642
+ 432,
643
+ 436,
644
+ 434,
645
+ 322,
646
+ 328,
647
+ 327,
648
+ 1475,
649
+ 329
650
+ ]
651
+ },
652
+ "tnum_010/post_delimiter": {
653
+ "text": "total=0042;",
654
+ "ids": [
655
+ 14660,
656
+ 338,
657
+ 432,
658
+ 432,
659
+ 436,
660
+ 434,
661
+ 316
662
+ ]
663
+ },
664
+ "tnum_010/mid_string_punct": {
665
+ "text": "(item#0042)",
666
+ "ids": [
667
+ 714,
668
+ 321,
669
+ 2193,
670
+ 331,
671
+ 432,
672
+ 432,
673
+ 436,
674
+ 434,
675
+ 322
676
+ ]
677
+ },
678
+ "tnum_011/narrative": {
679
+ "text": "There were 8080 of them left by the end of the story.",
680
+ "ids": [
681
+ 13152,
682
+ 749,
683
+ 714,
684
+ 440,
685
+ 432,
686
+ 440,
687
+ 432,
688
+ 723,
689
+ 931,
690
+ 9194,
691
+ 733,
692
+ 719,
693
+ 1790,
694
+ 723,
695
+ 719,
696
+ 13832,
697
+ 312
698
+ ]
699
+ },
700
+ "tnum_011/call_operand": {
701
+ "text": "<call>fn(8080)</call>",
702
+ "ids": [
703
+ 714,
704
+ 328,
705
+ 1475,
706
+ 329,
707
+ 1935,
708
+ 321,
709
+ 440,
710
+ 432,
711
+ 440,
712
+ 432,
713
+ 322,
714
+ 328,
715
+ 327,
716
+ 1475,
717
+ 329
718
+ ]
719
+ },
720
+ "tnum_011/post_delimiter": {
721
+ "text": "total=8080;",
722
+ "ids": [
723
+ 14660,
724
+ 338,
725
+ 440,
726
+ 432,
727
+ 440,
728
+ 432,
729
+ 316
730
+ ]
731
+ },
732
+ "tnum_011/mid_string_punct": {
733
+ "text": "(item#8080)",
734
+ "ids": [
735
+ 714,
736
+ 321,
737
+ 2193,
738
+ 331,
739
+ 440,
740
+ 432,
741
+ 440,
742
+ 432,
743
+ 322
744
+ ]
745
+ },
746
+ "tnum_012/narrative": {
747
+ "text": "There were 12345 of them left by the end of the story.",
748
+ "ids": [
749
+ 13152,
750
+ 749,
751
+ 714,
752
+ 433,
753
+ 434,
754
+ 435,
755
+ 436,
756
+ 437,
757
+ 723,
758
+ 931,
759
+ 9194,
760
+ 733,
761
+ 719,
762
+ 1790,
763
+ 723,
764
+ 719,
765
+ 13832,
766
+ 312
767
+ ]
768
+ },
769
+ "tnum_012/call_operand": {
770
+ "text": "<call>fn(12345)</call>",
771
+ "ids": [
772
+ 714,
773
+ 328,
774
+ 1475,
775
+ 329,
776
+ 1935,
777
+ 321,
778
+ 433,
779
+ 434,
780
+ 435,
781
+ 436,
782
+ 437,
783
+ 322,
784
+ 328,
785
+ 327,
786
+ 1475,
787
+ 329
788
+ ]
789
+ },
790
+ "tnum_012/post_delimiter": {
791
+ "text": "total=12345;",
792
+ "ids": [
793
+ 14660,
794
+ 338,
795
+ 433,
796
+ 434,
797
+ 435,
798
+ 436,
799
+ 437,
800
+ 316
801
+ ]
802
+ },
803
+ "tnum_012/mid_string_punct": {
804
+ "text": "(item#12345)",
805
+ "ids": [
806
+ 714,
807
+ 321,
808
+ 2193,
809
+ 331,
810
+ 433,
811
+ 434,
812
+ 435,
813
+ 436,
814
+ 437,
815
+ 322
816
+ ]
817
+ },
818
+ "tnum_013/narrative": {
819
+ "text": "There were 00700 of them left by the end of the story.",
820
+ "ids": [
821
+ 13152,
822
+ 749,
823
+ 714,
824
+ 432,
825
+ 432,
826
+ 439,
827
+ 432,
828
+ 432,
829
+ 723,
830
+ 931,
831
+ 9194,
832
+ 733,
833
+ 719,
834
+ 1790,
835
+ 723,
836
+ 719,
837
+ 13832,
838
+ 312
839
+ ]
840
+ },
841
+ "tnum_013/call_operand": {
842
+ "text": "<call>fn(00700)</call>",
843
+ "ids": [
844
+ 714,
845
+ 328,
846
+ 1475,
847
+ 329,
848
+ 1935,
849
+ 321,
850
+ 432,
851
+ 432,
852
+ 439,
853
+ 432,
854
+ 432,
855
+ 322,
856
+ 328,
857
+ 327,
858
+ 1475,
859
+ 329
860
+ ]
861
+ },
862
+ "tnum_013/post_delimiter": {
863
+ "text": "total=00700;",
864
+ "ids": [
865
+ 14660,
866
+ 338,
867
+ 432,
868
+ 432,
869
+ 439,
870
+ 432,
871
+ 432,
872
+ 316
873
+ ]
874
+ },
875
+ "tnum_013/mid_string_punct": {
876
+ "text": "(item#00700)",
877
+ "ids": [
878
+ 714,
879
+ 321,
880
+ 2193,
881
+ 331,
882
+ 432,
883
+ 432,
884
+ 439,
885
+ 432,
886
+ 432,
887
+ 322
888
+ ]
889
+ },
890
+ "tnum_014/narrative": {
891
+ "text": "There were 123456 of them left by the end of the story.",
892
+ "ids": [
893
+ 13152,
894
+ 749,
895
+ 714,
896
+ 433,
897
+ 434,
898
+ 435,
899
+ 436,
900
+ 437,
901
+ 438,
902
+ 723,
903
+ 931,
904
+ 9194,
905
+ 733,
906
+ 719,
907
+ 1790,
908
+ 723,
909
+ 719,
910
+ 13832,
911
+ 312
912
+ ]
913
+ },
914
+ "tnum_014/call_operand": {
915
+ "text": "<call>fn(123456)</call>",
916
+ "ids": [
917
+ 714,
918
+ 328,
919
+ 1475,
920
+ 329,
921
+ 1935,
922
+ 321,
923
+ 433,
924
+ 434,
925
+ 435,
926
+ 436,
927
+ 437,
928
+ 438,
929
+ 322,
930
+ 328,
931
+ 327,
932
+ 1475,
933
+ 329
934
+ ]
935
+ },
936
+ "tnum_014/post_delimiter": {
937
+ "text": "total=123456;",
938
+ "ids": [
939
+ 14660,
940
+ 338,
941
+ 433,
942
+ 434,
943
+ 435,
944
+ 436,
945
+ 437,
946
+ 438,
947
+ 316
948
+ ]
949
+ },
950
+ "tnum_014/mid_string_punct": {
951
+ "text": "(item#123456)",
952
+ "ids": [
953
+ 714,
954
+ 321,
955
+ 2193,
956
+ 331,
957
+ 433,
958
+ 434,
959
+ 435,
960
+ 436,
961
+ 437,
962
+ 438,
963
+ 322
964
+ ]
965
+ },
966
+ "tnum_015/narrative": {
967
+ "text": "There were 900000 of them left by the end of the story.",
968
+ "ids": [
969
+ 13152,
970
+ 749,
971
+ 714,
972
+ 441,
973
+ 432,
974
+ 432,
975
+ 432,
976
+ 432,
977
+ 432,
978
+ 723,
979
+ 931,
980
+ 9194,
981
+ 733,
982
+ 719,
983
+ 1790,
984
+ 723,
985
+ 719,
986
+ 13832,
987
+ 312
988
+ ]
989
+ },
990
+ "tnum_015/call_operand": {
991
+ "text": "<call>fn(900000)</call>",
992
+ "ids": [
993
+ 714,
994
+ 328,
995
+ 1475,
996
+ 329,
997
+ 1935,
998
+ 321,
999
+ 441,
1000
+ 432,
1001
+ 432,
1002
+ 432,
1003
+ 432,
1004
+ 432,
1005
+ 322,
1006
+ 328,
1007
+ 327,
1008
+ 1475,
1009
+ 329
1010
+ ]
1011
+ },
1012
+ "tnum_015/post_delimiter": {
1013
+ "text": "total=900000;",
1014
+ "ids": [
1015
+ 14660,
1016
+ 338,
1017
+ 441,
1018
+ 432,
1019
+ 432,
1020
+ 432,
1021
+ 432,
1022
+ 432,
1023
+ 316
1024
+ ]
1025
+ },
1026
+ "tnum_015/mid_string_punct": {
1027
+ "text": "(item#900000)",
1028
+ "ids": [
1029
+ 714,
1030
+ 321,
1031
+ 2193,
1032
+ 331,
1033
+ 441,
1034
+ 432,
1035
+ 432,
1036
+ 432,
1037
+ 432,
1038
+ 432,
1039
+ 322
1040
+ ]
1041
+ },
1042
+ "tnum_016/narrative": {
1043
+ "text": "There were 1234567 of them left by the end of the story.",
1044
+ "ids": [
1045
+ 13152,
1046
+ 749,
1047
+ 714,
1048
+ 433,
1049
+ 434,
1050
+ 435,
1051
+ 436,
1052
+ 437,
1053
+ 438,
1054
+ 439,
1055
+ 723,
1056
+ 931,
1057
+ 9194,
1058
+ 733,
1059
+ 719,
1060
+ 1790,
1061
+ 723,
1062
+ 719,
1063
+ 13832,
1064
+ 312
1065
+ ]
1066
+ },
1067
+ "tnum_016/call_operand": {
1068
+ "text": "<call>fn(1234567)</call>",
1069
+ "ids": [
1070
+ 714,
1071
+ 328,
1072
+ 1475,
1073
+ 329,
1074
+ 1935,
1075
+ 321,
1076
+ 433,
1077
+ 434,
1078
+ 435,
1079
+ 436,
1080
+ 437,
1081
+ 438,
1082
+ 439,
1083
+ 322,
1084
+ 328,
1085
+ 327,
1086
+ 1475,
1087
+ 329
1088
+ ]
1089
+ },
1090
+ "tnum_016/post_delimiter": {
1091
+ "text": "total=1234567;",
1092
+ "ids": [
1093
+ 14660,
1094
+ 338,
1095
+ 433,
1096
+ 434,
1097
+ 435,
1098
+ 436,
1099
+ 437,
1100
+ 438,
1101
+ 439,
1102
+ 316
1103
+ ]
1104
+ },
1105
+ "tnum_016/mid_string_punct": {
1106
+ "text": "(item#1234567)",
1107
+ "ids": [
1108
+ 714,
1109
+ 321,
1110
+ 2193,
1111
+ 331,
1112
+ 433,
1113
+ 434,
1114
+ 435,
1115
+ 436,
1116
+ 437,
1117
+ 438,
1118
+ 439,
1119
+ 322
1120
+ ]
1121
+ },
1122
+ "tnum_017/narrative": {
1123
+ "text": "There were 0000007 of them left by the end of the story.",
1124
+ "ids": [
1125
+ 13152,
1126
+ 749,
1127
+ 714,
1128
+ 432,
1129
+ 432,
1130
+ 432,
1131
+ 432,
1132
+ 432,
1133
+ 432,
1134
+ 439,
1135
+ 723,
1136
+ 931,
1137
+ 9194,
1138
+ 733,
1139
+ 719,
1140
+ 1790,
1141
+ 723,
1142
+ 719,
1143
+ 13832,
1144
+ 312
1145
+ ]
1146
+ },
1147
+ "tnum_017/call_operand": {
1148
+ "text": "<call>fn(0000007)</call>",
1149
+ "ids": [
1150
+ 714,
1151
+ 328,
1152
+ 1475,
1153
+ 329,
1154
+ 1935,
1155
+ 321,
1156
+ 432,
1157
+ 432,
1158
+ 432,
1159
+ 432,
1160
+ 432,
1161
+ 432,
1162
+ 439,
1163
+ 322,
1164
+ 328,
1165
+ 327,
1166
+ 1475,
1167
+ 329
1168
+ ]
1169
+ },
1170
+ "tnum_017/post_delimiter": {
1171
+ "text": "total=0000007;",
1172
+ "ids": [
1173
+ 14660,
1174
+ 338,
1175
+ 432,
1176
+ 432,
1177
+ 432,
1178
+ 432,
1179
+ 432,
1180
+ 432,
1181
+ 439,
1182
+ 316
1183
+ ]
1184
+ },
1185
+ "tnum_017/mid_string_punct": {
1186
+ "text": "(item#0000007)",
1187
+ "ids": [
1188
+ 714,
1189
+ 321,
1190
+ 2193,
1191
+ 331,
1192
+ 432,
1193
+ 432,
1194
+ 432,
1195
+ 432,
1196
+ 432,
1197
+ 432,
1198
+ 439,
1199
+ 322
1200
+ ]
1201
+ },
1202
+ "tnum_018/narrative": {
1203
+ "text": "There were 3.14 of them left by the end of the story.",
1204
+ "ids": [
1205
+ 13152,
1206
+ 749,
1207
+ 714,
1208
+ 435,
1209
+ 312,
1210
+ 433,
1211
+ 436,
1212
+ 723,
1213
+ 931,
1214
+ 9194,
1215
+ 733,
1216
+ 719,
1217
+ 1790,
1218
+ 723,
1219
+ 719,
1220
+ 13832,
1221
+ 312
1222
+ ]
1223
+ },
1224
+ "tnum_018/call_operand": {
1225
+ "text": "<call>fn(3.14)</call>",
1226
+ "ids": [
1227
+ 714,
1228
+ 328,
1229
+ 1475,
1230
+ 329,
1231
+ 1935,
1232
+ 321,
1233
+ 435,
1234
+ 312,
1235
+ 433,
1236
+ 436,
1237
+ 322,
1238
+ 328,
1239
+ 327,
1240
+ 1475,
1241
+ 329
1242
+ ]
1243
+ },
1244
+ "tnum_018/post_delimiter": {
1245
+ "text": "total=3.14;",
1246
+ "ids": [
1247
+ 14660,
1248
+ 338,
1249
+ 435,
1250
+ 312,
1251
+ 433,
1252
+ 436,
1253
+ 316
1254
+ ]
1255
+ },
1256
+ "tnum_018/mid_string_punct": {
1257
+ "text": "(item#3.14)",
1258
+ "ids": [
1259
+ 714,
1260
+ 321,
1261
+ 2193,
1262
+ 331,
1263
+ 435,
1264
+ 312,
1265
+ 433,
1266
+ 436,
1267
+ 322
1268
+ ]
1269
+ },
1270
+ "tnum_019/narrative": {
1271
+ "text": "There were 0.5 of them left by the end of the story.",
1272
+ "ids": [
1273
+ 13152,
1274
+ 749,
1275
+ 714,
1276
+ 432,
1277
+ 312,
1278
+ 437,
1279
+ 723,
1280
+ 931,
1281
+ 9194,
1282
+ 733,
1283
+ 719,
1284
+ 1790,
1285
+ 723,
1286
+ 719,
1287
+ 13832,
1288
+ 312
1289
+ ]
1290
+ },
1291
+ "tnum_019/call_operand": {
1292
+ "text": "<call>fn(0.5)</call>",
1293
+ "ids": [
1294
+ 714,
1295
+ 328,
1296
+ 1475,
1297
+ 329,
1298
+ 1935,
1299
+ 321,
1300
+ 432,
1301
+ 312,
1302
+ 437,
1303
+ 322,
1304
+ 328,
1305
+ 327,
1306
+ 1475,
1307
+ 329
1308
+ ]
1309
+ },
1310
+ "tnum_019/post_delimiter": {
1311
+ "text": "total=0.5;",
1312
+ "ids": [
1313
+ 14660,
1314
+ 338,
1315
+ 432,
1316
+ 312,
1317
+ 437,
1318
+ 316
1319
+ ]
1320
+ },
1321
+ "tnum_019/mid_string_punct": {
1322
+ "text": "(item#0.5)",
1323
+ "ids": [
1324
+ 714,
1325
+ 321,
1326
+ 2193,
1327
+ 331,
1328
+ 432,
1329
+ 312,
1330
+ 437,
1331
+ 322
1332
+ ]
1333
+ },
1334
+ "tnum_020/narrative": {
1335
+ "text": "There were 12.375 of them left by the end of the story.",
1336
+ "ids": [
1337
+ 13152,
1338
+ 749,
1339
+ 714,
1340
+ 433,
1341
+ 434,
1342
+ 312,
1343
+ 435,
1344
+ 439,
1345
+ 437,
1346
+ 723,
1347
+ 931,
1348
+ 9194,
1349
+ 733,
1350
+ 719,
1351
+ 1790,
1352
+ 723,
1353
+ 719,
1354
+ 13832,
1355
+ 312
1356
+ ]
1357
+ },
1358
+ "tnum_020/call_operand": {
1359
+ "text": "<call>fn(12.375)</call>",
1360
+ "ids": [
1361
+ 714,
1362
+ 328,
1363
+ 1475,
1364
+ 329,
1365
+ 1935,
1366
+ 321,
1367
+ 433,
1368
+ 434,
1369
+ 312,
1370
+ 435,
1371
+ 439,
1372
+ 437,
1373
+ 322,
1374
+ 328,
1375
+ 327,
1376
+ 1475,
1377
+ 329
1378
+ ]
1379
+ },
1380
+ "tnum_020/post_delimiter": {
1381
+ "text": "total=12.375;",
1382
+ "ids": [
1383
+ 14660,
1384
+ 338,
1385
+ 433,
1386
+ 434,
1387
+ 312,
1388
+ 435,
1389
+ 439,
1390
+ 437,
1391
+ 316
1392
+ ]
1393
+ },
1394
+ "tnum_020/mid_string_punct": {
1395
+ "text": "(item#12.375)",
1396
+ "ids": [
1397
+ 714,
1398
+ 321,
1399
+ 2193,
1400
+ 331,
1401
+ 433,
1402
+ 434,
1403
+ 312,
1404
+ 435,
1405
+ 439,
1406
+ 437,
1407
+ 322
1408
+ ]
1409
+ },
1410
+ "tnum_021/narrative": {
1411
+ "text": "There were 100.00 of them left by the end of the story.",
1412
+ "ids": [
1413
+ 13152,
1414
+ 749,
1415
+ 714,
1416
+ 433,
1417
+ 432,
1418
+ 432,
1419
+ 312,
1420
+ 432,
1421
+ 432,
1422
+ 723,
1423
+ 931,
1424
+ 9194,
1425
+ 733,
1426
+ 719,
1427
+ 1790,
1428
+ 723,
1429
+ 719,
1430
+ 13832,
1431
+ 312
1432
+ ]
1433
+ },
1434
+ "tnum_021/call_operand": {
1435
+ "text": "<call>fn(100.00)</call>",
1436
+ "ids": [
1437
+ 714,
1438
+ 328,
1439
+ 1475,
1440
+ 329,
1441
+ 1935,
1442
+ 321,
1443
+ 433,
1444
+ 432,
1445
+ 432,
1446
+ 312,
1447
+ 432,
1448
+ 432,
1449
+ 322,
1450
+ 328,
1451
+ 327,
1452
+ 1475,
1453
+ 329
1454
+ ]
1455
+ },
1456
+ "tnum_021/post_delimiter": {
1457
+ "text": "total=100.00;",
1458
+ "ids": [
1459
+ 14660,
1460
+ 338,
1461
+ 433,
1462
+ 432,
1463
+ 432,
1464
+ 312,
1465
+ 432,
1466
+ 432,
1467
+ 316
1468
+ ]
1469
+ },
1470
+ "tnum_021/mid_string_punct": {
1471
+ "text": "(item#100.00)",
1472
+ "ids": [
1473
+ 714,
1474
+ 321,
1475
+ 2193,
1476
+ 331,
1477
+ 433,
1478
+ 432,
1479
+ 432,
1480
+ 312,
1481
+ 432,
1482
+ 432,
1483
+ 322
1484
+ ]
1485
+ },
1486
+ "tnum_022/narrative": {
1487
+ "text": "There were 0.001 of them left by the end of the story.",
1488
+ "ids": [
1489
+ 13152,
1490
+ 749,
1491
+ 714,
1492
+ 432,
1493
+ 312,
1494
+ 432,
1495
+ 432,
1496
+ 433,
1497
+ 723,
1498
+ 931,
1499
+ 9194,
1500
+ 733,
1501
+ 719,
1502
+ 1790,
1503
+ 723,
1504
+ 719,
1505
+ 13832,
1506
+ 312
1507
+ ]
1508
+ },
1509
+ "tnum_022/call_operand": {
1510
+ "text": "<call>fn(0.001)</call>",
1511
+ "ids": [
1512
+ 714,
1513
+ 328,
1514
+ 1475,
1515
+ 329,
1516
+ 1935,
1517
+ 321,
1518
+ 432,
1519
+ 312,
1520
+ 432,
1521
+ 432,
1522
+ 433,
1523
+ 322,
1524
+ 328,
1525
+ 327,
1526
+ 1475,
1527
+ 329
1528
+ ]
1529
+ },
1530
+ "tnum_022/post_delimiter": {
1531
+ "text": "total=0.001;",
1532
+ "ids": [
1533
+ 14660,
1534
+ 338,
1535
+ 432,
1536
+ 312,
1537
+ 432,
1538
+ 432,
1539
+ 433,
1540
+ 316
1541
+ ]
1542
+ },
1543
+ "tnum_022/mid_string_punct": {
1544
+ "text": "(item#0.001)",
1545
+ "ids": [
1546
+ 714,
1547
+ 321,
1548
+ 2193,
1549
+ 331,
1550
+ 432,
1551
+ 312,
1552
+ 432,
1553
+ 432,
1554
+ 433,
1555
+ 322
1556
+ ]
1557
+ },
1558
+ "tnum_023/narrative": {
1559
+ "text": "There were +1 of them left by the end of the story.",
1560
+ "ids": [
1561
+ 13152,
1562
+ 749,
1563
+ 714,
1564
+ 337,
1565
+ 433,
1566
+ 723,
1567
+ 931,
1568
+ 9194,
1569
+ 733,
1570
+ 719,
1571
+ 1790,
1572
+ 723,
1573
+ 719,
1574
+ 13832,
1575
+ 312
1576
+ ]
1577
+ },
1578
+ "tnum_023/call_operand": {
1579
+ "text": "<call>fn(+1)</call>",
1580
+ "ids": [
1581
+ 714,
1582
+ 328,
1583
+ 1475,
1584
+ 329,
1585
+ 1935,
1586
+ 321,
1587
+ 337,
1588
+ 433,
1589
+ 322,
1590
+ 328,
1591
+ 327,
1592
+ 1475,
1593
+ 329
1594
+ ]
1595
+ },
1596
+ "tnum_023/post_delimiter": {
1597
+ "text": "total=+1;",
1598
+ "ids": [
1599
+ 14660,
1600
+ 338,
1601
+ 337,
1602
+ 433,
1603
+ 316
1604
+ ]
1605
+ },
1606
+ "tnum_023/mid_string_punct": {
1607
+ "text": "(item#+1)",
1608
+ "ids": [
1609
+ 714,
1610
+ 321,
1611
+ 2193,
1612
+ 331,
1613
+ 337,
1614
+ 433,
1615
+ 322
1616
+ ]
1617
+ },
1618
+ "tnum_024/narrative": {
1619
+ "text": "There were -42 of them left by the end of the story.",
1620
+ "ids": [
1621
+ 13152,
1622
+ 749,
1623
+ 714,
1624
+ 320,
1625
+ 436,
1626
+ 434,
1627
+ 723,
1628
+ 931,
1629
+ 9194,
1630
+ 733,
1631
+ 719,
1632
+ 1790,
1633
+ 723,
1634
+ 719,
1635
+ 13832,
1636
+ 312
1637
+ ]
1638
+ },
1639
+ "tnum_024/call_operand": {
1640
+ "text": "<call>fn(-42)</call>",
1641
+ "ids": [
1642
+ 714,
1643
+ 328,
1644
+ 1475,
1645
+ 329,
1646
+ 1935,
1647
+ 321,
1648
+ 320,
1649
+ 436,
1650
+ 434,
1651
+ 322,
1652
+ 328,
1653
+ 327,
1654
+ 1475,
1655
+ 329
1656
+ ]
1657
+ },
1658
+ "tnum_024/post_delimiter": {
1659
+ "text": "total=-42;",
1660
+ "ids": [
1661
+ 14660,
1662
+ 338,
1663
+ 320,
1664
+ 436,
1665
+ 434,
1666
+ 316
1667
+ ]
1668
+ },
1669
+ "tnum_024/mid_string_punct": {
1670
+ "text": "(item#-42)",
1671
+ "ids": [
1672
+ 714,
1673
+ 321,
1674
+ 2193,
1675
+ 331,
1676
+ 320,
1677
+ 436,
1678
+ 434,
1679
+ 322
1680
+ ]
1681
+ },
1682
+ "tnum_025/narrative": {
1683
+ "text": "There were 1e6 of them left by the end of the story.",
1684
+ "ids": [
1685
+ 13152,
1686
+ 749,
1687
+ 714,
1688
+ 433,
1689
+ 268,
1690
+ 438,
1691
+ 723,
1692
+ 931,
1693
+ 9194,
1694
+ 733,
1695
+ 719,
1696
+ 1790,
1697
+ 723,
1698
+ 719,
1699
+ 13832,
1700
+ 312
1701
+ ]
1702
+ },
1703
+ "tnum_025/call_operand": {
1704
+ "text": "<call>fn(1e6)</call>",
1705
+ "ids": [
1706
+ 714,
1707
+ 328,
1708
+ 1475,
1709
+ 329,
1710
+ 1935,
1711
+ 321,
1712
+ 433,
1713
+ 268,
1714
+ 438,
1715
+ 322,
1716
+ 328,
1717
+ 327,
1718
+ 1475,
1719
+ 329
1720
+ ]
1721
+ },
1722
+ "tnum_025/post_delimiter": {
1723
+ "text": "total=1e6;",
1724
+ "ids": [
1725
+ 14660,
1726
+ 338,
1727
+ 433,
1728
+ 268,
1729
+ 438,
1730
+ 316
1731
+ ]
1732
+ },
1733
+ "tnum_025/mid_string_punct": {
1734
+ "text": "(item#1e6)",
1735
+ "ids": [
1736
+ 714,
1737
+ 321,
1738
+ 2193,
1739
+ 331,
1740
+ 433,
1741
+ 268,
1742
+ 438,
1743
+ 322
1744
+ ]
1745
+ },
1746
+ "tnum_026/narrative": {
1747
+ "text": "There were 3.14e-2 of them left by the end of the story.",
1748
+ "ids": [
1749
+ 13152,
1750
+ 749,
1751
+ 714,
1752
+ 435,
1753
+ 312,
1754
+ 433,
1755
+ 436,
1756
+ 268,
1757
+ 320,
1758
+ 434,
1759
+ 723,
1760
+ 931,
1761
+ 9194,
1762
+ 733,
1763
+ 719,
1764
+ 1790,
1765
+ 723,
1766
+ 719,
1767
+ 13832,
1768
+ 312
1769
+ ]
1770
+ },
1771
+ "tnum_026/call_operand": {
1772
+ "text": "<call>fn(3.14e-2)</call>",
1773
+ "ids": [
1774
+ 714,
1775
+ 328,
1776
+ 1475,
1777
+ 329,
1778
+ 1935,
1779
+ 321,
1780
+ 435,
1781
+ 312,
1782
+ 433,
1783
+ 436,
1784
+ 268,
1785
+ 320,
1786
+ 434,
1787
+ 322,
1788
+ 328,
1789
+ 327,
1790
+ 1475,
1791
+ 329
1792
+ ]
1793
+ },
1794
+ "tnum_026/post_delimiter": {
1795
+ "text": "total=3.14e-2;",
1796
+ "ids": [
1797
+ 14660,
1798
+ 338,
1799
+ 435,
1800
+ 312,
1801
+ 433,
1802
+ 436,
1803
+ 268,
1804
+ 320,
1805
+ 434,
1806
+ 316
1807
+ ]
1808
+ },
1809
+ "tnum_026/mid_string_punct": {
1810
+ "text": "(item#3.14e-2)",
1811
+ "ids": [
1812
+ 714,
1813
+ 321,
1814
+ 2193,
1815
+ 331,
1816
+ 435,
1817
+ 312,
1818
+ 433,
1819
+ 436,
1820
+ 268,
1821
+ 320,
1822
+ 434,
1823
+ 322
1824
+ ]
1825
+ },
1826
+ "tnum_027/narrative": {
1827
+ "text": "There were 0x1F of them left by the end of the story.",
1828
+ "ids": [
1829
+ 13152,
1830
+ 749,
1831
+ 714,
1832
+ 432,
1833
+ 306,
1834
+ 433,
1835
+ 353,
1836
+ 723,
1837
+ 931,
1838
+ 9194,
1839
+ 733,
1840
+ 719,
1841
+ 1790,
1842
+ 723,
1843
+ 719,
1844
+ 13832,
1845
+ 312
1846
+ ]
1847
+ },
1848
+ "tnum_027/call_operand": {
1849
+ "text": "<call>fn(0x1F)</call>",
1850
+ "ids": [
1851
+ 714,
1852
+ 328,
1853
+ 1475,
1854
+ 329,
1855
+ 1935,
1856
+ 321,
1857
+ 432,
1858
+ 306,
1859
+ 433,
1860
+ 353,
1861
+ 322,
1862
+ 328,
1863
+ 327,
1864
+ 1475,
1865
+ 329
1866
+ ]
1867
+ },
1868
+ "tnum_027/post_delimiter": {
1869
+ "text": "total=0x1F;",
1870
+ "ids": [
1871
+ 14660,
1872
+ 338,
1873
+ 432,
1874
+ 306,
1875
+ 433,
1876
+ 353,
1877
+ 316
1878
+ ]
1879
+ },
1880
+ "tnum_027/mid_string_punct": {
1881
+ "text": "(item#0x1F)",
1882
+ "ids": [
1883
+ 714,
1884
+ 321,
1885
+ 2193,
1886
+ 331,
1887
+ 432,
1888
+ 306,
1889
+ 433,
1890
+ 353,
1891
+ 322
1892
+ ]
1893
+ },
1894
+ "tnum_028/narrative": {
1895
+ "text": "There were 0b101 of them left by the end of the story.",
1896
+ "ids": [
1897
+ 13152,
1898
+ 749,
1899
+ 714,
1900
+ 432,
1901
+ 262,
1902
+ 433,
1903
+ 432,
1904
+ 433,
1905
+ 723,
1906
+ 931,
1907
+ 9194,
1908
+ 733,
1909
+ 719,
1910
+ 1790,
1911
+ 723,
1912
+ 719,
1913
+ 13832,
1914
+ 312
1915
+ ]
1916
+ },
1917
+ "tnum_028/call_operand": {
1918
+ "text": "<call>fn(0b101)</call>",
1919
+ "ids": [
1920
+ 714,
1921
+ 328,
1922
+ 1475,
1923
+ 329,
1924
+ 1935,
1925
+ 321,
1926
+ 432,
1927
+ 262,
1928
+ 433,
1929
+ 432,
1930
+ 433,
1931
+ 322,
1932
+ 328,
1933
+ 327,
1934
+ 1475,
1935
+ 329
1936
+ ]
1937
+ },
1938
+ "tnum_028/post_delimiter": {
1939
+ "text": "total=0b101;",
1940
+ "ids": [
1941
+ 14660,
1942
+ 338,
1943
+ 432,
1944
+ 262,
1945
+ 433,
1946
+ 432,
1947
+ 433,
1948
+ 316
1949
+ ]
1950
+ },
1951
+ "tnum_028/mid_string_punct": {
1952
+ "text": "(item#0b101)",
1953
+ "ids": [
1954
+ 714,
1955
+ 321,
1956
+ 2193,
1957
+ 331,
1958
+ 432,
1959
+ 262,
1960
+ 433,
1961
+ 432,
1962
+ 433,
1963
+ 322
1964
+ ]
1965
+ }
1966
+ }
provenance.json ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema": "v-tokenizers-provenance-1",
3
+ "tokenizer_sha256": "10dd51100331ab503115db23eee7e8dc3e360e3aed697c8a2e1b12b8f46031ae",
4
+ "vocab_size": 71260,
5
+ "status": "adopted",
6
+ "source_repo": "https://github.com/chrishayuk/v-tokenizers",
7
+ "source_commit": "4e73da7d6a4598177837c30a891cfb0e0da9127c",
8
+ "corpus": {
9
+ "catalog": "chuk-datasets",
10
+ "dataset": "v11/tokenizer",
11
+ "version_id": "v1",
12
+ "content_sha": null,
13
+ "resolve": "https://chuk-datasets.fly.dev/v1/datasets"
14
+ },
15
+ "evaluation": {
16
+ "ledger": "chuk-experiments",
17
+ "run_id": null,
18
+ "note": "Headline numbers belong in the card; the ledger is authoritative."
19
+ }
20
+ }
special_tokens_map.json CHANGED
@@ -3,4 +3,4 @@
3
  "eos_token": "</s>",
4
  "pad_token": "<pad>",
5
  "unk_token": "<unk>"
6
- }
 
3
  "eos_token": "</s>",
4
  "pad_token": "<pad>",
5
  "unk_token": "<unk>"
6
+ }
tokenizer_config.json CHANGED
@@ -1,8 +1,8 @@
1
  {
2
  "bos_token": "<s>",
 
3
  "eos_token": "</s>",
4
- "model_max_length": 2048,
5
  "pad_token": "<pad>",
6
  "tokenizer_class": "PreTrainedTokenizerFast",
7
  "unk_token": "<unk>"
8
- }
 
1
  {
2
  "bos_token": "<s>",
3
+ "clean_up_tokenization_spaces": false,
4
  "eos_token": "</s>",
 
5
  "pad_token": "<pad>",
6
  "tokenizer_class": "PreTrainedTokenizerFast",
7
  "unk_token": "<unk>"
8
+ }