samuli commited on
Commit
9c2cf4c
·
verified ·
1 Parent(s): 948a8aa

Juurio 0.1.0: code, wheel, model card and licences

Browse files
.gitattributes CHANGED
@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  data/wordlist.json filter=lfs diff=lfs merge=lfs -text
 
 
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  data/wordlist.json filter=lfs diff=lfs merge=lfs -text
37
+ juurio-0.1.0-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
ATTRIBUTION.md ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Attribution and licensing
2
+
3
+ juurio has three separately licensed parts. The model checkpoints and data files are not bundled in the Python wheel. `juurio setup` downloads them from https://huggingface.co/samuli/juurio and caches them locally.
4
+
5
+ | part | files | licence |
6
+ |---|---|---|
7
+ | code | the `juurio` Python package (`juurio/`, the wheel) | MIT, see [`LICENSE`](LICENSE) |
8
+ | models | `checkpoints/` (weights, tokenizer and vocabularies), `data/char_vocab.json` | CC BY 4.0, see [`LICENSE-CC-BY-4.0`](LICENSE-CC-BY-4.0) |
9
+ | data derived from Giella `fin` | `data/wordlist.json`, `data/fst_table.jsonl`, `data/fst_tag_vocab.json` | GPL-3.0, see [`LICENSE-GPL-3.0`](LICENSE-GPL-3.0) |
10
+
11
+ For CC BY 4.0 attribution, credit "juurio by samuli, https://huggingface.co/samuli/juurio".
12
+
13
+ This file is a good-faith summary, not legal advice. Check the upstream licences linked below for your own use.
14
+
15
+ ## Data derived from Giella `fin` (GPL-3.0)
16
+
17
+ These files are generated by running the Giella `fin` finite-state transducer (through [uralicNLP](https://github.com/mikahama/uralicNLP)) over word forms and lemmas found in a Finnish web corpus, and storing its output. They are derivative works of [GiellaLT `lang-fin`](https://github.com/giellalt/lang-fin) and are distributed under GPL-3.0, the licence of `lang-fin`.
18
+
19
+ | file | what |
20
+ |---|---|
21
+ | `data/wordlist.json` | exception table: the inflected forms the neural realizers get wrong |
22
+ | `data/fst_table.jsonl` | morphology tags for the 100,000 most frequent word forms where the neural analyser disagrees with Giella (e-mail addresses, @handles, URLs, phone numbers and hashtags removed) |
23
+ | `data/fst_tag_vocab.json` | morphology tag vocabulary |
24
+
25
+ © 2004–present UiT The Arctic University of Norway / Giellatekno, and the `lang-fin` contributors. Source: https://github.com/giellalt/lang-fin (GPL-3.0). `lang-fin` derives from [Omorfi](https://github.com/flammie/omorfi) (GPL-3.0).
26
+
27
+ ## Models (CC BY 4.0)
28
+
29
+ | file | trained on |
30
+ |---|---|
31
+ | `checkpoints/realizer.pt`, `verbform.pt`, `finiteverb.pt` | inflected forms generated with Giella `fin` (GPL-3.0) |
32
+ | `checkpoints/parser.pt` | [UD_Finnish-TDT](https://universaldependencies.org/treebanks/fi_tdt/index.html) (CC BY-SA 4.0), silver parses of Finnish web text made with [Stanza](https://stanfordnlp.github.io/stanza/) `fi` (Apache-2.0, its model trained on UD_Finnish-TDT), and Giella `fin` morphology tags as input |
33
+ | `checkpoints/analyze.pt` | Giella `fin` morphology tags for word forms from Finnish web text |
34
+
35
+ Whether trained weights are a derivative work of their training data is legally unsettled. The table documents where the training data came from, so that users can make their own judgement, and credits the upstream sources.
36
+
37
+ ## Tools used to build the models and data
38
+
39
+ None of these are needed to run juurio.
40
+
41
+ | project | use | licence |
42
+ |---|---|---|
43
+ | [GiellaLT `lang-fin`](https://github.com/giellalt/lang-fin) through [uralicNLP](https://github.com/mikahama/uralicNLP) | the morphology the realizers learn from | GPL-3.0 |
44
+ | [Stanza](https://stanfordnlp.github.io/stanza/) | silver dependency parses | Apache-2.0 (code) |
45
+ | [UD_Finnish-TDT](https://universaldependencies.org/treebanks/fi_tdt/index.html) | parser training data, and the treebank behind Stanza's `fi` model | CC BY-SA 4.0 |
46
+ | [Voikko](https://voikko.puimula.org/) | checking generated forms | MPL-1.1 / GPL-2.0+ / LGPL-2.1+ (Finnish dictionary GPL) |
47
+ | [Omorfi](https://github.com/flammie/omorfi) | upstream of `lang-fin` | GPL-3.0 |
48
+
49
+ ## Runtime dependencies
50
+
51
+ `torch` (BSD-3-Clause), `transformers` (Apache-2.0), `numpy` (BSD-3-Clause), `huggingface-hub` (Apache-2.0).
LICENSE ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ MIT License
2
+
3
+ Copyright (c) 2026 samuli (https://huggingface.co/samuli)
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
LICENSE-CC-BY-4.0 ADDED
@@ -0,0 +1,396 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Attribution 4.0 International
2
+
3
+ =======================================================================
4
+
5
+ Creative Commons Corporation ("Creative Commons") is not a law firm and
6
+ does not provide legal services or legal advice. Distribution of
7
+ Creative Commons public licenses does not create a lawyer-client or
8
+ other relationship. Creative Commons makes its licenses and related
9
+ information available on an "as-is" basis. Creative Commons gives no
10
+ warranties regarding its licenses, any material licensed under their
11
+ terms and conditions, or any related information. Creative Commons
12
+ disclaims all liability for damages resulting from their use to the
13
+ fullest extent possible.
14
+
15
+ Using Creative Commons Public Licenses
16
+
17
+ Creative Commons public licenses provide a standard set of terms and
18
+ conditions that creators and other rights holders may use to share
19
+ original works of authorship and other material subject to copyright
20
+ and certain other rights specified in the public license below. The
21
+ following considerations are for informational purposes only, are not
22
+ exhaustive, and do not form part of our licenses.
23
+
24
+ Considerations for licensors: Our public licenses are
25
+ intended for use by those authorized to give the public
26
+ permission to use material in ways otherwise restricted by
27
+ copyright and certain other rights. Our licenses are
28
+ irrevocable. Licensors should read and understand the terms
29
+ and conditions of the license they choose before applying it.
30
+ Licensors should also secure all rights necessary before
31
+ applying our licenses so that the public can reuse the
32
+ material as expected. Licensors should clearly mark any
33
+ material not subject to the license. This includes other CC-
34
+ licensed material, or material used under an exception or
35
+ limitation to copyright. More considerations for licensors:
36
+ wiki.creativecommons.org/Considerations_for_licensors
37
+
38
+ Considerations for the public: By using one of our public
39
+ licenses, a licensor grants the public permission to use the
40
+ licensed material under specified terms and conditions. If
41
+ the licensor's permission is not necessary for any reason--for
42
+ example, because of any applicable exception or limitation to
43
+ copyright--then that use is not regulated by the license. Our
44
+ licenses grant only permissions under copyright and certain
45
+ other rights that a licensor has authority to grant. Use of
46
+ the licensed material may still be restricted for other
47
+ reasons, including because others have copyright or other
48
+ rights in the material. A licensor may make special requests,
49
+ such as asking that all changes be marked or described.
50
+ Although not required by our licenses, you are encouraged to
51
+ respect those requests where reasonable. More considerations
52
+ for the public:
53
+ wiki.creativecommons.org/Considerations_for_licensees
54
+
55
+ =======================================================================
56
+
57
+ Creative Commons Attribution 4.0 International Public License
58
+
59
+ By exercising the Licensed Rights (defined below), You accept and agree
60
+ to be bound by the terms and conditions of this Creative Commons
61
+ Attribution 4.0 International Public License ("Public License"). To the
62
+ extent this Public License may be interpreted as a contract, You are
63
+ granted the Licensed Rights in consideration of Your acceptance of
64
+ these terms and conditions, and the Licensor grants You such rights in
65
+ consideration of benefits the Licensor receives from making the
66
+ Licensed Material available under these terms and conditions.
67
+
68
+
69
+ Section 1 -- Definitions.
70
+
71
+ a. Adapted Material means material subject to Copyright and Similar
72
+ Rights that is derived from or based upon the Licensed Material
73
+ and in which the Licensed Material is translated, altered,
74
+ arranged, transformed, or otherwise modified in a manner requiring
75
+ permission under the Copyright and Similar Rights held by the
76
+ Licensor. For purposes of this Public License, where the Licensed
77
+ Material is a musical work, performance, or sound recording,
78
+ Adapted Material is always produced where the Licensed Material is
79
+ synched in timed relation with a moving image.
80
+
81
+ b. Adapter's License means the license You apply to Your Copyright
82
+ and Similar Rights in Your contributions to Adapted Material in
83
+ accordance with the terms and conditions of this Public License.
84
+
85
+ c. Copyright and Similar Rights means copyright and/or similar rights
86
+ closely related to copyright including, without limitation,
87
+ performance, broadcast, sound recording, and Sui Generis Database
88
+ Rights, without regard to how the rights are labeled or
89
+ categorized. For purposes of this Public License, the rights
90
+ specified in Section 2(b)(1)-(2) are not Copyright and Similar
91
+ Rights.
92
+
93
+ d. Effective Technological Measures means those measures that, in the
94
+ absence of proper authority, may not be circumvented under laws
95
+ fulfilling obligations under Article 11 of the WIPO Copyright
96
+ Treaty adopted on December 20, 1996, and/or similar international
97
+ agreements.
98
+
99
+ e. Exceptions and Limitations means fair use, fair dealing, and/or
100
+ any other exception or limitation to Copyright and Similar Rights
101
+ that applies to Your use of the Licensed Material.
102
+
103
+ f. Licensed Material means the artistic or literary work, database,
104
+ or other material to which the Licensor applied this Public
105
+ License.
106
+
107
+ g. Licensed Rights means the rights granted to You subject to the
108
+ terms and conditions of this Public License, which are limited to
109
+ all Copyright and Similar Rights that apply to Your use of the
110
+ Licensed Material and that the Licensor has authority to license.
111
+
112
+ h. Licensor means the individual(s) or entity(ies) granting rights
113
+ under this Public License.
114
+
115
+ i. Share means to provide material to the public by any means or
116
+ process that requires permission under the Licensed Rights, such
117
+ as reproduction, public display, public performance, distribution,
118
+ dissemination, communication, or importation, and to make material
119
+ available to the public including in ways that members of the
120
+ public may access the material from a place and at a time
121
+ individually chosen by them.
122
+
123
+ j. Sui Generis Database Rights means rights other than copyright
124
+ resulting from Directive 96/9/EC of the European Parliament and of
125
+ the Council of 11 March 1996 on the legal protection of databases,
126
+ as amended and/or succeeded, as well as other essentially
127
+ equivalent rights anywhere in the world.
128
+
129
+ k. You means the individual or entity exercising the Licensed Rights
130
+ under this Public License. Your has a corresponding meaning.
131
+
132
+
133
+ Section 2 -- Scope.
134
+
135
+ a. License grant.
136
+
137
+ 1. Subject to the terms and conditions of this Public License,
138
+ the Licensor hereby grants You a worldwide, royalty-free,
139
+ non-sublicensable, non-exclusive, irrevocable license to
140
+ exercise the Licensed Rights in the Licensed Material to:
141
+
142
+ a. reproduce and Share the Licensed Material, in whole or
143
+ in part; and
144
+
145
+ b. produce, reproduce, and Share Adapted Material.
146
+
147
+ 2. Exceptions and Limitations. For the avoidance of doubt, where
148
+ Exceptions and Limitations apply to Your use, this Public
149
+ License does not apply, and You do not need to comply with
150
+ its terms and conditions.
151
+
152
+ 3. Term. The term of this Public License is specified in Section
153
+ 6(a).
154
+
155
+ 4. Media and formats; technical modifications allowed. The
156
+ Licensor authorizes You to exercise the Licensed Rights in
157
+ all media and formats whether now known or hereafter created,
158
+ and to make technical modifications necessary to do so. The
159
+ Licensor waives and/or agrees not to assert any right or
160
+ authority to forbid You from making technical modifications
161
+ necessary to exercise the Licensed Rights, including
162
+ technical modifications necessary to circumvent Effective
163
+ Technological Measures. For purposes of this Public License,
164
+ simply making modifications authorized by this Section 2(a)
165
+ (4) never produces Adapted Material.
166
+
167
+ 5. Downstream recipients.
168
+
169
+ a. Offer from the Licensor -- Licensed Material. Every
170
+ recipient of the Licensed Material automatically
171
+ receives an offer from the Licensor to exercise the
172
+ Licensed Rights under the terms and conditions of this
173
+ Public License.
174
+
175
+ b. No downstream restrictions. You may not offer or impose
176
+ any additional or different terms or conditions on, or
177
+ apply any Effective Technological Measures to, the
178
+ Licensed Material if doing so restricts exercise of the
179
+ Licensed Rights by any recipient of the Licensed
180
+ Material.
181
+
182
+ 6. No endorsement. Nothing in this Public License constitutes or
183
+ may be construed as permission to assert or imply that You
184
+ are, or that Your use of the Licensed Material is, connected
185
+ with, or sponsored, endorsed, or granted official status by,
186
+ the Licensor or others designated to receive attribution as
187
+ provided in Section 3(a)(1)(A)(i).
188
+
189
+ b. Other rights.
190
+
191
+ 1. Moral rights, such as the right of integrity, are not
192
+ licensed under this Public License, nor are publicity,
193
+ privacy, and/or other similar personality rights; however, to
194
+ the extent possible, the Licensor waives and/or agrees not to
195
+ assert any such rights held by the Licensor to the limited
196
+ extent necessary to allow You to exercise the Licensed
197
+ Rights, but not otherwise.
198
+
199
+ 2. Patent and trademark rights are not licensed under this
200
+ Public License.
201
+
202
+ 3. To the extent possible, the Licensor waives any right to
203
+ collect royalties from You for the exercise of the Licensed
204
+ Rights, whether directly or through a collecting society
205
+ under any voluntary or waivable statutory or compulsory
206
+ licensing scheme. In all other cases the Licensor expressly
207
+ reserves any right to collect such royalties.
208
+
209
+
210
+ Section 3 -- License Conditions.
211
+
212
+ Your exercise of the Licensed Rights is expressly made subject to the
213
+ following conditions.
214
+
215
+ a. Attribution.
216
+
217
+ 1. If You Share the Licensed Material (including in modified
218
+ form), You must:
219
+
220
+ a. retain the following if it is supplied by the Licensor
221
+ with the Licensed Material:
222
+
223
+ i. identification of the creator(s) of the Licensed
224
+ Material and any others designated to receive
225
+ attribution, in any reasonable manner requested by
226
+ the Licensor (including by pseudonym if
227
+ designated);
228
+
229
+ ii. a copyright notice;
230
+
231
+ iii. a notice that refers to this Public License;
232
+
233
+ iv. a notice that refers to the disclaimer of
234
+ warranties;
235
+
236
+ v. a URI or hyperlink to the Licensed Material to the
237
+ extent reasonably practicable;
238
+
239
+ b. indicate if You modified the Licensed Material and
240
+ retain an indication of any previous modifications; and
241
+
242
+ c. indicate the Licensed Material is licensed under this
243
+ Public License, and include the text of, or the URI or
244
+ hyperlink to, this Public License.
245
+
246
+ 2. You may satisfy the conditions in Section 3(a)(1) in any
247
+ reasonable manner based on the medium, means, and context in
248
+ which You Share the Licensed Material. For example, it may be
249
+ reasonable to satisfy the conditions by providing a URI or
250
+ hyperlink to a resource that includes the required
251
+ information.
252
+
253
+ 3. If requested by the Licensor, You must remove any of the
254
+ information required by Section 3(a)(1)(A) to the extent
255
+ reasonably practicable.
256
+
257
+ 4. If You Share Adapted Material You produce, the Adapter's
258
+ License You apply must not prevent recipients of the Adapted
259
+ Material from complying with this Public License.
260
+
261
+
262
+ Section 4 -- Sui Generis Database Rights.
263
+
264
+ Where the Licensed Rights include Sui Generis Database Rights that
265
+ apply to Your use of the Licensed Material:
266
+
267
+ a. for the avoidance of doubt, Section 2(a)(1) grants You the right
268
+ to extract, reuse, reproduce, and Share all or a substantial
269
+ portion of the contents of the database;
270
+
271
+ b. if You include all or a substantial portion of the database
272
+ contents in a database in which You have Sui Generis Database
273
+ Rights, then the database in which You have Sui Generis Database
274
+ Rights (but not its individual contents) is Adapted Material; and
275
+
276
+ c. You must comply with the conditions in Section 3(a) if You Share
277
+ all or a substantial portion of the contents of the database.
278
+
279
+ For the avoidance of doubt, this Section 4 supplements and does not
280
+ replace Your obligations under this Public License where the Licensed
281
+ Rights include other Copyright and Similar Rights.
282
+
283
+
284
+ Section 5 -- Disclaimer of Warranties and Limitation of Liability.
285
+
286
+ a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
287
+ EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
288
+ AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
289
+ ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
290
+ IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
291
+ WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
292
+ PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
293
+ ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
294
+ KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
295
+ ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
296
+
297
+ b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
298
+ TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
299
+ NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
300
+ INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
301
+ COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
302
+ USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
303
+ ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
304
+ DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
305
+ IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
306
+
307
+ c. The disclaimer of warranties and limitation of liability provided
308
+ above shall be interpreted in a manner that, to the extent
309
+ possible, most closely approximates an absolute disclaimer and
310
+ waiver of all liability.
311
+
312
+
313
+ Section 6 -- Term and Termination.
314
+
315
+ a. This Public License applies for the term of the Copyright and
316
+ Similar Rights licensed here. However, if You fail to comply with
317
+ this Public License, then Your rights under this Public License
318
+ terminate automatically.
319
+
320
+ b. Where Your right to use the Licensed Material has terminated under
321
+ Section 6(a), it reinstates:
322
+
323
+ 1. automatically as of the date the violation is cured, provided
324
+ it is cured within 30 days of Your discovery of the
325
+ violation; or
326
+
327
+ 2. upon express reinstatement by the Licensor.
328
+
329
+ For the avoidance of doubt, this Section 6(b) does not affect any
330
+ right the Licensor may have to seek remedies for Your violations
331
+ of this Public License.
332
+
333
+ c. For the avoidance of doubt, the Licensor may also offer the
334
+ Licensed Material under separate terms or conditions or stop
335
+ distributing the Licensed Material at any time; however, doing so
336
+ will not terminate this Public License.
337
+
338
+ d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
339
+ License.
340
+
341
+
342
+ Section 7 -- Other Terms and Conditions.
343
+
344
+ a. The Licensor shall not be bound by any additional or different
345
+ terms or conditions communicated by You unless expressly agreed.
346
+
347
+ b. Any arrangements, understandings, or agreements regarding the
348
+ Licensed Material not stated herein are separate from and
349
+ independent of the terms and conditions of this Public License.
350
+
351
+
352
+ Section 8 -- Interpretation.
353
+
354
+ a. For the avoidance of doubt, this Public License does not, and
355
+ shall not be interpreted to, reduce, limit, restrict, or impose
356
+ conditions on any use of the Licensed Material that could lawfully
357
+ be made without permission under this Public License.
358
+
359
+ b. To the extent possible, if any provision of this Public License is
360
+ deemed unenforceable, it shall be automatically reformed to the
361
+ minimum extent necessary to make it enforceable. If the provision
362
+ cannot be reformed, it shall be severed from this Public License
363
+ without affecting the enforceability of the remaining terms and
364
+ conditions.
365
+
366
+ c. No term or condition of this Public License will be waived and no
367
+ failure to comply consented to unless expressly agreed to by the
368
+ Licensor.
369
+
370
+ d. Nothing in this Public License constitutes or may be interpreted
371
+ as a limitation upon, or waiver of, any privileges and immunities
372
+ that apply to the Licensor or You, including from the legal
373
+ processes of any jurisdiction or authority.
374
+
375
+
376
+ =======================================================================
377
+
378
+ Creative Commons is not a party to its public
379
+ licenses. Notwithstanding, Creative Commons may elect to apply one of
380
+ its public licenses to material it publishes and in those instances
381
+ will be considered the “Licensor.” The text of the Creative Commons
382
+ public licenses is dedicated to the public domain under the CC0 Public
383
+ Domain Dedication. Except for the limited purpose of indicating that
384
+ material is shared under a Creative Commons public license or as
385
+ otherwise permitted by the Creative Commons policies published at
386
+ creativecommons.org/policies, Creative Commons does not authorize the
387
+ use of the trademark "Creative Commons" or any other trademark or logo
388
+ of Creative Commons without its prior written consent including,
389
+ without limitation, in connection with any unauthorized modifications
390
+ to any of its public licenses or any other arrangements,
391
+ understandings, or agreements concerning use of licensed material. For
392
+ the avoidance of doubt, this paragraph does not form part of the
393
+ public licenses.
394
+
395
+ Creative Commons may be contacted at creativecommons.org.
396
+
LICENSE-GPL-3.0 ADDED
@@ -0,0 +1,674 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ GNU GENERAL PUBLIC LICENSE
2
+ Version 3, 29 June 2007
3
+
4
+ Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
5
+ Everyone is permitted to copy and distribute verbatim copies
6
+ of this license document, but changing it is not allowed.
7
+
8
+ Preamble
9
+
10
+ The GNU General Public License is a free, copyleft license for
11
+ software and other kinds of works.
12
+
13
+ The licenses for most software and other practical works are designed
14
+ to take away your freedom to share and change the works. By contrast,
15
+ the GNU General Public License is intended to guarantee your freedom to
16
+ share and change all versions of a program--to make sure it remains free
17
+ software for all its users. We, the Free Software Foundation, use the
18
+ GNU General Public License for most of our software; it applies also to
19
+ any other work released this way by its authors. You can apply it to
20
+ your programs, too.
21
+
22
+ When we speak of free software, we are referring to freedom, not
23
+ price. Our General Public Licenses are designed to make sure that you
24
+ have the freedom to distribute copies of free software (and charge for
25
+ them if you wish), that you receive source code or can get it if you
26
+ want it, that you can change the software or use pieces of it in new
27
+ free programs, and that you know you can do these things.
28
+
29
+ To protect your rights, we need to prevent others from denying you
30
+ these rights or asking you to surrender the rights. Therefore, you have
31
+ certain responsibilities if you distribute copies of the software, or if
32
+ you modify it: responsibilities to respect the freedom of others.
33
+
34
+ For example, if you distribute copies of such a program, whether
35
+ gratis or for a fee, you must pass on to the recipients the same
36
+ freedoms that you received. You must make sure that they, too, receive
37
+ or can get the source code. And you must show them these terms so they
38
+ know their rights.
39
+
40
+ Developers that use the GNU GPL protect your rights with two steps:
41
+ (1) assert copyright on the software, and (2) offer you this License
42
+ giving you legal permission to copy, distribute and/or modify it.
43
+
44
+ For the developers' and authors' protection, the GPL clearly explains
45
+ that there is no warranty for this free software. For both users' and
46
+ authors' sake, the GPL requires that modified versions be marked as
47
+ changed, so that their problems will not be attributed erroneously to
48
+ authors of previous versions.
49
+
50
+ Some devices are designed to deny users access to install or run
51
+ modified versions of the software inside them, although the manufacturer
52
+ can do so. This is fundamentally incompatible with the aim of
53
+ protecting users' freedom to change the software. The systematic
54
+ pattern of such abuse occurs in the area of products for individuals to
55
+ use, which is precisely where it is most unacceptable. Therefore, we
56
+ have designed this version of the GPL to prohibit the practice for those
57
+ products. If such problems arise substantially in other domains, we
58
+ stand ready to extend this provision to those domains in future versions
59
+ of the GPL, as needed to protect the freedom of users.
60
+
61
+ Finally, every program is threatened constantly by software patents.
62
+ States should not allow patents to restrict development and use of
63
+ software on general-purpose computers, but in those that do, we wish to
64
+ avoid the special danger that patents applied to a free program could
65
+ make it effectively proprietary. To prevent this, the GPL assures that
66
+ patents cannot be used to render the program non-free.
67
+
68
+ The precise terms and conditions for copying, distribution and
69
+ modification follow.
70
+
71
+ TERMS AND CONDITIONS
72
+
73
+ 0. Definitions.
74
+
75
+ "This License" refers to version 3 of the GNU General Public License.
76
+
77
+ "Copyright" also means copyright-like laws that apply to other kinds of
78
+ works, such as semiconductor masks.
79
+
80
+ "The Program" refers to any copyrightable work licensed under this
81
+ License. Each licensee is addressed as "you". "Licensees" and
82
+ "recipients" may be individuals or organizations.
83
+
84
+ To "modify" a work means to copy from or adapt all or part of the work
85
+ in a fashion requiring copyright permission, other than the making of an
86
+ exact copy. The resulting work is called a "modified version" of the
87
+ earlier work or a work "based on" the earlier work.
88
+
89
+ A "covered work" means either the unmodified Program or a work based
90
+ on the Program.
91
+
92
+ To "propagate" a work means to do anything with it that, without
93
+ permission, would make you directly or secondarily liable for
94
+ infringement under applicable copyright law, except executing it on a
95
+ computer or modifying a private copy. Propagation includes copying,
96
+ distribution (with or without modification), making available to the
97
+ public, and in some countries other activities as well.
98
+
99
+ To "convey" a work means any kind of propagation that enables other
100
+ parties to make or receive copies. Mere interaction with a user through
101
+ a computer network, with no transfer of a copy, is not conveying.
102
+
103
+ An interactive user interface displays "Appropriate Legal Notices"
104
+ to the extent that it includes a convenient and prominently visible
105
+ feature that (1) displays an appropriate copyright notice, and (2)
106
+ tells the user that there is no warranty for the work (except to the
107
+ extent that warranties are provided), that licensees may convey the
108
+ work under this License, and how to view a copy of this License. If
109
+ the interface presents a list of user commands or options, such as a
110
+ menu, a prominent item in the list meets this criterion.
111
+
112
+ 1. Source Code.
113
+
114
+ The "source code" for a work means the preferred form of the work
115
+ for making modifications to it. "Object code" means any non-source
116
+ form of a work.
117
+
118
+ A "Standard Interface" means an interface that either is an official
119
+ standard defined by a recognized standards body, or, in the case of
120
+ interfaces specified for a particular programming language, one that
121
+ is widely used among developers working in that language.
122
+
123
+ The "System Libraries" of an executable work include anything, other
124
+ than the work as a whole, that (a) is included in the normal form of
125
+ packaging a Major Component, but which is not part of that Major
126
+ Component, and (b) serves only to enable use of the work with that
127
+ Major Component, or to implement a Standard Interface for which an
128
+ implementation is available to the public in source code form. A
129
+ "Major Component", in this context, means a major essential component
130
+ (kernel, window system, and so on) of the specific operating system
131
+ (if any) on which the executable work runs, or a compiler used to
132
+ produce the work, or an object code interpreter used to run it.
133
+
134
+ The "Corresponding Source" for a work in object code form means all
135
+ the source code needed to generate, install, and (for an executable
136
+ work) run the object code and to modify the work, including scripts to
137
+ control those activities. However, it does not include the work's
138
+ System Libraries, or general-purpose tools or generally available free
139
+ programs which are used unmodified in performing those activities but
140
+ which are not part of the work. For example, Corresponding Source
141
+ includes interface definition files associated with source files for
142
+ the work, and the source code for shared libraries and dynamically
143
+ linked subprograms that the work is specifically designed to require,
144
+ such as by intimate data communication or control flow between those
145
+ subprograms and other parts of the work.
146
+
147
+ The Corresponding Source need not include anything that users
148
+ can regenerate automatically from other parts of the Corresponding
149
+ Source.
150
+
151
+ The Corresponding Source for a work in source code form is that
152
+ same work.
153
+
154
+ 2. Basic Permissions.
155
+
156
+ All rights granted under this License are granted for the term of
157
+ copyright on the Program, and are irrevocable provided the stated
158
+ conditions are met. This License explicitly affirms your unlimited
159
+ permission to run the unmodified Program. The output from running a
160
+ covered work is covered by this License only if the output, given its
161
+ content, constitutes a covered work. This License acknowledges your
162
+ rights of fair use or other equivalent, as provided by copyright law.
163
+
164
+ You may make, run and propagate covered works that you do not
165
+ convey, without conditions so long as your license otherwise remains
166
+ in force. You may convey covered works to others for the sole purpose
167
+ of having them make modifications exclusively for you, or provide you
168
+ with facilities for running those works, provided that you comply with
169
+ the terms of this License in conveying all material for which you do
170
+ not control copyright. Those thus making or running the covered works
171
+ for you must do so exclusively on your behalf, under your direction
172
+ and control, on terms that prohibit them from making any copies of
173
+ your copyrighted material outside their relationship with you.
174
+
175
+ Conveying under any other circumstances is permitted solely under
176
+ the conditions stated below. Sublicensing is not allowed; section 10
177
+ makes it unnecessary.
178
+
179
+ 3. Protecting Users' Legal Rights From Anti-Circumvention Law.
180
+
181
+ No covered work shall be deemed part of an effective technological
182
+ measure under any applicable law fulfilling obligations under article
183
+ 11 of the WIPO copyright treaty adopted on 20 December 1996, or
184
+ similar laws prohibiting or restricting circumvention of such
185
+ measures.
186
+
187
+ When you convey a covered work, you waive any legal power to forbid
188
+ circumvention of technological measures to the extent such circumvention
189
+ is effected by exercising rights under this License with respect to
190
+ the covered work, and you disclaim any intention to limit operation or
191
+ modification of the work as a means of enforcing, against the work's
192
+ users, your or third parties' legal rights to forbid circumvention of
193
+ technological measures.
194
+
195
+ 4. Conveying Verbatim Copies.
196
+
197
+ You may convey verbatim copies of the Program's source code as you
198
+ receive it, in any medium, provided that you conspicuously and
199
+ appropriately publish on each copy an appropriate copyright notice;
200
+ keep intact all notices stating that this License and any
201
+ non-permissive terms added in accord with section 7 apply to the code;
202
+ keep intact all notices of the absence of any warranty; and give all
203
+ recipients a copy of this License along with the Program.
204
+
205
+ You may charge any price or no price for each copy that you convey,
206
+ and you may offer support or warranty protection for a fee.
207
+
208
+ 5. Conveying Modified Source Versions.
209
+
210
+ You may convey a work based on the Program, or the modifications to
211
+ produce it from the Program, in the form of source code under the
212
+ terms of section 4, provided that you also meet all of these conditions:
213
+
214
+ a) The work must carry prominent notices stating that you modified
215
+ it, and giving a relevant date.
216
+
217
+ b) The work must carry prominent notices stating that it is
218
+ released under this License and any conditions added under section
219
+ 7. This requirement modifies the requirement in section 4 to
220
+ "keep intact all notices".
221
+
222
+ c) You must license the entire work, as a whole, under this
223
+ License to anyone who comes into possession of a copy. This
224
+ License will therefore apply, along with any applicable section 7
225
+ additional terms, to the whole of the work, and all its parts,
226
+ regardless of how they are packaged. This License gives no
227
+ permission to license the work in any other way, but it does not
228
+ invalidate such permission if you have separately received it.
229
+
230
+ d) If the work has interactive user interfaces, each must display
231
+ Appropriate Legal Notices; however, if the Program has interactive
232
+ interfaces that do not display Appropriate Legal Notices, your
233
+ work need not make them do so.
234
+
235
+ A compilation of a covered work with other separate and independent
236
+ works, which are not by their nature extensions of the covered work,
237
+ and which are not combined with it such as to form a larger program,
238
+ in or on a volume of a storage or distribution medium, is called an
239
+ "aggregate" if the compilation and its resulting copyright are not
240
+ used to limit the access or legal rights of the compilation's users
241
+ beyond what the individual works permit. Inclusion of a covered work
242
+ in an aggregate does not cause this License to apply to the other
243
+ parts of the aggregate.
244
+
245
+ 6. Conveying Non-Source Forms.
246
+
247
+ You may convey a covered work in object code form under the terms
248
+ of sections 4 and 5, provided that you also convey the
249
+ machine-readable Corresponding Source under the terms of this License,
250
+ in one of these ways:
251
+
252
+ a) Convey the object code in, or embodied in, a physical product
253
+ (including a physical distribution medium), accompanied by the
254
+ Corresponding Source fixed on a durable physical medium
255
+ customarily used for software interchange.
256
+
257
+ b) Convey the object code in, or embodied in, a physical product
258
+ (including a physical distribution medium), accompanied by a
259
+ written offer, valid for at least three years and valid for as
260
+ long as you offer spare parts or customer support for that product
261
+ model, to give anyone who possesses the object code either (1) a
262
+ copy of the Corresponding Source for all the software in the
263
+ product that is covered by this License, on a durable physical
264
+ medium customarily used for software interchange, for a price no
265
+ more than your reasonable cost of physically performing this
266
+ conveying of source, or (2) access to copy the
267
+ Corresponding Source from a network server at no charge.
268
+
269
+ c) Convey individual copies of the object code with a copy of the
270
+ written offer to provide the Corresponding Source. This
271
+ alternative is allowed only occasionally and noncommercially, and
272
+ only if you received the object code with such an offer, in accord
273
+ with subsection 6b.
274
+
275
+ d) Convey the object code by offering access from a designated
276
+ place (gratis or for a charge), and offer equivalent access to the
277
+ Corresponding Source in the same way through the same place at no
278
+ further charge. You need not require recipients to copy the
279
+ Corresponding Source along with the object code. If the place to
280
+ copy the object code is a network server, the Corresponding Source
281
+ may be on a different server (operated by you or a third party)
282
+ that supports equivalent copying facilities, provided you maintain
283
+ clear directions next to the object code saying where to find the
284
+ Corresponding Source. Regardless of what server hosts the
285
+ Corresponding Source, you remain obligated to ensure that it is
286
+ available for as long as needed to satisfy these requirements.
287
+
288
+ e) Convey the object code using peer-to-peer transmission, provided
289
+ you inform other peers where the object code and Corresponding
290
+ Source of the work are being offered to the general public at no
291
+ charge under subsection 6d.
292
+
293
+ A separable portion of the object code, whose source code is excluded
294
+ from the Corresponding Source as a System Library, need not be
295
+ included in conveying the object code work.
296
+
297
+ A "User Product" is either (1) a "consumer product", which means any
298
+ tangible personal property which is normally used for personal, family,
299
+ or household purposes, or (2) anything designed or sold for incorporation
300
+ into a dwelling. In determining whether a product is a consumer product,
301
+ doubtful cases shall be resolved in favor of coverage. For a particular
302
+ product received by a particular user, "normally used" refers to a
303
+ typical or common use of that class of product, regardless of the status
304
+ of the particular user or of the way in which the particular user
305
+ actually uses, or expects or is expected to use, the product. A product
306
+ is a consumer product regardless of whether the product has substantial
307
+ commercial, industrial or non-consumer uses, unless such uses represent
308
+ the only significant mode of use of the product.
309
+
310
+ "Installation Information" for a User Product means any methods,
311
+ procedures, authorization keys, or other information required to install
312
+ and execute modified versions of a covered work in that User Product from
313
+ a modified version of its Corresponding Source. The information must
314
+ suffice to ensure that the continued functioning of the modified object
315
+ code is in no case prevented or interfered with solely because
316
+ modification has been made.
317
+
318
+ If you convey an object code work under this section in, or with, or
319
+ specifically for use in, a User Product, and the conveying occurs as
320
+ part of a transaction in which the right of possession and use of the
321
+ User Product is transferred to the recipient in perpetuity or for a
322
+ fixed term (regardless of how the transaction is characterized), the
323
+ Corresponding Source conveyed under this section must be accompanied
324
+ by the Installation Information. But this requirement does not apply
325
+ if neither you nor any third party retains the ability to install
326
+ modified object code on the User Product (for example, the work has
327
+ been installed in ROM).
328
+
329
+ The requirement to provide Installation Information does not include a
330
+ requirement to continue to provide support service, warranty, or updates
331
+ for a work that has been modified or installed by the recipient, or for
332
+ the User Product in which it has been modified or installed. Access to a
333
+ network may be denied when the modification itself materially and
334
+ adversely affects the operation of the network or violates the rules and
335
+ protocols for communication across the network.
336
+
337
+ Corresponding Source conveyed, and Installation Information provided,
338
+ in accord with this section must be in a format that is publicly
339
+ documented (and with an implementation available to the public in
340
+ source code form), and must require no special password or key for
341
+ unpacking, reading or copying.
342
+
343
+ 7. Additional Terms.
344
+
345
+ "Additional permissions" are terms that supplement the terms of this
346
+ License by making exceptions from one or more of its conditions.
347
+ Additional permissions that are applicable to the entire Program shall
348
+ be treated as though they were included in this License, to the extent
349
+ that they are valid under applicable law. If additional permissions
350
+ apply only to part of the Program, that part may be used separately
351
+ under those permissions, but the entire Program remains governed by
352
+ this License without regard to the additional permissions.
353
+
354
+ When you convey a copy of a covered work, you may at your option
355
+ remove any additional permissions from that copy, or from any part of
356
+ it. (Additional permissions may be written to require their own
357
+ removal in certain cases when you modify the work.) You may place
358
+ additional permissions on material, added by you to a covered work,
359
+ for which you have or can give appropriate copyright permission.
360
+
361
+ Notwithstanding any other provision of this License, for material you
362
+ add to a covered work, you may (if authorized by the copyright holders of
363
+ that material) supplement the terms of this License with terms:
364
+
365
+ a) Disclaiming warranty or limiting liability differently from the
366
+ terms of sections 15 and 16 of this License; or
367
+
368
+ b) Requiring preservation of specified reasonable legal notices or
369
+ author attributions in that material or in the Appropriate Legal
370
+ Notices displayed by works containing it; or
371
+
372
+ c) Prohibiting misrepresentation of the origin of that material, or
373
+ requiring that modified versions of such material be marked in
374
+ reasonable ways as different from the original version; or
375
+
376
+ d) Limiting the use for publicity purposes of names of licensors or
377
+ authors of the material; or
378
+
379
+ e) Declining to grant rights under trademark law for use of some
380
+ trade names, trademarks, or service marks; or
381
+
382
+ f) Requiring indemnification of licensors and authors of that
383
+ material by anyone who conveys the material (or modified versions of
384
+ it) with contractual assumptions of liability to the recipient, for
385
+ any liability that these contractual assumptions directly impose on
386
+ those licensors and authors.
387
+
388
+ All other non-permissive additional terms are considered "further
389
+ restrictions" within the meaning of section 10. If the Program as you
390
+ received it, or any part of it, contains a notice stating that it is
391
+ governed by this License along with a term that is a further
392
+ restriction, you may remove that term. If a license document contains
393
+ a further restriction but permits relicensing or conveying under this
394
+ License, you may add to a covered work material governed by the terms
395
+ of that license document, provided that the further restriction does
396
+ not survive such relicensing or conveying.
397
+
398
+ If you add terms to a covered work in accord with this section, you
399
+ must place, in the relevant source files, a statement of the
400
+ additional terms that apply to those files, or a notice indicating
401
+ where to find the applicable terms.
402
+
403
+ Additional terms, permissive or non-permissive, may be stated in the
404
+ form of a separately written license, or stated as exceptions;
405
+ the above requirements apply either way.
406
+
407
+ 8. Termination.
408
+
409
+ You may not propagate or modify a covered work except as expressly
410
+ provided under this License. Any attempt otherwise to propagate or
411
+ modify it is void, and will automatically terminate your rights under
412
+ this License (including any patent licenses granted under the third
413
+ paragraph of section 11).
414
+
415
+ However, if you cease all violation of this License, then your
416
+ license from a particular copyright holder is reinstated (a)
417
+ provisionally, unless and until the copyright holder explicitly and
418
+ finally terminates your license, and (b) permanently, if the copyright
419
+ holder fails to notify you of the violation by some reasonable means
420
+ prior to 60 days after the cessation.
421
+
422
+ Moreover, your license from a particular copyright holder is
423
+ reinstated permanently if the copyright holder notifies you of the
424
+ violation by some reasonable means, this is the first time you have
425
+ received notice of violation of this License (for any work) from that
426
+ copyright holder, and you cure the violation prior to 30 days after
427
+ your receipt of the notice.
428
+
429
+ Termination of your rights under this section does not terminate the
430
+ licenses of parties who have received copies or rights from you under
431
+ this License. If your rights have been terminated and not permanently
432
+ reinstated, you do not qualify to receive new licenses for the same
433
+ material under section 10.
434
+
435
+ 9. Acceptance Not Required for Having Copies.
436
+
437
+ You are not required to accept this License in order to receive or
438
+ run a copy of the Program. Ancillary propagation of a covered work
439
+ occurring solely as a consequence of using peer-to-peer transmission
440
+ to receive a copy likewise does not require acceptance. However,
441
+ nothing other than this License grants you permission to propagate or
442
+ modify any covered work. These actions infringe copyright if you do
443
+ not accept this License. Therefore, by modifying or propagating a
444
+ covered work, you indicate your acceptance of this License to do so.
445
+
446
+ 10. Automatic Licensing of Downstream Recipients.
447
+
448
+ Each time you convey a covered work, the recipient automatically
449
+ receives a license from the original licensors, to run, modify and
450
+ propagate that work, subject to this License. You are not responsible
451
+ for enforcing compliance by third parties with this License.
452
+
453
+ An "entity transaction" is a transaction transferring control of an
454
+ organization, or substantially all assets of one, or subdividing an
455
+ organization, or merging organizations. If propagation of a covered
456
+ work results from an entity transaction, each party to that
457
+ transaction who receives a copy of the work also receives whatever
458
+ licenses to the work the party's predecessor in interest had or could
459
+ give under the previous paragraph, plus a right to possession of the
460
+ Corresponding Source of the work from the predecessor in interest, if
461
+ the predecessor has it or can get it with reasonable efforts.
462
+
463
+ You may not impose any further restrictions on the exercise of the
464
+ rights granted or affirmed under this License. For example, you may
465
+ not impose a license fee, royalty, or other charge for exercise of
466
+ rights granted under this License, and you may not initiate litigation
467
+ (including a cross-claim or counterclaim in a lawsuit) alleging that
468
+ any patent claim is infringed by making, using, selling, offering for
469
+ sale, or importing the Program or any portion of it.
470
+
471
+ 11. Patents.
472
+
473
+ A "contributor" is a copyright holder who authorizes use under this
474
+ License of the Program or a work on which the Program is based. The
475
+ work thus licensed is called the contributor's "contributor version".
476
+
477
+ A contributor's "essential patent claims" are all patent claims
478
+ owned or controlled by the contributor, whether already acquired or
479
+ hereafter acquired, that would be infringed by some manner, permitted
480
+ by this License, of making, using, or selling its contributor version,
481
+ but do not include claims that would be infringed only as a
482
+ consequence of further modification of the contributor version. For
483
+ purposes of this definition, "control" includes the right to grant
484
+ patent sublicenses in a manner consistent with the requirements of
485
+ this License.
486
+
487
+ Each contributor grants you a non-exclusive, worldwide, royalty-free
488
+ patent license under the contributor's essential patent claims, to
489
+ make, use, sell, offer for sale, import and otherwise run, modify and
490
+ propagate the contents of its contributor version.
491
+
492
+ In the following three paragraphs, a "patent license" is any express
493
+ agreement or commitment, however denominated, not to enforce a patent
494
+ (such as an express permission to practice a patent or covenant not to
495
+ sue for patent infringement). To "grant" such a patent license to a
496
+ party means to make such an agreement or commitment not to enforce a
497
+ patent against the party.
498
+
499
+ If you convey a covered work, knowingly relying on a patent license,
500
+ and the Corresponding Source of the work is not available for anyone
501
+ to copy, free of charge and under the terms of this License, through a
502
+ publicly available network server or other readily accessible means,
503
+ then you must either (1) cause the Corresponding Source to be so
504
+ available, or (2) arrange to deprive yourself of the benefit of the
505
+ patent license for this particular work, or (3) arrange, in a manner
506
+ consistent with the requirements of this License, to extend the patent
507
+ license to downstream recipients. "Knowingly relying" means you have
508
+ actual knowledge that, but for the patent license, your conveying the
509
+ covered work in a country, or your recipient's use of the covered work
510
+ in a country, would infringe one or more identifiable patents in that
511
+ country that you have reason to believe are valid.
512
+
513
+ If, pursuant to or in connection with a single transaction or
514
+ arrangement, you convey, or propagate by procuring conveyance of, a
515
+ covered work, and grant a patent license to some of the parties
516
+ receiving the covered work authorizing them to use, propagate, modify
517
+ or convey a specific copy of the covered work, then the patent license
518
+ you grant is automatically extended to all recipients of the covered
519
+ work and works based on it.
520
+
521
+ A patent license is "discriminatory" if it does not include within
522
+ the scope of its coverage, prohibits the exercise of, or is
523
+ conditioned on the non-exercise of one or more of the rights that are
524
+ specifically granted under this License. You may not convey a covered
525
+ work if you are a party to an arrangement with a third party that is
526
+ in the business of distributing software, under which you make payment
527
+ to the third party based on the extent of your activity of conveying
528
+ the work, and under which the third party grants, to any of the
529
+ parties who would receive the covered work from you, a discriminatory
530
+ patent license (a) in connection with copies of the covered work
531
+ conveyed by you (or copies made from those copies), or (b) primarily
532
+ for and in connection with specific products or compilations that
533
+ contain the covered work, unless you entered into that arrangement,
534
+ or that patent license was granted, prior to 28 March 2007.
535
+
536
+ Nothing in this License shall be construed as excluding or limiting
537
+ any implied license or other defenses to infringement that may
538
+ otherwise be available to you under applicable patent law.
539
+
540
+ 12. No Surrender of Others' Freedom.
541
+
542
+ If conditions are imposed on you (whether by court order, agreement or
543
+ otherwise) that contradict the conditions of this License, they do not
544
+ excuse you from the conditions of this License. If you cannot convey a
545
+ covered work so as to satisfy simultaneously your obligations under this
546
+ License and any other pertinent obligations, then as a consequence you may
547
+ not convey it at all. For example, if you agree to terms that obligate you
548
+ to collect a royalty for further conveying from those to whom you convey
549
+ the Program, the only way you could satisfy both those terms and this
550
+ License would be to refrain entirely from conveying the Program.
551
+
552
+ 13. Use with the GNU Affero General Public License.
553
+
554
+ Notwithstanding any other provision of this License, you have
555
+ permission to link or combine any covered work with a work licensed
556
+ under version 3 of the GNU Affero General Public License into a single
557
+ combined work, and to convey the resulting work. The terms of this
558
+ License will continue to apply to the part which is the covered work,
559
+ but the special requirements of the GNU Affero General Public License,
560
+ section 13, concerning interaction through a network will apply to the
561
+ combination as such.
562
+
563
+ 14. Revised Versions of this License.
564
+
565
+ The Free Software Foundation may publish revised and/or new versions of
566
+ the GNU General Public License from time to time. Such new versions will
567
+ be similar in spirit to the present version, but may differ in detail to
568
+ address new problems or concerns.
569
+
570
+ Each version is given a distinguishing version number. If the
571
+ Program specifies that a certain numbered version of the GNU General
572
+ Public License "or any later version" applies to it, you have the
573
+ option of following the terms and conditions either of that numbered
574
+ version or of any later version published by the Free Software
575
+ Foundation. If the Program does not specify a version number of the
576
+ GNU General Public License, you may choose any version ever published
577
+ by the Free Software Foundation.
578
+
579
+ If the Program specifies that a proxy can decide which future
580
+ versions of the GNU General Public License can be used, that proxy's
581
+ public statement of acceptance of a version permanently authorizes you
582
+ to choose that version for the Program.
583
+
584
+ Later license versions may give you additional or different
585
+ permissions. However, no additional obligations are imposed on any
586
+ author or copyright holder as a result of your choosing to follow a
587
+ later version.
588
+
589
+ 15. Disclaimer of Warranty.
590
+
591
+ THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
592
+ APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
593
+ HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
594
+ OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
595
+ THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
596
+ PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
597
+ IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
598
+ ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
599
+
600
+ 16. Limitation of Liability.
601
+
602
+ IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
603
+ WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
604
+ THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
605
+ GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
606
+ USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
607
+ DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
608
+ PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
609
+ EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
610
+ SUCH DAMAGES.
611
+
612
+ 17. Interpretation of Sections 15 and 16.
613
+
614
+ If the disclaimer of warranty and limitation of liability provided
615
+ above cannot be given local legal effect according to their terms,
616
+ reviewing courts shall apply local law that most closely approximates
617
+ an absolute waiver of all civil liability in connection with the
618
+ Program, unless a warranty or assumption of liability accompanies a
619
+ copy of the Program in return for a fee.
620
+
621
+ END OF TERMS AND CONDITIONS
622
+
623
+ How to Apply These Terms to Your New Programs
624
+
625
+ If you develop a new program, and you want it to be of the greatest
626
+ possible use to the public, the best way to achieve this is to make it
627
+ free software which everyone can redistribute and change under these terms.
628
+
629
+ To do so, attach the following notices to the program. It is safest
630
+ to attach them to the start of each source file to most effectively
631
+ state the exclusion of warranty; and each file should have at least
632
+ the "copyright" line and a pointer to where the full notice is found.
633
+
634
+ <one line to give the program's name and a brief idea of what it does.>
635
+ Copyright (C) <year> <name of author>
636
+
637
+ This program is free software: you can redistribute it and/or modify
638
+ it under the terms of the GNU General Public License as published by
639
+ the Free Software Foundation, either version 3 of the License, or
640
+ (at your option) any later version.
641
+
642
+ This program is distributed in the hope that it will be useful,
643
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
644
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
645
+ GNU General Public License for more details.
646
+
647
+ You should have received a copy of the GNU General Public License
648
+ along with this program. If not, see <https://www.gnu.org/licenses/>.
649
+
650
+ Also add information on how to contact you by electronic and paper mail.
651
+
652
+ If the program does terminal interaction, make it output a short
653
+ notice like this when it starts in an interactive mode:
654
+
655
+ <program> Copyright (C) <year> <name of author>
656
+ This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
657
+ This is free software, and you are welcome to redistribute it
658
+ under certain conditions; type `show c' for details.
659
+
660
+ The hypothetical commands `show w' and `show c' should show the appropriate
661
+ parts of the General Public License. Of course, your program's commands
662
+ might be different; for a GUI interface, you would use an "about box".
663
+
664
+ You should also get your employer (if you work as a programmer) or school,
665
+ if any, to sign a "copyright disclaimer" for the program, if necessary.
666
+ For more information on this, and how to apply and follow the GNU GPL, see
667
+ <https://www.gnu.org/licenses/>.
668
+
669
+ The GNU General Public License does not permit incorporating your program
670
+ into proprietary programs. If your program is a subroutine library, you
671
+ may consider it more useful to permit linking proprietary applications with
672
+ the library. If this is what you want to do, use the GNU Lesser General
673
+ Public License instead of this License. But first, please read
674
+ <https://www.gnu.org/licenses/why-not-lgpl.html>.
NOTICE ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ juurio: licences
2
+ ==================
3
+
4
+ Code (the juurio Python package and wheel): MIT, see LICENSE.
5
+
6
+ Models (checkpoints/, data/char_vocab.json): CC BY 4.0, see LICENSE-CC-BY-4.0.
7
+ Attribution: "juurio by samuli, https://huggingface.co/samuli/juurio".
8
+
9
+ Data derived from the Giella `fin` FST (data/wordlist.json, data/fst_table.jsonl,
10
+ data/fst_tag_vocab.json): GPL-3.0, see LICENSE-GPL-3.0.
11
+ (c) 2004-present UiT The Arctic University of Norway / Giellatekno and the lang-fin contributors.
12
+ Upstream source: https://github.com/giellalt/lang-fin
13
+
14
+ See ATTRIBUTION.md for where the training data came from. This is not legal advice.
README.md ADDED
@@ -0,0 +1,139 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - fi
4
+ license: other
5
+ license_name: mit-and-cc-by-4.0-and-gpl-3.0
6
+ license_link: https://huggingface.co/samuli/juurio/blob/main/ATTRIBUTION.md
7
+ library_name: pytorch
8
+ tags:
9
+ - finnish
10
+ - morphology
11
+ - inflection
12
+ - lemmatization
13
+ - dependency-parsing
14
+ ---
15
+
16
+ # Juurio
17
+
18
+ Open-vocabulary Finnish morphology over raw text. Give it a sentence and an operation (`past`, `passive`, `plural`, …) and it returns the transformed sentence or an explicit abstention. It also inflects out-of-vocabulary and loan words that a closed-lexicon morphology cannot (`striimata` → `striimannee`, `striimaaja` → `striimaajillanikin`).
19
+
20
+ ```python
21
+ from juurio import transform, realize_np, realize_verbform
22
+
23
+ transform("past", "Koira haukkuu pihalla.").text # 'Koira haukkui pihalla.'
24
+ transform("passive", "Mies osti talon.").text # 'Ostettiin talo.'
25
+ transform("pluperfect","Hän lukee kirjan.").text # 'Hän oli lukenut kirjan.' (compound tense)
26
+ transform("neg_imperative","Osta talo.").text # 'Älä osta taloa.' (imperative/2nd-person source)
27
+ transform("potential", "Nuori striimaa peliä.", lemma="striimata").text # 'Nuori striimannee peliä.' (OOV verb)
28
+ transform("plural+poss_1sg+clitic_kin", "Koira haukkuu pihalla.").text # 'Koiranikin haukkuvat pihalla.'
29
+
30
+ # phrase-level morphology utilities
31
+ realize_np("talo", case="Ine", number="Plur", px="1Sg").text # 'taloissani'
32
+ realize_np("striimaaja", case="Ade", number="Plur", px="1Sg", clitic="kin").text # 'striimaajillanikin' (OOV)
33
+ realize_verbform("mennä", form="inf3", case="Ade").text # 'menemällä'
34
+ realize_verbform("striimata", form="va").text # 'striimaava' (OOV participle)
35
+ ```
36
+
37
+ Every call returns a `Result` (`ok`, `text`, `reason`, `trace`, `warnings`). Input that is out of scope or that the operation does not apply to abstains, with `ok=False` and a documented reason. Forms produced by a neural model rather than the exception lexicon are listed in `trace["neural_unverified"]` and in `warnings`.
38
+
39
+ ## Install
40
+
41
+ ```bash
42
+ pip install https://huggingface.co/samuli/juurio/resolve/main/juurio-0.1.0-py3-none-any.whl
43
+ juurio setup # download and verify the model checkpoints and data
44
+ ```
45
+
46
+ Python 3.12 or newer. The install pulls in `torch`, `transformers`, `numpy` and `huggingface-hub`.
47
+
48
+ On a machine with a GPU, that installs the CUDA build of PyTorch. On a CPU-only machine the default PyTorch wheel also bundles CUDA, a download of several GB you can't use. To get the small CPU-only build instead, install it first:
49
+
50
+ ```bash
51
+ pip install torch --index-url https://download.pytorch.org/whl/cpu
52
+ pip install https://huggingface.co/samuli/juurio/resolve/main/juurio-0.1.0-py3-none-any.whl
53
+ ```
54
+
55
+ The wheel contains only the code. `juurio setup` downloads the model checkpoints and data from this repository (about 190 MB), checks their sha256 hashes and caches them. Without it, they are downloaded on first use. `juurio doctor` shows what is present and what is missing.
56
+
57
+ ## Background
58
+
59
+ Finnish is richly inflecting, and the standard open-source tools for it are excellent but closed-vocabulary and heavy at runtime. A finite-state morphology ([Giella `fin`](https://github.com/giellalt/lang-fin) and [Omorfi](https://github.com/flammie/omorfi), also the engine behind [Voikko](https://voikko.puimula.org/)) inflects only words already in its lexicon, and a neural dependency parser ([Stanza](https://stanfordnlp.github.io/stanza/)) brings a large dependency stack. A loan or slang word the lexicon has never seen (`striimata`, `striimaaja`) cannot be inflected.
60
+
61
+ Juurio distills those tools into small neural models and a compact exception lexicon, so at runtime it needs only PyTorch and it can inflect words the finite-state morphology does not know. The heavy tools were used only to build the models and data: Giella `fin` is the morphology the neural models learned from, Stanza produced silver parses for the parser, and Voikko was used to check generated forms.
62
+
63
+ ## Architecture
64
+
65
+ | concern | component |
66
+ |---|---|
67
+ | structure and scope (which clause, subject, object) | neural UD dependency parser |
68
+ | morphology of known words | exception lexicon (the forms the neural realizers get wrong) |
69
+ | open-vocabulary morphology (OOV and loan words) | small neural realizers: nominal, verb-form and finite-verb |
70
+
71
+ The rule engine handles all non-local structure (agreement, object case, subject drop). The neural realizers are called per word, only when the lexicon has no entry for the form.
72
+
73
+ ## Entry points
74
+
75
+ - `transform(op, text, lemma=None)`: finite-clause transforms that preserve grammar, such as negation, tense and mood (including the potential and the compound tenses), passive, question, the full imperative paradigm (with the negative `älä` series), number with agreement, degree, enclitics, possessives and `+`-composed nominal morphology.
76
+ - `realize_np(np, …)`: phrase-level nominal morphology, with the full case system, possessive suffixes, clitics and modifier agreement. Arbitrary case lives here because it is syntactically governed.
77
+ - `realize_verbform(verb, …)`: phrase-level non-finite verb morphology. Participles (VA, NUT, TAVA, TU and the agent participle MA) and infinitives, inflected in case and number with an optional possessive suffix.
78
+ - `inflect(word, …)`, `analyze(text)` and `lemmatize(text)`: single-word inflection, analysis and lemmatization.
79
+
80
+ Anything semantic, such as existential framing, is deliberately out of scope.
81
+
82
+ ## CLI
83
+
84
+ Every library function has a command of the same name. The CLI is a thin wrapper over the same `Transformer`.
85
+
86
+ ```
87
+ # one command per library function
88
+ juurio transform <op> "<text>" [--lemma L] [--trace] [--json] # transform()
89
+ juurio inflect "<word>" [spec…] [--case C] [--number N] [--degree D] [--px P] [--clitic K] [--pos P] [--like W] [--proper] [--json] # inflect()
90
+ juurio analyze "<text>" [--json] # analyze()
91
+ juurio lemmatize "<text>" [--json] # lemmatize()
92
+ juurio realize-np "<np>" [--case Ine] [--number Plur] [--px 1Sg] [--clitic kin] # realize_np()
93
+ juurio realize-verbform "<verb>" [--form va|nut|tava|tu|inf2|inf2_pass|inf3|ma] [--case Ine] [--px 3] # realize_verbform()
94
+
95
+ # infrastructure (no library counterpart)
96
+ juurio doctor # read-only environment diagnostics (dependencies, checkpoints + sha256)
97
+ juurio setup # download and verify the checkpoints and data
98
+ juurio list-ops # documented operation catalog
99
+ juurio batch <op> < lines # or per-line "op<TAB>text[<TAB>lemma]"
100
+ juurio demo | bench | repl
101
+ ```
102
+
103
+ `inflect` takes a case, number and degree phrase positionally (`juurio inflect talo mon gen` → `talojen`) or the equivalent `--case`, `--number` and `--degree` flags.
104
+
105
+ Exit status is `0` on success, `2` if any transform abstained (useful for scripting) and `1` on setup failure.
106
+
107
+ ## Files
108
+
109
+ | path | what |
110
+ |---|---|
111
+ | `juurio-0.1.0-py3-none-any.whl` | the Python package |
112
+ | `juurio/`, `pyproject.toml` | source code of the package |
113
+ | `docs/api.md` | API and CLI contract, scope boundary, reason codes |
114
+ | `checkpoints/` | parser, analyser and realizer models with their tokenizer and vocabularies |
115
+ | `data/` | exception lexicon, morphology tag table and vocabularies |
116
+ | `ATTRIBUTION.md` | licences and sources, file by file |
117
+
118
+ ## Training data
119
+
120
+ - The realizers learned from inflected forms generated with the Giella `fin` transducer, for lemmas found in a Finnish web corpus.
121
+ - The exception lexicon holds the Giella forms the realizers get wrong.
122
+ - The parser was trained on [UD_Finnish-TDT](https://universaldependencies.org/treebanks/fi_tdt/index.html) and on silver parses of Finnish web text made with Stanza.
123
+ - The morphology tag table lists Giella's tags for the 100,000 most frequent web-text word forms where the neural analyser disagrees with Giella. E-mail addresses, @handles, URLs, phone numbers and hashtags were removed from it.
124
+
125
+ ## Limitations
126
+
127
+ - `transform` handles one finite main clause in the active indicative, within limits of about twelve words and one finite verb. Subordinate and coordinated clauses, already negated clauses and multi-verb fragments abstain.
128
+ - Forms from the neural realizers are model guesses. They are flagged in `warnings` but not verified, and some are wrong: `realize_np("bugisovellus", case="Ine", px="1Pl", clitic="kin")` gives *bugisovelludessammekin* instead of *bugisovelluksessammekin*.
129
+ - The parser makes mistakes, and a wrong lemma or analysis can lead to a wrong form or an abstention. The `lemma=` argument overrides the lemma of the main verb.
130
+
131
+ ## License
132
+
133
+ Three parts with separate licences (details in [`ATTRIBUTION.md`](ATTRIBUTION.md)):
134
+
135
+ | part | licence |
136
+ |---|---|
137
+ | code (`juurio/`, the wheel) | [MIT](LICENSE) |
138
+ | models (`checkpoints/`, `data/char_vocab.json`) | [CC BY 4.0](LICENSE-CC-BY-4.0) |
139
+ | data derived from Giella `fin` (`data/wordlist.json`, `data/fst_table.jsonl`, `data/fst_tag_vocab.json`) | [GPL-3.0](LICENSE-GPL-3.0), the licence of [GiellaLT `lang-fin`](https://github.com/giellalt/lang-fin) (© UiT The Arctic University of Norway / Giellatekno) |
docs/api.md ADDED
@@ -0,0 +1,316 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # juurio — API & CLI contract
2
+
3
+ Open-vocabulary Finnish morphological transforms over raw text. You hand it a sentence and an
4
+ operation (`past`, `passive`, `plural`, …); it parses, transforms the finite main clause, and
5
+ returns the result or an explicit abstention. The differentiator is **OOV/loan words**: words the
6
+ exception lexicon does not cover (`striimata`, `tubettaja`) are inflected by small neural realizers.
7
+
8
+ ## Architecture (the "split thesis")
9
+
10
+ Three components, each owning what it is best at:
11
+
12
+ | Concern | Component |
13
+ |---|---|
14
+ | structure / scope (which clause, which subject/object) | neural UD dependency parser (Giella/Stanza-free) |
15
+ | idiosyncrasy-free morphology (known words) | net + lexicon exception table (byte-exact to the Giella FST) |
16
+ | open-vocabulary morphology (OOV/loan words) | tiny lemma-anchored neural realizers |
17
+
18
+ The serving runtime needs only torch/numpy/transformers — no Stanza, no Giella FST, no Voikko (those
19
+ are used only to build the models and data). The rule engine owns all non-local structure (agreement propagation, object
20
+ case, subject drop); the neural realizers are called **per token**, only when the lexicon has no
21
+ entry for the cell.
22
+
23
+ ## Scope boundary (a feature, not a limitation)
24
+
25
+ Two entry points with deliberately different promises:
26
+
27
+ - **`transform(op, text)` — finite-clause transforms** that preserve (or deliberately rewrite) the
28
+ clause's grammar: negation, tense/mood, passive, imperative, question, number agreement, possessive
29
+ on the subject head. Output is a grammatical sentence or an explicit abstain.
30
+ - **`realize_np(np, …)` — phrase-level nominal morphology**: inflect an NP into a Case/Number/Px(+
31
+ clitic). This is where *arbitrary case* lives, because case is **syntactically governed** — making
32
+ it a clause op would produce ungrammatical sentences (`Talo on iso` → `Talossa on iso`).
33
+
34
+ Anything **semantic** — e.g. existential/locative framing (`Taloissani on koira`), which introduces
35
+ and reorders arguments — is **out of scope** for both; it would be a separate future *generator* op,
36
+ not a morphological transform. Keeping these apart is what lets each give a clean guarantee.
37
+
38
+ ## Python API
39
+
40
+ ```python
41
+ from juurio import Transformer, transform, SUPPORTED_OPS, OP_CATALOG, REASONS, ABSTAIN_REASONS
42
+
43
+ t = Transformer() # loads parser + models once (heavy; reuse it)
44
+ r = t.transform("past", "Koira haukkuu pihalla.")
45
+ r.ok # True
46
+ r.text # 'Koira haukkui pihalla.'
47
+ r.reason # 'ok'
48
+
49
+ # one-shot convenience (reuses a process-wide Transformer):
50
+ transform("plural", "Striimaaja nukkuu.").text # 'Striimaajat nukkuvat.' (OOV noun, neural)
51
+ ```
52
+
53
+ `import juurio` is cheap; torch + the neural parser/realizers load lazily on the first
54
+ `Transformer()` / `transform()` (no Stanza/Giella at serving time).
55
+
56
+ ### `Transformer.transform(op, text, lemma=None, sent_index=0) -> Result`
57
+
58
+ - **op** — one of `SUPPORTED_OPS` (else `ValueError`).
59
+ - **text** — raw text; a leading dialogue dash/quote is handled and re-prepended.
60
+ - **lemma** — override the parser's lemma for the main verb. Use when a loan stem is mis-lemmatized
61
+ (`striimaa` → parser says `striä`); a confidence warning is raised automatically otherwise.
62
+ - **sent_index** — which parsed sentence to operate on (default 0).
63
+
64
+ ### `Result`
65
+
66
+ | field | meaning |
67
+ |---|---|
68
+ | `text` | transformed sentence, or `None` if abstained |
69
+ | `ok` | `True` iff a form was produced |
70
+ | `reason` | one of `REASONS` (`"ok"` on success, else in `ABSTAIN_REASONS`) |
71
+ | `trace` | provenance dict (see below) — `juurio transform --trace` prints it |
72
+ | `warnings` | low-confidence-lemma notes + **neural realization** notices (model guess, unverified against the lexicon) |
73
+
74
+ `trace` fields: `op`, `root_verb`, `parser_lemma`, `lemma_source` (`override`/`parser`); and, when
75
+ relevant, `neural_oov` (forms the realizer produced — model guesses), `oov_unrealizable` /
76
+ `oov_unverified` (words it could not/did-not-trust). `realize_np` traces `util`, `case`, `number`,
77
+ `px`, `clitic`, `head`. Inspect on the CLI with `--trace`, or as JSON with `--json`.
78
+
79
+ ### Performance
80
+
81
+ Models load once per process; the first call also loads the neural parser (the cold cost).
82
+ Indicative numbers (`juurio bench`, CPU parser + GPU realizer):
83
+
84
+ | | time |
85
+ |---|---|
86
+ | `Transformer()` construct | ~2 s |
87
+ | first `transform` (cold — parser load) | ~9 s |
88
+ | `transform` (warm) | ~70 ms |
89
+ | `realize_np` (warm) | ~30 ms |
90
+
91
+ Reuse one `Transformer` (or the module-level `transform`/`realize_np`, which cache a singleton).
92
+
93
+ ## Scope contract
94
+
95
+ `transform` operates on a **single finite Act/Ind main clause** (≈≤12 words, one finite verb).
96
+ Anything else **abstains** (`ok=False`) — it never guesses. Reason codes:
97
+
98
+ | reason | meaning |
99
+ |---|---|
100
+ | `ok` | a transformation was produced |
101
+ | `parse_empty` | the parser returned no usable sentence at `sent_index` |
102
+ | `no_clause` | no finite Act/Ind main clause (subordinate-only, fragment, or already-negated) |
103
+ | `out_of_scope` | exceeds scope limits (too long / >1 finite verb / coordination) |
104
+ | `op_not_applicable` | op does not apply to this clause's base state (e.g. `past` on a past clause) |
105
+ | `oov_unrealizable` | an open-class word needed inflecting but **neither** the lexicon **nor** the neural realizer could produce the target form |
106
+ | `oov_unverified` | the neural realizer produced a candidate for an OOV word but it failed validation (stem-inconsistent or near-tie) and was **suppressed**, not emitted |
107
+ | `conflicting_features` | `inflect` got two valid but contradictory inputs (spec `mon gen` + `number="sing"`; `proper=True` + a degree; `like=` + an explicit case/number/degree; a single-word control on a phrase) |
108
+ | `unanalyzable_model` | `inflect(like=…)` could not analyze the model word into a nominal case/number cell to copy |
109
+
110
+ **Neural OOV is surfaced, never silent.** When a form is produced by the neural realizer (not the
111
+ lexicon), it appears in `trace["neural_unverified"]` and a `warnings` entry. Treat it as a model guess.
112
+
113
+ ## Operations
114
+
115
+ `SUPPORTED_OPS` is the flat tuple; `OP_CATALOG` groups them with descriptions. Summary:
116
+
117
+ - **finite verb** — `neg`, `past`, `pres`, `cond`, `perfect`, `potential`, `passive`
118
+ - **compound tenses** (analytic olla auxiliary + the perfect participle) — `pluperfect`
119
+ (`olin sanonut`), `cond_perfect` (`olisin sanonut`), `pot_perfect` (`lienen sanonut`); OOV verbs
120
+ reuse the trained `perfect` participle + the lexicon's `olla` auxiliary
121
+ - **composites** (applied atomically) — `neg+past`, `neg+cond`, `neg+perfect`, `passive+past`,
122
+ `passive+cond`, `passive+perfect`, `neg+passive`, `neg+passive+past`, `neg+passive+cond`,
123
+ `neg+passive+perfect`
124
+ - **clause** — `quest` (polar question)
125
+ - **imperative paradigm** (drop subject, total object → Nom; negative forces partitive; trailing
126
+ punctuation preserved) — `imperative` (Sg2, OOV-capable), `imperative_pl2/sg3/pl1/pl3`, and the
127
+ negative `älä`-series `neg_imperative` (Sg2, OOV-capable), `neg_imperative_pl2/sg3/pl1/pl3`.
128
+ **Scope:** applies only when the command's addressee is in the clause — a subjectless/already-imperative
129
+ source (`osta talo.`) or a 2nd-person subject (`sinä/te`); a 3rd/1st-person subject abstains
130
+ (`out_of_scope`), since turning *"the man buys a house"* into a command would invent an addressee.
131
+ - **number** (re-agrees subject NP, modifiers, finite verb, copula predicate) — `plural`, `singular`
132
+ - **degree** (predicate adjective) — `comp`, `super`
133
+ - **enclitics** (on the finite verb) — `clitic_kin`, `clitic_kaan`, `clitic_han`, `clitic_pa`
134
+ - **possessive suffix** on the subject head, preserving its case/number — `poss_1sg`, `poss_2sg`,
135
+ `poss_3`, `poss_1pl`, `poss_2pl` (`talo`→`taloni`, `talossa`→`talossani`; OOV `striimaaja`→`striimaajani`)
136
+
137
+ `juurio list-ops` prints the live catalog.
138
+
139
+ ### Combined nominal morphology
140
+
141
+ Nominal ops compose with `+`: a `+`-join of **nominal atoms** (`plural`/`singular`, `poss_*`,
142
+ `clitic_*`) is applied to the subject head as **one realization** with the final
143
+ Number+Case+Px, then the clitic is attached — deterministic surface order `stem+case+number+Px+clitic`.
144
+ This is correct-by-construction (you cannot re-pluralize the already-realized string `taloni`).
145
+ Number changes also re-agree modifiers and the finite verb.
146
+
147
+ ```
148
+ poss_1sg+clitic_kin Koira haukkuu pihalla. -> Koiranikin haukkuu pihalla.
149
+ plural+poss_1sg Koira haukkuu pihalla. -> Koirani haukkuvat pihalla.
150
+ plural+poss_1sg+clitic_kin Koira haukkuu pihalla. -> Koiranikin haukkuvat pihalla.
151
+ plural+poss_1sg Striimaaja nukkuu. -> Striimaajani nukkuvat. (OOV head, neural)
152
+ ```
153
+
154
+ Each dimension may appear at most once. `+`-composed **verb** ops (`neg+passive+past`) are separate,
155
+ pre-registered atoms in the catalog. Mixed verb+nominal composition is not supported.
156
+
157
+ ## Phrase-level nominal morphology: `realize_np`
158
+
159
+ Finnish case is **syntactically governed**, so arbitrary case is *not* a clause transform (a
160
+ subject-case change would be ungrammatical). Instead it is a phrase utility, separate from
161
+ `transform`, which keeps grammaticality guarantees:
162
+
163
+ ```python
164
+ from juurio import realize_np
165
+ realize_np("talo", case="Ine", number="Plur", px="1Sg").text # 'taloissani'
166
+ realize_np("iso talo", case="Ine", number="Plur").text # 'isoissa taloissa'
167
+ realize_np("Helsinki", case="Ine").text # 'Helsingissä'
168
+ realize_np("striimaaja", case="Ade", number="Plur", px="1Sg", clitic="kin").text # 'striimaajillanikin' (OOV)
169
+ ```
170
+
171
+ `realize_np(np, case, number, px, clitic)` inflects an NP — a bare lemma or a short `adj* noun`
172
+ phrase — into the target Case/Number with an optional possessive and enclitic, agreeing modifiers,
173
+ and per-token lexicon→neural fallback (surface order `stem+case+number+Px+clitic`). It returns a `Result`
174
+ (`ok` with the NP, or an abstain reason). This is the showcase for open-vocabulary nominal
175
+ morphology; `transform` is for grammaticality-preserving clause operations. Sentence-level locatives
176
+ (`Taloissani on koira`) are a separate future generator op, not a case transform.
177
+
178
+ ## Phrase-level verb morphology: `realize_verbform`
179
+
180
+ The non-finite counterpart of `realize_np`: inflect a VERB into a **participle** or **infinitive**.
181
+ The four participles inflect like adjectives in case+number; the 2nd/3rd infinitives are case-marked.
182
+ A second open-vocabulary surface — the lexicon handles known verbs, a small **verbform realizer**
183
+ inflects OOV/loan verbs (`striimata`), flagged `unverified`.
184
+
185
+ ```python
186
+ from juurio import realize_verbform # form ∈ {va, nut, tava, tu, inf2, inf2_pass, inf3, ma}
187
+ realize_verbform("lukea", form="va").text # 'lukeva' (present active participle)
188
+ realize_verbform("lukea", form="va", case="Ine").text # 'lukevassa'
189
+ realize_verbform("lukea", form="tu").text # 'luettu' (past passive participle)
190
+ realize_verbform("mennä", form="inf3", case="Ade").text # 'menemällä' (3rd inf adessive)
191
+ realize_verbform("mennä", form="inf3", case="Ine").text # 'menemässä'
192
+ realize_verbform("tehdä", form="ma", case="Ine").text # 'tekemässä' (agent participle)
193
+ realize_verbform("tehdä", form="inf2_pass").text # 'tehtäessä' (passive 2nd inf, temporal)
194
+ realize_verbform("kuulla", form="tu", case="Par", px="3").text # 'kuultuansa' (temporal: 'after hearing')
195
+ realize_verbform("striimata", form="va").text # 'striimaava' (OOV verb, neural)
196
+ ```
197
+
198
+ `va`/`nut`/`tava`/`tu` = present/past active/passive participle; `inf2`/`inf3` = 2nd/3rd infinitive;
199
+ `inf2_pass` = passive 2nd infinitive (`-ttaessa` temporal, "while X is being done"; inessive-only,
200
+ case-invariant); `ma` = agent participle (-ma/-mä, "the X that someone does", inflects in case/number).
201
+ `px` (one of `None`/`1Sg`/`2Sg`/`3`/`1Pl`/`2Pl`) attaches a possessive suffix — participles inflect
202
+ like nominals, so `tu`+`Par`+`px` realizes the **temporal construction of anteriority** (`kuultuaan` /
203
+ the `-nsa` variant `kuultuansa` = "after hearing"). For OOV verbs the bare participle comes from the
204
+ verbform realizer and the case+Px from the noun realizer (both neural stages flagged unverified).
205
+ Returns a `Result` (`ok` with the form, or `oov_unrealizable`/`oov_unverified` when the morphology
206
+ engine cannot produce/trust it). Like `realize_np`, this is a phrase utility, not a clause transform.
207
+
208
+ ## Friendly inflection: `inflect`
209
+
210
+ The ergonomic single-word front door over `realize_np` — human-readable case/number/degree names, a
211
+ proper-noun toggle, and **inflect-by-example**. Returns a `Result` (never a bare string: abstention
212
+ stays first-class).
213
+
214
+ ```python
215
+ from juurio import inflect
216
+ inflect("talo", "gen").text # 'talon' (alias: gen/genitive/genetiivi)
217
+ inflect("talo", "mon gen").text # 'talojen' (combined spec, taivutin-style)
218
+ inflect("talo", case="inessive", number="plural").text # 'taloissa'
219
+ inflect("talo", px="1sg").text # 'taloni'
220
+ inflect("kaunis", "gen", degree="comp").text # 'kauniimman' (degree => adjective path)
221
+ inflect("meri", "gen").text # 'meren' (common noun)
222
+ inflect("Meri", "gen", proper=True).text # proper-noun reading (≠ 'meren')
223
+ inflect("makaronilaatikko", like="talojen").text # 'makaronilaatikkojen' (copy a model word's cell)
224
+ inflect("iso talo", "mon gen").text # 'isojen talojen' (phrase -> realize_np)
225
+ ```
226
+
227
+ `inflect(word, case="nom", number="sing", *, degree="pos", px="none", clitic=None, proper=None,
228
+ pos=None, like=None)`:
229
+
230
+ - **case** — a case alias, *or* a combined spec (`"mon gen"`, `"plural genitive comparative"`).
231
+ Accepts English abbrev/full and Finnish (`gen`/`genitive`/`genetiivi`, `mon`/`monikko`).
232
+ - **number / degree** — aliases too; an explicit value that contradicts the combined spec abstains
233
+ (`conflicting_features`).
234
+ - **px / clitic** — possessive suffix (`1sg`…/`None`) and enclitic, as in `realize_np`. `px=None` is
235
+ accepted (= no suffix). Rejected on a single adjective target.
236
+ - **proper** — `True`/`False` forces PROPN/NOUN; `None` (default) infers PROPN from a leading capital.
237
+ - **pos** — explicit `'noun'`/`'proper'`/`'adjective'` (wins over inference). PRON/NUM are not yet a
238
+ first-class `pos`. A degree implies the adjective path when `pos` is unset.
239
+ - **like** — a model word; its analyzed case (+number, +degree if adjectival) is copied onto `word`.
240
+ Mutually exclusive with `case`/`number`/`degree`; an unanalyzable model abstains
241
+ (`unanalyzable_model`).
242
+
243
+ An **unknown alias** raises `ValueError` (a typo fails fast); a **contradiction** abstains
244
+ (`conflicting_features`). A single token is realized **parser-free** (so `proper`/`pos` are honored);
245
+ a whitespace phrase delegates to `realize_np` (which applies `px`/`clitic` to the head but rejects the
246
+ single-word controls `proper`/`pos`/`degree`). An empty `word` raises `ValueError`.
247
+
248
+ ## Analysis: `analyze` / `lemmatize`
249
+
250
+ The inverse of generation (what a generate-only inflector cannot do): the serving neural parser
251
+ exposed directly.
252
+
253
+ ```python
254
+ from juurio import analyze, lemmatize, TokenAnalysis
255
+ sents = analyze("Koirat haukkuvat pihalla.") # list[list[TokenAnalysis]] (one list per sentence)
256
+ tok = sents[0][0]
257
+ tok.lemma, tok.upos, tok.feats # 'koira', 'NOUN', {'Case': 'Nom', 'Number': 'Plur'}
258
+ tok.asdict() # plain dict (id/form/lemma/upos/head/deprel/feats)
259
+ lemmatize("Koirat haukkuvat pihalla.") # ['koira', 'haukkua', 'piha', '.'] (flat)
260
+ ```
261
+
262
+ `TokenAnalysis` is a frozen record (`id`, `form`, `lemma`, `upos`, `head` — 1-based, 0=root —,
263
+ `deprel`, `feats` as a read-only UD mapping). `analyze` keeps sentence boundaries; `lemmatize`
264
+ returns a flat lemma list.
265
+
266
+ ## CLI
267
+
268
+ Each library function has a like-named command (the CLI is a thin wrapper over the same
269
+ `Transformer`); `realize_np`/`realize_verbform` keep the hyphenated spelling.
270
+
271
+ ```
272
+ # one command per library function
273
+ juurio transform <op> "<text>" [--lemma L] [--trace] [--json] # transform()
274
+ juurio inflect "<word>" [spec…] [--case C] [--number N] [--degree D] [--px P] [--clitic K] [--pos P] [--like W] [--proper] [--json] # inflect()
275
+ juurio analyze "<text>" [--json] # analyze()
276
+ juurio lemmatize "<text>" [--json] # lemmatize()
277
+ juurio realize-np "<np>" [--case Ine] [--number Plur] [--px 1Sg] [--clitic kin] [--json] # realize_np()
278
+ juurio realize-verbform "<verb>" [--form va|nut|tava|tu|inf2|inf2_pass|inf3|ma] [--case Ine] [--number Plur] [--px 3] # realize_verbform()
279
+
280
+ # infrastructure (no library counterpart)
281
+ juurio list-ops
282
+ juurio batch <op> [--lemma L] [--json] < lines # one op for every stdin line
283
+ juurio batch [--json] < tsv # per-line "op<TAB>text[<TAB>lemma]"
284
+ juurio demo # curated tour (incl. OOV) — also a smoke test
285
+ juurio repl # interactive shell; model loaded once
286
+ ```
287
+
288
+ `inflect` accepts a human case/number/degree phrase positionally (`juurio inflect talo mon gen`
289
+ → `talojen`, `juurio inflect kaunis gen --degree comp`), or as `--case/--number/--degree` flags;
290
+ a positional phrase wins over the flags. `--proper`/`--no-proper` force/forbid a proper-noun reading.
291
+
292
+ `juurio repl` is a read-eval loop over every feature (model loaded once): `inflect talo mon gen`,
293
+ `inflect makaronilaatikko like=talojen`, `transform passive Mies osti talon.`, `analyze …`,
294
+ `lemmatize …`, `np …`, `verb …`, `ops`, `help`. Options are `key=value` (`px=1sg proper=true
295
+ pos=adjective clitic=kin like=talojen`).
296
+
297
+ Also `python -m juurio …`. Without `--json`, prints the result (warnings appended after `⚠`) and
298
+ sends abstentions to stderr as `[abstain: <reason>] <input>`. With `--json`, one JSON object per
299
+ line (`text/ok/reason/warnings/trace`). **Exit status:** `0` if every transform produced a form,
300
+ `2` if any abstained — so scripts can branch on it.
301
+
302
+ ```console
303
+ $ juurio transform plural "Striimaaja nukkuu."
304
+ Striimaajat nukkuvat. ⚠ neural realization (unverified by FST): ['striimaaja->Striimaajat (NOUN)']
305
+ ```
306
+
307
+ ## Quality
308
+
309
+ The package is designed to **abstain rather than guess**: out-of-scope inputs return an explicit
310
+ abstention (`ok=False`) with a reason code instead of a speculative form, and neural OOV realizations
311
+ are always surfaced (in `trace["neural_unverified"]` and `warnings`) rather than emitted silently. Known
312
+ words are inflected to be byte-exact to the reference FST (through the exception lexicon); OOV/loan words are inflected by the neural
313
+ realizers, with stem-inconsistent or near-tie candidates suppressed (`oov_unverified`) rather than
314
+ returned.
315
+
316
+ Run `juurio demo` for a curated tour (including OOV cases) that doubles as a smoke test.
juurio-0.1.0-py3-none-any.whl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:88e74693dabe579bb500bbd66dd0bfc8282821770c1afb0b05b41d719c5c42cd
3
+ size 182902
juurio/__init__.py ADDED
@@ -0,0 +1,210 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """juurio — open-vocabulary Finnish morphological transforms over raw text.
2
+
3
+ Public, stable surface:
4
+
5
+ from juurio import Transformer, transform, SUPPORTED_OPS, REASONS, ABSTAIN_REASONS
6
+
7
+ t = Transformer() # loads parser + models once
8
+ r = t.transform("past", "Koira haukkuu pihalla.")
9
+ r.ok, r.text, r.reason # True, 'Koira haukkui pihalla.', 'ok'
10
+
11
+ `transform(op, text, lemma=None)` is a one-shot convenience that lazily builds and reuses a
12
+ process-wide `Transformer`. For anything beyond a quick call, construct a `Transformer` yourself.
13
+
14
+ Every transform returns a `Result`. `ok` is True only when a form was produced; otherwise `ok` is
15
+ False and `reason` is one of `ABSTAIN_REASONS`. Abstention is a first-class result, never a guess.
16
+ Neural realizations (forms the FST could not verify at runtime) are surfaced in `Result.warnings` and
17
+ `trace["neural_unverified"]` (deprecated alias `trace["neural_oov"]`) — model guesses, unverified by the
18
+ FST. "Unverified" is provenance (net- vs lexicon-served), not a claim the lemma is out-of-vocabulary.
19
+
20
+ Heavy deps (torch, stanza) are imported lazily, so `import juurio` is cheap; the cost is paid on
21
+ first `Transformer()` / `transform(...)`.
22
+ """
23
+ from __future__ import annotations
24
+
25
+ from . import features # pure alias-normalization (no heavy deps); supplies the UNSET sentinel
26
+
27
+ # The documented public op catalog, grouped by what they do. Keys are the strings `transform`
28
+ # accepts; the flat set is `SUPPORTED_OPS`. (Mirrors `algebra.ALL_OPS`; this is the stable view.)
29
+ OP_CATALOG = {
30
+ "polarity/tense/mood (finite verb)": {
31
+ "neg": "negate (connegative + ei-auxiliary)",
32
+ "past": "past tense (imperfect)",
33
+ "pres": "present tense",
34
+ "cond": "conditional mood (-isi-)",
35
+ "perfect": "perfect (analytic olla + past participle)",
36
+ "potential": "potential mood (-nee-; suppletive olla -> lienee)",
37
+ "passive": "impersonal passive (drop subject, total object -> Nom)",
38
+ },
39
+ "compound tenses (analytic olla auxiliary + the perfect participle)": {
40
+ "pluperfect": "pluperfect (olin sanonut; auxiliary in the past)",
41
+ "cond_perfect": "conditional perfect (olisin sanonut)",
42
+ "pot_perfect": "potential perfect (lienen sanonut; suppletive olla -> lienee)",
43
+ },
44
+ "composites (applied as one op)": {
45
+ "neg+past": "", "neg+cond": "", "neg+perfect": "",
46
+ "passive+past": "", "passive+cond": "", "passive+perfect": "",
47
+ "neg+passive": "", "neg+passive+past": "", "neg+passive+cond": "",
48
+ "neg+passive+perfect": "",
49
+ },
50
+ "clause": {
51
+ "quest": "polar question (verb-fronting + -ko/-kö)",
52
+ },
53
+ "imperative paradigm (drop subject, total object -> Nom, '.' -> '!')": {
54
+ "imperative": "affirmative Sg2 (Mene!) — the trained flag; OOV-capable",
55
+ "imperative_pl2": "affirmative Pl2 (Menkää!)",
56
+ "imperative_sg3": "affirmative Sg3 (Menköön!)",
57
+ "imperative_pl1": "affirmative Pl1 (Menkäämme! — 'let us')",
58
+ "imperative_pl3": "affirmative Pl3 (Menkööt!)",
59
+ "neg_imperative": "negative Sg2 (Älä mene!) — OOV-capable",
60
+ "neg_imperative_pl2": "negative Pl2 (Älkää menkö!)",
61
+ "neg_imperative_sg3": "negative Sg3 (Älköön menkö!)",
62
+ "neg_imperative_pl1": "negative Pl1 (Älkäämme menkö!)",
63
+ "neg_imperative_pl3": "negative Pl3 (Älkööt menkö!)",
64
+ },
65
+ "number (re-agrees subject, modifiers, finite verb, copula predicate)": {
66
+ "plural": "singular subject NP -> plural",
67
+ "singular": "plural subject NP -> singular",
68
+ },
69
+ "degree (predicate adjective)": {
70
+ "comp": "comparative (-mpi)",
71
+ "super": "superlative (-in)",
72
+ },
73
+ "enclitic particles (attached to the finite verb)": {
74
+ "clitic_kin": "-kin / -kään (additive: 'also / either')",
75
+ "clitic_kaan": "-kaan / -kään",
76
+ "clitic_han": "-han / -hän (familiarizing)",
77
+ "clitic_pa": "-pa / -pä (emphatic)",
78
+ },
79
+ "possessive suffix on the subject head (preserves its case/number)": {
80
+ "poss_1sg": "-ni ('my': talo -> taloni, talossa -> talossani)",
81
+ "poss_2sg": "-si ('your sg')",
82
+ "poss_3": "-nsa / -Vn ('his/her/their'; 3rd person, Sg & Pl)",
83
+ "poss_1pl": "-mme ('our')",
84
+ "poss_2pl": "-nne ('your pl')",
85
+ },
86
+ }
87
+
88
+ SUPPORTED_OPS: tuple[str, ...] = tuple(op for group in OP_CATALOG.values() for op in group)
89
+
90
+ _singleton = None
91
+
92
+
93
+ def transform(op: str, text: str, lemma: str | None = None, sent_index: int = 0):
94
+ """One-shot convenience: transform `text` with `op`, reusing a process-wide `Transformer`.
95
+
96
+ `lemma` overrides the parser's lemma for the main verb (use when a loan stem is mis-lemmatized).
97
+ Returns a `Result`. Builds the (heavy) Transformer on first call; subsequent calls reuse it."""
98
+ global _singleton
99
+ if _singleton is None:
100
+ from .pipeline import Transformer
101
+ _singleton = Transformer()
102
+ return _singleton.transform(op, text, lemma=lemma, sent_index=sent_index)
103
+
104
+
105
+ def realize_np(np: str, case: str = "Nom", number: str = "Sing",
106
+ px: str = "None", clitic: str | None = None):
107
+ """One-shot phrase-level nominal morphology (NOT a clause transform): inflect an NP — a bare
108
+ lemma or a short `adj* noun` phrase — into the target Case/Number, optional possessive + clitic.
109
+ Open-vocab (FST->neural fallback). Returns a `Result`. Reuses a process-wide `Transformer`.
110
+
111
+ realize_np("talo", case="Ine", number="Plur", px="1Sg").text # 'taloissani'
112
+ realize_np("striimaaja", case="Ade", number="Plur", px="1Sg", clitic="kin").text
113
+ realize_np("iso talo", case="Ine").text # 'isossa talossa'
114
+ """
115
+ global _singleton
116
+ if _singleton is None:
117
+ from .pipeline import Transformer
118
+ _singleton = Transformer()
119
+ return _singleton.realize_np(np, case=case, number=number, px=px, clitic=clitic)
120
+
121
+
122
+ def realize_verbform(verb: str, form: str = "va", case: str = "Nom", number: str = "Sing",
123
+ px: str = "None", lemma: str | None = None):
124
+ """One-shot non-finite verb morphology (NOT a clause transform): inflect a VERB into a participle
125
+ or infinitive, optionally with a possessive suffix `px`. Reuses a process-wide `Transformer`.
126
+ `form` is one of `VERBFORMS`.
127
+
128
+ realize_verbform("lukea", form="va", case="Ine").text # 'lukevassa' (present participle)
129
+ realize_verbform("mennä", form="inf3", case="Ade").text # 'menemällä' (3rd inf adessive)
130
+ realize_verbform("kuulla", form="tu", case="Par", px="3").text # 'kuultuansa' (temporal: 'after hearing')
131
+ """
132
+ global _singleton
133
+ if _singleton is None:
134
+ from .pipeline import Transformer
135
+ _singleton = Transformer()
136
+ return _singleton.realize_verbform(verb, form=form, case=case, number=number, px=px, lemma=lemma)
137
+
138
+
139
+ # the non-finite verb forms `realize_verbform` accepts (mirrors algebra.VERBFORMS without importing it)
140
+ VERBFORMS: tuple[str, ...] = ("va", "nut", "tava", "tu", "inf2", "inf2_pass", "inf3", "ma")
141
+
142
+
143
+ def inflect(word: str, case=features.UNSET, number=features.UNSET, degree=features.UNSET, *,
144
+ px=features.UNSET, clitic: str | None = None, proper: bool | None = None,
145
+ pos: str | None = None, like: str | None = None):
146
+ """One-shot friendly nominal inflection (the taivutin-style front door): inflect a single word
147
+ or short phrase into a target case/number with human-readable aliases, a proper-noun toggle, and
148
+ inflect-by-example (`like=`). Returns a `Result`. See `Transformer.inflect` for the full contract.
149
+
150
+ inflect("talo", "mon gen").text # 'talojen'
151
+ inflect("makaronilaatikko", like="talojen").text # 'makaronilaatikkojen'
152
+ """
153
+ return _get().inflect(word, case, number, degree, px=px, clitic=clitic, proper=proper,
154
+ pos=pos, like=like)
155
+
156
+
157
+ def analyze(text: str):
158
+ """One-shot analysis: parse `text` into sentences of `TokenAnalysis` (lemma, UPOS, head, deprel,
159
+ UD feats) — the inverse of generation. Reuses a process-wide `Transformer`."""
160
+ return _get().analyze(text)
161
+
162
+
163
+ def lemmatize(text: str):
164
+ """One-shot lemmatization: the lemma of every token in `text`, flat across sentences."""
165
+ return _get().lemmatize(text)
166
+
167
+
168
+ # Building a Transformer imports the INTERNAL `juurio.analyze` subpackage (the parser's hybrid
169
+ # analysis channel). Importing a submodule binds it as an attribute of this package, shadowing the
170
+ # public `analyze`/`lemmatize` convenience functions above (`import juurio; juurio.analyze(...)`
171
+ # would then hit the module, not the function). We capture the originals here and re-bind them after
172
+ # any Transformer construction (see `_restore_public_names`, called from `Transformer.__init__`).
173
+ # A rename of the subpackage was rejected as riskier (checkpoints may pickle `juurio.analyze.*`).
174
+ # We capture and restore `inflect` alongside analyze/lemmatize as a cheap safeguard against any future
175
+ # re-binding.
176
+ _ORIG_ANALYZE = analyze
177
+ _ORIG_LEMMATIZE = lemmatize
178
+ _ORIG_INFLECT = inflect
179
+
180
+
181
+ def _restore_public_names() -> None:
182
+ """Re-bind the public `analyze`/`lemmatize` functions, undoing the shadowing caused by importing
183
+ the internal `juurio.analyze` subpackage during Transformer/parser construction (`inflect` is
184
+ restored defensively too)."""
185
+ import sys
186
+ mod = sys.modules[__name__]
187
+ mod.analyze = _ORIG_ANALYZE
188
+ mod.lemmatize = _ORIG_LEMMATIZE
189
+ mod.inflect = _ORIG_INFLECT
190
+
191
+
192
+ def _get():
193
+ """Lazily build and reuse the process-wide `Transformer` (defers the heavy parser/model load)."""
194
+ global _singleton
195
+ if _singleton is None:
196
+ from .pipeline import Transformer
197
+ _singleton = Transformer()
198
+ return _singleton
199
+
200
+
201
+ def __getattr__(name): # lazy re-export so `import juurio` doesn't pull in torch/stanza
202
+ if name in ("Transformer", "Result", "TokenAnalysis", "REASONS", "ABSTAIN_REASONS"):
203
+ from . import pipeline
204
+ return getattr(pipeline, name)
205
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
206
+
207
+
208
+ __all__ = ["Transformer", "Result", "TokenAnalysis", "transform", "inflect", "analyze", "lemmatize",
209
+ "realize_np", "realize_verbform", "VERBFORMS", "features",
210
+ "SUPPORTED_OPS", "OP_CATALOG", "REASONS", "ABSTAIN_REASONS"]
juurio/__main__.py ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ import sys
2
+
3
+ from .cli import main
4
+
5
+ sys.exit(main())
juurio/algebra.py ADDED
@@ -0,0 +1,1438 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Rule-based engine for a family of Finnish grammatical clause transformations.
2
+
3
+ Given a parsed clause and an operation name, the engine applies the transformation by
4
+ deterministic rules, preserving the unchanged material, and realizes surface forms via the
5
+ Giella FST (known words) or the neural realizers (out-of-vocabulary words). It handles
6
+ transformations of *different kinds*:
7
+ neg — clause restructuring: insert `ei`, connegative/participle, object->partitive
8
+ quest — constituent movement: front the finite verb + `-ko/-kö` clitic (harmony)
9
+ comp — local morphology: positive adjective -> comparative
10
+ plus tense/mood, passive, the imperative paradigm, number±agreement, degree, enclitics,
11
+ possessives, and `+`-composed nominal morphology.
12
+
13
+ `OPS` / `EXT_OPS` / `ALL_OPS` are the op tables; `realize_np_form` and `realize_verbform_form`
14
+ are the phrase-level utilities (see juurio.realizer / verbforms / finiteverb for the realizers).
15
+ """
16
+
17
+ from __future__ import annotations
18
+
19
+ import os
20
+
21
+ from .device import DEV
22
+ from .morphforms import _main_verb, swap_initial, target_str as _tgt
23
+
24
+ CKPT = os.path.join(os.path.dirname(__file__), "..", "checkpoints", "algebra.pt")
25
+ BACK = set("aou") # back vowels -> -ko ; else -kö
26
+
27
+
28
+ # ===================== unified clause-feature core =====================
29
+ # The verb-feature ops (neg/past/pres/cond/passive) and their compositions all set
30
+ # features on the main clause and re-realize the verb cluster once, so morphological
31
+ # interactions are handled automatically (past+neg -> `ei ostanut`; passive+neg ->
32
+ # `ei osteta`). Object/subject case is then derived from the FINAL feature set.
33
+
34
+ NUMTAG = {"Sing": "Sg", "Plur": "Pl"}
35
+ AGREE = {"amod", "nummod", "det", "acl"}
36
+ _EI_FORMS = {"ei", "en", "et", "emme", "ette", "eivät"}
37
+
38
+
39
+ # onsets for nonce augmentation: single consonants + loan-like clusters. Word-initial only,
40
+ # so always paradigm-safe (Finnish gradation never touches the initial consonant).
41
+ ONSETS = list("kptsvnmlrhjdgbf") + ["tr", "kr", "pr", "gr", "str", "sp", "st", "sk",
42
+ "pl", "kl", "fl", "br", "dr"]
43
+
44
+
45
+ def _rand_onset(rng, init):
46
+ return rng.choice([o for o in ONSETS if o[0] != init])
47
+
48
+
49
+ def _vclass(lemma):
50
+ """Coarse Finnish verb conjugation type from the lemma's infinitive ending (productive).
51
+ Used only to rebalance nonce augmentation toward rare-but-regular classes (4/5/6),
52
+ which loanwords (googlata=4, tinderöidä=6) fall into and which the corpus under-samples."""
53
+ l = lemma.lower()
54
+ if l.endswith(("oida", "öidä")):
55
+ return 6
56
+ if l.endswith(("ella", "ellä")):
57
+ return 3
58
+ if l.endswith(("ata", "ätä", "ota", "ötä", "uta", "ytä", "eta", "etä", "ita", "itä")):
59
+ return 4
60
+ if l.endswith(("ta", "tä", "da", "dä")):
61
+ return 2
62
+ return 1
63
+
64
+
65
+ # extra nonce multiplier per class — inversely tracks corpus frequency (type 6 ~1% of tokens)
66
+ _CLASS_BOOST = {6: 4, 4: 2, 3: 2, 2: 1, 1: 1}
67
+
68
+
69
+ # --- lemma anchoring (product path only) ------------------------------------
70
+ # A single inflected form can be class-ambiguous: present `googlaa` (type-4 <googlata)
71
+ # is surface-identical to type-1 `maksaa`, but their pasts differ (googlasi vs maksoi).
72
+ # Prepending the dictionary form (`googlata § ...`) supplies the disambiguating signal a
73
+ # speaker always has — the model then only has to apply the PRODUCTIVE class rule. For
74
+ # nonce training we prepend the ORIGINAL (un-swapped) lemma, so its onset deliberately
75
+ # mismatches the swapped surface: that forces "class from lemma ending, stem from surface"
76
+ # rather than copying the lemma wholesale. The FST cannot do this (closed lexicon: it
77
+ # returns nothing for striimata/tubettaa/tinderöidä even GIVEN the lemma) — so it is a
78
+ # real open-vocabulary differentiator, not a reshuffle of rule-based capability.
79
+ LEMSEP = "§"
80
+
81
+
82
+ def _main_lemma(s, op):
83
+ if op in ("comp", "super"):
84
+ adjs = [t for t in s if t["upos"] == "ADJ" and t["feats"].get("Degree", "Pos") == "Pos"]
85
+ return adjs[0]["lemma"] if adjs else None
86
+ v, _ = _main_verb(s)
87
+ return v["lemma"] if v else None
88
+
89
+
90
+ def _wrap_src(src_str, s, op):
91
+ lem = _main_lemma(s, op)
92
+ return f"{lem} {LEMSEP} {src_str}" if lem else src_str
93
+
94
+
95
+ def _maybe_wrap(stoi, src_str, s, op):
96
+ """Wrap with the lemma prefix iff the loaded model is lemma-anchored (its vocab has §)."""
97
+ return _wrap_src(src_str, s, op) if LEMSEP in stoi else src_str
98
+
99
+
100
+ def _clitic(form):
101
+ return "ko" if any(c in BACK for c in form.lower()) else "kö"
102
+
103
+
104
+ def _children(sent):
105
+ ch = {}
106
+ for t in sent:
107
+ ch.setdefault(t["head"], []).append(t)
108
+ return ch
109
+
110
+
111
+ def _subtree_ids(sent, root_id):
112
+ ch = _children(sent)
113
+ out, stack = set(), [root_id]
114
+ while stack:
115
+ i = stack.pop(); out.add(i)
116
+ for c in ch.get(i, []):
117
+ stack.append(c["id"])
118
+ return out
119
+
120
+
121
+ def clause(sent):
122
+ """(verb_token, is_cop, base_features) or None. Skips already-negated clauses
123
+ (so negation gold never double-negates)."""
124
+ verb, is_cop = _main_verb(sent)
125
+ if verb is None:
126
+ return None
127
+ if any(t["lemma"] == "ei" or t["form"].lower() in _EI_FORMS for t in sent):
128
+ return None
129
+ f = verb["feats"]
130
+ base = dict(voice=f.get("Voice", "Act"), mood=f.get("Mood", "Ind"),
131
+ tense="Prt" if f.get("Tense") == "Past" else "Prs",
132
+ person=f.get("Person", "3"), number=f.get("Number", "Sing"), pol="Aff")
133
+ return verb, is_cop, base
134
+
135
+
136
+ _IMPERATIVE_MOODS = ("Ind", "Imp")
137
+
138
+
139
+ def _imperative_source_verb(sent):
140
+ """Root finite Act verb of an imperative-able clause, or None. Unlike `clause()` this also
141
+ accepts an already-imperative root (`osta …`, Mood=Imp), since the imperative ops generate from
142
+ the lemma regardless of source mood. Rejects copular clauses (root is the predicate, not a verb),
143
+ passives, and already-negated clauses. The 2nd-person/subjectless scope restriction is applied
144
+ separately by `imperative_in_scope`."""
145
+ root = next((t for t in sent if t["head"] == 0), None)
146
+ if root is None or root["upos"] not in ("VERB", "AUX"):
147
+ return None
148
+ f = root["feats"]
149
+ if (f.get("VerbForm") != "Fin" or f.get("Voice", "Act") != "Act"
150
+ or f.get("Mood", "Ind") not in _IMPERATIVE_MOODS):
151
+ return None
152
+ if any(t["lemma"] == "ei" or t["form"].lower() in _EI_FORMS for t in sent):
153
+ return None
154
+ return root
155
+
156
+
157
+ def _stem_overlap(a, b):
158
+ """Length of the shared leading stem of two surface strings (case-insensitive)."""
159
+ n = 0
160
+ for x, y in zip(a.lower(), b.lower()):
161
+ if x != y:
162
+ break
163
+ n += 1
164
+ return n
165
+
166
+
167
+ _OBJ_CASES = {"Par", "Gen", "Acc"} # core grammatical-object cases; Nom strays are subject candidates
168
+ _NOMINAL = {"NOUN", "PROPN", "PRON", "NUM", "ADJ"} # tokens that can be a clause argument
169
+
170
+
171
+ def repair_root_verb_with_lemma(sent, lemma):
172
+ """Hint repair for the ambiguous-loan-verb mis-tag. When a serving `lemma` hint names the clause's
173
+ main verb but the (neural) parser tagged the clause root as a non-verb — e.g. 'striimaa' →
174
+ NOUN/striima/Par instead of VERB/striimata, which makes `clause()` return None — re-tag that token as
175
+ a finite Act/Ind verb and normalize the broken tree into a single-root clause, so clause detection
176
+ finds it and downstream object/number/passive ops see real structure. Returns True iff a repair was
177
+ applied.
178
+
179
+ The real parse of this failure is MULTI-ROOT: the parser splits the clause, leaving both the
180
+ mis-tagged predicate and its argument(s) at head==0 (e.g. 'striimaa' AND 'peliä'). So the repair:
181
+ 1. promotes the mis-tagged root to a finite present Act/Ind verb;
182
+ 2. re-attaches every OTHER root to it (oblique-cased -> obj, else obl), guaranteeing a SINGLE root
183
+ (without this, `in_scope()` would accept a malformed multi-root tree and object/number/passive
184
+ ops would emit from broken structure rather than from the real arguments);
185
+ 3. recovers the subject (an existing nsubj, else a Nominative nominal dependent) and derives finite
186
+ agreement from it.
187
+
188
+ Conservative by design, so it cannot fabricate a clause from a genuinely out-of-scope input. It fires
189
+ ONLY when ALL hold: there is currently no finite main verb (correct clauses are untouched); the
190
+ sentence has at least one OTHER content (non-PUNCT) token (a bare fragment `Striimaaja.` stays out of
191
+ scope); the chosen token is an open-class ROOT (head==0) whose surface stem-matches the hint (≥3
192
+ chars) AND whose Case is NOT Nominative; and there is a recoverable nominal ARGUMENT (a stray-root
193
+ nominal that becomes the object, or a Nominative nominal that becomes the subject). The non-Nominative
194
+ guard is the mis-tag fingerprint — the parser reads the verb's ending as a case ending and assigns an
195
+ oblique case (Par/Gen/...), whereas a genuine noun/title fragment head (`Striima peli.`) is
196
+ Nominative. The argument requirement rejects argumentless oblique fragments (`Striimissä tänään.`,
197
+ whose only companion is an adverb) that the non-Nominative guard alone would not. Tense defaults to
198
+ present (the loan-verb base case); a genuinely past surface is normally tagged correctly by the parser
199
+ and does not reach this path."""
200
+ if not lemma or _main_verb(sent)[0] is not None:
201
+ return False
202
+ if len([t for t in sent if t["upos"] != "PUNCT"]) < 2: # bare fragment -> stays out of scope
203
+ return False
204
+ roots = [t for t in sent if t["head"] == 0]
205
+ root = next((t for t in roots
206
+ if t["upos"] in ("NOUN", "ADJ", "PROPN")
207
+ and t["feats"].get("Case") != "Nom"
208
+ and _stem_overlap(t["form"], lemma) >= 3), None)
209
+ if root is None:
210
+ return False
211
+ rid = root["id"]
212
+ # A real clause has at least one nominal ARGUMENT: another (stray-root) nominal that becomes the
213
+ # object, or a Nominative nominal that can be the subject. The non-Nominative root guard alone does
214
+ # not prove clausehood — an argumentless oblique fragment whose only companion is an adverb
215
+ # (`Striimissä tänään.`) has neither, so it must stay out of scope.
216
+ stray_args = [t for t in roots if t is not root and t["upos"] in _NOMINAL]
217
+ subj = next((t for t in sent if t["deprel"] in ("nsubj", "nsubj:cop")), None)
218
+ if subj is None: # a Nominative nominal dependent of the root
219
+ subj = next((t for t in sent if t["head"] == rid and t is not root
220
+ and t["upos"] in _NOMINAL and t["feats"].get("Case") == "Nom"), None)
221
+ if not stray_args and subj is None:
222
+ return False
223
+ root["upos"], root["lemma"], root["deprel"] = "VERB", lemma, "root"
224
+ for t in roots: # collapse stray roots into verb arguments
225
+ if t is root:
226
+ continue
227
+ t["head"] = rid
228
+ t["deprel"] = "obj" if t["feats"].get("Case") in _OBJ_CASES else "obl"
229
+ if subj is not None: # promote the recovered subject in place
230
+ subj["deprel"] = "nsubj"
231
+ root["feats"] = {"VerbForm": "Fin", "Voice": "Act", "Mood": "Ind", "Tense": "Pres",
232
+ "Person": subj["feats"].get("Person", "3") if subj else "3",
233
+ "Number": subj["feats"].get("Number", "Sing") if subj else "Sing"}
234
+ return True
235
+
236
+
237
+ def verb_cluster(lemma, voice, mood, tense, pol, person, number):
238
+ """Surface token(s) for the finite verb under the given features, or None.
239
+
240
+ PERFECT (tense=Prf) is ANALYTIC: olla-auxiliary + past participle. The participles
241
+ (`ostanut`/`ostettu`) and aux (`ole`/`on`) are tokens the model meets in other ops,
242
+ so an unseen perfect *composite* is a recombination of seen surface forms — unlike
243
+ the fused synthetic cells (`ostettaisi`) that collapse features into one novel form.
244
+ """
245
+ if not _NETLEX_BACKEND:
246
+ return None
247
+ return _netlex_verb_cluster(lemma, voice, mood, tense, pol, person, number)
248
+
249
+
250
+ # --- Giella-free OOV verb realization: the OOV verb's finite cells come from finiteverb.pt and its
251
+ # past participles from verbform.pt; the closed-class scaffolding (olla auxiliary, `ei`/`älä`
252
+ # negation particles) is served from the hardcoded tables since those words are ALWAYS known. The OOV
253
+ # lemma's own forms come from the neural nets.
254
+ # Set at inference by `pipeline.Transformer`.
255
+ _FINITEVERB_BUNDLE = None # juurio.finiteverb.load() bundle (None => OOV verbs abstain)
256
+
257
+
258
+ def _oov_finite(lemma, voice, mood, tense, person):
259
+ """One finite OOV verb form via finiteverb.pt, or None (no bundle / empty / stem-inconsistent).
260
+ `tense` is honoured only for the indicative (Cond/Pot/Imprt are tenseless), matching finiteverb's
261
+ cell space."""
262
+ if _FINITEVERB_BUNDLE is None:
263
+ return None
264
+ from . import finiteverb
265
+ cell = {"Voice": voice, "Mood": mood,
266
+ "Tense": (tense if mood == "Ind" else None), "Person": person}
267
+ out = finiteverb.decode_one(_FINITEVERB_BUNDLE, lemma, cell)
268
+ return out if (out and _stem_consistent(lemma, out)) else None
269
+
270
+
271
+ def _neural_imprt(lemma, tag):
272
+ """Imperative OOV form (Mood=Imprt) via finiteverb.pt, or None. `tag` is a person slot ('Sg2',
273
+ 'Pl2', …) for the affirmative or 'ConNeg' for the -kO connegative (`älkää …kö`). Records the
274
+ realized form in `_NEURAL_OOV` so the transform trace surfaces it — the same neural-realization
275
+ audit invariant `_realize_form` upholds (pipeline surfaces it as
276
+ `trace["neural_unverified"]`, with deprecated alias `trace["neural_oov"]`)."""
277
+ v = _oov_finite(lemma, "Act", "Imprt", None, tag)
278
+ if v:
279
+ _NEURAL_OOV.append(f"{lemma}->{v} (verb)")
280
+ return v
281
+
282
+
283
+ # --- net+lexicon KNOWN-word generation: serving of in-vocabulary words --------
284
+ # The dispatch is lexicon-first (`_GENLEX`, byte-exact on the cells the nets get wrong) → neural net →
285
+ # abstain(None). Closed-class suppletive words (olla auxiliary, the `ei` negative series) cannot be in
286
+ # the lexicon (it enumerates only open-class subsystems) and the nets aren't trained on them, so they
287
+ # are served from these hardcoded standard-Finnish tables.
288
+ _OLLA = { # olla auxiliary / olla-as-main-verb, keyed (Voice, Mood, Tense|None, Person|Sg/Pl|None)
289
+ ("Act", "Ind", "Prs", "Sg1"): "olen", ("Act", "Ind", "Prt", "Sg1"): "olin",
290
+ ("Act", "Cond", None, "Sg1"): "olisin", ("Act", "Pot", None, "Sg1"): "lienen",
291
+ ("Act", "Ind", "Prs", "Sg2"): "olet", ("Act", "Ind", "Prt", "Sg2"): "olit",
292
+ ("Act", "Cond", None, "Sg2"): "olisit", ("Act", "Pot", None, "Sg2"): "lienet",
293
+ ("Act", "Ind", "Prs", "Sg3"): "on", ("Act", "Ind", "Prt", "Sg3"): "oli",
294
+ ("Act", "Cond", None, "Sg3"): "olisi", ("Act", "Pot", None, "Sg3"): "lienee",
295
+ ("Act", "Ind", "Prs", "Pl1"): "olemme", ("Act", "Ind", "Prt", "Pl1"): "olimme",
296
+ ("Act", "Cond", None, "Pl1"): "olisimme", ("Act", "Pot", None, "Pl1"): "lienemme",
297
+ ("Act", "Ind", "Prs", "Pl2"): "olette", ("Act", "Ind", "Prt", "Pl2"): "olitte",
298
+ ("Act", "Cond", None, "Pl2"): "olisitte", ("Act", "Pot", None, "Pl2"): "lienette",
299
+ ("Act", "Ind", "Prs", "Pl3"): "ovat", ("Act", "Ind", "Prt", "Pl3"): "olivat",
300
+ ("Act", "Cond", None, "Pl3"): "olisivat", ("Act", "Pot", None, "Pl3"): "lienevät",
301
+ ("Act", "PrfPrc", None, "Sg"): "ollut", ("Act", "PrfPrc", None, "Pl"): "olleet",
302
+ ("Act", "Cond", "ConNeg", None): "olisi", ("Act", "Pot", "ConNeg", None): "liene",
303
+ }
304
+ _EI = {("Sing", "1"): "en", ("Sing", "2"): "et", ("Sing", "3"): "ei",
305
+ ("Plur", "1"): "emme", ("Plur", "2"): "ette", ("Plur", "3"): "eivät"}
306
+
307
+
308
+ def _netlex_ei(number, person):
309
+ """Negative auxiliary (`en/et/ei/emme/ette/eivät`) — closed-class table, FST-identical."""
310
+ return _EI.get((number, person), "ei")
311
+
312
+
313
+ def _netlex_npart(lemma, voice, number):
314
+ """Past participle (Act `nut`/Pss `tu`): lexicon-first then the verbform net, or None. The serving
315
+ analogue of `_npart` WITHOUT the OOV-safety abstain (we want the net's known-word form)."""
316
+ if lemma == "olla" and voice == "Act": # suppletive olla participle from the closed-class
317
+ return _OLLA.get(("Act", "PrfPrc", None, "Pl" if number == "Plur" else "Sg")) # ollut/olleet
318
+ form = "nut" if voice == "Act" else "tu"
319
+ num = "Plur" if (voice == "Act" and number == "Plur") else "Sing"
320
+ if _GENLEX is not None:
321
+ hit = _GENLEX.lookup_nonfin(lemma, form, "Nom", num)
322
+ if hit is not None:
323
+ return hit
324
+ if _VERBFORM_REALIZER is not None:
325
+ out, _conf = _VERBFORM_REALIZER(lemma, form, "Nom", num)
326
+ if out:
327
+ return out
328
+ return None
329
+
330
+
331
+ def _netlex_finite(lemma, voice, mood, tense, person):
332
+ """One finite form, Giella-free: olla via the closed-class table; every other lemma lexicon-first
333
+ then the finiteverb net. `person` is the finiteverb Person slot ('Sg3'/'Pe4'/'ConNeg'…). No
334
+ stem-consistency abstain (unlike `_oov_finite`) — for known words we serve the net's form."""
335
+ if lemma == "olla" and voice == "Act":
336
+ f = _OLLA.get(("Act", mood, (tense if mood == "Ind" else None), person))
337
+ if f is not None:
338
+ return f
339
+ cell = {"Voice": voice, "Mood": mood,
340
+ "Tense": (tense if mood == "Ind" else None), "Person": person}
341
+ if _GENLEX is not None: # lexicon FIRST — covers cells the net can't, incl.
342
+ hit = _GENLEX.lookup_finite(lemma, cell) # passive connegative (distilled from the FST)
343
+ if hit is not None:
344
+ return hit
345
+ # finiteverb has NO passive connegative cell (passive is only Pe4 — finiteverb.CELLS), so a
346
+ # Pss+ConNeg request the lexicon did NOT cover is out-of-distribution for the net; abstain (None)
347
+ # rather than emit garbage. Must come AFTER the lexicon lookup or shipped forms are dead. The
348
+ # passive PAST negative is unaffected (it is the tu-participle via `_netlex_npart`).
349
+ if voice == "Pss" and person == "ConNeg":
350
+ return None
351
+ if _FINITEVERB_BUNDLE is None:
352
+ return None
353
+ from . import finiteverb
354
+ return finiteverb.decode_one(_FINITEVERB_BUNDLE, lemma, cell)
355
+
356
+
357
+ def _netlex_perfect_aux(voice, mood, tense, pol, person, number):
358
+ """olla auxiliary for the perfect family — the closed-class `_OLLA`/`_EI` analogue of `_perfect_aux`
359
+ (byte-identical to the FST). The lexical participle is supplied separately by `_netlex_npart`."""
360
+ num = NUMTAG.get(number, "Sg")
361
+ pers = f"{num}{person}" if voice == "Act" else "Sg3" # passive auxiliary is impersonal
362
+ if pol == "Aff":
363
+ if tense == "Pqp":
364
+ aux = _OLLA.get(("Act", "Ind", "Prt", pers))
365
+ elif mood == "Cond":
366
+ aux = _OLLA.get(("Act", "Cond", None, pers))
367
+ elif mood == "Pot":
368
+ aux = _OLLA.get(("Act", "Pot", None, pers))
369
+ else:
370
+ aux = _OLLA.get(("Act", "Ind", "Prs", pers))
371
+ return [aux] if aux else None
372
+ ei = "ei" if voice == "Pss" else _netlex_ei(number, person)
373
+ if tense == "Pqp":
374
+ pl = "Pl" if (voice == "Act" and number == "Plur") else "Sg"
375
+ neg = _OLLA.get(("Act", "PrfPrc", None, pl))
376
+ elif mood == "Cond":
377
+ neg = _OLLA.get(("Act", "Cond", "ConNeg", None))
378
+ elif mood == "Pot":
379
+ neg = _OLLA.get(("Act", "Pot", "ConNeg", None))
380
+ else:
381
+ neg = "ole"
382
+ return [ei, neg] if (ei and neg) else None
383
+
384
+
385
+ def _netlex_verb_cluster(lemma, voice, mood, tense, pol, person, number):
386
+ """Giella-free finite verb cluster — structurally identical to `verb_cluster`, but every FST call is
387
+ a net+lexicon (`_netlex_finite`/`_netlex_npart`) or closed-class-table (`_netlex_perfect_aux`/`_EI`)
388
+ lookup. Returns the token list or None. Served when `_NETLEX_BACKEND` is on."""
389
+ num = NUMTAG.get(number, "Sg")
390
+ if tense in ("Prf", "Pqp"):
391
+ part = _netlex_npart(lemma, voice, number)
392
+ if not part:
393
+ return None
394
+ aux = _netlex_perfect_aux(voice, mood, tense, pol, person, number)
395
+ return aux + [part] if aux else None
396
+ if voice == "Act":
397
+ if pol == "Aff":
398
+ v = _netlex_finite(lemma, "Act", mood, tense, f"{num}{person}")
399
+ return [v] if v else None
400
+ ei = _netlex_ei(number, person)
401
+ nv = (_netlex_finite(lemma, "Act", "Cond", None, "ConNeg") if mood == "Cond"
402
+ else _netlex_finite(lemma, "Act", "Pot", None, "ConNeg") if mood == "Pot"
403
+ else _netlex_npart(lemma, "Act", number) if tense == "Prt"
404
+ else _netlex_finite(lemma, "Act", "Ind", "Prs", "ConNeg"))
405
+ return [ei, nv] if (ei and nv) else None
406
+ # passive (impersonal)
407
+ if pol == "Aff":
408
+ v = _netlex_finite(lemma, "Pss", mood, tense, "Pe4")
409
+ return [v] if v else None
410
+ nv = (_netlex_finite(lemma, "Pss", "Cond", None, "ConNeg") if mood == "Cond"
411
+ else _netlex_finite(lemma, "Pss", "Pot", None, "ConNeg") if mood == "Pot"
412
+ else _netlex_npart(lemma, "Pss", number) if tense == "Prt"
413
+ else _netlex_finite(lemma, "Pss", "Ind", "Prs", "ConNeg"))
414
+ return ["ei", nv] if nv else None
415
+
416
+
417
+ # Possessive suffixes that attach to the plural-genitive stem regularly (no vowel harmony): the
418
+ # genitive's final -n drops, the suffix appends. 1st/2nd person only — 3rd (-nsa/-nsä) is harmony-
419
+ # dependent and the Giella FST shows non-standard harmony for some words, so it is NOT derived here.
420
+ _GENPL_POSS_SUFFIX = {"1Sg": "ni", "2Sg": "si", "1Pl": "mme", "2Pl": "nne"}
421
+
422
+
423
+ def _derive_genpl_possessive(lemma, upos, feats, surface):
424
+ """Modern plural-genitive + (1st/2nd-person) possessive, derived from the engine's OWN Px=None
425
+ plural genitive — which is already the modern '-jen/-ten' base (talojen, lasten). Drop its final
426
+ -n, append the possessive suffix: talojen -> talojemme. Returns None if not applicable / the base
427
+ can't be realized / doesn't end in -n. Keeps the engine inside the FST's accepted set while
428
+ preferring the modern variant (vs the neural fallback's archaic '-in-' form, taloimme).
429
+
430
+ The base is realized lowercase (surface=None); the caller applies any leading capital once. If the
431
+ base itself came from the NEURAL realizer (an unverified guess, e.g. lasten/vastausten), the OOV
432
+ trace is re-recorded for the DERIVED form so its unverified provenance is preserved."""
433
+ suf = _GENPL_POSS_SUFFIX.get(feats.get("Px", "None"))
434
+ if suf is None or feats.get("Case") != "Gen" or feats.get("Number") != "Plur":
435
+ return None
436
+ noov = list(_NEURAL_OOV)
437
+ snap = (list(_OOV_UNVERIFIED), list(_OOV_UNREALIZABLE))
438
+ base = _netlex_nominal(lemma, upos, {k: v for k, v in feats.items() if k != "Px"}, None)
439
+ base_was_neural = len(_NEURAL_OOV) > len(noov)
440
+ # restore the OOV trace to its pre-call state; provenance is re-recorded for the DERIVED form only
441
+ _NEURAL_OOV[:], _OOV_UNVERIFIED[:], _OOV_UNREALIZABLE[:] = noov, snap[0], snap[1]
442
+ if not base or not base.endswith("n"):
443
+ return None
444
+ derived = base[:-1] + suf
445
+ if base_was_neural: # derived from an unverified neural base -> keep it auditable
446
+ _NEURAL_OOV.append(f"{lemma}->{derived} ({upos})")
447
+ return derived
448
+
449
+
450
+ # Partitive-plural endings, longest-first, so the comitative derive strips the whole ending before
451
+ # appending -ine. The comitative stem IS the partitive-plural stem (talo: taloja -> taloine; aarre:
452
+ # aarteita -> aarteine), so recovering it from the engine's own partitive plural inherits the correct
453
+ # stem AND gradation grade — which the neural realizer mis-guesses for e-/s-/alternation nominals.
454
+ _PARTITIVE_PLURAL_ENDINGS = ("ita", "itä", "ja", "jä", "ia", "iä", "ta", "tä", "a", "ä")
455
+
456
+
457
+ def _comitative_stem_from_partitive_plural(parpl):
458
+ """Strip the partitive-plural ending to recover the plural stem, then append the comitative -ine.
459
+ None if `parpl` is falsy or carries no known partitive ending."""
460
+ if not parpl:
461
+ return None
462
+ for suf in _PARTITIVE_PLURAL_ENDINGS:
463
+ if parpl.endswith(suf):
464
+ return parpl[:-len(suf)] + "ine"
465
+ return None
466
+
467
+
468
+ def _is_geminate_strong(strong, weak):
469
+ """True iff `strong` is `weak` with exactly one extra, doubled MEDIAL k/p/t — a weak->strong consonant-
470
+ gradation upgrade (laatikkoine vs laatikoine). Recognises the kpt-geminate comitative the partitive-
471
+ plural derive under-grades (its base is weak-grade, the comitative keeps the strong grade). Restricted
472
+ to the kk/pp/tt gradation pattern (the only geminates Finnish gradation produces) and to a medial
473
+ position (i>0): this predicate is the SOLE guard on adopting the net form, so it must match exactly the
474
+ linguistic invariant — b/d/g gradation is qualitative (single-consonant), never geminate, and an
475
+ initial doubling is impossible. The doubled consonant lands in the stem, before the -ine- suffix."""
476
+ if not strong or not weak or len(strong) != len(weak) + 1:
477
+ return False
478
+ for i in range(1, len(strong) - 1):
479
+ if strong[i] == strong[i + 1] and strong[i] in "kpt" and strong[:i] + strong[i + 1:] == weak:
480
+ return True
481
+ return False
482
+
483
+
484
+ def _derive_comitative(lemma, upos, feats, surface):
485
+ """Px=None comitative, derived from the engine's OWN partitive plural (comitative stem = plural stem
486
+ + 'ine'), which carries the correct stem and gradation grade — vs the neural realizer's mis-stemmed
487
+ guess (aarre->aarrehine for aarteine, valkokangas->valkokangaksine for valkokankaine). For kpt-
488
+ GEMINATE stems the partitive plural is weak-grade but the comitative is strong (laatikko: laatikoita
489
+ -> the derive gives 'laatikoine', but FST wants strong-grade 'laatikkoine'), so when the net supplies
490
+ exactly the doubled-consonant (strong) version of the derive, trust the net. Returns None (caller
491
+ falls back to the net) when not a Px=None comitative, or the partitive plural can't be realized /
492
+ lacks a known ending. This derive is markedly more FST-valid than the net's direct comitative.
493
+ The rare residual cases are compound lemmas whose partitive plural inflects BOTH parts (omatunto ->
494
+ omiatuntoine). Realized lowercase (surface=None); the caller applies any leading capital once."""
495
+ if upos != "NOUN" or feats.get("Case") != "Com" or feats.get("Px", "None") != "None":
496
+ return None
497
+ noov = list(_NEURAL_OOV)
498
+ snap = (list(_OOV_UNVERIFIED), list(_OOV_UNREALIZABLE))
499
+ parpl = _netlex_nominal(lemma, upos, {"Case": "Par", "Number": "Plur", "Px": "None"}, None)
500
+ base_was_neural = len(_NEURAL_OOV) > len(noov)
501
+ _NEURAL_OOV[:], _OOV_UNVERIFIED[:], _OOV_UNREALIZABLE[:] = noov, snap[0], snap[1]
502
+ derived = _comitative_stem_from_partitive_plural(parpl)
503
+ if derived is None:
504
+ return None
505
+ # kpt-geminate override: the comitative keeps the strong grade the partitive plural drops; the net
506
+ # gets that right (laatikkoine), so adopt the net form when it is exactly the geminate-strong version
507
+ # of the derive. The probe is trace-neutral (snapshot/restore around the realizer call).
508
+ adopted_net = False
509
+ if _NOUN_REALIZER is not None:
510
+ nf, _conf = _NOUN_REALIZER(lemma, upos, {**feats, "Number": "Sing"}, None)
511
+ _NEURAL_OOV[:], _OOV_UNVERIFIED[:], _OOV_UNREALIZABLE[:] = noov, snap[0], snap[1]
512
+ if nf and _is_geminate_strong(nf, derived):
513
+ derived, adopted_net = nf, True
514
+ if base_was_neural or adopted_net: # derived/adopted from an unverified neural form -> keep it auditable
515
+ _NEURAL_OOV.append(f"{lemma}->{derived} ({upos})")
516
+ return derived
517
+
518
+
519
+ # Opening-diphthong monosyllabic NOUN heads (full words, so suffix matching can't false-fire on a random
520
+ # -ie/-uo/-yö loan). Longest-distinguishing first: `työ` before `yö` so `käsityö` splits at `työ`.
521
+ _DIPHTHONG_HEADS = ("työ", "tie", "vyö", "suo", "vuo", "yö")
522
+
523
+
524
+ def _derive_diphthong_compound(lemma, upos, feats, surface):
525
+ """A NOUN compound whose HEAD is an opening-diphthong monosyllable (juhannus·yö, maan·tie, käsi·työ)
526
+ realizes as the invariant modifier prefix + the head's OWN realization. The net mis-stems some of these
527
+ (`juhannusyö`->`juhannusyyksia`, a silent wrong form that shares the lemma onset so `_stem_consistent`
528
+ misses it); the head inflects correctly on its own (post the _stem_consistent diphthong fix) and the
529
+ modifier prefix is a fixed string, so prefix+head is correct by construction. Fires only for a true
530
+ COMPOUND (lemma strictly longer than the head); the bare head goes straight to the net. Flags the
531
+ result neural-unverified iff the head realization was. Returns None when not applicable."""
532
+ if upos != "NOUN":
533
+ return None
534
+ low = lemma.lower()
535
+ # require a >=2-char modifier prefix so a real compound (käsi·työ, maan·tie) fires but the bare
536
+ # monosyllable does NOT (työ = t+yö would else match head `yö` with a 1-char prefix) — bare työ/vyö
537
+ # stay on their own net path (the _stem_consistent fix), and only genuine compounds are derived.
538
+ head = next((h for h in _DIPHTHONG_HEADS if low.endswith(h) and len(low) >= len(h) + 2), None)
539
+ if head is None:
540
+ return None
541
+ noov, snap = list(_NEURAL_OOV), (list(_OOV_UNVERIFIED), list(_OOV_UNREALIZABLE))
542
+ hf = _netlex_nominal(head, "NOUN", feats, None) # the head's own (correct) realization
543
+ head_was_neural = len(_NEURAL_OOV) > len(noov)
544
+ _NEURAL_OOV[:], _OOV_UNVERIFIED[:], _OOV_UNREALIZABLE[:] = noov, snap[0], snap[1]
545
+ if hf is None:
546
+ return None
547
+ derived = lemma[:-len(head)] + hf # invariant modifier prefix + inflected head
548
+ if head_was_neural:
549
+ _NEURAL_OOV.append(f"{lemma}->{derived} ({upos})")
550
+ return derived
551
+
552
+
553
+ def _netlex_nominal(lemma, upos, feats, surface=None):
554
+ """KNOWN-word nominal, Giella-free: lexicon-first then the nominal net, or None. PROPN is not shipped
555
+ in the lexicon (open class) so it is served by the net only. Preserves a leading capital from
556
+ `surface` to match `realize_strict`/the realizer."""
557
+ if _GENLEX is not None:
558
+ hit = _GENLEX.lookup_nominal(upos, lemma, feats) # None for PROPN by construction
559
+ if hit is not None:
560
+ if surface and surface[:1].isupper():
561
+ hit = hit[:1].upper() + hit[1:]
562
+ return hit # trusted lexicon (byte-exact) — no validation
563
+ # lexicon miss for a plural-genitive 1st/2nd-person possessive -> derive the MODERN form from the
564
+ # Px=None plural genitive (which the lexicon/realizer gives in the modern -jen/-ten stem), instead
565
+ # of the neural realizer's archaic -in- variant. Stays FST-valid (parity floor); see test suite.
566
+ # ...and a Px=None comitative -> derive from the engine's own partitive plural (correct stem +
567
+ # gradation) rather than the net's mis-stemmed guess. Both derives are mutually exclusive on feats
568
+ # (genpl needs Gen/Plur/1st-2nd-Px; comitative needs Com/Px=None) so the order is immaterial.
569
+ if upos == "NOUN":
570
+ # diphthong-compound first: it routes the head through this same function (handling the head's own
571
+ # genpl/comitative/net), so it correctly composes for every case of these compounds.
572
+ derived = _derive_diphthong_compound(lemma, upos, feats, surface)
573
+ if derived is None:
574
+ derived = _derive_genpl_possessive(lemma, upos, feats, surface)
575
+ if derived is None:
576
+ derived = _derive_comitative(lemma, upos, feats, surface)
577
+ if derived is not None:
578
+ if surface and surface[:1].isupper():
579
+ derived = derived[:1].upper() + derived[1:]
580
+ return derived
581
+ # lexicon miss -> neural realizer, but FAIL CLOSED on an untrusted guess: an OOV open-class
582
+ # word the shipped lexicon does not cover must abstain (None), never emit `<unk>`/truncated garbage or
583
+ # the bare lemma. Mirrors the FST path's abstain so the flip changes coverage, not output safety.
584
+ if _NOUN_REALIZER is not None:
585
+ nf, conf = _NOUN_REALIZER(lemma, upos, feats, surface)
586
+ if nf and _neural_oov_ok(lemma, nf, conf, conf_min=0.0): # net serves KNOWN words here
587
+ _NEURAL_OOV.append(f"{lemma}->{nf} ({upos})") # validated neural form (auditable)
588
+ return nf
589
+ if nf: # produced but failed validation -> suppress
590
+ _OOV_UNVERIFIED.append(f"{lemma}->{nf} ({upos}) conf={conf:.2f}")
591
+ return None
592
+ _OOV_UNREALIZABLE.append(f"{lemma} ({upos}) {feats.get('Case', 'Nom')}/{feats.get('Number', 'Sing')}")
593
+ return None
594
+
595
+
596
+ # Neural OOV noun/adj realizer: fn(lemma, upos, feats, surface) -> form | None.
597
+ # Set at inference so the noun/case/number ops fall back to the model when the FST cannot inflect a
598
+ # loan/OOV word, instead of silently no-op'ing to the lemma.
599
+ _NOUN_REALIZER = None
600
+ # OOV non-finite verb-form realizer: fn(lemma, form, case, number) -> (form, conf) | (None,0).
601
+ # Set at inference (like _NOUN_REALIZER) so realize_verbform can inflect a loan verb's participle/
602
+ # infinitive the FST cannot. Absent => OOV verb-forms abstain (oov_unrealizable).
603
+ _VERBFORM_REALIZER = None
604
+
605
+ # --- net+lexicon generation backend --------------------------------------------------------
606
+ # KNOWN-word inflection is served Giella-free by lexicon-first + neural-net-second (the same
607
+ # model+lexicon shape the analysis side uses). This is the only generation backend; the flag is the
608
+ # internal serving toggle set by `pipeline.Transformer` (and by tests that exercise the path directly).
609
+ _NETLEX_BACKEND = True
610
+ _GENLEX = None # genlex.GenerationLexicon | None (the static exception table; loaded at serving)
611
+
612
+
613
+ _OPEN_CLASS = ("NOUN", "ADJ", "PROPN") # POS that giella_query inflects + where OOV is the risk
614
+
615
+ # Words realized by a NEURAL fallback in the current transform (verb cluster or noun/adj). The FST
616
+ # cannot validate these (the whole point is they are OOV), so they are model GUESSES — recorded
617
+ # here and surfaced in the pipeline Result so an OOV-realized output is never silent/unaudited.
618
+ _NEURAL_OOV: list = []
619
+ # Open-class words an op needed to inflect but NEITHER the FST nor the neural realizer could —
620
+ # the op abstains and the pipeline reports `oov_unrealizable` (explicit "unsupported morphology",
621
+ # distinct from a structurally inapplicable op).
622
+ _OOV_UNREALIZABLE: list = []
623
+ # Open-class words where the neural realizer DID produce a candidate but it failed validation
624
+ # (stem-inconsistent, or below the near-tie confidence floor) — suppressed, never emitted. Distinct
625
+ # reason `oov_unverified`: "we had a guess and didn't trust it" vs "nothing could realize it".
626
+ _OOV_UNVERIFIED: list = []
627
+ # Near-tie confidence floor for a neural OOV form. NOTE: the top1-top2 margin is a WEAK error
628
+ # detector here — the model's rare misses are confidently wrong, so this only rejects genuine
629
+ # near-ties at ~zero cost to correct forms. The real safety is FST-first + `_stem_consistent` +
630
+ # mandatory unverified-flagging, not this threshold.
631
+ _OOV_CONF_MIN = 0.08
632
+
633
+
634
+ # Opening-diphthong monosyllables (tie/työ/yö/suo/vuo/vyö …) mutate the stem diphthong in the plural:
635
+ # ie->ei, uo->oi, yö->öi (tie->tei-, työ->töi-, yö->öi-, suo->soi-, vyö->vöi-). That changes the word's
636
+ # first two chars, so the bare first-2-char check below wrongly rejects their (correct) plural forms. Map
637
+ # each lemma-final diphthong to its plural stem nucleus (already including the plural -i-) so the documented
638
+ # mutation is allowed — and ONLY that: the output must begin with onset + nucleus, so garbage is rejected.
639
+ _OPENING_DIPHTHONG = {"ie": "ei", "uo": "oi", "yö": "öi"}
640
+
641
+
642
+ def _stem_consistent(src, out):
643
+ """Cheap validation of a neural OOV realization: Finnish word-initial never gradates, so a
644
+ genuine inflection of `src` shares its first 2 chars — EXCEPT the opening-diphthong monosyllables,
645
+ whose plural stem mutates the diphthong (työ->töitä). Rejects catastrophic model garbage."""
646
+ a, b = (src or "").lower(), (out or "").lower()
647
+ if not b:
648
+ return False
649
+ if a[:2] == b[:2]:
650
+ return True
651
+ for diph, nucleus in _OPENING_DIPHTHONG.items():
652
+ if a.endswith(diph) and b.startswith(a[:-2] + nucleus):
653
+ return True # onset + mutated plural nucleus (töi-/tei-/öi-/soi-/vöi-)
654
+ return False
655
+
656
+
657
+ def _neural_oov_ok(lemma, nf, conf, conf_min=_OOV_CONF_MIN):
658
+ """SHAPE-filter a NEURAL open-class realization before serving — fail closed on structurally-impossible
659
+ output, but do NOT gate stem-correctness here. A served net form is flagged `neural_unverified` (the
660
+ honest tiered claim); FST-parity coverage is expanded OFFLINE via the lexicon overrides, not by a
661
+ runtime heuristic.
662
+
663
+ The former char-prefix stem check (`_stem_consistent`, first-2-char) was REMOVED here: measured against
664
+ the FST it was anti-correlated with correctness — it caught none of the net's real errors (a wrong
665
+ inflection keeps the onset) while suppressing correct consonant-gradation forms (`aika`->`ajan`,
666
+ `apu`->`avun`), and the net produced no onset-drift garbage. So the onset heuristic only cost
667
+ correctness; it bought no safety. (It still guards the FST-FALLBACK path's true-OOV net at
668
+ `_realize_form`, where no FST gold exists to expand coverage offline.) Rejects only:
669
+ (a) the OOV-char marker `<unk>` (unseen compound seam / out-of-vocab char);
670
+ (b) truncation — a valid open-class inflection is never STRICTLY shorter than the citation lemma
671
+ (catches `suvaitsevaisuus`->`suvaitsevaisuu`);
672
+ (c) empty;
673
+ (d) a near-tie below `conf_min` (the netlex path passes 0; the FST-fallback path passes the OOV floor).
674
+ Returns True for a candidate safe to EMIT (flagged), not a guarantee of FST-correctness."""
675
+ if not nf or "<unk>" in nf:
676
+ return False
677
+ if len(nf) < len(lemma):
678
+ return False
679
+ return conf >= conf_min
680
+
681
+
682
+ def _realize_form(lemma, upos, feats, surface=None):
683
+ """One word-form: lexicon-first, neural OOV realizer second. The silent-fallback blind spot is only
684
+ OPEN-class (NOUN/ADJ/PROPN) words the lexicon can't inflect — for those a hard miss returns None so
685
+ case/number ops ABSTAIN instead of emitting a half-agreed NP. NUM/PRON are served lexicon-first (the
686
+ genlex numeral/pronoun overrides) and fall back to the lemma on a miss; the remaining closed classes
687
+ (DET, …) have a Nom form reliably equal to the lemma, so they keep the lemma fallback."""
688
+ if not _NETLEX_BACKEND:
689
+ return None
690
+ # Numerals + pronouns: closed classes served lexicon-first (NUM/PRON alias NOUN in the genlex). The
691
+ # nominal net mis-stems them (kaksi+Gen -> kaksin not kahden; minä+Gen -> minän not minun), so on a
692
+ # lexicon MISS fall back to the lemma (safe Nom, like the other closed classes below), NEVER the net.
693
+ if upos in ("NUM", "PRON"):
694
+ hit = _GENLEX.lookup_nominal(upos, lemma, feats) if _GENLEX is not None else None
695
+ if hit is not None and surface and surface[:1].isupper():
696
+ hit = hit[:1].upper() + hit[1:]
697
+ return hit if hit is not None else lemma
698
+ # Lexicon-first then the nominal net for open-class; other closed-class -> lemma.
699
+ # Abstain (None) on open-class miss.
700
+ if upos not in _OPEN_CLASS:
701
+ return lemma
702
+ return _netlex_nominal(lemma, upos, feats, surface)
703
+
704
+
705
+ def _flip_case(sent, obj, newcase):
706
+ """obj noun + agreeing modifiers -> newcase; {id: form} or None if any fails."""
707
+ out = {}
708
+ for w in [obj] + [c for c in sent if c["head"] == obj["id"] and c["deprel"] in AGREE
709
+ and c["feats"].get("Case") == obj["feats"].get("Case")]:
710
+ form = _realize_form(w["lemma"], w["upos"], {**w["feats"], "Case": newcase}, w["form"])
711
+ if not form: # FST miss + no neural fallback -> abstain (never lemma)
712
+ return None
713
+ out[w["id"]] = form
714
+ return out
715
+
716
+
717
+ # --- hybrid passive: deterministic structure + neural OOV verb realization ------
718
+ # The hard part of passive is non-local STRUCTURE (drop the subject subtree, nominativize the
719
+ # total object) — that belongs to the parse/rules (`apply_clause`), not the char model. The lexicon
720
+ # inflects known verbs; the verb cluster (`_netlex_verb_cluster`, finiteverb + verbform) realizes the
721
+ # OOV/loan verb forms the lexicon can't. The deterministic structure is the same either way.
722
+ _FORCE_NEURAL_VC = False # eval-only: pretend KNOWN-word lookup is empty for EVERY verb (simulate OOV)
723
+
724
+
725
+ def _features_to_op(voice, mood, tense, pol):
726
+ """Map a final feature-set to the trained op-name (canonical order neg+passive+tense/mood)."""
727
+ parts = []
728
+ if pol == "Neg":
729
+ parts.append("neg")
730
+ if voice == "Pss":
731
+ parts.append("passive")
732
+ if tense == "Prt":
733
+ parts.append("past")
734
+ elif tense == "Prf":
735
+ parts.append("perfect")
736
+ elif mood == "Cond":
737
+ parts.append("cond")
738
+ return "+".join(parts) or None
739
+
740
+
741
+ def apply_clause(sent, deltas, nonce_rng=None):
742
+ """Apply verb-feature deltas to the main clause -> (src_words, tgt_words) or None."""
743
+ c = clause(sent)
744
+ if c is None:
745
+ return None
746
+ verb, is_cop, base = c
747
+ final = {**base, **deltas}
748
+ if final == base or (final["voice"] == "Pss" and is_cop):
749
+ return None
750
+ vc = None if _FORCE_NEURAL_VC else verb_cluster(
751
+ verb["lemma"], final["voice"], final["mood"], final["tense"],
752
+ final["pol"], base["person"], base["number"])
753
+ if vc is None:
754
+ return None
755
+
756
+ drop, repl = set(), {}
757
+ going_passive = final["voice"] == "Pss" and base["voice"] == "Act"
758
+ if going_passive and not is_cop:
759
+ # demote any OVERT subject subtree. A missing nsubj is a valid pro-drop passive
760
+ # (`Tällaiset aarteet kotiutin` -> `Tällaiset aarteet kotiutettiin`), NOT noise —
761
+ # so we must not abort here (an earlier `return None` wrongly discarded these).
762
+ for t in sent:
763
+ if t["deprel"] in ("nsubj", "nsubj:cop"):
764
+ drop |= _subtree_ids(sent, t["id"])
765
+ if not is_cop:
766
+ for t in sent:
767
+ if t["deprel"] == "obj" and t["feats"].get("Case") in ("Gen", "Nom") \
768
+ and t["upos"] in ("NOUN", "PROPN", "ADJ", "NUM"):
769
+ newcase = "Par" if final["pol"] == "Neg" else ("Nom" if going_passive else None)
770
+ if newcase and newcase != t["feats"].get("Case"):
771
+ fl = _flip_case(sent, t, newcase)
772
+ if fl is None: # required object re-casing unrealizable (OOV) -> FAIL CLOSED
773
+ return None # abstain, never emit the original wrong-case object
774
+ repl.update(fl)
775
+
776
+ repl_c = None
777
+ if nonce_rng:
778
+ init = verb["form"][:1].lower()
779
+ repl_c = _rand_onset(nonce_rng, init)
780
+ vc = list(vc); vc[-1] = swap_initial(vc[-1], repl_c) # lexical verb (ei is first)
781
+
782
+ src = [swap_initial(t["form"], repl_c) if (repl_c and t["id"] == verb["id"]) else t["form"]
783
+ for t in sent]
784
+ tgt = []
785
+ for t in sent:
786
+ if t["id"] in drop:
787
+ continue
788
+ if t["id"] == verb["id"]:
789
+ tgt.extend(vc)
790
+ else:
791
+ tgt.append(repl.get(t["id"], t["form"]))
792
+ return src, tgt
793
+
794
+
795
+ # ----------------------------- standalone ops -----------------------------
796
+ def op_quest(sent, nonce_rng=None):
797
+ """Polar question: front the finite verb + -ko/-kö clitic (vowel harmony)."""
798
+ verb, _ = _main_verb(sent)
799
+ if verb is None:
800
+ return None
801
+ repl_c = None
802
+ if nonce_rng:
803
+ init = verb["form"][:1].lower()
804
+ repl_c = _rand_onset(nonce_rng, init)
805
+ vform = swap_initial(verb["form"], repl_c) if repl_c else verb["form"]
806
+ q = vform + _clitic(vform)
807
+ tgt = [q]
808
+ rest = [t for t in sent if t["id"] != verb["id"]
809
+ and not (t["upos"] == "PUNCT" and t["form"] in ".!")]
810
+ for i, t in enumerate(rest):
811
+ f = t["form"]
812
+ if i == 0 and t["upos"] != "PROPN" and f[:1].isupper():
813
+ f = f[:1].lower() + f[1:]
814
+ tgt.append(f)
815
+ tgt.append("?")
816
+ src = [swap_initial(t["form"], repl_c) if (repl_c and t["id"] == verb["id"]) else t["form"]
817
+ for t in sent]
818
+ return src, tgt
819
+
820
+
821
+ def _op_degree(degree):
822
+ """degree is the UD value 'Cmp' or 'Sup'. Routes through `_realize_form` (FST first, neural OOV
823
+ realizer second) so comparison works on OOV/loan/nonce adjectives the FST can't inflect, not
824
+ just FST-known ones — same open-vocabulary path as number/case."""
825
+ def fn(sent, nonce_rng=None):
826
+ repl = {}
827
+ for t in sent:
828
+ if t["upos"] == "ADJ" and t["feats"].get("Degree", "Pos") == "Pos":
829
+ c = _realize_form(t["lemma"], "ADJ", {**t["feats"], "Degree": degree}, t["form"])
830
+ if c and c != t["form"]:
831
+ repl[t["id"]] = c
832
+ if not repl:
833
+ return None
834
+ src = [t["form"] for t in sent]
835
+ tgt = [repl.get(t["id"], t["form"]) for t in sent]
836
+ if nonce_rng:
837
+ ids = list(repl)
838
+ tid = nonce_rng.choice(ids)
839
+ i = [t["id"] for t in sent].index(tid)
840
+ rc = _rand_onset(nonce_rng, src[i][:1].lower())
841
+ src[i] = swap_initial(src[i], rc); tgt[i] = swap_initial(tgt[i], rc)
842
+ return src, tgt
843
+ return fn
844
+
845
+
846
+ def _verbop(deltas, req):
847
+ def fn(sent, nonce_rng=None):
848
+ c = clause(sent)
849
+ if c is None:
850
+ return None
851
+ _, is_cop, base = c
852
+ if any(base.get(k) != v for k, v in req.items()):
853
+ return None
854
+ if deltas.get("voice") == "Pss" and is_cop:
855
+ return None
856
+ return apply_clause(sent, deltas, nonce_rng)
857
+ return fn
858
+
859
+
860
+ # verb-feature ops: (deltas, required base state)
861
+ VERB_OPS = {
862
+ "neg": ({"pol": "Neg"}, {}),
863
+ "past": ({"tense": "Prt"}, {"tense": "Prs"}),
864
+ "pres": ({"tense": "Prs"}, {"tense": "Prt"}),
865
+ "cond": ({"mood": "Cond"}, {"mood": "Ind"}),
866
+ "passive": ({"voice": "Pss"}, {"voice": "Act"}),
867
+ "perfect": ({"tense": "Prf"}, {"tense": "Prs", "mood": "Ind"}),
868
+ # potential mood is in the TRAINED set so the neural path covers OOV/loan verbs (`striimannee`)
869
+ # too, not just FST-known ones.
870
+ "potential": ({"mood": "Pot"}, {"mood": "Ind", "tense": "Prs"}),
871
+ }
872
+ # Composite (multi-feature) verb ops; each value is (delta-features, reset-baseline).
873
+ COMPOSITES = {
874
+ "neg+past": ({"pol": "Neg", "tense": "Prt"}, {"tense": "Prs"}),
875
+ "neg+cond": ({"pol": "Neg", "mood": "Cond"}, {"mood": "Ind"}),
876
+ "passive+past": ({"voice": "Pss", "tense": "Prt"}, {"voice": "Act", "tense": "Prs"}),
877
+ "passive+cond": ({"voice": "Pss", "mood": "Cond"}, {"voice": "Act", "mood": "Ind"}),
878
+ "neg+passive": ({"pol": "Neg", "voice": "Pss"}, {"voice": "Act"}),
879
+ # analytic perfect composites (e.g. `ei ole ostanut`)
880
+ "neg+perfect": ({"pol": "Neg", "tense": "Prf"}, {"tense": "Prs", "mood": "Ind"}),
881
+ "passive+perfect": ({"voice": "Pss", "tense": "Prf"}, {"voice": "Act", "tense": "Prs", "mood": "Ind"}),
882
+ "neg+passive+past": ({"pol": "Neg", "voice": "Pss", "tense": "Prt"},
883
+ {"voice": "Act", "tense": "Prs"}),
884
+ "neg+passive+cond": ({"pol": "Neg", "voice": "Pss", "mood": "Cond"},
885
+ {"voice": "Act", "mood": "Ind"}),
886
+ # three-feature analytic composite: `ei ole ostettu`
887
+ "neg+passive+perfect": ({"pol": "Neg", "voice": "Pss", "tense": "Prf"},
888
+ {"voice": "Act", "tense": "Prs", "mood": "Ind"}),
889
+ }
890
+ OPS = {k: _verbop(d, r) for k, (d, r) in {**VERB_OPS, **COMPOSITES}.items()}
891
+ OPS["quest"] = op_quest
892
+ OPS["comp"] = _op_degree("Cmp") # UD Degree values; _op_degree maps to FST/realizer
893
+ OPS["super"] = _op_degree("Sup")
894
+
895
+
896
+ # ===================== extended ops (EXT_OPS) =====================
897
+ # Extra ops that ride the rule engine + FST only (no neural flag model); they work on KNOWN
898
+ # words today:
899
+ # - potential / imperative : new VERB moods; FST handles known verbs.
900
+ # - clitic_* : pure string attach (vowel harmony).
901
+ # - plural / singular : NOUN number + verb re-agreement.
902
+ # Pipeline routes everything through ALL_OPS = {**OPS, **EXT_OPS}.
903
+
904
+ # suppletive personal pronouns (FST cannot derive minä<->me); keyed by (verb person, target number)
905
+ _PRON_BY = {("1", "Sing"): "minä", ("1", "Plur"): "me",
906
+ ("2", "Sing"): "sinä", ("2", "Plur"): "te",
907
+ ("3", "Sing"): "hän", ("3", "Plur"): "he"}
908
+
909
+
910
+ def _flip_number(sent, subj, target):
911
+ """subj noun + agreeing nominative modifiers -> Number=target; {id: form} or None."""
912
+ out = {}
913
+ for w in [subj] + [c for c in sent if c["head"] == subj["id"] and c["deprel"] in AGREE
914
+ and c["feats"].get("Case") == subj["feats"].get("Case")
915
+ and c["feats"].get("Number") == subj["feats"].get("Number")]:
916
+ form = _realize_form(w["lemma"], w["upos"], {**w["feats"], "Number": target}, w["form"])
917
+ if not form or form == w["form"]: # OOV miss, or vacuous change -> abstain
918
+ return None
919
+ out[w["id"]] = form
920
+ return out
921
+
922
+
923
+ def op_imperative(sent, nonce_rng=None):
924
+ """Affirmative 2sg imperative: verb -> Imprt+Sg2, drop the subject, total object -> Nom.
925
+ Accepts a finite Act root in indicative OR imperative mood; which clauses may be commanded
926
+ (2nd-person/subjectless) is enforced by `imperative_in_scope` at the pipeline gate."""
927
+ verb = _imperative_source_verb(sent)
928
+ if verb is None:
929
+ return None
930
+ repl_c = _rand_onset(nonce_rng, verb["form"][:1].lower()) if nonce_rng else None
931
+ v = _netlex_finite(verb["lemma"], "Act", "Imprt", None, "Sg2") if _NETLEX_BACKEND else None
932
+ if not v: # OOV/loan verb: Giella-free finiteverb (Imprt+Sg2)
933
+ v = _neural_imprt(verb["lemma"], "Sg2")
934
+ if not v:
935
+ return None
936
+ if repl_c:
937
+ v = swap_initial(v, repl_c)
938
+ drop = set()
939
+ for t in sent:
940
+ if t["deprel"] in ("nsubj", "nsubj:cop"):
941
+ drop |= _subtree_ids(sent, t["id"])
942
+ repl = {}
943
+ for t in sent:
944
+ if t["deprel"] == "obj" and t["feats"].get("Case") in ("Gen", "Nom") \
945
+ and t["upos"] in ("NOUN", "PROPN", "ADJ", "NUM", "PRON"):
946
+ fl = _flip_case(sent, t, "Nom")
947
+ if fl:
948
+ repl.update(fl)
949
+ src, tgt = [], []
950
+ for t in sent:
951
+ f = swap_initial(t["form"], repl_c) if (repl_c and t["id"] == verb["id"]) else t["form"]
952
+ src.append(f)
953
+ if t["id"] in drop:
954
+ continue
955
+ tgt.append(v if t["id"] == verb["id"] else repl.get(t["id"], t["form"]))
956
+ return src, tgt
957
+
958
+
959
+ # ---- full imperative paradigm (beyond Sg2) + negative imperative -----------------------
960
+ # `imperative` (Sg2 affirmative) stays the TRAINED flag (untouched). The other persons and the
961
+ # negative paradigm ride the FST + rules (known verbs), with OOV only for the Sg2 forms (which can
962
+ # borrow the trained `imperative` flag). The negative imperative is the älä-series particle +
963
+ # connegative imperative (`menkö`); Sg2 negative keeps the plain Sg2 form (`älä mene`).
964
+ _IMPRT_TAG = {"sg2": "Sg2", "sg3": "Sg3", "pl1": "Pl1", "pl2": "Pl2", "pl3": "Pl3"}
965
+ _ALA = {"sg2": "älä", "sg3": "älköön", "pl1": "älkäämme", "pl2": "älkää", "pl3": "älkööt"}
966
+
967
+
968
+ def _imprt_form(lemma, tag):
969
+ """Affirmative imperative surface for a person/number tag (lexicon-first, the standard literary
970
+ form — `mennä` -> 'mene', not the colloquial contraction 'mee'). None on a miss."""
971
+ if not _NETLEX_BACKEND:
972
+ return None
973
+ return _netlex_finite(lemma, "Act", "Imprt", None, tag)
974
+
975
+
976
+ def _op_imperative_person(slot, pol):
977
+ """Imperative for a person/number `slot` (sg2/sg3/pl1/pl2/pl3), affirmative or negative. Same
978
+ clause surgery as `op_imperative` (drop the subject, total object -> Nom). OOV (FST miss) is
979
+ supported only for the Sg2 forms via the trained `imperative` flag; other slots abstain."""
980
+ tag = _IMPRT_TAG[slot]
981
+
982
+ def fn(sent, nonce_rng=None):
983
+ verb = _imperative_source_verb(sent)
984
+ if verb is None:
985
+ return None
986
+ repl_c = _rand_onset(nonce_rng, verb["form"][:1].lower()) if nonce_rng else None
987
+ if pol == "Aff":
988
+ v = _imprt_form(verb["lemma"], tag)
989
+ if not v: # OOV/loan verb: Giella-free finiteverb (Imprt+tag)
990
+ v = _neural_imprt(verb["lemma"], tag)
991
+ if not v:
992
+ if verb["upos"] == "VERB":
993
+ _OOV_UNREALIZABLE.append(f"{verb['lemma']} (VERB) Imprt+{tag}")
994
+ return None
995
+ cluster = [v]
996
+ else: # negative: älä-series + connegative
997
+ if slot == "sg2": # `älä mene` keeps the plain Sg2 form
998
+ conneg = _imprt_form(verb["lemma"], "Sg2")
999
+ if not conneg: # OOV/loan verb: Giella-free finiteverb
1000
+ conneg = _neural_imprt(verb["lemma"], "Sg2")
1001
+ else: # `älkää menkö` uses the -kO connegative
1002
+ conneg = (_netlex_finite(verb["lemma"], "Act", "Imprt", None, "ConNeg")
1003
+ if _NETLEX_BACKEND else None)
1004
+ if not conneg: # OOV/loan verb: Giella-free finiteverb
1005
+ conneg = _neural_imprt(verb["lemma"], "ConNeg")
1006
+ if not conneg:
1007
+ if verb["upos"] == "VERB":
1008
+ _OOV_UNREALIZABLE.append(f"{verb['lemma']} (VERB) neg-Imprt+{tag}")
1009
+ return None
1010
+ cluster = [_ALA[slot], conneg]
1011
+ if repl_c:
1012
+ cluster[-1] = swap_initial(cluster[-1], repl_c)
1013
+ drop = set()
1014
+ for t in sent:
1015
+ if t["deprel"] in ("nsubj", "nsubj:cop"):
1016
+ drop |= _subtree_ids(sent, t["id"])
1017
+ newcase = "Nom" if pol == "Aff" else "Par" # negative imperative forces partitive object
1018
+ repl = {}
1019
+ for t in sent:
1020
+ if t["deprel"] == "obj" and t["feats"].get("Case") in ("Gen", "Nom") \
1021
+ and t["upos"] in ("NOUN", "PROPN", "ADJ", "NUM", "PRON"):
1022
+ fl = _flip_case(sent, t, newcase)
1023
+ if fl:
1024
+ repl.update(fl)
1025
+ src, tgt = [], []
1026
+ for t in sent:
1027
+ f = swap_initial(t["form"], repl_c) if (repl_c and t["id"] == verb["id"]) else t["form"]
1028
+ src.append(f)
1029
+ if t["id"] in drop:
1030
+ continue
1031
+ tgt.extend(cluster if t["id"] == verb["id"] else [repl.get(t["id"], t["form"])])
1032
+ return src, tgt
1033
+ return fn
1034
+
1035
+
1036
+ # accepted particle (canonical key OR a surface variant) -> (back-harmony variant, front-harmony variant).
1037
+ # -kin is harmony-invariant. Both members of a harmony pair are accepted as input so a caller may pass
1038
+ # either 'ko' or 'kö' (e.g. copied verbatim off an inflect-by-example model word).
1039
+ _CLITIC_VARIANTS = {
1040
+ "kin": ("kin", "kin"),
1041
+ "kaan": ("kaan", "kään"), "kään": ("kaan", "kään"),
1042
+ "ko": ("ko", "kö"), "kö": ("ko", "kö"),
1043
+ "han": ("han", "hän"), "hän": ("han", "hän"),
1044
+ "pa": ("pa", "pä"), "pä": ("pa", "pä"),
1045
+ }
1046
+
1047
+
1048
+ def _clitic_for(particle, harmony_word):
1049
+ """The vowel-harmony variant of an enclitic (-kin/-kaan/-ko/-han/-pa), chosen by the host word's
1050
+ vowels. Accepts the canonical particle or a surface variant ('ko'/'kö'). Returns None for an
1051
+ unrecognized particle so the caller can abstain instead of crashing (was a KeyError)."""
1052
+ pair = _CLITIC_VARIANTS.get(particle.lower()) if particle else None
1053
+ if pair is None:
1054
+ return None
1055
+ back = any(ch in BACK for ch in harmony_word.lower())
1056
+ return pair[0] if back else pair[1]
1057
+
1058
+
1059
+ def op_clitic(particle):
1060
+ """Attach a focus/pragmatic clitic (-kin/-kaan/-han/-pa) to the finite verb (vowel harmony)."""
1061
+ def fn(sent, nonce_rng=None):
1062
+ verb, _ = _main_verb(sent)
1063
+ if verb is None:
1064
+ return None
1065
+ form = verb["form"]
1066
+ repl_c = _rand_onset(nonce_rng, form[:1].lower()) if nonce_rng else None
1067
+ if repl_c:
1068
+ form = swap_initial(form, repl_c)
1069
+ suffix = _clitic_for(particle, form) # harmony follows the realized verb surface, not the lemma
1070
+ if suffix is None:
1071
+ return None
1072
+ new = form + suffix
1073
+ src = [swap_initial(t["form"], repl_c) if (repl_c and t["id"] == verb["id"]) else t["form"]
1074
+ for t in sent]
1075
+ tgt = [new if t["id"] == verb["id"] else t["form"] for t in sent]
1076
+ return src, tgt
1077
+ return fn
1078
+
1079
+
1080
+ def _op_number(target):
1081
+ """Change the (nominative) subject's number and re-agree the finite verb. target=Plur|Sing."""
1082
+ def fn(sent, nonce_rng=None):
1083
+ c = clause(sent)
1084
+ if c is None:
1085
+ return None
1086
+ verb, is_cop, base = c
1087
+ if base["number"] == target:
1088
+ return None
1089
+ subj = next((t for t in sent if t["deprel"] in ("nsubj", "nsubj:cop")), None)
1090
+ if subj is None or subj["feats"].get("Case", "Nom") != "Nom":
1091
+ return None
1092
+ repl = {}
1093
+ if subj["feats"].get("PronType") == "Prs": # suppletive personal pronoun
1094
+ new = _PRON_BY.get((base["person"], target))
1095
+ if new is None:
1096
+ return None
1097
+ repl[subj["id"]] = new[:1].upper() + new[1:] if subj["form"][:1].isupper() else new
1098
+ else:
1099
+ fl = _flip_number(sent, subj, target)
1100
+ if fl is None:
1101
+ return None
1102
+ repl.update(fl)
1103
+ vc = verb_cluster(verb["lemma"], base["voice"], base["mood"], base["tense"],
1104
+ base["pol"], base["person"], target)
1105
+ if vc is None:
1106
+ return None
1107
+ if is_cop: # copula clause: re-agree the predicate (root) + its conjuncts
1108
+ pred_id = verb["head"]
1109
+ for t in sent:
1110
+ if (t["id"] == pred_id or (t["head"] == pred_id and t["deprel"] == "conj")) \
1111
+ and t["upos"] in ("ADJ", "NOUN") and t["feats"].get("Case") == "Nom":
1112
+ pf = _realize_form(t["lemma"], t["upos"], {**t["feats"], "Number": target}, t["form"])
1113
+ if pf is None: # can't realize an agreeing predicate -> abstain, never
1114
+ return None # emit a half-agreed NP (subject Pl, predicate Sg)
1115
+ if pf != t["form"]:
1116
+ repl[t["id"]] = pf
1117
+ src, tgt = [], []
1118
+ for t in sent:
1119
+ src.append(t["form"])
1120
+ if t["id"] == verb["id"]:
1121
+ tgt.extend(vc)
1122
+ else:
1123
+ tgt.append(repl.get(t["id"], t["form"]))
1124
+ return src, tgt
1125
+ return fn
1126
+
1127
+
1128
+ _PX_PERSON = {"poss_1sg": "1Sg", "poss_2sg": "2Sg", "poss_3": "3",
1129
+ "poss_1pl": "1Pl", "poss_2pl": "2Pl"}
1130
+
1131
+
1132
+ def _op_poss(person):
1133
+ """Attach a possessive suffix (omistusliite) to the subject NP HEAD, preserving its case/number
1134
+ (talo->taloni, talossa->talossani). Structural rule: the engine picks the head noun;
1135
+ the realizer answers per-token (FST first, neural OOV second), so it works on loan/OOV heads
1136
+ (striimaaja->striimaajani). Agreeing modifiers (adjectives) keep their own form — Px is on the
1137
+ head only. Abstains if the subject is not a realizable nominal."""
1138
+ def fn(sent, nonce_rng=None):
1139
+ head = next((t for t in sent if t["deprel"] in ("nsubj", "nsubj:cop")
1140
+ and t["upos"] in ("NOUN", "PROPN")), None)
1141
+ if head is None:
1142
+ return None
1143
+ f = _realize_form(head["lemma"], head["upos"], {**head["feats"], "Px": person}, head["form"])
1144
+ if not f or f == head["form"]:
1145
+ return None
1146
+ src = [t["form"] for t in sent]
1147
+ tgt = [f if t["id"] == head["id"] else t["form"] for t in sent]
1148
+ return src, tgt
1149
+ return fn
1150
+
1151
+
1152
+ # ---- combined nominal morphology (compose number + Px + clitic on the subject head) ----
1153
+ def parse_nominal_bundle(op):
1154
+ """A '+'-composed op of >=2 NOMINAL atoms (plural/singular, poss_*, clitic_*) -> a feature
1155
+ bundle dict {number?, px?, clitic?}, else None. Each dimension may appear once. Single atoms and
1156
+ any verb/clause atom return None (they use the existing per-op path). This is what lets a user
1157
+ ask for `plural+poss_1sg+clitic_kin` and get ONE correctly-ordered realization, not a stack of
1158
+ string rewrites (you cannot pluralize the string `taloni`)."""
1159
+ atoms = op.split("+")
1160
+ if len(atoms) < 2:
1161
+ return None
1162
+ b = {}
1163
+ for a in atoms:
1164
+ if a in ("plural", "singular"):
1165
+ k, v = "number", ("Plur" if a == "plural" else "Sing")
1166
+ elif a in _PX_PERSON:
1167
+ k, v = "px", _PX_PERSON[a]
1168
+ elif a.startswith("clitic_"):
1169
+ k, v = "clitic", a.split("_", 1)[1]
1170
+ else:
1171
+ return None # a non-nominal atom -> not a nominal bundle
1172
+ if k in b:
1173
+ return None # same dimension twice -> ill-formed
1174
+ b[k] = v
1175
+ return b
1176
+
1177
+
1178
+ def _apply_nominal_bundle(sent, bundle):
1179
+ """Compose nominal morphology on the subject head in ONE realization: realize the head
1180
+ with the FINAL Number+Case+Px (never re-inflect an already-realized string), attach the clitic
1181
+ AFTER Px (surface order stem+case+number+Px+clitic), re-agree modifiers in number, and re-agree
1182
+ the finite verb if number changed. Case is preserved (a case-change op is separate)."""
1183
+ head = next((t for t in sent if t["deprel"] in ("nsubj", "nsubj:cop")
1184
+ and t["upos"] in ("NOUN", "PROPN")), None)
1185
+ if head is None:
1186
+ return None
1187
+ cur_num = head["feats"].get("Number", "Sing")
1188
+ number = bundle.get("number", cur_num)
1189
+ px = bundle.get("px", "None")
1190
+ feats = {**head["feats"], "Number": number, "Px": px} # Case preserved from the head
1191
+ new = _realize_form(head["lemma"], head["upos"], feats, head["form"])
1192
+ if not new:
1193
+ return None
1194
+ if "clitic" in bundle: # clitic attaches AFTER Px, on the head
1195
+ suffix = _clitic_for(bundle["clitic"], new) # harmony follows the realized surface host
1196
+ if suffix is None:
1197
+ return None
1198
+ new = new + suffix
1199
+ if new == head["form"]: # bundle was a no-op on this head
1200
+ return None
1201
+ repl = {head["id"]: new}
1202
+
1203
+ if number != cur_num: # number changed: agree modifiers + verb
1204
+ for m in sent:
1205
+ if m["head"] == head["id"] and m["deprel"] in AGREE and m["upos"] in ("ADJ", "NOUN"):
1206
+ mf = _realize_form(m["lemma"], m["upos"], {**m["feats"], "Number": number}, m["form"])
1207
+ if mf is None:
1208
+ return None
1209
+ repl[m["id"]] = mf
1210
+ c = clause(sent)
1211
+ if c is None:
1212
+ return None
1213
+ verb, _is_cop, base = c
1214
+ vc = verb_cluster(verb["lemma"], base["voice"], base["mood"], base["tense"],
1215
+ base["pol"], base["person"], number)
1216
+ if vc is None:
1217
+ return None
1218
+ src, tgt = [], []
1219
+ for t in sent:
1220
+ src.append(t["form"])
1221
+ tgt.extend(vc if t["id"] == verb["id"] else [repl.get(t["id"], t["form"])])
1222
+ return src, tgt
1223
+
1224
+ src = [t["form"] for t in sent]
1225
+ tgt = [repl.get(t["id"], t["form"]) for t in sent]
1226
+ return src, tgt
1227
+
1228
+
1229
+ # ---- phrase-level nominal morphology utility (NOT a clause transform) -----------------
1230
+ # Case is syntactically GOVERNED, so arbitrary case is exposed here as a phrase utility, never as a
1231
+ # finite-clause op — `transform` keeps grammaticality, `realize_np` does open-vocab nominal
1232
+ # morphology (`talo`->`talossa`, `striimaaja`->`striimaajillanikin`).
1233
+ def realize_np_form(lemma, upos="NOUN", case="Nom", number="Sing", px="None", clitic=None,
1234
+ surface=None):
1235
+ """Realize ONE nominal head: lemma -> target Case/Number(+Px), optional clitic after Px. Pure
1236
+ morphology, no parser; FST first then neural OOV via `_realize_form`. Returns the form or None
1237
+ (FST+neural both failed / suppressed). Does NOT clear the OOV trace lists — the caller does."""
1238
+ f = _realize_form(lemma, upos, {"Case": case, "Number": number, "Px": px}, surface or lemma)
1239
+ if f is None:
1240
+ return None
1241
+ if not clitic:
1242
+ return f
1243
+ suffix = _clitic_for(clitic, f) # harmony follows the realized SURFACE host, not the lemma (kuka->ketäkö)
1244
+ return f + suffix if suffix is not None else None
1245
+
1246
+
1247
+ # ---- non-finite verb forms (participles + infinitives) as a phrase utility ------------
1248
+ # A second phrase-level morphology utility (like realize_np): inflect a VERB lemma into a non-finite
1249
+ # form — the four participles (VA/NUT/TAVA/TU, which inflect like adjectives in case+number) and the
1250
+ # 2nd/3rd infinitive cases (mennessä/menemässä/menemästä/menemään/menemällä/menemättä). The FST query
1251
+ # layer (gen.giella_query) already supports these; this just exposes them with the FST→neural OOV
1252
+ # discipline. NUT is special-cased for OOV (it is the perfect participle, available via the trained
1253
+ # `perfect` flag); the other forms abstain for OOV verbs until a verbform realizer is trained.
1254
+ _VERBFORM = {
1255
+ "va": {"VerbForm": "Part", "PartForm": "Pres", "Voice": "Act"}, # present active (lukeva)
1256
+ "nut": {"VerbForm": "Part", "PartForm": "Past", "Voice": "Act"}, # past active (lukenut)
1257
+ "tava": {"VerbForm": "Part", "PartForm": "Pres", "Voice": "Pss"}, # present passive (luettava)
1258
+ "tu": {"VerbForm": "Part", "PartForm": "Past", "Voice": "Pss"}, # past passive (luettu)
1259
+ "inf2": {"VerbForm": "Inf", "InfForm": "2", "Voice": "Act"}, # 2nd infinitive (-essa/-en)
1260
+ "inf2_pass": {"VerbForm": "Inf", "InfForm": "2", "Voice": "Pss"}, # passive 2nd inf (tehtäessä; inessive-only)
1261
+ "inf3": {"VerbForm": "Inf", "InfForm": "3", "Voice": "Act"}, # 3rd infinitive (-ma- cases)
1262
+ "ma": {"VerbForm": "Part", "PartForm": "Agt"}, # agent participle (tekemä)
1263
+ }
1264
+ VERBFORMS = tuple(_VERBFORM)
1265
+
1266
+ # Possessive participle paradigm (the temporal construction, kuultuansa). The Px=None oblique participle
1267
+ # is already correct (sanoneessa); the Px form is that base's vowel stem + the possessive suffix, exactly
1268
+ # as for nominals (_derive_genpl_possessive) — so we DERIVE it instead of letting the noun realizer
1269
+ # re-inflect the participle from the wrong oblique stem. Singular + the five participles only; everything
1270
+ # else abstains (the caller keeps its OOV/flagged behaviour). Validated against the Giella FST
1271
+ # across conjugation types 1–6 × case × Px. We emit the uniform -nsa/-nsä
1272
+ # 3rd-person variant (the -Vn variant, sanoneessaan, is also FST-valid; -nsa is what the working tu+Par
1273
+ # path already produced as kuultuansa).
1274
+ _PARTICIPLE_PX_FORMS = frozenset({"va", "nut", "tava", "tu", "ma"})
1275
+ _PARTICIPLE_PX_SUFFIX = {"1Sg": "ni", "2Sg": "si", "1Pl": "mme", "2Pl": "nne"}
1276
+ _VOWELS = set("aeiouyäö")
1277
+
1278
+
1279
+ def _participle_px_stem(form, case, base_same, base_gen, base_nom):
1280
+ """The stem the possessive suffix attaches to (from the engine's own Px=None bases), or None to abstain.
1281
+ Three structural patterns: translative -ksi -> -kse-; nominative/genitive use the strong/oblique stem
1282
+ (tu: the strong NOM stem for gradation; nut: the genitive stem); every other case uses its own
1283
+ vowel-final form (or the vowel stem of an -n-final form, e.g. illative sanoneeseen)."""
1284
+ vow = lambda s: bool(s) and s[-1] in _VOWELS
1285
+ if case == "Tra": # -ksi -> -kse- (sanoneeksi -> sanoneekse-)
1286
+ return base_same[:-1] + "e" if (base_same and base_same.endswith("ksi")) else None
1287
+ if case in ("Nom", "Gen"):
1288
+ if form == "tu": # gradation: strong NOM stem (sanottu)
1289
+ return base_nom if vow(base_nom) else None
1290
+ if case == "Nom" and vow(base_nom): # va/ma: vowel-final nominative (sanova/sanoma)
1291
+ return base_nom
1292
+ return base_gen[:-1] if (base_gen and base_gen.endswith("n")) else None # nut: genitive stem (sanonee-)
1293
+ if vow(base_same): # general oblique (sanoneessa, sanonutta)
1294
+ return base_same
1295
+ return base_same[:-1] if (base_same and base_same.endswith("n")) else None
1296
+
1297
+
1298
+ def _derive_participle_possessive(lemma, form, case, number, px):
1299
+ """Modern possessive participle derived from the engine's own correct Px=None oblique base by a pure
1300
+ string rule, instead of the noun realizer re-inflecting the participle from the wrong oblique stem.
1301
+ Singular participles only; returns the form or None (abstain)."""
1302
+ if number != "Sing" or form not in _PARTICIPLE_PX_FORMS:
1303
+ return None
1304
+ suf = _PARTICIPLE_PX_SUFFIX.get(px)
1305
+ if suf is None:
1306
+ if px != "3":
1307
+ return None
1308
+ suf = "nsa" if any(c in BACK for c in lemma.lower()) else "nsä" # 3rd person: harmony from the lemma
1309
+ # the engine's own Px=None bases (FST-parity for known verbs; the verbform net for OOV). The derived
1310
+ # form inherits the base's provenance unchanged: the verbform Px=None serving path returns its net
1311
+ # output UNFLAGGED, so the derivation is unflagged too — consistent within the verbform subsystem.
1312
+ # (Verbform neural-provenance flagging is a broader pre-existing gap vs. the nominal path.)
1313
+ base_same = realize_verbform_form(lemma, form, case, number, "None")
1314
+ base_gen = realize_verbform_form(lemma, form, "Gen", number, "None")
1315
+ base_nom = realize_verbform_form(lemma, form, "Nom", number, "None")
1316
+ stem = _participle_px_stem(form, case, base_same, base_gen, base_nom)
1317
+ return stem + suf if stem is not None else None
1318
+
1319
+
1320
+ def realize_verbform_form(lemma, form="va", case="Nom", number="Sing", px="None", surface=None):
1321
+ """Realize ONE non-finite verb form: lemma -> participle/infinitive in the target Case/Number,
1322
+ optionally with a possessive suffix `px` (one of the realizer Px values). Px on a participle
1323
+ licenses the temporal construction, e.g. form='tu', case='Par', px='3' -> 'kuultuaan'. `form` is
1324
+ one of VERBFORMS. Lexicon-first then the verbform net; returns the form or None on a miss (OOV) —
1325
+ the caller decides whether to abstain. Pure morphology, no parser."""
1326
+ spec = _VERBFORM.get(form)
1327
+ if spec is None:
1328
+ return None
1329
+ if not _NETLEX_BACKEND:
1330
+ return None
1331
+ # Lexicon-first then the verbform net (keyed form+case+number, no Px dimension). A Px-bearing
1332
+ # temporal construction (e.g. 'kuultuansa') is DERIVED from the engine's own correct Px=None oblique
1333
+ # base by a pure string rule, instead of re-inflecting via the noun realizer (which used the wrong
1334
+ # oblique stem). Returns None -> abstain for plural / infinitives.
1335
+ if px and px != "None":
1336
+ return _derive_participle_possessive(lemma, form, case, number, px)
1337
+ if _GENLEX is not None:
1338
+ hit = _GENLEX.lookup_nonfin(lemma, form, case, number)
1339
+ if hit is not None:
1340
+ return hit # trusted lexicon (byte-exact) — no flag
1341
+ if _VERBFORM_REALIZER is not None:
1342
+ out, _conf = _VERBFORM_REALIZER(lemma, form, case, number)
1343
+ if out and "<unk>" not in out: # shape filter (mirror _neural_oov_ok); fail closed on <unk>
1344
+ # net-served verbform -> flag `neural_unverified` like the nominal path, so the best-effort
1345
+ # tail is HONESTLY marked (the verbform net has the same parity-or-flag contract; e.g.
1346
+ # `haukkua` va Ine -> `haukkuvussu` is wrong and must not pass as verified).
1347
+ _NEURAL_OOV.append(f"{lemma}->{out} ({form} {case}/{number})")
1348
+ return out
1349
+ return None
1350
+
1351
+
1352
+ # imperative is in the TRAINED set (registered now that op_imperative exists). potential rides
1353
+ # VERB_OPS->OPS. EXT_OPS keeps only the ops that don't need the neural model: clitics (pure string
1354
+ # attach) and number (would ride a future neural NOUN path, not this model).
1355
+ OPS["imperative"] = op_imperative
1356
+
1357
+ # compound-perfect tenses (FST for known verbs; OOV reuses the trained `perfect` participle
1358
+ # + an FST olla auxiliary). Each requires a present indicative base.
1359
+ _PERFECT_EXT = {
1360
+ "pluperfect": ({"tense": "Pqp"}, {"tense": "Prs", "mood": "Ind"}),
1361
+ "cond_perfect": ({"mood": "Cond", "tense": "Prf"}, {"tense": "Prs", "mood": "Ind"}),
1362
+ "pot_perfect": ({"mood": "Pot", "tense": "Prf"}, {"tense": "Prs", "mood": "Ind"}),
1363
+ }
1364
+ # imperative paradigm beyond the trained Sg2 flag, affirmative + negative (älä-series).
1365
+ _IMPRT_EXT = {f"imperative_{s}": _op_imperative_person(s, "Aff") for s in ("sg3", "pl1", "pl2", "pl3")}
1366
+ _IMPRT_EXT["neg_imperative"] = _op_imperative_person("sg2", "Neg")
1367
+ _IMPRT_EXT.update({f"neg_imperative_{s}": _op_imperative_person(s, "Neg")
1368
+ for s in ("sg3", "pl1", "pl2", "pl3")})
1369
+
1370
+ EXT_OPS = {
1371
+ "plural": _op_number("Plur"),
1372
+ "singular": _op_number("Sing"),
1373
+ "clitic_kin": op_clitic("kin"),
1374
+ "clitic_kaan": op_clitic("kaan"),
1375
+ "clitic_han": op_clitic("han"),
1376
+ "clitic_pa": op_clitic("pa"),
1377
+ # possessive suffix on the subject head (FST + neural OOV via the Px realizer)
1378
+ **{name: _op_poss(px) for name, px in _PX_PERSON.items()},
1379
+ # compound tenses + full imperative paradigm (FST/rules; targeted OOV reuse)
1380
+ **{name: _verbop(d, r) for name, (d, r) in _PERFECT_EXT.items()},
1381
+ **_IMPRT_EXT,
1382
+ }
1383
+ ALL_OPS = {**OPS, **EXT_OPS}
1384
+
1385
+
1386
+ # ----------------------------- clause scope + inference entry point -------------------------------
1387
+ def _nfin(s):
1388
+ return sum(1 for t in s if t["feats"].get("VerbForm") == "Fin")
1389
+
1390
+
1391
+ # Clause SCOPE: a single finite clause with one clear main predicate. Out-of-scope inputs
1392
+ # (subordinate/coordinate clauses, relatives, etc.) are abstained on — this removes the
1393
+ # wrong-target error class and makes "transform a clause" a well-defined primitive.
1394
+ _SUBORD = {"advcl", "ccomp", "acl", "acl:relcl", "parataxis", "csubj", "csubj:cop", "mark"}
1395
+
1396
+
1397
+ def in_scope(s, max_words=12):
1398
+ if _nfin(s) != 1 or clause(s) is None or len(s) > max_words:
1399
+ return False
1400
+ return not any(t["deprel"] in _SUBORD for t in s)
1401
+
1402
+
1403
+ # The imperative family applies only when the command's addressee is already in the clause: in scope
1404
+ # iff the clause is subjectless (incl. an already-imperative clause) OR its subject is 2nd-person
1405
+ # (`sinä`/`te`). A non-addressee (3rd/1st-person) subject would have to INVENT an addressee, so it
1406
+ # abstains — see `_imperative_source_verb` for the root requirements.
1407
+ def imperative_in_scope(s, max_words=12):
1408
+ verb = _imperative_source_verb(s)
1409
+ if verb is None or _nfin(s) != 1 or len(s) > max_words:
1410
+ return False
1411
+ if any(t["deprel"] in _SUBORD for t in s):
1412
+ return False
1413
+ has_subject = any(t["deprel"] in ("nsubj", "nsubj:cop") for t in s)
1414
+ return (not has_subject) or verb["feats"].get("Person") == "2"
1415
+
1416
+
1417
+ # Ops whose scope is governed by `imperative_in_scope` (not the general `clause`/`in_scope` gate).
1418
+ IMPERATIVE_OPS = frozenset(
1419
+ o for o in ALL_OPS
1420
+ if o == "imperative" or o.startswith("imperative_") or o.startswith("neg_imperative"))
1421
+
1422
+
1423
+ def hybrid_transform(sent, op, model_bundle=None, force_neural=False):
1424
+ """Inference: deterministic structure (`apply_clause`) + Giella-free neural realization for OOV
1425
+ verbs/nouns (the finiteverb/verbform/realizer nets, set on this module by `pipeline.Transformer`).
1426
+ `model_bundle` is accepted for backward compat but unused. Returns the transformed sentence
1427
+ string, or None if the op does not apply."""
1428
+ global _FORCE_NEURAL_VC
1429
+ _FORCE_NEURAL_VC = force_neural
1430
+ _NEURAL_OOV.clear() # record which words this transform realized neurally
1431
+ _OOV_UNREALIZABLE.clear() # ...which it could not realize at all (oov_unrealizable)
1432
+ _OOV_UNVERIFIED.clear() # ...and which it realized but suppressed (oov_unverified)
1433
+ bundle = parse_nominal_bundle(op) # composed nominal morphology, one realization
1434
+ try:
1435
+ r = _apply_nominal_bundle(sent, bundle) if bundle is not None else ALL_OPS[op](sent)
1436
+ finally:
1437
+ _FORCE_NEURAL_VC = False
1438
+ return _tgt(r) if r is not None else None
juurio/analyze/__init__.py ADDED
File without changes
juurio/analyze/data.py ADDED
@@ -0,0 +1,127 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """AnalyzeMux data layer — invert the FST cache into per-form multi-label training examples.
2
+
3
+ The parser-facing reverse model predicts, per surface form:
4
+ - a UNION tag-bag (multi-label over the FST morphtags, lemma excluded) — the exact thing the
5
+ parser's FST channel consumes; and
6
+ - a set of lemma EDIT-SCRIPTS (multi-label over the script inventory) — decode top-K to recover
7
+ candidate lemmas.
8
+
9
+ Training data is the inverted FST cache: each form already carries its full reading list
10
+ `[lemma, tag1, ...]`. We train on the full cache.
11
+
12
+ All functions here are pure + deterministic (no torch, no RNG) so the split and targets are reproducible.
13
+ """
14
+ from __future__ import annotations
15
+
16
+ import hashlib
17
+ import json
18
+ from collections import Counter
19
+ from pathlib import Path
20
+ from typing import Iterable
21
+
22
+ from juurio.parser.edit_scripts import extract_edit_script
23
+ from juurio.parser.fst_cache import UNK_ID as TAG_UNK_ID
24
+
25
+ SCRIPT_UNK_ID = 0 # reserved id 0 for unseen/rare edit-scripts; real scripts start at 1
26
+
27
+
28
+ def edit_script(form: str, lemma: str) -> str:
29
+ """Surface->lemma edit script as a canonical STRING key. The parser's extractor returns a tuple
30
+ `(delete_count, append_suffix)`; we serialize it `"<n>:<suffix>"` so it is hashable AND JSON-safe
31
+ (tuples round-trip through JSON as lists, which then break dict keys)."""
32
+ n, suffix = extract_edit_script(form, lemma)
33
+ return f"{n}:{suffix}"
34
+
35
+
36
+ class ScriptVocab:
37
+ """Multi-label inventory of surface->lemma edit-scripts. id 0 = UNK (rare/unseen); real from 1."""
38
+
39
+ def __init__(self, scripts: list[str]):
40
+ # `scripts` is the ordered real-script list (UNK is implicit at id 0).
41
+ self._scripts = list(scripts)
42
+ self._to_id = {s: i + 1 for i, s in enumerate(scripts)}
43
+
44
+ @property
45
+ def size(self) -> int:
46
+ """Total head width = UNK + real scripts."""
47
+ return len(self._scripts) + 1
48
+
49
+ @classmethod
50
+ def fit(cls, cache: dict[str, list[list[str]]], *, min_count: int = 1,
51
+ max_scripts: int | None = None) -> "ScriptVocab":
52
+ c: Counter[str] = Counter()
53
+ for form, readings in cache.items():
54
+ for r in readings:
55
+ if not r:
56
+ continue
57
+ try:
58
+ c[edit_script(form, r[0])] += 1
59
+ except Exception:
60
+ continue
61
+ kept = [s for s, n in c.most_common() if n >= min_count]
62
+ if max_scripts is not None:
63
+ kept = kept[:max_scripts]
64
+ return cls(kept)
65
+
66
+ def encode(self, script: str) -> int:
67
+ return self._to_id.get(script, SCRIPT_UNK_ID)
68
+
69
+ def decode(self, idx: int) -> str:
70
+ if idx == SCRIPT_UNK_ID or not (1 <= idx <= len(self._scripts)):
71
+ return "<UNK>"
72
+ return self._scripts[idx - 1]
73
+
74
+ def save(self, path: str) -> None:
75
+ Path(path).parent.mkdir(parents=True, exist_ok=True)
76
+ with open(path, "w") as fh:
77
+ json.dump({"scripts": self._scripts}, fh, ensure_ascii=False)
78
+
79
+ @classmethod
80
+ def load(cls, path: str) -> "ScriptVocab":
81
+ with open(path) as fh:
82
+ return cls(json.load(fh)["scripts"])
83
+
84
+
85
+ def reading_tagset(readings: list[list[str]]) -> set[str]:
86
+ """Union of morphtags across all readings, lemma (position 0) excluded."""
87
+ out: set[str] = set()
88
+ for r in readings:
89
+ for tag in r[1:]:
90
+ out.add(tag)
91
+ return out
92
+
93
+
94
+ def build_example(form: str, readings: list[list[str]], tag_vocab, script_vocab: ScriptVocab) -> dict:
95
+ """One training example: union tag ids (UNK-filtered, sorted unique) + lemma script ids (unique)."""
96
+ tag_ids = sorted({i for i in (tag_vocab.encode(t) for t in reading_tagset(readings))
97
+ if i != TAG_UNK_ID})
98
+ script_ids = sorted({script_vocab.encode(edit_script(form, r[0])) for r in readings if r})
99
+ return {"form": form, "tag_ids": tag_ids, "script_ids": script_ids}
100
+
101
+
102
+ # ----------------------------- lemma-family hold-out split ----------------------------------------
103
+ def lemma_held(lemma: str, holdout_pct: int) -> bool:
104
+ """Deterministic split: lemma is held out iff md5(lemma) % 100 < holdout_pct (stable across runs)."""
105
+ h = int(hashlib.md5(lemma.encode("utf-8")).hexdigest(), 16)
106
+ return (h % 100) < holdout_pct
107
+
108
+
109
+ def form_held(readings: list[list[str]], holdout_pct: int) -> bool:
110
+ """A form is held out if ANY of its reading lemmas is held — prevents a shared surface
111
+ form leaking held-family morphology into train."""
112
+ return any(lemma_held(r[0], holdout_pct) for r in readings if r)
113
+
114
+
115
+ def iter_examples(cache: dict[str, list[list[str]]], tag_vocab, script_vocab: ScriptVocab,
116
+ *, holdout_pct: int = 10, split: str | None = None) -> Iterable[dict]:
117
+ """Yield build_example() dicts; `split` in {None,'train','held'} filters by the lemma-family split."""
118
+ for form, readings in cache.items():
119
+ if not readings:
120
+ continue
121
+ if split is not None:
122
+ held = form_held(readings, holdout_pct)
123
+ if split == "train" and held:
124
+ continue
125
+ if split == "held" and not held:
126
+ continue
127
+ yield build_example(form, readings, tag_vocab, script_vocab)
juurio/analyze/hybrid.py ADDED
@@ -0,0 +1,44 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Build the hybrid morphology channel that feeds the parser: clean-FST tags for forms covered by the
2
+ FST-distilled lexicon, neural AnalyzeMux predictions for out-of-lexicon forms.
3
+
4
+ The central invariant (what lets the parser consume this with no retraining): in-lexicon forms emit the
5
+ EXACT clean-FST tag bag — including the empty bag for forms the lexicon analyses to nothing — and NEVER
6
+ fall through to the neural prediction. Neural is used only where the lexicon has no entry at all. The
7
+ hybrid channel is therefore a strict superset of the clean channel the parser trained on: identical
8
+ where the lexicon speaks, additive only on genuinely out-of-lexicon forms.
9
+ """
10
+ from __future__ import annotations
11
+
12
+ from juurio.parser.fst_cache import FstCache
13
+
14
+
15
+ def build_hybrid_channel(lexicon: FstCache, neural: FstCache, forms) -> tuple[FstCache, dict[str, int]]:
16
+ """Return (hybrid_cache, counts).
17
+
18
+ For each form in `forms`:
19
+ * form in `lexicon` -> the lexicon's clean tag bag (no fallthrough to neural, even if empty);
20
+ * otherwise -> the neural prediction.
21
+
22
+ `lexicon` is the full FST-distilled lexicon (full hybrid) or a compressed top-K table — in either
23
+ case membership (`form in lexicon`) decides clean-vs-neural. `counts` tallies lexicon / neural / empty.
24
+ """
25
+ data: dict[str, list[list[str]]] = {}
26
+ counts = {"lexicon": 0, "neural": 0, "empty": 0}
27
+ for form in forms:
28
+ if form in lexicon:
29
+ tags = sorted(lexicon.form_to_tagset(form))
30
+ if tags:
31
+ data[form] = [["<exc>", *tags]]
32
+ counts["lexicon"] += 1
33
+ else:
34
+ data[form] = [] # in-lexicon but no reading: stays empty, NO neural fallthrough
35
+ counts["empty"] += 1
36
+ else:
37
+ tags = sorted(neural.form_to_tagset(form))
38
+ if tags:
39
+ data[form] = [["<neural>", *tags]]
40
+ counts["neural"] += 1
41
+ else:
42
+ data[form] = []
43
+ counts["empty"] += 1
44
+ return FstCache(data), counts
juurio/analyze/infer.py ADDED
@@ -0,0 +1,74 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """AnalyzeMux inference: load a checkpoint, predict per-form union tag-sets, build a neural FST cache.
2
+
3
+ The reverse model's union-tag head emits ids in the SAME FstTagVocab space the parser channel consumes,
4
+ so we can decode predictions to tag strings and pack them into an `FstCache` the parser's collator reads
5
+ verbatim. Reserved PAD/UNK tag ids are never emitted.
6
+ """
7
+ from __future__ import annotations
8
+
9
+ import torch
10
+
11
+ from juurio.analyze.metrics import decode_multilabel
12
+ from juurio.analyze.model import AnalyzeMux, CharVocab
13
+ from juurio.parser.fst_cache import PAD_ID as TAG_PAD_ID
14
+ from juurio.parser.fst_cache import UNK_ID as TAG_UNK_ID
15
+ from juurio.parser.fst_cache import FstCache
16
+
17
+ TAG_RESERVED = (TAG_PAD_ID, TAG_UNK_ID)
18
+
19
+
20
+ @torch.no_grad()
21
+ def predict_tagsets(model, char_vocab: CharVocab, forms, *, tag_vocab,
22
+ threshold: float = 0.5, top_n: int = 16, device: str = "cpu",
23
+ batch_size: int = 1024) -> list[set[str]]:
24
+ """Predict each form's union tag-SET (decoded strings), reserved ids excluded, top_n-capped."""
25
+ model.eval()
26
+ out: list[set[str]] = []
27
+ for s in range(0, len(forms), batch_size):
28
+ chunk = forms[s:s + batch_size]
29
+ ids, mask = char_vocab.encode_batch(chunk)
30
+ logits = model(ids.to(device), mask.to(device))["tag_logits"].cpu()
31
+ for row in logits:
32
+ ids_pred = decode_multilabel(row.tolist(), threshold=threshold, top_n=top_n,
33
+ forbidden_ids=TAG_RESERVED)
34
+ out.append({tag_vocab.decode(i) for i in ids_pred})
35
+ return out
36
+
37
+
38
+ def build_neural_cache(model, char_vocab: CharVocab, forms, *, tag_vocab,
39
+ threshold: float = 0.5, top_n: int = 16, device: str = "cpu") -> FstCache:
40
+ """Build an FstCache mapping each form -> a single synthetic reading [<neural>, *predicted_tags].
41
+ The lemma slot is a placeholder (the channel excludes lemma); only the tag-set is meaningful."""
42
+ tagsets = predict_tagsets(model, char_vocab, list(forms), tag_vocab=tag_vocab,
43
+ threshold=threshold, top_n=top_n, device=device)
44
+ data: dict[str, list[list[str]]] = {}
45
+ for form, tags in zip(forms, tagsets):
46
+ data[form] = [["<neural>", *sorted(tags)]] if tags else []
47
+ return FstCache(data)
48
+
49
+
50
+ def load_analyze_mux(ckpt_path: str, *, char_vocab_path: str, device: str = "cpu", tag_vocab=None):
51
+ """Returns (model, char_vocab, calibrated_threshold).
52
+
53
+ If `tag_vocab` is given, verify the analyzer's tag head is compatible with it: the output
54
+ dimension must match the vocab size, and — when the checkpoint records `fst_tag_vocab_hash`
55
+ (stamped at training time) — the vocab content hash must match. This guards against decoding
56
+ the tag logits with a different / reordered FstTagVocab, which would silently corrupt the channel.
57
+ """
58
+ blob = torch.load(ckpt_path, map_location=device, weights_only=False)
59
+ cfg = blob["config"]
60
+ cv = CharVocab.load(char_vocab_path)
61
+ if tag_vocab is not None:
62
+ if cfg["n_tags"] != tag_vocab.size:
63
+ raise ValueError(f"AnalyzeMux tag head n_tags={cfg['n_tags']} != tag_vocab.size="
64
+ f"{tag_vocab.size}; analyzer and channel use different tag vocabs.")
65
+ want = cfg.get("fst_tag_vocab_hash")
66
+ if want is not None and want != tag_vocab.hash():
67
+ raise ValueError(f"AnalyzeMux was trained against fst_tag_vocab hash {want}, but the "
68
+ f"supplied tag_vocab hashes to {tag_vocab.hash()}.")
69
+ model = AnalyzeMux(char_vocab=cv.size, n_tags=cfg["n_tags"], n_scripts=cfg["n_scripts"],
70
+ d_model=cfg["d_model"], layers=cfg["layers"], heads=cfg["heads"],
71
+ ff=cfg["ff"]).to(device)
72
+ model.load_state_dict(blob["state_dict"])
73
+ model.eval()
74
+ return model, cv, cfg.get("calibrated_threshold", 0.5)
juurio/analyze/metrics.py ADDED
@@ -0,0 +1,66 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """AnalyzeMux eval metric primitives (pure, no torch).
2
+
3
+ Used to score the reverse model's predicted tag/script id-sets against the FST cache. Slicing
4
+ (held-family / rare / 3+-reading / nonce) is orchestration elsewhere; these are the kernels.
5
+ """
6
+ from __future__ import annotations
7
+
8
+ import math
9
+
10
+
11
+ def micro_prf(pred_sets, gold_sets) -> tuple[float, float, float]:
12
+ """Micro precision/recall/F1 over predicted-vs-gold id SETS (per-example, summed)."""
13
+ tp = fp = fn = 0
14
+ for pred, gold in zip(pred_sets, gold_sets):
15
+ pred, gold = set(pred), set(gold)
16
+ tp += len(pred & gold)
17
+ fp += len(pred - gold)
18
+ fn += len(gold - pred)
19
+ p = tp / (tp + fp) if (tp + fp) else 1.0
20
+ r = tp / (tp + fn) if (tp + fn) else 1.0
21
+ f = 2 * p * r / (p + r) if (p + r) else 0.0
22
+ return p, r, f
23
+
24
+
25
+ def script_recall_at_k(ranked_lists, gold_sets, *, k: int, exclude_ids=()) -> float:
26
+ """Micro recall@K: gold script ids appearing in each example's top-K / total gold ids.
27
+ `exclude_ids` (e.g. SCRIPT_UNK_ID) are dropped from BOTH gold and the ranked list before the top-K
28
+ cut — predicting UNK is not real lemma recovery."""
29
+ exclude = set(exclude_ids)
30
+ found = total = 0
31
+ for ranked, gold in zip(ranked_lists, gold_sets):
32
+ gold = set(gold) - exclude
33
+ ranked = [i for i in ranked if i not in exclude]
34
+ topk = set(ranked[:k])
35
+ total += len(gold)
36
+ found += len(gold & topk)
37
+ return found / total if total else 1.0
38
+
39
+
40
+ def fbeta(p: float, r: float, beta: float) -> float:
41
+ """F_beta: beta>1 weights recall higher than precision (the channel tolerates moderate
42
+ over-recall better than under-recall)."""
43
+ b2 = beta * beta
44
+ denom = b2 * p + r
45
+ return (1 + b2) * p * r / denom if denom else 0.0
46
+
47
+
48
+ def rank_ids(logits) -> list[int]:
49
+ """Indices sorted by descending logit (stable on ties by original index)."""
50
+ return sorted(range(len(logits)), key=lambda i: (-logits[i], i))
51
+
52
+
53
+ def _sigmoid(x: float) -> float:
54
+ return 1.0 / (1.0 + math.exp(-x))
55
+
56
+
57
+ def decode_multilabel(logits, *, threshold: float = 0.5, top_n: int = 16,
58
+ forbidden_ids=()) -> list[int]:
59
+ """Ids whose sigmoid(logit) > threshold, capped to the top_n highest-scoring (mirrors the parser
60
+ channel's max_tags cap — mirror what the parser actually receives). `forbidden_ids` (e.g. the
61
+ reserved PAD/UNK tag ids) are never decoded, regardless of logit."""
62
+ forbidden = set(forbidden_ids)
63
+ above = [i for i in range(len(logits))
64
+ if i not in forbidden and _sigmoid(logits[i]) > threshold]
65
+ above.sort(key=lambda i: (-logits[i], i))
66
+ return above[:top_n]
juurio/analyze/model.py ADDED
@@ -0,0 +1,95 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """AnalyzeMux model — char encoder + two multi-label heads.
2
+
3
+ surface form (chars) -> shared Transformer encoder -> masked mean-pool -> two heads:
4
+ - tag head: multi-label sigmoid over the FST morphtags (the parser channel union tag-bag);
5
+ - script head: multi-label sigmoid over the lemma edit-scripts (+UNK); top-K -> candidate lemmas.
6
+ """
7
+ from __future__ import annotations
8
+
9
+ import json
10
+ from pathlib import Path
11
+
12
+ import torch
13
+ import torch.nn as nn
14
+
15
+ CHAR_PAD_ID = 0
16
+ CHAR_UNK_ID = 1
17
+
18
+
19
+ class CharVocab:
20
+ """Char -> id. id 0 = PAD, id 1 = UNK, real chars from 2."""
21
+
22
+ def __init__(self, chars: list[str]):
23
+ self._chars = list(chars)
24
+ self._to_id = {c: i + 2 for i, c in enumerate(chars)}
25
+ self._from_id = {i + 2: c for i, c in enumerate(chars)}
26
+
27
+ @property
28
+ def size(self) -> int:
29
+ return len(self._chars) + 2
30
+
31
+ @property
32
+ def unk_id(self) -> int:
33
+ return CHAR_UNK_ID
34
+
35
+ @classmethod
36
+ def fit(cls, forms) -> "CharVocab":
37
+ seen: dict[str, None] = {}
38
+ for f in forms:
39
+ for ch in f:
40
+ seen.setdefault(ch, None)
41
+ return cls(sorted(seen))
42
+
43
+ def encode(self, form: str) -> list[int]:
44
+ return [self._to_id.get(ch, CHAR_UNK_ID) for ch in form]
45
+
46
+ def decode(self, idx: int) -> str:
47
+ return self._from_id.get(idx, "<UNK>")
48
+
49
+ def encode_batch(self, forms) -> tuple[torch.Tensor, torch.Tensor]:
50
+ seqs = [self.encode(f) for f in forms]
51
+ width = max((len(s) for s in seqs), default=1) or 1
52
+ batch = torch.zeros(len(seqs), width, dtype=torch.long)
53
+ mask = torch.zeros(len(seqs), width, dtype=torch.bool)
54
+ for i, s in enumerate(seqs):
55
+ if s:
56
+ batch[i, :len(s)] = torch.tensor(s, dtype=torch.long)
57
+ mask[i, :len(s)] = True
58
+ return batch, mask
59
+
60
+ def save(self, path: str) -> None:
61
+ Path(path).parent.mkdir(parents=True, exist_ok=True)
62
+ with open(path, "w") as fh:
63
+ json.dump({"chars": self._chars}, fh, ensure_ascii=False)
64
+
65
+ @classmethod
66
+ def load(cls, path: str) -> "CharVocab":
67
+ with open(path) as fh:
68
+ return cls(json.load(fh)["chars"])
69
+
70
+
71
+ class AnalyzeMux(nn.Module):
72
+ def __init__(self, char_vocab: int, n_tags: int, n_scripts: int, *,
73
+ d_model: int = 128, layers: int = 2, heads: int = 4, ff: int = 512,
74
+ maxlen: int = 64, dropout: float = 0.1):
75
+ super().__init__()
76
+ self.emb = nn.Embedding(char_vocab, d_model, padding_idx=CHAR_PAD_ID)
77
+ self.pos = nn.Embedding(maxlen, d_model)
78
+ self.enc = nn.TransformerEncoder(
79
+ nn.TransformerEncoderLayer(d_model, heads, ff, dropout, batch_first=True), layers)
80
+ self.tag_head = nn.Linear(d_model, n_tags)
81
+ self.script_head = nn.Linear(d_model, n_scripts)
82
+ self.maxlen = maxlen
83
+
84
+ def forward(self, ids: torch.Tensor, mask: torch.Tensor) -> dict[str, torch.Tensor]:
85
+ """ids/mask: (B, T). Returns tag_logits (B,n_tags), script_logits (B,n_scripts)."""
86
+ B, T = ids.shape
87
+ pos = torch.arange(T, device=ids.device).clamp_max(self.maxlen - 1)
88
+ x = self.emb(ids) + self.pos(pos)[None, :, :]
89
+ # TransformerEncoder: src_key_padding_mask is True where PAD (to be ignored).
90
+ h = self.enc(x, src_key_padding_mask=~mask) # (B, T, d)
91
+ # masked mean-pool over real positions only (PAD must not leak in)
92
+ m = mask.unsqueeze(-1).float()
93
+ pooled = (h * m).sum(dim=1) / m.sum(dim=1).clamp_min(1.0)
94
+ return {"tag_logits": self.tag_head(pooled),
95
+ "script_logits": self.script_head(pooled)}
juurio/bootstrap.py ADDED
@@ -0,0 +1,353 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Install-time bootstrap: acquire serving artifacts (model checkpoints + data) and diagnose the env.
2
+
3
+ Serving artifacts live in a HuggingFace repo (`samuli/juurio`) under CLEAN, stable filenames
4
+ (`checkpoints/parser.pt`, `data/wordlist.json`, …) decoupled from the messy dev filenames. Each MANIFEST
5
+ entry maps a stable LOGICAL key -> `{hf, source, sha256, size, …}`:
6
+ * `hf` — path in the HF repo (clean, version-free); downstream installs fetch this.
7
+ * `source` — the committed dev-tree file (versioned, e.g. `parser_v30b.pt`); used in source checkouts.
8
+
9
+ `bootstrap.artifact_path(key)` is the single resolver every serving load site goes through. Precedence:
10
+ 1. dev source (`ROOT/<source>`) when present — source-checkout/dev, authoritative, no runtime sha check;
11
+ 2. `JUURIO_CKPT_BASEURL` mirror (`<base>/<hf>`) — self-host override, sha256-verified;
12
+ 3. HuggingFace `samuli/juurio` @ pinned `_HF_REVISION` — sha256-verified, returned from the HF cache.
13
+
14
+ A new model release changes only the MANIFEST (`source`+`sha256`+`size`) and the pinned `_HF_REVISION`;
15
+ serving code and HF filenames never churn. `juurio setup` pre-fetches+verifies everything; `juurio
16
+ doctor` is read-only diagnostics (never downloads).
17
+ """
18
+ from __future__ import annotations
19
+
20
+ import hashlib
21
+ import importlib.metadata as _md
22
+ import os
23
+ import urllib.request
24
+
25
+ ROOT = os.path.normpath(os.path.join(os.path.dirname(__file__), ".."))
26
+
27
+ # HF repo holding the serving artifacts, pinned to the immutable upload commit so downstream
28
+ # installs fetch exactly these bytes.
29
+ HF_REPO = "samuli/juurio"
30
+ _HF_REVISION = "948a8aad56454eb3c24bb4c05c28a2dcbc90ba87"
31
+
32
+ # Serving artifacts, keyed by stable logical name. `hf` = clean path in HF_REPO; `source` = committed
33
+ # dev-tree file (resolved against ROOT). These are the files the served product loads.
34
+ MANIFEST = {
35
+ # --- served analysis: Giella/Stanza-free end-to-end neural parser ---
36
+ "parser": {
37
+ "hf": "checkpoints/parser.pt",
38
+ "source": "checkpoints/parser_v30b.pt",
39
+ "sha256": "bd29307b7af38086b6781cd73d9b70b5c73bb7b71d6cd2d37c4731de1a38c6ac",
40
+ "size": 70519101,
41
+ "what": "TinyFiUD dependency parser — UPOS/Feats/lemma/tree (served via neural_adapter.NeuralParser)",
42
+ },
43
+ "analyze": {
44
+ "hf": "checkpoints/analyze.pt",
45
+ "source": "checkpoints/analyze_mux_m2.pt",
46
+ "sha256": "b2d1bae63a43bb4cd9eb52dc7803448dcf3bbc7f7dd08b17e292d3c44548d4df",
47
+ "size": 50730757,
48
+ "what": "AnalyzeMux — neural morphology channel feeding the parser's morphology-tag input",
49
+ },
50
+ "realizer": {
51
+ "hf": "checkpoints/realizer.pt",
52
+ "source": "checkpoints/realizer.pt",
53
+ "sha256": "0311fbea4bb6a6ac8f385b3edca5c1eef4788ad4cc63a418e663a059f65631cd",
54
+ "size": 3951901,
55
+ "what": "nominal realizer (pointer-generator copy head) — OOV noun/adj/propn case/number/Px/degree",
56
+ },
57
+ "verbform": {
58
+ "hf": "checkpoints/verbform.pt",
59
+ "source": "checkpoints/verbform.pt",
60
+ "sha256": "c8ccffaed0a6468ea159147292f8d58d347cd6428c7ff6bf7a713e6a67046d59",
61
+ "size": 3950237,
62
+ "what": "verbform realizer (copy head) — OOV verb participles (VA/NUT/TAVA/TU + agent MA) + infinitives",
63
+ },
64
+ "finiteverb": {
65
+ "hf": "checkpoints/finiteverb.pt",
66
+ "source": "checkpoints/finiteverb.pt",
67
+ "sha256": "08bf5281d321dbae46625c0a83c9a8129eed3d2b9762a8058c584ce3a0a4fc83",
68
+ "size": 3816501,
69
+ "what": "finite-verb realizer — OOV finite verbs (Act/Pss x Ind/Cond/Pot/Imprt + connegatives)",
70
+ },
71
+ # Small vocab/tokenizer JSONs the NeuralParser loads. Committed in-tree for source checkouts; fetched
72
+ # from HF (like the checkpoints) for clean wheel installs that don't carry them.
73
+ "parser_tokenizer": {
74
+ "hf": "checkpoints/parser_tokenizer.json",
75
+ "source": "checkpoints/parser_tokenizer.json",
76
+ "sha256": "9ed387a7e9741863231174617d073af0cef179b841ca7de050e1dbce7ece157f",
77
+ "size": 537891,
78
+ "what": "parser BPE tokenizer (provenance-locked to the parser checkpoint)",
79
+ },
80
+ "parser_label_vocab": {
81
+ "hf": "checkpoints/parser_label_vocab.json",
82
+ "source": "checkpoints/parser_label_vocab.json",
83
+ "sha256": "7b13d5b2c770042034acda7c460b0afe15b1e1a1b19832474854f2d59aacbf36",
84
+ "size": 59183,
85
+ "what": "parser label vocabulary (UPOS/Feats/deprel ids; locked to the parser checkpoint)",
86
+ },
87
+ "analyze_charvocab": {
88
+ "hf": "checkpoints/analyze.charvocab.json",
89
+ "source": "checkpoints/analyze_mux_m2.charvocab.json",
90
+ "sha256": "ae2ef909670d08767b5a746a8733e8b43ab25e0d79f6dc285168961e4266bf90",
91
+ "size": 1443,
92
+ "what": "AnalyzeMux char vocabulary (locked to the analyze checkpoint)",
93
+ },
94
+ # --- generation lexicon + morphology-tag table (data/) ---
95
+ "wordlist": {
96
+ "hf": "data/wordlist.json",
97
+ "source": "data/shipped_wordlist.json",
98
+ "sha256": "248f89465a5a5c505dfaceb95ab0d8a98eaca32e17f6af253954a3a0bfeb85f0",
99
+ "size": 52555054,
100
+ "what": "generation lexicon (NOUN/ADJ/VERB cells the realizer nets get wrong) — companion to "
101
+ "realizer/verbform/finiteverb (juurio.genlex)",
102
+ },
103
+ "fst_table": {
104
+ "hf": "data/fst_table.jsonl",
105
+ "source": "data/fst_compressed_table_k100k_clean.jsonl",
106
+ "sha256": "04abe3cf35ab45256638ea1d5f63d39b99b0ed974751d5912eefa075c3183733",
107
+ "size": 7278625,
108
+ "what": "compressed morphology-tag table (7.3 MB) — clean tags for the top-100k forms, minus e-mail "
109
+ "addresses, @handles, URLs, phone numbers and hashtags; the neural channel fills the rest",
110
+ },
111
+ "fst_tag_vocab": {
112
+ "hf": "data/fst_tag_vocab.json",
113
+ "source": "data/fst_tag_vocab_v1.json",
114
+ "sha256": "6ca9e42d017890e2694fb0f3f53c2ccf7927261116f8f3045b969694c19a0846",
115
+ "size": 136521,
116
+ "what": "morphology-tag vocabulary (locked to the analyze/parser checkpoints)",
117
+ },
118
+ "char_vocab": {
119
+ "hf": "data/char_vocab.json",
120
+ "source": "data/char_vocab_v1.json",
121
+ "sha256": "7dca5353150021dccebf45f75e9f1cb9e8f8263677a6dc3823a2bd8a21abf59e",
122
+ "size": 5049,
123
+ "what": "parser char vocabulary (compound-splitter input; locked to the parser checkpoint)",
124
+ },
125
+ }
126
+
127
+ # Serving runtime deps — the served parser is the torch-only NeuralParser.
128
+ _RUNTIME_DEPS = ["torch", "numpy", "transformers", "huggingface_hub"]
129
+
130
+ _BASEURL_ENV = "JUURIO_CKPT_BASEURL" # self-host mirror of the clean HF tree (<base>/<hf>)
131
+ _OFFLINE_ENV = "JUURIO_OFFLINE" # truthy -> hf local_files_only (cache-only, no network)
132
+ _CACHE_ENV = "JUURIO_CACHE" # where the baseurl-mirror download lands (default ~/.cache/juurio)
133
+ # HF's own get_token() handles the standard sources (HF_TOKEN env + the `hf auth login` cached token).
134
+ # These non-standard langchain-style vars are a LAST-RESORT fallback only, so they never shadow a CLI login.
135
+ _LEGACY_TOKEN_ENVS = ("HUGGINGFACEHUB_API_TOKEN", "HUGGINGFACE_API_TOKEN")
136
+ # In-process memo of absolute paths whose bytes were sha256-verified this run — avoid re-hashing 70 MB
137
+ # per load(). Keyed by resolved PATH (not logical key) so a re-resolve to a different file re-verifies.
138
+ _VERIFIED: set[str] = set()
139
+
140
+
141
+ class ArtifactIntegrityError(RuntimeError):
142
+ """A fetched artifact failed sha256 verification — corrupt/tampered bytes. NEVER swallow this:
143
+ callers that otherwise degrade gracefully on a missing artifact must still hard-fail on a mismatch."""
144
+
145
+
146
+ def _source_path(key: str) -> str:
147
+ """Absolute path to the committed dev source for `key` (resolved against the repo ROOT)."""
148
+ return os.path.join(ROOT, MANIFEST[key]["source"])
149
+
150
+
151
+ def _hf_token() -> str | None:
152
+ """Resolve an HF token. Uses huggingface_hub's standard resolution first — HF_TOKEN env, else the
153
+ `hf auth login` cached session — then the legacy langchain-style env vars as a last resort (so they
154
+ can't shadow an interactive CLI login). Lazy hf import keeps `import juurio.bootstrap` hf-free."""
155
+ from huggingface_hub import get_token
156
+ tok = get_token()
157
+ if tok:
158
+ return tok
159
+ return next((os.environ[k] for k in _LEGACY_TOKEN_ENVS if os.environ.get(k)), None)
160
+
161
+
162
+ def _offline() -> bool:
163
+ return os.environ.get(_OFFLINE_ENV, "").strip().lower() in ("1", "true", "yes", "on")
164
+
165
+
166
+ def _cache_dir() -> str:
167
+ return os.environ.get(_CACHE_ENV) or os.path.join(os.path.expanduser("~"), ".cache", "juurio")
168
+
169
+
170
+ def _remote_url(key: str, base_url: str) -> str:
171
+ """The self-host mirror URL for `key` — mirrors the clean HF tree (`<base>/<hf>`)."""
172
+ return base_url.rstrip("/") + "/" + MANIFEST[key]["hf"]
173
+
174
+
175
+ def _sha256(path: str) -> str:
176
+ h = hashlib.sha256()
177
+ with open(path, "rb") as f:
178
+ for chunk in iter(lambda: f.read(1 << 20), b""):
179
+ h.update(chunk)
180
+ return h.hexdigest()
181
+
182
+
183
+ def _verify_source(key: str) -> str:
184
+ """'ok' | 'missing' | 'size-mismatch' | 'sha-mismatch' for the committed dev source (ROOT/<source>).
185
+ Guards manifest drift: a regenerated artifact whose manifest sha/size wasn't bumped fails here."""
186
+ p = _source_path(key)
187
+ meta = MANIFEST[key]
188
+ if not os.path.exists(p):
189
+ return "missing"
190
+ if os.path.getsize(p) != meta["size"]:
191
+ return "size-mismatch"
192
+ return "ok" if _sha256(p) == meta["sha256"] else "sha-mismatch"
193
+
194
+
195
+ def _fetch_hf(key: str) -> str:
196
+ """Download MANIFEST[key]['hf'] from HF_REPO@_HF_REVISION into the HF cache; return its path.
197
+ huggingface_hub is imported HERE (lazily) so `import juurio` never requires it for dev/source use."""
198
+ from huggingface_hub import hf_hub_download
199
+ return hf_hub_download(repo_id=HF_REPO, filename=MANIFEST[key]["hf"], revision=_HF_REVISION,
200
+ token=_hf_token(), local_files_only=_offline())
201
+
202
+
203
+ def _baseurl_dest(key: str) -> str:
204
+ """Local cache path a baseurl-mirror download of `key` lands at."""
205
+ return os.path.join(_cache_dir(), MANIFEST[key]["hf"])
206
+
207
+
208
+ def _download_baseurl(key: str) -> str:
209
+ """Mirror-download `key` from `JUURIO_CKPT_BASEURL/<hf>` into the juurio cache dir, sha-verified.
210
+ Reuses an already-verified cache file instead of re-downloading on every load."""
211
+ base = os.environ[_BASEURL_ENV]
212
+ meta = MANIFEST[key]
213
+ dest = _baseurl_dest(key)
214
+ if os.path.exists(dest) and _sha256(dest) == meta["sha256"]:
215
+ return dest
216
+ url = _remote_url(key, base)
217
+ os.makedirs(os.path.dirname(dest), exist_ok=True)
218
+ tmp = dest + ".part"
219
+ urllib.request.urlretrieve(url, tmp)
220
+ got = _sha256(tmp)
221
+ if got != meta["sha256"]:
222
+ os.remove(tmp)
223
+ raise ArtifactIntegrityError(
224
+ f"artifact {key!r} sha256 mismatch from {url} (got {got[:12]}…, want {meta['sha256'][:12]}…)")
225
+ os.replace(tmp, dest)
226
+ return dest
227
+
228
+
229
+ def artifact_path(key: str) -> str:
230
+ """Resolve serving artifact `key` to an absolute local path (see module docstring for precedence).
231
+ Downloads (HF / baseurl) are sha256-verified once per process; the dev-source path is trusted."""
232
+ if key not in MANIFEST:
233
+ raise KeyError(f"unknown artifact {key!r}; known: {sorted(MANIFEST)}")
234
+ meta = MANIFEST[key]
235
+ src = _source_path(key)
236
+ if os.path.exists(src):
237
+ return src # dev/source tree is authoritative (trusted, no runtime re-hash)
238
+ if os.environ.get(_BASEURL_ENV):
239
+ dest = _baseurl_dest(key)
240
+ if dest in _VERIFIED: # already verified this run — skip re-hashing the cache file
241
+ return dest
242
+ path = _download_baseurl(key) # verifies sha internally (reuses verified cache)
243
+ _VERIFIED.add(path)
244
+ return path
245
+ try:
246
+ path = _fetch_hf(key)
247
+ except Exception as e:
248
+ offline = " (JUURIO_OFFLINE set — not in local hf cache)" if _offline() else ""
249
+ raise RuntimeError(
250
+ f"could not fetch artifact {key!r} ({meta['hf']}) from {HF_REPO}@{_HF_REVISION}{offline}: {e}. "
251
+ f"Run `juurio setup`, authenticate with `hf auth login` (or set HF_TOKEN), "
252
+ f"or set {_BASEURL_ENV} to a mirror."
253
+ ) from e
254
+ if path not in _VERIFIED:
255
+ got = _sha256(path)
256
+ if got != meta["sha256"]:
257
+ raise ArtifactIntegrityError(
258
+ f"artifact {key!r} sha256 mismatch (got {got[:12]}…, want {meta['sha256'][:12]}…) "
259
+ f"from {HF_REPO}@{_HF_REVISION}:{meta['hf']} — refusing to load.")
260
+ _VERIFIED.add(path)
261
+ return path
262
+
263
+
264
+ def _located(key: str) -> tuple[str, str]:
265
+ """(location, status) for `key` WITHOUT downloading — for read-only diagnostics. location is one of
266
+ 'dev source' / 'mirror cache' / 'HF cache' / 'HF'; status is 'ok'|'missing'|'size-/sha-mismatch'."""
267
+ s = _verify_source(key)
268
+ if s != "missing":
269
+ return "dev source", s
270
+ if os.environ.get(_BASEURL_ENV):
271
+ # Baseurl active -> the resolver uses ONLY the mirror; do not fall through to the HF cache
272
+ # (that would let doctor report READY while artifact_path tries the mirror and fails).
273
+ dest = _baseurl_dest(key)
274
+ if os.path.exists(dest):
275
+ return "mirror cache", ("ok" if _sha256(dest) == MANIFEST[key]["sha256"] else "sha-mismatch")
276
+ return "mirror", "missing"
277
+ if _hf_cached(key):
278
+ return "HF cache", "ok"
279
+ return "HF", "missing"
280
+
281
+
282
+ def _hf_cached(key: str) -> bool:
283
+ """True if `key` is already in the local HF cache (no network). For read-only diagnostics."""
284
+ try:
285
+ from huggingface_hub import hf_hub_download
286
+ hf_hub_download(repo_id=HF_REPO, filename=MANIFEST[key]["hf"], revision=_HF_REVISION,
287
+ token=_hf_token(), local_files_only=True)
288
+ return True
289
+ except Exception:
290
+ return False
291
+
292
+
293
+ def _dep_versions(deps=_RUNTIME_DEPS) -> dict:
294
+ out = {}
295
+ for d in deps:
296
+ try:
297
+ out[d] = _md.version(d)
298
+ except Exception:
299
+ out[d] = None
300
+ return out
301
+
302
+
303
+ def doctor() -> bool:
304
+ """Read-only diagnostics (never downloads). Returns True iff everything needed to SERVE is present.
305
+ Each artifact is reported by where it resolves from: `dev source` (+ sha status) or `HF cache`."""
306
+ ok = True
307
+ print("juurio doctor\n" + "=" * 60)
308
+
309
+ print("\nRuntime Python deps (serving; floor pins in pyproject):")
310
+ for d, v in _dep_versions().items():
311
+ mark = "ok " if v else "MISSING"
312
+ ok = ok and bool(v)
313
+ print(f" [{mark}] {d:16s} {v or '-- run: uv sync'}")
314
+
315
+ print(f"\nServing artifacts (dev source, else HF cache {HF_REPO}@{_HF_REVISION}):")
316
+ for key, meta in MANIFEST.items():
317
+ loc, st = _located(key)
318
+ ok = ok and st == "ok"
319
+ flag = "ok " if st == "ok" else st.upper()
320
+ hint = "" if st == "ok" else " -- run `juurio setup`"
321
+ print(f" [{flag:>13s}] {key:18s} {loc:10s} {meta['hf']:34s} {meta['what']}{hint}")
322
+
323
+ print("\n" + ("READY — `juurio demo` / `juurio validate` should pass." if ok
324
+ else "NOT READY — run `juurio setup` (and `uv sync`)."))
325
+ return ok
326
+
327
+
328
+ def setup() -> bool:
329
+ """Acquire serving artifacts. Dev sources present + verified are used as-is; otherwise each artifact
330
+ is fetched from the baseurl mirror or HuggingFace and sha256-verified."""
331
+ print("juurio setup\n" + "=" * 60)
332
+
333
+ all_ok = True
334
+ for key, meta in MANIFEST.items():
335
+ s = _verify_source(key)
336
+ if s == "ok":
337
+ print(f" [ok] {key}: dev source present + verified ({meta['source']})")
338
+ continue
339
+ if s in ("size-mismatch", "sha-mismatch"):
340
+ all_ok = False
341
+ print(f" [WARN] {key}: dev source {meta['source']} {s} vs manifest — bump MANIFEST or rebuild")
342
+ continue
343
+ try:
344
+ path = artifact_path(key)
345
+ print(f" [ok] {key}: fetched + verified -> {path}")
346
+ except ArtifactIntegrityError:
347
+ raise # corrupt/tampered bytes — hard-fail, never fall back to [MISSING]
348
+ except Exception as e:
349
+ all_ok = False
350
+ print(f" [MISSING] {key}: {e}")
351
+
352
+ print("\n" + ("setup complete." if all_ok else "setup incomplete — see notes above."))
353
+ return all_ok
juurio/cli.py ADDED
@@ -0,0 +1,473 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """juurio command-line interface.
2
+
3
+ juurio doctor # read-only env diagnostics
4
+ juurio setup # acquire parser + model checkpoints
5
+ juurio list-ops # documented op catalog
6
+
7
+ # one command per library function (same names):
8
+ juurio transform <op> "<text>" [--lemma L] [--trace] [--json] # -> transform()
9
+ juurio inflect "<word>" [spec…] [--case C] [--number N] [--degree D] [--px P] [--clitic K] [--pos P] [--like W] [--proper] [--json] # -> inflect()
10
+ juurio analyze "<text>" [--json] # -> analyze()
11
+ juurio lemmatize "<text>" [--json] # -> lemmatize()
12
+ juurio realize-np "<np>" [--case Ine] [--number Plur] [--px 1Sg] [--clitic kin] [--json] # -> realize_np()
13
+ juurio realize-verbform "<verb>" [--form va|nut|tava|tu|inf2|inf2_pass|inf3|ma] [--case Ine] [--number Plur] [--px 3] # -> realize_verbform()
14
+
15
+ juurio batch <op> [--lemma L] [--json] < lines # one op over many lines (stdin)
16
+ juurio batch --json < tsv # per-line "op<TAB>text[<TAB>lemma]"
17
+ juurio demo # curated tour (incl. OOV) + smoke test
18
+ juurio repl # interactive shell (inflect/analyze/transform/...)
19
+ juurio bench # cold/warm latency numbers
20
+
21
+ Exit status: 0 on success; 2 if any transform abstained (a normal, documented outcome — useful for
22
+ scripting); 1 on a setup failure. `--json` emits one object per line (text/ok/reason/
23
+ warnings/trace). Also reachable as `python -m juurio ...`.
24
+ """
25
+ from __future__ import annotations
26
+
27
+ import argparse
28
+ import json
29
+ import sys
30
+
31
+ from . import OP_CATALOG, SUPPORTED_OPS
32
+
33
+ # A curated tour that doubles as a smoke test. Every entry is expected to succeed (ok=True).
34
+ _DEMO = [
35
+ ("past", "Koira haukkuu pihalla.", None), ("passive", "Mies osti talon.", None),
36
+ ("neg", "Lapsi nukkuu sängyssä.", None), ("quest", "Liisa tuli.", None),
37
+ ("imperative", "Sinä luet kirjan.", "lukea"),
38
+ ("past", "Nuori striimaa peliä.", "striimata"), # OOV verb
39
+ ("potential", "Nuori striimaa peliä.", "striimata"), # OOV verb, new mood
40
+ ("plural", "Striimaaja nukkuu.", None), # OOV noun head
41
+ ("poss_1sg", "Striimaaja nukkuu.", None), # OOV + possessive
42
+ ("plural+poss_1sg+clitic_kin", "Koira haukkuu pihalla.", None), # combined nominal morphology
43
+ ]
44
+ _DEMO_NP = [
45
+ ("talo", dict(case="Ine", number="Plur", px="1Sg")),
46
+ ("iso talo", dict(case="Ine", number="Plur")), # modifier agreement
47
+ ("striimaaja", dict(case="Ade", number="Plur", px="1Sg", clitic="kin")), # OOV
48
+ ("bugisovellus", dict(case="Ine", number="Sing", px="1Pl", clitic="kin")), # OOV compound loan
49
+ ]
50
+ _DEMO_VF = [
51
+ ("lukea", dict(form="va", case="Ine")), # known participle, inflected
52
+ ("mennä", dict(form="inf3", case="Ade")), # known 3rd infinitive
53
+ ("tehdä", dict(form="inf2_pass")), # known passive 2nd inf (tehtäessä)
54
+ ("kuulla", dict(form="tu", case="Par", px="3")), # temporal construction (kuultuansa)
55
+ ("striimata", dict(form="va", lemma="striimata")), # OOV participle (neural)
56
+ ("striimata", dict(form="inf3", case="Ine", lemma="striimata")), # OOV infinitive (neural)
57
+ ]
58
+
59
+
60
+ def _print_catalog() -> None:
61
+ for group, ops in OP_CATALOG.items():
62
+ print(f"\n{group}:")
63
+ for op, desc in ops.items():
64
+ print(f" {op:24s} {desc}")
65
+ print(f"\n{len(SUPPORTED_OPS)} ops total. Compose nominal ops with '+' (plural+poss_1sg+clitic_kin). "
66
+ "Out-of-scope / inapplicable input abstains (ok=False, reason in ABSTAIN_REASONS).")
67
+
68
+
69
+ def _emit(op: str, text: str, r, as_json: bool, trace: bool = False) -> None:
70
+ if as_json:
71
+ print(json.dumps({"op": op, "input": text, "text": r.text, "ok": r.ok,
72
+ "reason": r.reason, "warnings": r.warnings, "trace": r.trace},
73
+ ensure_ascii=False))
74
+ return
75
+ if r.ok:
76
+ tag = " ⚠ " + "; ".join(r.warnings) if r.warnings else ""
77
+ print(f"{r.text}{tag}")
78
+ else:
79
+ print(f"[abstain: {r.reason}] {text}", file=sys.stderr)
80
+ if trace:
81
+ for k, v in r.trace.items():
82
+ print(f" · {k}: {v}", file=sys.stderr)
83
+
84
+
85
+ def _bench() -> int:
86
+ import time
87
+ t0 = time.time()
88
+ from .pipeline import Transformer
89
+ t = Transformer()
90
+ t1 = time.time()
91
+ t.transform("past", "Koira haukkuu pihalla.") # first parse loads Stanza (cold)
92
+ t2 = time.time()
93
+ for _ in range(20):
94
+ t.transform("past", "Lapsi lukee kirjaa.")
95
+ t3 = time.time()
96
+ for _ in range(20):
97
+ t.realize_np("talo", case="Ine", number="Plur", px="1Sg")
98
+ t4 = time.time()
99
+ print("juurio bench (this machine)\n" + "=" * 40)
100
+ print(f" Transformer() construct : {t1 - t0:5.2f} s (models; Stanza loads lazily)")
101
+ print(f" first transform (cold) : {t2 - t1:5.2f} s (Stanza `fi` pipeline load)")
102
+ print(f" transform (warm, avg) : {(t3 - t2) / 20 * 1000:5.1f} ms")
103
+ print(f" realize_np (warm, avg) : {(t4 - t3) / 20 * 1000:5.1f} ms")
104
+ return 0
105
+
106
+
107
+ def _demo(t=None) -> int:
108
+ if t is None:
109
+ from .pipeline import Transformer
110
+ t = Transformer()
111
+ bad = 0
112
+ print("clause transforms (transform):")
113
+ for op, text, lemma in _DEMO:
114
+ r = t.transform(op, text, lemma=lemma)
115
+ oov = " [OOV→neural]" if any("neural" in w for w in r.warnings) else ""
116
+ print(f" {op:28s} {text!r:34s} -> {r.text!r}{oov}")
117
+ bad += not r.ok
118
+ print("\nphrase morphology (realize_np):")
119
+ for np, kw in _DEMO_NP:
120
+ r = t.realize_np(np, **kw)
121
+ oov = " [OOV→neural]" if any("neural" in w for w in r.warnings) else ""
122
+ print(f" {np!r:18s} {kw} -> {r.text!r}{oov}")
123
+ bad += not r.ok
124
+ print("\nverb morphology (realize_verbform):")
125
+ for verb, kw in _DEMO_VF:
126
+ r = t.realize_verbform(verb, **kw)
127
+ oov = " [OOV→neural]" if any("neural" in w for w in r.warnings) else ""
128
+ print(f" {verb!r:14s} {kw} -> {r.text!r}{oov}")
129
+ bad += not r.ok
130
+ print("\n" + ("demo OK" if not bad else f"demo FAILED: {bad} unexpected abstain(s)"))
131
+ return 0 if not bad else 1
132
+
133
+
134
+ _REPL_HELP = """commands (model is loaded once; every command reuses it):
135
+ transform <op> <text> clause transform e.g. transform passive Mies osti talon.
136
+ inflect <word> [spec] [k=v] inflect a nominal e.g. inflect talo mon gen
137
+ spec is a human case/number/degree phrase ('gen', 'mon gen', 'plural genitive comparative');
138
+ options as key=value: px=1sg clitic=kin proper=true pos=adjective like=talojen
139
+ np <phrase> [k=v] phrase nominal morphology e.g. np iso talo case=Ine number=Plur
140
+ options: case=Ine number=Plur px=1Sg clitic=kin
141
+ verb <verb> [k=v] non-finite verb form e.g. verb kuulla form=tu case=Par px=3
142
+ options: form=va|nut|tava|tu|inf2|inf2_pass|inf3|ma case= number= px= lemma=
143
+ analyze <text> dependency + morphology analysis (the inverse of generation)
144
+ lemmatize <text> lemmas only
145
+ ops list the transform ops
146
+ help this help quit | exit leave
147
+
148
+ examples:
149
+ inflect talo mon gen -> talojen
150
+ inflect Meri gen proper=true -> proper-noun reading (vs 'meri' -> meren)
151
+ inflect makaronilaatikko like=talojen -> makaronilaatikkojen (copy a model word's cell)
152
+ inflect kaunis gen degree=comp -> kauniimman (adjective comparative)
153
+ transform past Koira haukkuu pihalla.
154
+ analyze Koirat haukkuvat pihalla.
155
+ """
156
+
157
+
158
+ def _parse_opts(tokens: list[str]) -> tuple[list[str], dict]:
159
+ """Split REPL tokens into (positional words, {key: value}) by the presence of '='."""
160
+ words, opts = [], {}
161
+ for tok in tokens:
162
+ if "=" in tok:
163
+ k, v = tok.split("=", 1)
164
+ opts[k] = v
165
+ else:
166
+ words.append(tok)
167
+ return words, opts
168
+
169
+
170
+ def _inflect_kwargs(spec_words, *, case=None, number=None, degree=None,
171
+ px=None, clitic=None, pos=None, like=None, proper=None) -> dict:
172
+ """Map a positional case/number/degree phrase + named options onto `inflect()` kwargs.
173
+
174
+ A positional spec ('mon gen') wins over an explicit case= (mirrors the REPL). None-valued options
175
+ are dropped so `inflect()`'s own defaults apply. Shared by the `inflect` CLI command and the REPL."""
176
+ kw = {}
177
+ spec = " ".join(spec_words)
178
+ if spec:
179
+ kw["case"] = spec
180
+ elif case:
181
+ kw["case"] = case
182
+ for k, v in (("number", number), ("degree", degree), ("px", px),
183
+ ("clitic", clitic), ("pos", pos), ("like", like)):
184
+ if v is not None:
185
+ kw[k] = v
186
+ if proper is not None:
187
+ kw["proper"] = proper
188
+ return kw
189
+
190
+
191
+ def _truthy(v: str) -> bool:
192
+ s = v.strip().lower()
193
+ if s in ("1", "true", "t", "yes", "y", "on"):
194
+ return True
195
+ if s in ("0", "false", "f", "no", "n", "off"):
196
+ return False
197
+ raise ValueError(f"expected a boolean for proper= (true/false), got {v!r}")
198
+
199
+
200
+ def _print_analysis(sents) -> None:
201
+ for si, sent in enumerate(sents):
202
+ if len(sents) > 1:
203
+ print(f"sentence {si + 1}:")
204
+ print(f" {'id':>2} {'form':<16}{'lemma':<16}{'upos':<7}{'head':>4} {'deprel':<11}feats")
205
+ for tk in sent:
206
+ feats = ",".join(f"{k}={v}" for k, v in tk.feats.items())
207
+ print(f" {tk.id:>2} {tk.form:<16}{tk.lemma:<16}{tk.upos:<7}{tk.head:>4} "
208
+ f"{tk.deprel:<11}{feats}")
209
+
210
+
211
+ def _repl_dispatch(t, line: str) -> bool:
212
+ """Execute one REPL line against the (already-loaded) Transformer `t`. Returns False to signal the
213
+ loop should stop (quit/exit/q), True to keep going. A bad alias/op is reported, not fatal."""
214
+ cmd, *rest = line.split()
215
+ arg = line[len(cmd):].strip() # raw remainder (for free-text commands)
216
+ cmd = cmd.lower()
217
+ try:
218
+ if cmd in ("quit", "exit", "q"):
219
+ return False
220
+ elif cmd in ("help", "h", "?"):
221
+ print(_REPL_HELP)
222
+ elif cmd == "ops":
223
+ _print_catalog()
224
+ elif cmd in ("transform", "t"):
225
+ op, _, text = arg.partition(" ")
226
+ if not text:
227
+ print("usage: transform <op> <text>", file=sys.stderr)
228
+ return True
229
+ _emit(op, text, t.transform(op, text), False, trace=True)
230
+ elif cmd in ("inflect", "i"):
231
+ words, opts = _parse_opts(rest)
232
+ if not words:
233
+ print("usage: inflect <word> [spec] [px= clitic= proper= pos= like=]",
234
+ file=sys.stderr)
235
+ return True
236
+ kw = _inflect_kwargs(words[1:], case=opts.get("case"), number=opts.get("number"),
237
+ degree=opts.get("degree"), px=opts.get("px"),
238
+ clitic=opts.get("clitic"), pos=opts.get("pos"), like=opts.get("like"),
239
+ proper=_truthy(opts["proper"]) if "proper" in opts else None)
240
+ _emit("inflect", words[0], t.inflect(words[0], **kw), False, trace=True)
241
+ elif cmd == "np":
242
+ words, opts = _parse_opts(rest)
243
+ phrase = " ".join(words)
244
+ r = t.realize_np(phrase, case=opts.get("case", "Nom"),
245
+ number=opts.get("number", "Sing"),
246
+ px=opts.get("px", "None"), clitic=opts.get("clitic"))
247
+ _emit("realize-np", phrase, r, False, trace=True)
248
+ elif cmd == "verb":
249
+ words, opts = _parse_opts(rest)
250
+ verb = " ".join(words)
251
+ r = t.realize_verbform(verb, form=opts.get("form", "va"),
252
+ case=opts.get("case", "Nom"),
253
+ number=opts.get("number", "Sing"),
254
+ px=opts.get("px", "None"), lemma=opts.get("lemma"))
255
+ _emit(f"verbform:{opts.get('form', 'va')}", verb, r, False, trace=True)
256
+ elif cmd in ("analyze", "a"):
257
+ if not arg:
258
+ print("usage: analyze <text>", file=sys.stderr)
259
+ return True
260
+ _print_analysis(t.analyze(arg))
261
+ elif cmd in ("lemmatize", "lem"):
262
+ print(" ".join(t.lemmatize(arg)) if arg else "usage: lemmatize <text>")
263
+ else:
264
+ print(f"unknown command {cmd!r}; type 'help'", file=sys.stderr)
265
+ except ValueError as e: # unknown alias / unknown op / unsupported pos — keep the loop alive
266
+ print(f"error: {e}", file=sys.stderr)
267
+ return True
268
+
269
+
270
+ def _repl() -> int:
271
+ """Interactive read-eval loop exercising every public feature, with the model loaded once."""
272
+ print("loading model (parser + realizers)…", file=sys.stderr)
273
+ from .pipeline import Transformer
274
+ t = Transformer()
275
+ print("juurio interactive shell — type 'help' for commands, 'quit' to exit.")
276
+ while True:
277
+ try:
278
+ line = input("juurio> ").strip()
279
+ except (EOFError, KeyboardInterrupt):
280
+ print()
281
+ return 0
282
+ if not line:
283
+ continue
284
+ if not _repl_dispatch(t, line):
285
+ return 0
286
+ return 0
287
+
288
+
289
+ def _build_parser() -> argparse.ArgumentParser:
290
+ p = argparse.ArgumentParser(prog="juurio", description=__doc__,
291
+ formatter_class=argparse.RawDescriptionHelpFormatter)
292
+ p.add_argument("--version", action="version", version=f"juurio {_version()}")
293
+ sub = p.add_subparsers(dest="cmd", required=True)
294
+
295
+ sub.add_parser("doctor", help="read-only environment diagnostics")
296
+ sub.add_parser("setup", help="acquire parser + model checkpoints")
297
+ sub.add_parser("list-ops", help="print the documented op catalog")
298
+ sub.add_parser("demo", help="curated tour (incl. OOV) — doubles as a smoke test")
299
+ sub.add_parser("repl", help="interactive shell exercising every feature (model loaded once)")
300
+ sub.add_parser("bench", help="cold/warm latency numbers")
301
+
302
+ # Top-level commands mirror the library API one-for-one: transform/inflect/analyze/lemmatize +
303
+ # realize-np/realize-verbform (hyphenated). doctor/setup/list-ops/demo/repl/bench/batch are
304
+ # infrastructure-only and have no library counterpart.
305
+ pr = sub.add_parser("transform", help="clause transform (-> transform())")
306
+ pr.add_argument("op", metavar="OP",
307
+ help="an op from `list-ops`, or a '+'-composed nominal bundle "
308
+ "e.g. 'plural+poss_1sg+clitic_kin'")
309
+ pr.add_argument("text")
310
+ pr.add_argument("--lemma", help="override the parser lemma for the main verb")
311
+ pr.add_argument("--trace", action="store_true", help="print the Result.trace fields")
312
+ pr.add_argument("--json", action="store_true")
313
+
314
+ pi = sub.add_parser("inflect", help="inflect a single nominal (-> inflect())")
315
+ pi.add_argument("word", help="a bare lemma (or model word), e.g. 'talo'")
316
+ pi.add_argument("spec", nargs="*",
317
+ help="human case/number/degree phrase, e.g. 'mon gen' or 'plural genitive "
318
+ "comparative' (overrides --case/--number/--degree)")
319
+ pi.add_argument("--case", help="UD or human case (e.g. Gen / gen / genitive)")
320
+ pi.add_argument("--number", help="Sing/Plur (or 'yks'/'mon')")
321
+ pi.add_argument("--degree", help="pos/comp/sup (adjectives)")
322
+ pi.add_argument("--px", help="possessive suffix (1Sg/2Sg/3/1Pl/2Pl)")
323
+ pi.add_argument("--clitic", help="enclitic (kin/kaan/ko/han/pa)")
324
+ pi.add_argument("--pos", help="force a part-of-speech reading (e.g. adjective)")
325
+ pi.add_argument("--like", help="copy a model word's inflected cell, e.g. like=talojen")
326
+ pi.add_argument("--proper", action=argparse.BooleanOptionalAction, default=None,
327
+ help="force (or with --no-proper, forbid) a proper-noun reading")
328
+ pi.add_argument("--trace", action="store_true")
329
+ pi.add_argument("--json", action="store_true")
330
+
331
+ pa = sub.add_parser("analyze", help="dependency + morphology analysis (-> analyze())")
332
+ pa.add_argument("text")
333
+ pa.add_argument("--json", action="store_true")
334
+
335
+ pl = sub.add_parser("lemmatize", help="lemmas only (-> lemmatize())")
336
+ pl.add_argument("text")
337
+ pl.add_argument("--json", action="store_true")
338
+
339
+ pn = sub.add_parser("realize-np", help="phrase-level nominal morphology (NOT a clause transform)")
340
+ pn.add_argument("np", help="a bare lemma or short 'adj* noun' phrase, e.g. 'iso talo'")
341
+ pn.add_argument("--case", default="Nom", help="UD case (Nom/Gen/Par/Ine/Ela/Ill/Ade/Abl/All/...)")
342
+ pn.add_argument("--number", default="Sing", choices=["Sing", "Plur"])
343
+ pn.add_argument("--px", default="None", choices=["None", "1Sg", "2Sg", "3", "1Pl", "2Pl"],
344
+ help="possessive suffix")
345
+ pn.add_argument("--clitic", default=None, choices=["kin", "kaan", "ko", "han", "pa"])
346
+ pn.add_argument("--trace", action="store_true")
347
+ pn.add_argument("--json", action="store_true")
348
+
349
+ pv = sub.add_parser("realize-verbform",
350
+ help="non-finite verb morphology: participle/infinitive (NOT a clause transform)")
351
+ pv.add_argument("verb", help="a verb lemma or short clause, e.g. 'lukea' or 'Hän lukee kirjaa.'")
352
+ pv.add_argument("--form", default="va",
353
+ choices=["va", "nut", "tava", "tu", "inf2", "inf2_pass", "inf3", "ma"],
354
+ help="va/nut/tava/tu = present/past active/passive participle; inf2/inf3 = 2nd/3rd "
355
+ "infinitive; inf2_pass = passive 2nd inf (tehtäessä); ma = agent participle (tekemä)")
356
+ pv.add_argument("--case", default="Nom", help="UD case (participles: any; inf3: Ine/Ela/Ill/Ade/Abe)")
357
+ pv.add_argument("--number", default="Sing", choices=["Sing", "Plur"])
358
+ pv.add_argument("--px", default="None", choices=["None", "1Sg", "2Sg", "3", "1Pl", "2Pl"],
359
+ help="possessive suffix (temporal/possessive construction): tu+Par+3 -> kuultuansa")
360
+ pv.add_argument("--lemma", help="override the parser lemma (for a mis-lemmatized loan verb)")
361
+ pv.add_argument("--trace", action="store_true")
362
+ pv.add_argument("--json", action="store_true")
363
+
364
+ pb = sub.add_parser("batch", help="transform many lines from stdin")
365
+ pb.add_argument("op", nargs="?", metavar="OP",
366
+ help="op (or '+'-composed nominal bundle) for every line; omit to read "
367
+ "per-line 'op<TAB>text[<TAB>lemma]'")
368
+ pb.add_argument("--lemma", help="lemma override (single-op mode only)")
369
+ pb.add_argument("--json", action="store_true")
370
+ return p
371
+
372
+
373
+ def main(argv: list[str] | None = None) -> int:
374
+ args = _build_parser().parse_args(argv)
375
+
376
+ # commands that don't need the (heavy) Transformer
377
+ if args.cmd == "doctor":
378
+ from . import bootstrap
379
+ return 0 if bootstrap.doctor() else 1
380
+ if args.cmd == "setup":
381
+ from . import bootstrap
382
+ return 0 if bootstrap.setup() else 1
383
+ if args.cmd == "list-ops":
384
+ _print_catalog()
385
+ return 0
386
+ if args.cmd == "bench":
387
+ return _bench()
388
+ if args.cmd == "demo":
389
+ return _demo()
390
+ if args.cmd == "repl":
391
+ return _repl()
392
+
393
+ from .pipeline import Transformer
394
+ from . import algebra
395
+ t = Transformer()
396
+ any_abstain = False
397
+
398
+ def known(op):
399
+ return op in SUPPORTED_OPS or algebra.parse_nominal_bundle(op) is not None
400
+
401
+ if args.cmd == "transform":
402
+ r = t.transform(args.op, args.text, lemma=args.lemma)
403
+ _emit(args.op, args.text, r, args.json, trace=args.trace)
404
+ return 2 if not r.ok else 0
405
+
406
+ if args.cmd == "inflect":
407
+ kw = _inflect_kwargs(args.spec, case=args.case, number=args.number, degree=args.degree,
408
+ px=args.px, clitic=args.clitic, pos=args.pos, like=args.like,
409
+ proper=args.proper)
410
+ r = t.inflect(args.word, **kw)
411
+ _emit("inflect", args.word, r, args.json, trace=args.trace)
412
+ return 2 if not r.ok else 0
413
+
414
+ if args.cmd == "analyze":
415
+ sents = t.analyze(args.text)
416
+ if args.json:
417
+ print(json.dumps([[{"id": tk.id, "form": tk.form, "lemma": tk.lemma, "upos": tk.upos,
418
+ "head": tk.head, "deprel": tk.deprel, "feats": tk.feats}
419
+ for tk in s] for s in sents], ensure_ascii=False))
420
+ else:
421
+ _print_analysis(sents)
422
+ return 0
423
+
424
+ if args.cmd == "lemmatize":
425
+ lemmas = t.lemmatize(args.text)
426
+ print(json.dumps(lemmas, ensure_ascii=False) if args.json else " ".join(lemmas))
427
+ return 0
428
+
429
+ if args.cmd == "realize-np":
430
+ r = t.realize_np(args.np, case=args.case, number=args.number, px=args.px, clitic=args.clitic)
431
+ _emit("realize-np", args.np, r, args.json, trace=args.trace)
432
+ return 2 if not r.ok else 0
433
+
434
+ if args.cmd == "realize-verbform":
435
+ r = t.realize_verbform(args.verb, form=args.form, case=args.case, number=args.number,
436
+ px=args.px, lemma=args.lemma)
437
+ _emit(f"verbform:{args.form}", args.verb, r, args.json, trace=args.trace)
438
+ return 2 if not r.ok else 0
439
+
440
+ for line in sys.stdin:
441
+ line = line.rstrip("\n")
442
+ if not line.strip():
443
+ continue
444
+ if args.op: # single op for every line
445
+ op, text, lemma = args.op, line, args.lemma
446
+ else: # per-line "op<TAB>text[<TAB>lemma]"
447
+ parts = line.split("\t")
448
+ if len(parts) < 2:
449
+ print(f"[skip: need 'op<TAB>text'] {line}", file=sys.stderr)
450
+ any_abstain = True
451
+ continue
452
+ op, text = parts[0], parts[1]
453
+ lemma = parts[2] if len(parts) > 2 else None
454
+ if not known(op):
455
+ print(f"[skip: unknown op {op!r}] {line}", file=sys.stderr)
456
+ any_abstain = True
457
+ continue
458
+ r = t.transform(op, text, lemma=lemma)
459
+ _emit(op, text, r, args.json)
460
+ any_abstain = any_abstain or not r.ok
461
+ return 2 if any_abstain else 0
462
+
463
+
464
+ def _version() -> str:
465
+ import importlib.metadata as md
466
+ try:
467
+ return md.version("juurio")
468
+ except Exception:
469
+ return "0.1.0"
470
+
471
+
472
+ if __name__ == "__main__":
473
+ sys.exit(main())
juurio/copyrealizer.py ADDED
@@ -0,0 +1,162 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Pointer-generator char enc-dec — the copy mechanism.
2
+
3
+ The baseline net's exceptions are not morphology errors but STEM-COPYING failures: it corrupts
4
+ rare/long stem characters (abortoida->va: 'auortoiva' not 'abortoiva') while the affix is right. A
5
+ copy channel fixes exactly this: at each step the decoder may COPY a character from the source (the
6
+ lemma) instead of generating it from the vocab.
7
+
8
+ Mechanism (See et al. 2017, pointer-generator, char-level):
9
+ gen = softmax(proj(dec_out)) generation distribution over vocab
10
+ alpha = softmax(copy_q(dec_out) · copy_k(mem_src)) attention over SOURCE char positions
11
+ copy = scatter_add(alpha onto vocab by src char id)
12
+ p = sigmoid(gate(dec_out)) mix weight in [0,1]
13
+ P = p * gen + (1 - p) * copy final distribution
14
+ forward returns log P. The encoder/decoder are the SAME architecture as Realizer, bar three small
15
+ projections (copy_q, copy_k, gate).
16
+ """
17
+ from __future__ import annotations
18
+
19
+ import math
20
+
21
+ import torch
22
+ import torch.nn as nn
23
+
24
+ from .realizer import BOS, EOS, PAD, Realizer, encode_chars
25
+ from .device import DEV
26
+
27
+ _EPS = 1e-8
28
+
29
+
30
+ class CopyRealizer(Realizer):
31
+ def __init__(self, vocab, d_model=128, nhead=4, layers=2, ff=512, maxlen=64):
32
+ super().__init__(vocab, d_model, nhead, layers, ff, maxlen)
33
+ self.vocab = vocab
34
+ self.copy_q = nn.Linear(d_model, d_model)
35
+ self.copy_k = nn.Linear(d_model, d_model)
36
+ self.gate = nn.Linear(d_model, 1)
37
+ self._scale = 1.0 / math.sqrt(d_model)
38
+ self.use_copy = True # False -> pure generator (copy channel disabled)
39
+
40
+ def _pgen(self, dec_out):
41
+ """Per-step mix weight in [0,1] — how much to trust the generator vs the copy channel."""
42
+ return torch.sigmoid(self.gate(dec_out))
43
+
44
+ def _copy_dist(self, dec_out, mem, src, H):
45
+ """Copy distribution over the vocab: attention over the SOURCE char positions of memory
46
+ (mem[:, H:] — the part after the H header slots), scattered onto vocab indices by src char id.
47
+ Padded source positions are masked out, so they contribute no probability."""
48
+ B, T, _ = dec_out.shape
49
+ S = src.size(1)
50
+ mem_src = mem[:, H:, :] # [B, S, d] — source-char memory
51
+ q = self.copy_q(dec_out) # [B, T, d]
52
+ k = self.copy_k(mem_src) # [B, S, d]
53
+ scores = torch.bmm(q, k.transpose(1, 2)) * self._scale # [B, T, S]
54
+ pad = (src == PAD).unsqueeze(1) # [B, 1, S]
55
+ scores = scores.masked_fill(pad, float("-inf"))
56
+ alpha = torch.softmax(scores, dim=-1) # [B, T, S]
57
+ alpha = torch.nan_to_num(alpha, nan=0.0) # all-pad row -> zeros (no source)
58
+ pcopy = torch.zeros(B, T, self.vocab, device=dec_out.device, dtype=alpha.dtype)
59
+ idx = src.unsqueeze(1).expand(B, T, S) # [B, T, S] vocab ids to scatter onto
60
+ pcopy.scatter_add_(2, idx, alpha)
61
+ return pcopy
62
+
63
+ def _dec_out(self, src, header, tgt_in):
64
+ mem, mask = self.encode(src, header)
65
+ T = tgt_in.size(1)
66
+ cmask = torch.triu(torch.ones(T, T, device=src.device, dtype=torch.bool), 1)
67
+ dec_out = self.dec(self._tgt_embed(tgt_in), mem, tgt_mask=cmask,
68
+ tgt_key_padding_mask=(tgt_in == PAD), memory_key_padding_mask=mask)
69
+ return dec_out, mem
70
+
71
+ def forward(self, src, header, tgt_in):
72
+ """Return log P of shape [B, T, vocab] — train with NLLLoss(ignore_index=PAD). With use_copy
73
+ the mixed gen+copy distribution; without it the pure generator."""
74
+ dec_out, mem = self._dec_out(src, header, tgt_in)
75
+ if not self.use_copy:
76
+ return torch.log_softmax(self.proj(dec_out), dim=-1)
77
+ gen = torch.softmax(self.proj(dec_out), dim=-1) # [B, T, V]
78
+ copy = self._copy_dist(dec_out, mem, src, H=header.size(1))
79
+ p = self._pgen(dec_out) # [B, T, 1]
80
+ mixed = p * gen + (1.0 - p) * copy
81
+ return torch.log(mixed + _EPS)
82
+
83
+
84
+ def load(ckpt):
85
+ ck = torch.load(ckpt, map_location=DEV, weights_only=False)
86
+ m = CopyRealizer(len(ck["stoi"]), ck["cfg"]["d_model"], 4, ck["cfg"]["layers"]).to(DEV)
87
+ m.load_state_dict(ck["model"])
88
+ m.use_copy = ck["cfg"].get("use_copy", True)
89
+ m.eval()
90
+ return m, ck["stoi"], ck["itos"]
91
+
92
+
93
+ @torch.no_grad()
94
+ def decode_one(m, header, lemma, stoi, itos, maxlen=48, score=False):
95
+ """Greedy decode over the mixed gen+copy distribution. B=1. If `score`, also return a confidence =
96
+ min over steps of the top-1 margin (top1 - top2 prob) on the MIXED distribution — same uncertainty
97
+ contract as realizer._decode, just measured on the distribution the copy model actually predicts."""
98
+ m.eval()
99
+ dev = next(m.parameters()).device
100
+ src = torch.tensor([encode_chars(lemma, stoi)], dtype=torch.long, device=dev)
101
+ hdr = torch.tensor([header], dtype=torch.long, device=dev)
102
+ ids = [BOS]
103
+ out = []
104
+ margin = 1.0
105
+ for _ in range(maxlen):
106
+ ti = torch.tensor([ids], dtype=torch.long, device=dev)
107
+ logp = m(src, hdr, ti)[0, -1]
108
+ nxt = int(logp.argmax())
109
+ if score:
110
+ p = logp.float().exp() # logp is already log P (mixed), exp -> probs
111
+ top2 = torch.topk(p, 2).values
112
+ margin = min(margin, float(top2[0] - top2[1]))
113
+ if nxt == EOS:
114
+ break
115
+ out.append(nxt)
116
+ ids.append(nxt)
117
+ form = "".join(itos[t] for t in out if t in itos)
118
+ return (form, margin) if score else form
119
+
120
+
121
+ @torch.no_grad()
122
+ def decode_forms(items, bundle, batch_size=256, maxlen=48):
123
+ """Greedy decode over the mixed gen+copy distribution, batched; bit-identical to decode_one. The
124
+ prune pass MUST use this (not realizer.proj-only decode) or it would measure the generator alone.
125
+ items: iterable of (lemma, header_ids_list)."""
126
+ from .realizer import _pad
127
+ items = list(items)
128
+ if not items:
129
+ return []
130
+ m, stoi, itos = bundle
131
+ m.eval()
132
+ dev = next(m.parameters()).device
133
+ out = []
134
+ for start in range(0, len(items), batch_size):
135
+ chunk = items[start:start + batch_size]
136
+ B = len(chunk)
137
+ # respect caller casing (matches decode_one): PROPN comes in capitalized, NOUN/ADJ/verbs
138
+ # lowercase. The caller (batched/verbforms) is responsible for POS-correct casing.
139
+ src_lists = [encode_chars(lem, stoi) for lem, _ in chunk]
140
+ S = max((len(s) for s in src_lists), default=1) or 1
141
+ src = _pad(src_lists, S).to(dev)
142
+ hdr = torch.tensor([h for _, h in chunk], dtype=torch.long, device=dev)
143
+ ids = torch.full((B, 1), BOS, dtype=torch.long, device=dev)
144
+ finished = torch.zeros(B, dtype=torch.bool, device=dev)
145
+ collected = [[] for _ in range(B)]
146
+ for _ in range(maxlen):
147
+ logp = m(src, hdr, ids)[:, -1] # mixed log-probs at last position
148
+ nxt = logp.argmax(-1)
149
+ nxt_cpu = nxt.tolist()
150
+ for i in range(B):
151
+ if finished[i]:
152
+ continue
153
+ if nxt_cpu[i] == EOS:
154
+ finished[i] = True
155
+ else:
156
+ collected[i].append(nxt_cpu[i])
157
+ app = nxt.clone(); app[finished] = PAD
158
+ ids = torch.cat([ids, app.unsqueeze(1)], 1)
159
+ if bool(finished.all()):
160
+ break
161
+ out.extend("".join(itos[t] for t in row if t in itos) for row in collected)
162
+ return out
juurio/device.py ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ """The torch device string, factored out of `train` so the SERVING modules don't load the training
2
+ module (nanoGPT model + training loop) just to learn whether a GPU is present. `train.DEV` re-exports
3
+ this for back-compat. Serving needs only torch; importing this pulls in nothing else."""
4
+ from __future__ import annotations
5
+
6
+ import torch
7
+
8
+ DEV = "cuda" if torch.cuda.is_available() else "cpu"
juurio/features.py ADDED
@@ -0,0 +1,272 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Human-readable feature-alias normalization for the friendly generation API (`inflect`).
2
+
3
+ Callers should be able to say "gen"/"genitive"/"genetiivi" or a combined "mon gen" instead of the
4
+ engine's internal vocab. These helpers map every accepted alias onto the EXACT canonical strings the
5
+ engine expects (Case=Gen, Number=Plur, Degree=Cmp, Px=1Sg, …) and fail LOUD (ValueError) on an
6
+ unknown alias — an unrecognized case is a caller mistake, not data to silently coerce.
7
+
8
+ Pure module: no torch/parser/heavy deps, safe to import anywhere. The canonical vocab mirrors
9
+ `realizer.CASE_VALS/NUM_VALS/DEG_VALS/PX_VALS`."""
10
+ from __future__ import annotations
11
+
12
+ # canonical value -> accepted aliases (English abbrev/full + Finnish). The canonical string itself is
13
+ # always accepted too, case-insensitively (added in `_invert`).
14
+ _CASE = {
15
+ "Nom": ["nom", "nominative", "nominatiivi"],
16
+ "Gen": ["gen", "genitive", "genitiivi", "genetiivi"],
17
+ "Par": ["par", "ptv", "part", "partitive", "partitiivi"],
18
+ "Ess": ["ess", "essive", "essiivi"],
19
+ "Tra": ["tra", "transl", "translative", "translatiivi"],
20
+ "Ine": ["ine", "iness", "inessive", "inessiivi"],
21
+ "Ela": ["ela", "elat", "elative", "elatiivi"],
22
+ "Ill": ["ill", "illat", "illative", "illatiivi"],
23
+ "Ade": ["ade", "adess", "adessive", "adessiivi"],
24
+ "Abl": ["abl", "ablat", "ablative", "ablatiivi"],
25
+ "All": ["all", "allat", "allative", "allatiivi"],
26
+ }
27
+ _NUMBER = {
28
+ "Sing": ["sing", "sg", "singular", "yks", "yksikko", "yksikkö"],
29
+ "Plur": ["plur", "pl", "plural", "mon", "monikko"],
30
+ }
31
+ _DEGREE = {
32
+ "Pos": ["pos", "positive", "perusaste"],
33
+ "Cmp": ["comp", "cmp", "comparative", "komparatiivi", "vertailuaste"],
34
+ "Sup": ["sup", "super", "superl", "superlative", "superlatiivi", "yliaste"],
35
+ }
36
+ _PX = {
37
+ "None": ["none", ""],
38
+ "1Sg": ["1sg", "1s"],
39
+ "2Sg": ["2sg", "2s"],
40
+ "3": ["3", "3sg", "3pl", "3s", "3p"],
41
+ "1Pl": ["1pl", "1p"],
42
+ "2Pl": ["2pl", "2p"],
43
+ }
44
+ # Enclitic particles, canonical (back-harmony) form -> accepted surface variants. The harmony variant
45
+ # actually emitted is chosen from the host word at realization time (algebra._clitic_for), so both
46
+ # variants normalize to the same canonical key. -kin is harmony-invariant.
47
+ _CLITIC = {
48
+ "kin": ["kin"],
49
+ "kaan": ["kaan", "kään"],
50
+ "ko": ["ko", "kö"],
51
+ "han": ["han", "hän"],
52
+ "pa": ["pa", "pä"],
53
+ }
54
+
55
+
56
+ def _invert(table: dict[str, list[str]]) -> dict[str, str]:
57
+ out: dict[str, str] = {}
58
+ for canon, aliases in table.items():
59
+ out[canon.lower()] = canon
60
+ for a in aliases:
61
+ out[a.lower()] = canon
62
+ return out
63
+
64
+
65
+ _CASE_ALIASES = _invert(_CASE)
66
+ _NUMBER_ALIASES = _invert(_NUMBER)
67
+ _DEGREE_ALIASES = _invert(_DEGREE)
68
+ _PX_ALIASES = _invert(_PX)
69
+ _CLITIC_ALIASES = _invert(_CLITIC)
70
+
71
+
72
+ def _lookup(table: dict[str, str], value: str, dim: str) -> str:
73
+ key = value.strip().lower()
74
+ try:
75
+ return table[key]
76
+ except KeyError:
77
+ canon = sorted(set(table.values()))
78
+ raise ValueError(f"unknown {dim} {value!r}; accepted canonical values {canon} "
79
+ f"or their aliases") from None
80
+
81
+
82
+ def normalize_case(value: str) -> str:
83
+ """Map a case alias ('gen'/'genitive'/'genetiivi'/'Gen') to its canonical Case value ('Gen')."""
84
+ return _lookup(_CASE_ALIASES, value, "case")
85
+
86
+
87
+ def normalize_number(value: str) -> str:
88
+ """Map a number alias ('mon'/'plural'/'pl'/'Plur') to its canonical Number value ('Plur')."""
89
+ return _lookup(_NUMBER_ALIASES, value, "number")
90
+
91
+
92
+ def normalize_degree(value: str) -> str:
93
+ """Map a degree alias ('comp'/'comparative'/'Cmp') to its canonical Degree value ('Cmp')."""
94
+ return _lookup(_DEGREE_ALIASES, value, "degree")
95
+
96
+
97
+ def normalize_px(value: str | None) -> str:
98
+ """Map a possessive-suffix alias ('1sg'/'1Sg'/''/None) to its canonical Px value ('1Sg'). `None`
99
+ is accepted as "no possessive suffix" (mirrors clitic=None) and maps to 'None'."""
100
+ if value is None:
101
+ return "None"
102
+ return _lookup(_PX_ALIASES, value, "possessive (px)")
103
+
104
+
105
+ def normalize_clitic(value: str | None) -> str | None:
106
+ """Map an enclitic alias ('ko'/'kö'/'KO') to its canonical particle ('ko'); `None` (no clitic)
107
+ passes through as `None`. An unknown alias raises ValueError (a typo, not data to coerce)."""
108
+ if value is None:
109
+ return None
110
+ return _lookup(_CLITIC_ALIASES, value, "clitic")
111
+
112
+
113
+ # --- inflect-by-example enclitic peeling (pure heuristic; the engine applies the semantic gate) -----
114
+ _BACK_VOWELS = set("aou")
115
+ _VOWELS = set("aeiouyäöå")
116
+ # (surface, canonical) longest surface first so '-kään'/'-kaan' beat a spurious '-kin'-style short match.
117
+ _ENCLITIC_SURFACES = [("kään", "kaan"), ("kaan", "kaan"), ("kin", "kin"), ("han", "han"),
118
+ ("hän", "han"), ("ko", "ko"), ("kö", "ko"), ("pa", "pa"), ("pä", "pa")]
119
+ _FRONT_ENCLITICS = {"kään", "hän", "pä", "kö"} # may only follow a front/neutral-harmony stem
120
+ _BACK_ENCLITICS = {"kaan", "han", "pa", "ko"} # may only follow a back-harmony stem
121
+
122
+
123
+ def split_enclitic(word: str) -> tuple[str, str | None]:
124
+ """Peel a trailing Finnish focus/interrogative enclitic (-kO/-kin/-kAAn/-hAn/-pA) off a MODEL word
125
+ for inflect-by-example, returning (stem, canonical_particle) or (word, None) when nothing peels.
126
+
127
+ Pure string heuristic — three conservative gates so a lemma that merely *ends* in clitic letters
128
+ (aurinko, vihko, pako) is not mis-split: the variant must agree with the stem's vowel harmony
129
+ (front -kö only after a front stem, back -ko only after a back stem), the stem must be >= 3 chars,
130
+ and it must end in a vowel or -n/-t (the legal final segments of an inflected Finnish nominal —
131
+ rejects fragments like 'kaak'/'tark'). -kin is harmony-invariant. The CALLER still applies the
132
+ semantic gate (the stem must analyze to a genuine non-nominative inflected host and round-trip),
133
+ which removes the remaining lexical collisions; this function only proposes the candidate.
134
+ Combinations (-kohan, -pas) and -kaan after a vowel stem are out of scope."""
135
+ wl = word.lower()
136
+ for surf, canon in _ENCLITIC_SURFACES:
137
+ if not wl.endswith(surf) or len(word) - len(surf) < 3:
138
+ continue
139
+ stem = word[:-len(surf)]
140
+ stem_back = any(c in _BACK_VOWELS for c in stem.lower())
141
+ if surf in _FRONT_ENCLITICS and stem_back:
142
+ continue
143
+ if surf in _BACK_ENCLITICS and not stem_back:
144
+ continue
145
+ last = stem[-1].lower()
146
+ if last not in _VOWELS and last not in "nt":
147
+ continue
148
+ return stem, canon
149
+ return word, None
150
+
151
+
152
+ def px_from_psor(person: str | None, number: str | None) -> str:
153
+ """Map a parser's UD possessor features (Person[psor], Number[psor]) onto our canonical Px value
154
+ (for inflect-by-example). 1/2 distinguish singular vs plural possessor (1Sg/2Sg/1Pl/2Pl); 3 is
155
+ number-agnostic ('3'); no possessor (person is None / empty) maps to 'None'."""
156
+ if not person:
157
+ return "None"
158
+ if person == "3":
159
+ return "3"
160
+ return f"{person}{'Pl' if number == 'Plur' else 'Sg'}"
161
+
162
+
163
+ class FeatureConflict(ValueError):
164
+ """Two individually-valid feature inputs contradict each other (e.g. spec 'mon gen' together
165
+ with number='sing', a spec with two cases 'gen par', or proper=True with a comparative degree).
166
+ Distinct from a plain ValueError (an unknown alias / typo): a conflict is a CALLER contradiction
167
+ the friendly API surfaces as a first-class abstain (`conflicting_features`); a typo fails fast."""
168
+
169
+
170
+ class _Unset:
171
+ """Sentinel type for "argument not supplied" (so an explicit value that contradicts a combined
172
+ spec can be told apart from the default). Single instance `UNSET`, with a stable repr."""
173
+ _instance = None
174
+
175
+ def __new__(cls):
176
+ if cls._instance is None:
177
+ cls._instance = super().__new__(cls)
178
+ return cls._instance
179
+
180
+ def __repr__(self):
181
+ return "UNSET"
182
+
183
+
184
+ # sentinel so callers can tell "argument not supplied" from an explicit value (case/number/degree)
185
+ UNSET = _Unset()
186
+
187
+ # the dimensions a combined spec token may classify into (Px is a separate kwarg, never in a spec)
188
+ _SPEC_DIMS = (("Case", _CASE_ALIASES), ("Number", _NUMBER_ALIASES), ("Degree", _DEGREE_ALIASES))
189
+
190
+
191
+ def parse_feature_spec(spec: str) -> dict[str, str]:
192
+ """Parse a combined human spec ('mon gen' / 'plural genitive comparative') into a canonical
193
+ feature dict ({'Number':'Plur','Case':'Gen'[, 'Degree':...]}). Each whitespace token must
194
+ classify into exactly one dimension. An unknown token raises ValueError (a typo); two tokens
195
+ resolving to DIFFERENT values in the same dimension (a contradiction like 'gen par') raise
196
+ FeatureConflict. Repeated aliases of the same value ('gen genitive') are fine."""
197
+ out: dict[str, str] = {}
198
+ for tok in spec.split():
199
+ key = tok.lower()
200
+ for dim, table in _SPEC_DIMS:
201
+ if key in table:
202
+ val = table[key]
203
+ if dim in out and out[dim] != val:
204
+ raise FeatureConflict(f"conflicting {dim} in spec {spec!r}: {out[dim]} vs {val}")
205
+ out[dim] = val
206
+ break
207
+ else:
208
+ raise ValueError(f"unknown feature token {tok!r} in spec {spec!r}")
209
+ return out
210
+
211
+
212
+ # the UPOS values `inflect` supports as a single-word nominal (the realizer's POS_VALS). PRON/NUM are
213
+ # NOT trained — the realizer would silently clamp them to NOUN — so they are rejected, not advertised.
214
+ _POS_ALIASES = {
215
+ "noun": "NOUN", "n": "NOUN",
216
+ "proper": "PROPN", "propn": "PROPN", "proper noun": "PROPN", "name": "PROPN",
217
+ "adjective": "ADJ", "adj": "ADJ", "a": "ADJ",
218
+ }
219
+
220
+
221
+ def resolve_features(case: str = "nom", number=UNSET, degree=UNSET) -> dict[str, str]:
222
+ """Merge a (possibly combined) `case` spec with explicit `number`/`degree` kwargs into a
223
+ canonical {Case, Number, Degree} cell. `case` may be a single alias ('gen'), a number/degree
224
+ alias ('mon'), or a combined spec ('mon gen'). An explicit kwarg that contradicts the spec
225
+ raises FeatureConflict; an unknown alias raises plain ValueError. Missing dimensions default to
226
+ Case=Nom, Number=Sing, Degree=Pos."""
227
+ feats = parse_feature_spec(case)
228
+ feats.setdefault("Case", "Nom")
229
+ if number is not UNSET:
230
+ n = normalize_number(number)
231
+ if feats.get("Number", n) != n:
232
+ raise FeatureConflict(f"number={number!r} ({n}) conflicts with spec {case!r} "
233
+ f"(Number={feats['Number']})")
234
+ feats["Number"] = n
235
+ feats.setdefault("Number", "Sing")
236
+ if degree is not UNSET:
237
+ d = normalize_degree(degree)
238
+ if feats.get("Degree", d) != d:
239
+ raise FeatureConflict(f"degree={degree!r} ({d}) conflicts with spec {case!r} "
240
+ f"(Degree={feats['Degree']})")
241
+ feats["Degree"] = d
242
+ feats.setdefault("Degree", "Pos")
243
+ return feats
244
+
245
+
246
+ def resolve_pos(word: str, *, pos: str | None = None, proper: bool | None = None,
247
+ degree: str = "Pos") -> str:
248
+ """Decide the UPOS for a single-word `inflect`. Explicit `pos` ('noun'/'proper'/'adjective')
249
+ wins; `proper=True/False` is sugar for PROPN/NOUN; otherwise infer (canonical `degree` != Pos =>
250
+ ADJ; leading capital => PROPN; else NOUN). Contradictions raise FeatureConflict; an unsupported
251
+ `pos` (e.g. 'pronoun') raises plain ValueError."""
252
+ explicit: str | None = None
253
+ if pos is not None:
254
+ key = pos.strip().lower()
255
+ if key not in _POS_ALIASES:
256
+ raise ValueError(f"unsupported pos {pos!r}; inflect handles "
257
+ f"{sorted(set(_POS_ALIASES.values()))} (noun/proper/adjective)")
258
+ explicit = _POS_ALIASES[key]
259
+ if proper is not None:
260
+ sugar = "PROPN" if proper else "NOUN"
261
+ if explicit is not None and explicit != sugar:
262
+ raise FeatureConflict(f"proper={proper} conflicts with pos={pos!r}")
263
+ explicit = sugar
264
+ if explicit is not None:
265
+ if explicit != "ADJ" and degree != "Pos":
266
+ raise FeatureConflict(f"degree={degree} (comparative/superlative) is realized through the "
267
+ f"adjective path, but pos resolved to {explicit}; drop the degree or "
268
+ f"inflect it as an adjective")
269
+ return explicit
270
+ if degree != "Pos":
271
+ return "ADJ"
272
+ return "PROPN" if word[:1].isupper() else "NOUN"
juurio/finiteverb.py ADDED
@@ -0,0 +1,138 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Finite-verb conjugation net (inference).
2
+
3
+ realizer B covers nominals (NOUN/ADJ/PROPN); verbforms.py covers non-finite verb forms (participles,
4
+ infinitives). This module covers FINITE conjugation — `puhun` (I speak), `käveli` (s/he walked),
5
+ `menisivät` (they would go), `puhutaan` (one speaks) — the largest, most-used verb subsystem. It is the
6
+ same `Realizer` char enc-dec, with a categorical header over the finite grid:
7
+ Voice (Act/Pss) × Mood (Ind/Cond/Pot/Imprt) × Tense (Prs/Prt, Ind only) × Person (Sg1..Pl3/ConNeg;
8
+ passive = Pe4). 37 non-empty cells, pinned by tests/test_finiteverb_cells.py.
9
+
10
+ Mirrors verbforms.py (header_ids/load/decode) so the decode path is uniform across all POS.
11
+ """
12
+ from __future__ import annotations
13
+
14
+ import os
15
+
16
+ from juurio import bootstrap
17
+ from juurio.realizer import encode_chars
18
+
19
+ # --- finite grid (Voice, Mood, Tense, Person). Tense is None for Cond/Pot/Imprt. ----------------
20
+ VOICE = ["Act", "Pss"]
21
+ PERSONS = ["Sg1", "Sg2", "Sg3", "Pl1", "Pl2", "Pl3", "ConNeg"]
22
+ # (mood, [tenses]) — Ind carries tense; the others don't (tense None)
23
+ _ACT_MOODS = [("Ind", ["Prs", "Prt"]), ("Cond", [None]), ("Pot", [None]), ("Imprt", [None])]
24
+ _PSS_MOODS = [("Ind", ["Prs", "Prt"]), ("Cond", [None]), ("Pot", [None])]
25
+ # systematic empty cells that do not exist (excluded from the grid)
26
+ _EMPTY = {("Act", "Ind", "Prt", "ConNeg"), ("Act", "Imprt", "Sg1")}
27
+
28
+
29
+ def _build_cells():
30
+ cells = []
31
+ for mood, tenses in _ACT_MOODS:
32
+ for tense in tenses:
33
+ for per in PERSONS:
34
+ key = ("Act", mood, *((tense,) if tense else ()), per)
35
+ if key in _EMPTY:
36
+ continue
37
+ cells.append({"Voice": "Act", "Mood": mood, "Tense": tense, "Person": per})
38
+ for mood, tenses in _PSS_MOODS:
39
+ for tense in tenses:
40
+ cells.append({"Voice": "Pss", "Mood": mood, "Tense": tense, "Person": "Pe4"})
41
+ return cells
42
+
43
+
44
+ def cell_key(cell: dict):
45
+ """Order-stable identity tuple: (Voice, Mood, [Tense], Person), Tense omitted when None."""
46
+ t = cell.get("Tense")
47
+ if t is None:
48
+ return (cell["Voice"], cell["Mood"], cell["Person"])
49
+ return (cell["Voice"], cell["Mood"], t, cell["Person"])
50
+
51
+
52
+ CELLS = _build_cells()
53
+
54
+ # --- categorical header (fixed-length, clamp unknowns so the loaded net never KeyErrors) ----------
55
+ _MOODS = ["Ind", "Cond", "Pot", "Imprt"]
56
+ _TENSES = ["_None", "Prs", "Prt"]
57
+ _ALLPERS = PERSONS + ["Pe4"]
58
+ FEAT_GROUPS = [("Voice", VOICE), ("Mood", _MOODS), ("Tense", _TENSES), ("Person", _ALLPERS)]
59
+ FEAT_STOI = {f"{g}={v}": i for i, (g, v) in
60
+ enumerate((g, v) for g, vals in FEAT_GROUPS for v in vals)}
61
+ NFEAT_TOK = len(FEAT_STOI)
62
+ HLEN = len(FEAT_GROUPS)
63
+
64
+
65
+ def header_ids(cell: dict) -> list:
66
+ v = cell.get("Voice") if cell.get("Voice") in VOICE else "Act"
67
+ m = cell.get("Mood") if cell.get("Mood") in _MOODS else "Ind"
68
+ t = cell.get("Tense") or "_None"
69
+ t = t if t in _TENSES else "_None"
70
+ p = cell.get("Person") if cell.get("Person") in _ALLPERS else "Sg3"
71
+ return [FEAT_STOI[f"Voice={v}"], FEAT_STOI[f"Mood={m}"],
72
+ FEAT_STOI[f"Tense={t}"], FEAT_STOI[f"Person={p}"]]
73
+
74
+
75
+ CKPT = os.path.join(os.path.dirname(__file__), "..", "checkpoints", "finiteverb.pt")
76
+
77
+
78
+ def load(ckpt: str | None = None):
79
+ import torch
80
+ from juurio.realizer import Realizer
81
+ from juurio.device import DEV
82
+ if ckpt is None:
83
+ ckpt = bootstrap.artifact_path("finiteverb")
84
+ ck = torch.load(ckpt, map_location=DEV, weights_only=False)
85
+ m = Realizer(len(ck["stoi"]), ck["cfg"]["d_model"], 4, ck["cfg"]["layers"]).to(DEV)
86
+ m.load_state_dict(ck["model"]); m.eval()
87
+ return m, ck["stoi"], ck["itos"]
88
+
89
+
90
+ def decode_one(bundle, lemma: str, cell: dict, maxlen: int = 48) -> str:
91
+ from juurio.realizer import _decode
92
+ m, stoi, itos = bundle
93
+ return _decode(m, header_ids(cell), lemma.lower(), stoi, itos, maxlen=maxlen)
94
+
95
+
96
+ def decode_forms(items, bundle, batch_size: int = 256, maxlen: int = 48):
97
+ """Greedy-decode a finite form for each (lemma, cell). Batched; bit-identical to decode_one
98
+ (greedy argmax + per-seq attention). items: iterable of (lemma, cell-dict)."""
99
+ import torch
100
+ from juurio.realizer import BOS, EOS, PAD, _pad
101
+ from juurio.device import DEV
102
+ items = list(items)
103
+ if not items:
104
+ return []
105
+ m, stoi, itos = bundle
106
+ m.eval()
107
+ out = []
108
+ for start in range(0, len(items), batch_size):
109
+ chunk = items[start:start + batch_size]
110
+ B = len(chunk)
111
+ src_lists = [encode_chars(lem.lower(), stoi) for lem, _ in chunk]
112
+ S = max((len(s) for s in src_lists), default=1) or 1
113
+ src = _pad(src_lists, S).to(DEV)
114
+ hdr = torch.tensor([header_ids(cell) for _, cell in chunk], dtype=torch.long, device=DEV)
115
+ with torch.no_grad():
116
+ mem, mask = m.encode(src, hdr)
117
+ ids = torch.full((B, 1), BOS, dtype=torch.long, device=DEV)
118
+ finished = torch.zeros(B, dtype=torch.bool, device=DEV)
119
+ collected = [[] for _ in range(B)]
120
+ for _ in range(maxlen):
121
+ T = ids.size(1)
122
+ cmask = torch.triu(torch.ones(T, T, device=DEV, dtype=torch.bool), 1)
123
+ dout = m.dec(m._tgt_embed(ids), mem, tgt_mask=cmask, memory_key_padding_mask=mask)
124
+ nxt = m.proj(dout)[:, -1].argmax(-1)
125
+ nxt_cpu = nxt.tolist()
126
+ for i in range(B):
127
+ if finished[i]:
128
+ continue
129
+ if nxt_cpu[i] == EOS:
130
+ finished[i] = True
131
+ else:
132
+ collected[i].append(nxt_cpu[i])
133
+ app = nxt.clone(); app[finished] = PAD
134
+ ids = torch.cat([ids, app.unsqueeze(1)], 1)
135
+ if bool(finished.all()):
136
+ break
137
+ out.extend("".join(itos[t] for t in row if t in itos) for row in collected)
138
+ return out
juurio/genlex.py ADDED
@@ -0,0 +1,124 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Runtime generation lexicon — the static exception table that makes net+lexicon generation byte-exact.
2
+
3
+ At serving time the trained realizer nets (`realizer.pt` nominal, `verbform.pt` non-finite,
4
+ `finiteverb.pt` finite) are the productive engine; this lexicon (`data/shipped_wordlist.json`) carries
5
+ ONLY the (subsystem, lemma, cell) cells where a net disagrees with the Giella FST — look it up first,
6
+ fall through to the net on a miss. It is the GENERATION analogue of the analysis side's static FST-tag
7
+ table: model + lexicon, no live FST.
8
+
9
+ Byte-exactness is the whole point: a key the loader builds must equal the key the wordlist builder
10
+ wrote, or every override is silently missed. The key formats (per subsystem) are therefore mirrored
11
+ from that builder verbatim:
12
+
13
+ NOUN|<lemma>|('Case', V)+('Number', V)+('Px', V) # tuple(sorted(cell.items())) repr
14
+ ADJ|<lemma>|('Case', V)+('Degree', V)+('Number', V)
15
+ VERB_FIN|<lemma>|Voice+Mood[+Tense]+Person # finiteverb.cell_key(cell)
16
+ VERB_NONFIN|<lemma>|form+case+number
17
+
18
+ PROPN is intentionally NOT shipped (open class), so nominal lookups for PROPN always abstain.
19
+ """
20
+ from __future__ import annotations
21
+
22
+ import json
23
+ import os
24
+
25
+ from . import bootstrap
26
+
27
+ # Repo root (where data/ lives), so the default artifact resolves against the install, not the process
28
+ # CWD — mirrors parser.neural_adapter._ROOT / bootstrap.ROOT so serving works from any directory.
29
+ _ROOT = os.path.normpath(os.path.join(os.path.dirname(__file__), ".."))
30
+ CKPT = os.path.join("data", "shipped_wordlist.json")
31
+ # In-tree FST-parity overrides merged on top of the artifact at load time (resolved against the PACKAGE
32
+ # dir, not the repo root, so they ship with `juurio`). See `load`. Two files, both merged: the VERB
33
+ # register corrections and the net-unreliable common-NOUN paradigms. Independent so each regenerates
34
+ # without clobbering the other.
35
+ OVERRIDES = ("genlex_overrides.json", "genlex_overrides_noun.json")
36
+
37
+
38
+ def _resolve(path: str) -> str:
39
+ return path if os.path.isabs(path) else os.path.join(_ROOT, path)
40
+
41
+
42
+ class GenerationLexicon:
43
+ """The shipped exception table loaded into a flat dict, with per-subsystem byte-exact key lookups.
44
+
45
+ Construct via `GenerationLexicon.load()` (default `data/shipped_wordlist.json`, repo-root-resolved).
46
+ Each `lookup_*` returns the stored gold form or None (miss ⇒ the caller runs the net)."""
47
+
48
+ def __init__(self, table: dict):
49
+ self._table = table
50
+
51
+ @classmethod
52
+ def load(cls, path: str | None = None, override_path: str | None = None) -> "GenerationLexicon":
53
+ # Explicit path resolves against ROOT; default routes through the bootstrap resolver
54
+ # (dev source -> HF), so a clean wheel install fetches the shipped wordlist artifact.
55
+ resolved = _resolve(path) if path else bootstrap.artifact_path("wordlist")
56
+ try:
57
+ with open(resolved, encoding="utf-8") as fh:
58
+ table = json.load(fh)
59
+ except FileNotFoundError as e:
60
+ # The default net+lexicon generation backend loads this at Transformer() construction, so a
61
+ # missing artifact is a setup problem, not a programming bug — point the user at the fetcher.
62
+ raise FileNotFoundError(
63
+ f"generation lexicon not found at {resolved!r}; it is required by the default net+lexicon "
64
+ f"generation backend. Run `juurio setup` (or `juurio doctor`) to fetch the shipped "
65
+ f"artifacts listed in bootstrap.MANIFEST."
66
+ ) from e
67
+ # Merge the committed, in-tree FST-parity overrides ON TOP of the downloaded artifact: VERB cells
68
+ # where the artifact's variant picker shipped the wrong register (archaic `-oitse-`, colloquial
69
+ # `mee`), and NOUN paradigms for common lemmas the realizer net serves unreliably (aika->ajan,
70
+ # mökä->mökän), each corrected to the FST-valid form. Version-controlled, so they survive a
71
+ # re-download of the separately-pinned artifact; a missing file is a no-op. An explicit
72
+ # `override_path` (str or iterable of paths) takes precedence, used as-is; default resolves the
73
+ # OVERRIDES names against the PACKAGE dir.
74
+ if override_path is not None: # explicit: a single path (str/PathLike) or an iterable
75
+ ov_files = ([os.fspath(override_path)] if isinstance(override_path, (str, os.PathLike))
76
+ else [os.fspath(p) for p in override_path])
77
+ else:
78
+ ov_files = [os.path.join(os.path.dirname(__file__), n) for n in OVERRIDES]
79
+ for ov in ov_files:
80
+ try:
81
+ with open(ov, encoding="utf-8") as fh:
82
+ table.update(json.load(fh))
83
+ except FileNotFoundError:
84
+ pass
85
+ return cls(table)
86
+
87
+ def __len__(self) -> int:
88
+ return len(self._table)
89
+
90
+ # --- key builders: mirror the wordlist builder's flatten step EXACTLY -----------------------
91
+ @staticmethod
92
+ def _nominal_key(pos: str, lemma: str, cell: dict) -> str:
93
+ # _cellkey_nominal(cell) = tuple(sorted(cell.items())); flatten = pos|lemma|"+".join(str(x))
94
+ ck = tuple(sorted(cell.items()))
95
+ return f"{pos}|{lemma}|" + "+".join(str(x) for x in ck)
96
+
97
+ def lookup_nominal(self, upos: str, lemma: str, feats: dict) -> str | None:
98
+ """NOUN/ADJ exception form for `lemma` under UD `feats`, or None. Defaults (Px='None',
99
+ Degree='Pos', …) match the realizer/builder grid. PROPN is not shipped ⇒ always None. NUM and PRON
100
+ are lookup-only ALIASES of NOUN: simplex numerals and common pronouns are stored once under the NOUN
101
+ key, so a clause/parser token tagged NUM/PRON hits that same paradigm (no duplicate NUM|/PRON|
102
+ data)."""
103
+ if upos in ("NOUN", "NUM", "PRON"):
104
+ key_pos = "NOUN"
105
+ cell = {"Case": feats.get("Case", "Nom"), "Number": feats.get("Number", "Sing"),
106
+ "Px": feats.get("Px", "None")}
107
+ elif upos == "ADJ":
108
+ key_pos = "ADJ"
109
+ cell = {"Case": feats.get("Case", "Nom"), "Degree": feats.get("Degree", "Pos"),
110
+ "Number": feats.get("Number", "Sing")}
111
+ else:
112
+ return None
113
+ return self._table.get(self._nominal_key(key_pos, lemma, cell))
114
+
115
+ def lookup_finite(self, lemma: str, cell: dict) -> str | None:
116
+ """Finite-verb exception form for `lemma` at `cell` ({Voice,Mood,Tense?,Person}), or None."""
117
+ from . import finiteverb
118
+ ck = finiteverb.cell_key(cell) # (Voice, Mood, [Tense], Person), Tense omitted when None
119
+ return self._table.get(f"VERB_FIN|{lemma}|" + "+".join(str(x) for x in ck))
120
+
121
+ def lookup_nonfin(self, lemma: str, form: str, case: str = "Nom",
122
+ number: str = "Sing") -> str | None:
123
+ """Non-finite-verb exception form (participle/infinitive) for `lemma`, or None."""
124
+ return self._table.get(f"VERB_NONFIN|{lemma}|" + "+".join((form, case, number)))
juurio/genlex_overrides.json ADDED
@@ -0,0 +1,244 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "VERB_FIN|ahkeroida|Act+Imprt+Sg2": "ahkeroi",
3
+ "VERB_FIN|ahkeroida|Act+Ind+Prs+ConNeg": "ahkeroi",
4
+ "VERB_FIN|ahkeroida|Act+Ind+Prs+Pl1": "ahkeroimme",
5
+ "VERB_FIN|ahkeroida|Act+Ind+Prs+Pl3": "ahkeroivat",
6
+ "VERB_FIN|ahkeroida|Act+Ind+Prs+Sg2": "ahkeroit",
7
+ "VERB_FIN|ahkeroida|Act+Ind+Prs+Sg3": "ahkeroi",
8
+ "VERB_FIN|aprikoida|Act+Imprt+Sg2": "aprikoi",
9
+ "VERB_FIN|aprikoida|Act+Ind+Prs+ConNeg": "aprikoi",
10
+ "VERB_FIN|aprikoida|Act+Ind+Prs+Pl1": "aprikoimme",
11
+ "VERB_FIN|aprikoida|Act+Ind+Prs+Pl3": "aprikoivat",
12
+ "VERB_FIN|aprikoida|Act+Ind+Prs+Sg2": "aprikoit",
13
+ "VERB_FIN|aprikoida|Act+Ind+Prs+Sg3": "aprikoi",
14
+ "VERB_FIN|aterioida|Act+Imprt+Sg2": "aterioi",
15
+ "VERB_FIN|aterioida|Act+Ind+Prs+ConNeg": "aterioi",
16
+ "VERB_FIN|aterioida|Act+Ind+Prs+Pl1": "aterioimme",
17
+ "VERB_FIN|aterioida|Act+Ind+Prs+Pl3": "aterioivat",
18
+ "VERB_FIN|aterioida|Act+Ind+Prs+Sg2": "aterioit",
19
+ "VERB_FIN|aterioida|Act+Ind+Prs+Sg3": "aterioi",
20
+ "VERB_FIN|emännöidä|Act+Imprt+Sg2": "emännöi",
21
+ "VERB_FIN|emännöidä|Act+Ind+Prs+ConNeg": "emännöi",
22
+ "VERB_FIN|emännöidä|Act+Ind+Prs+Pl3": "emännöivät",
23
+ "VERB_FIN|emännöidä|Act+Ind+Prs+Sg3": "emännöi",
24
+ "VERB_FIN|haluta|Act+Imprt+Sg2": "halua",
25
+ "VERB_FIN|haluta|Act+Ind+Prs+ConNeg": "halua",
26
+ "VERB_FIN|haluta|Act+Ind+Prs+Pl1": "haluamme",
27
+ "VERB_FIN|haluta|Act+Ind+Prs+Pl2": "haluatte",
28
+ "VERB_FIN|haluta|Act+Ind+Prs+Pl3": "haluavat",
29
+ "VERB_FIN|haluta|Act+Ind+Prs+Sg1": "haluan",
30
+ "VERB_FIN|haluta|Act+Ind+Prs+Sg2": "haluat",
31
+ "VERB_FIN|haluta|Act+Ind+Prs+Sg3": "haluaa",
32
+ "VERB_FIN|haravoida|Act+Imprt+Sg2": "haravoi",
33
+ "VERB_FIN|haravoida|Act+Ind+Prs+ConNeg": "haravoi",
34
+ "VERB_FIN|haravoida|Act+Ind+Prs+Pl1": "haravoimme",
35
+ "VERB_FIN|haravoida|Act+Ind+Prs+Pl3": "haravoivat",
36
+ "VERB_FIN|haravoida|Act+Ind+Prs+Sg2": "haravoit",
37
+ "VERB_FIN|haravoida|Act+Ind+Prs+Sg3": "haravoi",
38
+ "VERB_FIN|hekumoida|Act+Imprt+Sg2": "hekumoi",
39
+ "VERB_FIN|hekumoida|Act+Ind+Prs+ConNeg": "hekumoi",
40
+ "VERB_FIN|hekumoida|Act+Ind+Prs+Pl1": "hekumoimme",
41
+ "VERB_FIN|hekumoida|Act+Ind+Prs+Pl3": "hekumoivat",
42
+ "VERB_FIN|hekumoida|Act+Ind+Prs+Sg2": "hekumoit",
43
+ "VERB_FIN|hekumoida|Act+Ind+Prs+Sg3": "hekumoi",
44
+ "VERB_FIN|ikävöidä|Act+Imprt+Sg2": "ikävöi",
45
+ "VERB_FIN|ikävöidä|Act+Ind+Prs+ConNeg": "ikävöi",
46
+ "VERB_FIN|ikävöidä|Act+Ind+Prs+Pl1": "ikävöimme",
47
+ "VERB_FIN|ikävöidä|Act+Ind+Prs+Pl3": "ikävöivät",
48
+ "VERB_FIN|ikävöidä|Act+Ind+Prs+Sg2": "ikävöit",
49
+ "VERB_FIN|ikävöidä|Act+Ind+Prs+Sg3": "ikävöi",
50
+ "VERB_FIN|ilakoida|Act+Imprt+Sg2": "ilakoi",
51
+ "VERB_FIN|ilakoida|Act+Ind+Prs+ConNeg": "ilakoi",
52
+ "VERB_FIN|ilakoida|Act+Ind+Prs+Pl1": "ilakoimme",
53
+ "VERB_FIN|ilakoida|Act+Ind+Prs+Pl3": "ilakoivat",
54
+ "VERB_FIN|ilakoida|Act+Ind+Prs+Sg2": "ilakoit",
55
+ "VERB_FIN|ilakoida|Act+Ind+Prs+Sg3": "ilakoi",
56
+ "VERB_FIN|imuroida|Act+Imprt+Sg2": "imuroi",
57
+ "VERB_FIN|imuroida|Act+Ind+Prs+ConNeg": "imuroi",
58
+ "VERB_FIN|imuroida|Act+Ind+Prs+Pl1": "imuroimme",
59
+ "VERB_FIN|imuroida|Act+Ind+Prs+Pl3": "imuroivat",
60
+ "VERB_FIN|imuroida|Act+Ind+Prs+Sg2": "imuroit",
61
+ "VERB_FIN|imuroida|Act+Ind+Prs+Sg3": "imuroi",
62
+ "VERB_FIN|isännöidä|Act+Imprt+Sg2": "isännöi",
63
+ "VERB_FIN|isännöidä|Act+Ind+Prs+ConNeg": "isännöi",
64
+ "VERB_FIN|isännöidä|Act+Ind+Prs+Pl3": "isännöivät",
65
+ "VERB_FIN|isännöidä|Act+Ind+Prs+Sg3": "isännöi",
66
+ "VERB_FIN|kapaloida|Act+Imprt+Sg2": "kapaloi",
67
+ "VERB_FIN|kapaloida|Act+Ind+Prs+ConNeg": "kapaloi",
68
+ "VERB_FIN|kapaloida|Act+Ind+Prs+Pl1": "kapaloimme",
69
+ "VERB_FIN|kapaloida|Act+Ind+Prs+Pl3": "kapaloivat",
70
+ "VERB_FIN|kapaloida|Act+Ind+Prs+Sg2": "kapaloit",
71
+ "VERB_FIN|kapaloida|Act+Ind+Prs+Sg3": "kapaloi",
72
+ "VERB_FIN|kapinoida|Act+Imprt+Sg2": "kapinoi",
73
+ "VERB_FIN|kapinoida|Act+Ind+Prs+ConNeg": "kapinoi",
74
+ "VERB_FIN|kapinoida|Act+Ind+Prs+Pl1": "kapinoimme",
75
+ "VERB_FIN|kapinoida|Act+Ind+Prs+Pl3": "kapinoivat",
76
+ "VERB_FIN|kapinoida|Act+Ind+Prs+Sg2": "kapinoit",
77
+ "VERB_FIN|kapinoida|Act+Ind+Prs+Sg3": "kapinoi",
78
+ "VERB_FIN|karkeloida|Act+Imprt+Sg2": "karkeloi",
79
+ "VERB_FIN|karkeloida|Act+Ind+Prs+ConNeg": "karkeloi",
80
+ "VERB_FIN|karkeloida|Act+Ind+Prs+Pl1": "karkeloimme",
81
+ "VERB_FIN|karkeloida|Act+Ind+Prs+Pl3": "karkeloivat",
82
+ "VERB_FIN|karkeloida|Act+Ind+Prs+Sg2": "karkeloit",
83
+ "VERB_FIN|karkeloida|Act+Ind+Prs+Sg3": "karkeloi",
84
+ "VERB_FIN|keikaroida|Act+Imprt+Sg2": "keikaroi",
85
+ "VERB_FIN|keikaroida|Act+Ind+Prs+ConNeg": "keikaroi",
86
+ "VERB_FIN|keikaroida|Act+Ind+Prs+Pl1": "keikaroimme",
87
+ "VERB_FIN|keikaroida|Act+Ind+Prs+Pl3": "keikaroivat",
88
+ "VERB_FIN|keikaroida|Act+Ind+Prs+Sg2": "keikaroit",
89
+ "VERB_FIN|keikaroida|Act+Ind+Prs+Sg3": "keikaroi",
90
+ "VERB_FIN|kihelmöidä|Act+Imprt+Sg2": "kihelmöi",
91
+ "VERB_FIN|kihelmöidä|Act+Ind+Prs+ConNeg": "kihelmöi",
92
+ "VERB_FIN|kihelmöidä|Act+Ind+Prs+Pl1": "kihelmöimme",
93
+ "VERB_FIN|kihelmöidä|Act+Ind+Prs+Pl3": "kihelmöivät",
94
+ "VERB_FIN|kihelmöidä|Act+Ind+Prs+Sg2": "kihelmöit",
95
+ "VERB_FIN|kihelmöidä|Act+Ind+Prs+Sg3": "kihelmöi",
96
+ "VERB_FIN|kipinöidä|Act+Imprt+Sg2": "kipinöi",
97
+ "VERB_FIN|kipinöidä|Act+Ind+Prs+ConNeg": "kipinöi",
98
+ "VERB_FIN|kipinöidä|Act+Ind+Prs+Pl3": "kipinöivät",
99
+ "VERB_FIN|kipinöidä|Act+Ind+Prs+Sg3": "kipinöi",
100
+ "VERB_FIN|kipunoida|Act+Imprt+Sg2": "kipunoi",
101
+ "VERB_FIN|kipunoida|Act+Ind+Prs+ConNeg": "kipunoi",
102
+ "VERB_FIN|kipunoida|Act+Ind+Prs+Pl1": "kipunoimme",
103
+ "VERB_FIN|kipunoida|Act+Ind+Prs+Pl3": "kipunoivat",
104
+ "VERB_FIN|kipunoida|Act+Ind+Prs+Sg2": "kipunoit",
105
+ "VERB_FIN|kipunoida|Act+Ind+Prs+Sg3": "kipunoi",
106
+ "VERB_FIN|kuutioida|Act+Imprt+Sg2": "kuutioi",
107
+ "VERB_FIN|kuutioida|Act+Ind+Prs+ConNeg": "kuutioi",
108
+ "VERB_FIN|kuutioida|Act+Ind+Prs+Pl1": "kuutioimme",
109
+ "VERB_FIN|kuutioida|Act+Ind+Prs+Pl3": "kuutioivat",
110
+ "VERB_FIN|kuutioida|Act+Ind+Prs+Sg2": "kuutioit",
111
+ "VERB_FIN|kuutioida|Act+Ind+Prs+Sg3": "kuutioi",
112
+ "VERB_FIN|käpälöidä|Act+Imprt+Sg2": "käpälöi",
113
+ "VERB_FIN|käpälöidä|Act+Ind+Prs+ConNeg": "käpälöi",
114
+ "VERB_FIN|käpälöidä|Act+Ind+Prs+Pl1": "käpälöimme",
115
+ "VERB_FIN|käpälöidä|Act+Ind+Prs+Pl3": "käpälöivät",
116
+ "VERB_FIN|käpälöidä|Act+Ind+Prs+Sg2": "käpälöit",
117
+ "VERB_FIN|käpälöidä|Act+Ind+Prs+Sg3": "käpälöi",
118
+ "VERB_FIN|liikennöidä|Act+Imprt+Sg2": "liikennöi",
119
+ "VERB_FIN|liikennöidä|Act+Ind+Prs+ConNeg": "liikennöi",
120
+ "VERB_FIN|liikennöidä|Act+Ind+Prs+Pl1": "liikennöimme",
121
+ "VERB_FIN|liikennöidä|Act+Ind+Prs+Pl3": "liikennöivät",
122
+ "VERB_FIN|liikennöidä|Act+Ind+Prs+Sg3": "liikennöi",
123
+ "VERB_FIN|luennoida|Act+Imprt+Sg2": "luennoi",
124
+ "VERB_FIN|luennoida|Act+Ind+Prs+ConNeg": "luennoi",
125
+ "VERB_FIN|luennoida|Act+Ind+Prs+Pl1": "luennoimme",
126
+ "VERB_FIN|luennoida|Act+Ind+Prs+Pl3": "luennoivat",
127
+ "VERB_FIN|luennoida|Act+Ind+Prs+Sg2": "luennoit",
128
+ "VERB_FIN|luennoida|Act+Ind+Prs+Sg3": "luennoi",
129
+ "VERB_FIN|mankeloida|Act+Imprt+Sg2": "mankeloi",
130
+ "VERB_FIN|mankeloida|Act+Ind+Prs+ConNeg": "mankeloi",
131
+ "VERB_FIN|mankeloida|Act+Ind+Prs+Pl1": "mankeloimme",
132
+ "VERB_FIN|mankeloida|Act+Ind+Prs+Pl3": "mankeloivat",
133
+ "VERB_FIN|mankeloida|Act+Ind+Prs+Sg2": "mankeloit",
134
+ "VERB_FIN|mankeloida|Act+Ind+Prs+Sg3": "mankeloi",
135
+ "VERB_FIN|mellakoida|Act+Imprt+Sg2": "mellakoi",
136
+ "VERB_FIN|mellakoida|Act+Ind+Prs+ConNeg": "mellakoi",
137
+ "VERB_FIN|mellakoida|Act+Ind+Prs+Pl1": "mellakoimme",
138
+ "VERB_FIN|mellakoida|Act+Ind+Prs+Pl3": "mellakoivat",
139
+ "VERB_FIN|mellakoida|Act+Ind+Prs+Sg2": "mellakoit",
140
+ "VERB_FIN|mellakoida|Act+Ind+Prs+Sg3": "mellakoi",
141
+ "VERB_FIN|mennä|Act+Imprt+Sg2": "mene",
142
+ "VERB_FIN|mennä|Act+Ind+Prs+Pl1": "menemme",
143
+ "VERB_FIN|mennä|Act+Ind+Prs+Pl2": "menette",
144
+ "VERB_FIN|mennä|Act+Ind+Prs+Sg1": "menen",
145
+ "VERB_FIN|mennä|Act+Ind+Prs+Sg2": "menet",
146
+ "VERB_FIN|metelöidä|Act+Imprt+Sg2": "metelöi",
147
+ "VERB_FIN|metelöidä|Act+Ind+Prs+ConNeg": "metelöi",
148
+ "VERB_FIN|metelöidä|Act+Ind+Prs+Pl1": "metelöimme",
149
+ "VERB_FIN|metelöidä|Act+Ind+Prs+Pl3": "metelöivät",
150
+ "VERB_FIN|metelöidä|Act+Ind+Prs+Sg2": "metelöit",
151
+ "VERB_FIN|metelöidä|Act+Ind+Prs+Sg3": "metelöi",
152
+ "VERB_FIN|pakinoida|Act+Imprt+Sg2": "pakinoi",
153
+ "VERB_FIN|pakinoida|Act+Ind+Prs+ConNeg": "pakinoi",
154
+ "VERB_FIN|pakinoida|Act+Ind+Prs+Pl3": "pakinoivat",
155
+ "VERB_FIN|pakinoida|Act+Ind+Prs+Sg2": "pakinoit",
156
+ "VERB_FIN|pakinoida|Act+Ind+Prs+Sg3": "pakinoi",
157
+ "VERB_FIN|panna|Act+Imprt+Sg2": "pane",
158
+ "VERB_FIN|panna|Act+Ind+Prs+Sg2": "panet",
159
+ "VERB_FIN|patikoida|Act+Imprt+Sg2": "patikoi",
160
+ "VERB_FIN|patikoida|Act+Ind+Prs+ConNeg": "patikoi",
161
+ "VERB_FIN|patikoida|Act+Ind+Prs+Pl1": "patikoimme",
162
+ "VERB_FIN|patikoida|Act+Ind+Prs+Pl3": "patikoivat",
163
+ "VERB_FIN|patikoida|Act+Ind+Prs+Sg2": "patikoit",
164
+ "VERB_FIN|patikoida|Act+Ind+Prs+Sg3": "patikoi",
165
+ "VERB_FIN|pokkuroida|Act+Imprt+Sg2": "pokkuroi",
166
+ "VERB_FIN|pokkuroida|Act+Ind+Prs+ConNeg": "pokkuroi",
167
+ "VERB_FIN|pokkuroida|Act+Ind+Prs+Pl1": "pokkuroimme",
168
+ "VERB_FIN|pokkuroida|Act+Ind+Prs+Pl3": "pokkuroivat",
169
+ "VERB_FIN|pokkuroida|Act+Ind+Prs+Sg2": "pokkuroit",
170
+ "VERB_FIN|pokkuroida|Act+Ind+Prs+Sg3": "pokkuroi",
171
+ "VERB_FIN|pullikoida|Act+Imprt+Sg2": "pullikoi",
172
+ "VERB_FIN|pullikoida|Act+Ind+Prs+ConNeg": "pullikoi",
173
+ "VERB_FIN|pullikoida|Act+Ind+Prs+Pl1": "pullikoimme",
174
+ "VERB_FIN|pullikoida|Act+Ind+Prs+Pl3": "pullikoivat",
175
+ "VERB_FIN|pullikoida|Act+Ind+Prs+Sg2": "pullikoit",
176
+ "VERB_FIN|pullikoida|Act+Ind+Prs+Sg3": "pullikoi",
177
+ "VERB_FIN|raueta|Act+Imprt+Sg2": "raukea",
178
+ "VERB_FIN|raueta|Act+Ind+Prs+ConNeg": "raukea",
179
+ "VERB_FIN|raueta|Act+Ind+Prs+Pl1": "raukeamme",
180
+ "VERB_FIN|raueta|Act+Ind+Prs+Pl3": "raukeavat",
181
+ "VERB_FIN|raueta|Act+Ind+Prs+Sg1": "raukean",
182
+ "VERB_FIN|raueta|Act+Ind+Prs+Sg2": "raukeat",
183
+ "VERB_FIN|raueta|Act+Ind+Prs+Sg3": "raukeaa",
184
+ "VERB_FIN|rettelöidä|Act+Imprt+Sg2": "rettelöi",
185
+ "VERB_FIN|rettelöidä|Act+Ind+Prs+ConNeg": "rettelöi",
186
+ "VERB_FIN|rettelöidä|Act+Ind+Prs+Pl1": "rettelöimme",
187
+ "VERB_FIN|rettelöidä|Act+Ind+Prs+Pl3": "rettelöivät",
188
+ "VERB_FIN|rettelöidä|Act+Ind+Prs+Sg2": "rettelöit",
189
+ "VERB_FIN|rettelöidä|Act+Ind+Prs+Sg3": "rettelöi",
190
+ "VERB_FIN|rähinöidä|Act+Imprt+Sg2": "rähinöi",
191
+ "VERB_FIN|rähinöidä|Act+Ind+Prs+ConNeg": "rähinöi",
192
+ "VERB_FIN|rähinöidä|Act+Ind+Prs+Pl1": "rähinöimme",
193
+ "VERB_FIN|rähinöidä|Act+Ind+Prs+Pl3": "rähinöivät",
194
+ "VERB_FIN|rähinöidä|Act+Ind+Prs+Sg3": "rähinöi",
195
+ "VERB_FIN|seppelöidä|Act+Imprt+Sg2": "seppelöi",
196
+ "VERB_FIN|seppelöidä|Act+Ind+Prs+ConNeg": "seppelöi",
197
+ "VERB_FIN|seppelöidä|Act+Ind+Prs+Pl1": "seppelöimme",
198
+ "VERB_FIN|seppelöidä|Act+Ind+Prs+Pl3": "seppelöivät",
199
+ "VERB_FIN|seppelöidä|Act+Ind+Prs+Sg3": "seppelöi",
200
+ "VERB_FIN|sukuloida|Act+Imprt+Sg2": "sukuloi",
201
+ "VERB_FIN|sukuloida|Act+Ind+Prs+ConNeg": "sukuloi",
202
+ "VERB_FIN|sukuloida|Act+Ind+Prs+Pl1": "sukuloimme",
203
+ "VERB_FIN|sukuloida|Act+Ind+Prs+Pl3": "sukuloivat",
204
+ "VERB_FIN|sukuloida|Act+Ind+Prs+Sg2": "sukuloit",
205
+ "VERB_FIN|sukuloida|Act+Ind+Prs+Sg3": "sukuloi",
206
+ "VERB_FIN|säikkyä|Act+Imprt+Sg2": "säiky",
207
+ "VERB_FIN|säikkyä|Act+Ind+Prs+ConNeg": "säiky",
208
+ "VERB_FIN|säikkyä|Act+Ind+Prs+Pl1": "säikymme",
209
+ "VERB_FIN|säikkyä|Act+Ind+Prs+Pl2": "säikytte",
210
+ "VERB_FIN|säikkyä|Act+Ind+Prs+Pl3": "säikkyvät",
211
+ "VERB_FIN|säikkyä|Act+Ind+Prs+Sg1": "säikyn",
212
+ "VERB_FIN|säikkyä|Act+Ind+Prs+Sg2": "säikyt",
213
+ "VERB_FIN|säikkyä|Act+Ind+Prs+Sg3": "säikkyy",
214
+ "VERB_FIN|tuikkaa|Act+Ind+Prs+Pl3": "tuikkavat",
215
+ "VERB_FIN|tulla|Act+Imprt+Sg2": "tule",
216
+ "VERB_FIN|tulla|Act+Ind+Prs+Pl1": "tulemme",
217
+ "VERB_FIN|tulla|Act+Ind+Prs+Pl2": "tulette",
218
+ "VERB_FIN|tulla|Act+Ind+Prs+Sg1": "tulen",
219
+ "VERB_FIN|tulla|Act+Ind+Prs+Sg2": "tulet",
220
+ "VERB_FIN|tupakoida|Act+Imprt+Sg2": "tupakoi",
221
+ "VERB_FIN|tupakoida|Act+Ind+Prs+ConNeg": "tupakoi",
222
+ "VERB_FIN|tupakoida|Act+Ind+Prs+Pl1": "tupakoimme",
223
+ "VERB_FIN|tupakoida|Act+Ind+Prs+Pl3": "tupakoivat",
224
+ "VERB_FIN|tupakoida|Act+Ind+Prs+Sg2": "tupakoit",
225
+ "VERB_FIN|tupakoida|Act+Ind+Prs+Sg3": "tupakoi",
226
+ "VERB_FIN|urakoida|Act+Imprt+Sg2": "urakoi",
227
+ "VERB_FIN|urakoida|Act+Ind+Prs+ConNeg": "urakoi",
228
+ "VERB_FIN|urakoida|Act+Ind+Prs+Pl1": "urakoimme",
229
+ "VERB_FIN|urakoida|Act+Ind+Prs+Pl3": "urakoivat",
230
+ "VERB_FIN|urakoida|Act+Ind+Prs+Sg2": "urakoit",
231
+ "VERB_FIN|urakoida|Act+Ind+Prs+Sg3": "urakoi",
232
+ "VERB_FIN|vihannoida|Act+Imprt+Sg2": "vihannoi",
233
+ "VERB_FIN|vihannoida|Act+Ind+Prs+ConNeg": "vihannoi",
234
+ "VERB_FIN|vihannoida|Act+Ind+Prs+Pl1": "vihannoimme",
235
+ "VERB_FIN|vihannoida|Act+Ind+Prs+Pl3": "vihannoivat",
236
+ "VERB_FIN|vihannoida|Act+Ind+Prs+Sg2": "vihannoit",
237
+ "VERB_FIN|vihannoida|Act+Ind+Prs+Sg3": "vihannoi",
238
+ "VERB_FIN|viheriöidä|Act+Imprt+Sg2": "viheriöi",
239
+ "VERB_FIN|viheriöidä|Act+Ind+Prs+ConNeg": "viheriöi",
240
+ "VERB_FIN|viheriöidä|Act+Ind+Prs+Pl1": "viheriöimme",
241
+ "VERB_FIN|viheriöidä|Act+Ind+Prs+Pl3": "viheriöivät",
242
+ "VERB_FIN|viheriöidä|Act+Ind+Prs+Sg2": "viheriöit",
243
+ "VERB_FIN|viheriöidä|Act+Ind+Prs+Sg3": "viheriöi"
244
+ }
juurio/genlex_overrides_noun.json ADDED
The diff for this file is too large to render. See raw diff
 
juurio/morphforms.py ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Shared rule-engine + realizer helpers.
2
+
3
+ Pure-Python utilities used across the transform engine (`algebra`) and the OOV realizers: locating a
4
+ clause's main finite verb, joining tokens back into a surface string, capitalizing the result, and the
5
+ nonce-wug initial-consonant swap. No world knowledge and no external morphology backend — just
6
+ grammar-mechanical string/parse helpers.
7
+ """
8
+
9
+ from __future__ import annotations
10
+
11
+ CONSONANTS = list("kptsvnmlrhj")
12
+
13
+
14
+ def _main_verb(sent):
15
+ """Return (verb_token, is_copula). The finite Act/Ind verb to operate on: the root if
16
+ it is one, else the clause's copula."""
17
+ root = next((t for t in sent if t["head"] == 0), None)
18
+ if root is None:
19
+ return None, False
20
+ def finite_act_ind(t):
21
+ f = t["feats"]
22
+ return (t["upos"] in ("VERB", "AUX") and f.get("VerbForm") == "Fin"
23
+ and f.get("Voice", "Act") == "Act" and f.get("Mood", "Ind") == "Ind"
24
+ and f.get("Tense") in ("Pres", "Past"))
25
+ if finite_act_ind(root):
26
+ return root, False
27
+ cop = next((t for t in sent if t["head"] == root["id"] and t["deprel"] == "cop"
28
+ and finite_act_ind(t)), None)
29
+ if cop:
30
+ return cop, True
31
+ return None, False
32
+
33
+
34
+ def _join(words):
35
+ s = " ".join(words)
36
+ for p in (" .", " ,", " ?", " !", " :", " ;"):
37
+ s = s.replace(p, p.strip())
38
+ for sup in ("²", "³", "¹"): # Stanza splits `km²`->`km ²`; gold keeps it joined
39
+ s = s.replace(" " + sup, sup)
40
+ return s
41
+
42
+
43
+ def target_str(r):
44
+ """Transformed sentence, first letter capitalized (an inserted leading word — e.g. `ei` — may now
45
+ lead)."""
46
+ s = _join(r[1])
47
+ return s[:1].upper() + s[1:] if s else s
48
+
49
+
50
+ # ---------------------------- nonce wug ----------------------------
51
+ def swap_initial(word, repl):
52
+ """Swap the word's first consonant (or prepend one if vowel-initial) — a stable,
53
+ paradigm-wide edit, since Finnish gradation never touches the word-initial
54
+ consonant. Applied identically to every form of the same verb."""
55
+ if not word:
56
+ return word
57
+ if word[0].lower() in CONSONANTS:
58
+ return repl + word[1:]
59
+ return repl + word
juurio/parser/__init__.py ADDED
@@ -0,0 +1 @@
 
 
1
+ """Compact Finnish UD parser."""
juurio/parser/case_augment.py ADDED
@@ -0,0 +1,465 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Targeted rare-case morphological augmentation.
2
+
3
+ Behaviour:
4
+ - Re-realize a single nominal head (NOUN/PROPN/ADJ) in a sentence
5
+ into a target Case; walk amod/det dependents and re-realize them
6
+ in agreement.
7
+ - Any failure (head doesn't realize, dependent doesn't realize,
8
+ Voikko rejects an output form) → skip the example entirely.
9
+ - Token count preserved by construction (single-token-in, single-
10
+ token-out per re-realized position).
11
+ - Used to grow rare-Case coverage where the BCE Feats head
12
+ under-fires on rarer cases (Tra/Abl/Ins/Ade/All).
13
+
14
+ This module does NOT modify the model. It produces AugSentence
15
+ objects (list of token dicts with updated form/feats); a downstream
16
+ loader emits the same collator batch keys but flips per-example loss
17
+ masks so only the Feats loss contributes to the gradient.
18
+
19
+ Eligibility:
20
+ - Heads: NOUN, PROPN, ADJ (with Case feature). NOT PRON, NOT NUM.
21
+ - Dependents re-realized: amod, det only. NOT nummod (deferred).
22
+ - Excluded from re-realization neighbourhoods:
23
+ flat, fixed, compound, appos, conj, punct — any nominal head
24
+ with one of these in its subtree is skipped.
25
+ """
26
+ from __future__ import annotations
27
+
28
+ import collections
29
+ import functools
30
+ import json
31
+ import os
32
+ import random
33
+ from dataclasses import dataclass, field
34
+ from typing import Iterable, Iterator
35
+
36
+ from .silver import SilverSentence, SilverToken
37
+
38
+ ELIGIBLE_HEAD_UPOS = ("NOUN", "PROPN", "ADJ")
39
+ AGREEMENT_DEPRELS = ("amod", "det") # nummod deferred
40
+ # Dependents are only re-realized if their UPOS is also in this allow-list.
41
+ # PRON/DET fall back to lemma in the current realizer, producing feat↔form
42
+ # mismatches like `samalla→sama`, `missään→mikään`. Add DET later behind a
43
+ # tested PRON/DET adapter.
44
+ ELIGIBLE_DEP_UPOS = ("ADJ", "NOUN")
45
+ EXCLUDED_DEPRELS_IN_SUBTREE = ("flat", "fixed", "compound", "appos", "conj")
46
+
47
+ # Target cases where the Feats head tends to under-fire:
48
+ PRIORITY_RARE_CASES = ("Tra", "Abl", "Ins", "Ade", "All", "Ess", "Abe", "Ill", "Ine")
49
+ ANCHOR_COMMON_CASES = ("Nom", "Gen", "Par")
50
+
51
+
52
+ @dataclass
53
+ class AugSentence:
54
+ """Augmented gold sentence. Same shape as SilverSentence; carries
55
+ the metadata a downstream collator needs to flip loss masks.
56
+
57
+ `feats_changed_token_ids` is for diagnostic only — we keep the
58
+ Feats loss as per-sentence-active, not per-token.
59
+ """
60
+ sentence: SilverSentence
61
+ target_case: str
62
+ target_number: str
63
+ head_token_id: int
64
+ dependents_changed: list[int] = field(default_factory=list)
65
+ feats_changed_token_ids: list[int] = field(default_factory=list)
66
+ # Per-sentence loss-mask flags:
67
+ # feats_mask=True (contribute), all others=False (no parser/upos/
68
+ # lemma/compound supervision on augmented examples).
69
+ feats_loss: bool = True
70
+ upos_loss: bool = False
71
+ lemma_loss: bool = False
72
+ parser_loss: bool = False
73
+ compound_loss: bool = False
74
+
75
+
76
+ def _parse_feats(s: str) -> dict[str, str]:
77
+ if not s or s == "_":
78
+ return {}
79
+ return dict(kv.split("=", 1) for kv in s.split("|") if "=" in kv)
80
+
81
+
82
+ def _format_feats(d: dict[str, str]) -> str:
83
+ if not d:
84
+ return "_"
85
+ return "|".join(f"{k}={v}" for k, v in sorted(d.items()))
86
+
87
+
88
+ def _children(sent: list, parent_id: int) -> list:
89
+ return [t for t in sent if t.head == parent_id]
90
+
91
+
92
+ def _subtree_token_ids(sent: list, root_id: int) -> set[int]:
93
+ out = {root_id}
94
+ stack = [root_id]
95
+ while stack:
96
+ x = stack.pop()
97
+ for c in _children(sent, x):
98
+ if c.id not in out:
99
+ out.add(c.id)
100
+ stack.append(c.id)
101
+ return out
102
+
103
+
104
+ def is_eligible_head(token, sent: list) -> bool:
105
+ """Eligibility: head must be NOUN/PROPN/ADJ with Case, and its
106
+ subtree must not contain flat/fixed/compound/appos/conj/punct
107
+ deprels."""
108
+ if token.upos not in ELIGIBLE_HEAD_UPOS:
109
+ return False
110
+ feats = _parse_feats(token.feats)
111
+ if "Case" not in feats:
112
+ return False
113
+ # Subtree check: walk the dependents of this token and reject if
114
+ # any are in the excluded set.
115
+ for cid in _subtree_token_ids(sent, token.id):
116
+ if cid == token.id:
117
+ continue
118
+ c = next((t for t in sent if t.id == cid), None)
119
+ if c is None:
120
+ continue
121
+ if c.deprel in EXCLUDED_DEPRELS_IN_SUBTREE:
122
+ return False
123
+ return True
124
+
125
+
126
+ def case_shift_one(
127
+ sent: SilverSentence,
128
+ head_token_id: int,
129
+ target_case: str,
130
+ *,
131
+ target_number: str | None = None,
132
+ voikko_check: bool = True,
133
+ ) -> AugSentence | None:
134
+ """Re-realize one nominal head + its amod/det agreement modifiers
135
+ in target_case. Return AugSentence or None (silent skip).
136
+
137
+ Args:
138
+ sent: gold SilverSentence (will not be mutated; a
139
+ new AugSentence with a new SilverSentence is
140
+ returned)
141
+ head_token_id: the id (1-indexed) of the head NOUN/ADJ/PROPN
142
+ target_case: one of the productive Finnish cases.
143
+ target_number: if None, keep the head's current Number.
144
+ voikko_check: if True (default), every re-realized form is
145
+ validated via uralicApi.analyze; any failure
146
+ → return None.
147
+
148
+ Returns:
149
+ AugSentence with feats_loss=True and all other masks False, or
150
+ None if eligibility/realizer/Voikko anything failed.
151
+ """
152
+ # Locate head token
153
+ head = next((t for t in sent.tokens if t.id == head_token_id), None)
154
+ if head is None or not is_eligible_head(head, sent.tokens):
155
+ return None
156
+ head_feats = _parse_feats(head.feats)
157
+ if target_number is None:
158
+ target_number = head_feats.get("Number", "Sing")
159
+
160
+ # Re-realize the head
161
+ head_form_new = _cached_realize_np_form(
162
+ head.lemma, head.upos, target_case, target_number,
163
+ )
164
+ if not head_form_new:
165
+ return None
166
+ # Preserve the original token's capitalization convention (the
167
+ # realizer always returns lower-case forms because lemmas are
168
+ # lower-case; without this, sentence-initial tokens become
169
+ # mid-sentence-looking).
170
+ head_form_new = _match_case_pattern(head.form, head_form_new)
171
+ if voikko_check and not _voikko_has_target(
172
+ head_form_new, target_case, target_number,
173
+ ):
174
+ return None
175
+
176
+ # Walk amod/det dependents and re-realize each in (target_case, target_number).
177
+ # Dep UPOS must be in ELIGIBLE_DEP_UPOS — PRON/DET would lemma-fallback
178
+ # in the current realizer, so we leave them surface-untouched (their Case
179
+ # feat stays original, which is the safe semantic: feat-form consistency).
180
+ deps_to_rewrite = [t for t in sent.tokens
181
+ if t.head == head_token_id
182
+ and t.deprel in AGREEMENT_DEPRELS
183
+ and t.upos in ELIGIBLE_DEP_UPOS]
184
+ dep_new_forms: dict[int, str] = {}
185
+ for d in deps_to_rewrite:
186
+ new = _cached_realize_np_form(
187
+ d.lemma, d.upos, target_case, target_number,
188
+ )
189
+ if not new:
190
+ return None
191
+ new = _match_case_pattern(d.form, new)
192
+ if voikko_check and not _voikko_has_target(
193
+ new, target_case, target_number,
194
+ ):
195
+ return None
196
+ dep_new_forms[d.id] = new
197
+
198
+ # Build the new token list with updated forms + Feats on head + deps
199
+ new_tokens = []
200
+ feats_changed_ids = []
201
+ changed_dep_ids = []
202
+ for t in sent.tokens:
203
+ if t.id == head.id:
204
+ new_feats = dict(head_feats)
205
+ new_feats["Case"] = target_case
206
+ new_feats["Number"] = target_number
207
+ new_tokens.append(SilverToken(
208
+ form=head_form_new,
209
+ lemma=t.lemma,
210
+ upos=t.upos,
211
+ feats=_format_feats(new_feats),
212
+ head=t.head, deprel=t.deprel,
213
+ ))
214
+ feats_changed_ids.append(t.id)
215
+ elif t.id in dep_new_forms:
216
+ d_feats = _parse_feats(t.feats)
217
+ d_feats["Case"] = target_case
218
+ d_feats["Number"] = target_number
219
+ new_tokens.append(SilverToken(
220
+ form=dep_new_forms[t.id],
221
+ lemma=t.lemma,
222
+ upos=t.upos,
223
+ feats=_format_feats(d_feats),
224
+ head=t.head, deprel=t.deprel,
225
+ ))
226
+ feats_changed_ids.append(t.id)
227
+ changed_dep_ids.append(t.id)
228
+ else:
229
+ new_tokens.append(t)
230
+
231
+ new_text = " ".join(tok.form for tok in new_tokens)
232
+ new_sent = SilverSentence(text=new_text, tokens=new_tokens)
233
+ return AugSentence(
234
+ sentence=new_sent,
235
+ target_case=target_case,
236
+ target_number=target_number,
237
+ head_token_id=head.id,
238
+ dependents_changed=changed_dep_ids,
239
+ feats_changed_token_ids=feats_changed_ids,
240
+ )
241
+
242
+
243
+ def _match_case_pattern(orig: str, new: str) -> str:
244
+ """Re-apply the original token's leading-capitalization convention
245
+ to the realizer's lower-case output. The realizer is lemma-driven
246
+ (lemmas are lower-case), so a sentence-initial token like `Jäällä`
247
+ comes back as `jääksi` without this fix.
248
+
249
+ - all-uppercase original (and length > 1) → all-upper new
250
+ - title-case original (leading upper, rest mixed) → leading-upper new
251
+ - everything else → leave as-is
252
+ """
253
+ if not orig or not new:
254
+ return new
255
+ if len(orig) > 1 and orig.isupper():
256
+ return new.upper()
257
+ if orig[:1].isupper():
258
+ return new[:1].upper() + new[1:]
259
+ return new
260
+
261
+
262
+ # Mapping from UD Case names to the Giella FST tag suffix used by
263
+ # uralicNLP's Voikko-fin transducer.
264
+ _UD_CASE_TO_FST_TAG = {
265
+ "Nom": "Nom", "Gen": "Gen", "Par": "Par",
266
+ "Ess": "Ess", "Tra": "Tra", "Ine": "Ine",
267
+ "Ela": "Ela", "Ill": "Ill", "Ade": "Ade",
268
+ "Abl": "Abl", "All": "All", "Abe": "Abe",
269
+ "Ins": "Ins", "Com": "Com",
270
+ }
271
+ # UD Number → FST tag (uralicApi uses Sg / Pl).
272
+ _UD_NUMBER_TO_FST_TAG = {"Sing": "Sg", "Plur": "Pl"}
273
+
274
+
275
+ @functools.lru_cache(maxsize=200_000)
276
+ def _voikko_analyses(form: str) -> tuple[str, ...]:
277
+ """Cached raw analyses for a form. Returns a tuple of analysis
278
+ strings like `vaihe+N+Sg+Ine`. Empty tuple means Voikko cannot
279
+ analyze the form at all (= the form is not Finnish).
280
+
281
+ uralicApi.analyze returns a list of `(analysis_string, weight)`
282
+ tuples — extract the string component only.
283
+ """
284
+ try:
285
+ from uralicNLP import uralicApi
286
+ raw = uralicApi.analyze(form, "fin")
287
+ return tuple(r[0] if isinstance(r, tuple) else r for r in raw)
288
+ except Exception:
289
+ return ()
290
+
291
+
292
+ def _voikko_ok(form: str) -> bool:
293
+ """Loose check: any Voikko analysis exists."""
294
+ return bool(_voikko_analyses(form))
295
+
296
+
297
+ def _voikko_has_target(form: str, case: str, number: str) -> bool:
298
+ """Strict check: require at least one Voikko analysis whose tag
299
+ string contains both the target Case tag (e.g. `+Tra`) and the
300
+ target Number tag (`+Sg` or `+Pl`). Catches the `sama → sama`
301
+ style lemma fallback where Voikko accepts the surface form (it IS
302
+ valid Finnish) but only as Nom-Sing, not as the requested target.
303
+ """
304
+ fst_case = _UD_CASE_TO_FST_TAG.get(case)
305
+ fst_num = _UD_NUMBER_TO_FST_TAG.get(number)
306
+ if fst_case is None or fst_num is None:
307
+ # Defensive: unknown case/number → fall back to loose check.
308
+ return _voikko_ok(form)
309
+ case_tag = "+" + fst_case
310
+ num_tag = "+" + fst_num
311
+ for a in _voikko_analyses(form):
312
+ if case_tag in a and num_tag in a:
313
+ return True
314
+ return False
315
+
316
+
317
+ @functools.lru_cache(maxsize=200_000)
318
+ def _cached_realize_np_form(lemma: str, upos: str, case: str,
319
+ number: str) -> str | None:
320
+ """Cached wrapper around juurio.algebra.realize_np_form. Cuts
321
+ walltime from minutes to seconds on a re-run."""
322
+ from ..algebra import realize_np_form
323
+ return realize_np_form(lemma, upos=upos, case=case, number=number)
324
+
325
+
326
+ def make_augmentation_corpus(
327
+ sents: list,
328
+ *,
329
+ target_cases: tuple = PRIORITY_RARE_CASES,
330
+ anchor_cases: tuple = ANCHOR_COMMON_CASES,
331
+ per_case_cap: int = 3000,
332
+ anchor_mix_pct: float = 0.10,
333
+ total_cap: int | None = 30000,
334
+ rng: random.Random | None = None,
335
+ voikko_check: bool = True,
336
+ ) -> list[AugSentence]:
337
+ """Stratified sampling. For each target_case, walk sents, collect
338
+ up to per_case_cap successful AugSentences. Then add anchor_mix_pct
339
+ of total_cap from anchor_cases (common cases) so the model doesn't
340
+ memorize 'augmented = rare'.
341
+
342
+ Returns the full list shuffled (so DataLoader doesn't see
343
+ target-case clumps).
344
+ """
345
+ rng = rng or random.Random(0)
346
+ out: list[AugSentence] = []
347
+ by_case: dict[str, list[AugSentence]] = collections.defaultdict(list)
348
+
349
+ # Pre-shuffle the sentences to vary head selection across cases
350
+ sent_order = list(sents)
351
+ rng.shuffle(sent_order)
352
+
353
+ for case in list(target_cases) + list(anchor_cases):
354
+ for sent in sent_order:
355
+ if len(by_case[case]) >= per_case_cap:
356
+ break
357
+ for tok in sent.tokens:
358
+ if tok.upos not in ELIGIBLE_HEAD_UPOS:
359
+ continue
360
+ aug = case_shift_one(sent, tok.id, case,
361
+ voikko_check=voikko_check)
362
+ if aug is not None:
363
+ by_case[case].append(aug)
364
+ break # one head per sentence
365
+ if len(by_case[case]) >= per_case_cap:
366
+ break
367
+
368
+ # Build the stratified corpus respecting total_cap + anchor_mix_pct
369
+ rare_cap = int((1.0 - anchor_mix_pct) * total_cap) if total_cap else None
370
+ anchor_cap = (total_cap - rare_cap) if total_cap else None
371
+ rare_pool = []
372
+ for case in target_cases:
373
+ rare_pool.extend(by_case[case])
374
+ rng.shuffle(rare_pool)
375
+ if rare_cap is not None and len(rare_pool) > rare_cap:
376
+ rare_pool = rare_pool[:rare_cap]
377
+ anchor_pool = []
378
+ for case in anchor_cases:
379
+ anchor_pool.extend(by_case[case])
380
+ rng.shuffle(anchor_pool)
381
+ if anchor_cap is not None and len(anchor_pool) > anchor_cap:
382
+ anchor_pool = anchor_pool[:anchor_cap]
383
+
384
+ final = rare_pool + anchor_pool
385
+ rng.shuffle(final)
386
+ return final
387
+
388
+
389
+ # ----------------------------- JSONL I/O ------------------------------------
390
+ # Persist the augmented corpus to disk so the training driver doesn't rebuild
391
+ # it on every restart (~10 min Voikko walk otherwise).
392
+
393
+ def _aug_to_dict(aug: "AugSentence") -> dict:
394
+ """Serialise an AugSentence to a JSON-able dict. SilverToken is a flat
395
+ dataclass; AugSentence wraps SilverSentence + per-sentence mask
396
+ flags + provenance metadata."""
397
+ s = aug.sentence
398
+ return {
399
+ "sentence": {
400
+ "text": s.text,
401
+ "tokens": [
402
+ {"form": t.form, "lemma": t.lemma, "upos": t.upos,
403
+ "feats": t.feats, "head": t.head, "deprel": t.deprel}
404
+ for t in s.tokens
405
+ ],
406
+ },
407
+ "target_case": aug.target_case,
408
+ "target_number": aug.target_number,
409
+ "head_token_id": aug.head_token_id,
410
+ "dependents_changed": list(aug.dependents_changed),
411
+ "feats_changed_token_ids": list(aug.feats_changed_token_ids),
412
+ "feats_loss": aug.feats_loss,
413
+ "upos_loss": aug.upos_loss,
414
+ "lemma_loss": aug.lemma_loss,
415
+ "parser_loss": aug.parser_loss,
416
+ "compound_loss": aug.compound_loss,
417
+ }
418
+
419
+
420
+ def _aug_from_dict(obj: dict) -> "AugSentence":
421
+ s = obj["sentence"]
422
+ sent = SilverSentence(
423
+ text=s["text"],
424
+ tokens=[SilverToken(**t) for t in s["tokens"]],
425
+ )
426
+ return AugSentence(
427
+ sentence=sent,
428
+ target_case=obj["target_case"],
429
+ target_number=obj["target_number"],
430
+ head_token_id=obj["head_token_id"],
431
+ dependents_changed=list(obj.get("dependents_changed", [])),
432
+ feats_changed_token_ids=list(obj.get("feats_changed_token_ids", [])),
433
+ feats_loss=obj.get("feats_loss", True),
434
+ upos_loss=obj.get("upos_loss", False),
435
+ lemma_loss=obj.get("lemma_loss", False),
436
+ parser_loss=obj.get("parser_loss", False),
437
+ compound_loss=obj.get("compound_loss", False),
438
+ )
439
+
440
+
441
+ def write_aug_jsonl(records: Iterable["AugSentence"], path: str,
442
+ mode: str = "w") -> int:
443
+ """Write `records` as one JSON object per line. Returns the count.
444
+ `mode='w'` truncates; `mode='a'` appends (resume-safe)."""
445
+ if mode not in ("w", "a"):
446
+ raise ValueError(f"mode must be 'w' or 'a', got {mode!r}")
447
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".",
448
+ exist_ok=True)
449
+ n = 0
450
+ with open(path, mode, encoding="utf-8") as fh:
451
+ for a in records:
452
+ fh.write(json.dumps(_aug_to_dict(a), ensure_ascii=False))
453
+ fh.write("\n")
454
+ n += 1
455
+ return n
456
+
457
+
458
+ def read_aug_jsonl(path: str) -> Iterator["AugSentence"]:
459
+ """Stream AugSentence records from a JSONL file."""
460
+ with open(path, encoding="utf-8") as fh:
461
+ for ln in fh:
462
+ ln = ln.strip()
463
+ if not ln:
464
+ continue
465
+ yield _aug_from_dict(json.loads(ln))
juurio/parser/char_vocab.py ADDED
@@ -0,0 +1,179 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Character vocabulary for the compound-lemma heads.
2
+
3
+ Mirrors `FstTagVocab` from `juurio.parser.fst_cache` in API shape and
4
+ implementation patterns, but operates on individual characters instead of
5
+ morphological tags.
6
+
7
+ Invariants:
8
+ - id 0 is always reserved for `<PAD>` (no-signal embedding).
9
+ - id 1 is always reserved for `<UNK>` (out-of-vocabulary character).
10
+ - Real characters start at id 2.
11
+ - The order of the character list is stable after `fit`; the SHA-256 hash
12
+ pins this order so a stale char vocab can't silently corrupt training.
13
+
14
+ Raises:
15
+ ValueError: if `chars` has fewer than 2 elements, or if `chars[0]` is not
16
+ `"<PAD>"` or `chars[1]` is not `"<UNK>"`. (Note: this was previously an
17
+ `AssertionError`, which is silently elided under ``python -O``.)
18
+ """
19
+ from __future__ import annotations
20
+
21
+ import hashlib
22
+ import json
23
+ from collections import Counter
24
+ from pathlib import Path
25
+ from typing import Iterable
26
+
27
+
28
+ PAD_ID = 0
29
+ UNK_ID = 1
30
+ PAD_CHAR = "<PAD>"
31
+ UNK_CHAR = "<UNK>"
32
+
33
+
34
+ class CharVocab:
35
+ """Maps individual character strings to integer ids.
36
+
37
+ Reserved ids:
38
+ PAD_ID = 0 — padding / no-signal position.
39
+ UNK_ID = 1 — out-of-vocabulary character.
40
+
41
+ Real characters start at id 2. The constructor raises `ValueError` if
42
+ `chars` has fewer than 2 entries, or if `chars[0]` is not `"<PAD>"` or
43
+ `chars[1]` is not `"<UNK>"`, to prevent accidental construction with wrong
44
+ ordering (which would silently feed the wrong embeddings).
45
+ """
46
+
47
+ PAD_ID: int = PAD_ID
48
+ UNK_ID: int = UNK_ID
49
+ PAD_CHAR: str = PAD_CHAR
50
+ UNK_CHAR: str = UNK_CHAR
51
+
52
+ def __init__(self, chars: list[str]):
53
+ """Construct a CharVocab from an ordered character list.
54
+
55
+ `chars[0]` must be `"<PAD>"` and `chars[1]` must be `"<UNK>"`;
56
+ real characters follow from index 2 onward. Violating this ordering
57
+ raises `ValueError`.
58
+
59
+ Raises:
60
+ ValueError: if `chars` has fewer than 2 elements, or if
61
+ `chars[0]` is not `"<PAD>"` or `chars[1]` is not `"<UNK>"`.
62
+ """
63
+ if len(chars) < 2:
64
+ raise ValueError(
65
+ "chars must have at least 2 entries; "
66
+ "PAD at index 0 and UNK at index 1"
67
+ )
68
+ if chars[0] != PAD_CHAR or chars[1] != UNK_CHAR:
69
+ raise ValueError(
70
+ f"chars[0] must be '{PAD_CHAR}' and chars[1] must be '{UNK_CHAR}'; "
71
+ f"got chars[0]={chars[0]!r}, chars[1]={chars[1]!r}"
72
+ )
73
+ self._chars = list(chars)
74
+ self._char_to_id = {c: i for i, c in enumerate(chars)}
75
+
76
+ @property
77
+ def size(self) -> int:
78
+ """Total vocabulary size including PAD and UNK."""
79
+ return len(self._chars)
80
+
81
+ @classmethod
82
+ def fit(
83
+ cls,
84
+ forms: Iterable[str],
85
+ *,
86
+ min_count: int = 1,
87
+ max_chars: int | None = None,
88
+ ) -> "CharVocab":
89
+ """Fit a CharVocab from an iterable of surface forms.
90
+
91
+ Collects all individual characters from every form, counts their
92
+ frequencies, drops chars seen fewer than `min_count` times, and
93
+ optionally truncates to the top `max_chars` by frequency. Returns a
94
+ new `CharVocab` with PAD and UNK prepended.
95
+
96
+ Args:
97
+ forms: Iterable of string forms (e.g. word surface forms).
98
+ min_count: Minimum character frequency to keep. Characters seen
99
+ fewer times are mapped to UNK at encoding time.
100
+ max_chars: If given, keep only the top-`max_chars` most frequent
101
+ characters (after the `min_count` filter).
102
+
103
+ Returns:
104
+ A new `CharVocab` instance.
105
+ """
106
+ c: Counter[str] = Counter()
107
+ for form in forms:
108
+ for ch in form:
109
+ c[ch] += 1
110
+ kept = [ch for ch, n in c.most_common() if n >= min_count]
111
+ if max_chars is not None:
112
+ kept = kept[:max_chars]
113
+ return cls([PAD_CHAR, UNK_CHAR] + kept)
114
+
115
+ def encode_char(self, c: str) -> int:
116
+ """Return the integer id for character `c`, or UNK_ID if unseen."""
117
+ return self._char_to_id.get(c, UNK_ID)
118
+
119
+ def encode_form(self, form: str, max_len: int) -> tuple[list[int], list[bool]]:
120
+ """Encode a surface form as a fixed-length sequence of char ids.
121
+
122
+ Forms shorter than `max_len` are padded on the right with PAD_ID;
123
+ forms longer than `max_len` are silently truncated to `max_len` chars.
124
+ The boolean mask is True for real character positions and False for
125
+ padding positions.
126
+
127
+ Args:
128
+ form: The surface form to encode.
129
+ max_len: Target sequence length (output length is always max_len).
130
+
131
+ Returns:
132
+ (char_ids, char_mask) — both have length `max_len`.
133
+ """
134
+ # Truncate to max_len
135
+ truncated = form[:max_len]
136
+ real_len = len(truncated)
137
+
138
+ char_ids: list[int] = [self._char_to_id.get(c, UNK_ID) for c in truncated]
139
+ char_mask: list[bool] = [True] * real_len
140
+
141
+ # Pad to max_len
142
+ pad_count = max_len - real_len
143
+ char_ids.extend([PAD_ID] * pad_count)
144
+ char_mask.extend([False] * pad_count)
145
+
146
+ return char_ids, char_mask
147
+
148
+ def hash(self) -> str:
149
+ """SHA-256 digest of the ordered character list (first 16 hex chars).
150
+
151
+ This short digest is used to pin the char vocab to a trained checkpoint
152
+ so that a stale or differently-fitted vocab can't silently feed wrong
153
+ char embeddings to a loaded model. The hash is order-sensitive: two
154
+ vocabs with identical character sets but different orderings produce
155
+ different hashes.
156
+ """
157
+ h = hashlib.sha256()
158
+ for c in self._chars:
159
+ h.update(c.encode("utf-8"))
160
+ h.update(b"\x00")
161
+ return h.hexdigest()[:16]
162
+
163
+ def save(self, path: str) -> None:
164
+ """Serialise the vocab to JSON at `path`.
165
+
166
+ Creates parent directories if they do not exist. The format is::
167
+
168
+ {"chars": ["<PAD>", "<UNK>", "a", "b", ...]}
169
+ """
170
+ Path(path).parent.mkdir(parents=True, exist_ok=True)
171
+ with open(path, "w") as fh:
172
+ json.dump({"chars": self._chars}, fh, ensure_ascii=False)
173
+
174
+ @classmethod
175
+ def load(cls, path: str) -> "CharVocab":
176
+ """Load a previously saved CharVocab from JSON."""
177
+ with open(path) as fh:
178
+ blob = json.load(fh)
179
+ return cls(blob["chars"])
juurio/parser/collator.py ADDED
@@ -0,0 +1,466 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Batch collator for the tiny Finnish UD parser.
2
+
3
+ Turns a list of `SilverSentence` records into model-ready batched tensors. Solves three problems:
4
+
5
+ 1. **Word ↔ subword mismatch.** Each `Word` (UPOS / lemma / head / deprel) may map to several
6
+ BPE subwords. We place labels only at the *first subword* of each Word; non-first subwords
7
+ get `<pad>` (id=0) so the cross-entropy loss with `ignore_index=0` skips them. A
8
+ `first_subw` bool mask says where the real labels are.
9
+ 2. **UD head index → subword position.** UD `head=k` (1-based Word index, 0=root) becomes a
10
+ subword position: `head=0` → 0 (the ROOT token at `ids[:, 0]`); `head=k>0` → the first-subword
11
+ position of Word k.
12
+ 3. **Length capping.** Sentences whose subword length (incl. ROOT) exceeds `max_len` are
13
+ dropped — silently filtered before tensor construction.
14
+
15
+ The model's `forward(ids, mask)` consumes `ids` and `mask`. The other keys are training-loss
16
+ targets. `first_subw` is exposed so the loss can scatter / gather per-Word predictions cleanly,
17
+ and so calibration metrics can be computed per Word.
18
+
19
+ Calling convention:
20
+
21
+ col = SilverCollator(tokenizer=tok, vocab=vocab, max_len=256)
22
+ batch = col(list_of_silver_sentences)
23
+ out = model(batch["ids"], batch["mask"])
24
+ # loss heads compare out[head] against batch[head], gated by batch["first_subw"].
25
+ """
26
+ from __future__ import annotations
27
+
28
+ import torch
29
+
30
+ from .char_vocab import CharVocab
31
+ from .compound import CompoundTargets
32
+ from .fst_cache import FstCache, FstTagVocab
33
+ from .label_vocab import LabelVocab
34
+ from .model import PAD_ID, ROOT_ID
35
+ from .silver import SilverSentence
36
+ from .tokenizer import BPETokenizer
37
+
38
+
39
+ class SilverCollator:
40
+ def __init__(self, tokenizer: BPETokenizer, vocab: LabelVocab, max_len: int = 256,
41
+ *, fst_cache: FstCache | None = None,
42
+ fst_tag_vocab: FstTagVocab | None = None,
43
+ fst_max_tags: int = 16,
44
+ char_vocab: CharVocab | None = None,
45
+ max_form_len: int = 64,
46
+ compound_enabled: bool = False):
47
+ """Construct a SilverCollator.
48
+
49
+ Args:
50
+ tokenizer: BPE tokenizer.
51
+ vocab: LabelVocab for all task heads.
52
+ max_len: Maximum subword sequence length; longer sentences are dropped.
53
+ fst_cache: Optional FST analysis cache.
54
+ fst_tag_vocab: Optional FST tag vocabulary.
55
+ fst_max_tags: Maximum number of morphtags per token in the FST channel.
56
+ char_vocab: Character vocabulary for compound-lemma heads.
57
+ Required when `compound_enabled=True`.
58
+ max_form_len: Maximum character sequence length per token. Longer forms
59
+ are truncated silently.
60
+ compound_enabled: When True, emit the six compound-target tensors in every
61
+ batch dict (`form_chars`, `form_char_mask`, `compound_K`,
62
+ `compound_split_pos`, `compound_tail_id`, `compound_noisy`).
63
+ When False (default), no new keys and no new allocations.
64
+
65
+ Raises:
66
+ ValueError: if `compound_enabled=True` but `char_vocab=None`.
67
+ """
68
+ if compound_enabled and char_vocab is None:
69
+ raise ValueError(
70
+ "compound_enabled=True requires a char_vocab; got char_vocab=None. "
71
+ "Fit a CharVocab on your training forms and pass it here."
72
+ )
73
+ self.tokenizer = tokenizer
74
+ self.vocab = vocab
75
+ self.max_len = max_len
76
+ # FST channel — None means the batch omits `fst_tags`, which the model
77
+ # treats as a no-op (same forward as no-FST). When both cache and vocab are
78
+ # provided, every batch carries a (B, T, fst_max_tags) int tensor with the
79
+ # per-first-subw morphtag ids; padding positions are PAD_ID = 0.
80
+ self.fst_cache = fst_cache
81
+ self.fst_tag_vocab = fst_tag_vocab
82
+ self.fst_max_tags = fst_max_tags
83
+ # Compound channel
84
+ self.char_vocab = char_vocab
85
+ self.max_form_len = max_form_len
86
+ self.compound_enabled = compound_enabled
87
+
88
+ def __call__(self, sentences: list) -> dict[str, torch.Tensor]:
89
+ """Encode a batch.
90
+
91
+ Inputs may be plain SilverSentence (gold; all four loss masks
92
+ will be True) or AugSentence (carries its own per-sentence
93
+ (feats/upos/lemma/parser)_loss flags). When ANY input is an
94
+ AugSentence, the returned batch dict carries feats_mask /
95
+ upos_mask / parser_mask / lemma_mask, each a (B,) bool tensor
96
+ aligned with kept_indices (NOT the pre-collation position).
97
+ When all inputs are SilverSentence, no mask keys are emitted.
98
+ """
99
+ # AugSentence wraps a SilverSentence and carries
100
+ # per-sentence mask flags. Detect mixed-or-aug input cheaply.
101
+ from .case_augment import AugSentence # local: cold path
102
+ has_aug = any(isinstance(s, AugSentence) for s in sentences)
103
+
104
+ def _unwrap(s):
105
+ return s.sentence if isinstance(s, AugSentence) else s
106
+
107
+ per_sentence = []
108
+ kept_indices: list[int] = []
109
+ kept_mask_flags: list[tuple[bool, bool, bool, bool]] = []
110
+ kept_has_teacher: list[bool] = []
111
+ for i, s in enumerate(sentences):
112
+ inner = _unwrap(s)
113
+ enc = self._encode_one(inner)
114
+ if enc is None:
115
+ continue
116
+ per_sentence.append(enc)
117
+ kept_indices.append(i)
118
+ # Per-sentence auxiliary-target eligibility = inner sentence
119
+ # carried teacher_tokens. AugSentence wraps the inner Silver
120
+ # so teacher attaches on the inner just like for plain silver.
121
+ kept_has_teacher.append(getattr(inner, "teacher_tokens", None) is not None)
122
+ if isinstance(s, AugSentence):
123
+ kept_mask_flags.append((
124
+ s.feats_loss, s.upos_loss, s.parser_loss, s.lemma_loss,
125
+ ))
126
+ else:
127
+ kept_mask_flags.append((True, True, True, True))
128
+ if not per_sentence:
129
+ return self._empty_batch()
130
+ batch = self._pad_and_stack(per_sentence)
131
+ batch["kept_indices"] = torch.tensor(kept_indices, dtype=torch.long)
132
+ if has_aug:
133
+ flags = torch.tensor(kept_mask_flags, dtype=torch.bool)
134
+ batch["feats_mask"] = flags[:, 0].clone()
135
+ batch["upos_mask"] = flags[:, 1].clone()
136
+ batch["parser_mask"] = flags[:, 2].clone()
137
+ batch["lemma_mask"] = flags[:, 3].clone()
138
+ # Per-sentence auxiliary-target apply mask (kd_apply). Only emit when at
139
+ # least one kept sentence had teacher_tokens (matches the
140
+ # has_teacher branch in _pad_and_stack).
141
+ if any(kept_has_teacher):
142
+ batch["kd_apply"] = torch.tensor(kept_has_teacher, dtype=torch.bool)
143
+ return batch
144
+
145
+ # ----------------------------- per-sentence encoding -------------------------------------
146
+ def _encode_one(self, s: SilverSentence) -> dict[str, list] | None:
147
+ """Encode one sentence to a dict of python lists (one per output tensor key). Returns
148
+ None if the sentence overflows `max_len`."""
149
+ ids = [ROOT_ID]
150
+ first_subw = [False] # ROOT is real but carries no labels
151
+ word_first_positions: list[int] = []
152
+
153
+ # tokenize each Word's surface form separately so we can record first-subword positions
154
+ for t in s.tokens:
155
+ wpiece = self.tokenizer.encode(t.form)
156
+ if not wpiece:
157
+ # extremely rare: a Word that BPE emits 0 ids for. Treat it as a single <unk>.
158
+ wpiece = [self.tokenizer.unk_id]
159
+ word_first_positions.append(len(ids))
160
+ ids.extend(wpiece)
161
+ first_subw.append(True)
162
+ first_subw.extend([False] * (len(wpiece) - 1))
163
+
164
+ if len(ids) > self.max_len:
165
+ return None
166
+
167
+ T = len(ids)
168
+ upos = [0] * T
169
+ lemma = [0] * T
170
+ rel = [0] * T
171
+ head = [0] * T
172
+ feats = [[0] * self.vocab.n_feat for _ in range(T)]
173
+
174
+ # Per-token Feats mask carried from the
175
+ # cleaned silver corpus. Default True (= include). When
176
+ # any token has the flag set, the collator emits a (B, T)
177
+ # feats_token_mask tensor; otherwise the key is absent.
178
+ feats_token_mask = [True] * T
179
+ any_feats_token_masked = False
180
+
181
+ for k, t in enumerate(s.tokens):
182
+ pos = word_first_positions[k]
183
+ upos[pos] = self.vocab.encode_upos(t.upos)
184
+ lemma[pos] = self.vocab.encode_edit_script(t.form, t.lemma)
185
+ rel[pos] = self.vocab.encode_deprel(t.deprel)
186
+ feats[pos] = self.vocab.encode_feats_multihot(t.feats)
187
+ if getattr(t, "feats_loss_masked", False):
188
+ feats_token_mask[pos] = False
189
+ any_feats_token_masked = True
190
+ # head → subword position. head=0 → ROOT at position 0.
191
+ if t.head == 0:
192
+ head[pos] = 0
193
+ elif 1 <= t.head <= len(s.tokens):
194
+ head[pos] = word_first_positions[t.head - 1]
195
+ else:
196
+ # Malformed head (out of range). Drop the whole sentence rather than
197
+ # silently fold to root — folding fabricates a UD arc the silver never asserted.
198
+ return None
199
+ out = {
200
+ "ids": ids,
201
+ "first_subw": first_subw,
202
+ "upos": upos,
203
+ "feats": feats,
204
+ "lemma": lemma,
205
+ "head": head,
206
+ "rel": rel,
207
+ "T": T,
208
+ }
209
+ if any_feats_token_masked:
210
+ out["feats_token_mask"] = feats_token_mask
211
+
212
+ # Optional per-token auxiliary targets (training only).
213
+ # Emit teacher_* lists ONLY when this sentence carries teacher_tokens;
214
+ # the per-sentence kd_apply mask is set at the __call__ level so the
215
+ # batch-side aggregator knows which sentences contribute.
216
+ teacher_tokens = getattr(s, "teacher_tokens", None)
217
+ if teacher_tokens is not None:
218
+ if len(teacher_tokens) != len(s.tokens):
219
+ raise ValueError(
220
+ f"teacher_tokens length {len(teacher_tokens)} != "
221
+ f"sentence tokens {len(s.tokens)} — cache misalignment"
222
+ )
223
+ teacher_upos = [0] * T
224
+ teacher_lemma_edit = [-100] * T # IGNORE_INDEX outside first-subw
225
+ teacher_rel = [0] * T
226
+ teacher_head = [0] * T
227
+ teacher_feats = [[0] * self.vocab.n_feat for _ in range(T)]
228
+ for k, tt in enumerate(teacher_tokens):
229
+ pos = word_first_positions[k]
230
+ teacher_upos[pos] = self.vocab.encode_upos(tt.upos)
231
+ teacher_lemma_edit[pos] = tt.lemma_edit # may be -100 (OOV)
232
+ teacher_rel[pos] = self.vocab.encode_deprel(tt.deprel)
233
+ teacher_feats[pos] = self.vocab.encode_feats_multihot(tt.feats)
234
+ if tt.head == 0:
235
+ teacher_head[pos] = 0
236
+ elif 1 <= tt.head <= len(s.tokens):
237
+ teacher_head[pos] = word_first_positions[tt.head - 1]
238
+ else:
239
+ # Defensive: Stanza emitted a malformed head. Mark as ROOT
240
+ # to keep the arc-CE finite, but the training-time auxiliary
241
+ # arc loss uses first_subw masking so the gradient is bounded.
242
+ teacher_head[pos] = 0
243
+ out["teacher_upos"] = teacher_upos
244
+ out["teacher_lemma_edit"] = teacher_lemma_edit
245
+ out["teacher_rel"] = teacher_rel
246
+ out["teacher_head"] = teacher_head
247
+ out["teacher_feats"] = teacher_feats
248
+ # Optional auxiliary per-word hidden-state targets (training only).
249
+ # `finbert_hidden`, when present, is a fp16 numpy array of shape
250
+ # `(n_words, hidden_dim)`, ROOT excluded. We forward the raw array
251
+ # + the per-word first-subw positions to `_pad_and_stack`, which
252
+ # scatters rows onto the (B, T, H) batch tensor.
253
+ finbert_hidden = getattr(s, "finbert_hidden", None)
254
+ if finbert_hidden is not None:
255
+ if finbert_hidden.shape[0] != len(s.tokens):
256
+ raise ValueError(
257
+ f"finbert_hidden.shape[0]={finbert_hidden.shape[0]} != "
258
+ f"len(sentence.tokens)={len(s.tokens)} — cache misalignment"
259
+ )
260
+ out["finbert_hidden_word"] = finbert_hidden
261
+ out["finbert_hidden_first_pos"] = list(word_first_positions)
262
+ # FST tags: per first-subw position emit a fixed-length list of tag ids
263
+ # (PAD = 0 elsewhere). Length T × fst_max_tags, allocated per sentence so
264
+ # padding to batch max-T at stack-time stays simple.
265
+ if self.fst_cache is not None and self.fst_tag_vocab is not None:
266
+ fst_tags = [[0] * self.fst_max_tags for _ in range(T)]
267
+ for k, t in enumerate(s.tokens):
268
+ pos = word_first_positions[k]
269
+ fst_tags[pos] = self.fst_cache.form_to_tag_ids(
270
+ t.form, self.fst_tag_vocab, max_tags=self.fst_max_tags)
271
+ out["fst_tags"] = fst_tags
272
+ # Compound channel: emitted only when compound_enabled is True.
273
+ # All non-first-subw positions (incl. ROOT at index 0) are zero/False;
274
+ # the loss uses `first_subw` to gate which positions are supervised.
275
+ if self.compound_enabled:
276
+ mfl = self.max_form_len
277
+ # Allocate per-position containers, all zeroed/Falsy.
278
+ form_chars = [[0] * mfl for _ in range(T)] # (T, mfl) int
279
+ form_char_mask = [[False] * mfl for _ in range(T)] # (T, mfl) bool
280
+ compound_K = [0] * T # (T,) int
281
+ compound_split_pos = [[False] * mfl for _ in range(T)] # (T, mfl) bool
282
+ compound_tail_id = [0] * T # (T,) int
283
+ compound_noisy = [False] * T # (T,) bool
284
+
285
+ for k, tok_item in enumerate(s.tokens):
286
+ pos = word_first_positions[k]
287
+ # Encode the surface form into char ids (truncate/pad to mfl).
288
+ char_ids, char_mask = self.char_vocab.encode_form(tok_item.form, mfl)
289
+ form_chars[pos] = char_ids
290
+ form_char_mask[pos] = char_mask
291
+ # Compute gold compound targets for this (form, lemma) pair.
292
+ targets = CompoundTargets.from_form_lemma(
293
+ tok_item.form, tok_item.lemma, self.vocab.edit_to_id)
294
+ compound_K[pos] = targets["K"]
295
+ compound_tail_id[pos] = targets["tail_script_id"] if targets["K"] > 0 else 0
296
+ compound_noisy[pos] = targets["noisy_label"]
297
+ # Multi-hot split position vector: True at each gold boundary char
298
+ # position in the form (up to mfl; positions beyond mfl are silently
299
+ # dropped since the form is truncated there anyway).
300
+ split_vec = [False] * mfl
301
+ for bp in targets["boundary_positions"]:
302
+ if bp < mfl:
303
+ split_vec[bp] = True
304
+ compound_split_pos[pos] = split_vec
305
+
306
+ out["form_chars"] = form_chars
307
+ out["form_char_mask"] = form_char_mask
308
+ out["compound_K"] = compound_K
309
+ out["compound_split_pos"] = compound_split_pos
310
+ out["compound_tail_id"] = compound_tail_id
311
+ out["compound_noisy"] = compound_noisy
312
+ return out
313
+
314
+ # ----------------------------- pad + stack -----------------------------------------------
315
+ def _pad_and_stack(self, encs: list[dict]) -> dict[str, torch.Tensor]:
316
+ T = max(e["T"] for e in encs)
317
+ B = len(encs)
318
+ ids = torch.full((B, T), PAD_ID, dtype=torch.long)
319
+ mask = torch.zeros(B, T, dtype=torch.bool)
320
+ first_subw = torch.zeros(B, T, dtype=torch.bool)
321
+ upos = torch.zeros(B, T, dtype=torch.long)
322
+ lemma = torch.zeros(B, T, dtype=torch.long)
323
+ rel = torch.zeros(B, T, dtype=torch.long)
324
+ head = torch.zeros(B, T, dtype=torch.long)
325
+ feats = torch.zeros(B, T, self.vocab.n_feat, dtype=torch.float)
326
+
327
+ # FST tags optional — allocate only if any sentence has them.
328
+ has_fst = "fst_tags" in encs[0]
329
+ fst_tags = (torch.zeros(B, T, self.fst_max_tags, dtype=torch.long)
330
+ if has_fst else None)
331
+
332
+ # Per-token Feats mask. Only allocate if
333
+ # ANY encoded sentence carries a feats_token_mask list.
334
+ has_feats_token_mask = any("feats_token_mask" in e for e in encs)
335
+ if has_feats_token_mask:
336
+ # Default True — non-flagged sentences and pad positions
337
+ # both keep "include in Feats loss" (the per-position
338
+ # gating is already done by first_subw downstream).
339
+ feats_token_mask_t = torch.ones(B, T, dtype=torch.bool)
340
+
341
+ # Compound tensors — allocate only when compound_enabled.
342
+ has_compound = "form_chars" in encs[0]
343
+ mfl = self.max_form_len
344
+ if has_compound:
345
+ form_chars = torch.zeros(B, T, mfl, dtype=torch.long)
346
+ form_char_mask = torch.zeros(B, T, mfl, dtype=torch.bool)
347
+ compound_K = torch.zeros(B, T, dtype=torch.long)
348
+ compound_split_pos = torch.zeros(B, T, mfl, dtype=torch.bool)
349
+ compound_tail_id = torch.zeros(B, T, dtype=torch.long)
350
+ compound_noisy = torch.zeros(B, T, dtype=torch.bool)
351
+
352
+ # Optional per-token auxiliary training targets. Allocate when ANY
353
+ # sentence in the batch has teacher_* (a mixed batch gets per-sentence
354
+ # zeros for the rows without them, and the per-sentence kd_apply mask
355
+ # gates the loss).
356
+ has_teacher = any("teacher_upos" in e for e in encs)
357
+ if has_teacher:
358
+ teacher_upos_t = torch.zeros(B, T, dtype=torch.long)
359
+ teacher_lemma_edit_t = torch.full((B, T), -100, dtype=torch.long)
360
+ teacher_rel_t = torch.zeros(B, T, dtype=torch.long)
361
+ teacher_head_t = torch.zeros(B, T, dtype=torch.long)
362
+ teacher_feats_t = torch.zeros(B, T, self.vocab.n_feat, dtype=torch.float)
363
+
364
+ # Hidden-state auxiliary targets.
365
+ # Allocate when ANY sentence in the batch carries
366
+ # `finbert_hidden_word` (cache attachment). Rows for cache-miss
367
+ # sentences stay all-zero with mask all-False so the loss path's
368
+ # mask gating naturally drops them.
369
+ has_finbert_hidden = any("finbert_hidden_word" in e for e in encs)
370
+ if has_finbert_hidden:
371
+ first_with_fb = next(e for e in encs if "finbert_hidden_word" in e)
372
+ fb_dim = int(first_with_fb["finbert_hidden_word"].shape[-1])
373
+ teacher_finbert_hidden = torch.zeros(B, T, fb_dim, dtype=torch.float16)
374
+ finbert_hidden_kd_mask = torch.zeros(B, T, dtype=torch.bool)
375
+
376
+ for b, e in enumerate(encs):
377
+ t = e["T"]
378
+ ids[b, :t] = torch.tensor(e["ids"], dtype=torch.long)
379
+ mask[b, :t] = True
380
+ first_subw[b, :t] = torch.tensor(e["first_subw"], dtype=torch.bool)
381
+ upos[b, :t] = torch.tensor(e["upos"], dtype=torch.long)
382
+ lemma[b, :t] = torch.tensor(e["lemma"], dtype=torch.long)
383
+ rel[b, :t] = torch.tensor(e["rel"], dtype=torch.long)
384
+ head[b, :t] = torch.tensor(e["head"], dtype=torch.long)
385
+ feats[b, :t] = torch.tensor(e["feats"], dtype=torch.float)
386
+ if has_fst:
387
+ fst_tags[b, :t] = torch.tensor(e["fst_tags"], dtype=torch.long)
388
+ if has_feats_token_mask and "feats_token_mask" in e:
389
+ feats_token_mask_t[b, :t] = torch.tensor(
390
+ e["feats_token_mask"], dtype=torch.bool)
391
+ if has_compound:
392
+ form_chars[b, :t] = torch.tensor(e["form_chars"], dtype=torch.long)
393
+ form_char_mask[b, :t] = torch.tensor(e["form_char_mask"], dtype=torch.bool)
394
+ compound_K[b, :t] = torch.tensor(e["compound_K"], dtype=torch.long)
395
+ compound_split_pos[b, :t] = torch.tensor(e["compound_split_pos"], dtype=torch.bool)
396
+ compound_tail_id[b, :t] = torch.tensor(e["compound_tail_id"], dtype=torch.long)
397
+ compound_noisy[b, :t] = torch.tensor(e["compound_noisy"], dtype=torch.bool)
398
+ if has_teacher and "teacher_upos" in e:
399
+ teacher_upos_t[b, :t] = torch.tensor(e["teacher_upos"], dtype=torch.long)
400
+ teacher_lemma_edit_t[b, :t] = torch.tensor(e["teacher_lemma_edit"], dtype=torch.long)
401
+ teacher_rel_t[b, :t] = torch.tensor(e["teacher_rel"], dtype=torch.long)
402
+ teacher_head_t[b, :t] = torch.tensor(e["teacher_head"], dtype=torch.long)
403
+ teacher_feats_t[b, :t] = torch.tensor(e["teacher_feats"], dtype=torch.float)
404
+ if has_finbert_hidden and "finbert_hidden_word" in e:
405
+ # `finbert_hidden_word` is fp16 (n_words, H); scatter
406
+ # rows onto the first-subword position of each UD word.
407
+ # ROOT (pos 0) and non-first-subword / pad positions
408
+ # stay zero with mask False — that is how the loss path
409
+ # excludes them.
410
+ rows = torch.from_numpy(e["finbert_hidden_word"]) # fp16
411
+ positions = e["finbert_hidden_first_pos"]
412
+ for k, pos in enumerate(positions):
413
+ teacher_finbert_hidden[b, pos] = rows[k]
414
+ finbert_hidden_kd_mask[b, pos] = True
415
+
416
+ out = {
417
+ "ids": ids,
418
+ "mask": mask,
419
+ "first_subw": first_subw,
420
+ "upos": upos,
421
+ "feats": feats,
422
+ "lemma": lemma,
423
+ "head": head,
424
+ "rel": rel,
425
+ }
426
+ if has_fst:
427
+ out["fst_tags"] = fst_tags
428
+ if has_feats_token_mask:
429
+ out["feats_token_mask"] = feats_token_mask_t
430
+ if has_compound:
431
+ out["form_chars"] = form_chars
432
+ out["form_char_mask"] = form_char_mask
433
+ out["compound_K"] = compound_K
434
+ out["compound_split_pos"] = compound_split_pos
435
+ out["compound_tail_id"] = compound_tail_id
436
+ out["compound_noisy"] = compound_noisy
437
+ if has_teacher:
438
+ out["teacher_upos"] = teacher_upos_t
439
+ out["teacher_lemma_edit"] = teacher_lemma_edit_t
440
+ out["teacher_rel"] = teacher_rel_t
441
+ out["teacher_head"] = teacher_head_t
442
+ out["teacher_feats"] = teacher_feats_t
443
+ if has_finbert_hidden:
444
+ out["teacher_finbert_hidden"] = teacher_finbert_hidden
445
+ out["finbert_hidden_kd_mask"] = finbert_hidden_kd_mask
446
+ return out
447
+
448
+ def _empty_batch(self) -> dict[str, torch.Tensor]:
449
+ z = torch.zeros(0, 0, dtype=torch.long)
450
+ out = {
451
+ "ids": z,
452
+ "mask": torch.zeros(0, 0, dtype=torch.bool),
453
+ "first_subw": torch.zeros(0, 0, dtype=torch.bool),
454
+ "upos": z, "lemma": z, "rel": z, "head": z,
455
+ "feats": torch.zeros(0, 0, self.vocab.n_feat),
456
+ "kept_indices": torch.zeros(0, dtype=torch.long),
457
+ }
458
+ if self.compound_enabled:
459
+ mfl = self.max_form_len
460
+ out["form_chars"] = torch.zeros(0, 0, mfl, dtype=torch.long)
461
+ out["form_char_mask"] = torch.zeros(0, 0, mfl, dtype=torch.bool)
462
+ out["compound_K"] = torch.zeros(0, 0, dtype=torch.long)
463
+ out["compound_split_pos"] = torch.zeros(0, 0, mfl, dtype=torch.bool)
464
+ out["compound_tail_id"] = torch.zeros(0, 0, dtype=torch.long)
465
+ out["compound_noisy"] = torch.zeros(0, 0, dtype=torch.bool)
466
+ return out
juurio/parser/compound.py ADDED
@@ -0,0 +1,154 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Compound-lemma supervision targets.
2
+
3
+ The lemma stack uses three heads to handle Finnish compound lemmatization
4
+ (`kiinankaali → kiinan#kaali`, `juhlamekkopaniikki → juhla#mekko#paniikki`).
5
+ Compounds account for a large share of lemma OOVs, so routing them through
6
+ dedicated heads recovers substantial lemma accuracy.
7
+
8
+ This module owns the *gold-target construction* for those new heads:
9
+
10
+ - boundary-count target K ∈ {0, 1, 2, 3+} → CE head
11
+ - per-position split target → per-char-pos BCE head
12
+ - tail-script target → CE head, reuses the existing
13
+ edit-script vocab from
14
+ `LabelVocab.edit_to_id`
15
+
16
+ The collator calls `CompoundTargets.from_form_lemma()` once per gold/silver
17
+ token to assemble the gold dict; the loss module then masks `L_split` and
18
+ `L_tail` on tokens flagged `noisy_label` (stem-changing compound prefixes) —
19
+ supervising the split scorer with noisy positions would poison it.
20
+
21
+ Boundary semantics:
22
+
23
+ For a lemma like `TGV#juna` and form `TGV-junia`, the gold `#` is the
24
+ boundary BEFORE the form's hyphen — position 3. Tail-form is then
25
+ `-junia`. The hyphen-fallback in the decoder strips the leading `-` from
26
+ the tail when looking up its edit-script.
27
+
28
+ Stem-changing prefixes (`lastenromaanit → lapsi#romaani`) are flagged
29
+ `noisy_label=True` — the form's prefix `lasten` is not a substring of
30
+ `lapsi`, so we can't project the gold `#` to a clean form position. Such
31
+ tokens still supervise the count head (gold K=1 is correct) but their
32
+ split target is empty.
33
+ """
34
+ from __future__ import annotations
35
+
36
+ from dataclasses import dataclass
37
+ from typing import Mapping
38
+
39
+ from .edit_scripts import extract_edit_script
40
+ from .label_vocab import _script_key
41
+
42
+ # Count-head cap. K ∈ {0, 1, 2, 3+}; everything ≥3 collapses to class 3.
43
+ COUNT_MAX = 3
44
+
45
+
46
+ @dataclass
47
+ class CompoundTargets:
48
+ """Gold targets for the three compound heads. `from_form_lemma` is the
49
+ only public constructor — keep target conventions in one place."""
50
+
51
+ @staticmethod
52
+ def from_form_lemma(form: str, lemma: str,
53
+ edit_to_id: Mapping[str, int]) -> dict:
54
+ """Build the gold-target dict for a single (form, lemma) pair.
55
+
56
+ Returns a dict with keys:
57
+ - `K` : int, capped at COUNT_MAX (3)
58
+ - `boundary_positions` : list[int], gold char positions in `form`,
59
+ one per gold `#`. EMPTY if K=0 OR if `noisy_label` is True.
60
+ - `tail_form` : str, the form slice from the last boundary on
61
+ (empty if K=0)
62
+ - `tail_lemma` : str, the last lemma piece (== lemma if K=0)
63
+ - `tail_script_id` : int id of the (tail_form → tail_lemma) script
64
+ in `edit_to_id`. OOV id if the script isn't in
65
+ the table. Set to OOV-id for K=0 (downstream
66
+ loss masks it).
67
+ - `noisy_label` : True iff any internal lemma piece is not a
68
+ verbatim case-insensitive substring of `form`
69
+ at the implied position.
70
+
71
+ Boundary projection algorithm: walk `pieces[:-1]` left-to-right and
72
+ require each piece to match `form[cursor:cursor+len(piece)]`
73
+ case-insensitively. If any piece fails, the entire example is noisy.
74
+ The last piece (tail) is computed from `form[positions[-1]:]` →
75
+ `pieces[-1]`; the last piece does NOT need to match verbatim (it's
76
+ inflected — that's what the tail-script head learns).
77
+ """
78
+ oov_id = edit_to_id.get("<oov>", 1)
79
+ pieces = lemma.split("#")
80
+ n_pieces = len(pieces)
81
+
82
+ # K=0 (no compound): trivial path.
83
+ if n_pieces == 1:
84
+ return {
85
+ "K": 0,
86
+ "boundary_positions": [],
87
+ "tail_form": "",
88
+ "tail_lemma": lemma,
89
+ "tail_script_id": oov_id, # unused; loss is masked
90
+ "noisy_label": False,
91
+ }
92
+
93
+ # Walk internal pieces; project each onto a char position in `form`.
94
+ # Between consecutive internal pieces, allow ONE separator char
95
+ # (`-` or `–`) — Finnish multi-component compounds use either no
96
+ # separator (`juhlamekkopaniikki → juhla#mekko#paniikki`) or a
97
+ # hyphen between every internal pair (`KIIS-FM-radiokanavalla →
98
+ # KIIS#FM#radio#kanava`). Without this the matcher would fail at the
99
+ # first internal hyphen and mark the example noisy, dropping
100
+ # multi-hyphen compounds from supervision unnecessarily.
101
+ cursor = 0
102
+ positions: list[int] = []
103
+ noisy = False
104
+ for i, piece in enumerate(pieces[:-1]):
105
+ n = len(piece)
106
+ if form[cursor:cursor + n].lower() != piece.lower():
107
+ noisy = True
108
+ break
109
+ cursor += n
110
+ positions.append(cursor)
111
+ # If the next char is `-` or `–` AND the next piece does not
112
+ # begin with the same separator, consume it (the hyphen IS the
113
+ # compound boundary marker in the form). Only do this between
114
+ # internal pieces — don't strip a leading hyphen from the tail
115
+ # here; the decoder's apply-time strip handles that case.
116
+ if cursor < len(form) and form[cursor] in ("-", "–"):
117
+ cursor += 1
118
+
119
+ # K target: real number of `#` in lemma, capped at COUNT_MAX.
120
+ K_real = n_pieces - 1
121
+ K = min(K_real, COUNT_MAX)
122
+
123
+ if noisy:
124
+ # Don't fabricate split positions; mask the split + tail losses
125
+ # downstream. Count head still supervised on the correct K.
126
+ return {
127
+ "K": K,
128
+ "boundary_positions": [],
129
+ "tail_form": "",
130
+ "tail_lemma": pieces[-1],
131
+ "tail_script_id": oov_id,
132
+ "noisy_label": True,
133
+ }
134
+
135
+ # Clean case: tail starts at positions[-1].
136
+ tail_start = positions[-1]
137
+ # Hyphen-fallback convention: both supervision and
138
+ # decode strip a leading `-`/`–` from the form-tail before computing
139
+ # / applying the edit script. `tail_form` therefore reflects the
140
+ # form-slice the script operates on (stripped), matching what the
141
+ # decoder sees at apply-time.
142
+ tail_form = form[tail_start:].lstrip("-–")
143
+ tail_lemma = pieces[-1]
144
+ es = extract_edit_script(tail_form, tail_lemma)
145
+ tail_script_id = edit_to_id.get(_script_key(es), oov_id)
146
+
147
+ return {
148
+ "K": K,
149
+ "boundary_positions": positions,
150
+ "tail_form": tail_form,
151
+ "tail_lemma": tail_lemma,
152
+ "tail_script_id": tail_script_id,
153
+ "noisy_label": False,
154
+ }
juurio/parser/edit_scripts.py ADDED
@@ -0,0 +1,155 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Longest-common-prefix lemma-edit-scripts.
2
+
3
+ The lemma head is a classifier over a finite inventory of edit scripts. This module is the
4
+ *representation* — given a `(form, lemma)` pair, it produces a `(delete_n, append_s)` script
5
+ that, applied to `form.lower()`, yields `lemma.lower()` exactly.
6
+
7
+ A script is `(delete_n: int, append_s: str)` meaning: lowercase the form, drop the last `delete_n`
8
+ characters, append `append_s`. The longest-common-prefix construction makes the script unique and
9
+ canonical for any pair (no decision about which edit is "right" — there is only one prefix-based
10
+ edit per pair). Full suppletion (`on -> olla`) is representable as `(2, "olla")`.
11
+
12
+ CLI:
13
+ python -m juurio.parser.edit_scripts probe # script coverage on UD-TDT-train/dev
14
+ """
15
+ from __future__ import annotations
16
+
17
+ from collections import Counter
18
+ from typing import Iterable
19
+
20
+ EditScript = tuple[int, str]
21
+
22
+
23
+ def extract_edit_script(form: str, lemma: str) -> EditScript:
24
+ """Compute the canonical longest-common-prefix edit script for `(form, lemma)`.
25
+
26
+ Both sides are lowercased before the common-prefix search; the resulting script applied to
27
+ `form.lower()` returns `lemma.lower()`. No-op on identity (delete=0, append="")."""
28
+ f = form.lower()
29
+ l = lemma.lower()
30
+ p = 0
31
+ while p < len(f) and p < len(l) and f[p] == l[p]:
32
+ p += 1
33
+ return (len(f) - p, l[p:])
34
+
35
+
36
+ def apply_edit_script(form: str, script: EditScript) -> str:
37
+ """Apply `script = (delete_n, append_s)` to `form.lower()` and return the result. Pathologically
38
+ over-long `delete_n` collapses to "drop the whole form" — keeps the function total."""
39
+ delete_n, append_s = script
40
+ f = form.lower()
41
+ if delete_n >= len(f):
42
+ return append_s
43
+ return f[: len(f) - delete_n] + append_s
44
+
45
+
46
+ def script_coverage(
47
+ pairs: Iterable[tuple[str, str]], ks: tuple[int, ...] = (200, 500, 1000, 2000)
48
+ ) -> dict[int, tuple[float, int]]:
49
+ """For each `k` in `ks`, return `(coverage, n_distinct)` where coverage is the fraction of
50
+ `pairs` whose extracted script is among the top-k most common scripts in `pairs` itself.
51
+
52
+ This is the *in-sample* oracle: the upper bound a classifier with vocab=k could ever reach if
53
+ the train and inference distributions were identical. The train/eval-split variant is below."""
54
+ pairs = list(pairs)
55
+ counts = Counter(extract_edit_script(f, l) for f, l in pairs)
56
+ n = len(pairs)
57
+ out = {}
58
+ for k in ks:
59
+ topk = {s for s, _ in counts.most_common(k)}
60
+ hits = sum(1 for f, l in pairs if extract_edit_script(f, l) in topk)
61
+ out[k] = (hits / n if n else 0.0, min(k, len(counts)))
62
+ return out
63
+
64
+
65
+ def script_coverage_train_eval(
66
+ train: Iterable[tuple[str, str]],
67
+ eval_: Iterable[tuple[str, str]],
68
+ ks: tuple[int, ...] = (200, 500, 1000, 2000),
69
+ ) -> dict[int, tuple[float, int]]:
70
+ """Build the top-k script vocabulary on `train`, measure coverage on `eval_`. Tests whether a
71
+ script vocabulary built on the train split generalises to the eval split."""
72
+ train = list(train)
73
+ eval_ = list(eval_)
74
+ counts = Counter(extract_edit_script(f, l) for f, l in train)
75
+ out = {}
76
+ for k in ks:
77
+ topk = {s for s, _ in counts.most_common(k)}
78
+ hits = sum(1 for f, l in eval_ if extract_edit_script(f, l) in topk)
79
+ out[k] = (hits / len(eval_) if eval_ else 0.0, min(k, len(counts)))
80
+ return out
81
+
82
+
83
+ # ----------------------------- CLI: script coverage ----------------------------------------------
84
+ def _read_conllu_pairs(path: str) -> list[tuple[str, str]]:
85
+ """(form, lemma) pairs from a CoNLL-U file, skipping comments, MWT lines, and empty lemmas.
86
+
87
+ A "real" content token in CoNLL-U has an integer `id` (MWT spans are `n-m` and are *also*
88
+ emitted but as separate per-component lines we want; the `n-m` ones we skip). Empty / `_`
89
+ lemmas are skipped — punctuation and some symbols carry no lemma."""
90
+ out = []
91
+ with open(path, encoding="utf-8") as fh:
92
+ for ln in fh:
93
+ ln = ln.strip()
94
+ if not ln or ln.startswith("#"):
95
+ continue
96
+ cols = ln.split("\t")
97
+ if len(cols) < 4:
98
+ continue
99
+ tid = cols[0]
100
+ if "-" in tid or "." in tid: # MWT span line / empty-node line — skip
101
+ continue
102
+ form, lemma = cols[1], cols[2]
103
+ if not form or not lemma or lemma == "_":
104
+ continue
105
+ out.append((form, lemma))
106
+ return out
107
+
108
+
109
+ def probe(train_path: str, dev_path: str) -> None:
110
+ """Report edit-script coverage on UD-TDT-train and UD-TDT-dev."""
111
+ train = _read_conllu_pairs(train_path)
112
+ dev = _read_conllu_pairs(dev_path)
113
+ print(f"UD-TDT: train {len(train)} tokens, dev {len(dev)} tokens")
114
+ print()
115
+
116
+ # in-sample (train coverage by train top-k)
117
+ print("== train in-sample coverage (oracle upper bound on a classifier head) ==")
118
+ cov_t = script_coverage(train, ks=(100, 200, 500, 1000, 2000, 4000))
119
+ for k, (c, n) in cov_t.items():
120
+ print(f" top-{k:5d} coverage {c*100:6.2f}% (n_distinct = {n})")
121
+
122
+ # out-of-sample (dev coverage by train top-k) — the actual generalisation question
123
+ print()
124
+ print("== dev coverage by train-fitted top-k vocab (Q3 generalisation) ==")
125
+ cov_d = script_coverage_train_eval(train, dev, ks=(100, 200, 500, 1000, 2000, 4000))
126
+ for k, (c, n) in cov_d.items():
127
+ print(f" top-{k:5d} dev-coverage {c*100:6.2f}% (vocab built from train)")
128
+
129
+ # uncoverable tail: how many distinct scripts total, and how many tokens does the ≥2000 tail eat
130
+ counts = Counter(extract_edit_script(f, l) for f, l in train)
131
+ print()
132
+ print(f"Total distinct scripts in train: {len(counts)}")
133
+ if len(counts) > 2000:
134
+ tail = sum(c for _, c in counts.most_common()[2000:])
135
+ print(f"Train tokens captured ONLY by tail scripts (>top-2000): {tail} ({tail/len(train)*100:.2f}%)")
136
+
137
+ print()
138
+ print("== sample of the rare-tail scripts (rank 1500–1510, just to eyeball oddities) ==")
139
+ for rank, (sc, n) in enumerate(counts.most_common()[1500:1510], start=1501):
140
+ delete_n, append_s = sc
141
+ print(f" rank {rank:4d} ({delete_n:2d}, {append_s!r:20s}) count {n}")
142
+
143
+
144
+ if __name__ == "__main__":
145
+ import os
146
+ import sys
147
+
148
+ repo = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
149
+ train = os.path.join(repo, "data", "ud", "fi_tdt-ud-train.conllu")
150
+ dev = os.path.join(repo, "data", "ud", "fi_tdt-ud-dev.conllu")
151
+ cmd = sys.argv[1] if len(sys.argv) > 1 else "probe"
152
+ if cmd == "probe":
153
+ probe(train, dev)
154
+ else:
155
+ raise SystemExit(f"unknown cmd {cmd!r}; usage: python -m juurio.parser.edit_scripts probe")
juurio/parser/eval.py ADDED
@@ -0,0 +1,158 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Checkpoint loader for the tiny Finnish UD parser.
2
+
3
+ Reconstructs a `TinyFiUD` model from a saved checkpoint blob, recovering the
4
+ channel/head configuration from the persisted config (and, for older ckpts,
5
+ inferring it from the state_dict shapes). The serving neural adapter imports
6
+ `_load_checkpoint` from here.
7
+ """
8
+ from __future__ import annotations
9
+
10
+ import torch
11
+
12
+ from .label_vocab import LabelVocab
13
+ from .model import TinyFiUD
14
+ from .tokenizer import BPETokenizer
15
+
16
+
17
+ # EXACT allowlist for unexpected keys in `_load_checkpoint`. Some training
18
+ # checkpoints contain `hidden_kd_proj.{weight, bias}` — an extra student
19
+ # projection used only during training; at eval time it is unused. We use
20
+ # an exact allowlist (rather than a prefix filter) so the strict=False load
21
+ # contract stays tight and never silently accepts arbitrary extra keys.
22
+ _ALLOWED_EXTRA_STATE_KEYS: frozenset[str] = frozenset({
23
+ "hidden_kd_proj.weight",
24
+ "hidden_kd_proj.bias",
25
+ })
26
+
27
+
28
+ def _load_checkpoint(path: str, vocab: LabelVocab, tokenizer: BPETokenizer, device: str) -> TinyFiUD:
29
+ blob = torch.load(path, map_location=device, weights_only=False)
30
+ cfg = blob["config"]
31
+ # Detect parser_bilstm_hidden from saved cfg or infer from state_dict keys
32
+ # (so this loader stays compatible with ckpts saved before the field existed).
33
+ sd = blob["state_dict"]
34
+ bilstm_hidden = cfg.get("parser_bilstm_hidden", 0)
35
+ if bilstm_hidden == 0 and "parser_bilstm.weight_ih_l0" in sd:
36
+ # weight_ih_l0 has shape (4*hidden, input) — recover hidden from row count
37
+ bilstm_hidden = sd["parser_bilstm.weight_ih_l0"].shape[0] // 4
38
+ # Detect arc_d from W_arc shape (older ckpts default to 200) and
39
+ # arc_distance_bias from the presence of the bias parameter.
40
+ arc_d = cfg.get("arc_d", sd["W_arc"].shape[0] if "W_arc" in sd else 200)
41
+ arc_distance_bias = cfg.get("arc_distance_bias", "arc_dist_bias" in sd)
42
+ # Detect FST channel (fst_tag_vocab_size + fst_tag_dim) from state_dict
43
+ # (so older ckpts load unchanged, and FST ckpts reconstruct the embedding bag + proj).
44
+ fst_tag_vocab_size = cfg.get("fst_tag_vocab_size", 0)
45
+ if fst_tag_vocab_size == 0 and "fst_tag_emb.weight" in sd:
46
+ fst_tag_vocab_size = sd["fst_tag_emb.weight"].shape[0]
47
+ fst_tag_dim = cfg.get("fst_tag_dim", 32)
48
+ if "fst_tag_emb.weight" in sd:
49
+ fst_tag_dim = sd["fst_tag_emb.weight"].shape[1]
50
+ # Reconstruct compound channel from cfg + state_dict.
51
+ # Older ckpts don't have these fields; the channel stays off.
52
+ compound_enabled = cfg.get("compound_enabled", False)
53
+ if not compound_enabled and "char_emb.weight" in sd:
54
+ compound_enabled = True
55
+ char_vocab_size = cfg.get("char_vocab_size", 0)
56
+ if char_vocab_size == 0 and "char_emb.weight" in sd:
57
+ char_vocab_size = sd["char_emb.weight"].shape[0]
58
+ char_emb_dim = cfg.get("char_emb_dim", 16)
59
+ if "char_emb.weight" in sd:
60
+ char_emb_dim = sd["char_emb.weight"].shape[1]
61
+ char_conv_channels = cfg.get("char_conv_channels", 64)
62
+ if "compound_split_conv.weight" in sd:
63
+ char_conv_channels = sd["compound_split_conv.weight"].shape[0]
64
+ max_form_len = cfg.get("max_form_len", 64)
65
+ # Detect parser-private FF block from state_dict.
66
+ # Hidden dim recovered from linear1.weight row count.
67
+ parser_private_hidden = cfg.get("parser_private_hidden", 0)
68
+ if parser_private_hidden == 0 and "parser_private_linear1.weight" in sd:
69
+ parser_private_hidden = sd["parser_private_linear1.weight"].shape[0]
70
+ # Reconstruct per-key Feats head config from the saved blob. Without
71
+ # this, eval would build a model without the per-key head and its
72
+ # state_dict keys would be unexpected — load_state_dict(strict=True)
73
+ # would fail. The trained ckpt persisted use_per_key_feats,
74
+ # value_counts, upos_emb_dim, and flat_ids; rebuild the head with the
75
+ # SAME shape it was trained with.
76
+ use_per_key_feats = cfg.get("use_per_key_feats", False)
77
+ feats_per_key_value_counts = cfg.get("feats_per_key_value_counts")
78
+ feats_per_key_upos_emb_dim = cfg.get("feats_per_key_upos_emb_dim", 32)
79
+ feats_per_key_flat_ids = cfg.get("feats_per_key_flat_ids")
80
+ # If the state_dict has per-key head weights but the config doesn't
81
+ # carry the metadata, fail loud — a ckpt without persisted config
82
+ # is a serialization bug we need to catch, not paper over.
83
+ has_per_key_sd = any(k.startswith("feats_per_key_head.") for k in sd.keys())
84
+ if has_per_key_sd and not use_per_key_feats:
85
+ raise ValueError(
86
+ "Checkpoint state_dict carries `feats_per_key_head.*` weights "
87
+ "but config.use_per_key_feats is False. The ckpt was saved without "
88
+ "the per-key metadata block. Re-save the ckpt via the training path "
89
+ "that persists the per-key config, or downgrade the eval to BCE-only."
90
+ )
91
+ # Near-identity capacity adapter.
92
+ # When a ckpt carries the `near_identity_adapter.*` namespace in its
93
+ # state_dict, the config must persist the flag; otherwise we'd
94
+ # silently rebuild a model without the adapter and load_state_dict would
95
+ # fail (missing keys). Same guard pattern as the per-key Feats head.
96
+ use_near_identity_adapter = cfg.get("use_near_identity_adapter", False)
97
+ near_identity_n_layers = cfg.get("near_identity_n_layers", 0)
98
+ near_identity_d_model = cfg.get("near_identity_d_model")
99
+ near_identity_heads = cfg.get("near_identity_heads")
100
+ near_identity_ff = cfg.get("near_identity_ff")
101
+ has_adapter_sd = any(k.startswith("near_identity_adapter.") for k in sd.keys())
102
+ if has_adapter_sd and not use_near_identity_adapter:
103
+ raise ValueError(
104
+ "Checkpoint state_dict carries `near_identity_adapter.*` weights "
105
+ "but config.use_near_identity_adapter is False. The ckpt was "
106
+ "saved without the adapter metadata block. Re-save via "
107
+ "`warm_start_near_identity` / current train.py path that persists "
108
+ "the adapter config."
109
+ )
110
+ model = TinyFiUD(
111
+ vocab=tokenizer.vocab_size, d_model=cfg["d_model"], layers=cfg["layers"],
112
+ heads=cfg["heads"], ff=cfg["ff"], n_upos=vocab.n_upos, n_feat=vocab.n_feat,
113
+ n_edit=vocab.n_edit, n_rel=vocab.n_rel, maxlen=cfg["max_len"],
114
+ parser_bilstm_hidden=bilstm_hidden,
115
+ arc_d=arc_d,
116
+ arc_distance_bias=arc_distance_bias,
117
+ fst_tag_vocab_size=fst_tag_vocab_size,
118
+ fst_tag_dim=fst_tag_dim,
119
+ compound_enabled=compound_enabled,
120
+ char_vocab_size=char_vocab_size,
121
+ char_emb_dim=char_emb_dim,
122
+ char_conv_channels=char_conv_channels,
123
+ max_form_len=max_form_len,
124
+ parser_private_hidden=parser_private_hidden,
125
+ use_per_key_feats=use_per_key_feats,
126
+ feats_per_key_value_counts=feats_per_key_value_counts,
127
+ feats_per_key_upos_emb_dim=feats_per_key_upos_emb_dim,
128
+ feats_per_key_flat_ids=feats_per_key_flat_ids,
129
+ use_near_identity_adapter=use_near_identity_adapter,
130
+ near_identity_n_layers=near_identity_n_layers,
131
+ near_identity_d_model=near_identity_d_model,
132
+ near_identity_heads=near_identity_heads,
133
+ near_identity_ff=near_identity_ff,
134
+ ).to(device)
135
+ # Some training checkpoints contain extra `hidden_kd_proj.*` projection
136
+ # keys that are unused at eval time. Tolerate them with strict=False
137
+ # rather than re-instantiate TinyFiUD with use_hidden_kd_proj=True
138
+ # (which would also need the cfg flag in the ckpt blob, which older
139
+ # ckpts lack). The allowlist is an EXACT set rather than a prefix
140
+ # filter so the strict=False contract stays tight and never silently
141
+ # accepts arbitrary extra child keys.
142
+ missing, unexpected = model.load_state_dict(sd, strict=False)
143
+ bad_unexpected = [k for k in unexpected if k not in _ALLOWED_EXTRA_STATE_KEYS]
144
+ if bad_unexpected or missing:
145
+ raise RuntimeError(
146
+ f"eval ckpt load mismatch — missing={missing} unexpected={bad_unexpected}"
147
+ )
148
+ # Pin the FST vocab hash on the loaded model so a consumer can refuse a
149
+ # stale vocab. Older ckpts won't have the field — that's fine.
150
+ expected_hash = cfg.get("fst_tag_vocab_hash")
151
+ if expected_hash is not None:
152
+ model._fst_tag_vocab_hash = expected_hash
153
+ # Same hash pin for the char vocab (analogous to the FST tag vocab).
154
+ # Older ckpts won't have it.
155
+ expected_char_hash = cfg.get("char_vocab_hash")
156
+ if expected_char_hash is not None:
157
+ model._char_vocab_hash = expected_char_hash
158
+ return model
juurio/parser/feats_per_key_head.py ADDED
@@ -0,0 +1,96 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Per-key biaffine Feats head.
2
+
3
+ Uses one biaffine scorer per UD feature key, conditioned on a UPOS embedding,
4
+ in place of a single `Linear(d_model → n_feat) + BCE` head.
5
+ Each scorer predicts an EXCLUSIVE value among that key's vocabulary
6
+ (plus an `<EMPTY>` class at column 0 that the model can predict
7
+ when the key is absent on this token).
8
+
9
+ Architecture (per key K with `n_values_K` values, including EMPTY):
10
+
11
+ logits_K[b, t, v] =
12
+ bilinear_K(h[b, t], upos_emb[b, t])[v] # (B, T) → (B, T, n_v)
13
+ + linear_h_K(h[b, t])[v]
14
+ + linear_u_K(upos_emb[b, t])[v]
15
+
16
+ `nn.Bilinear` provides the (h^T W_K[v] upos_emb + bias_K[v]) term;
17
+ the two `nn.Linear`s without bias add the Dozat-Manning x1- and
18
+ x2-affine terms. Total per-key parameter count is small
19
+ (O(d_model * upos_emb_dim * n_values_K)).
20
+
21
+ Notes:
22
+ - All keys are uniform, including singleton-value keys (n_values=2).
23
+ - The head owns its own UPOS embedding table and is fed by the UPOS
24
+ classifier's predicted-UPOS argmax at both train and eval, avoiding
25
+ a train/eval mismatch.
26
+ """
27
+ from __future__ import annotations
28
+
29
+ import torch
30
+ import torch.nn as nn
31
+
32
+
33
+ class PerKeyFeatsHead(nn.Module):
34
+ """Per-key biaffine Feats head.
35
+
36
+ Args:
37
+ d_model: hidden dim of the encoder's per-token representation.
38
+ upos_emb_dim: dim of the head-owned UPOS embedding (small).
39
+ value_counts: per-key value count INCLUDING the `<EMPTY>`
40
+ class at index 0. Length = number of keys.
41
+ n_upos: size of the UPOS vocabulary (for the head's embedding).
42
+ dropout: dropout applied to both `h` and UPOS embeddings before
43
+ scoring.
44
+
45
+ Forward signature:
46
+ h: `(B, T, d_model)` per-token hidden states.
47
+ upos_ids: `(B, T)` longs — UPOS argmax indices (the UPOS
48
+ head's predictions at both train and eval).
49
+
50
+ Returns:
51
+ `list[Tensor]` of length `n_keys`; element k has shape
52
+ `(B, T, value_counts[k])`. Column 0 is the `<EMPTY>` class.
53
+ """
54
+
55
+ def __init__(
56
+ self,
57
+ d_model: int,
58
+ upos_emb_dim: int,
59
+ value_counts: list[int],
60
+ n_upos: int,
61
+ dropout: float = 0.0,
62
+ ):
63
+ super().__init__()
64
+ self.d_model = d_model
65
+ self.upos_emb_dim = upos_emb_dim
66
+ self.value_counts = list(value_counts)
67
+ self.upos_emb = nn.Embedding(n_upos, upos_emb_dim)
68
+ self.dropout = nn.Dropout(dropout)
69
+ # One biaffine + two linear terms per key.
70
+ self.bilinear = nn.ModuleList(
71
+ nn.Bilinear(d_model, upos_emb_dim, n_v) for n_v in value_counts
72
+ )
73
+ self.linear_h = nn.ModuleList(
74
+ nn.Linear(d_model, n_v, bias=False) for n_v in value_counts
75
+ )
76
+ self.linear_u = nn.ModuleList(
77
+ nn.Linear(upos_emb_dim, n_v, bias=False) for n_v in value_counts
78
+ )
79
+
80
+ @property
81
+ def n_keys(self) -> int:
82
+ return len(self.value_counts)
83
+
84
+ def forward(
85
+ self,
86
+ h: torch.Tensor,
87
+ upos_ids: torch.Tensor,
88
+ ) -> list[torch.Tensor]:
89
+ upos_e = self.upos_emb(upos_ids) # (B, T, upos_emb_dim)
90
+ h_d = self.dropout(h)
91
+ u_d = self.dropout(upos_e)
92
+ out: list[torch.Tensor] = []
93
+ for bi, lh, lu in zip(self.bilinear, self.linear_h, self.linear_u):
94
+ logits = bi(h_d, u_d) + lh(h_d) + lu(u_d)
95
+ out.append(logits)
96
+ return out
juurio/parser/fst_cache.py ADDED
@@ -0,0 +1,147 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Tag vocabulary + per-form analysis cache for the FST channel.
2
+
3
+ The pipeline:
4
+ 1. Pre-compute an FST analysis for every unique surface form in the
5
+ training data → FST cache JSONL.
6
+ 2. At collator time, look up form → set of morphtags across all readings;
7
+ encode tags as int ids via `FstTagVocab`; pad to `max_tags` and emit a
8
+ (T, max_tags) int tensor parallel to subword positions.
9
+ 3. At model time, embed each tag id, sum across the `max_tags` axis (the
10
+ embedding for PAD=0 is constrained to zero so padded slots contribute
11
+ nothing), normalize, project to d_model, and add to the encoder input
12
+ at first-subword positions.
13
+
14
+ The vocab uses PAD=0, UNK=1 as fixed reserved ids; dataset tags start at 2.
15
+ The cache excludes the reading's lemma slot (`reading[0]`) when computing
16
+ the tag set, since lemma identity is handled by the dedicated edit-script
17
+ head and would otherwise blow up the tag vocab with lexical entries.
18
+ """
19
+ from __future__ import annotations
20
+
21
+ import hashlib
22
+ import json
23
+ from collections import Counter
24
+ from pathlib import Path
25
+ from typing import Iterable
26
+
27
+
28
+ PAD_ID = 0
29
+ UNK_ID = 1
30
+ PAD_TAG = "<PAD>"
31
+ UNK_TAG = "<UNK>"
32
+
33
+
34
+ class FstTagVocab:
35
+ """Maps morphtag strings to int ids."""
36
+
37
+ def __init__(self, tags: list[str]):
38
+ # tags is the ordered list with PAD at index 0, UNK at index 1, real tags from index 2.
39
+ assert tags[0] == PAD_TAG and tags[1] == UNK_TAG
40
+ self._tags = list(tags)
41
+ self._tag_to_id = {t: i for i, t in enumerate(tags)}
42
+
43
+ @property
44
+ def size(self) -> int:
45
+ return len(self._tags)
46
+
47
+ @classmethod
48
+ def fit(cls, readings: Iterable[list[str]], *, min_count: int = 1,
49
+ max_tags: int | None = None) -> "FstTagVocab":
50
+ """Fit a vocab from an iterable of readings. Each reading is
51
+ [lemma, tag1, tag2, ...] — the lemma is excluded automatically."""
52
+ c: Counter[str] = Counter()
53
+ for r in readings:
54
+ for tag in r[1:]: # skip lemma at position 0
55
+ c[tag] += 1
56
+ kept = [t for t, n in c.most_common() if n >= min_count]
57
+ if max_tags is not None:
58
+ kept = kept[:max_tags]
59
+ return cls([PAD_TAG, UNK_TAG] + kept)
60
+
61
+ def encode(self, tag: str) -> int:
62
+ return self._tag_to_id.get(tag, UNK_ID)
63
+
64
+ def decode(self, idx: int) -> str:
65
+ return self._tags[idx] if 0 <= idx < len(self._tags) else UNK_TAG
66
+
67
+ def hash(self) -> str:
68
+ """Stable hex digest of the ordered tag list — short enough to embed in
69
+ a ckpt config, strong enough to catch any reorder / refit / different-
70
+ min-count vocab. Used to pin the vocab to the trained ckpt so a stale
71
+ vocab can't silently feed wrong-IDed tags into a loaded model."""
72
+ h = hashlib.sha256()
73
+ for t in self._tags:
74
+ h.update(t.encode("utf-8"))
75
+ h.update(b"\x00")
76
+ return h.hexdigest()[:16]
77
+
78
+ def save(self, path: str) -> None:
79
+ Path(path).parent.mkdir(parents=True, exist_ok=True)
80
+ with open(path, "w") as fh:
81
+ json.dump({"tags": self._tags}, fh, ensure_ascii=False)
82
+
83
+ @classmethod
84
+ def load(cls, path: str) -> "FstTagVocab":
85
+ with open(path) as fh:
86
+ blob = json.load(fh)
87
+ return cls(blob["tags"])
88
+
89
+
90
+ class FstCache:
91
+ """form -> list of readings (each reading is [lemma, tag1, tag2, ...])."""
92
+
93
+ def __init__(self, data: dict[str, list[list[str]]] | None = None):
94
+ self._data: dict[str, list[list[str]]] = dict(data) if data else {}
95
+
96
+ def lookup(self, form: str) -> list[list[str]]:
97
+ return self._data.get(form, [])
98
+
99
+ def __len__(self) -> int:
100
+ return len(self._data)
101
+
102
+ def __contains__(self, form: str) -> bool:
103
+ return form in self._data
104
+
105
+ def form_to_tagset(self, form: str) -> set[str]:
106
+ """Union of all morphtags across all readings for `form`. The reading's
107
+ lemma slot (position 0) is excluded."""
108
+ out: set[str] = set()
109
+ for r in self._data.get(form, []):
110
+ for tag in r[1:]:
111
+ out.add(tag)
112
+ return out
113
+
114
+ def form_to_tag_ids(self, form: str, vocab: FstTagVocab, *, max_tags: int) -> list[int]:
115
+ """Encode the form's tagset as a fixed-length int list, padded with PAD_ID.
116
+ If more tags than `max_tags`, truncates (sorted by tag id for determinism)."""
117
+ tags = sorted(self.form_to_tagset(form))
118
+ ids = [vocab.encode(t) for t in tags if vocab.encode(t) != UNK_ID]
119
+ # de-duplicate while keeping order
120
+ seen: set[int] = set()
121
+ deduped = []
122
+ for i in ids:
123
+ if i not in seen:
124
+ seen.add(i)
125
+ deduped.append(i)
126
+ deduped = deduped[:max_tags]
127
+ # pad to max_tags
128
+ return deduped + [PAD_ID] * (max_tags - len(deduped))
129
+
130
+ def save(self, path: str) -> None:
131
+ Path(path).parent.mkdir(parents=True, exist_ok=True)
132
+ with open(path, "w") as fh:
133
+ for form, readings in self._data.items():
134
+ fh.write(json.dumps({"form": form, "readings": readings},
135
+ ensure_ascii=False) + "\n")
136
+
137
+ @classmethod
138
+ def load(cls, path: str) -> "FstCache":
139
+ data: dict[str, list[list[str]]] = {}
140
+ with open(path) as fh:
141
+ for line in fh:
142
+ line = line.strip()
143
+ if not line:
144
+ continue
145
+ rec = json.loads(line)
146
+ data[rec["form"]] = rec["readings"]
147
+ return cls(data)
juurio/parser/label_vocab.py ADDED
@@ -0,0 +1,192 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Label vocabularies for the compact Finnish UD parser.
2
+
3
+ Each task head needs to map between label strings and integer ids (model output). This
4
+ module bundles those mappings into a single `LabelVocab` object that:
5
+
6
+ - Is *fit once* on the training corpus (`LabelVocab.fit(records, top_edit=1000, top_feat=80)`).
7
+ - Reserves id 0 = `<pad>` and id 1 = `<oov>` for UPOS / deprel / edit-script so the training
8
+ loop can use `ignore_index=0` and OOV labels can be routed to a sink class.
9
+ - Frequency-caps feats and edit-scripts to keep head sizes small.
10
+ - Saves / loads as a single JSON blob alongside the tokenizer.
11
+
12
+ Edit-scripts are computed via `juurio.parser.edit_scripts.extract_edit_script`. Feats are
13
+ treated as multi-label: each `Key=Value` pair has its own id, the per-token target is multi-hot.
14
+ deprels are single-label (BCE-flavoured biaffine).
15
+ """
16
+ from __future__ import annotations
17
+
18
+ import json
19
+ import os
20
+ from collections import Counter
21
+ from typing import Iterable
22
+
23
+ from .edit_scripts import extract_edit_script
24
+ from .silver import SilverSentence
25
+
26
+
27
+ PAD = "<pad>"
28
+ OOV = "<oov>"
29
+ _RESERVED = (PAD, OOV)
30
+
31
+
32
+ class LabelVocab:
33
+ """Bundle of per-head label vocabularies. Frozen after `fit` or `load`."""
34
+
35
+ def __init__(
36
+ self,
37
+ upos_to_id: dict[str, int],
38
+ deprel_to_id: dict[str, int],
39
+ feat_to_id: dict[str, int],
40
+ edit_to_id: dict[str, int],
41
+ edit_scripts: list[tuple[int, str]],
42
+ ):
43
+ self.upos_to_id = dict(upos_to_id)
44
+ self.deprel_to_id = dict(deprel_to_id)
45
+ self.feat_to_id = dict(feat_to_id)
46
+ self.edit_to_id = dict(edit_to_id)
47
+ # parallel list so encoder can map back from id → (delete_n, append_s)
48
+ self.edit_scripts = list(edit_scripts)
49
+ # reverse lookup tables (for decoding model predictions back to strings)
50
+ self.id_to_upos = {i: t for t, i in self.upos_to_id.items()}
51
+ self.id_to_deprel = {i: t for t, i in self.deprel_to_id.items()}
52
+ self.id_to_feat = {i: t for t, i in self.feat_to_id.items()}
53
+ self.id_to_edit = {i: t for t, i in self.edit_to_id.items()}
54
+
55
+ # ----------------------------- sizes -------------------------------------------------------
56
+ @property
57
+ def n_upos(self) -> int:
58
+ return len(self.upos_to_id)
59
+
60
+ @property
61
+ def n_feat(self) -> int:
62
+ return len(self.feat_to_id)
63
+
64
+ @property
65
+ def n_edit(self) -> int:
66
+ return len(self.edit_to_id)
67
+
68
+ @property
69
+ def n_rel(self) -> int:
70
+ return len(self.deprel_to_id)
71
+
72
+ # ----------------------------- fit ---------------------------------------------------------
73
+ @classmethod
74
+ def fit(
75
+ cls,
76
+ records: Iterable[SilverSentence],
77
+ *,
78
+ top_edit: int = 1000,
79
+ top_feat: int = 80,
80
+ min_feat_freq: int = 1,
81
+ ) -> "LabelVocab":
82
+ """Build a fresh `LabelVocab` from the silver records. Vocabularies are deterministic
83
+ functions of frequency (ties broken alphabetically)."""
84
+ upos_c: Counter[str] = Counter()
85
+ deprel_c: Counter[str] = Counter()
86
+ feat_c: Counter[str] = Counter()
87
+ edit_c: Counter[tuple[int, str]] = Counter()
88
+
89
+ for s in records:
90
+ for t in s.tokens:
91
+ upos_c[t.upos] += 1
92
+ deprel_c[t.deprel] += 1
93
+ if t.feats:
94
+ items = (f"{k}={v}" for k, v in t.feats.items()) \
95
+ if isinstance(t.feats, dict) else t.feats.split("|")
96
+ for kv in items:
97
+ feat_c[kv] += 1
98
+ script = extract_edit_script(t.form, t.lemma)
99
+ edit_c[script] += 1
100
+
101
+ def _to_dict(counter: Counter, *, cap: int | None = None,
102
+ min_freq: int = 1) -> dict[str, int]:
103
+ out = {tok: i for i, tok in enumerate(_RESERVED)}
104
+ items = sorted(
105
+ ((k, v) for k, v in counter.items() if v >= min_freq),
106
+ key=lambda kv: (-kv[1], str(kv[0])),
107
+ )
108
+ if cap is not None:
109
+ items = items[:cap]
110
+ for k, _ in items:
111
+ if str(k) in out:
112
+ continue
113
+ out[str(k)] = len(out)
114
+ return out
115
+
116
+ upos_to_id = _to_dict(upos_c)
117
+ deprel_to_id = _to_dict(deprel_c)
118
+ feat_to_id = _to_dict(feat_c, cap=top_feat, min_freq=min_feat_freq)
119
+ # edit-scripts: cap and key the dict by the *string* form "<delete_n>:<append_s>"
120
+ edit_items_sorted = sorted(
121
+ edit_c.items(), key=lambda kv: (-kv[1], str(kv[0])),
122
+ )[:top_edit]
123
+ edit_to_id: dict[str, int] = {PAD: 0, OOV: 1}
124
+ edit_scripts: list[tuple[int, str]] = [(-1, ""), (-1, "")] # placeholders for reserved
125
+ for script, _ in edit_items_sorted:
126
+ key = _script_key(script)
127
+ if key in edit_to_id:
128
+ continue
129
+ edit_to_id[key] = len(edit_to_id)
130
+ edit_scripts.append(script)
131
+ return cls(upos_to_id, deprel_to_id, feat_to_id, edit_to_id, edit_scripts)
132
+
133
+ # ----------------------------- encode ------------------------------------------------------
134
+ def encode_upos(self, upos: str) -> int:
135
+ return self.upos_to_id.get(upos, self.upos_to_id[OOV])
136
+
137
+ def encode_deprel(self, deprel: str) -> int:
138
+ return self.deprel_to_id.get(deprel, self.deprel_to_id[OOV])
139
+
140
+ def encode_feats_multihot(self, feats) -> list[int]:
141
+ """Accepts either CoNLL-U style 'Case=Nom|Number=Sing' strings or
142
+ dict[str,str]. Empty/None → zero vector."""
143
+ vec = [0] * self.n_feat
144
+ if not feats:
145
+ return vec
146
+ if isinstance(feats, dict):
147
+ items = (f"{k}={v}" for k, v in feats.items())
148
+ else:
149
+ items = feats.split("|")
150
+ for kv in items:
151
+ i = self.feat_to_id.get(kv)
152
+ if i is not None:
153
+ vec[i] = 1
154
+ return vec
155
+
156
+ def encode_edit_script(self, form: str, lemma: str) -> int:
157
+ script = extract_edit_script(form, lemma)
158
+ return self.edit_to_id.get(_script_key(script), self.edit_to_id[OOV])
159
+
160
+ # ----------------------------- save / load -------------------------------------------------
161
+ def save(self, path: str) -> None:
162
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
163
+ blob = {
164
+ "upos_to_id": self.upos_to_id,
165
+ "deprel_to_id": self.deprel_to_id,
166
+ "feat_to_id": self.feat_to_id,
167
+ "edit_to_id": self.edit_to_id,
168
+ # store edit_scripts as list of [delete_n, append_s] for JSON
169
+ "edit_scripts": [[d, a] for (d, a) in self.edit_scripts],
170
+ }
171
+ with open(path, "w", encoding="utf-8") as fh:
172
+ json.dump(blob, fh, ensure_ascii=False, indent=2)
173
+
174
+ @classmethod
175
+ def load(cls, path: str) -> "LabelVocab":
176
+ with open(path, encoding="utf-8") as fh:
177
+ blob = json.load(fh)
178
+ edit_scripts = [(int(d), str(a)) for (d, a) in blob["edit_scripts"]]
179
+ return cls(
180
+ upos_to_id=blob["upos_to_id"],
181
+ deprel_to_id=blob["deprel_to_id"],
182
+ feat_to_id=blob["feat_to_id"],
183
+ edit_to_id=blob["edit_to_id"],
184
+ edit_scripts=edit_scripts,
185
+ )
186
+
187
+
188
+ def _script_key(script: tuple[int, str]) -> str:
189
+ """Canonical string form of an edit-script for use as a dict key. Uses ':' as a separator
190
+ because lemmas can contain almost anything but rarely contain colons in initial position."""
191
+ delete_n, append_s = script
192
+ return f"{delete_n}:{append_s}"
juurio/parser/model.py ADDED
@@ -0,0 +1,714 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Tiny distilled Finnish UD parser model.
2
+
3
+ Architecture rationale:
4
+ - A top-1000 edit-script vocabulary covers nearly all UD-TDT-dev lemmas, so
5
+ lemmatization is a classifier head over edit scripts.
6
+ - Multiword-token incidence is low, so there is no MWT head; instead MWTs are
7
+ detected and flagged at parse time.
8
+ - A rule-based tokenizer already reaches high F1, so there is no tokenizer head.
9
+
10
+ **Encoder-only transformer + four task heads, all reading the same per-token hidden state:**
11
+
12
+ token-ids ──BPE-embedded─┐
13
+ ├──TransformerEncoder──> h (B, T, d)
14
+ pos-embed ────────────────┘ │
15
+ │
16
+ ┌────────────────────────────────────────────────────┤
17
+ │ │ │ │
18
+ UPOS Feats Lemma Parser (biaffine)
19
+ Linear(d, 17) Linear(d, ~80) Linear(d,1000) head_mlp(d) → 200
20
+ dep_mlp(d) → 200
21
+ arc: dep · W_arc · head.T → (B,T,T)
22
+ rel: einsum bilinear → (B,T,T,40)
23
+
24
+ The success criteria (small parameter budget, low CPU latency per sentence, calibrated
25
+ confidence) are *post-training* properties; this file just lays down the architecture so the
26
+ training loop (next module) has a public surface to depend on. Shapes / padding contract are
27
+ nailed by `tests/test_tinyfiud.py`.
28
+
29
+ **Calling convention (must be respected by the data pipeline):**
30
+ - `ids` is a `(B, T)` long tensor; `ids[b, 0] == ROOT_ID` is the virtual UD root token. UD's
31
+ `head=0` (sentence root) maps to this position natively. The data pipeline prepends ROOT
32
+ before tokenizing the rest of the sentence.
33
+ - `mask` is `(B, T)` bool — True where the position holds a real token (incl. ROOT), False
34
+ where it's padding. The model uses `~mask` as `src_key_padding_mask` for the encoder and
35
+ masks padded HEAD positions in the arc/rel scores to `-inf` (the model owns the padding
36
+ contract — without this, MST/argmax could attach a dependent to a pad position).
37
+
38
+ **Architecture knobs to revisit if metrics miss target:**
39
+ - `head_mlp` and `dep_mlp` are shared between the arc and rel scorers. Standard Dozat-Manning
40
+ uses separate projections per task. If relation accuracy lags arc accuracy, splitting the
41
+ MLPs is the first architecture knob to try (adds ~80 K params).
42
+ """
43
+ from __future__ import annotations
44
+
45
+ import torch
46
+ import torch.nn as nn
47
+
48
+
49
+ # Reserved special token ids. The data pipeline (tokenizer + collator) must:
50
+ # - Use PAD_ID for padding.
51
+ # - Prepend a single ROOT_ID token at position 0 of every sentence (the virtual root for UD
52
+ # head selection). UD's head=0 ("root of sentence") then maps to this position natively.
53
+ PAD_ID = 0
54
+ ROOT_ID = 1
55
+
56
+
57
+ def per_token_confidence(logits: torch.Tensor) -> torch.Tensor:
58
+ """Per-token (top1 − top2) softmax margin — the same confidence signal the realizers use.
59
+ `logits` is `(..., n_classes)`; the return is `(...)` with one scalar per position.
60
+
61
+ Top-1 minus top-2 (not top-1 alone) because it correctly captures *competition*: a 0.51-vs-0.49
62
+ decision is uncertain even though top-1 is 0.51. A 0.95-vs-0.02 decision is confident."""
63
+ probs = torch.softmax(logits, dim=-1)
64
+ top2 = probs.topk(2, dim=-1).values
65
+ return top2[..., 0] - top2[..., 1]
66
+
67
+
68
+ class TinyFiUD(nn.Module):
69
+ """Tiny encoder-only transformer + four UD task heads.
70
+
71
+ Constructor args (all keyword):
72
+ vocab : BPE vocabulary size (target 8000)
73
+ d_model : hidden dim (target 256)
74
+ layers : encoder depth (target 6)
75
+ heads : attention heads (target 4)
76
+ ff : FFN inner dim (target 1024)
77
+ n_upos : number of UPOS labels (17)
78
+ n_feat : number of binary feature key=values for multi-label head (~80)
79
+ n_edit : edit-script vocabulary size (target 1000)
80
+ n_rel : number of dependency relation labels (~40)
81
+ maxlen : max sequence length the position embedding supports
82
+ dropout : transformer / MLP dropout
83
+ """
84
+
85
+ def __init__(
86
+ self,
87
+ vocab: int = 8000,
88
+ d_model: int = 256,
89
+ layers: int = 6,
90
+ heads: int = 4,
91
+ ff: int = 1024,
92
+ n_upos: int = 17,
93
+ n_feat: int = 80,
94
+ n_edit: int = 1000,
95
+ n_rel: int = 40,
96
+ maxlen: int = 512,
97
+ dropout: float = 0.1,
98
+ parser_bilstm_hidden: int = 0,
99
+ arc_d: int = 200,
100
+ arc_distance_bias: bool = False,
101
+ fst_tag_vocab_size: int = 0,
102
+ fst_tag_dim: int = 32,
103
+ compound_enabled: bool = False,
104
+ char_vocab_size: int = 0,
105
+ char_emb_dim: int = 16,
106
+ char_conv_channels: int = 64,
107
+ max_form_len: int = 64,
108
+ compound_detach_encoder: bool = False,
109
+ parser_private_hidden: int = 0,
110
+ # Optional projection of the final encoder hidden (`d_model`) up to
111
+ # an external hidden dim, used only as a training-time auxiliary
112
+ # target. When True, the forward emits `out["hidden"]` and
113
+ # `out["hidden_proj"]`. Default OFF preserves the base forward + ckpt
114
+ # shape bit-identically.
115
+ use_hidden_kd_proj: bool = False,
116
+ hidden_kd_proj_dim: int = 768,
117
+ # Per-key biaffine Feats head replacing flat BCE. When
118
+ # `use_per_key_feats=True`, the model instantiates
119
+ # `self.feats_per_key_head` and the forward path emits
120
+ # `out["feats_per_key"]` (list of per-key logits) in addition
121
+ # to the existing `out["feats"]`. The legacy BCE `self.feats`
122
+ # Linear stays in place so older ckpts load 1:1; the freeze
123
+ # utility silences it during per-key training.
124
+ use_per_key_feats: bool = False,
125
+ feats_per_key_value_counts: list[int] | None = None,
126
+ feats_per_key_upos_emb_dim: int = 32,
127
+ feats_per_key_dropout: float = 0.0,
128
+ # Per-key value index → flat label_vocab.feat_to_id
129
+ # for the eval-time decode round-trip. Length == n_keys; row k has
130
+ # `value_counts[k]` entries (-1 at index 0 for `<EMPTY>`, real flat
131
+ # ids elsewhere). Built once from `FeatsKeySplit.flat_id_for_kv`.
132
+ # When None and use_per_key_feats=True, the forward path emits
133
+ # `feats_per_key` but does NOT override `out["feats"]` — useful for
134
+ # training-only configurations; eval requires the table.
135
+ feats_per_key_flat_ids: list[list[int]] | None = None,
136
+ # Optional near-identity capacity adapter stacked on top of the
137
+ # encoder. At init it is bitwise identity, so forward outputs
138
+ # match the un-adapted model within float32 rounding. When False
139
+ # (default), no adapter is built; older ckpts load 1:1 with no
140
+ # extra state_dict keys.
141
+ use_near_identity_adapter: bool = False,
142
+ near_identity_n_layers: int = 0,
143
+ near_identity_d_model: int | None = None,
144
+ near_identity_heads: int | None = None,
145
+ near_identity_ff: int | None = None,
146
+ ):
147
+ super().__init__()
148
+ self.maxlen = maxlen
149
+ self.compound_enabled = compound_enabled
150
+ # When True, the compound branch consumes `h.detach()` instead
151
+ # of `h`. Together with the freeze whitelist in
152
+ # `juurio.parser.freeze`, this blocks the compound losses from
153
+ # re-shifting the shared encoder. Default False keeps the
154
+ # encoder shared with the compound branch.
155
+ self.compound_detach_encoder = compound_detach_encoder
156
+ self.max_form_len = max_form_len
157
+ self.n_edit = n_edit
158
+ # ---------- backbone ----------
159
+ self.tok_emb = nn.Embedding(vocab, d_model, padding_idx=0)
160
+ self.pos_emb = nn.Embedding(maxlen, d_model)
161
+ # ---------- FST-candidate input channel ----------
162
+ # Bag-of-tags morphtag input from uralicNLP analyses. Per first-subw
163
+ # position the collator emits a (max_tags,) int sequence of tag ids; we
164
+ # embed each, sum (pad embedding is constrained to zero so padded slots
165
+ # contribute nothing), and project through a zero-init linear so the
166
+ # encoder's output is bit-identical at initialization.
167
+ # Once gradients shape the projection the channel injects morphological
168
+ # disambiguation signal into the encoder, helping UPOS / Feats / Lemma /
169
+ # and indirectly LAS via better Case/Number agreement on long arcs.
170
+ # See `juurio/parser/fst_cache.py` for the cache + vocab format.
171
+ self.fst_tag_vocab_size = fst_tag_vocab_size
172
+ if fst_tag_vocab_size > 0:
173
+ self.fst_tag_emb = nn.Embedding(fst_tag_vocab_size, fst_tag_dim, padding_idx=0)
174
+ # zero PAD row explicitly (Embedding padding_idx zeros gradients but the
175
+ # initial values are random; force zero here so the bag sum at init is 0)
176
+ with torch.no_grad():
177
+ self.fst_tag_emb.weight[0].zero_()
178
+ self.fst_proj = nn.Linear(fst_tag_dim, d_model, bias=False)
179
+ nn.init.zeros_(self.fst_proj.weight) # zero-init → identity at start
180
+ else:
181
+ self.fst_tag_emb = None
182
+ self.fst_proj = None
183
+ self.encoder = nn.TransformerEncoder(
184
+ nn.TransformerEncoderLayer(d_model, heads, ff, dropout, batch_first=True),
185
+ num_layers=layers,
186
+ )
187
+ # Optional near-identity adapter on top of the encoder. None when
188
+ # disabled — keeps older ckpts byte-stable (no extra state_dict
189
+ # keys). When enabled, the adapter is `n_layers` near-identity
190
+ # blocks; at init each block is bit-for-bit identity so the adapted
191
+ # forward equals the un-adapted forward within float32 rounding.
192
+ self.near_identity_adapter = None
193
+ if use_near_identity_adapter and near_identity_n_layers > 0:
194
+ from .near_identity import NearIdentityAdapter
195
+ self.near_identity_adapter = NearIdentityAdapter(
196
+ d_model=(near_identity_d_model or d_model),
197
+ heads=(near_identity_heads or heads),
198
+ ff=(near_identity_ff or ff),
199
+ dropout=dropout,
200
+ n_layers=near_identity_n_layers,
201
+ )
202
+ # ---------- per-token task heads ----------
203
+ self.upos = nn.Linear(d_model, n_upos)
204
+ self.feats = nn.Linear(d_model, n_feat)
205
+ # Hidden KD projection. Built only when use_hidden_kd_proj;
206
+ # default None preserves the base ckpt shape.
207
+ self.hidden_kd_proj: nn.Linear | None = None
208
+ self._hidden_kd_proj_dim: int = hidden_kd_proj_dim
209
+ if use_hidden_kd_proj:
210
+ self.hidden_kd_proj = nn.Linear(d_model, hidden_kd_proj_dim)
211
+ # Feats-private adapter is not built at serving time; the
212
+ # Feats head consumes the encoder output directly. Kept as None so
213
+ # the forward path's `self.feats_adapter is not None` guard is a
214
+ # no-op and older checkpoints load unchanged (no extra keys).
215
+ self.feats_adapter = None
216
+ # Optional per-key biaffine Feats head.
217
+ # None when disabled (byte-stable; no extra
218
+ # state_dict keys). Builder validates value_counts presence
219
+ # and forbids the zero-key edge case.
220
+ self.feats_per_key_head = None
221
+ self.use_per_key_feats = use_per_key_feats
222
+ if use_per_key_feats:
223
+ if not feats_per_key_value_counts:
224
+ raise ValueError(
225
+ "use_per_key_feats=True requires `feats_per_key_value_counts` "
226
+ "(per-key value counts including the <EMPTY> class at 0). "
227
+ "Build via `FeatsKeySplit.from_label_vocab(vocab).value_counts`."
228
+ )
229
+ from .feats_per_key_head import PerKeyFeatsHead
230
+ self.feats_per_key_head = PerKeyFeatsHead(
231
+ d_model=d_model,
232
+ upos_emb_dim=feats_per_key_upos_emb_dim,
233
+ value_counts=list(feats_per_key_value_counts),
234
+ n_upos=n_upos,
235
+ dropout=feats_per_key_dropout,
236
+ )
237
+ # Build a padded lookup buffer (n_keys, max_n_values) of flat
238
+ # feat_to_id ids; entries past the per-key value count and
239
+ # `<EMPTY>` slots are -1 (no bit set). Used in forward to
240
+ # decode per-key argmax → multi-hot at eval time.
241
+ if feats_per_key_flat_ids is not None:
242
+ if len(feats_per_key_flat_ids) != len(feats_per_key_value_counts):
243
+ raise ValueError(
244
+ "feats_per_key_flat_ids length must equal value_counts length"
245
+ )
246
+ max_v = max(feats_per_key_value_counts)
247
+ lookup = torch.full((len(feats_per_key_value_counts), max_v),
248
+ -1, dtype=torch.long)
249
+ for k_idx, row in enumerate(feats_per_key_flat_ids):
250
+ n_v = feats_per_key_value_counts[k_idx]
251
+ # Accept rows that match the key's exact value count OR
252
+ # the padded max_v width (ckpt round-trip serializes the
253
+ # padded buffer). Trailing -1 entries are ignored.
254
+ if len(row) not in (n_v, max_v):
255
+ raise ValueError(
256
+ f"feats_per_key_flat_ids[{k_idx}] length {len(row)} "
257
+ f"!= value_counts[{k_idx}] {n_v} or max_v {max_v}"
258
+ )
259
+ for v_idx in range(n_v):
260
+ lookup[k_idx, v_idx] = row[v_idx]
261
+ self.register_buffer("_feats_per_key_flat_lookup", lookup,
262
+ persistent=False)
263
+ else:
264
+ self.register_buffer("_feats_per_key_flat_lookup", None,
265
+ persistent=False)
266
+ self.lemma_edit = nn.Linear(d_model, n_edit)
267
+ # ---------- optional BiLSTM in the parser path ----------
268
+ # Rationale: in long sentences, attachment accuracy degrades with arc
269
+ # distance — the transformer's global self-attention does not carry
270
+ # enough syntactic structure across long contexts for the biaffine
271
+ # scorer. A 1-layer BiLSTM placed before the arc/rel heads gives the
272
+ # scorer an explicit left-to-right + right-to-left summary. We do NOT
273
+ # feed this into upos/feats/lemma heads — they already score well and
274
+ # the added module shouldn't perturb a strong signal. Hidden=d_model/2
275
+ # so the bidirectional output is d_model again and feeds in via a clean
276
+ # residual.
277
+ #
278
+ # `parser_bilstm_hidden` is the actual LSTM hidden dim (not just a bool),
279
+ # and the forward path uses pack_padded_sequence so pad positions never
280
+ # leak into the backward direction.
281
+ if parser_bilstm_hidden > 0:
282
+ assert parser_bilstm_hidden * 2 == d_model, (
283
+ f"parser_bilstm_hidden={parser_bilstm_hidden} must equal d_model/2 "
284
+ f"({d_model // 2}) so the bidirectional output dim matches d_model "
285
+ f"and the residual is clean.")
286
+ self.parser_bilstm = nn.LSTM(
287
+ d_model, parser_bilstm_hidden,
288
+ num_layers=1, bidirectional=True, batch_first=True, dropout=0.0,
289
+ )
290
+ else:
291
+ self.parser_bilstm = None
292
+ # ---------- parser-private FF residual block ----------
293
+ # Residual FF inserted between the encoder output and the parser
294
+ # branch (bilstm → head_mlp/dep_mlp/biaffine). The non-parser heads
295
+ # (upos, feats, lemma_edit) still consume the raw encoder output `h`.
296
+ #
297
+ # Shape: d_model → parser_private_hidden → d_model. GELU + dropout
298
+ # between. No LayerNorm.
299
+ #
300
+ # Init: linear1 default Xavier; linear2 weight + bias both ZERO so
301
+ # parser_private(h) = 0 at step 0 → h_for_parser = h + 0 = h.
302
+ # Forward output bit-identical to a no-private model at init.
303
+ #
304
+ # Wake-up induction (2 steps): linear2 wakes at step 1 (intermediate
305
+ # = GELU(linear1(h)) is non-zero from random linear1 init). After
306
+ # an AdamW step linear2.weight is non-zero, then linear1 wakes at
307
+ # the next step. See tests/test_parser_private.py for the regression.
308
+ #
309
+ # Diagnostic: log ||self.parser_private_linear2.weight|| at eval
310
+ # time to tell whether the block failed to wake or woke but did
311
+ # not help.
312
+ self.parser_private_hidden = parser_private_hidden
313
+ if parser_private_hidden > 0:
314
+ self.parser_private_linear1 = nn.Linear(d_model, parser_private_hidden)
315
+ self.parser_private_linear2 = nn.Linear(parser_private_hidden, d_model)
316
+ self.parser_private_drop = nn.Dropout(dropout)
317
+ with torch.no_grad():
318
+ self.parser_private_linear2.weight.zero_()
319
+ self.parser_private_linear2.bias.zero_()
320
+ else:
321
+ self.parser_private_linear1 = None
322
+ self.parser_private_linear2 = None
323
+ self.parser_private_drop = None
324
+ # ---------- biaffine parser ----------
325
+ # arc_d is configurable. Dozat-Manning's paper used 100. The W_rel cost
326
+ # (n_rel × arc_d²) is a large share of model params for negligible
327
+ # label-given-head gain, so a smaller arc_d is usually a good tradeoff.
328
+ # head_mlp / dep_mlp: per-token arc_d-dim projection (Dozat & Manning scale)
329
+ self.arc_d = arc_d
330
+ self.head_mlp = nn.Sequential(nn.Linear(d_model, arc_d), nn.GELU(), nn.Dropout(dropout))
331
+ self.dep_mlp = nn.Sequential(nn.Linear(d_model, arc_d), nn.GELU(), nn.Dropout(dropout))
332
+ # arc bilinear: arc_score[b, dep, head] = dep_repr_b_dep · W_arc · head_repr_b_head + bias
333
+ self.W_arc = nn.Parameter(torch.empty(arc_d, arc_d))
334
+ nn.init.xavier_uniform_(self.W_arc)
335
+ self.head_bias = nn.Linear(arc_d, 1, bias=False) # head-only bias term (Dozat-Manning)
336
+ # rel bilinear: rel_score[b, dep, head, r] = dep · W_rel[r] · head + linear terms
337
+ self.W_rel = nn.Parameter(torch.empty(n_rel, arc_d, arc_d))
338
+ nn.init.xavier_uniform_(self.W_rel)
339
+ self.rel_lin_h = nn.Linear(arc_d, n_rel, bias=False)
340
+ self.rel_lin_d = nn.Linear(arc_d, n_rel, bias=False)
341
+ self.rel_bias = nn.Parameter(torch.zeros(n_rel))
342
+
343
+ # ---------- bucketed arc-distance × direction bias ----------
344
+ # In long sentences the model can label arcs well but struggles to find
345
+ # the right head at long distances. Add a tiny learned bias term that
346
+ # depends on signed bucketed (head_pos - dep_pos) — 11 scalars total
347
+ # (5 distance buckets × 2 directions + 1 root). Precompute the
348
+ # bucket-index table once as a buffer; in forward we just `[]`-index.
349
+ self.arc_distance_bias = arc_distance_bias
350
+ if arc_distance_bias:
351
+ # 11 buckets: 0=root, 1-5=left (head before dep), 6-10=right (head after dep)
352
+ # bucket boundaries match the eval bins: dist in {1, 2-3, 4-7, 8-15, 16+}
353
+ T = maxlen
354
+ idx = torch.zeros(T, T, dtype=torch.long)
355
+ for d in range(T):
356
+ for hp in range(T):
357
+ if hp == 0:
358
+ idx[d, hp] = 0 # root
359
+ continue
360
+ signed = hp - d
361
+ abs_d = abs(signed)
362
+ if abs_d == 1:
363
+ b = 0
364
+ elif abs_d <= 3:
365
+ b = 1
366
+ elif abs_d <= 7:
367
+ b = 2
368
+ elif abs_d <= 15:
369
+ b = 3
370
+ else:
371
+ b = 4
372
+ idx[d, hp] = 1 + b + (5 if signed > 0 else 0)
373
+ self.register_buffer("_arc_dist_idx", idx)
374
+ self.arc_dist_bias = nn.Parameter(torch.zeros(11))
375
+
376
+ # ---------- compound-lemma heads ----------
377
+ # Three heads attached to the per-token encoder hidden state (no new
378
+ # backbone params). When `compound_enabled=False` (default), this is a
379
+ # no-op — the forward dict stays surface-compatible.
380
+ #
381
+ # 1. count head: Linear(d_model, 4) → CE over {0, 1, 2, 3+} `#`s.
382
+ # 2. split scorer: char-level CNN over the token's form characters,
383
+ # FiLM-conditioned on `h_token`. Emits a per-char logit; decode
384
+ # picks the top-K legal positions.
385
+ # 3. tail-script head: Linear(d_model, n_edit). Separate from
386
+ # `lemma_edit` so the two heads can specialise (whole-form vs
387
+ # form-tail edit script).
388
+ #
389
+ # The char vocab includes PAD=0 and UNK=1; PAD-row embedding is
390
+ # constrained to zero so padded char slots inject no signal through
391
+ # the conv (same trick used for FST PAD).
392
+ if compound_enabled:
393
+ if char_vocab_size < 2:
394
+ raise ValueError(
395
+ f"compound_enabled requires char_vocab_size>=2 (got {char_vocab_size}); "
396
+ f"reserve PAD=0 and UNK=1 at minimum.")
397
+ self.char_emb = nn.Embedding(char_vocab_size, char_emb_dim, padding_idx=0)
398
+ with torch.no_grad():
399
+ self.char_emb.weight[0].zero_()
400
+ # 1D conv across char positions; pad to keep length=max_form_len.
401
+ self.compound_split_conv = nn.Conv1d(
402
+ in_channels=char_emb_dim,
403
+ out_channels=char_conv_channels,
404
+ kernel_size=3, padding=1, bias=True,
405
+ )
406
+ # FiLM: project h_token to (γ, β) of size char_conv_channels each.
407
+ # Init γ=1, β=0 so the modulation is identity at start of training.
408
+ # Zero the weights and set bias so the first `char_conv_channels`
409
+ # outputs (γ) are 1 and the second half (β) are 0.
410
+ self.compound_film = nn.Linear(d_model, 2 * char_conv_channels, bias=True)
411
+ with torch.no_grad():
412
+ self.compound_film.weight.zero_()
413
+ self.compound_film.bias[:char_conv_channels].fill_(1.0) # γ ← 1
414
+ self.compound_film.bias[char_conv_channels:].zero_() # β ← 0
415
+ # Per-position scalar logit on top of FiLM-modulated channels.
416
+ self.compound_split_out = nn.Linear(char_conv_channels, 1, bias=True)
417
+ # Count head + tail-script head read straight from h_token.
418
+ self.compound_count = nn.Linear(d_model, 4, bias=True)
419
+ self.compound_tail = nn.Linear(d_model, n_edit, bias=True)
420
+ else:
421
+ self.char_emb = None
422
+ self.compound_split_conv = None
423
+ self.compound_film = None
424
+ self.compound_split_out = None
425
+ self.compound_count = None
426
+ self.compound_tail = None
427
+
428
+ def _encode(self, ids: torch.Tensor, mask: torch.Tensor,
429
+ fst_tags: torch.Tensor | None = None) -> torch.Tensor:
430
+ """ids: (B, T) longs; mask: (B, T) bools (True = real token).
431
+ fst_tags: (B, T, max_tags) longs of morphtag ids per first-subword position
432
+ (zeros elsewhere, which is the PAD id so they contribute nothing).
433
+ Returns (B, T, d)."""
434
+ B, T = ids.shape
435
+ if T > self.maxlen:
436
+ raise ValueError(f"sequence length {T} exceeds maxlen {self.maxlen}")
437
+ positions = torch.arange(T, device=ids.device).unsqueeze(0).expand(B, T)
438
+ x = self.tok_emb(ids) + self.pos_emb(positions)
439
+ # FST channel: zero-init at start so this is a no-op until training shapes
440
+ # `fst_proj`. After init it injects per-token morphtag bag info at every
441
+ # position whose `fst_tags` row contains non-PAD ids.
442
+ if self.fst_tag_emb is not None and fst_tags is not None:
443
+ tag_emb = self.fst_tag_emb(fst_tags).sum(dim=2) # (B, T, fst_tag_dim)
444
+ x = x + self.fst_proj(tag_emb)
445
+ # nn.TransformerEncoder uses `src_key_padding_mask` where True = should be IGNORED
446
+ # (i.e. position is padding). Our `mask` is True for real tokens, so invert.
447
+ h = self.encoder(x, src_key_padding_mask=~mask)
448
+ # Optional near-identity adapter. At init this is bitwise identity
449
+ # on real-token positions, so the adapted forward equals the
450
+ # un-adapted forward within float32 rounding.
451
+ if self.near_identity_adapter is not None:
452
+ h = self.near_identity_adapter(h, mask=mask)
453
+ return h
454
+
455
+ def forward(self, ids: torch.Tensor, mask: torch.Tensor,
456
+ fst_tags: torch.Tensor | None = None,
457
+ form_chars: torch.Tensor | None = None,
458
+ form_char_mask: torch.Tensor | None = None,
459
+ ) -> dict[str, torch.Tensor]:
460
+ """Returns a dict with logits for each head:
461
+ upos : (B, T, n_upos)
462
+ feats : (B, T, n_feat)
463
+ lemma : (B, T, n_edit)
464
+ arc : (B, T, T) — arc[b, dep, head] is the score for `head` being the head of `dep`
465
+ rel : (B, T, T, n_rel) — rel[b, dep, head, r] is the score for relation r on that arc
466
+
467
+ When `compound_enabled`, three additional keys are returned:
468
+ count : (B, T, 4) — CE logits over {0, 1, 2, 3+} `#`s
469
+ split : (B, T, max_form_len) — per-char-pos boundary logits
470
+ tail : (B, T, n_edit) — tail-script logits
471
+
472
+ `form_chars` (B, T, max_form_len) and `form_char_mask` (B, T,
473
+ max_form_len) are required when compound is enabled; a missing pair
474
+ raises ValueError (same guard pattern as the FST channel — silent
475
+ disabling at eval is a footgun we don't repeat).
476
+ """
477
+ if self.compound_enabled and (form_chars is None or form_char_mask is None):
478
+ raise ValueError(
479
+ "compound_enabled=True but form_chars/form_char_mask were not "
480
+ "passed to forward(). Caller must provide both — silently disabling "
481
+ "the compound heads at eval would hide a real misconfiguration.")
482
+ h = self._encode(ids, mask, fst_tags=fst_tags)
483
+ # Parser-private FF residual block. Only the parser
484
+ # branch sees `h_for_parser`; the non-parser heads (upos, feats,
485
+ # lemma_edit) consume the raw encoder `h` further down. At step 0
486
+ # linear2.weight = 0 → parser_private(h) = 0 → h_for_parser = h
487
+ # (bit-identical-to-no-private forward). Once linear2 wakes,
488
+ # h_for_parser becomes a parser-private rewrite of h that the
489
+ # non-parser heads do NOT see.
490
+ if self.parser_private_linear1 is not None:
491
+ private = self.parser_private_linear2(
492
+ self.parser_private_drop(
493
+ torch.nn.functional.gelu(self.parser_private_linear1(h))))
494
+ h_for_parser = h + private
495
+ else:
496
+ h_for_parser = h
497
+ # Optional BiLSTM-augmented representation for the
498
+ # parser path only. The encoder's `src_key_padding_mask` only masks pads
499
+ # as keys/values, not as query outputs — pad positions still get embedded
500
+ # and have nonzero `h`. Without packing, the backward LSTM would read the
501
+ # pad suffix BEFORE real tokens, making real-token representations depend
502
+ # on batch padding. Pack via the real lengths so the LSTM never sees pads.
503
+ if self.parser_bilstm is not None:
504
+ lengths = mask.sum(dim=1).to("cpu", dtype=torch.long)
505
+ packed = nn.utils.rnn.pack_padded_sequence(
506
+ h_for_parser, lengths, batch_first=True, enforce_sorted=False)
507
+ packed_out, _ = self.parser_bilstm(packed)
508
+ h_lstm, _ = nn.utils.rnn.pad_packed_sequence(
509
+ packed_out, batch_first=True, total_length=h.size(1))
510
+ h_arc = h_for_parser + h_lstm # residual
511
+ else:
512
+ h_arc = h_for_parser
513
+ head_repr = self.head_mlp(h_arc) # (B, T, arc_d)
514
+ dep_repr = self.dep_mlp(h_arc) # (B, T, arc_d)
515
+
516
+ # arc_score[b, dep, head] = dep_repr_dep · W_arc · head_repr_head + head_bias(head_repr_head)
517
+ arc = torch.einsum("bdi,ij,bhj->bdh", dep_repr, self.W_arc, head_repr)
518
+ arc = arc + self.head_bias(head_repr).squeeze(-1).unsqueeze(1) # (B,1,T) broadcast over dep
519
+ # Optional bucketed arc-distance × direction bias targeting long-distance
520
+ # attachment errors. 11 learned scalars, sliced to current seq length and
521
+ # broadcast across batch.
522
+ if self.arc_distance_bias:
523
+ T = arc.shape[-1]
524
+ arc = arc + self.arc_dist_bias[self._arc_dist_idx[:T, :T]]
525
+
526
+ # rel_score[b, dep, head, r] = dep · W_rel[r] · head + lin_d(dep) + lin_h(head) + bias[r]
527
+ rel = torch.einsum("bdi,rij,bhj->bdhr", dep_repr, self.W_rel, head_repr)
528
+ rel = rel + self.rel_lin_d(dep_repr).unsqueeze(2) + self.rel_lin_h(head_repr).unsqueeze(1)
529
+ rel = rel + self.rel_bias
530
+
531
+ # Mask padded HEAD positions to -inf so MST/argmax can never attach to padding. Done inside
532
+ # the model (not the training loop) because the model owns the padding contract.
533
+ # mask shape (B, T) — True = real, False = pad. We broadcast over the dep axis.
534
+ pad_head_mask = ~mask # (B, T) True where head is pad
535
+ arc = arc.masked_fill(pad_head_mask.unsqueeze(1), float("-inf")) # (B, 1, T) → (B, T, T)
536
+ rel = rel.masked_fill(pad_head_mask.unsqueeze(1).unsqueeze(-1), float("-inf"))
537
+ # Mask the diagonal so a token can never be its own head.
538
+ # Without this, the arc loss has a phantom self-attachment class that
539
+ # routes gradients into the "left 2-3" arc_distance_bias bucket (since
540
+ # signed=0 falls into abs_d <= 3) and could win at argmax inference.
541
+ T_ = arc.shape[-1]
542
+ eye = torch.eye(T_, dtype=torch.bool, device=arc.device).unsqueeze(0) # (1, T, T)
543
+ arc = arc.masked_fill(eye, float("-inf"))
544
+ rel = rel.masked_fill(eye.unsqueeze(-1), float("-inf"))
545
+
546
+ upos_logits = self.upos(h)
547
+ out = {
548
+ "upos": upos_logits,
549
+ # Optional feats-private adapter. None → raw encoder hidden.
550
+ "feats": self.feats(
551
+ self.feats_adapter(h) if self.feats_adapter is not None else h
552
+ ),
553
+ "lemma": self.lemma_edit(h),
554
+ "arc": arc,
555
+ "rel": rel,
556
+ }
557
+ # Expose the final encoder hidden + an auxiliary projection when the
558
+ # projection head is built. `out["hidden"]` is the raw (B, T, d_model)
559
+ # tensor; `out["hidden_proj"]` is its (B, T, proj_dim) projection, used
560
+ # only by the optional auxiliary training loss. Both keys exist
561
+ # together; serving/eval callers ignore them.
562
+ if self.hidden_kd_proj is not None:
563
+ out["hidden"] = h
564
+ out["hidden_proj"] = self.hidden_kd_proj(h)
565
+ # Emit per-key biaffine logits when the per-key head is active.
566
+ # UPOS argmax is detached so the per-key CE loss cannot leak
567
+ # gradient back through the (frozen) UPOS head — the head sees
568
+ # UPOS as a discrete frozen input at both train and eval.
569
+ if self.feats_per_key_head is not None:
570
+ with torch.no_grad():
571
+ upos_argmax = upos_logits.argmax(dim=-1) # (B, T)
572
+ per_key_logits = self.feats_per_key_head(h, upos_argmax)
573
+ out["feats_per_key"] = per_key_logits
574
+ # Eval-time round-trip: override out["feats"] with FAKE
575
+ # logits decoded from per-key argmax via the flat-id lookup.
576
+ # When the per-key head is trained with per-key CE (not BCE),
577
+ # the frozen BCE Linear's predictions would be garbage — but the
578
+ # existing eval pipeline reads out["feats"] via sigmoid>0.5.
579
+ # We synthesize logits where each "1" bit is +12 (sigmoid
580
+ # ≈ 0.99) and each "0" bit is -12 (sigmoid ≈ 0.01), so the
581
+ # threshold reproduces the per-key argmax multi-hot exactly.
582
+ # Fail-loud: at eval time (model.training=False), the lookup
583
+ # MUST be present — otherwise out["feats"] would silently be
584
+ # the frozen, untrained BCE Linear's predictions and Feats-F1
585
+ # would be garbage. Training is OK without lookup (loss path
586
+ # uses out["feats_per_key"] directly).
587
+ lookup = getattr(self, "_feats_per_key_flat_lookup", None)
588
+ if lookup is None and not self.training:
589
+ raise RuntimeError(
590
+ "TinyFiUD: per-key Feats head is active but the flat-id "
591
+ "lookup is missing — out['feats'] cannot be overridden. "
592
+ "Eval would read the (frozen, untrained-in-Stage-2) BCE "
593
+ "Linear's predictions, which are garbage. Pass "
594
+ "`feats_per_key_flat_ids` at construction (built from "
595
+ "`FeatsKeySplit.flat_id_for_kv`)."
596
+ )
597
+ if lookup is not None:
598
+ B, T, _ = h.shape
599
+ n_feat = out["feats"].shape[-1]
600
+ if self.training:
601
+ # DIFFERENTIABLE per-key → flat projection so a flat-bit
602
+ # auxiliary loss retains gradient through the per-key head.
603
+ # Without this, the eval-only fake ±12 path below would
604
+ # also be used at training time and that loss term would
605
+ # become a non-gradient constant.
606
+ #
607
+ # Projection rule:
608
+ # flat_logit(K=v) = per_key_logits[K][v] -
609
+ # per_key_logits[K][<EMPTY>]
610
+ # When warm-started per-key from BCE, the EMPTY col is
611
+ # zero and value cols equal the source BCE logits, so
612
+ # the margin reproduces source BCE logits byte-equally
613
+ # (Feats-F1 at step 0 ≈ source BCE Feats-F1). Unmapped
614
+ # flat positions keep the eval-path's -12 constant so
615
+ # sigmoid → 0. `lookup` is a (n_keys, max_V) tensor
616
+ # padded with -1 to the widest per-key value count.
617
+ # Slice each row by the actual V_k from per_key_logits
618
+ # so the margins and flat-id columns line up.
619
+ margins_per_key: list[torch.Tensor] = []
620
+ flat_ids_per_key: list[torch.Tensor] = []
621
+ for k_idx, key_logits in enumerate(per_key_logits):
622
+ n_v = key_logits.shape[-1]
623
+ empty_col = key_logits[..., 0:1] # (B, T, 1)
624
+ margins_k = key_logits[..., 1:n_v] - empty_col # (B, T, V_k - 1)
625
+ flat_ids_k = lookup[k_idx][1:n_v] # (V_k - 1,)
626
+ margins_per_key.append(margins_k)
627
+ flat_ids_per_key.append(flat_ids_k)
628
+ all_margins = torch.cat(margins_per_key, dim=-1) # (B, T, N_total)
629
+ all_flat_ids = torch.cat(flat_ids_per_key, dim=0) # (N_total,)
630
+ valid = all_flat_ids >= 0
631
+ if not bool(valid.all()):
632
+ valid_idx = valid.nonzero(as_tuple=True)[0]
633
+ all_margins = all_margins.index_select(-1, valid_idx)
634
+ all_flat_ids = all_flat_ids.index_select(0, valid_idx)
635
+ flat_base = per_key_logits[0].new_full(
636
+ (B, T, n_feat), -12.0
637
+ )
638
+ idx = all_flat_ids.view(1, 1, -1).expand(B, T, -1)
639
+ flat_logits = flat_base.scatter(-1, idx, all_margins)
640
+ out["feats"] = flat_logits
641
+ else:
642
+ # Eval-time: argmax → multi-hot → ±12 fake logits.
643
+ argmax_list = [t.argmax(dim=-1) for t in per_key_logits]
644
+ multi_hot = torch.zeros(B, T, n_feat,
645
+ dtype=torch.bool, device=h.device)
646
+ for k_idx, argmax_k in enumerate(argmax_list):
647
+ flat_ids_k = lookup[k_idx][argmax_k] # (B, T)
648
+ set_mask = flat_ids_k >= 0
649
+ if set_mask.any():
650
+ b_idx, t_idx = set_mask.nonzero(as_tuple=True)
651
+ flat_idx = flat_ids_k[b_idx, t_idx]
652
+ multi_hot[b_idx, t_idx, flat_idx] = True
653
+ fake_logits = torch.where(
654
+ multi_hot,
655
+ torch.full_like(out["feats"], 12.0),
656
+ torch.full_like(out["feats"], -12.0),
657
+ )
658
+ out["feats"] = fake_logits
659
+
660
+ if self.compound_enabled:
661
+ # h_token (B, T, d) is the per-position encoder hidden state. The
662
+ # collator's `first_subw` gates the loss/decode onto real tokens
663
+ # downstream — at this layer we emit dense (B, T, ...) outputs.
664
+ #
665
+ # When `compound_detach_encoder=True`, the compound branch
666
+ # sees a detached copy of `h`. This blocks gradient from
667
+ # compound losses (count/split/tail) flowing back into the
668
+ # shared encoder. The detach is autograd-only — forward
669
+ # values are identical.
670
+ h_compound = h.detach() if self.compound_detach_encoder else h
671
+ B, T, _ = h.shape
672
+ F = self.max_form_len
673
+ # Shape/dtype guards, not asserts.
674
+ if form_chars.shape != (B, T, F):
675
+ raise ValueError(
676
+ f"form_chars shape {tuple(form_chars.shape)} != "
677
+ f"(B, T, max_form_len)=({B},{T},{F})")
678
+ if form_char_mask.shape != (B, T, F):
679
+ raise ValueError(
680
+ f"form_char_mask shape {tuple(form_char_mask.shape)} != "
681
+ f"(B, T, max_form_len)=({B},{T},{F})")
682
+ if form_char_mask.dtype != torch.bool:
683
+ raise ValueError(
684
+ f"form_char_mask dtype must be torch.bool, got {form_char_mask.dtype}")
685
+ # Char embeddings: (B, T, F, char_emb_dim). PAD-row is constrained
686
+ # zero (see __init__) so PAD slots inject no signal through the
687
+ # conv. We further zero out positions where form_char_mask is False
688
+ # — defensive, since UNK chars may be non-PAD-id but still padding.
689
+ ce = self.char_emb(form_chars) # (B, T, F, ce)
690
+ ce = ce * form_char_mask.unsqueeze(-1).to(ce.dtype)
691
+ # Conv expects (N, C, L). Flatten (B, T) into batch:
692
+ ce_bt = ce.reshape(B * T, F, -1).transpose(1, 2) # (B*T, ce, F)
693
+ conv_out = self.compound_split_conv(ce_bt) # (B*T, ch, F)
694
+ conv_out = torch.nn.functional.gelu(conv_out)
695
+ # FiLM modulation conditioned on h_token (B, T, d) → reshape to
696
+ # (B*T, 2*ch) → split into γ, β → (B*T, ch, 1) each, broadcast on F.
697
+ ch = self.compound_split_conv.out_channels
698
+ gb = self.compound_film(h_compound).reshape(B * T, 2 * ch) # (B*T, 2*ch)
699
+ gamma, beta = gb[:, :ch].unsqueeze(-1), gb[:, ch:].unsqueeze(-1)
700
+ mod = gamma * conv_out + beta # (B*T, ch, F)
701
+ mod = torch.nn.functional.gelu(mod)
702
+ # Per-char-position logit: (B*T, ch, F) → (B*T, F, ch) → Linear → (B*T, F, 1)
703
+ split_logits = self.compound_split_out(mod.transpose(1, 2)).squeeze(-1)
704
+ split_logits = split_logits.reshape(B, T, F)
705
+ # Do NOT mask padded positions to -inf here — BCEWithLogits
706
+ # would NaN on them. The decoder masks padded positions before
707
+ # its top-K; the loss ignores them via form_char_mask. Split
708
+ # logits stay finite.
709
+
710
+ out["count"] = self.compound_count(h_compound)
711
+ out["split"] = split_logits
712
+ out["tail"] = self.compound_tail(h_compound)
713
+
714
+ return out
juurio/parser/near_identity.py ADDED
@@ -0,0 +1,108 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Near-identity capacity adapter.
2
+
3
+ A stack of transformer blocks that each behave as the IDENTITY
4
+ function at initialization, regardless of input scale, mask, or
5
+ batch contents. This lets a warm-started model add parameter
6
+ capacity while producing bit-identical predictions (within float32
7
+ rounding) at init. Training can then grow the adapter's `alpha`
8
+ parameters away from zero and exploit the new capacity without
9
+ leaving the base model's behaviour uncontrolled.
10
+
11
+ The block design:
12
+ - inner: `nn.TransformerEncoderLayer`.
13
+ - zero-init `self_attn.out_proj.weight/bias`: attention output → 0.
14
+ - zero-init `linear2.weight/bias`: FFN output → 0.
15
+ - learnable residual-scale scalar `alpha`, init = 0:
16
+ block(x) = x + alpha * (layer(x) - x)
17
+ At init alpha=0, so block(x) ≡ x bit-for-bit. As training
18
+ grows alpha, the block contributes its (now non-zero) layer
19
+ delta. The zero-init projections ensure the layer starts from
20
+ a well-conditioned point — not random — so as alpha grows the
21
+ delta is meaningful from step 1.
22
+
23
+ Padding invariance: the underlying `nn.TransformerEncoderLayer`
24
+ respects `src_key_padding_mask`, so even at alpha > 0 the block
25
+ cannot leak information from padded positions into real-token
26
+ outputs via self-attention.
27
+
28
+ Tests at `tests/test_near_identity_warm_start.py`.
29
+ """
30
+ from __future__ import annotations
31
+
32
+ import torch
33
+ from torch import nn
34
+
35
+
36
+ class NearIdentityBlock(nn.Module):
37
+ """One near-identity transformer block.
38
+
39
+ At init `block(x) == x` bit-for-bit (within float32 rounding,
40
+ ≤ 1e-5 max abs diff). The learnable `alpha` parameter starts at
41
+ 0 and grows during training to let the wrapped layer contribute.
42
+ """
43
+
44
+ def __init__(self, d_model: int, heads: int, ff: int, dropout: float):
45
+ super().__init__()
46
+ self.layer = nn.TransformerEncoderLayer(
47
+ d_model=d_model,
48
+ nhead=heads,
49
+ dim_feedforward=ff,
50
+ dropout=dropout,
51
+ batch_first=True,
52
+ )
53
+ # Zero-init the final attention output projection so
54
+ # `self_attn(x) → 0` at init regardless of input.
55
+ nn.init.zeros_(self.layer.self_attn.out_proj.weight)
56
+ nn.init.zeros_(self.layer.self_attn.out_proj.bias)
57
+ # Zero-init the final FFN projection so `ff(x) → 0` at init.
58
+ nn.init.zeros_(self.layer.linear2.weight)
59
+ nn.init.zeros_(self.layer.linear2.bias)
60
+ # Residual scale α starts at 0 — block is exact identity at init.
61
+ # We use a 1-element vector so the parameter is always a tensor
62
+ # (not a 0-d scalar) which composes more cleanly with broadcasting.
63
+ self.alpha = nn.Parameter(torch.zeros(1))
64
+
65
+ def forward(self, x: torch.Tensor, *, mask: torch.Tensor) -> torch.Tensor:
66
+ """Apply the block.
67
+
68
+ Args:
69
+ x: (B, T, d_model) — encoder hidden states.
70
+ mask: (B, T) bool — True at real-token positions, False at
71
+ padding. We translate to PyTorch's
72
+ `src_key_padding_mask` (True = ignore) by inverting.
73
+
74
+ Returns:
75
+ (B, T, d_model). At init exactly equal to `x` at real-token
76
+ positions (within ≤ 1e-5 max abs diff).
77
+ """
78
+ # PyTorch convention: True in src_key_padding_mask = position
79
+ # is padding and should be IGNORED by attention.
80
+ y = self.layer(x, src_key_padding_mask=~mask)
81
+ # `alpha * (y - x)` is 0 at init → output = x. As alpha grows,
82
+ # the block smoothly interpolates toward the full layer output.
83
+ return x + self.alpha * (y - x)
84
+
85
+
86
+ class NearIdentityAdapter(nn.Module):
87
+ """A stack of `NearIdentityBlock`s applied after the encoder.
88
+
89
+ `n_layers=0` is a strict no-op (returns input verbatim) used as the
90
+ ablation baseline. `n_layers >= 1` adds parameter capacity that
91
+ starts as identity and grows during training.
92
+ """
93
+
94
+ def __init__(
95
+ self, d_model: int, heads: int, ff: int, dropout: float, n_layers: int
96
+ ):
97
+ super().__init__()
98
+ self.layers = nn.ModuleList(
99
+ [
100
+ NearIdentityBlock(d_model=d_model, heads=heads, ff=ff, dropout=dropout)
101
+ for _ in range(n_layers)
102
+ ]
103
+ )
104
+
105
+ def forward(self, x: torch.Tensor, *, mask: torch.Tensor) -> torch.Tensor:
106
+ for layer in self.layers:
107
+ x = layer(x, mask=mask)
108
+ return x
juurio/parser/neural_adapter.py ADDED
@@ -0,0 +1,188 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """The served UD parser: `simple_tokenize` + sentence-split → hybrid morphology
2
+ channel → the neural model → the `{id, form, lemma, upos, head, deprel, feats}`
3
+ token dicts the rule engine (`algebra`) consumes.
4
+
5
+ This is the production `ParserAdapter` behind `pipeline.Transformer`. It needs no
6
+ external morphology toolkit at inference:
7
+
8
+ * tokenization is the rule-based `simple_tokenize`, not an external tokenizer;
9
+ * the FST-tag input channel is fed by the neural AnalyzeMux (`build_neural_cache`)
10
+ over a *static* compressed table (`FstCache.lookup` is a pure dict — miss ⇒
11
+ empty, never a live FST call).
12
+
13
+ Feats are decoded from the per-key head (`out["feats_per_key"]`), NOT the flat
14
+ `out["feats"]` BCE head — the mode-independent pattern new consumers should follow.
15
+ """
16
+ from __future__ import annotations
17
+
18
+ import os
19
+ import re
20
+
21
+ import torch
22
+
23
+ from ..pipeline import ParserAdapter
24
+ from .char_vocab import CharVocab
25
+ from .collator import SilverCollator
26
+ from .edit_scripts import apply_edit_script
27
+ from .eval import _load_checkpoint
28
+ from .fst_cache import FstCache, FstTagVocab
29
+ from .label_vocab import LabelVocab
30
+ from .silver import SilverSentence, SilverToken
31
+ from .simple_tokenizer import simple_tokenize
32
+ from .tokenizer import BPETokenizer
33
+ from ..analyze.hybrid import build_hybrid_channel
34
+ from ..analyze.infer import build_neural_cache, load_analyze_mux
35
+ from .. import bootstrap
36
+
37
+ # Repo root (the from-source layout where checkpoints/ and data/ live). Default artifact paths and any
38
+ # relative path passed in are resolved against this — NOT the process CWD — so the served parser works
39
+ # from any working directory (mirrors bootstrap.ROOT). Same root bootstrap downloads artifacts into.
40
+ _ROOT = os.path.normpath(os.path.join(os.path.dirname(__file__), "..", ".."))
41
+
42
+
43
+ def _resolve(path: str | None) -> str | None:
44
+ """Resolve a (possibly relative) artifact path against the repo ROOT, leaving absolute paths as-is."""
45
+ if path is None or os.path.isabs(path):
46
+ return path
47
+ return os.path.join(_ROOT, path)
48
+
49
+
50
+ _SENT_BOUNDARY = re.compile(r"(?<=[.!?])\s+")
51
+
52
+
53
+ def _segment_and_tokenize(text: str) -> list[list[str]]:
54
+ """Naive `[.!?]` sentence split, then the rule-based tokenizer per sentence."""
55
+ out: list[list[str]] = []
56
+ for chunk in (_SENT_BOUNDARY.split(text.strip()) if text.strip() else []):
57
+ toks = simple_tokenize(chunk)
58
+ if toks:
59
+ out.append(toks)
60
+ return out
61
+
62
+
63
+ def _placeholder(tokens: list[str]) -> SilverSentence:
64
+ """A `SilverSentence` carrying `tokens` as forms with placeholder labels — the collator only reads
65
+ form/lemma for subword alignment; predict overwrites every label."""
66
+ return SilverSentence(text=" ".join(tokens),
67
+ tokens=[SilverToken(form=t, lemma=t.lower(), upos="X", feats="",
68
+ head=0 if i == 0 else 1,
69
+ deprel="root" if i == 0 else "dep")
70
+ for i, t in enumerate(tokens)])
71
+
72
+
73
+ class NeuralParser(ParserAdapter):
74
+ """UD parser backed by the morphology channel plus the neural model.
75
+
76
+ Pass `compressed_table=` for the shipped serving config; the full
77
+ `fst_cache_v1.jsonl` is the larger default. Either way, membership in the
78
+ table picks clean-vs-neural tags and the morphology channel fills everything
79
+ else. `device=None` auto-selects CUDA when available."""
80
+
81
+ def __init__(self, *, parser_ckpt: str | None = None,
82
+ analyze_ckpt: str | None = None,
83
+ analyze_char_vocab: str | None = None,
84
+ tokenizer: str | None = None,
85
+ label_vocab: str | None = None,
86
+ fst_tag_vocab: str | None = None,
87
+ parser_char_vocab: str | None = None,
88
+ fst_cache: str = "data/fst_cache_v1.jsonl", compressed_table: str | None = None,
89
+ fst_max_tags: int = 16, device: str | None = None):
90
+ self.device = device or ("cuda" if torch.cuda.is_available() else "cpu")
91
+ # Each serving artifact: an explicit path is resolved against the repo ROOT; None routes through
92
+ # the bootstrap resolver (dev source -> HuggingFace), so a clean wheel install fetches them.
93
+ tokenizer = _resolve(tokenizer) if tokenizer else bootstrap.artifact_path("parser_tokenizer")
94
+ label_vocab = _resolve(label_vocab) if label_vocab else bootstrap.artifact_path("parser_label_vocab")
95
+ fst_tag_vocab = _resolve(fst_tag_vocab) if fst_tag_vocab else bootstrap.artifact_path("fst_tag_vocab")
96
+ parser_char_vocab = _resolve(parser_char_vocab) if parser_char_vocab else bootstrap.artifact_path("char_vocab")
97
+ parser_ckpt = _resolve(parser_ckpt) if parser_ckpt else bootstrap.artifact_path("parser")
98
+ analyze_ckpt = _resolve(analyze_ckpt) if analyze_ckpt else bootstrap.artifact_path("analyze")
99
+ analyze_char_vocab = (_resolve(analyze_char_vocab) if analyze_char_vocab
100
+ else bootstrap.artifact_path("analyze_charvocab"))
101
+ # compressed_table/fst_cache stay opt-in: pipeline passes the resolved fst_table; a direct
102
+ # NeuralParser() with neither falls back to the full cache (data/fst_cache_v1.jsonl).
103
+ compressed_table, fst_cache = _resolve(compressed_table), _resolve(fst_cache)
104
+ self.tok = BPETokenizer.load(tokenizer)
105
+ self.vocab = LabelVocab.load(label_vocab)
106
+ self.tag_vocab = FstTagVocab.load(fst_tag_vocab)
107
+ self.char_vocab = CharVocab.load(parser_char_vocab)
108
+ self.model = _load_checkpoint(parser_ckpt, self.vocab, self.tok, self.device)
109
+ self.model.eval() # inference-only consumer: disable dropout etc. (we decode Feats from
110
+ # out["feats_per_key"] directly, so we do NOT rely on the eval-gated out["feats"] override)
111
+ assert (self.model.feats_per_key_head is not None
112
+ and self.model._feats_per_key_flat_lookup is not None), \
113
+ "NeuralParser decodes Feats from the per-key head; this checkpoint lacks it"
114
+ self.mux, self.mux_cv, self.thr = load_analyze_mux(
115
+ analyze_ckpt, char_vocab_path=analyze_char_vocab, device=self.device,
116
+ tag_vocab=self.tag_vocab)
117
+ # Prefer the compressed table; fall back to the full FST cache if the table isn't installed.
118
+ lexicon_path = compressed_table if (compressed_table and os.path.exists(compressed_table)) else fst_cache
119
+ self.lexicon = FstCache.load(lexicon_path)
120
+ self.fst_max_tags = fst_max_tags
121
+ self.compound = bool(getattr(self.model, "compound_enabled", True))
122
+ self.max_form_len = getattr(self.model, "max_form_len", 64)
123
+
124
+ @torch.no_grad()
125
+ def parse(self, text: str) -> list[list[dict]]:
126
+ sents_tokens = _segment_and_tokenize(text)
127
+ if not sents_tokens:
128
+ return []
129
+ forms = sorted({t for s in sents_tokens for t in s})
130
+ neural = build_neural_cache(self.mux, self.mux_cv, forms, tag_vocab=self.tag_vocab,
131
+ threshold=self.thr, top_n=self.fst_max_tags, device=self.device)
132
+ hybrid, _ = build_hybrid_channel(self.lexicon, neural, forms)
133
+ placeholders = [_placeholder(t) for t in sents_tokens]
134
+ col = SilverCollator(self.tok, self.vocab, fst_cache=hybrid, fst_tag_vocab=self.tag_vocab,
135
+ fst_max_tags=self.fst_max_tags,
136
+ char_vocab=self.char_vocab if self.compound else None,
137
+ max_form_len=self.max_form_len, compound_enabled=self.compound)
138
+ batch = col(placeholders)
139
+ if batch["ids"].shape[0] == 0:
140
+ return []
141
+ batch = {k: (v.to(self.device) if torch.is_tensor(v) else v) for k, v in batch.items()}
142
+ out = self.model(batch["ids"], batch["mask"], fst_tags=batch.get("fst_tags"),
143
+ form_chars=batch.get("form_chars"), form_char_mask=batch.get("form_char_mask"))
144
+ fs = batch["first_subw"]
145
+ valid = fs.clone(); valid[:, 0] = True
146
+ NEG = torch.finfo(out["arc"].dtype).min
147
+ arc_pred = out["arc"].masked_fill(~valid.unsqueeze(1), NEG).argmax(-1)
148
+ upos_pred = out["upos"].argmax(-1)
149
+ lemma_pred = out["lemma"].argmax(-1)
150
+ # Decode Feats from the per-key head (out["feats_per_key"]), NOT the legacy flat out["feats"].
151
+ # Each lookup row maps a key's value index → flat label_vocab id (-1 = <EMPTY> → no bit set).
152
+ lookup = self.model._feats_per_key_flat_lookup # (n_keys, max_V), -1 padded
153
+ feat_flat_ids = torch.stack(
154
+ [lookup[k][kl.argmax(-1)] for k, kl in enumerate(out["feats_per_key"])],
155
+ dim=-1) # (B, T, n_keys)
156
+ B, T, _, R = out["rel"].shape
157
+ gh = arc_pred.unsqueeze(-1).unsqueeze(-1).expand(B, T, 1, R)
158
+ rel_pred = out["rel"].gather(2, gh).squeeze(2).argmax(-1)
159
+
160
+ kept = batch["kept_indices"].tolist()
161
+ results: list[list[dict]] = []
162
+ for b, src_idx in enumerate(kept):
163
+ word_first = fs[b].nonzero(as_tuple=True)[0].tolist()
164
+ src_tokens = placeholders[src_idx].tokens
165
+ toks: list[dict] = []
166
+ for wi, (pos, src) in enumerate(zip(word_first, src_tokens), start=1):
167
+ upos = self.vocab.id_to_upos.get(upos_pred[b, pos].item(), "X")
168
+ if upos.startswith("<"):
169
+ upos = "X"
170
+ eid = lemma_pred[b, pos].item()
171
+ script = self.vocab.edit_scripts[eid] if 0 <= eid < len(self.vocab.edit_scripts) else (0, "")
172
+ lemma = src.form.lower() if script[0] == -1 else apply_edit_script(src.form, script)
173
+ feats: dict[str, str] = {}
174
+ for fid in feat_flat_ids[b, pos].tolist():
175
+ if fid < 0:
176
+ continue
177
+ kv = self.vocab.id_to_feat.get(fid, "")
178
+ if kv and not kv.startswith("<") and "=" in kv:
179
+ k, v = kv.split("=", 1)
180
+ feats[k] = v
181
+ hs = arc_pred[b, pos].item()
182
+ head = 0 if hs == 0 else (word_first.index(hs) + 1 if hs in word_first else 0)
183
+ rel = self.vocab.id_to_deprel.get(rel_pred[b, pos].item(), "dep")
184
+ rel = ("dep" if rel.startswith("<") else rel).split(":")[0]
185
+ toks.append({"id": wi, "form": src.form, "lemma": lemma, "upos": upos,
186
+ "head": head, "deprel": rel, "feats": feats})
187
+ results.append(toks)
188
+ return results
juurio/parser/silver.py ADDED
@@ -0,0 +1,252 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Silver-corpus generation for the compact Finnish UD parser.
2
+
3
+ Turns raw Finnish text into UD records via a UD parser pipeline, then caches the records as
4
+ JSONL on disk. The cached records serve as training labels for the parser.
5
+
6
+ **Schema** — one JSON object per line, one line per sentence:
7
+
8
+ {
9
+ "text": "Mies osti talon.",
10
+ "tokens": [
11
+ {"form": "Mies", "lemma": "mies", "upos": "NOUN",
12
+ "feats": "Case=Nom|Number=Sing", "head": 2, "deprel": "nsubj"},
13
+ ...
14
+ ]
15
+ }
16
+
17
+ Design notes:
18
+ - Argmax-only labels (no top-K logits): the parser's public API exposes only argmax
19
+ predictions, so the silver corpus stores argmax labels.
20
+ - MWT spans are dropped — we keep one record per `Word` (per-component lines), aligning with
21
+ the rule-based tokenizer. Sentences with MWT get parsed as their component forms; MWT
22
+ detection happens at *inference* time in the parser.
23
+ - JSONL is append-safe. `count_cached(path)` tells the driver how many sentences to skip in
24
+ the source corpus so a crashed run can resume without re-parsing.
25
+ """
26
+ from __future__ import annotations
27
+
28
+ import json
29
+ import os
30
+ from dataclasses import dataclass, asdict
31
+ from typing import Iterable, Iterator
32
+
33
+
34
+ @dataclass
35
+ class SilverToken:
36
+ """One token's argmax silver labels — surface form, lemma, UPOS, feats, head index, deprel.
37
+
38
+ `feats_loss_masked`: True when the cleanup pass could not produce a
39
+ high-confidence Feats label for this token. The collator emits a
40
+ `feats_token_mask` based on this flag so the Feats loss skips these
41
+ tokens. Defaults to False so older silver JSONL files load unchanged.
42
+ """
43
+ form: str
44
+ lemma: str
45
+ upos: str
46
+ feats: str # "" when Stanza returns None
47
+ head: int # 0 = UD root
48
+ deprel: str
49
+ feats_loss_masked: bool = False
50
+
51
+
52
+ @dataclass
53
+ class SilverSentence:
54
+ """One sentence's silver record.
55
+
56
+ `teacher_tokens`: optional parallel list aligned 1:1 with `tokens`, carrying
57
+ auxiliary per-token training targets. When present, the collator emits
58
+ per-token `teacher_*` tensors and a per-sentence `kd_apply=True`. Defaults
59
+ to None so existing call sites are unaffected (and serving never sets it).
60
+ """
61
+ text: str
62
+ tokens: list[SilverToken]
63
+ teacher_tokens: list | None = None
64
+
65
+
66
+ # ----------------------------- doc → silver ------------------------------------------------------
67
+ def doc_to_silver(doc) -> Iterator[SilverSentence]:
68
+ """Convert a Stanza `Document` (or duck-typed equivalent) to `SilverSentence` records.
69
+
70
+ `doc.sentences` is iterated; for each sentence, `sentence.words` (the per-Word view, not
71
+ `tokens`) is used so MWT components show up as separate `SilverToken`s and the head indices
72
+ stay in Word-space."""
73
+ for sent in doc.sentences:
74
+ out_tokens: list[SilverToken] = []
75
+ for w in sent.words:
76
+ out_tokens.append(SilverToken(
77
+ form=w.text,
78
+ lemma=w.lemma if w.lemma is not None else w.text,
79
+ upos=w.upos if w.upos is not None else "X",
80
+ feats=w.feats if w.feats is not None else "",
81
+ head=int(w.head) if w.head is not None else 0,
82
+ deprel=w.deprel if w.deprel is not None else "dep",
83
+ ))
84
+ yield SilverSentence(text=sent.text or "", tokens=out_tokens)
85
+
86
+
87
+ # ----------------------------- JSONL I/O ---------------------------------------------------------
88
+ def write_jsonl(records: Iterable[SilverSentence], path: str, mode: str = "a") -> int:
89
+ """Write `records` as one JSON object per line. Returns the number of records written.
90
+
91
+ `mode="a"` is append — resume-safe. `mode="w"` truncates first."""
92
+ if mode not in ("w", "a"):
93
+ raise ValueError(f"mode must be 'w' or 'a', got {mode!r}")
94
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
95
+ n = 0
96
+ with open(path, mode, encoding="utf-8") as fh:
97
+ for s in records:
98
+ fh.write(json.dumps(asdict(s), ensure_ascii=False))
99
+ fh.write("\n")
100
+ n += 1
101
+ return n
102
+
103
+
104
+ _SILVER_TOKEN_FIELDS = frozenset(SilverToken.__dataclass_fields__.keys())
105
+
106
+
107
+ def read_jsonl(path: str) -> Iterator[SilverSentence]:
108
+ """Stream `SilverSentence` records from a JSONL file.
109
+
110
+ Unknown token fields are silently ignored — keeps the reader
111
+ forward-compatible with cleanup-pass extensions (e.g. a
112
+ `feats_loss_masked` flag; future passes may add more).
113
+ """
114
+ with open(path, encoding="utf-8") as fh:
115
+ for ln in fh:
116
+ ln = ln.strip()
117
+ if not ln:
118
+ continue
119
+ obj = json.loads(ln)
120
+ yield SilverSentence(
121
+ text=obj["text"],
122
+ tokens=[
123
+ SilverToken(**{k: v for k, v in t.items()
124
+ if k in _SILVER_TOKEN_FIELDS})
125
+ for t in obj["tokens"]
126
+ ],
127
+ )
128
+
129
+
130
+ def count_cached(path: str) -> int:
131
+ """Return the number of records already cached at `path`. 0 if the file doesn't exist."""
132
+ if not os.path.exists(path):
133
+ return 0
134
+ n = 0
135
+ with open(path, encoding="utf-8") as fh:
136
+ for ln in fh:
137
+ if ln.strip():
138
+ n += 1
139
+ return n
140
+
141
+
142
+ # ----------------------------- driver: parse a corpus --------------------------------------------
143
+ def parse_corpus(
144
+ nlp,
145
+ corpus: Iterable[str],
146
+ out_path: str,
147
+ *,
148
+ batch_size: int = 32,
149
+ flush_every: int = 200,
150
+ skip: int = 0,
151
+ ) -> int:
152
+ """Run `nlp` over `corpus` and stream silver records to `out_path`.
153
+
154
+ - `nlp` is a configured Stanza pipeline (caller controls GPU, processors, etc.).
155
+ - `batch_size`: Stanza is faster with batches; we feed it `batch_size` sentences at a time.
156
+ - `flush_every`: how often to flush the output file so a kill -9 loses at most this many records.
157
+ - `skip`: how many input sentences to drop from the head — for resume; pass `count_cached(out_path)`.
158
+
159
+ Returns the number of sentences newly written to `out_path`."""
160
+ mode = "a" if skip > 0 or os.path.exists(out_path) else "w"
161
+ os.makedirs(os.path.dirname(os.path.abspath(out_path)) or ".", exist_ok=True)
162
+ written = 0
163
+ buf: list[str] = []
164
+ with open(out_path, mode, encoding="utf-8") as fh:
165
+ seen = 0
166
+ for text in corpus:
167
+ seen += 1
168
+ if seen <= skip:
169
+ continue
170
+ buf.append(text)
171
+ if len(buf) >= batch_size:
172
+ written += _flush_batch(nlp, buf, fh)
173
+ buf.clear()
174
+ if written % flush_every == 0:
175
+ fh.flush()
176
+ if buf:
177
+ written += _flush_batch(nlp, buf, fh)
178
+ fh.flush()
179
+ return written
180
+
181
+
182
+ def _flush_batch(nlp, texts: list[str], fh) -> int:
183
+ """Parse a batch and dump silver records to the file handle. Returns count written."""
184
+ # Stanza accepts a list[str] in newer versions for batched parsing; concatenated string with
185
+ # blank-line separators is the wire-compatible fallback. We try list[str] first.
186
+ try:
187
+ docs = nlp.bulk_process(texts)
188
+ except (AttributeError, TypeError):
189
+ # older Stanza: parse joined string, take whole doc; works since corpus chunks are
190
+ # roughly sentence-sized
191
+ docs = [nlp(t) for t in texts]
192
+ n = 0
193
+ for doc in docs:
194
+ for s in doc_to_silver(doc):
195
+ fh.write(json.dumps(asdict(s), ensure_ascii=False))
196
+ fh.write("\n")
197
+ n += 1
198
+ return n
199
+
200
+
201
+ # ----------------------------- CLI: smoke-test on 10K sentences ----------------------------------
202
+ def _iter_corpus(path: str, max_sentences: int) -> Iterator[str]:
203
+ """Stream sentence-sized chunks from a JSONL corpus. Same chunking as `tokenizer.py`."""
204
+ n = 0
205
+ with open(path, encoding="utf-8") as fh:
206
+ for ln in fh:
207
+ ln = ln.strip()
208
+ if not ln:
209
+ continue
210
+ obj = json.loads(ln)
211
+ text = obj.get("text", "")
212
+ for chunk in text.split("\n"):
213
+ chunk = chunk.strip()
214
+ if chunk:
215
+ yield chunk
216
+ n += 1
217
+ if n >= max_sentences:
218
+ return
219
+
220
+
221
+ if __name__ == "__main__":
222
+ import argparse
223
+ import time
224
+
225
+ ap = argparse.ArgumentParser(description="Silver-parse Finnish text with Stanza → JSONL.")
226
+ ap.add_argument("--corpus", required=True,
227
+ help="input jsonl with a 'text' field (one document per line)")
228
+ ap.add_argument("--out", default="checkpoints/silver_smoke_10k.jsonl")
229
+ ap.add_argument("--n", type=int, default=10_000, help="number of input chunks to parse")
230
+ ap.add_argument("--batch", type=int, default=32)
231
+ ap.add_argument("--gpu", action="store_true")
232
+ args = ap.parse_args()
233
+
234
+ import stanza
235
+ t0 = time.time()
236
+ print(f"[silver] loading Stanza fi pipeline (gpu={args.gpu})")
237
+ nlp = stanza.Pipeline(
238
+ "fi", processors="tokenize,mwt,pos,lemma,depparse",
239
+ use_gpu=args.gpu, verbose=False, download_method=None,
240
+ )
241
+
242
+ skip = count_cached(args.out)
243
+ if skip:
244
+ print(f"[silver] resuming — {skip} records already cached at {args.out}")
245
+ n_written = parse_corpus(
246
+ nlp, _iter_corpus(args.corpus, args.n), args.out,
247
+ batch_size=args.batch, skip=skip,
248
+ )
249
+ dt = time.time() - t0
250
+ print(f"[silver] wrote {n_written} new records in {dt:.1f}s "
251
+ f"({n_written/max(dt,1e-6):.1f} sent/s)")
252
+ print(f"[silver] total cached: {count_cached(args.out)} → {args.out}")
juurio/parser/simple_tokenizer.py ADDED
@@ -0,0 +1,205 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Coarse rule-based Finnish tokenizer.
2
+
3
+ A lightweight, dependency-free tokenizer that gets close to UD-TDT's gold tokenization. The
4
+ `boundary_f1` helper measures token-boundary F1 against gold so callers can decide whether the
5
+ rule-based output is good enough or whether a learned tokenizer is needed.
6
+
7
+ Tokenization rules (intentionally minimal):
8
+ - Split on whitespace runs.
9
+ - Punctuation characters at the *edges* of a whitespace-delimited chunk become their own tokens.
10
+ - Internal punctuation between alphanumerics (`H&M`, `1990-luvulla`, `a@b.fi`) stays inside the
11
+ token — this captures most Finnish hyphenated compounds and loan brand names.
12
+
13
+ CLI:
14
+ python -m juurio.parser.simple_tokenizer probe # F1 vs UD-TDT-dev
15
+ """
16
+ from __future__ import annotations
17
+
18
+ import re
19
+
20
+ _WS = re.compile(r"\s+")
21
+ # Punctuation we'd like to split off at the edges. Everything else stays internal.
22
+ _EDGE_PUNCT = '.,;:!?"\'()[]{}«»‹›„""\'\'…'
23
+
24
+ # Common Finnish abbreviations whose trailing period is PART of the token — UD-TDT keeps `n.`, `esim.`
25
+ # etc. as single tokens. Stored lowercased without the period. Intentionally NOT exhaustive: the
26
+ # ambiguous number/ordinal (`1990.` = year+'.' vs ordinal) and emoticon cases are left to the edge
27
+ # splitter — that irreducible tail is what the module docstring flags for a learned tokenizer.
28
+ _ABBREV = {
29
+ "n", "nn", "mm", "esim", "ns", "nk", "jne", "yms", "ym", "tms", "vrt", "ent",
30
+ "os", "ko", "nro", "mrd", "milj", "tri", "prof", "huom", "ks", "oik",
31
+ }
32
+
33
+
34
+ def _keeps_trailing_period(core: str) -> bool:
35
+ """True if a period following `core` should stay attached: a known abbreviation, or a single
36
+ uppercase initial (`J.`, `P.`, `C.`). Numbers/ordinals are deliberately excluded (ambiguous with
37
+ sentence-final periods)."""
38
+ if not core:
39
+ return False
40
+ if core.lower() in _ABBREV:
41
+ return True
42
+ return len(core) == 1 and core.isalpha() and core.isupper()
43
+
44
+
45
+ def _strip_edges(chunk: str) -> list[str]:
46
+ """Pull leading and trailing punctuation off a whitespace-delimited chunk, recursively, while
47
+ leaving alphanumeric-internal punctuation alone. `"Hei!"` -> [`"`, `Hei`, `!`, `"`]. An
48
+ abbreviation/initial period (`esim.`, `J.`) is kept inside the token rather than split off."""
49
+ out: list[str] = []
50
+ leading: list[str] = []
51
+ trailing: list[str] = []
52
+ while chunk and chunk[0] in _EDGE_PUNCT:
53
+ leading.append(chunk[0])
54
+ chunk = chunk[1:]
55
+ while chunk and chunk[-1] in _EDGE_PUNCT:
56
+ if chunk[-1] == "." and _keeps_trailing_period(chunk[:-1]):
57
+ break # period belongs to the abbreviation / initial — keep it inside the token
58
+ trailing.append(chunk[-1])
59
+ chunk = chunk[:-1]
60
+ out.extend(leading)
61
+ if chunk:
62
+ out.append(chunk)
63
+ out.extend(reversed(trailing))
64
+ return out
65
+
66
+
67
+ def simple_tokenize(text: str) -> list[str]:
68
+ """Tokenize `text` into a list of surface tokens. Minimal, rule-based — see module docstring."""
69
+ if not text or not text.strip():
70
+ return []
71
+ out: list[str] = []
72
+ for chunk in _WS.split(text.strip()):
73
+ if chunk:
74
+ out.extend(_strip_edges(chunk))
75
+ return out
76
+
77
+
78
+ # ----------------------------- boundary F1 evaluation --------------------------------------------
79
+ def _read_text_and_tokens(path: str) -> list[tuple[str, list[str]]]:
80
+ """Yield (raw_text, gold_tokens) pairs from a CoNLL-U file.
81
+
82
+ `# text =` comments carry the raw sentence; the per-token form columns carry the gold
83
+ tokenization. We skip MWT span lines (the per-component lines are what align with the raw
84
+ text) and empty-node lines."""
85
+ out: list[tuple[str, list[str]]] = []
86
+ text = None
87
+ tokens: list[str] = []
88
+ with open(path, encoding="utf-8") as fh:
89
+ for ln in fh:
90
+ ln = ln.rstrip("\n")
91
+ if not ln:
92
+ if text is not None and tokens:
93
+ out.append((text, tokens))
94
+ text = None
95
+ tokens = []
96
+ continue
97
+ if ln.startswith("# text = "):
98
+ text = ln[len("# text = "):]
99
+ continue
100
+ if ln.startswith("#"):
101
+ continue
102
+ cols = ln.split("\t")
103
+ if len(cols) < 2:
104
+ continue
105
+ tid = cols[0]
106
+ if "-" in tid or "." in tid: # skip MWT span / empty-node lines
107
+ continue
108
+ tokens.append(cols[1])
109
+ if text is not None and tokens:
110
+ out.append((text, tokens))
111
+ return out
112
+
113
+
114
+ def _token_spans(text: str, tokens: list[str]) -> list[tuple[int, int]] | None:
115
+ """Greedy left-to-right alignment of `tokens` against `text`, returning their `(start, end)`
116
+ char spans. Returns `None` if alignment fails (a token does not appear at the expected
117
+ position) — those sentences are dropped from the F1 calculation as alignment failures.
118
+
119
+ Both gold and predicted token lists go through this routine so the F1 is computed over the
120
+ same alignment convention."""
121
+ out = []
122
+ i = 0
123
+ n = len(text)
124
+ for t in tokens:
125
+ # skip leading whitespace
126
+ while i < n and text[i].isspace():
127
+ i += 1
128
+ # exact match at i?
129
+ if text[i:i + len(t)] == t:
130
+ out.append((i, i + len(t)))
131
+ i += len(t)
132
+ else:
133
+ return None
134
+ return out
135
+
136
+
137
+ def boundary_f1(path: str) -> dict[str, float]:
138
+ """Compute token-boundary F1 of `simple_tokenize` against UD-TDT gold on the given file.
139
+
140
+ Boundaries are `(start, end)` char-span sets over each sentence. Sentences whose gold or
141
+ predicted alignment fails are tallied separately."""
142
+ pairs = _read_text_and_tokens(path)
143
+ tp = fp = fn = 0
144
+ n_sent = 0
145
+ n_perfect = 0
146
+ n_align_fail_gold = 0
147
+ n_align_fail_pred = 0
148
+ for text, gold_tokens in pairs:
149
+ gold_spans = _token_spans(text, gold_tokens)
150
+ if gold_spans is None:
151
+ n_align_fail_gold += 1
152
+ continue
153
+ pred_tokens = simple_tokenize(text)
154
+ pred_spans = _token_spans(text, pred_tokens)
155
+ if pred_spans is None:
156
+ n_align_fail_pred += 1
157
+ continue
158
+ n_sent += 1
159
+ g, p = set(gold_spans), set(pred_spans)
160
+ tp += len(g & p)
161
+ fp += len(p - g)
162
+ fn += len(g - p)
163
+ if g == p:
164
+ n_perfect += 1
165
+ prec = tp / (tp + fp) if (tp + fp) else 0.0
166
+ rec = tp / (tp + fn) if (tp + fn) else 0.0
167
+ f1 = 2 * prec * rec / (prec + rec) if (prec + rec) else 0.0
168
+ return {
169
+ "sentences_eval": n_sent,
170
+ "sentences_perfect": n_perfect,
171
+ "sentence_perfect_rate": n_perfect / n_sent if n_sent else 0.0,
172
+ "precision": prec,
173
+ "recall": rec,
174
+ "f1": f1,
175
+ "alignment_failed_gold": n_align_fail_gold,
176
+ "alignment_failed_pred": n_align_fail_pred,
177
+ }
178
+
179
+
180
+ def probe(paths: dict[str, str]) -> None:
181
+ print("Ablation §0.3 — rule-based tokenizer boundary F1 vs UD-TDT")
182
+ print("=" * 78)
183
+ for name, path in paths.items():
184
+ r = boundary_f1(path)
185
+ print(f" {name:6s} sentences-eval {r['sentences_eval']:5d} "
186
+ f"P {r['precision']:.4f} R {r['recall']:.4f} F1 {r['f1']:.4f} "
187
+ f"perfect-sent {r['sentence_perfect_rate']*100:5.2f}% "
188
+ f"align-fail gold/pred {r['alignment_failed_gold']}/{r['alignment_failed_pred']}")
189
+
190
+
191
+ if __name__ == "__main__":
192
+ import os
193
+ import sys
194
+
195
+ repo = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
196
+ paths = {
197
+ "train": os.path.join(repo, "data", "ud", "fi_tdt-ud-train.conllu"),
198
+ "dev": os.path.join(repo, "data", "ud", "fi_tdt-ud-dev.conllu"),
199
+ "test": os.path.join(repo, "data", "ud", "fi_tdt-ud-test.conllu"),
200
+ }
201
+ cmd = sys.argv[1] if len(sys.argv) > 1 else "probe"
202
+ if cmd == "probe":
203
+ probe(paths)
204
+ else:
205
+ raise SystemExit(f"unknown cmd {cmd!r}; usage: python -m juurio.parser.simple_tokenizer probe")
juurio/parser/tokenizer.py ADDED
@@ -0,0 +1,172 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """BPE tokenizer for the compact Finnish UD parser.
2
+
3
+ Wraps HuggingFace `tokenizers` to give a thin, fixed API the parser depends on:
4
+
5
+ - `BPETokenizer.train(corpus_iter, vocab_size=8000)` trains from scratch on an iterable of
6
+ Finnish text strings. Reserves PAD_ID=0, ROOT_ID=1, UNK,
7
+ BOS, EOS as the first five vocab slots so the model's `PAD_ID` / `ROOT_ID` constants stay
8
+ consistent across training and inference.
9
+ - `encode(text) -> list[int]` produces a plain id list.
10
+ - `encode_with_root(text)` prepends ROOT_ID — the convention for parser input
11
+ (the UD virtual root is the head=0 attachment target).
12
+ - `decode(ids) -> str` round-trips back to a surface string (BPE joiner handled).
13
+ - `save(path) / load(path)` go through a single `tokenizer.json` (HF format).
14
+
15
+ The locked special-token order is:
16
+
17
+ 0: <pad> (PAD_ID, also `padding_idx` of the embedding)
18
+ 1: <root> (ROOT_ID, the virtual UD root token)
19
+ 2: <unk>
20
+ 3: <bos>
21
+ 4: <eos>
22
+
23
+ These ids are baked into both this module and `juurio/parser/model.py`. Changing them is a
24
+ *data-pipeline contract break* — only do it together with a matching model-checkpoint retrain.
25
+ """
26
+ from __future__ import annotations
27
+
28
+ import os
29
+ from typing import Iterable
30
+
31
+ from tokenizers import Tokenizer, decoders, models, pre_tokenizers, trainers
32
+ from tokenizers.processors import TemplateProcessing
33
+
34
+ from .model import PAD_ID, ROOT_ID
35
+
36
+ # Locked special-token order. See module docstring.
37
+ PAD_TOKEN = "<pad>"
38
+ ROOT_TOKEN = "<root>"
39
+ UNK_TOKEN = "<unk>"
40
+ BOS_TOKEN = "<bos>"
41
+ EOS_TOKEN = "<eos>"
42
+ _SPECIAL_TOKENS = [PAD_TOKEN, ROOT_TOKEN, UNK_TOKEN, BOS_TOKEN, EOS_TOKEN]
43
+
44
+
45
+ class BPETokenizer:
46
+ """Thin wrapper around an HF BPE `Tokenizer` with our locked special-token contract."""
47
+
48
+ def __init__(self, tok: Tokenizer):
49
+ self._tok = tok
50
+ # sanity: special tokens must be in the locked positions
51
+ assert tok.token_to_id(PAD_TOKEN) == PAD_ID, "PAD token at wrong id"
52
+ assert tok.token_to_id(ROOT_TOKEN) == ROOT_ID, "ROOT token at wrong id"
53
+
54
+ # ----------------------------- properties --------------------------------------------------
55
+ @property
56
+ def pad_id(self) -> int:
57
+ return PAD_ID
58
+
59
+ @property
60
+ def root_id(self) -> int:
61
+ return ROOT_ID
62
+
63
+ @property
64
+ def unk_id(self) -> int:
65
+ return self._tok.token_to_id(UNK_TOKEN)
66
+
67
+ @property
68
+ def vocab_size(self) -> int:
69
+ return self._tok.get_vocab_size()
70
+
71
+ # ----------------------------- encode / decode ---------------------------------------------
72
+ def encode(self, text: str) -> list[int]:
73
+ """Encode `text` to a list of integer token ids. Does NOT prepend ROOT; see
74
+ `encode_with_root` for parser input."""
75
+ return self._tok.encode(text).ids
76
+
77
+ def encode_with_root(self, text: str) -> list[int]:
78
+ """Encode `text` and prepend ROOT_ID — the canonical parser input format."""
79
+ return [ROOT_ID, *self.encode(text)]
80
+
81
+ def decode(self, ids: list[int]) -> str:
82
+ """Decode an id sequence back to text. Special tokens are skipped."""
83
+ return self._tok.decode(ids, skip_special_tokens=True)
84
+
85
+ # ----------------------------- save / load -------------------------------------------------
86
+ def save(self, path: str) -> None:
87
+ os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
88
+ self._tok.save(path)
89
+
90
+ @classmethod
91
+ def load(cls, path: str) -> "BPETokenizer":
92
+ return cls(Tokenizer.from_file(path))
93
+
94
+ # ----------------------------- training ----------------------------------------------------
95
+ @classmethod
96
+ def train(cls, corpus: Iterable[str], vocab_size: int = 8000) -> "BPETokenizer":
97
+ """Train a fresh byte-level BPE tokenizer over `corpus`.
98
+
99
+ Byte-level BPE is the safe default for Finnish: it never produces `<unk>` on UTF-8 input,
100
+ handles the full inflectional zoo without language-specific pre-tokenization, and matches
101
+ what modern subword models use internally.
102
+ """
103
+ tok = Tokenizer(models.BPE(unk_token=UNK_TOKEN))
104
+ tok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
105
+ tok.decoder = decoders.ByteLevel()
106
+ trainer = trainers.BpeTrainer(
107
+ vocab_size=vocab_size,
108
+ special_tokens=_SPECIAL_TOKENS,
109
+ initial_alphabet=pre_tokenizers.ByteLevel.alphabet(),
110
+ show_progress=False,
111
+ )
112
+ tok.train_from_iterator(list(corpus), trainer=trainer)
113
+ # Post-processor: don't add BOS/EOS automatically — the parser data pipeline owns ROOT
114
+ # insertion via `encode_with_root`. Keep encode() plain for full caller control.
115
+ tok.post_processor = TemplateProcessing(
116
+ single="$A",
117
+ special_tokens=[(BOS_TOKEN, tok.token_to_id(BOS_TOKEN)),
118
+ (EOS_TOKEN, tok.token_to_id(EOS_TOKEN))],
119
+ )
120
+ return cls(tok)
121
+
122
+
123
+ # ----------------------------- CLI: train on a text corpus ---------------------------------------
124
+ def _iter_corpus(path: str, max_sentences: int | None = None) -> Iterable[str]:
125
+ """Yield text strings from a JSONL corpus file. Each line is `{"text": "..."}`."""
126
+ import json
127
+ n = 0
128
+ with open(path, encoding="utf-8") as fh:
129
+ for ln in fh:
130
+ ln = ln.strip()
131
+ if not ln:
132
+ continue
133
+ obj = json.loads(ln)
134
+ text = obj.get("text", "")
135
+ if not text:
136
+ continue
137
+ # Corpus records are full documents; split on newlines to feed BPE roughly
138
+ # sentence-sized chunks (BPE doesn't care about sentence boundaries but this caps
139
+ # memory).
140
+ for chunk in text.split("\n"):
141
+ chunk = chunk.strip()
142
+ if chunk:
143
+ yield chunk
144
+ n += 1
145
+ if max_sentences is not None and n >= max_sentences:
146
+ return
147
+
148
+
149
+ if __name__ == "__main__":
150
+ import argparse
151
+ import time
152
+
153
+ ap = argparse.ArgumentParser(description="Train an 8K BPE tokenizer for the tiny UD parser.")
154
+ ap.add_argument("--corpus", required=True,
155
+ help="input jsonl with a 'text' field (one document per line)")
156
+ ap.add_argument("--out", default="checkpoints/parser_tokenizer.json")
157
+ ap.add_argument("--vocab", type=int, default=8000)
158
+ ap.add_argument("--max-sentences", type=int, default=2_000_000,
159
+ help="cap on training chunks; 2 M is plenty for an 8K BPE.")
160
+ args = ap.parse_args()
161
+
162
+ t0 = time.time()
163
+ print(f"[BPE] streaming up to {args.max_sentences:,} chunks from {args.corpus}")
164
+ tok = BPETokenizer.train(_iter_corpus(args.corpus, args.max_sentences), vocab_size=args.vocab)
165
+ tok.save(args.out)
166
+ print(f"[BPE] vocab={tok.vocab_size} PAD={tok.pad_id} ROOT={tok.root_id} UNK={tok.unk_id}")
167
+ print(f"[BPE] saved → {args.out} ({time.time()-t0:.1f}s)")
168
+ demo = "Mies ostaa talon Helsingistä."
169
+ ids = tok.encode_with_root(demo)
170
+ print(f"[BPE] demo: {demo!r}")
171
+ print(f" ids: {ids}")
172
+ print(f" decoded: {tok.decode(ids)!r}")
juurio/pipeline.py ADDED
@@ -0,0 +1,705 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Raw-text product pipeline: string -> parse -> scope/abstain -> grammatical transform.
2
+
3
+ This is the product surface over the core. It wires three pieces assigned to different components:
4
+ - STRUCTURE / scope -> a UD dependency parser (default: the torch-only Giella/Stanza-free
5
+ end-to-end neural parser `parser.neural_adapter.NeuralParser`; `StanzaParser` is an opt-in
6
+ backend), behind a swappable adapter
7
+ - idiosyncrasy-free morphology -> the net+lexicon backend: lexicon-first, neural realizers second
8
+ (`algebra.verb_cluster` + the nominal/verbform/finiteverb nets)
9
+ - open-vocabulary morphology -> the tiny lemma-anchored transformer (neural fallback)
10
+
11
+ `transform(op, text, lemma=None)` returns a `Result` with the output, an explicit abstain
12
+ reason, a trace, and warnings — abstention is a first-class result, not a silent failure.
13
+
14
+ SCOPE CONTRACT (what `transform` promises). It operates on a SINGLE finite main clause. It returns
15
+ a `Result` whose `ok` is True only when it produced a transformation; otherwise `ok=False` and
16
+ `reason` is one of `ABSTAIN_REASONS` (see `REASONS` for descriptions). In scope: one finite
17
+ Act/Ind main clause within `algebra.in_scope` limits (≈≤12 words, one finite verb). Out of scope
18
+ (→ abstain, never a guess): subordinate/coordinated clauses, already-negated clauses, multi-verb
19
+ fragments, op not applicable to the clause's base state, and OOV open-class words neither the lexicon
20
+ nor the neural realizer can inflect (`oov_unrealizable`). Neural realizations that DID succeed are
21
+ surfaced in `trace["neural_unverified"]` (deprecated alias `trace["neural_oov"]`) + a warning (model
22
+ guess, unverified against the lexicon — a provenance flag, not a claim the lemma is out-of-vocabulary).
23
+
24
+ Residual risks this module guards:
25
+ 1. lemma-error propagation: the parser lemmatizer can mangle a loan stem (`striimaa`->`striä`),
26
+ which corrupts the class anchor. Mitigations: a `lemma=` override, and a confidence
27
+ warning when the parser lemma diverges early from the surface form.
28
+ 2. parser subject-attachment differences.
29
+
30
+ When the optional Stanza backend is used, Stanza runs on CPU here; the transformer still uses the GPU.
31
+ """
32
+ from __future__ import annotations
33
+
34
+ import os
35
+ import warnings
36
+ from collections.abc import Mapping
37
+ from dataclasses import dataclass, field
38
+ from types import MappingProxyType
39
+
40
+ from . import algebra
41
+ from . import features
42
+ from . import bootstrap
43
+
44
+ # stock nn.TransformerDecoder emits a prototype "nested tensors" UserWarning on every forward;
45
+ # it is benign and noise on the CLI. Silence just that one message.
46
+ warnings.filterwarnings("ignore", message=".*nested tensor.*", category=UserWarning)
47
+
48
+
49
+ # ----------------------------- parser adapter -----------------------------
50
+ class ParserAdapter:
51
+ """Abstraction over a UD parser so the backend is swappable. A backend must turn raw text
52
+ into a list of sentences, each a list of token dicts in our format:
53
+ {id, form, lemma, upos, head(int, 0=root), deprel(base label), feats(dict)}."""
54
+
55
+ def parse(self, text: str) -> list[list[dict]]:
56
+ raise NotImplementedError
57
+
58
+
59
+ class StanzaParser(ParserAdapter):
60
+ _nlp = None # class-level cache: the model is expensive to load
61
+
62
+ def __init__(self, device: str = "cpu"):
63
+ self.device = device
64
+
65
+ def _pipe(self):
66
+ if StanzaParser._nlp is None:
67
+ import stanza
68
+ StanzaParser._nlp = stanza.Pipeline(
69
+ "fi", processors="tokenize,pos,lemma,depparse",
70
+ download_method=None, verbose=False, device=self.device)
71
+ return StanzaParser._nlp
72
+
73
+ @staticmethod
74
+ def _feats(s: str | None) -> dict:
75
+ return dict(kv.split("=", 1) for kv in s.split("|")) if s and s != "_" else {}
76
+
77
+ def parse(self, text: str) -> list[list[dict]]:
78
+ doc = self._pipe()(text)
79
+ out = []
80
+ for sent in doc.sentences:
81
+ out.append([{
82
+ "id": w.id, "form": w.text, "lemma": w.lemma or w.text, "upos": w.upos,
83
+ "head": w.head, "deprel": (w.deprel or "").split(":")[0],
84
+ "feats": self._feats(w.feats),
85
+ } for w in sent.words])
86
+ return out
87
+
88
+
89
+ # ----------------------------- result type + contract -----------------------------
90
+ # The full, documented set of `Result.reason` codes (the abstention contract).
91
+ REASONS = {
92
+ "ok": "a transformation was produced (Result.ok is True)",
93
+ "parse_empty": "the parser returned no usable sentence at sent_index",
94
+ "no_clause": "no finite Act/Ind main clause found (subordinate-only, fragment, "
95
+ "or already-negated clause that would double-negate)",
96
+ "out_of_scope": "clause exceeds scope limits (too long / >1 finite verb / coordination)",
97
+ "op_not_applicable": "the op does not apply to this clause's base state "
98
+ "(e.g. past on an already-past clause, passive on a copula clause)",
99
+ "oov_unrealizable": "an open-class word needed inflecting but NEITHER the FST nor the neural "
100
+ "realizer could produce the target form (explicit unsupported morphology)",
101
+ "oov_unverified": "the neural realizer produced a candidate for an OOV open-class word but it "
102
+ "failed validation (stem-inconsistent or near-tie) and was suppressed, not emitted",
103
+ "conflicting_features": "inflect() received two valid but contradictory inputs (e.g. spec "
104
+ "'mon gen' with number='sing', proper=True with a degree, or like= alongside "
105
+ "an explicit case/number/degree)",
106
+ "unanalyzable_model": "inflect(like=…) could not analyze the model word into a nominal "
107
+ "case/number cell to copy",
108
+ }
109
+ ABSTAIN_REASONS = frozenset(REASONS) - {"ok"}
110
+
111
+
112
+ @dataclass
113
+ class Result:
114
+ text: str | None # transformed sentence, or None if abstained/failed
115
+ ok: bool # True iff a transformation was produced
116
+ reason: str # one of REASONS (ok => ABSTAIN_REASONS otherwise)
117
+ trace: dict = field(default_factory=dict) # op, root_verb, parser_lemma, lemma_source,
118
+ warnings: list[str] = field(default_factory=list) # neural_unverified / oov_unrealizable detail, lemma warnings
119
+
120
+
121
+ @dataclass(frozen=True)
122
+ class TokenAnalysis:
123
+ """One analyzed token from `analyze()` — the parser's native fields as a stable, typed record
124
+ (so the public contract is these named fields, not "whatever dict the parser emitted today").
125
+ `head` is the 1-based id of the syntactic head (0 = root); `feats` is the UD morphology mapping
126
+ (e.g. {'Case': 'Gen', 'Number': 'Plur'}), stored read-only so the record is truly immutable."""
127
+ id: int
128
+ form: str
129
+ lemma: str
130
+ upos: str
131
+ head: int
132
+ deprel: str
133
+ feats: Mapping[str, str]
134
+
135
+ def __post_init__(self):
136
+ # frozen dataclass: bypass the immutability guard to wrap feats read-only (a defensive copy)
137
+ object.__setattr__(self, "feats", MappingProxyType(dict(self.feats)))
138
+
139
+ def asdict(self) -> dict:
140
+ return {"id": self.id, "form": self.form, "lemma": self.lemma, "upos": self.upos,
141
+ "head": self.head, "deprel": self.deprel, "feats": dict(self.feats)}
142
+
143
+
144
+ # ----------------------------- engine -----------------------------
145
+ _QUOTES = "\"'«»“”„‘’*•"
146
+ _DASHES = "-–—"
147
+
148
+
149
+ def _strip_dialogue_prefix(text: str):
150
+ """Stanza over-segments a leading dialogue marker (— or quote) into its OWN one-token sentence,
151
+ so `sents[0]` is just the punctuation and the real clause is lost -> false `no_clause`. Strip a
152
+ leading run of quotes/dialogue-dashes (a dash only when followed by space, so `-5` is safe) and
153
+ re-prepend it to the output. Returns (prefix, core)."""
154
+ i = 0
155
+ while i < len(text):
156
+ c = text[i]
157
+ if c.isspace() or c in _QUOTES:
158
+ i += 1
159
+ elif c in _DASHES and i + 1 < len(text) and text[i + 1].isspace():
160
+ i += 1
161
+ else:
162
+ break
163
+ return text[:i], text[i:]
164
+
165
+
166
+ def _common_prefix(a: str, b: str) -> int:
167
+ n = 0
168
+ for x, y in zip(a, b):
169
+ if x != y:
170
+ break
171
+ n += 1
172
+ return n
173
+
174
+
175
+ def _lemma_confidence_warning(form: str, lemma: str) -> str | None:
176
+ """Flag a likely-bad parser lemma (the dominant residual risk), as a WARNING (override via
177
+ lemma=), not a hard abstain. Giella-FREE check (the served analysis path no longer calls the FST):
178
+ a correct lemma shares a long prefix with the form (striimaa/striimata=7), a mangled one diverges
179
+ early (striimaa/striä=4). This covers all words, since the neural parser supplies the lemma. An
180
+ earlier FST-round-trip branch (confirm the lemma is among `uralicApi.analyze(form)`'s readings) was
181
+ removed to keep generation+analysis Giella-free at serving."""
182
+ f, l = form.lower(), lemma.lower()
183
+ cp = _common_prefix(f, l)
184
+ if cp < 0.6 * len(f):
185
+ return f"low-confidence lemma '{lemma}' for form '{form}' (prefix {cp}/{len(f)}); pass lemma= to override"
186
+ return None
187
+
188
+
189
+ def _optional_artifact(key: str) -> str | None:
190
+ """Resolve serving artifact `key` via bootstrap, or None if it can't be fetched. Used for the OOV
191
+ realizers, which degrade gracefully (abstain) rather than hard-fail when their artifact is absent.
192
+ A sha256 mismatch (corrupt/tampered bytes) is NOT graceful — it always propagates."""
193
+ try:
194
+ return bootstrap.artifact_path(key)
195
+ except bootstrap.ArtifactIntegrityError:
196
+ raise
197
+ except Exception:
198
+ return None
199
+
200
+
201
+ def _default_parser(**kwargs) -> ParserAdapter:
202
+ """Default SERVING parser: the Giella/Stanza-free end-to-end neural parser
203
+ (`juurio.parser.neural_adapter.NeuralParser`).
204
+ Prefers the 7.3 MB compressed FST-tag table over the 67.9 MB full cache when present. `StanzaParser`
205
+ remains available as an opt-in backend (`Transformer(parser=StanzaParser())`) for benchmarking, but
206
+ is no longer a runtime dependency. Extra kwargs pass through to `NeuralParser` (e.g. `device=`)."""
207
+ from .parser.neural_adapter import NeuralParser
208
+ # Prefer the 7.3 MB compressed table; resolve it via bootstrap (dev source -> HuggingFace) so a clean
209
+ # wheel install fetches it. NeuralParser falls back to the full 67.9 MB FST cache only if unset.
210
+ # Guard the resolve so an explicit caller override (or None) doesn't trigger a fetch (offline-safe).
211
+ if "compressed_table" not in kwargs:
212
+ kwargs["compressed_table"] = bootstrap.artifact_path("fst_table")
213
+ return NeuralParser(**kwargs)
214
+
215
+
216
+ def _oov_abstain_result(trace: dict) -> Result:
217
+ """Abstain Result for a token the morphology engine could not realize, with the reason taken from
218
+ the algebra OOV trace flags: an unverified neural candidate (suppressed) outranks nothing-producible.
219
+ Shared by the nominal realization paths (`realize_np`, `inflect`) so the mapping can't drift."""
220
+ if algebra._OOV_UNVERIFIED:
221
+ trace["oov_unverified"] = list(algebra._OOV_UNVERIFIED)
222
+ return Result(None, False, "oov_unverified", trace, [])
223
+ trace["oov_unrealizable"] = list(algebra._OOV_UNREALIZABLE)
224
+ return Result(None, False, "oov_unrealizable", trace, [])
225
+
226
+
227
+ def _record_neural_unverified(trace: dict, items: list) -> list:
228
+ """Record net-realized forms in the trace and return them. These are model GUESSES the FST could not
229
+ verify at runtime — a PROVENANCE flag (neural- vs lexicon-served), NOT a vocabulary claim: the lemma
230
+ may be perfectly known (the shipped lexicon is exception-only, so common regular words like `talo`
231
+ are net-served). Hence the label is "neural realization (unverified by FST)", not "neural OOV…".
232
+ `neural_unverified` is the canonical trace key; `neural_oov` is a DEPRECATED back-compat alias kept
233
+ for one release cycle (prefer `neural_unverified`)."""
234
+ trace["neural_unverified"] = items
235
+ trace["neural_oov"] = items
236
+ return items
237
+
238
+
239
+ def _ok_with_oov_warnings(text: str, trace: dict) -> Result:
240
+ """Success Result, surfacing any neural realizations on this call as unverified warnings."""
241
+ warnings_out: list[str] = []
242
+ if algebra._NEURAL_OOV:
243
+ _record_neural_unverified(trace, list(algebra._NEURAL_OOV))
244
+ warnings_out.append(f"neural realization (unverified by FST): {algebra._NEURAL_OOV}")
245
+ return Result(text, True, "ok", trace, warnings_out)
246
+
247
+
248
+ class Transformer:
249
+ """Loads the parser + the lemma-anchored model once, then serves `transform`."""
250
+
251
+ def __init__(self, parser: ParserAdapter | None = None, ckpt: str | None = None,
252
+ noun_realizer: bool = True):
253
+ try:
254
+ self.parser = parser or _default_parser()
255
+ finally:
256
+ # Building the parser imports the internal `juurio.analyze` subpackage, which shadows the
257
+ # public `analyze`/`lemmatize` convenience funcs as package attrs. Restore them in `finally`
258
+ # so the module-level API stays callable even if construction below fails (see __init__.py).
259
+ from . import _restore_public_names
260
+ _restore_public_names()
261
+ self._bundle = None # unused; kept for back-compat with the hybrid_transform signature
262
+ # OOV verb/noun realizers (Giella-free): the FST inflects known words; when it can't (loan/OOV
263
+ # word) these neural nets fill in. All optional + guarded — an absent checkpoint just means
264
+ # that OOV class abstains. `ckpt` is accepted for API compat but no longer loads a verb model.
265
+ # Reset the module globals first so a later construction (e.g. a missing checkpoint, or
266
+ # noun_realizer=False) cannot inherit a previous Transformer's realizers in a long-lived process.
267
+ algebra._NOUN_REALIZER = None
268
+ algebra._VERBFORM_REALIZER = None
269
+ algebra._FINITEVERB_BUNDLE = None
270
+ # Generation backend: Giella-free lexicon-first + net-second. Reset the globals so a prior
271
+ # construction never leaks its backend state into a new one.
272
+ algebra._NETLEX_BACKEND = True
273
+ algebra._GENLEX = None
274
+ # OOV finite/non-finite verb realizers: these are independent of `noun_realizer` (which only
275
+ # gates the nominal net) — they are the Giella-free OOV verb path and load whenever their
276
+ # checkpoints are present. finiteverb covers finite cells (incl. imperative + connegative)
277
+ # and verbform the past participles; both feed `algebra._netlex_verb_cluster`/`_neural_imprt`.
278
+ # Resolve each realizer artifact through bootstrap (dev source -> HuggingFace). These OOV
279
+ # realizers stay OPTIONAL: if an artifact can't be resolved (e.g. offline with an empty cache)
280
+ # we leave the global None and the engine abstains on OOV — same graceful degradation as a
281
+ # missing checkpoint before. The netlex backend's wordlist (below) is the hard require.
282
+ from . import verbforms
283
+ vf = _optional_artifact("verbform")
284
+ if vf:
285
+ algebra._VERBFORM_REALIZER = verbforms.make_verbform_realizer(ckpt=vf)
286
+ from . import finiteverb
287
+ fv = _optional_artifact("finiteverb")
288
+ if fv:
289
+ algebra._FINITEVERB_BUNDLE = finiteverb.load(ckpt=fv)
290
+ if noun_realizer:
291
+ from . import realizer
292
+ nr = _optional_artifact("realizer")
293
+ if nr:
294
+ algebra._NOUN_REALIZER = realizer.make_noun_realizer(ckpt=nr)
295
+ # Net+lexicon backend reuses the realizer bundles loaded above + the static exception table.
296
+ from . import genlex
297
+ algebra._GENLEX = genlex.GenerationLexicon.load()
298
+
299
+ def transform(self, op: str, text: str, lemma: str | None = None,
300
+ sent_index: int = 0) -> Result:
301
+ if op not in algebra.ALL_OPS and algebra.parse_nominal_bundle(op) is None:
302
+ raise ValueError(f"unknown op {op!r}; known atoms: {sorted(algebra.ALL_OPS)}; or a "
303
+ f"'+'-composed nominal bundle e.g. 'plural+poss_1sg+clitic_kin'")
304
+ prefix, core = _strip_dialogue_prefix(text) # un-split leading dialogue dash/quote
305
+ sents = self.parser.parse(core)
306
+ if not sents or sent_index >= len(sents):
307
+ return Result(None, False, "parse_empty")
308
+ sent = sents[sent_index]
309
+
310
+ # lemma/UPOS hint repair: if the caller named the main verb but the parser mis-tagged the clause
311
+ # root as a non-verb (the ambiguous-loan-verb failure, e.g. 'striimaa'->NOUN), re-tag it so
312
+ # clause detection succeeds. Conservative: only when there is no finite verb + the hint stem-
313
+ # matches (see algebra.repair_root_verb_with_lemma).
314
+ lemma_repaired = bool(lemma) and algebra.repair_root_verb_with_lemma(sent, lemma)
315
+
316
+ verb, _ = algebra._main_verb(sent)
317
+ trace = {"op": op, "root_verb": verb["form"] if verb else None,
318
+ "parser_lemma": verb["lemma"] if verb else None}
319
+ if lemma_repaired:
320
+ trace["lemma_repair"] = True
321
+ warnings: list[str] = []
322
+
323
+ # scope / abstention is a first-class result. The imperative family has its own scope
324
+ # (subjectless / 2nd-person source — see algebra.imperative_in_scope); every other op uses
325
+ # the general finite Act/Ind clause gate.
326
+ if op in algebra.IMPERATIVE_OPS:
327
+ if not algebra.imperative_in_scope(sent):
328
+ trace["scope"] = "out_of_scope"
329
+ return Result(None, False, "out_of_scope", trace, warnings)
330
+ else:
331
+ if algebra.clause(sent) is None:
332
+ return Result(None, False, "no_clause", trace, warnings)
333
+ if not algebra.in_scope(sent):
334
+ trace["scope"] = "out_of_scope"
335
+ return Result(None, False, "out_of_scope", trace, warnings)
336
+
337
+ # lemma override (the parser lemma is used otherwise) + confidence guard
338
+ if lemma is not None and verb is not None:
339
+ trace["lemma_source"] = "override"
340
+ verb["lemma"] = lemma
341
+ elif verb is not None:
342
+ trace["lemma_source"] = "parser"
343
+ w = _lemma_confidence_warning(verb["form"], verb["lemma"])
344
+ if w:
345
+ warnings.append(w)
346
+
347
+ # transform: deterministic structure + FST verb, neural fallback for OOV (hybrid)
348
+ out = algebra.hybrid_transform(sent, op, self._bundle)
349
+ if out is None:
350
+ if algebra._OOV_UNVERIFIED: # neural produced a candidate but we suppressed it
351
+ trace["oov_unverified"] = list(algebra._OOV_UNVERIFIED)
352
+ return Result(None, False, "oov_unverified", trace, warnings)
353
+ if algebra._OOV_UNREALIZABLE: # FST+neural both failed on an open-class word
354
+ trace["oov_unrealizable"] = list(algebra._OOV_UNREALIZABLE)
355
+ return Result(None, False, "oov_unrealizable", trace, warnings)
356
+ return Result(None, False, "op_not_applicable", trace, warnings)
357
+ # surface any neural realizations: these are model guesses the FST could not verify at
358
+ # runtime (provenance, not vocabulary), so the caller can weigh them appropriately
359
+ if algebra._NEURAL_OOV:
360
+ _record_neural_unverified(trace, list(algebra._NEURAL_OOV))
361
+ warnings.append(f"neural realization (unverified by FST): {algebra._NEURAL_OOV}")
362
+ return Result(prefix + out, True, "ok", trace, warnings)
363
+
364
+ def realize_np(self, np: str, case: str = "Nom", number: str = "Sing",
365
+ px: str = "None", clitic: str | None = None) -> Result:
366
+ """PHRASE-LEVEL nominal morphology utility — NOT a clause transform (case is syntactically
367
+ governed). Inflect an NP (a bare lemma or a short `adj* noun` phrase) into
368
+ the target Case/Number, optional possessive (`px`) + enclitic, with agreement on modifiers
369
+ and per-token FST->neural fallback. Surface order stem+case+number+Px+clitic. Returns a `Result`:
370
+ `ok` with the realized NP, or an abstain reason (`parse_empty`/`no_clause` for no nominal
371
+ head, `oov_unverified`/`oov_unrealizable` for a head the morphology engine cannot produce)."""
372
+ algebra._NEURAL_OOV.clear()
373
+ algebra._OOV_UNREALIZABLE.clear()
374
+ algebra._OOV_UNVERIFIED.clear()
375
+ sents = self.parser.parse(np)
376
+ toks = sents[0] if sents else []
377
+ head = next((t for t in reversed(toks) if t["upos"] in ("NOUN", "PROPN")), None)
378
+ if head is None and len(toks) == 1 and toks[0]["upos"] not in ("PUNCT", "SYM"):
379
+ # A single bare word handed to realize_np (a NOMINAL utility) is a noun, even when the
380
+ # context-free parser reads it as a homograph verb (suo=suoda, tuli=past of tulla). Force the
381
+ # NOUN reading so the morphology runs instead of abstaining `no_clause` on a one-word NP.
382
+ toks = [{**toks[0], "upos": "NOUN", "lemma": toks[0]["form"]}]
383
+ head = toks[0]
384
+ trace = {"util": "realize_np", "case": case, "number": number, "px": px, "clitic": clitic,
385
+ "head": head["form"] if head else None}
386
+ if head is None:
387
+ return Result(None, False, "no_clause", trace, []) # no nominal head to inflect
388
+ out = []
389
+ # The nominal HEAD is a citation lemma per realize_np's contract, so its SURFACE already IS its
390
+ # lemma — use `t["form"]`, NOT the parser's re-lemmatisation: the neural lemmatiser mis-maps
391
+ # lexicalised nominals to verb infinitives (valtuutettu->valtuuttaa, reititin->reitittää,
392
+ # vietti->viettää) or the wrong stem (neitsyt->neitsy), which would inflect into a non-word.
393
+ # Agreeing MODIFIERS keep the parser lemma: the parser carries their Degree in `feats` and supplies
394
+ # the base adjective, so a comparative input ("isompi talo") realizes from lemma `iso`+Degree=Cmp
395
+ # (-> isommassa); using the surface `isompi` as the lemma would double-apply degree and abstain.
396
+ for t in toks:
397
+ if t["id"] == head["id"]:
398
+ f = algebra.realize_np_form(t["form"], t["upos"], case, number, px, clitic, t["form"])
399
+ elif t["upos"] in ("ADJ", "NUM") or t["deprel"] in algebra.AGREE:
400
+ f = algebra._realize_form(t["lemma"], t["upos"],
401
+ {**t["feats"], "Case": case, "Number": number}, t["form"])
402
+ else:
403
+ f = t["form"] # leave non-agreeing tokens as-is
404
+ if f is None:
405
+ return _oov_abstain_result(trace)
406
+ out.append(f)
407
+ return _ok_with_oov_warnings(" ".join(out), trace)
408
+
409
+ def realize_verbform(self, verb: str, form: str = "va", case: str = "Nom",
410
+ number: str = "Sing", px: str = "None", lemma: str | None = None) -> Result:
411
+ """PHRASE-LEVEL non-finite verb morphology — NOT a clause transform. Inflect a VERB into a
412
+ participle (`va`/`nut`/`tava`/`tu`/`ma`, which inflect in case+number) or an infinitive
413
+ (`inf2`/`inf2_pass`/`inf3`, case-marked: menemässä/menemästä/menemään/menemällä/menemättä).
414
+ Pass `px` (a possessive-suffix value) for the temporal/possessive construction: form='tu',
415
+ case='Par', px='3' -> 'kuultuaan' ('after hearing'). `verb` may be a bare lemma or a short
416
+ clause (the main verb is used); pass `lemma=` to override the parser for a mis-lemmatized loan.
417
+ FST-backed (known verbs); for an OOV verb the verbform realizer serves the bare participle and,
418
+ with `px`, the noun realizer attaches case+Px (participles inflect like nominals) — both neural
419
+ stages flagged unverified. Absent the realizers, abstains. `form` is one of `algebra.VERBFORMS`."""
420
+ algebra._NEURAL_OOV.clear()
421
+ algebra._OOV_UNREALIZABLE.clear()
422
+ algebra._OOV_UNVERIFIED.clear()
423
+ if form not in algebra.VERBFORMS:
424
+ raise ValueError(f"unknown verb form {form!r}; known: {algebra.VERBFORMS}")
425
+ if lemma is None:
426
+ sents = self.parser.parse(verb)
427
+ toks = sents[0] if sents else []
428
+ vtok = next((t for t in toks if t["upos"] == "VERB"), None)
429
+ if vtok is not None:
430
+ lemma = vtok["lemma"]
431
+ elif len(verb.split()) == 1:
432
+ # single bare word with no VERB reading: this is a verb utility, so trust the caller's
433
+ # word as the verb lemma. The parser is context-free and re-lemmatises homographs to the
434
+ # wrong POS ('haukkua' -> noun 'haukku' -> garbage participle 'haukkuvussu'); honouring
435
+ # the raw infinitive keeps it correct ('haukkuvassa').
436
+ lemma = verb.strip()
437
+ else:
438
+ # genuine multiword clause with no VERB token: best-effort last-token fallback.
439
+ lemma = toks[-1]["lemma"] if toks else verb
440
+ trace = {"util": "realize_verbform", "form": form, "case": case, "number": number,
441
+ "px": px, "lemma": lemma}
442
+ # 1. FST (known verbs), Px-aware (kuulla,tu,Par,3 -> kuultuansa). In netlex mode this also serves
443
+ # the verbform NET, flagging via algebra._NEURAL_OOV; propagate that so a net-served verbform is
444
+ # honestly marked `neural_unverified` (mirrors the nominal paths), not passed off as FST-verified.
445
+ f = algebra.realize_verbform_form(lemma, form, case, number, px=px)
446
+ if f is not None:
447
+ if algebra._NEURAL_OOV:
448
+ _record_neural_unverified(trace, list(algebra._NEURAL_OOV))
449
+ return Result(f, True, "ok", trace,
450
+ [f"neural realization (unverified by FST): {list(algebra._NEURAL_OOV)}"])
451
+ return Result(f, True, "ok", trace, [])
452
+ # 2. OOV. Without a possessive suffix this is the single-stage verbform realizer.
453
+ if px == "None":
454
+ if algebra._VERBFORM_REALIZER is not None:
455
+ vf, conf = algebra._VERBFORM_REALIZER(lemma, form, case, number)
456
+ if vf and algebra._stem_consistent(lemma, vf) and conf >= algebra._OOV_CONF_MIN:
457
+ _record_neural_unverified(trace, [f"{lemma}->{vf} ({form})"])
458
+ return Result(vf, True, "ok", trace,
459
+ [f"neural realization (unverified by FST): ['{lemma}->{vf} ({form})']"])
460
+ if vf:
461
+ trace["oov_unverified"] = [f"{lemma}->{vf} ({form}) conf={conf:.2f}"]
462
+ return Result(None, False, "oov_unverified", trace, [])
463
+ trace["oov_unrealizable"] = [f"{lemma} (VERB) {form}/{case}/{number}"]
464
+ return Result(None, False, "oov_unrealizable", trace, [])
465
+ # 2b. OOV + Px: compose. Bare participle stem from the verbform realizer, then case+Px from the
466
+ # noun realizer (participles inflect like nominals). Two neural hops -> flagged unverified.
467
+ if algebra._VERBFORM_REALIZER is not None and algebra._NOUN_REALIZER is not None:
468
+ stem, conf = algebra._VERBFORM_REALIZER(lemma, form, "Nom", "Sing")
469
+ if stem and algebra._stem_consistent(lemma, stem) and conf >= algebra._OOV_CONF_MIN:
470
+ out = algebra._realize_form(stem, "NOUN",
471
+ {"Case": case, "Number": number, "Px": px}, stem)
472
+ if out is not None:
473
+ _record_neural_unverified(trace, [f"{lemma}->{stem}->{out} ({form}+Px{px})"])
474
+ return Result(out, True, "ok", trace,
475
+ [f"neural realization (unverified by FST): ['{lemma}->{out} ({form}+Px)']"])
476
+ # _realize_form recorded why the Px step failed (unverified vs unrealizable)
477
+ if algebra._OOV_UNVERIFIED:
478
+ trace["oov_unverified"] = list(algebra._OOV_UNVERIFIED)
479
+ return Result(None, False, "oov_unverified", trace, [])
480
+ trace["oov_unrealizable"] = [f"{lemma} (VERB) {form}/{case}/{number}/Px{px}"]
481
+ return Result(None, False, "oov_unrealizable", trace, [])
482
+
483
+ # --------------------------- friendly nominal inflection -----------------------------------
484
+ def inflect(self, word: str, case=features.UNSET, number=features.UNSET,
485
+ degree=features.UNSET, *, px=features.UNSET, clitic: str | None = None,
486
+ proper: bool | None = None, pos: str | None = None,
487
+ like: str | None = None) -> Result:
488
+ """Inflect a single nominal (or short `adj* noun` phrase) into a target case/number, the
489
+ ergonomic front door over `realize_np`. Accepts human-readable aliases:
490
+
491
+ inflect("talo", "gen").text # 'talon' (English/Finnish abbrev)
492
+ inflect("talo", "mon gen").text # 'talojen' (combined spec, taivutin-style)
493
+ inflect("talo", case="inessive", number="plural").text # 'taloissa'
494
+ inflect("talo", px="1sg").text # 'taloni'
495
+ inflect("kaunis", "gen", degree="comp").text # 'kauniimman' (-> ADJ)
496
+ inflect("Meri", "gen", proper=True).text # proper-noun reading (vs 'meri' -> 'meren')
497
+ inflect("makaronilaatikko", like="talojen").text # copy 'talojen's cell -> plural gen
498
+
499
+ Aliases are normalized in `juurio.features`; an UNKNOWN alias raises ValueError (a typo
500
+ fails fast). Contradictory-but-valid inputs ABSTAIN with reason `conflicting_features`
501
+ (e.g. spec 'mon gen' + number='sing'; proper=True + a degree; like= + an explicit
502
+ case/number/degree/px — like= copies the model's whole cell, possessive included).
503
+ like= transfers Case/Number/Px always; a model's Degree (the model was an adjective) transfers
504
+ ONLY to an explicit adjective target (`pos='adjective'`) — on any other target it is inapplicable
505
+ and dropped (recorded in `trace['dropped_features']`), never realized as a non-word.
506
+ POS: explicit `pos` ('noun'/'proper'/'adjective') wins; `proper` is sugar; otherwise inferred
507
+ (degree => ADJ, leading capital => PROPN, else NOUN). A single token is realized parser-free
508
+ (so `proper`/`pos` are honored); a whitespace phrase delegates to `realize_np` (which rejects
509
+ the single-word controls `proper`/`pos`/`degree`, but — like realize_np — still applies
510
+ `px`/`clitic` to the phrase head). `px`/`clitic` are rejected on a single ADJ target (only
511
+ nominal heads take them). An empty/whitespace `word` raises ValueError. Returns a `Result`."""
512
+ word = word.strip()
513
+ if not word:
514
+ raise ValueError("inflect() requires a non-empty word")
515
+ trace: dict = {"util": "inflect", "word": word}
516
+ # --- features: from a model word (like=) or from case/number/degree (with spec support) ---
517
+ if like is not None:
518
+ # like= copies the model word's WHOLE nominal cell (case/number/degree, any possessive, AND
519
+ # a trailing focus/interrogative enclitic), so an explicit case/number/degree/px/clitic
520
+ # alongside it is contradictory -> abstain.
521
+ if (case is not features.UNSET or number is not features.UNSET
522
+ or degree is not features.UNSET or px is not features.UNSET
523
+ or clitic is not None):
524
+ return Result(None, False, "conflicting_features",
525
+ {**trace, "like": like,
526
+ "error": "like= is exclusive with case/number/degree/px/clitic"}, [])
527
+ cell = self._analyze_nominal_cell(like)
528
+ if cell is None:
529
+ return Result(None, False, "unanalyzable_model", {**trace, "like": like}, [])
530
+ feats = {"Case": cell.get("Case", "Nom"), "Number": cell.get("Number", "Sing"),
531
+ "Degree": cell.get("Degree", "Pos")}
532
+ px_v = cell.get("Px", "None") # a possessive on the model (e.g. koiriemme -> '1Pl') carries over
533
+ clitic = cell.get("Clitic") # an enclitic on the model (makaronilaatikkoonko -> 'ko') carries over
534
+ trace["like"] = like
535
+ # like_cell mirrors the model's OWN cell: show Degree only if the model actually carried one
536
+ # (a noun model has no degree -> don't fabricate a phantom Degree=Pos in the trace).
537
+ trace["like_cell"] = {"Case": feats["Case"], "Number": feats["Number"],
538
+ **({"Degree": feats["Degree"]} if cell.get("Degree") else {}),
539
+ **({"Px": px_v} if px_v != "None" else {}),
540
+ **({"Clitic": clitic} if clitic else {})}
541
+ # A Degree copied from a like= model (i.e. the model word was an adjective) is only an
542
+ # APPLICABLE feature when the TARGET is itself an adjective — which the user states with
543
+ # pos='adjective'. Left in for any other target it would coerce the POS to ADJ (degree => ADJ
544
+ # in resolve_pos) and fabricate a non-word (talo + kauniimmassa -> 'taloimmassa'). So on a
545
+ # non-adjective target the inapplicable degree is dropped (recorded), never realized.
546
+ if feats["Degree"] != "Pos":
547
+ try:
548
+ target_is_adj = features.resolve_pos(word, pos=pos, proper=proper) == "ADJ"
549
+ except (features.FeatureConflict, ValueError):
550
+ target_is_adj = False # a real pos/proper conflict surfaces below; just don't keep degree
551
+ if not target_is_adj:
552
+ trace["dropped_features"] = ["Degree"]
553
+ feats["Degree"] = "Pos"
554
+ else:
555
+ try:
556
+ feats = features.resolve_features(
557
+ "nom" if case is features.UNSET else case, number, degree)
558
+ except features.FeatureConflict as e:
559
+ return Result(None, False, "conflicting_features", {**trace, "error": str(e)}, [])
560
+ # a plain ValueError (unknown alias) propagates: a typo fails fast
561
+ px_v = "None" if px is features.UNSET else features.normalize_px(px)
562
+ clitic = features.normalize_clitic(clitic) # 'ko'/'kö'/'kaan' -> canonical; unknown -> ValueError
563
+ trace.update(case=feats["Case"], number=feats["Number"], degree=feats["Degree"],
564
+ px=px_v, clitic=clitic)
565
+
566
+ # --- phrase vs single token ---------------------------------------------------------------
567
+ if len(word.split()) > 1:
568
+ if proper is not None or pos is not None or feats["Degree"] != "Pos":
569
+ return Result(None, False, "conflicting_features",
570
+ {**trace, "error": "proper=/pos=/degree are single-word only; a phrase "
571
+ "inflects its head with modifier agreement via realize_np"}, [])
572
+ res = self.realize_np(word, case=feats["Case"], number=feats["Number"],
573
+ px=px_v, clitic=clitic)
574
+ # realize_np returns its own trace; carry forward a recorded like= drop so the annotation
575
+ # isn't lost for phrase targets (single-token targets keep `trace` and so already show it).
576
+ if res.ok and "dropped_features" in trace:
577
+ res.trace.setdefault("dropped_features", trace["dropped_features"])
578
+ return res
579
+
580
+ # --- single token: resolve POS, then realize parser-free ----------------------------------
581
+ try:
582
+ upos = features.resolve_pos(word, pos=pos, proper=proper, degree=feats["Degree"])
583
+ except features.FeatureConflict as e:
584
+ return Result(None, False, "conflicting_features", {**trace, "error": str(e)}, [])
585
+ trace["upos"] = upos
586
+
587
+ algebra._NEURAL_OOV.clear()
588
+ algebra._OOV_UNREALIZABLE.clear()
589
+ algebra._OOV_UNVERIFIED.clear()
590
+ if upos == "ADJ":
591
+ if px_v != "None" or clitic is not None:
592
+ return Result(None, False, "conflicting_features",
593
+ {**trace, "error": "px/clitic are not supported on adjectives"}, [])
594
+ f = algebra._realize_form(word, "ADJ", {"Case": feats["Case"],
595
+ "Number": feats["Number"], "Degree": feats["Degree"]}, word)
596
+ else: # NOUN / PROPN
597
+ f = algebra.realize_np_form(word, upos, feats["Case"], feats["Number"], px_v, clitic, word)
598
+ if f is None:
599
+ return _oov_abstain_result(trace)
600
+ return _ok_with_oov_warnings(f, trace)
601
+
602
+ def _nominal_head(self, word: str) -> dict | None:
603
+ """Parse `word` and return its rightmost nominal head token (NOUN/PROPN/ADJ/PRON/NUM), or None."""
604
+ sents = self.parser.parse(word)
605
+ toks = sents[0] if sents else []
606
+ return next((tk for tk in reversed(toks)
607
+ if tk["upos"] in ("NOUN", "PROPN", "ADJ", "PRON", "NUM")), None)
608
+
609
+ @staticmethod
610
+ def _cell_from_head(head: dict) -> dict:
611
+ """Build the nominal {Case[,Number][,Degree][,Px]} cell from a parsed head token."""
612
+ feats = head.get("feats", {})
613
+ cell = {k: feats[k] for k in ("Case", "Number") if k in feats}
614
+ if head["upos"] == "ADJ" and "Degree" in feats:
615
+ cell["Degree"] = feats["Degree"]
616
+ # carry a possessive suffix too (UD splits it into Person[psor]/Number[psor]); a model word
617
+ # like 'koiriemme' is gen-plur-1Pl, so the whole cell — not just case/number — must copy over.
618
+ px = features.px_from_psor(feats.get("Person[psor]"), feats.get("Number[psor]"))
619
+ if px != "None":
620
+ cell["Px"] = px
621
+ return cell
622
+
623
+ def _enclitic_host_cell(self, stem: str) -> dict | None:
624
+ """Semantic gate for inflect-by-example enclitic peeling: return the stem's cell IFF the stem is
625
+ a GENUINE inflected nominal host, else None. The serving analyzer absorbs an enclitic into the
626
+ lemma (it emits no Clitic feature), so to copy '-ko' off a model word (makaronilaatikkoonko) we
627
+ peel it heuristically (features.split_enclitic) and verify the remainder here. Two checks reject
628
+ a lemma that merely ends in clitic letters: (1) a bare nominative singular with no possessive is
629
+ a lemma, not an inflected host (so 'kissa' from a spurious 'kissa|kin' split is refused — at the
630
+ cost of not copying a clitic off a Nom-Sing model, a documented limitation); (2) a round-trip —
631
+ the stem must regenerate from its own analysis (rejects hallucinated parses of fragments like
632
+ 'aurin' from 'aurinko'). The round-trip realize would dirty the OOV trace, so snapshot/restore."""
633
+ host = self._nominal_head(stem)
634
+ if host is None:
635
+ return None
636
+ cell = self._cell_from_head(host)
637
+ if (cell.get("Case", "Nom") == "Nom" and cell.get("Number", "Sing") == "Sing"
638
+ and "Px" not in cell):
639
+ return None
640
+ upos = host["upos"] if host["upos"] in ("NOUN", "PROPN") else "NOUN"
641
+ # the round-trip realize can append to all three OOV trace lists; snapshot/restore every one so
642
+ # the gate is trace-neutral (the phrase-target path never reaches inflect()'s own clear at L528).
643
+ saved = (list(algebra._NEURAL_OOV), list(algebra._OOV_UNVERIFIED), list(algebra._OOV_UNREALIZABLE))
644
+ try:
645
+ regen = algebra.realize_np_form(host["lemma"], upos, cell.get("Case", "Nom"),
646
+ cell.get("Number", "Sing"), cell.get("Px", "None"),
647
+ None, host["lemma"])
648
+ finally:
649
+ (algebra._NEURAL_OOV[:], algebra._OOV_UNVERIFIED[:],
650
+ algebra._OOV_UNREALIZABLE[:]) = saved
651
+ if regen is None or regen.lower() != stem.lower():
652
+ return None
653
+ return cell
654
+
655
+ def _analyze_nominal_cell(self, model: str) -> dict | None:
656
+ """Analyze a model word and return its nominal {Case[,Number][,Degree][,Px][,Clitic]} cell (for
657
+ inflect-by-example), or None if it has no nominal head to copy from. A trailing focus/
658
+ interrogative enclitic (-kO/-kin/-kAAn/-hAn/-pA) is peeled and copied as Clitic when the
659
+ remainder passes the `_enclitic_host_cell` gate (so `like=makaronilaatikkoonko` -> +ko)."""
660
+ stem, particle = features.split_enclitic(model)
661
+ if particle is not None and stem != model:
662
+ host_cell = self._enclitic_host_cell(stem)
663
+ if host_cell is not None:
664
+ return {**host_cell, "Clitic": particle}
665
+ head = self._nominal_head(model)
666
+ if head is None:
667
+ return None
668
+ return self._cell_from_head(head) or None
669
+
670
+ # --------------------------- analysis convenience -----------------------------------------
671
+ def analyze(self, text: str) -> list[list["TokenAnalysis"]]:
672
+ """Analyze raw text into sentences of `TokenAnalysis` (lemma, UPOS, head, deprel, UD feats).
673
+ The generation side's inverse — what taivutin cannot do. Reuses the serving parser."""
674
+ return [[TokenAnalysis(tk["id"], tk["form"], tk["lemma"], tk["upos"], tk["head"],
675
+ tk["deprel"], dict(tk.get("feats", {}))) for tk in sent]
676
+ for sent in self.parser.parse(text)]
677
+
678
+ def lemmatize(self, text: str) -> list[str]:
679
+ """Return the lemma of every token in `text`, flat across sentences."""
680
+ return [tk["lemma"] for sent in self.parser.parse(text) for tk in sent]
681
+
682
+
683
+ if __name__ == "__main__":
684
+ import sys
685
+ t = Transformer()
686
+ if len(sys.argv) > 2:
687
+ op, text = sys.argv[1], sys.argv[2]
688
+ lemma = sys.argv[3] if len(sys.argv) > 3 else None
689
+ r = t.transform(op, text, lemma=lemma)
690
+ print(r)
691
+ else:
692
+ # self-test: raw text in, grammatical transform out, incl. OOV loans + abstention
693
+ cases = [
694
+ ("passive", "Poika tubettaa videon.", None),
695
+ ("past", "Käyttäjä googlaa vastauksen.", None),
696
+ ("neg", "Nuori striimaa peliä.", None), # parser lemma may be bad
697
+ ("neg", "Nuori striimaa peliä.", "striimata"), # ...fixed by override
698
+ ("passive", "Kissa nukkui, koska se oli väsynyt.", None), # multi-clause -> abstain
699
+ ]
700
+ for op, text, lem in cases:
701
+ r = t.transform(op, text, lemma=lem)
702
+ tag = f"[{op}{'/lemma='+lem if lem else ''}]"
703
+ print(f"{tag:24s} {text}")
704
+ print(f" -> {r.text!r} ({r.reason})"
705
+ + (f" warn={r.warnings}" if r.warnings else ""))
juurio/realizer.py ADDED
@@ -0,0 +1,196 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """OOV noun/adjective realizer (inference).
2
+
3
+ A small PER-TOKEN realizer the rule engine calls when an open-class loan/OOV word has no lexicon
4
+ entry. Given `lemma + POS + Case + Number + Degree (+ Px)`, it emits the target surface form, so it
5
+ generalizes to unseen stems (e.g. `striimaaja -> striimaajat`) instead of memorizing lemmas.
6
+
7
+ Separate model + checkpoint from the sentence-transform flag model. Header is CATEGORICAL
8
+ (POS/Case/Number/Degree/Px), not binary flags. Wiring: `pipeline`/callers set
9
+ `algebra._NOUN_REALIZER = make_noun_realizer(...)`; `_realize_form` tries the lexicon first, this
10
+ second, else abstains.
11
+ """
12
+ from __future__ import annotations
13
+ import os
14
+
15
+ import torch
16
+ import torch.nn as nn
17
+
18
+ from .device import DEV
19
+ from . import bootstrap
20
+
21
+ CKPT = os.path.join(os.path.dirname(__file__), "..", "checkpoints", "realizer.pt")
22
+ PAD, BOS, EOS, UNK = 0, 1, 2, 3
23
+ SPECIAL = ["<pad>", "<bos>", "<eos>", "<unk>"]
24
+
25
+ # Fixed comprehensive cased char set: every char that can appear in a Finnish surface/lemma — incl.
26
+ # foreign loan letters (w/x/z/q/...), CAPITALS (PROPN), digits, hyphen/apostrophe.
27
+ # A pointer-copy head can only reproduce a char that has its own vocab id, so the vocab is fixed.
28
+ _LOWER = "abcdefghijklmnopqrstuvwxyzåäöšž"
29
+ _UPPER = "ABCDEFGHIJKLMNOPQRSTUVWXYZÅÄÖŠŽ"
30
+ _DIGITS = "0123456789"
31
+ _PUNCT = "-' "
32
+ FIXED_CHARS = _LOWER + _UPPER + _DIGITS + _PUNCT
33
+
34
+ # categorical feature space (the header). Cases = the UD case set.
35
+ POS_VALS = ["NOUN", "ADJ", "PROPN"]
36
+ CASE_VALS = ["Nom", "Gen", "Par", "Ess", "Tra", "Ine", "Ela", "Ill", "Ade", "Abl", "All",
37
+ "Abe", "Com", "Ins"]
38
+ NUM_VALS = ["Sing", "Plur"]
39
+ DEG_VALS = ["Pos", "Cmp", "Sup"]
40
+ # possessive suffix (omistusliite). 3rd person merges Sg/Pl (-nsa/-nsä/-Vn), so one "3" value.
41
+ # Px belongs IN the realizer header — it attaches after case/number and interacts with stem
42
+ # allomorphy (taloni vs talossani), so it is part of the inflected form, not a clitic.
43
+ PX_VALS = ["None", "1Sg", "2Sg", "3", "1Pl", "2Pl"]
44
+ FEAT_GROUPS = [("POS", POS_VALS), ("Case", CASE_VALS), ("Number", NUM_VALS),
45
+ ("Degree", DEG_VALS), ("Px", PX_VALS)]
46
+ FEAT_STOI = {f"{g}={v}": i for i, (g, v) in
47
+ enumerate((g, v) for g, vals in FEAT_GROUPS for v in vals)} # flat contiguous index
48
+ NFEAT_TOK = len(FEAT_STOI) # distinct categorical tokens
49
+ HLEN = len(FEAT_GROUPS) # header length (5 slots)
50
+
51
+
52
+ def case_lemma(pos, s):
53
+ """PROPN lemmas/forms are capitalized (Wagner, Åbo) so the copy head copies the capital position;
54
+ NOUN/ADJ/verbs stay as given (lowercase). Never lowercases internal capitals (McDonald)."""
55
+ return (s[:1].upper() + s[1:]) if pos == "PROPN" else s
56
+
57
+
58
+ def header_ids(pos, case, num, deg, px="None"):
59
+ """The categorical token-ids for a target slot (unknown values clamp to the default)."""
60
+ pos = pos if pos in POS_VALS else "NOUN"
61
+ case = case if case in CASE_VALS else "Nom"
62
+ num = num if num in NUM_VALS else "Sing"
63
+ deg = deg if deg in DEG_VALS else "Pos"
64
+ px = px if px in PX_VALS else "None"
65
+ return [FEAT_STOI[f"POS={pos}"], FEAT_STOI[f"Case={case}"], FEAT_STOI[f"Number={num}"],
66
+ FEAT_STOI[f"Degree={deg}"], FEAT_STOI[f"Px={px}"]]
67
+
68
+
69
+ def encode_chars(s, stoi):
70
+ """Encode a string to ids. New cased vocabs carry <unk>: map OOV→<unk> (never silently drop, the
71
+ bug behind `warettaa`→`arettaa`). Legacy 30-char checkpoints have no <unk>: preserve their original
72
+ drop-OOV behaviour so they load + decode unchanged."""
73
+ unk = stoi.get("<unk>")
74
+ if unk is None:
75
+ return [stoi[c] for c in s if c in stoi]
76
+ return [stoi.get(c, unk) for c in s]
77
+
78
+
79
+ # ----------------------------- model -----------------------------
80
+ class Realizer(nn.Module):
81
+ def __init__(self, vocab, d_model=128, nhead=4, layers=2, ff=512, maxlen=64):
82
+ super().__init__()
83
+ self.maxlen = maxlen
84
+ self.emb = nn.Embedding(vocab, d_model, padding_idx=PAD)
85
+ self.featemb = nn.Embedding(NFEAT_TOK, d_model)
86
+ self.pos = nn.Embedding(maxlen, d_model)
87
+ self.enc = nn.TransformerEncoder(
88
+ nn.TransformerEncoderLayer(d_model, nhead, ff, 0.1, batch_first=True), layers)
89
+ self.dec = nn.TransformerDecoder(
90
+ nn.TransformerDecoderLayer(d_model, nhead, ff, 0.1, batch_first=True), layers)
91
+ self.proj = nn.Linear(d_model, vocab)
92
+ self.proj.weight = self.emb.weight
93
+
94
+ def encode(self, src, header):
95
+ B, S = src.shape
96
+ H = header.size(1)
97
+ seq = torch.cat([self.featemb(header), self.emb(src)], 1)
98
+ seq = seq + self.pos(torch.arange(H + S, device=src.device))
99
+ mask = torch.cat([torch.zeros(B, H, dtype=torch.bool, device=src.device), src == PAD], 1)
100
+ return self.enc(seq, src_key_padding_mask=mask), mask
101
+
102
+ def _tgt_embed(self, ids):
103
+ T = ids.size(1)
104
+ return self.emb(ids) + self.pos(torch.arange(T, device=ids.device))
105
+
106
+ def forward(self, src, header, tgt_in):
107
+ mem, mask = self.encode(src, header)
108
+ T = tgt_in.size(1)
109
+ cmask = torch.triu(torch.ones(T, T, device=src.device, dtype=torch.bool), 1)
110
+ out = self.dec(self._tgt_embed(tgt_in), mem, tgt_mask=cmask,
111
+ tgt_key_padding_mask=(tgt_in == PAD), memory_key_padding_mask=mask)
112
+ return self.proj(out)
113
+
114
+
115
+ def _pad(seqs, n):
116
+ return torch.tensor([s + [PAD] * (n - len(s)) for s in seqs], dtype=torch.long)
117
+
118
+
119
+ @torch.no_grad()
120
+ def _decode(m, header, lemma, stoi, itos, maxlen=48, score=False):
121
+ """Greedy decode. If `score`, also return a confidence = min over steps of the top-1 softmax
122
+ margin (top1 - top2 prob) — a per-cell uncertainty signal used to suppress shaky OOV guesses."""
123
+ m.eval()
124
+ src = torch.tensor([encode_chars(lemma, stoi)], dtype=torch.long, device=DEV)
125
+ hdr = torch.tensor([header], dtype=torch.long, device=DEV)
126
+ mem, mask = m.encode(src, hdr)
127
+ ids = [BOS]
128
+ margin = 1.0
129
+ for _ in range(maxlen):
130
+ ti = torch.tensor([ids], dtype=torch.long, device=DEV)
131
+ T = ti.size(1)
132
+ cmask = torch.triu(torch.ones(T, T, device=DEV, dtype=torch.bool), 1)
133
+ out = m.dec(m._tgt_embed(ti), mem, tgt_mask=cmask, memory_key_padding_mask=mask)
134
+ logits = m.proj(out)[0, -1]
135
+ nxt = int(logits.argmax())
136
+ if score:
137
+ p = torch.softmax(logits.float(), -1)
138
+ top2 = torch.topk(p, 2).values
139
+ margin = min(margin, float(top2[0] - top2[1]))
140
+ if nxt == EOS:
141
+ break
142
+ ids.append(nxt)
143
+ form = "".join(itos[i] for i in ids[1:] if i in itos)
144
+ return (form, margin) if score else form
145
+
146
+
147
+ # ----------------------------- inference hook -----------------------------
148
+ def load(ckpt=None):
149
+ """Load realizer B. A copy-head checkpoint (pointer-generator — carries copy_q/copy_k/gate) is
150
+ reconstructed as a CopyRealizer so NOUN/ADJ/PROPN are served by the mixed gen+copy distribution; a
151
+ plain checkpoint stays a generator-only Realizer. Detected by weights, so old and new checkpoints
152
+ both load. `ckpt=None` resolves the shipped artifact via bootstrap."""
153
+ if ckpt is None:
154
+ ckpt = bootstrap.artifact_path("realizer")
155
+ ck = torch.load(ckpt, map_location=DEV, weights_only=False)
156
+ if "copy_q.weight" in ck["model"]:
157
+ from .copyrealizer import load as _copy_load
158
+ return _copy_load(ckpt)
159
+ m = Realizer(len(ck["stoi"]), ck["cfg"]["d_model"], 4, ck["cfg"]["layers"]).to(DEV)
160
+ m.load_state_dict(ck["model"]); m.eval()
161
+ return m, ck["stoi"], ck["itos"]
162
+
163
+
164
+ def make_noun_realizer(bundle=None, ckpt=None):
165
+ """Return fn(lemma, upos, feats, surface)->form for `algebra._NOUN_REALIZER` (None on failure).
166
+ Preserves the source word's capitalisation (proper names / sentence-initial). Decodes over the
167
+ mixed gen+copy distribution for a copy checkpoint, the generator for a plain one."""
168
+ m, stoi, itos = bundle or load(ckpt)
169
+ from .copyrealizer import CopyRealizer, decode_one as _copy_decode
170
+ is_copy = isinstance(m, CopyRealizer)
171
+
172
+ def realize(lemma, upos, feats, surface=None):
173
+ """Return (form, confidence). confidence in [0,1] is the min top1-top2 margin over decode steps
174
+ (on the mixed distribution for a copy model); the caller (`_realize_form`) abstains below a
175
+ threshold for true OOVs."""
176
+ h = header_ids(upos, feats.get("Case", "Nom"), feats.get("Number", "Sing"),
177
+ feats.get("Degree", "Pos"), feats.get("Px", "None")) # clamps unknowns
178
+ decode = _copy_decode if is_copy else _decode
179
+ # case the INPUT by POS (PROPN capitalized -> copy head copies the capital; NOUN/ADJ lowercase).
180
+ # Trailing surface-based re-capitalization still covers sentence-initial nouns.
181
+ out, conf = decode(m, h, case_lemma(upos, lemma.lower()), stoi, itos, score=True)
182
+ if not out:
183
+ return None, 0.0
184
+ if surface and surface[:1].isupper():
185
+ out = out[:1].upper() + out[1:]
186
+ return out, conf
187
+
188
+ return realize
189
+
190
+
191
+ _FEAT_ITOS = {i: k for k, i in FEAT_STOI.items()}
192
+
193
+
194
+ def _hdr_labels(h):
195
+ """Invert a header_ids list back to (POS, Case, Number, Degree) label strings."""
196
+ return tuple(_FEAT_ITOS[i].split("=", 1)[1] for i in h)
juurio/verbforms.py ADDED
@@ -0,0 +1,136 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """OOV non-finite verb-form realizer (inference).
2
+
3
+ Extends the open-vocabulary morphology story from nouns/adjectives (`realizer.py`) to VERB participles
4
+ and infinitives: a categorical header over (non-finite FORM x Case x Number). Given a verb lemma + that
5
+ header, emit the participle/infinitive surface, generalizing to loan verbs the lexicon cannot inflect
6
+ (`striimata` -> `striimaava` / `striimaamassa`).
7
+
8
+ The lexicon handles non-finite forms of KNOWN verbs; this model is the fallback for OOV/loan verbs,
9
+ exactly as the noun realizer is for OOV nouns/adjectives.
10
+
11
+ Wiring: callers set `algebra._VERBFORM_REALIZER = make_verbform_realizer()`; `pipeline.realize_verbform`
12
+ tries the lexicon first, this second, else abstains. Separate model + checkpoint.
13
+ """
14
+ from __future__ import annotations
15
+ import os
16
+
17
+ import torch
18
+
19
+ from .device import DEV
20
+ from . import bootstrap
21
+ from .realizer import Realizer, _pad, _decode, encode_chars # reuse the char enc-dec + decode
22
+ from .copyrealizer import CopyRealizer # pointer-generator copy head
23
+ from .algebra import VERBFORMS
24
+
25
+ CKPT = os.path.join(os.path.dirname(__file__), "..", "checkpoints", "verbform.pt")
26
+ PAD, BOS, EOS = 0, 1, 2
27
+
28
+ # categorical header: the non-finite FORM family x Case x Number. Cases are the full UD set
29
+ # (participles inflect like adjectives across all of them; infinitive forms only exist for a subset).
30
+ FORM_VALS = list(VERBFORMS) # va nut tava tu inf2 inf3
31
+ CASE_VALS = ["Nom", "Gen", "Par", "Ess", "Tra", "Ine", "Ela", "Ill", "Ade", "Abl", "All",
32
+ "Abe", "Com", "Ins"]
33
+ NUM_VALS = ["Sing", "Plur"]
34
+ FEAT_GROUPS = [("Form", FORM_VALS), ("Case", CASE_VALS), ("Number", NUM_VALS)]
35
+ FEAT_STOI = {f"{g}={v}": i for i, (g, v) in
36
+ enumerate((g, v) for g, vals in FEAT_GROUPS for v in vals)}
37
+ NFEAT_TOK = len(FEAT_STOI) # fits within realizer.NFEAT_TOK, so Realizer's featemb is reused
38
+ HLEN = len(FEAT_GROUPS) # 3 header slots
39
+
40
+
41
+ def header_ids(form, case, num):
42
+ form = form if form in FORM_VALS else "va"
43
+ case = case if case in CASE_VALS else "Nom"
44
+ num = num if num in NUM_VALS else "Sing"
45
+ if form == "inf2_pass": # passive 2nd inf is inessive-only; pin the header to its one cell
46
+ case, num = "Ine", "Sing"
47
+ return [FEAT_STOI[f"Form={form}"], FEAT_STOI[f"Case={case}"], FEAT_STOI[f"Number={num}"]]
48
+
49
+
50
+ def load(ckpt=None):
51
+ """Load the non-finite net. A copy-head checkpoint (pointer-generator — carries copy_q/copy_k/gate)
52
+ is reconstructed as a CopyRealizer so the served distribution is the mixed gen+copy one; a plain
53
+ checkpoint stays a generator-only Realizer. Detection is by weights, so old and new checkpoints
54
+ both load. `ckpt=None` resolves the shipped artifact via bootstrap."""
55
+ if ckpt is None:
56
+ ckpt = bootstrap.artifact_path("verbform")
57
+ ck = torch.load(ckpt, map_location=DEV, weights_only=False)
58
+ if "copy_q.weight" in ck["model"]:
59
+ from .copyrealizer import load as _copy_load
60
+ return _copy_load(ckpt)
61
+ m = Realizer(len(ck["stoi"]), ck["cfg"]["d_model"], 4, ck["cfg"]["layers"]).to(DEV)
62
+ m.load_state_dict(ck["model"]); m.eval()
63
+ return m, ck["stoi"], ck["itos"]
64
+
65
+
66
+ def decode_forms(items, bundle, batch_size: int = 256, maxlen: int = 48):
67
+ """Greedy-decode a non-finite form for each (lemma, (form, case, num)). Batched; bit-identical to
68
+ the single-item _decode (greedy argmax + per-seq attention). items: iterable of
69
+ (lemma, (form, case, num)). Mirrors finiteverb.decode_forms — same Realizer enc-dec, just this
70
+ module's 3-slot header."""
71
+ items = list(items)
72
+ if not items:
73
+ return []
74
+ m, stoi, itos = bundle
75
+ if isinstance(m, CopyRealizer):
76
+ # copy model: decode over the mixed gen+copy distribution (CopyRealizer.decode_forms takes
77
+ # (lemma, header_ids)); the generator-only proj path below would measure the wrong model.
78
+ from . import copyrealizer as CR
79
+ return CR.decode_forms([(lem.lower(), header_ids(f, c, n)) for lem, (f, c, n) in items],
80
+ bundle, batch_size=batch_size, maxlen=maxlen)
81
+ m.eval()
82
+ out = []
83
+ for start in range(0, len(items), batch_size):
84
+ chunk = items[start:start + batch_size]
85
+ B = len(chunk)
86
+ src_lists = [encode_chars(lem.lower(), stoi) for lem, _ in chunk]
87
+ S = max((len(s) for s in src_lists), default=1) or 1
88
+ src = _pad(src_lists, S).to(DEV)
89
+ hdr = torch.tensor([header_ids(f, c, n) for _, (f, c, n) in chunk],
90
+ dtype=torch.long, device=DEV)
91
+ with torch.no_grad():
92
+ mem, mask = m.encode(src, hdr)
93
+ ids = torch.full((B, 1), BOS, dtype=torch.long, device=DEV)
94
+ finished = torch.zeros(B, dtype=torch.bool, device=DEV)
95
+ collected = [[] for _ in range(B)]
96
+ for _ in range(maxlen):
97
+ T = ids.size(1)
98
+ cmask = torch.triu(torch.ones(T, T, device=DEV, dtype=torch.bool), 1)
99
+ dout = m.dec(m._tgt_embed(ids), mem, tgt_mask=cmask, memory_key_padding_mask=mask)
100
+ nxt = m.proj(dout)[:, -1].argmax(-1)
101
+ nxt_cpu = nxt.tolist()
102
+ for i in range(B):
103
+ if finished[i]:
104
+ continue
105
+ if nxt_cpu[i] == EOS:
106
+ finished[i] = True
107
+ else:
108
+ collected[i].append(nxt_cpu[i])
109
+ app = nxt.clone(); app[finished] = PAD
110
+ ids = torch.cat([ids, app.unsqueeze(1)], 1)
111
+ if bool(finished.all()):
112
+ break
113
+ out.extend("".join(itos[t] for t in row if t in itos) for row in collected)
114
+ return out
115
+
116
+
117
+ def make_verbform_realizer(bundle=None, ckpt=None):
118
+ """Return fn(lemma, form, case, number) -> (form, confidence) for `algebra._VERBFORM_REALIZER`
119
+ (None on failure). confidence is the min top1-top2 margin over decode steps — measured on the
120
+ mixed gen+copy distribution for a copy checkpoint, on the generator for a plain one."""
121
+ m, stoi, itos = bundle or load(ckpt)
122
+ if isinstance(m, CopyRealizer):
123
+ from .copyrealizer import decode_one as _copy_decode
124
+
125
+ def realize(lemma, form, case="Nom", number="Sing"):
126
+ out, conf = _copy_decode(m, header_ids(form, case, number), lemma.lower(), stoi, itos,
127
+ score=True)
128
+ return (out, conf) if out else (None, 0.0)
129
+
130
+ return realize
131
+
132
+ def realize(lemma, form, case="Nom", number="Sing"):
133
+ out, conf = _decode(m, header_ids(form, case, number), lemma.lower(), stoi, itos, score=True)
134
+ return (out, conf) if out else (None, 0.0)
135
+
136
+ return realize
pyproject.toml ADDED
@@ -0,0 +1,52 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [build-system]
2
+ requires = ["hatchling"]
3
+ build-backend = "hatchling.build"
4
+
5
+ [project]
6
+ name = "juurio"
7
+ version = "0.1.0"
8
+ description = "Open-vocabulary Finnish morphology: a neural parser, an exception lexicon and tiny neural realizers for OOV/loan word forms"
9
+ readme = "README.md"
10
+ requires-python = ">=3.12"
11
+ license = { text = "MIT" }
12
+ license-files = ["LICENSE"]
13
+ authors = [{ name = "samuli" }]
14
+ keywords = ["finnish", "morphology", "nlp", "inflection", "dependency-parsing", "fst", "uralic"]
15
+ classifiers = [
16
+ "Development Status :: 4 - Beta",
17
+ "Intended Audience :: Developers",
18
+ "License :: OSI Approved :: MIT License",
19
+ "Natural Language :: Finnish",
20
+ "Programming Language :: Python :: 3",
21
+ "Programming Language :: Python :: 3.12",
22
+ "Topic :: Scientific/Engineering :: Artificial Intelligence",
23
+ "Topic :: Text Processing :: Linguistic",
24
+ ]
25
+ # Version floors track the tested versions and MUST NOT exceed what's installed/tested (guarded by
26
+ # tests/test_runtime_dep_floors.py) — a floor above the tested version makes a clean `pip install` pull
27
+ # an untested-newer dep. Inference needs only torch / numpy / transformers (the analysis path is the
28
+ # torch-only neural parser; generation uses the neural realizers + lexicon). huggingface-hub: it
29
+ # fetches the model/data artifacts from a HuggingFace repo via bootstrap.artifact_path.
30
+ dependencies = [
31
+ "transformers>=5.7",
32
+ "torch>=2.4",
33
+ "numpy>=1.26",
34
+ "huggingface-hub>=0.34",
35
+ ]
36
+
37
+ [project.urls]
38
+ Homepage = "https://huggingface.co/samuli/juurio"
39
+
40
+ [project.optional-dependencies]
41
+ dev = ["pytest>=8", "packaging>=23"]
42
+
43
+ [project.scripts]
44
+ juurio = "juurio.cli:main"
45
+
46
+ # ─── Wheel ship-list ─────────────────────────────────────────────────────────────────────────────
47
+ # Ships the `juurio` package (top-level modules + the genlex override JSONs, juurio.analyze,
48
+ # and juurio.parser). The bulk serving artifacts (model checkpoints, wordlist, compressed tag table)
49
+ # are NOT bundled — they are fetched at runtime from the HuggingFace repo by `juurio setup`
50
+ # (bootstrap.py), sha256-verified. Guarded by tests/test_serving_wheel_contents.py.
51
+ [tool.hatch.build.targets.wheel]
52
+ only-include = ["juurio"]