Juurio 0.1.0: code, wheel, model card and licences
Browse files- .gitattributes +1 -0
- ATTRIBUTION.md +51 -0
- LICENSE +21 -0
- LICENSE-CC-BY-4.0 +396 -0
- LICENSE-GPL-3.0 +674 -0
- NOTICE +14 -0
- README.md +139 -0
- docs/api.md +316 -0
- juurio-0.1.0-py3-none-any.whl +3 -0
- juurio/__init__.py +210 -0
- juurio/__main__.py +5 -0
- juurio/algebra.py +1438 -0
- juurio/analyze/__init__.py +0 -0
- juurio/analyze/data.py +127 -0
- juurio/analyze/hybrid.py +44 -0
- juurio/analyze/infer.py +74 -0
- juurio/analyze/metrics.py +66 -0
- juurio/analyze/model.py +95 -0
- juurio/bootstrap.py +353 -0
- juurio/cli.py +473 -0
- juurio/copyrealizer.py +162 -0
- juurio/device.py +8 -0
- juurio/features.py +272 -0
- juurio/finiteverb.py +138 -0
- juurio/genlex.py +124 -0
- juurio/genlex_overrides.json +244 -0
- juurio/genlex_overrides_noun.json +0 -0
- juurio/morphforms.py +59 -0
- juurio/parser/__init__.py +1 -0
- juurio/parser/case_augment.py +465 -0
- juurio/parser/char_vocab.py +179 -0
- juurio/parser/collator.py +466 -0
- juurio/parser/compound.py +154 -0
- juurio/parser/edit_scripts.py +155 -0
- juurio/parser/eval.py +158 -0
- juurio/parser/feats_per_key_head.py +96 -0
- juurio/parser/fst_cache.py +147 -0
- juurio/parser/label_vocab.py +192 -0
- juurio/parser/model.py +714 -0
- juurio/parser/near_identity.py +108 -0
- juurio/parser/neural_adapter.py +188 -0
- juurio/parser/silver.py +252 -0
- juurio/parser/simple_tokenizer.py +205 -0
- juurio/parser/tokenizer.py +172 -0
- juurio/pipeline.py +705 -0
- juurio/realizer.py +196 -0
- juurio/verbforms.py +136 -0
- pyproject.toml +52 -0
.gitattributes
CHANGED
|
@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
data/wordlist.json filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
data/wordlist.json filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
juurio-0.1.0-py3-none-any.whl filter=lfs diff=lfs merge=lfs -text
|
ATTRIBUTION.md
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Attribution and licensing
|
| 2 |
+
|
| 3 |
+
juurio has three separately licensed parts. The model checkpoints and data files are not bundled in the Python wheel. `juurio setup` downloads them from https://huggingface.co/samuli/juurio and caches them locally.
|
| 4 |
+
|
| 5 |
+
| part | files | licence |
|
| 6 |
+
|---|---|---|
|
| 7 |
+
| code | the `juurio` Python package (`juurio/`, the wheel) | MIT, see [`LICENSE`](LICENSE) |
|
| 8 |
+
| models | `checkpoints/` (weights, tokenizer and vocabularies), `data/char_vocab.json` | CC BY 4.0, see [`LICENSE-CC-BY-4.0`](LICENSE-CC-BY-4.0) |
|
| 9 |
+
| data derived from Giella `fin` | `data/wordlist.json`, `data/fst_table.jsonl`, `data/fst_tag_vocab.json` | GPL-3.0, see [`LICENSE-GPL-3.0`](LICENSE-GPL-3.0) |
|
| 10 |
+
|
| 11 |
+
For CC BY 4.0 attribution, credit "juurio by samuli, https://huggingface.co/samuli/juurio".
|
| 12 |
+
|
| 13 |
+
This file is a good-faith summary, not legal advice. Check the upstream licences linked below for your own use.
|
| 14 |
+
|
| 15 |
+
## Data derived from Giella `fin` (GPL-3.0)
|
| 16 |
+
|
| 17 |
+
These files are generated by running the Giella `fin` finite-state transducer (through [uralicNLP](https://github.com/mikahama/uralicNLP)) over word forms and lemmas found in a Finnish web corpus, and storing its output. They are derivative works of [GiellaLT `lang-fin`](https://github.com/giellalt/lang-fin) and are distributed under GPL-3.0, the licence of `lang-fin`.
|
| 18 |
+
|
| 19 |
+
| file | what |
|
| 20 |
+
|---|---|
|
| 21 |
+
| `data/wordlist.json` | exception table: the inflected forms the neural realizers get wrong |
|
| 22 |
+
| `data/fst_table.jsonl` | morphology tags for the 100,000 most frequent word forms where the neural analyser disagrees with Giella (e-mail addresses, @handles, URLs, phone numbers and hashtags removed) |
|
| 23 |
+
| `data/fst_tag_vocab.json` | morphology tag vocabulary |
|
| 24 |
+
|
| 25 |
+
© 2004–present UiT The Arctic University of Norway / Giellatekno, and the `lang-fin` contributors. Source: https://github.com/giellalt/lang-fin (GPL-3.0). `lang-fin` derives from [Omorfi](https://github.com/flammie/omorfi) (GPL-3.0).
|
| 26 |
+
|
| 27 |
+
## Models (CC BY 4.0)
|
| 28 |
+
|
| 29 |
+
| file | trained on |
|
| 30 |
+
|---|---|
|
| 31 |
+
| `checkpoints/realizer.pt`, `verbform.pt`, `finiteverb.pt` | inflected forms generated with Giella `fin` (GPL-3.0) |
|
| 32 |
+
| `checkpoints/parser.pt` | [UD_Finnish-TDT](https://universaldependencies.org/treebanks/fi_tdt/index.html) (CC BY-SA 4.0), silver parses of Finnish web text made with [Stanza](https://stanfordnlp.github.io/stanza/) `fi` (Apache-2.0, its model trained on UD_Finnish-TDT), and Giella `fin` morphology tags as input |
|
| 33 |
+
| `checkpoints/analyze.pt` | Giella `fin` morphology tags for word forms from Finnish web text |
|
| 34 |
+
|
| 35 |
+
Whether trained weights are a derivative work of their training data is legally unsettled. The table documents where the training data came from, so that users can make their own judgement, and credits the upstream sources.
|
| 36 |
+
|
| 37 |
+
## Tools used to build the models and data
|
| 38 |
+
|
| 39 |
+
None of these are needed to run juurio.
|
| 40 |
+
|
| 41 |
+
| project | use | licence |
|
| 42 |
+
|---|---|---|
|
| 43 |
+
| [GiellaLT `lang-fin`](https://github.com/giellalt/lang-fin) through [uralicNLP](https://github.com/mikahama/uralicNLP) | the morphology the realizers learn from | GPL-3.0 |
|
| 44 |
+
| [Stanza](https://stanfordnlp.github.io/stanza/) | silver dependency parses | Apache-2.0 (code) |
|
| 45 |
+
| [UD_Finnish-TDT](https://universaldependencies.org/treebanks/fi_tdt/index.html) | parser training data, and the treebank behind Stanza's `fi` model | CC BY-SA 4.0 |
|
| 46 |
+
| [Voikko](https://voikko.puimula.org/) | checking generated forms | MPL-1.1 / GPL-2.0+ / LGPL-2.1+ (Finnish dictionary GPL) |
|
| 47 |
+
| [Omorfi](https://github.com/flammie/omorfi) | upstream of `lang-fin` | GPL-3.0 |
|
| 48 |
+
|
| 49 |
+
## Runtime dependencies
|
| 50 |
+
|
| 51 |
+
`torch` (BSD-3-Clause), `transformers` (Apache-2.0), `numpy` (BSD-3-Clause), `huggingface-hub` (Apache-2.0).
|
LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
MIT License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2026 samuli (https://huggingface.co/samuli)
|
| 4 |
+
|
| 5 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
| 6 |
+
of this software and associated documentation files (the "Software"), to deal
|
| 7 |
+
in the Software without restriction, including without limitation the rights
|
| 8 |
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
| 9 |
+
copies of the Software, and to permit persons to whom the Software is
|
| 10 |
+
furnished to do so, subject to the following conditions:
|
| 11 |
+
|
| 12 |
+
The above copyright notice and this permission notice shall be included in all
|
| 13 |
+
copies or substantial portions of the Software.
|
| 14 |
+
|
| 15 |
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
| 16 |
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
| 17 |
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
| 18 |
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
| 19 |
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
| 20 |
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
| 21 |
+
SOFTWARE.
|
LICENSE-CC-BY-4.0
ADDED
|
@@ -0,0 +1,396 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Attribution 4.0 International
|
| 2 |
+
|
| 3 |
+
=======================================================================
|
| 4 |
+
|
| 5 |
+
Creative Commons Corporation ("Creative Commons") is not a law firm and
|
| 6 |
+
does not provide legal services or legal advice. Distribution of
|
| 7 |
+
Creative Commons public licenses does not create a lawyer-client or
|
| 8 |
+
other relationship. Creative Commons makes its licenses and related
|
| 9 |
+
information available on an "as-is" basis. Creative Commons gives no
|
| 10 |
+
warranties regarding its licenses, any material licensed under their
|
| 11 |
+
terms and conditions, or any related information. Creative Commons
|
| 12 |
+
disclaims all liability for damages resulting from their use to the
|
| 13 |
+
fullest extent possible.
|
| 14 |
+
|
| 15 |
+
Using Creative Commons Public Licenses
|
| 16 |
+
|
| 17 |
+
Creative Commons public licenses provide a standard set of terms and
|
| 18 |
+
conditions that creators and other rights holders may use to share
|
| 19 |
+
original works of authorship and other material subject to copyright
|
| 20 |
+
and certain other rights specified in the public license below. The
|
| 21 |
+
following considerations are for informational purposes only, are not
|
| 22 |
+
exhaustive, and do not form part of our licenses.
|
| 23 |
+
|
| 24 |
+
Considerations for licensors: Our public licenses are
|
| 25 |
+
intended for use by those authorized to give the public
|
| 26 |
+
permission to use material in ways otherwise restricted by
|
| 27 |
+
copyright and certain other rights. Our licenses are
|
| 28 |
+
irrevocable. Licensors should read and understand the terms
|
| 29 |
+
and conditions of the license they choose before applying it.
|
| 30 |
+
Licensors should also secure all rights necessary before
|
| 31 |
+
applying our licenses so that the public can reuse the
|
| 32 |
+
material as expected. Licensors should clearly mark any
|
| 33 |
+
material not subject to the license. This includes other CC-
|
| 34 |
+
licensed material, or material used under an exception or
|
| 35 |
+
limitation to copyright. More considerations for licensors:
|
| 36 |
+
wiki.creativecommons.org/Considerations_for_licensors
|
| 37 |
+
|
| 38 |
+
Considerations for the public: By using one of our public
|
| 39 |
+
licenses, a licensor grants the public permission to use the
|
| 40 |
+
licensed material under specified terms and conditions. If
|
| 41 |
+
the licensor's permission is not necessary for any reason--for
|
| 42 |
+
example, because of any applicable exception or limitation to
|
| 43 |
+
copyright--then that use is not regulated by the license. Our
|
| 44 |
+
licenses grant only permissions under copyright and certain
|
| 45 |
+
other rights that a licensor has authority to grant. Use of
|
| 46 |
+
the licensed material may still be restricted for other
|
| 47 |
+
reasons, including because others have copyright or other
|
| 48 |
+
rights in the material. A licensor may make special requests,
|
| 49 |
+
such as asking that all changes be marked or described.
|
| 50 |
+
Although not required by our licenses, you are encouraged to
|
| 51 |
+
respect those requests where reasonable. More considerations
|
| 52 |
+
for the public:
|
| 53 |
+
wiki.creativecommons.org/Considerations_for_licensees
|
| 54 |
+
|
| 55 |
+
=======================================================================
|
| 56 |
+
|
| 57 |
+
Creative Commons Attribution 4.0 International Public License
|
| 58 |
+
|
| 59 |
+
By exercising the Licensed Rights (defined below), You accept and agree
|
| 60 |
+
to be bound by the terms and conditions of this Creative Commons
|
| 61 |
+
Attribution 4.0 International Public License ("Public License"). To the
|
| 62 |
+
extent this Public License may be interpreted as a contract, You are
|
| 63 |
+
granted the Licensed Rights in consideration of Your acceptance of
|
| 64 |
+
these terms and conditions, and the Licensor grants You such rights in
|
| 65 |
+
consideration of benefits the Licensor receives from making the
|
| 66 |
+
Licensed Material available under these terms and conditions.
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
Section 1 -- Definitions.
|
| 70 |
+
|
| 71 |
+
a. Adapted Material means material subject to Copyright and Similar
|
| 72 |
+
Rights that is derived from or based upon the Licensed Material
|
| 73 |
+
and in which the Licensed Material is translated, altered,
|
| 74 |
+
arranged, transformed, or otherwise modified in a manner requiring
|
| 75 |
+
permission under the Copyright and Similar Rights held by the
|
| 76 |
+
Licensor. For purposes of this Public License, where the Licensed
|
| 77 |
+
Material is a musical work, performance, or sound recording,
|
| 78 |
+
Adapted Material is always produced where the Licensed Material is
|
| 79 |
+
synched in timed relation with a moving image.
|
| 80 |
+
|
| 81 |
+
b. Adapter's License means the license You apply to Your Copyright
|
| 82 |
+
and Similar Rights in Your contributions to Adapted Material in
|
| 83 |
+
accordance with the terms and conditions of this Public License.
|
| 84 |
+
|
| 85 |
+
c. Copyright and Similar Rights means copyright and/or similar rights
|
| 86 |
+
closely related to copyright including, without limitation,
|
| 87 |
+
performance, broadcast, sound recording, and Sui Generis Database
|
| 88 |
+
Rights, without regard to how the rights are labeled or
|
| 89 |
+
categorized. For purposes of this Public License, the rights
|
| 90 |
+
specified in Section 2(b)(1)-(2) are not Copyright and Similar
|
| 91 |
+
Rights.
|
| 92 |
+
|
| 93 |
+
d. Effective Technological Measures means those measures that, in the
|
| 94 |
+
absence of proper authority, may not be circumvented under laws
|
| 95 |
+
fulfilling obligations under Article 11 of the WIPO Copyright
|
| 96 |
+
Treaty adopted on December 20, 1996, and/or similar international
|
| 97 |
+
agreements.
|
| 98 |
+
|
| 99 |
+
e. Exceptions and Limitations means fair use, fair dealing, and/or
|
| 100 |
+
any other exception or limitation to Copyright and Similar Rights
|
| 101 |
+
that applies to Your use of the Licensed Material.
|
| 102 |
+
|
| 103 |
+
f. Licensed Material means the artistic or literary work, database,
|
| 104 |
+
or other material to which the Licensor applied this Public
|
| 105 |
+
License.
|
| 106 |
+
|
| 107 |
+
g. Licensed Rights means the rights granted to You subject to the
|
| 108 |
+
terms and conditions of this Public License, which are limited to
|
| 109 |
+
all Copyright and Similar Rights that apply to Your use of the
|
| 110 |
+
Licensed Material and that the Licensor has authority to license.
|
| 111 |
+
|
| 112 |
+
h. Licensor means the individual(s) or entity(ies) granting rights
|
| 113 |
+
under this Public License.
|
| 114 |
+
|
| 115 |
+
i. Share means to provide material to the public by any means or
|
| 116 |
+
process that requires permission under the Licensed Rights, such
|
| 117 |
+
as reproduction, public display, public performance, distribution,
|
| 118 |
+
dissemination, communication, or importation, and to make material
|
| 119 |
+
available to the public including in ways that members of the
|
| 120 |
+
public may access the material from a place and at a time
|
| 121 |
+
individually chosen by them.
|
| 122 |
+
|
| 123 |
+
j. Sui Generis Database Rights means rights other than copyright
|
| 124 |
+
resulting from Directive 96/9/EC of the European Parliament and of
|
| 125 |
+
the Council of 11 March 1996 on the legal protection of databases,
|
| 126 |
+
as amended and/or succeeded, as well as other essentially
|
| 127 |
+
equivalent rights anywhere in the world.
|
| 128 |
+
|
| 129 |
+
k. You means the individual or entity exercising the Licensed Rights
|
| 130 |
+
under this Public License. Your has a corresponding meaning.
|
| 131 |
+
|
| 132 |
+
|
| 133 |
+
Section 2 -- Scope.
|
| 134 |
+
|
| 135 |
+
a. License grant.
|
| 136 |
+
|
| 137 |
+
1. Subject to the terms and conditions of this Public License,
|
| 138 |
+
the Licensor hereby grants You a worldwide, royalty-free,
|
| 139 |
+
non-sublicensable, non-exclusive, irrevocable license to
|
| 140 |
+
exercise the Licensed Rights in the Licensed Material to:
|
| 141 |
+
|
| 142 |
+
a. reproduce and Share the Licensed Material, in whole or
|
| 143 |
+
in part; and
|
| 144 |
+
|
| 145 |
+
b. produce, reproduce, and Share Adapted Material.
|
| 146 |
+
|
| 147 |
+
2. Exceptions and Limitations. For the avoidance of doubt, where
|
| 148 |
+
Exceptions and Limitations apply to Your use, this Public
|
| 149 |
+
License does not apply, and You do not need to comply with
|
| 150 |
+
its terms and conditions.
|
| 151 |
+
|
| 152 |
+
3. Term. The term of this Public License is specified in Section
|
| 153 |
+
6(a).
|
| 154 |
+
|
| 155 |
+
4. Media and formats; technical modifications allowed. The
|
| 156 |
+
Licensor authorizes You to exercise the Licensed Rights in
|
| 157 |
+
all media and formats whether now known or hereafter created,
|
| 158 |
+
and to make technical modifications necessary to do so. The
|
| 159 |
+
Licensor waives and/or agrees not to assert any right or
|
| 160 |
+
authority to forbid You from making technical modifications
|
| 161 |
+
necessary to exercise the Licensed Rights, including
|
| 162 |
+
technical modifications necessary to circumvent Effective
|
| 163 |
+
Technological Measures. For purposes of this Public License,
|
| 164 |
+
simply making modifications authorized by this Section 2(a)
|
| 165 |
+
(4) never produces Adapted Material.
|
| 166 |
+
|
| 167 |
+
5. Downstream recipients.
|
| 168 |
+
|
| 169 |
+
a. Offer from the Licensor -- Licensed Material. Every
|
| 170 |
+
recipient of the Licensed Material automatically
|
| 171 |
+
receives an offer from the Licensor to exercise the
|
| 172 |
+
Licensed Rights under the terms and conditions of this
|
| 173 |
+
Public License.
|
| 174 |
+
|
| 175 |
+
b. No downstream restrictions. You may not offer or impose
|
| 176 |
+
any additional or different terms or conditions on, or
|
| 177 |
+
apply any Effective Technological Measures to, the
|
| 178 |
+
Licensed Material if doing so restricts exercise of the
|
| 179 |
+
Licensed Rights by any recipient of the Licensed
|
| 180 |
+
Material.
|
| 181 |
+
|
| 182 |
+
6. No endorsement. Nothing in this Public License constitutes or
|
| 183 |
+
may be construed as permission to assert or imply that You
|
| 184 |
+
are, or that Your use of the Licensed Material is, connected
|
| 185 |
+
with, or sponsored, endorsed, or granted official status by,
|
| 186 |
+
the Licensor or others designated to receive attribution as
|
| 187 |
+
provided in Section 3(a)(1)(A)(i).
|
| 188 |
+
|
| 189 |
+
b. Other rights.
|
| 190 |
+
|
| 191 |
+
1. Moral rights, such as the right of integrity, are not
|
| 192 |
+
licensed under this Public License, nor are publicity,
|
| 193 |
+
privacy, and/or other similar personality rights; however, to
|
| 194 |
+
the extent possible, the Licensor waives and/or agrees not to
|
| 195 |
+
assert any such rights held by the Licensor to the limited
|
| 196 |
+
extent necessary to allow You to exercise the Licensed
|
| 197 |
+
Rights, but not otherwise.
|
| 198 |
+
|
| 199 |
+
2. Patent and trademark rights are not licensed under this
|
| 200 |
+
Public License.
|
| 201 |
+
|
| 202 |
+
3. To the extent possible, the Licensor waives any right to
|
| 203 |
+
collect royalties from You for the exercise of the Licensed
|
| 204 |
+
Rights, whether directly or through a collecting society
|
| 205 |
+
under any voluntary or waivable statutory or compulsory
|
| 206 |
+
licensing scheme. In all other cases the Licensor expressly
|
| 207 |
+
reserves any right to collect such royalties.
|
| 208 |
+
|
| 209 |
+
|
| 210 |
+
Section 3 -- License Conditions.
|
| 211 |
+
|
| 212 |
+
Your exercise of the Licensed Rights is expressly made subject to the
|
| 213 |
+
following conditions.
|
| 214 |
+
|
| 215 |
+
a. Attribution.
|
| 216 |
+
|
| 217 |
+
1. If You Share the Licensed Material (including in modified
|
| 218 |
+
form), You must:
|
| 219 |
+
|
| 220 |
+
a. retain the following if it is supplied by the Licensor
|
| 221 |
+
with the Licensed Material:
|
| 222 |
+
|
| 223 |
+
i. identification of the creator(s) of the Licensed
|
| 224 |
+
Material and any others designated to receive
|
| 225 |
+
attribution, in any reasonable manner requested by
|
| 226 |
+
the Licensor (including by pseudonym if
|
| 227 |
+
designated);
|
| 228 |
+
|
| 229 |
+
ii. a copyright notice;
|
| 230 |
+
|
| 231 |
+
iii. a notice that refers to this Public License;
|
| 232 |
+
|
| 233 |
+
iv. a notice that refers to the disclaimer of
|
| 234 |
+
warranties;
|
| 235 |
+
|
| 236 |
+
v. a URI or hyperlink to the Licensed Material to the
|
| 237 |
+
extent reasonably practicable;
|
| 238 |
+
|
| 239 |
+
b. indicate if You modified the Licensed Material and
|
| 240 |
+
retain an indication of any previous modifications; and
|
| 241 |
+
|
| 242 |
+
c. indicate the Licensed Material is licensed under this
|
| 243 |
+
Public License, and include the text of, or the URI or
|
| 244 |
+
hyperlink to, this Public License.
|
| 245 |
+
|
| 246 |
+
2. You may satisfy the conditions in Section 3(a)(1) in any
|
| 247 |
+
reasonable manner based on the medium, means, and context in
|
| 248 |
+
which You Share the Licensed Material. For example, it may be
|
| 249 |
+
reasonable to satisfy the conditions by providing a URI or
|
| 250 |
+
hyperlink to a resource that includes the required
|
| 251 |
+
information.
|
| 252 |
+
|
| 253 |
+
3. If requested by the Licensor, You must remove any of the
|
| 254 |
+
information required by Section 3(a)(1)(A) to the extent
|
| 255 |
+
reasonably practicable.
|
| 256 |
+
|
| 257 |
+
4. If You Share Adapted Material You produce, the Adapter's
|
| 258 |
+
License You apply must not prevent recipients of the Adapted
|
| 259 |
+
Material from complying with this Public License.
|
| 260 |
+
|
| 261 |
+
|
| 262 |
+
Section 4 -- Sui Generis Database Rights.
|
| 263 |
+
|
| 264 |
+
Where the Licensed Rights include Sui Generis Database Rights that
|
| 265 |
+
apply to Your use of the Licensed Material:
|
| 266 |
+
|
| 267 |
+
a. for the avoidance of doubt, Section 2(a)(1) grants You the right
|
| 268 |
+
to extract, reuse, reproduce, and Share all or a substantial
|
| 269 |
+
portion of the contents of the database;
|
| 270 |
+
|
| 271 |
+
b. if You include all or a substantial portion of the database
|
| 272 |
+
contents in a database in which You have Sui Generis Database
|
| 273 |
+
Rights, then the database in which You have Sui Generis Database
|
| 274 |
+
Rights (but not its individual contents) is Adapted Material; and
|
| 275 |
+
|
| 276 |
+
c. You must comply with the conditions in Section 3(a) if You Share
|
| 277 |
+
all or a substantial portion of the contents of the database.
|
| 278 |
+
|
| 279 |
+
For the avoidance of doubt, this Section 4 supplements and does not
|
| 280 |
+
replace Your obligations under this Public License where the Licensed
|
| 281 |
+
Rights include other Copyright and Similar Rights.
|
| 282 |
+
|
| 283 |
+
|
| 284 |
+
Section 5 -- Disclaimer of Warranties and Limitation of Liability.
|
| 285 |
+
|
| 286 |
+
a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
|
| 287 |
+
EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
|
| 288 |
+
AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
|
| 289 |
+
ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
|
| 290 |
+
IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
|
| 291 |
+
WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
|
| 292 |
+
PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
|
| 293 |
+
ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
|
| 294 |
+
KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
|
| 295 |
+
ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
|
| 296 |
+
|
| 297 |
+
b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
|
| 298 |
+
TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
|
| 299 |
+
NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
|
| 300 |
+
INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
|
| 301 |
+
COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
|
| 302 |
+
USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
|
| 303 |
+
ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
|
| 304 |
+
DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
|
| 305 |
+
IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
|
| 306 |
+
|
| 307 |
+
c. The disclaimer of warranties and limitation of liability provided
|
| 308 |
+
above shall be interpreted in a manner that, to the extent
|
| 309 |
+
possible, most closely approximates an absolute disclaimer and
|
| 310 |
+
waiver of all liability.
|
| 311 |
+
|
| 312 |
+
|
| 313 |
+
Section 6 -- Term and Termination.
|
| 314 |
+
|
| 315 |
+
a. This Public License applies for the term of the Copyright and
|
| 316 |
+
Similar Rights licensed here. However, if You fail to comply with
|
| 317 |
+
this Public License, then Your rights under this Public License
|
| 318 |
+
terminate automatically.
|
| 319 |
+
|
| 320 |
+
b. Where Your right to use the Licensed Material has terminated under
|
| 321 |
+
Section 6(a), it reinstates:
|
| 322 |
+
|
| 323 |
+
1. automatically as of the date the violation is cured, provided
|
| 324 |
+
it is cured within 30 days of Your discovery of the
|
| 325 |
+
violation; or
|
| 326 |
+
|
| 327 |
+
2. upon express reinstatement by the Licensor.
|
| 328 |
+
|
| 329 |
+
For the avoidance of doubt, this Section 6(b) does not affect any
|
| 330 |
+
right the Licensor may have to seek remedies for Your violations
|
| 331 |
+
of this Public License.
|
| 332 |
+
|
| 333 |
+
c. For the avoidance of doubt, the Licensor may also offer the
|
| 334 |
+
Licensed Material under separate terms or conditions or stop
|
| 335 |
+
distributing the Licensed Material at any time; however, doing so
|
| 336 |
+
will not terminate this Public License.
|
| 337 |
+
|
| 338 |
+
d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
|
| 339 |
+
License.
|
| 340 |
+
|
| 341 |
+
|
| 342 |
+
Section 7 -- Other Terms and Conditions.
|
| 343 |
+
|
| 344 |
+
a. The Licensor shall not be bound by any additional or different
|
| 345 |
+
terms or conditions communicated by You unless expressly agreed.
|
| 346 |
+
|
| 347 |
+
b. Any arrangements, understandings, or agreements regarding the
|
| 348 |
+
Licensed Material not stated herein are separate from and
|
| 349 |
+
independent of the terms and conditions of this Public License.
|
| 350 |
+
|
| 351 |
+
|
| 352 |
+
Section 8 -- Interpretation.
|
| 353 |
+
|
| 354 |
+
a. For the avoidance of doubt, this Public License does not, and
|
| 355 |
+
shall not be interpreted to, reduce, limit, restrict, or impose
|
| 356 |
+
conditions on any use of the Licensed Material that could lawfully
|
| 357 |
+
be made without permission under this Public License.
|
| 358 |
+
|
| 359 |
+
b. To the extent possible, if any provision of this Public License is
|
| 360 |
+
deemed unenforceable, it shall be automatically reformed to the
|
| 361 |
+
minimum extent necessary to make it enforceable. If the provision
|
| 362 |
+
cannot be reformed, it shall be severed from this Public License
|
| 363 |
+
without affecting the enforceability of the remaining terms and
|
| 364 |
+
conditions.
|
| 365 |
+
|
| 366 |
+
c. No term or condition of this Public License will be waived and no
|
| 367 |
+
failure to comply consented to unless expressly agreed to by the
|
| 368 |
+
Licensor.
|
| 369 |
+
|
| 370 |
+
d. Nothing in this Public License constitutes or may be interpreted
|
| 371 |
+
as a limitation upon, or waiver of, any privileges and immunities
|
| 372 |
+
that apply to the Licensor or You, including from the legal
|
| 373 |
+
processes of any jurisdiction or authority.
|
| 374 |
+
|
| 375 |
+
|
| 376 |
+
=======================================================================
|
| 377 |
+
|
| 378 |
+
Creative Commons is not a party to its public
|
| 379 |
+
licenses. Notwithstanding, Creative Commons may elect to apply one of
|
| 380 |
+
its public licenses to material it publishes and in those instances
|
| 381 |
+
will be considered the “Licensor.” The text of the Creative Commons
|
| 382 |
+
public licenses is dedicated to the public domain under the CC0 Public
|
| 383 |
+
Domain Dedication. Except for the limited purpose of indicating that
|
| 384 |
+
material is shared under a Creative Commons public license or as
|
| 385 |
+
otherwise permitted by the Creative Commons policies published at
|
| 386 |
+
creativecommons.org/policies, Creative Commons does not authorize the
|
| 387 |
+
use of the trademark "Creative Commons" or any other trademark or logo
|
| 388 |
+
of Creative Commons without its prior written consent including,
|
| 389 |
+
without limitation, in connection with any unauthorized modifications
|
| 390 |
+
to any of its public licenses or any other arrangements,
|
| 391 |
+
understandings, or agreements concerning use of licensed material. For
|
| 392 |
+
the avoidance of doubt, this paragraph does not form part of the
|
| 393 |
+
public licenses.
|
| 394 |
+
|
| 395 |
+
Creative Commons may be contacted at creativecommons.org.
|
| 396 |
+
|
LICENSE-GPL-3.0
ADDED
|
@@ -0,0 +1,674 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
GNU GENERAL PUBLIC LICENSE
|
| 2 |
+
Version 3, 29 June 2007
|
| 3 |
+
|
| 4 |
+
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
|
| 5 |
+
Everyone is permitted to copy and distribute verbatim copies
|
| 6 |
+
of this license document, but changing it is not allowed.
|
| 7 |
+
|
| 8 |
+
Preamble
|
| 9 |
+
|
| 10 |
+
The GNU General Public License is a free, copyleft license for
|
| 11 |
+
software and other kinds of works.
|
| 12 |
+
|
| 13 |
+
The licenses for most software and other practical works are designed
|
| 14 |
+
to take away your freedom to share and change the works. By contrast,
|
| 15 |
+
the GNU General Public License is intended to guarantee your freedom to
|
| 16 |
+
share and change all versions of a program--to make sure it remains free
|
| 17 |
+
software for all its users. We, the Free Software Foundation, use the
|
| 18 |
+
GNU General Public License for most of our software; it applies also to
|
| 19 |
+
any other work released this way by its authors. You can apply it to
|
| 20 |
+
your programs, too.
|
| 21 |
+
|
| 22 |
+
When we speak of free software, we are referring to freedom, not
|
| 23 |
+
price. Our General Public Licenses are designed to make sure that you
|
| 24 |
+
have the freedom to distribute copies of free software (and charge for
|
| 25 |
+
them if you wish), that you receive source code or can get it if you
|
| 26 |
+
want it, that you can change the software or use pieces of it in new
|
| 27 |
+
free programs, and that you know you can do these things.
|
| 28 |
+
|
| 29 |
+
To protect your rights, we need to prevent others from denying you
|
| 30 |
+
these rights or asking you to surrender the rights. Therefore, you have
|
| 31 |
+
certain responsibilities if you distribute copies of the software, or if
|
| 32 |
+
you modify it: responsibilities to respect the freedom of others.
|
| 33 |
+
|
| 34 |
+
For example, if you distribute copies of such a program, whether
|
| 35 |
+
gratis or for a fee, you must pass on to the recipients the same
|
| 36 |
+
freedoms that you received. You must make sure that they, too, receive
|
| 37 |
+
or can get the source code. And you must show them these terms so they
|
| 38 |
+
know their rights.
|
| 39 |
+
|
| 40 |
+
Developers that use the GNU GPL protect your rights with two steps:
|
| 41 |
+
(1) assert copyright on the software, and (2) offer you this License
|
| 42 |
+
giving you legal permission to copy, distribute and/or modify it.
|
| 43 |
+
|
| 44 |
+
For the developers' and authors' protection, the GPL clearly explains
|
| 45 |
+
that there is no warranty for this free software. For both users' and
|
| 46 |
+
authors' sake, the GPL requires that modified versions be marked as
|
| 47 |
+
changed, so that their problems will not be attributed erroneously to
|
| 48 |
+
authors of previous versions.
|
| 49 |
+
|
| 50 |
+
Some devices are designed to deny users access to install or run
|
| 51 |
+
modified versions of the software inside them, although the manufacturer
|
| 52 |
+
can do so. This is fundamentally incompatible with the aim of
|
| 53 |
+
protecting users' freedom to change the software. The systematic
|
| 54 |
+
pattern of such abuse occurs in the area of products for individuals to
|
| 55 |
+
use, which is precisely where it is most unacceptable. Therefore, we
|
| 56 |
+
have designed this version of the GPL to prohibit the practice for those
|
| 57 |
+
products. If such problems arise substantially in other domains, we
|
| 58 |
+
stand ready to extend this provision to those domains in future versions
|
| 59 |
+
of the GPL, as needed to protect the freedom of users.
|
| 60 |
+
|
| 61 |
+
Finally, every program is threatened constantly by software patents.
|
| 62 |
+
States should not allow patents to restrict development and use of
|
| 63 |
+
software on general-purpose computers, but in those that do, we wish to
|
| 64 |
+
avoid the special danger that patents applied to a free program could
|
| 65 |
+
make it effectively proprietary. To prevent this, the GPL assures that
|
| 66 |
+
patents cannot be used to render the program non-free.
|
| 67 |
+
|
| 68 |
+
The precise terms and conditions for copying, distribution and
|
| 69 |
+
modification follow.
|
| 70 |
+
|
| 71 |
+
TERMS AND CONDITIONS
|
| 72 |
+
|
| 73 |
+
0. Definitions.
|
| 74 |
+
|
| 75 |
+
"This License" refers to version 3 of the GNU General Public License.
|
| 76 |
+
|
| 77 |
+
"Copyright" also means copyright-like laws that apply to other kinds of
|
| 78 |
+
works, such as semiconductor masks.
|
| 79 |
+
|
| 80 |
+
"The Program" refers to any copyrightable work licensed under this
|
| 81 |
+
License. Each licensee is addressed as "you". "Licensees" and
|
| 82 |
+
"recipients" may be individuals or organizations.
|
| 83 |
+
|
| 84 |
+
To "modify" a work means to copy from or adapt all or part of the work
|
| 85 |
+
in a fashion requiring copyright permission, other than the making of an
|
| 86 |
+
exact copy. The resulting work is called a "modified version" of the
|
| 87 |
+
earlier work or a work "based on" the earlier work.
|
| 88 |
+
|
| 89 |
+
A "covered work" means either the unmodified Program or a work based
|
| 90 |
+
on the Program.
|
| 91 |
+
|
| 92 |
+
To "propagate" a work means to do anything with it that, without
|
| 93 |
+
permission, would make you directly or secondarily liable for
|
| 94 |
+
infringement under applicable copyright law, except executing it on a
|
| 95 |
+
computer or modifying a private copy. Propagation includes copying,
|
| 96 |
+
distribution (with or without modification), making available to the
|
| 97 |
+
public, and in some countries other activities as well.
|
| 98 |
+
|
| 99 |
+
To "convey" a work means any kind of propagation that enables other
|
| 100 |
+
parties to make or receive copies. Mere interaction with a user through
|
| 101 |
+
a computer network, with no transfer of a copy, is not conveying.
|
| 102 |
+
|
| 103 |
+
An interactive user interface displays "Appropriate Legal Notices"
|
| 104 |
+
to the extent that it includes a convenient and prominently visible
|
| 105 |
+
feature that (1) displays an appropriate copyright notice, and (2)
|
| 106 |
+
tells the user that there is no warranty for the work (except to the
|
| 107 |
+
extent that warranties are provided), that licensees may convey the
|
| 108 |
+
work under this License, and how to view a copy of this License. If
|
| 109 |
+
the interface presents a list of user commands or options, such as a
|
| 110 |
+
menu, a prominent item in the list meets this criterion.
|
| 111 |
+
|
| 112 |
+
1. Source Code.
|
| 113 |
+
|
| 114 |
+
The "source code" for a work means the preferred form of the work
|
| 115 |
+
for making modifications to it. "Object code" means any non-source
|
| 116 |
+
form of a work.
|
| 117 |
+
|
| 118 |
+
A "Standard Interface" means an interface that either is an official
|
| 119 |
+
standard defined by a recognized standards body, or, in the case of
|
| 120 |
+
interfaces specified for a particular programming language, one that
|
| 121 |
+
is widely used among developers working in that language.
|
| 122 |
+
|
| 123 |
+
The "System Libraries" of an executable work include anything, other
|
| 124 |
+
than the work as a whole, that (a) is included in the normal form of
|
| 125 |
+
packaging a Major Component, but which is not part of that Major
|
| 126 |
+
Component, and (b) serves only to enable use of the work with that
|
| 127 |
+
Major Component, or to implement a Standard Interface for which an
|
| 128 |
+
implementation is available to the public in source code form. A
|
| 129 |
+
"Major Component", in this context, means a major essential component
|
| 130 |
+
(kernel, window system, and so on) of the specific operating system
|
| 131 |
+
(if any) on which the executable work runs, or a compiler used to
|
| 132 |
+
produce the work, or an object code interpreter used to run it.
|
| 133 |
+
|
| 134 |
+
The "Corresponding Source" for a work in object code form means all
|
| 135 |
+
the source code needed to generate, install, and (for an executable
|
| 136 |
+
work) run the object code and to modify the work, including scripts to
|
| 137 |
+
control those activities. However, it does not include the work's
|
| 138 |
+
System Libraries, or general-purpose tools or generally available free
|
| 139 |
+
programs which are used unmodified in performing those activities but
|
| 140 |
+
which are not part of the work. For example, Corresponding Source
|
| 141 |
+
includes interface definition files associated with source files for
|
| 142 |
+
the work, and the source code for shared libraries and dynamically
|
| 143 |
+
linked subprograms that the work is specifically designed to require,
|
| 144 |
+
such as by intimate data communication or control flow between those
|
| 145 |
+
subprograms and other parts of the work.
|
| 146 |
+
|
| 147 |
+
The Corresponding Source need not include anything that users
|
| 148 |
+
can regenerate automatically from other parts of the Corresponding
|
| 149 |
+
Source.
|
| 150 |
+
|
| 151 |
+
The Corresponding Source for a work in source code form is that
|
| 152 |
+
same work.
|
| 153 |
+
|
| 154 |
+
2. Basic Permissions.
|
| 155 |
+
|
| 156 |
+
All rights granted under this License are granted for the term of
|
| 157 |
+
copyright on the Program, and are irrevocable provided the stated
|
| 158 |
+
conditions are met. This License explicitly affirms your unlimited
|
| 159 |
+
permission to run the unmodified Program. The output from running a
|
| 160 |
+
covered work is covered by this License only if the output, given its
|
| 161 |
+
content, constitutes a covered work. This License acknowledges your
|
| 162 |
+
rights of fair use or other equivalent, as provided by copyright law.
|
| 163 |
+
|
| 164 |
+
You may make, run and propagate covered works that you do not
|
| 165 |
+
convey, without conditions so long as your license otherwise remains
|
| 166 |
+
in force. You may convey covered works to others for the sole purpose
|
| 167 |
+
of having them make modifications exclusively for you, or provide you
|
| 168 |
+
with facilities for running those works, provided that you comply with
|
| 169 |
+
the terms of this License in conveying all material for which you do
|
| 170 |
+
not control copyright. Those thus making or running the covered works
|
| 171 |
+
for you must do so exclusively on your behalf, under your direction
|
| 172 |
+
and control, on terms that prohibit them from making any copies of
|
| 173 |
+
your copyrighted material outside their relationship with you.
|
| 174 |
+
|
| 175 |
+
Conveying under any other circumstances is permitted solely under
|
| 176 |
+
the conditions stated below. Sublicensing is not allowed; section 10
|
| 177 |
+
makes it unnecessary.
|
| 178 |
+
|
| 179 |
+
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
|
| 180 |
+
|
| 181 |
+
No covered work shall be deemed part of an effective technological
|
| 182 |
+
measure under any applicable law fulfilling obligations under article
|
| 183 |
+
11 of the WIPO copyright treaty adopted on 20 December 1996, or
|
| 184 |
+
similar laws prohibiting or restricting circumvention of such
|
| 185 |
+
measures.
|
| 186 |
+
|
| 187 |
+
When you convey a covered work, you waive any legal power to forbid
|
| 188 |
+
circumvention of technological measures to the extent such circumvention
|
| 189 |
+
is effected by exercising rights under this License with respect to
|
| 190 |
+
the covered work, and you disclaim any intention to limit operation or
|
| 191 |
+
modification of the work as a means of enforcing, against the work's
|
| 192 |
+
users, your or third parties' legal rights to forbid circumvention of
|
| 193 |
+
technological measures.
|
| 194 |
+
|
| 195 |
+
4. Conveying Verbatim Copies.
|
| 196 |
+
|
| 197 |
+
You may convey verbatim copies of the Program's source code as you
|
| 198 |
+
receive it, in any medium, provided that you conspicuously and
|
| 199 |
+
appropriately publish on each copy an appropriate copyright notice;
|
| 200 |
+
keep intact all notices stating that this License and any
|
| 201 |
+
non-permissive terms added in accord with section 7 apply to the code;
|
| 202 |
+
keep intact all notices of the absence of any warranty; and give all
|
| 203 |
+
recipients a copy of this License along with the Program.
|
| 204 |
+
|
| 205 |
+
You may charge any price or no price for each copy that you convey,
|
| 206 |
+
and you may offer support or warranty protection for a fee.
|
| 207 |
+
|
| 208 |
+
5. Conveying Modified Source Versions.
|
| 209 |
+
|
| 210 |
+
You may convey a work based on the Program, or the modifications to
|
| 211 |
+
produce it from the Program, in the form of source code under the
|
| 212 |
+
terms of section 4, provided that you also meet all of these conditions:
|
| 213 |
+
|
| 214 |
+
a) The work must carry prominent notices stating that you modified
|
| 215 |
+
it, and giving a relevant date.
|
| 216 |
+
|
| 217 |
+
b) The work must carry prominent notices stating that it is
|
| 218 |
+
released under this License and any conditions added under section
|
| 219 |
+
7. This requirement modifies the requirement in section 4 to
|
| 220 |
+
"keep intact all notices".
|
| 221 |
+
|
| 222 |
+
c) You must license the entire work, as a whole, under this
|
| 223 |
+
License to anyone who comes into possession of a copy. This
|
| 224 |
+
License will therefore apply, along with any applicable section 7
|
| 225 |
+
additional terms, to the whole of the work, and all its parts,
|
| 226 |
+
regardless of how they are packaged. This License gives no
|
| 227 |
+
permission to license the work in any other way, but it does not
|
| 228 |
+
invalidate such permission if you have separately received it.
|
| 229 |
+
|
| 230 |
+
d) If the work has interactive user interfaces, each must display
|
| 231 |
+
Appropriate Legal Notices; however, if the Program has interactive
|
| 232 |
+
interfaces that do not display Appropriate Legal Notices, your
|
| 233 |
+
work need not make them do so.
|
| 234 |
+
|
| 235 |
+
A compilation of a covered work with other separate and independent
|
| 236 |
+
works, which are not by their nature extensions of the covered work,
|
| 237 |
+
and which are not combined with it such as to form a larger program,
|
| 238 |
+
in or on a volume of a storage or distribution medium, is called an
|
| 239 |
+
"aggregate" if the compilation and its resulting copyright are not
|
| 240 |
+
used to limit the access or legal rights of the compilation's users
|
| 241 |
+
beyond what the individual works permit. Inclusion of a covered work
|
| 242 |
+
in an aggregate does not cause this License to apply to the other
|
| 243 |
+
parts of the aggregate.
|
| 244 |
+
|
| 245 |
+
6. Conveying Non-Source Forms.
|
| 246 |
+
|
| 247 |
+
You may convey a covered work in object code form under the terms
|
| 248 |
+
of sections 4 and 5, provided that you also convey the
|
| 249 |
+
machine-readable Corresponding Source under the terms of this License,
|
| 250 |
+
in one of these ways:
|
| 251 |
+
|
| 252 |
+
a) Convey the object code in, or embodied in, a physical product
|
| 253 |
+
(including a physical distribution medium), accompanied by the
|
| 254 |
+
Corresponding Source fixed on a durable physical medium
|
| 255 |
+
customarily used for software interchange.
|
| 256 |
+
|
| 257 |
+
b) Convey the object code in, or embodied in, a physical product
|
| 258 |
+
(including a physical distribution medium), accompanied by a
|
| 259 |
+
written offer, valid for at least three years and valid for as
|
| 260 |
+
long as you offer spare parts or customer support for that product
|
| 261 |
+
model, to give anyone who possesses the object code either (1) a
|
| 262 |
+
copy of the Corresponding Source for all the software in the
|
| 263 |
+
product that is covered by this License, on a durable physical
|
| 264 |
+
medium customarily used for software interchange, for a price no
|
| 265 |
+
more than your reasonable cost of physically performing this
|
| 266 |
+
conveying of source, or (2) access to copy the
|
| 267 |
+
Corresponding Source from a network server at no charge.
|
| 268 |
+
|
| 269 |
+
c) Convey individual copies of the object code with a copy of the
|
| 270 |
+
written offer to provide the Corresponding Source. This
|
| 271 |
+
alternative is allowed only occasionally and noncommercially, and
|
| 272 |
+
only if you received the object code with such an offer, in accord
|
| 273 |
+
with subsection 6b.
|
| 274 |
+
|
| 275 |
+
d) Convey the object code by offering access from a designated
|
| 276 |
+
place (gratis or for a charge), and offer equivalent access to the
|
| 277 |
+
Corresponding Source in the same way through the same place at no
|
| 278 |
+
further charge. You need not require recipients to copy the
|
| 279 |
+
Corresponding Source along with the object code. If the place to
|
| 280 |
+
copy the object code is a network server, the Corresponding Source
|
| 281 |
+
may be on a different server (operated by you or a third party)
|
| 282 |
+
that supports equivalent copying facilities, provided you maintain
|
| 283 |
+
clear directions next to the object code saying where to find the
|
| 284 |
+
Corresponding Source. Regardless of what server hosts the
|
| 285 |
+
Corresponding Source, you remain obligated to ensure that it is
|
| 286 |
+
available for as long as needed to satisfy these requirements.
|
| 287 |
+
|
| 288 |
+
e) Convey the object code using peer-to-peer transmission, provided
|
| 289 |
+
you inform other peers where the object code and Corresponding
|
| 290 |
+
Source of the work are being offered to the general public at no
|
| 291 |
+
charge under subsection 6d.
|
| 292 |
+
|
| 293 |
+
A separable portion of the object code, whose source code is excluded
|
| 294 |
+
from the Corresponding Source as a System Library, need not be
|
| 295 |
+
included in conveying the object code work.
|
| 296 |
+
|
| 297 |
+
A "User Product" is either (1) a "consumer product", which means any
|
| 298 |
+
tangible personal property which is normally used for personal, family,
|
| 299 |
+
or household purposes, or (2) anything designed or sold for incorporation
|
| 300 |
+
into a dwelling. In determining whether a product is a consumer product,
|
| 301 |
+
doubtful cases shall be resolved in favor of coverage. For a particular
|
| 302 |
+
product received by a particular user, "normally used" refers to a
|
| 303 |
+
typical or common use of that class of product, regardless of the status
|
| 304 |
+
of the particular user or of the way in which the particular user
|
| 305 |
+
actually uses, or expects or is expected to use, the product. A product
|
| 306 |
+
is a consumer product regardless of whether the product has substantial
|
| 307 |
+
commercial, industrial or non-consumer uses, unless such uses represent
|
| 308 |
+
the only significant mode of use of the product.
|
| 309 |
+
|
| 310 |
+
"Installation Information" for a User Product means any methods,
|
| 311 |
+
procedures, authorization keys, or other information required to install
|
| 312 |
+
and execute modified versions of a covered work in that User Product from
|
| 313 |
+
a modified version of its Corresponding Source. The information must
|
| 314 |
+
suffice to ensure that the continued functioning of the modified object
|
| 315 |
+
code is in no case prevented or interfered with solely because
|
| 316 |
+
modification has been made.
|
| 317 |
+
|
| 318 |
+
If you convey an object code work under this section in, or with, or
|
| 319 |
+
specifically for use in, a User Product, and the conveying occurs as
|
| 320 |
+
part of a transaction in which the right of possession and use of the
|
| 321 |
+
User Product is transferred to the recipient in perpetuity or for a
|
| 322 |
+
fixed term (regardless of how the transaction is characterized), the
|
| 323 |
+
Corresponding Source conveyed under this section must be accompanied
|
| 324 |
+
by the Installation Information. But this requirement does not apply
|
| 325 |
+
if neither you nor any third party retains the ability to install
|
| 326 |
+
modified object code on the User Product (for example, the work has
|
| 327 |
+
been installed in ROM).
|
| 328 |
+
|
| 329 |
+
The requirement to provide Installation Information does not include a
|
| 330 |
+
requirement to continue to provide support service, warranty, or updates
|
| 331 |
+
for a work that has been modified or installed by the recipient, or for
|
| 332 |
+
the User Product in which it has been modified or installed. Access to a
|
| 333 |
+
network may be denied when the modification itself materially and
|
| 334 |
+
adversely affects the operation of the network or violates the rules and
|
| 335 |
+
protocols for communication across the network.
|
| 336 |
+
|
| 337 |
+
Corresponding Source conveyed, and Installation Information provided,
|
| 338 |
+
in accord with this section must be in a format that is publicly
|
| 339 |
+
documented (and with an implementation available to the public in
|
| 340 |
+
source code form), and must require no special password or key for
|
| 341 |
+
unpacking, reading or copying.
|
| 342 |
+
|
| 343 |
+
7. Additional Terms.
|
| 344 |
+
|
| 345 |
+
"Additional permissions" are terms that supplement the terms of this
|
| 346 |
+
License by making exceptions from one or more of its conditions.
|
| 347 |
+
Additional permissions that are applicable to the entire Program shall
|
| 348 |
+
be treated as though they were included in this License, to the extent
|
| 349 |
+
that they are valid under applicable law. If additional permissions
|
| 350 |
+
apply only to part of the Program, that part may be used separately
|
| 351 |
+
under those permissions, but the entire Program remains governed by
|
| 352 |
+
this License without regard to the additional permissions.
|
| 353 |
+
|
| 354 |
+
When you convey a copy of a covered work, you may at your option
|
| 355 |
+
remove any additional permissions from that copy, or from any part of
|
| 356 |
+
it. (Additional permissions may be written to require their own
|
| 357 |
+
removal in certain cases when you modify the work.) You may place
|
| 358 |
+
additional permissions on material, added by you to a covered work,
|
| 359 |
+
for which you have or can give appropriate copyright permission.
|
| 360 |
+
|
| 361 |
+
Notwithstanding any other provision of this License, for material you
|
| 362 |
+
add to a covered work, you may (if authorized by the copyright holders of
|
| 363 |
+
that material) supplement the terms of this License with terms:
|
| 364 |
+
|
| 365 |
+
a) Disclaiming warranty or limiting liability differently from the
|
| 366 |
+
terms of sections 15 and 16 of this License; or
|
| 367 |
+
|
| 368 |
+
b) Requiring preservation of specified reasonable legal notices or
|
| 369 |
+
author attributions in that material or in the Appropriate Legal
|
| 370 |
+
Notices displayed by works containing it; or
|
| 371 |
+
|
| 372 |
+
c) Prohibiting misrepresentation of the origin of that material, or
|
| 373 |
+
requiring that modified versions of such material be marked in
|
| 374 |
+
reasonable ways as different from the original version; or
|
| 375 |
+
|
| 376 |
+
d) Limiting the use for publicity purposes of names of licensors or
|
| 377 |
+
authors of the material; or
|
| 378 |
+
|
| 379 |
+
e) Declining to grant rights under trademark law for use of some
|
| 380 |
+
trade names, trademarks, or service marks; or
|
| 381 |
+
|
| 382 |
+
f) Requiring indemnification of licensors and authors of that
|
| 383 |
+
material by anyone who conveys the material (or modified versions of
|
| 384 |
+
it) with contractual assumptions of liability to the recipient, for
|
| 385 |
+
any liability that these contractual assumptions directly impose on
|
| 386 |
+
those licensors and authors.
|
| 387 |
+
|
| 388 |
+
All other non-permissive additional terms are considered "further
|
| 389 |
+
restrictions" within the meaning of section 10. If the Program as you
|
| 390 |
+
received it, or any part of it, contains a notice stating that it is
|
| 391 |
+
governed by this License along with a term that is a further
|
| 392 |
+
restriction, you may remove that term. If a license document contains
|
| 393 |
+
a further restriction but permits relicensing or conveying under this
|
| 394 |
+
License, you may add to a covered work material governed by the terms
|
| 395 |
+
of that license document, provided that the further restriction does
|
| 396 |
+
not survive such relicensing or conveying.
|
| 397 |
+
|
| 398 |
+
If you add terms to a covered work in accord with this section, you
|
| 399 |
+
must place, in the relevant source files, a statement of the
|
| 400 |
+
additional terms that apply to those files, or a notice indicating
|
| 401 |
+
where to find the applicable terms.
|
| 402 |
+
|
| 403 |
+
Additional terms, permissive or non-permissive, may be stated in the
|
| 404 |
+
form of a separately written license, or stated as exceptions;
|
| 405 |
+
the above requirements apply either way.
|
| 406 |
+
|
| 407 |
+
8. Termination.
|
| 408 |
+
|
| 409 |
+
You may not propagate or modify a covered work except as expressly
|
| 410 |
+
provided under this License. Any attempt otherwise to propagate or
|
| 411 |
+
modify it is void, and will automatically terminate your rights under
|
| 412 |
+
this License (including any patent licenses granted under the third
|
| 413 |
+
paragraph of section 11).
|
| 414 |
+
|
| 415 |
+
However, if you cease all violation of this License, then your
|
| 416 |
+
license from a particular copyright holder is reinstated (a)
|
| 417 |
+
provisionally, unless and until the copyright holder explicitly and
|
| 418 |
+
finally terminates your license, and (b) permanently, if the copyright
|
| 419 |
+
holder fails to notify you of the violation by some reasonable means
|
| 420 |
+
prior to 60 days after the cessation.
|
| 421 |
+
|
| 422 |
+
Moreover, your license from a particular copyright holder is
|
| 423 |
+
reinstated permanently if the copyright holder notifies you of the
|
| 424 |
+
violation by some reasonable means, this is the first time you have
|
| 425 |
+
received notice of violation of this License (for any work) from that
|
| 426 |
+
copyright holder, and you cure the violation prior to 30 days after
|
| 427 |
+
your receipt of the notice.
|
| 428 |
+
|
| 429 |
+
Termination of your rights under this section does not terminate the
|
| 430 |
+
licenses of parties who have received copies or rights from you under
|
| 431 |
+
this License. If your rights have been terminated and not permanently
|
| 432 |
+
reinstated, you do not qualify to receive new licenses for the same
|
| 433 |
+
material under section 10.
|
| 434 |
+
|
| 435 |
+
9. Acceptance Not Required for Having Copies.
|
| 436 |
+
|
| 437 |
+
You are not required to accept this License in order to receive or
|
| 438 |
+
run a copy of the Program. Ancillary propagation of a covered work
|
| 439 |
+
occurring solely as a consequence of using peer-to-peer transmission
|
| 440 |
+
to receive a copy likewise does not require acceptance. However,
|
| 441 |
+
nothing other than this License grants you permission to propagate or
|
| 442 |
+
modify any covered work. These actions infringe copyright if you do
|
| 443 |
+
not accept this License. Therefore, by modifying or propagating a
|
| 444 |
+
covered work, you indicate your acceptance of this License to do so.
|
| 445 |
+
|
| 446 |
+
10. Automatic Licensing of Downstream Recipients.
|
| 447 |
+
|
| 448 |
+
Each time you convey a covered work, the recipient automatically
|
| 449 |
+
receives a license from the original licensors, to run, modify and
|
| 450 |
+
propagate that work, subject to this License. You are not responsible
|
| 451 |
+
for enforcing compliance by third parties with this License.
|
| 452 |
+
|
| 453 |
+
An "entity transaction" is a transaction transferring control of an
|
| 454 |
+
organization, or substantially all assets of one, or subdividing an
|
| 455 |
+
organization, or merging organizations. If propagation of a covered
|
| 456 |
+
work results from an entity transaction, each party to that
|
| 457 |
+
transaction who receives a copy of the work also receives whatever
|
| 458 |
+
licenses to the work the party's predecessor in interest had or could
|
| 459 |
+
give under the previous paragraph, plus a right to possession of the
|
| 460 |
+
Corresponding Source of the work from the predecessor in interest, if
|
| 461 |
+
the predecessor has it or can get it with reasonable efforts.
|
| 462 |
+
|
| 463 |
+
You may not impose any further restrictions on the exercise of the
|
| 464 |
+
rights granted or affirmed under this License. For example, you may
|
| 465 |
+
not impose a license fee, royalty, or other charge for exercise of
|
| 466 |
+
rights granted under this License, and you may not initiate litigation
|
| 467 |
+
(including a cross-claim or counterclaim in a lawsuit) alleging that
|
| 468 |
+
any patent claim is infringed by making, using, selling, offering for
|
| 469 |
+
sale, or importing the Program or any portion of it.
|
| 470 |
+
|
| 471 |
+
11. Patents.
|
| 472 |
+
|
| 473 |
+
A "contributor" is a copyright holder who authorizes use under this
|
| 474 |
+
License of the Program or a work on which the Program is based. The
|
| 475 |
+
work thus licensed is called the contributor's "contributor version".
|
| 476 |
+
|
| 477 |
+
A contributor's "essential patent claims" are all patent claims
|
| 478 |
+
owned or controlled by the contributor, whether already acquired or
|
| 479 |
+
hereafter acquired, that would be infringed by some manner, permitted
|
| 480 |
+
by this License, of making, using, or selling its contributor version,
|
| 481 |
+
but do not include claims that would be infringed only as a
|
| 482 |
+
consequence of further modification of the contributor version. For
|
| 483 |
+
purposes of this definition, "control" includes the right to grant
|
| 484 |
+
patent sublicenses in a manner consistent with the requirements of
|
| 485 |
+
this License.
|
| 486 |
+
|
| 487 |
+
Each contributor grants you a non-exclusive, worldwide, royalty-free
|
| 488 |
+
patent license under the contributor's essential patent claims, to
|
| 489 |
+
make, use, sell, offer for sale, import and otherwise run, modify and
|
| 490 |
+
propagate the contents of its contributor version.
|
| 491 |
+
|
| 492 |
+
In the following three paragraphs, a "patent license" is any express
|
| 493 |
+
agreement or commitment, however denominated, not to enforce a patent
|
| 494 |
+
(such as an express permission to practice a patent or covenant not to
|
| 495 |
+
sue for patent infringement). To "grant" such a patent license to a
|
| 496 |
+
party means to make such an agreement or commitment not to enforce a
|
| 497 |
+
patent against the party.
|
| 498 |
+
|
| 499 |
+
If you convey a covered work, knowingly relying on a patent license,
|
| 500 |
+
and the Corresponding Source of the work is not available for anyone
|
| 501 |
+
to copy, free of charge and under the terms of this License, through a
|
| 502 |
+
publicly available network server or other readily accessible means,
|
| 503 |
+
then you must either (1) cause the Corresponding Source to be so
|
| 504 |
+
available, or (2) arrange to deprive yourself of the benefit of the
|
| 505 |
+
patent license for this particular work, or (3) arrange, in a manner
|
| 506 |
+
consistent with the requirements of this License, to extend the patent
|
| 507 |
+
license to downstream recipients. "Knowingly relying" means you have
|
| 508 |
+
actual knowledge that, but for the patent license, your conveying the
|
| 509 |
+
covered work in a country, or your recipient's use of the covered work
|
| 510 |
+
in a country, would infringe one or more identifiable patents in that
|
| 511 |
+
country that you have reason to believe are valid.
|
| 512 |
+
|
| 513 |
+
If, pursuant to or in connection with a single transaction or
|
| 514 |
+
arrangement, you convey, or propagate by procuring conveyance of, a
|
| 515 |
+
covered work, and grant a patent license to some of the parties
|
| 516 |
+
receiving the covered work authorizing them to use, propagate, modify
|
| 517 |
+
or convey a specific copy of the covered work, then the patent license
|
| 518 |
+
you grant is automatically extended to all recipients of the covered
|
| 519 |
+
work and works based on it.
|
| 520 |
+
|
| 521 |
+
A patent license is "discriminatory" if it does not include within
|
| 522 |
+
the scope of its coverage, prohibits the exercise of, or is
|
| 523 |
+
conditioned on the non-exercise of one or more of the rights that are
|
| 524 |
+
specifically granted under this License. You may not convey a covered
|
| 525 |
+
work if you are a party to an arrangement with a third party that is
|
| 526 |
+
in the business of distributing software, under which you make payment
|
| 527 |
+
to the third party based on the extent of your activity of conveying
|
| 528 |
+
the work, and under which the third party grants, to any of the
|
| 529 |
+
parties who would receive the covered work from you, a discriminatory
|
| 530 |
+
patent license (a) in connection with copies of the covered work
|
| 531 |
+
conveyed by you (or copies made from those copies), or (b) primarily
|
| 532 |
+
for and in connection with specific products or compilations that
|
| 533 |
+
contain the covered work, unless you entered into that arrangement,
|
| 534 |
+
or that patent license was granted, prior to 28 March 2007.
|
| 535 |
+
|
| 536 |
+
Nothing in this License shall be construed as excluding or limiting
|
| 537 |
+
any implied license or other defenses to infringement that may
|
| 538 |
+
otherwise be available to you under applicable patent law.
|
| 539 |
+
|
| 540 |
+
12. No Surrender of Others' Freedom.
|
| 541 |
+
|
| 542 |
+
If conditions are imposed on you (whether by court order, agreement or
|
| 543 |
+
otherwise) that contradict the conditions of this License, they do not
|
| 544 |
+
excuse you from the conditions of this License. If you cannot convey a
|
| 545 |
+
covered work so as to satisfy simultaneously your obligations under this
|
| 546 |
+
License and any other pertinent obligations, then as a consequence you may
|
| 547 |
+
not convey it at all. For example, if you agree to terms that obligate you
|
| 548 |
+
to collect a royalty for further conveying from those to whom you convey
|
| 549 |
+
the Program, the only way you could satisfy both those terms and this
|
| 550 |
+
License would be to refrain entirely from conveying the Program.
|
| 551 |
+
|
| 552 |
+
13. Use with the GNU Affero General Public License.
|
| 553 |
+
|
| 554 |
+
Notwithstanding any other provision of this License, you have
|
| 555 |
+
permission to link or combine any covered work with a work licensed
|
| 556 |
+
under version 3 of the GNU Affero General Public License into a single
|
| 557 |
+
combined work, and to convey the resulting work. The terms of this
|
| 558 |
+
License will continue to apply to the part which is the covered work,
|
| 559 |
+
but the special requirements of the GNU Affero General Public License,
|
| 560 |
+
section 13, concerning interaction through a network will apply to the
|
| 561 |
+
combination as such.
|
| 562 |
+
|
| 563 |
+
14. Revised Versions of this License.
|
| 564 |
+
|
| 565 |
+
The Free Software Foundation may publish revised and/or new versions of
|
| 566 |
+
the GNU General Public License from time to time. Such new versions will
|
| 567 |
+
be similar in spirit to the present version, but may differ in detail to
|
| 568 |
+
address new problems or concerns.
|
| 569 |
+
|
| 570 |
+
Each version is given a distinguishing version number. If the
|
| 571 |
+
Program specifies that a certain numbered version of the GNU General
|
| 572 |
+
Public License "or any later version" applies to it, you have the
|
| 573 |
+
option of following the terms and conditions either of that numbered
|
| 574 |
+
version or of any later version published by the Free Software
|
| 575 |
+
Foundation. If the Program does not specify a version number of the
|
| 576 |
+
GNU General Public License, you may choose any version ever published
|
| 577 |
+
by the Free Software Foundation.
|
| 578 |
+
|
| 579 |
+
If the Program specifies that a proxy can decide which future
|
| 580 |
+
versions of the GNU General Public License can be used, that proxy's
|
| 581 |
+
public statement of acceptance of a version permanently authorizes you
|
| 582 |
+
to choose that version for the Program.
|
| 583 |
+
|
| 584 |
+
Later license versions may give you additional or different
|
| 585 |
+
permissions. However, no additional obligations are imposed on any
|
| 586 |
+
author or copyright holder as a result of your choosing to follow a
|
| 587 |
+
later version.
|
| 588 |
+
|
| 589 |
+
15. Disclaimer of Warranty.
|
| 590 |
+
|
| 591 |
+
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
|
| 592 |
+
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
|
| 593 |
+
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
|
| 594 |
+
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
|
| 595 |
+
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
| 596 |
+
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
|
| 597 |
+
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
|
| 598 |
+
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
|
| 599 |
+
|
| 600 |
+
16. Limitation of Liability.
|
| 601 |
+
|
| 602 |
+
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
|
| 603 |
+
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
|
| 604 |
+
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
|
| 605 |
+
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
|
| 606 |
+
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
|
| 607 |
+
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
|
| 608 |
+
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
|
| 609 |
+
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
|
| 610 |
+
SUCH DAMAGES.
|
| 611 |
+
|
| 612 |
+
17. Interpretation of Sections 15 and 16.
|
| 613 |
+
|
| 614 |
+
If the disclaimer of warranty and limitation of liability provided
|
| 615 |
+
above cannot be given local legal effect according to their terms,
|
| 616 |
+
reviewing courts shall apply local law that most closely approximates
|
| 617 |
+
an absolute waiver of all civil liability in connection with the
|
| 618 |
+
Program, unless a warranty or assumption of liability accompanies a
|
| 619 |
+
copy of the Program in return for a fee.
|
| 620 |
+
|
| 621 |
+
END OF TERMS AND CONDITIONS
|
| 622 |
+
|
| 623 |
+
How to Apply These Terms to Your New Programs
|
| 624 |
+
|
| 625 |
+
If you develop a new program, and you want it to be of the greatest
|
| 626 |
+
possible use to the public, the best way to achieve this is to make it
|
| 627 |
+
free software which everyone can redistribute and change under these terms.
|
| 628 |
+
|
| 629 |
+
To do so, attach the following notices to the program. It is safest
|
| 630 |
+
to attach them to the start of each source file to most effectively
|
| 631 |
+
state the exclusion of warranty; and each file should have at least
|
| 632 |
+
the "copyright" line and a pointer to where the full notice is found.
|
| 633 |
+
|
| 634 |
+
<one line to give the program's name and a brief idea of what it does.>
|
| 635 |
+
Copyright (C) <year> <name of author>
|
| 636 |
+
|
| 637 |
+
This program is free software: you can redistribute it and/or modify
|
| 638 |
+
it under the terms of the GNU General Public License as published by
|
| 639 |
+
the Free Software Foundation, either version 3 of the License, or
|
| 640 |
+
(at your option) any later version.
|
| 641 |
+
|
| 642 |
+
This program is distributed in the hope that it will be useful,
|
| 643 |
+
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
| 644 |
+
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
| 645 |
+
GNU General Public License for more details.
|
| 646 |
+
|
| 647 |
+
You should have received a copy of the GNU General Public License
|
| 648 |
+
along with this program. If not, see <https://www.gnu.org/licenses/>.
|
| 649 |
+
|
| 650 |
+
Also add information on how to contact you by electronic and paper mail.
|
| 651 |
+
|
| 652 |
+
If the program does terminal interaction, make it output a short
|
| 653 |
+
notice like this when it starts in an interactive mode:
|
| 654 |
+
|
| 655 |
+
<program> Copyright (C) <year> <name of author>
|
| 656 |
+
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
|
| 657 |
+
This is free software, and you are welcome to redistribute it
|
| 658 |
+
under certain conditions; type `show c' for details.
|
| 659 |
+
|
| 660 |
+
The hypothetical commands `show w' and `show c' should show the appropriate
|
| 661 |
+
parts of the General Public License. Of course, your program's commands
|
| 662 |
+
might be different; for a GUI interface, you would use an "about box".
|
| 663 |
+
|
| 664 |
+
You should also get your employer (if you work as a programmer) or school,
|
| 665 |
+
if any, to sign a "copyright disclaimer" for the program, if necessary.
|
| 666 |
+
For more information on this, and how to apply and follow the GNU GPL, see
|
| 667 |
+
<https://www.gnu.org/licenses/>.
|
| 668 |
+
|
| 669 |
+
The GNU General Public License does not permit incorporating your program
|
| 670 |
+
into proprietary programs. If your program is a subroutine library, you
|
| 671 |
+
may consider it more useful to permit linking proprietary applications with
|
| 672 |
+
the library. If this is what you want to do, use the GNU Lesser General
|
| 673 |
+
Public License instead of this License. But first, please read
|
| 674 |
+
<https://www.gnu.org/licenses/why-not-lgpl.html>.
|
NOTICE
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
juurio: licences
|
| 2 |
+
==================
|
| 3 |
+
|
| 4 |
+
Code (the juurio Python package and wheel): MIT, see LICENSE.
|
| 5 |
+
|
| 6 |
+
Models (checkpoints/, data/char_vocab.json): CC BY 4.0, see LICENSE-CC-BY-4.0.
|
| 7 |
+
Attribution: "juurio by samuli, https://huggingface.co/samuli/juurio".
|
| 8 |
+
|
| 9 |
+
Data derived from the Giella `fin` FST (data/wordlist.json, data/fst_table.jsonl,
|
| 10 |
+
data/fst_tag_vocab.json): GPL-3.0, see LICENSE-GPL-3.0.
|
| 11 |
+
(c) 2004-present UiT The Arctic University of Norway / Giellatekno and the lang-fin contributors.
|
| 12 |
+
Upstream source: https://github.com/giellalt/lang-fin
|
| 13 |
+
|
| 14 |
+
See ATTRIBUTION.md for where the training data came from. This is not legal advice.
|
README.md
ADDED
|
@@ -0,0 +1,139 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- fi
|
| 4 |
+
license: other
|
| 5 |
+
license_name: mit-and-cc-by-4.0-and-gpl-3.0
|
| 6 |
+
license_link: https://huggingface.co/samuli/juurio/blob/main/ATTRIBUTION.md
|
| 7 |
+
library_name: pytorch
|
| 8 |
+
tags:
|
| 9 |
+
- finnish
|
| 10 |
+
- morphology
|
| 11 |
+
- inflection
|
| 12 |
+
- lemmatization
|
| 13 |
+
- dependency-parsing
|
| 14 |
+
---
|
| 15 |
+
|
| 16 |
+
# Juurio
|
| 17 |
+
|
| 18 |
+
Open-vocabulary Finnish morphology over raw text. Give it a sentence and an operation (`past`, `passive`, `plural`, …) and it returns the transformed sentence or an explicit abstention. It also inflects out-of-vocabulary and loan words that a closed-lexicon morphology cannot (`striimata` → `striimannee`, `striimaaja` → `striimaajillanikin`).
|
| 19 |
+
|
| 20 |
+
```python
|
| 21 |
+
from juurio import transform, realize_np, realize_verbform
|
| 22 |
+
|
| 23 |
+
transform("past", "Koira haukkuu pihalla.").text # 'Koira haukkui pihalla.'
|
| 24 |
+
transform("passive", "Mies osti talon.").text # 'Ostettiin talo.'
|
| 25 |
+
transform("pluperfect","Hän lukee kirjan.").text # 'Hän oli lukenut kirjan.' (compound tense)
|
| 26 |
+
transform("neg_imperative","Osta talo.").text # 'Älä osta taloa.' (imperative/2nd-person source)
|
| 27 |
+
transform("potential", "Nuori striimaa peliä.", lemma="striimata").text # 'Nuori striimannee peliä.' (OOV verb)
|
| 28 |
+
transform("plural+poss_1sg+clitic_kin", "Koira haukkuu pihalla.").text # 'Koiranikin haukkuvat pihalla.'
|
| 29 |
+
|
| 30 |
+
# phrase-level morphology utilities
|
| 31 |
+
realize_np("talo", case="Ine", number="Plur", px="1Sg").text # 'taloissani'
|
| 32 |
+
realize_np("striimaaja", case="Ade", number="Plur", px="1Sg", clitic="kin").text # 'striimaajillanikin' (OOV)
|
| 33 |
+
realize_verbform("mennä", form="inf3", case="Ade").text # 'menemällä'
|
| 34 |
+
realize_verbform("striimata", form="va").text # 'striimaava' (OOV participle)
|
| 35 |
+
```
|
| 36 |
+
|
| 37 |
+
Every call returns a `Result` (`ok`, `text`, `reason`, `trace`, `warnings`). Input that is out of scope or that the operation does not apply to abstains, with `ok=False` and a documented reason. Forms produced by a neural model rather than the exception lexicon are listed in `trace["neural_unverified"]` and in `warnings`.
|
| 38 |
+
|
| 39 |
+
## Install
|
| 40 |
+
|
| 41 |
+
```bash
|
| 42 |
+
pip install https://huggingface.co/samuli/juurio/resolve/main/juurio-0.1.0-py3-none-any.whl
|
| 43 |
+
juurio setup # download and verify the model checkpoints and data
|
| 44 |
+
```
|
| 45 |
+
|
| 46 |
+
Python 3.12 or newer. The install pulls in `torch`, `transformers`, `numpy` and `huggingface-hub`.
|
| 47 |
+
|
| 48 |
+
On a machine with a GPU, that installs the CUDA build of PyTorch. On a CPU-only machine the default PyTorch wheel also bundles CUDA, a download of several GB you can't use. To get the small CPU-only build instead, install it first:
|
| 49 |
+
|
| 50 |
+
```bash
|
| 51 |
+
pip install torch --index-url https://download.pytorch.org/whl/cpu
|
| 52 |
+
pip install https://huggingface.co/samuli/juurio/resolve/main/juurio-0.1.0-py3-none-any.whl
|
| 53 |
+
```
|
| 54 |
+
|
| 55 |
+
The wheel contains only the code. `juurio setup` downloads the model checkpoints and data from this repository (about 190 MB), checks their sha256 hashes and caches them. Without it, they are downloaded on first use. `juurio doctor` shows what is present and what is missing.
|
| 56 |
+
|
| 57 |
+
## Background
|
| 58 |
+
|
| 59 |
+
Finnish is richly inflecting, and the standard open-source tools for it are excellent but closed-vocabulary and heavy at runtime. A finite-state morphology ([Giella `fin`](https://github.com/giellalt/lang-fin) and [Omorfi](https://github.com/flammie/omorfi), also the engine behind [Voikko](https://voikko.puimula.org/)) inflects only words already in its lexicon, and a neural dependency parser ([Stanza](https://stanfordnlp.github.io/stanza/)) brings a large dependency stack. A loan or slang word the lexicon has never seen (`striimata`, `striimaaja`) cannot be inflected.
|
| 60 |
+
|
| 61 |
+
Juurio distills those tools into small neural models and a compact exception lexicon, so at runtime it needs only PyTorch and it can inflect words the finite-state morphology does not know. The heavy tools were used only to build the models and data: Giella `fin` is the morphology the neural models learned from, Stanza produced silver parses for the parser, and Voikko was used to check generated forms.
|
| 62 |
+
|
| 63 |
+
## Architecture
|
| 64 |
+
|
| 65 |
+
| concern | component |
|
| 66 |
+
|---|---|
|
| 67 |
+
| structure and scope (which clause, subject, object) | neural UD dependency parser |
|
| 68 |
+
| morphology of known words | exception lexicon (the forms the neural realizers get wrong) |
|
| 69 |
+
| open-vocabulary morphology (OOV and loan words) | small neural realizers: nominal, verb-form and finite-verb |
|
| 70 |
+
|
| 71 |
+
The rule engine handles all non-local structure (agreement, object case, subject drop). The neural realizers are called per word, only when the lexicon has no entry for the form.
|
| 72 |
+
|
| 73 |
+
## Entry points
|
| 74 |
+
|
| 75 |
+
- `transform(op, text, lemma=None)`: finite-clause transforms that preserve grammar, such as negation, tense and mood (including the potential and the compound tenses), passive, question, the full imperative paradigm (with the negative `älä` series), number with agreement, degree, enclitics, possessives and `+`-composed nominal morphology.
|
| 76 |
+
- `realize_np(np, …)`: phrase-level nominal morphology, with the full case system, possessive suffixes, clitics and modifier agreement. Arbitrary case lives here because it is syntactically governed.
|
| 77 |
+
- `realize_verbform(verb, …)`: phrase-level non-finite verb morphology. Participles (VA, NUT, TAVA, TU and the agent participle MA) and infinitives, inflected in case and number with an optional possessive suffix.
|
| 78 |
+
- `inflect(word, …)`, `analyze(text)` and `lemmatize(text)`: single-word inflection, analysis and lemmatization.
|
| 79 |
+
|
| 80 |
+
Anything semantic, such as existential framing, is deliberately out of scope.
|
| 81 |
+
|
| 82 |
+
## CLI
|
| 83 |
+
|
| 84 |
+
Every library function has a command of the same name. The CLI is a thin wrapper over the same `Transformer`.
|
| 85 |
+
|
| 86 |
+
```
|
| 87 |
+
# one command per library function
|
| 88 |
+
juurio transform <op> "<text>" [--lemma L] [--trace] [--json] # transform()
|
| 89 |
+
juurio inflect "<word>" [spec…] [--case C] [--number N] [--degree D] [--px P] [--clitic K] [--pos P] [--like W] [--proper] [--json] # inflect()
|
| 90 |
+
juurio analyze "<text>" [--json] # analyze()
|
| 91 |
+
juurio lemmatize "<text>" [--json] # lemmatize()
|
| 92 |
+
juurio realize-np "<np>" [--case Ine] [--number Plur] [--px 1Sg] [--clitic kin] # realize_np()
|
| 93 |
+
juurio realize-verbform "<verb>" [--form va|nut|tava|tu|inf2|inf2_pass|inf3|ma] [--case Ine] [--px 3] # realize_verbform()
|
| 94 |
+
|
| 95 |
+
# infrastructure (no library counterpart)
|
| 96 |
+
juurio doctor # read-only environment diagnostics (dependencies, checkpoints + sha256)
|
| 97 |
+
juurio setup # download and verify the checkpoints and data
|
| 98 |
+
juurio list-ops # documented operation catalog
|
| 99 |
+
juurio batch <op> < lines # or per-line "op<TAB>text[<TAB>lemma]"
|
| 100 |
+
juurio demo | bench | repl
|
| 101 |
+
```
|
| 102 |
+
|
| 103 |
+
`inflect` takes a case, number and degree phrase positionally (`juurio inflect talo mon gen` → `talojen`) or the equivalent `--case`, `--number` and `--degree` flags.
|
| 104 |
+
|
| 105 |
+
Exit status is `0` on success, `2` if any transform abstained (useful for scripting) and `1` on setup failure.
|
| 106 |
+
|
| 107 |
+
## Files
|
| 108 |
+
|
| 109 |
+
| path | what |
|
| 110 |
+
|---|---|
|
| 111 |
+
| `juurio-0.1.0-py3-none-any.whl` | the Python package |
|
| 112 |
+
| `juurio/`, `pyproject.toml` | source code of the package |
|
| 113 |
+
| `docs/api.md` | API and CLI contract, scope boundary, reason codes |
|
| 114 |
+
| `checkpoints/` | parser, analyser and realizer models with their tokenizer and vocabularies |
|
| 115 |
+
| `data/` | exception lexicon, morphology tag table and vocabularies |
|
| 116 |
+
| `ATTRIBUTION.md` | licences and sources, file by file |
|
| 117 |
+
|
| 118 |
+
## Training data
|
| 119 |
+
|
| 120 |
+
- The realizers learned from inflected forms generated with the Giella `fin` transducer, for lemmas found in a Finnish web corpus.
|
| 121 |
+
- The exception lexicon holds the Giella forms the realizers get wrong.
|
| 122 |
+
- The parser was trained on [UD_Finnish-TDT](https://universaldependencies.org/treebanks/fi_tdt/index.html) and on silver parses of Finnish web text made with Stanza.
|
| 123 |
+
- The morphology tag table lists Giella's tags for the 100,000 most frequent web-text word forms where the neural analyser disagrees with Giella. E-mail addresses, @handles, URLs, phone numbers and hashtags were removed from it.
|
| 124 |
+
|
| 125 |
+
## Limitations
|
| 126 |
+
|
| 127 |
+
- `transform` handles one finite main clause in the active indicative, within limits of about twelve words and one finite verb. Subordinate and coordinated clauses, already negated clauses and multi-verb fragments abstain.
|
| 128 |
+
- Forms from the neural realizers are model guesses. They are flagged in `warnings` but not verified, and some are wrong: `realize_np("bugisovellus", case="Ine", px="1Pl", clitic="kin")` gives *bugisovelludessammekin* instead of *bugisovelluksessammekin*.
|
| 129 |
+
- The parser makes mistakes, and a wrong lemma or analysis can lead to a wrong form or an abstention. The `lemma=` argument overrides the lemma of the main verb.
|
| 130 |
+
|
| 131 |
+
## License
|
| 132 |
+
|
| 133 |
+
Three parts with separate licences (details in [`ATTRIBUTION.md`](ATTRIBUTION.md)):
|
| 134 |
+
|
| 135 |
+
| part | licence |
|
| 136 |
+
|---|---|
|
| 137 |
+
| code (`juurio/`, the wheel) | [MIT](LICENSE) |
|
| 138 |
+
| models (`checkpoints/`, `data/char_vocab.json`) | [CC BY 4.0](LICENSE-CC-BY-4.0) |
|
| 139 |
+
| data derived from Giella `fin` (`data/wordlist.json`, `data/fst_table.jsonl`, `data/fst_tag_vocab.json`) | [GPL-3.0](LICENSE-GPL-3.0), the licence of [GiellaLT `lang-fin`](https://github.com/giellalt/lang-fin) (© UiT The Arctic University of Norway / Giellatekno) |
|
docs/api.md
ADDED
|
@@ -0,0 +1,316 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# juurio — API & CLI contract
|
| 2 |
+
|
| 3 |
+
Open-vocabulary Finnish morphological transforms over raw text. You hand it a sentence and an
|
| 4 |
+
operation (`past`, `passive`, `plural`, …); it parses, transforms the finite main clause, and
|
| 5 |
+
returns the result or an explicit abstention. The differentiator is **OOV/loan words**: words the
|
| 6 |
+
exception lexicon does not cover (`striimata`, `tubettaja`) are inflected by small neural realizers.
|
| 7 |
+
|
| 8 |
+
## Architecture (the "split thesis")
|
| 9 |
+
|
| 10 |
+
Three components, each owning what it is best at:
|
| 11 |
+
|
| 12 |
+
| Concern | Component |
|
| 13 |
+
|---|---|
|
| 14 |
+
| structure / scope (which clause, which subject/object) | neural UD dependency parser (Giella/Stanza-free) |
|
| 15 |
+
| idiosyncrasy-free morphology (known words) | net + lexicon exception table (byte-exact to the Giella FST) |
|
| 16 |
+
| open-vocabulary morphology (OOV/loan words) | tiny lemma-anchored neural realizers |
|
| 17 |
+
|
| 18 |
+
The serving runtime needs only torch/numpy/transformers — no Stanza, no Giella FST, no Voikko (those
|
| 19 |
+
are used only to build the models and data). The rule engine owns all non-local structure (agreement propagation, object
|
| 20 |
+
case, subject drop); the neural realizers are called **per token**, only when the lexicon has no
|
| 21 |
+
entry for the cell.
|
| 22 |
+
|
| 23 |
+
## Scope boundary (a feature, not a limitation)
|
| 24 |
+
|
| 25 |
+
Two entry points with deliberately different promises:
|
| 26 |
+
|
| 27 |
+
- **`transform(op, text)` — finite-clause transforms** that preserve (or deliberately rewrite) the
|
| 28 |
+
clause's grammar: negation, tense/mood, passive, imperative, question, number agreement, possessive
|
| 29 |
+
on the subject head. Output is a grammatical sentence or an explicit abstain.
|
| 30 |
+
- **`realize_np(np, …)` — phrase-level nominal morphology**: inflect an NP into a Case/Number/Px(+
|
| 31 |
+
clitic). This is where *arbitrary case* lives, because case is **syntactically governed** — making
|
| 32 |
+
it a clause op would produce ungrammatical sentences (`Talo on iso` → `Talossa on iso`).
|
| 33 |
+
|
| 34 |
+
Anything **semantic** — e.g. existential/locative framing (`Taloissani on koira`), which introduces
|
| 35 |
+
and reorders arguments — is **out of scope** for both; it would be a separate future *generator* op,
|
| 36 |
+
not a morphological transform. Keeping these apart is what lets each give a clean guarantee.
|
| 37 |
+
|
| 38 |
+
## Python API
|
| 39 |
+
|
| 40 |
+
```python
|
| 41 |
+
from juurio import Transformer, transform, SUPPORTED_OPS, OP_CATALOG, REASONS, ABSTAIN_REASONS
|
| 42 |
+
|
| 43 |
+
t = Transformer() # loads parser + models once (heavy; reuse it)
|
| 44 |
+
r = t.transform("past", "Koira haukkuu pihalla.")
|
| 45 |
+
r.ok # True
|
| 46 |
+
r.text # 'Koira haukkui pihalla.'
|
| 47 |
+
r.reason # 'ok'
|
| 48 |
+
|
| 49 |
+
# one-shot convenience (reuses a process-wide Transformer):
|
| 50 |
+
transform("plural", "Striimaaja nukkuu.").text # 'Striimaajat nukkuvat.' (OOV noun, neural)
|
| 51 |
+
```
|
| 52 |
+
|
| 53 |
+
`import juurio` is cheap; torch + the neural parser/realizers load lazily on the first
|
| 54 |
+
`Transformer()` / `transform()` (no Stanza/Giella at serving time).
|
| 55 |
+
|
| 56 |
+
### `Transformer.transform(op, text, lemma=None, sent_index=0) -> Result`
|
| 57 |
+
|
| 58 |
+
- **op** — one of `SUPPORTED_OPS` (else `ValueError`).
|
| 59 |
+
- **text** — raw text; a leading dialogue dash/quote is handled and re-prepended.
|
| 60 |
+
- **lemma** — override the parser's lemma for the main verb. Use when a loan stem is mis-lemmatized
|
| 61 |
+
(`striimaa` → parser says `striä`); a confidence warning is raised automatically otherwise.
|
| 62 |
+
- **sent_index** — which parsed sentence to operate on (default 0).
|
| 63 |
+
|
| 64 |
+
### `Result`
|
| 65 |
+
|
| 66 |
+
| field | meaning |
|
| 67 |
+
|---|---|
|
| 68 |
+
| `text` | transformed sentence, or `None` if abstained |
|
| 69 |
+
| `ok` | `True` iff a form was produced |
|
| 70 |
+
| `reason` | one of `REASONS` (`"ok"` on success, else in `ABSTAIN_REASONS`) |
|
| 71 |
+
| `trace` | provenance dict (see below) — `juurio transform --trace` prints it |
|
| 72 |
+
| `warnings` | low-confidence-lemma notes + **neural realization** notices (model guess, unverified against the lexicon) |
|
| 73 |
+
|
| 74 |
+
`trace` fields: `op`, `root_verb`, `parser_lemma`, `lemma_source` (`override`/`parser`); and, when
|
| 75 |
+
relevant, `neural_oov` (forms the realizer produced — model guesses), `oov_unrealizable` /
|
| 76 |
+
`oov_unverified` (words it could not/did-not-trust). `realize_np` traces `util`, `case`, `number`,
|
| 77 |
+
`px`, `clitic`, `head`. Inspect on the CLI with `--trace`, or as JSON with `--json`.
|
| 78 |
+
|
| 79 |
+
### Performance
|
| 80 |
+
|
| 81 |
+
Models load once per process; the first call also loads the neural parser (the cold cost).
|
| 82 |
+
Indicative numbers (`juurio bench`, CPU parser + GPU realizer):
|
| 83 |
+
|
| 84 |
+
| | time |
|
| 85 |
+
|---|---|
|
| 86 |
+
| `Transformer()` construct | ~2 s |
|
| 87 |
+
| first `transform` (cold — parser load) | ~9 s |
|
| 88 |
+
| `transform` (warm) | ~70 ms |
|
| 89 |
+
| `realize_np` (warm) | ~30 ms |
|
| 90 |
+
|
| 91 |
+
Reuse one `Transformer` (or the module-level `transform`/`realize_np`, which cache a singleton).
|
| 92 |
+
|
| 93 |
+
## Scope contract
|
| 94 |
+
|
| 95 |
+
`transform` operates on a **single finite Act/Ind main clause** (≈≤12 words, one finite verb).
|
| 96 |
+
Anything else **abstains** (`ok=False`) — it never guesses. Reason codes:
|
| 97 |
+
|
| 98 |
+
| reason | meaning |
|
| 99 |
+
|---|---|
|
| 100 |
+
| `ok` | a transformation was produced |
|
| 101 |
+
| `parse_empty` | the parser returned no usable sentence at `sent_index` |
|
| 102 |
+
| `no_clause` | no finite Act/Ind main clause (subordinate-only, fragment, or already-negated) |
|
| 103 |
+
| `out_of_scope` | exceeds scope limits (too long / >1 finite verb / coordination) |
|
| 104 |
+
| `op_not_applicable` | op does not apply to this clause's base state (e.g. `past` on a past clause) |
|
| 105 |
+
| `oov_unrealizable` | an open-class word needed inflecting but **neither** the lexicon **nor** the neural realizer could produce the target form |
|
| 106 |
+
| `oov_unverified` | the neural realizer produced a candidate for an OOV word but it failed validation (stem-inconsistent or near-tie) and was **suppressed**, not emitted |
|
| 107 |
+
| `conflicting_features` | `inflect` got two valid but contradictory inputs (spec `mon gen` + `number="sing"`; `proper=True` + a degree; `like=` + an explicit case/number/degree; a single-word control on a phrase) |
|
| 108 |
+
| `unanalyzable_model` | `inflect(like=…)` could not analyze the model word into a nominal case/number cell to copy |
|
| 109 |
+
|
| 110 |
+
**Neural OOV is surfaced, never silent.** When a form is produced by the neural realizer (not the
|
| 111 |
+
lexicon), it appears in `trace["neural_unverified"]` and a `warnings` entry. Treat it as a model guess.
|
| 112 |
+
|
| 113 |
+
## Operations
|
| 114 |
+
|
| 115 |
+
`SUPPORTED_OPS` is the flat tuple; `OP_CATALOG` groups them with descriptions. Summary:
|
| 116 |
+
|
| 117 |
+
- **finite verb** — `neg`, `past`, `pres`, `cond`, `perfect`, `potential`, `passive`
|
| 118 |
+
- **compound tenses** (analytic olla auxiliary + the perfect participle) — `pluperfect`
|
| 119 |
+
(`olin sanonut`), `cond_perfect` (`olisin sanonut`), `pot_perfect` (`lienen sanonut`); OOV verbs
|
| 120 |
+
reuse the trained `perfect` participle + the lexicon's `olla` auxiliary
|
| 121 |
+
- **composites** (applied atomically) — `neg+past`, `neg+cond`, `neg+perfect`, `passive+past`,
|
| 122 |
+
`passive+cond`, `passive+perfect`, `neg+passive`, `neg+passive+past`, `neg+passive+cond`,
|
| 123 |
+
`neg+passive+perfect`
|
| 124 |
+
- **clause** — `quest` (polar question)
|
| 125 |
+
- **imperative paradigm** (drop subject, total object → Nom; negative forces partitive; trailing
|
| 126 |
+
punctuation preserved) — `imperative` (Sg2, OOV-capable), `imperative_pl2/sg3/pl1/pl3`, and the
|
| 127 |
+
negative `älä`-series `neg_imperative` (Sg2, OOV-capable), `neg_imperative_pl2/sg3/pl1/pl3`.
|
| 128 |
+
**Scope:** applies only when the command's addressee is in the clause — a subjectless/already-imperative
|
| 129 |
+
source (`osta talo.`) or a 2nd-person subject (`sinä/te`); a 3rd/1st-person subject abstains
|
| 130 |
+
(`out_of_scope`), since turning *"the man buys a house"* into a command would invent an addressee.
|
| 131 |
+
- **number** (re-agrees subject NP, modifiers, finite verb, copula predicate) — `plural`, `singular`
|
| 132 |
+
- **degree** (predicate adjective) — `comp`, `super`
|
| 133 |
+
- **enclitics** (on the finite verb) — `clitic_kin`, `clitic_kaan`, `clitic_han`, `clitic_pa`
|
| 134 |
+
- **possessive suffix** on the subject head, preserving its case/number — `poss_1sg`, `poss_2sg`,
|
| 135 |
+
`poss_3`, `poss_1pl`, `poss_2pl` (`talo`→`taloni`, `talossa`→`talossani`; OOV `striimaaja`→`striimaajani`)
|
| 136 |
+
|
| 137 |
+
`juurio list-ops` prints the live catalog.
|
| 138 |
+
|
| 139 |
+
### Combined nominal morphology
|
| 140 |
+
|
| 141 |
+
Nominal ops compose with `+`: a `+`-join of **nominal atoms** (`plural`/`singular`, `poss_*`,
|
| 142 |
+
`clitic_*`) is applied to the subject head as **one realization** with the final
|
| 143 |
+
Number+Case+Px, then the clitic is attached — deterministic surface order `stem+case+number+Px+clitic`.
|
| 144 |
+
This is correct-by-construction (you cannot re-pluralize the already-realized string `taloni`).
|
| 145 |
+
Number changes also re-agree modifiers and the finite verb.
|
| 146 |
+
|
| 147 |
+
```
|
| 148 |
+
poss_1sg+clitic_kin Koira haukkuu pihalla. -> Koiranikin haukkuu pihalla.
|
| 149 |
+
plural+poss_1sg Koira haukkuu pihalla. -> Koirani haukkuvat pihalla.
|
| 150 |
+
plural+poss_1sg+clitic_kin Koira haukkuu pihalla. -> Koiranikin haukkuvat pihalla.
|
| 151 |
+
plural+poss_1sg Striimaaja nukkuu. -> Striimaajani nukkuvat. (OOV head, neural)
|
| 152 |
+
```
|
| 153 |
+
|
| 154 |
+
Each dimension may appear at most once. `+`-composed **verb** ops (`neg+passive+past`) are separate,
|
| 155 |
+
pre-registered atoms in the catalog. Mixed verb+nominal composition is not supported.
|
| 156 |
+
|
| 157 |
+
## Phrase-level nominal morphology: `realize_np`
|
| 158 |
+
|
| 159 |
+
Finnish case is **syntactically governed**, so arbitrary case is *not* a clause transform (a
|
| 160 |
+
subject-case change would be ungrammatical). Instead it is a phrase utility, separate from
|
| 161 |
+
`transform`, which keeps grammaticality guarantees:
|
| 162 |
+
|
| 163 |
+
```python
|
| 164 |
+
from juurio import realize_np
|
| 165 |
+
realize_np("talo", case="Ine", number="Plur", px="1Sg").text # 'taloissani'
|
| 166 |
+
realize_np("iso talo", case="Ine", number="Plur").text # 'isoissa taloissa'
|
| 167 |
+
realize_np("Helsinki", case="Ine").text # 'Helsingissä'
|
| 168 |
+
realize_np("striimaaja", case="Ade", number="Plur", px="1Sg", clitic="kin").text # 'striimaajillanikin' (OOV)
|
| 169 |
+
```
|
| 170 |
+
|
| 171 |
+
`realize_np(np, case, number, px, clitic)` inflects an NP — a bare lemma or a short `adj* noun`
|
| 172 |
+
phrase — into the target Case/Number with an optional possessive and enclitic, agreeing modifiers,
|
| 173 |
+
and per-token lexicon→neural fallback (surface order `stem+case+number+Px+clitic`). It returns a `Result`
|
| 174 |
+
(`ok` with the NP, or an abstain reason). This is the showcase for open-vocabulary nominal
|
| 175 |
+
morphology; `transform` is for grammaticality-preserving clause operations. Sentence-level locatives
|
| 176 |
+
(`Taloissani on koira`) are a separate future generator op, not a case transform.
|
| 177 |
+
|
| 178 |
+
## Phrase-level verb morphology: `realize_verbform`
|
| 179 |
+
|
| 180 |
+
The non-finite counterpart of `realize_np`: inflect a VERB into a **participle** or **infinitive**.
|
| 181 |
+
The four participles inflect like adjectives in case+number; the 2nd/3rd infinitives are case-marked.
|
| 182 |
+
A second open-vocabulary surface — the lexicon handles known verbs, a small **verbform realizer**
|
| 183 |
+
inflects OOV/loan verbs (`striimata`), flagged `unverified`.
|
| 184 |
+
|
| 185 |
+
```python
|
| 186 |
+
from juurio import realize_verbform # form ∈ {va, nut, tava, tu, inf2, inf2_pass, inf3, ma}
|
| 187 |
+
realize_verbform("lukea", form="va").text # 'lukeva' (present active participle)
|
| 188 |
+
realize_verbform("lukea", form="va", case="Ine").text # 'lukevassa'
|
| 189 |
+
realize_verbform("lukea", form="tu").text # 'luettu' (past passive participle)
|
| 190 |
+
realize_verbform("mennä", form="inf3", case="Ade").text # 'menemällä' (3rd inf adessive)
|
| 191 |
+
realize_verbform("mennä", form="inf3", case="Ine").text # 'menemässä'
|
| 192 |
+
realize_verbform("tehdä", form="ma", case="Ine").text # 'tekemässä' (agent participle)
|
| 193 |
+
realize_verbform("tehdä", form="inf2_pass").text # 'tehtäessä' (passive 2nd inf, temporal)
|
| 194 |
+
realize_verbform("kuulla", form="tu", case="Par", px="3").text # 'kuultuansa' (temporal: 'after hearing')
|
| 195 |
+
realize_verbform("striimata", form="va").text # 'striimaava' (OOV verb, neural)
|
| 196 |
+
```
|
| 197 |
+
|
| 198 |
+
`va`/`nut`/`tava`/`tu` = present/past active/passive participle; `inf2`/`inf3` = 2nd/3rd infinitive;
|
| 199 |
+
`inf2_pass` = passive 2nd infinitive (`-ttaessa` temporal, "while X is being done"; inessive-only,
|
| 200 |
+
case-invariant); `ma` = agent participle (-ma/-mä, "the X that someone does", inflects in case/number).
|
| 201 |
+
`px` (one of `None`/`1Sg`/`2Sg`/`3`/`1Pl`/`2Pl`) attaches a possessive suffix — participles inflect
|
| 202 |
+
like nominals, so `tu`+`Par`+`px` realizes the **temporal construction of anteriority** (`kuultuaan` /
|
| 203 |
+
the `-nsa` variant `kuultuansa` = "after hearing"). For OOV verbs the bare participle comes from the
|
| 204 |
+
verbform realizer and the case+Px from the noun realizer (both neural stages flagged unverified).
|
| 205 |
+
Returns a `Result` (`ok` with the form, or `oov_unrealizable`/`oov_unverified` when the morphology
|
| 206 |
+
engine cannot produce/trust it). Like `realize_np`, this is a phrase utility, not a clause transform.
|
| 207 |
+
|
| 208 |
+
## Friendly inflection: `inflect`
|
| 209 |
+
|
| 210 |
+
The ergonomic single-word front door over `realize_np` — human-readable case/number/degree names, a
|
| 211 |
+
proper-noun toggle, and **inflect-by-example**. Returns a `Result` (never a bare string: abstention
|
| 212 |
+
stays first-class).
|
| 213 |
+
|
| 214 |
+
```python
|
| 215 |
+
from juurio import inflect
|
| 216 |
+
inflect("talo", "gen").text # 'talon' (alias: gen/genitive/genetiivi)
|
| 217 |
+
inflect("talo", "mon gen").text # 'talojen' (combined spec, taivutin-style)
|
| 218 |
+
inflect("talo", case="inessive", number="plural").text # 'taloissa'
|
| 219 |
+
inflect("talo", px="1sg").text # 'taloni'
|
| 220 |
+
inflect("kaunis", "gen", degree="comp").text # 'kauniimman' (degree => adjective path)
|
| 221 |
+
inflect("meri", "gen").text # 'meren' (common noun)
|
| 222 |
+
inflect("Meri", "gen", proper=True).text # proper-noun reading (≠ 'meren')
|
| 223 |
+
inflect("makaronilaatikko", like="talojen").text # 'makaronilaatikkojen' (copy a model word's cell)
|
| 224 |
+
inflect("iso talo", "mon gen").text # 'isojen talojen' (phrase -> realize_np)
|
| 225 |
+
```
|
| 226 |
+
|
| 227 |
+
`inflect(word, case="nom", number="sing", *, degree="pos", px="none", clitic=None, proper=None,
|
| 228 |
+
pos=None, like=None)`:
|
| 229 |
+
|
| 230 |
+
- **case** — a case alias, *or* a combined spec (`"mon gen"`, `"plural genitive comparative"`).
|
| 231 |
+
Accepts English abbrev/full and Finnish (`gen`/`genitive`/`genetiivi`, `mon`/`monikko`).
|
| 232 |
+
- **number / degree** — aliases too; an explicit value that contradicts the combined spec abstains
|
| 233 |
+
(`conflicting_features`).
|
| 234 |
+
- **px / clitic** — possessive suffix (`1sg`…/`None`) and enclitic, as in `realize_np`. `px=None` is
|
| 235 |
+
accepted (= no suffix). Rejected on a single adjective target.
|
| 236 |
+
- **proper** — `True`/`False` forces PROPN/NOUN; `None` (default) infers PROPN from a leading capital.
|
| 237 |
+
- **pos** — explicit `'noun'`/`'proper'`/`'adjective'` (wins over inference). PRON/NUM are not yet a
|
| 238 |
+
first-class `pos`. A degree implies the adjective path when `pos` is unset.
|
| 239 |
+
- **like** — a model word; its analyzed case (+number, +degree if adjectival) is copied onto `word`.
|
| 240 |
+
Mutually exclusive with `case`/`number`/`degree`; an unanalyzable model abstains
|
| 241 |
+
(`unanalyzable_model`).
|
| 242 |
+
|
| 243 |
+
An **unknown alias** raises `ValueError` (a typo fails fast); a **contradiction** abstains
|
| 244 |
+
(`conflicting_features`). A single token is realized **parser-free** (so `proper`/`pos` are honored);
|
| 245 |
+
a whitespace phrase delegates to `realize_np` (which applies `px`/`clitic` to the head but rejects the
|
| 246 |
+
single-word controls `proper`/`pos`/`degree`). An empty `word` raises `ValueError`.
|
| 247 |
+
|
| 248 |
+
## Analysis: `analyze` / `lemmatize`
|
| 249 |
+
|
| 250 |
+
The inverse of generation (what a generate-only inflector cannot do): the serving neural parser
|
| 251 |
+
exposed directly.
|
| 252 |
+
|
| 253 |
+
```python
|
| 254 |
+
from juurio import analyze, lemmatize, TokenAnalysis
|
| 255 |
+
sents = analyze("Koirat haukkuvat pihalla.") # list[list[TokenAnalysis]] (one list per sentence)
|
| 256 |
+
tok = sents[0][0]
|
| 257 |
+
tok.lemma, tok.upos, tok.feats # 'koira', 'NOUN', {'Case': 'Nom', 'Number': 'Plur'}
|
| 258 |
+
tok.asdict() # plain dict (id/form/lemma/upos/head/deprel/feats)
|
| 259 |
+
lemmatize("Koirat haukkuvat pihalla.") # ['koira', 'haukkua', 'piha', '.'] (flat)
|
| 260 |
+
```
|
| 261 |
+
|
| 262 |
+
`TokenAnalysis` is a frozen record (`id`, `form`, `lemma`, `upos`, `head` — 1-based, 0=root —,
|
| 263 |
+
`deprel`, `feats` as a read-only UD mapping). `analyze` keeps sentence boundaries; `lemmatize`
|
| 264 |
+
returns a flat lemma list.
|
| 265 |
+
|
| 266 |
+
## CLI
|
| 267 |
+
|
| 268 |
+
Each library function has a like-named command (the CLI is a thin wrapper over the same
|
| 269 |
+
`Transformer`); `realize_np`/`realize_verbform` keep the hyphenated spelling.
|
| 270 |
+
|
| 271 |
+
```
|
| 272 |
+
# one command per library function
|
| 273 |
+
juurio transform <op> "<text>" [--lemma L] [--trace] [--json] # transform()
|
| 274 |
+
juurio inflect "<word>" [spec…] [--case C] [--number N] [--degree D] [--px P] [--clitic K] [--pos P] [--like W] [--proper] [--json] # inflect()
|
| 275 |
+
juurio analyze "<text>" [--json] # analyze()
|
| 276 |
+
juurio lemmatize "<text>" [--json] # lemmatize()
|
| 277 |
+
juurio realize-np "<np>" [--case Ine] [--number Plur] [--px 1Sg] [--clitic kin] [--json] # realize_np()
|
| 278 |
+
juurio realize-verbform "<verb>" [--form va|nut|tava|tu|inf2|inf2_pass|inf3|ma] [--case Ine] [--number Plur] [--px 3] # realize_verbform()
|
| 279 |
+
|
| 280 |
+
# infrastructure (no library counterpart)
|
| 281 |
+
juurio list-ops
|
| 282 |
+
juurio batch <op> [--lemma L] [--json] < lines # one op for every stdin line
|
| 283 |
+
juurio batch [--json] < tsv # per-line "op<TAB>text[<TAB>lemma]"
|
| 284 |
+
juurio demo # curated tour (incl. OOV) — also a smoke test
|
| 285 |
+
juurio repl # interactive shell; model loaded once
|
| 286 |
+
```
|
| 287 |
+
|
| 288 |
+
`inflect` accepts a human case/number/degree phrase positionally (`juurio inflect talo mon gen`
|
| 289 |
+
→ `talojen`, `juurio inflect kaunis gen --degree comp`), or as `--case/--number/--degree` flags;
|
| 290 |
+
a positional phrase wins over the flags. `--proper`/`--no-proper` force/forbid a proper-noun reading.
|
| 291 |
+
|
| 292 |
+
`juurio repl` is a read-eval loop over every feature (model loaded once): `inflect talo mon gen`,
|
| 293 |
+
`inflect makaronilaatikko like=talojen`, `transform passive Mies osti talon.`, `analyze …`,
|
| 294 |
+
`lemmatize …`, `np …`, `verb …`, `ops`, `help`. Options are `key=value` (`px=1sg proper=true
|
| 295 |
+
pos=adjective clitic=kin like=talojen`).
|
| 296 |
+
|
| 297 |
+
Also `python -m juurio …`. Without `--json`, prints the result (warnings appended after `⚠`) and
|
| 298 |
+
sends abstentions to stderr as `[abstain: <reason>] <input>`. With `--json`, one JSON object per
|
| 299 |
+
line (`text/ok/reason/warnings/trace`). **Exit status:** `0` if every transform produced a form,
|
| 300 |
+
`2` if any abstained — so scripts can branch on it.
|
| 301 |
+
|
| 302 |
+
```console
|
| 303 |
+
$ juurio transform plural "Striimaaja nukkuu."
|
| 304 |
+
Striimaajat nukkuvat. ⚠ neural realization (unverified by FST): ['striimaaja->Striimaajat (NOUN)']
|
| 305 |
+
```
|
| 306 |
+
|
| 307 |
+
## Quality
|
| 308 |
+
|
| 309 |
+
The package is designed to **abstain rather than guess**: out-of-scope inputs return an explicit
|
| 310 |
+
abstention (`ok=False`) with a reason code instead of a speculative form, and neural OOV realizations
|
| 311 |
+
are always surfaced (in `trace["neural_unverified"]` and `warnings`) rather than emitted silently. Known
|
| 312 |
+
words are inflected to be byte-exact to the reference FST (through the exception lexicon); OOV/loan words are inflected by the neural
|
| 313 |
+
realizers, with stem-inconsistent or near-tie candidates suppressed (`oov_unverified`) rather than
|
| 314 |
+
returned.
|
| 315 |
+
|
| 316 |
+
Run `juurio demo` for a curated tour (including OOV cases) that doubles as a smoke test.
|
juurio-0.1.0-py3-none-any.whl
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:88e74693dabe579bb500bbd66dd0bfc8282821770c1afb0b05b41d719c5c42cd
|
| 3 |
+
size 182902
|
juurio/__init__.py
ADDED
|
@@ -0,0 +1,210 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""juurio — open-vocabulary Finnish morphological transforms over raw text.
|
| 2 |
+
|
| 3 |
+
Public, stable surface:
|
| 4 |
+
|
| 5 |
+
from juurio import Transformer, transform, SUPPORTED_OPS, REASONS, ABSTAIN_REASONS
|
| 6 |
+
|
| 7 |
+
t = Transformer() # loads parser + models once
|
| 8 |
+
r = t.transform("past", "Koira haukkuu pihalla.")
|
| 9 |
+
r.ok, r.text, r.reason # True, 'Koira haukkui pihalla.', 'ok'
|
| 10 |
+
|
| 11 |
+
`transform(op, text, lemma=None)` is a one-shot convenience that lazily builds and reuses a
|
| 12 |
+
process-wide `Transformer`. For anything beyond a quick call, construct a `Transformer` yourself.
|
| 13 |
+
|
| 14 |
+
Every transform returns a `Result`. `ok` is True only when a form was produced; otherwise `ok` is
|
| 15 |
+
False and `reason` is one of `ABSTAIN_REASONS`. Abstention is a first-class result, never a guess.
|
| 16 |
+
Neural realizations (forms the FST could not verify at runtime) are surfaced in `Result.warnings` and
|
| 17 |
+
`trace["neural_unverified"]` (deprecated alias `trace["neural_oov"]`) — model guesses, unverified by the
|
| 18 |
+
FST. "Unverified" is provenance (net- vs lexicon-served), not a claim the lemma is out-of-vocabulary.
|
| 19 |
+
|
| 20 |
+
Heavy deps (torch, stanza) are imported lazily, so `import juurio` is cheap; the cost is paid on
|
| 21 |
+
first `Transformer()` / `transform(...)`.
|
| 22 |
+
"""
|
| 23 |
+
from __future__ import annotations
|
| 24 |
+
|
| 25 |
+
from . import features # pure alias-normalization (no heavy deps); supplies the UNSET sentinel
|
| 26 |
+
|
| 27 |
+
# The documented public op catalog, grouped by what they do. Keys are the strings `transform`
|
| 28 |
+
# accepts; the flat set is `SUPPORTED_OPS`. (Mirrors `algebra.ALL_OPS`; this is the stable view.)
|
| 29 |
+
OP_CATALOG = {
|
| 30 |
+
"polarity/tense/mood (finite verb)": {
|
| 31 |
+
"neg": "negate (connegative + ei-auxiliary)",
|
| 32 |
+
"past": "past tense (imperfect)",
|
| 33 |
+
"pres": "present tense",
|
| 34 |
+
"cond": "conditional mood (-isi-)",
|
| 35 |
+
"perfect": "perfect (analytic olla + past participle)",
|
| 36 |
+
"potential": "potential mood (-nee-; suppletive olla -> lienee)",
|
| 37 |
+
"passive": "impersonal passive (drop subject, total object -> Nom)",
|
| 38 |
+
},
|
| 39 |
+
"compound tenses (analytic olla auxiliary + the perfect participle)": {
|
| 40 |
+
"pluperfect": "pluperfect (olin sanonut; auxiliary in the past)",
|
| 41 |
+
"cond_perfect": "conditional perfect (olisin sanonut)",
|
| 42 |
+
"pot_perfect": "potential perfect (lienen sanonut; suppletive olla -> lienee)",
|
| 43 |
+
},
|
| 44 |
+
"composites (applied as one op)": {
|
| 45 |
+
"neg+past": "", "neg+cond": "", "neg+perfect": "",
|
| 46 |
+
"passive+past": "", "passive+cond": "", "passive+perfect": "",
|
| 47 |
+
"neg+passive": "", "neg+passive+past": "", "neg+passive+cond": "",
|
| 48 |
+
"neg+passive+perfect": "",
|
| 49 |
+
},
|
| 50 |
+
"clause": {
|
| 51 |
+
"quest": "polar question (verb-fronting + -ko/-kö)",
|
| 52 |
+
},
|
| 53 |
+
"imperative paradigm (drop subject, total object -> Nom, '.' -> '!')": {
|
| 54 |
+
"imperative": "affirmative Sg2 (Mene!) — the trained flag; OOV-capable",
|
| 55 |
+
"imperative_pl2": "affirmative Pl2 (Menkää!)",
|
| 56 |
+
"imperative_sg3": "affirmative Sg3 (Menköön!)",
|
| 57 |
+
"imperative_pl1": "affirmative Pl1 (Menkäämme! — 'let us')",
|
| 58 |
+
"imperative_pl3": "affirmative Pl3 (Menkööt!)",
|
| 59 |
+
"neg_imperative": "negative Sg2 (Älä mene!) — OOV-capable",
|
| 60 |
+
"neg_imperative_pl2": "negative Pl2 (Älkää menkö!)",
|
| 61 |
+
"neg_imperative_sg3": "negative Sg3 (Älköön menkö!)",
|
| 62 |
+
"neg_imperative_pl1": "negative Pl1 (Älkäämme menkö!)",
|
| 63 |
+
"neg_imperative_pl3": "negative Pl3 (Älkööt menkö!)",
|
| 64 |
+
},
|
| 65 |
+
"number (re-agrees subject, modifiers, finite verb, copula predicate)": {
|
| 66 |
+
"plural": "singular subject NP -> plural",
|
| 67 |
+
"singular": "plural subject NP -> singular",
|
| 68 |
+
},
|
| 69 |
+
"degree (predicate adjective)": {
|
| 70 |
+
"comp": "comparative (-mpi)",
|
| 71 |
+
"super": "superlative (-in)",
|
| 72 |
+
},
|
| 73 |
+
"enclitic particles (attached to the finite verb)": {
|
| 74 |
+
"clitic_kin": "-kin / -kään (additive: 'also / either')",
|
| 75 |
+
"clitic_kaan": "-kaan / -kään",
|
| 76 |
+
"clitic_han": "-han / -hän (familiarizing)",
|
| 77 |
+
"clitic_pa": "-pa / -pä (emphatic)",
|
| 78 |
+
},
|
| 79 |
+
"possessive suffix on the subject head (preserves its case/number)": {
|
| 80 |
+
"poss_1sg": "-ni ('my': talo -> taloni, talossa -> talossani)",
|
| 81 |
+
"poss_2sg": "-si ('your sg')",
|
| 82 |
+
"poss_3": "-nsa / -Vn ('his/her/their'; 3rd person, Sg & Pl)",
|
| 83 |
+
"poss_1pl": "-mme ('our')",
|
| 84 |
+
"poss_2pl": "-nne ('your pl')",
|
| 85 |
+
},
|
| 86 |
+
}
|
| 87 |
+
|
| 88 |
+
SUPPORTED_OPS: tuple[str, ...] = tuple(op for group in OP_CATALOG.values() for op in group)
|
| 89 |
+
|
| 90 |
+
_singleton = None
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
def transform(op: str, text: str, lemma: str | None = None, sent_index: int = 0):
|
| 94 |
+
"""One-shot convenience: transform `text` with `op`, reusing a process-wide `Transformer`.
|
| 95 |
+
|
| 96 |
+
`lemma` overrides the parser's lemma for the main verb (use when a loan stem is mis-lemmatized).
|
| 97 |
+
Returns a `Result`. Builds the (heavy) Transformer on first call; subsequent calls reuse it."""
|
| 98 |
+
global _singleton
|
| 99 |
+
if _singleton is None:
|
| 100 |
+
from .pipeline import Transformer
|
| 101 |
+
_singleton = Transformer()
|
| 102 |
+
return _singleton.transform(op, text, lemma=lemma, sent_index=sent_index)
|
| 103 |
+
|
| 104 |
+
|
| 105 |
+
def realize_np(np: str, case: str = "Nom", number: str = "Sing",
|
| 106 |
+
px: str = "None", clitic: str | None = None):
|
| 107 |
+
"""One-shot phrase-level nominal morphology (NOT a clause transform): inflect an NP — a bare
|
| 108 |
+
lemma or a short `adj* noun` phrase — into the target Case/Number, optional possessive + clitic.
|
| 109 |
+
Open-vocab (FST->neural fallback). Returns a `Result`. Reuses a process-wide `Transformer`.
|
| 110 |
+
|
| 111 |
+
realize_np("talo", case="Ine", number="Plur", px="1Sg").text # 'taloissani'
|
| 112 |
+
realize_np("striimaaja", case="Ade", number="Plur", px="1Sg", clitic="kin").text
|
| 113 |
+
realize_np("iso talo", case="Ine").text # 'isossa talossa'
|
| 114 |
+
"""
|
| 115 |
+
global _singleton
|
| 116 |
+
if _singleton is None:
|
| 117 |
+
from .pipeline import Transformer
|
| 118 |
+
_singleton = Transformer()
|
| 119 |
+
return _singleton.realize_np(np, case=case, number=number, px=px, clitic=clitic)
|
| 120 |
+
|
| 121 |
+
|
| 122 |
+
def realize_verbform(verb: str, form: str = "va", case: str = "Nom", number: str = "Sing",
|
| 123 |
+
px: str = "None", lemma: str | None = None):
|
| 124 |
+
"""One-shot non-finite verb morphology (NOT a clause transform): inflect a VERB into a participle
|
| 125 |
+
or infinitive, optionally with a possessive suffix `px`. Reuses a process-wide `Transformer`.
|
| 126 |
+
`form` is one of `VERBFORMS`.
|
| 127 |
+
|
| 128 |
+
realize_verbform("lukea", form="va", case="Ine").text # 'lukevassa' (present participle)
|
| 129 |
+
realize_verbform("mennä", form="inf3", case="Ade").text # 'menemällä' (3rd inf adessive)
|
| 130 |
+
realize_verbform("kuulla", form="tu", case="Par", px="3").text # 'kuultuansa' (temporal: 'after hearing')
|
| 131 |
+
"""
|
| 132 |
+
global _singleton
|
| 133 |
+
if _singleton is None:
|
| 134 |
+
from .pipeline import Transformer
|
| 135 |
+
_singleton = Transformer()
|
| 136 |
+
return _singleton.realize_verbform(verb, form=form, case=case, number=number, px=px, lemma=lemma)
|
| 137 |
+
|
| 138 |
+
|
| 139 |
+
# the non-finite verb forms `realize_verbform` accepts (mirrors algebra.VERBFORMS without importing it)
|
| 140 |
+
VERBFORMS: tuple[str, ...] = ("va", "nut", "tava", "tu", "inf2", "inf2_pass", "inf3", "ma")
|
| 141 |
+
|
| 142 |
+
|
| 143 |
+
def inflect(word: str, case=features.UNSET, number=features.UNSET, degree=features.UNSET, *,
|
| 144 |
+
px=features.UNSET, clitic: str | None = None, proper: bool | None = None,
|
| 145 |
+
pos: str | None = None, like: str | None = None):
|
| 146 |
+
"""One-shot friendly nominal inflection (the taivutin-style front door): inflect a single word
|
| 147 |
+
or short phrase into a target case/number with human-readable aliases, a proper-noun toggle, and
|
| 148 |
+
inflect-by-example (`like=`). Returns a `Result`. See `Transformer.inflect` for the full contract.
|
| 149 |
+
|
| 150 |
+
inflect("talo", "mon gen").text # 'talojen'
|
| 151 |
+
inflect("makaronilaatikko", like="talojen").text # 'makaronilaatikkojen'
|
| 152 |
+
"""
|
| 153 |
+
return _get().inflect(word, case, number, degree, px=px, clitic=clitic, proper=proper,
|
| 154 |
+
pos=pos, like=like)
|
| 155 |
+
|
| 156 |
+
|
| 157 |
+
def analyze(text: str):
|
| 158 |
+
"""One-shot analysis: parse `text` into sentences of `TokenAnalysis` (lemma, UPOS, head, deprel,
|
| 159 |
+
UD feats) — the inverse of generation. Reuses a process-wide `Transformer`."""
|
| 160 |
+
return _get().analyze(text)
|
| 161 |
+
|
| 162 |
+
|
| 163 |
+
def lemmatize(text: str):
|
| 164 |
+
"""One-shot lemmatization: the lemma of every token in `text`, flat across sentences."""
|
| 165 |
+
return _get().lemmatize(text)
|
| 166 |
+
|
| 167 |
+
|
| 168 |
+
# Building a Transformer imports the INTERNAL `juurio.analyze` subpackage (the parser's hybrid
|
| 169 |
+
# analysis channel). Importing a submodule binds it as an attribute of this package, shadowing the
|
| 170 |
+
# public `analyze`/`lemmatize` convenience functions above (`import juurio; juurio.analyze(...)`
|
| 171 |
+
# would then hit the module, not the function). We capture the originals here and re-bind them after
|
| 172 |
+
# any Transformer construction (see `_restore_public_names`, called from `Transformer.__init__`).
|
| 173 |
+
# A rename of the subpackage was rejected as riskier (checkpoints may pickle `juurio.analyze.*`).
|
| 174 |
+
# We capture and restore `inflect` alongside analyze/lemmatize as a cheap safeguard against any future
|
| 175 |
+
# re-binding.
|
| 176 |
+
_ORIG_ANALYZE = analyze
|
| 177 |
+
_ORIG_LEMMATIZE = lemmatize
|
| 178 |
+
_ORIG_INFLECT = inflect
|
| 179 |
+
|
| 180 |
+
|
| 181 |
+
def _restore_public_names() -> None:
|
| 182 |
+
"""Re-bind the public `analyze`/`lemmatize` functions, undoing the shadowing caused by importing
|
| 183 |
+
the internal `juurio.analyze` subpackage during Transformer/parser construction (`inflect` is
|
| 184 |
+
restored defensively too)."""
|
| 185 |
+
import sys
|
| 186 |
+
mod = sys.modules[__name__]
|
| 187 |
+
mod.analyze = _ORIG_ANALYZE
|
| 188 |
+
mod.lemmatize = _ORIG_LEMMATIZE
|
| 189 |
+
mod.inflect = _ORIG_INFLECT
|
| 190 |
+
|
| 191 |
+
|
| 192 |
+
def _get():
|
| 193 |
+
"""Lazily build and reuse the process-wide `Transformer` (defers the heavy parser/model load)."""
|
| 194 |
+
global _singleton
|
| 195 |
+
if _singleton is None:
|
| 196 |
+
from .pipeline import Transformer
|
| 197 |
+
_singleton = Transformer()
|
| 198 |
+
return _singleton
|
| 199 |
+
|
| 200 |
+
|
| 201 |
+
def __getattr__(name): # lazy re-export so `import juurio` doesn't pull in torch/stanza
|
| 202 |
+
if name in ("Transformer", "Result", "TokenAnalysis", "REASONS", "ABSTAIN_REASONS"):
|
| 203 |
+
from . import pipeline
|
| 204 |
+
return getattr(pipeline, name)
|
| 205 |
+
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
|
| 206 |
+
|
| 207 |
+
|
| 208 |
+
__all__ = ["Transformer", "Result", "TokenAnalysis", "transform", "inflect", "analyze", "lemmatize",
|
| 209 |
+
"realize_np", "realize_verbform", "VERBFORMS", "features",
|
| 210 |
+
"SUPPORTED_OPS", "OP_CATALOG", "REASONS", "ABSTAIN_REASONS"]
|
juurio/__main__.py
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import sys
|
| 2 |
+
|
| 3 |
+
from .cli import main
|
| 4 |
+
|
| 5 |
+
sys.exit(main())
|
juurio/algebra.py
ADDED
|
@@ -0,0 +1,1438 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Rule-based engine for a family of Finnish grammatical clause transformations.
|
| 2 |
+
|
| 3 |
+
Given a parsed clause and an operation name, the engine applies the transformation by
|
| 4 |
+
deterministic rules, preserving the unchanged material, and realizes surface forms via the
|
| 5 |
+
Giella FST (known words) or the neural realizers (out-of-vocabulary words). It handles
|
| 6 |
+
transformations of *different kinds*:
|
| 7 |
+
neg — clause restructuring: insert `ei`, connegative/participle, object->partitive
|
| 8 |
+
quest — constituent movement: front the finite verb + `-ko/-kö` clitic (harmony)
|
| 9 |
+
comp — local morphology: positive adjective -> comparative
|
| 10 |
+
plus tense/mood, passive, the imperative paradigm, number±agreement, degree, enclitics,
|
| 11 |
+
possessives, and `+`-composed nominal morphology.
|
| 12 |
+
|
| 13 |
+
`OPS` / `EXT_OPS` / `ALL_OPS` are the op tables; `realize_np_form` and `realize_verbform_form`
|
| 14 |
+
are the phrase-level utilities (see juurio.realizer / verbforms / finiteverb for the realizers).
|
| 15 |
+
"""
|
| 16 |
+
|
| 17 |
+
from __future__ import annotations
|
| 18 |
+
|
| 19 |
+
import os
|
| 20 |
+
|
| 21 |
+
from .device import DEV
|
| 22 |
+
from .morphforms import _main_verb, swap_initial, target_str as _tgt
|
| 23 |
+
|
| 24 |
+
CKPT = os.path.join(os.path.dirname(__file__), "..", "checkpoints", "algebra.pt")
|
| 25 |
+
BACK = set("aou") # back vowels -> -ko ; else -kö
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
# ===================== unified clause-feature core =====================
|
| 29 |
+
# The verb-feature ops (neg/past/pres/cond/passive) and their compositions all set
|
| 30 |
+
# features on the main clause and re-realize the verb cluster once, so morphological
|
| 31 |
+
# interactions are handled automatically (past+neg -> `ei ostanut`; passive+neg ->
|
| 32 |
+
# `ei osteta`). Object/subject case is then derived from the FINAL feature set.
|
| 33 |
+
|
| 34 |
+
NUMTAG = {"Sing": "Sg", "Plur": "Pl"}
|
| 35 |
+
AGREE = {"amod", "nummod", "det", "acl"}
|
| 36 |
+
_EI_FORMS = {"ei", "en", "et", "emme", "ette", "eivät"}
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
# onsets for nonce augmentation: single consonants + loan-like clusters. Word-initial only,
|
| 40 |
+
# so always paradigm-safe (Finnish gradation never touches the initial consonant).
|
| 41 |
+
ONSETS = list("kptsvnmlrhjdgbf") + ["tr", "kr", "pr", "gr", "str", "sp", "st", "sk",
|
| 42 |
+
"pl", "kl", "fl", "br", "dr"]
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
def _rand_onset(rng, init):
|
| 46 |
+
return rng.choice([o for o in ONSETS if o[0] != init])
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
def _vclass(lemma):
|
| 50 |
+
"""Coarse Finnish verb conjugation type from the lemma's infinitive ending (productive).
|
| 51 |
+
Used only to rebalance nonce augmentation toward rare-but-regular classes (4/5/6),
|
| 52 |
+
which loanwords (googlata=4, tinderöidä=6) fall into and which the corpus under-samples."""
|
| 53 |
+
l = lemma.lower()
|
| 54 |
+
if l.endswith(("oida", "öidä")):
|
| 55 |
+
return 6
|
| 56 |
+
if l.endswith(("ella", "ellä")):
|
| 57 |
+
return 3
|
| 58 |
+
if l.endswith(("ata", "ätä", "ota", "ötä", "uta", "ytä", "eta", "etä", "ita", "itä")):
|
| 59 |
+
return 4
|
| 60 |
+
if l.endswith(("ta", "tä", "da", "dä")):
|
| 61 |
+
return 2
|
| 62 |
+
return 1
|
| 63 |
+
|
| 64 |
+
|
| 65 |
+
# extra nonce multiplier per class — inversely tracks corpus frequency (type 6 ~1% of tokens)
|
| 66 |
+
_CLASS_BOOST = {6: 4, 4: 2, 3: 2, 2: 1, 1: 1}
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
# --- lemma anchoring (product path only) ------------------------------------
|
| 70 |
+
# A single inflected form can be class-ambiguous: present `googlaa` (type-4 <googlata)
|
| 71 |
+
# is surface-identical to type-1 `maksaa`, but their pasts differ (googlasi vs maksoi).
|
| 72 |
+
# Prepending the dictionary form (`googlata § ...`) supplies the disambiguating signal a
|
| 73 |
+
# speaker always has — the model then only has to apply the PRODUCTIVE class rule. For
|
| 74 |
+
# nonce training we prepend the ORIGINAL (un-swapped) lemma, so its onset deliberately
|
| 75 |
+
# mismatches the swapped surface: that forces "class from lemma ending, stem from surface"
|
| 76 |
+
# rather than copying the lemma wholesale. The FST cannot do this (closed lexicon: it
|
| 77 |
+
# returns nothing for striimata/tubettaa/tinderöidä even GIVEN the lemma) — so it is a
|
| 78 |
+
# real open-vocabulary differentiator, not a reshuffle of rule-based capability.
|
| 79 |
+
LEMSEP = "§"
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
def _main_lemma(s, op):
|
| 83 |
+
if op in ("comp", "super"):
|
| 84 |
+
adjs = [t for t in s if t["upos"] == "ADJ" and t["feats"].get("Degree", "Pos") == "Pos"]
|
| 85 |
+
return adjs[0]["lemma"] if adjs else None
|
| 86 |
+
v, _ = _main_verb(s)
|
| 87 |
+
return v["lemma"] if v else None
|
| 88 |
+
|
| 89 |
+
|
| 90 |
+
def _wrap_src(src_str, s, op):
|
| 91 |
+
lem = _main_lemma(s, op)
|
| 92 |
+
return f"{lem} {LEMSEP} {src_str}" if lem else src_str
|
| 93 |
+
|
| 94 |
+
|
| 95 |
+
def _maybe_wrap(stoi, src_str, s, op):
|
| 96 |
+
"""Wrap with the lemma prefix iff the loaded model is lemma-anchored (its vocab has §)."""
|
| 97 |
+
return _wrap_src(src_str, s, op) if LEMSEP in stoi else src_str
|
| 98 |
+
|
| 99 |
+
|
| 100 |
+
def _clitic(form):
|
| 101 |
+
return "ko" if any(c in BACK for c in form.lower()) else "kö"
|
| 102 |
+
|
| 103 |
+
|
| 104 |
+
def _children(sent):
|
| 105 |
+
ch = {}
|
| 106 |
+
for t in sent:
|
| 107 |
+
ch.setdefault(t["head"], []).append(t)
|
| 108 |
+
return ch
|
| 109 |
+
|
| 110 |
+
|
| 111 |
+
def _subtree_ids(sent, root_id):
|
| 112 |
+
ch = _children(sent)
|
| 113 |
+
out, stack = set(), [root_id]
|
| 114 |
+
while stack:
|
| 115 |
+
i = stack.pop(); out.add(i)
|
| 116 |
+
for c in ch.get(i, []):
|
| 117 |
+
stack.append(c["id"])
|
| 118 |
+
return out
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
def clause(sent):
|
| 122 |
+
"""(verb_token, is_cop, base_features) or None. Skips already-negated clauses
|
| 123 |
+
(so negation gold never double-negates)."""
|
| 124 |
+
verb, is_cop = _main_verb(sent)
|
| 125 |
+
if verb is None:
|
| 126 |
+
return None
|
| 127 |
+
if any(t["lemma"] == "ei" or t["form"].lower() in _EI_FORMS for t in sent):
|
| 128 |
+
return None
|
| 129 |
+
f = verb["feats"]
|
| 130 |
+
base = dict(voice=f.get("Voice", "Act"), mood=f.get("Mood", "Ind"),
|
| 131 |
+
tense="Prt" if f.get("Tense") == "Past" else "Prs",
|
| 132 |
+
person=f.get("Person", "3"), number=f.get("Number", "Sing"), pol="Aff")
|
| 133 |
+
return verb, is_cop, base
|
| 134 |
+
|
| 135 |
+
|
| 136 |
+
_IMPERATIVE_MOODS = ("Ind", "Imp")
|
| 137 |
+
|
| 138 |
+
|
| 139 |
+
def _imperative_source_verb(sent):
|
| 140 |
+
"""Root finite Act verb of an imperative-able clause, or None. Unlike `clause()` this also
|
| 141 |
+
accepts an already-imperative root (`osta …`, Mood=Imp), since the imperative ops generate from
|
| 142 |
+
the lemma regardless of source mood. Rejects copular clauses (root is the predicate, not a verb),
|
| 143 |
+
passives, and already-negated clauses. The 2nd-person/subjectless scope restriction is applied
|
| 144 |
+
separately by `imperative_in_scope`."""
|
| 145 |
+
root = next((t for t in sent if t["head"] == 0), None)
|
| 146 |
+
if root is None or root["upos"] not in ("VERB", "AUX"):
|
| 147 |
+
return None
|
| 148 |
+
f = root["feats"]
|
| 149 |
+
if (f.get("VerbForm") != "Fin" or f.get("Voice", "Act") != "Act"
|
| 150 |
+
or f.get("Mood", "Ind") not in _IMPERATIVE_MOODS):
|
| 151 |
+
return None
|
| 152 |
+
if any(t["lemma"] == "ei" or t["form"].lower() in _EI_FORMS for t in sent):
|
| 153 |
+
return None
|
| 154 |
+
return root
|
| 155 |
+
|
| 156 |
+
|
| 157 |
+
def _stem_overlap(a, b):
|
| 158 |
+
"""Length of the shared leading stem of two surface strings (case-insensitive)."""
|
| 159 |
+
n = 0
|
| 160 |
+
for x, y in zip(a.lower(), b.lower()):
|
| 161 |
+
if x != y:
|
| 162 |
+
break
|
| 163 |
+
n += 1
|
| 164 |
+
return n
|
| 165 |
+
|
| 166 |
+
|
| 167 |
+
_OBJ_CASES = {"Par", "Gen", "Acc"} # core grammatical-object cases; Nom strays are subject candidates
|
| 168 |
+
_NOMINAL = {"NOUN", "PROPN", "PRON", "NUM", "ADJ"} # tokens that can be a clause argument
|
| 169 |
+
|
| 170 |
+
|
| 171 |
+
def repair_root_verb_with_lemma(sent, lemma):
|
| 172 |
+
"""Hint repair for the ambiguous-loan-verb mis-tag. When a serving `lemma` hint names the clause's
|
| 173 |
+
main verb but the (neural) parser tagged the clause root as a non-verb — e.g. 'striimaa' →
|
| 174 |
+
NOUN/striima/Par instead of VERB/striimata, which makes `clause()` return None — re-tag that token as
|
| 175 |
+
a finite Act/Ind verb and normalize the broken tree into a single-root clause, so clause detection
|
| 176 |
+
finds it and downstream object/number/passive ops see real structure. Returns True iff a repair was
|
| 177 |
+
applied.
|
| 178 |
+
|
| 179 |
+
The real parse of this failure is MULTI-ROOT: the parser splits the clause, leaving both the
|
| 180 |
+
mis-tagged predicate and its argument(s) at head==0 (e.g. 'striimaa' AND 'peliä'). So the repair:
|
| 181 |
+
1. promotes the mis-tagged root to a finite present Act/Ind verb;
|
| 182 |
+
2. re-attaches every OTHER root to it (oblique-cased -> obj, else obl), guaranteeing a SINGLE root
|
| 183 |
+
(without this, `in_scope()` would accept a malformed multi-root tree and object/number/passive
|
| 184 |
+
ops would emit from broken structure rather than from the real arguments);
|
| 185 |
+
3. recovers the subject (an existing nsubj, else a Nominative nominal dependent) and derives finite
|
| 186 |
+
agreement from it.
|
| 187 |
+
|
| 188 |
+
Conservative by design, so it cannot fabricate a clause from a genuinely out-of-scope input. It fires
|
| 189 |
+
ONLY when ALL hold: there is currently no finite main verb (correct clauses are untouched); the
|
| 190 |
+
sentence has at least one OTHER content (non-PUNCT) token (a bare fragment `Striimaaja.` stays out of
|
| 191 |
+
scope); the chosen token is an open-class ROOT (head==0) whose surface stem-matches the hint (≥3
|
| 192 |
+
chars) AND whose Case is NOT Nominative; and there is a recoverable nominal ARGUMENT (a stray-root
|
| 193 |
+
nominal that becomes the object, or a Nominative nominal that becomes the subject). The non-Nominative
|
| 194 |
+
guard is the mis-tag fingerprint — the parser reads the verb's ending as a case ending and assigns an
|
| 195 |
+
oblique case (Par/Gen/...), whereas a genuine noun/title fragment head (`Striima peli.`) is
|
| 196 |
+
Nominative. The argument requirement rejects argumentless oblique fragments (`Striimissä tänään.`,
|
| 197 |
+
whose only companion is an adverb) that the non-Nominative guard alone would not. Tense defaults to
|
| 198 |
+
present (the loan-verb base case); a genuinely past surface is normally tagged correctly by the parser
|
| 199 |
+
and does not reach this path."""
|
| 200 |
+
if not lemma or _main_verb(sent)[0] is not None:
|
| 201 |
+
return False
|
| 202 |
+
if len([t for t in sent if t["upos"] != "PUNCT"]) < 2: # bare fragment -> stays out of scope
|
| 203 |
+
return False
|
| 204 |
+
roots = [t for t in sent if t["head"] == 0]
|
| 205 |
+
root = next((t for t in roots
|
| 206 |
+
if t["upos"] in ("NOUN", "ADJ", "PROPN")
|
| 207 |
+
and t["feats"].get("Case") != "Nom"
|
| 208 |
+
and _stem_overlap(t["form"], lemma) >= 3), None)
|
| 209 |
+
if root is None:
|
| 210 |
+
return False
|
| 211 |
+
rid = root["id"]
|
| 212 |
+
# A real clause has at least one nominal ARGUMENT: another (stray-root) nominal that becomes the
|
| 213 |
+
# object, or a Nominative nominal that can be the subject. The non-Nominative root guard alone does
|
| 214 |
+
# not prove clausehood — an argumentless oblique fragment whose only companion is an adverb
|
| 215 |
+
# (`Striimissä tänään.`) has neither, so it must stay out of scope.
|
| 216 |
+
stray_args = [t for t in roots if t is not root and t["upos"] in _NOMINAL]
|
| 217 |
+
subj = next((t for t in sent if t["deprel"] in ("nsubj", "nsubj:cop")), None)
|
| 218 |
+
if subj is None: # a Nominative nominal dependent of the root
|
| 219 |
+
subj = next((t for t in sent if t["head"] == rid and t is not root
|
| 220 |
+
and t["upos"] in _NOMINAL and t["feats"].get("Case") == "Nom"), None)
|
| 221 |
+
if not stray_args and subj is None:
|
| 222 |
+
return False
|
| 223 |
+
root["upos"], root["lemma"], root["deprel"] = "VERB", lemma, "root"
|
| 224 |
+
for t in roots: # collapse stray roots into verb arguments
|
| 225 |
+
if t is root:
|
| 226 |
+
continue
|
| 227 |
+
t["head"] = rid
|
| 228 |
+
t["deprel"] = "obj" if t["feats"].get("Case") in _OBJ_CASES else "obl"
|
| 229 |
+
if subj is not None: # promote the recovered subject in place
|
| 230 |
+
subj["deprel"] = "nsubj"
|
| 231 |
+
root["feats"] = {"VerbForm": "Fin", "Voice": "Act", "Mood": "Ind", "Tense": "Pres",
|
| 232 |
+
"Person": subj["feats"].get("Person", "3") if subj else "3",
|
| 233 |
+
"Number": subj["feats"].get("Number", "Sing") if subj else "Sing"}
|
| 234 |
+
return True
|
| 235 |
+
|
| 236 |
+
|
| 237 |
+
def verb_cluster(lemma, voice, mood, tense, pol, person, number):
|
| 238 |
+
"""Surface token(s) for the finite verb under the given features, or None.
|
| 239 |
+
|
| 240 |
+
PERFECT (tense=Prf) is ANALYTIC: olla-auxiliary + past participle. The participles
|
| 241 |
+
(`ostanut`/`ostettu`) and aux (`ole`/`on`) are tokens the model meets in other ops,
|
| 242 |
+
so an unseen perfect *composite* is a recombination of seen surface forms — unlike
|
| 243 |
+
the fused synthetic cells (`ostettaisi`) that collapse features into one novel form.
|
| 244 |
+
"""
|
| 245 |
+
if not _NETLEX_BACKEND:
|
| 246 |
+
return None
|
| 247 |
+
return _netlex_verb_cluster(lemma, voice, mood, tense, pol, person, number)
|
| 248 |
+
|
| 249 |
+
|
| 250 |
+
# --- Giella-free OOV verb realization: the OOV verb's finite cells come from finiteverb.pt and its
|
| 251 |
+
# past participles from verbform.pt; the closed-class scaffolding (olla auxiliary, `ei`/`älä`
|
| 252 |
+
# negation particles) is served from the hardcoded tables since those words are ALWAYS known. The OOV
|
| 253 |
+
# lemma's own forms come from the neural nets.
|
| 254 |
+
# Set at inference by `pipeline.Transformer`.
|
| 255 |
+
_FINITEVERB_BUNDLE = None # juurio.finiteverb.load() bundle (None => OOV verbs abstain)
|
| 256 |
+
|
| 257 |
+
|
| 258 |
+
def _oov_finite(lemma, voice, mood, tense, person):
|
| 259 |
+
"""One finite OOV verb form via finiteverb.pt, or None (no bundle / empty / stem-inconsistent).
|
| 260 |
+
`tense` is honoured only for the indicative (Cond/Pot/Imprt are tenseless), matching finiteverb's
|
| 261 |
+
cell space."""
|
| 262 |
+
if _FINITEVERB_BUNDLE is None:
|
| 263 |
+
return None
|
| 264 |
+
from . import finiteverb
|
| 265 |
+
cell = {"Voice": voice, "Mood": mood,
|
| 266 |
+
"Tense": (tense if mood == "Ind" else None), "Person": person}
|
| 267 |
+
out = finiteverb.decode_one(_FINITEVERB_BUNDLE, lemma, cell)
|
| 268 |
+
return out if (out and _stem_consistent(lemma, out)) else None
|
| 269 |
+
|
| 270 |
+
|
| 271 |
+
def _neural_imprt(lemma, tag):
|
| 272 |
+
"""Imperative OOV form (Mood=Imprt) via finiteverb.pt, or None. `tag` is a person slot ('Sg2',
|
| 273 |
+
'Pl2', …) for the affirmative or 'ConNeg' for the -kO connegative (`älkää …kö`). Records the
|
| 274 |
+
realized form in `_NEURAL_OOV` so the transform trace surfaces it — the same neural-realization
|
| 275 |
+
audit invariant `_realize_form` upholds (pipeline surfaces it as
|
| 276 |
+
`trace["neural_unverified"]`, with deprecated alias `trace["neural_oov"]`)."""
|
| 277 |
+
v = _oov_finite(lemma, "Act", "Imprt", None, tag)
|
| 278 |
+
if v:
|
| 279 |
+
_NEURAL_OOV.append(f"{lemma}->{v} (verb)")
|
| 280 |
+
return v
|
| 281 |
+
|
| 282 |
+
|
| 283 |
+
# --- net+lexicon KNOWN-word generation: serving of in-vocabulary words --------
|
| 284 |
+
# The dispatch is lexicon-first (`_GENLEX`, byte-exact on the cells the nets get wrong) → neural net →
|
| 285 |
+
# abstain(None). Closed-class suppletive words (olla auxiliary, the `ei` negative series) cannot be in
|
| 286 |
+
# the lexicon (it enumerates only open-class subsystems) and the nets aren't trained on them, so they
|
| 287 |
+
# are served from these hardcoded standard-Finnish tables.
|
| 288 |
+
_OLLA = { # olla auxiliary / olla-as-main-verb, keyed (Voice, Mood, Tense|None, Person|Sg/Pl|None)
|
| 289 |
+
("Act", "Ind", "Prs", "Sg1"): "olen", ("Act", "Ind", "Prt", "Sg1"): "olin",
|
| 290 |
+
("Act", "Cond", None, "Sg1"): "olisin", ("Act", "Pot", None, "Sg1"): "lienen",
|
| 291 |
+
("Act", "Ind", "Prs", "Sg2"): "olet", ("Act", "Ind", "Prt", "Sg2"): "olit",
|
| 292 |
+
("Act", "Cond", None, "Sg2"): "olisit", ("Act", "Pot", None, "Sg2"): "lienet",
|
| 293 |
+
("Act", "Ind", "Prs", "Sg3"): "on", ("Act", "Ind", "Prt", "Sg3"): "oli",
|
| 294 |
+
("Act", "Cond", None, "Sg3"): "olisi", ("Act", "Pot", None, "Sg3"): "lienee",
|
| 295 |
+
("Act", "Ind", "Prs", "Pl1"): "olemme", ("Act", "Ind", "Prt", "Pl1"): "olimme",
|
| 296 |
+
("Act", "Cond", None, "Pl1"): "olisimme", ("Act", "Pot", None, "Pl1"): "lienemme",
|
| 297 |
+
("Act", "Ind", "Prs", "Pl2"): "olette", ("Act", "Ind", "Prt", "Pl2"): "olitte",
|
| 298 |
+
("Act", "Cond", None, "Pl2"): "olisitte", ("Act", "Pot", None, "Pl2"): "lienette",
|
| 299 |
+
("Act", "Ind", "Prs", "Pl3"): "ovat", ("Act", "Ind", "Prt", "Pl3"): "olivat",
|
| 300 |
+
("Act", "Cond", None, "Pl3"): "olisivat", ("Act", "Pot", None, "Pl3"): "lienevät",
|
| 301 |
+
("Act", "PrfPrc", None, "Sg"): "ollut", ("Act", "PrfPrc", None, "Pl"): "olleet",
|
| 302 |
+
("Act", "Cond", "ConNeg", None): "olisi", ("Act", "Pot", "ConNeg", None): "liene",
|
| 303 |
+
}
|
| 304 |
+
_EI = {("Sing", "1"): "en", ("Sing", "2"): "et", ("Sing", "3"): "ei",
|
| 305 |
+
("Plur", "1"): "emme", ("Plur", "2"): "ette", ("Plur", "3"): "eivät"}
|
| 306 |
+
|
| 307 |
+
|
| 308 |
+
def _netlex_ei(number, person):
|
| 309 |
+
"""Negative auxiliary (`en/et/ei/emme/ette/eivät`) — closed-class table, FST-identical."""
|
| 310 |
+
return _EI.get((number, person), "ei")
|
| 311 |
+
|
| 312 |
+
|
| 313 |
+
def _netlex_npart(lemma, voice, number):
|
| 314 |
+
"""Past participle (Act `nut`/Pss `tu`): lexicon-first then the verbform net, or None. The serving
|
| 315 |
+
analogue of `_npart` WITHOUT the OOV-safety abstain (we want the net's known-word form)."""
|
| 316 |
+
if lemma == "olla" and voice == "Act": # suppletive olla participle from the closed-class
|
| 317 |
+
return _OLLA.get(("Act", "PrfPrc", None, "Pl" if number == "Plur" else "Sg")) # ollut/olleet
|
| 318 |
+
form = "nut" if voice == "Act" else "tu"
|
| 319 |
+
num = "Plur" if (voice == "Act" and number == "Plur") else "Sing"
|
| 320 |
+
if _GENLEX is not None:
|
| 321 |
+
hit = _GENLEX.lookup_nonfin(lemma, form, "Nom", num)
|
| 322 |
+
if hit is not None:
|
| 323 |
+
return hit
|
| 324 |
+
if _VERBFORM_REALIZER is not None:
|
| 325 |
+
out, _conf = _VERBFORM_REALIZER(lemma, form, "Nom", num)
|
| 326 |
+
if out:
|
| 327 |
+
return out
|
| 328 |
+
return None
|
| 329 |
+
|
| 330 |
+
|
| 331 |
+
def _netlex_finite(lemma, voice, mood, tense, person):
|
| 332 |
+
"""One finite form, Giella-free: olla via the closed-class table; every other lemma lexicon-first
|
| 333 |
+
then the finiteverb net. `person` is the finiteverb Person slot ('Sg3'/'Pe4'/'ConNeg'…). No
|
| 334 |
+
stem-consistency abstain (unlike `_oov_finite`) — for known words we serve the net's form."""
|
| 335 |
+
if lemma == "olla" and voice == "Act":
|
| 336 |
+
f = _OLLA.get(("Act", mood, (tense if mood == "Ind" else None), person))
|
| 337 |
+
if f is not None:
|
| 338 |
+
return f
|
| 339 |
+
cell = {"Voice": voice, "Mood": mood,
|
| 340 |
+
"Tense": (tense if mood == "Ind" else None), "Person": person}
|
| 341 |
+
if _GENLEX is not None: # lexicon FIRST — covers cells the net can't, incl.
|
| 342 |
+
hit = _GENLEX.lookup_finite(lemma, cell) # passive connegative (distilled from the FST)
|
| 343 |
+
if hit is not None:
|
| 344 |
+
return hit
|
| 345 |
+
# finiteverb has NO passive connegative cell (passive is only Pe4 — finiteverb.CELLS), so a
|
| 346 |
+
# Pss+ConNeg request the lexicon did NOT cover is out-of-distribution for the net; abstain (None)
|
| 347 |
+
# rather than emit garbage. Must come AFTER the lexicon lookup or shipped forms are dead. The
|
| 348 |
+
# passive PAST negative is unaffected (it is the tu-participle via `_netlex_npart`).
|
| 349 |
+
if voice == "Pss" and person == "ConNeg":
|
| 350 |
+
return None
|
| 351 |
+
if _FINITEVERB_BUNDLE is None:
|
| 352 |
+
return None
|
| 353 |
+
from . import finiteverb
|
| 354 |
+
return finiteverb.decode_one(_FINITEVERB_BUNDLE, lemma, cell)
|
| 355 |
+
|
| 356 |
+
|
| 357 |
+
def _netlex_perfect_aux(voice, mood, tense, pol, person, number):
|
| 358 |
+
"""olla auxiliary for the perfect family — the closed-class `_OLLA`/`_EI` analogue of `_perfect_aux`
|
| 359 |
+
(byte-identical to the FST). The lexical participle is supplied separately by `_netlex_npart`."""
|
| 360 |
+
num = NUMTAG.get(number, "Sg")
|
| 361 |
+
pers = f"{num}{person}" if voice == "Act" else "Sg3" # passive auxiliary is impersonal
|
| 362 |
+
if pol == "Aff":
|
| 363 |
+
if tense == "Pqp":
|
| 364 |
+
aux = _OLLA.get(("Act", "Ind", "Prt", pers))
|
| 365 |
+
elif mood == "Cond":
|
| 366 |
+
aux = _OLLA.get(("Act", "Cond", None, pers))
|
| 367 |
+
elif mood == "Pot":
|
| 368 |
+
aux = _OLLA.get(("Act", "Pot", None, pers))
|
| 369 |
+
else:
|
| 370 |
+
aux = _OLLA.get(("Act", "Ind", "Prs", pers))
|
| 371 |
+
return [aux] if aux else None
|
| 372 |
+
ei = "ei" if voice == "Pss" else _netlex_ei(number, person)
|
| 373 |
+
if tense == "Pqp":
|
| 374 |
+
pl = "Pl" if (voice == "Act" and number == "Plur") else "Sg"
|
| 375 |
+
neg = _OLLA.get(("Act", "PrfPrc", None, pl))
|
| 376 |
+
elif mood == "Cond":
|
| 377 |
+
neg = _OLLA.get(("Act", "Cond", "ConNeg", None))
|
| 378 |
+
elif mood == "Pot":
|
| 379 |
+
neg = _OLLA.get(("Act", "Pot", "ConNeg", None))
|
| 380 |
+
else:
|
| 381 |
+
neg = "ole"
|
| 382 |
+
return [ei, neg] if (ei and neg) else None
|
| 383 |
+
|
| 384 |
+
|
| 385 |
+
def _netlex_verb_cluster(lemma, voice, mood, tense, pol, person, number):
|
| 386 |
+
"""Giella-free finite verb cluster — structurally identical to `verb_cluster`, but every FST call is
|
| 387 |
+
a net+lexicon (`_netlex_finite`/`_netlex_npart`) or closed-class-table (`_netlex_perfect_aux`/`_EI`)
|
| 388 |
+
lookup. Returns the token list or None. Served when `_NETLEX_BACKEND` is on."""
|
| 389 |
+
num = NUMTAG.get(number, "Sg")
|
| 390 |
+
if tense in ("Prf", "Pqp"):
|
| 391 |
+
part = _netlex_npart(lemma, voice, number)
|
| 392 |
+
if not part:
|
| 393 |
+
return None
|
| 394 |
+
aux = _netlex_perfect_aux(voice, mood, tense, pol, person, number)
|
| 395 |
+
return aux + [part] if aux else None
|
| 396 |
+
if voice == "Act":
|
| 397 |
+
if pol == "Aff":
|
| 398 |
+
v = _netlex_finite(lemma, "Act", mood, tense, f"{num}{person}")
|
| 399 |
+
return [v] if v else None
|
| 400 |
+
ei = _netlex_ei(number, person)
|
| 401 |
+
nv = (_netlex_finite(lemma, "Act", "Cond", None, "ConNeg") if mood == "Cond"
|
| 402 |
+
else _netlex_finite(lemma, "Act", "Pot", None, "ConNeg") if mood == "Pot"
|
| 403 |
+
else _netlex_npart(lemma, "Act", number) if tense == "Prt"
|
| 404 |
+
else _netlex_finite(lemma, "Act", "Ind", "Prs", "ConNeg"))
|
| 405 |
+
return [ei, nv] if (ei and nv) else None
|
| 406 |
+
# passive (impersonal)
|
| 407 |
+
if pol == "Aff":
|
| 408 |
+
v = _netlex_finite(lemma, "Pss", mood, tense, "Pe4")
|
| 409 |
+
return [v] if v else None
|
| 410 |
+
nv = (_netlex_finite(lemma, "Pss", "Cond", None, "ConNeg") if mood == "Cond"
|
| 411 |
+
else _netlex_finite(lemma, "Pss", "Pot", None, "ConNeg") if mood == "Pot"
|
| 412 |
+
else _netlex_npart(lemma, "Pss", number) if tense == "Prt"
|
| 413 |
+
else _netlex_finite(lemma, "Pss", "Ind", "Prs", "ConNeg"))
|
| 414 |
+
return ["ei", nv] if nv else None
|
| 415 |
+
|
| 416 |
+
|
| 417 |
+
# Possessive suffixes that attach to the plural-genitive stem regularly (no vowel harmony): the
|
| 418 |
+
# genitive's final -n drops, the suffix appends. 1st/2nd person only — 3rd (-nsa/-nsä) is harmony-
|
| 419 |
+
# dependent and the Giella FST shows non-standard harmony for some words, so it is NOT derived here.
|
| 420 |
+
_GENPL_POSS_SUFFIX = {"1Sg": "ni", "2Sg": "si", "1Pl": "mme", "2Pl": "nne"}
|
| 421 |
+
|
| 422 |
+
|
| 423 |
+
def _derive_genpl_possessive(lemma, upos, feats, surface):
|
| 424 |
+
"""Modern plural-genitive + (1st/2nd-person) possessive, derived from the engine's OWN Px=None
|
| 425 |
+
plural genitive — which is already the modern '-jen/-ten' base (talojen, lasten). Drop its final
|
| 426 |
+
-n, append the possessive suffix: talojen -> talojemme. Returns None if not applicable / the base
|
| 427 |
+
can't be realized / doesn't end in -n. Keeps the engine inside the FST's accepted set while
|
| 428 |
+
preferring the modern variant (vs the neural fallback's archaic '-in-' form, taloimme).
|
| 429 |
+
|
| 430 |
+
The base is realized lowercase (surface=None); the caller applies any leading capital once. If the
|
| 431 |
+
base itself came from the NEURAL realizer (an unverified guess, e.g. lasten/vastausten), the OOV
|
| 432 |
+
trace is re-recorded for the DERIVED form so its unverified provenance is preserved."""
|
| 433 |
+
suf = _GENPL_POSS_SUFFIX.get(feats.get("Px", "None"))
|
| 434 |
+
if suf is None or feats.get("Case") != "Gen" or feats.get("Number") != "Plur":
|
| 435 |
+
return None
|
| 436 |
+
noov = list(_NEURAL_OOV)
|
| 437 |
+
snap = (list(_OOV_UNVERIFIED), list(_OOV_UNREALIZABLE))
|
| 438 |
+
base = _netlex_nominal(lemma, upos, {k: v for k, v in feats.items() if k != "Px"}, None)
|
| 439 |
+
base_was_neural = len(_NEURAL_OOV) > len(noov)
|
| 440 |
+
# restore the OOV trace to its pre-call state; provenance is re-recorded for the DERIVED form only
|
| 441 |
+
_NEURAL_OOV[:], _OOV_UNVERIFIED[:], _OOV_UNREALIZABLE[:] = noov, snap[0], snap[1]
|
| 442 |
+
if not base or not base.endswith("n"):
|
| 443 |
+
return None
|
| 444 |
+
derived = base[:-1] + suf
|
| 445 |
+
if base_was_neural: # derived from an unverified neural base -> keep it auditable
|
| 446 |
+
_NEURAL_OOV.append(f"{lemma}->{derived} ({upos})")
|
| 447 |
+
return derived
|
| 448 |
+
|
| 449 |
+
|
| 450 |
+
# Partitive-plural endings, longest-first, so the comitative derive strips the whole ending before
|
| 451 |
+
# appending -ine. The comitative stem IS the partitive-plural stem (talo: taloja -> taloine; aarre:
|
| 452 |
+
# aarteita -> aarteine), so recovering it from the engine's own partitive plural inherits the correct
|
| 453 |
+
# stem AND gradation grade — which the neural realizer mis-guesses for e-/s-/alternation nominals.
|
| 454 |
+
_PARTITIVE_PLURAL_ENDINGS = ("ita", "itä", "ja", "jä", "ia", "iä", "ta", "tä", "a", "ä")
|
| 455 |
+
|
| 456 |
+
|
| 457 |
+
def _comitative_stem_from_partitive_plural(parpl):
|
| 458 |
+
"""Strip the partitive-plural ending to recover the plural stem, then append the comitative -ine.
|
| 459 |
+
None if `parpl` is falsy or carries no known partitive ending."""
|
| 460 |
+
if not parpl:
|
| 461 |
+
return None
|
| 462 |
+
for suf in _PARTITIVE_PLURAL_ENDINGS:
|
| 463 |
+
if parpl.endswith(suf):
|
| 464 |
+
return parpl[:-len(suf)] + "ine"
|
| 465 |
+
return None
|
| 466 |
+
|
| 467 |
+
|
| 468 |
+
def _is_geminate_strong(strong, weak):
|
| 469 |
+
"""True iff `strong` is `weak` with exactly one extra, doubled MEDIAL k/p/t — a weak->strong consonant-
|
| 470 |
+
gradation upgrade (laatikkoine vs laatikoine). Recognises the kpt-geminate comitative the partitive-
|
| 471 |
+
plural derive under-grades (its base is weak-grade, the comitative keeps the strong grade). Restricted
|
| 472 |
+
to the kk/pp/tt gradation pattern (the only geminates Finnish gradation produces) and to a medial
|
| 473 |
+
position (i>0): this predicate is the SOLE guard on adopting the net form, so it must match exactly the
|
| 474 |
+
linguistic invariant — b/d/g gradation is qualitative (single-consonant), never geminate, and an
|
| 475 |
+
initial doubling is impossible. The doubled consonant lands in the stem, before the -ine- suffix."""
|
| 476 |
+
if not strong or not weak or len(strong) != len(weak) + 1:
|
| 477 |
+
return False
|
| 478 |
+
for i in range(1, len(strong) - 1):
|
| 479 |
+
if strong[i] == strong[i + 1] and strong[i] in "kpt" and strong[:i] + strong[i + 1:] == weak:
|
| 480 |
+
return True
|
| 481 |
+
return False
|
| 482 |
+
|
| 483 |
+
|
| 484 |
+
def _derive_comitative(lemma, upos, feats, surface):
|
| 485 |
+
"""Px=None comitative, derived from the engine's OWN partitive plural (comitative stem = plural stem
|
| 486 |
+
+ 'ine'), which carries the correct stem and gradation grade — vs the neural realizer's mis-stemmed
|
| 487 |
+
guess (aarre->aarrehine for aarteine, valkokangas->valkokangaksine for valkokankaine). For kpt-
|
| 488 |
+
GEMINATE stems the partitive plural is weak-grade but the comitative is strong (laatikko: laatikoita
|
| 489 |
+
-> the derive gives 'laatikoine', but FST wants strong-grade 'laatikkoine'), so when the net supplies
|
| 490 |
+
exactly the doubled-consonant (strong) version of the derive, trust the net. Returns None (caller
|
| 491 |
+
falls back to the net) when not a Px=None comitative, or the partitive plural can't be realized /
|
| 492 |
+
lacks a known ending. This derive is markedly more FST-valid than the net's direct comitative.
|
| 493 |
+
The rare residual cases are compound lemmas whose partitive plural inflects BOTH parts (omatunto ->
|
| 494 |
+
omiatuntoine). Realized lowercase (surface=None); the caller applies any leading capital once."""
|
| 495 |
+
if upos != "NOUN" or feats.get("Case") != "Com" or feats.get("Px", "None") != "None":
|
| 496 |
+
return None
|
| 497 |
+
noov = list(_NEURAL_OOV)
|
| 498 |
+
snap = (list(_OOV_UNVERIFIED), list(_OOV_UNREALIZABLE))
|
| 499 |
+
parpl = _netlex_nominal(lemma, upos, {"Case": "Par", "Number": "Plur", "Px": "None"}, None)
|
| 500 |
+
base_was_neural = len(_NEURAL_OOV) > len(noov)
|
| 501 |
+
_NEURAL_OOV[:], _OOV_UNVERIFIED[:], _OOV_UNREALIZABLE[:] = noov, snap[0], snap[1]
|
| 502 |
+
derived = _comitative_stem_from_partitive_plural(parpl)
|
| 503 |
+
if derived is None:
|
| 504 |
+
return None
|
| 505 |
+
# kpt-geminate override: the comitative keeps the strong grade the partitive plural drops; the net
|
| 506 |
+
# gets that right (laatikkoine), so adopt the net form when it is exactly the geminate-strong version
|
| 507 |
+
# of the derive. The probe is trace-neutral (snapshot/restore around the realizer call).
|
| 508 |
+
adopted_net = False
|
| 509 |
+
if _NOUN_REALIZER is not None:
|
| 510 |
+
nf, _conf = _NOUN_REALIZER(lemma, upos, {**feats, "Number": "Sing"}, None)
|
| 511 |
+
_NEURAL_OOV[:], _OOV_UNVERIFIED[:], _OOV_UNREALIZABLE[:] = noov, snap[0], snap[1]
|
| 512 |
+
if nf and _is_geminate_strong(nf, derived):
|
| 513 |
+
derived, adopted_net = nf, True
|
| 514 |
+
if base_was_neural or adopted_net: # derived/adopted from an unverified neural form -> keep it auditable
|
| 515 |
+
_NEURAL_OOV.append(f"{lemma}->{derived} ({upos})")
|
| 516 |
+
return derived
|
| 517 |
+
|
| 518 |
+
|
| 519 |
+
# Opening-diphthong monosyllabic NOUN heads (full words, so suffix matching can't false-fire on a random
|
| 520 |
+
# -ie/-uo/-yö loan). Longest-distinguishing first: `työ` before `yö` so `käsityö` splits at `työ`.
|
| 521 |
+
_DIPHTHONG_HEADS = ("työ", "tie", "vyö", "suo", "vuo", "yö")
|
| 522 |
+
|
| 523 |
+
|
| 524 |
+
def _derive_diphthong_compound(lemma, upos, feats, surface):
|
| 525 |
+
"""A NOUN compound whose HEAD is an opening-diphthong monosyllable (juhannus·yö, maan·tie, käsi·työ)
|
| 526 |
+
realizes as the invariant modifier prefix + the head's OWN realization. The net mis-stems some of these
|
| 527 |
+
(`juhannusyö`->`juhannusyyksia`, a silent wrong form that shares the lemma onset so `_stem_consistent`
|
| 528 |
+
misses it); the head inflects correctly on its own (post the _stem_consistent diphthong fix) and the
|
| 529 |
+
modifier prefix is a fixed string, so prefix+head is correct by construction. Fires only for a true
|
| 530 |
+
COMPOUND (lemma strictly longer than the head); the bare head goes straight to the net. Flags the
|
| 531 |
+
result neural-unverified iff the head realization was. Returns None when not applicable."""
|
| 532 |
+
if upos != "NOUN":
|
| 533 |
+
return None
|
| 534 |
+
low = lemma.lower()
|
| 535 |
+
# require a >=2-char modifier prefix so a real compound (käsi·työ, maan·tie) fires but the bare
|
| 536 |
+
# monosyllable does NOT (työ = t+yö would else match head `yö` with a 1-char prefix) — bare työ/vyö
|
| 537 |
+
# stay on their own net path (the _stem_consistent fix), and only genuine compounds are derived.
|
| 538 |
+
head = next((h for h in _DIPHTHONG_HEADS if low.endswith(h) and len(low) >= len(h) + 2), None)
|
| 539 |
+
if head is None:
|
| 540 |
+
return None
|
| 541 |
+
noov, snap = list(_NEURAL_OOV), (list(_OOV_UNVERIFIED), list(_OOV_UNREALIZABLE))
|
| 542 |
+
hf = _netlex_nominal(head, "NOUN", feats, None) # the head's own (correct) realization
|
| 543 |
+
head_was_neural = len(_NEURAL_OOV) > len(noov)
|
| 544 |
+
_NEURAL_OOV[:], _OOV_UNVERIFIED[:], _OOV_UNREALIZABLE[:] = noov, snap[0], snap[1]
|
| 545 |
+
if hf is None:
|
| 546 |
+
return None
|
| 547 |
+
derived = lemma[:-len(head)] + hf # invariant modifier prefix + inflected head
|
| 548 |
+
if head_was_neural:
|
| 549 |
+
_NEURAL_OOV.append(f"{lemma}->{derived} ({upos})")
|
| 550 |
+
return derived
|
| 551 |
+
|
| 552 |
+
|
| 553 |
+
def _netlex_nominal(lemma, upos, feats, surface=None):
|
| 554 |
+
"""KNOWN-word nominal, Giella-free: lexicon-first then the nominal net, or None. PROPN is not shipped
|
| 555 |
+
in the lexicon (open class) so it is served by the net only. Preserves a leading capital from
|
| 556 |
+
`surface` to match `realize_strict`/the realizer."""
|
| 557 |
+
if _GENLEX is not None:
|
| 558 |
+
hit = _GENLEX.lookup_nominal(upos, lemma, feats) # None for PROPN by construction
|
| 559 |
+
if hit is not None:
|
| 560 |
+
if surface and surface[:1].isupper():
|
| 561 |
+
hit = hit[:1].upper() + hit[1:]
|
| 562 |
+
return hit # trusted lexicon (byte-exact) — no validation
|
| 563 |
+
# lexicon miss for a plural-genitive 1st/2nd-person possessive -> derive the MODERN form from the
|
| 564 |
+
# Px=None plural genitive (which the lexicon/realizer gives in the modern -jen/-ten stem), instead
|
| 565 |
+
# of the neural realizer's archaic -in- variant. Stays FST-valid (parity floor); see test suite.
|
| 566 |
+
# ...and a Px=None comitative -> derive from the engine's own partitive plural (correct stem +
|
| 567 |
+
# gradation) rather than the net's mis-stemmed guess. Both derives are mutually exclusive on feats
|
| 568 |
+
# (genpl needs Gen/Plur/1st-2nd-Px; comitative needs Com/Px=None) so the order is immaterial.
|
| 569 |
+
if upos == "NOUN":
|
| 570 |
+
# diphthong-compound first: it routes the head through this same function (handling the head's own
|
| 571 |
+
# genpl/comitative/net), so it correctly composes for every case of these compounds.
|
| 572 |
+
derived = _derive_diphthong_compound(lemma, upos, feats, surface)
|
| 573 |
+
if derived is None:
|
| 574 |
+
derived = _derive_genpl_possessive(lemma, upos, feats, surface)
|
| 575 |
+
if derived is None:
|
| 576 |
+
derived = _derive_comitative(lemma, upos, feats, surface)
|
| 577 |
+
if derived is not None:
|
| 578 |
+
if surface and surface[:1].isupper():
|
| 579 |
+
derived = derived[:1].upper() + derived[1:]
|
| 580 |
+
return derived
|
| 581 |
+
# lexicon miss -> neural realizer, but FAIL CLOSED on an untrusted guess: an OOV open-class
|
| 582 |
+
# word the shipped lexicon does not cover must abstain (None), never emit `<unk>`/truncated garbage or
|
| 583 |
+
# the bare lemma. Mirrors the FST path's abstain so the flip changes coverage, not output safety.
|
| 584 |
+
if _NOUN_REALIZER is not None:
|
| 585 |
+
nf, conf = _NOUN_REALIZER(lemma, upos, feats, surface)
|
| 586 |
+
if nf and _neural_oov_ok(lemma, nf, conf, conf_min=0.0): # net serves KNOWN words here
|
| 587 |
+
_NEURAL_OOV.append(f"{lemma}->{nf} ({upos})") # validated neural form (auditable)
|
| 588 |
+
return nf
|
| 589 |
+
if nf: # produced but failed validation -> suppress
|
| 590 |
+
_OOV_UNVERIFIED.append(f"{lemma}->{nf} ({upos}) conf={conf:.2f}")
|
| 591 |
+
return None
|
| 592 |
+
_OOV_UNREALIZABLE.append(f"{lemma} ({upos}) {feats.get('Case', 'Nom')}/{feats.get('Number', 'Sing')}")
|
| 593 |
+
return None
|
| 594 |
+
|
| 595 |
+
|
| 596 |
+
# Neural OOV noun/adj realizer: fn(lemma, upos, feats, surface) -> form | None.
|
| 597 |
+
# Set at inference so the noun/case/number ops fall back to the model when the FST cannot inflect a
|
| 598 |
+
# loan/OOV word, instead of silently no-op'ing to the lemma.
|
| 599 |
+
_NOUN_REALIZER = None
|
| 600 |
+
# OOV non-finite verb-form realizer: fn(lemma, form, case, number) -> (form, conf) | (None,0).
|
| 601 |
+
# Set at inference (like _NOUN_REALIZER) so realize_verbform can inflect a loan verb's participle/
|
| 602 |
+
# infinitive the FST cannot. Absent => OOV verb-forms abstain (oov_unrealizable).
|
| 603 |
+
_VERBFORM_REALIZER = None
|
| 604 |
+
|
| 605 |
+
# --- net+lexicon generation backend --------------------------------------------------------
|
| 606 |
+
# KNOWN-word inflection is served Giella-free by lexicon-first + neural-net-second (the same
|
| 607 |
+
# model+lexicon shape the analysis side uses). This is the only generation backend; the flag is the
|
| 608 |
+
# internal serving toggle set by `pipeline.Transformer` (and by tests that exercise the path directly).
|
| 609 |
+
_NETLEX_BACKEND = True
|
| 610 |
+
_GENLEX = None # genlex.GenerationLexicon | None (the static exception table; loaded at serving)
|
| 611 |
+
|
| 612 |
+
|
| 613 |
+
_OPEN_CLASS = ("NOUN", "ADJ", "PROPN") # POS that giella_query inflects + where OOV is the risk
|
| 614 |
+
|
| 615 |
+
# Words realized by a NEURAL fallback in the current transform (verb cluster or noun/adj). The FST
|
| 616 |
+
# cannot validate these (the whole point is they are OOV), so they are model GUESSES — recorded
|
| 617 |
+
# here and surfaced in the pipeline Result so an OOV-realized output is never silent/unaudited.
|
| 618 |
+
_NEURAL_OOV: list = []
|
| 619 |
+
# Open-class words an op needed to inflect but NEITHER the FST nor the neural realizer could —
|
| 620 |
+
# the op abstains and the pipeline reports `oov_unrealizable` (explicit "unsupported morphology",
|
| 621 |
+
# distinct from a structurally inapplicable op).
|
| 622 |
+
_OOV_UNREALIZABLE: list = []
|
| 623 |
+
# Open-class words where the neural realizer DID produce a candidate but it failed validation
|
| 624 |
+
# (stem-inconsistent, or below the near-tie confidence floor) — suppressed, never emitted. Distinct
|
| 625 |
+
# reason `oov_unverified`: "we had a guess and didn't trust it" vs "nothing could realize it".
|
| 626 |
+
_OOV_UNVERIFIED: list = []
|
| 627 |
+
# Near-tie confidence floor for a neural OOV form. NOTE: the top1-top2 margin is a WEAK error
|
| 628 |
+
# detector here — the model's rare misses are confidently wrong, so this only rejects genuine
|
| 629 |
+
# near-ties at ~zero cost to correct forms. The real safety is FST-first + `_stem_consistent` +
|
| 630 |
+
# mandatory unverified-flagging, not this threshold.
|
| 631 |
+
_OOV_CONF_MIN = 0.08
|
| 632 |
+
|
| 633 |
+
|
| 634 |
+
# Opening-diphthong monosyllables (tie/työ/yö/suo/vuo/vyö …) mutate the stem diphthong in the plural:
|
| 635 |
+
# ie->ei, uo->oi, yö->öi (tie->tei-, työ->töi-, yö->öi-, suo->soi-, vyö->vöi-). That changes the word's
|
| 636 |
+
# first two chars, so the bare first-2-char check below wrongly rejects their (correct) plural forms. Map
|
| 637 |
+
# each lemma-final diphthong to its plural stem nucleus (already including the plural -i-) so the documented
|
| 638 |
+
# mutation is allowed — and ONLY that: the output must begin with onset + nucleus, so garbage is rejected.
|
| 639 |
+
_OPENING_DIPHTHONG = {"ie": "ei", "uo": "oi", "yö": "öi"}
|
| 640 |
+
|
| 641 |
+
|
| 642 |
+
def _stem_consistent(src, out):
|
| 643 |
+
"""Cheap validation of a neural OOV realization: Finnish word-initial never gradates, so a
|
| 644 |
+
genuine inflection of `src` shares its first 2 chars — EXCEPT the opening-diphthong monosyllables,
|
| 645 |
+
whose plural stem mutates the diphthong (työ->töitä). Rejects catastrophic model garbage."""
|
| 646 |
+
a, b = (src or "").lower(), (out or "").lower()
|
| 647 |
+
if not b:
|
| 648 |
+
return False
|
| 649 |
+
if a[:2] == b[:2]:
|
| 650 |
+
return True
|
| 651 |
+
for diph, nucleus in _OPENING_DIPHTHONG.items():
|
| 652 |
+
if a.endswith(diph) and b.startswith(a[:-2] + nucleus):
|
| 653 |
+
return True # onset + mutated plural nucleus (töi-/tei-/öi-/soi-/vöi-)
|
| 654 |
+
return False
|
| 655 |
+
|
| 656 |
+
|
| 657 |
+
def _neural_oov_ok(lemma, nf, conf, conf_min=_OOV_CONF_MIN):
|
| 658 |
+
"""SHAPE-filter a NEURAL open-class realization before serving — fail closed on structurally-impossible
|
| 659 |
+
output, but do NOT gate stem-correctness here. A served net form is flagged `neural_unverified` (the
|
| 660 |
+
honest tiered claim); FST-parity coverage is expanded OFFLINE via the lexicon overrides, not by a
|
| 661 |
+
runtime heuristic.
|
| 662 |
+
|
| 663 |
+
The former char-prefix stem check (`_stem_consistent`, first-2-char) was REMOVED here: measured against
|
| 664 |
+
the FST it was anti-correlated with correctness — it caught none of the net's real errors (a wrong
|
| 665 |
+
inflection keeps the onset) while suppressing correct consonant-gradation forms (`aika`->`ajan`,
|
| 666 |
+
`apu`->`avun`), and the net produced no onset-drift garbage. So the onset heuristic only cost
|
| 667 |
+
correctness; it bought no safety. (It still guards the FST-FALLBACK path's true-OOV net at
|
| 668 |
+
`_realize_form`, where no FST gold exists to expand coverage offline.) Rejects only:
|
| 669 |
+
(a) the OOV-char marker `<unk>` (unseen compound seam / out-of-vocab char);
|
| 670 |
+
(b) truncation — a valid open-class inflection is never STRICTLY shorter than the citation lemma
|
| 671 |
+
(catches `suvaitsevaisuus`->`suvaitsevaisuu`);
|
| 672 |
+
(c) empty;
|
| 673 |
+
(d) a near-tie below `conf_min` (the netlex path passes 0; the FST-fallback path passes the OOV floor).
|
| 674 |
+
Returns True for a candidate safe to EMIT (flagged), not a guarantee of FST-correctness."""
|
| 675 |
+
if not nf or "<unk>" in nf:
|
| 676 |
+
return False
|
| 677 |
+
if len(nf) < len(lemma):
|
| 678 |
+
return False
|
| 679 |
+
return conf >= conf_min
|
| 680 |
+
|
| 681 |
+
|
| 682 |
+
def _realize_form(lemma, upos, feats, surface=None):
|
| 683 |
+
"""One word-form: lexicon-first, neural OOV realizer second. The silent-fallback blind spot is only
|
| 684 |
+
OPEN-class (NOUN/ADJ/PROPN) words the lexicon can't inflect — for those a hard miss returns None so
|
| 685 |
+
case/number ops ABSTAIN instead of emitting a half-agreed NP. NUM/PRON are served lexicon-first (the
|
| 686 |
+
genlex numeral/pronoun overrides) and fall back to the lemma on a miss; the remaining closed classes
|
| 687 |
+
(DET, …) have a Nom form reliably equal to the lemma, so they keep the lemma fallback."""
|
| 688 |
+
if not _NETLEX_BACKEND:
|
| 689 |
+
return None
|
| 690 |
+
# Numerals + pronouns: closed classes served lexicon-first (NUM/PRON alias NOUN in the genlex). The
|
| 691 |
+
# nominal net mis-stems them (kaksi+Gen -> kaksin not kahden; minä+Gen -> minän not minun), so on a
|
| 692 |
+
# lexicon MISS fall back to the lemma (safe Nom, like the other closed classes below), NEVER the net.
|
| 693 |
+
if upos in ("NUM", "PRON"):
|
| 694 |
+
hit = _GENLEX.lookup_nominal(upos, lemma, feats) if _GENLEX is not None else None
|
| 695 |
+
if hit is not None and surface and surface[:1].isupper():
|
| 696 |
+
hit = hit[:1].upper() + hit[1:]
|
| 697 |
+
return hit if hit is not None else lemma
|
| 698 |
+
# Lexicon-first then the nominal net for open-class; other closed-class -> lemma.
|
| 699 |
+
# Abstain (None) on open-class miss.
|
| 700 |
+
if upos not in _OPEN_CLASS:
|
| 701 |
+
return lemma
|
| 702 |
+
return _netlex_nominal(lemma, upos, feats, surface)
|
| 703 |
+
|
| 704 |
+
|
| 705 |
+
def _flip_case(sent, obj, newcase):
|
| 706 |
+
"""obj noun + agreeing modifiers -> newcase; {id: form} or None if any fails."""
|
| 707 |
+
out = {}
|
| 708 |
+
for w in [obj] + [c for c in sent if c["head"] == obj["id"] and c["deprel"] in AGREE
|
| 709 |
+
and c["feats"].get("Case") == obj["feats"].get("Case")]:
|
| 710 |
+
form = _realize_form(w["lemma"], w["upos"], {**w["feats"], "Case": newcase}, w["form"])
|
| 711 |
+
if not form: # FST miss + no neural fallback -> abstain (never lemma)
|
| 712 |
+
return None
|
| 713 |
+
out[w["id"]] = form
|
| 714 |
+
return out
|
| 715 |
+
|
| 716 |
+
|
| 717 |
+
# --- hybrid passive: deterministic structure + neural OOV verb realization ------
|
| 718 |
+
# The hard part of passive is non-local STRUCTURE (drop the subject subtree, nominativize the
|
| 719 |
+
# total object) — that belongs to the parse/rules (`apply_clause`), not the char model. The lexicon
|
| 720 |
+
# inflects known verbs; the verb cluster (`_netlex_verb_cluster`, finiteverb + verbform) realizes the
|
| 721 |
+
# OOV/loan verb forms the lexicon can't. The deterministic structure is the same either way.
|
| 722 |
+
_FORCE_NEURAL_VC = False # eval-only: pretend KNOWN-word lookup is empty for EVERY verb (simulate OOV)
|
| 723 |
+
|
| 724 |
+
|
| 725 |
+
def _features_to_op(voice, mood, tense, pol):
|
| 726 |
+
"""Map a final feature-set to the trained op-name (canonical order neg+passive+tense/mood)."""
|
| 727 |
+
parts = []
|
| 728 |
+
if pol == "Neg":
|
| 729 |
+
parts.append("neg")
|
| 730 |
+
if voice == "Pss":
|
| 731 |
+
parts.append("passive")
|
| 732 |
+
if tense == "Prt":
|
| 733 |
+
parts.append("past")
|
| 734 |
+
elif tense == "Prf":
|
| 735 |
+
parts.append("perfect")
|
| 736 |
+
elif mood == "Cond":
|
| 737 |
+
parts.append("cond")
|
| 738 |
+
return "+".join(parts) or None
|
| 739 |
+
|
| 740 |
+
|
| 741 |
+
def apply_clause(sent, deltas, nonce_rng=None):
|
| 742 |
+
"""Apply verb-feature deltas to the main clause -> (src_words, tgt_words) or None."""
|
| 743 |
+
c = clause(sent)
|
| 744 |
+
if c is None:
|
| 745 |
+
return None
|
| 746 |
+
verb, is_cop, base = c
|
| 747 |
+
final = {**base, **deltas}
|
| 748 |
+
if final == base or (final["voice"] == "Pss" and is_cop):
|
| 749 |
+
return None
|
| 750 |
+
vc = None if _FORCE_NEURAL_VC else verb_cluster(
|
| 751 |
+
verb["lemma"], final["voice"], final["mood"], final["tense"],
|
| 752 |
+
final["pol"], base["person"], base["number"])
|
| 753 |
+
if vc is None:
|
| 754 |
+
return None
|
| 755 |
+
|
| 756 |
+
drop, repl = set(), {}
|
| 757 |
+
going_passive = final["voice"] == "Pss" and base["voice"] == "Act"
|
| 758 |
+
if going_passive and not is_cop:
|
| 759 |
+
# demote any OVERT subject subtree. A missing nsubj is a valid pro-drop passive
|
| 760 |
+
# (`Tällaiset aarteet kotiutin` -> `Tällaiset aarteet kotiutettiin`), NOT noise —
|
| 761 |
+
# so we must not abort here (an earlier `return None` wrongly discarded these).
|
| 762 |
+
for t in sent:
|
| 763 |
+
if t["deprel"] in ("nsubj", "nsubj:cop"):
|
| 764 |
+
drop |= _subtree_ids(sent, t["id"])
|
| 765 |
+
if not is_cop:
|
| 766 |
+
for t in sent:
|
| 767 |
+
if t["deprel"] == "obj" and t["feats"].get("Case") in ("Gen", "Nom") \
|
| 768 |
+
and t["upos"] in ("NOUN", "PROPN", "ADJ", "NUM"):
|
| 769 |
+
newcase = "Par" if final["pol"] == "Neg" else ("Nom" if going_passive else None)
|
| 770 |
+
if newcase and newcase != t["feats"].get("Case"):
|
| 771 |
+
fl = _flip_case(sent, t, newcase)
|
| 772 |
+
if fl is None: # required object re-casing unrealizable (OOV) -> FAIL CLOSED
|
| 773 |
+
return None # abstain, never emit the original wrong-case object
|
| 774 |
+
repl.update(fl)
|
| 775 |
+
|
| 776 |
+
repl_c = None
|
| 777 |
+
if nonce_rng:
|
| 778 |
+
init = verb["form"][:1].lower()
|
| 779 |
+
repl_c = _rand_onset(nonce_rng, init)
|
| 780 |
+
vc = list(vc); vc[-1] = swap_initial(vc[-1], repl_c) # lexical verb (ei is first)
|
| 781 |
+
|
| 782 |
+
src = [swap_initial(t["form"], repl_c) if (repl_c and t["id"] == verb["id"]) else t["form"]
|
| 783 |
+
for t in sent]
|
| 784 |
+
tgt = []
|
| 785 |
+
for t in sent:
|
| 786 |
+
if t["id"] in drop:
|
| 787 |
+
continue
|
| 788 |
+
if t["id"] == verb["id"]:
|
| 789 |
+
tgt.extend(vc)
|
| 790 |
+
else:
|
| 791 |
+
tgt.append(repl.get(t["id"], t["form"]))
|
| 792 |
+
return src, tgt
|
| 793 |
+
|
| 794 |
+
|
| 795 |
+
# ----------------------------- standalone ops -----------------------------
|
| 796 |
+
def op_quest(sent, nonce_rng=None):
|
| 797 |
+
"""Polar question: front the finite verb + -ko/-kö clitic (vowel harmony)."""
|
| 798 |
+
verb, _ = _main_verb(sent)
|
| 799 |
+
if verb is None:
|
| 800 |
+
return None
|
| 801 |
+
repl_c = None
|
| 802 |
+
if nonce_rng:
|
| 803 |
+
init = verb["form"][:1].lower()
|
| 804 |
+
repl_c = _rand_onset(nonce_rng, init)
|
| 805 |
+
vform = swap_initial(verb["form"], repl_c) if repl_c else verb["form"]
|
| 806 |
+
q = vform + _clitic(vform)
|
| 807 |
+
tgt = [q]
|
| 808 |
+
rest = [t for t in sent if t["id"] != verb["id"]
|
| 809 |
+
and not (t["upos"] == "PUNCT" and t["form"] in ".!")]
|
| 810 |
+
for i, t in enumerate(rest):
|
| 811 |
+
f = t["form"]
|
| 812 |
+
if i == 0 and t["upos"] != "PROPN" and f[:1].isupper():
|
| 813 |
+
f = f[:1].lower() + f[1:]
|
| 814 |
+
tgt.append(f)
|
| 815 |
+
tgt.append("?")
|
| 816 |
+
src = [swap_initial(t["form"], repl_c) if (repl_c and t["id"] == verb["id"]) else t["form"]
|
| 817 |
+
for t in sent]
|
| 818 |
+
return src, tgt
|
| 819 |
+
|
| 820 |
+
|
| 821 |
+
def _op_degree(degree):
|
| 822 |
+
"""degree is the UD value 'Cmp' or 'Sup'. Routes through `_realize_form` (FST first, neural OOV
|
| 823 |
+
realizer second) so comparison works on OOV/loan/nonce adjectives the FST can't inflect, not
|
| 824 |
+
just FST-known ones — same open-vocabulary path as number/case."""
|
| 825 |
+
def fn(sent, nonce_rng=None):
|
| 826 |
+
repl = {}
|
| 827 |
+
for t in sent:
|
| 828 |
+
if t["upos"] == "ADJ" and t["feats"].get("Degree", "Pos") == "Pos":
|
| 829 |
+
c = _realize_form(t["lemma"], "ADJ", {**t["feats"], "Degree": degree}, t["form"])
|
| 830 |
+
if c and c != t["form"]:
|
| 831 |
+
repl[t["id"]] = c
|
| 832 |
+
if not repl:
|
| 833 |
+
return None
|
| 834 |
+
src = [t["form"] for t in sent]
|
| 835 |
+
tgt = [repl.get(t["id"], t["form"]) for t in sent]
|
| 836 |
+
if nonce_rng:
|
| 837 |
+
ids = list(repl)
|
| 838 |
+
tid = nonce_rng.choice(ids)
|
| 839 |
+
i = [t["id"] for t in sent].index(tid)
|
| 840 |
+
rc = _rand_onset(nonce_rng, src[i][:1].lower())
|
| 841 |
+
src[i] = swap_initial(src[i], rc); tgt[i] = swap_initial(tgt[i], rc)
|
| 842 |
+
return src, tgt
|
| 843 |
+
return fn
|
| 844 |
+
|
| 845 |
+
|
| 846 |
+
def _verbop(deltas, req):
|
| 847 |
+
def fn(sent, nonce_rng=None):
|
| 848 |
+
c = clause(sent)
|
| 849 |
+
if c is None:
|
| 850 |
+
return None
|
| 851 |
+
_, is_cop, base = c
|
| 852 |
+
if any(base.get(k) != v for k, v in req.items()):
|
| 853 |
+
return None
|
| 854 |
+
if deltas.get("voice") == "Pss" and is_cop:
|
| 855 |
+
return None
|
| 856 |
+
return apply_clause(sent, deltas, nonce_rng)
|
| 857 |
+
return fn
|
| 858 |
+
|
| 859 |
+
|
| 860 |
+
# verb-feature ops: (deltas, required base state)
|
| 861 |
+
VERB_OPS = {
|
| 862 |
+
"neg": ({"pol": "Neg"}, {}),
|
| 863 |
+
"past": ({"tense": "Prt"}, {"tense": "Prs"}),
|
| 864 |
+
"pres": ({"tense": "Prs"}, {"tense": "Prt"}),
|
| 865 |
+
"cond": ({"mood": "Cond"}, {"mood": "Ind"}),
|
| 866 |
+
"passive": ({"voice": "Pss"}, {"voice": "Act"}),
|
| 867 |
+
"perfect": ({"tense": "Prf"}, {"tense": "Prs", "mood": "Ind"}),
|
| 868 |
+
# potential mood is in the TRAINED set so the neural path covers OOV/loan verbs (`striimannee`)
|
| 869 |
+
# too, not just FST-known ones.
|
| 870 |
+
"potential": ({"mood": "Pot"}, {"mood": "Ind", "tense": "Prs"}),
|
| 871 |
+
}
|
| 872 |
+
# Composite (multi-feature) verb ops; each value is (delta-features, reset-baseline).
|
| 873 |
+
COMPOSITES = {
|
| 874 |
+
"neg+past": ({"pol": "Neg", "tense": "Prt"}, {"tense": "Prs"}),
|
| 875 |
+
"neg+cond": ({"pol": "Neg", "mood": "Cond"}, {"mood": "Ind"}),
|
| 876 |
+
"passive+past": ({"voice": "Pss", "tense": "Prt"}, {"voice": "Act", "tense": "Prs"}),
|
| 877 |
+
"passive+cond": ({"voice": "Pss", "mood": "Cond"}, {"voice": "Act", "mood": "Ind"}),
|
| 878 |
+
"neg+passive": ({"pol": "Neg", "voice": "Pss"}, {"voice": "Act"}),
|
| 879 |
+
# analytic perfect composites (e.g. `ei ole ostanut`)
|
| 880 |
+
"neg+perfect": ({"pol": "Neg", "tense": "Prf"}, {"tense": "Prs", "mood": "Ind"}),
|
| 881 |
+
"passive+perfect": ({"voice": "Pss", "tense": "Prf"}, {"voice": "Act", "tense": "Prs", "mood": "Ind"}),
|
| 882 |
+
"neg+passive+past": ({"pol": "Neg", "voice": "Pss", "tense": "Prt"},
|
| 883 |
+
{"voice": "Act", "tense": "Prs"}),
|
| 884 |
+
"neg+passive+cond": ({"pol": "Neg", "voice": "Pss", "mood": "Cond"},
|
| 885 |
+
{"voice": "Act", "mood": "Ind"}),
|
| 886 |
+
# three-feature analytic composite: `ei ole ostettu`
|
| 887 |
+
"neg+passive+perfect": ({"pol": "Neg", "voice": "Pss", "tense": "Prf"},
|
| 888 |
+
{"voice": "Act", "tense": "Prs", "mood": "Ind"}),
|
| 889 |
+
}
|
| 890 |
+
OPS = {k: _verbop(d, r) for k, (d, r) in {**VERB_OPS, **COMPOSITES}.items()}
|
| 891 |
+
OPS["quest"] = op_quest
|
| 892 |
+
OPS["comp"] = _op_degree("Cmp") # UD Degree values; _op_degree maps to FST/realizer
|
| 893 |
+
OPS["super"] = _op_degree("Sup")
|
| 894 |
+
|
| 895 |
+
|
| 896 |
+
# ===================== extended ops (EXT_OPS) =====================
|
| 897 |
+
# Extra ops that ride the rule engine + FST only (no neural flag model); they work on KNOWN
|
| 898 |
+
# words today:
|
| 899 |
+
# - potential / imperative : new VERB moods; FST handles known verbs.
|
| 900 |
+
# - clitic_* : pure string attach (vowel harmony).
|
| 901 |
+
# - plural / singular : NOUN number + verb re-agreement.
|
| 902 |
+
# Pipeline routes everything through ALL_OPS = {**OPS, **EXT_OPS}.
|
| 903 |
+
|
| 904 |
+
# suppletive personal pronouns (FST cannot derive minä<->me); keyed by (verb person, target number)
|
| 905 |
+
_PRON_BY = {("1", "Sing"): "minä", ("1", "Plur"): "me",
|
| 906 |
+
("2", "Sing"): "sinä", ("2", "Plur"): "te",
|
| 907 |
+
("3", "Sing"): "hän", ("3", "Plur"): "he"}
|
| 908 |
+
|
| 909 |
+
|
| 910 |
+
def _flip_number(sent, subj, target):
|
| 911 |
+
"""subj noun + agreeing nominative modifiers -> Number=target; {id: form} or None."""
|
| 912 |
+
out = {}
|
| 913 |
+
for w in [subj] + [c for c in sent if c["head"] == subj["id"] and c["deprel"] in AGREE
|
| 914 |
+
and c["feats"].get("Case") == subj["feats"].get("Case")
|
| 915 |
+
and c["feats"].get("Number") == subj["feats"].get("Number")]:
|
| 916 |
+
form = _realize_form(w["lemma"], w["upos"], {**w["feats"], "Number": target}, w["form"])
|
| 917 |
+
if not form or form == w["form"]: # OOV miss, or vacuous change -> abstain
|
| 918 |
+
return None
|
| 919 |
+
out[w["id"]] = form
|
| 920 |
+
return out
|
| 921 |
+
|
| 922 |
+
|
| 923 |
+
def op_imperative(sent, nonce_rng=None):
|
| 924 |
+
"""Affirmative 2sg imperative: verb -> Imprt+Sg2, drop the subject, total object -> Nom.
|
| 925 |
+
Accepts a finite Act root in indicative OR imperative mood; which clauses may be commanded
|
| 926 |
+
(2nd-person/subjectless) is enforced by `imperative_in_scope` at the pipeline gate."""
|
| 927 |
+
verb = _imperative_source_verb(sent)
|
| 928 |
+
if verb is None:
|
| 929 |
+
return None
|
| 930 |
+
repl_c = _rand_onset(nonce_rng, verb["form"][:1].lower()) if nonce_rng else None
|
| 931 |
+
v = _netlex_finite(verb["lemma"], "Act", "Imprt", None, "Sg2") if _NETLEX_BACKEND else None
|
| 932 |
+
if not v: # OOV/loan verb: Giella-free finiteverb (Imprt+Sg2)
|
| 933 |
+
v = _neural_imprt(verb["lemma"], "Sg2")
|
| 934 |
+
if not v:
|
| 935 |
+
return None
|
| 936 |
+
if repl_c:
|
| 937 |
+
v = swap_initial(v, repl_c)
|
| 938 |
+
drop = set()
|
| 939 |
+
for t in sent:
|
| 940 |
+
if t["deprel"] in ("nsubj", "nsubj:cop"):
|
| 941 |
+
drop |= _subtree_ids(sent, t["id"])
|
| 942 |
+
repl = {}
|
| 943 |
+
for t in sent:
|
| 944 |
+
if t["deprel"] == "obj" and t["feats"].get("Case") in ("Gen", "Nom") \
|
| 945 |
+
and t["upos"] in ("NOUN", "PROPN", "ADJ", "NUM", "PRON"):
|
| 946 |
+
fl = _flip_case(sent, t, "Nom")
|
| 947 |
+
if fl:
|
| 948 |
+
repl.update(fl)
|
| 949 |
+
src, tgt = [], []
|
| 950 |
+
for t in sent:
|
| 951 |
+
f = swap_initial(t["form"], repl_c) if (repl_c and t["id"] == verb["id"]) else t["form"]
|
| 952 |
+
src.append(f)
|
| 953 |
+
if t["id"] in drop:
|
| 954 |
+
continue
|
| 955 |
+
tgt.append(v if t["id"] == verb["id"] else repl.get(t["id"], t["form"]))
|
| 956 |
+
return src, tgt
|
| 957 |
+
|
| 958 |
+
|
| 959 |
+
# ---- full imperative paradigm (beyond Sg2) + negative imperative -----------------------
|
| 960 |
+
# `imperative` (Sg2 affirmative) stays the TRAINED flag (untouched). The other persons and the
|
| 961 |
+
# negative paradigm ride the FST + rules (known verbs), with OOV only for the Sg2 forms (which can
|
| 962 |
+
# borrow the trained `imperative` flag). The negative imperative is the älä-series particle +
|
| 963 |
+
# connegative imperative (`menkö`); Sg2 negative keeps the plain Sg2 form (`älä mene`).
|
| 964 |
+
_IMPRT_TAG = {"sg2": "Sg2", "sg3": "Sg3", "pl1": "Pl1", "pl2": "Pl2", "pl3": "Pl3"}
|
| 965 |
+
_ALA = {"sg2": "älä", "sg3": "älköön", "pl1": "älkäämme", "pl2": "älkää", "pl3": "älkööt"}
|
| 966 |
+
|
| 967 |
+
|
| 968 |
+
def _imprt_form(lemma, tag):
|
| 969 |
+
"""Affirmative imperative surface for a person/number tag (lexicon-first, the standard literary
|
| 970 |
+
form — `mennä` -> 'mene', not the colloquial contraction 'mee'). None on a miss."""
|
| 971 |
+
if not _NETLEX_BACKEND:
|
| 972 |
+
return None
|
| 973 |
+
return _netlex_finite(lemma, "Act", "Imprt", None, tag)
|
| 974 |
+
|
| 975 |
+
|
| 976 |
+
def _op_imperative_person(slot, pol):
|
| 977 |
+
"""Imperative for a person/number `slot` (sg2/sg3/pl1/pl2/pl3), affirmative or negative. Same
|
| 978 |
+
clause surgery as `op_imperative` (drop the subject, total object -> Nom). OOV (FST miss) is
|
| 979 |
+
supported only for the Sg2 forms via the trained `imperative` flag; other slots abstain."""
|
| 980 |
+
tag = _IMPRT_TAG[slot]
|
| 981 |
+
|
| 982 |
+
def fn(sent, nonce_rng=None):
|
| 983 |
+
verb = _imperative_source_verb(sent)
|
| 984 |
+
if verb is None:
|
| 985 |
+
return None
|
| 986 |
+
repl_c = _rand_onset(nonce_rng, verb["form"][:1].lower()) if nonce_rng else None
|
| 987 |
+
if pol == "Aff":
|
| 988 |
+
v = _imprt_form(verb["lemma"], tag)
|
| 989 |
+
if not v: # OOV/loan verb: Giella-free finiteverb (Imprt+tag)
|
| 990 |
+
v = _neural_imprt(verb["lemma"], tag)
|
| 991 |
+
if not v:
|
| 992 |
+
if verb["upos"] == "VERB":
|
| 993 |
+
_OOV_UNREALIZABLE.append(f"{verb['lemma']} (VERB) Imprt+{tag}")
|
| 994 |
+
return None
|
| 995 |
+
cluster = [v]
|
| 996 |
+
else: # negative: älä-series + connegative
|
| 997 |
+
if slot == "sg2": # `älä mene` keeps the plain Sg2 form
|
| 998 |
+
conneg = _imprt_form(verb["lemma"], "Sg2")
|
| 999 |
+
if not conneg: # OOV/loan verb: Giella-free finiteverb
|
| 1000 |
+
conneg = _neural_imprt(verb["lemma"], "Sg2")
|
| 1001 |
+
else: # `älkää menkö` uses the -kO connegative
|
| 1002 |
+
conneg = (_netlex_finite(verb["lemma"], "Act", "Imprt", None, "ConNeg")
|
| 1003 |
+
if _NETLEX_BACKEND else None)
|
| 1004 |
+
if not conneg: # OOV/loan verb: Giella-free finiteverb
|
| 1005 |
+
conneg = _neural_imprt(verb["lemma"], "ConNeg")
|
| 1006 |
+
if not conneg:
|
| 1007 |
+
if verb["upos"] == "VERB":
|
| 1008 |
+
_OOV_UNREALIZABLE.append(f"{verb['lemma']} (VERB) neg-Imprt+{tag}")
|
| 1009 |
+
return None
|
| 1010 |
+
cluster = [_ALA[slot], conneg]
|
| 1011 |
+
if repl_c:
|
| 1012 |
+
cluster[-1] = swap_initial(cluster[-1], repl_c)
|
| 1013 |
+
drop = set()
|
| 1014 |
+
for t in sent:
|
| 1015 |
+
if t["deprel"] in ("nsubj", "nsubj:cop"):
|
| 1016 |
+
drop |= _subtree_ids(sent, t["id"])
|
| 1017 |
+
newcase = "Nom" if pol == "Aff" else "Par" # negative imperative forces partitive object
|
| 1018 |
+
repl = {}
|
| 1019 |
+
for t in sent:
|
| 1020 |
+
if t["deprel"] == "obj" and t["feats"].get("Case") in ("Gen", "Nom") \
|
| 1021 |
+
and t["upos"] in ("NOUN", "PROPN", "ADJ", "NUM", "PRON"):
|
| 1022 |
+
fl = _flip_case(sent, t, newcase)
|
| 1023 |
+
if fl:
|
| 1024 |
+
repl.update(fl)
|
| 1025 |
+
src, tgt = [], []
|
| 1026 |
+
for t in sent:
|
| 1027 |
+
f = swap_initial(t["form"], repl_c) if (repl_c and t["id"] == verb["id"]) else t["form"]
|
| 1028 |
+
src.append(f)
|
| 1029 |
+
if t["id"] in drop:
|
| 1030 |
+
continue
|
| 1031 |
+
tgt.extend(cluster if t["id"] == verb["id"] else [repl.get(t["id"], t["form"])])
|
| 1032 |
+
return src, tgt
|
| 1033 |
+
return fn
|
| 1034 |
+
|
| 1035 |
+
|
| 1036 |
+
# accepted particle (canonical key OR a surface variant) -> (back-harmony variant, front-harmony variant).
|
| 1037 |
+
# -kin is harmony-invariant. Both members of a harmony pair are accepted as input so a caller may pass
|
| 1038 |
+
# either 'ko' or 'kö' (e.g. copied verbatim off an inflect-by-example model word).
|
| 1039 |
+
_CLITIC_VARIANTS = {
|
| 1040 |
+
"kin": ("kin", "kin"),
|
| 1041 |
+
"kaan": ("kaan", "kään"), "kään": ("kaan", "kään"),
|
| 1042 |
+
"ko": ("ko", "kö"), "kö": ("ko", "kö"),
|
| 1043 |
+
"han": ("han", "hän"), "hän": ("han", "hän"),
|
| 1044 |
+
"pa": ("pa", "pä"), "pä": ("pa", "pä"),
|
| 1045 |
+
}
|
| 1046 |
+
|
| 1047 |
+
|
| 1048 |
+
def _clitic_for(particle, harmony_word):
|
| 1049 |
+
"""The vowel-harmony variant of an enclitic (-kin/-kaan/-ko/-han/-pa), chosen by the host word's
|
| 1050 |
+
vowels. Accepts the canonical particle or a surface variant ('ko'/'kö'). Returns None for an
|
| 1051 |
+
unrecognized particle so the caller can abstain instead of crashing (was a KeyError)."""
|
| 1052 |
+
pair = _CLITIC_VARIANTS.get(particle.lower()) if particle else None
|
| 1053 |
+
if pair is None:
|
| 1054 |
+
return None
|
| 1055 |
+
back = any(ch in BACK for ch in harmony_word.lower())
|
| 1056 |
+
return pair[0] if back else pair[1]
|
| 1057 |
+
|
| 1058 |
+
|
| 1059 |
+
def op_clitic(particle):
|
| 1060 |
+
"""Attach a focus/pragmatic clitic (-kin/-kaan/-han/-pa) to the finite verb (vowel harmony)."""
|
| 1061 |
+
def fn(sent, nonce_rng=None):
|
| 1062 |
+
verb, _ = _main_verb(sent)
|
| 1063 |
+
if verb is None:
|
| 1064 |
+
return None
|
| 1065 |
+
form = verb["form"]
|
| 1066 |
+
repl_c = _rand_onset(nonce_rng, form[:1].lower()) if nonce_rng else None
|
| 1067 |
+
if repl_c:
|
| 1068 |
+
form = swap_initial(form, repl_c)
|
| 1069 |
+
suffix = _clitic_for(particle, form) # harmony follows the realized verb surface, not the lemma
|
| 1070 |
+
if suffix is None:
|
| 1071 |
+
return None
|
| 1072 |
+
new = form + suffix
|
| 1073 |
+
src = [swap_initial(t["form"], repl_c) if (repl_c and t["id"] == verb["id"]) else t["form"]
|
| 1074 |
+
for t in sent]
|
| 1075 |
+
tgt = [new if t["id"] == verb["id"] else t["form"] for t in sent]
|
| 1076 |
+
return src, tgt
|
| 1077 |
+
return fn
|
| 1078 |
+
|
| 1079 |
+
|
| 1080 |
+
def _op_number(target):
|
| 1081 |
+
"""Change the (nominative) subject's number and re-agree the finite verb. target=Plur|Sing."""
|
| 1082 |
+
def fn(sent, nonce_rng=None):
|
| 1083 |
+
c = clause(sent)
|
| 1084 |
+
if c is None:
|
| 1085 |
+
return None
|
| 1086 |
+
verb, is_cop, base = c
|
| 1087 |
+
if base["number"] == target:
|
| 1088 |
+
return None
|
| 1089 |
+
subj = next((t for t in sent if t["deprel"] in ("nsubj", "nsubj:cop")), None)
|
| 1090 |
+
if subj is None or subj["feats"].get("Case", "Nom") != "Nom":
|
| 1091 |
+
return None
|
| 1092 |
+
repl = {}
|
| 1093 |
+
if subj["feats"].get("PronType") == "Prs": # suppletive personal pronoun
|
| 1094 |
+
new = _PRON_BY.get((base["person"], target))
|
| 1095 |
+
if new is None:
|
| 1096 |
+
return None
|
| 1097 |
+
repl[subj["id"]] = new[:1].upper() + new[1:] if subj["form"][:1].isupper() else new
|
| 1098 |
+
else:
|
| 1099 |
+
fl = _flip_number(sent, subj, target)
|
| 1100 |
+
if fl is None:
|
| 1101 |
+
return None
|
| 1102 |
+
repl.update(fl)
|
| 1103 |
+
vc = verb_cluster(verb["lemma"], base["voice"], base["mood"], base["tense"],
|
| 1104 |
+
base["pol"], base["person"], target)
|
| 1105 |
+
if vc is None:
|
| 1106 |
+
return None
|
| 1107 |
+
if is_cop: # copula clause: re-agree the predicate (root) + its conjuncts
|
| 1108 |
+
pred_id = verb["head"]
|
| 1109 |
+
for t in sent:
|
| 1110 |
+
if (t["id"] == pred_id or (t["head"] == pred_id and t["deprel"] == "conj")) \
|
| 1111 |
+
and t["upos"] in ("ADJ", "NOUN") and t["feats"].get("Case") == "Nom":
|
| 1112 |
+
pf = _realize_form(t["lemma"], t["upos"], {**t["feats"], "Number": target}, t["form"])
|
| 1113 |
+
if pf is None: # can't realize an agreeing predicate -> abstain, never
|
| 1114 |
+
return None # emit a half-agreed NP (subject Pl, predicate Sg)
|
| 1115 |
+
if pf != t["form"]:
|
| 1116 |
+
repl[t["id"]] = pf
|
| 1117 |
+
src, tgt = [], []
|
| 1118 |
+
for t in sent:
|
| 1119 |
+
src.append(t["form"])
|
| 1120 |
+
if t["id"] == verb["id"]:
|
| 1121 |
+
tgt.extend(vc)
|
| 1122 |
+
else:
|
| 1123 |
+
tgt.append(repl.get(t["id"], t["form"]))
|
| 1124 |
+
return src, tgt
|
| 1125 |
+
return fn
|
| 1126 |
+
|
| 1127 |
+
|
| 1128 |
+
_PX_PERSON = {"poss_1sg": "1Sg", "poss_2sg": "2Sg", "poss_3": "3",
|
| 1129 |
+
"poss_1pl": "1Pl", "poss_2pl": "2Pl"}
|
| 1130 |
+
|
| 1131 |
+
|
| 1132 |
+
def _op_poss(person):
|
| 1133 |
+
"""Attach a possessive suffix (omistusliite) to the subject NP HEAD, preserving its case/number
|
| 1134 |
+
(talo->taloni, talossa->talossani). Structural rule: the engine picks the head noun;
|
| 1135 |
+
the realizer answers per-token (FST first, neural OOV second), so it works on loan/OOV heads
|
| 1136 |
+
(striimaaja->striimaajani). Agreeing modifiers (adjectives) keep their own form — Px is on the
|
| 1137 |
+
head only. Abstains if the subject is not a realizable nominal."""
|
| 1138 |
+
def fn(sent, nonce_rng=None):
|
| 1139 |
+
head = next((t for t in sent if t["deprel"] in ("nsubj", "nsubj:cop")
|
| 1140 |
+
and t["upos"] in ("NOUN", "PROPN")), None)
|
| 1141 |
+
if head is None:
|
| 1142 |
+
return None
|
| 1143 |
+
f = _realize_form(head["lemma"], head["upos"], {**head["feats"], "Px": person}, head["form"])
|
| 1144 |
+
if not f or f == head["form"]:
|
| 1145 |
+
return None
|
| 1146 |
+
src = [t["form"] for t in sent]
|
| 1147 |
+
tgt = [f if t["id"] == head["id"] else t["form"] for t in sent]
|
| 1148 |
+
return src, tgt
|
| 1149 |
+
return fn
|
| 1150 |
+
|
| 1151 |
+
|
| 1152 |
+
# ---- combined nominal morphology (compose number + Px + clitic on the subject head) ----
|
| 1153 |
+
def parse_nominal_bundle(op):
|
| 1154 |
+
"""A '+'-composed op of >=2 NOMINAL atoms (plural/singular, poss_*, clitic_*) -> a feature
|
| 1155 |
+
bundle dict {number?, px?, clitic?}, else None. Each dimension may appear once. Single atoms and
|
| 1156 |
+
any verb/clause atom return None (they use the existing per-op path). This is what lets a user
|
| 1157 |
+
ask for `plural+poss_1sg+clitic_kin` and get ONE correctly-ordered realization, not a stack of
|
| 1158 |
+
string rewrites (you cannot pluralize the string `taloni`)."""
|
| 1159 |
+
atoms = op.split("+")
|
| 1160 |
+
if len(atoms) < 2:
|
| 1161 |
+
return None
|
| 1162 |
+
b = {}
|
| 1163 |
+
for a in atoms:
|
| 1164 |
+
if a in ("plural", "singular"):
|
| 1165 |
+
k, v = "number", ("Plur" if a == "plural" else "Sing")
|
| 1166 |
+
elif a in _PX_PERSON:
|
| 1167 |
+
k, v = "px", _PX_PERSON[a]
|
| 1168 |
+
elif a.startswith("clitic_"):
|
| 1169 |
+
k, v = "clitic", a.split("_", 1)[1]
|
| 1170 |
+
else:
|
| 1171 |
+
return None # a non-nominal atom -> not a nominal bundle
|
| 1172 |
+
if k in b:
|
| 1173 |
+
return None # same dimension twice -> ill-formed
|
| 1174 |
+
b[k] = v
|
| 1175 |
+
return b
|
| 1176 |
+
|
| 1177 |
+
|
| 1178 |
+
def _apply_nominal_bundle(sent, bundle):
|
| 1179 |
+
"""Compose nominal morphology on the subject head in ONE realization: realize the head
|
| 1180 |
+
with the FINAL Number+Case+Px (never re-inflect an already-realized string), attach the clitic
|
| 1181 |
+
AFTER Px (surface order stem+case+number+Px+clitic), re-agree modifiers in number, and re-agree
|
| 1182 |
+
the finite verb if number changed. Case is preserved (a case-change op is separate)."""
|
| 1183 |
+
head = next((t for t in sent if t["deprel"] in ("nsubj", "nsubj:cop")
|
| 1184 |
+
and t["upos"] in ("NOUN", "PROPN")), None)
|
| 1185 |
+
if head is None:
|
| 1186 |
+
return None
|
| 1187 |
+
cur_num = head["feats"].get("Number", "Sing")
|
| 1188 |
+
number = bundle.get("number", cur_num)
|
| 1189 |
+
px = bundle.get("px", "None")
|
| 1190 |
+
feats = {**head["feats"], "Number": number, "Px": px} # Case preserved from the head
|
| 1191 |
+
new = _realize_form(head["lemma"], head["upos"], feats, head["form"])
|
| 1192 |
+
if not new:
|
| 1193 |
+
return None
|
| 1194 |
+
if "clitic" in bundle: # clitic attaches AFTER Px, on the head
|
| 1195 |
+
suffix = _clitic_for(bundle["clitic"], new) # harmony follows the realized surface host
|
| 1196 |
+
if suffix is None:
|
| 1197 |
+
return None
|
| 1198 |
+
new = new + suffix
|
| 1199 |
+
if new == head["form"]: # bundle was a no-op on this head
|
| 1200 |
+
return None
|
| 1201 |
+
repl = {head["id"]: new}
|
| 1202 |
+
|
| 1203 |
+
if number != cur_num: # number changed: agree modifiers + verb
|
| 1204 |
+
for m in sent:
|
| 1205 |
+
if m["head"] == head["id"] and m["deprel"] in AGREE and m["upos"] in ("ADJ", "NOUN"):
|
| 1206 |
+
mf = _realize_form(m["lemma"], m["upos"], {**m["feats"], "Number": number}, m["form"])
|
| 1207 |
+
if mf is None:
|
| 1208 |
+
return None
|
| 1209 |
+
repl[m["id"]] = mf
|
| 1210 |
+
c = clause(sent)
|
| 1211 |
+
if c is None:
|
| 1212 |
+
return None
|
| 1213 |
+
verb, _is_cop, base = c
|
| 1214 |
+
vc = verb_cluster(verb["lemma"], base["voice"], base["mood"], base["tense"],
|
| 1215 |
+
base["pol"], base["person"], number)
|
| 1216 |
+
if vc is None:
|
| 1217 |
+
return None
|
| 1218 |
+
src, tgt = [], []
|
| 1219 |
+
for t in sent:
|
| 1220 |
+
src.append(t["form"])
|
| 1221 |
+
tgt.extend(vc if t["id"] == verb["id"] else [repl.get(t["id"], t["form"])])
|
| 1222 |
+
return src, tgt
|
| 1223 |
+
|
| 1224 |
+
src = [t["form"] for t in sent]
|
| 1225 |
+
tgt = [repl.get(t["id"], t["form"]) for t in sent]
|
| 1226 |
+
return src, tgt
|
| 1227 |
+
|
| 1228 |
+
|
| 1229 |
+
# ---- phrase-level nominal morphology utility (NOT a clause transform) -----------------
|
| 1230 |
+
# Case is syntactically GOVERNED, so arbitrary case is exposed here as a phrase utility, never as a
|
| 1231 |
+
# finite-clause op — `transform` keeps grammaticality, `realize_np` does open-vocab nominal
|
| 1232 |
+
# morphology (`talo`->`talossa`, `striimaaja`->`striimaajillanikin`).
|
| 1233 |
+
def realize_np_form(lemma, upos="NOUN", case="Nom", number="Sing", px="None", clitic=None,
|
| 1234 |
+
surface=None):
|
| 1235 |
+
"""Realize ONE nominal head: lemma -> target Case/Number(+Px), optional clitic after Px. Pure
|
| 1236 |
+
morphology, no parser; FST first then neural OOV via `_realize_form`. Returns the form or None
|
| 1237 |
+
(FST+neural both failed / suppressed). Does NOT clear the OOV trace lists — the caller does."""
|
| 1238 |
+
f = _realize_form(lemma, upos, {"Case": case, "Number": number, "Px": px}, surface or lemma)
|
| 1239 |
+
if f is None:
|
| 1240 |
+
return None
|
| 1241 |
+
if not clitic:
|
| 1242 |
+
return f
|
| 1243 |
+
suffix = _clitic_for(clitic, f) # harmony follows the realized SURFACE host, not the lemma (kuka->ketäkö)
|
| 1244 |
+
return f + suffix if suffix is not None else None
|
| 1245 |
+
|
| 1246 |
+
|
| 1247 |
+
# ---- non-finite verb forms (participles + infinitives) as a phrase utility ------------
|
| 1248 |
+
# A second phrase-level morphology utility (like realize_np): inflect a VERB lemma into a non-finite
|
| 1249 |
+
# form — the four participles (VA/NUT/TAVA/TU, which inflect like adjectives in case+number) and the
|
| 1250 |
+
# 2nd/3rd infinitive cases (mennessä/menemässä/menemästä/menemään/menemällä/menemättä). The FST query
|
| 1251 |
+
# layer (gen.giella_query) already supports these; this just exposes them with the FST→neural OOV
|
| 1252 |
+
# discipline. NUT is special-cased for OOV (it is the perfect participle, available via the trained
|
| 1253 |
+
# `perfect` flag); the other forms abstain for OOV verbs until a verbform realizer is trained.
|
| 1254 |
+
_VERBFORM = {
|
| 1255 |
+
"va": {"VerbForm": "Part", "PartForm": "Pres", "Voice": "Act"}, # present active (lukeva)
|
| 1256 |
+
"nut": {"VerbForm": "Part", "PartForm": "Past", "Voice": "Act"}, # past active (lukenut)
|
| 1257 |
+
"tava": {"VerbForm": "Part", "PartForm": "Pres", "Voice": "Pss"}, # present passive (luettava)
|
| 1258 |
+
"tu": {"VerbForm": "Part", "PartForm": "Past", "Voice": "Pss"}, # past passive (luettu)
|
| 1259 |
+
"inf2": {"VerbForm": "Inf", "InfForm": "2", "Voice": "Act"}, # 2nd infinitive (-essa/-en)
|
| 1260 |
+
"inf2_pass": {"VerbForm": "Inf", "InfForm": "2", "Voice": "Pss"}, # passive 2nd inf (tehtäessä; inessive-only)
|
| 1261 |
+
"inf3": {"VerbForm": "Inf", "InfForm": "3", "Voice": "Act"}, # 3rd infinitive (-ma- cases)
|
| 1262 |
+
"ma": {"VerbForm": "Part", "PartForm": "Agt"}, # agent participle (tekemä)
|
| 1263 |
+
}
|
| 1264 |
+
VERBFORMS = tuple(_VERBFORM)
|
| 1265 |
+
|
| 1266 |
+
# Possessive participle paradigm (the temporal construction, kuultuansa). The Px=None oblique participle
|
| 1267 |
+
# is already correct (sanoneessa); the Px form is that base's vowel stem + the possessive suffix, exactly
|
| 1268 |
+
# as for nominals (_derive_genpl_possessive) — so we DERIVE it instead of letting the noun realizer
|
| 1269 |
+
# re-inflect the participle from the wrong oblique stem. Singular + the five participles only; everything
|
| 1270 |
+
# else abstains (the caller keeps its OOV/flagged behaviour). Validated against the Giella FST
|
| 1271 |
+
# across conjugation types 1–6 × case × Px. We emit the uniform -nsa/-nsä
|
| 1272 |
+
# 3rd-person variant (the -Vn variant, sanoneessaan, is also FST-valid; -nsa is what the working tu+Par
|
| 1273 |
+
# path already produced as kuultuansa).
|
| 1274 |
+
_PARTICIPLE_PX_FORMS = frozenset({"va", "nut", "tava", "tu", "ma"})
|
| 1275 |
+
_PARTICIPLE_PX_SUFFIX = {"1Sg": "ni", "2Sg": "si", "1Pl": "mme", "2Pl": "nne"}
|
| 1276 |
+
_VOWELS = set("aeiouyäö")
|
| 1277 |
+
|
| 1278 |
+
|
| 1279 |
+
def _participle_px_stem(form, case, base_same, base_gen, base_nom):
|
| 1280 |
+
"""The stem the possessive suffix attaches to (from the engine's own Px=None bases), or None to abstain.
|
| 1281 |
+
Three structural patterns: translative -ksi -> -kse-; nominative/genitive use the strong/oblique stem
|
| 1282 |
+
(tu: the strong NOM stem for gradation; nut: the genitive stem); every other case uses its own
|
| 1283 |
+
vowel-final form (or the vowel stem of an -n-final form, e.g. illative sanoneeseen)."""
|
| 1284 |
+
vow = lambda s: bool(s) and s[-1] in _VOWELS
|
| 1285 |
+
if case == "Tra": # -ksi -> -kse- (sanoneeksi -> sanoneekse-)
|
| 1286 |
+
return base_same[:-1] + "e" if (base_same and base_same.endswith("ksi")) else None
|
| 1287 |
+
if case in ("Nom", "Gen"):
|
| 1288 |
+
if form == "tu": # gradation: strong NOM stem (sanottu)
|
| 1289 |
+
return base_nom if vow(base_nom) else None
|
| 1290 |
+
if case == "Nom" and vow(base_nom): # va/ma: vowel-final nominative (sanova/sanoma)
|
| 1291 |
+
return base_nom
|
| 1292 |
+
return base_gen[:-1] if (base_gen and base_gen.endswith("n")) else None # nut: genitive stem (sanonee-)
|
| 1293 |
+
if vow(base_same): # general oblique (sanoneessa, sanonutta)
|
| 1294 |
+
return base_same
|
| 1295 |
+
return base_same[:-1] if (base_same and base_same.endswith("n")) else None
|
| 1296 |
+
|
| 1297 |
+
|
| 1298 |
+
def _derive_participle_possessive(lemma, form, case, number, px):
|
| 1299 |
+
"""Modern possessive participle derived from the engine's own correct Px=None oblique base by a pure
|
| 1300 |
+
string rule, instead of the noun realizer re-inflecting the participle from the wrong oblique stem.
|
| 1301 |
+
Singular participles only; returns the form or None (abstain)."""
|
| 1302 |
+
if number != "Sing" or form not in _PARTICIPLE_PX_FORMS:
|
| 1303 |
+
return None
|
| 1304 |
+
suf = _PARTICIPLE_PX_SUFFIX.get(px)
|
| 1305 |
+
if suf is None:
|
| 1306 |
+
if px != "3":
|
| 1307 |
+
return None
|
| 1308 |
+
suf = "nsa" if any(c in BACK for c in lemma.lower()) else "nsä" # 3rd person: harmony from the lemma
|
| 1309 |
+
# the engine's own Px=None bases (FST-parity for known verbs; the verbform net for OOV). The derived
|
| 1310 |
+
# form inherits the base's provenance unchanged: the verbform Px=None serving path returns its net
|
| 1311 |
+
# output UNFLAGGED, so the derivation is unflagged too — consistent within the verbform subsystem.
|
| 1312 |
+
# (Verbform neural-provenance flagging is a broader pre-existing gap vs. the nominal path.)
|
| 1313 |
+
base_same = realize_verbform_form(lemma, form, case, number, "None")
|
| 1314 |
+
base_gen = realize_verbform_form(lemma, form, "Gen", number, "None")
|
| 1315 |
+
base_nom = realize_verbform_form(lemma, form, "Nom", number, "None")
|
| 1316 |
+
stem = _participle_px_stem(form, case, base_same, base_gen, base_nom)
|
| 1317 |
+
return stem + suf if stem is not None else None
|
| 1318 |
+
|
| 1319 |
+
|
| 1320 |
+
def realize_verbform_form(lemma, form="va", case="Nom", number="Sing", px="None", surface=None):
|
| 1321 |
+
"""Realize ONE non-finite verb form: lemma -> participle/infinitive in the target Case/Number,
|
| 1322 |
+
optionally with a possessive suffix `px` (one of the realizer Px values). Px on a participle
|
| 1323 |
+
licenses the temporal construction, e.g. form='tu', case='Par', px='3' -> 'kuultuaan'. `form` is
|
| 1324 |
+
one of VERBFORMS. Lexicon-first then the verbform net; returns the form or None on a miss (OOV) —
|
| 1325 |
+
the caller decides whether to abstain. Pure morphology, no parser."""
|
| 1326 |
+
spec = _VERBFORM.get(form)
|
| 1327 |
+
if spec is None:
|
| 1328 |
+
return None
|
| 1329 |
+
if not _NETLEX_BACKEND:
|
| 1330 |
+
return None
|
| 1331 |
+
# Lexicon-first then the verbform net (keyed form+case+number, no Px dimension). A Px-bearing
|
| 1332 |
+
# temporal construction (e.g. 'kuultuansa') is DERIVED from the engine's own correct Px=None oblique
|
| 1333 |
+
# base by a pure string rule, instead of re-inflecting via the noun realizer (which used the wrong
|
| 1334 |
+
# oblique stem). Returns None -> abstain for plural / infinitives.
|
| 1335 |
+
if px and px != "None":
|
| 1336 |
+
return _derive_participle_possessive(lemma, form, case, number, px)
|
| 1337 |
+
if _GENLEX is not None:
|
| 1338 |
+
hit = _GENLEX.lookup_nonfin(lemma, form, case, number)
|
| 1339 |
+
if hit is not None:
|
| 1340 |
+
return hit # trusted lexicon (byte-exact) — no flag
|
| 1341 |
+
if _VERBFORM_REALIZER is not None:
|
| 1342 |
+
out, _conf = _VERBFORM_REALIZER(lemma, form, case, number)
|
| 1343 |
+
if out and "<unk>" not in out: # shape filter (mirror _neural_oov_ok); fail closed on <unk>
|
| 1344 |
+
# net-served verbform -> flag `neural_unverified` like the nominal path, so the best-effort
|
| 1345 |
+
# tail is HONESTLY marked (the verbform net has the same parity-or-flag contract; e.g.
|
| 1346 |
+
# `haukkua` va Ine -> `haukkuvussu` is wrong and must not pass as verified).
|
| 1347 |
+
_NEURAL_OOV.append(f"{lemma}->{out} ({form} {case}/{number})")
|
| 1348 |
+
return out
|
| 1349 |
+
return None
|
| 1350 |
+
|
| 1351 |
+
|
| 1352 |
+
# imperative is in the TRAINED set (registered now that op_imperative exists). potential rides
|
| 1353 |
+
# VERB_OPS->OPS. EXT_OPS keeps only the ops that don't need the neural model: clitics (pure string
|
| 1354 |
+
# attach) and number (would ride a future neural NOUN path, not this model).
|
| 1355 |
+
OPS["imperative"] = op_imperative
|
| 1356 |
+
|
| 1357 |
+
# compound-perfect tenses (FST for known verbs; OOV reuses the trained `perfect` participle
|
| 1358 |
+
# + an FST olla auxiliary). Each requires a present indicative base.
|
| 1359 |
+
_PERFECT_EXT = {
|
| 1360 |
+
"pluperfect": ({"tense": "Pqp"}, {"tense": "Prs", "mood": "Ind"}),
|
| 1361 |
+
"cond_perfect": ({"mood": "Cond", "tense": "Prf"}, {"tense": "Prs", "mood": "Ind"}),
|
| 1362 |
+
"pot_perfect": ({"mood": "Pot", "tense": "Prf"}, {"tense": "Prs", "mood": "Ind"}),
|
| 1363 |
+
}
|
| 1364 |
+
# imperative paradigm beyond the trained Sg2 flag, affirmative + negative (älä-series).
|
| 1365 |
+
_IMPRT_EXT = {f"imperative_{s}": _op_imperative_person(s, "Aff") for s in ("sg3", "pl1", "pl2", "pl3")}
|
| 1366 |
+
_IMPRT_EXT["neg_imperative"] = _op_imperative_person("sg2", "Neg")
|
| 1367 |
+
_IMPRT_EXT.update({f"neg_imperative_{s}": _op_imperative_person(s, "Neg")
|
| 1368 |
+
for s in ("sg3", "pl1", "pl2", "pl3")})
|
| 1369 |
+
|
| 1370 |
+
EXT_OPS = {
|
| 1371 |
+
"plural": _op_number("Plur"),
|
| 1372 |
+
"singular": _op_number("Sing"),
|
| 1373 |
+
"clitic_kin": op_clitic("kin"),
|
| 1374 |
+
"clitic_kaan": op_clitic("kaan"),
|
| 1375 |
+
"clitic_han": op_clitic("han"),
|
| 1376 |
+
"clitic_pa": op_clitic("pa"),
|
| 1377 |
+
# possessive suffix on the subject head (FST + neural OOV via the Px realizer)
|
| 1378 |
+
**{name: _op_poss(px) for name, px in _PX_PERSON.items()},
|
| 1379 |
+
# compound tenses + full imperative paradigm (FST/rules; targeted OOV reuse)
|
| 1380 |
+
**{name: _verbop(d, r) for name, (d, r) in _PERFECT_EXT.items()},
|
| 1381 |
+
**_IMPRT_EXT,
|
| 1382 |
+
}
|
| 1383 |
+
ALL_OPS = {**OPS, **EXT_OPS}
|
| 1384 |
+
|
| 1385 |
+
|
| 1386 |
+
# ----------------------------- clause scope + inference entry point -------------------------------
|
| 1387 |
+
def _nfin(s):
|
| 1388 |
+
return sum(1 for t in s if t["feats"].get("VerbForm") == "Fin")
|
| 1389 |
+
|
| 1390 |
+
|
| 1391 |
+
# Clause SCOPE: a single finite clause with one clear main predicate. Out-of-scope inputs
|
| 1392 |
+
# (subordinate/coordinate clauses, relatives, etc.) are abstained on — this removes the
|
| 1393 |
+
# wrong-target error class and makes "transform a clause" a well-defined primitive.
|
| 1394 |
+
_SUBORD = {"advcl", "ccomp", "acl", "acl:relcl", "parataxis", "csubj", "csubj:cop", "mark"}
|
| 1395 |
+
|
| 1396 |
+
|
| 1397 |
+
def in_scope(s, max_words=12):
|
| 1398 |
+
if _nfin(s) != 1 or clause(s) is None or len(s) > max_words:
|
| 1399 |
+
return False
|
| 1400 |
+
return not any(t["deprel"] in _SUBORD for t in s)
|
| 1401 |
+
|
| 1402 |
+
|
| 1403 |
+
# The imperative family applies only when the command's addressee is already in the clause: in scope
|
| 1404 |
+
# iff the clause is subjectless (incl. an already-imperative clause) OR its subject is 2nd-person
|
| 1405 |
+
# (`sinä`/`te`). A non-addressee (3rd/1st-person) subject would have to INVENT an addressee, so it
|
| 1406 |
+
# abstains — see `_imperative_source_verb` for the root requirements.
|
| 1407 |
+
def imperative_in_scope(s, max_words=12):
|
| 1408 |
+
verb = _imperative_source_verb(s)
|
| 1409 |
+
if verb is None or _nfin(s) != 1 or len(s) > max_words:
|
| 1410 |
+
return False
|
| 1411 |
+
if any(t["deprel"] in _SUBORD for t in s):
|
| 1412 |
+
return False
|
| 1413 |
+
has_subject = any(t["deprel"] in ("nsubj", "nsubj:cop") for t in s)
|
| 1414 |
+
return (not has_subject) or verb["feats"].get("Person") == "2"
|
| 1415 |
+
|
| 1416 |
+
|
| 1417 |
+
# Ops whose scope is governed by `imperative_in_scope` (not the general `clause`/`in_scope` gate).
|
| 1418 |
+
IMPERATIVE_OPS = frozenset(
|
| 1419 |
+
o for o in ALL_OPS
|
| 1420 |
+
if o == "imperative" or o.startswith("imperative_") or o.startswith("neg_imperative"))
|
| 1421 |
+
|
| 1422 |
+
|
| 1423 |
+
def hybrid_transform(sent, op, model_bundle=None, force_neural=False):
|
| 1424 |
+
"""Inference: deterministic structure (`apply_clause`) + Giella-free neural realization for OOV
|
| 1425 |
+
verbs/nouns (the finiteverb/verbform/realizer nets, set on this module by `pipeline.Transformer`).
|
| 1426 |
+
`model_bundle` is accepted for backward compat but unused. Returns the transformed sentence
|
| 1427 |
+
string, or None if the op does not apply."""
|
| 1428 |
+
global _FORCE_NEURAL_VC
|
| 1429 |
+
_FORCE_NEURAL_VC = force_neural
|
| 1430 |
+
_NEURAL_OOV.clear() # record which words this transform realized neurally
|
| 1431 |
+
_OOV_UNREALIZABLE.clear() # ...which it could not realize at all (oov_unrealizable)
|
| 1432 |
+
_OOV_UNVERIFIED.clear() # ...and which it realized but suppressed (oov_unverified)
|
| 1433 |
+
bundle = parse_nominal_bundle(op) # composed nominal morphology, one realization
|
| 1434 |
+
try:
|
| 1435 |
+
r = _apply_nominal_bundle(sent, bundle) if bundle is not None else ALL_OPS[op](sent)
|
| 1436 |
+
finally:
|
| 1437 |
+
_FORCE_NEURAL_VC = False
|
| 1438 |
+
return _tgt(r) if r is not None else None
|
juurio/analyze/__init__.py
ADDED
|
File without changes
|
juurio/analyze/data.py
ADDED
|
@@ -0,0 +1,127 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""AnalyzeMux data layer — invert the FST cache into per-form multi-label training examples.
|
| 2 |
+
|
| 3 |
+
The parser-facing reverse model predicts, per surface form:
|
| 4 |
+
- a UNION tag-bag (multi-label over the FST morphtags, lemma excluded) — the exact thing the
|
| 5 |
+
parser's FST channel consumes; and
|
| 6 |
+
- a set of lemma EDIT-SCRIPTS (multi-label over the script inventory) — decode top-K to recover
|
| 7 |
+
candidate lemmas.
|
| 8 |
+
|
| 9 |
+
Training data is the inverted FST cache: each form already carries its full reading list
|
| 10 |
+
`[lemma, tag1, ...]`. We train on the full cache.
|
| 11 |
+
|
| 12 |
+
All functions here are pure + deterministic (no torch, no RNG) so the split and targets are reproducible.
|
| 13 |
+
"""
|
| 14 |
+
from __future__ import annotations
|
| 15 |
+
|
| 16 |
+
import hashlib
|
| 17 |
+
import json
|
| 18 |
+
from collections import Counter
|
| 19 |
+
from pathlib import Path
|
| 20 |
+
from typing import Iterable
|
| 21 |
+
|
| 22 |
+
from juurio.parser.edit_scripts import extract_edit_script
|
| 23 |
+
from juurio.parser.fst_cache import UNK_ID as TAG_UNK_ID
|
| 24 |
+
|
| 25 |
+
SCRIPT_UNK_ID = 0 # reserved id 0 for unseen/rare edit-scripts; real scripts start at 1
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
def edit_script(form: str, lemma: str) -> str:
|
| 29 |
+
"""Surface->lemma edit script as a canonical STRING key. The parser's extractor returns a tuple
|
| 30 |
+
`(delete_count, append_suffix)`; we serialize it `"<n>:<suffix>"` so it is hashable AND JSON-safe
|
| 31 |
+
(tuples round-trip through JSON as lists, which then break dict keys)."""
|
| 32 |
+
n, suffix = extract_edit_script(form, lemma)
|
| 33 |
+
return f"{n}:{suffix}"
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
class ScriptVocab:
|
| 37 |
+
"""Multi-label inventory of surface->lemma edit-scripts. id 0 = UNK (rare/unseen); real from 1."""
|
| 38 |
+
|
| 39 |
+
def __init__(self, scripts: list[str]):
|
| 40 |
+
# `scripts` is the ordered real-script list (UNK is implicit at id 0).
|
| 41 |
+
self._scripts = list(scripts)
|
| 42 |
+
self._to_id = {s: i + 1 for i, s in enumerate(scripts)}
|
| 43 |
+
|
| 44 |
+
@property
|
| 45 |
+
def size(self) -> int:
|
| 46 |
+
"""Total head width = UNK + real scripts."""
|
| 47 |
+
return len(self._scripts) + 1
|
| 48 |
+
|
| 49 |
+
@classmethod
|
| 50 |
+
def fit(cls, cache: dict[str, list[list[str]]], *, min_count: int = 1,
|
| 51 |
+
max_scripts: int | None = None) -> "ScriptVocab":
|
| 52 |
+
c: Counter[str] = Counter()
|
| 53 |
+
for form, readings in cache.items():
|
| 54 |
+
for r in readings:
|
| 55 |
+
if not r:
|
| 56 |
+
continue
|
| 57 |
+
try:
|
| 58 |
+
c[edit_script(form, r[0])] += 1
|
| 59 |
+
except Exception:
|
| 60 |
+
continue
|
| 61 |
+
kept = [s for s, n in c.most_common() if n >= min_count]
|
| 62 |
+
if max_scripts is not None:
|
| 63 |
+
kept = kept[:max_scripts]
|
| 64 |
+
return cls(kept)
|
| 65 |
+
|
| 66 |
+
def encode(self, script: str) -> int:
|
| 67 |
+
return self._to_id.get(script, SCRIPT_UNK_ID)
|
| 68 |
+
|
| 69 |
+
def decode(self, idx: int) -> str:
|
| 70 |
+
if idx == SCRIPT_UNK_ID or not (1 <= idx <= len(self._scripts)):
|
| 71 |
+
return "<UNK>"
|
| 72 |
+
return self._scripts[idx - 1]
|
| 73 |
+
|
| 74 |
+
def save(self, path: str) -> None:
|
| 75 |
+
Path(path).parent.mkdir(parents=True, exist_ok=True)
|
| 76 |
+
with open(path, "w") as fh:
|
| 77 |
+
json.dump({"scripts": self._scripts}, fh, ensure_ascii=False)
|
| 78 |
+
|
| 79 |
+
@classmethod
|
| 80 |
+
def load(cls, path: str) -> "ScriptVocab":
|
| 81 |
+
with open(path) as fh:
|
| 82 |
+
return cls(json.load(fh)["scripts"])
|
| 83 |
+
|
| 84 |
+
|
| 85 |
+
def reading_tagset(readings: list[list[str]]) -> set[str]:
|
| 86 |
+
"""Union of morphtags across all readings, lemma (position 0) excluded."""
|
| 87 |
+
out: set[str] = set()
|
| 88 |
+
for r in readings:
|
| 89 |
+
for tag in r[1:]:
|
| 90 |
+
out.add(tag)
|
| 91 |
+
return out
|
| 92 |
+
|
| 93 |
+
|
| 94 |
+
def build_example(form: str, readings: list[list[str]], tag_vocab, script_vocab: ScriptVocab) -> dict:
|
| 95 |
+
"""One training example: union tag ids (UNK-filtered, sorted unique) + lemma script ids (unique)."""
|
| 96 |
+
tag_ids = sorted({i for i in (tag_vocab.encode(t) for t in reading_tagset(readings))
|
| 97 |
+
if i != TAG_UNK_ID})
|
| 98 |
+
script_ids = sorted({script_vocab.encode(edit_script(form, r[0])) for r in readings if r})
|
| 99 |
+
return {"form": form, "tag_ids": tag_ids, "script_ids": script_ids}
|
| 100 |
+
|
| 101 |
+
|
| 102 |
+
# ----------------------------- lemma-family hold-out split ----------------------------------------
|
| 103 |
+
def lemma_held(lemma: str, holdout_pct: int) -> bool:
|
| 104 |
+
"""Deterministic split: lemma is held out iff md5(lemma) % 100 < holdout_pct (stable across runs)."""
|
| 105 |
+
h = int(hashlib.md5(lemma.encode("utf-8")).hexdigest(), 16)
|
| 106 |
+
return (h % 100) < holdout_pct
|
| 107 |
+
|
| 108 |
+
|
| 109 |
+
def form_held(readings: list[list[str]], holdout_pct: int) -> bool:
|
| 110 |
+
"""A form is held out if ANY of its reading lemmas is held — prevents a shared surface
|
| 111 |
+
form leaking held-family morphology into train."""
|
| 112 |
+
return any(lemma_held(r[0], holdout_pct) for r in readings if r)
|
| 113 |
+
|
| 114 |
+
|
| 115 |
+
def iter_examples(cache: dict[str, list[list[str]]], tag_vocab, script_vocab: ScriptVocab,
|
| 116 |
+
*, holdout_pct: int = 10, split: str | None = None) -> Iterable[dict]:
|
| 117 |
+
"""Yield build_example() dicts; `split` in {None,'train','held'} filters by the lemma-family split."""
|
| 118 |
+
for form, readings in cache.items():
|
| 119 |
+
if not readings:
|
| 120 |
+
continue
|
| 121 |
+
if split is not None:
|
| 122 |
+
held = form_held(readings, holdout_pct)
|
| 123 |
+
if split == "train" and held:
|
| 124 |
+
continue
|
| 125 |
+
if split == "held" and not held:
|
| 126 |
+
continue
|
| 127 |
+
yield build_example(form, readings, tag_vocab, script_vocab)
|
juurio/analyze/hybrid.py
ADDED
|
@@ -0,0 +1,44 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Build the hybrid morphology channel that feeds the parser: clean-FST tags for forms covered by the
|
| 2 |
+
FST-distilled lexicon, neural AnalyzeMux predictions for out-of-lexicon forms.
|
| 3 |
+
|
| 4 |
+
The central invariant (what lets the parser consume this with no retraining): in-lexicon forms emit the
|
| 5 |
+
EXACT clean-FST tag bag — including the empty bag for forms the lexicon analyses to nothing — and NEVER
|
| 6 |
+
fall through to the neural prediction. Neural is used only where the lexicon has no entry at all. The
|
| 7 |
+
hybrid channel is therefore a strict superset of the clean channel the parser trained on: identical
|
| 8 |
+
where the lexicon speaks, additive only on genuinely out-of-lexicon forms.
|
| 9 |
+
"""
|
| 10 |
+
from __future__ import annotations
|
| 11 |
+
|
| 12 |
+
from juurio.parser.fst_cache import FstCache
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
def build_hybrid_channel(lexicon: FstCache, neural: FstCache, forms) -> tuple[FstCache, dict[str, int]]:
|
| 16 |
+
"""Return (hybrid_cache, counts).
|
| 17 |
+
|
| 18 |
+
For each form in `forms`:
|
| 19 |
+
* form in `lexicon` -> the lexicon's clean tag bag (no fallthrough to neural, even if empty);
|
| 20 |
+
* otherwise -> the neural prediction.
|
| 21 |
+
|
| 22 |
+
`lexicon` is the full FST-distilled lexicon (full hybrid) or a compressed top-K table — in either
|
| 23 |
+
case membership (`form in lexicon`) decides clean-vs-neural. `counts` tallies lexicon / neural / empty.
|
| 24 |
+
"""
|
| 25 |
+
data: dict[str, list[list[str]]] = {}
|
| 26 |
+
counts = {"lexicon": 0, "neural": 0, "empty": 0}
|
| 27 |
+
for form in forms:
|
| 28 |
+
if form in lexicon:
|
| 29 |
+
tags = sorted(lexicon.form_to_tagset(form))
|
| 30 |
+
if tags:
|
| 31 |
+
data[form] = [["<exc>", *tags]]
|
| 32 |
+
counts["lexicon"] += 1
|
| 33 |
+
else:
|
| 34 |
+
data[form] = [] # in-lexicon but no reading: stays empty, NO neural fallthrough
|
| 35 |
+
counts["empty"] += 1
|
| 36 |
+
else:
|
| 37 |
+
tags = sorted(neural.form_to_tagset(form))
|
| 38 |
+
if tags:
|
| 39 |
+
data[form] = [["<neural>", *tags]]
|
| 40 |
+
counts["neural"] += 1
|
| 41 |
+
else:
|
| 42 |
+
data[form] = []
|
| 43 |
+
counts["empty"] += 1
|
| 44 |
+
return FstCache(data), counts
|
juurio/analyze/infer.py
ADDED
|
@@ -0,0 +1,74 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""AnalyzeMux inference: load a checkpoint, predict per-form union tag-sets, build a neural FST cache.
|
| 2 |
+
|
| 3 |
+
The reverse model's union-tag head emits ids in the SAME FstTagVocab space the parser channel consumes,
|
| 4 |
+
so we can decode predictions to tag strings and pack them into an `FstCache` the parser's collator reads
|
| 5 |
+
verbatim. Reserved PAD/UNK tag ids are never emitted.
|
| 6 |
+
"""
|
| 7 |
+
from __future__ import annotations
|
| 8 |
+
|
| 9 |
+
import torch
|
| 10 |
+
|
| 11 |
+
from juurio.analyze.metrics import decode_multilabel
|
| 12 |
+
from juurio.analyze.model import AnalyzeMux, CharVocab
|
| 13 |
+
from juurio.parser.fst_cache import PAD_ID as TAG_PAD_ID
|
| 14 |
+
from juurio.parser.fst_cache import UNK_ID as TAG_UNK_ID
|
| 15 |
+
from juurio.parser.fst_cache import FstCache
|
| 16 |
+
|
| 17 |
+
TAG_RESERVED = (TAG_PAD_ID, TAG_UNK_ID)
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
@torch.no_grad()
|
| 21 |
+
def predict_tagsets(model, char_vocab: CharVocab, forms, *, tag_vocab,
|
| 22 |
+
threshold: float = 0.5, top_n: int = 16, device: str = "cpu",
|
| 23 |
+
batch_size: int = 1024) -> list[set[str]]:
|
| 24 |
+
"""Predict each form's union tag-SET (decoded strings), reserved ids excluded, top_n-capped."""
|
| 25 |
+
model.eval()
|
| 26 |
+
out: list[set[str]] = []
|
| 27 |
+
for s in range(0, len(forms), batch_size):
|
| 28 |
+
chunk = forms[s:s + batch_size]
|
| 29 |
+
ids, mask = char_vocab.encode_batch(chunk)
|
| 30 |
+
logits = model(ids.to(device), mask.to(device))["tag_logits"].cpu()
|
| 31 |
+
for row in logits:
|
| 32 |
+
ids_pred = decode_multilabel(row.tolist(), threshold=threshold, top_n=top_n,
|
| 33 |
+
forbidden_ids=TAG_RESERVED)
|
| 34 |
+
out.append({tag_vocab.decode(i) for i in ids_pred})
|
| 35 |
+
return out
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
def build_neural_cache(model, char_vocab: CharVocab, forms, *, tag_vocab,
|
| 39 |
+
threshold: float = 0.5, top_n: int = 16, device: str = "cpu") -> FstCache:
|
| 40 |
+
"""Build an FstCache mapping each form -> a single synthetic reading [<neural>, *predicted_tags].
|
| 41 |
+
The lemma slot is a placeholder (the channel excludes lemma); only the tag-set is meaningful."""
|
| 42 |
+
tagsets = predict_tagsets(model, char_vocab, list(forms), tag_vocab=tag_vocab,
|
| 43 |
+
threshold=threshold, top_n=top_n, device=device)
|
| 44 |
+
data: dict[str, list[list[str]]] = {}
|
| 45 |
+
for form, tags in zip(forms, tagsets):
|
| 46 |
+
data[form] = [["<neural>", *sorted(tags)]] if tags else []
|
| 47 |
+
return FstCache(data)
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
def load_analyze_mux(ckpt_path: str, *, char_vocab_path: str, device: str = "cpu", tag_vocab=None):
|
| 51 |
+
"""Returns (model, char_vocab, calibrated_threshold).
|
| 52 |
+
|
| 53 |
+
If `tag_vocab` is given, verify the analyzer's tag head is compatible with it: the output
|
| 54 |
+
dimension must match the vocab size, and — when the checkpoint records `fst_tag_vocab_hash`
|
| 55 |
+
(stamped at training time) — the vocab content hash must match. This guards against decoding
|
| 56 |
+
the tag logits with a different / reordered FstTagVocab, which would silently corrupt the channel.
|
| 57 |
+
"""
|
| 58 |
+
blob = torch.load(ckpt_path, map_location=device, weights_only=False)
|
| 59 |
+
cfg = blob["config"]
|
| 60 |
+
cv = CharVocab.load(char_vocab_path)
|
| 61 |
+
if tag_vocab is not None:
|
| 62 |
+
if cfg["n_tags"] != tag_vocab.size:
|
| 63 |
+
raise ValueError(f"AnalyzeMux tag head n_tags={cfg['n_tags']} != tag_vocab.size="
|
| 64 |
+
f"{tag_vocab.size}; analyzer and channel use different tag vocabs.")
|
| 65 |
+
want = cfg.get("fst_tag_vocab_hash")
|
| 66 |
+
if want is not None and want != tag_vocab.hash():
|
| 67 |
+
raise ValueError(f"AnalyzeMux was trained against fst_tag_vocab hash {want}, but the "
|
| 68 |
+
f"supplied tag_vocab hashes to {tag_vocab.hash()}.")
|
| 69 |
+
model = AnalyzeMux(char_vocab=cv.size, n_tags=cfg["n_tags"], n_scripts=cfg["n_scripts"],
|
| 70 |
+
d_model=cfg["d_model"], layers=cfg["layers"], heads=cfg["heads"],
|
| 71 |
+
ff=cfg["ff"]).to(device)
|
| 72 |
+
model.load_state_dict(blob["state_dict"])
|
| 73 |
+
model.eval()
|
| 74 |
+
return model, cv, cfg.get("calibrated_threshold", 0.5)
|
juurio/analyze/metrics.py
ADDED
|
@@ -0,0 +1,66 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""AnalyzeMux eval metric primitives (pure, no torch).
|
| 2 |
+
|
| 3 |
+
Used to score the reverse model's predicted tag/script id-sets against the FST cache. Slicing
|
| 4 |
+
(held-family / rare / 3+-reading / nonce) is orchestration elsewhere; these are the kernels.
|
| 5 |
+
"""
|
| 6 |
+
from __future__ import annotations
|
| 7 |
+
|
| 8 |
+
import math
|
| 9 |
+
|
| 10 |
+
|
| 11 |
+
def micro_prf(pred_sets, gold_sets) -> tuple[float, float, float]:
|
| 12 |
+
"""Micro precision/recall/F1 over predicted-vs-gold id SETS (per-example, summed)."""
|
| 13 |
+
tp = fp = fn = 0
|
| 14 |
+
for pred, gold in zip(pred_sets, gold_sets):
|
| 15 |
+
pred, gold = set(pred), set(gold)
|
| 16 |
+
tp += len(pred & gold)
|
| 17 |
+
fp += len(pred - gold)
|
| 18 |
+
fn += len(gold - pred)
|
| 19 |
+
p = tp / (tp + fp) if (tp + fp) else 1.0
|
| 20 |
+
r = tp / (tp + fn) if (tp + fn) else 1.0
|
| 21 |
+
f = 2 * p * r / (p + r) if (p + r) else 0.0
|
| 22 |
+
return p, r, f
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def script_recall_at_k(ranked_lists, gold_sets, *, k: int, exclude_ids=()) -> float:
|
| 26 |
+
"""Micro recall@K: gold script ids appearing in each example's top-K / total gold ids.
|
| 27 |
+
`exclude_ids` (e.g. SCRIPT_UNK_ID) are dropped from BOTH gold and the ranked list before the top-K
|
| 28 |
+
cut — predicting UNK is not real lemma recovery."""
|
| 29 |
+
exclude = set(exclude_ids)
|
| 30 |
+
found = total = 0
|
| 31 |
+
for ranked, gold in zip(ranked_lists, gold_sets):
|
| 32 |
+
gold = set(gold) - exclude
|
| 33 |
+
ranked = [i for i in ranked if i not in exclude]
|
| 34 |
+
topk = set(ranked[:k])
|
| 35 |
+
total += len(gold)
|
| 36 |
+
found += len(gold & topk)
|
| 37 |
+
return found / total if total else 1.0
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
def fbeta(p: float, r: float, beta: float) -> float:
|
| 41 |
+
"""F_beta: beta>1 weights recall higher than precision (the channel tolerates moderate
|
| 42 |
+
over-recall better than under-recall)."""
|
| 43 |
+
b2 = beta * beta
|
| 44 |
+
denom = b2 * p + r
|
| 45 |
+
return (1 + b2) * p * r / denom if denom else 0.0
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
def rank_ids(logits) -> list[int]:
|
| 49 |
+
"""Indices sorted by descending logit (stable on ties by original index)."""
|
| 50 |
+
return sorted(range(len(logits)), key=lambda i: (-logits[i], i))
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
def _sigmoid(x: float) -> float:
|
| 54 |
+
return 1.0 / (1.0 + math.exp(-x))
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
def decode_multilabel(logits, *, threshold: float = 0.5, top_n: int = 16,
|
| 58 |
+
forbidden_ids=()) -> list[int]:
|
| 59 |
+
"""Ids whose sigmoid(logit) > threshold, capped to the top_n highest-scoring (mirrors the parser
|
| 60 |
+
channel's max_tags cap — mirror what the parser actually receives). `forbidden_ids` (e.g. the
|
| 61 |
+
reserved PAD/UNK tag ids) are never decoded, regardless of logit."""
|
| 62 |
+
forbidden = set(forbidden_ids)
|
| 63 |
+
above = [i for i in range(len(logits))
|
| 64 |
+
if i not in forbidden and _sigmoid(logits[i]) > threshold]
|
| 65 |
+
above.sort(key=lambda i: (-logits[i], i))
|
| 66 |
+
return above[:top_n]
|
juurio/analyze/model.py
ADDED
|
@@ -0,0 +1,95 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""AnalyzeMux model — char encoder + two multi-label heads.
|
| 2 |
+
|
| 3 |
+
surface form (chars) -> shared Transformer encoder -> masked mean-pool -> two heads:
|
| 4 |
+
- tag head: multi-label sigmoid over the FST morphtags (the parser channel union tag-bag);
|
| 5 |
+
- script head: multi-label sigmoid over the lemma edit-scripts (+UNK); top-K -> candidate lemmas.
|
| 6 |
+
"""
|
| 7 |
+
from __future__ import annotations
|
| 8 |
+
|
| 9 |
+
import json
|
| 10 |
+
from pathlib import Path
|
| 11 |
+
|
| 12 |
+
import torch
|
| 13 |
+
import torch.nn as nn
|
| 14 |
+
|
| 15 |
+
CHAR_PAD_ID = 0
|
| 16 |
+
CHAR_UNK_ID = 1
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
class CharVocab:
|
| 20 |
+
"""Char -> id. id 0 = PAD, id 1 = UNK, real chars from 2."""
|
| 21 |
+
|
| 22 |
+
def __init__(self, chars: list[str]):
|
| 23 |
+
self._chars = list(chars)
|
| 24 |
+
self._to_id = {c: i + 2 for i, c in enumerate(chars)}
|
| 25 |
+
self._from_id = {i + 2: c for i, c in enumerate(chars)}
|
| 26 |
+
|
| 27 |
+
@property
|
| 28 |
+
def size(self) -> int:
|
| 29 |
+
return len(self._chars) + 2
|
| 30 |
+
|
| 31 |
+
@property
|
| 32 |
+
def unk_id(self) -> int:
|
| 33 |
+
return CHAR_UNK_ID
|
| 34 |
+
|
| 35 |
+
@classmethod
|
| 36 |
+
def fit(cls, forms) -> "CharVocab":
|
| 37 |
+
seen: dict[str, None] = {}
|
| 38 |
+
for f in forms:
|
| 39 |
+
for ch in f:
|
| 40 |
+
seen.setdefault(ch, None)
|
| 41 |
+
return cls(sorted(seen))
|
| 42 |
+
|
| 43 |
+
def encode(self, form: str) -> list[int]:
|
| 44 |
+
return [self._to_id.get(ch, CHAR_UNK_ID) for ch in form]
|
| 45 |
+
|
| 46 |
+
def decode(self, idx: int) -> str:
|
| 47 |
+
return self._from_id.get(idx, "<UNK>")
|
| 48 |
+
|
| 49 |
+
def encode_batch(self, forms) -> tuple[torch.Tensor, torch.Tensor]:
|
| 50 |
+
seqs = [self.encode(f) for f in forms]
|
| 51 |
+
width = max((len(s) for s in seqs), default=1) or 1
|
| 52 |
+
batch = torch.zeros(len(seqs), width, dtype=torch.long)
|
| 53 |
+
mask = torch.zeros(len(seqs), width, dtype=torch.bool)
|
| 54 |
+
for i, s in enumerate(seqs):
|
| 55 |
+
if s:
|
| 56 |
+
batch[i, :len(s)] = torch.tensor(s, dtype=torch.long)
|
| 57 |
+
mask[i, :len(s)] = True
|
| 58 |
+
return batch, mask
|
| 59 |
+
|
| 60 |
+
def save(self, path: str) -> None:
|
| 61 |
+
Path(path).parent.mkdir(parents=True, exist_ok=True)
|
| 62 |
+
with open(path, "w") as fh:
|
| 63 |
+
json.dump({"chars": self._chars}, fh, ensure_ascii=False)
|
| 64 |
+
|
| 65 |
+
@classmethod
|
| 66 |
+
def load(cls, path: str) -> "CharVocab":
|
| 67 |
+
with open(path) as fh:
|
| 68 |
+
return cls(json.load(fh)["chars"])
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
class AnalyzeMux(nn.Module):
|
| 72 |
+
def __init__(self, char_vocab: int, n_tags: int, n_scripts: int, *,
|
| 73 |
+
d_model: int = 128, layers: int = 2, heads: int = 4, ff: int = 512,
|
| 74 |
+
maxlen: int = 64, dropout: float = 0.1):
|
| 75 |
+
super().__init__()
|
| 76 |
+
self.emb = nn.Embedding(char_vocab, d_model, padding_idx=CHAR_PAD_ID)
|
| 77 |
+
self.pos = nn.Embedding(maxlen, d_model)
|
| 78 |
+
self.enc = nn.TransformerEncoder(
|
| 79 |
+
nn.TransformerEncoderLayer(d_model, heads, ff, dropout, batch_first=True), layers)
|
| 80 |
+
self.tag_head = nn.Linear(d_model, n_tags)
|
| 81 |
+
self.script_head = nn.Linear(d_model, n_scripts)
|
| 82 |
+
self.maxlen = maxlen
|
| 83 |
+
|
| 84 |
+
def forward(self, ids: torch.Tensor, mask: torch.Tensor) -> dict[str, torch.Tensor]:
|
| 85 |
+
"""ids/mask: (B, T). Returns tag_logits (B,n_tags), script_logits (B,n_scripts)."""
|
| 86 |
+
B, T = ids.shape
|
| 87 |
+
pos = torch.arange(T, device=ids.device).clamp_max(self.maxlen - 1)
|
| 88 |
+
x = self.emb(ids) + self.pos(pos)[None, :, :]
|
| 89 |
+
# TransformerEncoder: src_key_padding_mask is True where PAD (to be ignored).
|
| 90 |
+
h = self.enc(x, src_key_padding_mask=~mask) # (B, T, d)
|
| 91 |
+
# masked mean-pool over real positions only (PAD must not leak in)
|
| 92 |
+
m = mask.unsqueeze(-1).float()
|
| 93 |
+
pooled = (h * m).sum(dim=1) / m.sum(dim=1).clamp_min(1.0)
|
| 94 |
+
return {"tag_logits": self.tag_head(pooled),
|
| 95 |
+
"script_logits": self.script_head(pooled)}
|
juurio/bootstrap.py
ADDED
|
@@ -0,0 +1,353 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Install-time bootstrap: acquire serving artifacts (model checkpoints + data) and diagnose the env.
|
| 2 |
+
|
| 3 |
+
Serving artifacts live in a HuggingFace repo (`samuli/juurio`) under CLEAN, stable filenames
|
| 4 |
+
(`checkpoints/parser.pt`, `data/wordlist.json`, …) decoupled from the messy dev filenames. Each MANIFEST
|
| 5 |
+
entry maps a stable LOGICAL key -> `{hf, source, sha256, size, …}`:
|
| 6 |
+
* `hf` — path in the HF repo (clean, version-free); downstream installs fetch this.
|
| 7 |
+
* `source` — the committed dev-tree file (versioned, e.g. `parser_v30b.pt`); used in source checkouts.
|
| 8 |
+
|
| 9 |
+
`bootstrap.artifact_path(key)` is the single resolver every serving load site goes through. Precedence:
|
| 10 |
+
1. dev source (`ROOT/<source>`) when present — source-checkout/dev, authoritative, no runtime sha check;
|
| 11 |
+
2. `JUURIO_CKPT_BASEURL` mirror (`<base>/<hf>`) — self-host override, sha256-verified;
|
| 12 |
+
3. HuggingFace `samuli/juurio` @ pinned `_HF_REVISION` — sha256-verified, returned from the HF cache.
|
| 13 |
+
|
| 14 |
+
A new model release changes only the MANIFEST (`source`+`sha256`+`size`) and the pinned `_HF_REVISION`;
|
| 15 |
+
serving code and HF filenames never churn. `juurio setup` pre-fetches+verifies everything; `juurio
|
| 16 |
+
doctor` is read-only diagnostics (never downloads).
|
| 17 |
+
"""
|
| 18 |
+
from __future__ import annotations
|
| 19 |
+
|
| 20 |
+
import hashlib
|
| 21 |
+
import importlib.metadata as _md
|
| 22 |
+
import os
|
| 23 |
+
import urllib.request
|
| 24 |
+
|
| 25 |
+
ROOT = os.path.normpath(os.path.join(os.path.dirname(__file__), ".."))
|
| 26 |
+
|
| 27 |
+
# HF repo holding the serving artifacts, pinned to the immutable upload commit so downstream
|
| 28 |
+
# installs fetch exactly these bytes.
|
| 29 |
+
HF_REPO = "samuli/juurio"
|
| 30 |
+
_HF_REVISION = "948a8aad56454eb3c24bb4c05c28a2dcbc90ba87"
|
| 31 |
+
|
| 32 |
+
# Serving artifacts, keyed by stable logical name. `hf` = clean path in HF_REPO; `source` = committed
|
| 33 |
+
# dev-tree file (resolved against ROOT). These are the files the served product loads.
|
| 34 |
+
MANIFEST = {
|
| 35 |
+
# --- served analysis: Giella/Stanza-free end-to-end neural parser ---
|
| 36 |
+
"parser": {
|
| 37 |
+
"hf": "checkpoints/parser.pt",
|
| 38 |
+
"source": "checkpoints/parser_v30b.pt",
|
| 39 |
+
"sha256": "bd29307b7af38086b6781cd73d9b70b5c73bb7b71d6cd2d37c4731de1a38c6ac",
|
| 40 |
+
"size": 70519101,
|
| 41 |
+
"what": "TinyFiUD dependency parser — UPOS/Feats/lemma/tree (served via neural_adapter.NeuralParser)",
|
| 42 |
+
},
|
| 43 |
+
"analyze": {
|
| 44 |
+
"hf": "checkpoints/analyze.pt",
|
| 45 |
+
"source": "checkpoints/analyze_mux_m2.pt",
|
| 46 |
+
"sha256": "b2d1bae63a43bb4cd9eb52dc7803448dcf3bbc7f7dd08b17e292d3c44548d4df",
|
| 47 |
+
"size": 50730757,
|
| 48 |
+
"what": "AnalyzeMux — neural morphology channel feeding the parser's morphology-tag input",
|
| 49 |
+
},
|
| 50 |
+
"realizer": {
|
| 51 |
+
"hf": "checkpoints/realizer.pt",
|
| 52 |
+
"source": "checkpoints/realizer.pt",
|
| 53 |
+
"sha256": "0311fbea4bb6a6ac8f385b3edca5c1eef4788ad4cc63a418e663a059f65631cd",
|
| 54 |
+
"size": 3951901,
|
| 55 |
+
"what": "nominal realizer (pointer-generator copy head) — OOV noun/adj/propn case/number/Px/degree",
|
| 56 |
+
},
|
| 57 |
+
"verbform": {
|
| 58 |
+
"hf": "checkpoints/verbform.pt",
|
| 59 |
+
"source": "checkpoints/verbform.pt",
|
| 60 |
+
"sha256": "c8ccffaed0a6468ea159147292f8d58d347cd6428c7ff6bf7a713e6a67046d59",
|
| 61 |
+
"size": 3950237,
|
| 62 |
+
"what": "verbform realizer (copy head) — OOV verb participles (VA/NUT/TAVA/TU + agent MA) + infinitives",
|
| 63 |
+
},
|
| 64 |
+
"finiteverb": {
|
| 65 |
+
"hf": "checkpoints/finiteverb.pt",
|
| 66 |
+
"source": "checkpoints/finiteverb.pt",
|
| 67 |
+
"sha256": "08bf5281d321dbae46625c0a83c9a8129eed3d2b9762a8058c584ce3a0a4fc83",
|
| 68 |
+
"size": 3816501,
|
| 69 |
+
"what": "finite-verb realizer — OOV finite verbs (Act/Pss x Ind/Cond/Pot/Imprt + connegatives)",
|
| 70 |
+
},
|
| 71 |
+
# Small vocab/tokenizer JSONs the NeuralParser loads. Committed in-tree for source checkouts; fetched
|
| 72 |
+
# from HF (like the checkpoints) for clean wheel installs that don't carry them.
|
| 73 |
+
"parser_tokenizer": {
|
| 74 |
+
"hf": "checkpoints/parser_tokenizer.json",
|
| 75 |
+
"source": "checkpoints/parser_tokenizer.json",
|
| 76 |
+
"sha256": "9ed387a7e9741863231174617d073af0cef179b841ca7de050e1dbce7ece157f",
|
| 77 |
+
"size": 537891,
|
| 78 |
+
"what": "parser BPE tokenizer (provenance-locked to the parser checkpoint)",
|
| 79 |
+
},
|
| 80 |
+
"parser_label_vocab": {
|
| 81 |
+
"hf": "checkpoints/parser_label_vocab.json",
|
| 82 |
+
"source": "checkpoints/parser_label_vocab.json",
|
| 83 |
+
"sha256": "7b13d5b2c770042034acda7c460b0afe15b1e1a1b19832474854f2d59aacbf36",
|
| 84 |
+
"size": 59183,
|
| 85 |
+
"what": "parser label vocabulary (UPOS/Feats/deprel ids; locked to the parser checkpoint)",
|
| 86 |
+
},
|
| 87 |
+
"analyze_charvocab": {
|
| 88 |
+
"hf": "checkpoints/analyze.charvocab.json",
|
| 89 |
+
"source": "checkpoints/analyze_mux_m2.charvocab.json",
|
| 90 |
+
"sha256": "ae2ef909670d08767b5a746a8733e8b43ab25e0d79f6dc285168961e4266bf90",
|
| 91 |
+
"size": 1443,
|
| 92 |
+
"what": "AnalyzeMux char vocabulary (locked to the analyze checkpoint)",
|
| 93 |
+
},
|
| 94 |
+
# --- generation lexicon + morphology-tag table (data/) ---
|
| 95 |
+
"wordlist": {
|
| 96 |
+
"hf": "data/wordlist.json",
|
| 97 |
+
"source": "data/shipped_wordlist.json",
|
| 98 |
+
"sha256": "248f89465a5a5c505dfaceb95ab0d8a98eaca32e17f6af253954a3a0bfeb85f0",
|
| 99 |
+
"size": 52555054,
|
| 100 |
+
"what": "generation lexicon (NOUN/ADJ/VERB cells the realizer nets get wrong) — companion to "
|
| 101 |
+
"realizer/verbform/finiteverb (juurio.genlex)",
|
| 102 |
+
},
|
| 103 |
+
"fst_table": {
|
| 104 |
+
"hf": "data/fst_table.jsonl",
|
| 105 |
+
"source": "data/fst_compressed_table_k100k_clean.jsonl",
|
| 106 |
+
"sha256": "04abe3cf35ab45256638ea1d5f63d39b99b0ed974751d5912eefa075c3183733",
|
| 107 |
+
"size": 7278625,
|
| 108 |
+
"what": "compressed morphology-tag table (7.3 MB) — clean tags for the top-100k forms, minus e-mail "
|
| 109 |
+
"addresses, @handles, URLs, phone numbers and hashtags; the neural channel fills the rest",
|
| 110 |
+
},
|
| 111 |
+
"fst_tag_vocab": {
|
| 112 |
+
"hf": "data/fst_tag_vocab.json",
|
| 113 |
+
"source": "data/fst_tag_vocab_v1.json",
|
| 114 |
+
"sha256": "6ca9e42d017890e2694fb0f3f53c2ccf7927261116f8f3045b969694c19a0846",
|
| 115 |
+
"size": 136521,
|
| 116 |
+
"what": "morphology-tag vocabulary (locked to the analyze/parser checkpoints)",
|
| 117 |
+
},
|
| 118 |
+
"char_vocab": {
|
| 119 |
+
"hf": "data/char_vocab.json",
|
| 120 |
+
"source": "data/char_vocab_v1.json",
|
| 121 |
+
"sha256": "7dca5353150021dccebf45f75e9f1cb9e8f8263677a6dc3823a2bd8a21abf59e",
|
| 122 |
+
"size": 5049,
|
| 123 |
+
"what": "parser char vocabulary (compound-splitter input; locked to the parser checkpoint)",
|
| 124 |
+
},
|
| 125 |
+
}
|
| 126 |
+
|
| 127 |
+
# Serving runtime deps — the served parser is the torch-only NeuralParser.
|
| 128 |
+
_RUNTIME_DEPS = ["torch", "numpy", "transformers", "huggingface_hub"]
|
| 129 |
+
|
| 130 |
+
_BASEURL_ENV = "JUURIO_CKPT_BASEURL" # self-host mirror of the clean HF tree (<base>/<hf>)
|
| 131 |
+
_OFFLINE_ENV = "JUURIO_OFFLINE" # truthy -> hf local_files_only (cache-only, no network)
|
| 132 |
+
_CACHE_ENV = "JUURIO_CACHE" # where the baseurl-mirror download lands (default ~/.cache/juurio)
|
| 133 |
+
# HF's own get_token() handles the standard sources (HF_TOKEN env + the `hf auth login` cached token).
|
| 134 |
+
# These non-standard langchain-style vars are a LAST-RESORT fallback only, so they never shadow a CLI login.
|
| 135 |
+
_LEGACY_TOKEN_ENVS = ("HUGGINGFACEHUB_API_TOKEN", "HUGGINGFACE_API_TOKEN")
|
| 136 |
+
# In-process memo of absolute paths whose bytes were sha256-verified this run — avoid re-hashing 70 MB
|
| 137 |
+
# per load(). Keyed by resolved PATH (not logical key) so a re-resolve to a different file re-verifies.
|
| 138 |
+
_VERIFIED: set[str] = set()
|
| 139 |
+
|
| 140 |
+
|
| 141 |
+
class ArtifactIntegrityError(RuntimeError):
|
| 142 |
+
"""A fetched artifact failed sha256 verification — corrupt/tampered bytes. NEVER swallow this:
|
| 143 |
+
callers that otherwise degrade gracefully on a missing artifact must still hard-fail on a mismatch."""
|
| 144 |
+
|
| 145 |
+
|
| 146 |
+
def _source_path(key: str) -> str:
|
| 147 |
+
"""Absolute path to the committed dev source for `key` (resolved against the repo ROOT)."""
|
| 148 |
+
return os.path.join(ROOT, MANIFEST[key]["source"])
|
| 149 |
+
|
| 150 |
+
|
| 151 |
+
def _hf_token() -> str | None:
|
| 152 |
+
"""Resolve an HF token. Uses huggingface_hub's standard resolution first — HF_TOKEN env, else the
|
| 153 |
+
`hf auth login` cached session — then the legacy langchain-style env vars as a last resort (so they
|
| 154 |
+
can't shadow an interactive CLI login). Lazy hf import keeps `import juurio.bootstrap` hf-free."""
|
| 155 |
+
from huggingface_hub import get_token
|
| 156 |
+
tok = get_token()
|
| 157 |
+
if tok:
|
| 158 |
+
return tok
|
| 159 |
+
return next((os.environ[k] for k in _LEGACY_TOKEN_ENVS if os.environ.get(k)), None)
|
| 160 |
+
|
| 161 |
+
|
| 162 |
+
def _offline() -> bool:
|
| 163 |
+
return os.environ.get(_OFFLINE_ENV, "").strip().lower() in ("1", "true", "yes", "on")
|
| 164 |
+
|
| 165 |
+
|
| 166 |
+
def _cache_dir() -> str:
|
| 167 |
+
return os.environ.get(_CACHE_ENV) or os.path.join(os.path.expanduser("~"), ".cache", "juurio")
|
| 168 |
+
|
| 169 |
+
|
| 170 |
+
def _remote_url(key: str, base_url: str) -> str:
|
| 171 |
+
"""The self-host mirror URL for `key` — mirrors the clean HF tree (`<base>/<hf>`)."""
|
| 172 |
+
return base_url.rstrip("/") + "/" + MANIFEST[key]["hf"]
|
| 173 |
+
|
| 174 |
+
|
| 175 |
+
def _sha256(path: str) -> str:
|
| 176 |
+
h = hashlib.sha256()
|
| 177 |
+
with open(path, "rb") as f:
|
| 178 |
+
for chunk in iter(lambda: f.read(1 << 20), b""):
|
| 179 |
+
h.update(chunk)
|
| 180 |
+
return h.hexdigest()
|
| 181 |
+
|
| 182 |
+
|
| 183 |
+
def _verify_source(key: str) -> str:
|
| 184 |
+
"""'ok' | 'missing' | 'size-mismatch' | 'sha-mismatch' for the committed dev source (ROOT/<source>).
|
| 185 |
+
Guards manifest drift: a regenerated artifact whose manifest sha/size wasn't bumped fails here."""
|
| 186 |
+
p = _source_path(key)
|
| 187 |
+
meta = MANIFEST[key]
|
| 188 |
+
if not os.path.exists(p):
|
| 189 |
+
return "missing"
|
| 190 |
+
if os.path.getsize(p) != meta["size"]:
|
| 191 |
+
return "size-mismatch"
|
| 192 |
+
return "ok" if _sha256(p) == meta["sha256"] else "sha-mismatch"
|
| 193 |
+
|
| 194 |
+
|
| 195 |
+
def _fetch_hf(key: str) -> str:
|
| 196 |
+
"""Download MANIFEST[key]['hf'] from HF_REPO@_HF_REVISION into the HF cache; return its path.
|
| 197 |
+
huggingface_hub is imported HERE (lazily) so `import juurio` never requires it for dev/source use."""
|
| 198 |
+
from huggingface_hub import hf_hub_download
|
| 199 |
+
return hf_hub_download(repo_id=HF_REPO, filename=MANIFEST[key]["hf"], revision=_HF_REVISION,
|
| 200 |
+
token=_hf_token(), local_files_only=_offline())
|
| 201 |
+
|
| 202 |
+
|
| 203 |
+
def _baseurl_dest(key: str) -> str:
|
| 204 |
+
"""Local cache path a baseurl-mirror download of `key` lands at."""
|
| 205 |
+
return os.path.join(_cache_dir(), MANIFEST[key]["hf"])
|
| 206 |
+
|
| 207 |
+
|
| 208 |
+
def _download_baseurl(key: str) -> str:
|
| 209 |
+
"""Mirror-download `key` from `JUURIO_CKPT_BASEURL/<hf>` into the juurio cache dir, sha-verified.
|
| 210 |
+
Reuses an already-verified cache file instead of re-downloading on every load."""
|
| 211 |
+
base = os.environ[_BASEURL_ENV]
|
| 212 |
+
meta = MANIFEST[key]
|
| 213 |
+
dest = _baseurl_dest(key)
|
| 214 |
+
if os.path.exists(dest) and _sha256(dest) == meta["sha256"]:
|
| 215 |
+
return dest
|
| 216 |
+
url = _remote_url(key, base)
|
| 217 |
+
os.makedirs(os.path.dirname(dest), exist_ok=True)
|
| 218 |
+
tmp = dest + ".part"
|
| 219 |
+
urllib.request.urlretrieve(url, tmp)
|
| 220 |
+
got = _sha256(tmp)
|
| 221 |
+
if got != meta["sha256"]:
|
| 222 |
+
os.remove(tmp)
|
| 223 |
+
raise ArtifactIntegrityError(
|
| 224 |
+
f"artifact {key!r} sha256 mismatch from {url} (got {got[:12]}…, want {meta['sha256'][:12]}…)")
|
| 225 |
+
os.replace(tmp, dest)
|
| 226 |
+
return dest
|
| 227 |
+
|
| 228 |
+
|
| 229 |
+
def artifact_path(key: str) -> str:
|
| 230 |
+
"""Resolve serving artifact `key` to an absolute local path (see module docstring for precedence).
|
| 231 |
+
Downloads (HF / baseurl) are sha256-verified once per process; the dev-source path is trusted."""
|
| 232 |
+
if key not in MANIFEST:
|
| 233 |
+
raise KeyError(f"unknown artifact {key!r}; known: {sorted(MANIFEST)}")
|
| 234 |
+
meta = MANIFEST[key]
|
| 235 |
+
src = _source_path(key)
|
| 236 |
+
if os.path.exists(src):
|
| 237 |
+
return src # dev/source tree is authoritative (trusted, no runtime re-hash)
|
| 238 |
+
if os.environ.get(_BASEURL_ENV):
|
| 239 |
+
dest = _baseurl_dest(key)
|
| 240 |
+
if dest in _VERIFIED: # already verified this run — skip re-hashing the cache file
|
| 241 |
+
return dest
|
| 242 |
+
path = _download_baseurl(key) # verifies sha internally (reuses verified cache)
|
| 243 |
+
_VERIFIED.add(path)
|
| 244 |
+
return path
|
| 245 |
+
try:
|
| 246 |
+
path = _fetch_hf(key)
|
| 247 |
+
except Exception as e:
|
| 248 |
+
offline = " (JUURIO_OFFLINE set — not in local hf cache)" if _offline() else ""
|
| 249 |
+
raise RuntimeError(
|
| 250 |
+
f"could not fetch artifact {key!r} ({meta['hf']}) from {HF_REPO}@{_HF_REVISION}{offline}: {e}. "
|
| 251 |
+
f"Run `juurio setup`, authenticate with `hf auth login` (or set HF_TOKEN), "
|
| 252 |
+
f"or set {_BASEURL_ENV} to a mirror."
|
| 253 |
+
) from e
|
| 254 |
+
if path not in _VERIFIED:
|
| 255 |
+
got = _sha256(path)
|
| 256 |
+
if got != meta["sha256"]:
|
| 257 |
+
raise ArtifactIntegrityError(
|
| 258 |
+
f"artifact {key!r} sha256 mismatch (got {got[:12]}…, want {meta['sha256'][:12]}…) "
|
| 259 |
+
f"from {HF_REPO}@{_HF_REVISION}:{meta['hf']} — refusing to load.")
|
| 260 |
+
_VERIFIED.add(path)
|
| 261 |
+
return path
|
| 262 |
+
|
| 263 |
+
|
| 264 |
+
def _located(key: str) -> tuple[str, str]:
|
| 265 |
+
"""(location, status) for `key` WITHOUT downloading — for read-only diagnostics. location is one of
|
| 266 |
+
'dev source' / 'mirror cache' / 'HF cache' / 'HF'; status is 'ok'|'missing'|'size-/sha-mismatch'."""
|
| 267 |
+
s = _verify_source(key)
|
| 268 |
+
if s != "missing":
|
| 269 |
+
return "dev source", s
|
| 270 |
+
if os.environ.get(_BASEURL_ENV):
|
| 271 |
+
# Baseurl active -> the resolver uses ONLY the mirror; do not fall through to the HF cache
|
| 272 |
+
# (that would let doctor report READY while artifact_path tries the mirror and fails).
|
| 273 |
+
dest = _baseurl_dest(key)
|
| 274 |
+
if os.path.exists(dest):
|
| 275 |
+
return "mirror cache", ("ok" if _sha256(dest) == MANIFEST[key]["sha256"] else "sha-mismatch")
|
| 276 |
+
return "mirror", "missing"
|
| 277 |
+
if _hf_cached(key):
|
| 278 |
+
return "HF cache", "ok"
|
| 279 |
+
return "HF", "missing"
|
| 280 |
+
|
| 281 |
+
|
| 282 |
+
def _hf_cached(key: str) -> bool:
|
| 283 |
+
"""True if `key` is already in the local HF cache (no network). For read-only diagnostics."""
|
| 284 |
+
try:
|
| 285 |
+
from huggingface_hub import hf_hub_download
|
| 286 |
+
hf_hub_download(repo_id=HF_REPO, filename=MANIFEST[key]["hf"], revision=_HF_REVISION,
|
| 287 |
+
token=_hf_token(), local_files_only=True)
|
| 288 |
+
return True
|
| 289 |
+
except Exception:
|
| 290 |
+
return False
|
| 291 |
+
|
| 292 |
+
|
| 293 |
+
def _dep_versions(deps=_RUNTIME_DEPS) -> dict:
|
| 294 |
+
out = {}
|
| 295 |
+
for d in deps:
|
| 296 |
+
try:
|
| 297 |
+
out[d] = _md.version(d)
|
| 298 |
+
except Exception:
|
| 299 |
+
out[d] = None
|
| 300 |
+
return out
|
| 301 |
+
|
| 302 |
+
|
| 303 |
+
def doctor() -> bool:
|
| 304 |
+
"""Read-only diagnostics (never downloads). Returns True iff everything needed to SERVE is present.
|
| 305 |
+
Each artifact is reported by where it resolves from: `dev source` (+ sha status) or `HF cache`."""
|
| 306 |
+
ok = True
|
| 307 |
+
print("juurio doctor\n" + "=" * 60)
|
| 308 |
+
|
| 309 |
+
print("\nRuntime Python deps (serving; floor pins in pyproject):")
|
| 310 |
+
for d, v in _dep_versions().items():
|
| 311 |
+
mark = "ok " if v else "MISSING"
|
| 312 |
+
ok = ok and bool(v)
|
| 313 |
+
print(f" [{mark}] {d:16s} {v or '-- run: uv sync'}")
|
| 314 |
+
|
| 315 |
+
print(f"\nServing artifacts (dev source, else HF cache {HF_REPO}@{_HF_REVISION}):")
|
| 316 |
+
for key, meta in MANIFEST.items():
|
| 317 |
+
loc, st = _located(key)
|
| 318 |
+
ok = ok and st == "ok"
|
| 319 |
+
flag = "ok " if st == "ok" else st.upper()
|
| 320 |
+
hint = "" if st == "ok" else " -- run `juurio setup`"
|
| 321 |
+
print(f" [{flag:>13s}] {key:18s} {loc:10s} {meta['hf']:34s} {meta['what']}{hint}")
|
| 322 |
+
|
| 323 |
+
print("\n" + ("READY — `juurio demo` / `juurio validate` should pass." if ok
|
| 324 |
+
else "NOT READY — run `juurio setup` (and `uv sync`)."))
|
| 325 |
+
return ok
|
| 326 |
+
|
| 327 |
+
|
| 328 |
+
def setup() -> bool:
|
| 329 |
+
"""Acquire serving artifacts. Dev sources present + verified are used as-is; otherwise each artifact
|
| 330 |
+
is fetched from the baseurl mirror or HuggingFace and sha256-verified."""
|
| 331 |
+
print("juurio setup\n" + "=" * 60)
|
| 332 |
+
|
| 333 |
+
all_ok = True
|
| 334 |
+
for key, meta in MANIFEST.items():
|
| 335 |
+
s = _verify_source(key)
|
| 336 |
+
if s == "ok":
|
| 337 |
+
print(f" [ok] {key}: dev source present + verified ({meta['source']})")
|
| 338 |
+
continue
|
| 339 |
+
if s in ("size-mismatch", "sha-mismatch"):
|
| 340 |
+
all_ok = False
|
| 341 |
+
print(f" [WARN] {key}: dev source {meta['source']} {s} vs manifest — bump MANIFEST or rebuild")
|
| 342 |
+
continue
|
| 343 |
+
try:
|
| 344 |
+
path = artifact_path(key)
|
| 345 |
+
print(f" [ok] {key}: fetched + verified -> {path}")
|
| 346 |
+
except ArtifactIntegrityError:
|
| 347 |
+
raise # corrupt/tampered bytes — hard-fail, never fall back to [MISSING]
|
| 348 |
+
except Exception as e:
|
| 349 |
+
all_ok = False
|
| 350 |
+
print(f" [MISSING] {key}: {e}")
|
| 351 |
+
|
| 352 |
+
print("\n" + ("setup complete." if all_ok else "setup incomplete — see notes above."))
|
| 353 |
+
return all_ok
|
juurio/cli.py
ADDED
|
@@ -0,0 +1,473 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""juurio command-line interface.
|
| 2 |
+
|
| 3 |
+
juurio doctor # read-only env diagnostics
|
| 4 |
+
juurio setup # acquire parser + model checkpoints
|
| 5 |
+
juurio list-ops # documented op catalog
|
| 6 |
+
|
| 7 |
+
# one command per library function (same names):
|
| 8 |
+
juurio transform <op> "<text>" [--lemma L] [--trace] [--json] # -> transform()
|
| 9 |
+
juurio inflect "<word>" [spec…] [--case C] [--number N] [--degree D] [--px P] [--clitic K] [--pos P] [--like W] [--proper] [--json] # -> inflect()
|
| 10 |
+
juurio analyze "<text>" [--json] # -> analyze()
|
| 11 |
+
juurio lemmatize "<text>" [--json] # -> lemmatize()
|
| 12 |
+
juurio realize-np "<np>" [--case Ine] [--number Plur] [--px 1Sg] [--clitic kin] [--json] # -> realize_np()
|
| 13 |
+
juurio realize-verbform "<verb>" [--form va|nut|tava|tu|inf2|inf2_pass|inf3|ma] [--case Ine] [--number Plur] [--px 3] # -> realize_verbform()
|
| 14 |
+
|
| 15 |
+
juurio batch <op> [--lemma L] [--json] < lines # one op over many lines (stdin)
|
| 16 |
+
juurio batch --json < tsv # per-line "op<TAB>text[<TAB>lemma]"
|
| 17 |
+
juurio demo # curated tour (incl. OOV) + smoke test
|
| 18 |
+
juurio repl # interactive shell (inflect/analyze/transform/...)
|
| 19 |
+
juurio bench # cold/warm latency numbers
|
| 20 |
+
|
| 21 |
+
Exit status: 0 on success; 2 if any transform abstained (a normal, documented outcome — useful for
|
| 22 |
+
scripting); 1 on a setup failure. `--json` emits one object per line (text/ok/reason/
|
| 23 |
+
warnings/trace). Also reachable as `python -m juurio ...`.
|
| 24 |
+
"""
|
| 25 |
+
from __future__ import annotations
|
| 26 |
+
|
| 27 |
+
import argparse
|
| 28 |
+
import json
|
| 29 |
+
import sys
|
| 30 |
+
|
| 31 |
+
from . import OP_CATALOG, SUPPORTED_OPS
|
| 32 |
+
|
| 33 |
+
# A curated tour that doubles as a smoke test. Every entry is expected to succeed (ok=True).
|
| 34 |
+
_DEMO = [
|
| 35 |
+
("past", "Koira haukkuu pihalla.", None), ("passive", "Mies osti talon.", None),
|
| 36 |
+
("neg", "Lapsi nukkuu sängyssä.", None), ("quest", "Liisa tuli.", None),
|
| 37 |
+
("imperative", "Sinä luet kirjan.", "lukea"),
|
| 38 |
+
("past", "Nuori striimaa peliä.", "striimata"), # OOV verb
|
| 39 |
+
("potential", "Nuori striimaa peliä.", "striimata"), # OOV verb, new mood
|
| 40 |
+
("plural", "Striimaaja nukkuu.", None), # OOV noun head
|
| 41 |
+
("poss_1sg", "Striimaaja nukkuu.", None), # OOV + possessive
|
| 42 |
+
("plural+poss_1sg+clitic_kin", "Koira haukkuu pihalla.", None), # combined nominal morphology
|
| 43 |
+
]
|
| 44 |
+
_DEMO_NP = [
|
| 45 |
+
("talo", dict(case="Ine", number="Plur", px="1Sg")),
|
| 46 |
+
("iso talo", dict(case="Ine", number="Plur")), # modifier agreement
|
| 47 |
+
("striimaaja", dict(case="Ade", number="Plur", px="1Sg", clitic="kin")), # OOV
|
| 48 |
+
("bugisovellus", dict(case="Ine", number="Sing", px="1Pl", clitic="kin")), # OOV compound loan
|
| 49 |
+
]
|
| 50 |
+
_DEMO_VF = [
|
| 51 |
+
("lukea", dict(form="va", case="Ine")), # known participle, inflected
|
| 52 |
+
("mennä", dict(form="inf3", case="Ade")), # known 3rd infinitive
|
| 53 |
+
("tehdä", dict(form="inf2_pass")), # known passive 2nd inf (tehtäessä)
|
| 54 |
+
("kuulla", dict(form="tu", case="Par", px="3")), # temporal construction (kuultuansa)
|
| 55 |
+
("striimata", dict(form="va", lemma="striimata")), # OOV participle (neural)
|
| 56 |
+
("striimata", dict(form="inf3", case="Ine", lemma="striimata")), # OOV infinitive (neural)
|
| 57 |
+
]
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
def _print_catalog() -> None:
|
| 61 |
+
for group, ops in OP_CATALOG.items():
|
| 62 |
+
print(f"\n{group}:")
|
| 63 |
+
for op, desc in ops.items():
|
| 64 |
+
print(f" {op:24s} {desc}")
|
| 65 |
+
print(f"\n{len(SUPPORTED_OPS)} ops total. Compose nominal ops with '+' (plural+poss_1sg+clitic_kin). "
|
| 66 |
+
"Out-of-scope / inapplicable input abstains (ok=False, reason in ABSTAIN_REASONS).")
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
def _emit(op: str, text: str, r, as_json: bool, trace: bool = False) -> None:
|
| 70 |
+
if as_json:
|
| 71 |
+
print(json.dumps({"op": op, "input": text, "text": r.text, "ok": r.ok,
|
| 72 |
+
"reason": r.reason, "warnings": r.warnings, "trace": r.trace},
|
| 73 |
+
ensure_ascii=False))
|
| 74 |
+
return
|
| 75 |
+
if r.ok:
|
| 76 |
+
tag = " ⚠ " + "; ".join(r.warnings) if r.warnings else ""
|
| 77 |
+
print(f"{r.text}{tag}")
|
| 78 |
+
else:
|
| 79 |
+
print(f"[abstain: {r.reason}] {text}", file=sys.stderr)
|
| 80 |
+
if trace:
|
| 81 |
+
for k, v in r.trace.items():
|
| 82 |
+
print(f" · {k}: {v}", file=sys.stderr)
|
| 83 |
+
|
| 84 |
+
|
| 85 |
+
def _bench() -> int:
|
| 86 |
+
import time
|
| 87 |
+
t0 = time.time()
|
| 88 |
+
from .pipeline import Transformer
|
| 89 |
+
t = Transformer()
|
| 90 |
+
t1 = time.time()
|
| 91 |
+
t.transform("past", "Koira haukkuu pihalla.") # first parse loads Stanza (cold)
|
| 92 |
+
t2 = time.time()
|
| 93 |
+
for _ in range(20):
|
| 94 |
+
t.transform("past", "Lapsi lukee kirjaa.")
|
| 95 |
+
t3 = time.time()
|
| 96 |
+
for _ in range(20):
|
| 97 |
+
t.realize_np("talo", case="Ine", number="Plur", px="1Sg")
|
| 98 |
+
t4 = time.time()
|
| 99 |
+
print("juurio bench (this machine)\n" + "=" * 40)
|
| 100 |
+
print(f" Transformer() construct : {t1 - t0:5.2f} s (models; Stanza loads lazily)")
|
| 101 |
+
print(f" first transform (cold) : {t2 - t1:5.2f} s (Stanza `fi` pipeline load)")
|
| 102 |
+
print(f" transform (warm, avg) : {(t3 - t2) / 20 * 1000:5.1f} ms")
|
| 103 |
+
print(f" realize_np (warm, avg) : {(t4 - t3) / 20 * 1000:5.1f} ms")
|
| 104 |
+
return 0
|
| 105 |
+
|
| 106 |
+
|
| 107 |
+
def _demo(t=None) -> int:
|
| 108 |
+
if t is None:
|
| 109 |
+
from .pipeline import Transformer
|
| 110 |
+
t = Transformer()
|
| 111 |
+
bad = 0
|
| 112 |
+
print("clause transforms (transform):")
|
| 113 |
+
for op, text, lemma in _DEMO:
|
| 114 |
+
r = t.transform(op, text, lemma=lemma)
|
| 115 |
+
oov = " [OOV→neural]" if any("neural" in w for w in r.warnings) else ""
|
| 116 |
+
print(f" {op:28s} {text!r:34s} -> {r.text!r}{oov}")
|
| 117 |
+
bad += not r.ok
|
| 118 |
+
print("\nphrase morphology (realize_np):")
|
| 119 |
+
for np, kw in _DEMO_NP:
|
| 120 |
+
r = t.realize_np(np, **kw)
|
| 121 |
+
oov = " [OOV→neural]" if any("neural" in w for w in r.warnings) else ""
|
| 122 |
+
print(f" {np!r:18s} {kw} -> {r.text!r}{oov}")
|
| 123 |
+
bad += not r.ok
|
| 124 |
+
print("\nverb morphology (realize_verbform):")
|
| 125 |
+
for verb, kw in _DEMO_VF:
|
| 126 |
+
r = t.realize_verbform(verb, **kw)
|
| 127 |
+
oov = " [OOV→neural]" if any("neural" in w for w in r.warnings) else ""
|
| 128 |
+
print(f" {verb!r:14s} {kw} -> {r.text!r}{oov}")
|
| 129 |
+
bad += not r.ok
|
| 130 |
+
print("\n" + ("demo OK" if not bad else f"demo FAILED: {bad} unexpected abstain(s)"))
|
| 131 |
+
return 0 if not bad else 1
|
| 132 |
+
|
| 133 |
+
|
| 134 |
+
_REPL_HELP = """commands (model is loaded once; every command reuses it):
|
| 135 |
+
transform <op> <text> clause transform e.g. transform passive Mies osti talon.
|
| 136 |
+
inflect <word> [spec] [k=v] inflect a nominal e.g. inflect talo mon gen
|
| 137 |
+
spec is a human case/number/degree phrase ('gen', 'mon gen', 'plural genitive comparative');
|
| 138 |
+
options as key=value: px=1sg clitic=kin proper=true pos=adjective like=talojen
|
| 139 |
+
np <phrase> [k=v] phrase nominal morphology e.g. np iso talo case=Ine number=Plur
|
| 140 |
+
options: case=Ine number=Plur px=1Sg clitic=kin
|
| 141 |
+
verb <verb> [k=v] non-finite verb form e.g. verb kuulla form=tu case=Par px=3
|
| 142 |
+
options: form=va|nut|tava|tu|inf2|inf2_pass|inf3|ma case= number= px= lemma=
|
| 143 |
+
analyze <text> dependency + morphology analysis (the inverse of generation)
|
| 144 |
+
lemmatize <text> lemmas only
|
| 145 |
+
ops list the transform ops
|
| 146 |
+
help this help quit | exit leave
|
| 147 |
+
|
| 148 |
+
examples:
|
| 149 |
+
inflect talo mon gen -> talojen
|
| 150 |
+
inflect Meri gen proper=true -> proper-noun reading (vs 'meri' -> meren)
|
| 151 |
+
inflect makaronilaatikko like=talojen -> makaronilaatikkojen (copy a model word's cell)
|
| 152 |
+
inflect kaunis gen degree=comp -> kauniimman (adjective comparative)
|
| 153 |
+
transform past Koira haukkuu pihalla.
|
| 154 |
+
analyze Koirat haukkuvat pihalla.
|
| 155 |
+
"""
|
| 156 |
+
|
| 157 |
+
|
| 158 |
+
def _parse_opts(tokens: list[str]) -> tuple[list[str], dict]:
|
| 159 |
+
"""Split REPL tokens into (positional words, {key: value}) by the presence of '='."""
|
| 160 |
+
words, opts = [], {}
|
| 161 |
+
for tok in tokens:
|
| 162 |
+
if "=" in tok:
|
| 163 |
+
k, v = tok.split("=", 1)
|
| 164 |
+
opts[k] = v
|
| 165 |
+
else:
|
| 166 |
+
words.append(tok)
|
| 167 |
+
return words, opts
|
| 168 |
+
|
| 169 |
+
|
| 170 |
+
def _inflect_kwargs(spec_words, *, case=None, number=None, degree=None,
|
| 171 |
+
px=None, clitic=None, pos=None, like=None, proper=None) -> dict:
|
| 172 |
+
"""Map a positional case/number/degree phrase + named options onto `inflect()` kwargs.
|
| 173 |
+
|
| 174 |
+
A positional spec ('mon gen') wins over an explicit case= (mirrors the REPL). None-valued options
|
| 175 |
+
are dropped so `inflect()`'s own defaults apply. Shared by the `inflect` CLI command and the REPL."""
|
| 176 |
+
kw = {}
|
| 177 |
+
spec = " ".join(spec_words)
|
| 178 |
+
if spec:
|
| 179 |
+
kw["case"] = spec
|
| 180 |
+
elif case:
|
| 181 |
+
kw["case"] = case
|
| 182 |
+
for k, v in (("number", number), ("degree", degree), ("px", px),
|
| 183 |
+
("clitic", clitic), ("pos", pos), ("like", like)):
|
| 184 |
+
if v is not None:
|
| 185 |
+
kw[k] = v
|
| 186 |
+
if proper is not None:
|
| 187 |
+
kw["proper"] = proper
|
| 188 |
+
return kw
|
| 189 |
+
|
| 190 |
+
|
| 191 |
+
def _truthy(v: str) -> bool:
|
| 192 |
+
s = v.strip().lower()
|
| 193 |
+
if s in ("1", "true", "t", "yes", "y", "on"):
|
| 194 |
+
return True
|
| 195 |
+
if s in ("0", "false", "f", "no", "n", "off"):
|
| 196 |
+
return False
|
| 197 |
+
raise ValueError(f"expected a boolean for proper= (true/false), got {v!r}")
|
| 198 |
+
|
| 199 |
+
|
| 200 |
+
def _print_analysis(sents) -> None:
|
| 201 |
+
for si, sent in enumerate(sents):
|
| 202 |
+
if len(sents) > 1:
|
| 203 |
+
print(f"sentence {si + 1}:")
|
| 204 |
+
print(f" {'id':>2} {'form':<16}{'lemma':<16}{'upos':<7}{'head':>4} {'deprel':<11}feats")
|
| 205 |
+
for tk in sent:
|
| 206 |
+
feats = ",".join(f"{k}={v}" for k, v in tk.feats.items())
|
| 207 |
+
print(f" {tk.id:>2} {tk.form:<16}{tk.lemma:<16}{tk.upos:<7}{tk.head:>4} "
|
| 208 |
+
f"{tk.deprel:<11}{feats}")
|
| 209 |
+
|
| 210 |
+
|
| 211 |
+
def _repl_dispatch(t, line: str) -> bool:
|
| 212 |
+
"""Execute one REPL line against the (already-loaded) Transformer `t`. Returns False to signal the
|
| 213 |
+
loop should stop (quit/exit/q), True to keep going. A bad alias/op is reported, not fatal."""
|
| 214 |
+
cmd, *rest = line.split()
|
| 215 |
+
arg = line[len(cmd):].strip() # raw remainder (for free-text commands)
|
| 216 |
+
cmd = cmd.lower()
|
| 217 |
+
try:
|
| 218 |
+
if cmd in ("quit", "exit", "q"):
|
| 219 |
+
return False
|
| 220 |
+
elif cmd in ("help", "h", "?"):
|
| 221 |
+
print(_REPL_HELP)
|
| 222 |
+
elif cmd == "ops":
|
| 223 |
+
_print_catalog()
|
| 224 |
+
elif cmd in ("transform", "t"):
|
| 225 |
+
op, _, text = arg.partition(" ")
|
| 226 |
+
if not text:
|
| 227 |
+
print("usage: transform <op> <text>", file=sys.stderr)
|
| 228 |
+
return True
|
| 229 |
+
_emit(op, text, t.transform(op, text), False, trace=True)
|
| 230 |
+
elif cmd in ("inflect", "i"):
|
| 231 |
+
words, opts = _parse_opts(rest)
|
| 232 |
+
if not words:
|
| 233 |
+
print("usage: inflect <word> [spec] [px= clitic= proper= pos= like=]",
|
| 234 |
+
file=sys.stderr)
|
| 235 |
+
return True
|
| 236 |
+
kw = _inflect_kwargs(words[1:], case=opts.get("case"), number=opts.get("number"),
|
| 237 |
+
degree=opts.get("degree"), px=opts.get("px"),
|
| 238 |
+
clitic=opts.get("clitic"), pos=opts.get("pos"), like=opts.get("like"),
|
| 239 |
+
proper=_truthy(opts["proper"]) if "proper" in opts else None)
|
| 240 |
+
_emit("inflect", words[0], t.inflect(words[0], **kw), False, trace=True)
|
| 241 |
+
elif cmd == "np":
|
| 242 |
+
words, opts = _parse_opts(rest)
|
| 243 |
+
phrase = " ".join(words)
|
| 244 |
+
r = t.realize_np(phrase, case=opts.get("case", "Nom"),
|
| 245 |
+
number=opts.get("number", "Sing"),
|
| 246 |
+
px=opts.get("px", "None"), clitic=opts.get("clitic"))
|
| 247 |
+
_emit("realize-np", phrase, r, False, trace=True)
|
| 248 |
+
elif cmd == "verb":
|
| 249 |
+
words, opts = _parse_opts(rest)
|
| 250 |
+
verb = " ".join(words)
|
| 251 |
+
r = t.realize_verbform(verb, form=opts.get("form", "va"),
|
| 252 |
+
case=opts.get("case", "Nom"),
|
| 253 |
+
number=opts.get("number", "Sing"),
|
| 254 |
+
px=opts.get("px", "None"), lemma=opts.get("lemma"))
|
| 255 |
+
_emit(f"verbform:{opts.get('form', 'va')}", verb, r, False, trace=True)
|
| 256 |
+
elif cmd in ("analyze", "a"):
|
| 257 |
+
if not arg:
|
| 258 |
+
print("usage: analyze <text>", file=sys.stderr)
|
| 259 |
+
return True
|
| 260 |
+
_print_analysis(t.analyze(arg))
|
| 261 |
+
elif cmd in ("lemmatize", "lem"):
|
| 262 |
+
print(" ".join(t.lemmatize(arg)) if arg else "usage: lemmatize <text>")
|
| 263 |
+
else:
|
| 264 |
+
print(f"unknown command {cmd!r}; type 'help'", file=sys.stderr)
|
| 265 |
+
except ValueError as e: # unknown alias / unknown op / unsupported pos — keep the loop alive
|
| 266 |
+
print(f"error: {e}", file=sys.stderr)
|
| 267 |
+
return True
|
| 268 |
+
|
| 269 |
+
|
| 270 |
+
def _repl() -> int:
|
| 271 |
+
"""Interactive read-eval loop exercising every public feature, with the model loaded once."""
|
| 272 |
+
print("loading model (parser + realizers)…", file=sys.stderr)
|
| 273 |
+
from .pipeline import Transformer
|
| 274 |
+
t = Transformer()
|
| 275 |
+
print("juurio interactive shell — type 'help' for commands, 'quit' to exit.")
|
| 276 |
+
while True:
|
| 277 |
+
try:
|
| 278 |
+
line = input("juurio> ").strip()
|
| 279 |
+
except (EOFError, KeyboardInterrupt):
|
| 280 |
+
print()
|
| 281 |
+
return 0
|
| 282 |
+
if not line:
|
| 283 |
+
continue
|
| 284 |
+
if not _repl_dispatch(t, line):
|
| 285 |
+
return 0
|
| 286 |
+
return 0
|
| 287 |
+
|
| 288 |
+
|
| 289 |
+
def _build_parser() -> argparse.ArgumentParser:
|
| 290 |
+
p = argparse.ArgumentParser(prog="juurio", description=__doc__,
|
| 291 |
+
formatter_class=argparse.RawDescriptionHelpFormatter)
|
| 292 |
+
p.add_argument("--version", action="version", version=f"juurio {_version()}")
|
| 293 |
+
sub = p.add_subparsers(dest="cmd", required=True)
|
| 294 |
+
|
| 295 |
+
sub.add_parser("doctor", help="read-only environment diagnostics")
|
| 296 |
+
sub.add_parser("setup", help="acquire parser + model checkpoints")
|
| 297 |
+
sub.add_parser("list-ops", help="print the documented op catalog")
|
| 298 |
+
sub.add_parser("demo", help="curated tour (incl. OOV) — doubles as a smoke test")
|
| 299 |
+
sub.add_parser("repl", help="interactive shell exercising every feature (model loaded once)")
|
| 300 |
+
sub.add_parser("bench", help="cold/warm latency numbers")
|
| 301 |
+
|
| 302 |
+
# Top-level commands mirror the library API one-for-one: transform/inflect/analyze/lemmatize +
|
| 303 |
+
# realize-np/realize-verbform (hyphenated). doctor/setup/list-ops/demo/repl/bench/batch are
|
| 304 |
+
# infrastructure-only and have no library counterpart.
|
| 305 |
+
pr = sub.add_parser("transform", help="clause transform (-> transform())")
|
| 306 |
+
pr.add_argument("op", metavar="OP",
|
| 307 |
+
help="an op from `list-ops`, or a '+'-composed nominal bundle "
|
| 308 |
+
"e.g. 'plural+poss_1sg+clitic_kin'")
|
| 309 |
+
pr.add_argument("text")
|
| 310 |
+
pr.add_argument("--lemma", help="override the parser lemma for the main verb")
|
| 311 |
+
pr.add_argument("--trace", action="store_true", help="print the Result.trace fields")
|
| 312 |
+
pr.add_argument("--json", action="store_true")
|
| 313 |
+
|
| 314 |
+
pi = sub.add_parser("inflect", help="inflect a single nominal (-> inflect())")
|
| 315 |
+
pi.add_argument("word", help="a bare lemma (or model word), e.g. 'talo'")
|
| 316 |
+
pi.add_argument("spec", nargs="*",
|
| 317 |
+
help="human case/number/degree phrase, e.g. 'mon gen' or 'plural genitive "
|
| 318 |
+
"comparative' (overrides --case/--number/--degree)")
|
| 319 |
+
pi.add_argument("--case", help="UD or human case (e.g. Gen / gen / genitive)")
|
| 320 |
+
pi.add_argument("--number", help="Sing/Plur (or 'yks'/'mon')")
|
| 321 |
+
pi.add_argument("--degree", help="pos/comp/sup (adjectives)")
|
| 322 |
+
pi.add_argument("--px", help="possessive suffix (1Sg/2Sg/3/1Pl/2Pl)")
|
| 323 |
+
pi.add_argument("--clitic", help="enclitic (kin/kaan/ko/han/pa)")
|
| 324 |
+
pi.add_argument("--pos", help="force a part-of-speech reading (e.g. adjective)")
|
| 325 |
+
pi.add_argument("--like", help="copy a model word's inflected cell, e.g. like=talojen")
|
| 326 |
+
pi.add_argument("--proper", action=argparse.BooleanOptionalAction, default=None,
|
| 327 |
+
help="force (or with --no-proper, forbid) a proper-noun reading")
|
| 328 |
+
pi.add_argument("--trace", action="store_true")
|
| 329 |
+
pi.add_argument("--json", action="store_true")
|
| 330 |
+
|
| 331 |
+
pa = sub.add_parser("analyze", help="dependency + morphology analysis (-> analyze())")
|
| 332 |
+
pa.add_argument("text")
|
| 333 |
+
pa.add_argument("--json", action="store_true")
|
| 334 |
+
|
| 335 |
+
pl = sub.add_parser("lemmatize", help="lemmas only (-> lemmatize())")
|
| 336 |
+
pl.add_argument("text")
|
| 337 |
+
pl.add_argument("--json", action="store_true")
|
| 338 |
+
|
| 339 |
+
pn = sub.add_parser("realize-np", help="phrase-level nominal morphology (NOT a clause transform)")
|
| 340 |
+
pn.add_argument("np", help="a bare lemma or short 'adj* noun' phrase, e.g. 'iso talo'")
|
| 341 |
+
pn.add_argument("--case", default="Nom", help="UD case (Nom/Gen/Par/Ine/Ela/Ill/Ade/Abl/All/...)")
|
| 342 |
+
pn.add_argument("--number", default="Sing", choices=["Sing", "Plur"])
|
| 343 |
+
pn.add_argument("--px", default="None", choices=["None", "1Sg", "2Sg", "3", "1Pl", "2Pl"],
|
| 344 |
+
help="possessive suffix")
|
| 345 |
+
pn.add_argument("--clitic", default=None, choices=["kin", "kaan", "ko", "han", "pa"])
|
| 346 |
+
pn.add_argument("--trace", action="store_true")
|
| 347 |
+
pn.add_argument("--json", action="store_true")
|
| 348 |
+
|
| 349 |
+
pv = sub.add_parser("realize-verbform",
|
| 350 |
+
help="non-finite verb morphology: participle/infinitive (NOT a clause transform)")
|
| 351 |
+
pv.add_argument("verb", help="a verb lemma or short clause, e.g. 'lukea' or 'Hän lukee kirjaa.'")
|
| 352 |
+
pv.add_argument("--form", default="va",
|
| 353 |
+
choices=["va", "nut", "tava", "tu", "inf2", "inf2_pass", "inf3", "ma"],
|
| 354 |
+
help="va/nut/tava/tu = present/past active/passive participle; inf2/inf3 = 2nd/3rd "
|
| 355 |
+
"infinitive; inf2_pass = passive 2nd inf (tehtäessä); ma = agent participle (tekemä)")
|
| 356 |
+
pv.add_argument("--case", default="Nom", help="UD case (participles: any; inf3: Ine/Ela/Ill/Ade/Abe)")
|
| 357 |
+
pv.add_argument("--number", default="Sing", choices=["Sing", "Plur"])
|
| 358 |
+
pv.add_argument("--px", default="None", choices=["None", "1Sg", "2Sg", "3", "1Pl", "2Pl"],
|
| 359 |
+
help="possessive suffix (temporal/possessive construction): tu+Par+3 -> kuultuansa")
|
| 360 |
+
pv.add_argument("--lemma", help="override the parser lemma (for a mis-lemmatized loan verb)")
|
| 361 |
+
pv.add_argument("--trace", action="store_true")
|
| 362 |
+
pv.add_argument("--json", action="store_true")
|
| 363 |
+
|
| 364 |
+
pb = sub.add_parser("batch", help="transform many lines from stdin")
|
| 365 |
+
pb.add_argument("op", nargs="?", metavar="OP",
|
| 366 |
+
help="op (or '+'-composed nominal bundle) for every line; omit to read "
|
| 367 |
+
"per-line 'op<TAB>text[<TAB>lemma]'")
|
| 368 |
+
pb.add_argument("--lemma", help="lemma override (single-op mode only)")
|
| 369 |
+
pb.add_argument("--json", action="store_true")
|
| 370 |
+
return p
|
| 371 |
+
|
| 372 |
+
|
| 373 |
+
def main(argv: list[str] | None = None) -> int:
|
| 374 |
+
args = _build_parser().parse_args(argv)
|
| 375 |
+
|
| 376 |
+
# commands that don't need the (heavy) Transformer
|
| 377 |
+
if args.cmd == "doctor":
|
| 378 |
+
from . import bootstrap
|
| 379 |
+
return 0 if bootstrap.doctor() else 1
|
| 380 |
+
if args.cmd == "setup":
|
| 381 |
+
from . import bootstrap
|
| 382 |
+
return 0 if bootstrap.setup() else 1
|
| 383 |
+
if args.cmd == "list-ops":
|
| 384 |
+
_print_catalog()
|
| 385 |
+
return 0
|
| 386 |
+
if args.cmd == "bench":
|
| 387 |
+
return _bench()
|
| 388 |
+
if args.cmd == "demo":
|
| 389 |
+
return _demo()
|
| 390 |
+
if args.cmd == "repl":
|
| 391 |
+
return _repl()
|
| 392 |
+
|
| 393 |
+
from .pipeline import Transformer
|
| 394 |
+
from . import algebra
|
| 395 |
+
t = Transformer()
|
| 396 |
+
any_abstain = False
|
| 397 |
+
|
| 398 |
+
def known(op):
|
| 399 |
+
return op in SUPPORTED_OPS or algebra.parse_nominal_bundle(op) is not None
|
| 400 |
+
|
| 401 |
+
if args.cmd == "transform":
|
| 402 |
+
r = t.transform(args.op, args.text, lemma=args.lemma)
|
| 403 |
+
_emit(args.op, args.text, r, args.json, trace=args.trace)
|
| 404 |
+
return 2 if not r.ok else 0
|
| 405 |
+
|
| 406 |
+
if args.cmd == "inflect":
|
| 407 |
+
kw = _inflect_kwargs(args.spec, case=args.case, number=args.number, degree=args.degree,
|
| 408 |
+
px=args.px, clitic=args.clitic, pos=args.pos, like=args.like,
|
| 409 |
+
proper=args.proper)
|
| 410 |
+
r = t.inflect(args.word, **kw)
|
| 411 |
+
_emit("inflect", args.word, r, args.json, trace=args.trace)
|
| 412 |
+
return 2 if not r.ok else 0
|
| 413 |
+
|
| 414 |
+
if args.cmd == "analyze":
|
| 415 |
+
sents = t.analyze(args.text)
|
| 416 |
+
if args.json:
|
| 417 |
+
print(json.dumps([[{"id": tk.id, "form": tk.form, "lemma": tk.lemma, "upos": tk.upos,
|
| 418 |
+
"head": tk.head, "deprel": tk.deprel, "feats": tk.feats}
|
| 419 |
+
for tk in s] for s in sents], ensure_ascii=False))
|
| 420 |
+
else:
|
| 421 |
+
_print_analysis(sents)
|
| 422 |
+
return 0
|
| 423 |
+
|
| 424 |
+
if args.cmd == "lemmatize":
|
| 425 |
+
lemmas = t.lemmatize(args.text)
|
| 426 |
+
print(json.dumps(lemmas, ensure_ascii=False) if args.json else " ".join(lemmas))
|
| 427 |
+
return 0
|
| 428 |
+
|
| 429 |
+
if args.cmd == "realize-np":
|
| 430 |
+
r = t.realize_np(args.np, case=args.case, number=args.number, px=args.px, clitic=args.clitic)
|
| 431 |
+
_emit("realize-np", args.np, r, args.json, trace=args.trace)
|
| 432 |
+
return 2 if not r.ok else 0
|
| 433 |
+
|
| 434 |
+
if args.cmd == "realize-verbform":
|
| 435 |
+
r = t.realize_verbform(args.verb, form=args.form, case=args.case, number=args.number,
|
| 436 |
+
px=args.px, lemma=args.lemma)
|
| 437 |
+
_emit(f"verbform:{args.form}", args.verb, r, args.json, trace=args.trace)
|
| 438 |
+
return 2 if not r.ok else 0
|
| 439 |
+
|
| 440 |
+
for line in sys.stdin:
|
| 441 |
+
line = line.rstrip("\n")
|
| 442 |
+
if not line.strip():
|
| 443 |
+
continue
|
| 444 |
+
if args.op: # single op for every line
|
| 445 |
+
op, text, lemma = args.op, line, args.lemma
|
| 446 |
+
else: # per-line "op<TAB>text[<TAB>lemma]"
|
| 447 |
+
parts = line.split("\t")
|
| 448 |
+
if len(parts) < 2:
|
| 449 |
+
print(f"[skip: need 'op<TAB>text'] {line}", file=sys.stderr)
|
| 450 |
+
any_abstain = True
|
| 451 |
+
continue
|
| 452 |
+
op, text = parts[0], parts[1]
|
| 453 |
+
lemma = parts[2] if len(parts) > 2 else None
|
| 454 |
+
if not known(op):
|
| 455 |
+
print(f"[skip: unknown op {op!r}] {line}", file=sys.stderr)
|
| 456 |
+
any_abstain = True
|
| 457 |
+
continue
|
| 458 |
+
r = t.transform(op, text, lemma=lemma)
|
| 459 |
+
_emit(op, text, r, args.json)
|
| 460 |
+
any_abstain = any_abstain or not r.ok
|
| 461 |
+
return 2 if any_abstain else 0
|
| 462 |
+
|
| 463 |
+
|
| 464 |
+
def _version() -> str:
|
| 465 |
+
import importlib.metadata as md
|
| 466 |
+
try:
|
| 467 |
+
return md.version("juurio")
|
| 468 |
+
except Exception:
|
| 469 |
+
return "0.1.0"
|
| 470 |
+
|
| 471 |
+
|
| 472 |
+
if __name__ == "__main__":
|
| 473 |
+
sys.exit(main())
|
juurio/copyrealizer.py
ADDED
|
@@ -0,0 +1,162 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Pointer-generator char enc-dec — the copy mechanism.
|
| 2 |
+
|
| 3 |
+
The baseline net's exceptions are not morphology errors but STEM-COPYING failures: it corrupts
|
| 4 |
+
rare/long stem characters (abortoida->va: 'auortoiva' not 'abortoiva') while the affix is right. A
|
| 5 |
+
copy channel fixes exactly this: at each step the decoder may COPY a character from the source (the
|
| 6 |
+
lemma) instead of generating it from the vocab.
|
| 7 |
+
|
| 8 |
+
Mechanism (See et al. 2017, pointer-generator, char-level):
|
| 9 |
+
gen = softmax(proj(dec_out)) generation distribution over vocab
|
| 10 |
+
alpha = softmax(copy_q(dec_out) · copy_k(mem_src)) attention over SOURCE char positions
|
| 11 |
+
copy = scatter_add(alpha onto vocab by src char id)
|
| 12 |
+
p = sigmoid(gate(dec_out)) mix weight in [0,1]
|
| 13 |
+
P = p * gen + (1 - p) * copy final distribution
|
| 14 |
+
forward returns log P. The encoder/decoder are the SAME architecture as Realizer, bar three small
|
| 15 |
+
projections (copy_q, copy_k, gate).
|
| 16 |
+
"""
|
| 17 |
+
from __future__ import annotations
|
| 18 |
+
|
| 19 |
+
import math
|
| 20 |
+
|
| 21 |
+
import torch
|
| 22 |
+
import torch.nn as nn
|
| 23 |
+
|
| 24 |
+
from .realizer import BOS, EOS, PAD, Realizer, encode_chars
|
| 25 |
+
from .device import DEV
|
| 26 |
+
|
| 27 |
+
_EPS = 1e-8
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
class CopyRealizer(Realizer):
|
| 31 |
+
def __init__(self, vocab, d_model=128, nhead=4, layers=2, ff=512, maxlen=64):
|
| 32 |
+
super().__init__(vocab, d_model, nhead, layers, ff, maxlen)
|
| 33 |
+
self.vocab = vocab
|
| 34 |
+
self.copy_q = nn.Linear(d_model, d_model)
|
| 35 |
+
self.copy_k = nn.Linear(d_model, d_model)
|
| 36 |
+
self.gate = nn.Linear(d_model, 1)
|
| 37 |
+
self._scale = 1.0 / math.sqrt(d_model)
|
| 38 |
+
self.use_copy = True # False -> pure generator (copy channel disabled)
|
| 39 |
+
|
| 40 |
+
def _pgen(self, dec_out):
|
| 41 |
+
"""Per-step mix weight in [0,1] — how much to trust the generator vs the copy channel."""
|
| 42 |
+
return torch.sigmoid(self.gate(dec_out))
|
| 43 |
+
|
| 44 |
+
def _copy_dist(self, dec_out, mem, src, H):
|
| 45 |
+
"""Copy distribution over the vocab: attention over the SOURCE char positions of memory
|
| 46 |
+
(mem[:, H:] — the part after the H header slots), scattered onto vocab indices by src char id.
|
| 47 |
+
Padded source positions are masked out, so they contribute no probability."""
|
| 48 |
+
B, T, _ = dec_out.shape
|
| 49 |
+
S = src.size(1)
|
| 50 |
+
mem_src = mem[:, H:, :] # [B, S, d] — source-char memory
|
| 51 |
+
q = self.copy_q(dec_out) # [B, T, d]
|
| 52 |
+
k = self.copy_k(mem_src) # [B, S, d]
|
| 53 |
+
scores = torch.bmm(q, k.transpose(1, 2)) * self._scale # [B, T, S]
|
| 54 |
+
pad = (src == PAD).unsqueeze(1) # [B, 1, S]
|
| 55 |
+
scores = scores.masked_fill(pad, float("-inf"))
|
| 56 |
+
alpha = torch.softmax(scores, dim=-1) # [B, T, S]
|
| 57 |
+
alpha = torch.nan_to_num(alpha, nan=0.0) # all-pad row -> zeros (no source)
|
| 58 |
+
pcopy = torch.zeros(B, T, self.vocab, device=dec_out.device, dtype=alpha.dtype)
|
| 59 |
+
idx = src.unsqueeze(1).expand(B, T, S) # [B, T, S] vocab ids to scatter onto
|
| 60 |
+
pcopy.scatter_add_(2, idx, alpha)
|
| 61 |
+
return pcopy
|
| 62 |
+
|
| 63 |
+
def _dec_out(self, src, header, tgt_in):
|
| 64 |
+
mem, mask = self.encode(src, header)
|
| 65 |
+
T = tgt_in.size(1)
|
| 66 |
+
cmask = torch.triu(torch.ones(T, T, device=src.device, dtype=torch.bool), 1)
|
| 67 |
+
dec_out = self.dec(self._tgt_embed(tgt_in), mem, tgt_mask=cmask,
|
| 68 |
+
tgt_key_padding_mask=(tgt_in == PAD), memory_key_padding_mask=mask)
|
| 69 |
+
return dec_out, mem
|
| 70 |
+
|
| 71 |
+
def forward(self, src, header, tgt_in):
|
| 72 |
+
"""Return log P of shape [B, T, vocab] — train with NLLLoss(ignore_index=PAD). With use_copy
|
| 73 |
+
the mixed gen+copy distribution; without it the pure generator."""
|
| 74 |
+
dec_out, mem = self._dec_out(src, header, tgt_in)
|
| 75 |
+
if not self.use_copy:
|
| 76 |
+
return torch.log_softmax(self.proj(dec_out), dim=-1)
|
| 77 |
+
gen = torch.softmax(self.proj(dec_out), dim=-1) # [B, T, V]
|
| 78 |
+
copy = self._copy_dist(dec_out, mem, src, H=header.size(1))
|
| 79 |
+
p = self._pgen(dec_out) # [B, T, 1]
|
| 80 |
+
mixed = p * gen + (1.0 - p) * copy
|
| 81 |
+
return torch.log(mixed + _EPS)
|
| 82 |
+
|
| 83 |
+
|
| 84 |
+
def load(ckpt):
|
| 85 |
+
ck = torch.load(ckpt, map_location=DEV, weights_only=False)
|
| 86 |
+
m = CopyRealizer(len(ck["stoi"]), ck["cfg"]["d_model"], 4, ck["cfg"]["layers"]).to(DEV)
|
| 87 |
+
m.load_state_dict(ck["model"])
|
| 88 |
+
m.use_copy = ck["cfg"].get("use_copy", True)
|
| 89 |
+
m.eval()
|
| 90 |
+
return m, ck["stoi"], ck["itos"]
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
@torch.no_grad()
|
| 94 |
+
def decode_one(m, header, lemma, stoi, itos, maxlen=48, score=False):
|
| 95 |
+
"""Greedy decode over the mixed gen+copy distribution. B=1. If `score`, also return a confidence =
|
| 96 |
+
min over steps of the top-1 margin (top1 - top2 prob) on the MIXED distribution — same uncertainty
|
| 97 |
+
contract as realizer._decode, just measured on the distribution the copy model actually predicts."""
|
| 98 |
+
m.eval()
|
| 99 |
+
dev = next(m.parameters()).device
|
| 100 |
+
src = torch.tensor([encode_chars(lemma, stoi)], dtype=torch.long, device=dev)
|
| 101 |
+
hdr = torch.tensor([header], dtype=torch.long, device=dev)
|
| 102 |
+
ids = [BOS]
|
| 103 |
+
out = []
|
| 104 |
+
margin = 1.0
|
| 105 |
+
for _ in range(maxlen):
|
| 106 |
+
ti = torch.tensor([ids], dtype=torch.long, device=dev)
|
| 107 |
+
logp = m(src, hdr, ti)[0, -1]
|
| 108 |
+
nxt = int(logp.argmax())
|
| 109 |
+
if score:
|
| 110 |
+
p = logp.float().exp() # logp is already log P (mixed), exp -> probs
|
| 111 |
+
top2 = torch.topk(p, 2).values
|
| 112 |
+
margin = min(margin, float(top2[0] - top2[1]))
|
| 113 |
+
if nxt == EOS:
|
| 114 |
+
break
|
| 115 |
+
out.append(nxt)
|
| 116 |
+
ids.append(nxt)
|
| 117 |
+
form = "".join(itos[t] for t in out if t in itos)
|
| 118 |
+
return (form, margin) if score else form
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
@torch.no_grad()
|
| 122 |
+
def decode_forms(items, bundle, batch_size=256, maxlen=48):
|
| 123 |
+
"""Greedy decode over the mixed gen+copy distribution, batched; bit-identical to decode_one. The
|
| 124 |
+
prune pass MUST use this (not realizer.proj-only decode) or it would measure the generator alone.
|
| 125 |
+
items: iterable of (lemma, header_ids_list)."""
|
| 126 |
+
from .realizer import _pad
|
| 127 |
+
items = list(items)
|
| 128 |
+
if not items:
|
| 129 |
+
return []
|
| 130 |
+
m, stoi, itos = bundle
|
| 131 |
+
m.eval()
|
| 132 |
+
dev = next(m.parameters()).device
|
| 133 |
+
out = []
|
| 134 |
+
for start in range(0, len(items), batch_size):
|
| 135 |
+
chunk = items[start:start + batch_size]
|
| 136 |
+
B = len(chunk)
|
| 137 |
+
# respect caller casing (matches decode_one): PROPN comes in capitalized, NOUN/ADJ/verbs
|
| 138 |
+
# lowercase. The caller (batched/verbforms) is responsible for POS-correct casing.
|
| 139 |
+
src_lists = [encode_chars(lem, stoi) for lem, _ in chunk]
|
| 140 |
+
S = max((len(s) for s in src_lists), default=1) or 1
|
| 141 |
+
src = _pad(src_lists, S).to(dev)
|
| 142 |
+
hdr = torch.tensor([h for _, h in chunk], dtype=torch.long, device=dev)
|
| 143 |
+
ids = torch.full((B, 1), BOS, dtype=torch.long, device=dev)
|
| 144 |
+
finished = torch.zeros(B, dtype=torch.bool, device=dev)
|
| 145 |
+
collected = [[] for _ in range(B)]
|
| 146 |
+
for _ in range(maxlen):
|
| 147 |
+
logp = m(src, hdr, ids)[:, -1] # mixed log-probs at last position
|
| 148 |
+
nxt = logp.argmax(-1)
|
| 149 |
+
nxt_cpu = nxt.tolist()
|
| 150 |
+
for i in range(B):
|
| 151 |
+
if finished[i]:
|
| 152 |
+
continue
|
| 153 |
+
if nxt_cpu[i] == EOS:
|
| 154 |
+
finished[i] = True
|
| 155 |
+
else:
|
| 156 |
+
collected[i].append(nxt_cpu[i])
|
| 157 |
+
app = nxt.clone(); app[finished] = PAD
|
| 158 |
+
ids = torch.cat([ids, app.unsqueeze(1)], 1)
|
| 159 |
+
if bool(finished.all()):
|
| 160 |
+
break
|
| 161 |
+
out.extend("".join(itos[t] for t in row if t in itos) for row in collected)
|
| 162 |
+
return out
|
juurio/device.py
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""The torch device string, factored out of `train` so the SERVING modules don't load the training
|
| 2 |
+
module (nanoGPT model + training loop) just to learn whether a GPU is present. `train.DEV` re-exports
|
| 3 |
+
this for back-compat. Serving needs only torch; importing this pulls in nothing else."""
|
| 4 |
+
from __future__ import annotations
|
| 5 |
+
|
| 6 |
+
import torch
|
| 7 |
+
|
| 8 |
+
DEV = "cuda" if torch.cuda.is_available() else "cpu"
|
juurio/features.py
ADDED
|
@@ -0,0 +1,272 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Human-readable feature-alias normalization for the friendly generation API (`inflect`).
|
| 2 |
+
|
| 3 |
+
Callers should be able to say "gen"/"genitive"/"genetiivi" or a combined "mon gen" instead of the
|
| 4 |
+
engine's internal vocab. These helpers map every accepted alias onto the EXACT canonical strings the
|
| 5 |
+
engine expects (Case=Gen, Number=Plur, Degree=Cmp, Px=1Sg, …) and fail LOUD (ValueError) on an
|
| 6 |
+
unknown alias — an unrecognized case is a caller mistake, not data to silently coerce.
|
| 7 |
+
|
| 8 |
+
Pure module: no torch/parser/heavy deps, safe to import anywhere. The canonical vocab mirrors
|
| 9 |
+
`realizer.CASE_VALS/NUM_VALS/DEG_VALS/PX_VALS`."""
|
| 10 |
+
from __future__ import annotations
|
| 11 |
+
|
| 12 |
+
# canonical value -> accepted aliases (English abbrev/full + Finnish). The canonical string itself is
|
| 13 |
+
# always accepted too, case-insensitively (added in `_invert`).
|
| 14 |
+
_CASE = {
|
| 15 |
+
"Nom": ["nom", "nominative", "nominatiivi"],
|
| 16 |
+
"Gen": ["gen", "genitive", "genitiivi", "genetiivi"],
|
| 17 |
+
"Par": ["par", "ptv", "part", "partitive", "partitiivi"],
|
| 18 |
+
"Ess": ["ess", "essive", "essiivi"],
|
| 19 |
+
"Tra": ["tra", "transl", "translative", "translatiivi"],
|
| 20 |
+
"Ine": ["ine", "iness", "inessive", "inessiivi"],
|
| 21 |
+
"Ela": ["ela", "elat", "elative", "elatiivi"],
|
| 22 |
+
"Ill": ["ill", "illat", "illative", "illatiivi"],
|
| 23 |
+
"Ade": ["ade", "adess", "adessive", "adessiivi"],
|
| 24 |
+
"Abl": ["abl", "ablat", "ablative", "ablatiivi"],
|
| 25 |
+
"All": ["all", "allat", "allative", "allatiivi"],
|
| 26 |
+
}
|
| 27 |
+
_NUMBER = {
|
| 28 |
+
"Sing": ["sing", "sg", "singular", "yks", "yksikko", "yksikkö"],
|
| 29 |
+
"Plur": ["plur", "pl", "plural", "mon", "monikko"],
|
| 30 |
+
}
|
| 31 |
+
_DEGREE = {
|
| 32 |
+
"Pos": ["pos", "positive", "perusaste"],
|
| 33 |
+
"Cmp": ["comp", "cmp", "comparative", "komparatiivi", "vertailuaste"],
|
| 34 |
+
"Sup": ["sup", "super", "superl", "superlative", "superlatiivi", "yliaste"],
|
| 35 |
+
}
|
| 36 |
+
_PX = {
|
| 37 |
+
"None": ["none", ""],
|
| 38 |
+
"1Sg": ["1sg", "1s"],
|
| 39 |
+
"2Sg": ["2sg", "2s"],
|
| 40 |
+
"3": ["3", "3sg", "3pl", "3s", "3p"],
|
| 41 |
+
"1Pl": ["1pl", "1p"],
|
| 42 |
+
"2Pl": ["2pl", "2p"],
|
| 43 |
+
}
|
| 44 |
+
# Enclitic particles, canonical (back-harmony) form -> accepted surface variants. The harmony variant
|
| 45 |
+
# actually emitted is chosen from the host word at realization time (algebra._clitic_for), so both
|
| 46 |
+
# variants normalize to the same canonical key. -kin is harmony-invariant.
|
| 47 |
+
_CLITIC = {
|
| 48 |
+
"kin": ["kin"],
|
| 49 |
+
"kaan": ["kaan", "kään"],
|
| 50 |
+
"ko": ["ko", "kö"],
|
| 51 |
+
"han": ["han", "hän"],
|
| 52 |
+
"pa": ["pa", "pä"],
|
| 53 |
+
}
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
def _invert(table: dict[str, list[str]]) -> dict[str, str]:
|
| 57 |
+
out: dict[str, str] = {}
|
| 58 |
+
for canon, aliases in table.items():
|
| 59 |
+
out[canon.lower()] = canon
|
| 60 |
+
for a in aliases:
|
| 61 |
+
out[a.lower()] = canon
|
| 62 |
+
return out
|
| 63 |
+
|
| 64 |
+
|
| 65 |
+
_CASE_ALIASES = _invert(_CASE)
|
| 66 |
+
_NUMBER_ALIASES = _invert(_NUMBER)
|
| 67 |
+
_DEGREE_ALIASES = _invert(_DEGREE)
|
| 68 |
+
_PX_ALIASES = _invert(_PX)
|
| 69 |
+
_CLITIC_ALIASES = _invert(_CLITIC)
|
| 70 |
+
|
| 71 |
+
|
| 72 |
+
def _lookup(table: dict[str, str], value: str, dim: str) -> str:
|
| 73 |
+
key = value.strip().lower()
|
| 74 |
+
try:
|
| 75 |
+
return table[key]
|
| 76 |
+
except KeyError:
|
| 77 |
+
canon = sorted(set(table.values()))
|
| 78 |
+
raise ValueError(f"unknown {dim} {value!r}; accepted canonical values {canon} "
|
| 79 |
+
f"or their aliases") from None
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
def normalize_case(value: str) -> str:
|
| 83 |
+
"""Map a case alias ('gen'/'genitive'/'genetiivi'/'Gen') to its canonical Case value ('Gen')."""
|
| 84 |
+
return _lookup(_CASE_ALIASES, value, "case")
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
def normalize_number(value: str) -> str:
|
| 88 |
+
"""Map a number alias ('mon'/'plural'/'pl'/'Plur') to its canonical Number value ('Plur')."""
|
| 89 |
+
return _lookup(_NUMBER_ALIASES, value, "number")
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
def normalize_degree(value: str) -> str:
|
| 93 |
+
"""Map a degree alias ('comp'/'comparative'/'Cmp') to its canonical Degree value ('Cmp')."""
|
| 94 |
+
return _lookup(_DEGREE_ALIASES, value, "degree")
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
def normalize_px(value: str | None) -> str:
|
| 98 |
+
"""Map a possessive-suffix alias ('1sg'/'1Sg'/''/None) to its canonical Px value ('1Sg'). `None`
|
| 99 |
+
is accepted as "no possessive suffix" (mirrors clitic=None) and maps to 'None'."""
|
| 100 |
+
if value is None:
|
| 101 |
+
return "None"
|
| 102 |
+
return _lookup(_PX_ALIASES, value, "possessive (px)")
|
| 103 |
+
|
| 104 |
+
|
| 105 |
+
def normalize_clitic(value: str | None) -> str | None:
|
| 106 |
+
"""Map an enclitic alias ('ko'/'kö'/'KO') to its canonical particle ('ko'); `None` (no clitic)
|
| 107 |
+
passes through as `None`. An unknown alias raises ValueError (a typo, not data to coerce)."""
|
| 108 |
+
if value is None:
|
| 109 |
+
return None
|
| 110 |
+
return _lookup(_CLITIC_ALIASES, value, "clitic")
|
| 111 |
+
|
| 112 |
+
|
| 113 |
+
# --- inflect-by-example enclitic peeling (pure heuristic; the engine applies the semantic gate) -----
|
| 114 |
+
_BACK_VOWELS = set("aou")
|
| 115 |
+
_VOWELS = set("aeiouyäöå")
|
| 116 |
+
# (surface, canonical) longest surface first so '-kään'/'-kaan' beat a spurious '-kin'-style short match.
|
| 117 |
+
_ENCLITIC_SURFACES = [("kään", "kaan"), ("kaan", "kaan"), ("kin", "kin"), ("han", "han"),
|
| 118 |
+
("hän", "han"), ("ko", "ko"), ("kö", "ko"), ("pa", "pa"), ("pä", "pa")]
|
| 119 |
+
_FRONT_ENCLITICS = {"kään", "hän", "pä", "kö"} # may only follow a front/neutral-harmony stem
|
| 120 |
+
_BACK_ENCLITICS = {"kaan", "han", "pa", "ko"} # may only follow a back-harmony stem
|
| 121 |
+
|
| 122 |
+
|
| 123 |
+
def split_enclitic(word: str) -> tuple[str, str | None]:
|
| 124 |
+
"""Peel a trailing Finnish focus/interrogative enclitic (-kO/-kin/-kAAn/-hAn/-pA) off a MODEL word
|
| 125 |
+
for inflect-by-example, returning (stem, canonical_particle) or (word, None) when nothing peels.
|
| 126 |
+
|
| 127 |
+
Pure string heuristic — three conservative gates so a lemma that merely *ends* in clitic letters
|
| 128 |
+
(aurinko, vihko, pako) is not mis-split: the variant must agree with the stem's vowel harmony
|
| 129 |
+
(front -kö only after a front stem, back -ko only after a back stem), the stem must be >= 3 chars,
|
| 130 |
+
and it must end in a vowel or -n/-t (the legal final segments of an inflected Finnish nominal —
|
| 131 |
+
rejects fragments like 'kaak'/'tark'). -kin is harmony-invariant. The CALLER still applies the
|
| 132 |
+
semantic gate (the stem must analyze to a genuine non-nominative inflected host and round-trip),
|
| 133 |
+
which removes the remaining lexical collisions; this function only proposes the candidate.
|
| 134 |
+
Combinations (-kohan, -pas) and -kaan after a vowel stem are out of scope."""
|
| 135 |
+
wl = word.lower()
|
| 136 |
+
for surf, canon in _ENCLITIC_SURFACES:
|
| 137 |
+
if not wl.endswith(surf) or len(word) - len(surf) < 3:
|
| 138 |
+
continue
|
| 139 |
+
stem = word[:-len(surf)]
|
| 140 |
+
stem_back = any(c in _BACK_VOWELS for c in stem.lower())
|
| 141 |
+
if surf in _FRONT_ENCLITICS and stem_back:
|
| 142 |
+
continue
|
| 143 |
+
if surf in _BACK_ENCLITICS and not stem_back:
|
| 144 |
+
continue
|
| 145 |
+
last = stem[-1].lower()
|
| 146 |
+
if last not in _VOWELS and last not in "nt":
|
| 147 |
+
continue
|
| 148 |
+
return stem, canon
|
| 149 |
+
return word, None
|
| 150 |
+
|
| 151 |
+
|
| 152 |
+
def px_from_psor(person: str | None, number: str | None) -> str:
|
| 153 |
+
"""Map a parser's UD possessor features (Person[psor], Number[psor]) onto our canonical Px value
|
| 154 |
+
(for inflect-by-example). 1/2 distinguish singular vs plural possessor (1Sg/2Sg/1Pl/2Pl); 3 is
|
| 155 |
+
number-agnostic ('3'); no possessor (person is None / empty) maps to 'None'."""
|
| 156 |
+
if not person:
|
| 157 |
+
return "None"
|
| 158 |
+
if person == "3":
|
| 159 |
+
return "3"
|
| 160 |
+
return f"{person}{'Pl' if number == 'Plur' else 'Sg'}"
|
| 161 |
+
|
| 162 |
+
|
| 163 |
+
class FeatureConflict(ValueError):
|
| 164 |
+
"""Two individually-valid feature inputs contradict each other (e.g. spec 'mon gen' together
|
| 165 |
+
with number='sing', a spec with two cases 'gen par', or proper=True with a comparative degree).
|
| 166 |
+
Distinct from a plain ValueError (an unknown alias / typo): a conflict is a CALLER contradiction
|
| 167 |
+
the friendly API surfaces as a first-class abstain (`conflicting_features`); a typo fails fast."""
|
| 168 |
+
|
| 169 |
+
|
| 170 |
+
class _Unset:
|
| 171 |
+
"""Sentinel type for "argument not supplied" (so an explicit value that contradicts a combined
|
| 172 |
+
spec can be told apart from the default). Single instance `UNSET`, with a stable repr."""
|
| 173 |
+
_instance = None
|
| 174 |
+
|
| 175 |
+
def __new__(cls):
|
| 176 |
+
if cls._instance is None:
|
| 177 |
+
cls._instance = super().__new__(cls)
|
| 178 |
+
return cls._instance
|
| 179 |
+
|
| 180 |
+
def __repr__(self):
|
| 181 |
+
return "UNSET"
|
| 182 |
+
|
| 183 |
+
|
| 184 |
+
# sentinel so callers can tell "argument not supplied" from an explicit value (case/number/degree)
|
| 185 |
+
UNSET = _Unset()
|
| 186 |
+
|
| 187 |
+
# the dimensions a combined spec token may classify into (Px is a separate kwarg, never in a spec)
|
| 188 |
+
_SPEC_DIMS = (("Case", _CASE_ALIASES), ("Number", _NUMBER_ALIASES), ("Degree", _DEGREE_ALIASES))
|
| 189 |
+
|
| 190 |
+
|
| 191 |
+
def parse_feature_spec(spec: str) -> dict[str, str]:
|
| 192 |
+
"""Parse a combined human spec ('mon gen' / 'plural genitive comparative') into a canonical
|
| 193 |
+
feature dict ({'Number':'Plur','Case':'Gen'[, 'Degree':...]}). Each whitespace token must
|
| 194 |
+
classify into exactly one dimension. An unknown token raises ValueError (a typo); two tokens
|
| 195 |
+
resolving to DIFFERENT values in the same dimension (a contradiction like 'gen par') raise
|
| 196 |
+
FeatureConflict. Repeated aliases of the same value ('gen genitive') are fine."""
|
| 197 |
+
out: dict[str, str] = {}
|
| 198 |
+
for tok in spec.split():
|
| 199 |
+
key = tok.lower()
|
| 200 |
+
for dim, table in _SPEC_DIMS:
|
| 201 |
+
if key in table:
|
| 202 |
+
val = table[key]
|
| 203 |
+
if dim in out and out[dim] != val:
|
| 204 |
+
raise FeatureConflict(f"conflicting {dim} in spec {spec!r}: {out[dim]} vs {val}")
|
| 205 |
+
out[dim] = val
|
| 206 |
+
break
|
| 207 |
+
else:
|
| 208 |
+
raise ValueError(f"unknown feature token {tok!r} in spec {spec!r}")
|
| 209 |
+
return out
|
| 210 |
+
|
| 211 |
+
|
| 212 |
+
# the UPOS values `inflect` supports as a single-word nominal (the realizer's POS_VALS). PRON/NUM are
|
| 213 |
+
# NOT trained — the realizer would silently clamp them to NOUN — so they are rejected, not advertised.
|
| 214 |
+
_POS_ALIASES = {
|
| 215 |
+
"noun": "NOUN", "n": "NOUN",
|
| 216 |
+
"proper": "PROPN", "propn": "PROPN", "proper noun": "PROPN", "name": "PROPN",
|
| 217 |
+
"adjective": "ADJ", "adj": "ADJ", "a": "ADJ",
|
| 218 |
+
}
|
| 219 |
+
|
| 220 |
+
|
| 221 |
+
def resolve_features(case: str = "nom", number=UNSET, degree=UNSET) -> dict[str, str]:
|
| 222 |
+
"""Merge a (possibly combined) `case` spec with explicit `number`/`degree` kwargs into a
|
| 223 |
+
canonical {Case, Number, Degree} cell. `case` may be a single alias ('gen'), a number/degree
|
| 224 |
+
alias ('mon'), or a combined spec ('mon gen'). An explicit kwarg that contradicts the spec
|
| 225 |
+
raises FeatureConflict; an unknown alias raises plain ValueError. Missing dimensions default to
|
| 226 |
+
Case=Nom, Number=Sing, Degree=Pos."""
|
| 227 |
+
feats = parse_feature_spec(case)
|
| 228 |
+
feats.setdefault("Case", "Nom")
|
| 229 |
+
if number is not UNSET:
|
| 230 |
+
n = normalize_number(number)
|
| 231 |
+
if feats.get("Number", n) != n:
|
| 232 |
+
raise FeatureConflict(f"number={number!r} ({n}) conflicts with spec {case!r} "
|
| 233 |
+
f"(Number={feats['Number']})")
|
| 234 |
+
feats["Number"] = n
|
| 235 |
+
feats.setdefault("Number", "Sing")
|
| 236 |
+
if degree is not UNSET:
|
| 237 |
+
d = normalize_degree(degree)
|
| 238 |
+
if feats.get("Degree", d) != d:
|
| 239 |
+
raise FeatureConflict(f"degree={degree!r} ({d}) conflicts with spec {case!r} "
|
| 240 |
+
f"(Degree={feats['Degree']})")
|
| 241 |
+
feats["Degree"] = d
|
| 242 |
+
feats.setdefault("Degree", "Pos")
|
| 243 |
+
return feats
|
| 244 |
+
|
| 245 |
+
|
| 246 |
+
def resolve_pos(word: str, *, pos: str | None = None, proper: bool | None = None,
|
| 247 |
+
degree: str = "Pos") -> str:
|
| 248 |
+
"""Decide the UPOS for a single-word `inflect`. Explicit `pos` ('noun'/'proper'/'adjective')
|
| 249 |
+
wins; `proper=True/False` is sugar for PROPN/NOUN; otherwise infer (canonical `degree` != Pos =>
|
| 250 |
+
ADJ; leading capital => PROPN; else NOUN). Contradictions raise FeatureConflict; an unsupported
|
| 251 |
+
`pos` (e.g. 'pronoun') raises plain ValueError."""
|
| 252 |
+
explicit: str | None = None
|
| 253 |
+
if pos is not None:
|
| 254 |
+
key = pos.strip().lower()
|
| 255 |
+
if key not in _POS_ALIASES:
|
| 256 |
+
raise ValueError(f"unsupported pos {pos!r}; inflect handles "
|
| 257 |
+
f"{sorted(set(_POS_ALIASES.values()))} (noun/proper/adjective)")
|
| 258 |
+
explicit = _POS_ALIASES[key]
|
| 259 |
+
if proper is not None:
|
| 260 |
+
sugar = "PROPN" if proper else "NOUN"
|
| 261 |
+
if explicit is not None and explicit != sugar:
|
| 262 |
+
raise FeatureConflict(f"proper={proper} conflicts with pos={pos!r}")
|
| 263 |
+
explicit = sugar
|
| 264 |
+
if explicit is not None:
|
| 265 |
+
if explicit != "ADJ" and degree != "Pos":
|
| 266 |
+
raise FeatureConflict(f"degree={degree} (comparative/superlative) is realized through the "
|
| 267 |
+
f"adjective path, but pos resolved to {explicit}; drop the degree or "
|
| 268 |
+
f"inflect it as an adjective")
|
| 269 |
+
return explicit
|
| 270 |
+
if degree != "Pos":
|
| 271 |
+
return "ADJ"
|
| 272 |
+
return "PROPN" if word[:1].isupper() else "NOUN"
|
juurio/finiteverb.py
ADDED
|
@@ -0,0 +1,138 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Finite-verb conjugation net (inference).
|
| 2 |
+
|
| 3 |
+
realizer B covers nominals (NOUN/ADJ/PROPN); verbforms.py covers non-finite verb forms (participles,
|
| 4 |
+
infinitives). This module covers FINITE conjugation — `puhun` (I speak), `käveli` (s/he walked),
|
| 5 |
+
`menisivät` (they would go), `puhutaan` (one speaks) — the largest, most-used verb subsystem. It is the
|
| 6 |
+
same `Realizer` char enc-dec, with a categorical header over the finite grid:
|
| 7 |
+
Voice (Act/Pss) × Mood (Ind/Cond/Pot/Imprt) × Tense (Prs/Prt, Ind only) × Person (Sg1..Pl3/ConNeg;
|
| 8 |
+
passive = Pe4). 37 non-empty cells, pinned by tests/test_finiteverb_cells.py.
|
| 9 |
+
|
| 10 |
+
Mirrors verbforms.py (header_ids/load/decode) so the decode path is uniform across all POS.
|
| 11 |
+
"""
|
| 12 |
+
from __future__ import annotations
|
| 13 |
+
|
| 14 |
+
import os
|
| 15 |
+
|
| 16 |
+
from juurio import bootstrap
|
| 17 |
+
from juurio.realizer import encode_chars
|
| 18 |
+
|
| 19 |
+
# --- finite grid (Voice, Mood, Tense, Person). Tense is None for Cond/Pot/Imprt. ----------------
|
| 20 |
+
VOICE = ["Act", "Pss"]
|
| 21 |
+
PERSONS = ["Sg1", "Sg2", "Sg3", "Pl1", "Pl2", "Pl3", "ConNeg"]
|
| 22 |
+
# (mood, [tenses]) — Ind carries tense; the others don't (tense None)
|
| 23 |
+
_ACT_MOODS = [("Ind", ["Prs", "Prt"]), ("Cond", [None]), ("Pot", [None]), ("Imprt", [None])]
|
| 24 |
+
_PSS_MOODS = [("Ind", ["Prs", "Prt"]), ("Cond", [None]), ("Pot", [None])]
|
| 25 |
+
# systematic empty cells that do not exist (excluded from the grid)
|
| 26 |
+
_EMPTY = {("Act", "Ind", "Prt", "ConNeg"), ("Act", "Imprt", "Sg1")}
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
def _build_cells():
|
| 30 |
+
cells = []
|
| 31 |
+
for mood, tenses in _ACT_MOODS:
|
| 32 |
+
for tense in tenses:
|
| 33 |
+
for per in PERSONS:
|
| 34 |
+
key = ("Act", mood, *((tense,) if tense else ()), per)
|
| 35 |
+
if key in _EMPTY:
|
| 36 |
+
continue
|
| 37 |
+
cells.append({"Voice": "Act", "Mood": mood, "Tense": tense, "Person": per})
|
| 38 |
+
for mood, tenses in _PSS_MOODS:
|
| 39 |
+
for tense in tenses:
|
| 40 |
+
cells.append({"Voice": "Pss", "Mood": mood, "Tense": tense, "Person": "Pe4"})
|
| 41 |
+
return cells
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
def cell_key(cell: dict):
|
| 45 |
+
"""Order-stable identity tuple: (Voice, Mood, [Tense], Person), Tense omitted when None."""
|
| 46 |
+
t = cell.get("Tense")
|
| 47 |
+
if t is None:
|
| 48 |
+
return (cell["Voice"], cell["Mood"], cell["Person"])
|
| 49 |
+
return (cell["Voice"], cell["Mood"], t, cell["Person"])
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
CELLS = _build_cells()
|
| 53 |
+
|
| 54 |
+
# --- categorical header (fixed-length, clamp unknowns so the loaded net never KeyErrors) ----------
|
| 55 |
+
_MOODS = ["Ind", "Cond", "Pot", "Imprt"]
|
| 56 |
+
_TENSES = ["_None", "Prs", "Prt"]
|
| 57 |
+
_ALLPERS = PERSONS + ["Pe4"]
|
| 58 |
+
FEAT_GROUPS = [("Voice", VOICE), ("Mood", _MOODS), ("Tense", _TENSES), ("Person", _ALLPERS)]
|
| 59 |
+
FEAT_STOI = {f"{g}={v}": i for i, (g, v) in
|
| 60 |
+
enumerate((g, v) for g, vals in FEAT_GROUPS for v in vals)}
|
| 61 |
+
NFEAT_TOK = len(FEAT_STOI)
|
| 62 |
+
HLEN = len(FEAT_GROUPS)
|
| 63 |
+
|
| 64 |
+
|
| 65 |
+
def header_ids(cell: dict) -> list:
|
| 66 |
+
v = cell.get("Voice") if cell.get("Voice") in VOICE else "Act"
|
| 67 |
+
m = cell.get("Mood") if cell.get("Mood") in _MOODS else "Ind"
|
| 68 |
+
t = cell.get("Tense") or "_None"
|
| 69 |
+
t = t if t in _TENSES else "_None"
|
| 70 |
+
p = cell.get("Person") if cell.get("Person") in _ALLPERS else "Sg3"
|
| 71 |
+
return [FEAT_STOI[f"Voice={v}"], FEAT_STOI[f"Mood={m}"],
|
| 72 |
+
FEAT_STOI[f"Tense={t}"], FEAT_STOI[f"Person={p}"]]
|
| 73 |
+
|
| 74 |
+
|
| 75 |
+
CKPT = os.path.join(os.path.dirname(__file__), "..", "checkpoints", "finiteverb.pt")
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
def load(ckpt: str | None = None):
|
| 79 |
+
import torch
|
| 80 |
+
from juurio.realizer import Realizer
|
| 81 |
+
from juurio.device import DEV
|
| 82 |
+
if ckpt is None:
|
| 83 |
+
ckpt = bootstrap.artifact_path("finiteverb")
|
| 84 |
+
ck = torch.load(ckpt, map_location=DEV, weights_only=False)
|
| 85 |
+
m = Realizer(len(ck["stoi"]), ck["cfg"]["d_model"], 4, ck["cfg"]["layers"]).to(DEV)
|
| 86 |
+
m.load_state_dict(ck["model"]); m.eval()
|
| 87 |
+
return m, ck["stoi"], ck["itos"]
|
| 88 |
+
|
| 89 |
+
|
| 90 |
+
def decode_one(bundle, lemma: str, cell: dict, maxlen: int = 48) -> str:
|
| 91 |
+
from juurio.realizer import _decode
|
| 92 |
+
m, stoi, itos = bundle
|
| 93 |
+
return _decode(m, header_ids(cell), lemma.lower(), stoi, itos, maxlen=maxlen)
|
| 94 |
+
|
| 95 |
+
|
| 96 |
+
def decode_forms(items, bundle, batch_size: int = 256, maxlen: int = 48):
|
| 97 |
+
"""Greedy-decode a finite form for each (lemma, cell). Batched; bit-identical to decode_one
|
| 98 |
+
(greedy argmax + per-seq attention). items: iterable of (lemma, cell-dict)."""
|
| 99 |
+
import torch
|
| 100 |
+
from juurio.realizer import BOS, EOS, PAD, _pad
|
| 101 |
+
from juurio.device import DEV
|
| 102 |
+
items = list(items)
|
| 103 |
+
if not items:
|
| 104 |
+
return []
|
| 105 |
+
m, stoi, itos = bundle
|
| 106 |
+
m.eval()
|
| 107 |
+
out = []
|
| 108 |
+
for start in range(0, len(items), batch_size):
|
| 109 |
+
chunk = items[start:start + batch_size]
|
| 110 |
+
B = len(chunk)
|
| 111 |
+
src_lists = [encode_chars(lem.lower(), stoi) for lem, _ in chunk]
|
| 112 |
+
S = max((len(s) for s in src_lists), default=1) or 1
|
| 113 |
+
src = _pad(src_lists, S).to(DEV)
|
| 114 |
+
hdr = torch.tensor([header_ids(cell) for _, cell in chunk], dtype=torch.long, device=DEV)
|
| 115 |
+
with torch.no_grad():
|
| 116 |
+
mem, mask = m.encode(src, hdr)
|
| 117 |
+
ids = torch.full((B, 1), BOS, dtype=torch.long, device=DEV)
|
| 118 |
+
finished = torch.zeros(B, dtype=torch.bool, device=DEV)
|
| 119 |
+
collected = [[] for _ in range(B)]
|
| 120 |
+
for _ in range(maxlen):
|
| 121 |
+
T = ids.size(1)
|
| 122 |
+
cmask = torch.triu(torch.ones(T, T, device=DEV, dtype=torch.bool), 1)
|
| 123 |
+
dout = m.dec(m._tgt_embed(ids), mem, tgt_mask=cmask, memory_key_padding_mask=mask)
|
| 124 |
+
nxt = m.proj(dout)[:, -1].argmax(-1)
|
| 125 |
+
nxt_cpu = nxt.tolist()
|
| 126 |
+
for i in range(B):
|
| 127 |
+
if finished[i]:
|
| 128 |
+
continue
|
| 129 |
+
if nxt_cpu[i] == EOS:
|
| 130 |
+
finished[i] = True
|
| 131 |
+
else:
|
| 132 |
+
collected[i].append(nxt_cpu[i])
|
| 133 |
+
app = nxt.clone(); app[finished] = PAD
|
| 134 |
+
ids = torch.cat([ids, app.unsqueeze(1)], 1)
|
| 135 |
+
if bool(finished.all()):
|
| 136 |
+
break
|
| 137 |
+
out.extend("".join(itos[t] for t in row if t in itos) for row in collected)
|
| 138 |
+
return out
|
juurio/genlex.py
ADDED
|
@@ -0,0 +1,124 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Runtime generation lexicon — the static exception table that makes net+lexicon generation byte-exact.
|
| 2 |
+
|
| 3 |
+
At serving time the trained realizer nets (`realizer.pt` nominal, `verbform.pt` non-finite,
|
| 4 |
+
`finiteverb.pt` finite) are the productive engine; this lexicon (`data/shipped_wordlist.json`) carries
|
| 5 |
+
ONLY the (subsystem, lemma, cell) cells where a net disagrees with the Giella FST — look it up first,
|
| 6 |
+
fall through to the net on a miss. It is the GENERATION analogue of the analysis side's static FST-tag
|
| 7 |
+
table: model + lexicon, no live FST.
|
| 8 |
+
|
| 9 |
+
Byte-exactness is the whole point: a key the loader builds must equal the key the wordlist builder
|
| 10 |
+
wrote, or every override is silently missed. The key formats (per subsystem) are therefore mirrored
|
| 11 |
+
from that builder verbatim:
|
| 12 |
+
|
| 13 |
+
NOUN|<lemma>|('Case', V)+('Number', V)+('Px', V) # tuple(sorted(cell.items())) repr
|
| 14 |
+
ADJ|<lemma>|('Case', V)+('Degree', V)+('Number', V)
|
| 15 |
+
VERB_FIN|<lemma>|Voice+Mood[+Tense]+Person # finiteverb.cell_key(cell)
|
| 16 |
+
VERB_NONFIN|<lemma>|form+case+number
|
| 17 |
+
|
| 18 |
+
PROPN is intentionally NOT shipped (open class), so nominal lookups for PROPN always abstain.
|
| 19 |
+
"""
|
| 20 |
+
from __future__ import annotations
|
| 21 |
+
|
| 22 |
+
import json
|
| 23 |
+
import os
|
| 24 |
+
|
| 25 |
+
from . import bootstrap
|
| 26 |
+
|
| 27 |
+
# Repo root (where data/ lives), so the default artifact resolves against the install, not the process
|
| 28 |
+
# CWD — mirrors parser.neural_adapter._ROOT / bootstrap.ROOT so serving works from any directory.
|
| 29 |
+
_ROOT = os.path.normpath(os.path.join(os.path.dirname(__file__), ".."))
|
| 30 |
+
CKPT = os.path.join("data", "shipped_wordlist.json")
|
| 31 |
+
# In-tree FST-parity overrides merged on top of the artifact at load time (resolved against the PACKAGE
|
| 32 |
+
# dir, not the repo root, so they ship with `juurio`). See `load`. Two files, both merged: the VERB
|
| 33 |
+
# register corrections and the net-unreliable common-NOUN paradigms. Independent so each regenerates
|
| 34 |
+
# without clobbering the other.
|
| 35 |
+
OVERRIDES = ("genlex_overrides.json", "genlex_overrides_noun.json")
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
def _resolve(path: str) -> str:
|
| 39 |
+
return path if os.path.isabs(path) else os.path.join(_ROOT, path)
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
class GenerationLexicon:
|
| 43 |
+
"""The shipped exception table loaded into a flat dict, with per-subsystem byte-exact key lookups.
|
| 44 |
+
|
| 45 |
+
Construct via `GenerationLexicon.load()` (default `data/shipped_wordlist.json`, repo-root-resolved).
|
| 46 |
+
Each `lookup_*` returns the stored gold form or None (miss ⇒ the caller runs the net)."""
|
| 47 |
+
|
| 48 |
+
def __init__(self, table: dict):
|
| 49 |
+
self._table = table
|
| 50 |
+
|
| 51 |
+
@classmethod
|
| 52 |
+
def load(cls, path: str | None = None, override_path: str | None = None) -> "GenerationLexicon":
|
| 53 |
+
# Explicit path resolves against ROOT; default routes through the bootstrap resolver
|
| 54 |
+
# (dev source -> HF), so a clean wheel install fetches the shipped wordlist artifact.
|
| 55 |
+
resolved = _resolve(path) if path else bootstrap.artifact_path("wordlist")
|
| 56 |
+
try:
|
| 57 |
+
with open(resolved, encoding="utf-8") as fh:
|
| 58 |
+
table = json.load(fh)
|
| 59 |
+
except FileNotFoundError as e:
|
| 60 |
+
# The default net+lexicon generation backend loads this at Transformer() construction, so a
|
| 61 |
+
# missing artifact is a setup problem, not a programming bug — point the user at the fetcher.
|
| 62 |
+
raise FileNotFoundError(
|
| 63 |
+
f"generation lexicon not found at {resolved!r}; it is required by the default net+lexicon "
|
| 64 |
+
f"generation backend. Run `juurio setup` (or `juurio doctor`) to fetch the shipped "
|
| 65 |
+
f"artifacts listed in bootstrap.MANIFEST."
|
| 66 |
+
) from e
|
| 67 |
+
# Merge the committed, in-tree FST-parity overrides ON TOP of the downloaded artifact: VERB cells
|
| 68 |
+
# where the artifact's variant picker shipped the wrong register (archaic `-oitse-`, colloquial
|
| 69 |
+
# `mee`), and NOUN paradigms for common lemmas the realizer net serves unreliably (aika->ajan,
|
| 70 |
+
# mökä->mökän), each corrected to the FST-valid form. Version-controlled, so they survive a
|
| 71 |
+
# re-download of the separately-pinned artifact; a missing file is a no-op. An explicit
|
| 72 |
+
# `override_path` (str or iterable of paths) takes precedence, used as-is; default resolves the
|
| 73 |
+
# OVERRIDES names against the PACKAGE dir.
|
| 74 |
+
if override_path is not None: # explicit: a single path (str/PathLike) or an iterable
|
| 75 |
+
ov_files = ([os.fspath(override_path)] if isinstance(override_path, (str, os.PathLike))
|
| 76 |
+
else [os.fspath(p) for p in override_path])
|
| 77 |
+
else:
|
| 78 |
+
ov_files = [os.path.join(os.path.dirname(__file__), n) for n in OVERRIDES]
|
| 79 |
+
for ov in ov_files:
|
| 80 |
+
try:
|
| 81 |
+
with open(ov, encoding="utf-8") as fh:
|
| 82 |
+
table.update(json.load(fh))
|
| 83 |
+
except FileNotFoundError:
|
| 84 |
+
pass
|
| 85 |
+
return cls(table)
|
| 86 |
+
|
| 87 |
+
def __len__(self) -> int:
|
| 88 |
+
return len(self._table)
|
| 89 |
+
|
| 90 |
+
# --- key builders: mirror the wordlist builder's flatten step EXACTLY -----------------------
|
| 91 |
+
@staticmethod
|
| 92 |
+
def _nominal_key(pos: str, lemma: str, cell: dict) -> str:
|
| 93 |
+
# _cellkey_nominal(cell) = tuple(sorted(cell.items())); flatten = pos|lemma|"+".join(str(x))
|
| 94 |
+
ck = tuple(sorted(cell.items()))
|
| 95 |
+
return f"{pos}|{lemma}|" + "+".join(str(x) for x in ck)
|
| 96 |
+
|
| 97 |
+
def lookup_nominal(self, upos: str, lemma: str, feats: dict) -> str | None:
|
| 98 |
+
"""NOUN/ADJ exception form for `lemma` under UD `feats`, or None. Defaults (Px='None',
|
| 99 |
+
Degree='Pos', …) match the realizer/builder grid. PROPN is not shipped ⇒ always None. NUM and PRON
|
| 100 |
+
are lookup-only ALIASES of NOUN: simplex numerals and common pronouns are stored once under the NOUN
|
| 101 |
+
key, so a clause/parser token tagged NUM/PRON hits that same paradigm (no duplicate NUM|/PRON|
|
| 102 |
+
data)."""
|
| 103 |
+
if upos in ("NOUN", "NUM", "PRON"):
|
| 104 |
+
key_pos = "NOUN"
|
| 105 |
+
cell = {"Case": feats.get("Case", "Nom"), "Number": feats.get("Number", "Sing"),
|
| 106 |
+
"Px": feats.get("Px", "None")}
|
| 107 |
+
elif upos == "ADJ":
|
| 108 |
+
key_pos = "ADJ"
|
| 109 |
+
cell = {"Case": feats.get("Case", "Nom"), "Degree": feats.get("Degree", "Pos"),
|
| 110 |
+
"Number": feats.get("Number", "Sing")}
|
| 111 |
+
else:
|
| 112 |
+
return None
|
| 113 |
+
return self._table.get(self._nominal_key(key_pos, lemma, cell))
|
| 114 |
+
|
| 115 |
+
def lookup_finite(self, lemma: str, cell: dict) -> str | None:
|
| 116 |
+
"""Finite-verb exception form for `lemma` at `cell` ({Voice,Mood,Tense?,Person}), or None."""
|
| 117 |
+
from . import finiteverb
|
| 118 |
+
ck = finiteverb.cell_key(cell) # (Voice, Mood, [Tense], Person), Tense omitted when None
|
| 119 |
+
return self._table.get(f"VERB_FIN|{lemma}|" + "+".join(str(x) for x in ck))
|
| 120 |
+
|
| 121 |
+
def lookup_nonfin(self, lemma: str, form: str, case: str = "Nom",
|
| 122 |
+
number: str = "Sing") -> str | None:
|
| 123 |
+
"""Non-finite-verb exception form (participle/infinitive) for `lemma`, or None."""
|
| 124 |
+
return self._table.get(f"VERB_NONFIN|{lemma}|" + "+".join((form, case, number)))
|
juurio/genlex_overrides.json
ADDED
|
@@ -0,0 +1,244 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"VERB_FIN|ahkeroida|Act+Imprt+Sg2": "ahkeroi",
|
| 3 |
+
"VERB_FIN|ahkeroida|Act+Ind+Prs+ConNeg": "ahkeroi",
|
| 4 |
+
"VERB_FIN|ahkeroida|Act+Ind+Prs+Pl1": "ahkeroimme",
|
| 5 |
+
"VERB_FIN|ahkeroida|Act+Ind+Prs+Pl3": "ahkeroivat",
|
| 6 |
+
"VERB_FIN|ahkeroida|Act+Ind+Prs+Sg2": "ahkeroit",
|
| 7 |
+
"VERB_FIN|ahkeroida|Act+Ind+Prs+Sg3": "ahkeroi",
|
| 8 |
+
"VERB_FIN|aprikoida|Act+Imprt+Sg2": "aprikoi",
|
| 9 |
+
"VERB_FIN|aprikoida|Act+Ind+Prs+ConNeg": "aprikoi",
|
| 10 |
+
"VERB_FIN|aprikoida|Act+Ind+Prs+Pl1": "aprikoimme",
|
| 11 |
+
"VERB_FIN|aprikoida|Act+Ind+Prs+Pl3": "aprikoivat",
|
| 12 |
+
"VERB_FIN|aprikoida|Act+Ind+Prs+Sg2": "aprikoit",
|
| 13 |
+
"VERB_FIN|aprikoida|Act+Ind+Prs+Sg3": "aprikoi",
|
| 14 |
+
"VERB_FIN|aterioida|Act+Imprt+Sg2": "aterioi",
|
| 15 |
+
"VERB_FIN|aterioida|Act+Ind+Prs+ConNeg": "aterioi",
|
| 16 |
+
"VERB_FIN|aterioida|Act+Ind+Prs+Pl1": "aterioimme",
|
| 17 |
+
"VERB_FIN|aterioida|Act+Ind+Prs+Pl3": "aterioivat",
|
| 18 |
+
"VERB_FIN|aterioida|Act+Ind+Prs+Sg2": "aterioit",
|
| 19 |
+
"VERB_FIN|aterioida|Act+Ind+Prs+Sg3": "aterioi",
|
| 20 |
+
"VERB_FIN|emännöidä|Act+Imprt+Sg2": "emännöi",
|
| 21 |
+
"VERB_FIN|emännöidä|Act+Ind+Prs+ConNeg": "emännöi",
|
| 22 |
+
"VERB_FIN|emännöidä|Act+Ind+Prs+Pl3": "emännöivät",
|
| 23 |
+
"VERB_FIN|emännöidä|Act+Ind+Prs+Sg3": "emännöi",
|
| 24 |
+
"VERB_FIN|haluta|Act+Imprt+Sg2": "halua",
|
| 25 |
+
"VERB_FIN|haluta|Act+Ind+Prs+ConNeg": "halua",
|
| 26 |
+
"VERB_FIN|haluta|Act+Ind+Prs+Pl1": "haluamme",
|
| 27 |
+
"VERB_FIN|haluta|Act+Ind+Prs+Pl2": "haluatte",
|
| 28 |
+
"VERB_FIN|haluta|Act+Ind+Prs+Pl3": "haluavat",
|
| 29 |
+
"VERB_FIN|haluta|Act+Ind+Prs+Sg1": "haluan",
|
| 30 |
+
"VERB_FIN|haluta|Act+Ind+Prs+Sg2": "haluat",
|
| 31 |
+
"VERB_FIN|haluta|Act+Ind+Prs+Sg3": "haluaa",
|
| 32 |
+
"VERB_FIN|haravoida|Act+Imprt+Sg2": "haravoi",
|
| 33 |
+
"VERB_FIN|haravoida|Act+Ind+Prs+ConNeg": "haravoi",
|
| 34 |
+
"VERB_FIN|haravoida|Act+Ind+Prs+Pl1": "haravoimme",
|
| 35 |
+
"VERB_FIN|haravoida|Act+Ind+Prs+Pl3": "haravoivat",
|
| 36 |
+
"VERB_FIN|haravoida|Act+Ind+Prs+Sg2": "haravoit",
|
| 37 |
+
"VERB_FIN|haravoida|Act+Ind+Prs+Sg3": "haravoi",
|
| 38 |
+
"VERB_FIN|hekumoida|Act+Imprt+Sg2": "hekumoi",
|
| 39 |
+
"VERB_FIN|hekumoida|Act+Ind+Prs+ConNeg": "hekumoi",
|
| 40 |
+
"VERB_FIN|hekumoida|Act+Ind+Prs+Pl1": "hekumoimme",
|
| 41 |
+
"VERB_FIN|hekumoida|Act+Ind+Prs+Pl3": "hekumoivat",
|
| 42 |
+
"VERB_FIN|hekumoida|Act+Ind+Prs+Sg2": "hekumoit",
|
| 43 |
+
"VERB_FIN|hekumoida|Act+Ind+Prs+Sg3": "hekumoi",
|
| 44 |
+
"VERB_FIN|ikävöidä|Act+Imprt+Sg2": "ikävöi",
|
| 45 |
+
"VERB_FIN|ikävöidä|Act+Ind+Prs+ConNeg": "ikävöi",
|
| 46 |
+
"VERB_FIN|ikävöidä|Act+Ind+Prs+Pl1": "ikävöimme",
|
| 47 |
+
"VERB_FIN|ikävöidä|Act+Ind+Prs+Pl3": "ikävöivät",
|
| 48 |
+
"VERB_FIN|ikävöidä|Act+Ind+Prs+Sg2": "ikävöit",
|
| 49 |
+
"VERB_FIN|ikävöidä|Act+Ind+Prs+Sg3": "ikävöi",
|
| 50 |
+
"VERB_FIN|ilakoida|Act+Imprt+Sg2": "ilakoi",
|
| 51 |
+
"VERB_FIN|ilakoida|Act+Ind+Prs+ConNeg": "ilakoi",
|
| 52 |
+
"VERB_FIN|ilakoida|Act+Ind+Prs+Pl1": "ilakoimme",
|
| 53 |
+
"VERB_FIN|ilakoida|Act+Ind+Prs+Pl3": "ilakoivat",
|
| 54 |
+
"VERB_FIN|ilakoida|Act+Ind+Prs+Sg2": "ilakoit",
|
| 55 |
+
"VERB_FIN|ilakoida|Act+Ind+Prs+Sg3": "ilakoi",
|
| 56 |
+
"VERB_FIN|imuroida|Act+Imprt+Sg2": "imuroi",
|
| 57 |
+
"VERB_FIN|imuroida|Act+Ind+Prs+ConNeg": "imuroi",
|
| 58 |
+
"VERB_FIN|imuroida|Act+Ind+Prs+Pl1": "imuroimme",
|
| 59 |
+
"VERB_FIN|imuroida|Act+Ind+Prs+Pl3": "imuroivat",
|
| 60 |
+
"VERB_FIN|imuroida|Act+Ind+Prs+Sg2": "imuroit",
|
| 61 |
+
"VERB_FIN|imuroida|Act+Ind+Prs+Sg3": "imuroi",
|
| 62 |
+
"VERB_FIN|isännöidä|Act+Imprt+Sg2": "isännöi",
|
| 63 |
+
"VERB_FIN|isännöidä|Act+Ind+Prs+ConNeg": "isännöi",
|
| 64 |
+
"VERB_FIN|isännöidä|Act+Ind+Prs+Pl3": "isännöivät",
|
| 65 |
+
"VERB_FIN|isännöidä|Act+Ind+Prs+Sg3": "isännöi",
|
| 66 |
+
"VERB_FIN|kapaloida|Act+Imprt+Sg2": "kapaloi",
|
| 67 |
+
"VERB_FIN|kapaloida|Act+Ind+Prs+ConNeg": "kapaloi",
|
| 68 |
+
"VERB_FIN|kapaloida|Act+Ind+Prs+Pl1": "kapaloimme",
|
| 69 |
+
"VERB_FIN|kapaloida|Act+Ind+Prs+Pl3": "kapaloivat",
|
| 70 |
+
"VERB_FIN|kapaloida|Act+Ind+Prs+Sg2": "kapaloit",
|
| 71 |
+
"VERB_FIN|kapaloida|Act+Ind+Prs+Sg3": "kapaloi",
|
| 72 |
+
"VERB_FIN|kapinoida|Act+Imprt+Sg2": "kapinoi",
|
| 73 |
+
"VERB_FIN|kapinoida|Act+Ind+Prs+ConNeg": "kapinoi",
|
| 74 |
+
"VERB_FIN|kapinoida|Act+Ind+Prs+Pl1": "kapinoimme",
|
| 75 |
+
"VERB_FIN|kapinoida|Act+Ind+Prs+Pl3": "kapinoivat",
|
| 76 |
+
"VERB_FIN|kapinoida|Act+Ind+Prs+Sg2": "kapinoit",
|
| 77 |
+
"VERB_FIN|kapinoida|Act+Ind+Prs+Sg3": "kapinoi",
|
| 78 |
+
"VERB_FIN|karkeloida|Act+Imprt+Sg2": "karkeloi",
|
| 79 |
+
"VERB_FIN|karkeloida|Act+Ind+Prs+ConNeg": "karkeloi",
|
| 80 |
+
"VERB_FIN|karkeloida|Act+Ind+Prs+Pl1": "karkeloimme",
|
| 81 |
+
"VERB_FIN|karkeloida|Act+Ind+Prs+Pl3": "karkeloivat",
|
| 82 |
+
"VERB_FIN|karkeloida|Act+Ind+Prs+Sg2": "karkeloit",
|
| 83 |
+
"VERB_FIN|karkeloida|Act+Ind+Prs+Sg3": "karkeloi",
|
| 84 |
+
"VERB_FIN|keikaroida|Act+Imprt+Sg2": "keikaroi",
|
| 85 |
+
"VERB_FIN|keikaroida|Act+Ind+Prs+ConNeg": "keikaroi",
|
| 86 |
+
"VERB_FIN|keikaroida|Act+Ind+Prs+Pl1": "keikaroimme",
|
| 87 |
+
"VERB_FIN|keikaroida|Act+Ind+Prs+Pl3": "keikaroivat",
|
| 88 |
+
"VERB_FIN|keikaroida|Act+Ind+Prs+Sg2": "keikaroit",
|
| 89 |
+
"VERB_FIN|keikaroida|Act+Ind+Prs+Sg3": "keikaroi",
|
| 90 |
+
"VERB_FIN|kihelmöidä|Act+Imprt+Sg2": "kihelmöi",
|
| 91 |
+
"VERB_FIN|kihelmöidä|Act+Ind+Prs+ConNeg": "kihelmöi",
|
| 92 |
+
"VERB_FIN|kihelmöidä|Act+Ind+Prs+Pl1": "kihelmöimme",
|
| 93 |
+
"VERB_FIN|kihelmöidä|Act+Ind+Prs+Pl3": "kihelmöivät",
|
| 94 |
+
"VERB_FIN|kihelmöidä|Act+Ind+Prs+Sg2": "kihelmöit",
|
| 95 |
+
"VERB_FIN|kihelmöidä|Act+Ind+Prs+Sg3": "kihelmöi",
|
| 96 |
+
"VERB_FIN|kipinöidä|Act+Imprt+Sg2": "kipinöi",
|
| 97 |
+
"VERB_FIN|kipinöidä|Act+Ind+Prs+ConNeg": "kipinöi",
|
| 98 |
+
"VERB_FIN|kipinöidä|Act+Ind+Prs+Pl3": "kipinöivät",
|
| 99 |
+
"VERB_FIN|kipinöidä|Act+Ind+Prs+Sg3": "kipinöi",
|
| 100 |
+
"VERB_FIN|kipunoida|Act+Imprt+Sg2": "kipunoi",
|
| 101 |
+
"VERB_FIN|kipunoida|Act+Ind+Prs+ConNeg": "kipunoi",
|
| 102 |
+
"VERB_FIN|kipunoida|Act+Ind+Prs+Pl1": "kipunoimme",
|
| 103 |
+
"VERB_FIN|kipunoida|Act+Ind+Prs+Pl3": "kipunoivat",
|
| 104 |
+
"VERB_FIN|kipunoida|Act+Ind+Prs+Sg2": "kipunoit",
|
| 105 |
+
"VERB_FIN|kipunoida|Act+Ind+Prs+Sg3": "kipunoi",
|
| 106 |
+
"VERB_FIN|kuutioida|Act+Imprt+Sg2": "kuutioi",
|
| 107 |
+
"VERB_FIN|kuutioida|Act+Ind+Prs+ConNeg": "kuutioi",
|
| 108 |
+
"VERB_FIN|kuutioida|Act+Ind+Prs+Pl1": "kuutioimme",
|
| 109 |
+
"VERB_FIN|kuutioida|Act+Ind+Prs+Pl3": "kuutioivat",
|
| 110 |
+
"VERB_FIN|kuutioida|Act+Ind+Prs+Sg2": "kuutioit",
|
| 111 |
+
"VERB_FIN|kuutioida|Act+Ind+Prs+Sg3": "kuutioi",
|
| 112 |
+
"VERB_FIN|käpälöidä|Act+Imprt+Sg2": "käpälöi",
|
| 113 |
+
"VERB_FIN|käpälöidä|Act+Ind+Prs+ConNeg": "käpälöi",
|
| 114 |
+
"VERB_FIN|käpälöidä|Act+Ind+Prs+Pl1": "käpälöimme",
|
| 115 |
+
"VERB_FIN|käpälöidä|Act+Ind+Prs+Pl3": "käpälöivät",
|
| 116 |
+
"VERB_FIN|käpälöidä|Act+Ind+Prs+Sg2": "käpälöit",
|
| 117 |
+
"VERB_FIN|käpälöidä|Act+Ind+Prs+Sg3": "käpälöi",
|
| 118 |
+
"VERB_FIN|liikennöidä|Act+Imprt+Sg2": "liikennöi",
|
| 119 |
+
"VERB_FIN|liikennöidä|Act+Ind+Prs+ConNeg": "liikennöi",
|
| 120 |
+
"VERB_FIN|liikennöidä|Act+Ind+Prs+Pl1": "liikennöimme",
|
| 121 |
+
"VERB_FIN|liikennöidä|Act+Ind+Prs+Pl3": "liikennöivät",
|
| 122 |
+
"VERB_FIN|liikennöidä|Act+Ind+Prs+Sg3": "liikennöi",
|
| 123 |
+
"VERB_FIN|luennoida|Act+Imprt+Sg2": "luennoi",
|
| 124 |
+
"VERB_FIN|luennoida|Act+Ind+Prs+ConNeg": "luennoi",
|
| 125 |
+
"VERB_FIN|luennoida|Act+Ind+Prs+Pl1": "luennoimme",
|
| 126 |
+
"VERB_FIN|luennoida|Act+Ind+Prs+Pl3": "luennoivat",
|
| 127 |
+
"VERB_FIN|luennoida|Act+Ind+Prs+Sg2": "luennoit",
|
| 128 |
+
"VERB_FIN|luennoida|Act+Ind+Prs+Sg3": "luennoi",
|
| 129 |
+
"VERB_FIN|mankeloida|Act+Imprt+Sg2": "mankeloi",
|
| 130 |
+
"VERB_FIN|mankeloida|Act+Ind+Prs+ConNeg": "mankeloi",
|
| 131 |
+
"VERB_FIN|mankeloida|Act+Ind+Prs+Pl1": "mankeloimme",
|
| 132 |
+
"VERB_FIN|mankeloida|Act+Ind+Prs+Pl3": "mankeloivat",
|
| 133 |
+
"VERB_FIN|mankeloida|Act+Ind+Prs+Sg2": "mankeloit",
|
| 134 |
+
"VERB_FIN|mankeloida|Act+Ind+Prs+Sg3": "mankeloi",
|
| 135 |
+
"VERB_FIN|mellakoida|Act+Imprt+Sg2": "mellakoi",
|
| 136 |
+
"VERB_FIN|mellakoida|Act+Ind+Prs+ConNeg": "mellakoi",
|
| 137 |
+
"VERB_FIN|mellakoida|Act+Ind+Prs+Pl1": "mellakoimme",
|
| 138 |
+
"VERB_FIN|mellakoida|Act+Ind+Prs+Pl3": "mellakoivat",
|
| 139 |
+
"VERB_FIN|mellakoida|Act+Ind+Prs+Sg2": "mellakoit",
|
| 140 |
+
"VERB_FIN|mellakoida|Act+Ind+Prs+Sg3": "mellakoi",
|
| 141 |
+
"VERB_FIN|mennä|Act+Imprt+Sg2": "mene",
|
| 142 |
+
"VERB_FIN|mennä|Act+Ind+Prs+Pl1": "menemme",
|
| 143 |
+
"VERB_FIN|mennä|Act+Ind+Prs+Pl2": "menette",
|
| 144 |
+
"VERB_FIN|mennä|Act+Ind+Prs+Sg1": "menen",
|
| 145 |
+
"VERB_FIN|mennä|Act+Ind+Prs+Sg2": "menet",
|
| 146 |
+
"VERB_FIN|metelöidä|Act+Imprt+Sg2": "metelöi",
|
| 147 |
+
"VERB_FIN|metelöidä|Act+Ind+Prs+ConNeg": "metelöi",
|
| 148 |
+
"VERB_FIN|metelöidä|Act+Ind+Prs+Pl1": "metelöimme",
|
| 149 |
+
"VERB_FIN|metelöidä|Act+Ind+Prs+Pl3": "metelöivät",
|
| 150 |
+
"VERB_FIN|metelöidä|Act+Ind+Prs+Sg2": "metelöit",
|
| 151 |
+
"VERB_FIN|metelöidä|Act+Ind+Prs+Sg3": "metelöi",
|
| 152 |
+
"VERB_FIN|pakinoida|Act+Imprt+Sg2": "pakinoi",
|
| 153 |
+
"VERB_FIN|pakinoida|Act+Ind+Prs+ConNeg": "pakinoi",
|
| 154 |
+
"VERB_FIN|pakinoida|Act+Ind+Prs+Pl3": "pakinoivat",
|
| 155 |
+
"VERB_FIN|pakinoida|Act+Ind+Prs+Sg2": "pakinoit",
|
| 156 |
+
"VERB_FIN|pakinoida|Act+Ind+Prs+Sg3": "pakinoi",
|
| 157 |
+
"VERB_FIN|panna|Act+Imprt+Sg2": "pane",
|
| 158 |
+
"VERB_FIN|panna|Act+Ind+Prs+Sg2": "panet",
|
| 159 |
+
"VERB_FIN|patikoida|Act+Imprt+Sg2": "patikoi",
|
| 160 |
+
"VERB_FIN|patikoida|Act+Ind+Prs+ConNeg": "patikoi",
|
| 161 |
+
"VERB_FIN|patikoida|Act+Ind+Prs+Pl1": "patikoimme",
|
| 162 |
+
"VERB_FIN|patikoida|Act+Ind+Prs+Pl3": "patikoivat",
|
| 163 |
+
"VERB_FIN|patikoida|Act+Ind+Prs+Sg2": "patikoit",
|
| 164 |
+
"VERB_FIN|patikoida|Act+Ind+Prs+Sg3": "patikoi",
|
| 165 |
+
"VERB_FIN|pokkuroida|Act+Imprt+Sg2": "pokkuroi",
|
| 166 |
+
"VERB_FIN|pokkuroida|Act+Ind+Prs+ConNeg": "pokkuroi",
|
| 167 |
+
"VERB_FIN|pokkuroida|Act+Ind+Prs+Pl1": "pokkuroimme",
|
| 168 |
+
"VERB_FIN|pokkuroida|Act+Ind+Prs+Pl3": "pokkuroivat",
|
| 169 |
+
"VERB_FIN|pokkuroida|Act+Ind+Prs+Sg2": "pokkuroit",
|
| 170 |
+
"VERB_FIN|pokkuroida|Act+Ind+Prs+Sg3": "pokkuroi",
|
| 171 |
+
"VERB_FIN|pullikoida|Act+Imprt+Sg2": "pullikoi",
|
| 172 |
+
"VERB_FIN|pullikoida|Act+Ind+Prs+ConNeg": "pullikoi",
|
| 173 |
+
"VERB_FIN|pullikoida|Act+Ind+Prs+Pl1": "pullikoimme",
|
| 174 |
+
"VERB_FIN|pullikoida|Act+Ind+Prs+Pl3": "pullikoivat",
|
| 175 |
+
"VERB_FIN|pullikoida|Act+Ind+Prs+Sg2": "pullikoit",
|
| 176 |
+
"VERB_FIN|pullikoida|Act+Ind+Prs+Sg3": "pullikoi",
|
| 177 |
+
"VERB_FIN|raueta|Act+Imprt+Sg2": "raukea",
|
| 178 |
+
"VERB_FIN|raueta|Act+Ind+Prs+ConNeg": "raukea",
|
| 179 |
+
"VERB_FIN|raueta|Act+Ind+Prs+Pl1": "raukeamme",
|
| 180 |
+
"VERB_FIN|raueta|Act+Ind+Prs+Pl3": "raukeavat",
|
| 181 |
+
"VERB_FIN|raueta|Act+Ind+Prs+Sg1": "raukean",
|
| 182 |
+
"VERB_FIN|raueta|Act+Ind+Prs+Sg2": "raukeat",
|
| 183 |
+
"VERB_FIN|raueta|Act+Ind+Prs+Sg3": "raukeaa",
|
| 184 |
+
"VERB_FIN|rettelöidä|Act+Imprt+Sg2": "rettelöi",
|
| 185 |
+
"VERB_FIN|rettelöidä|Act+Ind+Prs+ConNeg": "rettelöi",
|
| 186 |
+
"VERB_FIN|rettelöidä|Act+Ind+Prs+Pl1": "rettelöimme",
|
| 187 |
+
"VERB_FIN|rettelöidä|Act+Ind+Prs+Pl3": "rettelöivät",
|
| 188 |
+
"VERB_FIN|rettelöidä|Act+Ind+Prs+Sg2": "rettelöit",
|
| 189 |
+
"VERB_FIN|rettelöidä|Act+Ind+Prs+Sg3": "rettelöi",
|
| 190 |
+
"VERB_FIN|rähinöidä|Act+Imprt+Sg2": "rähinöi",
|
| 191 |
+
"VERB_FIN|rähinöidä|Act+Ind+Prs+ConNeg": "rähinöi",
|
| 192 |
+
"VERB_FIN|rähinöidä|Act+Ind+Prs+Pl1": "rähinöimme",
|
| 193 |
+
"VERB_FIN|rähinöidä|Act+Ind+Prs+Pl3": "rähinöivät",
|
| 194 |
+
"VERB_FIN|rähinöidä|Act+Ind+Prs+Sg3": "rähinöi",
|
| 195 |
+
"VERB_FIN|seppelöidä|Act+Imprt+Sg2": "seppelöi",
|
| 196 |
+
"VERB_FIN|seppelöidä|Act+Ind+Prs+ConNeg": "seppelöi",
|
| 197 |
+
"VERB_FIN|seppelöidä|Act+Ind+Prs+Pl1": "seppelöimme",
|
| 198 |
+
"VERB_FIN|seppelöidä|Act+Ind+Prs+Pl3": "seppelöivät",
|
| 199 |
+
"VERB_FIN|seppelöidä|Act+Ind+Prs+Sg3": "seppelöi",
|
| 200 |
+
"VERB_FIN|sukuloida|Act+Imprt+Sg2": "sukuloi",
|
| 201 |
+
"VERB_FIN|sukuloida|Act+Ind+Prs+ConNeg": "sukuloi",
|
| 202 |
+
"VERB_FIN|sukuloida|Act+Ind+Prs+Pl1": "sukuloimme",
|
| 203 |
+
"VERB_FIN|sukuloida|Act+Ind+Prs+Pl3": "sukuloivat",
|
| 204 |
+
"VERB_FIN|sukuloida|Act+Ind+Prs+Sg2": "sukuloit",
|
| 205 |
+
"VERB_FIN|sukuloida|Act+Ind+Prs+Sg3": "sukuloi",
|
| 206 |
+
"VERB_FIN|säikkyä|Act+Imprt+Sg2": "säiky",
|
| 207 |
+
"VERB_FIN|säikkyä|Act+Ind+Prs+ConNeg": "säiky",
|
| 208 |
+
"VERB_FIN|säikkyä|Act+Ind+Prs+Pl1": "säikymme",
|
| 209 |
+
"VERB_FIN|säikkyä|Act+Ind+Prs+Pl2": "säikytte",
|
| 210 |
+
"VERB_FIN|säikkyä|Act+Ind+Prs+Pl3": "säikkyvät",
|
| 211 |
+
"VERB_FIN|säikkyä|Act+Ind+Prs+Sg1": "säikyn",
|
| 212 |
+
"VERB_FIN|säikkyä|Act+Ind+Prs+Sg2": "säikyt",
|
| 213 |
+
"VERB_FIN|säikkyä|Act+Ind+Prs+Sg3": "säikkyy",
|
| 214 |
+
"VERB_FIN|tuikkaa|Act+Ind+Prs+Pl3": "tuikkavat",
|
| 215 |
+
"VERB_FIN|tulla|Act+Imprt+Sg2": "tule",
|
| 216 |
+
"VERB_FIN|tulla|Act+Ind+Prs+Pl1": "tulemme",
|
| 217 |
+
"VERB_FIN|tulla|Act+Ind+Prs+Pl2": "tulette",
|
| 218 |
+
"VERB_FIN|tulla|Act+Ind+Prs+Sg1": "tulen",
|
| 219 |
+
"VERB_FIN|tulla|Act+Ind+Prs+Sg2": "tulet",
|
| 220 |
+
"VERB_FIN|tupakoida|Act+Imprt+Sg2": "tupakoi",
|
| 221 |
+
"VERB_FIN|tupakoida|Act+Ind+Prs+ConNeg": "tupakoi",
|
| 222 |
+
"VERB_FIN|tupakoida|Act+Ind+Prs+Pl1": "tupakoimme",
|
| 223 |
+
"VERB_FIN|tupakoida|Act+Ind+Prs+Pl3": "tupakoivat",
|
| 224 |
+
"VERB_FIN|tupakoida|Act+Ind+Prs+Sg2": "tupakoit",
|
| 225 |
+
"VERB_FIN|tupakoida|Act+Ind+Prs+Sg3": "tupakoi",
|
| 226 |
+
"VERB_FIN|urakoida|Act+Imprt+Sg2": "urakoi",
|
| 227 |
+
"VERB_FIN|urakoida|Act+Ind+Prs+ConNeg": "urakoi",
|
| 228 |
+
"VERB_FIN|urakoida|Act+Ind+Prs+Pl1": "urakoimme",
|
| 229 |
+
"VERB_FIN|urakoida|Act+Ind+Prs+Pl3": "urakoivat",
|
| 230 |
+
"VERB_FIN|urakoida|Act+Ind+Prs+Sg2": "urakoit",
|
| 231 |
+
"VERB_FIN|urakoida|Act+Ind+Prs+Sg3": "urakoi",
|
| 232 |
+
"VERB_FIN|vihannoida|Act+Imprt+Sg2": "vihannoi",
|
| 233 |
+
"VERB_FIN|vihannoida|Act+Ind+Prs+ConNeg": "vihannoi",
|
| 234 |
+
"VERB_FIN|vihannoida|Act+Ind+Prs+Pl1": "vihannoimme",
|
| 235 |
+
"VERB_FIN|vihannoida|Act+Ind+Prs+Pl3": "vihannoivat",
|
| 236 |
+
"VERB_FIN|vihannoida|Act+Ind+Prs+Sg2": "vihannoit",
|
| 237 |
+
"VERB_FIN|vihannoida|Act+Ind+Prs+Sg3": "vihannoi",
|
| 238 |
+
"VERB_FIN|viheriöidä|Act+Imprt+Sg2": "viheriöi",
|
| 239 |
+
"VERB_FIN|viheriöidä|Act+Ind+Prs+ConNeg": "viheriöi",
|
| 240 |
+
"VERB_FIN|viheriöidä|Act+Ind+Prs+Pl1": "viheriöimme",
|
| 241 |
+
"VERB_FIN|viheriöidä|Act+Ind+Prs+Pl3": "viheriöivät",
|
| 242 |
+
"VERB_FIN|viheriöidä|Act+Ind+Prs+Sg2": "viheriöit",
|
| 243 |
+
"VERB_FIN|viheriöidä|Act+Ind+Prs+Sg3": "viheriöi"
|
| 244 |
+
}
|
juurio/genlex_overrides_noun.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
juurio/morphforms.py
ADDED
|
@@ -0,0 +1,59 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Shared rule-engine + realizer helpers.
|
| 2 |
+
|
| 3 |
+
Pure-Python utilities used across the transform engine (`algebra`) and the OOV realizers: locating a
|
| 4 |
+
clause's main finite verb, joining tokens back into a surface string, capitalizing the result, and the
|
| 5 |
+
nonce-wug initial-consonant swap. No world knowledge and no external morphology backend — just
|
| 6 |
+
grammar-mechanical string/parse helpers.
|
| 7 |
+
"""
|
| 8 |
+
|
| 9 |
+
from __future__ import annotations
|
| 10 |
+
|
| 11 |
+
CONSONANTS = list("kptsvnmlrhj")
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
def _main_verb(sent):
|
| 15 |
+
"""Return (verb_token, is_copula). The finite Act/Ind verb to operate on: the root if
|
| 16 |
+
it is one, else the clause's copula."""
|
| 17 |
+
root = next((t for t in sent if t["head"] == 0), None)
|
| 18 |
+
if root is None:
|
| 19 |
+
return None, False
|
| 20 |
+
def finite_act_ind(t):
|
| 21 |
+
f = t["feats"]
|
| 22 |
+
return (t["upos"] in ("VERB", "AUX") and f.get("VerbForm") == "Fin"
|
| 23 |
+
and f.get("Voice", "Act") == "Act" and f.get("Mood", "Ind") == "Ind"
|
| 24 |
+
and f.get("Tense") in ("Pres", "Past"))
|
| 25 |
+
if finite_act_ind(root):
|
| 26 |
+
return root, False
|
| 27 |
+
cop = next((t for t in sent if t["head"] == root["id"] and t["deprel"] == "cop"
|
| 28 |
+
and finite_act_ind(t)), None)
|
| 29 |
+
if cop:
|
| 30 |
+
return cop, True
|
| 31 |
+
return None, False
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
def _join(words):
|
| 35 |
+
s = " ".join(words)
|
| 36 |
+
for p in (" .", " ,", " ?", " !", " :", " ;"):
|
| 37 |
+
s = s.replace(p, p.strip())
|
| 38 |
+
for sup in ("²", "³", "¹"): # Stanza splits `km²`->`km ²`; gold keeps it joined
|
| 39 |
+
s = s.replace(" " + sup, sup)
|
| 40 |
+
return s
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def target_str(r):
|
| 44 |
+
"""Transformed sentence, first letter capitalized (an inserted leading word — e.g. `ei` — may now
|
| 45 |
+
lead)."""
|
| 46 |
+
s = _join(r[1])
|
| 47 |
+
return s[:1].upper() + s[1:] if s else s
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
# ---------------------------- nonce wug ----------------------------
|
| 51 |
+
def swap_initial(word, repl):
|
| 52 |
+
"""Swap the word's first consonant (or prepend one if vowel-initial) — a stable,
|
| 53 |
+
paradigm-wide edit, since Finnish gradation never touches the word-initial
|
| 54 |
+
consonant. Applied identically to every form of the same verb."""
|
| 55 |
+
if not word:
|
| 56 |
+
return word
|
| 57 |
+
if word[0].lower() in CONSONANTS:
|
| 58 |
+
return repl + word[1:]
|
| 59 |
+
return repl + word
|
juurio/parser/__init__.py
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
"""Compact Finnish UD parser."""
|
juurio/parser/case_augment.py
ADDED
|
@@ -0,0 +1,465 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Targeted rare-case morphological augmentation.
|
| 2 |
+
|
| 3 |
+
Behaviour:
|
| 4 |
+
- Re-realize a single nominal head (NOUN/PROPN/ADJ) in a sentence
|
| 5 |
+
into a target Case; walk amod/det dependents and re-realize them
|
| 6 |
+
in agreement.
|
| 7 |
+
- Any failure (head doesn't realize, dependent doesn't realize,
|
| 8 |
+
Voikko rejects an output form) → skip the example entirely.
|
| 9 |
+
- Token count preserved by construction (single-token-in, single-
|
| 10 |
+
token-out per re-realized position).
|
| 11 |
+
- Used to grow rare-Case coverage where the BCE Feats head
|
| 12 |
+
under-fires on rarer cases (Tra/Abl/Ins/Ade/All).
|
| 13 |
+
|
| 14 |
+
This module does NOT modify the model. It produces AugSentence
|
| 15 |
+
objects (list of token dicts with updated form/feats); a downstream
|
| 16 |
+
loader emits the same collator batch keys but flips per-example loss
|
| 17 |
+
masks so only the Feats loss contributes to the gradient.
|
| 18 |
+
|
| 19 |
+
Eligibility:
|
| 20 |
+
- Heads: NOUN, PROPN, ADJ (with Case feature). NOT PRON, NOT NUM.
|
| 21 |
+
- Dependents re-realized: amod, det only. NOT nummod (deferred).
|
| 22 |
+
- Excluded from re-realization neighbourhoods:
|
| 23 |
+
flat, fixed, compound, appos, conj, punct — any nominal head
|
| 24 |
+
with one of these in its subtree is skipped.
|
| 25 |
+
"""
|
| 26 |
+
from __future__ import annotations
|
| 27 |
+
|
| 28 |
+
import collections
|
| 29 |
+
import functools
|
| 30 |
+
import json
|
| 31 |
+
import os
|
| 32 |
+
import random
|
| 33 |
+
from dataclasses import dataclass, field
|
| 34 |
+
from typing import Iterable, Iterator
|
| 35 |
+
|
| 36 |
+
from .silver import SilverSentence, SilverToken
|
| 37 |
+
|
| 38 |
+
ELIGIBLE_HEAD_UPOS = ("NOUN", "PROPN", "ADJ")
|
| 39 |
+
AGREEMENT_DEPRELS = ("amod", "det") # nummod deferred
|
| 40 |
+
# Dependents are only re-realized if their UPOS is also in this allow-list.
|
| 41 |
+
# PRON/DET fall back to lemma in the current realizer, producing feat↔form
|
| 42 |
+
# mismatches like `samalla→sama`, `missään→mikään`. Add DET later behind a
|
| 43 |
+
# tested PRON/DET adapter.
|
| 44 |
+
ELIGIBLE_DEP_UPOS = ("ADJ", "NOUN")
|
| 45 |
+
EXCLUDED_DEPRELS_IN_SUBTREE = ("flat", "fixed", "compound", "appos", "conj")
|
| 46 |
+
|
| 47 |
+
# Target cases where the Feats head tends to under-fire:
|
| 48 |
+
PRIORITY_RARE_CASES = ("Tra", "Abl", "Ins", "Ade", "All", "Ess", "Abe", "Ill", "Ine")
|
| 49 |
+
ANCHOR_COMMON_CASES = ("Nom", "Gen", "Par")
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
@dataclass
|
| 53 |
+
class AugSentence:
|
| 54 |
+
"""Augmented gold sentence. Same shape as SilverSentence; carries
|
| 55 |
+
the metadata a downstream collator needs to flip loss masks.
|
| 56 |
+
|
| 57 |
+
`feats_changed_token_ids` is for diagnostic only — we keep the
|
| 58 |
+
Feats loss as per-sentence-active, not per-token.
|
| 59 |
+
"""
|
| 60 |
+
sentence: SilverSentence
|
| 61 |
+
target_case: str
|
| 62 |
+
target_number: str
|
| 63 |
+
head_token_id: int
|
| 64 |
+
dependents_changed: list[int] = field(default_factory=list)
|
| 65 |
+
feats_changed_token_ids: list[int] = field(default_factory=list)
|
| 66 |
+
# Per-sentence loss-mask flags:
|
| 67 |
+
# feats_mask=True (contribute), all others=False (no parser/upos/
|
| 68 |
+
# lemma/compound supervision on augmented examples).
|
| 69 |
+
feats_loss: bool = True
|
| 70 |
+
upos_loss: bool = False
|
| 71 |
+
lemma_loss: bool = False
|
| 72 |
+
parser_loss: bool = False
|
| 73 |
+
compound_loss: bool = False
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
def _parse_feats(s: str) -> dict[str, str]:
|
| 77 |
+
if not s or s == "_":
|
| 78 |
+
return {}
|
| 79 |
+
return dict(kv.split("=", 1) for kv in s.split("|") if "=" in kv)
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
def _format_feats(d: dict[str, str]) -> str:
|
| 83 |
+
if not d:
|
| 84 |
+
return "_"
|
| 85 |
+
return "|".join(f"{k}={v}" for k, v in sorted(d.items()))
|
| 86 |
+
|
| 87 |
+
|
| 88 |
+
def _children(sent: list, parent_id: int) -> list:
|
| 89 |
+
return [t for t in sent if t.head == parent_id]
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
def _subtree_token_ids(sent: list, root_id: int) -> set[int]:
|
| 93 |
+
out = {root_id}
|
| 94 |
+
stack = [root_id]
|
| 95 |
+
while stack:
|
| 96 |
+
x = stack.pop()
|
| 97 |
+
for c in _children(sent, x):
|
| 98 |
+
if c.id not in out:
|
| 99 |
+
out.add(c.id)
|
| 100 |
+
stack.append(c.id)
|
| 101 |
+
return out
|
| 102 |
+
|
| 103 |
+
|
| 104 |
+
def is_eligible_head(token, sent: list) -> bool:
|
| 105 |
+
"""Eligibility: head must be NOUN/PROPN/ADJ with Case, and its
|
| 106 |
+
subtree must not contain flat/fixed/compound/appos/conj/punct
|
| 107 |
+
deprels."""
|
| 108 |
+
if token.upos not in ELIGIBLE_HEAD_UPOS:
|
| 109 |
+
return False
|
| 110 |
+
feats = _parse_feats(token.feats)
|
| 111 |
+
if "Case" not in feats:
|
| 112 |
+
return False
|
| 113 |
+
# Subtree check: walk the dependents of this token and reject if
|
| 114 |
+
# any are in the excluded set.
|
| 115 |
+
for cid in _subtree_token_ids(sent, token.id):
|
| 116 |
+
if cid == token.id:
|
| 117 |
+
continue
|
| 118 |
+
c = next((t for t in sent if t.id == cid), None)
|
| 119 |
+
if c is None:
|
| 120 |
+
continue
|
| 121 |
+
if c.deprel in EXCLUDED_DEPRELS_IN_SUBTREE:
|
| 122 |
+
return False
|
| 123 |
+
return True
|
| 124 |
+
|
| 125 |
+
|
| 126 |
+
def case_shift_one(
|
| 127 |
+
sent: SilverSentence,
|
| 128 |
+
head_token_id: int,
|
| 129 |
+
target_case: str,
|
| 130 |
+
*,
|
| 131 |
+
target_number: str | None = None,
|
| 132 |
+
voikko_check: bool = True,
|
| 133 |
+
) -> AugSentence | None:
|
| 134 |
+
"""Re-realize one nominal head + its amod/det agreement modifiers
|
| 135 |
+
in target_case. Return AugSentence or None (silent skip).
|
| 136 |
+
|
| 137 |
+
Args:
|
| 138 |
+
sent: gold SilverSentence (will not be mutated; a
|
| 139 |
+
new AugSentence with a new SilverSentence is
|
| 140 |
+
returned)
|
| 141 |
+
head_token_id: the id (1-indexed) of the head NOUN/ADJ/PROPN
|
| 142 |
+
target_case: one of the productive Finnish cases.
|
| 143 |
+
target_number: if None, keep the head's current Number.
|
| 144 |
+
voikko_check: if True (default), every re-realized form is
|
| 145 |
+
validated via uralicApi.analyze; any failure
|
| 146 |
+
→ return None.
|
| 147 |
+
|
| 148 |
+
Returns:
|
| 149 |
+
AugSentence with feats_loss=True and all other masks False, or
|
| 150 |
+
None if eligibility/realizer/Voikko anything failed.
|
| 151 |
+
"""
|
| 152 |
+
# Locate head token
|
| 153 |
+
head = next((t for t in sent.tokens if t.id == head_token_id), None)
|
| 154 |
+
if head is None or not is_eligible_head(head, sent.tokens):
|
| 155 |
+
return None
|
| 156 |
+
head_feats = _parse_feats(head.feats)
|
| 157 |
+
if target_number is None:
|
| 158 |
+
target_number = head_feats.get("Number", "Sing")
|
| 159 |
+
|
| 160 |
+
# Re-realize the head
|
| 161 |
+
head_form_new = _cached_realize_np_form(
|
| 162 |
+
head.lemma, head.upos, target_case, target_number,
|
| 163 |
+
)
|
| 164 |
+
if not head_form_new:
|
| 165 |
+
return None
|
| 166 |
+
# Preserve the original token's capitalization convention (the
|
| 167 |
+
# realizer always returns lower-case forms because lemmas are
|
| 168 |
+
# lower-case; without this, sentence-initial tokens become
|
| 169 |
+
# mid-sentence-looking).
|
| 170 |
+
head_form_new = _match_case_pattern(head.form, head_form_new)
|
| 171 |
+
if voikko_check and not _voikko_has_target(
|
| 172 |
+
head_form_new, target_case, target_number,
|
| 173 |
+
):
|
| 174 |
+
return None
|
| 175 |
+
|
| 176 |
+
# Walk amod/det dependents and re-realize each in (target_case, target_number).
|
| 177 |
+
# Dep UPOS must be in ELIGIBLE_DEP_UPOS — PRON/DET would lemma-fallback
|
| 178 |
+
# in the current realizer, so we leave them surface-untouched (their Case
|
| 179 |
+
# feat stays original, which is the safe semantic: feat-form consistency).
|
| 180 |
+
deps_to_rewrite = [t for t in sent.tokens
|
| 181 |
+
if t.head == head_token_id
|
| 182 |
+
and t.deprel in AGREEMENT_DEPRELS
|
| 183 |
+
and t.upos in ELIGIBLE_DEP_UPOS]
|
| 184 |
+
dep_new_forms: dict[int, str] = {}
|
| 185 |
+
for d in deps_to_rewrite:
|
| 186 |
+
new = _cached_realize_np_form(
|
| 187 |
+
d.lemma, d.upos, target_case, target_number,
|
| 188 |
+
)
|
| 189 |
+
if not new:
|
| 190 |
+
return None
|
| 191 |
+
new = _match_case_pattern(d.form, new)
|
| 192 |
+
if voikko_check and not _voikko_has_target(
|
| 193 |
+
new, target_case, target_number,
|
| 194 |
+
):
|
| 195 |
+
return None
|
| 196 |
+
dep_new_forms[d.id] = new
|
| 197 |
+
|
| 198 |
+
# Build the new token list with updated forms + Feats on head + deps
|
| 199 |
+
new_tokens = []
|
| 200 |
+
feats_changed_ids = []
|
| 201 |
+
changed_dep_ids = []
|
| 202 |
+
for t in sent.tokens:
|
| 203 |
+
if t.id == head.id:
|
| 204 |
+
new_feats = dict(head_feats)
|
| 205 |
+
new_feats["Case"] = target_case
|
| 206 |
+
new_feats["Number"] = target_number
|
| 207 |
+
new_tokens.append(SilverToken(
|
| 208 |
+
form=head_form_new,
|
| 209 |
+
lemma=t.lemma,
|
| 210 |
+
upos=t.upos,
|
| 211 |
+
feats=_format_feats(new_feats),
|
| 212 |
+
head=t.head, deprel=t.deprel,
|
| 213 |
+
))
|
| 214 |
+
feats_changed_ids.append(t.id)
|
| 215 |
+
elif t.id in dep_new_forms:
|
| 216 |
+
d_feats = _parse_feats(t.feats)
|
| 217 |
+
d_feats["Case"] = target_case
|
| 218 |
+
d_feats["Number"] = target_number
|
| 219 |
+
new_tokens.append(SilverToken(
|
| 220 |
+
form=dep_new_forms[t.id],
|
| 221 |
+
lemma=t.lemma,
|
| 222 |
+
upos=t.upos,
|
| 223 |
+
feats=_format_feats(d_feats),
|
| 224 |
+
head=t.head, deprel=t.deprel,
|
| 225 |
+
))
|
| 226 |
+
feats_changed_ids.append(t.id)
|
| 227 |
+
changed_dep_ids.append(t.id)
|
| 228 |
+
else:
|
| 229 |
+
new_tokens.append(t)
|
| 230 |
+
|
| 231 |
+
new_text = " ".join(tok.form for tok in new_tokens)
|
| 232 |
+
new_sent = SilverSentence(text=new_text, tokens=new_tokens)
|
| 233 |
+
return AugSentence(
|
| 234 |
+
sentence=new_sent,
|
| 235 |
+
target_case=target_case,
|
| 236 |
+
target_number=target_number,
|
| 237 |
+
head_token_id=head.id,
|
| 238 |
+
dependents_changed=changed_dep_ids,
|
| 239 |
+
feats_changed_token_ids=feats_changed_ids,
|
| 240 |
+
)
|
| 241 |
+
|
| 242 |
+
|
| 243 |
+
def _match_case_pattern(orig: str, new: str) -> str:
|
| 244 |
+
"""Re-apply the original token's leading-capitalization convention
|
| 245 |
+
to the realizer's lower-case output. The realizer is lemma-driven
|
| 246 |
+
(lemmas are lower-case), so a sentence-initial token like `Jäällä`
|
| 247 |
+
comes back as `jääksi` without this fix.
|
| 248 |
+
|
| 249 |
+
- all-uppercase original (and length > 1) → all-upper new
|
| 250 |
+
- title-case original (leading upper, rest mixed) → leading-upper new
|
| 251 |
+
- everything else → leave as-is
|
| 252 |
+
"""
|
| 253 |
+
if not orig or not new:
|
| 254 |
+
return new
|
| 255 |
+
if len(orig) > 1 and orig.isupper():
|
| 256 |
+
return new.upper()
|
| 257 |
+
if orig[:1].isupper():
|
| 258 |
+
return new[:1].upper() + new[1:]
|
| 259 |
+
return new
|
| 260 |
+
|
| 261 |
+
|
| 262 |
+
# Mapping from UD Case names to the Giella FST tag suffix used by
|
| 263 |
+
# uralicNLP's Voikko-fin transducer.
|
| 264 |
+
_UD_CASE_TO_FST_TAG = {
|
| 265 |
+
"Nom": "Nom", "Gen": "Gen", "Par": "Par",
|
| 266 |
+
"Ess": "Ess", "Tra": "Tra", "Ine": "Ine",
|
| 267 |
+
"Ela": "Ela", "Ill": "Ill", "Ade": "Ade",
|
| 268 |
+
"Abl": "Abl", "All": "All", "Abe": "Abe",
|
| 269 |
+
"Ins": "Ins", "Com": "Com",
|
| 270 |
+
}
|
| 271 |
+
# UD Number → FST tag (uralicApi uses Sg / Pl).
|
| 272 |
+
_UD_NUMBER_TO_FST_TAG = {"Sing": "Sg", "Plur": "Pl"}
|
| 273 |
+
|
| 274 |
+
|
| 275 |
+
@functools.lru_cache(maxsize=200_000)
|
| 276 |
+
def _voikko_analyses(form: str) -> tuple[str, ...]:
|
| 277 |
+
"""Cached raw analyses for a form. Returns a tuple of analysis
|
| 278 |
+
strings like `vaihe+N+Sg+Ine`. Empty tuple means Voikko cannot
|
| 279 |
+
analyze the form at all (= the form is not Finnish).
|
| 280 |
+
|
| 281 |
+
uralicApi.analyze returns a list of `(analysis_string, weight)`
|
| 282 |
+
tuples — extract the string component only.
|
| 283 |
+
"""
|
| 284 |
+
try:
|
| 285 |
+
from uralicNLP import uralicApi
|
| 286 |
+
raw = uralicApi.analyze(form, "fin")
|
| 287 |
+
return tuple(r[0] if isinstance(r, tuple) else r for r in raw)
|
| 288 |
+
except Exception:
|
| 289 |
+
return ()
|
| 290 |
+
|
| 291 |
+
|
| 292 |
+
def _voikko_ok(form: str) -> bool:
|
| 293 |
+
"""Loose check: any Voikko analysis exists."""
|
| 294 |
+
return bool(_voikko_analyses(form))
|
| 295 |
+
|
| 296 |
+
|
| 297 |
+
def _voikko_has_target(form: str, case: str, number: str) -> bool:
|
| 298 |
+
"""Strict check: require at least one Voikko analysis whose tag
|
| 299 |
+
string contains both the target Case tag (e.g. `+Tra`) and the
|
| 300 |
+
target Number tag (`+Sg` or `+Pl`). Catches the `sama → sama`
|
| 301 |
+
style lemma fallback where Voikko accepts the surface form (it IS
|
| 302 |
+
valid Finnish) but only as Nom-Sing, not as the requested target.
|
| 303 |
+
"""
|
| 304 |
+
fst_case = _UD_CASE_TO_FST_TAG.get(case)
|
| 305 |
+
fst_num = _UD_NUMBER_TO_FST_TAG.get(number)
|
| 306 |
+
if fst_case is None or fst_num is None:
|
| 307 |
+
# Defensive: unknown case/number → fall back to loose check.
|
| 308 |
+
return _voikko_ok(form)
|
| 309 |
+
case_tag = "+" + fst_case
|
| 310 |
+
num_tag = "+" + fst_num
|
| 311 |
+
for a in _voikko_analyses(form):
|
| 312 |
+
if case_tag in a and num_tag in a:
|
| 313 |
+
return True
|
| 314 |
+
return False
|
| 315 |
+
|
| 316 |
+
|
| 317 |
+
@functools.lru_cache(maxsize=200_000)
|
| 318 |
+
def _cached_realize_np_form(lemma: str, upos: str, case: str,
|
| 319 |
+
number: str) -> str | None:
|
| 320 |
+
"""Cached wrapper around juurio.algebra.realize_np_form. Cuts
|
| 321 |
+
walltime from minutes to seconds on a re-run."""
|
| 322 |
+
from ..algebra import realize_np_form
|
| 323 |
+
return realize_np_form(lemma, upos=upos, case=case, number=number)
|
| 324 |
+
|
| 325 |
+
|
| 326 |
+
def make_augmentation_corpus(
|
| 327 |
+
sents: list,
|
| 328 |
+
*,
|
| 329 |
+
target_cases: tuple = PRIORITY_RARE_CASES,
|
| 330 |
+
anchor_cases: tuple = ANCHOR_COMMON_CASES,
|
| 331 |
+
per_case_cap: int = 3000,
|
| 332 |
+
anchor_mix_pct: float = 0.10,
|
| 333 |
+
total_cap: int | None = 30000,
|
| 334 |
+
rng: random.Random | None = None,
|
| 335 |
+
voikko_check: bool = True,
|
| 336 |
+
) -> list[AugSentence]:
|
| 337 |
+
"""Stratified sampling. For each target_case, walk sents, collect
|
| 338 |
+
up to per_case_cap successful AugSentences. Then add anchor_mix_pct
|
| 339 |
+
of total_cap from anchor_cases (common cases) so the model doesn't
|
| 340 |
+
memorize 'augmented = rare'.
|
| 341 |
+
|
| 342 |
+
Returns the full list shuffled (so DataLoader doesn't see
|
| 343 |
+
target-case clumps).
|
| 344 |
+
"""
|
| 345 |
+
rng = rng or random.Random(0)
|
| 346 |
+
out: list[AugSentence] = []
|
| 347 |
+
by_case: dict[str, list[AugSentence]] = collections.defaultdict(list)
|
| 348 |
+
|
| 349 |
+
# Pre-shuffle the sentences to vary head selection across cases
|
| 350 |
+
sent_order = list(sents)
|
| 351 |
+
rng.shuffle(sent_order)
|
| 352 |
+
|
| 353 |
+
for case in list(target_cases) + list(anchor_cases):
|
| 354 |
+
for sent in sent_order:
|
| 355 |
+
if len(by_case[case]) >= per_case_cap:
|
| 356 |
+
break
|
| 357 |
+
for tok in sent.tokens:
|
| 358 |
+
if tok.upos not in ELIGIBLE_HEAD_UPOS:
|
| 359 |
+
continue
|
| 360 |
+
aug = case_shift_one(sent, tok.id, case,
|
| 361 |
+
voikko_check=voikko_check)
|
| 362 |
+
if aug is not None:
|
| 363 |
+
by_case[case].append(aug)
|
| 364 |
+
break # one head per sentence
|
| 365 |
+
if len(by_case[case]) >= per_case_cap:
|
| 366 |
+
break
|
| 367 |
+
|
| 368 |
+
# Build the stratified corpus respecting total_cap + anchor_mix_pct
|
| 369 |
+
rare_cap = int((1.0 - anchor_mix_pct) * total_cap) if total_cap else None
|
| 370 |
+
anchor_cap = (total_cap - rare_cap) if total_cap else None
|
| 371 |
+
rare_pool = []
|
| 372 |
+
for case in target_cases:
|
| 373 |
+
rare_pool.extend(by_case[case])
|
| 374 |
+
rng.shuffle(rare_pool)
|
| 375 |
+
if rare_cap is not None and len(rare_pool) > rare_cap:
|
| 376 |
+
rare_pool = rare_pool[:rare_cap]
|
| 377 |
+
anchor_pool = []
|
| 378 |
+
for case in anchor_cases:
|
| 379 |
+
anchor_pool.extend(by_case[case])
|
| 380 |
+
rng.shuffle(anchor_pool)
|
| 381 |
+
if anchor_cap is not None and len(anchor_pool) > anchor_cap:
|
| 382 |
+
anchor_pool = anchor_pool[:anchor_cap]
|
| 383 |
+
|
| 384 |
+
final = rare_pool + anchor_pool
|
| 385 |
+
rng.shuffle(final)
|
| 386 |
+
return final
|
| 387 |
+
|
| 388 |
+
|
| 389 |
+
# ----------------------------- JSONL I/O ------------------------------------
|
| 390 |
+
# Persist the augmented corpus to disk so the training driver doesn't rebuild
|
| 391 |
+
# it on every restart (~10 min Voikko walk otherwise).
|
| 392 |
+
|
| 393 |
+
def _aug_to_dict(aug: "AugSentence") -> dict:
|
| 394 |
+
"""Serialise an AugSentence to a JSON-able dict. SilverToken is a flat
|
| 395 |
+
dataclass; AugSentence wraps SilverSentence + per-sentence mask
|
| 396 |
+
flags + provenance metadata."""
|
| 397 |
+
s = aug.sentence
|
| 398 |
+
return {
|
| 399 |
+
"sentence": {
|
| 400 |
+
"text": s.text,
|
| 401 |
+
"tokens": [
|
| 402 |
+
{"form": t.form, "lemma": t.lemma, "upos": t.upos,
|
| 403 |
+
"feats": t.feats, "head": t.head, "deprel": t.deprel}
|
| 404 |
+
for t in s.tokens
|
| 405 |
+
],
|
| 406 |
+
},
|
| 407 |
+
"target_case": aug.target_case,
|
| 408 |
+
"target_number": aug.target_number,
|
| 409 |
+
"head_token_id": aug.head_token_id,
|
| 410 |
+
"dependents_changed": list(aug.dependents_changed),
|
| 411 |
+
"feats_changed_token_ids": list(aug.feats_changed_token_ids),
|
| 412 |
+
"feats_loss": aug.feats_loss,
|
| 413 |
+
"upos_loss": aug.upos_loss,
|
| 414 |
+
"lemma_loss": aug.lemma_loss,
|
| 415 |
+
"parser_loss": aug.parser_loss,
|
| 416 |
+
"compound_loss": aug.compound_loss,
|
| 417 |
+
}
|
| 418 |
+
|
| 419 |
+
|
| 420 |
+
def _aug_from_dict(obj: dict) -> "AugSentence":
|
| 421 |
+
s = obj["sentence"]
|
| 422 |
+
sent = SilverSentence(
|
| 423 |
+
text=s["text"],
|
| 424 |
+
tokens=[SilverToken(**t) for t in s["tokens"]],
|
| 425 |
+
)
|
| 426 |
+
return AugSentence(
|
| 427 |
+
sentence=sent,
|
| 428 |
+
target_case=obj["target_case"],
|
| 429 |
+
target_number=obj["target_number"],
|
| 430 |
+
head_token_id=obj["head_token_id"],
|
| 431 |
+
dependents_changed=list(obj.get("dependents_changed", [])),
|
| 432 |
+
feats_changed_token_ids=list(obj.get("feats_changed_token_ids", [])),
|
| 433 |
+
feats_loss=obj.get("feats_loss", True),
|
| 434 |
+
upos_loss=obj.get("upos_loss", False),
|
| 435 |
+
lemma_loss=obj.get("lemma_loss", False),
|
| 436 |
+
parser_loss=obj.get("parser_loss", False),
|
| 437 |
+
compound_loss=obj.get("compound_loss", False),
|
| 438 |
+
)
|
| 439 |
+
|
| 440 |
+
|
| 441 |
+
def write_aug_jsonl(records: Iterable["AugSentence"], path: str,
|
| 442 |
+
mode: str = "w") -> int:
|
| 443 |
+
"""Write `records` as one JSON object per line. Returns the count.
|
| 444 |
+
`mode='w'` truncates; `mode='a'` appends (resume-safe)."""
|
| 445 |
+
if mode not in ("w", "a"):
|
| 446 |
+
raise ValueError(f"mode must be 'w' or 'a', got {mode!r}")
|
| 447 |
+
os.makedirs(os.path.dirname(os.path.abspath(path)) or ".",
|
| 448 |
+
exist_ok=True)
|
| 449 |
+
n = 0
|
| 450 |
+
with open(path, mode, encoding="utf-8") as fh:
|
| 451 |
+
for a in records:
|
| 452 |
+
fh.write(json.dumps(_aug_to_dict(a), ensure_ascii=False))
|
| 453 |
+
fh.write("\n")
|
| 454 |
+
n += 1
|
| 455 |
+
return n
|
| 456 |
+
|
| 457 |
+
|
| 458 |
+
def read_aug_jsonl(path: str) -> Iterator["AugSentence"]:
|
| 459 |
+
"""Stream AugSentence records from a JSONL file."""
|
| 460 |
+
with open(path, encoding="utf-8") as fh:
|
| 461 |
+
for ln in fh:
|
| 462 |
+
ln = ln.strip()
|
| 463 |
+
if not ln:
|
| 464 |
+
continue
|
| 465 |
+
yield _aug_from_dict(json.loads(ln))
|
juurio/parser/char_vocab.py
ADDED
|
@@ -0,0 +1,179 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Character vocabulary for the compound-lemma heads.
|
| 2 |
+
|
| 3 |
+
Mirrors `FstTagVocab` from `juurio.parser.fst_cache` in API shape and
|
| 4 |
+
implementation patterns, but operates on individual characters instead of
|
| 5 |
+
morphological tags.
|
| 6 |
+
|
| 7 |
+
Invariants:
|
| 8 |
+
- id 0 is always reserved for `<PAD>` (no-signal embedding).
|
| 9 |
+
- id 1 is always reserved for `<UNK>` (out-of-vocabulary character).
|
| 10 |
+
- Real characters start at id 2.
|
| 11 |
+
- The order of the character list is stable after `fit`; the SHA-256 hash
|
| 12 |
+
pins this order so a stale char vocab can't silently corrupt training.
|
| 13 |
+
|
| 14 |
+
Raises:
|
| 15 |
+
ValueError: if `chars` has fewer than 2 elements, or if `chars[0]` is not
|
| 16 |
+
`"<PAD>"` or `chars[1]` is not `"<UNK>"`. (Note: this was previously an
|
| 17 |
+
`AssertionError`, which is silently elided under ``python -O``.)
|
| 18 |
+
"""
|
| 19 |
+
from __future__ import annotations
|
| 20 |
+
|
| 21 |
+
import hashlib
|
| 22 |
+
import json
|
| 23 |
+
from collections import Counter
|
| 24 |
+
from pathlib import Path
|
| 25 |
+
from typing import Iterable
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
PAD_ID = 0
|
| 29 |
+
UNK_ID = 1
|
| 30 |
+
PAD_CHAR = "<PAD>"
|
| 31 |
+
UNK_CHAR = "<UNK>"
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
class CharVocab:
|
| 35 |
+
"""Maps individual character strings to integer ids.
|
| 36 |
+
|
| 37 |
+
Reserved ids:
|
| 38 |
+
PAD_ID = 0 — padding / no-signal position.
|
| 39 |
+
UNK_ID = 1 — out-of-vocabulary character.
|
| 40 |
+
|
| 41 |
+
Real characters start at id 2. The constructor raises `ValueError` if
|
| 42 |
+
`chars` has fewer than 2 entries, or if `chars[0]` is not `"<PAD>"` or
|
| 43 |
+
`chars[1]` is not `"<UNK>"`, to prevent accidental construction with wrong
|
| 44 |
+
ordering (which would silently feed the wrong embeddings).
|
| 45 |
+
"""
|
| 46 |
+
|
| 47 |
+
PAD_ID: int = PAD_ID
|
| 48 |
+
UNK_ID: int = UNK_ID
|
| 49 |
+
PAD_CHAR: str = PAD_CHAR
|
| 50 |
+
UNK_CHAR: str = UNK_CHAR
|
| 51 |
+
|
| 52 |
+
def __init__(self, chars: list[str]):
|
| 53 |
+
"""Construct a CharVocab from an ordered character list.
|
| 54 |
+
|
| 55 |
+
`chars[0]` must be `"<PAD>"` and `chars[1]` must be `"<UNK>"`;
|
| 56 |
+
real characters follow from index 2 onward. Violating this ordering
|
| 57 |
+
raises `ValueError`.
|
| 58 |
+
|
| 59 |
+
Raises:
|
| 60 |
+
ValueError: if `chars` has fewer than 2 elements, or if
|
| 61 |
+
`chars[0]` is not `"<PAD>"` or `chars[1]` is not `"<UNK>"`.
|
| 62 |
+
"""
|
| 63 |
+
if len(chars) < 2:
|
| 64 |
+
raise ValueError(
|
| 65 |
+
"chars must have at least 2 entries; "
|
| 66 |
+
"PAD at index 0 and UNK at index 1"
|
| 67 |
+
)
|
| 68 |
+
if chars[0] != PAD_CHAR or chars[1] != UNK_CHAR:
|
| 69 |
+
raise ValueError(
|
| 70 |
+
f"chars[0] must be '{PAD_CHAR}' and chars[1] must be '{UNK_CHAR}'; "
|
| 71 |
+
f"got chars[0]={chars[0]!r}, chars[1]={chars[1]!r}"
|
| 72 |
+
)
|
| 73 |
+
self._chars = list(chars)
|
| 74 |
+
self._char_to_id = {c: i for i, c in enumerate(chars)}
|
| 75 |
+
|
| 76 |
+
@property
|
| 77 |
+
def size(self) -> int:
|
| 78 |
+
"""Total vocabulary size including PAD and UNK."""
|
| 79 |
+
return len(self._chars)
|
| 80 |
+
|
| 81 |
+
@classmethod
|
| 82 |
+
def fit(
|
| 83 |
+
cls,
|
| 84 |
+
forms: Iterable[str],
|
| 85 |
+
*,
|
| 86 |
+
min_count: int = 1,
|
| 87 |
+
max_chars: int | None = None,
|
| 88 |
+
) -> "CharVocab":
|
| 89 |
+
"""Fit a CharVocab from an iterable of surface forms.
|
| 90 |
+
|
| 91 |
+
Collects all individual characters from every form, counts their
|
| 92 |
+
frequencies, drops chars seen fewer than `min_count` times, and
|
| 93 |
+
optionally truncates to the top `max_chars` by frequency. Returns a
|
| 94 |
+
new `CharVocab` with PAD and UNK prepended.
|
| 95 |
+
|
| 96 |
+
Args:
|
| 97 |
+
forms: Iterable of string forms (e.g. word surface forms).
|
| 98 |
+
min_count: Minimum character frequency to keep. Characters seen
|
| 99 |
+
fewer times are mapped to UNK at encoding time.
|
| 100 |
+
max_chars: If given, keep only the top-`max_chars` most frequent
|
| 101 |
+
characters (after the `min_count` filter).
|
| 102 |
+
|
| 103 |
+
Returns:
|
| 104 |
+
A new `CharVocab` instance.
|
| 105 |
+
"""
|
| 106 |
+
c: Counter[str] = Counter()
|
| 107 |
+
for form in forms:
|
| 108 |
+
for ch in form:
|
| 109 |
+
c[ch] += 1
|
| 110 |
+
kept = [ch for ch, n in c.most_common() if n >= min_count]
|
| 111 |
+
if max_chars is not None:
|
| 112 |
+
kept = kept[:max_chars]
|
| 113 |
+
return cls([PAD_CHAR, UNK_CHAR] + kept)
|
| 114 |
+
|
| 115 |
+
def encode_char(self, c: str) -> int:
|
| 116 |
+
"""Return the integer id for character `c`, or UNK_ID if unseen."""
|
| 117 |
+
return self._char_to_id.get(c, UNK_ID)
|
| 118 |
+
|
| 119 |
+
def encode_form(self, form: str, max_len: int) -> tuple[list[int], list[bool]]:
|
| 120 |
+
"""Encode a surface form as a fixed-length sequence of char ids.
|
| 121 |
+
|
| 122 |
+
Forms shorter than `max_len` are padded on the right with PAD_ID;
|
| 123 |
+
forms longer than `max_len` are silently truncated to `max_len` chars.
|
| 124 |
+
The boolean mask is True for real character positions and False for
|
| 125 |
+
padding positions.
|
| 126 |
+
|
| 127 |
+
Args:
|
| 128 |
+
form: The surface form to encode.
|
| 129 |
+
max_len: Target sequence length (output length is always max_len).
|
| 130 |
+
|
| 131 |
+
Returns:
|
| 132 |
+
(char_ids, char_mask) — both have length `max_len`.
|
| 133 |
+
"""
|
| 134 |
+
# Truncate to max_len
|
| 135 |
+
truncated = form[:max_len]
|
| 136 |
+
real_len = len(truncated)
|
| 137 |
+
|
| 138 |
+
char_ids: list[int] = [self._char_to_id.get(c, UNK_ID) for c in truncated]
|
| 139 |
+
char_mask: list[bool] = [True] * real_len
|
| 140 |
+
|
| 141 |
+
# Pad to max_len
|
| 142 |
+
pad_count = max_len - real_len
|
| 143 |
+
char_ids.extend([PAD_ID] * pad_count)
|
| 144 |
+
char_mask.extend([False] * pad_count)
|
| 145 |
+
|
| 146 |
+
return char_ids, char_mask
|
| 147 |
+
|
| 148 |
+
def hash(self) -> str:
|
| 149 |
+
"""SHA-256 digest of the ordered character list (first 16 hex chars).
|
| 150 |
+
|
| 151 |
+
This short digest is used to pin the char vocab to a trained checkpoint
|
| 152 |
+
so that a stale or differently-fitted vocab can't silently feed wrong
|
| 153 |
+
char embeddings to a loaded model. The hash is order-sensitive: two
|
| 154 |
+
vocabs with identical character sets but different orderings produce
|
| 155 |
+
different hashes.
|
| 156 |
+
"""
|
| 157 |
+
h = hashlib.sha256()
|
| 158 |
+
for c in self._chars:
|
| 159 |
+
h.update(c.encode("utf-8"))
|
| 160 |
+
h.update(b"\x00")
|
| 161 |
+
return h.hexdigest()[:16]
|
| 162 |
+
|
| 163 |
+
def save(self, path: str) -> None:
|
| 164 |
+
"""Serialise the vocab to JSON at `path`.
|
| 165 |
+
|
| 166 |
+
Creates parent directories if they do not exist. The format is::
|
| 167 |
+
|
| 168 |
+
{"chars": ["<PAD>", "<UNK>", "a", "b", ...]}
|
| 169 |
+
"""
|
| 170 |
+
Path(path).parent.mkdir(parents=True, exist_ok=True)
|
| 171 |
+
with open(path, "w") as fh:
|
| 172 |
+
json.dump({"chars": self._chars}, fh, ensure_ascii=False)
|
| 173 |
+
|
| 174 |
+
@classmethod
|
| 175 |
+
def load(cls, path: str) -> "CharVocab":
|
| 176 |
+
"""Load a previously saved CharVocab from JSON."""
|
| 177 |
+
with open(path) as fh:
|
| 178 |
+
blob = json.load(fh)
|
| 179 |
+
return cls(blob["chars"])
|
juurio/parser/collator.py
ADDED
|
@@ -0,0 +1,466 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Batch collator for the tiny Finnish UD parser.
|
| 2 |
+
|
| 3 |
+
Turns a list of `SilverSentence` records into model-ready batched tensors. Solves three problems:
|
| 4 |
+
|
| 5 |
+
1. **Word ↔ subword mismatch.** Each `Word` (UPOS / lemma / head / deprel) may map to several
|
| 6 |
+
BPE subwords. We place labels only at the *first subword* of each Word; non-first subwords
|
| 7 |
+
get `<pad>` (id=0) so the cross-entropy loss with `ignore_index=0` skips them. A
|
| 8 |
+
`first_subw` bool mask says where the real labels are.
|
| 9 |
+
2. **UD head index → subword position.** UD `head=k` (1-based Word index, 0=root) becomes a
|
| 10 |
+
subword position: `head=0` → 0 (the ROOT token at `ids[:, 0]`); `head=k>0` → the first-subword
|
| 11 |
+
position of Word k.
|
| 12 |
+
3. **Length capping.** Sentences whose subword length (incl. ROOT) exceeds `max_len` are
|
| 13 |
+
dropped — silently filtered before tensor construction.
|
| 14 |
+
|
| 15 |
+
The model's `forward(ids, mask)` consumes `ids` and `mask`. The other keys are training-loss
|
| 16 |
+
targets. `first_subw` is exposed so the loss can scatter / gather per-Word predictions cleanly,
|
| 17 |
+
and so calibration metrics can be computed per Word.
|
| 18 |
+
|
| 19 |
+
Calling convention:
|
| 20 |
+
|
| 21 |
+
col = SilverCollator(tokenizer=tok, vocab=vocab, max_len=256)
|
| 22 |
+
batch = col(list_of_silver_sentences)
|
| 23 |
+
out = model(batch["ids"], batch["mask"])
|
| 24 |
+
# loss heads compare out[head] against batch[head], gated by batch["first_subw"].
|
| 25 |
+
"""
|
| 26 |
+
from __future__ import annotations
|
| 27 |
+
|
| 28 |
+
import torch
|
| 29 |
+
|
| 30 |
+
from .char_vocab import CharVocab
|
| 31 |
+
from .compound import CompoundTargets
|
| 32 |
+
from .fst_cache import FstCache, FstTagVocab
|
| 33 |
+
from .label_vocab import LabelVocab
|
| 34 |
+
from .model import PAD_ID, ROOT_ID
|
| 35 |
+
from .silver import SilverSentence
|
| 36 |
+
from .tokenizer import BPETokenizer
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
class SilverCollator:
|
| 40 |
+
def __init__(self, tokenizer: BPETokenizer, vocab: LabelVocab, max_len: int = 256,
|
| 41 |
+
*, fst_cache: FstCache | None = None,
|
| 42 |
+
fst_tag_vocab: FstTagVocab | None = None,
|
| 43 |
+
fst_max_tags: int = 16,
|
| 44 |
+
char_vocab: CharVocab | None = None,
|
| 45 |
+
max_form_len: int = 64,
|
| 46 |
+
compound_enabled: bool = False):
|
| 47 |
+
"""Construct a SilverCollator.
|
| 48 |
+
|
| 49 |
+
Args:
|
| 50 |
+
tokenizer: BPE tokenizer.
|
| 51 |
+
vocab: LabelVocab for all task heads.
|
| 52 |
+
max_len: Maximum subword sequence length; longer sentences are dropped.
|
| 53 |
+
fst_cache: Optional FST analysis cache.
|
| 54 |
+
fst_tag_vocab: Optional FST tag vocabulary.
|
| 55 |
+
fst_max_tags: Maximum number of morphtags per token in the FST channel.
|
| 56 |
+
char_vocab: Character vocabulary for compound-lemma heads.
|
| 57 |
+
Required when `compound_enabled=True`.
|
| 58 |
+
max_form_len: Maximum character sequence length per token. Longer forms
|
| 59 |
+
are truncated silently.
|
| 60 |
+
compound_enabled: When True, emit the six compound-target tensors in every
|
| 61 |
+
batch dict (`form_chars`, `form_char_mask`, `compound_K`,
|
| 62 |
+
`compound_split_pos`, `compound_tail_id`, `compound_noisy`).
|
| 63 |
+
When False (default), no new keys and no new allocations.
|
| 64 |
+
|
| 65 |
+
Raises:
|
| 66 |
+
ValueError: if `compound_enabled=True` but `char_vocab=None`.
|
| 67 |
+
"""
|
| 68 |
+
if compound_enabled and char_vocab is None:
|
| 69 |
+
raise ValueError(
|
| 70 |
+
"compound_enabled=True requires a char_vocab; got char_vocab=None. "
|
| 71 |
+
"Fit a CharVocab on your training forms and pass it here."
|
| 72 |
+
)
|
| 73 |
+
self.tokenizer = tokenizer
|
| 74 |
+
self.vocab = vocab
|
| 75 |
+
self.max_len = max_len
|
| 76 |
+
# FST channel — None means the batch omits `fst_tags`, which the model
|
| 77 |
+
# treats as a no-op (same forward as no-FST). When both cache and vocab are
|
| 78 |
+
# provided, every batch carries a (B, T, fst_max_tags) int tensor with the
|
| 79 |
+
# per-first-subw morphtag ids; padding positions are PAD_ID = 0.
|
| 80 |
+
self.fst_cache = fst_cache
|
| 81 |
+
self.fst_tag_vocab = fst_tag_vocab
|
| 82 |
+
self.fst_max_tags = fst_max_tags
|
| 83 |
+
# Compound channel
|
| 84 |
+
self.char_vocab = char_vocab
|
| 85 |
+
self.max_form_len = max_form_len
|
| 86 |
+
self.compound_enabled = compound_enabled
|
| 87 |
+
|
| 88 |
+
def __call__(self, sentences: list) -> dict[str, torch.Tensor]:
|
| 89 |
+
"""Encode a batch.
|
| 90 |
+
|
| 91 |
+
Inputs may be plain SilverSentence (gold; all four loss masks
|
| 92 |
+
will be True) or AugSentence (carries its own per-sentence
|
| 93 |
+
(feats/upos/lemma/parser)_loss flags). When ANY input is an
|
| 94 |
+
AugSentence, the returned batch dict carries feats_mask /
|
| 95 |
+
upos_mask / parser_mask / lemma_mask, each a (B,) bool tensor
|
| 96 |
+
aligned with kept_indices (NOT the pre-collation position).
|
| 97 |
+
When all inputs are SilverSentence, no mask keys are emitted.
|
| 98 |
+
"""
|
| 99 |
+
# AugSentence wraps a SilverSentence and carries
|
| 100 |
+
# per-sentence mask flags. Detect mixed-or-aug input cheaply.
|
| 101 |
+
from .case_augment import AugSentence # local: cold path
|
| 102 |
+
has_aug = any(isinstance(s, AugSentence) for s in sentences)
|
| 103 |
+
|
| 104 |
+
def _unwrap(s):
|
| 105 |
+
return s.sentence if isinstance(s, AugSentence) else s
|
| 106 |
+
|
| 107 |
+
per_sentence = []
|
| 108 |
+
kept_indices: list[int] = []
|
| 109 |
+
kept_mask_flags: list[tuple[bool, bool, bool, bool]] = []
|
| 110 |
+
kept_has_teacher: list[bool] = []
|
| 111 |
+
for i, s in enumerate(sentences):
|
| 112 |
+
inner = _unwrap(s)
|
| 113 |
+
enc = self._encode_one(inner)
|
| 114 |
+
if enc is None:
|
| 115 |
+
continue
|
| 116 |
+
per_sentence.append(enc)
|
| 117 |
+
kept_indices.append(i)
|
| 118 |
+
# Per-sentence auxiliary-target eligibility = inner sentence
|
| 119 |
+
# carried teacher_tokens. AugSentence wraps the inner Silver
|
| 120 |
+
# so teacher attaches on the inner just like for plain silver.
|
| 121 |
+
kept_has_teacher.append(getattr(inner, "teacher_tokens", None) is not None)
|
| 122 |
+
if isinstance(s, AugSentence):
|
| 123 |
+
kept_mask_flags.append((
|
| 124 |
+
s.feats_loss, s.upos_loss, s.parser_loss, s.lemma_loss,
|
| 125 |
+
))
|
| 126 |
+
else:
|
| 127 |
+
kept_mask_flags.append((True, True, True, True))
|
| 128 |
+
if not per_sentence:
|
| 129 |
+
return self._empty_batch()
|
| 130 |
+
batch = self._pad_and_stack(per_sentence)
|
| 131 |
+
batch["kept_indices"] = torch.tensor(kept_indices, dtype=torch.long)
|
| 132 |
+
if has_aug:
|
| 133 |
+
flags = torch.tensor(kept_mask_flags, dtype=torch.bool)
|
| 134 |
+
batch["feats_mask"] = flags[:, 0].clone()
|
| 135 |
+
batch["upos_mask"] = flags[:, 1].clone()
|
| 136 |
+
batch["parser_mask"] = flags[:, 2].clone()
|
| 137 |
+
batch["lemma_mask"] = flags[:, 3].clone()
|
| 138 |
+
# Per-sentence auxiliary-target apply mask (kd_apply). Only emit when at
|
| 139 |
+
# least one kept sentence had teacher_tokens (matches the
|
| 140 |
+
# has_teacher branch in _pad_and_stack).
|
| 141 |
+
if any(kept_has_teacher):
|
| 142 |
+
batch["kd_apply"] = torch.tensor(kept_has_teacher, dtype=torch.bool)
|
| 143 |
+
return batch
|
| 144 |
+
|
| 145 |
+
# ----------------------------- per-sentence encoding -------------------------------------
|
| 146 |
+
def _encode_one(self, s: SilverSentence) -> dict[str, list] | None:
|
| 147 |
+
"""Encode one sentence to a dict of python lists (one per output tensor key). Returns
|
| 148 |
+
None if the sentence overflows `max_len`."""
|
| 149 |
+
ids = [ROOT_ID]
|
| 150 |
+
first_subw = [False] # ROOT is real but carries no labels
|
| 151 |
+
word_first_positions: list[int] = []
|
| 152 |
+
|
| 153 |
+
# tokenize each Word's surface form separately so we can record first-subword positions
|
| 154 |
+
for t in s.tokens:
|
| 155 |
+
wpiece = self.tokenizer.encode(t.form)
|
| 156 |
+
if not wpiece:
|
| 157 |
+
# extremely rare: a Word that BPE emits 0 ids for. Treat it as a single <unk>.
|
| 158 |
+
wpiece = [self.tokenizer.unk_id]
|
| 159 |
+
word_first_positions.append(len(ids))
|
| 160 |
+
ids.extend(wpiece)
|
| 161 |
+
first_subw.append(True)
|
| 162 |
+
first_subw.extend([False] * (len(wpiece) - 1))
|
| 163 |
+
|
| 164 |
+
if len(ids) > self.max_len:
|
| 165 |
+
return None
|
| 166 |
+
|
| 167 |
+
T = len(ids)
|
| 168 |
+
upos = [0] * T
|
| 169 |
+
lemma = [0] * T
|
| 170 |
+
rel = [0] * T
|
| 171 |
+
head = [0] * T
|
| 172 |
+
feats = [[0] * self.vocab.n_feat for _ in range(T)]
|
| 173 |
+
|
| 174 |
+
# Per-token Feats mask carried from the
|
| 175 |
+
# cleaned silver corpus. Default True (= include). When
|
| 176 |
+
# any token has the flag set, the collator emits a (B, T)
|
| 177 |
+
# feats_token_mask tensor; otherwise the key is absent.
|
| 178 |
+
feats_token_mask = [True] * T
|
| 179 |
+
any_feats_token_masked = False
|
| 180 |
+
|
| 181 |
+
for k, t in enumerate(s.tokens):
|
| 182 |
+
pos = word_first_positions[k]
|
| 183 |
+
upos[pos] = self.vocab.encode_upos(t.upos)
|
| 184 |
+
lemma[pos] = self.vocab.encode_edit_script(t.form, t.lemma)
|
| 185 |
+
rel[pos] = self.vocab.encode_deprel(t.deprel)
|
| 186 |
+
feats[pos] = self.vocab.encode_feats_multihot(t.feats)
|
| 187 |
+
if getattr(t, "feats_loss_masked", False):
|
| 188 |
+
feats_token_mask[pos] = False
|
| 189 |
+
any_feats_token_masked = True
|
| 190 |
+
# head → subword position. head=0 → ROOT at position 0.
|
| 191 |
+
if t.head == 0:
|
| 192 |
+
head[pos] = 0
|
| 193 |
+
elif 1 <= t.head <= len(s.tokens):
|
| 194 |
+
head[pos] = word_first_positions[t.head - 1]
|
| 195 |
+
else:
|
| 196 |
+
# Malformed head (out of range). Drop the whole sentence rather than
|
| 197 |
+
# silently fold to root — folding fabricates a UD arc the silver never asserted.
|
| 198 |
+
return None
|
| 199 |
+
out = {
|
| 200 |
+
"ids": ids,
|
| 201 |
+
"first_subw": first_subw,
|
| 202 |
+
"upos": upos,
|
| 203 |
+
"feats": feats,
|
| 204 |
+
"lemma": lemma,
|
| 205 |
+
"head": head,
|
| 206 |
+
"rel": rel,
|
| 207 |
+
"T": T,
|
| 208 |
+
}
|
| 209 |
+
if any_feats_token_masked:
|
| 210 |
+
out["feats_token_mask"] = feats_token_mask
|
| 211 |
+
|
| 212 |
+
# Optional per-token auxiliary targets (training only).
|
| 213 |
+
# Emit teacher_* lists ONLY when this sentence carries teacher_tokens;
|
| 214 |
+
# the per-sentence kd_apply mask is set at the __call__ level so the
|
| 215 |
+
# batch-side aggregator knows which sentences contribute.
|
| 216 |
+
teacher_tokens = getattr(s, "teacher_tokens", None)
|
| 217 |
+
if teacher_tokens is not None:
|
| 218 |
+
if len(teacher_tokens) != len(s.tokens):
|
| 219 |
+
raise ValueError(
|
| 220 |
+
f"teacher_tokens length {len(teacher_tokens)} != "
|
| 221 |
+
f"sentence tokens {len(s.tokens)} — cache misalignment"
|
| 222 |
+
)
|
| 223 |
+
teacher_upos = [0] * T
|
| 224 |
+
teacher_lemma_edit = [-100] * T # IGNORE_INDEX outside first-subw
|
| 225 |
+
teacher_rel = [0] * T
|
| 226 |
+
teacher_head = [0] * T
|
| 227 |
+
teacher_feats = [[0] * self.vocab.n_feat for _ in range(T)]
|
| 228 |
+
for k, tt in enumerate(teacher_tokens):
|
| 229 |
+
pos = word_first_positions[k]
|
| 230 |
+
teacher_upos[pos] = self.vocab.encode_upos(tt.upos)
|
| 231 |
+
teacher_lemma_edit[pos] = tt.lemma_edit # may be -100 (OOV)
|
| 232 |
+
teacher_rel[pos] = self.vocab.encode_deprel(tt.deprel)
|
| 233 |
+
teacher_feats[pos] = self.vocab.encode_feats_multihot(tt.feats)
|
| 234 |
+
if tt.head == 0:
|
| 235 |
+
teacher_head[pos] = 0
|
| 236 |
+
elif 1 <= tt.head <= len(s.tokens):
|
| 237 |
+
teacher_head[pos] = word_first_positions[tt.head - 1]
|
| 238 |
+
else:
|
| 239 |
+
# Defensive: Stanza emitted a malformed head. Mark as ROOT
|
| 240 |
+
# to keep the arc-CE finite, but the training-time auxiliary
|
| 241 |
+
# arc loss uses first_subw masking so the gradient is bounded.
|
| 242 |
+
teacher_head[pos] = 0
|
| 243 |
+
out["teacher_upos"] = teacher_upos
|
| 244 |
+
out["teacher_lemma_edit"] = teacher_lemma_edit
|
| 245 |
+
out["teacher_rel"] = teacher_rel
|
| 246 |
+
out["teacher_head"] = teacher_head
|
| 247 |
+
out["teacher_feats"] = teacher_feats
|
| 248 |
+
# Optional auxiliary per-word hidden-state targets (training only).
|
| 249 |
+
# `finbert_hidden`, when present, is a fp16 numpy array of shape
|
| 250 |
+
# `(n_words, hidden_dim)`, ROOT excluded. We forward the raw array
|
| 251 |
+
# + the per-word first-subw positions to `_pad_and_stack`, which
|
| 252 |
+
# scatters rows onto the (B, T, H) batch tensor.
|
| 253 |
+
finbert_hidden = getattr(s, "finbert_hidden", None)
|
| 254 |
+
if finbert_hidden is not None:
|
| 255 |
+
if finbert_hidden.shape[0] != len(s.tokens):
|
| 256 |
+
raise ValueError(
|
| 257 |
+
f"finbert_hidden.shape[0]={finbert_hidden.shape[0]} != "
|
| 258 |
+
f"len(sentence.tokens)={len(s.tokens)} — cache misalignment"
|
| 259 |
+
)
|
| 260 |
+
out["finbert_hidden_word"] = finbert_hidden
|
| 261 |
+
out["finbert_hidden_first_pos"] = list(word_first_positions)
|
| 262 |
+
# FST tags: per first-subw position emit a fixed-length list of tag ids
|
| 263 |
+
# (PAD = 0 elsewhere). Length T × fst_max_tags, allocated per sentence so
|
| 264 |
+
# padding to batch max-T at stack-time stays simple.
|
| 265 |
+
if self.fst_cache is not None and self.fst_tag_vocab is not None:
|
| 266 |
+
fst_tags = [[0] * self.fst_max_tags for _ in range(T)]
|
| 267 |
+
for k, t in enumerate(s.tokens):
|
| 268 |
+
pos = word_first_positions[k]
|
| 269 |
+
fst_tags[pos] = self.fst_cache.form_to_tag_ids(
|
| 270 |
+
t.form, self.fst_tag_vocab, max_tags=self.fst_max_tags)
|
| 271 |
+
out["fst_tags"] = fst_tags
|
| 272 |
+
# Compound channel: emitted only when compound_enabled is True.
|
| 273 |
+
# All non-first-subw positions (incl. ROOT at index 0) are zero/False;
|
| 274 |
+
# the loss uses `first_subw` to gate which positions are supervised.
|
| 275 |
+
if self.compound_enabled:
|
| 276 |
+
mfl = self.max_form_len
|
| 277 |
+
# Allocate per-position containers, all zeroed/Falsy.
|
| 278 |
+
form_chars = [[0] * mfl for _ in range(T)] # (T, mfl) int
|
| 279 |
+
form_char_mask = [[False] * mfl for _ in range(T)] # (T, mfl) bool
|
| 280 |
+
compound_K = [0] * T # (T,) int
|
| 281 |
+
compound_split_pos = [[False] * mfl for _ in range(T)] # (T, mfl) bool
|
| 282 |
+
compound_tail_id = [0] * T # (T,) int
|
| 283 |
+
compound_noisy = [False] * T # (T,) bool
|
| 284 |
+
|
| 285 |
+
for k, tok_item in enumerate(s.tokens):
|
| 286 |
+
pos = word_first_positions[k]
|
| 287 |
+
# Encode the surface form into char ids (truncate/pad to mfl).
|
| 288 |
+
char_ids, char_mask = self.char_vocab.encode_form(tok_item.form, mfl)
|
| 289 |
+
form_chars[pos] = char_ids
|
| 290 |
+
form_char_mask[pos] = char_mask
|
| 291 |
+
# Compute gold compound targets for this (form, lemma) pair.
|
| 292 |
+
targets = CompoundTargets.from_form_lemma(
|
| 293 |
+
tok_item.form, tok_item.lemma, self.vocab.edit_to_id)
|
| 294 |
+
compound_K[pos] = targets["K"]
|
| 295 |
+
compound_tail_id[pos] = targets["tail_script_id"] if targets["K"] > 0 else 0
|
| 296 |
+
compound_noisy[pos] = targets["noisy_label"]
|
| 297 |
+
# Multi-hot split position vector: True at each gold boundary char
|
| 298 |
+
# position in the form (up to mfl; positions beyond mfl are silently
|
| 299 |
+
# dropped since the form is truncated there anyway).
|
| 300 |
+
split_vec = [False] * mfl
|
| 301 |
+
for bp in targets["boundary_positions"]:
|
| 302 |
+
if bp < mfl:
|
| 303 |
+
split_vec[bp] = True
|
| 304 |
+
compound_split_pos[pos] = split_vec
|
| 305 |
+
|
| 306 |
+
out["form_chars"] = form_chars
|
| 307 |
+
out["form_char_mask"] = form_char_mask
|
| 308 |
+
out["compound_K"] = compound_K
|
| 309 |
+
out["compound_split_pos"] = compound_split_pos
|
| 310 |
+
out["compound_tail_id"] = compound_tail_id
|
| 311 |
+
out["compound_noisy"] = compound_noisy
|
| 312 |
+
return out
|
| 313 |
+
|
| 314 |
+
# ----------------------------- pad + stack -----------------------------------------------
|
| 315 |
+
def _pad_and_stack(self, encs: list[dict]) -> dict[str, torch.Tensor]:
|
| 316 |
+
T = max(e["T"] for e in encs)
|
| 317 |
+
B = len(encs)
|
| 318 |
+
ids = torch.full((B, T), PAD_ID, dtype=torch.long)
|
| 319 |
+
mask = torch.zeros(B, T, dtype=torch.bool)
|
| 320 |
+
first_subw = torch.zeros(B, T, dtype=torch.bool)
|
| 321 |
+
upos = torch.zeros(B, T, dtype=torch.long)
|
| 322 |
+
lemma = torch.zeros(B, T, dtype=torch.long)
|
| 323 |
+
rel = torch.zeros(B, T, dtype=torch.long)
|
| 324 |
+
head = torch.zeros(B, T, dtype=torch.long)
|
| 325 |
+
feats = torch.zeros(B, T, self.vocab.n_feat, dtype=torch.float)
|
| 326 |
+
|
| 327 |
+
# FST tags optional — allocate only if any sentence has them.
|
| 328 |
+
has_fst = "fst_tags" in encs[0]
|
| 329 |
+
fst_tags = (torch.zeros(B, T, self.fst_max_tags, dtype=torch.long)
|
| 330 |
+
if has_fst else None)
|
| 331 |
+
|
| 332 |
+
# Per-token Feats mask. Only allocate if
|
| 333 |
+
# ANY encoded sentence carries a feats_token_mask list.
|
| 334 |
+
has_feats_token_mask = any("feats_token_mask" in e for e in encs)
|
| 335 |
+
if has_feats_token_mask:
|
| 336 |
+
# Default True — non-flagged sentences and pad positions
|
| 337 |
+
# both keep "include in Feats loss" (the per-position
|
| 338 |
+
# gating is already done by first_subw downstream).
|
| 339 |
+
feats_token_mask_t = torch.ones(B, T, dtype=torch.bool)
|
| 340 |
+
|
| 341 |
+
# Compound tensors — allocate only when compound_enabled.
|
| 342 |
+
has_compound = "form_chars" in encs[0]
|
| 343 |
+
mfl = self.max_form_len
|
| 344 |
+
if has_compound:
|
| 345 |
+
form_chars = torch.zeros(B, T, mfl, dtype=torch.long)
|
| 346 |
+
form_char_mask = torch.zeros(B, T, mfl, dtype=torch.bool)
|
| 347 |
+
compound_K = torch.zeros(B, T, dtype=torch.long)
|
| 348 |
+
compound_split_pos = torch.zeros(B, T, mfl, dtype=torch.bool)
|
| 349 |
+
compound_tail_id = torch.zeros(B, T, dtype=torch.long)
|
| 350 |
+
compound_noisy = torch.zeros(B, T, dtype=torch.bool)
|
| 351 |
+
|
| 352 |
+
# Optional per-token auxiliary training targets. Allocate when ANY
|
| 353 |
+
# sentence in the batch has teacher_* (a mixed batch gets per-sentence
|
| 354 |
+
# zeros for the rows without them, and the per-sentence kd_apply mask
|
| 355 |
+
# gates the loss).
|
| 356 |
+
has_teacher = any("teacher_upos" in e for e in encs)
|
| 357 |
+
if has_teacher:
|
| 358 |
+
teacher_upos_t = torch.zeros(B, T, dtype=torch.long)
|
| 359 |
+
teacher_lemma_edit_t = torch.full((B, T), -100, dtype=torch.long)
|
| 360 |
+
teacher_rel_t = torch.zeros(B, T, dtype=torch.long)
|
| 361 |
+
teacher_head_t = torch.zeros(B, T, dtype=torch.long)
|
| 362 |
+
teacher_feats_t = torch.zeros(B, T, self.vocab.n_feat, dtype=torch.float)
|
| 363 |
+
|
| 364 |
+
# Hidden-state auxiliary targets.
|
| 365 |
+
# Allocate when ANY sentence in the batch carries
|
| 366 |
+
# `finbert_hidden_word` (cache attachment). Rows for cache-miss
|
| 367 |
+
# sentences stay all-zero with mask all-False so the loss path's
|
| 368 |
+
# mask gating naturally drops them.
|
| 369 |
+
has_finbert_hidden = any("finbert_hidden_word" in e for e in encs)
|
| 370 |
+
if has_finbert_hidden:
|
| 371 |
+
first_with_fb = next(e for e in encs if "finbert_hidden_word" in e)
|
| 372 |
+
fb_dim = int(first_with_fb["finbert_hidden_word"].shape[-1])
|
| 373 |
+
teacher_finbert_hidden = torch.zeros(B, T, fb_dim, dtype=torch.float16)
|
| 374 |
+
finbert_hidden_kd_mask = torch.zeros(B, T, dtype=torch.bool)
|
| 375 |
+
|
| 376 |
+
for b, e in enumerate(encs):
|
| 377 |
+
t = e["T"]
|
| 378 |
+
ids[b, :t] = torch.tensor(e["ids"], dtype=torch.long)
|
| 379 |
+
mask[b, :t] = True
|
| 380 |
+
first_subw[b, :t] = torch.tensor(e["first_subw"], dtype=torch.bool)
|
| 381 |
+
upos[b, :t] = torch.tensor(e["upos"], dtype=torch.long)
|
| 382 |
+
lemma[b, :t] = torch.tensor(e["lemma"], dtype=torch.long)
|
| 383 |
+
rel[b, :t] = torch.tensor(e["rel"], dtype=torch.long)
|
| 384 |
+
head[b, :t] = torch.tensor(e["head"], dtype=torch.long)
|
| 385 |
+
feats[b, :t] = torch.tensor(e["feats"], dtype=torch.float)
|
| 386 |
+
if has_fst:
|
| 387 |
+
fst_tags[b, :t] = torch.tensor(e["fst_tags"], dtype=torch.long)
|
| 388 |
+
if has_feats_token_mask and "feats_token_mask" in e:
|
| 389 |
+
feats_token_mask_t[b, :t] = torch.tensor(
|
| 390 |
+
e["feats_token_mask"], dtype=torch.bool)
|
| 391 |
+
if has_compound:
|
| 392 |
+
form_chars[b, :t] = torch.tensor(e["form_chars"], dtype=torch.long)
|
| 393 |
+
form_char_mask[b, :t] = torch.tensor(e["form_char_mask"], dtype=torch.bool)
|
| 394 |
+
compound_K[b, :t] = torch.tensor(e["compound_K"], dtype=torch.long)
|
| 395 |
+
compound_split_pos[b, :t] = torch.tensor(e["compound_split_pos"], dtype=torch.bool)
|
| 396 |
+
compound_tail_id[b, :t] = torch.tensor(e["compound_tail_id"], dtype=torch.long)
|
| 397 |
+
compound_noisy[b, :t] = torch.tensor(e["compound_noisy"], dtype=torch.bool)
|
| 398 |
+
if has_teacher and "teacher_upos" in e:
|
| 399 |
+
teacher_upos_t[b, :t] = torch.tensor(e["teacher_upos"], dtype=torch.long)
|
| 400 |
+
teacher_lemma_edit_t[b, :t] = torch.tensor(e["teacher_lemma_edit"], dtype=torch.long)
|
| 401 |
+
teacher_rel_t[b, :t] = torch.tensor(e["teacher_rel"], dtype=torch.long)
|
| 402 |
+
teacher_head_t[b, :t] = torch.tensor(e["teacher_head"], dtype=torch.long)
|
| 403 |
+
teacher_feats_t[b, :t] = torch.tensor(e["teacher_feats"], dtype=torch.float)
|
| 404 |
+
if has_finbert_hidden and "finbert_hidden_word" in e:
|
| 405 |
+
# `finbert_hidden_word` is fp16 (n_words, H); scatter
|
| 406 |
+
# rows onto the first-subword position of each UD word.
|
| 407 |
+
# ROOT (pos 0) and non-first-subword / pad positions
|
| 408 |
+
# stay zero with mask False — that is how the loss path
|
| 409 |
+
# excludes them.
|
| 410 |
+
rows = torch.from_numpy(e["finbert_hidden_word"]) # fp16
|
| 411 |
+
positions = e["finbert_hidden_first_pos"]
|
| 412 |
+
for k, pos in enumerate(positions):
|
| 413 |
+
teacher_finbert_hidden[b, pos] = rows[k]
|
| 414 |
+
finbert_hidden_kd_mask[b, pos] = True
|
| 415 |
+
|
| 416 |
+
out = {
|
| 417 |
+
"ids": ids,
|
| 418 |
+
"mask": mask,
|
| 419 |
+
"first_subw": first_subw,
|
| 420 |
+
"upos": upos,
|
| 421 |
+
"feats": feats,
|
| 422 |
+
"lemma": lemma,
|
| 423 |
+
"head": head,
|
| 424 |
+
"rel": rel,
|
| 425 |
+
}
|
| 426 |
+
if has_fst:
|
| 427 |
+
out["fst_tags"] = fst_tags
|
| 428 |
+
if has_feats_token_mask:
|
| 429 |
+
out["feats_token_mask"] = feats_token_mask_t
|
| 430 |
+
if has_compound:
|
| 431 |
+
out["form_chars"] = form_chars
|
| 432 |
+
out["form_char_mask"] = form_char_mask
|
| 433 |
+
out["compound_K"] = compound_K
|
| 434 |
+
out["compound_split_pos"] = compound_split_pos
|
| 435 |
+
out["compound_tail_id"] = compound_tail_id
|
| 436 |
+
out["compound_noisy"] = compound_noisy
|
| 437 |
+
if has_teacher:
|
| 438 |
+
out["teacher_upos"] = teacher_upos_t
|
| 439 |
+
out["teacher_lemma_edit"] = teacher_lemma_edit_t
|
| 440 |
+
out["teacher_rel"] = teacher_rel_t
|
| 441 |
+
out["teacher_head"] = teacher_head_t
|
| 442 |
+
out["teacher_feats"] = teacher_feats_t
|
| 443 |
+
if has_finbert_hidden:
|
| 444 |
+
out["teacher_finbert_hidden"] = teacher_finbert_hidden
|
| 445 |
+
out["finbert_hidden_kd_mask"] = finbert_hidden_kd_mask
|
| 446 |
+
return out
|
| 447 |
+
|
| 448 |
+
def _empty_batch(self) -> dict[str, torch.Tensor]:
|
| 449 |
+
z = torch.zeros(0, 0, dtype=torch.long)
|
| 450 |
+
out = {
|
| 451 |
+
"ids": z,
|
| 452 |
+
"mask": torch.zeros(0, 0, dtype=torch.bool),
|
| 453 |
+
"first_subw": torch.zeros(0, 0, dtype=torch.bool),
|
| 454 |
+
"upos": z, "lemma": z, "rel": z, "head": z,
|
| 455 |
+
"feats": torch.zeros(0, 0, self.vocab.n_feat),
|
| 456 |
+
"kept_indices": torch.zeros(0, dtype=torch.long),
|
| 457 |
+
}
|
| 458 |
+
if self.compound_enabled:
|
| 459 |
+
mfl = self.max_form_len
|
| 460 |
+
out["form_chars"] = torch.zeros(0, 0, mfl, dtype=torch.long)
|
| 461 |
+
out["form_char_mask"] = torch.zeros(0, 0, mfl, dtype=torch.bool)
|
| 462 |
+
out["compound_K"] = torch.zeros(0, 0, dtype=torch.long)
|
| 463 |
+
out["compound_split_pos"] = torch.zeros(0, 0, mfl, dtype=torch.bool)
|
| 464 |
+
out["compound_tail_id"] = torch.zeros(0, 0, dtype=torch.long)
|
| 465 |
+
out["compound_noisy"] = torch.zeros(0, 0, dtype=torch.bool)
|
| 466 |
+
return out
|
juurio/parser/compound.py
ADDED
|
@@ -0,0 +1,154 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Compound-lemma supervision targets.
|
| 2 |
+
|
| 3 |
+
The lemma stack uses three heads to handle Finnish compound lemmatization
|
| 4 |
+
(`kiinankaali → kiinan#kaali`, `juhlamekkopaniikki → juhla#mekko#paniikki`).
|
| 5 |
+
Compounds account for a large share of lemma OOVs, so routing them through
|
| 6 |
+
dedicated heads recovers substantial lemma accuracy.
|
| 7 |
+
|
| 8 |
+
This module owns the *gold-target construction* for those new heads:
|
| 9 |
+
|
| 10 |
+
- boundary-count target K ∈ {0, 1, 2, 3+} → CE head
|
| 11 |
+
- per-position split target → per-char-pos BCE head
|
| 12 |
+
- tail-script target → CE head, reuses the existing
|
| 13 |
+
edit-script vocab from
|
| 14 |
+
`LabelVocab.edit_to_id`
|
| 15 |
+
|
| 16 |
+
The collator calls `CompoundTargets.from_form_lemma()` once per gold/silver
|
| 17 |
+
token to assemble the gold dict; the loss module then masks `L_split` and
|
| 18 |
+
`L_tail` on tokens flagged `noisy_label` (stem-changing compound prefixes) —
|
| 19 |
+
supervising the split scorer with noisy positions would poison it.
|
| 20 |
+
|
| 21 |
+
Boundary semantics:
|
| 22 |
+
|
| 23 |
+
For a lemma like `TGV#juna` and form `TGV-junia`, the gold `#` is the
|
| 24 |
+
boundary BEFORE the form's hyphen — position 3. Tail-form is then
|
| 25 |
+
`-junia`. The hyphen-fallback in the decoder strips the leading `-` from
|
| 26 |
+
the tail when looking up its edit-script.
|
| 27 |
+
|
| 28 |
+
Stem-changing prefixes (`lastenromaanit → lapsi#romaani`) are flagged
|
| 29 |
+
`noisy_label=True` — the form's prefix `lasten` is not a substring of
|
| 30 |
+
`lapsi`, so we can't project the gold `#` to a clean form position. Such
|
| 31 |
+
tokens still supervise the count head (gold K=1 is correct) but their
|
| 32 |
+
split target is empty.
|
| 33 |
+
"""
|
| 34 |
+
from __future__ import annotations
|
| 35 |
+
|
| 36 |
+
from dataclasses import dataclass
|
| 37 |
+
from typing import Mapping
|
| 38 |
+
|
| 39 |
+
from .edit_scripts import extract_edit_script
|
| 40 |
+
from .label_vocab import _script_key
|
| 41 |
+
|
| 42 |
+
# Count-head cap. K ∈ {0, 1, 2, 3+}; everything ≥3 collapses to class 3.
|
| 43 |
+
COUNT_MAX = 3
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
@dataclass
|
| 47 |
+
class CompoundTargets:
|
| 48 |
+
"""Gold targets for the three compound heads. `from_form_lemma` is the
|
| 49 |
+
only public constructor — keep target conventions in one place."""
|
| 50 |
+
|
| 51 |
+
@staticmethod
|
| 52 |
+
def from_form_lemma(form: str, lemma: str,
|
| 53 |
+
edit_to_id: Mapping[str, int]) -> dict:
|
| 54 |
+
"""Build the gold-target dict for a single (form, lemma) pair.
|
| 55 |
+
|
| 56 |
+
Returns a dict with keys:
|
| 57 |
+
- `K` : int, capped at COUNT_MAX (3)
|
| 58 |
+
- `boundary_positions` : list[int], gold char positions in `form`,
|
| 59 |
+
one per gold `#`. EMPTY if K=0 OR if `noisy_label` is True.
|
| 60 |
+
- `tail_form` : str, the form slice from the last boundary on
|
| 61 |
+
(empty if K=0)
|
| 62 |
+
- `tail_lemma` : str, the last lemma piece (== lemma if K=0)
|
| 63 |
+
- `tail_script_id` : int id of the (tail_form → tail_lemma) script
|
| 64 |
+
in `edit_to_id`. OOV id if the script isn't in
|
| 65 |
+
the table. Set to OOV-id for K=0 (downstream
|
| 66 |
+
loss masks it).
|
| 67 |
+
- `noisy_label` : True iff any internal lemma piece is not a
|
| 68 |
+
verbatim case-insensitive substring of `form`
|
| 69 |
+
at the implied position.
|
| 70 |
+
|
| 71 |
+
Boundary projection algorithm: walk `pieces[:-1]` left-to-right and
|
| 72 |
+
require each piece to match `form[cursor:cursor+len(piece)]`
|
| 73 |
+
case-insensitively. If any piece fails, the entire example is noisy.
|
| 74 |
+
The last piece (tail) is computed from `form[positions[-1]:]` →
|
| 75 |
+
`pieces[-1]`; the last piece does NOT need to match verbatim (it's
|
| 76 |
+
inflected — that's what the tail-script head learns).
|
| 77 |
+
"""
|
| 78 |
+
oov_id = edit_to_id.get("<oov>", 1)
|
| 79 |
+
pieces = lemma.split("#")
|
| 80 |
+
n_pieces = len(pieces)
|
| 81 |
+
|
| 82 |
+
# K=0 (no compound): trivial path.
|
| 83 |
+
if n_pieces == 1:
|
| 84 |
+
return {
|
| 85 |
+
"K": 0,
|
| 86 |
+
"boundary_positions": [],
|
| 87 |
+
"tail_form": "",
|
| 88 |
+
"tail_lemma": lemma,
|
| 89 |
+
"tail_script_id": oov_id, # unused; loss is masked
|
| 90 |
+
"noisy_label": False,
|
| 91 |
+
}
|
| 92 |
+
|
| 93 |
+
# Walk internal pieces; project each onto a char position in `form`.
|
| 94 |
+
# Between consecutive internal pieces, allow ONE separator char
|
| 95 |
+
# (`-` or `–`) — Finnish multi-component compounds use either no
|
| 96 |
+
# separator (`juhlamekkopaniikki → juhla#mekko#paniikki`) or a
|
| 97 |
+
# hyphen between every internal pair (`KIIS-FM-radiokanavalla →
|
| 98 |
+
# KIIS#FM#radio#kanava`). Without this the matcher would fail at the
|
| 99 |
+
# first internal hyphen and mark the example noisy, dropping
|
| 100 |
+
# multi-hyphen compounds from supervision unnecessarily.
|
| 101 |
+
cursor = 0
|
| 102 |
+
positions: list[int] = []
|
| 103 |
+
noisy = False
|
| 104 |
+
for i, piece in enumerate(pieces[:-1]):
|
| 105 |
+
n = len(piece)
|
| 106 |
+
if form[cursor:cursor + n].lower() != piece.lower():
|
| 107 |
+
noisy = True
|
| 108 |
+
break
|
| 109 |
+
cursor += n
|
| 110 |
+
positions.append(cursor)
|
| 111 |
+
# If the next char is `-` or `–` AND the next piece does not
|
| 112 |
+
# begin with the same separator, consume it (the hyphen IS the
|
| 113 |
+
# compound boundary marker in the form). Only do this between
|
| 114 |
+
# internal pieces — don't strip a leading hyphen from the tail
|
| 115 |
+
# here; the decoder's apply-time strip handles that case.
|
| 116 |
+
if cursor < len(form) and form[cursor] in ("-", "–"):
|
| 117 |
+
cursor += 1
|
| 118 |
+
|
| 119 |
+
# K target: real number of `#` in lemma, capped at COUNT_MAX.
|
| 120 |
+
K_real = n_pieces - 1
|
| 121 |
+
K = min(K_real, COUNT_MAX)
|
| 122 |
+
|
| 123 |
+
if noisy:
|
| 124 |
+
# Don't fabricate split positions; mask the split + tail losses
|
| 125 |
+
# downstream. Count head still supervised on the correct K.
|
| 126 |
+
return {
|
| 127 |
+
"K": K,
|
| 128 |
+
"boundary_positions": [],
|
| 129 |
+
"tail_form": "",
|
| 130 |
+
"tail_lemma": pieces[-1],
|
| 131 |
+
"tail_script_id": oov_id,
|
| 132 |
+
"noisy_label": True,
|
| 133 |
+
}
|
| 134 |
+
|
| 135 |
+
# Clean case: tail starts at positions[-1].
|
| 136 |
+
tail_start = positions[-1]
|
| 137 |
+
# Hyphen-fallback convention: both supervision and
|
| 138 |
+
# decode strip a leading `-`/`–` from the form-tail before computing
|
| 139 |
+
# / applying the edit script. `tail_form` therefore reflects the
|
| 140 |
+
# form-slice the script operates on (stripped), matching what the
|
| 141 |
+
# decoder sees at apply-time.
|
| 142 |
+
tail_form = form[tail_start:].lstrip("-–")
|
| 143 |
+
tail_lemma = pieces[-1]
|
| 144 |
+
es = extract_edit_script(tail_form, tail_lemma)
|
| 145 |
+
tail_script_id = edit_to_id.get(_script_key(es), oov_id)
|
| 146 |
+
|
| 147 |
+
return {
|
| 148 |
+
"K": K,
|
| 149 |
+
"boundary_positions": positions,
|
| 150 |
+
"tail_form": tail_form,
|
| 151 |
+
"tail_lemma": tail_lemma,
|
| 152 |
+
"tail_script_id": tail_script_id,
|
| 153 |
+
"noisy_label": False,
|
| 154 |
+
}
|
juurio/parser/edit_scripts.py
ADDED
|
@@ -0,0 +1,155 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Longest-common-prefix lemma-edit-scripts.
|
| 2 |
+
|
| 3 |
+
The lemma head is a classifier over a finite inventory of edit scripts. This module is the
|
| 4 |
+
*representation* — given a `(form, lemma)` pair, it produces a `(delete_n, append_s)` script
|
| 5 |
+
that, applied to `form.lower()`, yields `lemma.lower()` exactly.
|
| 6 |
+
|
| 7 |
+
A script is `(delete_n: int, append_s: str)` meaning: lowercase the form, drop the last `delete_n`
|
| 8 |
+
characters, append `append_s`. The longest-common-prefix construction makes the script unique and
|
| 9 |
+
canonical for any pair (no decision about which edit is "right" — there is only one prefix-based
|
| 10 |
+
edit per pair). Full suppletion (`on -> olla`) is representable as `(2, "olla")`.
|
| 11 |
+
|
| 12 |
+
CLI:
|
| 13 |
+
python -m juurio.parser.edit_scripts probe # script coverage on UD-TDT-train/dev
|
| 14 |
+
"""
|
| 15 |
+
from __future__ import annotations
|
| 16 |
+
|
| 17 |
+
from collections import Counter
|
| 18 |
+
from typing import Iterable
|
| 19 |
+
|
| 20 |
+
EditScript = tuple[int, str]
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
def extract_edit_script(form: str, lemma: str) -> EditScript:
|
| 24 |
+
"""Compute the canonical longest-common-prefix edit script for `(form, lemma)`.
|
| 25 |
+
|
| 26 |
+
Both sides are lowercased before the common-prefix search; the resulting script applied to
|
| 27 |
+
`form.lower()` returns `lemma.lower()`. No-op on identity (delete=0, append="")."""
|
| 28 |
+
f = form.lower()
|
| 29 |
+
l = lemma.lower()
|
| 30 |
+
p = 0
|
| 31 |
+
while p < len(f) and p < len(l) and f[p] == l[p]:
|
| 32 |
+
p += 1
|
| 33 |
+
return (len(f) - p, l[p:])
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def apply_edit_script(form: str, script: EditScript) -> str:
|
| 37 |
+
"""Apply `script = (delete_n, append_s)` to `form.lower()` and return the result. Pathologically
|
| 38 |
+
over-long `delete_n` collapses to "drop the whole form" — keeps the function total."""
|
| 39 |
+
delete_n, append_s = script
|
| 40 |
+
f = form.lower()
|
| 41 |
+
if delete_n >= len(f):
|
| 42 |
+
return append_s
|
| 43 |
+
return f[: len(f) - delete_n] + append_s
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
def script_coverage(
|
| 47 |
+
pairs: Iterable[tuple[str, str]], ks: tuple[int, ...] = (200, 500, 1000, 2000)
|
| 48 |
+
) -> dict[int, tuple[float, int]]:
|
| 49 |
+
"""For each `k` in `ks`, return `(coverage, n_distinct)` where coverage is the fraction of
|
| 50 |
+
`pairs` whose extracted script is among the top-k most common scripts in `pairs` itself.
|
| 51 |
+
|
| 52 |
+
This is the *in-sample* oracle: the upper bound a classifier with vocab=k could ever reach if
|
| 53 |
+
the train and inference distributions were identical. The train/eval-split variant is below."""
|
| 54 |
+
pairs = list(pairs)
|
| 55 |
+
counts = Counter(extract_edit_script(f, l) for f, l in pairs)
|
| 56 |
+
n = len(pairs)
|
| 57 |
+
out = {}
|
| 58 |
+
for k in ks:
|
| 59 |
+
topk = {s for s, _ in counts.most_common(k)}
|
| 60 |
+
hits = sum(1 for f, l in pairs if extract_edit_script(f, l) in topk)
|
| 61 |
+
out[k] = (hits / n if n else 0.0, min(k, len(counts)))
|
| 62 |
+
return out
|
| 63 |
+
|
| 64 |
+
|
| 65 |
+
def script_coverage_train_eval(
|
| 66 |
+
train: Iterable[tuple[str, str]],
|
| 67 |
+
eval_: Iterable[tuple[str, str]],
|
| 68 |
+
ks: tuple[int, ...] = (200, 500, 1000, 2000),
|
| 69 |
+
) -> dict[int, tuple[float, int]]:
|
| 70 |
+
"""Build the top-k script vocabulary on `train`, measure coverage on `eval_`. Tests whether a
|
| 71 |
+
script vocabulary built on the train split generalises to the eval split."""
|
| 72 |
+
train = list(train)
|
| 73 |
+
eval_ = list(eval_)
|
| 74 |
+
counts = Counter(extract_edit_script(f, l) for f, l in train)
|
| 75 |
+
out = {}
|
| 76 |
+
for k in ks:
|
| 77 |
+
topk = {s for s, _ in counts.most_common(k)}
|
| 78 |
+
hits = sum(1 for f, l in eval_ if extract_edit_script(f, l) in topk)
|
| 79 |
+
out[k] = (hits / len(eval_) if eval_ else 0.0, min(k, len(counts)))
|
| 80 |
+
return out
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
# ----------------------------- CLI: script coverage ----------------------------------------------
|
| 84 |
+
def _read_conllu_pairs(path: str) -> list[tuple[str, str]]:
|
| 85 |
+
"""(form, lemma) pairs from a CoNLL-U file, skipping comments, MWT lines, and empty lemmas.
|
| 86 |
+
|
| 87 |
+
A "real" content token in CoNLL-U has an integer `id` (MWT spans are `n-m` and are *also*
|
| 88 |
+
emitted but as separate per-component lines we want; the `n-m` ones we skip). Empty / `_`
|
| 89 |
+
lemmas are skipped — punctuation and some symbols carry no lemma."""
|
| 90 |
+
out = []
|
| 91 |
+
with open(path, encoding="utf-8") as fh:
|
| 92 |
+
for ln in fh:
|
| 93 |
+
ln = ln.strip()
|
| 94 |
+
if not ln or ln.startswith("#"):
|
| 95 |
+
continue
|
| 96 |
+
cols = ln.split("\t")
|
| 97 |
+
if len(cols) < 4:
|
| 98 |
+
continue
|
| 99 |
+
tid = cols[0]
|
| 100 |
+
if "-" in tid or "." in tid: # MWT span line / empty-node line — skip
|
| 101 |
+
continue
|
| 102 |
+
form, lemma = cols[1], cols[2]
|
| 103 |
+
if not form or not lemma or lemma == "_":
|
| 104 |
+
continue
|
| 105 |
+
out.append((form, lemma))
|
| 106 |
+
return out
|
| 107 |
+
|
| 108 |
+
|
| 109 |
+
def probe(train_path: str, dev_path: str) -> None:
|
| 110 |
+
"""Report edit-script coverage on UD-TDT-train and UD-TDT-dev."""
|
| 111 |
+
train = _read_conllu_pairs(train_path)
|
| 112 |
+
dev = _read_conllu_pairs(dev_path)
|
| 113 |
+
print(f"UD-TDT: train {len(train)} tokens, dev {len(dev)} tokens")
|
| 114 |
+
print()
|
| 115 |
+
|
| 116 |
+
# in-sample (train coverage by train top-k)
|
| 117 |
+
print("== train in-sample coverage (oracle upper bound on a classifier head) ==")
|
| 118 |
+
cov_t = script_coverage(train, ks=(100, 200, 500, 1000, 2000, 4000))
|
| 119 |
+
for k, (c, n) in cov_t.items():
|
| 120 |
+
print(f" top-{k:5d} coverage {c*100:6.2f}% (n_distinct = {n})")
|
| 121 |
+
|
| 122 |
+
# out-of-sample (dev coverage by train top-k) — the actual generalisation question
|
| 123 |
+
print()
|
| 124 |
+
print("== dev coverage by train-fitted top-k vocab (Q3 generalisation) ==")
|
| 125 |
+
cov_d = script_coverage_train_eval(train, dev, ks=(100, 200, 500, 1000, 2000, 4000))
|
| 126 |
+
for k, (c, n) in cov_d.items():
|
| 127 |
+
print(f" top-{k:5d} dev-coverage {c*100:6.2f}% (vocab built from train)")
|
| 128 |
+
|
| 129 |
+
# uncoverable tail: how many distinct scripts total, and how many tokens does the ≥2000 tail eat
|
| 130 |
+
counts = Counter(extract_edit_script(f, l) for f, l in train)
|
| 131 |
+
print()
|
| 132 |
+
print(f"Total distinct scripts in train: {len(counts)}")
|
| 133 |
+
if len(counts) > 2000:
|
| 134 |
+
tail = sum(c for _, c in counts.most_common()[2000:])
|
| 135 |
+
print(f"Train tokens captured ONLY by tail scripts (>top-2000): {tail} ({tail/len(train)*100:.2f}%)")
|
| 136 |
+
|
| 137 |
+
print()
|
| 138 |
+
print("== sample of the rare-tail scripts (rank 1500–1510, just to eyeball oddities) ==")
|
| 139 |
+
for rank, (sc, n) in enumerate(counts.most_common()[1500:1510], start=1501):
|
| 140 |
+
delete_n, append_s = sc
|
| 141 |
+
print(f" rank {rank:4d} ({delete_n:2d}, {append_s!r:20s}) count {n}")
|
| 142 |
+
|
| 143 |
+
|
| 144 |
+
if __name__ == "__main__":
|
| 145 |
+
import os
|
| 146 |
+
import sys
|
| 147 |
+
|
| 148 |
+
repo = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
| 149 |
+
train = os.path.join(repo, "data", "ud", "fi_tdt-ud-train.conllu")
|
| 150 |
+
dev = os.path.join(repo, "data", "ud", "fi_tdt-ud-dev.conllu")
|
| 151 |
+
cmd = sys.argv[1] if len(sys.argv) > 1 else "probe"
|
| 152 |
+
if cmd == "probe":
|
| 153 |
+
probe(train, dev)
|
| 154 |
+
else:
|
| 155 |
+
raise SystemExit(f"unknown cmd {cmd!r}; usage: python -m juurio.parser.edit_scripts probe")
|
juurio/parser/eval.py
ADDED
|
@@ -0,0 +1,158 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Checkpoint loader for the tiny Finnish UD parser.
|
| 2 |
+
|
| 3 |
+
Reconstructs a `TinyFiUD` model from a saved checkpoint blob, recovering the
|
| 4 |
+
channel/head configuration from the persisted config (and, for older ckpts,
|
| 5 |
+
inferring it from the state_dict shapes). The serving neural adapter imports
|
| 6 |
+
`_load_checkpoint` from here.
|
| 7 |
+
"""
|
| 8 |
+
from __future__ import annotations
|
| 9 |
+
|
| 10 |
+
import torch
|
| 11 |
+
|
| 12 |
+
from .label_vocab import LabelVocab
|
| 13 |
+
from .model import TinyFiUD
|
| 14 |
+
from .tokenizer import BPETokenizer
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
# EXACT allowlist for unexpected keys in `_load_checkpoint`. Some training
|
| 18 |
+
# checkpoints contain `hidden_kd_proj.{weight, bias}` — an extra student
|
| 19 |
+
# projection used only during training; at eval time it is unused. We use
|
| 20 |
+
# an exact allowlist (rather than a prefix filter) so the strict=False load
|
| 21 |
+
# contract stays tight and never silently accepts arbitrary extra keys.
|
| 22 |
+
_ALLOWED_EXTRA_STATE_KEYS: frozenset[str] = frozenset({
|
| 23 |
+
"hidden_kd_proj.weight",
|
| 24 |
+
"hidden_kd_proj.bias",
|
| 25 |
+
})
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
def _load_checkpoint(path: str, vocab: LabelVocab, tokenizer: BPETokenizer, device: str) -> TinyFiUD:
|
| 29 |
+
blob = torch.load(path, map_location=device, weights_only=False)
|
| 30 |
+
cfg = blob["config"]
|
| 31 |
+
# Detect parser_bilstm_hidden from saved cfg or infer from state_dict keys
|
| 32 |
+
# (so this loader stays compatible with ckpts saved before the field existed).
|
| 33 |
+
sd = blob["state_dict"]
|
| 34 |
+
bilstm_hidden = cfg.get("parser_bilstm_hidden", 0)
|
| 35 |
+
if bilstm_hidden == 0 and "parser_bilstm.weight_ih_l0" in sd:
|
| 36 |
+
# weight_ih_l0 has shape (4*hidden, input) — recover hidden from row count
|
| 37 |
+
bilstm_hidden = sd["parser_bilstm.weight_ih_l0"].shape[0] // 4
|
| 38 |
+
# Detect arc_d from W_arc shape (older ckpts default to 200) and
|
| 39 |
+
# arc_distance_bias from the presence of the bias parameter.
|
| 40 |
+
arc_d = cfg.get("arc_d", sd["W_arc"].shape[0] if "W_arc" in sd else 200)
|
| 41 |
+
arc_distance_bias = cfg.get("arc_distance_bias", "arc_dist_bias" in sd)
|
| 42 |
+
# Detect FST channel (fst_tag_vocab_size + fst_tag_dim) from state_dict
|
| 43 |
+
# (so older ckpts load unchanged, and FST ckpts reconstruct the embedding bag + proj).
|
| 44 |
+
fst_tag_vocab_size = cfg.get("fst_tag_vocab_size", 0)
|
| 45 |
+
if fst_tag_vocab_size == 0 and "fst_tag_emb.weight" in sd:
|
| 46 |
+
fst_tag_vocab_size = sd["fst_tag_emb.weight"].shape[0]
|
| 47 |
+
fst_tag_dim = cfg.get("fst_tag_dim", 32)
|
| 48 |
+
if "fst_tag_emb.weight" in sd:
|
| 49 |
+
fst_tag_dim = sd["fst_tag_emb.weight"].shape[1]
|
| 50 |
+
# Reconstruct compound channel from cfg + state_dict.
|
| 51 |
+
# Older ckpts don't have these fields; the channel stays off.
|
| 52 |
+
compound_enabled = cfg.get("compound_enabled", False)
|
| 53 |
+
if not compound_enabled and "char_emb.weight" in sd:
|
| 54 |
+
compound_enabled = True
|
| 55 |
+
char_vocab_size = cfg.get("char_vocab_size", 0)
|
| 56 |
+
if char_vocab_size == 0 and "char_emb.weight" in sd:
|
| 57 |
+
char_vocab_size = sd["char_emb.weight"].shape[0]
|
| 58 |
+
char_emb_dim = cfg.get("char_emb_dim", 16)
|
| 59 |
+
if "char_emb.weight" in sd:
|
| 60 |
+
char_emb_dim = sd["char_emb.weight"].shape[1]
|
| 61 |
+
char_conv_channels = cfg.get("char_conv_channels", 64)
|
| 62 |
+
if "compound_split_conv.weight" in sd:
|
| 63 |
+
char_conv_channels = sd["compound_split_conv.weight"].shape[0]
|
| 64 |
+
max_form_len = cfg.get("max_form_len", 64)
|
| 65 |
+
# Detect parser-private FF block from state_dict.
|
| 66 |
+
# Hidden dim recovered from linear1.weight row count.
|
| 67 |
+
parser_private_hidden = cfg.get("parser_private_hidden", 0)
|
| 68 |
+
if parser_private_hidden == 0 and "parser_private_linear1.weight" in sd:
|
| 69 |
+
parser_private_hidden = sd["parser_private_linear1.weight"].shape[0]
|
| 70 |
+
# Reconstruct per-key Feats head config from the saved blob. Without
|
| 71 |
+
# this, eval would build a model without the per-key head and its
|
| 72 |
+
# state_dict keys would be unexpected — load_state_dict(strict=True)
|
| 73 |
+
# would fail. The trained ckpt persisted use_per_key_feats,
|
| 74 |
+
# value_counts, upos_emb_dim, and flat_ids; rebuild the head with the
|
| 75 |
+
# SAME shape it was trained with.
|
| 76 |
+
use_per_key_feats = cfg.get("use_per_key_feats", False)
|
| 77 |
+
feats_per_key_value_counts = cfg.get("feats_per_key_value_counts")
|
| 78 |
+
feats_per_key_upos_emb_dim = cfg.get("feats_per_key_upos_emb_dim", 32)
|
| 79 |
+
feats_per_key_flat_ids = cfg.get("feats_per_key_flat_ids")
|
| 80 |
+
# If the state_dict has per-key head weights but the config doesn't
|
| 81 |
+
# carry the metadata, fail loud — a ckpt without persisted config
|
| 82 |
+
# is a serialization bug we need to catch, not paper over.
|
| 83 |
+
has_per_key_sd = any(k.startswith("feats_per_key_head.") for k in sd.keys())
|
| 84 |
+
if has_per_key_sd and not use_per_key_feats:
|
| 85 |
+
raise ValueError(
|
| 86 |
+
"Checkpoint state_dict carries `feats_per_key_head.*` weights "
|
| 87 |
+
"but config.use_per_key_feats is False. The ckpt was saved without "
|
| 88 |
+
"the per-key metadata block. Re-save the ckpt via the training path "
|
| 89 |
+
"that persists the per-key config, or downgrade the eval to BCE-only."
|
| 90 |
+
)
|
| 91 |
+
# Near-identity capacity adapter.
|
| 92 |
+
# When a ckpt carries the `near_identity_adapter.*` namespace in its
|
| 93 |
+
# state_dict, the config must persist the flag; otherwise we'd
|
| 94 |
+
# silently rebuild a model without the adapter and load_state_dict would
|
| 95 |
+
# fail (missing keys). Same guard pattern as the per-key Feats head.
|
| 96 |
+
use_near_identity_adapter = cfg.get("use_near_identity_adapter", False)
|
| 97 |
+
near_identity_n_layers = cfg.get("near_identity_n_layers", 0)
|
| 98 |
+
near_identity_d_model = cfg.get("near_identity_d_model")
|
| 99 |
+
near_identity_heads = cfg.get("near_identity_heads")
|
| 100 |
+
near_identity_ff = cfg.get("near_identity_ff")
|
| 101 |
+
has_adapter_sd = any(k.startswith("near_identity_adapter.") for k in sd.keys())
|
| 102 |
+
if has_adapter_sd and not use_near_identity_adapter:
|
| 103 |
+
raise ValueError(
|
| 104 |
+
"Checkpoint state_dict carries `near_identity_adapter.*` weights "
|
| 105 |
+
"but config.use_near_identity_adapter is False. The ckpt was "
|
| 106 |
+
"saved without the adapter metadata block. Re-save via "
|
| 107 |
+
"`warm_start_near_identity` / current train.py path that persists "
|
| 108 |
+
"the adapter config."
|
| 109 |
+
)
|
| 110 |
+
model = TinyFiUD(
|
| 111 |
+
vocab=tokenizer.vocab_size, d_model=cfg["d_model"], layers=cfg["layers"],
|
| 112 |
+
heads=cfg["heads"], ff=cfg["ff"], n_upos=vocab.n_upos, n_feat=vocab.n_feat,
|
| 113 |
+
n_edit=vocab.n_edit, n_rel=vocab.n_rel, maxlen=cfg["max_len"],
|
| 114 |
+
parser_bilstm_hidden=bilstm_hidden,
|
| 115 |
+
arc_d=arc_d,
|
| 116 |
+
arc_distance_bias=arc_distance_bias,
|
| 117 |
+
fst_tag_vocab_size=fst_tag_vocab_size,
|
| 118 |
+
fst_tag_dim=fst_tag_dim,
|
| 119 |
+
compound_enabled=compound_enabled,
|
| 120 |
+
char_vocab_size=char_vocab_size,
|
| 121 |
+
char_emb_dim=char_emb_dim,
|
| 122 |
+
char_conv_channels=char_conv_channels,
|
| 123 |
+
max_form_len=max_form_len,
|
| 124 |
+
parser_private_hidden=parser_private_hidden,
|
| 125 |
+
use_per_key_feats=use_per_key_feats,
|
| 126 |
+
feats_per_key_value_counts=feats_per_key_value_counts,
|
| 127 |
+
feats_per_key_upos_emb_dim=feats_per_key_upos_emb_dim,
|
| 128 |
+
feats_per_key_flat_ids=feats_per_key_flat_ids,
|
| 129 |
+
use_near_identity_adapter=use_near_identity_adapter,
|
| 130 |
+
near_identity_n_layers=near_identity_n_layers,
|
| 131 |
+
near_identity_d_model=near_identity_d_model,
|
| 132 |
+
near_identity_heads=near_identity_heads,
|
| 133 |
+
near_identity_ff=near_identity_ff,
|
| 134 |
+
).to(device)
|
| 135 |
+
# Some training checkpoints contain extra `hidden_kd_proj.*` projection
|
| 136 |
+
# keys that are unused at eval time. Tolerate them with strict=False
|
| 137 |
+
# rather than re-instantiate TinyFiUD with use_hidden_kd_proj=True
|
| 138 |
+
# (which would also need the cfg flag in the ckpt blob, which older
|
| 139 |
+
# ckpts lack). The allowlist is an EXACT set rather than a prefix
|
| 140 |
+
# filter so the strict=False contract stays tight and never silently
|
| 141 |
+
# accepts arbitrary extra child keys.
|
| 142 |
+
missing, unexpected = model.load_state_dict(sd, strict=False)
|
| 143 |
+
bad_unexpected = [k for k in unexpected if k not in _ALLOWED_EXTRA_STATE_KEYS]
|
| 144 |
+
if bad_unexpected or missing:
|
| 145 |
+
raise RuntimeError(
|
| 146 |
+
f"eval ckpt load mismatch — missing={missing} unexpected={bad_unexpected}"
|
| 147 |
+
)
|
| 148 |
+
# Pin the FST vocab hash on the loaded model so a consumer can refuse a
|
| 149 |
+
# stale vocab. Older ckpts won't have the field — that's fine.
|
| 150 |
+
expected_hash = cfg.get("fst_tag_vocab_hash")
|
| 151 |
+
if expected_hash is not None:
|
| 152 |
+
model._fst_tag_vocab_hash = expected_hash
|
| 153 |
+
# Same hash pin for the char vocab (analogous to the FST tag vocab).
|
| 154 |
+
# Older ckpts won't have it.
|
| 155 |
+
expected_char_hash = cfg.get("char_vocab_hash")
|
| 156 |
+
if expected_char_hash is not None:
|
| 157 |
+
model._char_vocab_hash = expected_char_hash
|
| 158 |
+
return model
|
juurio/parser/feats_per_key_head.py
ADDED
|
@@ -0,0 +1,96 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Per-key biaffine Feats head.
|
| 2 |
+
|
| 3 |
+
Uses one biaffine scorer per UD feature key, conditioned on a UPOS embedding,
|
| 4 |
+
in place of a single `Linear(d_model → n_feat) + BCE` head.
|
| 5 |
+
Each scorer predicts an EXCLUSIVE value among that key's vocabulary
|
| 6 |
+
(plus an `<EMPTY>` class at column 0 that the model can predict
|
| 7 |
+
when the key is absent on this token).
|
| 8 |
+
|
| 9 |
+
Architecture (per key K with `n_values_K` values, including EMPTY):
|
| 10 |
+
|
| 11 |
+
logits_K[b, t, v] =
|
| 12 |
+
bilinear_K(h[b, t], upos_emb[b, t])[v] # (B, T) → (B, T, n_v)
|
| 13 |
+
+ linear_h_K(h[b, t])[v]
|
| 14 |
+
+ linear_u_K(upos_emb[b, t])[v]
|
| 15 |
+
|
| 16 |
+
`nn.Bilinear` provides the (h^T W_K[v] upos_emb + bias_K[v]) term;
|
| 17 |
+
the two `nn.Linear`s without bias add the Dozat-Manning x1- and
|
| 18 |
+
x2-affine terms. Total per-key parameter count is small
|
| 19 |
+
(O(d_model * upos_emb_dim * n_values_K)).
|
| 20 |
+
|
| 21 |
+
Notes:
|
| 22 |
+
- All keys are uniform, including singleton-value keys (n_values=2).
|
| 23 |
+
- The head owns its own UPOS embedding table and is fed by the UPOS
|
| 24 |
+
classifier's predicted-UPOS argmax at both train and eval, avoiding
|
| 25 |
+
a train/eval mismatch.
|
| 26 |
+
"""
|
| 27 |
+
from __future__ import annotations
|
| 28 |
+
|
| 29 |
+
import torch
|
| 30 |
+
import torch.nn as nn
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
class PerKeyFeatsHead(nn.Module):
|
| 34 |
+
"""Per-key biaffine Feats head.
|
| 35 |
+
|
| 36 |
+
Args:
|
| 37 |
+
d_model: hidden dim of the encoder's per-token representation.
|
| 38 |
+
upos_emb_dim: dim of the head-owned UPOS embedding (small).
|
| 39 |
+
value_counts: per-key value count INCLUDING the `<EMPTY>`
|
| 40 |
+
class at index 0. Length = number of keys.
|
| 41 |
+
n_upos: size of the UPOS vocabulary (for the head's embedding).
|
| 42 |
+
dropout: dropout applied to both `h` and UPOS embeddings before
|
| 43 |
+
scoring.
|
| 44 |
+
|
| 45 |
+
Forward signature:
|
| 46 |
+
h: `(B, T, d_model)` per-token hidden states.
|
| 47 |
+
upos_ids: `(B, T)` longs — UPOS argmax indices (the UPOS
|
| 48 |
+
head's predictions at both train and eval).
|
| 49 |
+
|
| 50 |
+
Returns:
|
| 51 |
+
`list[Tensor]` of length `n_keys`; element k has shape
|
| 52 |
+
`(B, T, value_counts[k])`. Column 0 is the `<EMPTY>` class.
|
| 53 |
+
"""
|
| 54 |
+
|
| 55 |
+
def __init__(
|
| 56 |
+
self,
|
| 57 |
+
d_model: int,
|
| 58 |
+
upos_emb_dim: int,
|
| 59 |
+
value_counts: list[int],
|
| 60 |
+
n_upos: int,
|
| 61 |
+
dropout: float = 0.0,
|
| 62 |
+
):
|
| 63 |
+
super().__init__()
|
| 64 |
+
self.d_model = d_model
|
| 65 |
+
self.upos_emb_dim = upos_emb_dim
|
| 66 |
+
self.value_counts = list(value_counts)
|
| 67 |
+
self.upos_emb = nn.Embedding(n_upos, upos_emb_dim)
|
| 68 |
+
self.dropout = nn.Dropout(dropout)
|
| 69 |
+
# One biaffine + two linear terms per key.
|
| 70 |
+
self.bilinear = nn.ModuleList(
|
| 71 |
+
nn.Bilinear(d_model, upos_emb_dim, n_v) for n_v in value_counts
|
| 72 |
+
)
|
| 73 |
+
self.linear_h = nn.ModuleList(
|
| 74 |
+
nn.Linear(d_model, n_v, bias=False) for n_v in value_counts
|
| 75 |
+
)
|
| 76 |
+
self.linear_u = nn.ModuleList(
|
| 77 |
+
nn.Linear(upos_emb_dim, n_v, bias=False) for n_v in value_counts
|
| 78 |
+
)
|
| 79 |
+
|
| 80 |
+
@property
|
| 81 |
+
def n_keys(self) -> int:
|
| 82 |
+
return len(self.value_counts)
|
| 83 |
+
|
| 84 |
+
def forward(
|
| 85 |
+
self,
|
| 86 |
+
h: torch.Tensor,
|
| 87 |
+
upos_ids: torch.Tensor,
|
| 88 |
+
) -> list[torch.Tensor]:
|
| 89 |
+
upos_e = self.upos_emb(upos_ids) # (B, T, upos_emb_dim)
|
| 90 |
+
h_d = self.dropout(h)
|
| 91 |
+
u_d = self.dropout(upos_e)
|
| 92 |
+
out: list[torch.Tensor] = []
|
| 93 |
+
for bi, lh, lu in zip(self.bilinear, self.linear_h, self.linear_u):
|
| 94 |
+
logits = bi(h_d, u_d) + lh(h_d) + lu(u_d)
|
| 95 |
+
out.append(logits)
|
| 96 |
+
return out
|
juurio/parser/fst_cache.py
ADDED
|
@@ -0,0 +1,147 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Tag vocabulary + per-form analysis cache for the FST channel.
|
| 2 |
+
|
| 3 |
+
The pipeline:
|
| 4 |
+
1. Pre-compute an FST analysis for every unique surface form in the
|
| 5 |
+
training data → FST cache JSONL.
|
| 6 |
+
2. At collator time, look up form → set of morphtags across all readings;
|
| 7 |
+
encode tags as int ids via `FstTagVocab`; pad to `max_tags` and emit a
|
| 8 |
+
(T, max_tags) int tensor parallel to subword positions.
|
| 9 |
+
3. At model time, embed each tag id, sum across the `max_tags` axis (the
|
| 10 |
+
embedding for PAD=0 is constrained to zero so padded slots contribute
|
| 11 |
+
nothing), normalize, project to d_model, and add to the encoder input
|
| 12 |
+
at first-subword positions.
|
| 13 |
+
|
| 14 |
+
The vocab uses PAD=0, UNK=1 as fixed reserved ids; dataset tags start at 2.
|
| 15 |
+
The cache excludes the reading's lemma slot (`reading[0]`) when computing
|
| 16 |
+
the tag set, since lemma identity is handled by the dedicated edit-script
|
| 17 |
+
head and would otherwise blow up the tag vocab with lexical entries.
|
| 18 |
+
"""
|
| 19 |
+
from __future__ import annotations
|
| 20 |
+
|
| 21 |
+
import hashlib
|
| 22 |
+
import json
|
| 23 |
+
from collections import Counter
|
| 24 |
+
from pathlib import Path
|
| 25 |
+
from typing import Iterable
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
PAD_ID = 0
|
| 29 |
+
UNK_ID = 1
|
| 30 |
+
PAD_TAG = "<PAD>"
|
| 31 |
+
UNK_TAG = "<UNK>"
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
class FstTagVocab:
|
| 35 |
+
"""Maps morphtag strings to int ids."""
|
| 36 |
+
|
| 37 |
+
def __init__(self, tags: list[str]):
|
| 38 |
+
# tags is the ordered list with PAD at index 0, UNK at index 1, real tags from index 2.
|
| 39 |
+
assert tags[0] == PAD_TAG and tags[1] == UNK_TAG
|
| 40 |
+
self._tags = list(tags)
|
| 41 |
+
self._tag_to_id = {t: i for i, t in enumerate(tags)}
|
| 42 |
+
|
| 43 |
+
@property
|
| 44 |
+
def size(self) -> int:
|
| 45 |
+
return len(self._tags)
|
| 46 |
+
|
| 47 |
+
@classmethod
|
| 48 |
+
def fit(cls, readings: Iterable[list[str]], *, min_count: int = 1,
|
| 49 |
+
max_tags: int | None = None) -> "FstTagVocab":
|
| 50 |
+
"""Fit a vocab from an iterable of readings. Each reading is
|
| 51 |
+
[lemma, tag1, tag2, ...] — the lemma is excluded automatically."""
|
| 52 |
+
c: Counter[str] = Counter()
|
| 53 |
+
for r in readings:
|
| 54 |
+
for tag in r[1:]: # skip lemma at position 0
|
| 55 |
+
c[tag] += 1
|
| 56 |
+
kept = [t for t, n in c.most_common() if n >= min_count]
|
| 57 |
+
if max_tags is not None:
|
| 58 |
+
kept = kept[:max_tags]
|
| 59 |
+
return cls([PAD_TAG, UNK_TAG] + kept)
|
| 60 |
+
|
| 61 |
+
def encode(self, tag: str) -> int:
|
| 62 |
+
return self._tag_to_id.get(tag, UNK_ID)
|
| 63 |
+
|
| 64 |
+
def decode(self, idx: int) -> str:
|
| 65 |
+
return self._tags[idx] if 0 <= idx < len(self._tags) else UNK_TAG
|
| 66 |
+
|
| 67 |
+
def hash(self) -> str:
|
| 68 |
+
"""Stable hex digest of the ordered tag list — short enough to embed in
|
| 69 |
+
a ckpt config, strong enough to catch any reorder / refit / different-
|
| 70 |
+
min-count vocab. Used to pin the vocab to the trained ckpt so a stale
|
| 71 |
+
vocab can't silently feed wrong-IDed tags into a loaded model."""
|
| 72 |
+
h = hashlib.sha256()
|
| 73 |
+
for t in self._tags:
|
| 74 |
+
h.update(t.encode("utf-8"))
|
| 75 |
+
h.update(b"\x00")
|
| 76 |
+
return h.hexdigest()[:16]
|
| 77 |
+
|
| 78 |
+
def save(self, path: str) -> None:
|
| 79 |
+
Path(path).parent.mkdir(parents=True, exist_ok=True)
|
| 80 |
+
with open(path, "w") as fh:
|
| 81 |
+
json.dump({"tags": self._tags}, fh, ensure_ascii=False)
|
| 82 |
+
|
| 83 |
+
@classmethod
|
| 84 |
+
def load(cls, path: str) -> "FstTagVocab":
|
| 85 |
+
with open(path) as fh:
|
| 86 |
+
blob = json.load(fh)
|
| 87 |
+
return cls(blob["tags"])
|
| 88 |
+
|
| 89 |
+
|
| 90 |
+
class FstCache:
|
| 91 |
+
"""form -> list of readings (each reading is [lemma, tag1, tag2, ...])."""
|
| 92 |
+
|
| 93 |
+
def __init__(self, data: dict[str, list[list[str]]] | None = None):
|
| 94 |
+
self._data: dict[str, list[list[str]]] = dict(data) if data else {}
|
| 95 |
+
|
| 96 |
+
def lookup(self, form: str) -> list[list[str]]:
|
| 97 |
+
return self._data.get(form, [])
|
| 98 |
+
|
| 99 |
+
def __len__(self) -> int:
|
| 100 |
+
return len(self._data)
|
| 101 |
+
|
| 102 |
+
def __contains__(self, form: str) -> bool:
|
| 103 |
+
return form in self._data
|
| 104 |
+
|
| 105 |
+
def form_to_tagset(self, form: str) -> set[str]:
|
| 106 |
+
"""Union of all morphtags across all readings for `form`. The reading's
|
| 107 |
+
lemma slot (position 0) is excluded."""
|
| 108 |
+
out: set[str] = set()
|
| 109 |
+
for r in self._data.get(form, []):
|
| 110 |
+
for tag in r[1:]:
|
| 111 |
+
out.add(tag)
|
| 112 |
+
return out
|
| 113 |
+
|
| 114 |
+
def form_to_tag_ids(self, form: str, vocab: FstTagVocab, *, max_tags: int) -> list[int]:
|
| 115 |
+
"""Encode the form's tagset as a fixed-length int list, padded with PAD_ID.
|
| 116 |
+
If more tags than `max_tags`, truncates (sorted by tag id for determinism)."""
|
| 117 |
+
tags = sorted(self.form_to_tagset(form))
|
| 118 |
+
ids = [vocab.encode(t) for t in tags if vocab.encode(t) != UNK_ID]
|
| 119 |
+
# de-duplicate while keeping order
|
| 120 |
+
seen: set[int] = set()
|
| 121 |
+
deduped = []
|
| 122 |
+
for i in ids:
|
| 123 |
+
if i not in seen:
|
| 124 |
+
seen.add(i)
|
| 125 |
+
deduped.append(i)
|
| 126 |
+
deduped = deduped[:max_tags]
|
| 127 |
+
# pad to max_tags
|
| 128 |
+
return deduped + [PAD_ID] * (max_tags - len(deduped))
|
| 129 |
+
|
| 130 |
+
def save(self, path: str) -> None:
|
| 131 |
+
Path(path).parent.mkdir(parents=True, exist_ok=True)
|
| 132 |
+
with open(path, "w") as fh:
|
| 133 |
+
for form, readings in self._data.items():
|
| 134 |
+
fh.write(json.dumps({"form": form, "readings": readings},
|
| 135 |
+
ensure_ascii=False) + "\n")
|
| 136 |
+
|
| 137 |
+
@classmethod
|
| 138 |
+
def load(cls, path: str) -> "FstCache":
|
| 139 |
+
data: dict[str, list[list[str]]] = {}
|
| 140 |
+
with open(path) as fh:
|
| 141 |
+
for line in fh:
|
| 142 |
+
line = line.strip()
|
| 143 |
+
if not line:
|
| 144 |
+
continue
|
| 145 |
+
rec = json.loads(line)
|
| 146 |
+
data[rec["form"]] = rec["readings"]
|
| 147 |
+
return cls(data)
|
juurio/parser/label_vocab.py
ADDED
|
@@ -0,0 +1,192 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Label vocabularies for the compact Finnish UD parser.
|
| 2 |
+
|
| 3 |
+
Each task head needs to map between label strings and integer ids (model output). This
|
| 4 |
+
module bundles those mappings into a single `LabelVocab` object that:
|
| 5 |
+
|
| 6 |
+
- Is *fit once* on the training corpus (`LabelVocab.fit(records, top_edit=1000, top_feat=80)`).
|
| 7 |
+
- Reserves id 0 = `<pad>` and id 1 = `<oov>` for UPOS / deprel / edit-script so the training
|
| 8 |
+
loop can use `ignore_index=0` and OOV labels can be routed to a sink class.
|
| 9 |
+
- Frequency-caps feats and edit-scripts to keep head sizes small.
|
| 10 |
+
- Saves / loads as a single JSON blob alongside the tokenizer.
|
| 11 |
+
|
| 12 |
+
Edit-scripts are computed via `juurio.parser.edit_scripts.extract_edit_script`. Feats are
|
| 13 |
+
treated as multi-label: each `Key=Value` pair has its own id, the per-token target is multi-hot.
|
| 14 |
+
deprels are single-label (BCE-flavoured biaffine).
|
| 15 |
+
"""
|
| 16 |
+
from __future__ import annotations
|
| 17 |
+
|
| 18 |
+
import json
|
| 19 |
+
import os
|
| 20 |
+
from collections import Counter
|
| 21 |
+
from typing import Iterable
|
| 22 |
+
|
| 23 |
+
from .edit_scripts import extract_edit_script
|
| 24 |
+
from .silver import SilverSentence
|
| 25 |
+
|
| 26 |
+
|
| 27 |
+
PAD = "<pad>"
|
| 28 |
+
OOV = "<oov>"
|
| 29 |
+
_RESERVED = (PAD, OOV)
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
class LabelVocab:
|
| 33 |
+
"""Bundle of per-head label vocabularies. Frozen after `fit` or `load`."""
|
| 34 |
+
|
| 35 |
+
def __init__(
|
| 36 |
+
self,
|
| 37 |
+
upos_to_id: dict[str, int],
|
| 38 |
+
deprel_to_id: dict[str, int],
|
| 39 |
+
feat_to_id: dict[str, int],
|
| 40 |
+
edit_to_id: dict[str, int],
|
| 41 |
+
edit_scripts: list[tuple[int, str]],
|
| 42 |
+
):
|
| 43 |
+
self.upos_to_id = dict(upos_to_id)
|
| 44 |
+
self.deprel_to_id = dict(deprel_to_id)
|
| 45 |
+
self.feat_to_id = dict(feat_to_id)
|
| 46 |
+
self.edit_to_id = dict(edit_to_id)
|
| 47 |
+
# parallel list so encoder can map back from id → (delete_n, append_s)
|
| 48 |
+
self.edit_scripts = list(edit_scripts)
|
| 49 |
+
# reverse lookup tables (for decoding model predictions back to strings)
|
| 50 |
+
self.id_to_upos = {i: t for t, i in self.upos_to_id.items()}
|
| 51 |
+
self.id_to_deprel = {i: t for t, i in self.deprel_to_id.items()}
|
| 52 |
+
self.id_to_feat = {i: t for t, i in self.feat_to_id.items()}
|
| 53 |
+
self.id_to_edit = {i: t for t, i in self.edit_to_id.items()}
|
| 54 |
+
|
| 55 |
+
# ----------------------------- sizes -------------------------------------------------------
|
| 56 |
+
@property
|
| 57 |
+
def n_upos(self) -> int:
|
| 58 |
+
return len(self.upos_to_id)
|
| 59 |
+
|
| 60 |
+
@property
|
| 61 |
+
def n_feat(self) -> int:
|
| 62 |
+
return len(self.feat_to_id)
|
| 63 |
+
|
| 64 |
+
@property
|
| 65 |
+
def n_edit(self) -> int:
|
| 66 |
+
return len(self.edit_to_id)
|
| 67 |
+
|
| 68 |
+
@property
|
| 69 |
+
def n_rel(self) -> int:
|
| 70 |
+
return len(self.deprel_to_id)
|
| 71 |
+
|
| 72 |
+
# ----------------------------- fit ---------------------------------------------------------
|
| 73 |
+
@classmethod
|
| 74 |
+
def fit(
|
| 75 |
+
cls,
|
| 76 |
+
records: Iterable[SilverSentence],
|
| 77 |
+
*,
|
| 78 |
+
top_edit: int = 1000,
|
| 79 |
+
top_feat: int = 80,
|
| 80 |
+
min_feat_freq: int = 1,
|
| 81 |
+
) -> "LabelVocab":
|
| 82 |
+
"""Build a fresh `LabelVocab` from the silver records. Vocabularies are deterministic
|
| 83 |
+
functions of frequency (ties broken alphabetically)."""
|
| 84 |
+
upos_c: Counter[str] = Counter()
|
| 85 |
+
deprel_c: Counter[str] = Counter()
|
| 86 |
+
feat_c: Counter[str] = Counter()
|
| 87 |
+
edit_c: Counter[tuple[int, str]] = Counter()
|
| 88 |
+
|
| 89 |
+
for s in records:
|
| 90 |
+
for t in s.tokens:
|
| 91 |
+
upos_c[t.upos] += 1
|
| 92 |
+
deprel_c[t.deprel] += 1
|
| 93 |
+
if t.feats:
|
| 94 |
+
items = (f"{k}={v}" for k, v in t.feats.items()) \
|
| 95 |
+
if isinstance(t.feats, dict) else t.feats.split("|")
|
| 96 |
+
for kv in items:
|
| 97 |
+
feat_c[kv] += 1
|
| 98 |
+
script = extract_edit_script(t.form, t.lemma)
|
| 99 |
+
edit_c[script] += 1
|
| 100 |
+
|
| 101 |
+
def _to_dict(counter: Counter, *, cap: int | None = None,
|
| 102 |
+
min_freq: int = 1) -> dict[str, int]:
|
| 103 |
+
out = {tok: i for i, tok in enumerate(_RESERVED)}
|
| 104 |
+
items = sorted(
|
| 105 |
+
((k, v) for k, v in counter.items() if v >= min_freq),
|
| 106 |
+
key=lambda kv: (-kv[1], str(kv[0])),
|
| 107 |
+
)
|
| 108 |
+
if cap is not None:
|
| 109 |
+
items = items[:cap]
|
| 110 |
+
for k, _ in items:
|
| 111 |
+
if str(k) in out:
|
| 112 |
+
continue
|
| 113 |
+
out[str(k)] = len(out)
|
| 114 |
+
return out
|
| 115 |
+
|
| 116 |
+
upos_to_id = _to_dict(upos_c)
|
| 117 |
+
deprel_to_id = _to_dict(deprel_c)
|
| 118 |
+
feat_to_id = _to_dict(feat_c, cap=top_feat, min_freq=min_feat_freq)
|
| 119 |
+
# edit-scripts: cap and key the dict by the *string* form "<delete_n>:<append_s>"
|
| 120 |
+
edit_items_sorted = sorted(
|
| 121 |
+
edit_c.items(), key=lambda kv: (-kv[1], str(kv[0])),
|
| 122 |
+
)[:top_edit]
|
| 123 |
+
edit_to_id: dict[str, int] = {PAD: 0, OOV: 1}
|
| 124 |
+
edit_scripts: list[tuple[int, str]] = [(-1, ""), (-1, "")] # placeholders for reserved
|
| 125 |
+
for script, _ in edit_items_sorted:
|
| 126 |
+
key = _script_key(script)
|
| 127 |
+
if key in edit_to_id:
|
| 128 |
+
continue
|
| 129 |
+
edit_to_id[key] = len(edit_to_id)
|
| 130 |
+
edit_scripts.append(script)
|
| 131 |
+
return cls(upos_to_id, deprel_to_id, feat_to_id, edit_to_id, edit_scripts)
|
| 132 |
+
|
| 133 |
+
# ----------------------------- encode ------------------------------------------------------
|
| 134 |
+
def encode_upos(self, upos: str) -> int:
|
| 135 |
+
return self.upos_to_id.get(upos, self.upos_to_id[OOV])
|
| 136 |
+
|
| 137 |
+
def encode_deprel(self, deprel: str) -> int:
|
| 138 |
+
return self.deprel_to_id.get(deprel, self.deprel_to_id[OOV])
|
| 139 |
+
|
| 140 |
+
def encode_feats_multihot(self, feats) -> list[int]:
|
| 141 |
+
"""Accepts either CoNLL-U style 'Case=Nom|Number=Sing' strings or
|
| 142 |
+
dict[str,str]. Empty/None → zero vector."""
|
| 143 |
+
vec = [0] * self.n_feat
|
| 144 |
+
if not feats:
|
| 145 |
+
return vec
|
| 146 |
+
if isinstance(feats, dict):
|
| 147 |
+
items = (f"{k}={v}" for k, v in feats.items())
|
| 148 |
+
else:
|
| 149 |
+
items = feats.split("|")
|
| 150 |
+
for kv in items:
|
| 151 |
+
i = self.feat_to_id.get(kv)
|
| 152 |
+
if i is not None:
|
| 153 |
+
vec[i] = 1
|
| 154 |
+
return vec
|
| 155 |
+
|
| 156 |
+
def encode_edit_script(self, form: str, lemma: str) -> int:
|
| 157 |
+
script = extract_edit_script(form, lemma)
|
| 158 |
+
return self.edit_to_id.get(_script_key(script), self.edit_to_id[OOV])
|
| 159 |
+
|
| 160 |
+
# ----------------------------- save / load -------------------------------------------------
|
| 161 |
+
def save(self, path: str) -> None:
|
| 162 |
+
os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
|
| 163 |
+
blob = {
|
| 164 |
+
"upos_to_id": self.upos_to_id,
|
| 165 |
+
"deprel_to_id": self.deprel_to_id,
|
| 166 |
+
"feat_to_id": self.feat_to_id,
|
| 167 |
+
"edit_to_id": self.edit_to_id,
|
| 168 |
+
# store edit_scripts as list of [delete_n, append_s] for JSON
|
| 169 |
+
"edit_scripts": [[d, a] for (d, a) in self.edit_scripts],
|
| 170 |
+
}
|
| 171 |
+
with open(path, "w", encoding="utf-8") as fh:
|
| 172 |
+
json.dump(blob, fh, ensure_ascii=False, indent=2)
|
| 173 |
+
|
| 174 |
+
@classmethod
|
| 175 |
+
def load(cls, path: str) -> "LabelVocab":
|
| 176 |
+
with open(path, encoding="utf-8") as fh:
|
| 177 |
+
blob = json.load(fh)
|
| 178 |
+
edit_scripts = [(int(d), str(a)) for (d, a) in blob["edit_scripts"]]
|
| 179 |
+
return cls(
|
| 180 |
+
upos_to_id=blob["upos_to_id"],
|
| 181 |
+
deprel_to_id=blob["deprel_to_id"],
|
| 182 |
+
feat_to_id=blob["feat_to_id"],
|
| 183 |
+
edit_to_id=blob["edit_to_id"],
|
| 184 |
+
edit_scripts=edit_scripts,
|
| 185 |
+
)
|
| 186 |
+
|
| 187 |
+
|
| 188 |
+
def _script_key(script: tuple[int, str]) -> str:
|
| 189 |
+
"""Canonical string form of an edit-script for use as a dict key. Uses ':' as a separator
|
| 190 |
+
because lemmas can contain almost anything but rarely contain colons in initial position."""
|
| 191 |
+
delete_n, append_s = script
|
| 192 |
+
return f"{delete_n}:{append_s}"
|
juurio/parser/model.py
ADDED
|
@@ -0,0 +1,714 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Tiny distilled Finnish UD parser model.
|
| 2 |
+
|
| 3 |
+
Architecture rationale:
|
| 4 |
+
- A top-1000 edit-script vocabulary covers nearly all UD-TDT-dev lemmas, so
|
| 5 |
+
lemmatization is a classifier head over edit scripts.
|
| 6 |
+
- Multiword-token incidence is low, so there is no MWT head; instead MWTs are
|
| 7 |
+
detected and flagged at parse time.
|
| 8 |
+
- A rule-based tokenizer already reaches high F1, so there is no tokenizer head.
|
| 9 |
+
|
| 10 |
+
**Encoder-only transformer + four task heads, all reading the same per-token hidden state:**
|
| 11 |
+
|
| 12 |
+
token-ids ──BPE-embedded─┐
|
| 13 |
+
├──TransformerEncoder──> h (B, T, d)
|
| 14 |
+
pos-embed ────────────────┘ │
|
| 15 |
+
│
|
| 16 |
+
┌────────────────────────────────────────────────────┤
|
| 17 |
+
│ │ │ │
|
| 18 |
+
UPOS Feats Lemma Parser (biaffine)
|
| 19 |
+
Linear(d, 17) Linear(d, ~80) Linear(d,1000) head_mlp(d) → 200
|
| 20 |
+
dep_mlp(d) → 200
|
| 21 |
+
arc: dep · W_arc · head.T → (B,T,T)
|
| 22 |
+
rel: einsum bilinear → (B,T,T,40)
|
| 23 |
+
|
| 24 |
+
The success criteria (small parameter budget, low CPU latency per sentence, calibrated
|
| 25 |
+
confidence) are *post-training* properties; this file just lays down the architecture so the
|
| 26 |
+
training loop (next module) has a public surface to depend on. Shapes / padding contract are
|
| 27 |
+
nailed by `tests/test_tinyfiud.py`.
|
| 28 |
+
|
| 29 |
+
**Calling convention (must be respected by the data pipeline):**
|
| 30 |
+
- `ids` is a `(B, T)` long tensor; `ids[b, 0] == ROOT_ID` is the virtual UD root token. UD's
|
| 31 |
+
`head=0` (sentence root) maps to this position natively. The data pipeline prepends ROOT
|
| 32 |
+
before tokenizing the rest of the sentence.
|
| 33 |
+
- `mask` is `(B, T)` bool — True where the position holds a real token (incl. ROOT), False
|
| 34 |
+
where it's padding. The model uses `~mask` as `src_key_padding_mask` for the encoder and
|
| 35 |
+
masks padded HEAD positions in the arc/rel scores to `-inf` (the model owns the padding
|
| 36 |
+
contract — without this, MST/argmax could attach a dependent to a pad position).
|
| 37 |
+
|
| 38 |
+
**Architecture knobs to revisit if metrics miss target:**
|
| 39 |
+
- `head_mlp` and `dep_mlp` are shared between the arc and rel scorers. Standard Dozat-Manning
|
| 40 |
+
uses separate projections per task. If relation accuracy lags arc accuracy, splitting the
|
| 41 |
+
MLPs is the first architecture knob to try (adds ~80 K params).
|
| 42 |
+
"""
|
| 43 |
+
from __future__ import annotations
|
| 44 |
+
|
| 45 |
+
import torch
|
| 46 |
+
import torch.nn as nn
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
# Reserved special token ids. The data pipeline (tokenizer + collator) must:
|
| 50 |
+
# - Use PAD_ID for padding.
|
| 51 |
+
# - Prepend a single ROOT_ID token at position 0 of every sentence (the virtual root for UD
|
| 52 |
+
# head selection). UD's head=0 ("root of sentence") then maps to this position natively.
|
| 53 |
+
PAD_ID = 0
|
| 54 |
+
ROOT_ID = 1
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
def per_token_confidence(logits: torch.Tensor) -> torch.Tensor:
|
| 58 |
+
"""Per-token (top1 − top2) softmax margin — the same confidence signal the realizers use.
|
| 59 |
+
`logits` is `(..., n_classes)`; the return is `(...)` with one scalar per position.
|
| 60 |
+
|
| 61 |
+
Top-1 minus top-2 (not top-1 alone) because it correctly captures *competition*: a 0.51-vs-0.49
|
| 62 |
+
decision is uncertain even though top-1 is 0.51. A 0.95-vs-0.02 decision is confident."""
|
| 63 |
+
probs = torch.softmax(logits, dim=-1)
|
| 64 |
+
top2 = probs.topk(2, dim=-1).values
|
| 65 |
+
return top2[..., 0] - top2[..., 1]
|
| 66 |
+
|
| 67 |
+
|
| 68 |
+
class TinyFiUD(nn.Module):
|
| 69 |
+
"""Tiny encoder-only transformer + four UD task heads.
|
| 70 |
+
|
| 71 |
+
Constructor args (all keyword):
|
| 72 |
+
vocab : BPE vocabulary size (target 8000)
|
| 73 |
+
d_model : hidden dim (target 256)
|
| 74 |
+
layers : encoder depth (target 6)
|
| 75 |
+
heads : attention heads (target 4)
|
| 76 |
+
ff : FFN inner dim (target 1024)
|
| 77 |
+
n_upos : number of UPOS labels (17)
|
| 78 |
+
n_feat : number of binary feature key=values for multi-label head (~80)
|
| 79 |
+
n_edit : edit-script vocabulary size (target 1000)
|
| 80 |
+
n_rel : number of dependency relation labels (~40)
|
| 81 |
+
maxlen : max sequence length the position embedding supports
|
| 82 |
+
dropout : transformer / MLP dropout
|
| 83 |
+
"""
|
| 84 |
+
|
| 85 |
+
def __init__(
|
| 86 |
+
self,
|
| 87 |
+
vocab: int = 8000,
|
| 88 |
+
d_model: int = 256,
|
| 89 |
+
layers: int = 6,
|
| 90 |
+
heads: int = 4,
|
| 91 |
+
ff: int = 1024,
|
| 92 |
+
n_upos: int = 17,
|
| 93 |
+
n_feat: int = 80,
|
| 94 |
+
n_edit: int = 1000,
|
| 95 |
+
n_rel: int = 40,
|
| 96 |
+
maxlen: int = 512,
|
| 97 |
+
dropout: float = 0.1,
|
| 98 |
+
parser_bilstm_hidden: int = 0,
|
| 99 |
+
arc_d: int = 200,
|
| 100 |
+
arc_distance_bias: bool = False,
|
| 101 |
+
fst_tag_vocab_size: int = 0,
|
| 102 |
+
fst_tag_dim: int = 32,
|
| 103 |
+
compound_enabled: bool = False,
|
| 104 |
+
char_vocab_size: int = 0,
|
| 105 |
+
char_emb_dim: int = 16,
|
| 106 |
+
char_conv_channels: int = 64,
|
| 107 |
+
max_form_len: int = 64,
|
| 108 |
+
compound_detach_encoder: bool = False,
|
| 109 |
+
parser_private_hidden: int = 0,
|
| 110 |
+
# Optional projection of the final encoder hidden (`d_model`) up to
|
| 111 |
+
# an external hidden dim, used only as a training-time auxiliary
|
| 112 |
+
# target. When True, the forward emits `out["hidden"]` and
|
| 113 |
+
# `out["hidden_proj"]`. Default OFF preserves the base forward + ckpt
|
| 114 |
+
# shape bit-identically.
|
| 115 |
+
use_hidden_kd_proj: bool = False,
|
| 116 |
+
hidden_kd_proj_dim: int = 768,
|
| 117 |
+
# Per-key biaffine Feats head replacing flat BCE. When
|
| 118 |
+
# `use_per_key_feats=True`, the model instantiates
|
| 119 |
+
# `self.feats_per_key_head` and the forward path emits
|
| 120 |
+
# `out["feats_per_key"]` (list of per-key logits) in addition
|
| 121 |
+
# to the existing `out["feats"]`. The legacy BCE `self.feats`
|
| 122 |
+
# Linear stays in place so older ckpts load 1:1; the freeze
|
| 123 |
+
# utility silences it during per-key training.
|
| 124 |
+
use_per_key_feats: bool = False,
|
| 125 |
+
feats_per_key_value_counts: list[int] | None = None,
|
| 126 |
+
feats_per_key_upos_emb_dim: int = 32,
|
| 127 |
+
feats_per_key_dropout: float = 0.0,
|
| 128 |
+
# Per-key value index → flat label_vocab.feat_to_id
|
| 129 |
+
# for the eval-time decode round-trip. Length == n_keys; row k has
|
| 130 |
+
# `value_counts[k]` entries (-1 at index 0 for `<EMPTY>`, real flat
|
| 131 |
+
# ids elsewhere). Built once from `FeatsKeySplit.flat_id_for_kv`.
|
| 132 |
+
# When None and use_per_key_feats=True, the forward path emits
|
| 133 |
+
# `feats_per_key` but does NOT override `out["feats"]` — useful for
|
| 134 |
+
# training-only configurations; eval requires the table.
|
| 135 |
+
feats_per_key_flat_ids: list[list[int]] | None = None,
|
| 136 |
+
# Optional near-identity capacity adapter stacked on top of the
|
| 137 |
+
# encoder. At init it is bitwise identity, so forward outputs
|
| 138 |
+
# match the un-adapted model within float32 rounding. When False
|
| 139 |
+
# (default), no adapter is built; older ckpts load 1:1 with no
|
| 140 |
+
# extra state_dict keys.
|
| 141 |
+
use_near_identity_adapter: bool = False,
|
| 142 |
+
near_identity_n_layers: int = 0,
|
| 143 |
+
near_identity_d_model: int | None = None,
|
| 144 |
+
near_identity_heads: int | None = None,
|
| 145 |
+
near_identity_ff: int | None = None,
|
| 146 |
+
):
|
| 147 |
+
super().__init__()
|
| 148 |
+
self.maxlen = maxlen
|
| 149 |
+
self.compound_enabled = compound_enabled
|
| 150 |
+
# When True, the compound branch consumes `h.detach()` instead
|
| 151 |
+
# of `h`. Together with the freeze whitelist in
|
| 152 |
+
# `juurio.parser.freeze`, this blocks the compound losses from
|
| 153 |
+
# re-shifting the shared encoder. Default False keeps the
|
| 154 |
+
# encoder shared with the compound branch.
|
| 155 |
+
self.compound_detach_encoder = compound_detach_encoder
|
| 156 |
+
self.max_form_len = max_form_len
|
| 157 |
+
self.n_edit = n_edit
|
| 158 |
+
# ---------- backbone ----------
|
| 159 |
+
self.tok_emb = nn.Embedding(vocab, d_model, padding_idx=0)
|
| 160 |
+
self.pos_emb = nn.Embedding(maxlen, d_model)
|
| 161 |
+
# ---------- FST-candidate input channel ----------
|
| 162 |
+
# Bag-of-tags morphtag input from uralicNLP analyses. Per first-subw
|
| 163 |
+
# position the collator emits a (max_tags,) int sequence of tag ids; we
|
| 164 |
+
# embed each, sum (pad embedding is constrained to zero so padded slots
|
| 165 |
+
# contribute nothing), and project through a zero-init linear so the
|
| 166 |
+
# encoder's output is bit-identical at initialization.
|
| 167 |
+
# Once gradients shape the projection the channel injects morphological
|
| 168 |
+
# disambiguation signal into the encoder, helping UPOS / Feats / Lemma /
|
| 169 |
+
# and indirectly LAS via better Case/Number agreement on long arcs.
|
| 170 |
+
# See `juurio/parser/fst_cache.py` for the cache + vocab format.
|
| 171 |
+
self.fst_tag_vocab_size = fst_tag_vocab_size
|
| 172 |
+
if fst_tag_vocab_size > 0:
|
| 173 |
+
self.fst_tag_emb = nn.Embedding(fst_tag_vocab_size, fst_tag_dim, padding_idx=0)
|
| 174 |
+
# zero PAD row explicitly (Embedding padding_idx zeros gradients but the
|
| 175 |
+
# initial values are random; force zero here so the bag sum at init is 0)
|
| 176 |
+
with torch.no_grad():
|
| 177 |
+
self.fst_tag_emb.weight[0].zero_()
|
| 178 |
+
self.fst_proj = nn.Linear(fst_tag_dim, d_model, bias=False)
|
| 179 |
+
nn.init.zeros_(self.fst_proj.weight) # zero-init → identity at start
|
| 180 |
+
else:
|
| 181 |
+
self.fst_tag_emb = None
|
| 182 |
+
self.fst_proj = None
|
| 183 |
+
self.encoder = nn.TransformerEncoder(
|
| 184 |
+
nn.TransformerEncoderLayer(d_model, heads, ff, dropout, batch_first=True),
|
| 185 |
+
num_layers=layers,
|
| 186 |
+
)
|
| 187 |
+
# Optional near-identity adapter on top of the encoder. None when
|
| 188 |
+
# disabled — keeps older ckpts byte-stable (no extra state_dict
|
| 189 |
+
# keys). When enabled, the adapter is `n_layers` near-identity
|
| 190 |
+
# blocks; at init each block is bit-for-bit identity so the adapted
|
| 191 |
+
# forward equals the un-adapted forward within float32 rounding.
|
| 192 |
+
self.near_identity_adapter = None
|
| 193 |
+
if use_near_identity_adapter and near_identity_n_layers > 0:
|
| 194 |
+
from .near_identity import NearIdentityAdapter
|
| 195 |
+
self.near_identity_adapter = NearIdentityAdapter(
|
| 196 |
+
d_model=(near_identity_d_model or d_model),
|
| 197 |
+
heads=(near_identity_heads or heads),
|
| 198 |
+
ff=(near_identity_ff or ff),
|
| 199 |
+
dropout=dropout,
|
| 200 |
+
n_layers=near_identity_n_layers,
|
| 201 |
+
)
|
| 202 |
+
# ---------- per-token task heads ----------
|
| 203 |
+
self.upos = nn.Linear(d_model, n_upos)
|
| 204 |
+
self.feats = nn.Linear(d_model, n_feat)
|
| 205 |
+
# Hidden KD projection. Built only when use_hidden_kd_proj;
|
| 206 |
+
# default None preserves the base ckpt shape.
|
| 207 |
+
self.hidden_kd_proj: nn.Linear | None = None
|
| 208 |
+
self._hidden_kd_proj_dim: int = hidden_kd_proj_dim
|
| 209 |
+
if use_hidden_kd_proj:
|
| 210 |
+
self.hidden_kd_proj = nn.Linear(d_model, hidden_kd_proj_dim)
|
| 211 |
+
# Feats-private adapter is not built at serving time; the
|
| 212 |
+
# Feats head consumes the encoder output directly. Kept as None so
|
| 213 |
+
# the forward path's `self.feats_adapter is not None` guard is a
|
| 214 |
+
# no-op and older checkpoints load unchanged (no extra keys).
|
| 215 |
+
self.feats_adapter = None
|
| 216 |
+
# Optional per-key biaffine Feats head.
|
| 217 |
+
# None when disabled (byte-stable; no extra
|
| 218 |
+
# state_dict keys). Builder validates value_counts presence
|
| 219 |
+
# and forbids the zero-key edge case.
|
| 220 |
+
self.feats_per_key_head = None
|
| 221 |
+
self.use_per_key_feats = use_per_key_feats
|
| 222 |
+
if use_per_key_feats:
|
| 223 |
+
if not feats_per_key_value_counts:
|
| 224 |
+
raise ValueError(
|
| 225 |
+
"use_per_key_feats=True requires `feats_per_key_value_counts` "
|
| 226 |
+
"(per-key value counts including the <EMPTY> class at 0). "
|
| 227 |
+
"Build via `FeatsKeySplit.from_label_vocab(vocab).value_counts`."
|
| 228 |
+
)
|
| 229 |
+
from .feats_per_key_head import PerKeyFeatsHead
|
| 230 |
+
self.feats_per_key_head = PerKeyFeatsHead(
|
| 231 |
+
d_model=d_model,
|
| 232 |
+
upos_emb_dim=feats_per_key_upos_emb_dim,
|
| 233 |
+
value_counts=list(feats_per_key_value_counts),
|
| 234 |
+
n_upos=n_upos,
|
| 235 |
+
dropout=feats_per_key_dropout,
|
| 236 |
+
)
|
| 237 |
+
# Build a padded lookup buffer (n_keys, max_n_values) of flat
|
| 238 |
+
# feat_to_id ids; entries past the per-key value count and
|
| 239 |
+
# `<EMPTY>` slots are -1 (no bit set). Used in forward to
|
| 240 |
+
# decode per-key argmax → multi-hot at eval time.
|
| 241 |
+
if feats_per_key_flat_ids is not None:
|
| 242 |
+
if len(feats_per_key_flat_ids) != len(feats_per_key_value_counts):
|
| 243 |
+
raise ValueError(
|
| 244 |
+
"feats_per_key_flat_ids length must equal value_counts length"
|
| 245 |
+
)
|
| 246 |
+
max_v = max(feats_per_key_value_counts)
|
| 247 |
+
lookup = torch.full((len(feats_per_key_value_counts), max_v),
|
| 248 |
+
-1, dtype=torch.long)
|
| 249 |
+
for k_idx, row in enumerate(feats_per_key_flat_ids):
|
| 250 |
+
n_v = feats_per_key_value_counts[k_idx]
|
| 251 |
+
# Accept rows that match the key's exact value count OR
|
| 252 |
+
# the padded max_v width (ckpt round-trip serializes the
|
| 253 |
+
# padded buffer). Trailing -1 entries are ignored.
|
| 254 |
+
if len(row) not in (n_v, max_v):
|
| 255 |
+
raise ValueError(
|
| 256 |
+
f"feats_per_key_flat_ids[{k_idx}] length {len(row)} "
|
| 257 |
+
f"!= value_counts[{k_idx}] {n_v} or max_v {max_v}"
|
| 258 |
+
)
|
| 259 |
+
for v_idx in range(n_v):
|
| 260 |
+
lookup[k_idx, v_idx] = row[v_idx]
|
| 261 |
+
self.register_buffer("_feats_per_key_flat_lookup", lookup,
|
| 262 |
+
persistent=False)
|
| 263 |
+
else:
|
| 264 |
+
self.register_buffer("_feats_per_key_flat_lookup", None,
|
| 265 |
+
persistent=False)
|
| 266 |
+
self.lemma_edit = nn.Linear(d_model, n_edit)
|
| 267 |
+
# ---------- optional BiLSTM in the parser path ----------
|
| 268 |
+
# Rationale: in long sentences, attachment accuracy degrades with arc
|
| 269 |
+
# distance — the transformer's global self-attention does not carry
|
| 270 |
+
# enough syntactic structure across long contexts for the biaffine
|
| 271 |
+
# scorer. A 1-layer BiLSTM placed before the arc/rel heads gives the
|
| 272 |
+
# scorer an explicit left-to-right + right-to-left summary. We do NOT
|
| 273 |
+
# feed this into upos/feats/lemma heads — they already score well and
|
| 274 |
+
# the added module shouldn't perturb a strong signal. Hidden=d_model/2
|
| 275 |
+
# so the bidirectional output is d_model again and feeds in via a clean
|
| 276 |
+
# residual.
|
| 277 |
+
#
|
| 278 |
+
# `parser_bilstm_hidden` is the actual LSTM hidden dim (not just a bool),
|
| 279 |
+
# and the forward path uses pack_padded_sequence so pad positions never
|
| 280 |
+
# leak into the backward direction.
|
| 281 |
+
if parser_bilstm_hidden > 0:
|
| 282 |
+
assert parser_bilstm_hidden * 2 == d_model, (
|
| 283 |
+
f"parser_bilstm_hidden={parser_bilstm_hidden} must equal d_model/2 "
|
| 284 |
+
f"({d_model // 2}) so the bidirectional output dim matches d_model "
|
| 285 |
+
f"and the residual is clean.")
|
| 286 |
+
self.parser_bilstm = nn.LSTM(
|
| 287 |
+
d_model, parser_bilstm_hidden,
|
| 288 |
+
num_layers=1, bidirectional=True, batch_first=True, dropout=0.0,
|
| 289 |
+
)
|
| 290 |
+
else:
|
| 291 |
+
self.parser_bilstm = None
|
| 292 |
+
# ---------- parser-private FF residual block ----------
|
| 293 |
+
# Residual FF inserted between the encoder output and the parser
|
| 294 |
+
# branch (bilstm → head_mlp/dep_mlp/biaffine). The non-parser heads
|
| 295 |
+
# (upos, feats, lemma_edit) still consume the raw encoder output `h`.
|
| 296 |
+
#
|
| 297 |
+
# Shape: d_model → parser_private_hidden → d_model. GELU + dropout
|
| 298 |
+
# between. No LayerNorm.
|
| 299 |
+
#
|
| 300 |
+
# Init: linear1 default Xavier; linear2 weight + bias both ZERO so
|
| 301 |
+
# parser_private(h) = 0 at step 0 → h_for_parser = h + 0 = h.
|
| 302 |
+
# Forward output bit-identical to a no-private model at init.
|
| 303 |
+
#
|
| 304 |
+
# Wake-up induction (2 steps): linear2 wakes at step 1 (intermediate
|
| 305 |
+
# = GELU(linear1(h)) is non-zero from random linear1 init). After
|
| 306 |
+
# an AdamW step linear2.weight is non-zero, then linear1 wakes at
|
| 307 |
+
# the next step. See tests/test_parser_private.py for the regression.
|
| 308 |
+
#
|
| 309 |
+
# Diagnostic: log ||self.parser_private_linear2.weight|| at eval
|
| 310 |
+
# time to tell whether the block failed to wake or woke but did
|
| 311 |
+
# not help.
|
| 312 |
+
self.parser_private_hidden = parser_private_hidden
|
| 313 |
+
if parser_private_hidden > 0:
|
| 314 |
+
self.parser_private_linear1 = nn.Linear(d_model, parser_private_hidden)
|
| 315 |
+
self.parser_private_linear2 = nn.Linear(parser_private_hidden, d_model)
|
| 316 |
+
self.parser_private_drop = nn.Dropout(dropout)
|
| 317 |
+
with torch.no_grad():
|
| 318 |
+
self.parser_private_linear2.weight.zero_()
|
| 319 |
+
self.parser_private_linear2.bias.zero_()
|
| 320 |
+
else:
|
| 321 |
+
self.parser_private_linear1 = None
|
| 322 |
+
self.parser_private_linear2 = None
|
| 323 |
+
self.parser_private_drop = None
|
| 324 |
+
# ---------- biaffine parser ----------
|
| 325 |
+
# arc_d is configurable. Dozat-Manning's paper used 100. The W_rel cost
|
| 326 |
+
# (n_rel × arc_d²) is a large share of model params for negligible
|
| 327 |
+
# label-given-head gain, so a smaller arc_d is usually a good tradeoff.
|
| 328 |
+
# head_mlp / dep_mlp: per-token arc_d-dim projection (Dozat & Manning scale)
|
| 329 |
+
self.arc_d = arc_d
|
| 330 |
+
self.head_mlp = nn.Sequential(nn.Linear(d_model, arc_d), nn.GELU(), nn.Dropout(dropout))
|
| 331 |
+
self.dep_mlp = nn.Sequential(nn.Linear(d_model, arc_d), nn.GELU(), nn.Dropout(dropout))
|
| 332 |
+
# arc bilinear: arc_score[b, dep, head] = dep_repr_b_dep · W_arc · head_repr_b_head + bias
|
| 333 |
+
self.W_arc = nn.Parameter(torch.empty(arc_d, arc_d))
|
| 334 |
+
nn.init.xavier_uniform_(self.W_arc)
|
| 335 |
+
self.head_bias = nn.Linear(arc_d, 1, bias=False) # head-only bias term (Dozat-Manning)
|
| 336 |
+
# rel bilinear: rel_score[b, dep, head, r] = dep · W_rel[r] · head + linear terms
|
| 337 |
+
self.W_rel = nn.Parameter(torch.empty(n_rel, arc_d, arc_d))
|
| 338 |
+
nn.init.xavier_uniform_(self.W_rel)
|
| 339 |
+
self.rel_lin_h = nn.Linear(arc_d, n_rel, bias=False)
|
| 340 |
+
self.rel_lin_d = nn.Linear(arc_d, n_rel, bias=False)
|
| 341 |
+
self.rel_bias = nn.Parameter(torch.zeros(n_rel))
|
| 342 |
+
|
| 343 |
+
# ---------- bucketed arc-distance × direction bias ----------
|
| 344 |
+
# In long sentences the model can label arcs well but struggles to find
|
| 345 |
+
# the right head at long distances. Add a tiny learned bias term that
|
| 346 |
+
# depends on signed bucketed (head_pos - dep_pos) — 11 scalars total
|
| 347 |
+
# (5 distance buckets × 2 directions + 1 root). Precompute the
|
| 348 |
+
# bucket-index table once as a buffer; in forward we just `[]`-index.
|
| 349 |
+
self.arc_distance_bias = arc_distance_bias
|
| 350 |
+
if arc_distance_bias:
|
| 351 |
+
# 11 buckets: 0=root, 1-5=left (head before dep), 6-10=right (head after dep)
|
| 352 |
+
# bucket boundaries match the eval bins: dist in {1, 2-3, 4-7, 8-15, 16+}
|
| 353 |
+
T = maxlen
|
| 354 |
+
idx = torch.zeros(T, T, dtype=torch.long)
|
| 355 |
+
for d in range(T):
|
| 356 |
+
for hp in range(T):
|
| 357 |
+
if hp == 0:
|
| 358 |
+
idx[d, hp] = 0 # root
|
| 359 |
+
continue
|
| 360 |
+
signed = hp - d
|
| 361 |
+
abs_d = abs(signed)
|
| 362 |
+
if abs_d == 1:
|
| 363 |
+
b = 0
|
| 364 |
+
elif abs_d <= 3:
|
| 365 |
+
b = 1
|
| 366 |
+
elif abs_d <= 7:
|
| 367 |
+
b = 2
|
| 368 |
+
elif abs_d <= 15:
|
| 369 |
+
b = 3
|
| 370 |
+
else:
|
| 371 |
+
b = 4
|
| 372 |
+
idx[d, hp] = 1 + b + (5 if signed > 0 else 0)
|
| 373 |
+
self.register_buffer("_arc_dist_idx", idx)
|
| 374 |
+
self.arc_dist_bias = nn.Parameter(torch.zeros(11))
|
| 375 |
+
|
| 376 |
+
# ---------- compound-lemma heads ----------
|
| 377 |
+
# Three heads attached to the per-token encoder hidden state (no new
|
| 378 |
+
# backbone params). When `compound_enabled=False` (default), this is a
|
| 379 |
+
# no-op — the forward dict stays surface-compatible.
|
| 380 |
+
#
|
| 381 |
+
# 1. count head: Linear(d_model, 4) → CE over {0, 1, 2, 3+} `#`s.
|
| 382 |
+
# 2. split scorer: char-level CNN over the token's form characters,
|
| 383 |
+
# FiLM-conditioned on `h_token`. Emits a per-char logit; decode
|
| 384 |
+
# picks the top-K legal positions.
|
| 385 |
+
# 3. tail-script head: Linear(d_model, n_edit). Separate from
|
| 386 |
+
# `lemma_edit` so the two heads can specialise (whole-form vs
|
| 387 |
+
# form-tail edit script).
|
| 388 |
+
#
|
| 389 |
+
# The char vocab includes PAD=0 and UNK=1; PAD-row embedding is
|
| 390 |
+
# constrained to zero so padded char slots inject no signal through
|
| 391 |
+
# the conv (same trick used for FST PAD).
|
| 392 |
+
if compound_enabled:
|
| 393 |
+
if char_vocab_size < 2:
|
| 394 |
+
raise ValueError(
|
| 395 |
+
f"compound_enabled requires char_vocab_size>=2 (got {char_vocab_size}); "
|
| 396 |
+
f"reserve PAD=0 and UNK=1 at minimum.")
|
| 397 |
+
self.char_emb = nn.Embedding(char_vocab_size, char_emb_dim, padding_idx=0)
|
| 398 |
+
with torch.no_grad():
|
| 399 |
+
self.char_emb.weight[0].zero_()
|
| 400 |
+
# 1D conv across char positions; pad to keep length=max_form_len.
|
| 401 |
+
self.compound_split_conv = nn.Conv1d(
|
| 402 |
+
in_channels=char_emb_dim,
|
| 403 |
+
out_channels=char_conv_channels,
|
| 404 |
+
kernel_size=3, padding=1, bias=True,
|
| 405 |
+
)
|
| 406 |
+
# FiLM: project h_token to (γ, β) of size char_conv_channels each.
|
| 407 |
+
# Init γ=1, β=0 so the modulation is identity at start of training.
|
| 408 |
+
# Zero the weights and set bias so the first `char_conv_channels`
|
| 409 |
+
# outputs (γ) are 1 and the second half (β) are 0.
|
| 410 |
+
self.compound_film = nn.Linear(d_model, 2 * char_conv_channels, bias=True)
|
| 411 |
+
with torch.no_grad():
|
| 412 |
+
self.compound_film.weight.zero_()
|
| 413 |
+
self.compound_film.bias[:char_conv_channels].fill_(1.0) # γ ← 1
|
| 414 |
+
self.compound_film.bias[char_conv_channels:].zero_() # β ← 0
|
| 415 |
+
# Per-position scalar logit on top of FiLM-modulated channels.
|
| 416 |
+
self.compound_split_out = nn.Linear(char_conv_channels, 1, bias=True)
|
| 417 |
+
# Count head + tail-script head read straight from h_token.
|
| 418 |
+
self.compound_count = nn.Linear(d_model, 4, bias=True)
|
| 419 |
+
self.compound_tail = nn.Linear(d_model, n_edit, bias=True)
|
| 420 |
+
else:
|
| 421 |
+
self.char_emb = None
|
| 422 |
+
self.compound_split_conv = None
|
| 423 |
+
self.compound_film = None
|
| 424 |
+
self.compound_split_out = None
|
| 425 |
+
self.compound_count = None
|
| 426 |
+
self.compound_tail = None
|
| 427 |
+
|
| 428 |
+
def _encode(self, ids: torch.Tensor, mask: torch.Tensor,
|
| 429 |
+
fst_tags: torch.Tensor | None = None) -> torch.Tensor:
|
| 430 |
+
"""ids: (B, T) longs; mask: (B, T) bools (True = real token).
|
| 431 |
+
fst_tags: (B, T, max_tags) longs of morphtag ids per first-subword position
|
| 432 |
+
(zeros elsewhere, which is the PAD id so they contribute nothing).
|
| 433 |
+
Returns (B, T, d)."""
|
| 434 |
+
B, T = ids.shape
|
| 435 |
+
if T > self.maxlen:
|
| 436 |
+
raise ValueError(f"sequence length {T} exceeds maxlen {self.maxlen}")
|
| 437 |
+
positions = torch.arange(T, device=ids.device).unsqueeze(0).expand(B, T)
|
| 438 |
+
x = self.tok_emb(ids) + self.pos_emb(positions)
|
| 439 |
+
# FST channel: zero-init at start so this is a no-op until training shapes
|
| 440 |
+
# `fst_proj`. After init it injects per-token morphtag bag info at every
|
| 441 |
+
# position whose `fst_tags` row contains non-PAD ids.
|
| 442 |
+
if self.fst_tag_emb is not None and fst_tags is not None:
|
| 443 |
+
tag_emb = self.fst_tag_emb(fst_tags).sum(dim=2) # (B, T, fst_tag_dim)
|
| 444 |
+
x = x + self.fst_proj(tag_emb)
|
| 445 |
+
# nn.TransformerEncoder uses `src_key_padding_mask` where True = should be IGNORED
|
| 446 |
+
# (i.e. position is padding). Our `mask` is True for real tokens, so invert.
|
| 447 |
+
h = self.encoder(x, src_key_padding_mask=~mask)
|
| 448 |
+
# Optional near-identity adapter. At init this is bitwise identity
|
| 449 |
+
# on real-token positions, so the adapted forward equals the
|
| 450 |
+
# un-adapted forward within float32 rounding.
|
| 451 |
+
if self.near_identity_adapter is not None:
|
| 452 |
+
h = self.near_identity_adapter(h, mask=mask)
|
| 453 |
+
return h
|
| 454 |
+
|
| 455 |
+
def forward(self, ids: torch.Tensor, mask: torch.Tensor,
|
| 456 |
+
fst_tags: torch.Tensor | None = None,
|
| 457 |
+
form_chars: torch.Tensor | None = None,
|
| 458 |
+
form_char_mask: torch.Tensor | None = None,
|
| 459 |
+
) -> dict[str, torch.Tensor]:
|
| 460 |
+
"""Returns a dict with logits for each head:
|
| 461 |
+
upos : (B, T, n_upos)
|
| 462 |
+
feats : (B, T, n_feat)
|
| 463 |
+
lemma : (B, T, n_edit)
|
| 464 |
+
arc : (B, T, T) — arc[b, dep, head] is the score for `head` being the head of `dep`
|
| 465 |
+
rel : (B, T, T, n_rel) — rel[b, dep, head, r] is the score for relation r on that arc
|
| 466 |
+
|
| 467 |
+
When `compound_enabled`, three additional keys are returned:
|
| 468 |
+
count : (B, T, 4) — CE logits over {0, 1, 2, 3+} `#`s
|
| 469 |
+
split : (B, T, max_form_len) — per-char-pos boundary logits
|
| 470 |
+
tail : (B, T, n_edit) — tail-script logits
|
| 471 |
+
|
| 472 |
+
`form_chars` (B, T, max_form_len) and `form_char_mask` (B, T,
|
| 473 |
+
max_form_len) are required when compound is enabled; a missing pair
|
| 474 |
+
raises ValueError (same guard pattern as the FST channel — silent
|
| 475 |
+
disabling at eval is a footgun we don't repeat).
|
| 476 |
+
"""
|
| 477 |
+
if self.compound_enabled and (form_chars is None or form_char_mask is None):
|
| 478 |
+
raise ValueError(
|
| 479 |
+
"compound_enabled=True but form_chars/form_char_mask were not "
|
| 480 |
+
"passed to forward(). Caller must provide both — silently disabling "
|
| 481 |
+
"the compound heads at eval would hide a real misconfiguration.")
|
| 482 |
+
h = self._encode(ids, mask, fst_tags=fst_tags)
|
| 483 |
+
# Parser-private FF residual block. Only the parser
|
| 484 |
+
# branch sees `h_for_parser`; the non-parser heads (upos, feats,
|
| 485 |
+
# lemma_edit) consume the raw encoder `h` further down. At step 0
|
| 486 |
+
# linear2.weight = 0 → parser_private(h) = 0 → h_for_parser = h
|
| 487 |
+
# (bit-identical-to-no-private forward). Once linear2 wakes,
|
| 488 |
+
# h_for_parser becomes a parser-private rewrite of h that the
|
| 489 |
+
# non-parser heads do NOT see.
|
| 490 |
+
if self.parser_private_linear1 is not None:
|
| 491 |
+
private = self.parser_private_linear2(
|
| 492 |
+
self.parser_private_drop(
|
| 493 |
+
torch.nn.functional.gelu(self.parser_private_linear1(h))))
|
| 494 |
+
h_for_parser = h + private
|
| 495 |
+
else:
|
| 496 |
+
h_for_parser = h
|
| 497 |
+
# Optional BiLSTM-augmented representation for the
|
| 498 |
+
# parser path only. The encoder's `src_key_padding_mask` only masks pads
|
| 499 |
+
# as keys/values, not as query outputs — pad positions still get embedded
|
| 500 |
+
# and have nonzero `h`. Without packing, the backward LSTM would read the
|
| 501 |
+
# pad suffix BEFORE real tokens, making real-token representations depend
|
| 502 |
+
# on batch padding. Pack via the real lengths so the LSTM never sees pads.
|
| 503 |
+
if self.parser_bilstm is not None:
|
| 504 |
+
lengths = mask.sum(dim=1).to("cpu", dtype=torch.long)
|
| 505 |
+
packed = nn.utils.rnn.pack_padded_sequence(
|
| 506 |
+
h_for_parser, lengths, batch_first=True, enforce_sorted=False)
|
| 507 |
+
packed_out, _ = self.parser_bilstm(packed)
|
| 508 |
+
h_lstm, _ = nn.utils.rnn.pad_packed_sequence(
|
| 509 |
+
packed_out, batch_first=True, total_length=h.size(1))
|
| 510 |
+
h_arc = h_for_parser + h_lstm # residual
|
| 511 |
+
else:
|
| 512 |
+
h_arc = h_for_parser
|
| 513 |
+
head_repr = self.head_mlp(h_arc) # (B, T, arc_d)
|
| 514 |
+
dep_repr = self.dep_mlp(h_arc) # (B, T, arc_d)
|
| 515 |
+
|
| 516 |
+
# arc_score[b, dep, head] = dep_repr_dep · W_arc · head_repr_head + head_bias(head_repr_head)
|
| 517 |
+
arc = torch.einsum("bdi,ij,bhj->bdh", dep_repr, self.W_arc, head_repr)
|
| 518 |
+
arc = arc + self.head_bias(head_repr).squeeze(-1).unsqueeze(1) # (B,1,T) broadcast over dep
|
| 519 |
+
# Optional bucketed arc-distance × direction bias targeting long-distance
|
| 520 |
+
# attachment errors. 11 learned scalars, sliced to current seq length and
|
| 521 |
+
# broadcast across batch.
|
| 522 |
+
if self.arc_distance_bias:
|
| 523 |
+
T = arc.shape[-1]
|
| 524 |
+
arc = arc + self.arc_dist_bias[self._arc_dist_idx[:T, :T]]
|
| 525 |
+
|
| 526 |
+
# rel_score[b, dep, head, r] = dep · W_rel[r] · head + lin_d(dep) + lin_h(head) + bias[r]
|
| 527 |
+
rel = torch.einsum("bdi,rij,bhj->bdhr", dep_repr, self.W_rel, head_repr)
|
| 528 |
+
rel = rel + self.rel_lin_d(dep_repr).unsqueeze(2) + self.rel_lin_h(head_repr).unsqueeze(1)
|
| 529 |
+
rel = rel + self.rel_bias
|
| 530 |
+
|
| 531 |
+
# Mask padded HEAD positions to -inf so MST/argmax can never attach to padding. Done inside
|
| 532 |
+
# the model (not the training loop) because the model owns the padding contract.
|
| 533 |
+
# mask shape (B, T) — True = real, False = pad. We broadcast over the dep axis.
|
| 534 |
+
pad_head_mask = ~mask # (B, T) True where head is pad
|
| 535 |
+
arc = arc.masked_fill(pad_head_mask.unsqueeze(1), float("-inf")) # (B, 1, T) → (B, T, T)
|
| 536 |
+
rel = rel.masked_fill(pad_head_mask.unsqueeze(1).unsqueeze(-1), float("-inf"))
|
| 537 |
+
# Mask the diagonal so a token can never be its own head.
|
| 538 |
+
# Without this, the arc loss has a phantom self-attachment class that
|
| 539 |
+
# routes gradients into the "left 2-3" arc_distance_bias bucket (since
|
| 540 |
+
# signed=0 falls into abs_d <= 3) and could win at argmax inference.
|
| 541 |
+
T_ = arc.shape[-1]
|
| 542 |
+
eye = torch.eye(T_, dtype=torch.bool, device=arc.device).unsqueeze(0) # (1, T, T)
|
| 543 |
+
arc = arc.masked_fill(eye, float("-inf"))
|
| 544 |
+
rel = rel.masked_fill(eye.unsqueeze(-1), float("-inf"))
|
| 545 |
+
|
| 546 |
+
upos_logits = self.upos(h)
|
| 547 |
+
out = {
|
| 548 |
+
"upos": upos_logits,
|
| 549 |
+
# Optional feats-private adapter. None → raw encoder hidden.
|
| 550 |
+
"feats": self.feats(
|
| 551 |
+
self.feats_adapter(h) if self.feats_adapter is not None else h
|
| 552 |
+
),
|
| 553 |
+
"lemma": self.lemma_edit(h),
|
| 554 |
+
"arc": arc,
|
| 555 |
+
"rel": rel,
|
| 556 |
+
}
|
| 557 |
+
# Expose the final encoder hidden + an auxiliary projection when the
|
| 558 |
+
# projection head is built. `out["hidden"]` is the raw (B, T, d_model)
|
| 559 |
+
# tensor; `out["hidden_proj"]` is its (B, T, proj_dim) projection, used
|
| 560 |
+
# only by the optional auxiliary training loss. Both keys exist
|
| 561 |
+
# together; serving/eval callers ignore them.
|
| 562 |
+
if self.hidden_kd_proj is not None:
|
| 563 |
+
out["hidden"] = h
|
| 564 |
+
out["hidden_proj"] = self.hidden_kd_proj(h)
|
| 565 |
+
# Emit per-key biaffine logits when the per-key head is active.
|
| 566 |
+
# UPOS argmax is detached so the per-key CE loss cannot leak
|
| 567 |
+
# gradient back through the (frozen) UPOS head — the head sees
|
| 568 |
+
# UPOS as a discrete frozen input at both train and eval.
|
| 569 |
+
if self.feats_per_key_head is not None:
|
| 570 |
+
with torch.no_grad():
|
| 571 |
+
upos_argmax = upos_logits.argmax(dim=-1) # (B, T)
|
| 572 |
+
per_key_logits = self.feats_per_key_head(h, upos_argmax)
|
| 573 |
+
out["feats_per_key"] = per_key_logits
|
| 574 |
+
# Eval-time round-trip: override out["feats"] with FAKE
|
| 575 |
+
# logits decoded from per-key argmax via the flat-id lookup.
|
| 576 |
+
# When the per-key head is trained with per-key CE (not BCE),
|
| 577 |
+
# the frozen BCE Linear's predictions would be garbage — but the
|
| 578 |
+
# existing eval pipeline reads out["feats"] via sigmoid>0.5.
|
| 579 |
+
# We synthesize logits where each "1" bit is +12 (sigmoid
|
| 580 |
+
# ≈ 0.99) and each "0" bit is -12 (sigmoid ≈ 0.01), so the
|
| 581 |
+
# threshold reproduces the per-key argmax multi-hot exactly.
|
| 582 |
+
# Fail-loud: at eval time (model.training=False), the lookup
|
| 583 |
+
# MUST be present — otherwise out["feats"] would silently be
|
| 584 |
+
# the frozen, untrained BCE Linear's predictions and Feats-F1
|
| 585 |
+
# would be garbage. Training is OK without lookup (loss path
|
| 586 |
+
# uses out["feats_per_key"] directly).
|
| 587 |
+
lookup = getattr(self, "_feats_per_key_flat_lookup", None)
|
| 588 |
+
if lookup is None and not self.training:
|
| 589 |
+
raise RuntimeError(
|
| 590 |
+
"TinyFiUD: per-key Feats head is active but the flat-id "
|
| 591 |
+
"lookup is missing — out['feats'] cannot be overridden. "
|
| 592 |
+
"Eval would read the (frozen, untrained-in-Stage-2) BCE "
|
| 593 |
+
"Linear's predictions, which are garbage. Pass "
|
| 594 |
+
"`feats_per_key_flat_ids` at construction (built from "
|
| 595 |
+
"`FeatsKeySplit.flat_id_for_kv`)."
|
| 596 |
+
)
|
| 597 |
+
if lookup is not None:
|
| 598 |
+
B, T, _ = h.shape
|
| 599 |
+
n_feat = out["feats"].shape[-1]
|
| 600 |
+
if self.training:
|
| 601 |
+
# DIFFERENTIABLE per-key → flat projection so a flat-bit
|
| 602 |
+
# auxiliary loss retains gradient through the per-key head.
|
| 603 |
+
# Without this, the eval-only fake ±12 path below would
|
| 604 |
+
# also be used at training time and that loss term would
|
| 605 |
+
# become a non-gradient constant.
|
| 606 |
+
#
|
| 607 |
+
# Projection rule:
|
| 608 |
+
# flat_logit(K=v) = per_key_logits[K][v] -
|
| 609 |
+
# per_key_logits[K][<EMPTY>]
|
| 610 |
+
# When warm-started per-key from BCE, the EMPTY col is
|
| 611 |
+
# zero and value cols equal the source BCE logits, so
|
| 612 |
+
# the margin reproduces source BCE logits byte-equally
|
| 613 |
+
# (Feats-F1 at step 0 ≈ source BCE Feats-F1). Unmapped
|
| 614 |
+
# flat positions keep the eval-path's -12 constant so
|
| 615 |
+
# sigmoid → 0. `lookup` is a (n_keys, max_V) tensor
|
| 616 |
+
# padded with -1 to the widest per-key value count.
|
| 617 |
+
# Slice each row by the actual V_k from per_key_logits
|
| 618 |
+
# so the margins and flat-id columns line up.
|
| 619 |
+
margins_per_key: list[torch.Tensor] = []
|
| 620 |
+
flat_ids_per_key: list[torch.Tensor] = []
|
| 621 |
+
for k_idx, key_logits in enumerate(per_key_logits):
|
| 622 |
+
n_v = key_logits.shape[-1]
|
| 623 |
+
empty_col = key_logits[..., 0:1] # (B, T, 1)
|
| 624 |
+
margins_k = key_logits[..., 1:n_v] - empty_col # (B, T, V_k - 1)
|
| 625 |
+
flat_ids_k = lookup[k_idx][1:n_v] # (V_k - 1,)
|
| 626 |
+
margins_per_key.append(margins_k)
|
| 627 |
+
flat_ids_per_key.append(flat_ids_k)
|
| 628 |
+
all_margins = torch.cat(margins_per_key, dim=-1) # (B, T, N_total)
|
| 629 |
+
all_flat_ids = torch.cat(flat_ids_per_key, dim=0) # (N_total,)
|
| 630 |
+
valid = all_flat_ids >= 0
|
| 631 |
+
if not bool(valid.all()):
|
| 632 |
+
valid_idx = valid.nonzero(as_tuple=True)[0]
|
| 633 |
+
all_margins = all_margins.index_select(-1, valid_idx)
|
| 634 |
+
all_flat_ids = all_flat_ids.index_select(0, valid_idx)
|
| 635 |
+
flat_base = per_key_logits[0].new_full(
|
| 636 |
+
(B, T, n_feat), -12.0
|
| 637 |
+
)
|
| 638 |
+
idx = all_flat_ids.view(1, 1, -1).expand(B, T, -1)
|
| 639 |
+
flat_logits = flat_base.scatter(-1, idx, all_margins)
|
| 640 |
+
out["feats"] = flat_logits
|
| 641 |
+
else:
|
| 642 |
+
# Eval-time: argmax → multi-hot → ±12 fake logits.
|
| 643 |
+
argmax_list = [t.argmax(dim=-1) for t in per_key_logits]
|
| 644 |
+
multi_hot = torch.zeros(B, T, n_feat,
|
| 645 |
+
dtype=torch.bool, device=h.device)
|
| 646 |
+
for k_idx, argmax_k in enumerate(argmax_list):
|
| 647 |
+
flat_ids_k = lookup[k_idx][argmax_k] # (B, T)
|
| 648 |
+
set_mask = flat_ids_k >= 0
|
| 649 |
+
if set_mask.any():
|
| 650 |
+
b_idx, t_idx = set_mask.nonzero(as_tuple=True)
|
| 651 |
+
flat_idx = flat_ids_k[b_idx, t_idx]
|
| 652 |
+
multi_hot[b_idx, t_idx, flat_idx] = True
|
| 653 |
+
fake_logits = torch.where(
|
| 654 |
+
multi_hot,
|
| 655 |
+
torch.full_like(out["feats"], 12.0),
|
| 656 |
+
torch.full_like(out["feats"], -12.0),
|
| 657 |
+
)
|
| 658 |
+
out["feats"] = fake_logits
|
| 659 |
+
|
| 660 |
+
if self.compound_enabled:
|
| 661 |
+
# h_token (B, T, d) is the per-position encoder hidden state. The
|
| 662 |
+
# collator's `first_subw` gates the loss/decode onto real tokens
|
| 663 |
+
# downstream — at this layer we emit dense (B, T, ...) outputs.
|
| 664 |
+
#
|
| 665 |
+
# When `compound_detach_encoder=True`, the compound branch
|
| 666 |
+
# sees a detached copy of `h`. This blocks gradient from
|
| 667 |
+
# compound losses (count/split/tail) flowing back into the
|
| 668 |
+
# shared encoder. The detach is autograd-only — forward
|
| 669 |
+
# values are identical.
|
| 670 |
+
h_compound = h.detach() if self.compound_detach_encoder else h
|
| 671 |
+
B, T, _ = h.shape
|
| 672 |
+
F = self.max_form_len
|
| 673 |
+
# Shape/dtype guards, not asserts.
|
| 674 |
+
if form_chars.shape != (B, T, F):
|
| 675 |
+
raise ValueError(
|
| 676 |
+
f"form_chars shape {tuple(form_chars.shape)} != "
|
| 677 |
+
f"(B, T, max_form_len)=({B},{T},{F})")
|
| 678 |
+
if form_char_mask.shape != (B, T, F):
|
| 679 |
+
raise ValueError(
|
| 680 |
+
f"form_char_mask shape {tuple(form_char_mask.shape)} != "
|
| 681 |
+
f"(B, T, max_form_len)=({B},{T},{F})")
|
| 682 |
+
if form_char_mask.dtype != torch.bool:
|
| 683 |
+
raise ValueError(
|
| 684 |
+
f"form_char_mask dtype must be torch.bool, got {form_char_mask.dtype}")
|
| 685 |
+
# Char embeddings: (B, T, F, char_emb_dim). PAD-row is constrained
|
| 686 |
+
# zero (see __init__) so PAD slots inject no signal through the
|
| 687 |
+
# conv. We further zero out positions where form_char_mask is False
|
| 688 |
+
# — defensive, since UNK chars may be non-PAD-id but still padding.
|
| 689 |
+
ce = self.char_emb(form_chars) # (B, T, F, ce)
|
| 690 |
+
ce = ce * form_char_mask.unsqueeze(-1).to(ce.dtype)
|
| 691 |
+
# Conv expects (N, C, L). Flatten (B, T) into batch:
|
| 692 |
+
ce_bt = ce.reshape(B * T, F, -1).transpose(1, 2) # (B*T, ce, F)
|
| 693 |
+
conv_out = self.compound_split_conv(ce_bt) # (B*T, ch, F)
|
| 694 |
+
conv_out = torch.nn.functional.gelu(conv_out)
|
| 695 |
+
# FiLM modulation conditioned on h_token (B, T, d) → reshape to
|
| 696 |
+
# (B*T, 2*ch) → split into γ, β → (B*T, ch, 1) each, broadcast on F.
|
| 697 |
+
ch = self.compound_split_conv.out_channels
|
| 698 |
+
gb = self.compound_film(h_compound).reshape(B * T, 2 * ch) # (B*T, 2*ch)
|
| 699 |
+
gamma, beta = gb[:, :ch].unsqueeze(-1), gb[:, ch:].unsqueeze(-1)
|
| 700 |
+
mod = gamma * conv_out + beta # (B*T, ch, F)
|
| 701 |
+
mod = torch.nn.functional.gelu(mod)
|
| 702 |
+
# Per-char-position logit: (B*T, ch, F) → (B*T, F, ch) → Linear → (B*T, F, 1)
|
| 703 |
+
split_logits = self.compound_split_out(mod.transpose(1, 2)).squeeze(-1)
|
| 704 |
+
split_logits = split_logits.reshape(B, T, F)
|
| 705 |
+
# Do NOT mask padded positions to -inf here — BCEWithLogits
|
| 706 |
+
# would NaN on them. The decoder masks padded positions before
|
| 707 |
+
# its top-K; the loss ignores them via form_char_mask. Split
|
| 708 |
+
# logits stay finite.
|
| 709 |
+
|
| 710 |
+
out["count"] = self.compound_count(h_compound)
|
| 711 |
+
out["split"] = split_logits
|
| 712 |
+
out["tail"] = self.compound_tail(h_compound)
|
| 713 |
+
|
| 714 |
+
return out
|
juurio/parser/near_identity.py
ADDED
|
@@ -0,0 +1,108 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Near-identity capacity adapter.
|
| 2 |
+
|
| 3 |
+
A stack of transformer blocks that each behave as the IDENTITY
|
| 4 |
+
function at initialization, regardless of input scale, mask, or
|
| 5 |
+
batch contents. This lets a warm-started model add parameter
|
| 6 |
+
capacity while producing bit-identical predictions (within float32
|
| 7 |
+
rounding) at init. Training can then grow the adapter's `alpha`
|
| 8 |
+
parameters away from zero and exploit the new capacity without
|
| 9 |
+
leaving the base model's behaviour uncontrolled.
|
| 10 |
+
|
| 11 |
+
The block design:
|
| 12 |
+
- inner: `nn.TransformerEncoderLayer`.
|
| 13 |
+
- zero-init `self_attn.out_proj.weight/bias`: attention output → 0.
|
| 14 |
+
- zero-init `linear2.weight/bias`: FFN output → 0.
|
| 15 |
+
- learnable residual-scale scalar `alpha`, init = 0:
|
| 16 |
+
block(x) = x + alpha * (layer(x) - x)
|
| 17 |
+
At init alpha=0, so block(x) ≡ x bit-for-bit. As training
|
| 18 |
+
grows alpha, the block contributes its (now non-zero) layer
|
| 19 |
+
delta. The zero-init projections ensure the layer starts from
|
| 20 |
+
a well-conditioned point — not random — so as alpha grows the
|
| 21 |
+
delta is meaningful from step 1.
|
| 22 |
+
|
| 23 |
+
Padding invariance: the underlying `nn.TransformerEncoderLayer`
|
| 24 |
+
respects `src_key_padding_mask`, so even at alpha > 0 the block
|
| 25 |
+
cannot leak information from padded positions into real-token
|
| 26 |
+
outputs via self-attention.
|
| 27 |
+
|
| 28 |
+
Tests at `tests/test_near_identity_warm_start.py`.
|
| 29 |
+
"""
|
| 30 |
+
from __future__ import annotations
|
| 31 |
+
|
| 32 |
+
import torch
|
| 33 |
+
from torch import nn
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
class NearIdentityBlock(nn.Module):
|
| 37 |
+
"""One near-identity transformer block.
|
| 38 |
+
|
| 39 |
+
At init `block(x) == x` bit-for-bit (within float32 rounding,
|
| 40 |
+
≤ 1e-5 max abs diff). The learnable `alpha` parameter starts at
|
| 41 |
+
0 and grows during training to let the wrapped layer contribute.
|
| 42 |
+
"""
|
| 43 |
+
|
| 44 |
+
def __init__(self, d_model: int, heads: int, ff: int, dropout: float):
|
| 45 |
+
super().__init__()
|
| 46 |
+
self.layer = nn.TransformerEncoderLayer(
|
| 47 |
+
d_model=d_model,
|
| 48 |
+
nhead=heads,
|
| 49 |
+
dim_feedforward=ff,
|
| 50 |
+
dropout=dropout,
|
| 51 |
+
batch_first=True,
|
| 52 |
+
)
|
| 53 |
+
# Zero-init the final attention output projection so
|
| 54 |
+
# `self_attn(x) → 0` at init regardless of input.
|
| 55 |
+
nn.init.zeros_(self.layer.self_attn.out_proj.weight)
|
| 56 |
+
nn.init.zeros_(self.layer.self_attn.out_proj.bias)
|
| 57 |
+
# Zero-init the final FFN projection so `ff(x) → 0` at init.
|
| 58 |
+
nn.init.zeros_(self.layer.linear2.weight)
|
| 59 |
+
nn.init.zeros_(self.layer.linear2.bias)
|
| 60 |
+
# Residual scale α starts at 0 — block is exact identity at init.
|
| 61 |
+
# We use a 1-element vector so the parameter is always a tensor
|
| 62 |
+
# (not a 0-d scalar) which composes more cleanly with broadcasting.
|
| 63 |
+
self.alpha = nn.Parameter(torch.zeros(1))
|
| 64 |
+
|
| 65 |
+
def forward(self, x: torch.Tensor, *, mask: torch.Tensor) -> torch.Tensor:
|
| 66 |
+
"""Apply the block.
|
| 67 |
+
|
| 68 |
+
Args:
|
| 69 |
+
x: (B, T, d_model) — encoder hidden states.
|
| 70 |
+
mask: (B, T) bool — True at real-token positions, False at
|
| 71 |
+
padding. We translate to PyTorch's
|
| 72 |
+
`src_key_padding_mask` (True = ignore) by inverting.
|
| 73 |
+
|
| 74 |
+
Returns:
|
| 75 |
+
(B, T, d_model). At init exactly equal to `x` at real-token
|
| 76 |
+
positions (within ≤ 1e-5 max abs diff).
|
| 77 |
+
"""
|
| 78 |
+
# PyTorch convention: True in src_key_padding_mask = position
|
| 79 |
+
# is padding and should be IGNORED by attention.
|
| 80 |
+
y = self.layer(x, src_key_padding_mask=~mask)
|
| 81 |
+
# `alpha * (y - x)` is 0 at init → output = x. As alpha grows,
|
| 82 |
+
# the block smoothly interpolates toward the full layer output.
|
| 83 |
+
return x + self.alpha * (y - x)
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
class NearIdentityAdapter(nn.Module):
|
| 87 |
+
"""A stack of `NearIdentityBlock`s applied after the encoder.
|
| 88 |
+
|
| 89 |
+
`n_layers=0` is a strict no-op (returns input verbatim) used as the
|
| 90 |
+
ablation baseline. `n_layers >= 1` adds parameter capacity that
|
| 91 |
+
starts as identity and grows during training.
|
| 92 |
+
"""
|
| 93 |
+
|
| 94 |
+
def __init__(
|
| 95 |
+
self, d_model: int, heads: int, ff: int, dropout: float, n_layers: int
|
| 96 |
+
):
|
| 97 |
+
super().__init__()
|
| 98 |
+
self.layers = nn.ModuleList(
|
| 99 |
+
[
|
| 100 |
+
NearIdentityBlock(d_model=d_model, heads=heads, ff=ff, dropout=dropout)
|
| 101 |
+
for _ in range(n_layers)
|
| 102 |
+
]
|
| 103 |
+
)
|
| 104 |
+
|
| 105 |
+
def forward(self, x: torch.Tensor, *, mask: torch.Tensor) -> torch.Tensor:
|
| 106 |
+
for layer in self.layers:
|
| 107 |
+
x = layer(x, mask=mask)
|
| 108 |
+
return x
|
juurio/parser/neural_adapter.py
ADDED
|
@@ -0,0 +1,188 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""The served UD parser: `simple_tokenize` + sentence-split → hybrid morphology
|
| 2 |
+
channel → the neural model → the `{id, form, lemma, upos, head, deprel, feats}`
|
| 3 |
+
token dicts the rule engine (`algebra`) consumes.
|
| 4 |
+
|
| 5 |
+
This is the production `ParserAdapter` behind `pipeline.Transformer`. It needs no
|
| 6 |
+
external morphology toolkit at inference:
|
| 7 |
+
|
| 8 |
+
* tokenization is the rule-based `simple_tokenize`, not an external tokenizer;
|
| 9 |
+
* the FST-tag input channel is fed by the neural AnalyzeMux (`build_neural_cache`)
|
| 10 |
+
over a *static* compressed table (`FstCache.lookup` is a pure dict — miss ⇒
|
| 11 |
+
empty, never a live FST call).
|
| 12 |
+
|
| 13 |
+
Feats are decoded from the per-key head (`out["feats_per_key"]`), NOT the flat
|
| 14 |
+
`out["feats"]` BCE head — the mode-independent pattern new consumers should follow.
|
| 15 |
+
"""
|
| 16 |
+
from __future__ import annotations
|
| 17 |
+
|
| 18 |
+
import os
|
| 19 |
+
import re
|
| 20 |
+
|
| 21 |
+
import torch
|
| 22 |
+
|
| 23 |
+
from ..pipeline import ParserAdapter
|
| 24 |
+
from .char_vocab import CharVocab
|
| 25 |
+
from .collator import SilverCollator
|
| 26 |
+
from .edit_scripts import apply_edit_script
|
| 27 |
+
from .eval import _load_checkpoint
|
| 28 |
+
from .fst_cache import FstCache, FstTagVocab
|
| 29 |
+
from .label_vocab import LabelVocab
|
| 30 |
+
from .silver import SilverSentence, SilverToken
|
| 31 |
+
from .simple_tokenizer import simple_tokenize
|
| 32 |
+
from .tokenizer import BPETokenizer
|
| 33 |
+
from ..analyze.hybrid import build_hybrid_channel
|
| 34 |
+
from ..analyze.infer import build_neural_cache, load_analyze_mux
|
| 35 |
+
from .. import bootstrap
|
| 36 |
+
|
| 37 |
+
# Repo root (the from-source layout where checkpoints/ and data/ live). Default artifact paths and any
|
| 38 |
+
# relative path passed in are resolved against this — NOT the process CWD — so the served parser works
|
| 39 |
+
# from any working directory (mirrors bootstrap.ROOT). Same root bootstrap downloads artifacts into.
|
| 40 |
+
_ROOT = os.path.normpath(os.path.join(os.path.dirname(__file__), "..", ".."))
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def _resolve(path: str | None) -> str | None:
|
| 44 |
+
"""Resolve a (possibly relative) artifact path against the repo ROOT, leaving absolute paths as-is."""
|
| 45 |
+
if path is None or os.path.isabs(path):
|
| 46 |
+
return path
|
| 47 |
+
return os.path.join(_ROOT, path)
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
_SENT_BOUNDARY = re.compile(r"(?<=[.!?])\s+")
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
def _segment_and_tokenize(text: str) -> list[list[str]]:
|
| 54 |
+
"""Naive `[.!?]` sentence split, then the rule-based tokenizer per sentence."""
|
| 55 |
+
out: list[list[str]] = []
|
| 56 |
+
for chunk in (_SENT_BOUNDARY.split(text.strip()) if text.strip() else []):
|
| 57 |
+
toks = simple_tokenize(chunk)
|
| 58 |
+
if toks:
|
| 59 |
+
out.append(toks)
|
| 60 |
+
return out
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
def _placeholder(tokens: list[str]) -> SilverSentence:
|
| 64 |
+
"""A `SilverSentence` carrying `tokens` as forms with placeholder labels — the collator only reads
|
| 65 |
+
form/lemma for subword alignment; predict overwrites every label."""
|
| 66 |
+
return SilverSentence(text=" ".join(tokens),
|
| 67 |
+
tokens=[SilverToken(form=t, lemma=t.lower(), upos="X", feats="",
|
| 68 |
+
head=0 if i == 0 else 1,
|
| 69 |
+
deprel="root" if i == 0 else "dep")
|
| 70 |
+
for i, t in enumerate(tokens)])
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
class NeuralParser(ParserAdapter):
|
| 74 |
+
"""UD parser backed by the morphology channel plus the neural model.
|
| 75 |
+
|
| 76 |
+
Pass `compressed_table=` for the shipped serving config; the full
|
| 77 |
+
`fst_cache_v1.jsonl` is the larger default. Either way, membership in the
|
| 78 |
+
table picks clean-vs-neural tags and the morphology channel fills everything
|
| 79 |
+
else. `device=None` auto-selects CUDA when available."""
|
| 80 |
+
|
| 81 |
+
def __init__(self, *, parser_ckpt: str | None = None,
|
| 82 |
+
analyze_ckpt: str | None = None,
|
| 83 |
+
analyze_char_vocab: str | None = None,
|
| 84 |
+
tokenizer: str | None = None,
|
| 85 |
+
label_vocab: str | None = None,
|
| 86 |
+
fst_tag_vocab: str | None = None,
|
| 87 |
+
parser_char_vocab: str | None = None,
|
| 88 |
+
fst_cache: str = "data/fst_cache_v1.jsonl", compressed_table: str | None = None,
|
| 89 |
+
fst_max_tags: int = 16, device: str | None = None):
|
| 90 |
+
self.device = device or ("cuda" if torch.cuda.is_available() else "cpu")
|
| 91 |
+
# Each serving artifact: an explicit path is resolved against the repo ROOT; None routes through
|
| 92 |
+
# the bootstrap resolver (dev source -> HuggingFace), so a clean wheel install fetches them.
|
| 93 |
+
tokenizer = _resolve(tokenizer) if tokenizer else bootstrap.artifact_path("parser_tokenizer")
|
| 94 |
+
label_vocab = _resolve(label_vocab) if label_vocab else bootstrap.artifact_path("parser_label_vocab")
|
| 95 |
+
fst_tag_vocab = _resolve(fst_tag_vocab) if fst_tag_vocab else bootstrap.artifact_path("fst_tag_vocab")
|
| 96 |
+
parser_char_vocab = _resolve(parser_char_vocab) if parser_char_vocab else bootstrap.artifact_path("char_vocab")
|
| 97 |
+
parser_ckpt = _resolve(parser_ckpt) if parser_ckpt else bootstrap.artifact_path("parser")
|
| 98 |
+
analyze_ckpt = _resolve(analyze_ckpt) if analyze_ckpt else bootstrap.artifact_path("analyze")
|
| 99 |
+
analyze_char_vocab = (_resolve(analyze_char_vocab) if analyze_char_vocab
|
| 100 |
+
else bootstrap.artifact_path("analyze_charvocab"))
|
| 101 |
+
# compressed_table/fst_cache stay opt-in: pipeline passes the resolved fst_table; a direct
|
| 102 |
+
# NeuralParser() with neither falls back to the full cache (data/fst_cache_v1.jsonl).
|
| 103 |
+
compressed_table, fst_cache = _resolve(compressed_table), _resolve(fst_cache)
|
| 104 |
+
self.tok = BPETokenizer.load(tokenizer)
|
| 105 |
+
self.vocab = LabelVocab.load(label_vocab)
|
| 106 |
+
self.tag_vocab = FstTagVocab.load(fst_tag_vocab)
|
| 107 |
+
self.char_vocab = CharVocab.load(parser_char_vocab)
|
| 108 |
+
self.model = _load_checkpoint(parser_ckpt, self.vocab, self.tok, self.device)
|
| 109 |
+
self.model.eval() # inference-only consumer: disable dropout etc. (we decode Feats from
|
| 110 |
+
# out["feats_per_key"] directly, so we do NOT rely on the eval-gated out["feats"] override)
|
| 111 |
+
assert (self.model.feats_per_key_head is not None
|
| 112 |
+
and self.model._feats_per_key_flat_lookup is not None), \
|
| 113 |
+
"NeuralParser decodes Feats from the per-key head; this checkpoint lacks it"
|
| 114 |
+
self.mux, self.mux_cv, self.thr = load_analyze_mux(
|
| 115 |
+
analyze_ckpt, char_vocab_path=analyze_char_vocab, device=self.device,
|
| 116 |
+
tag_vocab=self.tag_vocab)
|
| 117 |
+
# Prefer the compressed table; fall back to the full FST cache if the table isn't installed.
|
| 118 |
+
lexicon_path = compressed_table if (compressed_table and os.path.exists(compressed_table)) else fst_cache
|
| 119 |
+
self.lexicon = FstCache.load(lexicon_path)
|
| 120 |
+
self.fst_max_tags = fst_max_tags
|
| 121 |
+
self.compound = bool(getattr(self.model, "compound_enabled", True))
|
| 122 |
+
self.max_form_len = getattr(self.model, "max_form_len", 64)
|
| 123 |
+
|
| 124 |
+
@torch.no_grad()
|
| 125 |
+
def parse(self, text: str) -> list[list[dict]]:
|
| 126 |
+
sents_tokens = _segment_and_tokenize(text)
|
| 127 |
+
if not sents_tokens:
|
| 128 |
+
return []
|
| 129 |
+
forms = sorted({t for s in sents_tokens for t in s})
|
| 130 |
+
neural = build_neural_cache(self.mux, self.mux_cv, forms, tag_vocab=self.tag_vocab,
|
| 131 |
+
threshold=self.thr, top_n=self.fst_max_tags, device=self.device)
|
| 132 |
+
hybrid, _ = build_hybrid_channel(self.lexicon, neural, forms)
|
| 133 |
+
placeholders = [_placeholder(t) for t in sents_tokens]
|
| 134 |
+
col = SilverCollator(self.tok, self.vocab, fst_cache=hybrid, fst_tag_vocab=self.tag_vocab,
|
| 135 |
+
fst_max_tags=self.fst_max_tags,
|
| 136 |
+
char_vocab=self.char_vocab if self.compound else None,
|
| 137 |
+
max_form_len=self.max_form_len, compound_enabled=self.compound)
|
| 138 |
+
batch = col(placeholders)
|
| 139 |
+
if batch["ids"].shape[0] == 0:
|
| 140 |
+
return []
|
| 141 |
+
batch = {k: (v.to(self.device) if torch.is_tensor(v) else v) for k, v in batch.items()}
|
| 142 |
+
out = self.model(batch["ids"], batch["mask"], fst_tags=batch.get("fst_tags"),
|
| 143 |
+
form_chars=batch.get("form_chars"), form_char_mask=batch.get("form_char_mask"))
|
| 144 |
+
fs = batch["first_subw"]
|
| 145 |
+
valid = fs.clone(); valid[:, 0] = True
|
| 146 |
+
NEG = torch.finfo(out["arc"].dtype).min
|
| 147 |
+
arc_pred = out["arc"].masked_fill(~valid.unsqueeze(1), NEG).argmax(-1)
|
| 148 |
+
upos_pred = out["upos"].argmax(-1)
|
| 149 |
+
lemma_pred = out["lemma"].argmax(-1)
|
| 150 |
+
# Decode Feats from the per-key head (out["feats_per_key"]), NOT the legacy flat out["feats"].
|
| 151 |
+
# Each lookup row maps a key's value index → flat label_vocab id (-1 = <EMPTY> → no bit set).
|
| 152 |
+
lookup = self.model._feats_per_key_flat_lookup # (n_keys, max_V), -1 padded
|
| 153 |
+
feat_flat_ids = torch.stack(
|
| 154 |
+
[lookup[k][kl.argmax(-1)] for k, kl in enumerate(out["feats_per_key"])],
|
| 155 |
+
dim=-1) # (B, T, n_keys)
|
| 156 |
+
B, T, _, R = out["rel"].shape
|
| 157 |
+
gh = arc_pred.unsqueeze(-1).unsqueeze(-1).expand(B, T, 1, R)
|
| 158 |
+
rel_pred = out["rel"].gather(2, gh).squeeze(2).argmax(-1)
|
| 159 |
+
|
| 160 |
+
kept = batch["kept_indices"].tolist()
|
| 161 |
+
results: list[list[dict]] = []
|
| 162 |
+
for b, src_idx in enumerate(kept):
|
| 163 |
+
word_first = fs[b].nonzero(as_tuple=True)[0].tolist()
|
| 164 |
+
src_tokens = placeholders[src_idx].tokens
|
| 165 |
+
toks: list[dict] = []
|
| 166 |
+
for wi, (pos, src) in enumerate(zip(word_first, src_tokens), start=1):
|
| 167 |
+
upos = self.vocab.id_to_upos.get(upos_pred[b, pos].item(), "X")
|
| 168 |
+
if upos.startswith("<"):
|
| 169 |
+
upos = "X"
|
| 170 |
+
eid = lemma_pred[b, pos].item()
|
| 171 |
+
script = self.vocab.edit_scripts[eid] if 0 <= eid < len(self.vocab.edit_scripts) else (0, "")
|
| 172 |
+
lemma = src.form.lower() if script[0] == -1 else apply_edit_script(src.form, script)
|
| 173 |
+
feats: dict[str, str] = {}
|
| 174 |
+
for fid in feat_flat_ids[b, pos].tolist():
|
| 175 |
+
if fid < 0:
|
| 176 |
+
continue
|
| 177 |
+
kv = self.vocab.id_to_feat.get(fid, "")
|
| 178 |
+
if kv and not kv.startswith("<") and "=" in kv:
|
| 179 |
+
k, v = kv.split("=", 1)
|
| 180 |
+
feats[k] = v
|
| 181 |
+
hs = arc_pred[b, pos].item()
|
| 182 |
+
head = 0 if hs == 0 else (word_first.index(hs) + 1 if hs in word_first else 0)
|
| 183 |
+
rel = self.vocab.id_to_deprel.get(rel_pred[b, pos].item(), "dep")
|
| 184 |
+
rel = ("dep" if rel.startswith("<") else rel).split(":")[0]
|
| 185 |
+
toks.append({"id": wi, "form": src.form, "lemma": lemma, "upos": upos,
|
| 186 |
+
"head": head, "deprel": rel, "feats": feats})
|
| 187 |
+
results.append(toks)
|
| 188 |
+
return results
|
juurio/parser/silver.py
ADDED
|
@@ -0,0 +1,252 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Silver-corpus generation for the compact Finnish UD parser.
|
| 2 |
+
|
| 3 |
+
Turns raw Finnish text into UD records via a UD parser pipeline, then caches the records as
|
| 4 |
+
JSONL on disk. The cached records serve as training labels for the parser.
|
| 5 |
+
|
| 6 |
+
**Schema** — one JSON object per line, one line per sentence:
|
| 7 |
+
|
| 8 |
+
{
|
| 9 |
+
"text": "Mies osti talon.",
|
| 10 |
+
"tokens": [
|
| 11 |
+
{"form": "Mies", "lemma": "mies", "upos": "NOUN",
|
| 12 |
+
"feats": "Case=Nom|Number=Sing", "head": 2, "deprel": "nsubj"},
|
| 13 |
+
...
|
| 14 |
+
]
|
| 15 |
+
}
|
| 16 |
+
|
| 17 |
+
Design notes:
|
| 18 |
+
- Argmax-only labels (no top-K logits): the parser's public API exposes only argmax
|
| 19 |
+
predictions, so the silver corpus stores argmax labels.
|
| 20 |
+
- MWT spans are dropped — we keep one record per `Word` (per-component lines), aligning with
|
| 21 |
+
the rule-based tokenizer. Sentences with MWT get parsed as their component forms; MWT
|
| 22 |
+
detection happens at *inference* time in the parser.
|
| 23 |
+
- JSONL is append-safe. `count_cached(path)` tells the driver how many sentences to skip in
|
| 24 |
+
the source corpus so a crashed run can resume without re-parsing.
|
| 25 |
+
"""
|
| 26 |
+
from __future__ import annotations
|
| 27 |
+
|
| 28 |
+
import json
|
| 29 |
+
import os
|
| 30 |
+
from dataclasses import dataclass, asdict
|
| 31 |
+
from typing import Iterable, Iterator
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
@dataclass
|
| 35 |
+
class SilverToken:
|
| 36 |
+
"""One token's argmax silver labels — surface form, lemma, UPOS, feats, head index, deprel.
|
| 37 |
+
|
| 38 |
+
`feats_loss_masked`: True when the cleanup pass could not produce a
|
| 39 |
+
high-confidence Feats label for this token. The collator emits a
|
| 40 |
+
`feats_token_mask` based on this flag so the Feats loss skips these
|
| 41 |
+
tokens. Defaults to False so older silver JSONL files load unchanged.
|
| 42 |
+
"""
|
| 43 |
+
form: str
|
| 44 |
+
lemma: str
|
| 45 |
+
upos: str
|
| 46 |
+
feats: str # "" when Stanza returns None
|
| 47 |
+
head: int # 0 = UD root
|
| 48 |
+
deprel: str
|
| 49 |
+
feats_loss_masked: bool = False
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
@dataclass
|
| 53 |
+
class SilverSentence:
|
| 54 |
+
"""One sentence's silver record.
|
| 55 |
+
|
| 56 |
+
`teacher_tokens`: optional parallel list aligned 1:1 with `tokens`, carrying
|
| 57 |
+
auxiliary per-token training targets. When present, the collator emits
|
| 58 |
+
per-token `teacher_*` tensors and a per-sentence `kd_apply=True`. Defaults
|
| 59 |
+
to None so existing call sites are unaffected (and serving never sets it).
|
| 60 |
+
"""
|
| 61 |
+
text: str
|
| 62 |
+
tokens: list[SilverToken]
|
| 63 |
+
teacher_tokens: list | None = None
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
# ----------------------------- doc → silver ------------------------------------------------------
|
| 67 |
+
def doc_to_silver(doc) -> Iterator[SilverSentence]:
|
| 68 |
+
"""Convert a Stanza `Document` (or duck-typed equivalent) to `SilverSentence` records.
|
| 69 |
+
|
| 70 |
+
`doc.sentences` is iterated; for each sentence, `sentence.words` (the per-Word view, not
|
| 71 |
+
`tokens`) is used so MWT components show up as separate `SilverToken`s and the head indices
|
| 72 |
+
stay in Word-space."""
|
| 73 |
+
for sent in doc.sentences:
|
| 74 |
+
out_tokens: list[SilverToken] = []
|
| 75 |
+
for w in sent.words:
|
| 76 |
+
out_tokens.append(SilverToken(
|
| 77 |
+
form=w.text,
|
| 78 |
+
lemma=w.lemma if w.lemma is not None else w.text,
|
| 79 |
+
upos=w.upos if w.upos is not None else "X",
|
| 80 |
+
feats=w.feats if w.feats is not None else "",
|
| 81 |
+
head=int(w.head) if w.head is not None else 0,
|
| 82 |
+
deprel=w.deprel if w.deprel is not None else "dep",
|
| 83 |
+
))
|
| 84 |
+
yield SilverSentence(text=sent.text or "", tokens=out_tokens)
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
# ----------------------------- JSONL I/O ---------------------------------------------------------
|
| 88 |
+
def write_jsonl(records: Iterable[SilverSentence], path: str, mode: str = "a") -> int:
|
| 89 |
+
"""Write `records` as one JSON object per line. Returns the number of records written.
|
| 90 |
+
|
| 91 |
+
`mode="a"` is append — resume-safe. `mode="w"` truncates first."""
|
| 92 |
+
if mode not in ("w", "a"):
|
| 93 |
+
raise ValueError(f"mode must be 'w' or 'a', got {mode!r}")
|
| 94 |
+
os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
|
| 95 |
+
n = 0
|
| 96 |
+
with open(path, mode, encoding="utf-8") as fh:
|
| 97 |
+
for s in records:
|
| 98 |
+
fh.write(json.dumps(asdict(s), ensure_ascii=False))
|
| 99 |
+
fh.write("\n")
|
| 100 |
+
n += 1
|
| 101 |
+
return n
|
| 102 |
+
|
| 103 |
+
|
| 104 |
+
_SILVER_TOKEN_FIELDS = frozenset(SilverToken.__dataclass_fields__.keys())
|
| 105 |
+
|
| 106 |
+
|
| 107 |
+
def read_jsonl(path: str) -> Iterator[SilverSentence]:
|
| 108 |
+
"""Stream `SilverSentence` records from a JSONL file.
|
| 109 |
+
|
| 110 |
+
Unknown token fields are silently ignored — keeps the reader
|
| 111 |
+
forward-compatible with cleanup-pass extensions (e.g. a
|
| 112 |
+
`feats_loss_masked` flag; future passes may add more).
|
| 113 |
+
"""
|
| 114 |
+
with open(path, encoding="utf-8") as fh:
|
| 115 |
+
for ln in fh:
|
| 116 |
+
ln = ln.strip()
|
| 117 |
+
if not ln:
|
| 118 |
+
continue
|
| 119 |
+
obj = json.loads(ln)
|
| 120 |
+
yield SilverSentence(
|
| 121 |
+
text=obj["text"],
|
| 122 |
+
tokens=[
|
| 123 |
+
SilverToken(**{k: v for k, v in t.items()
|
| 124 |
+
if k in _SILVER_TOKEN_FIELDS})
|
| 125 |
+
for t in obj["tokens"]
|
| 126 |
+
],
|
| 127 |
+
)
|
| 128 |
+
|
| 129 |
+
|
| 130 |
+
def count_cached(path: str) -> int:
|
| 131 |
+
"""Return the number of records already cached at `path`. 0 if the file doesn't exist."""
|
| 132 |
+
if not os.path.exists(path):
|
| 133 |
+
return 0
|
| 134 |
+
n = 0
|
| 135 |
+
with open(path, encoding="utf-8") as fh:
|
| 136 |
+
for ln in fh:
|
| 137 |
+
if ln.strip():
|
| 138 |
+
n += 1
|
| 139 |
+
return n
|
| 140 |
+
|
| 141 |
+
|
| 142 |
+
# ----------------------------- driver: parse a corpus --------------------------------------------
|
| 143 |
+
def parse_corpus(
|
| 144 |
+
nlp,
|
| 145 |
+
corpus: Iterable[str],
|
| 146 |
+
out_path: str,
|
| 147 |
+
*,
|
| 148 |
+
batch_size: int = 32,
|
| 149 |
+
flush_every: int = 200,
|
| 150 |
+
skip: int = 0,
|
| 151 |
+
) -> int:
|
| 152 |
+
"""Run `nlp` over `corpus` and stream silver records to `out_path`.
|
| 153 |
+
|
| 154 |
+
- `nlp` is a configured Stanza pipeline (caller controls GPU, processors, etc.).
|
| 155 |
+
- `batch_size`: Stanza is faster with batches; we feed it `batch_size` sentences at a time.
|
| 156 |
+
- `flush_every`: how often to flush the output file so a kill -9 loses at most this many records.
|
| 157 |
+
- `skip`: how many input sentences to drop from the head — for resume; pass `count_cached(out_path)`.
|
| 158 |
+
|
| 159 |
+
Returns the number of sentences newly written to `out_path`."""
|
| 160 |
+
mode = "a" if skip > 0 or os.path.exists(out_path) else "w"
|
| 161 |
+
os.makedirs(os.path.dirname(os.path.abspath(out_path)) or ".", exist_ok=True)
|
| 162 |
+
written = 0
|
| 163 |
+
buf: list[str] = []
|
| 164 |
+
with open(out_path, mode, encoding="utf-8") as fh:
|
| 165 |
+
seen = 0
|
| 166 |
+
for text in corpus:
|
| 167 |
+
seen += 1
|
| 168 |
+
if seen <= skip:
|
| 169 |
+
continue
|
| 170 |
+
buf.append(text)
|
| 171 |
+
if len(buf) >= batch_size:
|
| 172 |
+
written += _flush_batch(nlp, buf, fh)
|
| 173 |
+
buf.clear()
|
| 174 |
+
if written % flush_every == 0:
|
| 175 |
+
fh.flush()
|
| 176 |
+
if buf:
|
| 177 |
+
written += _flush_batch(nlp, buf, fh)
|
| 178 |
+
fh.flush()
|
| 179 |
+
return written
|
| 180 |
+
|
| 181 |
+
|
| 182 |
+
def _flush_batch(nlp, texts: list[str], fh) -> int:
|
| 183 |
+
"""Parse a batch and dump silver records to the file handle. Returns count written."""
|
| 184 |
+
# Stanza accepts a list[str] in newer versions for batched parsing; concatenated string with
|
| 185 |
+
# blank-line separators is the wire-compatible fallback. We try list[str] first.
|
| 186 |
+
try:
|
| 187 |
+
docs = nlp.bulk_process(texts)
|
| 188 |
+
except (AttributeError, TypeError):
|
| 189 |
+
# older Stanza: parse joined string, take whole doc; works since corpus chunks are
|
| 190 |
+
# roughly sentence-sized
|
| 191 |
+
docs = [nlp(t) for t in texts]
|
| 192 |
+
n = 0
|
| 193 |
+
for doc in docs:
|
| 194 |
+
for s in doc_to_silver(doc):
|
| 195 |
+
fh.write(json.dumps(asdict(s), ensure_ascii=False))
|
| 196 |
+
fh.write("\n")
|
| 197 |
+
n += 1
|
| 198 |
+
return n
|
| 199 |
+
|
| 200 |
+
|
| 201 |
+
# ----------------------------- CLI: smoke-test on 10K sentences ----------------------------------
|
| 202 |
+
def _iter_corpus(path: str, max_sentences: int) -> Iterator[str]:
|
| 203 |
+
"""Stream sentence-sized chunks from a JSONL corpus. Same chunking as `tokenizer.py`."""
|
| 204 |
+
n = 0
|
| 205 |
+
with open(path, encoding="utf-8") as fh:
|
| 206 |
+
for ln in fh:
|
| 207 |
+
ln = ln.strip()
|
| 208 |
+
if not ln:
|
| 209 |
+
continue
|
| 210 |
+
obj = json.loads(ln)
|
| 211 |
+
text = obj.get("text", "")
|
| 212 |
+
for chunk in text.split("\n"):
|
| 213 |
+
chunk = chunk.strip()
|
| 214 |
+
if chunk:
|
| 215 |
+
yield chunk
|
| 216 |
+
n += 1
|
| 217 |
+
if n >= max_sentences:
|
| 218 |
+
return
|
| 219 |
+
|
| 220 |
+
|
| 221 |
+
if __name__ == "__main__":
|
| 222 |
+
import argparse
|
| 223 |
+
import time
|
| 224 |
+
|
| 225 |
+
ap = argparse.ArgumentParser(description="Silver-parse Finnish text with Stanza → JSONL.")
|
| 226 |
+
ap.add_argument("--corpus", required=True,
|
| 227 |
+
help="input jsonl with a 'text' field (one document per line)")
|
| 228 |
+
ap.add_argument("--out", default="checkpoints/silver_smoke_10k.jsonl")
|
| 229 |
+
ap.add_argument("--n", type=int, default=10_000, help="number of input chunks to parse")
|
| 230 |
+
ap.add_argument("--batch", type=int, default=32)
|
| 231 |
+
ap.add_argument("--gpu", action="store_true")
|
| 232 |
+
args = ap.parse_args()
|
| 233 |
+
|
| 234 |
+
import stanza
|
| 235 |
+
t0 = time.time()
|
| 236 |
+
print(f"[silver] loading Stanza fi pipeline (gpu={args.gpu})")
|
| 237 |
+
nlp = stanza.Pipeline(
|
| 238 |
+
"fi", processors="tokenize,mwt,pos,lemma,depparse",
|
| 239 |
+
use_gpu=args.gpu, verbose=False, download_method=None,
|
| 240 |
+
)
|
| 241 |
+
|
| 242 |
+
skip = count_cached(args.out)
|
| 243 |
+
if skip:
|
| 244 |
+
print(f"[silver] resuming — {skip} records already cached at {args.out}")
|
| 245 |
+
n_written = parse_corpus(
|
| 246 |
+
nlp, _iter_corpus(args.corpus, args.n), args.out,
|
| 247 |
+
batch_size=args.batch, skip=skip,
|
| 248 |
+
)
|
| 249 |
+
dt = time.time() - t0
|
| 250 |
+
print(f"[silver] wrote {n_written} new records in {dt:.1f}s "
|
| 251 |
+
f"({n_written/max(dt,1e-6):.1f} sent/s)")
|
| 252 |
+
print(f"[silver] total cached: {count_cached(args.out)} → {args.out}")
|
juurio/parser/simple_tokenizer.py
ADDED
|
@@ -0,0 +1,205 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Coarse rule-based Finnish tokenizer.
|
| 2 |
+
|
| 3 |
+
A lightweight, dependency-free tokenizer that gets close to UD-TDT's gold tokenization. The
|
| 4 |
+
`boundary_f1` helper measures token-boundary F1 against gold so callers can decide whether the
|
| 5 |
+
rule-based output is good enough or whether a learned tokenizer is needed.
|
| 6 |
+
|
| 7 |
+
Tokenization rules (intentionally minimal):
|
| 8 |
+
- Split on whitespace runs.
|
| 9 |
+
- Punctuation characters at the *edges* of a whitespace-delimited chunk become their own tokens.
|
| 10 |
+
- Internal punctuation between alphanumerics (`H&M`, `1990-luvulla`, `a@b.fi`) stays inside the
|
| 11 |
+
token — this captures most Finnish hyphenated compounds and loan brand names.
|
| 12 |
+
|
| 13 |
+
CLI:
|
| 14 |
+
python -m juurio.parser.simple_tokenizer probe # F1 vs UD-TDT-dev
|
| 15 |
+
"""
|
| 16 |
+
from __future__ import annotations
|
| 17 |
+
|
| 18 |
+
import re
|
| 19 |
+
|
| 20 |
+
_WS = re.compile(r"\s+")
|
| 21 |
+
# Punctuation we'd like to split off at the edges. Everything else stays internal.
|
| 22 |
+
_EDGE_PUNCT = '.,;:!?"\'()[]{}«»‹›„""\'\'…'
|
| 23 |
+
|
| 24 |
+
# Common Finnish abbreviations whose trailing period is PART of the token — UD-TDT keeps `n.`, `esim.`
|
| 25 |
+
# etc. as single tokens. Stored lowercased without the period. Intentionally NOT exhaustive: the
|
| 26 |
+
# ambiguous number/ordinal (`1990.` = year+'.' vs ordinal) and emoticon cases are left to the edge
|
| 27 |
+
# splitter — that irreducible tail is what the module docstring flags for a learned tokenizer.
|
| 28 |
+
_ABBREV = {
|
| 29 |
+
"n", "nn", "mm", "esim", "ns", "nk", "jne", "yms", "ym", "tms", "vrt", "ent",
|
| 30 |
+
"os", "ko", "nro", "mrd", "milj", "tri", "prof", "huom", "ks", "oik",
|
| 31 |
+
}
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
def _keeps_trailing_period(core: str) -> bool:
|
| 35 |
+
"""True if a period following `core` should stay attached: a known abbreviation, or a single
|
| 36 |
+
uppercase initial (`J.`, `P.`, `C.`). Numbers/ordinals are deliberately excluded (ambiguous with
|
| 37 |
+
sentence-final periods)."""
|
| 38 |
+
if not core:
|
| 39 |
+
return False
|
| 40 |
+
if core.lower() in _ABBREV:
|
| 41 |
+
return True
|
| 42 |
+
return len(core) == 1 and core.isalpha() and core.isupper()
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
def _strip_edges(chunk: str) -> list[str]:
|
| 46 |
+
"""Pull leading and trailing punctuation off a whitespace-delimited chunk, recursively, while
|
| 47 |
+
leaving alphanumeric-internal punctuation alone. `"Hei!"` -> [`"`, `Hei`, `!`, `"`]. An
|
| 48 |
+
abbreviation/initial period (`esim.`, `J.`) is kept inside the token rather than split off."""
|
| 49 |
+
out: list[str] = []
|
| 50 |
+
leading: list[str] = []
|
| 51 |
+
trailing: list[str] = []
|
| 52 |
+
while chunk and chunk[0] in _EDGE_PUNCT:
|
| 53 |
+
leading.append(chunk[0])
|
| 54 |
+
chunk = chunk[1:]
|
| 55 |
+
while chunk and chunk[-1] in _EDGE_PUNCT:
|
| 56 |
+
if chunk[-1] == "." and _keeps_trailing_period(chunk[:-1]):
|
| 57 |
+
break # period belongs to the abbreviation / initial — keep it inside the token
|
| 58 |
+
trailing.append(chunk[-1])
|
| 59 |
+
chunk = chunk[:-1]
|
| 60 |
+
out.extend(leading)
|
| 61 |
+
if chunk:
|
| 62 |
+
out.append(chunk)
|
| 63 |
+
out.extend(reversed(trailing))
|
| 64 |
+
return out
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
def simple_tokenize(text: str) -> list[str]:
|
| 68 |
+
"""Tokenize `text` into a list of surface tokens. Minimal, rule-based — see module docstring."""
|
| 69 |
+
if not text or not text.strip():
|
| 70 |
+
return []
|
| 71 |
+
out: list[str] = []
|
| 72 |
+
for chunk in _WS.split(text.strip()):
|
| 73 |
+
if chunk:
|
| 74 |
+
out.extend(_strip_edges(chunk))
|
| 75 |
+
return out
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
# ----------------------------- boundary F1 evaluation --------------------------------------------
|
| 79 |
+
def _read_text_and_tokens(path: str) -> list[tuple[str, list[str]]]:
|
| 80 |
+
"""Yield (raw_text, gold_tokens) pairs from a CoNLL-U file.
|
| 81 |
+
|
| 82 |
+
`# text =` comments carry the raw sentence; the per-token form columns carry the gold
|
| 83 |
+
tokenization. We skip MWT span lines (the per-component lines are what align with the raw
|
| 84 |
+
text) and empty-node lines."""
|
| 85 |
+
out: list[tuple[str, list[str]]] = []
|
| 86 |
+
text = None
|
| 87 |
+
tokens: list[str] = []
|
| 88 |
+
with open(path, encoding="utf-8") as fh:
|
| 89 |
+
for ln in fh:
|
| 90 |
+
ln = ln.rstrip("\n")
|
| 91 |
+
if not ln:
|
| 92 |
+
if text is not None and tokens:
|
| 93 |
+
out.append((text, tokens))
|
| 94 |
+
text = None
|
| 95 |
+
tokens = []
|
| 96 |
+
continue
|
| 97 |
+
if ln.startswith("# text = "):
|
| 98 |
+
text = ln[len("# text = "):]
|
| 99 |
+
continue
|
| 100 |
+
if ln.startswith("#"):
|
| 101 |
+
continue
|
| 102 |
+
cols = ln.split("\t")
|
| 103 |
+
if len(cols) < 2:
|
| 104 |
+
continue
|
| 105 |
+
tid = cols[0]
|
| 106 |
+
if "-" in tid or "." in tid: # skip MWT span / empty-node lines
|
| 107 |
+
continue
|
| 108 |
+
tokens.append(cols[1])
|
| 109 |
+
if text is not None and tokens:
|
| 110 |
+
out.append((text, tokens))
|
| 111 |
+
return out
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
def _token_spans(text: str, tokens: list[str]) -> list[tuple[int, int]] | None:
|
| 115 |
+
"""Greedy left-to-right alignment of `tokens` against `text`, returning their `(start, end)`
|
| 116 |
+
char spans. Returns `None` if alignment fails (a token does not appear at the expected
|
| 117 |
+
position) — those sentences are dropped from the F1 calculation as alignment failures.
|
| 118 |
+
|
| 119 |
+
Both gold and predicted token lists go through this routine so the F1 is computed over the
|
| 120 |
+
same alignment convention."""
|
| 121 |
+
out = []
|
| 122 |
+
i = 0
|
| 123 |
+
n = len(text)
|
| 124 |
+
for t in tokens:
|
| 125 |
+
# skip leading whitespace
|
| 126 |
+
while i < n and text[i].isspace():
|
| 127 |
+
i += 1
|
| 128 |
+
# exact match at i?
|
| 129 |
+
if text[i:i + len(t)] == t:
|
| 130 |
+
out.append((i, i + len(t)))
|
| 131 |
+
i += len(t)
|
| 132 |
+
else:
|
| 133 |
+
return None
|
| 134 |
+
return out
|
| 135 |
+
|
| 136 |
+
|
| 137 |
+
def boundary_f1(path: str) -> dict[str, float]:
|
| 138 |
+
"""Compute token-boundary F1 of `simple_tokenize` against UD-TDT gold on the given file.
|
| 139 |
+
|
| 140 |
+
Boundaries are `(start, end)` char-span sets over each sentence. Sentences whose gold or
|
| 141 |
+
predicted alignment fails are tallied separately."""
|
| 142 |
+
pairs = _read_text_and_tokens(path)
|
| 143 |
+
tp = fp = fn = 0
|
| 144 |
+
n_sent = 0
|
| 145 |
+
n_perfect = 0
|
| 146 |
+
n_align_fail_gold = 0
|
| 147 |
+
n_align_fail_pred = 0
|
| 148 |
+
for text, gold_tokens in pairs:
|
| 149 |
+
gold_spans = _token_spans(text, gold_tokens)
|
| 150 |
+
if gold_spans is None:
|
| 151 |
+
n_align_fail_gold += 1
|
| 152 |
+
continue
|
| 153 |
+
pred_tokens = simple_tokenize(text)
|
| 154 |
+
pred_spans = _token_spans(text, pred_tokens)
|
| 155 |
+
if pred_spans is None:
|
| 156 |
+
n_align_fail_pred += 1
|
| 157 |
+
continue
|
| 158 |
+
n_sent += 1
|
| 159 |
+
g, p = set(gold_spans), set(pred_spans)
|
| 160 |
+
tp += len(g & p)
|
| 161 |
+
fp += len(p - g)
|
| 162 |
+
fn += len(g - p)
|
| 163 |
+
if g == p:
|
| 164 |
+
n_perfect += 1
|
| 165 |
+
prec = tp / (tp + fp) if (tp + fp) else 0.0
|
| 166 |
+
rec = tp / (tp + fn) if (tp + fn) else 0.0
|
| 167 |
+
f1 = 2 * prec * rec / (prec + rec) if (prec + rec) else 0.0
|
| 168 |
+
return {
|
| 169 |
+
"sentences_eval": n_sent,
|
| 170 |
+
"sentences_perfect": n_perfect,
|
| 171 |
+
"sentence_perfect_rate": n_perfect / n_sent if n_sent else 0.0,
|
| 172 |
+
"precision": prec,
|
| 173 |
+
"recall": rec,
|
| 174 |
+
"f1": f1,
|
| 175 |
+
"alignment_failed_gold": n_align_fail_gold,
|
| 176 |
+
"alignment_failed_pred": n_align_fail_pred,
|
| 177 |
+
}
|
| 178 |
+
|
| 179 |
+
|
| 180 |
+
def probe(paths: dict[str, str]) -> None:
|
| 181 |
+
print("Ablation §0.3 — rule-based tokenizer boundary F1 vs UD-TDT")
|
| 182 |
+
print("=" * 78)
|
| 183 |
+
for name, path in paths.items():
|
| 184 |
+
r = boundary_f1(path)
|
| 185 |
+
print(f" {name:6s} sentences-eval {r['sentences_eval']:5d} "
|
| 186 |
+
f"P {r['precision']:.4f} R {r['recall']:.4f} F1 {r['f1']:.4f} "
|
| 187 |
+
f"perfect-sent {r['sentence_perfect_rate']*100:5.2f}% "
|
| 188 |
+
f"align-fail gold/pred {r['alignment_failed_gold']}/{r['alignment_failed_pred']}")
|
| 189 |
+
|
| 190 |
+
|
| 191 |
+
if __name__ == "__main__":
|
| 192 |
+
import os
|
| 193 |
+
import sys
|
| 194 |
+
|
| 195 |
+
repo = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
| 196 |
+
paths = {
|
| 197 |
+
"train": os.path.join(repo, "data", "ud", "fi_tdt-ud-train.conllu"),
|
| 198 |
+
"dev": os.path.join(repo, "data", "ud", "fi_tdt-ud-dev.conllu"),
|
| 199 |
+
"test": os.path.join(repo, "data", "ud", "fi_tdt-ud-test.conllu"),
|
| 200 |
+
}
|
| 201 |
+
cmd = sys.argv[1] if len(sys.argv) > 1 else "probe"
|
| 202 |
+
if cmd == "probe":
|
| 203 |
+
probe(paths)
|
| 204 |
+
else:
|
| 205 |
+
raise SystemExit(f"unknown cmd {cmd!r}; usage: python -m juurio.parser.simple_tokenizer probe")
|
juurio/parser/tokenizer.py
ADDED
|
@@ -0,0 +1,172 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""BPE tokenizer for the compact Finnish UD parser.
|
| 2 |
+
|
| 3 |
+
Wraps HuggingFace `tokenizers` to give a thin, fixed API the parser depends on:
|
| 4 |
+
|
| 5 |
+
- `BPETokenizer.train(corpus_iter, vocab_size=8000)` trains from scratch on an iterable of
|
| 6 |
+
Finnish text strings. Reserves PAD_ID=0, ROOT_ID=1, UNK,
|
| 7 |
+
BOS, EOS as the first five vocab slots so the model's `PAD_ID` / `ROOT_ID` constants stay
|
| 8 |
+
consistent across training and inference.
|
| 9 |
+
- `encode(text) -> list[int]` produces a plain id list.
|
| 10 |
+
- `encode_with_root(text)` prepends ROOT_ID — the convention for parser input
|
| 11 |
+
(the UD virtual root is the head=0 attachment target).
|
| 12 |
+
- `decode(ids) -> str` round-trips back to a surface string (BPE joiner handled).
|
| 13 |
+
- `save(path) / load(path)` go through a single `tokenizer.json` (HF format).
|
| 14 |
+
|
| 15 |
+
The locked special-token order is:
|
| 16 |
+
|
| 17 |
+
0: <pad> (PAD_ID, also `padding_idx` of the embedding)
|
| 18 |
+
1: <root> (ROOT_ID, the virtual UD root token)
|
| 19 |
+
2: <unk>
|
| 20 |
+
3: <bos>
|
| 21 |
+
4: <eos>
|
| 22 |
+
|
| 23 |
+
These ids are baked into both this module and `juurio/parser/model.py`. Changing them is a
|
| 24 |
+
*data-pipeline contract break* — only do it together with a matching model-checkpoint retrain.
|
| 25 |
+
"""
|
| 26 |
+
from __future__ import annotations
|
| 27 |
+
|
| 28 |
+
import os
|
| 29 |
+
from typing import Iterable
|
| 30 |
+
|
| 31 |
+
from tokenizers import Tokenizer, decoders, models, pre_tokenizers, trainers
|
| 32 |
+
from tokenizers.processors import TemplateProcessing
|
| 33 |
+
|
| 34 |
+
from .model import PAD_ID, ROOT_ID
|
| 35 |
+
|
| 36 |
+
# Locked special-token order. See module docstring.
|
| 37 |
+
PAD_TOKEN = "<pad>"
|
| 38 |
+
ROOT_TOKEN = "<root>"
|
| 39 |
+
UNK_TOKEN = "<unk>"
|
| 40 |
+
BOS_TOKEN = "<bos>"
|
| 41 |
+
EOS_TOKEN = "<eos>"
|
| 42 |
+
_SPECIAL_TOKENS = [PAD_TOKEN, ROOT_TOKEN, UNK_TOKEN, BOS_TOKEN, EOS_TOKEN]
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
class BPETokenizer:
|
| 46 |
+
"""Thin wrapper around an HF BPE `Tokenizer` with our locked special-token contract."""
|
| 47 |
+
|
| 48 |
+
def __init__(self, tok: Tokenizer):
|
| 49 |
+
self._tok = tok
|
| 50 |
+
# sanity: special tokens must be in the locked positions
|
| 51 |
+
assert tok.token_to_id(PAD_TOKEN) == PAD_ID, "PAD token at wrong id"
|
| 52 |
+
assert tok.token_to_id(ROOT_TOKEN) == ROOT_ID, "ROOT token at wrong id"
|
| 53 |
+
|
| 54 |
+
# ----------------------------- properties --------------------------------------------------
|
| 55 |
+
@property
|
| 56 |
+
def pad_id(self) -> int:
|
| 57 |
+
return PAD_ID
|
| 58 |
+
|
| 59 |
+
@property
|
| 60 |
+
def root_id(self) -> int:
|
| 61 |
+
return ROOT_ID
|
| 62 |
+
|
| 63 |
+
@property
|
| 64 |
+
def unk_id(self) -> int:
|
| 65 |
+
return self._tok.token_to_id(UNK_TOKEN)
|
| 66 |
+
|
| 67 |
+
@property
|
| 68 |
+
def vocab_size(self) -> int:
|
| 69 |
+
return self._tok.get_vocab_size()
|
| 70 |
+
|
| 71 |
+
# ----------------------------- encode / decode ---------------------------------------------
|
| 72 |
+
def encode(self, text: str) -> list[int]:
|
| 73 |
+
"""Encode `text` to a list of integer token ids. Does NOT prepend ROOT; see
|
| 74 |
+
`encode_with_root` for parser input."""
|
| 75 |
+
return self._tok.encode(text).ids
|
| 76 |
+
|
| 77 |
+
def encode_with_root(self, text: str) -> list[int]:
|
| 78 |
+
"""Encode `text` and prepend ROOT_ID — the canonical parser input format."""
|
| 79 |
+
return [ROOT_ID, *self.encode(text)]
|
| 80 |
+
|
| 81 |
+
def decode(self, ids: list[int]) -> str:
|
| 82 |
+
"""Decode an id sequence back to text. Special tokens are skipped."""
|
| 83 |
+
return self._tok.decode(ids, skip_special_tokens=True)
|
| 84 |
+
|
| 85 |
+
# ----------------------------- save / load -------------------------------------------------
|
| 86 |
+
def save(self, path: str) -> None:
|
| 87 |
+
os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)
|
| 88 |
+
self._tok.save(path)
|
| 89 |
+
|
| 90 |
+
@classmethod
|
| 91 |
+
def load(cls, path: str) -> "BPETokenizer":
|
| 92 |
+
return cls(Tokenizer.from_file(path))
|
| 93 |
+
|
| 94 |
+
# ----------------------------- training ----------------------------------------------------
|
| 95 |
+
@classmethod
|
| 96 |
+
def train(cls, corpus: Iterable[str], vocab_size: int = 8000) -> "BPETokenizer":
|
| 97 |
+
"""Train a fresh byte-level BPE tokenizer over `corpus`.
|
| 98 |
+
|
| 99 |
+
Byte-level BPE is the safe default for Finnish: it never produces `<unk>` on UTF-8 input,
|
| 100 |
+
handles the full inflectional zoo without language-specific pre-tokenization, and matches
|
| 101 |
+
what modern subword models use internally.
|
| 102 |
+
"""
|
| 103 |
+
tok = Tokenizer(models.BPE(unk_token=UNK_TOKEN))
|
| 104 |
+
tok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
|
| 105 |
+
tok.decoder = decoders.ByteLevel()
|
| 106 |
+
trainer = trainers.BpeTrainer(
|
| 107 |
+
vocab_size=vocab_size,
|
| 108 |
+
special_tokens=_SPECIAL_TOKENS,
|
| 109 |
+
initial_alphabet=pre_tokenizers.ByteLevel.alphabet(),
|
| 110 |
+
show_progress=False,
|
| 111 |
+
)
|
| 112 |
+
tok.train_from_iterator(list(corpus), trainer=trainer)
|
| 113 |
+
# Post-processor: don't add BOS/EOS automatically — the parser data pipeline owns ROOT
|
| 114 |
+
# insertion via `encode_with_root`. Keep encode() plain for full caller control.
|
| 115 |
+
tok.post_processor = TemplateProcessing(
|
| 116 |
+
single="$A",
|
| 117 |
+
special_tokens=[(BOS_TOKEN, tok.token_to_id(BOS_TOKEN)),
|
| 118 |
+
(EOS_TOKEN, tok.token_to_id(EOS_TOKEN))],
|
| 119 |
+
)
|
| 120 |
+
return cls(tok)
|
| 121 |
+
|
| 122 |
+
|
| 123 |
+
# ----------------------------- CLI: train on a text corpus ---------------------------------------
|
| 124 |
+
def _iter_corpus(path: str, max_sentences: int | None = None) -> Iterable[str]:
|
| 125 |
+
"""Yield text strings from a JSONL corpus file. Each line is `{"text": "..."}`."""
|
| 126 |
+
import json
|
| 127 |
+
n = 0
|
| 128 |
+
with open(path, encoding="utf-8") as fh:
|
| 129 |
+
for ln in fh:
|
| 130 |
+
ln = ln.strip()
|
| 131 |
+
if not ln:
|
| 132 |
+
continue
|
| 133 |
+
obj = json.loads(ln)
|
| 134 |
+
text = obj.get("text", "")
|
| 135 |
+
if not text:
|
| 136 |
+
continue
|
| 137 |
+
# Corpus records are full documents; split on newlines to feed BPE roughly
|
| 138 |
+
# sentence-sized chunks (BPE doesn't care about sentence boundaries but this caps
|
| 139 |
+
# memory).
|
| 140 |
+
for chunk in text.split("\n"):
|
| 141 |
+
chunk = chunk.strip()
|
| 142 |
+
if chunk:
|
| 143 |
+
yield chunk
|
| 144 |
+
n += 1
|
| 145 |
+
if max_sentences is not None and n >= max_sentences:
|
| 146 |
+
return
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
if __name__ == "__main__":
|
| 150 |
+
import argparse
|
| 151 |
+
import time
|
| 152 |
+
|
| 153 |
+
ap = argparse.ArgumentParser(description="Train an 8K BPE tokenizer for the tiny UD parser.")
|
| 154 |
+
ap.add_argument("--corpus", required=True,
|
| 155 |
+
help="input jsonl with a 'text' field (one document per line)")
|
| 156 |
+
ap.add_argument("--out", default="checkpoints/parser_tokenizer.json")
|
| 157 |
+
ap.add_argument("--vocab", type=int, default=8000)
|
| 158 |
+
ap.add_argument("--max-sentences", type=int, default=2_000_000,
|
| 159 |
+
help="cap on training chunks; 2 M is plenty for an 8K BPE.")
|
| 160 |
+
args = ap.parse_args()
|
| 161 |
+
|
| 162 |
+
t0 = time.time()
|
| 163 |
+
print(f"[BPE] streaming up to {args.max_sentences:,} chunks from {args.corpus}")
|
| 164 |
+
tok = BPETokenizer.train(_iter_corpus(args.corpus, args.max_sentences), vocab_size=args.vocab)
|
| 165 |
+
tok.save(args.out)
|
| 166 |
+
print(f"[BPE] vocab={tok.vocab_size} PAD={tok.pad_id} ROOT={tok.root_id} UNK={tok.unk_id}")
|
| 167 |
+
print(f"[BPE] saved → {args.out} ({time.time()-t0:.1f}s)")
|
| 168 |
+
demo = "Mies ostaa talon Helsingistä."
|
| 169 |
+
ids = tok.encode_with_root(demo)
|
| 170 |
+
print(f"[BPE] demo: {demo!r}")
|
| 171 |
+
print(f" ids: {ids}")
|
| 172 |
+
print(f" decoded: {tok.decode(ids)!r}")
|
juurio/pipeline.py
ADDED
|
@@ -0,0 +1,705 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Raw-text product pipeline: string -> parse -> scope/abstain -> grammatical transform.
|
| 2 |
+
|
| 3 |
+
This is the product surface over the core. It wires three pieces assigned to different components:
|
| 4 |
+
- STRUCTURE / scope -> a UD dependency parser (default: the torch-only Giella/Stanza-free
|
| 5 |
+
end-to-end neural parser `parser.neural_adapter.NeuralParser`; `StanzaParser` is an opt-in
|
| 6 |
+
backend), behind a swappable adapter
|
| 7 |
+
- idiosyncrasy-free morphology -> the net+lexicon backend: lexicon-first, neural realizers second
|
| 8 |
+
(`algebra.verb_cluster` + the nominal/verbform/finiteverb nets)
|
| 9 |
+
- open-vocabulary morphology -> the tiny lemma-anchored transformer (neural fallback)
|
| 10 |
+
|
| 11 |
+
`transform(op, text, lemma=None)` returns a `Result` with the output, an explicit abstain
|
| 12 |
+
reason, a trace, and warnings — abstention is a first-class result, not a silent failure.
|
| 13 |
+
|
| 14 |
+
SCOPE CONTRACT (what `transform` promises). It operates on a SINGLE finite main clause. It returns
|
| 15 |
+
a `Result` whose `ok` is True only when it produced a transformation; otherwise `ok=False` and
|
| 16 |
+
`reason` is one of `ABSTAIN_REASONS` (see `REASONS` for descriptions). In scope: one finite
|
| 17 |
+
Act/Ind main clause within `algebra.in_scope` limits (≈≤12 words, one finite verb). Out of scope
|
| 18 |
+
(→ abstain, never a guess): subordinate/coordinated clauses, already-negated clauses, multi-verb
|
| 19 |
+
fragments, op not applicable to the clause's base state, and OOV open-class words neither the lexicon
|
| 20 |
+
nor the neural realizer can inflect (`oov_unrealizable`). Neural realizations that DID succeed are
|
| 21 |
+
surfaced in `trace["neural_unverified"]` (deprecated alias `trace["neural_oov"]`) + a warning (model
|
| 22 |
+
guess, unverified against the lexicon — a provenance flag, not a claim the lemma is out-of-vocabulary).
|
| 23 |
+
|
| 24 |
+
Residual risks this module guards:
|
| 25 |
+
1. lemma-error propagation: the parser lemmatizer can mangle a loan stem (`striimaa`->`striä`),
|
| 26 |
+
which corrupts the class anchor. Mitigations: a `lemma=` override, and a confidence
|
| 27 |
+
warning when the parser lemma diverges early from the surface form.
|
| 28 |
+
2. parser subject-attachment differences.
|
| 29 |
+
|
| 30 |
+
When the optional Stanza backend is used, Stanza runs on CPU here; the transformer still uses the GPU.
|
| 31 |
+
"""
|
| 32 |
+
from __future__ import annotations
|
| 33 |
+
|
| 34 |
+
import os
|
| 35 |
+
import warnings
|
| 36 |
+
from collections.abc import Mapping
|
| 37 |
+
from dataclasses import dataclass, field
|
| 38 |
+
from types import MappingProxyType
|
| 39 |
+
|
| 40 |
+
from . import algebra
|
| 41 |
+
from . import features
|
| 42 |
+
from . import bootstrap
|
| 43 |
+
|
| 44 |
+
# stock nn.TransformerDecoder emits a prototype "nested tensors" UserWarning on every forward;
|
| 45 |
+
# it is benign and noise on the CLI. Silence just that one message.
|
| 46 |
+
warnings.filterwarnings("ignore", message=".*nested tensor.*", category=UserWarning)
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
# ----------------------------- parser adapter -----------------------------
|
| 50 |
+
class ParserAdapter:
|
| 51 |
+
"""Abstraction over a UD parser so the backend is swappable. A backend must turn raw text
|
| 52 |
+
into a list of sentences, each a list of token dicts in our format:
|
| 53 |
+
{id, form, lemma, upos, head(int, 0=root), deprel(base label), feats(dict)}."""
|
| 54 |
+
|
| 55 |
+
def parse(self, text: str) -> list[list[dict]]:
|
| 56 |
+
raise NotImplementedError
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
class StanzaParser(ParserAdapter):
|
| 60 |
+
_nlp = None # class-level cache: the model is expensive to load
|
| 61 |
+
|
| 62 |
+
def __init__(self, device: str = "cpu"):
|
| 63 |
+
self.device = device
|
| 64 |
+
|
| 65 |
+
def _pipe(self):
|
| 66 |
+
if StanzaParser._nlp is None:
|
| 67 |
+
import stanza
|
| 68 |
+
StanzaParser._nlp = stanza.Pipeline(
|
| 69 |
+
"fi", processors="tokenize,pos,lemma,depparse",
|
| 70 |
+
download_method=None, verbose=False, device=self.device)
|
| 71 |
+
return StanzaParser._nlp
|
| 72 |
+
|
| 73 |
+
@staticmethod
|
| 74 |
+
def _feats(s: str | None) -> dict:
|
| 75 |
+
return dict(kv.split("=", 1) for kv in s.split("|")) if s and s != "_" else {}
|
| 76 |
+
|
| 77 |
+
def parse(self, text: str) -> list[list[dict]]:
|
| 78 |
+
doc = self._pipe()(text)
|
| 79 |
+
out = []
|
| 80 |
+
for sent in doc.sentences:
|
| 81 |
+
out.append([{
|
| 82 |
+
"id": w.id, "form": w.text, "lemma": w.lemma or w.text, "upos": w.upos,
|
| 83 |
+
"head": w.head, "deprel": (w.deprel or "").split(":")[0],
|
| 84 |
+
"feats": self._feats(w.feats),
|
| 85 |
+
} for w in sent.words])
|
| 86 |
+
return out
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
# ----------------------------- result type + contract -----------------------------
|
| 90 |
+
# The full, documented set of `Result.reason` codes (the abstention contract).
|
| 91 |
+
REASONS = {
|
| 92 |
+
"ok": "a transformation was produced (Result.ok is True)",
|
| 93 |
+
"parse_empty": "the parser returned no usable sentence at sent_index",
|
| 94 |
+
"no_clause": "no finite Act/Ind main clause found (subordinate-only, fragment, "
|
| 95 |
+
"or already-negated clause that would double-negate)",
|
| 96 |
+
"out_of_scope": "clause exceeds scope limits (too long / >1 finite verb / coordination)",
|
| 97 |
+
"op_not_applicable": "the op does not apply to this clause's base state "
|
| 98 |
+
"(e.g. past on an already-past clause, passive on a copula clause)",
|
| 99 |
+
"oov_unrealizable": "an open-class word needed inflecting but NEITHER the FST nor the neural "
|
| 100 |
+
"realizer could produce the target form (explicit unsupported morphology)",
|
| 101 |
+
"oov_unverified": "the neural realizer produced a candidate for an OOV open-class word but it "
|
| 102 |
+
"failed validation (stem-inconsistent or near-tie) and was suppressed, not emitted",
|
| 103 |
+
"conflicting_features": "inflect() received two valid but contradictory inputs (e.g. spec "
|
| 104 |
+
"'mon gen' with number='sing', proper=True with a degree, or like= alongside "
|
| 105 |
+
"an explicit case/number/degree)",
|
| 106 |
+
"unanalyzable_model": "inflect(like=…) could not analyze the model word into a nominal "
|
| 107 |
+
"case/number cell to copy",
|
| 108 |
+
}
|
| 109 |
+
ABSTAIN_REASONS = frozenset(REASONS) - {"ok"}
|
| 110 |
+
|
| 111 |
+
|
| 112 |
+
@dataclass
|
| 113 |
+
class Result:
|
| 114 |
+
text: str | None # transformed sentence, or None if abstained/failed
|
| 115 |
+
ok: bool # True iff a transformation was produced
|
| 116 |
+
reason: str # one of REASONS (ok => ABSTAIN_REASONS otherwise)
|
| 117 |
+
trace: dict = field(default_factory=dict) # op, root_verb, parser_lemma, lemma_source,
|
| 118 |
+
warnings: list[str] = field(default_factory=list) # neural_unverified / oov_unrealizable detail, lemma warnings
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
@dataclass(frozen=True)
|
| 122 |
+
class TokenAnalysis:
|
| 123 |
+
"""One analyzed token from `analyze()` — the parser's native fields as a stable, typed record
|
| 124 |
+
(so the public contract is these named fields, not "whatever dict the parser emitted today").
|
| 125 |
+
`head` is the 1-based id of the syntactic head (0 = root); `feats` is the UD morphology mapping
|
| 126 |
+
(e.g. {'Case': 'Gen', 'Number': 'Plur'}), stored read-only so the record is truly immutable."""
|
| 127 |
+
id: int
|
| 128 |
+
form: str
|
| 129 |
+
lemma: str
|
| 130 |
+
upos: str
|
| 131 |
+
head: int
|
| 132 |
+
deprel: str
|
| 133 |
+
feats: Mapping[str, str]
|
| 134 |
+
|
| 135 |
+
def __post_init__(self):
|
| 136 |
+
# frozen dataclass: bypass the immutability guard to wrap feats read-only (a defensive copy)
|
| 137 |
+
object.__setattr__(self, "feats", MappingProxyType(dict(self.feats)))
|
| 138 |
+
|
| 139 |
+
def asdict(self) -> dict:
|
| 140 |
+
return {"id": self.id, "form": self.form, "lemma": self.lemma, "upos": self.upos,
|
| 141 |
+
"head": self.head, "deprel": self.deprel, "feats": dict(self.feats)}
|
| 142 |
+
|
| 143 |
+
|
| 144 |
+
# ----------------------------- engine -----------------------------
|
| 145 |
+
_QUOTES = "\"'«»“”„‘’*•"
|
| 146 |
+
_DASHES = "-–—"
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
def _strip_dialogue_prefix(text: str):
|
| 150 |
+
"""Stanza over-segments a leading dialogue marker (— or quote) into its OWN one-token sentence,
|
| 151 |
+
so `sents[0]` is just the punctuation and the real clause is lost -> false `no_clause`. Strip a
|
| 152 |
+
leading run of quotes/dialogue-dashes (a dash only when followed by space, so `-5` is safe) and
|
| 153 |
+
re-prepend it to the output. Returns (prefix, core)."""
|
| 154 |
+
i = 0
|
| 155 |
+
while i < len(text):
|
| 156 |
+
c = text[i]
|
| 157 |
+
if c.isspace() or c in _QUOTES:
|
| 158 |
+
i += 1
|
| 159 |
+
elif c in _DASHES and i + 1 < len(text) and text[i + 1].isspace():
|
| 160 |
+
i += 1
|
| 161 |
+
else:
|
| 162 |
+
break
|
| 163 |
+
return text[:i], text[i:]
|
| 164 |
+
|
| 165 |
+
|
| 166 |
+
def _common_prefix(a: str, b: str) -> int:
|
| 167 |
+
n = 0
|
| 168 |
+
for x, y in zip(a, b):
|
| 169 |
+
if x != y:
|
| 170 |
+
break
|
| 171 |
+
n += 1
|
| 172 |
+
return n
|
| 173 |
+
|
| 174 |
+
|
| 175 |
+
def _lemma_confidence_warning(form: str, lemma: str) -> str | None:
|
| 176 |
+
"""Flag a likely-bad parser lemma (the dominant residual risk), as a WARNING (override via
|
| 177 |
+
lemma=), not a hard abstain. Giella-FREE check (the served analysis path no longer calls the FST):
|
| 178 |
+
a correct lemma shares a long prefix with the form (striimaa/striimata=7), a mangled one diverges
|
| 179 |
+
early (striimaa/striä=4). This covers all words, since the neural parser supplies the lemma. An
|
| 180 |
+
earlier FST-round-trip branch (confirm the lemma is among `uralicApi.analyze(form)`'s readings) was
|
| 181 |
+
removed to keep generation+analysis Giella-free at serving."""
|
| 182 |
+
f, l = form.lower(), lemma.lower()
|
| 183 |
+
cp = _common_prefix(f, l)
|
| 184 |
+
if cp < 0.6 * len(f):
|
| 185 |
+
return f"low-confidence lemma '{lemma}' for form '{form}' (prefix {cp}/{len(f)}); pass lemma= to override"
|
| 186 |
+
return None
|
| 187 |
+
|
| 188 |
+
|
| 189 |
+
def _optional_artifact(key: str) -> str | None:
|
| 190 |
+
"""Resolve serving artifact `key` via bootstrap, or None if it can't be fetched. Used for the OOV
|
| 191 |
+
realizers, which degrade gracefully (abstain) rather than hard-fail when their artifact is absent.
|
| 192 |
+
A sha256 mismatch (corrupt/tampered bytes) is NOT graceful — it always propagates."""
|
| 193 |
+
try:
|
| 194 |
+
return bootstrap.artifact_path(key)
|
| 195 |
+
except bootstrap.ArtifactIntegrityError:
|
| 196 |
+
raise
|
| 197 |
+
except Exception:
|
| 198 |
+
return None
|
| 199 |
+
|
| 200 |
+
|
| 201 |
+
def _default_parser(**kwargs) -> ParserAdapter:
|
| 202 |
+
"""Default SERVING parser: the Giella/Stanza-free end-to-end neural parser
|
| 203 |
+
(`juurio.parser.neural_adapter.NeuralParser`).
|
| 204 |
+
Prefers the 7.3 MB compressed FST-tag table over the 67.9 MB full cache when present. `StanzaParser`
|
| 205 |
+
remains available as an opt-in backend (`Transformer(parser=StanzaParser())`) for benchmarking, but
|
| 206 |
+
is no longer a runtime dependency. Extra kwargs pass through to `NeuralParser` (e.g. `device=`)."""
|
| 207 |
+
from .parser.neural_adapter import NeuralParser
|
| 208 |
+
# Prefer the 7.3 MB compressed table; resolve it via bootstrap (dev source -> HuggingFace) so a clean
|
| 209 |
+
# wheel install fetches it. NeuralParser falls back to the full 67.9 MB FST cache only if unset.
|
| 210 |
+
# Guard the resolve so an explicit caller override (or None) doesn't trigger a fetch (offline-safe).
|
| 211 |
+
if "compressed_table" not in kwargs:
|
| 212 |
+
kwargs["compressed_table"] = bootstrap.artifact_path("fst_table")
|
| 213 |
+
return NeuralParser(**kwargs)
|
| 214 |
+
|
| 215 |
+
|
| 216 |
+
def _oov_abstain_result(trace: dict) -> Result:
|
| 217 |
+
"""Abstain Result for a token the morphology engine could not realize, with the reason taken from
|
| 218 |
+
the algebra OOV trace flags: an unverified neural candidate (suppressed) outranks nothing-producible.
|
| 219 |
+
Shared by the nominal realization paths (`realize_np`, `inflect`) so the mapping can't drift."""
|
| 220 |
+
if algebra._OOV_UNVERIFIED:
|
| 221 |
+
trace["oov_unverified"] = list(algebra._OOV_UNVERIFIED)
|
| 222 |
+
return Result(None, False, "oov_unverified", trace, [])
|
| 223 |
+
trace["oov_unrealizable"] = list(algebra._OOV_UNREALIZABLE)
|
| 224 |
+
return Result(None, False, "oov_unrealizable", trace, [])
|
| 225 |
+
|
| 226 |
+
|
| 227 |
+
def _record_neural_unverified(trace: dict, items: list) -> list:
|
| 228 |
+
"""Record net-realized forms in the trace and return them. These are model GUESSES the FST could not
|
| 229 |
+
verify at runtime — a PROVENANCE flag (neural- vs lexicon-served), NOT a vocabulary claim: the lemma
|
| 230 |
+
may be perfectly known (the shipped lexicon is exception-only, so common regular words like `talo`
|
| 231 |
+
are net-served). Hence the label is "neural realization (unverified by FST)", not "neural OOV…".
|
| 232 |
+
`neural_unverified` is the canonical trace key; `neural_oov` is a DEPRECATED back-compat alias kept
|
| 233 |
+
for one release cycle (prefer `neural_unverified`)."""
|
| 234 |
+
trace["neural_unverified"] = items
|
| 235 |
+
trace["neural_oov"] = items
|
| 236 |
+
return items
|
| 237 |
+
|
| 238 |
+
|
| 239 |
+
def _ok_with_oov_warnings(text: str, trace: dict) -> Result:
|
| 240 |
+
"""Success Result, surfacing any neural realizations on this call as unverified warnings."""
|
| 241 |
+
warnings_out: list[str] = []
|
| 242 |
+
if algebra._NEURAL_OOV:
|
| 243 |
+
_record_neural_unverified(trace, list(algebra._NEURAL_OOV))
|
| 244 |
+
warnings_out.append(f"neural realization (unverified by FST): {algebra._NEURAL_OOV}")
|
| 245 |
+
return Result(text, True, "ok", trace, warnings_out)
|
| 246 |
+
|
| 247 |
+
|
| 248 |
+
class Transformer:
|
| 249 |
+
"""Loads the parser + the lemma-anchored model once, then serves `transform`."""
|
| 250 |
+
|
| 251 |
+
def __init__(self, parser: ParserAdapter | None = None, ckpt: str | None = None,
|
| 252 |
+
noun_realizer: bool = True):
|
| 253 |
+
try:
|
| 254 |
+
self.parser = parser or _default_parser()
|
| 255 |
+
finally:
|
| 256 |
+
# Building the parser imports the internal `juurio.analyze` subpackage, which shadows the
|
| 257 |
+
# public `analyze`/`lemmatize` convenience funcs as package attrs. Restore them in `finally`
|
| 258 |
+
# so the module-level API stays callable even if construction below fails (see __init__.py).
|
| 259 |
+
from . import _restore_public_names
|
| 260 |
+
_restore_public_names()
|
| 261 |
+
self._bundle = None # unused; kept for back-compat with the hybrid_transform signature
|
| 262 |
+
# OOV verb/noun realizers (Giella-free): the FST inflects known words; when it can't (loan/OOV
|
| 263 |
+
# word) these neural nets fill in. All optional + guarded — an absent checkpoint just means
|
| 264 |
+
# that OOV class abstains. `ckpt` is accepted for API compat but no longer loads a verb model.
|
| 265 |
+
# Reset the module globals first so a later construction (e.g. a missing checkpoint, or
|
| 266 |
+
# noun_realizer=False) cannot inherit a previous Transformer's realizers in a long-lived process.
|
| 267 |
+
algebra._NOUN_REALIZER = None
|
| 268 |
+
algebra._VERBFORM_REALIZER = None
|
| 269 |
+
algebra._FINITEVERB_BUNDLE = None
|
| 270 |
+
# Generation backend: Giella-free lexicon-first + net-second. Reset the globals so a prior
|
| 271 |
+
# construction never leaks its backend state into a new one.
|
| 272 |
+
algebra._NETLEX_BACKEND = True
|
| 273 |
+
algebra._GENLEX = None
|
| 274 |
+
# OOV finite/non-finite verb realizers: these are independent of `noun_realizer` (which only
|
| 275 |
+
# gates the nominal net) — they are the Giella-free OOV verb path and load whenever their
|
| 276 |
+
# checkpoints are present. finiteverb covers finite cells (incl. imperative + connegative)
|
| 277 |
+
# and verbform the past participles; both feed `algebra._netlex_verb_cluster`/`_neural_imprt`.
|
| 278 |
+
# Resolve each realizer artifact through bootstrap (dev source -> HuggingFace). These OOV
|
| 279 |
+
# realizers stay OPTIONAL: if an artifact can't be resolved (e.g. offline with an empty cache)
|
| 280 |
+
# we leave the global None and the engine abstains on OOV — same graceful degradation as a
|
| 281 |
+
# missing checkpoint before. The netlex backend's wordlist (below) is the hard require.
|
| 282 |
+
from . import verbforms
|
| 283 |
+
vf = _optional_artifact("verbform")
|
| 284 |
+
if vf:
|
| 285 |
+
algebra._VERBFORM_REALIZER = verbforms.make_verbform_realizer(ckpt=vf)
|
| 286 |
+
from . import finiteverb
|
| 287 |
+
fv = _optional_artifact("finiteverb")
|
| 288 |
+
if fv:
|
| 289 |
+
algebra._FINITEVERB_BUNDLE = finiteverb.load(ckpt=fv)
|
| 290 |
+
if noun_realizer:
|
| 291 |
+
from . import realizer
|
| 292 |
+
nr = _optional_artifact("realizer")
|
| 293 |
+
if nr:
|
| 294 |
+
algebra._NOUN_REALIZER = realizer.make_noun_realizer(ckpt=nr)
|
| 295 |
+
# Net+lexicon backend reuses the realizer bundles loaded above + the static exception table.
|
| 296 |
+
from . import genlex
|
| 297 |
+
algebra._GENLEX = genlex.GenerationLexicon.load()
|
| 298 |
+
|
| 299 |
+
def transform(self, op: str, text: str, lemma: str | None = None,
|
| 300 |
+
sent_index: int = 0) -> Result:
|
| 301 |
+
if op not in algebra.ALL_OPS and algebra.parse_nominal_bundle(op) is None:
|
| 302 |
+
raise ValueError(f"unknown op {op!r}; known atoms: {sorted(algebra.ALL_OPS)}; or a "
|
| 303 |
+
f"'+'-composed nominal bundle e.g. 'plural+poss_1sg+clitic_kin'")
|
| 304 |
+
prefix, core = _strip_dialogue_prefix(text) # un-split leading dialogue dash/quote
|
| 305 |
+
sents = self.parser.parse(core)
|
| 306 |
+
if not sents or sent_index >= len(sents):
|
| 307 |
+
return Result(None, False, "parse_empty")
|
| 308 |
+
sent = sents[sent_index]
|
| 309 |
+
|
| 310 |
+
# lemma/UPOS hint repair: if the caller named the main verb but the parser mis-tagged the clause
|
| 311 |
+
# root as a non-verb (the ambiguous-loan-verb failure, e.g. 'striimaa'->NOUN), re-tag it so
|
| 312 |
+
# clause detection succeeds. Conservative: only when there is no finite verb + the hint stem-
|
| 313 |
+
# matches (see algebra.repair_root_verb_with_lemma).
|
| 314 |
+
lemma_repaired = bool(lemma) and algebra.repair_root_verb_with_lemma(sent, lemma)
|
| 315 |
+
|
| 316 |
+
verb, _ = algebra._main_verb(sent)
|
| 317 |
+
trace = {"op": op, "root_verb": verb["form"] if verb else None,
|
| 318 |
+
"parser_lemma": verb["lemma"] if verb else None}
|
| 319 |
+
if lemma_repaired:
|
| 320 |
+
trace["lemma_repair"] = True
|
| 321 |
+
warnings: list[str] = []
|
| 322 |
+
|
| 323 |
+
# scope / abstention is a first-class result. The imperative family has its own scope
|
| 324 |
+
# (subjectless / 2nd-person source — see algebra.imperative_in_scope); every other op uses
|
| 325 |
+
# the general finite Act/Ind clause gate.
|
| 326 |
+
if op in algebra.IMPERATIVE_OPS:
|
| 327 |
+
if not algebra.imperative_in_scope(sent):
|
| 328 |
+
trace["scope"] = "out_of_scope"
|
| 329 |
+
return Result(None, False, "out_of_scope", trace, warnings)
|
| 330 |
+
else:
|
| 331 |
+
if algebra.clause(sent) is None:
|
| 332 |
+
return Result(None, False, "no_clause", trace, warnings)
|
| 333 |
+
if not algebra.in_scope(sent):
|
| 334 |
+
trace["scope"] = "out_of_scope"
|
| 335 |
+
return Result(None, False, "out_of_scope", trace, warnings)
|
| 336 |
+
|
| 337 |
+
# lemma override (the parser lemma is used otherwise) + confidence guard
|
| 338 |
+
if lemma is not None and verb is not None:
|
| 339 |
+
trace["lemma_source"] = "override"
|
| 340 |
+
verb["lemma"] = lemma
|
| 341 |
+
elif verb is not None:
|
| 342 |
+
trace["lemma_source"] = "parser"
|
| 343 |
+
w = _lemma_confidence_warning(verb["form"], verb["lemma"])
|
| 344 |
+
if w:
|
| 345 |
+
warnings.append(w)
|
| 346 |
+
|
| 347 |
+
# transform: deterministic structure + FST verb, neural fallback for OOV (hybrid)
|
| 348 |
+
out = algebra.hybrid_transform(sent, op, self._bundle)
|
| 349 |
+
if out is None:
|
| 350 |
+
if algebra._OOV_UNVERIFIED: # neural produced a candidate but we suppressed it
|
| 351 |
+
trace["oov_unverified"] = list(algebra._OOV_UNVERIFIED)
|
| 352 |
+
return Result(None, False, "oov_unverified", trace, warnings)
|
| 353 |
+
if algebra._OOV_UNREALIZABLE: # FST+neural both failed on an open-class word
|
| 354 |
+
trace["oov_unrealizable"] = list(algebra._OOV_UNREALIZABLE)
|
| 355 |
+
return Result(None, False, "oov_unrealizable", trace, warnings)
|
| 356 |
+
return Result(None, False, "op_not_applicable", trace, warnings)
|
| 357 |
+
# surface any neural realizations: these are model guesses the FST could not verify at
|
| 358 |
+
# runtime (provenance, not vocabulary), so the caller can weigh them appropriately
|
| 359 |
+
if algebra._NEURAL_OOV:
|
| 360 |
+
_record_neural_unverified(trace, list(algebra._NEURAL_OOV))
|
| 361 |
+
warnings.append(f"neural realization (unverified by FST): {algebra._NEURAL_OOV}")
|
| 362 |
+
return Result(prefix + out, True, "ok", trace, warnings)
|
| 363 |
+
|
| 364 |
+
def realize_np(self, np: str, case: str = "Nom", number: str = "Sing",
|
| 365 |
+
px: str = "None", clitic: str | None = None) -> Result:
|
| 366 |
+
"""PHRASE-LEVEL nominal morphology utility — NOT a clause transform (case is syntactically
|
| 367 |
+
governed). Inflect an NP (a bare lemma or a short `adj* noun` phrase) into
|
| 368 |
+
the target Case/Number, optional possessive (`px`) + enclitic, with agreement on modifiers
|
| 369 |
+
and per-token FST->neural fallback. Surface order stem+case+number+Px+clitic. Returns a `Result`:
|
| 370 |
+
`ok` with the realized NP, or an abstain reason (`parse_empty`/`no_clause` for no nominal
|
| 371 |
+
head, `oov_unverified`/`oov_unrealizable` for a head the morphology engine cannot produce)."""
|
| 372 |
+
algebra._NEURAL_OOV.clear()
|
| 373 |
+
algebra._OOV_UNREALIZABLE.clear()
|
| 374 |
+
algebra._OOV_UNVERIFIED.clear()
|
| 375 |
+
sents = self.parser.parse(np)
|
| 376 |
+
toks = sents[0] if sents else []
|
| 377 |
+
head = next((t for t in reversed(toks) if t["upos"] in ("NOUN", "PROPN")), None)
|
| 378 |
+
if head is None and len(toks) == 1 and toks[0]["upos"] not in ("PUNCT", "SYM"):
|
| 379 |
+
# A single bare word handed to realize_np (a NOMINAL utility) is a noun, even when the
|
| 380 |
+
# context-free parser reads it as a homograph verb (suo=suoda, tuli=past of tulla). Force the
|
| 381 |
+
# NOUN reading so the morphology runs instead of abstaining `no_clause` on a one-word NP.
|
| 382 |
+
toks = [{**toks[0], "upos": "NOUN", "lemma": toks[0]["form"]}]
|
| 383 |
+
head = toks[0]
|
| 384 |
+
trace = {"util": "realize_np", "case": case, "number": number, "px": px, "clitic": clitic,
|
| 385 |
+
"head": head["form"] if head else None}
|
| 386 |
+
if head is None:
|
| 387 |
+
return Result(None, False, "no_clause", trace, []) # no nominal head to inflect
|
| 388 |
+
out = []
|
| 389 |
+
# The nominal HEAD is a citation lemma per realize_np's contract, so its SURFACE already IS its
|
| 390 |
+
# lemma — use `t["form"]`, NOT the parser's re-lemmatisation: the neural lemmatiser mis-maps
|
| 391 |
+
# lexicalised nominals to verb infinitives (valtuutettu->valtuuttaa, reititin->reitittää,
|
| 392 |
+
# vietti->viettää) or the wrong stem (neitsyt->neitsy), which would inflect into a non-word.
|
| 393 |
+
# Agreeing MODIFIERS keep the parser lemma: the parser carries their Degree in `feats` and supplies
|
| 394 |
+
# the base adjective, so a comparative input ("isompi talo") realizes from lemma `iso`+Degree=Cmp
|
| 395 |
+
# (-> isommassa); using the surface `isompi` as the lemma would double-apply degree and abstain.
|
| 396 |
+
for t in toks:
|
| 397 |
+
if t["id"] == head["id"]:
|
| 398 |
+
f = algebra.realize_np_form(t["form"], t["upos"], case, number, px, clitic, t["form"])
|
| 399 |
+
elif t["upos"] in ("ADJ", "NUM") or t["deprel"] in algebra.AGREE:
|
| 400 |
+
f = algebra._realize_form(t["lemma"], t["upos"],
|
| 401 |
+
{**t["feats"], "Case": case, "Number": number}, t["form"])
|
| 402 |
+
else:
|
| 403 |
+
f = t["form"] # leave non-agreeing tokens as-is
|
| 404 |
+
if f is None:
|
| 405 |
+
return _oov_abstain_result(trace)
|
| 406 |
+
out.append(f)
|
| 407 |
+
return _ok_with_oov_warnings(" ".join(out), trace)
|
| 408 |
+
|
| 409 |
+
def realize_verbform(self, verb: str, form: str = "va", case: str = "Nom",
|
| 410 |
+
number: str = "Sing", px: str = "None", lemma: str | None = None) -> Result:
|
| 411 |
+
"""PHRASE-LEVEL non-finite verb morphology — NOT a clause transform. Inflect a VERB into a
|
| 412 |
+
participle (`va`/`nut`/`tava`/`tu`/`ma`, which inflect in case+number) or an infinitive
|
| 413 |
+
(`inf2`/`inf2_pass`/`inf3`, case-marked: menemässä/menemästä/menemään/menemällä/menemättä).
|
| 414 |
+
Pass `px` (a possessive-suffix value) for the temporal/possessive construction: form='tu',
|
| 415 |
+
case='Par', px='3' -> 'kuultuaan' ('after hearing'). `verb` may be a bare lemma or a short
|
| 416 |
+
clause (the main verb is used); pass `lemma=` to override the parser for a mis-lemmatized loan.
|
| 417 |
+
FST-backed (known verbs); for an OOV verb the verbform realizer serves the bare participle and,
|
| 418 |
+
with `px`, the noun realizer attaches case+Px (participles inflect like nominals) — both neural
|
| 419 |
+
stages flagged unverified. Absent the realizers, abstains. `form` is one of `algebra.VERBFORMS`."""
|
| 420 |
+
algebra._NEURAL_OOV.clear()
|
| 421 |
+
algebra._OOV_UNREALIZABLE.clear()
|
| 422 |
+
algebra._OOV_UNVERIFIED.clear()
|
| 423 |
+
if form not in algebra.VERBFORMS:
|
| 424 |
+
raise ValueError(f"unknown verb form {form!r}; known: {algebra.VERBFORMS}")
|
| 425 |
+
if lemma is None:
|
| 426 |
+
sents = self.parser.parse(verb)
|
| 427 |
+
toks = sents[0] if sents else []
|
| 428 |
+
vtok = next((t for t in toks if t["upos"] == "VERB"), None)
|
| 429 |
+
if vtok is not None:
|
| 430 |
+
lemma = vtok["lemma"]
|
| 431 |
+
elif len(verb.split()) == 1:
|
| 432 |
+
# single bare word with no VERB reading: this is a verb utility, so trust the caller's
|
| 433 |
+
# word as the verb lemma. The parser is context-free and re-lemmatises homographs to the
|
| 434 |
+
# wrong POS ('haukkua' -> noun 'haukku' -> garbage participle 'haukkuvussu'); honouring
|
| 435 |
+
# the raw infinitive keeps it correct ('haukkuvassa').
|
| 436 |
+
lemma = verb.strip()
|
| 437 |
+
else:
|
| 438 |
+
# genuine multiword clause with no VERB token: best-effort last-token fallback.
|
| 439 |
+
lemma = toks[-1]["lemma"] if toks else verb
|
| 440 |
+
trace = {"util": "realize_verbform", "form": form, "case": case, "number": number,
|
| 441 |
+
"px": px, "lemma": lemma}
|
| 442 |
+
# 1. FST (known verbs), Px-aware (kuulla,tu,Par,3 -> kuultuansa). In netlex mode this also serves
|
| 443 |
+
# the verbform NET, flagging via algebra._NEURAL_OOV; propagate that so a net-served verbform is
|
| 444 |
+
# honestly marked `neural_unverified` (mirrors the nominal paths), not passed off as FST-verified.
|
| 445 |
+
f = algebra.realize_verbform_form(lemma, form, case, number, px=px)
|
| 446 |
+
if f is not None:
|
| 447 |
+
if algebra._NEURAL_OOV:
|
| 448 |
+
_record_neural_unverified(trace, list(algebra._NEURAL_OOV))
|
| 449 |
+
return Result(f, True, "ok", trace,
|
| 450 |
+
[f"neural realization (unverified by FST): {list(algebra._NEURAL_OOV)}"])
|
| 451 |
+
return Result(f, True, "ok", trace, [])
|
| 452 |
+
# 2. OOV. Without a possessive suffix this is the single-stage verbform realizer.
|
| 453 |
+
if px == "None":
|
| 454 |
+
if algebra._VERBFORM_REALIZER is not None:
|
| 455 |
+
vf, conf = algebra._VERBFORM_REALIZER(lemma, form, case, number)
|
| 456 |
+
if vf and algebra._stem_consistent(lemma, vf) and conf >= algebra._OOV_CONF_MIN:
|
| 457 |
+
_record_neural_unverified(trace, [f"{lemma}->{vf} ({form})"])
|
| 458 |
+
return Result(vf, True, "ok", trace,
|
| 459 |
+
[f"neural realization (unverified by FST): ['{lemma}->{vf} ({form})']"])
|
| 460 |
+
if vf:
|
| 461 |
+
trace["oov_unverified"] = [f"{lemma}->{vf} ({form}) conf={conf:.2f}"]
|
| 462 |
+
return Result(None, False, "oov_unverified", trace, [])
|
| 463 |
+
trace["oov_unrealizable"] = [f"{lemma} (VERB) {form}/{case}/{number}"]
|
| 464 |
+
return Result(None, False, "oov_unrealizable", trace, [])
|
| 465 |
+
# 2b. OOV + Px: compose. Bare participle stem from the verbform realizer, then case+Px from the
|
| 466 |
+
# noun realizer (participles inflect like nominals). Two neural hops -> flagged unverified.
|
| 467 |
+
if algebra._VERBFORM_REALIZER is not None and algebra._NOUN_REALIZER is not None:
|
| 468 |
+
stem, conf = algebra._VERBFORM_REALIZER(lemma, form, "Nom", "Sing")
|
| 469 |
+
if stem and algebra._stem_consistent(lemma, stem) and conf >= algebra._OOV_CONF_MIN:
|
| 470 |
+
out = algebra._realize_form(stem, "NOUN",
|
| 471 |
+
{"Case": case, "Number": number, "Px": px}, stem)
|
| 472 |
+
if out is not None:
|
| 473 |
+
_record_neural_unverified(trace, [f"{lemma}->{stem}->{out} ({form}+Px{px})"])
|
| 474 |
+
return Result(out, True, "ok", trace,
|
| 475 |
+
[f"neural realization (unverified by FST): ['{lemma}->{out} ({form}+Px)']"])
|
| 476 |
+
# _realize_form recorded why the Px step failed (unverified vs unrealizable)
|
| 477 |
+
if algebra._OOV_UNVERIFIED:
|
| 478 |
+
trace["oov_unverified"] = list(algebra._OOV_UNVERIFIED)
|
| 479 |
+
return Result(None, False, "oov_unverified", trace, [])
|
| 480 |
+
trace["oov_unrealizable"] = [f"{lemma} (VERB) {form}/{case}/{number}/Px{px}"]
|
| 481 |
+
return Result(None, False, "oov_unrealizable", trace, [])
|
| 482 |
+
|
| 483 |
+
# --------------------------- friendly nominal inflection -----------------------------------
|
| 484 |
+
def inflect(self, word: str, case=features.UNSET, number=features.UNSET,
|
| 485 |
+
degree=features.UNSET, *, px=features.UNSET, clitic: str | None = None,
|
| 486 |
+
proper: bool | None = None, pos: str | None = None,
|
| 487 |
+
like: str | None = None) -> Result:
|
| 488 |
+
"""Inflect a single nominal (or short `adj* noun` phrase) into a target case/number, the
|
| 489 |
+
ergonomic front door over `realize_np`. Accepts human-readable aliases:
|
| 490 |
+
|
| 491 |
+
inflect("talo", "gen").text # 'talon' (English/Finnish abbrev)
|
| 492 |
+
inflect("talo", "mon gen").text # 'talojen' (combined spec, taivutin-style)
|
| 493 |
+
inflect("talo", case="inessive", number="plural").text # 'taloissa'
|
| 494 |
+
inflect("talo", px="1sg").text # 'taloni'
|
| 495 |
+
inflect("kaunis", "gen", degree="comp").text # 'kauniimman' (-> ADJ)
|
| 496 |
+
inflect("Meri", "gen", proper=True).text # proper-noun reading (vs 'meri' -> 'meren')
|
| 497 |
+
inflect("makaronilaatikko", like="talojen").text # copy 'talojen's cell -> plural gen
|
| 498 |
+
|
| 499 |
+
Aliases are normalized in `juurio.features`; an UNKNOWN alias raises ValueError (a typo
|
| 500 |
+
fails fast). Contradictory-but-valid inputs ABSTAIN with reason `conflicting_features`
|
| 501 |
+
(e.g. spec 'mon gen' + number='sing'; proper=True + a degree; like= + an explicit
|
| 502 |
+
case/number/degree/px — like= copies the model's whole cell, possessive included).
|
| 503 |
+
like= transfers Case/Number/Px always; a model's Degree (the model was an adjective) transfers
|
| 504 |
+
ONLY to an explicit adjective target (`pos='adjective'`) — on any other target it is inapplicable
|
| 505 |
+
and dropped (recorded in `trace['dropped_features']`), never realized as a non-word.
|
| 506 |
+
POS: explicit `pos` ('noun'/'proper'/'adjective') wins; `proper` is sugar; otherwise inferred
|
| 507 |
+
(degree => ADJ, leading capital => PROPN, else NOUN). A single token is realized parser-free
|
| 508 |
+
(so `proper`/`pos` are honored); a whitespace phrase delegates to `realize_np` (which rejects
|
| 509 |
+
the single-word controls `proper`/`pos`/`degree`, but — like realize_np — still applies
|
| 510 |
+
`px`/`clitic` to the phrase head). `px`/`clitic` are rejected on a single ADJ target (only
|
| 511 |
+
nominal heads take them). An empty/whitespace `word` raises ValueError. Returns a `Result`."""
|
| 512 |
+
word = word.strip()
|
| 513 |
+
if not word:
|
| 514 |
+
raise ValueError("inflect() requires a non-empty word")
|
| 515 |
+
trace: dict = {"util": "inflect", "word": word}
|
| 516 |
+
# --- features: from a model word (like=) or from case/number/degree (with spec support) ---
|
| 517 |
+
if like is not None:
|
| 518 |
+
# like= copies the model word's WHOLE nominal cell (case/number/degree, any possessive, AND
|
| 519 |
+
# a trailing focus/interrogative enclitic), so an explicit case/number/degree/px/clitic
|
| 520 |
+
# alongside it is contradictory -> abstain.
|
| 521 |
+
if (case is not features.UNSET or number is not features.UNSET
|
| 522 |
+
or degree is not features.UNSET or px is not features.UNSET
|
| 523 |
+
or clitic is not None):
|
| 524 |
+
return Result(None, False, "conflicting_features",
|
| 525 |
+
{**trace, "like": like,
|
| 526 |
+
"error": "like= is exclusive with case/number/degree/px/clitic"}, [])
|
| 527 |
+
cell = self._analyze_nominal_cell(like)
|
| 528 |
+
if cell is None:
|
| 529 |
+
return Result(None, False, "unanalyzable_model", {**trace, "like": like}, [])
|
| 530 |
+
feats = {"Case": cell.get("Case", "Nom"), "Number": cell.get("Number", "Sing"),
|
| 531 |
+
"Degree": cell.get("Degree", "Pos")}
|
| 532 |
+
px_v = cell.get("Px", "None") # a possessive on the model (e.g. koiriemme -> '1Pl') carries over
|
| 533 |
+
clitic = cell.get("Clitic") # an enclitic on the model (makaronilaatikkoonko -> 'ko') carries over
|
| 534 |
+
trace["like"] = like
|
| 535 |
+
# like_cell mirrors the model's OWN cell: show Degree only if the model actually carried one
|
| 536 |
+
# (a noun model has no degree -> don't fabricate a phantom Degree=Pos in the trace).
|
| 537 |
+
trace["like_cell"] = {"Case": feats["Case"], "Number": feats["Number"],
|
| 538 |
+
**({"Degree": feats["Degree"]} if cell.get("Degree") else {}),
|
| 539 |
+
**({"Px": px_v} if px_v != "None" else {}),
|
| 540 |
+
**({"Clitic": clitic} if clitic else {})}
|
| 541 |
+
# A Degree copied from a like= model (i.e. the model word was an adjective) is only an
|
| 542 |
+
# APPLICABLE feature when the TARGET is itself an adjective — which the user states with
|
| 543 |
+
# pos='adjective'. Left in for any other target it would coerce the POS to ADJ (degree => ADJ
|
| 544 |
+
# in resolve_pos) and fabricate a non-word (talo + kauniimmassa -> 'taloimmassa'). So on a
|
| 545 |
+
# non-adjective target the inapplicable degree is dropped (recorded), never realized.
|
| 546 |
+
if feats["Degree"] != "Pos":
|
| 547 |
+
try:
|
| 548 |
+
target_is_adj = features.resolve_pos(word, pos=pos, proper=proper) == "ADJ"
|
| 549 |
+
except (features.FeatureConflict, ValueError):
|
| 550 |
+
target_is_adj = False # a real pos/proper conflict surfaces below; just don't keep degree
|
| 551 |
+
if not target_is_adj:
|
| 552 |
+
trace["dropped_features"] = ["Degree"]
|
| 553 |
+
feats["Degree"] = "Pos"
|
| 554 |
+
else:
|
| 555 |
+
try:
|
| 556 |
+
feats = features.resolve_features(
|
| 557 |
+
"nom" if case is features.UNSET else case, number, degree)
|
| 558 |
+
except features.FeatureConflict as e:
|
| 559 |
+
return Result(None, False, "conflicting_features", {**trace, "error": str(e)}, [])
|
| 560 |
+
# a plain ValueError (unknown alias) propagates: a typo fails fast
|
| 561 |
+
px_v = "None" if px is features.UNSET else features.normalize_px(px)
|
| 562 |
+
clitic = features.normalize_clitic(clitic) # 'ko'/'kö'/'kaan' -> canonical; unknown -> ValueError
|
| 563 |
+
trace.update(case=feats["Case"], number=feats["Number"], degree=feats["Degree"],
|
| 564 |
+
px=px_v, clitic=clitic)
|
| 565 |
+
|
| 566 |
+
# --- phrase vs single token ---------------------------------------------------------------
|
| 567 |
+
if len(word.split()) > 1:
|
| 568 |
+
if proper is not None or pos is not None or feats["Degree"] != "Pos":
|
| 569 |
+
return Result(None, False, "conflicting_features",
|
| 570 |
+
{**trace, "error": "proper=/pos=/degree are single-word only; a phrase "
|
| 571 |
+
"inflects its head with modifier agreement via realize_np"}, [])
|
| 572 |
+
res = self.realize_np(word, case=feats["Case"], number=feats["Number"],
|
| 573 |
+
px=px_v, clitic=clitic)
|
| 574 |
+
# realize_np returns its own trace; carry forward a recorded like= drop so the annotation
|
| 575 |
+
# isn't lost for phrase targets (single-token targets keep `trace` and so already show it).
|
| 576 |
+
if res.ok and "dropped_features" in trace:
|
| 577 |
+
res.trace.setdefault("dropped_features", trace["dropped_features"])
|
| 578 |
+
return res
|
| 579 |
+
|
| 580 |
+
# --- single token: resolve POS, then realize parser-free ----------------------------------
|
| 581 |
+
try:
|
| 582 |
+
upos = features.resolve_pos(word, pos=pos, proper=proper, degree=feats["Degree"])
|
| 583 |
+
except features.FeatureConflict as e:
|
| 584 |
+
return Result(None, False, "conflicting_features", {**trace, "error": str(e)}, [])
|
| 585 |
+
trace["upos"] = upos
|
| 586 |
+
|
| 587 |
+
algebra._NEURAL_OOV.clear()
|
| 588 |
+
algebra._OOV_UNREALIZABLE.clear()
|
| 589 |
+
algebra._OOV_UNVERIFIED.clear()
|
| 590 |
+
if upos == "ADJ":
|
| 591 |
+
if px_v != "None" or clitic is not None:
|
| 592 |
+
return Result(None, False, "conflicting_features",
|
| 593 |
+
{**trace, "error": "px/clitic are not supported on adjectives"}, [])
|
| 594 |
+
f = algebra._realize_form(word, "ADJ", {"Case": feats["Case"],
|
| 595 |
+
"Number": feats["Number"], "Degree": feats["Degree"]}, word)
|
| 596 |
+
else: # NOUN / PROPN
|
| 597 |
+
f = algebra.realize_np_form(word, upos, feats["Case"], feats["Number"], px_v, clitic, word)
|
| 598 |
+
if f is None:
|
| 599 |
+
return _oov_abstain_result(trace)
|
| 600 |
+
return _ok_with_oov_warnings(f, trace)
|
| 601 |
+
|
| 602 |
+
def _nominal_head(self, word: str) -> dict | None:
|
| 603 |
+
"""Parse `word` and return its rightmost nominal head token (NOUN/PROPN/ADJ/PRON/NUM), or None."""
|
| 604 |
+
sents = self.parser.parse(word)
|
| 605 |
+
toks = sents[0] if sents else []
|
| 606 |
+
return next((tk for tk in reversed(toks)
|
| 607 |
+
if tk["upos"] in ("NOUN", "PROPN", "ADJ", "PRON", "NUM")), None)
|
| 608 |
+
|
| 609 |
+
@staticmethod
|
| 610 |
+
def _cell_from_head(head: dict) -> dict:
|
| 611 |
+
"""Build the nominal {Case[,Number][,Degree][,Px]} cell from a parsed head token."""
|
| 612 |
+
feats = head.get("feats", {})
|
| 613 |
+
cell = {k: feats[k] for k in ("Case", "Number") if k in feats}
|
| 614 |
+
if head["upos"] == "ADJ" and "Degree" in feats:
|
| 615 |
+
cell["Degree"] = feats["Degree"]
|
| 616 |
+
# carry a possessive suffix too (UD splits it into Person[psor]/Number[psor]); a model word
|
| 617 |
+
# like 'koiriemme' is gen-plur-1Pl, so the whole cell — not just case/number — must copy over.
|
| 618 |
+
px = features.px_from_psor(feats.get("Person[psor]"), feats.get("Number[psor]"))
|
| 619 |
+
if px != "None":
|
| 620 |
+
cell["Px"] = px
|
| 621 |
+
return cell
|
| 622 |
+
|
| 623 |
+
def _enclitic_host_cell(self, stem: str) -> dict | None:
|
| 624 |
+
"""Semantic gate for inflect-by-example enclitic peeling: return the stem's cell IFF the stem is
|
| 625 |
+
a GENUINE inflected nominal host, else None. The serving analyzer absorbs an enclitic into the
|
| 626 |
+
lemma (it emits no Clitic feature), so to copy '-ko' off a model word (makaronilaatikkoonko) we
|
| 627 |
+
peel it heuristically (features.split_enclitic) and verify the remainder here. Two checks reject
|
| 628 |
+
a lemma that merely ends in clitic letters: (1) a bare nominative singular with no possessive is
|
| 629 |
+
a lemma, not an inflected host (so 'kissa' from a spurious 'kissa|kin' split is refused — at the
|
| 630 |
+
cost of not copying a clitic off a Nom-Sing model, a documented limitation); (2) a round-trip —
|
| 631 |
+
the stem must regenerate from its own analysis (rejects hallucinated parses of fragments like
|
| 632 |
+
'aurin' from 'aurinko'). The round-trip realize would dirty the OOV trace, so snapshot/restore."""
|
| 633 |
+
host = self._nominal_head(stem)
|
| 634 |
+
if host is None:
|
| 635 |
+
return None
|
| 636 |
+
cell = self._cell_from_head(host)
|
| 637 |
+
if (cell.get("Case", "Nom") == "Nom" and cell.get("Number", "Sing") == "Sing"
|
| 638 |
+
and "Px" not in cell):
|
| 639 |
+
return None
|
| 640 |
+
upos = host["upos"] if host["upos"] in ("NOUN", "PROPN") else "NOUN"
|
| 641 |
+
# the round-trip realize can append to all three OOV trace lists; snapshot/restore every one so
|
| 642 |
+
# the gate is trace-neutral (the phrase-target path never reaches inflect()'s own clear at L528).
|
| 643 |
+
saved = (list(algebra._NEURAL_OOV), list(algebra._OOV_UNVERIFIED), list(algebra._OOV_UNREALIZABLE))
|
| 644 |
+
try:
|
| 645 |
+
regen = algebra.realize_np_form(host["lemma"], upos, cell.get("Case", "Nom"),
|
| 646 |
+
cell.get("Number", "Sing"), cell.get("Px", "None"),
|
| 647 |
+
None, host["lemma"])
|
| 648 |
+
finally:
|
| 649 |
+
(algebra._NEURAL_OOV[:], algebra._OOV_UNVERIFIED[:],
|
| 650 |
+
algebra._OOV_UNREALIZABLE[:]) = saved
|
| 651 |
+
if regen is None or regen.lower() != stem.lower():
|
| 652 |
+
return None
|
| 653 |
+
return cell
|
| 654 |
+
|
| 655 |
+
def _analyze_nominal_cell(self, model: str) -> dict | None:
|
| 656 |
+
"""Analyze a model word and return its nominal {Case[,Number][,Degree][,Px][,Clitic]} cell (for
|
| 657 |
+
inflect-by-example), or None if it has no nominal head to copy from. A trailing focus/
|
| 658 |
+
interrogative enclitic (-kO/-kin/-kAAn/-hAn/-pA) is peeled and copied as Clitic when the
|
| 659 |
+
remainder passes the `_enclitic_host_cell` gate (so `like=makaronilaatikkoonko` -> +ko)."""
|
| 660 |
+
stem, particle = features.split_enclitic(model)
|
| 661 |
+
if particle is not None and stem != model:
|
| 662 |
+
host_cell = self._enclitic_host_cell(stem)
|
| 663 |
+
if host_cell is not None:
|
| 664 |
+
return {**host_cell, "Clitic": particle}
|
| 665 |
+
head = self._nominal_head(model)
|
| 666 |
+
if head is None:
|
| 667 |
+
return None
|
| 668 |
+
return self._cell_from_head(head) or None
|
| 669 |
+
|
| 670 |
+
# --------------------------- analysis convenience -----------------------------------------
|
| 671 |
+
def analyze(self, text: str) -> list[list["TokenAnalysis"]]:
|
| 672 |
+
"""Analyze raw text into sentences of `TokenAnalysis` (lemma, UPOS, head, deprel, UD feats).
|
| 673 |
+
The generation side's inverse — what taivutin cannot do. Reuses the serving parser."""
|
| 674 |
+
return [[TokenAnalysis(tk["id"], tk["form"], tk["lemma"], tk["upos"], tk["head"],
|
| 675 |
+
tk["deprel"], dict(tk.get("feats", {}))) for tk in sent]
|
| 676 |
+
for sent in self.parser.parse(text)]
|
| 677 |
+
|
| 678 |
+
def lemmatize(self, text: str) -> list[str]:
|
| 679 |
+
"""Return the lemma of every token in `text`, flat across sentences."""
|
| 680 |
+
return [tk["lemma"] for sent in self.parser.parse(text) for tk in sent]
|
| 681 |
+
|
| 682 |
+
|
| 683 |
+
if __name__ == "__main__":
|
| 684 |
+
import sys
|
| 685 |
+
t = Transformer()
|
| 686 |
+
if len(sys.argv) > 2:
|
| 687 |
+
op, text = sys.argv[1], sys.argv[2]
|
| 688 |
+
lemma = sys.argv[3] if len(sys.argv) > 3 else None
|
| 689 |
+
r = t.transform(op, text, lemma=lemma)
|
| 690 |
+
print(r)
|
| 691 |
+
else:
|
| 692 |
+
# self-test: raw text in, grammatical transform out, incl. OOV loans + abstention
|
| 693 |
+
cases = [
|
| 694 |
+
("passive", "Poika tubettaa videon.", None),
|
| 695 |
+
("past", "Käyttäjä googlaa vastauksen.", None),
|
| 696 |
+
("neg", "Nuori striimaa peliä.", None), # parser lemma may be bad
|
| 697 |
+
("neg", "Nuori striimaa peliä.", "striimata"), # ...fixed by override
|
| 698 |
+
("passive", "Kissa nukkui, koska se oli väsynyt.", None), # multi-clause -> abstain
|
| 699 |
+
]
|
| 700 |
+
for op, text, lem in cases:
|
| 701 |
+
r = t.transform(op, text, lemma=lem)
|
| 702 |
+
tag = f"[{op}{'/lemma='+lem if lem else ''}]"
|
| 703 |
+
print(f"{tag:24s} {text}")
|
| 704 |
+
print(f" -> {r.text!r} ({r.reason})"
|
| 705 |
+
+ (f" warn={r.warnings}" if r.warnings else ""))
|
juurio/realizer.py
ADDED
|
@@ -0,0 +1,196 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""OOV noun/adjective realizer (inference).
|
| 2 |
+
|
| 3 |
+
A small PER-TOKEN realizer the rule engine calls when an open-class loan/OOV word has no lexicon
|
| 4 |
+
entry. Given `lemma + POS + Case + Number + Degree (+ Px)`, it emits the target surface form, so it
|
| 5 |
+
generalizes to unseen stems (e.g. `striimaaja -> striimaajat`) instead of memorizing lemmas.
|
| 6 |
+
|
| 7 |
+
Separate model + checkpoint from the sentence-transform flag model. Header is CATEGORICAL
|
| 8 |
+
(POS/Case/Number/Degree/Px), not binary flags. Wiring: `pipeline`/callers set
|
| 9 |
+
`algebra._NOUN_REALIZER = make_noun_realizer(...)`; `_realize_form` tries the lexicon first, this
|
| 10 |
+
second, else abstains.
|
| 11 |
+
"""
|
| 12 |
+
from __future__ import annotations
|
| 13 |
+
import os
|
| 14 |
+
|
| 15 |
+
import torch
|
| 16 |
+
import torch.nn as nn
|
| 17 |
+
|
| 18 |
+
from .device import DEV
|
| 19 |
+
from . import bootstrap
|
| 20 |
+
|
| 21 |
+
CKPT = os.path.join(os.path.dirname(__file__), "..", "checkpoints", "realizer.pt")
|
| 22 |
+
PAD, BOS, EOS, UNK = 0, 1, 2, 3
|
| 23 |
+
SPECIAL = ["<pad>", "<bos>", "<eos>", "<unk>"]
|
| 24 |
+
|
| 25 |
+
# Fixed comprehensive cased char set: every char that can appear in a Finnish surface/lemma — incl.
|
| 26 |
+
# foreign loan letters (w/x/z/q/...), CAPITALS (PROPN), digits, hyphen/apostrophe.
|
| 27 |
+
# A pointer-copy head can only reproduce a char that has its own vocab id, so the vocab is fixed.
|
| 28 |
+
_LOWER = "abcdefghijklmnopqrstuvwxyzåäöšž"
|
| 29 |
+
_UPPER = "ABCDEFGHIJKLMNOPQRSTUVWXYZÅÄÖŠŽ"
|
| 30 |
+
_DIGITS = "0123456789"
|
| 31 |
+
_PUNCT = "-' "
|
| 32 |
+
FIXED_CHARS = _LOWER + _UPPER + _DIGITS + _PUNCT
|
| 33 |
+
|
| 34 |
+
# categorical feature space (the header). Cases = the UD case set.
|
| 35 |
+
POS_VALS = ["NOUN", "ADJ", "PROPN"]
|
| 36 |
+
CASE_VALS = ["Nom", "Gen", "Par", "Ess", "Tra", "Ine", "Ela", "Ill", "Ade", "Abl", "All",
|
| 37 |
+
"Abe", "Com", "Ins"]
|
| 38 |
+
NUM_VALS = ["Sing", "Plur"]
|
| 39 |
+
DEG_VALS = ["Pos", "Cmp", "Sup"]
|
| 40 |
+
# possessive suffix (omistusliite). 3rd person merges Sg/Pl (-nsa/-nsä/-Vn), so one "3" value.
|
| 41 |
+
# Px belongs IN the realizer header — it attaches after case/number and interacts with stem
|
| 42 |
+
# allomorphy (taloni vs talossani), so it is part of the inflected form, not a clitic.
|
| 43 |
+
PX_VALS = ["None", "1Sg", "2Sg", "3", "1Pl", "2Pl"]
|
| 44 |
+
FEAT_GROUPS = [("POS", POS_VALS), ("Case", CASE_VALS), ("Number", NUM_VALS),
|
| 45 |
+
("Degree", DEG_VALS), ("Px", PX_VALS)]
|
| 46 |
+
FEAT_STOI = {f"{g}={v}": i for i, (g, v) in
|
| 47 |
+
enumerate((g, v) for g, vals in FEAT_GROUPS for v in vals)} # flat contiguous index
|
| 48 |
+
NFEAT_TOK = len(FEAT_STOI) # distinct categorical tokens
|
| 49 |
+
HLEN = len(FEAT_GROUPS) # header length (5 slots)
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
def case_lemma(pos, s):
|
| 53 |
+
"""PROPN lemmas/forms are capitalized (Wagner, Åbo) so the copy head copies the capital position;
|
| 54 |
+
NOUN/ADJ/verbs stay as given (lowercase). Never lowercases internal capitals (McDonald)."""
|
| 55 |
+
return (s[:1].upper() + s[1:]) if pos == "PROPN" else s
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
def header_ids(pos, case, num, deg, px="None"):
|
| 59 |
+
"""The categorical token-ids for a target slot (unknown values clamp to the default)."""
|
| 60 |
+
pos = pos if pos in POS_VALS else "NOUN"
|
| 61 |
+
case = case if case in CASE_VALS else "Nom"
|
| 62 |
+
num = num if num in NUM_VALS else "Sing"
|
| 63 |
+
deg = deg if deg in DEG_VALS else "Pos"
|
| 64 |
+
px = px if px in PX_VALS else "None"
|
| 65 |
+
return [FEAT_STOI[f"POS={pos}"], FEAT_STOI[f"Case={case}"], FEAT_STOI[f"Number={num}"],
|
| 66 |
+
FEAT_STOI[f"Degree={deg}"], FEAT_STOI[f"Px={px}"]]
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
def encode_chars(s, stoi):
|
| 70 |
+
"""Encode a string to ids. New cased vocabs carry <unk>: map OOV→<unk> (never silently drop, the
|
| 71 |
+
bug behind `warettaa`→`arettaa`). Legacy 30-char checkpoints have no <unk>: preserve their original
|
| 72 |
+
drop-OOV behaviour so they load + decode unchanged."""
|
| 73 |
+
unk = stoi.get("<unk>")
|
| 74 |
+
if unk is None:
|
| 75 |
+
return [stoi[c] for c in s if c in stoi]
|
| 76 |
+
return [stoi.get(c, unk) for c in s]
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
# ----------------------------- model -----------------------------
|
| 80 |
+
class Realizer(nn.Module):
|
| 81 |
+
def __init__(self, vocab, d_model=128, nhead=4, layers=2, ff=512, maxlen=64):
|
| 82 |
+
super().__init__()
|
| 83 |
+
self.maxlen = maxlen
|
| 84 |
+
self.emb = nn.Embedding(vocab, d_model, padding_idx=PAD)
|
| 85 |
+
self.featemb = nn.Embedding(NFEAT_TOK, d_model)
|
| 86 |
+
self.pos = nn.Embedding(maxlen, d_model)
|
| 87 |
+
self.enc = nn.TransformerEncoder(
|
| 88 |
+
nn.TransformerEncoderLayer(d_model, nhead, ff, 0.1, batch_first=True), layers)
|
| 89 |
+
self.dec = nn.TransformerDecoder(
|
| 90 |
+
nn.TransformerDecoderLayer(d_model, nhead, ff, 0.1, batch_first=True), layers)
|
| 91 |
+
self.proj = nn.Linear(d_model, vocab)
|
| 92 |
+
self.proj.weight = self.emb.weight
|
| 93 |
+
|
| 94 |
+
def encode(self, src, header):
|
| 95 |
+
B, S = src.shape
|
| 96 |
+
H = header.size(1)
|
| 97 |
+
seq = torch.cat([self.featemb(header), self.emb(src)], 1)
|
| 98 |
+
seq = seq + self.pos(torch.arange(H + S, device=src.device))
|
| 99 |
+
mask = torch.cat([torch.zeros(B, H, dtype=torch.bool, device=src.device), src == PAD], 1)
|
| 100 |
+
return self.enc(seq, src_key_padding_mask=mask), mask
|
| 101 |
+
|
| 102 |
+
def _tgt_embed(self, ids):
|
| 103 |
+
T = ids.size(1)
|
| 104 |
+
return self.emb(ids) + self.pos(torch.arange(T, device=ids.device))
|
| 105 |
+
|
| 106 |
+
def forward(self, src, header, tgt_in):
|
| 107 |
+
mem, mask = self.encode(src, header)
|
| 108 |
+
T = tgt_in.size(1)
|
| 109 |
+
cmask = torch.triu(torch.ones(T, T, device=src.device, dtype=torch.bool), 1)
|
| 110 |
+
out = self.dec(self._tgt_embed(tgt_in), mem, tgt_mask=cmask,
|
| 111 |
+
tgt_key_padding_mask=(tgt_in == PAD), memory_key_padding_mask=mask)
|
| 112 |
+
return self.proj(out)
|
| 113 |
+
|
| 114 |
+
|
| 115 |
+
def _pad(seqs, n):
|
| 116 |
+
return torch.tensor([s + [PAD] * (n - len(s)) for s in seqs], dtype=torch.long)
|
| 117 |
+
|
| 118 |
+
|
| 119 |
+
@torch.no_grad()
|
| 120 |
+
def _decode(m, header, lemma, stoi, itos, maxlen=48, score=False):
|
| 121 |
+
"""Greedy decode. If `score`, also return a confidence = min over steps of the top-1 softmax
|
| 122 |
+
margin (top1 - top2 prob) — a per-cell uncertainty signal used to suppress shaky OOV guesses."""
|
| 123 |
+
m.eval()
|
| 124 |
+
src = torch.tensor([encode_chars(lemma, stoi)], dtype=torch.long, device=DEV)
|
| 125 |
+
hdr = torch.tensor([header], dtype=torch.long, device=DEV)
|
| 126 |
+
mem, mask = m.encode(src, hdr)
|
| 127 |
+
ids = [BOS]
|
| 128 |
+
margin = 1.0
|
| 129 |
+
for _ in range(maxlen):
|
| 130 |
+
ti = torch.tensor([ids], dtype=torch.long, device=DEV)
|
| 131 |
+
T = ti.size(1)
|
| 132 |
+
cmask = torch.triu(torch.ones(T, T, device=DEV, dtype=torch.bool), 1)
|
| 133 |
+
out = m.dec(m._tgt_embed(ti), mem, tgt_mask=cmask, memory_key_padding_mask=mask)
|
| 134 |
+
logits = m.proj(out)[0, -1]
|
| 135 |
+
nxt = int(logits.argmax())
|
| 136 |
+
if score:
|
| 137 |
+
p = torch.softmax(logits.float(), -1)
|
| 138 |
+
top2 = torch.topk(p, 2).values
|
| 139 |
+
margin = min(margin, float(top2[0] - top2[1]))
|
| 140 |
+
if nxt == EOS:
|
| 141 |
+
break
|
| 142 |
+
ids.append(nxt)
|
| 143 |
+
form = "".join(itos[i] for i in ids[1:] if i in itos)
|
| 144 |
+
return (form, margin) if score else form
|
| 145 |
+
|
| 146 |
+
|
| 147 |
+
# ----------------------------- inference hook -----------------------------
|
| 148 |
+
def load(ckpt=None):
|
| 149 |
+
"""Load realizer B. A copy-head checkpoint (pointer-generator — carries copy_q/copy_k/gate) is
|
| 150 |
+
reconstructed as a CopyRealizer so NOUN/ADJ/PROPN are served by the mixed gen+copy distribution; a
|
| 151 |
+
plain checkpoint stays a generator-only Realizer. Detected by weights, so old and new checkpoints
|
| 152 |
+
both load. `ckpt=None` resolves the shipped artifact via bootstrap."""
|
| 153 |
+
if ckpt is None:
|
| 154 |
+
ckpt = bootstrap.artifact_path("realizer")
|
| 155 |
+
ck = torch.load(ckpt, map_location=DEV, weights_only=False)
|
| 156 |
+
if "copy_q.weight" in ck["model"]:
|
| 157 |
+
from .copyrealizer import load as _copy_load
|
| 158 |
+
return _copy_load(ckpt)
|
| 159 |
+
m = Realizer(len(ck["stoi"]), ck["cfg"]["d_model"], 4, ck["cfg"]["layers"]).to(DEV)
|
| 160 |
+
m.load_state_dict(ck["model"]); m.eval()
|
| 161 |
+
return m, ck["stoi"], ck["itos"]
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
def make_noun_realizer(bundle=None, ckpt=None):
|
| 165 |
+
"""Return fn(lemma, upos, feats, surface)->form for `algebra._NOUN_REALIZER` (None on failure).
|
| 166 |
+
Preserves the source word's capitalisation (proper names / sentence-initial). Decodes over the
|
| 167 |
+
mixed gen+copy distribution for a copy checkpoint, the generator for a plain one."""
|
| 168 |
+
m, stoi, itos = bundle or load(ckpt)
|
| 169 |
+
from .copyrealizer import CopyRealizer, decode_one as _copy_decode
|
| 170 |
+
is_copy = isinstance(m, CopyRealizer)
|
| 171 |
+
|
| 172 |
+
def realize(lemma, upos, feats, surface=None):
|
| 173 |
+
"""Return (form, confidence). confidence in [0,1] is the min top1-top2 margin over decode steps
|
| 174 |
+
(on the mixed distribution for a copy model); the caller (`_realize_form`) abstains below a
|
| 175 |
+
threshold for true OOVs."""
|
| 176 |
+
h = header_ids(upos, feats.get("Case", "Nom"), feats.get("Number", "Sing"),
|
| 177 |
+
feats.get("Degree", "Pos"), feats.get("Px", "None")) # clamps unknowns
|
| 178 |
+
decode = _copy_decode if is_copy else _decode
|
| 179 |
+
# case the INPUT by POS (PROPN capitalized -> copy head copies the capital; NOUN/ADJ lowercase).
|
| 180 |
+
# Trailing surface-based re-capitalization still covers sentence-initial nouns.
|
| 181 |
+
out, conf = decode(m, h, case_lemma(upos, lemma.lower()), stoi, itos, score=True)
|
| 182 |
+
if not out:
|
| 183 |
+
return None, 0.0
|
| 184 |
+
if surface and surface[:1].isupper():
|
| 185 |
+
out = out[:1].upper() + out[1:]
|
| 186 |
+
return out, conf
|
| 187 |
+
|
| 188 |
+
return realize
|
| 189 |
+
|
| 190 |
+
|
| 191 |
+
_FEAT_ITOS = {i: k for k, i in FEAT_STOI.items()}
|
| 192 |
+
|
| 193 |
+
|
| 194 |
+
def _hdr_labels(h):
|
| 195 |
+
"""Invert a header_ids list back to (POS, Case, Number, Degree) label strings."""
|
| 196 |
+
return tuple(_FEAT_ITOS[i].split("=", 1)[1] for i in h)
|
juurio/verbforms.py
ADDED
|
@@ -0,0 +1,136 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""OOV non-finite verb-form realizer (inference).
|
| 2 |
+
|
| 3 |
+
Extends the open-vocabulary morphology story from nouns/adjectives (`realizer.py`) to VERB participles
|
| 4 |
+
and infinitives: a categorical header over (non-finite FORM x Case x Number). Given a verb lemma + that
|
| 5 |
+
header, emit the participle/infinitive surface, generalizing to loan verbs the lexicon cannot inflect
|
| 6 |
+
(`striimata` -> `striimaava` / `striimaamassa`).
|
| 7 |
+
|
| 8 |
+
The lexicon handles non-finite forms of KNOWN verbs; this model is the fallback for OOV/loan verbs,
|
| 9 |
+
exactly as the noun realizer is for OOV nouns/adjectives.
|
| 10 |
+
|
| 11 |
+
Wiring: callers set `algebra._VERBFORM_REALIZER = make_verbform_realizer()`; `pipeline.realize_verbform`
|
| 12 |
+
tries the lexicon first, this second, else abstains. Separate model + checkpoint.
|
| 13 |
+
"""
|
| 14 |
+
from __future__ import annotations
|
| 15 |
+
import os
|
| 16 |
+
|
| 17 |
+
import torch
|
| 18 |
+
|
| 19 |
+
from .device import DEV
|
| 20 |
+
from . import bootstrap
|
| 21 |
+
from .realizer import Realizer, _pad, _decode, encode_chars # reuse the char enc-dec + decode
|
| 22 |
+
from .copyrealizer import CopyRealizer # pointer-generator copy head
|
| 23 |
+
from .algebra import VERBFORMS
|
| 24 |
+
|
| 25 |
+
CKPT = os.path.join(os.path.dirname(__file__), "..", "checkpoints", "verbform.pt")
|
| 26 |
+
PAD, BOS, EOS = 0, 1, 2
|
| 27 |
+
|
| 28 |
+
# categorical header: the non-finite FORM family x Case x Number. Cases are the full UD set
|
| 29 |
+
# (participles inflect like adjectives across all of them; infinitive forms only exist for a subset).
|
| 30 |
+
FORM_VALS = list(VERBFORMS) # va nut tava tu inf2 inf3
|
| 31 |
+
CASE_VALS = ["Nom", "Gen", "Par", "Ess", "Tra", "Ine", "Ela", "Ill", "Ade", "Abl", "All",
|
| 32 |
+
"Abe", "Com", "Ins"]
|
| 33 |
+
NUM_VALS = ["Sing", "Plur"]
|
| 34 |
+
FEAT_GROUPS = [("Form", FORM_VALS), ("Case", CASE_VALS), ("Number", NUM_VALS)]
|
| 35 |
+
FEAT_STOI = {f"{g}={v}": i for i, (g, v) in
|
| 36 |
+
enumerate((g, v) for g, vals in FEAT_GROUPS for v in vals)}
|
| 37 |
+
NFEAT_TOK = len(FEAT_STOI) # fits within realizer.NFEAT_TOK, so Realizer's featemb is reused
|
| 38 |
+
HLEN = len(FEAT_GROUPS) # 3 header slots
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
def header_ids(form, case, num):
|
| 42 |
+
form = form if form in FORM_VALS else "va"
|
| 43 |
+
case = case if case in CASE_VALS else "Nom"
|
| 44 |
+
num = num if num in NUM_VALS else "Sing"
|
| 45 |
+
if form == "inf2_pass": # passive 2nd inf is inessive-only; pin the header to its one cell
|
| 46 |
+
case, num = "Ine", "Sing"
|
| 47 |
+
return [FEAT_STOI[f"Form={form}"], FEAT_STOI[f"Case={case}"], FEAT_STOI[f"Number={num}"]]
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
def load(ckpt=None):
|
| 51 |
+
"""Load the non-finite net. A copy-head checkpoint (pointer-generator — carries copy_q/copy_k/gate)
|
| 52 |
+
is reconstructed as a CopyRealizer so the served distribution is the mixed gen+copy one; a plain
|
| 53 |
+
checkpoint stays a generator-only Realizer. Detection is by weights, so old and new checkpoints
|
| 54 |
+
both load. `ckpt=None` resolves the shipped artifact via bootstrap."""
|
| 55 |
+
if ckpt is None:
|
| 56 |
+
ckpt = bootstrap.artifact_path("verbform")
|
| 57 |
+
ck = torch.load(ckpt, map_location=DEV, weights_only=False)
|
| 58 |
+
if "copy_q.weight" in ck["model"]:
|
| 59 |
+
from .copyrealizer import load as _copy_load
|
| 60 |
+
return _copy_load(ckpt)
|
| 61 |
+
m = Realizer(len(ck["stoi"]), ck["cfg"]["d_model"], 4, ck["cfg"]["layers"]).to(DEV)
|
| 62 |
+
m.load_state_dict(ck["model"]); m.eval()
|
| 63 |
+
return m, ck["stoi"], ck["itos"]
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
def decode_forms(items, bundle, batch_size: int = 256, maxlen: int = 48):
|
| 67 |
+
"""Greedy-decode a non-finite form for each (lemma, (form, case, num)). Batched; bit-identical to
|
| 68 |
+
the single-item _decode (greedy argmax + per-seq attention). items: iterable of
|
| 69 |
+
(lemma, (form, case, num)). Mirrors finiteverb.decode_forms — same Realizer enc-dec, just this
|
| 70 |
+
module's 3-slot header."""
|
| 71 |
+
items = list(items)
|
| 72 |
+
if not items:
|
| 73 |
+
return []
|
| 74 |
+
m, stoi, itos = bundle
|
| 75 |
+
if isinstance(m, CopyRealizer):
|
| 76 |
+
# copy model: decode over the mixed gen+copy distribution (CopyRealizer.decode_forms takes
|
| 77 |
+
# (lemma, header_ids)); the generator-only proj path below would measure the wrong model.
|
| 78 |
+
from . import copyrealizer as CR
|
| 79 |
+
return CR.decode_forms([(lem.lower(), header_ids(f, c, n)) for lem, (f, c, n) in items],
|
| 80 |
+
bundle, batch_size=batch_size, maxlen=maxlen)
|
| 81 |
+
m.eval()
|
| 82 |
+
out = []
|
| 83 |
+
for start in range(0, len(items), batch_size):
|
| 84 |
+
chunk = items[start:start + batch_size]
|
| 85 |
+
B = len(chunk)
|
| 86 |
+
src_lists = [encode_chars(lem.lower(), stoi) for lem, _ in chunk]
|
| 87 |
+
S = max((len(s) for s in src_lists), default=1) or 1
|
| 88 |
+
src = _pad(src_lists, S).to(DEV)
|
| 89 |
+
hdr = torch.tensor([header_ids(f, c, n) for _, (f, c, n) in chunk],
|
| 90 |
+
dtype=torch.long, device=DEV)
|
| 91 |
+
with torch.no_grad():
|
| 92 |
+
mem, mask = m.encode(src, hdr)
|
| 93 |
+
ids = torch.full((B, 1), BOS, dtype=torch.long, device=DEV)
|
| 94 |
+
finished = torch.zeros(B, dtype=torch.bool, device=DEV)
|
| 95 |
+
collected = [[] for _ in range(B)]
|
| 96 |
+
for _ in range(maxlen):
|
| 97 |
+
T = ids.size(1)
|
| 98 |
+
cmask = torch.triu(torch.ones(T, T, device=DEV, dtype=torch.bool), 1)
|
| 99 |
+
dout = m.dec(m._tgt_embed(ids), mem, tgt_mask=cmask, memory_key_padding_mask=mask)
|
| 100 |
+
nxt = m.proj(dout)[:, -1].argmax(-1)
|
| 101 |
+
nxt_cpu = nxt.tolist()
|
| 102 |
+
for i in range(B):
|
| 103 |
+
if finished[i]:
|
| 104 |
+
continue
|
| 105 |
+
if nxt_cpu[i] == EOS:
|
| 106 |
+
finished[i] = True
|
| 107 |
+
else:
|
| 108 |
+
collected[i].append(nxt_cpu[i])
|
| 109 |
+
app = nxt.clone(); app[finished] = PAD
|
| 110 |
+
ids = torch.cat([ids, app.unsqueeze(1)], 1)
|
| 111 |
+
if bool(finished.all()):
|
| 112 |
+
break
|
| 113 |
+
out.extend("".join(itos[t] for t in row if t in itos) for row in collected)
|
| 114 |
+
return out
|
| 115 |
+
|
| 116 |
+
|
| 117 |
+
def make_verbform_realizer(bundle=None, ckpt=None):
|
| 118 |
+
"""Return fn(lemma, form, case, number) -> (form, confidence) for `algebra._VERBFORM_REALIZER`
|
| 119 |
+
(None on failure). confidence is the min top1-top2 margin over decode steps — measured on the
|
| 120 |
+
mixed gen+copy distribution for a copy checkpoint, on the generator for a plain one."""
|
| 121 |
+
m, stoi, itos = bundle or load(ckpt)
|
| 122 |
+
if isinstance(m, CopyRealizer):
|
| 123 |
+
from .copyrealizer import decode_one as _copy_decode
|
| 124 |
+
|
| 125 |
+
def realize(lemma, form, case="Nom", number="Sing"):
|
| 126 |
+
out, conf = _copy_decode(m, header_ids(form, case, number), lemma.lower(), stoi, itos,
|
| 127 |
+
score=True)
|
| 128 |
+
return (out, conf) if out else (None, 0.0)
|
| 129 |
+
|
| 130 |
+
return realize
|
| 131 |
+
|
| 132 |
+
def realize(lemma, form, case="Nom", number="Sing"):
|
| 133 |
+
out, conf = _decode(m, header_ids(form, case, number), lemma.lower(), stoi, itos, score=True)
|
| 134 |
+
return (out, conf) if out else (None, 0.0)
|
| 135 |
+
|
| 136 |
+
return realize
|
pyproject.toml
ADDED
|
@@ -0,0 +1,52 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[build-system]
|
| 2 |
+
requires = ["hatchling"]
|
| 3 |
+
build-backend = "hatchling.build"
|
| 4 |
+
|
| 5 |
+
[project]
|
| 6 |
+
name = "juurio"
|
| 7 |
+
version = "0.1.0"
|
| 8 |
+
description = "Open-vocabulary Finnish morphology: a neural parser, an exception lexicon and tiny neural realizers for OOV/loan word forms"
|
| 9 |
+
readme = "README.md"
|
| 10 |
+
requires-python = ">=3.12"
|
| 11 |
+
license = { text = "MIT" }
|
| 12 |
+
license-files = ["LICENSE"]
|
| 13 |
+
authors = [{ name = "samuli" }]
|
| 14 |
+
keywords = ["finnish", "morphology", "nlp", "inflection", "dependency-parsing", "fst", "uralic"]
|
| 15 |
+
classifiers = [
|
| 16 |
+
"Development Status :: 4 - Beta",
|
| 17 |
+
"Intended Audience :: Developers",
|
| 18 |
+
"License :: OSI Approved :: MIT License",
|
| 19 |
+
"Natural Language :: Finnish",
|
| 20 |
+
"Programming Language :: Python :: 3",
|
| 21 |
+
"Programming Language :: Python :: 3.12",
|
| 22 |
+
"Topic :: Scientific/Engineering :: Artificial Intelligence",
|
| 23 |
+
"Topic :: Text Processing :: Linguistic",
|
| 24 |
+
]
|
| 25 |
+
# Version floors track the tested versions and MUST NOT exceed what's installed/tested (guarded by
|
| 26 |
+
# tests/test_runtime_dep_floors.py) — a floor above the tested version makes a clean `pip install` pull
|
| 27 |
+
# an untested-newer dep. Inference needs only torch / numpy / transformers (the analysis path is the
|
| 28 |
+
# torch-only neural parser; generation uses the neural realizers + lexicon). huggingface-hub: it
|
| 29 |
+
# fetches the model/data artifacts from a HuggingFace repo via bootstrap.artifact_path.
|
| 30 |
+
dependencies = [
|
| 31 |
+
"transformers>=5.7",
|
| 32 |
+
"torch>=2.4",
|
| 33 |
+
"numpy>=1.26",
|
| 34 |
+
"huggingface-hub>=0.34",
|
| 35 |
+
]
|
| 36 |
+
|
| 37 |
+
[project.urls]
|
| 38 |
+
Homepage = "https://huggingface.co/samuli/juurio"
|
| 39 |
+
|
| 40 |
+
[project.optional-dependencies]
|
| 41 |
+
dev = ["pytest>=8", "packaging>=23"]
|
| 42 |
+
|
| 43 |
+
[project.scripts]
|
| 44 |
+
juurio = "juurio.cli:main"
|
| 45 |
+
|
| 46 |
+
# ─── Wheel ship-list ─────────────────────────────────────────────────────────────────────────────
|
| 47 |
+
# Ships the `juurio` package (top-level modules + the genlex override JSONs, juurio.analyze,
|
| 48 |
+
# and juurio.parser). The bulk serving artifacts (model checkpoints, wordlist, compressed tag table)
|
| 49 |
+
# are NOT bundled — they are fetched at runtime from the HuggingFace repo by `juurio setup`
|
| 50 |
+
# (bootstrap.py), sha256-verified. Guarded by tests/test_serving_wheel_contents.py.
|
| 51 |
+
[tool.hatch.build.targets.wheel]
|
| 52 |
+
only-include = ["juurio"]
|