Release frozen VimeML tiny-ja-v1 FP32 and Core ML INT8 (GPL-2.0)

LICENSE

@@ -0,0 +1,339 @@
+                    GNU GENERAL PUBLIC LICENSE
+                       Version 2, June 1991
+
+ Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
+ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+                            Preamble
+
+  The licenses for most software are designed to take away your
+freedom to share and change it.  By contrast, the GNU General Public
+License is intended to guarantee your freedom to share and change free
+software--to make sure the software is free for all its users.  This
+General Public License applies to most of the Free Software
+Foundation's software and to any other program whose authors commit to
+using it.  (Some other Free Software Foundation software is covered by
+the GNU Lesser General Public License instead.)  You can apply it to
+your programs, too.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+this service if you wish), that you receive source code or can get it
+if you want it, that you can change the software or use pieces of it
+in new free programs; and that you know you can do these things.
+
+  To protect your rights, we need to make restrictions that forbid
+anyone to deny you these rights or to ask you to surrender the rights.
+These restrictions translate to certain responsibilities for you if you
+distribute copies of the software, or if you modify it.
+
+  For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must give the recipients all the rights that
+you have.  You must make sure that they, too, receive or can get the
+source code.  And you must show them these terms so they know their
+rights.
+
+  We protect your rights with two steps: (1) copyright the software, and
+(2) offer you this license which gives you legal permission to copy,
+distribute and/or modify the software.
+
+  Also, for each author's protection and ours, we want to make certain
+that everyone understands that there is no warranty for this free
+software.  If the software is modified by someone else and passed on, we
+want its recipients to know that what they have is not the original, so
+that any problems introduced by others will not reflect on the original
+authors' reputations.
+
+  Finally, any free program is threatened constantly by software
+patents.  We wish to avoid the danger that redistributors of a free
+program will individually obtain patent licenses, in effect making the
+program proprietary.  To prevent this, we have made it clear that any
+patent must be licensed for everyone's free use or not licensed at all.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+                    GNU GENERAL PUBLIC LICENSE
+   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+
+  0. This License applies to any program or other work which contains
+a notice placed by the copyright holder saying it may be distributed
+under the terms of this General Public License.  The "Program", below,
+refers to any such program or work, and a "work based on the Program"
+means either the Program or any derivative work under copyright law:
+that is to say, a work containing the Program or a portion of it,
+either verbatim or with modifications and/or translated into another
+language.  (Hereinafter, translation is included without limitation in
+the term "modification".)  Each licensee is addressed as "you".
+
+Activities other than copying, distribution and modification are not
+covered by this License; they are outside its scope.  The act of
+running the Program is not restricted, and the output from the Program
+is covered only if its contents constitute a work based on the
+Program (independent of having been made by running the Program).
+Whether that is true depends on what the Program does.
+
+  1. You may copy and distribute verbatim copies of the Program's
+source code as you receive it, in any medium, provided that you
+conspicuously and appropriately publish on each copy an appropriate
+copyright notice and disclaimer of warranty; keep intact all the
+notices that refer to this License and to the absence of any warranty;
+and give any other recipients of the Program a copy of this License
+along with the Program.
+
+You may charge a fee for the physical act of transferring a copy, and
+you may at your option offer warranty protection in exchange for a fee.
+
+  2. You may modify your copy or copies of the Program or any portion
+of it, thus forming a work based on the Program, and copy and
+distribute such modifications or work under the terms of Section 1
+above, provided that you also meet all of these conditions:
+
+    a) You must cause the modified files to carry prominent notices
+    stating that you changed the files and the date of any change.
+
+    b) You must cause any work that you distribute or publish, that in
+    whole or in part contains or is derived from the Program or any
+    part thereof, to be licensed as a whole at no charge to all third
+    parties under the terms of this License.
+
+    c) If the modified program normally reads commands interactively
+    when run, you must cause it, when started running for such
+    interactive use in the most ordinary way, to print or display an
+    announcement including an appropriate copyright notice and a
+    notice that there is no warranty (or else, saying that you provide
+    a warranty) and that users may redistribute the program under
+    these conditions, and telling the user how to view a copy of this
+    License.  (Exception: if the Program itself is interactive but
+    does not normally print such an announcement, your work based on
+    the Program is not required to print an announcement.)
+
+These requirements apply to the modified work as a whole.  If
+identifiable sections of that work are not derived from the Program,
+and can be reasonably considered independent and separate works in
+themselves, then this License, and its terms, do not apply to those
+sections when you distribute them as separate works.  But when you
+distribute the same sections as part of a whole which is a work based
+on the Program, the distribution of the whole must be on the terms of
+this License, whose permissions for other licensees extend to the
+entire whole, and thus to each and every part regardless of who wrote it.
+
+Thus, it is not the intent of this section to claim rights or contest
+your rights to work written entirely by you; rather, the intent is to
+exercise the right to control the distribution of derivative or
+collective works based on the Program.
+
+In addition, mere aggregation of another work not based on the Program
+with the Program (or with a work based on the Program) on a volume of
+a storage or distribution medium does not bring the other work under
+the scope of this License.
+
+  3. You may copy and distribute the Program (or a work based on it,
+under Section 2) in object code or executable form under the terms of
+Sections 1 and 2 above provided that you also do one of the following:
+
+    a) Accompany it with the complete corresponding machine-readable
+    source code, which must be distributed under the terms of Sections
+    1 and 2 above on a medium customarily used for software interchange; or,
+
+    b) Accompany it with a written offer, valid for at least three
+    years, to give any third party, for a charge no more than your
+    cost of physically performing source distribution, a complete
+    machine-readable copy of the corresponding source code, to be
+    distributed under the terms of Sections 1 and 2 above on a medium
+    customarily used for software interchange; or,
+
+    c) Accompany it with the information you received as to the offer
+    to distribute corresponding source code.  (This alternative is
+    allowed only for noncommercial distribution and only if you
+    received the program in object code or executable form with such
+    an offer, in accord with Subsection b above.)
+
+The source code for a work means the preferred form of the work for
+making modifications to it.  For an executable work, complete source
+code means all the source code for all modules it contains, plus any
+associated interface definition files, plus the scripts used to
+control compilation and installation of the executable.  However, as a
+special exception, the source code distributed need not include
+anything that is normally distributed (in either source or binary
+form) with the major components (compiler, kernel, and so on) of the
+operating system on which the executable runs, unless that component
+itself accompanies the executable.
+
+If distribution of executable or object code is made by offering
+access to copy from a designated place, then offering equivalent
+access to copy the source code from the same place counts as
+distribution of the source code, even though third parties are not
+compelled to copy the source along with the object code.
+
+  4. You may not copy, modify, sublicense, or distribute the Program
+except as expressly provided under this License.  Any attempt
+otherwise to copy, modify, sublicense or distribute the Program is
+void, and will automatically terminate your rights under this License.
+However, parties who have received copies, or rights, from you under
+this License will not have their licenses terminated so long as such
+parties remain in full compliance.
+
+  5. You are not required to accept this License, since you have not
+signed it.  However, nothing else grants you permission to modify or
+distribute the Program or its derivative works.  These actions are
+prohibited by law if you do not accept this License.  Therefore, by
+modifying or distributing the Program (or any work based on the
+Program), you indicate your acceptance of this License to do so, and
+all its terms and conditions for copying, distributing or modifying
+the Program or works based on it.
+
+  6. Each time you redistribute the Program (or any work based on the
+Program), the recipient automatically receives a license from the
+original licensor to copy, distribute or modify the Program subject to
+these terms and conditions.  You may not impose any further
+restrictions on the recipients' exercise of the rights granted herein.
+You are not responsible for enforcing compliance by third parties to
+this License.
+
+  7. If, as a consequence of a court judgment or allegation of patent
+infringement or for any other reason (not limited to patent issues),
+conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot
+distribute so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you
+may not distribute the Program at all.  For example, if a patent
+license would not permit royalty-free redistribution of the Program by
+all those who receive copies directly or indirectly through you, then
+the only way you could satisfy both it and this License would be to
+refrain entirely from distribution of the Program.
+
+If any portion of this section is held invalid or unenforceable under
+any particular circumstance, the balance of the section is intended to
+apply and the section as a whole is intended to apply in other
+circumstances.
+
+It is not the purpose of this section to induce you to infringe any
+patents or other property right claims or to contest validity of any
+such claims; this section has the sole purpose of protecting the
+integrity of the free software distribution system, which is
+implemented by public license practices.  Many people have made
+generous contributions to the wide range of software distributed
+through that system in reliance on consistent application of that
+system; it is up to the author/donor to decide if he or she is willing
+to distribute software through any other system and a licensee cannot
+impose that choice.
+
+This section is intended to make thoroughly clear what is believed to
+be a consequence of the rest of this License.
+
+  8. If the distribution and/or use of the Program is restricted in
+certain countries either by patents or by copyrighted interfaces, the
+original copyright holder who places the Program under this License
+may add an explicit geographical distribution limitation excluding
+those countries, so that distribution is permitted only in or among
+countries not thus excluded.  In such case, this License incorporates
+the limitation as if written in the body of this License.
+
+  9. The Free Software Foundation may publish revised and/or new versions
+of the General Public License from time to time.  Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+Each version is given a distinguishing version number.  If the Program
+specifies a version number of this License which applies to it and "any
+later version", you have the option of following the terms and conditions
+either of that version or of any later version published by the Free
+Software Foundation.  If the Program does not specify a version number of
+this License, you may choose any version ever published by the Free Software
+Foundation.
+
+  10. If you wish to incorporate parts of the Program into other free
+programs whose distribution conditions are different, write to the author
+to ask for permission.  For software which is copyrighted by the Free
+Software Foundation, write to the Free Software Foundation; we sometimes
+make exceptions for this.  Our decision will be guided by the two goals
+of preserving the free status of all derivatives of our free software and
+of promoting the sharing and reuse of software generally.
+
+                            NO WARRANTY
+
+  11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
+FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN
+OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
+PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
+OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS
+TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD THE
+PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
+REPAIR OR CORRECTION.
+
+  12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
+REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
+INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
+OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
+TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
+YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
+PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGES.
+
+                     END OF TERMS AND CONDITIONS
+
+            How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+convey the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+Also add information on how to contact you by electronic and paper mail.
+
+If the program is interactive, make it output a short notice like this
+when it starts in an interactive mode:
+
+    Gnomovision version 69, Copyright (C) year name of author
+    Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+    This is free software, and you are welcome to redistribute it
+    under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License.  Of course, the commands you use may
+be called something other than `show w' and `show c'; they could even be
+mouse-clicks or menu items--whatever suits your program.
+
+You should also get your employer (if you work as a programmer) or your
+school, if any, to sign a "copyright disclaimer" for the program, if
+necessary.  Here is a sample; alter the names:
+
+  Yoyodyne, Inc., hereby disclaims all copyright interest in the program
+  `Gnomovision' (which makes passes at compilers) written by James Hacker.
+
+  <signature of Ty Coon>, 1 April 1989
+  Ty Coon, President of Vice
+
+This General Public License does not permit incorporating your program into
+proprietary programs.  If your program is a subroutine library, you may
+consider it more useful to permit linking proprietary applications with the
+library.  If this is what you want to do, use the GNU Lesser General
+Public License instead of this License.

README.md

@@ -1,3 +1,114 @@
 ---
-license: apache-2.0
+language:
+- ja
+license: gpl-2.0
+pipeline_tag: text-generation
+tags:
+- pytorch
+- coreml
+- japanese
+- causal-lm
+- contextual-reranking
+- tiny-gpt
 ---
+
+# VimeML Tiny Japanese GPT v1
+
+A frozen **7,386,624-parameter**, decoder-only Japanese language model for local contextual candidate reranking and short within-sentence continuation. This repository provides the FP32 inference bundle and an iOS18 INT8-weight Core ML variant. SentencePiece tokenization, kana candidate retrieval, scoring and search run outside the Core ML graph.
+
+This is a custom PyTorch model, not a Transformers `AutoModel` integration or an instruction/chat model. The source model, loader, scoring and conversion code are included under `source/`; no `trust_remote_code` or training checkpoint is needed.
+
+## Files and identity
+
+| Path | Contents |
+| --- | --- |
+| `inference/` | Unique FP32 state_dict, config, original SentencePiece model, token manifest and provenance manifest; no optimizer |
+| `coreml/ios18-int8-block32/` | Original Core ML `.mlpackage` and manifest |
+| `source/` | Frozen VimeML Python source, conversion scripts, configuration and unified requirements |
+| `infer.py` | Read-only wrapper around the original fingerprint-checked loader, generation and scoring |
+| `evaluation/summary.json` | Curated results and source report hashes, without benchmark texts or device traces |
+| `RELEASE.json`, `SHA256SUMS.txt`, `verify_release.py` | Release inventory and local byte verification |
+
+- Source repository: [Voltline/VimeML](https://github.com/Voltline/VimeML), commit **9f0409ef7970c468efb622b207b1089de09651c1**.
+- Inference manifest SHA256: `4284cdc5491513cc817633081677b363b14f59c0b4ffb78cdf7fcfb221f71b3c`.
+- Core ML manifest SHA256: `a9d516233fa57aaadd09477e2e909760972ff49b5577919f8a441c489f5c2245`.
+- Tokenizer SHA256: `d9f1ba1e456ce72dd9c06a62b12e804cf14090c30179e6078cd51d03063b2982`.
+- Original training checkpoint SHA256: `93b6139aa05031df02038efcaf26788bc9916712919f3d2a4c4f398615cef75b`; the training checkpoint is not distributed here.
+
+The original manifests remain byte-for-byte intact. Their old absolute paths describe provenance, not paths required on a downloader's machine. Core ML conversion scripts evolved after the original experiment; the old manifest's script hash is retained.
+
+## Architecture and tokenization
+
+Pre-LayerNorm GPT with GELU, learned absolute positions and causal attention: vocabulary 16,384; context length 128; width 256; 4 layers; 4 heads; FFN width 1024. Input embedding and LM head share one parameter. There is no KV cache or sliding context. Each generation step recomputes the entire prefix.
+
+SentencePiece 0.2.1 unigram, identity normalization, whitespace preservation and byte fallback. PAD/UNK/BOS/EOS IDs are 0/1/2/3. The exact original `tokenizer.model` is required; retraining a tokenizer does not preserve this vocabulary binding.
+
+## Quick start: downloaded files
+
+Python ≥3.11. After downloading the repository, install the included unified requirements. Select the appropriate PyTorch build first if needed; do not use the PyTorch-only package index for the whole requirements file. The Darwin branch pins the original Core ML tool environment; no Core ML prediction occurs during FP32 inference.
+
+```sh
+python -m pip install -r source/requirements.txt
+python verify_release.py
+python infer.py --prompt "今日は雨が降っているので、" --max-new-tokens 8
+python infer.py --prompt "雨が降っているので、" --candidate "傘を持っていく" --candidate "橋を渡っていく"
+```
+
+The second inference command ranks the supplied strings using the original contextual suffix logP sum. It does not perform kana conversion or dictionary lookup. Invalid/overlong candidates raise in this diagnostic wrapper; the application implementation retains the dictionary order on eligibility failure.
+
+The FP32 loader checks all bundle files and the included source fingerprints, restores the shared head, and uses `torch.load(..., weights_only=True)`. Keep the source, tokenizer and inference files together. `verify_release.py` checks every release file without loading a model; checksums detect altered bytes but are not a digital signature.
+
+## Candidate scoring
+
+Encode `context + candidate` jointly. Cancel the common token prefix across BOS+context and the jointly encoded candidate sequences, then sum original full-vocabulary log-softmax values over the remaining suffix. Do not add a candidate-final EOS. Ties preserve the original order. This token suffix likelihood is a scoring proxy, not exact string conditional probability.
+
+Deployment uses **pure LM scoring**; AzooKey retrieves kana candidates. The older FP32 `AzooKey score + 2 × LM sum` policy is a historical comparison, not the INT8 deployment policy. Quantization does not establish that λ=2 remains optimal.
+
+## Training and evaluation
+
+Training used nine local shards of [FineWeb2-Edu Japanese](https://huggingface.co/datasets/hotchpotch/fineweb-2-edu-japanese) and Japanese [Tatoeba](https://tatoeba.org/), cleaned, split into sentences, exactly deduplicated and grouped across 98%/1%/1% train/validation/test splits. There were 25,713,003 unique sentences and 25,185,368 training sentences. The tokenizer learned from 2 million sampled training sentences.
+
+One data pass, 196,853 updates and 573,295,237 prediction targets. The frozen best checkpoint's complete validation loss is 4.5415708951, perplexity 93.8380942. Fixed-subset validation selected the checkpoint; the test split was not used for training, checkpoint selection or tuning. No test perplexity is claimed.
+
+Reranking uses fixed N-best 20 AzooKey candidates, without injecting correct answers. [AJIMEE JWTD_v2/v1](https://github.com/azooKey/AJIMEE-Bench) has 200 cases and allows multiple accepted spellings. The synthetic development set has 137 reviewed cases.
+
+| Frozen pool | AzooKey original Top-1 | FP32 pure LM Top-1 / Top-5 | INT8 pure LM Top-1 / Top-5 | Pool coverage |
+| --- | ---: | ---: | ---: | ---: |
+| Development | 111/137 | 122/137 / 134/137 | 122/137 / 134/137 | 136/137 |
+| AJIMEE | 87/200 | 124/200 / 151/200 | 124/200 / 151/200 | 162/200 |
+
+INT8 vs Windows FP32: development changes 119 complete rankings and 0 first choices; AJIMEE changes 163 complete rankings and 2 first choices, both between acceptable spellings. The original development compression report instead compares against uncompressed conservative Core ML: 118 complete rankings change. Equal hit counts do not mean equal scores or complete rankings. See the release summary for the separate baselines and hashes.
+
+## Core ML variant
+
+**Weight-only symmetric INT8, block size 32; FP32 computation; CPU_ONLY; minimum iOS 18.** It is not INT8 activation or arithmetic inference. Logical `.mlpackage` size is **8,077,801 bytes**. This is not compiled size, IPA size or resident memory.
+
+Input `input_ids`: INT32 `[1,T]`, 1≤T≤128. Output `logits`: FLOAT32 `[1,T,16384]`, full raw vocabulary logits. Only right-side PAD is supported. Shared embedding/head are retained. Use `.cpuOnly`; GPU/ALL paths for this package triggered a quantized-gather assertion in the client experiments.
+
+On Mac, compile manually for the target platform before integrating:
+
+```sh
+xcrun coremlcompiler compile coreml/ios18-int8-block32/model.mlpackage compiled-int8 --deployment-target 18.0
+```
+
+The INT8 model **fails the strict FP32 logits alignment gate** (maximum absolute difference about 3.66). Candidate fixtures, causal and right-PAD checks passed, and the frozen pool hit counts above were preserved. Do not describe the model as numerically identical to FP32.
+
+An AJIMEE-text diagnostic reported mean KL(FP32‖INT8)≈0.00589, mean NLL 5.56698→5.57289 and position top-choice agreement 94.4%. These are means of per-example position means, not a corpus token-weighted aggregate. Text content was capped at 127 tokens plus BOS; the old report did not record truncation counts. Similar aggregate KL to a simulated INT8 recipe does not prove exact conversion equivalence. Tested palette4 group16 degraded substantially and is not included.
+
+## Performance evidence and limits
+
+Recorded iPhone 16 Pro Max optimized CPU_ONLY **test-host App** results: approximately 14 candidate reranking p50 10.4ms/p95 12.7ms; next-word p50 5.8ms; T128 prediction 3.0ms. Host memory increased by about 24.2MB and peaked at 74.8MB. These are not keyboard-extension latency or memory measurements.
+
+A separate 152.7 second manual real-keyboard recording produced 127 approximately 1 second samples: physical footprint peak 21.25MiB and private resident peak 51.922MiB. The user confirmed LM ranking and next-word suggestions; installed build/model hashes were not independently captured. Short spikes may have been missed, the tail still grew slowly, and long-session stability is not certified. The different memory metrics cannot be substituted for one another. No energy claim is made.
+
+## Intended use and limitations
+
+Japanese within-sentence reranking and short completion experiments, primarily for on-device input methods. The model is small, not instruction-tuned, and can repeat text, drift, truncate or invent facts. Outputs need product-level review; the fixed beam suite had only 10/20 prefixes with clearly natural suggestions in an AI review. These offline pool metrics do not measure application-wide accuracy or guarantee keyboard performance.
+
+Semantic near-duplicates and training overlap with public evaluation text have not been comprehensively audited. AJIMEE has been used for error analysis and is not a fresh blind test. Development labels and review came from AI, not native-speaker adjudication. Official benchmark context can cross sentences, while the client uses current-sentence context. Candidate coverage limits reranking accuracy.
+
+## License and attribution
+
+Model weights and the included VimeML code are released under **GNU GPL v2.0**, chosen by the author. See `LICENSE` and `source/LICENSE`. Training data and third-party software retain their own terms; they are not relicensed by this model repository.
+
+FineWeb2-Edu Japanese's dataset card declares ODC-BY; [Tatoeba text terms](https://tatoeba.org/en/terms_of_use) default to CC-BY 2.0 FR with author attribution requirements. No corpus text is redistributed here. AJIMEE's source data is CC-BY-SA 3.0; this release distributes aggregate metrics, not its evaluation text. SentencePiece runtime code is not vendored into this release; installed dependencies retain their upstream licenses. Source configuration and training/conversion implementations are included, while datasets, API responses, optimizer state, device traces and the separate Vime client are excluded.

RELEASE.json

@@ -0,0 +1,481 @@
+{
+  "format": "vimeml_hf_release_v1",
+  "repo_id": "Voltline/vimeml-tiny-ja-v1",
+  "license": "gpl-2.0",
+  "source_repository": "https://github.com/Voltline/VimeML",
+  "source_commit": "9f0409ef7970c468efb622b207b1089de09651c1",
+  "created_utc": "2026-10-06T14:54:06.938207+00:00",
+  "bundle_manifest_sha256": "4284cdc5491513cc817633081677b363b14f59c0b4ffb78cdf7fcfb221f71b3c",
+  "coreml_manifest_sha256": "a9d516233fa57aaadd09477e2e909760972ff49b5577919f8a441c489f5c2245",
+  "files": {
+    "coreml/ios18-int8-block32/manifest.json": {
+      "bytes": 13273,
+      "sha256": "a9d516233fa57aaadd09477e2e909760972ff49b5577919f8a441c489f5c2245"
+    },
+    "coreml/ios18-int8-block32/model.mlpackage/Data/com.apple.CoreML/model.mlmodel": {
+      "bytes": 46400,
+      "sha256": "22e80193abc7672bbcf2a3fc214a186ae6e64d76c1e1b42ead38fef5a003cd37"
+    },
+    "coreml/ios18-int8-block32/model.mlpackage/Data/com.apple.CoreML/weights/weight.bin": {
+      "bytes": 8030784,
+      "sha256": "d3a063455371e624cf208fb6ed76fc9ac6addd8c2f87eaebf4653649f85b8109"
+    },
+    "coreml/ios18-int8-block32/model.mlpackage/Manifest.json": {
+      "bytes": 617,
+      "sha256": "31b13734848024036806cbd5bacf405eddba6c4451e5659b25f714ffe1766ee5"
+    },
+    "evaluation/summary.json": {
+      "bytes": 9055,
+      "sha256": "d0fe405a4f68c3e5f94541834ab9f60cf0937b4f2fd6c50d25737e6ab828ad0e"
+    },
+    "infer.py": {
+      "bytes": 1528,
+      "sha256": "8ed8d241d349a4e88840dd841da9db3191ec9f0db84c8f8165a951fa84720621"
+    },
+    "inference/config.json": {
+      "bytes": 145,
+      "sha256": "8950524f2f3a0373f3a0ebbcde79a8feaa54949ed926830c287b384306adf9ca"
+    },
+    "inference/manifest.json": {
+      "bytes": 4746,
+      "sha256": "4284cdc5491513cc817633081677b363b14f59c0b4ffb78cdf7fcfb221f71b3c"
+    },
+    "inference/token-manifest.json": {
+      "bytes": 6208,
+      "sha256": "c7ebf3672d46cfc75d12013c34f35fb7ac9eac613f29f3454be0cb0e991854dc"
+    },
+    "inference/tokenizer.model": {
+      "bytes": 284118,
+      "sha256": "d9f1ba1e456ce72dd9c06a62b12e804cf14090c30179e6078cd51d03063b2982"
+    },
+    "inference/weights.pt": {
+      "bytes": 29562119,
+      "sha256": "1cdac16913068bfdcfa94b139ecb34d945d0e9483a863373a6dca4397e3cd355"
+    },
+    "LICENSE": {
+      "bytes": 18431,
+      "sha256": "189b1af95d661151e054cea10c91b3d754e4de4d3fecfb074c1fb29476f7167b"
+    },
+    "README.md": {
+      "bytes": 10728,
+      "sha256": "366194211cc7db2f7de4f82471796b543f829e4c646a984758f2a5d589556f70"
+    },
+    "source/configs/corpus-parallel.toml": {
+      "bytes": 1239,
+      "sha256": "67e3bf5c3166c9c5fa847ec515c44dd41dfebee56134265df2c5b13bec14ad9a"
+    },
+    "source/configs/corpus-sharded.toml": {
+      "bytes": 1179,
+      "sha256": "49f06564931dc4be63f76cdde8cd3be78ddeb81a45cbada768e0a0ca8191a49d"
+    },
+    "source/configs/ime-eval-v1.json": {
+      "bytes": 8368,
+      "sha256": "46d0b4eba6293a2fa2e7ef16d9132c6320e9d2da80921d60aa939e90e35dc222"
+    },
+    "source/configs/ime-homophones.json": {
+      "bytes": 1195,
+      "sha256": "61c475fba63964b540e55be39d70e786886c059ce58fee94465517f648f021fe"
+    },
+    "source/configs/inference-prompts.json": {
+      "bytes": 685,
+      "sha256": "e701173b553617086ad2ab679313423eabb19e89d9f51fd31b321336584652ea"
+    },
+    "source/configs/loader.toml": {
+      "bytes": 249,
+      "sha256": "e8baa76da3e223bcb3b333f96994ef4e86eeecb62cea18503eb8f7ad9c4a74cf"
+    },
+    "source/configs/phrase-demo-prompts.json": {
+      "bytes": 1500,
+      "sha256": "b51e101d6a4bc34c4a375439191e4adbd63d511ccc62b49c4d5b758decee17ed"
+    },
+    "source/configs/tokenizer.toml": {
+      "bytes": 658,
+      "sha256": "28396838808e5745c4d2251e90e095d2a2e816033526bcb154d1c74c98dfe9b5"
+    },
+    "source/configs/train-smoke.toml": {
+      "bytes": 791,
+      "sha256": "2d386dd8efe0ffd0da6df319226daf7bab8948b1bcb2abc7afb59c45d98d28b3"
+    },
+    "source/configs/train-v1.toml": {
+      "bytes": 900,
+      "sha256": "44ba93246d406d5187c496ad2cf23d1533e6f581a8d6dc6f7fff03fca22b8323"
+    },
+    "source/LICENSE": {
+      "bytes": 18431,
+      "sha256": "189b1af95d661151e054cea10c91b3d754e4de4d3fecfb074c1fb29476f7167b"
+    },
+    "source/README.md": {
+      "bytes": 274,
+      "sha256": "5184082e7c857b4ab965d3c5b7366a0d5b56d98cdb5e40511dda6d0c2e298290"
+    },
+    "source/requirements.txt": {
+      "bytes": 968,
+      "sha256": "0fac5689cead9021745847b2753294a9a4d95c002a61762f7f25254db2a26489"
+    },
+    "source/scripts/benchmarks/evaluate.py": {
+      "bytes": 278,
+      "sha256": "34a2a24d14c0e8b20c44dafab720075cbc23261a7fdbe71091dff12200e7fc24"
+    },
+    "source/scripts/benchmarks/evaluate_ajimee.py": {
+      "bytes": 291,
+      "sha256": "a28fc24da1f8cbb6972b49dc0223361e16c7b093872d92df6bd52aabd00bd54a"
+    },
+    "source/scripts/benchmarks/export_cached_development.py": {
+      "bytes": 1024,
+      "sha256": "d328ff258e4a8fa83e51be099f5fa8fbf480f2d377579c475c355f9f01263bf6"
+    },
+    "source/scripts/benchmarks/generate.py": {
+      "bytes": 269,
+      "sha256": "de9866c56cd846073a3a3a82b1553ab737ccb911001d17de71839865057d0293"
+    },
+    "source/scripts/benchmarks/generate_development.py": {
+      "bytes": 279,
+      "sha256": "3b3b4192535a035bbf57e3f51f1fb16a0647b1a3fc060a179b7ae9fb1127cc41"
+    },
+    "source/scripts/benchmarks/hybrid.py": {
+      "bytes": 281,
+      "sha256": "64dbd0fcee7e294bd290bbf9f2057e81656ede4e0b328cdae710360da24716b3"
+    },
+    "source/scripts/benchmarks/prepare_ajimee.py": {
+      "bytes": 272,
+      "sha256": "3935e16a81c389c5a3dd74da6da38732ff32b668ea3c6d0484c52f017657706a"
+    },
+    "source/scripts/corpus/build.py": {
+      "bytes": 271,
+      "sha256": "0e4d753adf6cf97cc19b975e4f35d3cea41a4f519782f6b8eeea3309260486df"
+    },
+    "source/scripts/corpus/merge.py": {
+      "bytes": 1335,
+      "sha256": "dd6639d55b9aecdbf46d20fec98ab78422ff53ad14ec5a4a39d6a43e824cd0d6"
+    },
+    "source/scripts/corpus/preprocess.py": {
+      "bytes": 279,
+      "sha256": "f8181f1b055661a1097d2bf80654ac59af4cf6a8b74afecb8f44b40e3d94d4f8"
+    },
+    "source/scripts/deployment/ane_output_probe.py": {
+      "bytes": 3786,
+      "sha256": "e3e12fd3373119539eb3ed23eaccbc1457cafa7a22ae444dd4b187c118d883ce"
+    },
+    "source/scripts/deployment/compare_quantization.py": {
+      "bytes": 6534,
+      "sha256": "9a0761dbd3ece9b730ed0e52010ef965a8415b4320188c31e1863ab0c874d2fd"
+    },
+    "source/scripts/deployment/compute_plan.py": {
+      "bytes": 2759,
+      "sha256": "2aa26fb2f2b59a630cc9cafb7eb684f237c44c780c1ec820badbbd07d8f2bf41"
+    },
+    "source/scripts/deployment/coreml.py": {
+      "bytes": 278,
+      "sha256": "7f321ab43b41acd4765045a93edc5c3ec874dcb741541a3f7460d13f8c27f50e"
+    },
+    "source/scripts/deployment/coreml_ane.py": {
+      "bytes": 4674,
+      "sha256": "28c67b5b30c3ad93a46703dad62236b2cd81ec5478ebf2cd9974e7b336001871"
+    },
+    "source/scripts/deployment/coreml_conservative.py": {
+      "bytes": 14097,
+      "sha256": "61b290658385ee40325263be24b1a1dffdc18fe5bc494dd02a1bf93bf301b986"
+    },
+    "source/scripts/deployment/package_handoff.py": {
+      "bytes": 6406,
+      "sha256": "65db41ee70efb5409738ece23eb41750e695f1746e42b85314198d72710fa7a7"
+    },
+    "source/scripts/deployment/package_ios.py": {
+      "bytes": 4266,
+      "sha256": "1228adeb409e3cf095aa7890390307b9cd518c535fc23365f5a8ffc01b144448"
+    },
+    "source/scripts/deployment/summarize_keyboard_trace.py": {
+      "bytes": 4106,
+      "sha256": "0857bab4b1dcf81ec8b1ae5e680672cf512d5547aa3246c1e2a7ad7bfc0b79e5"
+    },
+    "source/scripts/deployment/summarize_memory.py": {
+      "bytes": 3339,
+      "sha256": "a798a4478ba907f0b4c88246c90d865063fa5c7c02f10456e7faceea7a7966ba"
+    },
+    "source/scripts/publishing/hub.py": {
+      "bytes": 2825,
+      "sha256": "bbd7f36ed4682bcdcb5e4c5d87b42733106dbaebf6f949fcd60cd498dc059274"
+    },
+    "source/scripts/publishing/prepare_hf.py": {
+      "bytes": 12512,
+      "sha256": "994c690a95f8d13d15a1ab1671185c8e2744537f48d667c176f9a3079b3630e9"
+    },
+    "source/scripts/review/approve.py": {
+      "bytes": 273,
+      "sha256": "555b275e74482c597a26b3cbc23ffb1648e636801bc5ddeccf7c25df176414d0"
+    },
+    "source/scripts/review/prepare.py": {
+      "bytes": 273,
+      "sha256": "77578100f9aaae50cabac68a85996dcf02479179f6d17fbf30368ea7aa70ab5f"
+    },
+    "source/scripts/review/run.py": {
+      "bytes": 269,
+      "sha256": "d18a127f4a08b84ac03a398f3c96c935a0b6275fe0e7b1af2f3e3d4d92ec49b1"
+    },
+    "source/scripts/review/run_multi_key.py": {
+      "bytes": 282,
+      "sha256": "702ba0084944c8db433f1d12f64e6933659965a9a125434cce80f916f63c6f47"
+    },
+    "source/scripts/tokenizer/encode.py": {
+      "bytes": 284,
+      "sha256": "46af106351c8835ba0e001295c4d2122c8c629d799c913feb8292d2e83100670"
+    },
+    "source/scripts/tokenizer/train.py": {
+      "bytes": 274,
+      "sha256": "1001c67f565711d91c59335b342ab1a3e3fdb40f7f5642a8a49ce9ac46a4823d"
+    },
+    "source/scripts/tools/check_corpus.py": {
+      "bytes": 282,
+      "sha256": "231f867355c0cd2806e0ee20378c9671443309f2716db9dbf1426fda6bbe9eb6"
+    },
+    "source/scripts/tools/inspect_data.py": {
+      "bytes": 350,
+      "sha256": "fd5a05f0c74e4d3bd95097a37be009422ec348c3fd3e36f1ca12740c02309760"
+    },
+    "source/scripts/tools/phrase_demo.py": {
+      "bytes": 274,
+      "sha256": "09d81377f63d9c967e444ffdaaec83a7228de8109c4492a5175b46a8590387bc"
+    },
+    "source/scripts/training/check_data.py": {
+      "bytes": 278,
+      "sha256": "fce755bfd04595f95d6a021c5d2533599d3463542239da198d65e4c26a716e0a"
+    },
+    "source/scripts/training/evaluate_ime.py": {
+      "bytes": 274,
+      "sha256": "81729525388462eb93ede97f917f0f6eae1ef6518c35d169e19dc745f37ffa4d"
+    },
+    "source/scripts/training/infer.py": {
+      "bytes": 252,
+      "sha256": "6f409f7761d543e65d5f4c8886c8f59e0929b4b299977a8439bd0d13623755f4"
+    },
+    "source/scripts/training/train.py": {
+      "bytes": 263,
+      "sha256": "56b5781f98b3440d23d1516d830e83356db3a0b04c9aab134368c2ef9cefb91c"
+    },
+    "source/scripts/training/train_wandb.py": {
+      "bytes": 3345,
+      "sha256": "db40a3f34315dcd409f0b22d8633c569f71f6807841fb2551a9ca423869ca149"
+    },
+    "source/src/vimeml/__init__.py": {
+      "bytes": 52,
+      "sha256": "ff0f42adee5b707f35ea0ba1307e65d8a2f632a55c3281c8f446211032f0fbc1"
+    },
+    "source/src/vimeml/benchmarks/__init__.py": {
+      "bytes": 77,
+      "sha256": "03a0b8a9e43ef3ecf56189a86fe7e33d9633d4a766edf02583e911aef9bde581"
+    },
+    "source/src/vimeml/benchmarks/ajimee.py": {
+      "bytes": 6566,
+      "sha256": "4916651168f3f883161ebbbc24db5e1ae5b1bc4cd6abf8c7b83560ef960406cb"
+    },
+    "source/src/vimeml/benchmarks/development.py": {
+      "bytes": 36457,
+      "sha256": "d74e8f9e888c115c6a9ce27f5b08ea40efb332c087d94e372d4c33364b2dc2d4"
+    },
+    "source/src/vimeml/benchmarks/evaluate.py": {
+      "bytes": 9097,
+      "sha256": "867cd2f41b49043dd985e7f05e788f3f83a0ac47a4d0d9b2d6306dcd0c96b02a"
+    },
+    "source/src/vimeml/benchmarks/evaluate_ajimee.py": {
+      "bytes": 13783,
+      "sha256": "459a8c0d1bb4f8ffba57e92ee2c8bb3f5278c3e4a1a5bcc5dfda48a21bde3716"
+    },
+    "source/src/vimeml/benchmarks/generate.py": {
+      "bytes": 28505,
+      "sha256": "9a468c23ea3ebf02f1184b1742cbe11a025f2a0d7ae57e0ee2e54468de72a1d7"
+    },
+    "source/src/vimeml/benchmarks/hybrid.py": {
+      "bytes": 20442,
+      "sha256": "71267f445ac92736f1dabf03c3e77193e9f2a1323f23fa1bc060aff4fbe4ea7e"
+    },
+    "source/src/vimeml/corpus/__init__.py": {
+      "bytes": 32,
+      "sha256": "55cdd3f899dd7ad156f8987cf2ba2bbbc76a95dfaba33a319058b4d01e2f9947"
+    },
+    "source/src/vimeml/corpus/build.py": {
+      "bytes": 12675,
+      "sha256": "895b6894c4b85e72c8655cf6b42e21d51b51736d649ec2adb6841e033f4ac1f9"
+    },
+    "source/src/vimeml/data/__init__.py": {
+      "bytes": 30,
+      "sha256": "86d655d2d715fcb2f73269c1ffbfc9e87d9080a1baaf389f88e30de6022fdc19"
+    },
+    "source/src/vimeml/data/annotations.py": {
+      "bytes": 6610,
+      "sha256": "ba62919c8084d24cb3f41c451e970b5448e7b0f5407340c28cca0b220d387395"
+    },
+    "source/src/vimeml/data/build.py": {
+      "bytes": 29679,
+      "sha256": "d7d63e1ea9a2f9da9fe92ff2773b792c7ca89911b2d7058a2709fcbbc443081e"
+    },
+    "source/src/vimeml/data/clean.py": {
+      "bytes": 1791,
+      "sha256": "ee88adbb74d5a03f591231484928becf0a62557c85702efddd214c9bc8a2354f"
+    },
+    "source/src/vimeml/data/corpus_export.py": {
+      "bytes": 8056,
+      "sha256": "d2e943aae1e2942076b1db8430067e048fcdc71d46761bb329f7fa460e843280"
+    },
+    "source/src/vimeml/data/corpus_parts.py": {
+      "bytes": 10178,
+      "sha256": "c196a78e5096e659eb8c536602e009c98de346bd82084c29f9384e526055f6f6"
+    },
+    "source/src/vimeml/data/merge_parts.py": {
+      "bytes": 8755,
+      "sha256": "43cb87845d17e52261675073b010269c53e4c79a8e5e16237d2627c9e07116e6"
+    },
+    "source/src/vimeml/data/merge_parts_fast.py": {
+      "bytes": 27413,
+      "sha256": "600dc56899dfa6ac2eda639cf79f6ad5e789642b99d6c68569262f593bfc3642"
+    },
+    "source/src/vimeml/data/preprocess_part.py": {
+      "bytes": 6553,
+      "sha256": "433a93f18809aa1b11a13c8e63edce2a73864140fbaf4a4bcff0c7ab475be164"
+    },
+    "source/src/vimeml/data/readers.py": {
+      "bytes": 4734,
+      "sha256": "837017b0282f62ed68c9a47c69da3ed12fe39b74e7f88bc3ae16113f55b679b6"
+    },
+    "source/src/vimeml/data/segment.py": {
+      "bytes": 8309,
+      "sha256": "9834f4867e126f229e86a7fc5b7481f4c803c05d00a21966a78ec293095b0f8c"
+    },
+    "source/src/vimeml/deployment/__init__.py": {
+      "bytes": 86,
+      "sha256": "9301bfc92623e4a8b2589e01d7ca98aac917152544b83562b23628bf4cfe414f"
+    },
+    "source/src/vimeml/deployment/bundle.py": {
+      "bytes": 7978,
+      "sha256": "4f66f215e576ea1267971da0f49d995f49b09417920cbbf9e159e2d1c8083043"
+    },
+    "source/src/vimeml/deployment/cli.py": {
+      "bytes": 6394,
+      "sha256": "55d65451e04721c3c05a131b9c422f2e7c1835d6340cc8b76c4936243521dd6a"
+    },
+    "source/src/vimeml/deployment/coreml.py": {
+      "bytes": 8328,
+      "sha256": "8281456e241e32ed79f9223d63f5c476e02b97b263723be9ad46c0a75435ee52"
+    },
+    "source/src/vimeml/deployment/graph.py": {
+      "bytes": 2306,
+      "sha256": "e2474a97e4006c4e1fe14552b11874d08c52fa217b41c9d0ab0dbab7b0a79a67"
+    },
+    "source/src/vimeml/deployment/validation.py": {
+      "bytes": 15542,
+      "sha256": "9e5d526dbd8e9be72f9d8782ac402623007226ee9e1ca14f8ce005a52a994c95"
+    },
+    "source/src/vimeml/review/__init__.py": {
+      "bytes": 32,
+      "sha256": "6422f9e2042a114155c7caf2721c1f6bd8e5de4d153009703af1281f2a1c76c3"
+    },
+    "source/src/vimeml/review/approve.py": {
+      "bytes": 3982,
+      "sha256": "b81ca9f1dcbe1b8e36f189723e81f2a3d44e9ef51f5a1c6f9ad0ce96a9b00536"
+    },
+    "source/src/vimeml/review/multi_key.py": {
+      "bytes": 8932,
+      "sha256": "e21ebd1e94159d17c26234191f2f1d1ee95e542c54a9762fe0a5717a73812e4a"
+    },
+    "source/src/vimeml/review/prepare.py": {
+      "bytes": 27338,
+      "sha256": "968209bb52d010c75066d2296a93bf27b30d264e614927d5e72bbe7c0a0b9323"
+    },
+    "source/src/vimeml/review/run.py": {
+      "bytes": 27866,
+      "sha256": "8507f1850e5b8af416108ff20b4283637d03fa267add33548674dedd7445b036"
+    },
+    "source/src/vimeml/tokenizer/__init__.py": {
+      "bytes": 35,
+      "sha256": "77c08158b861bafd5e902c886a0878a08a4afea25a5277dd33be60ae219ff624"
+    },
+    "source/src/vimeml/tokenizer/encode.py": {
+      "bytes": 9961,
+      "sha256": "92bdbdda6fa80f4dbb975f8882c021d8c7c0f9017f669c89dfae31dc2eecaa69"
+    },
+    "source/src/vimeml/tokenizer/encode_parallel.py": {
+      "bytes": 21649,
+      "sha256": "c194cb241d40b49669515f68f44278b55bcfbae210756f220b375cdee22ab009"
+    },
+    "source/src/vimeml/tokenizer/monitor.py": {
+      "bytes": 4352,
+      "sha256": "184a02f06f477a6037368ad2c2f43a32299fef83d009d851083e452ce91cf7d9"
+    },
+    "source/src/vimeml/tokenizer/native_train.py": {
+      "bytes": 387,
+      "sha256": "f42ef20a1e2b4aaa2bc6c04a2357b3c0375bb20a6055b49225318d08760ab01a"
+    },
+    "source/src/vimeml/tokenizer/store.py": {
+      "bytes": 3243,
+      "sha256": "53b285db5bfff0127281c8a35acc1beba8f5dfee1c6ae05e46923e91f9befd91"
+    },
+    "source/src/vimeml/tokenizer/train.py": {
+      "bytes": 21332,
+      "sha256": "81c218f4185345e443a5bfe1f80da024ad1260b6e3653ceb52938824cd60d752"
+    },
+    "source/src/vimeml/tools/__init__.py": {
+      "bytes": 31,
+      "sha256": "0a0511ce3c52639d6a3b0a1f030162d003bb930a2a7c795cc5f15fa68bdcb6b7"
+    },
+    "source/src/vimeml/tools/check_corpus.py": {
+      "bytes": 3242,
+      "sha256": "770f251e80a34626b3b36a193c9fd9ebaa3f4c78885382cef6cb060db7b3d762"
+    },
+    "source/src/vimeml/tools/inspect_data.py": {
+      "bytes": 4402,
+      "sha256": "5f180d035a6e73d3931959d1ae9a2eece8442390a59e522d0e908285205eb6c1"
+    },
+    "source/src/vimeml/tools/phrase_demo.html": {
+      "bytes": 9275,
+      "sha256": "c298c93b65c6c81782d5a4edea52d72e58d78a1db6c28412e15e338d97f409b5"
+    },
+    "source/src/vimeml/tools/phrase_demo.py": {
+      "bytes": 11017,
+      "sha256": "07023e63ece7f683038d0a823bac49722619940a0b1a387a989696effaaea16c"
+    },
+    "source/src/vimeml/training/__init__.py": {
+      "bytes": 64,
+      "sha256": "30be98c6d0dc0b68bb12915c37cb84d97cc3c47bc1c0734e45435ac033a02180"
+    },
+    "source/src/vimeml/training/check_data.py": {
+      "bytes": 7618,
+      "sha256": "c2530ba10df54c190211f59d1756f36de4792a4344dc78d14b88305e16a2f4be"
+    },
+    "source/src/vimeml/training/data.py": {
+      "bytes": 16536,
+      "sha256": "1627a4db9bb3c179977dc25e08b9443e84c0a966ff8416c3e5872c456b5b3ae3"
+    },
+    "source/src/vimeml/training/evaluate_ime.py": {
+      "bytes": 18317,
+      "sha256": "9a36b75893f58f61228ad7a6bbb2a34b52aaf312ca6675383580964500d639cd"
+    },
+    "source/src/vimeml/training/infer.py": {
+      "bytes": 8545,
+      "sha256": "7a093392ed15377c54bd1842cfbf32c6ab0257f3e956a81e84e41ad4624a9e9c"
+    },
+    "source/src/vimeml/training/model.py": {
+      "bytes": 4652,
+      "sha256": "dde6a0f8d7bd2d3a76e48fb503c22296da4c1156f976125bdb773d902bc7e07e"
+    },
+    "source/src/vimeml/training/train.py": {
+      "bytes": 25943,
+      "sha256": "dd34aa271f9a62af55aa877fc4090a1db0001cdce656faadd37304524eb59c5b"
+    },
+    "source/templates/huggingface/infer.py": {
+      "bytes": 1528,
+      "sha256": "8ed8d241d349a4e88840dd841da9db3191ec9f0db84c8f8165a951fa84720621"
+    },
+    "source/templates/huggingface/README.md": {
+      "bytes": 10512,
+      "sha256": "169add32b203da3142840133dc27456a63a576baecb5f9a9273d923464a1fcff"
+    },
+    "source/templates/huggingface/verify_release.py": {
+      "bytes": 2997,
+      "sha256": "80dd64ae4dac2eddbe864a2d2c4f429cf8b0bb6317b8eaf962fd378e11449b90"
+    },
+    "verify_release.py": {
+      "bytes": 2997,
+      "sha256": "80dd64ae4dac2eddbe864a2d2c4f429cf8b0bb6317b8eaf962fd378e11449b90"
+    }
+  },
+  "policy": "Local preparation only. No training, conversion, compression, device access or upload."
+}

SHA256SUMS.txt

@@ -0,0 +1,118 @@
+189b1af95d661151e054cea10c91b3d754e4de4d3fecfb074c1fb29476f7167b  LICENSE
+366194211cc7db2f7de4f82471796b543f829e4c646a984758f2a5d589556f70  README.md
+9a1394b38c9f68911b1fa90ad06a371757fa069837bf9c24177c02761684fc24  RELEASE.json
+a9d516233fa57aaadd09477e2e909760972ff49b5577919f8a441c489f5c2245  coreml/ios18-int8-block32/manifest.json
+22e80193abc7672bbcf2a3fc214a186ae6e64d76c1e1b42ead38fef5a003cd37  coreml/ios18-int8-block32/model.mlpackage/Data/com.apple.CoreML/model.mlmodel
+d3a063455371e624cf208fb6ed76fc9ac6addd8c2f87eaebf4653649f85b8109  coreml/ios18-int8-block32/model.mlpackage/Data/com.apple.CoreML/weights/weight.bin
+31b13734848024036806cbd5bacf405eddba6c4451e5659b25f714ffe1766ee5  coreml/ios18-int8-block32/model.mlpackage/Manifest.json
+d0fe405a4f68c3e5f94541834ab9f60cf0937b4f2fd6c50d25737e6ab828ad0e  evaluation/summary.json
+8ed8d241d349a4e88840dd841da9db3191ec9f0db84c8f8165a951fa84720621  infer.py
+8950524f2f3a0373f3a0ebbcde79a8feaa54949ed926830c287b384306adf9ca  inference/config.json
+4284cdc5491513cc817633081677b363b14f59c0b4ffb78cdf7fcfb221f71b3c  inference/manifest.json
+c7ebf3672d46cfc75d12013c34f35fb7ac9eac613f29f3454be0cb0e991854dc  inference/token-manifest.json
+d9f1ba1e456ce72dd9c06a62b12e804cf14090c30179e6078cd51d03063b2982  inference/tokenizer.model
+1cdac16913068bfdcfa94b139ecb34d945d0e9483a863373a6dca4397e3cd355  inference/weights.pt
+189b1af95d661151e054cea10c91b3d754e4de4d3fecfb074c1fb29476f7167b  source/LICENSE
+5184082e7c857b4ab965d3c5b7366a0d5b56d98cdb5e40511dda6d0c2e298290  source/README.md
+67e3bf5c3166c9c5fa847ec515c44dd41dfebee56134265df2c5b13bec14ad9a  source/configs/corpus-parallel.toml
+49f06564931dc4be63f76cdde8cd3be78ddeb81a45cbada768e0a0ca8191a49d  source/configs/corpus-sharded.toml
+46d0b4eba6293a2fa2e7ef16d9132c6320e9d2da80921d60aa939e90e35dc222  source/configs/ime-eval-v1.json
+61c475fba63964b540e55be39d70e786886c059ce58fee94465517f648f021fe  source/configs/ime-homophones.json
+e701173b553617086ad2ab679313423eabb19e89d9f51fd31b321336584652ea  source/configs/inference-prompts.json
+e8baa76da3e223bcb3b333f96994ef4e86eeecb62cea18503eb8f7ad9c4a74cf  source/configs/loader.toml
+b51e101d6a4bc34c4a375439191e4adbd63d511ccc62b49c4d5b758decee17ed  source/configs/phrase-demo-prompts.json
+28396838808e5745c4d2251e90e095d2a2e816033526bcb154d1c74c98dfe9b5  source/configs/tokenizer.toml
+2d386dd8efe0ffd0da6df319226daf7bab8948b1bcb2abc7afb59c45d98d28b3  source/configs/train-smoke.toml
+44ba93246d406d5187c496ad2cf23d1533e6f581a8d6dc6f7fff03fca22b8323  source/configs/train-v1.toml
+0fac5689cead9021745847b2753294a9a4d95c002a61762f7f25254db2a26489  source/requirements.txt
+34a2a24d14c0e8b20c44dafab720075cbc23261a7fdbe71091dff12200e7fc24  source/scripts/benchmarks/evaluate.py
+a28fc24da1f8cbb6972b49dc0223361e16c7b093872d92df6bd52aabd00bd54a  source/scripts/benchmarks/evaluate_ajimee.py
+d328ff258e4a8fa83e51be099f5fa8fbf480f2d377579c475c355f9f01263bf6  source/scripts/benchmarks/export_cached_development.py
+de9866c56cd846073a3a3a82b1553ab737ccb911001d17de71839865057d0293  source/scripts/benchmarks/generate.py
+3b3b4192535a035bbf57e3f51f1fb16a0647b1a3fc060a179b7ae9fb1127cc41  source/scripts/benchmarks/generate_development.py
+64dbd0fcee7e294bd290bbf9f2057e81656ede4e0b328cdae710360da24716b3  source/scripts/benchmarks/hybrid.py
+3935e16a81c389c5a3dd74da6da38732ff32b668ea3c6d0484c52f017657706a  source/scripts/benchmarks/prepare_ajimee.py
+0e4d753adf6cf97cc19b975e4f35d3cea41a4f519782f6b8eeea3309260486df  source/scripts/corpus/build.py
+dd6639d55b9aecdbf46d20fec98ab78422ff53ad14ec5a4a39d6a43e824cd0d6  source/scripts/corpus/merge.py
+f8181f1b055661a1097d2bf80654ac59af4cf6a8b74afecb8f44b40e3d94d4f8  source/scripts/corpus/preprocess.py
+e3e12fd3373119539eb3ed23eaccbc1457cafa7a22ae444dd4b187c118d883ce  source/scripts/deployment/ane_output_probe.py
+9a0761dbd3ece9b730ed0e52010ef965a8415b4320188c31e1863ab0c874d2fd  source/scripts/deployment/compare_quantization.py
+2aa26fb2f2b59a630cc9cafb7eb684f237c44c780c1ec820badbbd07d8f2bf41  source/scripts/deployment/compute_plan.py
+7f321ab43b41acd4765045a93edc5c3ec874dcb741541a3f7460d13f8c27f50e  source/scripts/deployment/coreml.py
+28c67b5b30c3ad93a46703dad62236b2cd81ec5478ebf2cd9974e7b336001871  source/scripts/deployment/coreml_ane.py
+61b290658385ee40325263be24b1a1dffdc18fe5bc494dd02a1bf93bf301b986  source/scripts/deployment/coreml_conservative.py
+65db41ee70efb5409738ece23eb41750e695f1746e42b85314198d72710fa7a7  source/scripts/deployment/package_handoff.py
+1228adeb409e3cf095aa7890390307b9cd518c535fc23365f5a8ffc01b144448  source/scripts/deployment/package_ios.py
+0857bab4b1dcf81ec8b1ae5e680672cf512d5547aa3246c1e2a7ad7bfc0b79e5  source/scripts/deployment/summarize_keyboard_trace.py
+a798a4478ba907f0b4c88246c90d865063fa5c7c02f10456e7faceea7a7966ba  source/scripts/deployment/summarize_memory.py
+bbd7f36ed4682bcdcb5e4c5d87b42733106dbaebf6f949fcd60cd498dc059274  source/scripts/publishing/hub.py
+994c690a95f8d13d15a1ab1671185c8e2744537f48d667c176f9a3079b3630e9  source/scripts/publishing/prepare_hf.py
+555b275e74482c597a26b3cbc23ffb1648e636801bc5ddeccf7c25df176414d0  source/scripts/review/approve.py
+77578100f9aaae50cabac68a85996dcf02479179f6d17fbf30368ea7aa70ab5f  source/scripts/review/prepare.py
+d18a127f4a08b84ac03a398f3c96c935a0b6275fe0e7b1af2f3e3d4d92ec49b1  source/scripts/review/run.py
+702ba0084944c8db433f1d12f64e6933659965a9a125434cce80f916f63c6f47  source/scripts/review/run_multi_key.py
+46af106351c8835ba0e001295c4d2122c8c629d799c913feb8292d2e83100670  source/scripts/tokenizer/encode.py
+1001c67f565711d91c59335b342ab1a3e3fdb40f7f5642a8a49ce9ac46a4823d  source/scripts/tokenizer/train.py
+231f867355c0cd2806e0ee20378c9671443309f2716db9dbf1426fda6bbe9eb6  source/scripts/tools/check_corpus.py
+fd5a05f0c74e4d3bd95097a37be009422ec348c3fd3e36f1ca12740c02309760  source/scripts/tools/inspect_data.py
+09d81377f63d9c967e444ffdaaec83a7228de8109c4492a5175b46a8590387bc  source/scripts/tools/phrase_demo.py
+fce755bfd04595f95d6a021c5d2533599d3463542239da198d65e4c26a716e0a  source/scripts/training/check_data.py
+81729525388462eb93ede97f917f0f6eae1ef6518c35d169e19dc745f37ffa4d  source/scripts/training/evaluate_ime.py
+6f409f7761d543e65d5f4c8886c8f59e0929b4b299977a8439bd0d13623755f4  source/scripts/training/infer.py
+56b5781f98b3440d23d1516d830e83356db3a0b04c9aab134368c2ef9cefb91c  source/scripts/training/train.py
+db40a3f34315dcd409f0b22d8633c569f71f6807841fb2551a9ca423869ca149  source/scripts/training/train_wandb.py
+ff0f42adee5b707f35ea0ba1307e65d8a2f632a55c3281c8f446211032f0fbc1  source/src/vimeml/__init__.py
+03a0b8a9e43ef3ecf56189a86fe7e33d9633d4a766edf02583e911aef9bde581  source/src/vimeml/benchmarks/__init__.py
+4916651168f3f883161ebbbc24db5e1ae5b1bc4cd6abf8c7b83560ef960406cb  source/src/vimeml/benchmarks/ajimee.py
+d74e8f9e888c115c6a9ce27f5b08ea40efb332c087d94e372d4c33364b2dc2d4  source/src/vimeml/benchmarks/development.py
+867cd2f41b49043dd985e7f05e788f3f83a0ac47a4d0d9b2d6306dcd0c96b02a  source/src/vimeml/benchmarks/evaluate.py
+459a8c0d1bb4f8ffba57e92ee2c8bb3f5278c3e4a1a5bcc5dfda48a21bde3716  source/src/vimeml/benchmarks/evaluate_ajimee.py
+9a468c23ea3ebf02f1184b1742cbe11a025f2a0d7ae57e0ee2e54468de72a1d7  source/src/vimeml/benchmarks/generate.py
+71267f445ac92736f1dabf03c3e77193e9f2a1323f23fa1bc060aff4fbe4ea7e  source/src/vimeml/benchmarks/hybrid.py
+55cdd3f899dd7ad156f8987cf2ba2bbbc76a95dfaba33a319058b4d01e2f9947  source/src/vimeml/corpus/__init__.py
+895b6894c4b85e72c8655cf6b42e21d51b51736d649ec2adb6841e033f4ac1f9  source/src/vimeml/corpus/build.py
+86d655d2d715fcb2f73269c1ffbfc9e87d9080a1baaf389f88e30de6022fdc19  source/src/vimeml/data/__init__.py
+ba62919c8084d24cb3f41c451e970b5448e7b0f5407340c28cca0b220d387395  source/src/vimeml/data/annotations.py
+d7d63e1ea9a2f9da9fe92ff2773b792c7ca89911b2d7058a2709fcbbc443081e  source/src/vimeml/data/build.py
+ee88adbb74d5a03f591231484928becf0a62557c85702efddd214c9bc8a2354f  source/src/vimeml/data/clean.py
+d2e943aae1e2942076b1db8430067e048fcdc71d46761bb329f7fa460e843280  source/src/vimeml/data/corpus_export.py
+c196a78e5096e659eb8c536602e009c98de346bd82084c29f9384e526055f6f6  source/src/vimeml/data/corpus_parts.py
+43cb87845d17e52261675073b010269c53e4c79a8e5e16237d2627c9e07116e6  source/src/vimeml/data/merge_parts.py
+600dc56899dfa6ac2eda639cf79f6ad5e789642b99d6c68569262f593bfc3642  source/src/vimeml/data/merge_parts_fast.py
+433a93f18809aa1b11a13c8e63edce2a73864140fbaf4a4bcff0c7ab475be164  source/src/vimeml/data/preprocess_part.py
+837017b0282f62ed68c9a47c69da3ed12fe39b74e7f88bc3ae16113f55b679b6  source/src/vimeml/data/readers.py
+9834f4867e126f229e86a7fc5b7481f4c803c05d00a21966a78ec293095b0f8c  source/src/vimeml/data/segment.py
+9301bfc92623e4a8b2589e01d7ca98aac917152544b83562b23628bf4cfe414f  source/src/vimeml/deployment/__init__.py
+4f66f215e576ea1267971da0f49d995f49b09417920cbbf9e159e2d1c8083043  source/src/vimeml/deployment/bundle.py
+55d65451e04721c3c05a131b9c422f2e7c1835d6340cc8b76c4936243521dd6a  source/src/vimeml/deployment/cli.py
+8281456e241e32ed79f9223d63f5c476e02b97b263723be9ad46c0a75435ee52  source/src/vimeml/deployment/coreml.py
+e2474a97e4006c4e1fe14552b11874d08c52fa217b41c9d0ab0dbab7b0a79a67  source/src/vimeml/deployment/graph.py
+9e5d526dbd8e9be72f9d8782ac402623007226ee9e1ca14f8ce005a52a994c95  source/src/vimeml/deployment/validation.py
+6422f9e2042a114155c7caf2721c1f6bd8e5de4d153009703af1281f2a1c76c3  source/src/vimeml/review/__init__.py
+b81ca9f1dcbe1b8e36f189723e81f2a3d44e9ef51f5a1c6f9ad0ce96a9b00536  source/src/vimeml/review/approve.py
+e21ebd1e94159d17c26234191f2f1d1ee95e542c54a9762fe0a5717a73812e4a  source/src/vimeml/review/multi_key.py
+968209bb52d010c75066d2296a93bf27b30d264e614927d5e72bbe7c0a0b9323  source/src/vimeml/review/prepare.py
+8507f1850e5b8af416108ff20b4283637d03fa267add33548674dedd7445b036  source/src/vimeml/review/run.py
+77c08158b861bafd5e902c886a0878a08a4afea25a5277dd33be60ae219ff624  source/src/vimeml/tokenizer/__init__.py
+92bdbdda6fa80f4dbb975f8882c021d8c7c0f9017f669c89dfae31dc2eecaa69  source/src/vimeml/tokenizer/encode.py
+c194cb241d40b49669515f68f44278b55bcfbae210756f220b375cdee22ab009  source/src/vimeml/tokenizer/encode_parallel.py
+184a02f06f477a6037368ad2c2f43a32299fef83d009d851083e452ce91cf7d9  source/src/vimeml/tokenizer/monitor.py
+f42ef20a1e2b4aaa2bc6c04a2357b3c0375bb20a6055b49225318d08760ab01a  source/src/vimeml/tokenizer/native_train.py
+53b285db5bfff0127281c8a35acc1beba8f5dfee1c6ae05e46923e91f9befd91  source/src/vimeml/tokenizer/store.py
+81c218f4185345e443a5bfe1f80da024ad1260b6e3653ceb52938824cd60d752  source/src/vimeml/tokenizer/train.py
+0a0511ce3c52639d6a3b0a1f030162d003bb930a2a7c795cc5f15fa68bdcb6b7  source/src/vimeml/tools/__init__.py
+770f251e80a34626b3b36a193c9fd9ebaa3f4c78885382cef6cb060db7b3d762  source/src/vimeml/tools/check_corpus.py
+5f180d035a6e73d3931959d1ae9a2eece8442390a59e522d0e908285205eb6c1  source/src/vimeml/tools/inspect_data.py
+c298c93b65c6c81782d5a4edea52d72e58d78a1db6c28412e15e338d97f409b5  source/src/vimeml/tools/phrase_demo.html
+07023e63ece7f683038d0a823bac49722619940a0b1a387a989696effaaea16c  source/src/vimeml/tools/phrase_demo.py
+30be98c6d0dc0b68bb12915c37cb84d97cc3c47bc1c0734e45435ac033a02180  source/src/vimeml/training/__init__.py
+c2530ba10df54c190211f59d1756f36de4792a4344dc78d14b88305e16a2f4be  source/src/vimeml/training/check_data.py
+1627a4db9bb3c179977dc25e08b9443e84c0a966ff8416c3e5872c456b5b3ae3  source/src/vimeml/training/data.py
+9a36b75893f58f61228ad7a6bbb2a34b52aaf312ca6675383580964500d639cd  source/src/vimeml/training/evaluate_ime.py
+7a093392ed15377c54bd1842cfbf32c6ab0257f3e956a81e84e41ad4624a9e9c  source/src/vimeml/training/infer.py
+dde6a0f8d7bd2d3a76e48fb503c22296da4c1156f976125bdb773d902bc7e07e  source/src/vimeml/training/model.py
+dd34aa271f9a62af55aa877fc4090a1db0001cdce656faadd37304524eb59c5b  source/src/vimeml/training/train.py
+169add32b203da3142840133dc27456a63a576baecb5f9a9273d923464a1fcff  source/templates/huggingface/README.md
+8ed8d241d349a4e88840dd841da9db3191ec9f0db84c8f8165a951fa84720621  source/templates/huggingface/infer.py
+80dd64ae4dac2eddbe864a2d2c4f429cf8b0bb6317b8eaf962fd378e11449b90  source/templates/huggingface/verify_release.py
+80dd64ae4dac2eddbe864a2d2c4f429cf8b0bb6317b8eaf962fd378e11449b90  verify_release.py

coreml/ios18-int8-block32/manifest.json

@@ -0,0 +1,491 @@
+{
+  "format": "vimeml_coreml_v1",
+  "status": "complete",
+  "kind": "linear8_fp32_compute",
+  "minimum_ios": 18,
+  "bundle": {
+    "checkpoint_sha256": "93b6139aa05031df02038efcaf26788bc9916712919f3d2a4c4f398615cef75b",
+    "tokenizer_sha256": "d9f1ba1e456ce72dd9c06a62b12e804cf14090c30179e6078cd51d03063b2982",
+    "bundle_manifest_sha256": "4284cdc5491513cc817633081677b363b14f59c0b4ffb78cdf7fcfb221f71b3c",
+    "model": {
+      "vocab_size": 16384,
+      "context_length": 128,
+      "d_model": 256,
+      "n_heads": 4,
+      "n_layers": 4,
+      "d_ff": 1024,
+      "dropout": 0.0
+    },
+    "precision": "fp32",
+    "device": "cpu",
+    "environment": {
+      "python": "3.11.17",
+      "platform": "macOS-27.2-arm64-arm-64bit",
+      "machine": "arm64",
+      "packages": {
+        "torch": "2.7.1",
+        "numpy": "2.3.5",
+        "sentencepiece": "0.2.1",
+        "coremltools": "9.0"
+      }
+    }
+  },
+  "interface": {
+    "input_ids": "int32 [1,T], 1<=T<=128; right PAD only",
+    "logits": "float32 [1,T,16384]; FP32 computation; no softmax"
+  },
+  "source_manifest_sha256": "f5591ad1512b931a3a20765eb2218771f68e49699a3d23601e0cc0566d9c6d9d",
+  "fp16_alignment_sha256": "6bfead3f03f53f3e43780427515b3c5f12e6713845fd9f3a459df95aa23f800a",
+  "conversion_script_sha256": "4674f9d9e298562c8f517636674bf973532b8fc6ddc802aae8ca2aa939642912",
+  "precision_policy": {
+    "source_recipe": "fp16_weights_fp32_compute_preserve_positions_qkv_v1",
+    "computation": "FP32; token embedding gather is FP16 then cast to FP32"
+  },
+  "compression": {
+    "method": "linear",
+    "bits": 8,
+    "group_size": 16,
+    "block_size": 32,
+    "weight_threshold": 2048,
+    "selection": {
+      "policy": "linear_weights_and_named_embedding_tables_v1",
+      "selected_constants": [
+        {
+          "name": "model_position_embedding_weight",
+          "shape": [
+            128,
+            256
+          ],
+          "dtype": "float32",
+          "elements": 32768,
+          "consumers": [
+            {
+              "name": "op_143",
+              "type": "gather",
+              "inputs": {
+                "x": "model_position_embedding_weight"
+              }
+            }
+          ]
+        },
+        {
+          "name": "model_blocks_0_attention_qkv_weight",
+          "shape": [
+            768,
+            256
+          ],
+          "dtype": "float32",
+          "elements": 196608,
+          "consumers": [
+            {
+              "name": "linear_0",
+              "type": "linear",
+              "inputs": {
+                "weight": "model_blocks_0_attention_qkv_weight"
+              }
+            }
+          ]
+        },
+        {
+          "name": "model_blocks_1_attention_qkv_weight",
+          "shape": [
+            768,
+            256
+          ],
+          "dtype": "float32",
+          "elements": 196608,
+          "consumers": [
+            {
+              "name": "linear_4",
+              "type": "linear",
+              "inputs": {
+                "weight": "model_blocks_1_attention_qkv_weight"
+              }
+            }
+          ]
+        },
+        {
+          "name": "model_blocks_2_attention_qkv_weight",
+          "shape": [
+            768,
+            256
+          ],
+          "dtype": "float32",
+          "elements": 196608,
+          "consumers": [
+            {
+              "name": "linear_8",
+              "type": "linear",
+              "inputs": {
+                "weight": "model_blocks_2_attention_qkv_weight"
+              }
+            }
+          ]
+        },
+        {
+          "name": "model_blocks_3_attention_qkv_weight",
+          "shape": [
+            768,
+            256
+          ],
+          "dtype": "float32",
+          "elements": 196608,
+          "consumers": [
+            {
+              "name": "linear_12",
+              "type": "linear",
+              "inputs": {
+                "weight": "model_blocks_3_attention_qkv_weight"
+              }
+            }
+          ]
+        },
+        {
+          "name": "model_token_embedding_weight_storage_fp16",
+          "shape": [
+            16384,
+            256
+          ],
+          "dtype": "float16",
+          "elements": 4194304,
+          "consumers": [
+            {
+              "name": "op_139_storage_fp16",
+              "type": "gather",
+              "inputs": {
+                "x": "model_token_embedding_weight_storage_fp16"
+              }
+            },
+            {
+              "name": "linear_16_compute_fp32",
+              "type": "linear",
+              "inputs": {
+                "weight": "model_token_embedding_weight_storage_fp16",
+                "bias": "linear_16_bias_0_storage_fp16"
+              }
+            }
+          ]
+        },
+        {
+          "name": "model_blocks_0_attention_projection_weight_storage_fp16",
+          "shape": [
+            256,
+            256
+          ],
+          "dtype": "float16",
+          "elements": 65536,
+          "consumers": [
+            {
+              "name": "linear_1_compute_fp32",
+              "type": "linear",
+              "inputs": {
+                "weight": "model_blocks_0_attention_projection_weight_storage_fp16"
+              }
+            }
+          ]
+        },
+        {
+          "name": "model_blocks_0_ffn_0_weight_storage_fp16",
+          "shape": [
+            1024,
+            256
+          ],
+          "dtype": "float16",
+          "elements": 262144,
+          "consumers": [
+            {
+              "name": "linear_2_compute_fp32",
+              "type": "linear",
+              "inputs": {
+                "weight": "model_blocks_0_ffn_0_weight_storage_fp16"
+              }
+            }
+          ]
+        },
+        {
+          "name": "model_blocks_0_ffn_2_weight_storage_fp16",
+          "shape": [
+            256,
+            1024
+          ],
+          "dtype": "float16",
+          "elements": 262144,
+          "consumers": [
+            {
+              "name": "linear_3_compute_fp32",
+              "type": "linear",
+              "inputs": {
+                "weight": "model_blocks_0_ffn_2_weight_storage_fp16"
+              }
+            }
+          ]
+        },
+        {
+          "name": "model_blocks_1_attention_projection_weight_storage_fp16",
+          "shape": [
+            256,
+            256
+          ],
+          "dtype": "float16",
+          "elements": 65536,
+          "consumers": [
+            {
+              "name": "linear_5_compute_fp32",
+              "type": "linear",
+              "inputs": {
+                "weight": "model_blocks_1_attention_projection_weight_storage_fp16"
+              }
+            }
+          ]
+        },
+        {
+          "name": "model_blocks_1_ffn_0_weight_storage_fp16",
+          "shape": [
+            1024,
+            256
+          ],
+          "dtype": "float16",
+          "elements": 262144,
+          "consumers": [
+            {
+              "name": "linear_6_compute_fp32",
+              "type": "linear",
+              "inputs": {
+                "weight": "model_blocks_1_ffn_0_weight_storage_fp16"
+              }
+            }
+          ]
+        },
+        {
+          "name": "model_blocks_1_ffn_2_weight_storage_fp16",
+          "shape": [
+            256,
+            1024
+          ],
+          "dtype": "float16",
+          "elements": 262144,
+          "consumers": [
+            {
+              "name": "linear_7_compute_fp32",
+              "type": "linear",
+              "inputs": {
+                "weight": "model_blocks_1_ffn_2_weight_storage_fp16"
+              }
+            }
+          ]
+        },
+        {
+          "name": "model_blocks_2_attention_projection_weight_storage_fp16",
+          "shape": [
+            256,
+            256
+          ],
+          "dtype": "float16",
+          "elements": 65536,
+          "consumers": [
+            {
+              "name": "linear_9_compute_fp32",
+              "type": "linear",
+              "inputs": {
+                "weight": "model_blocks_2_attention_projection_weight_storage_fp16"
+              }
+            }
+          ]
+        },
+        {
+          "name": "model_blocks_2_ffn_0_weight_storage_fp16",
+          "shape": [
+            1024,
+            256
+          ],
+          "dtype": "float16",
+          "elements": 262144,
+          "consumers": [
+            {
+              "name": "linear_10_compute_fp32",
+              "type": "linear",
+              "inputs": {
+                "weight": "model_blocks_2_ffn_0_weight_storage_fp16"
+              }
+            }
+          ]
+        },
+        {
+          "name": "model_blocks_2_ffn_2_weight_storage_fp16",
+          "shape": [
+            256,
+            1024
+          ],
+          "dtype": "float16",
+          "elements": 262144,
+          "consumers": [
+            {
+              "name": "linear_11_compute_fp32",
+              "type": "linear",
+              "inputs": {
+                "weight": "model_blocks_2_ffn_2_weight_storage_fp16"
+              }
+            }
+          ]
+        },
+        {
+          "name": "model_blocks_3_attention_projection_weight_storage_fp16",
+          "shape": [
+            256,
+            256
+          ],
+          "dtype": "float16",
+          "elements": 65536,
+          "consumers": [
+            {
+              "name": "linear_13_compute_fp32",
+              "type": "linear",
+              "inputs": {
+                "weight": "model_blocks_3_attention_projection_weight_storage_fp16"
+              }
+            }
+          ]
+        },
+        {
+          "name": "model_blocks_3_ffn_0_weight_storage_fp16",
+          "shape": [
+            1024,
+            256
+          ],
+          "dtype": "float16",
+          "elements": 262144,
+          "consumers": [
+            {
+              "name": "linear_14_compute_fp32",
+              "type": "linear",
+              "inputs": {
+                "weight": "model_blocks_3_ffn_0_weight_storage_fp16"
+              }
+            }
+          ]
+        },
+        {
+          "name": "model_blocks_3_ffn_2_weight_storage_fp16",
+          "shape": [
+            256,
+            1024
+          ],
+          "dtype": "float16",
+          "elements": 262144,
+          "consumers": [
+            {
+              "name": "linear_15_compute_fp32",
+              "type": "linear",
+              "inputs": {
+                "weight": "model_blocks_3_ffn_2_weight_storage_fp16"
+              }
+            }
+          ]
+        }
+      ],
+      "excluded_constants": [
+        {
+          "name": "causal_mask",
+          "shape": [
+            128,
+            128
+          ],
+          "dtype": "float32",
+          "elements": 16384,
+          "consumers": [
+            {
+              "name": "op_182",
+              "type": "slice_by_index",
+              "inputs": {
+                "x": "causal_mask"
+              }
+            }
+          ],
+          "reason": "Not a linear weight or named embedding parameter",
+          "nonfinite_elements": 8128
+        },
+        {
+          "name": "linear_16_bias_0_storage_fp16",
+          "shape": [
+            16384
+          ],
+          "dtype": "float16",
+          "elements": 16384,
+          "consumers": [
+            {
+              "name": "linear_16_compute_fp32",
+              "type": "linear",
+              "inputs": {
+                "weight": "model_token_embedding_weight_storage_fp16",
+                "bias": "linear_16_bias_0_storage_fp16"
+              }
+            }
+          ],
+          "reason": "Not a linear weight or named embedding parameter",
+          "nonfinite_elements": 0
+        }
+      ],
+      "selected_elements": 7372800
+    },
+    "scope": "Eligible learned matrices, including FP32 QKV; structural constants/masks unchanged; no activation quantization or retraining."
+  },
+  "environment": {
+    "python": "3.11.17",
+    "platform": "macOS-27.2-arm64-arm-64bit",
+    "machine": "arm64",
+    "packages": {
+      "torch": "2.7.1",
+      "numpy": "2.3.5",
+      "sentencepiece": "0.2.1",
+      "coremltools": "9.0"
+    }
+  },
+  "spec": {
+    "specification_version": 9,
+    "operations": {
+      "const": 153,
+      "constexpr_blockwise_shift_scale": 18,
+      "shape": 1,
+      "gather": 3,
+      "range_1d": 1,
+      "greater_equal": 6,
+      "add": 19,
+      "select": 6,
+      "cast": 1,
+      "layer_norm": 9,
+      "linear": 17,
+      "reshape": 8,
+      "transpose": 8,
+      "split": 4,
+      "squeeze": 12,
+      "matmul": 8,
+      "mul": 4,
+      "concat": 2,
+      "slice_by_index": 2,
+      "softmax": 4,
+      "gelu": 4
+    },
+    "inputs": [
+      "input_ids"
+    ],
+    "outputs": [
+      "logits"
+    ]
+  },
+  "package_files": {
+    "Data/com.apple.CoreML/model.mlmodel": {
+      "bytes": 46400,
+      "sha256": "22e80193abc7672bbcf2a3fc214a186ae6e64d76c1e1b42ead38fef5a003cd37"
+    },
+    "Data/com.apple.CoreML/weights/weight.bin": {
+      "bytes": 8030784,
+      "sha256": "d3a063455371e624cf208fb6ed76fc9ac6addd8c2f87eaebf4653649f85b8109"
+    },
+    "Manifest.json": {
+      "bytes": 617,
+      "sha256": "31b13734848024036806cbd5bacf405eddba6c4451e5659b25f714ffe1766ee5"
+    }
+  },
+  "package_bytes": 8077801,
+  "note": "Logical file bytes, not compiled size, IPA size or resident memory. Inspect weight.bin and const/constexpr ops for tied-weight duplication."
+}

coreml/ios18-int8-block32/model.mlpackage/Data/com.apple.CoreML/model.mlmodel

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:22e80193abc7672bbcf2a3fc214a186ae6e64d76c1e1b42ead38fef5a003cd37
+size 46400

coreml/ios18-int8-block32/model.mlpackage/Data/com.apple.CoreML/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d3a063455371e624cf208fb6ed76fc9ac6addd8c2f87eaebf4653649f85b8109
+size 8030784

coreml/ios18-int8-block32/model.mlpackage/Manifest.json

@@ -0,0 +1,18 @@
+{
+    "fileFormatVersion": "1.0.0",
+    "itemInfoEntries": {
+        "B5BC0804-8142-444E-8A75-43E553098EA8": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Weights",
+            "name": "weights",
+            "path": "com.apple.CoreML/weights"
+        },
+        "C4D1258E-24AB-4839-B8C6-61581D74BA25": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Specification",
+            "name": "model.mlmodel",
+            "path": "com.apple.CoreML/model.mlmodel"
+        }
+    },
+    "rootModelIdentifier": "C4D1258E-24AB-4839-B8C6-61581D74BA25"
+}

evaluation/summary.json

@@ -0,0 +1,239 @@
+{
+  "format": "vimeml_public_evaluation_v1",
+  "bundle_manifest_sha256": "4284cdc5491513cc817633081677b363b14f59c0b4ffb78cdf7fcfb221f71b3c",
+  "coreml_manifest_sha256": "a9d516233fa57aaadd09477e2e909760972ff49b5577919f8a441c489f5c2245",
+  "training": {
+    "prediction_targets": 573295237,
+    "data_passes": 1.0,
+    "full_validation": {
+      "loss": 4.541570895102848,
+      "prediction_pairs": 5890326,
+      "perplexity": 93.83809420004854
+    },
+    "test_used_for_selection": false
+  },
+  "ranking": {
+    "dev": {
+      "original_order": {
+        "cases": 137,
+        "top1_correct": 111,
+        "top1_accuracy": 0.8102189781021898,
+        "top5_correct": 130,
+        "top5_accuracy": 0.948905109489051,
+        "candidate_pool_covered": 136,
+        "candidate_pool_recall": 0.9927007299270073,
+        "covered_top1_accuracy": 0.8161764705882353,
+        "mean_min_cer": 0.04746998743349109,
+        "mrr": 0.8745925911071898,
+        "corrected_vs_azookey": 0,
+        "regressed_vs_azookey": 0,
+        "fallback_cases": 0
+      },
+      "fp32_lm": {
+        "cases": 137,
+        "top1_correct": 122,
+        "top1_accuracy": 0.8905109489051095,
+        "top5_correct": 134,
+        "top5_accuracy": 0.9781021897810219,
+        "candidate_pool_covered": 136,
+        "candidate_pool_recall": 0.9927007299270073,
+        "covered_top1_accuracy": 0.8970588235294118,
+        "mean_min_cer": 0.03045798483754688,
+        "mrr": 0.935349322210636,
+        "corrected_vs_azookey": 15,
+        "regressed_vs_azookey": 4,
+        "fallback_cases": 0
+      },
+      "int8_lm": {
+        "cases": 137,
+        "top1_correct": 122,
+        "top1_accuracy": 0.8905109489051095,
+        "top5_correct": 134,
+        "top5_accuracy": 0.9781021897810219,
+        "candidate_pool_covered": 136,
+        "candidate_pool_recall": 0.9927007299270073,
+        "covered_top1_accuracy": 0.8970588235294118,
+        "mean_min_cer": 0.03045798483754688,
+        "mrr": 0.935349322210636,
+        "corrected_vs_azookey": 15,
+        "regressed_vs_azookey": 4,
+        "fallback_cases": 0
+      },
+      "windows_fp32_comparison": {
+        "order_changed_cases": 119,
+        "top1_changed_cases": 0,
+        "max_score_sum_abs": 1.9853296279907227,
+        "mean_score_sum_abs": 0.20199488408841776
+      },
+      "original_report_comparison_baseline": "conservative Core ML",
+      "original_report_comparison": {
+        "order_changed_cases": 118,
+        "top1_changed_cases": 0,
+        "max_score_sum_abs": 1.994759202003479,
+        "mean_score_sum_abs": 0.2013680215637894
+      },
+      "benchmark_source_sha256": "a945f2117570530ef73b1dd4ef53b9dc69b07fa69ac98a6d67c9268b540eafa8",
+      "candidate_export_sha256": "ab192509dce1cb4f4b20273e623f7bfd4237b1c3691950438e6921b8a510ae21"
+    },
+    "ajimee": {
+      "original_order": {
+        "cases": 200,
+        "top1_correct": 87,
+        "top1_accuracy": 0.435,
+        "top5_correct": 143,
+        "top5_accuracy": 0.715,
+        "candidate_pool_covered": 162,
+        "candidate_pool_recall": 0.81,
+        "covered_top1_accuracy": 0.5370370370370371,
+        "mean_min_cer": 0.0821630611981915,
+        "mrr": 0.554232017982018,
+        "corrected_vs_azookey": 0,
+        "regressed_vs_azookey": 0,
+        "fallback_cases": 0
+      },
+      "fp32_lm": {
+        "cases": 200,
+        "top1_correct": 124,
+        "top1_accuracy": 0.62,
+        "top5_correct": 151,
+        "top5_accuracy": 0.755,
+        "candidate_pool_covered": 162,
+        "candidate_pool_recall": 0.81,
+        "covered_top1_accuracy": 0.7654320987654321,
+        "mean_min_cer": 0.05657040945427151,
+        "mrr": 0.684061974789916,
+        "corrected_vs_azookey": 48,
+        "regressed_vs_azookey": 11,
+        "fallback_cases": 0
+      },
+      "int8_lm": {
+        "cases": 200,
+        "top1_correct": 124,
+        "top1_accuracy": 0.62,
+        "top5_correct": 151,
+        "top5_accuracy": 0.755,
+        "candidate_pool_covered": 162,
+        "candidate_pool_recall": 0.81,
+        "covered_top1_accuracy": 0.7654320987654321,
+        "mean_min_cer": 0.05657040945427151,
+        "mrr": 0.6841512605042016,
+        "corrected_vs_azookey": 48,
+        "regressed_vs_azookey": 11,
+        "fallback_cases": 0
+      },
+      "windows_fp32_comparison": {
+        "order_changed_cases": 163,
+        "top1_changed_cases": 2,
+        "max_score_sum_abs": 2.6928324103355408,
+        "mean_score_sum_abs": 0.33380034679328247
+      },
+      "original_report_comparison_baseline": "Windows FP32",
+      "original_report_comparison": {
+        "order_changed_cases": 163,
+        "top1_changed_cases": 2,
+        "max_score_sum_abs": 2.6928324103355408,
+        "mean_score_sum_abs": 0.33380034679328247
+      },
+      "benchmark_source_sha256": "e9eb668fd6aa14b1e26436f429b5550108af0a1dfd443b8cea0bcb3ab3028fca",
+      "candidate_export_sha256": "6b532af2e7b1536ded98e9d75cbba4fd7338b4204987fc4dafb16df7edfdf223"
+    }
+  },
+  "alignment": {
+    "strict_passed": false,
+    "max_logit_abs": 3.6601905822753906,
+    "invariants": {
+      "right_pad": {
+        "max_abs": 2.8133392333984375e-05,
+        "mean_abs": 2.5544544911072975e-06,
+        "rmse": 3.420821628891477e-06,
+        "outside_tolerance": 0,
+        "values": 114688
+      },
+      "causal": {
+        "max_abs": 0.0,
+        "mean_abs": 0.0,
+        "rmse": 0.0,
+        "outside_tolerance": 0,
+        "values": 262144
+      }
+    }
+  },
+  "distribution": {
+    "original_report": {
+      "model": "artifacts/deployment/tiny-ja-v1-conservative-int8-b32-v1",
+      "compute_units": "CPU_ONLY",
+      "benchmark_text": {
+        "kl_coreml": {
+          "mean": 0.005891128288349137,
+          "max": 0.025890197604894638
+        },
+        "kl_simulated": {
+          "mean": 0.005977873008232564,
+          "max": 0.028363749384880066
+        },
+        "nll_fp32": {
+          "mean": 5.566977231502533,
+          "max": 8.531479835510254
+        },
+        "nll_coreml": {
+          "mean": 5.572893209457398,
+          "max": 8.633211135864258
+        },
+        "top1_agreement": {
+          "mean": 0.943656535744667,
+          "max": 1.0
+        },
+        "max_logprob_diff": {
+          "mean": 0.9077084493637085,
+          "max": 3.5512266159057617
+        }
+      },
+      "random_tokens": {
+        "kl_coreml": {
+          "mean": 0.015501508116722107,
+          "max": 0.01807662844657898
+        },
+        "kl_simulated": {
+          "mean": 0.015804098173975944,
+          "max": 0.01901794597506523
+        },
+        "nll_fp32": {
+          "mean": 11.800455474853516,
+          "max": 12.100545883178711
+        },
+        "nll_coreml": {
+          "mean": 11.795885848999024,
+          "max": 12.123295783996582
+        },
+        "top1_agreement": {
+          "mean": 0.9140625,
+          "max": 0.9296875
+        },
+        "max_logprob_diff": {
+          "mean": 2.206966495513916,
+          "max": 2.564694404602051
+        }
+      }
+    },
+    "aggregation": "Mean of per-example position means; not corpus token-weighted.",
+    "content_token_limit": 127,
+    "truncation_count_recorded": false,
+    "original_report_has_full_identity_metadata": false
+  },
+  "source_reports_sha256": {
+    "outputs/deployment/conservative-int8-b32-dev-v1/metrics.json": "839e220c3024ec9798c921fb39d8862c5f874889c9dd9c2066cb339894658329",
+    "outputs/deployment/conservative-int8-b32-dev-v1/scores.jsonl": "f00a6b56b96e6be1db3ba7c58abe53bc91e7614c2f7f7959c27b0a8cce8db936",
+    "outputs/ime-eval/tiny-ja-v1-dev-v2-scores/manifest.json": "16bbd3b5b921e54896cfdca880d77c78fedd76b2aa8956fbaaa4c45bd4a4030e",
+    "outputs/ime-eval/tiny-ja-v1-dev-v2-scores/scores.jsonl": "2710b88428b5f73c6aeac5d45d442fa0d9bca1e038db2928c6059be993f3cfcb",
+    "outputs/deployment/conservative-dev-v3/metrics.json": "387d80e1b52a2c8dbcbb2196471ae548ae869b1e3f259dc1ee972c592e6b4863",
+    "outputs/deployment/conservative-dev-v3/scores.jsonl": "bb4fa9f52a074144a3cb432777ec6c94ad07637cdecd5b2126ad5d64a4e6a1d2",
+    "outputs/deployment/conservative-int8-b32-ajimee-v1/metrics.json": "3b59e396173328c78e8164f809da24764474685a2bd35765457fac5df644aac5",
+    "outputs/deployment/conservative-int8-b32-ajimee-v1/scores.jsonl": "9daf557818ab0aef12fb74e68b8cd4ae502806650d3d7e0cd23eeb2e1499e89e",
+    "outputs/ime-eval/tiny-ja-v1-ajimee/metrics.json": "219e15ec8ee54b09b455bd14ce60ab99bff239510ab9df344b69d2afc4b21196",
+    "outputs/ime-eval/tiny-ja-v1-ajimee/scores.jsonl": "247b29b59285551acaf73fabf9bee79c9617360623329e6bb60deb2d33ee7ddf",
+    "outputs/deployment/conservative-int8-b32-alignment-v1/alignment.json": "6773fafbd84e27dc1c349234fe601898e9f2af56ec5e2dcc6a32b12fbe14d018",
+    "outputs/deployment/conservative-int8-b32-distribution-v1/report.json": "6970eafdbb0d8e8b075d5b38b241cfa8fa46777961106ac32714755950722852",
+    "artifacts/models/tiny-ja-v1/summary.json": "4d9417b3bb8702b025b1ce4962d38ea36ad6d52928d19267dccdcaed3bf6c022"
+  },
+  "scope": "Offline fixed pools and saved Mac observations; not app-wide accuracy or a Windows Core ML runtime rerun."
+}

infer.py

@@ -0,0 +1,36 @@
+"""Read-only FP32 inference using the included, fingerprint-checked VimeML source."""
+import argparse
+import json
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--prompt', default='今日は雨が降っているので、')
+    parser.add_argument('--candidate', action='append', help='Rerank these strings after --prompt; repeat for each candidate.')
+    parser.add_argument('--max-new-tokens', type=int, default=8)
+    parser.add_argument('--threads', type=int, default=4)
+    args = parser.parse_args()
+    if args.threads < 1 or args.max_new_tokens < 1:
+        parser.error('threads and max-new-tokens must be positive.')
+    # The wrapper has no model math or scoring changes. Original code lives in source/.
+    sys.path.insert(0, str(ROOT / 'source/src'))
+    import torch
+    from vimeml.deployment.bundle import BundleLM
+    from vimeml.training.evaluate_ime import score_candidates
+    torch.set_num_threads(args.threads)
+    lm = BundleLM(ROOT / 'inference')
+    if args.candidate:
+        result = score_candidates(lm, args.prompt, args.candidate)
+        result['ranked_candidates'] = sorted(result['candidates'],
+            key=lambda item: item['log_probability_sum'], reverse=True)
+    else:
+        result = lm.generate(args.prompt, max_new_tokens=args.max_new_tokens, temperature=0)
+    print(json.dumps(result, ensure_ascii=False, indent=2))
+
+
+if __name__ == '__main__':
+    main()

inference/config.json

@@ -0,0 +1,9 @@
+{
+  "vocab_size": 16384,
+  "context_length": 128,
+  "d_model": 256,
+  "n_heads": 4,
+  "n_layers": 4,
+  "d_ff": 1024,
+  "dropout": 0.0
+}

inference/manifest.json

@@ -0,0 +1,108 @@
+{
+  "format": "vimeml_inference_bundle_v1",
+  "status": "complete",
+  "parameter_count": 7386624,
+  "precision": "fp32",
+  "special_ids": {
+    "pad": 0,
+    "unk": 1,
+    "bos": 2,
+    "eos": 3
+  },
+  "tied_lm_head": true,
+  "checkpoint_sha256": "93b6139aa05031df02038efcaf26788bc9916712919f3d2a4c4f398615cef75b",
+  "checkpoint_step": 196853,
+  "training_signatures": {
+    "tokens": "c7ebf3672d46cfc75d12013c34f35fb7ac9eac613f29f3454be0cb0e991854dc",
+    "windows": "4cdc90177a3fac335ad39434156f54ad20aa143190700950a122d914fd05d617",
+    "training_code": {
+      "train.py": "dd34aa271f9a62af55aa877fc4090a1db0001cdce656faadd37304524eb59c5b",
+      "model.py": "dde6a0f8d7bd2d3a76e48fb503c22296da4c1156f976125bdb773d902bc7e07e",
+      "data.py": "1627a4db9bb3c179977dc25e08b9443e84c0a966ff8416c3e5872c456b5b3ae3"
+    }
+  },
+  "source_fingerprints": {
+    "src/vimeml/training/model.py": {
+      "sha256": "dde6a0f8d7bd2d3a76e48fb503c22296da4c1156f976125bdb773d902bc7e07e",
+      "lf_sha256": "dde6a0f8d7bd2d3a76e48fb503c22296da4c1156f976125bdb773d902bc7e07e"
+    },
+    "src/vimeml/training/data.py": {
+      "sha256": "1627a4db9bb3c179977dc25e08b9443e84c0a966ff8416c3e5872c456b5b3ae3",
+      "lf_sha256": "1627a4db9bb3c179977dc25e08b9443e84c0a966ff8416c3e5872c456b5b3ae3"
+    },
+    "src/vimeml/training/train.py": {
+      "sha256": "dd34aa271f9a62af55aa877fc4090a1db0001cdce656faadd37304524eb59c5b",
+      "lf_sha256": "dd34aa271f9a62af55aa877fc4090a1db0001cdce656faadd37304524eb59c5b"
+    },
+    "src/vimeml/training/infer.py": {
+      "sha256": "7a093392ed15377c54bd1842cfbf32c6ab0257f3e956a81e84e41ad4624a9e9c",
+      "lf_sha256": "7a093392ed15377c54bd1842cfbf32c6ab0257f3e956a81e84e41ad4624a9e9c"
+    },
+    "src/vimeml/training/evaluate_ime.py": {
+      "sha256": "9a36b75893f58f61228ad7a6bbb2a34b52aaf312ca6675383580964500d639cd",
+      "lf_sha256": "9a36b75893f58f61228ad7a6bbb2a34b52aaf312ca6675383580964500d639cd"
+    },
+    "src/vimeml/benchmarks/evaluate_ajimee.py": {
+      "sha256": "459a8c0d1bb4f8ffba57e92ee2c8bb3f5278c3e4a1a5bcc5dfda48a21bde3716",
+      "lf_sha256": "459a8c0d1bb4f8ffba57e92ee2c8bb3f5278c3e4a1a5bcc5dfda48a21bde3716"
+    },
+    "src/vimeml/tools/phrase_demo.py": {
+      "sha256": "07023e63ece7f683038d0a823bac49722619940a0b1a387a989696effaaea16c",
+      "lf_sha256": "07023e63ece7f683038d0a823bac49722619940a0b1a387a989696effaaea16c"
+    },
+    "src/vimeml/deployment/__init__.py": {
+      "sha256": "9301bfc92623e4a8b2589e01d7ca98aac917152544b83562b23628bf4cfe414f",
+      "lf_sha256": "9301bfc92623e4a8b2589e01d7ca98aac917152544b83562b23628bf4cfe414f"
+    },
+    "src/vimeml/deployment/bundle.py": {
+      "sha256": "4f66f215e576ea1267971da0f49d995f49b09417920cbbf9e159e2d1c8083043",
+      "lf_sha256": "4f66f215e576ea1267971da0f49d995f49b09417920cbbf9e159e2d1c8083043"
+    },
+    "src/vimeml/deployment/cli.py": {
+      "sha256": "55d65451e04721c3c05a131b9c422f2e7c1835d6340cc8b76c4936243521dd6a",
+      "lf_sha256": "55d65451e04721c3c05a131b9c422f2e7c1835d6340cc8b76c4936243521dd6a"
+    },
+    "src/vimeml/deployment/coreml.py": {
+      "sha256": "8281456e241e32ed79f9223d63f5c476e02b97b263723be9ad46c0a75435ee52",
+      "lf_sha256": "8281456e241e32ed79f9223d63f5c476e02b97b263723be9ad46c0a75435ee52"
+    },
+    "src/vimeml/deployment/graph.py": {
+      "sha256": "e2474a97e4006c4e1fe14552b11874d08c52fa217b41c9d0ab0dbab7b0a79a67",
+      "lf_sha256": "e2474a97e4006c4e1fe14552b11874d08c52fa217b41c9d0ab0dbab7b0a79a67"
+    },
+    "src/vimeml/deployment/validation.py": {
+      "sha256": "9e5d526dbd8e9be72f9d8782ac402623007226ee9e1ca14f8ce005a52a994c95",
+      "lf_sha256": "9e5d526dbd8e9be72f9d8782ac402623007226ee9e1ca14f8ce005a52a994c95"
+    }
+  },
+  "environment": {
+    "python": "3.13.5",
+    "platform": "Windows-11-10.0.26300-SP0",
+    "machine": "AMD64",
+    "packages": {
+      "torch": "2.10.0+cu128",
+      "numpy": "2.5.3",
+      "sentencepiece": "0.2.1",
+      "coremltools": null
+    }
+  },
+  "files": {
+    "config.json": {
+      "bytes": 145,
+      "sha256": "8950524f2f3a0373f3a0ebbcde79a8feaa54949ed926830c287b384306adf9ca"
+    },
+    "token-manifest.json": {
+      "bytes": 6208,
+      "sha256": "c7ebf3672d46cfc75d12013c34f35fb7ac9eac613f29f3454be0cb0e991854dc"
+    },
+    "tokenizer.model": {
+      "bytes": 284118,
+      "sha256": "d9f1ba1e456ce72dd9c06a62b12e804cf14090c30179e6078cd51d03063b2982"
+    },
+    "weights.pt": {
+      "bytes": 29562119,
+      "sha256": "1cdac16913068bfdcfa94b139ecb34d945d0e9483a863373a6dca4397e3cd355"
+    }
+  },
+  "policy": "Weights only; no optimizer, RNG or training state. LM-only contextual suffix logP sum; no extra EOS; no KV cache."
+}

inference/token-manifest.json

@@ -0,0 +1,119 @@
+{
+  "status": "complete",
+  "format": "vimeml_sentence_tokens_v1",
+  "vocab_size": 16384,
+  "token_dtype": "uint16_le",
+  "offset_dtype": "uint64_le",
+  "offset_unit": "tokens",
+  "special_ids": {
+    "pad": 0,
+    "unk": 1,
+    "bos": 2,
+    "eos": 3
+  },
+  "sentencepiece_version": "0.2.1",
+  "splits": {
+    "train": {
+      "sentences": 25185368,
+      "stored_tokens": 598480605,
+      "content_tokens": 548109869,
+      "characters": 1080952698,
+      "prediction_pairs": 573295237,
+      "max_sequence_tokens": 590,
+      "primary_sources": {
+        "fineweb": 24938228,
+        "tatoeba": 247140
+      },
+      "file_bytes": {
+        "tokens.bin": 1196961210,
+        "offsets.bin": 201482952,
+        "rows.bin": 201482944,
+        "sources.bin": 25185368
+      }
+    },
+    "validation": {
+      "sentences": 260337,
+      "stored_tokens": 6150663,
+      "content_tokens": 5629989,
+      "characters": 11110489,
+      "prediction_pairs": 5890326,
+      "max_sequence_tokens": 439,
+      "primary_sources": {
+        "fineweb": 257916,
+        "tatoeba": 2421
+      },
+      "file_bytes": {
+        "tokens.bin": 12301326,
+        "offsets.bin": 2082704,
+        "rows.bin": 2082696,
+        "sources.bin": 260337
+      }
+    },
+    "test": {
+      "sentences": 267298,
+      "stored_tokens": 6258211,
+      "content_tokens": 5723615,
+      "characters": 11296824,
+      "prediction_pairs": 5990913,
+      "max_sequence_tokens": 497,
+      "primary_sources": {
+        "fineweb": 264698,
+        "tatoeba": 2600
+      },
+      "file_bytes": {
+        "tokens.bin": 12516422,
+        "offsets.bin": 2138392,
+        "rows.bin": 2138384,
+        "sources.bin": 267298
+      }
+    }
+  },
+  "source_ids": {
+    "fineweb": 0,
+    "tatoeba": 1
+  },
+  "sequence_policy": "Each sequence is BOS + complete encoded sentence + EOS; no truncation or concatenation.",
+  "prediction_policy": "x=s[:-1], y=s[1:]; sentence boundaries remain separate.",
+  "provenance_policy": "Index equals zero-based corpus JSONL/TXT row. rows.bin stores uint64 little-endian byte positions in the source JSONL; sources.bin stores uint8 primary source IDs. Full original metadata stays in corpus JSONL.",
+  "corpus_dir": "D:\\Sources\\VimeML\\outputs\\corpus-fast-v1",
+  "tokenizer_dir": "D:\\Sources\\VimeML\\artifacts\\tokenizers\\ja-unigram-16k-v1",
+  "input_sha256": {
+    "D:\\Sources\\VimeML\\outputs\\corpus-fast-v1\\train.txt": "9c1b4585adaf0518dc0275e7f8086e967db26f9797fb571e1ff481aeadf2ff8f",
+    "D:\\Sources\\VimeML\\outputs\\corpus-fast-v1\\train.jsonl": "0c8bb3db59cd820db6315c63cd1b7adc92e7afcb7e8bc82ee39820f1e3d7ea21",
+    "D:\\Sources\\VimeML\\outputs\\corpus-fast-v1\\validation.txt": "f188bea10a8b464dd385d8e0aa310225de07e2a0a170e90ce63503c150bf4cdd",
+    "D:\\Sources\\VimeML\\outputs\\corpus-fast-v1\\validation.jsonl": "544773d7bd40ed22244a618db4508675109441eb69959d41a191e689fcbdfd58",
+    "D:\\Sources\\VimeML\\outputs\\corpus-fast-v1\\test.txt": "92e99a5062473a749254d90d9268ede03a804c9a8ac5806aa0a43e8b63222972",
+    "D:\\Sources\\VimeML\\outputs\\corpus-fast-v1\\test.jsonl": "28b0a0493f5c77c5ecdb768d6e023400b191d6855f5542a68a0a87a766586ef3",
+    "D:\\Sources\\VimeML\\outputs\\corpus-fast-v1\\manifest.json": "4a9d25d2a4abc6d9df8b008b49e34c060b917606c700dfa3b9f0dd4006030233",
+    "D:\\Sources\\VimeML\\outputs\\corpus-fast-v1\\stats.json": "fd7fc5077f0fdccd46591c1664735990a8f28fa8c93fa12a0e0030352ac7ec8b",
+    "D:\\Sources\\VimeML\\artifacts\\tokenizers\\ja-unigram-16k-v1\\tokenizer.model": "d9f1ba1e456ce72dd9c06a62b12e804cf14090c30179e6078cd51d03063b2982",
+    "D:\\Sources\\VimeML\\artifacts\\tokenizers\\ja-unigram-16k-v1\\manifest.json": "e88e66dee784ffa087a90e584793c2f41fb1f135499cf1bdd9eaa7cfb72ee921",
+    "D:\\Sources\\VimeML\\artifacts\\tokenizers\\ja-unigram-16k-v1\\stats.json": "9232bce676e86718940cdb11b5ad5d92b123816723ab68fa3d47212adca60ba8",
+    "D:\\Sources\\VimeML\\src\\vimeml\\tokenizer\\encode_parallel.py": "c194cb241d40b49669515f68f44278b55bcfbae210756f220b375cdee22ab009",
+    "D:\\Sources\\VimeML\\src\\vimeml\\tokenizer\\encode.py": "92bdbdda6fa80f4dbb975f8882c021d8c7c0f9017f669c89dfae31dc2eecaa69",
+    "D:\\Sources\\VimeML\\src\\vimeml\\tokenizer\\monitor.py": "c8cae279ea7faa0021b85bbc8dd2151510360a18db87982638920cc15a726b3a",
+    "D:\\Sources\\VimeML\\src\\vimeml\\tokenizer\\train.py": "3dc553d88962feb67fb919ee1f08879b2f9eac0d5b2783db7eed385505d3ee37"
+  },
+  "output_sha256": {
+    "train.tokens.bin": "5d8bafb458cdc510c0a9e9182d55d084c03cbc651eb348e387a965ac1b882834",
+    "train.offsets.bin": "8cee1724543d454e353fbd023cb40b0c0a9b2f9eb9b24b9e8ddae85afb46e762",
+    "train.rows.bin": "7cc9d8960e16d8df1c4ad1798255ade4def3bd066072439ebc3b7e8336674756",
+    "train.sources.bin": "e0e5184a6cc40fd6df0c4858cb56d79af838b07f07b86913b2a65395d325e616",
+    "validation.tokens.bin": "1bc211def7323bb620f164172a906e160c1860c1832c7d03c2de9699472f96cd",
+    "validation.offsets.bin": "e193c626736661525e2757186fb3588556d081a8019144fec0337de217dbfe1c",
+    "validation.rows.bin": "e63780d4e9b8378cee413683b6e861392dcb15a8af10308c63a8bcf68402bb79",
+    "validation.sources.bin": "dd57fbd72bf36133403751dc583943a5adf0409cf2a77bdb56ac6cc461944c34",
+    "test.tokens.bin": "ad5e5f6d5e5805bc7dbc883bbcd221df8d5699126c5a9210e86b86dad71a4e2c",
+    "test.offsets.bin": "89cdbcd19a39d0a2ff6a120aa1aeb526fc30aab03eec76b0fe50e51ca311dec3",
+    "test.rows.bin": "9653952ecabd27980f28ad7da21787b8e51046f55141a19cb21032e6de9b2faa",
+    "test.sources.bin": "473cc664829aa10e3c423389129427821b9985118febf6b276d01bd523056e37",
+    "stats.json": "563df68f55d54ed273681eb51145fd7b2cf4f2b4c1f86a053f07c666f9bc919b"
+  },
+  "corpus_quality_mode": "rules_with_approved_annotations",
+  "corpus_ready_for_lm_training": false,
+  "purpose": "Full token data for the accepted first corpus baseline; known extraction and near-duplicate limitations remain.",
+  "workers": 16,
+  "native_threads_per_worker": 1,
+  "signature": "53badf97e9290a595598f4c1cb270a118e5b92870dd406769752d119c31d963a",
+  "script_sha256": "c194cb241d40b49669515f68f44278b55bcfbae210756f220b375cdee22ab009"
+}

inference/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d9f1ba1e456ce72dd9c06a62b12e804cf14090c30179e6078cd51d03063b2982
+size 284118

inference/weights.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1cdac16913068bfdcfa94b139ecb34d945d0e9483a863373a6dca4397e3cd355
+size 29562119

source/LICENSE

@@ -0,0 +1,339 @@
+                    GNU GENERAL PUBLIC LICENSE
+                       Version 2, June 1991
+
+ Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
+ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+                            Preamble
+
+  The licenses for most software are designed to take away your
+freedom to share and change it.  By contrast, the GNU General Public
+License is intended to guarantee your freedom to share and change free
+software--to make sure the software is free for all its users.  This
+General Public License applies to most of the Free Software
+Foundation's software and to any other program whose authors commit to
+using it.  (Some other Free Software Foundation software is covered by
+the GNU Lesser General Public License instead.)  You can apply it to
+your programs, too.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+this service if you wish), that you receive source code or can get it
+if you want it, that you can change the software or use pieces of it
+in new free programs; and that you know you can do these things.
+
+  To protect your rights, we need to make restrictions that forbid
+anyone to deny you these rights or to ask you to surrender the rights.
+These restrictions translate to certain responsibilities for you if you
+distribute copies of the software, or if you modify it.
+
+  For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must give the recipients all the rights that
+you have.  You must make sure that they, too, receive or can get the
+source code.  And you must show them these terms so they know their
+rights.
+
+  We protect your rights with two steps: (1) copyright the software, and
+(2) offer you this license which gives you legal permission to copy,
+distribute and/or modify the software.
+
+  Also, for each author's protection and ours, we want to make certain
+that everyone understands that there is no warranty for this free
+software.  If the software is modified by someone else and passed on, we
+want its recipients to know that what they have is not the original, so
+that any problems introduced by others will not reflect on the original
+authors' reputations.
+
+  Finally, any free program is threatened constantly by software
+patents.  We wish to avoid the danger that redistributors of a free
+program will individually obtain patent licenses, in effect making the
+program proprietary.  To prevent this, we have made it clear that any
+patent must be licensed for everyone's free use or not licensed at all.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+                    GNU GENERAL PUBLIC LICENSE
+   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+
+  0. This License applies to any program or other work which contains
+a notice placed by the copyright holder saying it may be distributed
+under the terms of this General Public License.  The "Program", below,
+refers to any such program or work, and a "work based on the Program"
+means either the Program or any derivative work under copyright law:
+that is to say, a work containing the Program or a portion of it,
+either verbatim or with modifications and/or translated into another
+language.  (Hereinafter, translation is included without limitation in
+the term "modification".)  Each licensee is addressed as "you".
+
+Activities other than copying, distribution and modification are not
+covered by this License; they are outside its scope.  The act of
+running the Program is not restricted, and the output from the Program
+is covered only if its contents constitute a work based on the
+Program (independent of having been made by running the Program).
+Whether that is true depends on what the Program does.
+
+  1. You may copy and distribute verbatim copies of the Program's
+source code as you receive it, in any medium, provided that you
+conspicuously and appropriately publish on each copy an appropriate
+copyright notice and disclaimer of warranty; keep intact all the
+notices that refer to this License and to the absence of any warranty;
+and give any other recipients of the Program a copy of this License
+along with the Program.
+
+You may charge a fee for the physical act of transferring a copy, and
+you may at your option offer warranty protection in exchange for a fee.
+
+  2. You may modify your copy or copies of the Program or any portion
+of it, thus forming a work based on the Program, and copy and
+distribute such modifications or work under the terms of Section 1
+above, provided that you also meet all of these conditions:
+
+    a) You must cause the modified files to carry prominent notices
+    stating that you changed the files and the date of any change.
+
+    b) You must cause any work that you distribute or publish, that in
+    whole or in part contains or is derived from the Program or any
+    part thereof, to be licensed as a whole at no charge to all third
+    parties under the terms of this License.
+
+    c) If the modified program normally reads commands interactively
+    when run, you must cause it, when started running for such
+    interactive use in the most ordinary way, to print or display an
+    announcement including an appropriate copyright notice and a
+    notice that there is no warranty (or else, saying that you provide
+    a warranty) and that users may redistribute the program under
+    these conditions, and telling the user how to view a copy of this
+    License.  (Exception: if the Program itself is interactive but
+    does not normally print such an announcement, your work based on
+    the Program is not required to print an announcement.)
+
+These requirements apply to the modified work as a whole.  If
+identifiable sections of that work are not derived from the Program,
+and can be reasonably considered independent and separate works in
+themselves, then this License, and its terms, do not apply to those
+sections when you distribute them as separate works.  But when you
+distribute the same sections as part of a whole which is a work based
+on the Program, the distribution of the whole must be on the terms of
+this License, whose permissions for other licensees extend to the
+entire whole, and thus to each and every part regardless of who wrote it.
+
+Thus, it is not the intent of this section to claim rights or contest
+your rights to work written entirely by you; rather, the intent is to
+exercise the right to control the distribution of derivative or
+collective works based on the Program.
+
+In addition, mere aggregation of another work not based on the Program
+with the Program (or with a work based on the Program) on a volume of
+a storage or distribution medium does not bring the other work under
+the scope of this License.
+
+  3. You may copy and distribute the Program (or a work based on it,
+under Section 2) in object code or executable form under the terms of
+Sections 1 and 2 above provided that you also do one of the following:
+
+    a) Accompany it with the complete corresponding machine-readable
+    source code, which must be distributed under the terms of Sections
+    1 and 2 above on a medium customarily used for software interchange; or,
+
+    b) Accompany it with a written offer, valid for at least three
+    years, to give any third party, for a charge no more than your
+    cost of physically performing source distribution, a complete
+    machine-readable copy of the corresponding source code, to be
+    distributed under the terms of Sections 1 and 2 above on a medium
+    customarily used for software interchange; or,
+
+    c) Accompany it with the information you received as to the offer
+    to distribute corresponding source code.  (This alternative is
+    allowed only for noncommercial distribution and only if you
+    received the program in object code or executable form with such
+    an offer, in accord with Subsection b above.)
+
+The source code for a work means the preferred form of the work for
+making modifications to it.  For an executable work, complete source
+code means all the source code for all modules it contains, plus any
+associated interface definition files, plus the scripts used to
+control compilation and installation of the executable.  However, as a
+special exception, the source code distributed need not include
+anything that is normally distributed (in either source or binary
+form) with the major components (compiler, kernel, and so on) of the
+operating system on which the executable runs, unless that component
+itself accompanies the executable.
+
+If distribution of executable or object code is made by offering
+access to copy from a designated place, then offering equivalent
+access to copy the source code from the same place counts as
+distribution of the source code, even though third parties are not
+compelled to copy the source along with the object code.
+
+  4. You may not copy, modify, sublicense, or distribute the Program
+except as expressly provided under this License.  Any attempt
+otherwise to copy, modify, sublicense or distribute the Program is
+void, and will automatically terminate your rights under this License.
+However, parties who have received copies, or rights, from you under
+this License will not have their licenses terminated so long as such
+parties remain in full compliance.
+
+  5. You are not required to accept this License, since you have not
+signed it.  However, nothing else grants you permission to modify or
+distribute the Program or its derivative works.  These actions are
+prohibited by law if you do not accept this License.  Therefore, by
+modifying or distributing the Program (or any work based on the
+Program), you indicate your acceptance of this License to do so, and
+all its terms and conditions for copying, distributing or modifying
+the Program or works based on it.
+
+  6. Each time you redistribute the Program (or any work based on the
+Program), the recipient automatically receives a license from the
+original licensor to copy, distribute or modify the Program subject to
+these terms and conditions.  You may not impose any further
+restrictions on the recipients' exercise of the rights granted herein.
+You are not responsible for enforcing compliance by third parties to
+this License.
+
+  7. If, as a consequence of a court judgment or allegation of patent
+infringement or for any other reason (not limited to patent issues),
+conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot
+distribute so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you
+may not distribute the Program at all.  For example, if a patent
+license would not permit royalty-free redistribution of the Program by
+all those who receive copies directly or indirectly through you, then
+the only way you could satisfy both it and this License would be to
+refrain entirely from distribution of the Program.
+
+If any portion of this section is held invalid or unenforceable under
+any particular circumstance, the balance of the section is intended to
+apply and the section as a whole is intended to apply in other
+circumstances.
+
+It is not the purpose of this section to induce you to infringe any
+patents or other property right claims or to contest validity of any
+such claims; this section has the sole purpose of protecting the
+integrity of the free software distribution system, which is
+implemented by public license practices.  Many people have made
+generous contributions to the wide range of software distributed
+through that system in reliance on consistent application of that
+system; it is up to the author/donor to decide if he or she is willing
+to distribute software through any other system and a licensee cannot
+impose that choice.
+
+This section is intended to make thoroughly clear what is believed to
+be a consequence of the rest of this License.
+
+  8. If the distribution and/or use of the Program is restricted in
+certain countries either by patents or by copyrighted interfaces, the
+original copyright holder who places the Program under this License
+may add an explicit geographical distribution limitation excluding
+those countries, so that distribution is permitted only in or among
+countries not thus excluded.  In such case, this License incorporates
+the limitation as if written in the body of this License.
+
+  9. The Free Software Foundation may publish revised and/or new versions
+of the General Public License from time to time.  Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+Each version is given a distinguishing version number.  If the Program
+specifies a version number of this License which applies to it and "any
+later version", you have the option of following the terms and conditions
+either of that version or of any later version published by the Free
+Software Foundation.  If the Program does not specify a version number of
+this License, you may choose any version ever published by the Free Software
+Foundation.
+
+  10. If you wish to incorporate parts of the Program into other free
+programs whose distribution conditions are different, write to the author
+to ask for permission.  For software which is copyrighted by the Free
+Software Foundation, write to the Free Software Foundation; we sometimes
+make exceptions for this.  Our decision will be guided by the two goals
+of preserving the free status of all derivatives of our free software and
+of promoting the sharing and reuse of software generally.
+
+                            NO WARRANTY
+
+  11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
+FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN
+OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
+PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
+OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS
+TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD THE
+PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
+REPAIR OR CORRECTION.
+
+  12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
+REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
+INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
+OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
+TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
+YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
+PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGES.
+
+                     END OF TERMS AND CONDITIONS
+
+            How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+convey the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 2 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License along
+    with this program; if not, write to the Free Software Foundation, Inc.,
+    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+Also add information on how to contact you by electronic and paper mail.
+
+If the program is interactive, make it output a short notice like this
+when it starts in an interactive mode:
+
+    Gnomovision version 69, Copyright (C) year name of author
+    Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+    This is free software, and you are welcome to redistribute it
+    under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License.  Of course, the commands you use may
+be called something other than `show w' and `show c'; they could even be
+mouse-clicks or menu items--whatever suits your program.
+
+You should also get your employer (if you work as a programmer) or your
+school, if any, to sign a "copyright disclaimer" for the program, if
+necessary.  Here is a sample; alter the names:
+
+  Yoyodyne, Inc., hereby disclaims all copyright interest in the program
+  `Gnomovision' (which makes passes at compilers) written by James Hacker.
+
+  <signature of Ty Coon>, 1 April 1989
+  Ty Coon, President of Vice
+
+This General Public License does not permit incorporating your program into
+proprietary programs.  If your program is a subroutine library, you may
+consider it more useful to permit linking proprietary applications with the
+library.  If this is what you want to do, use the GNU Lesser General
+Public License instead of this License.

source/README.md

@@ -0,0 +1,3 @@
+VimeML source/configuration from Git commit 9f0409ef7970c468efb622b207b1089de09651c1. GPL-2.0; see LICENSE.
+The FP32 loader checks the original model and deployment code fingerprints.
+Corpus texts, annotations, optimizer state and the separate Vime client are not included.

source/configs/corpus-parallel.toml

@@ -0,0 +1,35 @@
+# Local parallel runner: one command starts all workers and merges automatically.
+[inputs]
+fineweb_paths = [
+  "datasets/fineweb-2-edu-japanese/train-00010-of-00283.parquet",
+  "datasets/fineweb-2-edu-japanese/train-00030-of-00283.parquet",
+  "datasets/fineweb-2-edu-japanese/train-00050-of-00283.parquet",
+  "datasets/fineweb-2-edu-japanese/train-00070-of-00283.parquet",
+  "datasets/fineweb-2-edu-japanese/train-00090-of-00283.parquet",
+  "datasets/fineweb-2-edu-japanese/train-00110-of-00283.parquet",
+  "datasets/fineweb-2-edu-japanese/train-00130-of-00283.parquet",
+  "datasets/fineweb-2-edu-japanese/train-00150-of-00283.parquet",
+  "datasets/fineweb-2-edu-japanese/train-00170-of-00283.parquet",
+]
+tatoeba_path = "datasets/Tatoeba/jpn_sentences.tsv"
+calibration_ids = "annotations/calibration-documents-v1.json"
+
+[build]
+quality_mode = "rules_with_approved_annotations"
+output_dir = "outputs/corpus-parallel-v1"
+seed = 42
+batch_size = 1024
+max_join_chars = 2048
+fineweb_documents_per_shard = 0
+tatoeba_documents = 0
+commit_every = 5000
+progress_every = 5000
+
+[review]
+annotations = "annotations/approved-v1.jsonl"
+require_all_annotations = true
+
+[split]
+train = 0.98
+validation = 0.01
+test = 0.01

source/configs/corpus-sharded.toml

@@ -0,0 +1,35 @@
+# Same file on both machines; only --part and --output differ.
+[inputs]
+fineweb_paths = [
+  "datasets/fineweb-2-edu-japanese/train-00010-of-00283.parquet",
+  "datasets/fineweb-2-edu-japanese/train-00030-of-00283.parquet",
+  "datasets/fineweb-2-edu-japanese/train-00050-of-00283.parquet",
+  "datasets/fineweb-2-edu-japanese/train-00070-of-00283.parquet",
+  "datasets/fineweb-2-edu-japanese/train-00090-of-00283.parquet",
+  "datasets/fineweb-2-edu-japanese/train-00110-of-00283.parquet",
+  "datasets/fineweb-2-edu-japanese/train-00130-of-00283.parquet",
+  "datasets/fineweb-2-edu-japanese/train-00150-of-00283.parquet",
+  "datasets/fineweb-2-edu-japanese/train-00170-of-00283.parquet",
+]
+tatoeba_path = "datasets/Tatoeba/jpn_sentences.tsv"
+calibration_ids = "annotations/calibration-documents-v1.json"
+
+[build]
+quality_mode = "rules_with_approved_annotations"
+output_dir = "outputs/corpus-parts"
+seed = 42
+batch_size = 1024
+max_join_chars = 2048
+fineweb_documents_per_shard = 0
+tatoeba_documents = 0
+commit_every = 5000
+progress_every = 5000
+
+[review]
+annotations = "annotations/approved-v1.jsonl"
+require_all_annotations = true
+
+[split]
+train = 0.98
+validation = 0.01
+test = 0.01

source/configs/ime-eval-v1.json

@@ -0,0 +1,54 @@
+{
+  "version": "ime_diagnostic_v1",
+  "provenance": "Handwritten Japanese diagnostic cases, frozen before first evaluation. Synthetic homophone candidate lists, not AzooKey output. Training overlap unknown. References are non-exhaustive.",
+  "ranking": [
+    {"id":"hashi-food","group":"hashi","reading":"はし","context":"うどんを食べるために","candidates":["箸","橋","端"],"acceptable":["箸"]},
+    {"id":"hashi-river","group":"hashi","reading":"はし","context":"川を渡るための","candidates":["箸","橋","端"],"acceptable":["橋"]},
+    {"id":"hashi-edge","group":"hashi","reading":"はし","context":"机の中央ではなく","candidates":["箸","橋","端"],"acceptable":["端"]},
+    {"id":"au-friend","group":"au","reading":"あう","context":"駅で友人に","candidates":["会う","合う","遭う"],"acceptable":["会う"]},
+    {"id":"au-fit","group":"au","reading":"あう","context":"この服は体に","candidates":["会う","合う","遭う"],"acceptable":["合う"]},
+    {"id":"au-accident","group":"au","reading":"あう","context":"帰り道で事故に","candidates":["会う","合う","遭う"],"acceptable":["遭う"]},
+    {"id":"kikai-machine","group":"kikai","reading":"きかい","context":"工場で使う大きな","candidates":["機会","機械"],"acceptable":["機械"]},
+    {"id":"kikai-chance","group":"kikai","reading":"きかい","context":"もう一度挑戦する","candidates":["機会","機械"],"acceptable":["機会"]},
+    {"id":"kami-paper","group":"kami","reading":"かみ","context":"メモを書くための","candidates":["紙","髪","神"],"acceptable":["紙"]},
+    {"id":"kami-hair","group":"kami","reading":"かみ","context":"美容室で切ってもらった","candidates":["紙","髪","神"],"acceptable":["髪"]},
+    {"id":"kami-god","group":"kami","reading":"かみ","context":"神社に祀られている","candidates":["紙","髪","神"],"acceptable":["神"]},
+    {"id":"kouen-park","group":"kouen","reading":"こうえん","context":"子供たちが遊具で遊ぶ","candidates":["公園","講演","公演"],"acceptable":["公園"]},
+    {"id":"kouen-lecture","group":"kouen","reading":"こうえん","context":"大学の先生が研究成果について行う","candidates":["公園","講演","公演"],"acceptable":["講演"]},
+    {"id":"kouen-show","group":"kouen","reading":"こうえん","context":"劇場で行われるミュージカルの","candidates":["公園","講演","公演"],"acceptable":["公演"]},
+    {"id":"kanshou-art","group":"kanshou","reading":"かんしょう","context":"美術館で絵画を","candidates":["鑑賞","干渉"],"acceptable":["鑑賞"]},
+    {"id":"kanshou-interfere","group":"kanshou","reading":"かんしょう","context":"他人の私生活に過度に","candidates":["鑑賞","干渉"],"acceptable":["干渉"]},
+    {"id":"seikaku-personality","group":"seikaku","reading":"せいかく","context":"彼は明るく穏やかな","candidates":["性格","正確"],"acceptable":["性格"]},
+    {"id":"seikaku-accurate","group":"seikaku","reading":"せいかく","context":"この測定結果は非常に","candidates":["性格","正確"],"acceptable":["正確"]},
+    {"id":"hoshou-warranty","group":"hoshou","reading":"ほしょう","context":"製品が故障した場合の無料修理を","candidates":["保証","保障","補償"],"acceptable":["保証"]},
+    {"id":"hoshou-rights","group":"hoshou","reading":"ほしょう","context":"すべての国民の基本的人権を","candidates":["保証","保障","補償"],"acceptable":["保障"]},
+    {"id":"hoshou-damage","group":"hoshou","reading":"ほしょう","context":"事故による被害者の損害を","candidates":["保証","保障","補償"],"acceptable":["補償"]},
+    {"id":"ishi-stone","group":"ishi","reading":"いし","context":"川辺で拾った丸い","candidates":["石","意思","医師","意志"],"acceptable":["石"]},
+    {"id":"ishi-doctor","group":"ishi","reading":"いし","context":"病院で患者を診察する","candidates":["石","意思","医師","意志"],"acceptable":["医師"]},
+    {"id":"ishi-intent","group":"ishi","reading":"いし","context":"本人の同意や","candidates":["石","意思","医師","意志"],"acceptable":["意思","意志"],"note":"意思 is the preferred spelling here; 意志 is accepted conservatively as a meaningful variant."},
+    {"id":"atsui-weather","group":"atsui","reading":"あつい","context":"真夏の屋外はとても","candidates":["暑い","熱い","厚い"],"acceptable":["暑い"]},
+    {"id":"atsui-water","group":"atsui","reading":"あつい","context":"沸かしたばかりのお湯は","candidates":["暑い","熱い","厚い"],"acceptable":["熱い"]},
+    {"id":"atsui-book","group":"atsui","reading":"あつい","context":"この本はページが多くて","candidates":["暑い","熱い","厚い"],"acceptable":["厚い"]},
+    {"id":"naosu-repair","group":"naosu","reading":"なおす","context":"壊れたパソコンを","candidates":["直す","治す"],"acceptable":["直す"]},
+    {"id":"naosu-cure","group":"naosu","reading":"なおす","context":"薬を飲んで風邪を","candidates":["直す","治す"],"acceptable":["治す"]},
+    {"id":"kiku-medicine","group":"kiku","reading":"きく","context":"この薬は頭痛によく","candidates":["聞く","聴く","効く","利く"],"acceptable":["効く"]},
+    {"id":"kiku-listen","group":"kiku","reading":"きく","context":"会議で上司の説明を","candidates":["聞く","聴く","効く","利く"],"acceptable":["聞く","聴く"],"note":"Both listening spellings are accepted."},
+    {"id":"kiku-flexible","group":"kiku","reading":"きく","context":"小さな変更でも融通が","candidates":["聞く","聴く","効く","利く"],"acceptable":["利く"]}
+  ],
+  "suggestions": [
+    {"id":"weather","prompt":"今日はとても","references":["いい天気","良い天気","暑い","寒い","楽しい","忙しい","暖かい","寒かった","暑かった","よかった","良かった"]},
+    {"id":"formal-thanks","prompt":"本日はお忙しい中","references":["ありがとうございます","ありがとうございました","お越しいただき","ご参加いただき","お集まりいただき","お時間をいただき"]},
+    {"id":"busy-request","prompt":"お忙しいところ","references":["ありがとうございます","ありがとうございました","恐れ入ります","申し訳ありません","恐縮ですが","お手数ですが","すみません"]},
+    {"id":"confirmation","prompt":"ご確認のほど","references":["よろしくお願いいたします","よろしくお願いします","お願いいたします","お願いします","お願い申し上げます"]},
+    {"id":"walk","prompt":"駅まで歩いて","references":["行きます","行く","向かいます","向かう","移動","約","五分","5分","10分"]},
+    {"id":"password","prompt":"パスワードを入力して","references":["ログイン","ください","下さい","次へ","認証","送信"]},
+    {"id":"form","prompt":"お名前とメールアドレスを","references":["入力してください","ご入力ください","記入してください","お知らせください","登録してください","入力","記入"]},
+    {"id":"cold","prompt":"風邪をひいたので","references":["休み","休む","病院","薬","寝","早く寝","会社を休","学校を休","今日は休","自宅で","家で","休養"]},
+    {"id":"attachment","prompt":"添付した資料の内容を","references":["ご確認ください","確認してください","ご確認いただけますか","確認","ご確認"]},
+    {"id":"reply","prompt":"お返事を","references":["お待ち","いただけ","ください","お願い","頂け"]},
+    {"id":"application","prompt":"このアプリは","references":["無料","簡単","スマートフォン","使い","便利","利用","対応"]},
+    {"id":"study","prompt":"日本語を勉強する","references":["ため","上で","とき","時","方法","こと"]},
+    {"id":"morning","prompt":"明日の朝は","references":["早く","早起き","雨","晴れ","学校","仕事","出発","午前"]},
+    {"id":"dinner","prompt":"夕食のあとに","references":["散歩","お風呂","デザート","お茶","家族","テレビ","寝","勉強"]}
+  ]
+}

source/configs/ime-homophones.json

@@ -0,0 +1,18 @@
+{
+  "version": "handchecked_homophones_v1",
+  "note": "Controlled same-reading candidate pools. Contexts and gold suitability still require blind verification. Alternative listening/intent spellings are excluded to reduce ambiguity.",
+  "groups": [
+    {"id":"hashi","reading":"はし","candidates":["箸","橋","端"]},
+    {"id":"au","reading":"あう","candidates":["会う","合う","遭う"]},
+    {"id":"kikai","reading":"きかい","candidates":["機会","機械"]},
+    {"id":"kami","reading":"かみ","candidates":["紙","髪","神"]},
+    {"id":"kouen","reading":"こうえん","candidates":["公園","講演","公演"]},
+    {"id":"kanshou","reading":"かんしょう","candidates":["鑑賞","干渉"]},
+    {"id":"seikaku","reading":"せいかく","candidates":["性格","正確"]},
+    {"id":"hoshou","reading":"ほしょう","candidates":["保証","保障","補償"]},
+    {"id":"ishi","reading":"いし","candidates":["石","医師"]},
+    {"id":"atsui","reading":"あつい","candidates":["暑い","熱い","厚い"]},
+    {"id":"naosu","reading":"なおす","candidates":["直す","治す"]},
+    {"id":"kiku","reading":"きく","candidates":["聞く","効く","利く"]}
+  ]
+}

source/configs/inference-prompts.json

@@ -0,0 +1,14 @@
+[
+  {"kind": "daily", "prompt": "今日はとても"},
+  {"kind": "daily", "prompt": "明日の朝は"},
+  {"kind": "daily", "prompt": "駅まで歩いて"},
+  {"kind": "daily", "prompt": "この本を読んで"},
+  {"kind": "learning", "prompt": "日本語を勉強する"},
+  {"kind": "business", "prompt": "お忙しいところ"},
+  {"kind": "business", "prompt": "本日はお忙しい中"},
+  {"kind": "technology", "prompt": "このアプリは"},
+  {"kind": "daily", "prompt": "最近、仕事が忙しくて"},
+  {"kind": "travel", "prompt": "東京駅から電車で"},
+  {"kind": "digits", "prompt": "2026年10月"},
+  {"kind": "latin", "prompt": "iPhoneで日本語を"}
+]

source/configs/loader.toml

@@ -0,0 +1,8 @@
+# Data validation only. No model is trained by this entry point.
+token_dir = "artifacts/token-data/corpus-v1-16k"
+index_dir = "artifacts/training-data/corpus-v1-c128"
+context_length = 128
+batch_size = 128
+num_workers = 4
+seed = 42
+check_batches = 8

source/configs/phrase-demo-prompts.json

@@ -0,0 +1,22 @@
+[
+  {"id":"rain","scene":"天气","prompt":"今日は雨が降っているので、"},
+  {"id":"meeting","scene":"工作","prompt":"明日の会議までに、"},
+  {"id":"station","scene":"出行","prompt":"駅に着いたら、"},
+  {"id":"food","scene":"日常","prompt":"お腹が空いたので、"},
+  {"id":"thanks","scene":"商务","prompt":"本日はお忙しい中、"},
+  {"id":"confirmation","scene":"商务","prompt":"ご確認のほど、"},
+  {"id":"attachment","scene":"工作","prompt":"添付した資料を"},
+  {"id":"password","scene":"技术","prompt":"パスワードを入力して"},
+  {"id":"update","scene":"技术","prompt":"アプリを更新したら、"},
+  {"id":"travel","scene":"出行","prompt":"旅行に出かける前に、"},
+  {"id":"cold","scene":"日常","prompt":"風邪をひいたので、"},
+  {"id":"reading","scene":"日常","prompt":"寝る前に本を"},
+  {"id":"clothes","scene":"穿着","prompt":"外は寒いので、コートを"},
+  {"id":"movie","scene":"媒体","prompt":"昨日見た映画は"},
+  {"id":"practice","scene":"学习","prompt":"日本語が上手になるために、"},
+  {"id":"schedule","scene":"安排","prompt":"来週の月曜日は"},
+  {"id":"reply","scene":"工作","prompt":"お返事を"},
+  {"id":"dinner","scene":"日常","prompt":"夕食を食べた後、"},
+  {"id":"coffee","scene":"日常","prompt":"朝起きたら、コーヒーを"},
+  {"id":"friend","scene":"交流","prompt":"久しぶりに友達と"}
+]

source/configs/tokenizer.toml

@@ -0,0 +1,25 @@
+# Formal baseline: learn the vocabulary from a random sample of all train rows.
+# LM training still uses the full corpus. Only a manual command starts training.
+[paths]
+corpus_dir = "outputs/corpus-fast-v1"
+output_dir = "artifacts/tokenizers/ja-unigram-16k-v1"
+
+[training]
+model_type = "unigram"
+vocab_size = 16384
+character_coverage = 0.9995
+byte_fallback = true
+normalization_rule_name = "identity"
+remove_extra_whitespaces = false
+num_threads = 12
+input_sentence_size = 2000000
+seed = 42
+max_sentence_length = 8192
+
+[measurement]
+# Independent processes; each native encoder uses one thread.
+workers = 8
+batch_size = 4096
+
+[monitoring]
+tensorboard = true

source/configs/train-smoke.toml

@@ -0,0 +1,37 @@
+# First manual run: pipeline correctness and measured throughput, not a final LM.
+token_dir = "artifacts/token-data/corpus-v1-16k"
+index_dir = "artifacts/training-data/corpus-v1-c128"
+output_dir = "artifacts/models/tiny-ja-smoke-v1"
+log_dir = "runs/training/tiny-ja-smoke-v1"
+
+[model]
+vocab_size = 16384
+context_length = 128
+d_model = 256
+n_heads = 4
+n_layers = 4
+d_ff = 1024
+dropout = 0.0
+
+[training]
+device = "cuda"
+precision = "bf16"
+seed = 42
+cpu_threads = 4
+batch_size = 128
+num_workers = 4
+bucket_multiplier = 64
+gradient_accumulation = 1
+max_steps = 300
+learning_rate = 0.0003
+min_learning_rate = 0.00003
+warmup_steps = 30
+weight_decay = 0.1
+beta1 = 0.9
+beta2 = 0.95
+grad_clip = 1.0
+log_every = 10
+eval_every = 100
+eval_batches = 16
+checkpoint_every = 100
+benchmark_warmup_steps = 20

source/configs/train-v1.toml

@@ -0,0 +1,40 @@
+# First formal baseline: exactly one full pass through the frozen corpus.
+# 25,197,117 windows / batch128 = 196,853 updates, including the partial batch.
+token_dir = "artifacts/token-data/corpus-v1-16k"
+index_dir = "artifacts/training-data/corpus-v1-c128"
+output_dir = "artifacts/models/tiny-ja-v1"
+log_dir = "runs/training/tiny-ja-v1"
+
+[model]
+vocab_size = 16384
+context_length = 128
+d_model = 256
+n_heads = 4
+n_layers = 4
+d_ff = 1024
+dropout = 0.0
+
+[training]
+device = "cuda"
+precision = "bf16"
+seed = 42
+cpu_threads = 4
+batch_size = 128
+num_workers = 4
+bucket_multiplier = 64
+gradient_accumulation = 1
+epochs = 1
+max_steps = 196853
+learning_rate = 0.0003
+min_learning_rate = 0.00003
+warmup_steps = 2000
+weight_decay = 0.1
+beta1 = 0.9
+beta2 = 0.95
+grad_clip = 1.0
+log_every = 100
+eval_every = 5000
+eval_batches = 128
+checkpoint_every = 5000
+benchmark_warmup_steps = 100
+validate_full_at_end = true

source/requirements.txt

@@ -0,0 +1,20 @@
+# Python >=3.11. Install from PyPI after selecting the appropriate PyTorch build.
+# Windows/Linux CUDA 12.8: install torch==2.10.0 from the cu128 index first;
+# then install this file using the default PyPI index. See docs/training.md.
+numpy>=2,<3; sys_platform != "darwin"
+# Core ML Tools 9.0's Torch scalar casts are incompatible with NumPy 2.4+.
+numpy==2.3.5; sys_platform == "darwin"
+pyarrow==25.0.1
+sentencepiece==0.2.1
+# Keep Windows training frozen; Core ML 9 lists PyTorch 2.7 support.
+torch==2.10.0; sys_platform != "darwin"
+torch==2.7.1; sys_platform == "darwin"
+coremltools==9.0; sys_platform == "darwin"
+# K-means palettes also process FP32 QKV weights in the conservative route.
+scikit-learn==1.5.1; sys_platform == "darwin"
+tensorboard==2.20.0
+# TensorBoard 2.20 imports pkg_resources, removed in newer setuptools.
+setuptools==80.9.0
+wandb==0.30.0
+# Manual model release authentication/upload/download; never runs during training.
+huggingface_hub==2.1.1

source/scripts/benchmarks/evaluate.py

@@ -0,0 +1,11 @@
+"""Score a frozen, verified synthetic benchmark with a local checkpoint."""
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(ROOT / "src"))
+
+from vimeml.benchmarks.evaluate import main
+
+if __name__ == "__main__":
+    main()

source/scripts/benchmarks/evaluate_ajimee.py

@@ -0,0 +1,11 @@
+"""Evaluate a frozen AJIMEE candidate export using the local Tiny Japanese LM."""
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(ROOT / "src"))
+
+from vimeml.benchmarks.evaluate_ajimee import main
+
+if __name__ == "__main__":
+    main()

source/scripts/benchmarks/export_cached_development.py

@@ -0,0 +1,28 @@
+"""Export successful generation caches for review, even if other batches failed."""
+import argparse
+import json
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(ROOT / "src"))
+from vimeml.benchmarks.development import export_cached_drafts
+from vimeml.review.multi_key import output_lock
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--source", type=Path, required=True)
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--exclude", type=Path, default=ROOT / "artifacts/benchmarks/ajimee-jwtd-v2-v1/evaluation_items.json")
+    args = parser.parse_args()
+    try:
+        with output_lock(args.output):
+            stats = export_cached_drafts(args.source, args.output, args.exclude)
+        print(json.dumps(stats, ensure_ascii=False, indent=2))
+    except (OSError, ValueError, KeyError) as error:
+        parser.exit(2, f"Error: {error}\n")
+
+
+if __name__ == "__main__":
+    main()

source/scripts/benchmarks/generate.py

@@ -0,0 +1,11 @@
+"""Generate and blind-check a synthetic benchmark; no training."""
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(ROOT / "src"))
+
+from vimeml.benchmarks.generate import main
+
+if __name__ == "__main__":
+    main()

source/scripts/benchmarks/generate_development.py

@@ -0,0 +1,10 @@
+"""Generate and independently verify kana-conversion development drafts."""
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0,str(ROOT / 'src'))
+from vimeml.benchmarks.development import main
+
+if __name__ == '__main__':
+    main()

source/scripts/benchmarks/hybrid.py

@@ -0,0 +1,10 @@
+"""Prepare development inputs, cache model scores and evaluate hybrid ranking."""
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(ROOT / "src"))
+from vimeml.benchmarks.hybrid import main
+
+if __name__ == "__main__":
+    main()

source/scripts/benchmarks/prepare_ajimee.py

@@ -0,0 +1,11 @@
+"""Prepare downloaded AJIMEE data for the AzooKey CLI (stdlib only)."""
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(ROOT / "src"))
+
+from vimeml.benchmarks.ajimee import main
+
+if __name__ == "__main__":
+    main()

source/scripts/corpus/build.py

@@ -0,0 +1,12 @@
+"""Command entry point; implementation lives in src/vimeml."""
+
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(ROOT / "src"))
+
+from vimeml.corpus.build import main
+
+if __name__ == "__main__":
+    main()

source/scripts/corpus/merge.py

@@ -0,0 +1,33 @@
+"""Merge completed cleaning parts using the verified parallel exporter."""
+
+import argparse
+import os
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(ROOT / "src"))
+from vimeml.data.merge_parts_fast import run
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--config", type=Path, default=ROOT / "configs/corpus-parallel.toml")
+    sources = parser.add_mutually_exclusive_group(required=True)
+    sources.add_argument("--parts", nargs="+", type=Path)
+    sources.add_argument("--parts-dir", type=Path)
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--work", type=Path)
+    parser.add_argument("--workers", type=int, default=min(8, os.cpu_count() or 1))
+    parser.add_argument("--buckets", type=int, default=64, choices=(1, 4, 16, 64, 256))
+    parser.add_argument("--cache-mb", type=int, default=128)
+    args = parser.parse_args()
+    parts = args.parts if args.parts is not None else sorted(p.parent for p in args.parts_dir.glob("*/manifest.json"))
+    if not parts:
+        parser.error("No completed parts found")
+    run(args.config, parts, args.output, workers=args.workers, buckets=args.buckets,
+        cache_mb=args.cache_mb, work=args.work)
+
+
+if __name__ == "__main__":
+    main()

source/scripts/corpus/preprocess.py

@@ -0,0 +1,12 @@
+"""Command entry point; implementation lives in src/vimeml."""
+
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(ROOT / "src"))
+
+from vimeml.data.preprocess_part import main
+
+if __name__ == "__main__":
+    main()

source/scripts/deployment/ane_output_probe.py

@@ -0,0 +1,72 @@
+"""Manual reduced-output ANE experiment; not the deployment interface.
+
+Usage: python scripts/deployment/ane_output_probe.py NEW_OUTPUT.mlpackage
+Conversion, timing and accuracy checks start only when this command is run.
+"""
+import argparse
+import sys
+import time
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(ROOT / "src"))
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("output", type=Path, help="New scratch .mlpackage path")
+    args = parser.parse_args()
+    out = args.output
+    if out.exists():
+        parser.error("Output exists; choose a new experiment path.")
+    if out.suffix != ".mlpackage":
+        parser.error("Output must end with .mlpackage.")
+    if sys.platform != "darwin":
+        parser.error("Run this experiment manually on Mac.")
+    import numpy as np, torch, coremltools as ct
+    from torch import nn
+    from vimeml.deployment.bundle import BundleLM
+    from vimeml.deployment.graph import ConversionGraph
+    lm = BundleLM(ROOT/"artifacts/deployment/tiny-ja-v1-inference-v1")
+    K = 16
+    class Head(nn.Module):
+        def __init__(s, model): super().__init__(); s.g = ConversionGraph(model)
+        def forward(s, input_ids, target_ids):
+            logp = torch.log_softmax(s.g(input_ids), dim=-1)
+            target = torch.gather(logp, 2, target_ids.long().unsqueeze(-1)).squeeze(-1)
+            top, idx = torch.topk(logp, K, dim=-1)
+            return target, top, idx.int()
+    h = Head(lm.model).eval()
+    ex = (torch.zeros(1,16,dtype=torch.int32), torch.zeros(1,16,dtype=torch.int32))
+    tr = torch.jit.trace(h, ex)
+    shapes = ct.EnumeratedShapes(shapes=[(1,L) for L in (16,32,64,128)], default=(1,16))
+    m = ct.convert(tr, convert_to="mlprogram", minimum_deployment_target=ct.target.iOS18, compute_precision=ct.precision.FLOAT16,
+        inputs=[ct.TensorType(name="input_ids", shape=shapes, dtype=np.int32), ct.TensorType(name="target_ids", shape=shapes, dtype=np.int32)],
+        outputs=[ct.TensorType(name="target_logprobs", dtype=np.float32), ct.TensorType(name="top_logprobs", dtype=np.float32), ct.TensorType(name="top_ids", dtype=np.int32)])
+    m.save(str(out))
+    for units in ("CPU_ONLY","CPU_AND_NE"):
+        mm = ct.models.MLModel(str(out), compute_units=getattr(ct.ComputeUnit, units)); row=[units]
+        for L in (16,64,128):
+            x={"input_ids":np.full((1,L),5,np.int32),"target_ids":np.full((1,L),6,np.int32)}; mm.predict(x); ts=[]
+            for _ in range(20): t=time.perf_counter(); mm.predict(x); ts.append((time.perf_counter()-t)*1000)
+            row.append(f"T{L}={np.median(ts):.2f}ms")
+        print(" ".join(row))
+    # accuracy vs FP32 on AJIMEE text
+    import json
+    items=json.load(open(ROOT/"artifacts/benchmarks/ajimee-jwtd-v2-v1/evaluation_items.json"))
+    mm = ct.models.MLModel(str(out), compute_units=ct.ComputeUnit.CPU_AND_NE)
+    diffs=[]; topagree=[]
+    for it in items[:100]:
+        ids=[2]+lm.processor.encode((it.get("context_text") or "")+it["expected_output"][0])[:127]
+        n=len(ids)-1; L=next(b for b in (16,32,64,128) if b>=n)
+        inp=np.zeros((1,L),np.int32); inp[0,:n]=ids[:-1]; tgt=np.zeros((1,L),np.int32); tgt[0,:n]=ids[1:]
+        o=mm.predict({"input_ids":inp,"target_ids":tgt})
+        with torch.no_grad(): ref=lm.model(torch.tensor([ids[:-1]])).float().log_softmax(-1)[0]
+        r=ref.gather(1,torch.tensor(ids[1:])[:,None]).squeeze(1).numpy()
+        diffs.append(abs(o["target_logprobs"][0,:n]-r).max())
+        topagree.append(np.mean(o["top_ids"][0,:n,0]==ref.argmax(-1).numpy()))
+    print(f"ANE target logprob max|diff| mean={np.mean(diffs):.4f} max={np.max(diffs):.4f}; top1 agree={np.mean(topagree):.4f}")
+
+
+if __name__ == "__main__":
+    main()

source/scripts/deployment/compare_quantization.py

@@ -0,0 +1,119 @@
+"""Distribution-level check of a Core ML package against the FP32 bundle.
+
+Distribution diagnostics supplement the strict logits, PAD and ranking checks.
+They do not establish exact conversion equivalence. This reports KL(FP32 || model),
+NLL on real text, top-1 agreement and, as a control, the same metrics for a
+PyTorch simulation of symmetric INT8 block quantization. Read-only.
+"""
+import argparse
+import copy
+import json
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(ROOT / "src"))
+
+import numpy as np
+import torch
+
+from vimeml.deployment.bundle import BundleLM, environment
+from vimeml.deployment.coreml import verify_package
+from vimeml.training.data import file_sha
+
+
+def simulate_int8(model, block):
+    """Mirror the conservative recipe: FP16 storage except QKV/positions, then INT8 per block."""
+    if block <= 0 or any(p.ndim == 2 and p.shape[1] % block for p in model.parameters()):
+        raise ValueError("Block size must be positive and divide every matrix input dimension.")
+    result = copy.deepcopy(model)
+    with torch.no_grad():
+        for name, parameter in result.named_parameters():
+            if parameter.ndim != 2:
+                continue
+            weight = parameter if ("qkv" in name or "position" in name) else parameter.half().float()
+            rows, columns = weight.shape
+            blocks = weight.reshape(rows, columns // block, block)
+            scale = blocks.abs().amax(-1, keepdim=True).clamp_min(1e-12) / 127
+            parameter.copy_((torch.round(blocks / scale).clamp(-127, 127) * scale).reshape(rows, columns))
+    return result
+
+
+def texts(benchmark):
+    items = json.loads((benchmark / "evaluation_items.json").read_text(encoding="utf-8"))
+    return [(item.get("context_text") or "") + item["expected_output"][0] for item in items]
+
+
+def compare(lm, model, mlmodel, sequences):
+    metrics = {key: [] for key in ("kl_coreml", "kl_simulated", "nll_fp32", "nll_coreml",
+                                   "top1_agreement", "max_logprob_diff")}
+    for ids in sequences:
+        tensor = torch.tensor([ids])
+        with torch.no_grad():
+            reference = lm.model(tensor)[0].float().log_softmax(-1)
+            simulated = model(tensor)[0].float().log_softmax(-1)
+        logits = mlmodel.predict({"input_ids": np.array([ids], dtype=np.int32)})["logits"][0]
+        coreml = torch.tensor(logits).log_softmax(-1)
+        probabilities = reference.exp()
+        metrics["kl_coreml"].append((probabilities * (reference - coreml)).sum(-1).mean().item())
+        metrics["kl_simulated"].append((probabilities * (reference - simulated)).sum(-1).mean().item())
+        targets = tensor[0, 1:, None]
+        metrics["nll_fp32"].append(-reference[:-1].gather(-1, targets).mean().item())
+        metrics["nll_coreml"].append(-coreml[:-1].gather(-1, targets).mean().item())
+        metrics["top1_agreement"].append((reference.argmax(-1) == coreml.argmax(-1)).float().mean().item())
+        metrics["max_logprob_diff"].append((reference - coreml).abs().max().item())
+    return {key: {"mean": float(np.mean(values)), "max": float(np.max(values))} for key, values in metrics.items()}
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--bundle", type=Path, default=ROOT / "artifacts/deployment/tiny-ja-v1-inference-v1")
+    parser.add_argument("--model", type=Path, required=True, help="Core ML experiment directory with model.mlpackage")
+    parser.add_argument("--benchmark", type=Path, default=ROOT / "artifacts/benchmarks/ajimee-jwtd-v2-v1")
+    parser.add_argument("--block-size", type=int, default=32)
+    parser.add_argument("--compute-units", choices=("CPU_ONLY", "CPU_AND_GPU", "CPU_AND_NE", "ALL"), default="CPU_ONLY")
+    parser.add_argument("--output", type=Path, help="Optional JSON report path; must not exist.")
+    args = parser.parse_args()
+    if args.output and args.output.exists():
+        parser.error("Output exists.")
+    lm = BundleLM(args.bundle)
+    manifest = verify_package(args.model)
+    if manifest["bundle"]["bundle_manifest_sha256"] != lm.metadata["bundle_manifest_sha256"]:
+        raise ValueError("Core ML model and FP32 bundle identities differ.")
+    if sys.platform != "darwin":
+        parser.error("Run Core ML prediction manually on Mac.")
+    import coremltools as ct
+    torch.set_num_threads(4)
+    mlmodel = ct.models.MLModel(str(args.model / "model.mlpackage"),
+                                compute_units=getattr(ct.ComputeUnit, args.compute_units))
+    simulated = simulate_int8(lm.model, args.block_size).eval()
+    limit = lm.model.config.context_length - 1
+    encoded = [lm.processor.encode(text) for text in texts(args.benchmark)]
+    if not encoded or any(not ids for ids in encoded):
+        raise ValueError("Benchmark must have at least one nonempty target token per example.")
+    real = [[lm.special["bos"], *ids[:limit]] for ids in encoded]
+    rng = np.random.default_rng(0)
+    random = [[lm.special["bos"], *rng.integers(4, lm.model.config.vocab_size, limit).tolist()] for _ in range(5)]
+    report = {"format": "vimeml_distribution_diagnostic_v2", "model": str(args.model),
+              "compute_units": args.compute_units, "environment": environment(),
+              "bundle_manifest_sha256": lm.metadata["bundle_manifest_sha256"],
+              "coreml_manifest_sha256": file_sha(args.model / "manifest.json"),
+              "benchmark_items_sha256": file_sha(args.benchmark / "evaluation_items.json"),
+              "simulation": {"block_size": args.block_size,
+                  "note": "FP32 scales and symmetric +/-127 rounding; not exact Core ML scale storage or kernels."},
+              "sampling": {"examples": len(encoded), "content_token_limit": limit,
+                  "truncated_examples": sum(len(ids) > limit for ids in encoded),
+                  "aggregation": "Mean of each example's position mean; not token-weighted corpus mean.",
+                  "kl_top1_include_last_position": True, "nll_excludes_last_position": True,
+                  "random_sequences": 5, "random_seed": 0},
+              "benchmark_text": compare(lm, simulated, mlmodel, real),
+              "random_tokens": compare(lm, simulated, mlmodel, random)}
+    text = json.dumps(report, ensure_ascii=False, indent=2)
+    print(text)
+    if args.output:
+        args.output.parent.mkdir(parents=True, exist_ok=True)
+        args.output.write_text(text + "\n", encoding="utf-8")
+
+
+if __name__ == "__main__":
+    main()

source/scripts/deployment/compute_plan.py

@@ -0,0 +1,57 @@
+"""Report where Core ML would place each op of a package (CPU / GPU / Neural Engine).
+
+Uses MLComputePlan on this Mac. Placement on iPhone can differ; confirm with a device run.
+"""
+import argparse
+import json
+import shutil
+import tempfile
+from collections import Counter
+from pathlib import Path
+
+
+def plan(package, units):
+    import coremltools as ct
+    from coremltools.models.compute_plan import MLComputePlan
+    from coremltools.models.compute_device import MLCPUComputeDevice, MLGPUComputeDevice, MLNeuralEngineComputeDevice
+
+    names = {MLCPUComputeDevice: "cpu", MLGPUComputeDevice: "gpu", MLNeuralEngineComputeDevice: "ane"}
+    temporary = Path(tempfile.mkdtemp())
+    try:
+        compiled = ct.utils.compile_model(str(package), str(temporary / "model.mlmodelc"))
+        compute_plan = MLComputePlan.load_from_path(path=str(compiled), compute_units=getattr(ct.ComputeUnit, units))
+        program = compute_plan.model_structure.program
+        placement, cost, details = Counter(), Counter(), []
+        for function in program.functions.values():
+            for operation in function.block.operations:
+                usage = compute_plan.get_compute_device_usage_for_mlprogram_operation(operation)
+                if usage is None:
+                    continue
+                device = names.get(type(usage.preferred_compute_device), "other")
+                estimate = compute_plan.get_estimated_cost_for_mlprogram_operation(operation)
+                weight = estimate.weight if estimate else 0.0
+                placement[device] += 1
+                cost[device] += weight
+                details.append({"op": operation.operator_name, "device": device, "cost": weight,
+                                "supported": sorted({names.get(type(d), "other")
+                                                     for d in usage.supported_compute_devices})})
+        return {"package": str(package), "compute_units": units, "ops_by_device": dict(placement),
+                "estimated_cost_by_device": {k: round(v, 4) for k, v in cost.items()}, "ops": details}
+    finally:
+        shutil.rmtree(temporary, ignore_errors=True)
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--model", type=Path, required=True, help="Experiment directory with model.mlpackage")
+    parser.add_argument("--compute-units", default="CPU_AND_NE", choices=("CPU_ONLY", "CPU_AND_GPU", "CPU_AND_NE", "ALL"))
+    parser.add_argument("--details", action="store_true", help="Print every op")
+    args = parser.parse_args()
+    report = plan(args.model / "model.mlpackage", args.compute_units)
+    if not args.details:
+        report.pop("ops")
+    print(json.dumps(report, indent=2))
+
+
+if __name__ == "__main__":
+    main()

source/scripts/deployment/coreml.py

@@ -0,0 +1,11 @@
+"""Manual deployment CLI: no training, API calls or automatic stage chaining."""
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(ROOT / "src"))
+
+from vimeml.deployment.cli import main
+
+if __name__ == "__main__":
+    main()

source/scripts/deployment/coreml_ane.py

@@ -0,0 +1,91 @@
+"""Neural Engine variant: FP16 computation, enumerated input lengths, optional INT8 weights.
+
+The conservative package computes in FP32, which the Neural Engine cannot execute, so every
+op falls back to CPU. This entry converts the same traced graph with FP16 computation and a
+small set of fixed lengths (callers right-pad to the next length; PAD never affects valid
+positions under the causal mask), then optionally applies per-channel INT8 weight quantization.
+"""
+import argparse
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(ROOT / "src"))
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+
+import numpy as np
+import torch
+
+from vimeml.deployment.bundle import BundleLM
+from vimeml.deployment.coreml import coremltools, finish, inspect_spec, verify_package
+from vimeml.deployment.graph import trace_graph
+from vimeml.training.data import file_sha
+from coreml_conservative import parameter_compression_config
+
+LENGTHS = (16, 32, 64, 128)
+
+
+def convert(bundle, output, lengths):
+    ct = coremltools()
+    lm = BundleLM(bundle)
+    shapes = ct.EnumeratedShapes(shapes=[(1, length) for length in lengths], default=(1, lengths[0]))
+    model = ct.convert(trace_graph(lm.model), source="pytorch", convert_to="mlprogram",
+        minimum_deployment_target=ct.target.iOS18, compute_precision=ct.precision.FLOAT16,
+        inputs=[ct.TensorType(name="input_ids", shape=shapes, dtype=np.int32)],
+        outputs=[ct.TensorType(name="logits", dtype=np.float32)], skip_model_load=True)
+    model.short_description = "Frozen TinyGPT; FP16 computation for the Neural Engine; enumerated lengths; no KV cache."
+    model.user_defined_metadata["bundle_manifest_sha256"] = lm.metadata["bundle_manifest_sha256"]
+    finish(output, model, {"kind": "fp16_ane", "minimum_ios": 18, "bundle": lm.metadata, "lengths": list(lengths),
+        "conversion_script_sha256": file_sha(Path(__file__)),
+        "interface": {"input_ids": f"int32 [1,T], T in {list(lengths)}; right-pad with PAD=0",
+                      "logits": "float32 [1,T,16384]; FP16 computation; no softmax"}})
+
+
+def compress(source, output):
+    ct = coremltools()
+    original = verify_package(source)
+    if original["kind"] != "fp16_ane":
+        raise ValueError("Compress an uncompressed fp16_ane conversion.")
+    from coremltools.optimize import coreml as opt
+    model = ct.models.MLModel(str(source / "model.mlpackage"), skip_model_load=True)
+    config = opt.OpLinearQuantizerConfig(mode="linear_symmetric", dtype="int8",
+                                         granularity="per_channel", weight_threshold=2048)
+    optimization_config, selection = parameter_compression_config(opt, model, config)
+    result = opt.linear_quantize_weights(model, config=optimization_config)
+    if not any(name.startswith("constexpr_") for name in inspect_spec(result)["operations"]):
+        raise ValueError("No compressed constexpr weights found.")
+    finish(output, result, {"kind": "linear8_fp16_ane", "minimum_ios": 18, "bundle": original["bundle"],
+        "lengths": original["lengths"], "interface": original["interface"],
+        "source_manifest_sha256": file_sha(source / "manifest.json"),
+        "conversion_script_sha256": file_sha(Path(__file__)),
+        "compression": {"method": "linear", "bits": 8, "granularity": "per_channel", "weight_threshold": 2048,
+                        "selection": selection,
+                        "scope": "Learned matrices only; masks and small constants unchanged; FP16 activations."}})
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    sub = parser.add_subparsers(dest="command", required=True)
+    convert_parser = sub.add_parser("convert")
+    convert_parser.add_argument("--bundle", type=Path, required=True)
+    convert_parser.add_argument("--lengths", type=int, nargs="+", default=list(LENGTHS))
+    compress_parser = sub.add_parser("compress")
+    compress_parser.add_argument("--source", type=Path, required=True)
+    for command in (convert_parser, compress_parser):
+        command.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    if args.output.exists():
+        parser.error("Output exists; use a new versioned directory.")
+    torch.set_num_threads(4)
+    if args.command == "convert":
+        lengths = sorted(set(args.lengths))
+        if lengths[0] < 1 or lengths[-1] != 128:
+            parser.error("Lengths must be positive and include 128.")
+        convert(args.bundle, args.output, lengths)
+    else:
+        compress(args.source, args.output)
+    print(f"Output: {args.output.resolve()}")
+
+
+if __name__ == "__main__":
+    main()

source/scripts/deployment/coreml_conservative.py

@@ -0,0 +1,229 @@
+"""FP16 weight-storage baseline and weight-only quantization with FP32 computation.
+
+Kept outside the frozen deployment sources so an existing inference bundle
+remains verifiable. This converter's hash and exact precision policy are saved
+in the package manifest. Stages preserve prior experiment directories.
+"""
+import argparse
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(ROOT / "src"))
+
+import numpy as np
+import torch
+
+from vimeml.deployment.bundle import BundleLM, read_json
+from vimeml.deployment.coreml import coremltools, finish, inspect_spec, verify_package
+from vimeml.deployment.graph import trace_graph
+from vimeml.training.data import file_sha
+
+
+def weight_storage_graph(ct, traced, context_length, target=18):
+    from coremltools.converters.mil import Builder as mb
+    from coremltools.converters.mil.mil.scope import ScopeInfo
+
+    program = ct.convert(traced, source="pytorch", convert_to="milinternal",
+        minimum_deployment_target=getattr(ct.target, f"iOS{target}"), compute_precision=ct.precision.FLOAT32,
+        inputs=[ct.TensorType(name="input_ids", shape=(1, ct.RangeDim(lower_bound=1,
+            upper_bound=context_length, default=min(16, context_length))), dtype=np.int32)],
+        outputs=[ct.TensorType(name="logits", dtype=np.float32)])
+    changed = {"fp16_linear_layers": 0, "fp16_token_embedding_tables": 0,
+               "fp32_qkv_layers": 0, "fp32_position_embedding_tables": 0}
+    for function in program.functions.values():
+        with function:
+            for op in list(function.operations):
+                if op.op_type == "linear":
+                    if "attention.qkv" in op.weight.name:
+                        changed["fp32_qkv_layers"] += 1
+                        continue
+                    with mb.scope(*[ScopeInfo(source=source, data=data)
+                                    for source, data in op.scopes.items()]), mb.set_before_op(op):
+                        weight = mb.const(val=op.weight.val.astype(np.float16),
+                                          name=op.weight.name + "_storage_fp16")
+                        arguments = {"weight": weight}
+                        if op.bias is not None:
+                            arguments["bias"] = mb.const(val=op.bias.val.astype(np.float16),
+                                                         name=op.bias.name + "_storage_fp16")
+                        result = mb.linear(x=op.x, name=op.name + "_compute_fp32", **arguments)
+                    function.replace_uses_of_var_after_op(anchor_op=op,
+                        old_var=op.outputs[0], new_var=result)
+                    function.remove_ops([op])
+                    changed["fp16_linear_layers"] += 1
+                elif (op.op_type == "gather" and op.x.val is not None and
+                      op.x.val.dtype == np.float32):
+                    if "position_embedding" in op.x.name:
+                        changed["fp32_position_embedding_tables"] += 1
+                        continue
+                    if "token_embedding" not in op.x.name:
+                        raise ValueError("Unexpected floating-point gather table; inspect the graph.")
+                    with mb.scope(*[ScopeInfo(source=source, data=data)
+                                    for source, data in op.scopes.items()]), mb.set_before_op(op):
+                        table = mb.const(val=op.x.val.astype(np.float16),
+                                         name=op.x.name + "_storage_fp16")
+                        gathered = mb.gather(x=table, indices=op.indices, axis=op.axis,
+                                             name=op.name + "_storage_fp16")
+                        result = mb.cast(x=gathered, dtype="fp32", name=op.name + "_compute_fp32")
+                    function.replace_uses_of_var_after_op(anchor_op=op,
+                        old_var=op.outputs[0], new_var=result)
+                    function.remove_ops([op])
+                    changed["fp16_token_embedding_tables"] += 1
+    if not all(changed.values()):
+        raise ValueError(f"Expected TinyGPT precision boundaries were not found: {changed}")
+    program.functions["main"].outputs[0].set_name("logits")
+    return program, changed
+
+
+def convert(bundle, output, target=18):
+    ct = coremltools()
+    if output.exists():
+        raise ValueError("Output exists; choose a new conversion experiment.")
+    lm = BundleLM(bundle)
+    program, changed = weight_storage_graph(ct, trace_graph(lm.model), lm.model.config.context_length, target)
+    model = ct.convert(program, convert_to="mlprogram", minimum_deployment_target=getattr(ct.target, f"iOS{target}"),
+                       compute_precision=ct.precision.FLOAT32, skip_model_load=True)
+    policy = {"recipe": "fp16_weights_fp32_compute_preserve_positions_qkv_v1",
+              "computation": "FP32; token embedding gather is FP16 then cast to FP32",
+              "fp16_parameters": "Token embedding; non-QKV linear weights and biases, including LM head",
+              "fp32_parameters": "Position embedding; attention QKV weights/biases; LayerNorm parameters",
+              "graph_changes": changed}
+    model.short_description = "Frozen TinyGPT; mixed FP16/FP32 weights, FP32 computation; no KV cache."
+    model.user_defined_metadata["bundle_manifest_sha256"] = lm.metadata["bundle_manifest_sha256"]
+    model.user_defined_metadata["precision_recipe"] = policy["recipe"]
+    finish(output, model, {"kind": "fp16_weights_fp32_compute", "minimum_ios": target, "bundle": lm.metadata,
+        "precision_policy": policy, "conversion_script_sha256": file_sha(Path(__file__)),
+        "interface": {"input_ids": "int32 [1,T], 1<=T<=128; right PAD only",
+                      "logits": "float32 [1,T,16384]; FP32 computation; no softmax"}})
+
+
+def parameter_compression_config(opt, model, config):
+    """Allow only parameter matrices; global defaults must leave other consts intact."""
+    metadata = opt.get_weights_metadata(model, weight_threshold=config.weight_threshold)
+    selected, excluded = [], []
+    configs = {}
+    for name, entry in metadata.items():
+        value = entry.val
+        consumers = [{"name": child.name, "type": child.op_type,
+                      "inputs": dict(child.params_name_mapping)} for child in entry.child_ops]
+        record = {"name": name, "shape": [int(dim) for dim in value.shape],
+                  "dtype": str(value.dtype), "elements": int(value.size), "consumers": consumers}
+        normalized_name = name.replace(".", "_")
+        embedding = normalized_name.startswith(("model_token_embedding_weight",
+                                               "model_position_embedding_weight"))
+        parameter_consumers = all(
+            (child.op_type == "linear" and child.params_name_mapping.get("weight") == name) or
+            (embedding and child.op_type == "gather" and child.params_name_mapping.get("x") == name)
+            for child in entry.child_ops)
+        if (not consumers or not parameter_consumers or value.ndim != 2 or
+                not np.issubdtype(value.dtype, np.floating)):
+            record.update(reason="Not a linear weight or named embedding parameter",
+                          nonfinite_elements=int((~np.isfinite(value)).sum()))
+            excluded.append(record)
+            continue
+        if value.size <= config.weight_threshold:
+            record["reason"] = "Does not exceed weight threshold"
+            excluded.append(record)
+            continue
+        if not np.isfinite(value).all():
+            raise ValueError(f"Nonfinite learned parameter {name}; refusing compression.")
+        selected.append(record)
+        configs[name] = config
+    if not selected:
+        raise ValueError("No eligible learned parameter matrices found; inspect the model graph.")
+    selection = {"policy": "linear_weights_and_named_embedding_tables_v1",
+                 "selected_constants": selected, "excluded_constants": excluded,
+                 "selected_elements": sum(item["elements"] for item in selected)}
+    print(f"Compression selection: {len(selected)} parameter matrices; "
+          f"{len(excluded)} large constants excluded.", flush=True)
+    for item in excluded:
+        print(f"  Excluded {item['name']}: {item['reason']}", flush=True)
+    return opt.OptimizationConfig(global_config=None, op_name_configs=configs), selection
+
+
+def compress(source, output, validation, method, bits, group_size, block_size, granularity="per_block"):
+    """Same exact-package alignment gate, extended to this declared source kind."""
+    ct = coremltools()
+    if output.exists():
+        raise ValueError("Output exists; choose a new compression experiment.")
+    original = verify_package(source)
+    if original["kind"] != "fp16_weights_fp32_compute":
+        raise ValueError("Use this command only with an uncompressed conservative conversion.")
+    gate = read_json(validation)
+    if (gate.get("format") != "vimeml_alignment_v1" or not gate.get("passed") or
+            gate.get("coreml_manifest_sha256") != file_sha(source / "manifest.json")):
+        raise ValueError("First pass alignment for this exact source and supply its alignment.json.")
+    if method == "palette" and group_size and original["minimum_ios"] < 18:
+        raise ValueError("Grouped palette requires iOS18; use per-tensor palette or linear INT8 on iOS17.")
+    if method == "linear" and (granularity == "per_block" or bits == 4) and original["minimum_ios"] < 18:
+        raise ValueError("Blockwise/4bit linear quantization requires iOS18; use INT8 per-channel on iOS17.")
+    from coremltools.optimize import coreml as opt
+    if method == "palette":
+        try:
+            from sklearn.cluster import KMeans  # noqa: F401: required by Core ML k-means fallback
+        except ImportError as error:
+            raise RuntimeError("Install the Mac requirements first; palette compression needs "
+                               "scikit-learn==1.5.1 (uv pip install --python venv/coreml/bin/python "
+                               "scikit-learn==1.5.1).") from error
+    model = ct.models.MLModel(str(source / "model.mlpackage"), skip_model_load=True)
+    if method == "palette":
+        config = opt.OpPalettizerConfig(mode="kmeans", nbits=bits, weight_threshold=2048,
+            granularity="per_grouped_channel" if group_size else "per_tensor", group_size=group_size or 32)
+    else:
+        config = opt.OpLinearQuantizerConfig(mode="linear_symmetric", dtype=f"int{bits}",
+            granularity=granularity, block_size=block_size if granularity == "per_block" else 0, weight_threshold=2048)
+    optimization_config, selection = parameter_compression_config(opt, model, config)
+    if method == "palette":
+        result = opt.palettize_weights(model, config=optimization_config)
+    else:
+        result = opt.linear_quantize_weights(model, config=optimization_config)
+    if not any(name.startswith("constexpr_") for name in inspect_spec(result)["operations"]):
+        raise ValueError("No compressed constexpr weights found.")
+    finish(output, result, {"kind": f"{method}{bits}_fp32_compute", "minimum_ios": original["minimum_ios"],
+        "bundle": original["bundle"], "interface": original["interface"],
+        "source_manifest_sha256": file_sha(source / "manifest.json"),
+        "fp16_alignment_sha256": file_sha(validation),
+        "conversion_script_sha256": file_sha(Path(__file__)),
+        "precision_policy": {"source_recipe": original["precision_policy"]["recipe"],
+                             "computation": original["precision_policy"]["computation"]},
+        "compression": {"method": method, "bits": bits, "group_size": group_size,
+                        "block_size": block_size if method == "linear" and granularity == "per_block" else None,
+                        "granularity": granularity if method == "linear" else "per_grouped_channel" if group_size else "per_tensor",
+                        "weight_threshold": 2048,
+                        "selection": selection,
+                        "scope": "Eligible learned matrices, including FP32 QKV; structural constants/masks unchanged; no activation quantization or retraining."}})
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--threads", type=int, default=4)
+    sub = parser.add_subparsers(dest="command", required=True)
+    convert_parser = sub.add_parser("convert", help="Mixed weight storage and FP32 computation, iOS18.")
+    convert_parser.add_argument("--bundle", type=Path, required=True)
+    convert_parser.add_argument("--target", type=int, choices=(17, 18), default=18)
+    compress_parser = sub.add_parser("compress", help="Compress this exact validated conservative baseline.")
+    compress_parser.add_argument("--source", type=Path, required=True)
+    compress_parser.add_argument("--fp16-alignment", type=Path, required=True)
+    compress_parser.add_argument("--method", choices=("palette", "linear"), default="palette")
+    compress_parser.add_argument("--bits", type=int, choices=(4, 8), default=4)
+    compress_parser.add_argument("--group-size", type=int, choices=(0, 8, 16, 32), default=16)
+    compress_parser.add_argument("--block-size", type=int, choices=(16, 32, 64, 128), default=32)
+    compress_parser.add_argument("--granularity", choices=("per_channel", "per_block"), default="per_block")
+    for command in (convert_parser, compress_parser):
+        command.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    if args.threads < 1:
+        parser.error("threads must be positive.")
+    if args.output.exists():
+        parser.error("Output exists; use a new versioned directory.")
+    torch.set_num_threads(args.threads)
+    if args.command == "convert":
+        convert(args.bundle, args.output, args.target)
+    else:
+        compress(args.source, args.output, args.fp16_alignment, args.method, args.bits,
+                 args.group_size, args.block_size, args.granularity)
+    print(f"Output: {args.output.resolve()}")
+
+
+if __name__ == "__main__":
+    main()

source/scripts/deployment/package_handoff.py

@@ -0,0 +1,117 @@
+"""Archive Mac project snapshots, experiment evidence and Vime client for local handoff."""
+import argparse
+import hashlib
+import json
+import subprocess
+import zipfile
+from pathlib import Path
+from datetime import date
+
+
+SKIP_PARTS = {'.git', '__pycache__', '.pytest_cache', '.build', '.swiftpm', 'xcuserdata'}
+SKIP_NAMES = {'.DS_Store'}
+
+
+def digest(source):
+    if isinstance(source, bytes):
+        return hashlib.sha256(source).hexdigest()
+    with source.open('rb') as stream:
+        return hashlib.file_digest(stream, 'sha256').hexdigest()
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--project', type=Path, required=True)
+    parser.add_argument('--client', type=Path, required=True)
+    parser.add_argument('--output', type=Path, required=True)
+    args = parser.parse_args()
+    root, client, out = args.project.resolve(), args.client.resolve(), args.output.resolve()
+    if out.exists():
+        raise SystemExit('Output already exists; use a new handoff version.')
+    out.mkdir(parents=True)
+    manifest = {'format': 'vimeml_mac_handoff_v1', 'date': date.today().isoformat(),
+                'vime_git_head': subprocess.check_output(['git', 'rev-parse', 'HEAD'], cwd=client, text=True).strip(),
+                'vime_git_status': subprocess.check_output(['git', 'status', '--short'], cwd=client, text=True),
+                'archives': {}, 'excluded': [],
+                'policy': 'Local transfer only; snapshots are not automatic merge instructions or Git commits.'}
+
+    def entries(base, roots, prefix):
+        files = []
+        for name in roots:
+            source = base / name
+            if not source.exists():
+                continue
+            paths = [source] if source.is_file() else sorted(source.rglob('*'))
+            for path in paths:
+                relative = path.relative_to(base)
+                if not path.is_file():
+                    continue
+                if (SKIP_PARTS.intersection(relative.parts) or path.name in SKIP_NAMES or path.suffix == '.pyc'
+                        or path.name == '.env' or path.name.startswith('.env.')
+                        or path.name in {'.pypirc', '.envrc'}):
+                    manifest['excluded'].append(str(Path(prefix) / relative))
+                    continue
+                if path.is_symlink():
+                    raise SystemExit(f'Unexpected symlink requires review: {path}')
+                files.append(((Path(prefix) / relative).as_posix(), path))
+        return files
+
+    def archive(name, files):
+        indexed = []
+        with zipfile.ZipFile(out / name, 'w', compression=zipfile.ZIP_DEFLATED, compresslevel=3) as zip_file:
+            for relative, data in files:
+                value = digest(data)
+                if isinstance(data, bytes):
+                    zip_file.writestr(relative, data)
+                    size = len(data)
+                else:
+                    zip_file.write(data, relative)
+                    size = data.stat().st_size
+                indexed.append({'path': relative, 'bytes': size, 'sha256': value})
+        # Verify every entry against its source hash, including models and raw trace files.
+        with zipfile.ZipFile(out / name) as zip_file:
+            for entry in indexed:
+                with zip_file.open(entry['path']) as stream:
+                    value = hashlib.file_digest(stream, 'sha256').hexdigest()
+                if value != entry['sha256']:
+                    raise SystemExit(f'Archive verification failed: {entry["path"]}')
+        manifest['archives'][name] = {'file_count': len(indexed),
+                                     'uncompressed_bytes': sum(e['bytes'] for e in indexed),
+                                     'archive_bytes': (out / name).stat().st_size,
+                                     'entries': indexed}
+        print(f'{name}: {len(indexed)} files, {(out / name).stat().st_size / 1048576:.1f} MiB; SHA256 verified', flush=True)
+
+    source_roots = ['annotations', 'configs', 'docs', 'examples', 'scripts', 'src', 'tests']
+    source_roots += [p.name for p in root.iterdir() if p.is_file() and p.name not in SKIP_NAMES]
+    archive('VimeML-source.zip', entries(root, source_roots, 'VimeML'))
+    results_roots = ['outputs', 'runs'] + [str(p.relative_to(root)) for p in (root / 'artifacts').iterdir()
+                                          if p.name not in {'models', 'token-data'}]
+    archive('VimeML-results.zip', entries(root, results_roots, 'VimeML'))
+    archive('VimeML-training-inputs.zip', entries(root, ['artifacts/models', 'artifacts/token-data'], 'VimeML'))
+
+    client_paths = subprocess.check_output(['git', 'ls-files', '-z', '--cached', '--others', '--exclude-standard'], cwd=client).decode().split('\0')
+    client_entries = entries(client, sorted(set(p for p in client_paths if p)), 'Vime')
+    patch = subprocess.check_output(['git', 'diff', '--binary', 'HEAD', '--', '.', ':(exclude)**/xcuserdata/**'], cwd=client)
+    client_entries += [('CLIENT_TRACKED_CHANGES.patch', patch),
+                       ('CLIENT_GIT_STATUS.txt', manifest['vime_git_status'].encode()),
+                       ('CLIENT_GIT_HEAD.txt', (manifest['vime_git_head'] + '\n').encode()),
+                       ('Vime/Docs/MacStageHandoff.md', (root / 'docs/reports/mac-20261006/handoff.md').read_bytes()),
+                       ('Vime/Docs/iPhoneKeyboardMemory.md', (root / 'docs/reports/mac-20261006/iphone-keyboard-memory.md').read_bytes())]
+    archive('Vime-client.zip', client_entries)
+    manifest['not_archived_roots'] = ['VimeML/venv (rebuild from requirements and environment snapshot)',
+                                      'VimeML/handoff (this generated output)',
+                                      'Vime/.git (base commit and patch preserved instead)']
+    (out / 'README.md').write_bytes((root / 'docs/reports/mac-20261006/handoff.md').read_bytes())
+    (out / 'MANIFEST.json').write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + '\n')
+    sums = []
+    for path in sorted(out.iterdir()):
+        if path.is_file():
+            with path.open('rb') as stream:
+                value = hashlib.file_digest(stream, 'sha256').hexdigest()
+            sums.append(f'{value}  {path.name}')
+    (out / 'SHA256SUMS.txt').write_text('\n'.join(sums) + '\n')
+    print(f'Handoff ready: {out}', flush=True)
+
+
+if __name__ == '__main__':
+    main()

source/scripts/deployment/package_ios.py

@@ -0,0 +1,76 @@
+"""Package an evaluated INT8 candidate-reranking deployment without overwriting experiments."""
+import argparse
+import json
+import shutil
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(ROOT / "src"))
+from vimeml.deployment.coreml import verify_package
+from vimeml.training.data import file_sha
+
+
+def package(source, compiled, tokenizer, review, output):
+    if output.exists():
+        raise ValueError("Output exists; choose a new deployment version.")
+    model = verify_package(source)
+    decision = json.loads(review.read_text())
+    if (model["kind"] != "linear8_fp32_compute" or model["minimum_ios"] != 18 or
+            model["compression"]["bits"] != 8 or
+            decision["source_manifest_sha256"] != file_sha(source / "manifest.json") or
+            decision["decision"] != "integrate_cpu_candidate_reranking_for_simulator_validation"):
+        raise ValueError("Review/source identity or reviewed INT8 deployment policy mismatch.")
+    if file_sha(tokenizer) != model["bundle"]["tokenizer_sha256"]:
+        raise ValueError("Wrong tokenizer for this frozen model.")
+    if not (compiled / "coremldata.bin").is_file():
+        raise ValueError("Supply an iOS18 coremlcompiler output directory.")
+    inventory = {p.relative_to(compiled).as_posix(): file_sha(p)
+                 for p in sorted(compiled.rglob("*")) if p.is_file()}
+    output.mkdir(parents=True)
+    resources = output / "Resources"
+    resources.mkdir()
+    shutil.copytree(compiled, resources / "TinyJapaneseINT8.mlmodelc")
+    shutil.copyfile(tokenizer, resources / "VimeJapaneseTokenizer.model")
+    manifest = {"format": "vime_ios_lm_v1", "minimum_ios": 18, "vocab_size": 16384,
+        "context_length": 128, "compute_units": "CPU_ONLY", "tokenizer_sha256": file_sha(tokenizer),
+        "compiled_files_sha256": inventory, "source_manifest_sha256": file_sha(source / "manifest.json"),
+        "bundle_manifest_sha256": model["bundle"]["bundle_manifest_sha256"], "kind": model["kind"],
+        "quantization": decision["quantization"],
+        "strict_fp32_logits_alignment_passed": decision["strict_fp32_alignment_passed"],
+        "quality_review": "See the dated review.json; this packaging action does not validate the current client or installed device build.",
+        "integration_snapshot": "examples/ios/integration: first candidate-reranking integration, not the latest Vime client"}
+    (resources / "VimeLMManifest.json").write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n")
+    shutil.copyfile(review, output / "review.json")
+    shutil.copyfile(source / "manifest.json", output / "coreml-source-manifest.json")
+    shutil.copytree(ROOT / "examples/ios/VimeSentencePiece", output / "Dependencies/VimeSentencePiece")
+    shutil.copytree(ROOT / "examples/ios/integration", output / "Integration")
+    (output / "README.md").write_text("""# Vime iOS18 INT8 resource/reference package
+
+Weight-only symmetric INT8 block32, FP32 computation, CPU_ONLY, iOS18.
+Resources and tokenizer hashes are in Resources/VimeLMManifest.json.
+Official SentencePiece0.2.1 and third-party license notices are preserved.
+
+review.json is the dated first-integration review, not a fresh device acceptance.
+Strict FP32 logits alignment failed; fixture/PAD/causal and dev/AJIMEE evidence
+supported a limited candidate-reranking integration. Packaging does not run tests.
+
+Integration/ is the first examples/ios/integration snapshot. It predates the latest
+Vime client's sentence context, next-word suggestions, preferences and cancellation.
+Use the latest Vime repository (or the archived Vime-client.zip) for those behaviors.
+This package does not attest installed build identity, enabled settings or device stability.
+See docs/coreml.md and docs/reports/mac-20261006/ for the subsequent device evidence.
+""", encoding="utf-8")
+    print(output.resolve())
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    for name in ("source", "compiled", "tokenizer", "review", "output"):
+        parser.add_argument("--" + name, type=Path, required=True)
+    args = parser.parse_args()
+    package(args.source, args.compiled, args.tokenizer, args.review, args.output)
+
+
+if __name__ == "__main__":
+    main()

source/scripts/deployment/summarize_keyboard_trace.py

@@ -0,0 +1,77 @@
+"""Summarize exported Instruments sysmon-process XML for Vime processes."""
+import argparse
+import json
+import statistics
+import xml.etree.ElementTree as ET
+from pathlib import Path
+
+
+def summarize(source):
+    root = ET.parse(source).getroot()
+    columns = [c.findtext('mnemonic') for c in root.find('.//schema').findall('col')]
+    references = {e.attrib['id']: e for e in root.iter() if 'id' in e.attrib}
+
+    def resolve(element):
+        while 'ref' in element.attrib:
+            element = references[element.attrib['ref']]
+        return element
+
+    def number(element):
+        element = resolve(element)
+        return None if element.tag == 'sentinel' or element.text is None else float(element.text)
+
+    processes = {}
+    for row in root.findall('.//row'):
+        values = dict(zip(columns, row))
+        label = resolve(values['process']).get('fmt', '')
+        if not (label.startswith('VimeKeyboard (') or label.startswith('Vime (')):
+            continue
+        footprint = number(values['memory-physical-footprint'])
+        if footprint is None:
+            continue
+        item = {'seconds': number(values['time']) / 1e9,
+                'physical_mib': footprint / 1048576,
+                'resident_mib': number(values['memory-resident-size']) / 1048576,
+                'private_resident_mib': number(values['memory-real-private']) / 1048576,
+                'shared_resident_mib': number(values['memory-real-shared']) / 1048576,
+                'threads': number(values['thread-count']),
+                'recently_died': bool(number(values['recently-died']))}
+        processes.setdefault(label, []).append(item)
+    summary = {}
+    for label, samples in processes.items():
+        samples.sort(key=lambda s: s['seconds'])
+        peak = max(samples, key=lambda s: s['physical_mib'])
+        windows = []
+        for start in range(0, int(samples[-1]['seconds']) + 1, 30):
+            selected = [s['physical_mib'] for s in samples if start <= s['seconds'] < start + 30]
+            if selected:
+                windows.append({'start_seconds': start, 'end_seconds': start + 30,
+                                'samples': len(selected), 'min_mib': min(selected),
+                                'median_mib': statistics.median(selected), 'max_mib': max(selected)})
+        intervals = [b['seconds'] - a['seconds'] for a, b in zip(samples, samples[1:])]
+        summary[label] = {'sample_count': len(samples), 'first': samples[0], 'last': samples[-1],
+                          'sampled_peak': peak, 'sampled_min_mib': min(s['physical_mib'] for s in samples),
+                          'resident_metrics': {key: {'first_mib': samples[0][key],
+                                                     'sampled_peak_mib': max(s[key] for s in samples),
+                                                     'last_mib': samples[-1][key]}
+                                               for key in ('private_resident_mib', 'shared_resident_mib', 'resident_mib')},
+                          'median_interval_seconds': statistics.median(intervals) if intervals else None,
+                          'recently_died_samples': sum(s['recently_died'] for s in samples),
+                          'windows': windows, 'samples': samples}
+    return {'source': str(source), 'metric': 'sysmon physical footprint; sampled peak, not lifetime high-water mark',
+            'processes': summary}
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--input', type=Path, required=True)
+    parser.add_argument('--output', type=Path, required=True)
+    args = parser.parse_args()
+    if args.output.exists():
+        parser.error("Output exists; choose a new report path.")
+    report = summarize(args.input)
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    with args.output.open('x', encoding='utf-8') as stream:
+        stream.write(json.dumps(report, indent=2) + '\n')
+    print(json.dumps({k: {key: value for key, value in v.items() if key != 'samples'}
+                      for k, v in report['processes'].items()}, indent=2))

source/scripts/deployment/summarize_memory.py

@@ -0,0 +1,59 @@
+"""Summarize preserved fresh-process simulator memory audits (MiB, not iPhone estimates)."""
+import argparse
+import json
+from pathlib import Path
+
+
+def summarize(directory):
+    reports = {name: json.loads((directory / (name + '.json')).read_text())
+               for name in ('direct', 'reload', 'engine', 'lm')}
+    repeat = directory / 'lm_words-repeat.json'
+    if repeat.exists():
+        reports['lm_words'] = json.loads(repeat.read_text())
+    rows = []
+    for name in ('engine', 'lm', 'lm_words'):
+        if name not in reports:
+            continue
+        report = reports[name]
+        steady = next(s for s in report['stages'] if s['stage'] == 'worker_cycles_61_80')['after']['physical_mib']
+        values = [x['memory']['physical_mib'] for x in report['checkpoints']]
+        rows.append({'scenario': name, 'baseline_mib': report['baseline']['physical_mib'],
+                     'warm_after_80_cycles_mib': steady, 'process_peak_mib': report['final']['process_lifetime_peak_mib'],
+                     'warm_checkpoints_range_mib': max(values) - min(values),
+                     'after_worker_release_mib': report['final']['physical_mib'],
+                     'cycles_with_suggestions': report.get('cycles_with_published_suggestions')})
+    direct = reports['direct']
+    deltas = {}
+    for phase in ('scores', 'next_words'):
+        values = [x['memory']['physical_mib'] for x in direct['checkpoints'] if x.get('stage') == phase]
+        deltas[phase + '_20_to_100_growth_mib'] = values[-1] - values[0]
+    reload_values = [x['memory']['physical_mib'] for x in reports['reload']['checkpoints']]
+    prior = json.loads((directory / 'prior-keyboard-trace.json').read_text())
+    failed = directory / 'lm_words-failed.json'
+    summary = {'scope': 'optimized iOS27.0 arm64 Simulator test host, CPU_ONLY; not actual keyboard extension',
+               'rows': rows, 'bounded_growth': deltas,
+               'reload_1_to_20_growth_mib': reload_values[-1] - reload_values[0],
+               'reload_10_to_20_growth_mib': reload_values[-1] - reload_values[9],
+               'first_lm_words_failed': failed.exists(),
+               'prior_extension_sampled_peak_mib': prior['peak_physical_mib'],
+               'prior_extension_lm_loaded_proven': False,
+               'limitations': ['Single runs with varying baselines; no fixed keyboard memory limit inferred.',
+                   'No new device connection, installation or recording.',
+                   'A bounded plateau does not exclude all leaks.',
+                   'Prior extension trace settings/source/workload do not prove LM-active measurement.']}
+    return summary
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--directory', type=Path, required=True)
+    parser.add_argument('--output', type=Path, help='New report path; default DIRECTORY/summary.json')
+    args = parser.parse_args()
+    output = args.output or args.directory / 'summary.json'
+    if output.exists():
+        parser.error('Output exists; use --output with a new report path.')
+    report = summarize(args.directory)
+    output.parent.mkdir(parents=True, exist_ok=True)
+    with output.open('x', encoding='utf-8') as stream:
+        stream.write(json.dumps(report, indent=2) + '\n')
+    print(json.dumps(report, indent=2))

source/scripts/publishing/hub.py

@@ -0,0 +1,50 @@
+"""Manual Hub upload/download. Network work starts only with an explicit subcommand."""
+import argparse
+import json
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from prepare_hf import verifier
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    sub = parser.add_subparsers(dest='command', required=True)
+    upload = sub.add_parser('upload', help='Manually upload only the verified inventory into the existing model repo main branch.')
+    upload.add_argument('--release', type=Path, required=True)
+    download = sub.add_parser('download', help='Download a fixed Hub commit into a fresh directory, then checksum-verify.')
+    download.add_argument('--revision', required=True, help='Full 40-character Hub commit from upload output')
+    download.add_argument('--output', type=Path, required=True)
+    for command in (upload, download):
+        command.add_argument('--repo-id', default='Voltline/vimeml-tiny-ja-v1')
+    args = parser.parse_args()
+    check = verifier()
+    if args.command == 'upload':
+        manifest, files = check.verify(args.release)
+        if manifest['repo_id'] != args.repo_id:
+            parser.error('Release and target repository identities differ.')
+        from huggingface_hub import CommitOperationAdd, HfApi
+        api = HfApi()
+        parent = api.repo_info(args.repo_id, repo_type='model', revision='main').sha
+        result = api.create_commit(repo_id=args.repo_id, repo_type='model', revision='main', parent_commit=parent,
+            operations=[CommitOperationAdd(path_in_repo=name, path_or_fileobj=str(args.release / name)) for name in files],
+            commit_message='Release frozen VimeML tiny-ja-v1 FP32 and Core ML INT8 (GPL-2.0)')
+        print(json.dumps({'repo_id': args.repo_id, 'hub_commit': result.oid,
+                          'commit_url': result.commit_url, 'uploaded_files': len(files)}, indent=2))
+    else:
+        if len(args.revision) != 40 or any(c not in '0123456789abcdef' for c in args.revision):
+            parser.error('Use a full immutable 40-character Hub commit, not main.')
+        if args.output.exists():
+            parser.error('Output exists; use a fresh download directory.')
+        from huggingface_hub import snapshot_download
+        snapshot_download(repo_id=args.repo_id, repo_type='model', revision=args.revision, local_dir=str(args.output))
+        manifest, files = check.verify(args.output)
+        if manifest['repo_id'] != args.repo_id:
+            raise ValueError('Downloaded release belongs to a different repository.')
+        print(json.dumps({'repo_id': args.repo_id, 'hub_commit': args.revision,
+                          'verified_files': len(files), 'source_commit': manifest['source_commit']}, indent=2))
+
+
+if __name__ == '__main__':
+    main()

source/scripts/publishing/prepare_hf.py

@@ -0,0 +1,212 @@
+"""Prepare or verify a local GPL-2.0 v1 release. Never trains, converts or uploads."""
+import argparse
+import importlib.util
+import json
+import re
+import shutil
+import subprocess
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[2]
+sys.path.insert(0, str(ROOT / 'src'))
+
+
+def verifier():
+    spec = importlib.util.spec_from_file_location('vimeml_release_verifier', ROOT / 'templates/huggingface/verify_release.py')
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+def write_json(path, value):
+    path.parent.mkdir(parents=True, exist_ok=True)
+    path.write_text(json.dumps(value, ensure_ascii=False, indent=2, allow_nan=False) + '\n', encoding='utf-8', newline='\n')
+
+
+def git(*args):
+    return subprocess.check_output(['git', *args], cwd=ROOT).decode('utf-8').strip()
+
+
+def copy_checked(source, target, digest=None):
+    check = verifier()
+    source = Path(source)
+    if source.is_symlink() or not source.is_file():
+        raise ValueError(f'Expected a regular source file: {source}')
+    expected = digest or check.sha(source)
+    target.parent.mkdir(parents=True, exist_ok=True)
+    if target.exists():
+        raise ValueError(f'Refusing to overwrite: {target}')
+    shutil.copyfile(source, target)
+    if check.sha(target) != expected:
+        raise ValueError(f'Copy checksum mismatch: {target}')
+
+
+def evaluation_summary(bundle_manifest_sha, coreml_manifest_sha):
+    from vimeml.benchmarks.evaluate_ajimee import load_export, summarize
+    from vimeml.deployment.bundle import read_json
+    from vimeml.deployment.validation import compare_rows
+    sha = verifier().sha
+    hashes, results = {}, {}
+
+    def read(path):
+        hashes[path.relative_to(ROOT).as_posix()] = sha(path)
+        return read_json(path)
+
+    def rows(path, metadata):
+        expected = metadata['files_sha256']['scores.jsonl']
+        if sha(path) != expected:
+            raise ValueError(f'Score file identity mismatch: {path}')
+        hashes[path.relative_to(ROOT).as_posix()] = expected
+        return [json.loads(line) for line in path.read_text(encoding='utf-8').splitlines()]
+
+    for role, benchmark, baseline in [('dev', 'ime-dev-v2', 'tiny-ja-v1-dev-v2-scores'),
+                                      ('ajimee', 'ajimee-jwtd-v2-v1', 'tiny-ja-v1-ajimee')]:
+        directory = ROOT / f'outputs/deployment/conservative-int8-b32-{role}-v1'
+        report = read(directory / 'metrics.json')
+        if (report['model']['coreml_manifest_sha256'] != coreml_manifest_sha or
+                report['model']['bundle_manifest_sha256'] != bundle_manifest_sha):
+            raise ValueError('Saved evaluation uses a different model.')
+        current = rows(directory / 'scores.jsonl', report)
+        metrics = summarize(current)
+        if metrics != report['metrics']:
+            raise ValueError('Saved metrics do not match their score rows.')
+        frozen, provenance, inputs = load_export(ROOT / 'artifacts/benchmarks' / benchmark)
+        if frozen != report['benchmark_manifest'] or provenance != report['export_provenance'] or len(inputs) != len(current):
+            raise ValueError('Frozen candidate export identity changed.')
+        for left, right in zip(inputs, current):
+            if any(left[key] != right[key] for key in ('id', 'query', 'left_context', 'answers', 'candidates')):
+                raise ValueError('Saved candidate/input mismatch.')
+        prior_directory = ROOT / 'outputs/ime-eval' / baseline
+        prior_path = prior_directory / ('manifest.json' if (prior_directory / 'manifest.json').exists() else 'metrics.json')
+        prior_report = read(prior_path)
+        if any(prior_report['model'][key] != report['model'][key]
+               for key in ('checkpoint_sha256', 'tokenizer_sha256')):
+            raise ValueError('FP32 baseline belongs to different frozen weights or tokenizer.')
+        prior = rows(prior_directory / 'scores.jsonl', prior_report)
+        fp32_metrics = summarize(prior)
+        comparison = compare_rows(prior, current)
+        reported_before = prior
+        if role == 'dev':
+            source = ROOT / 'outputs/deployment/conservative-dev-v3'
+            source_metadata = read(source / 'metrics.json')
+            reported_before = rows(source / 'scores.jsonl', source_metadata)
+        if compare_rows(reported_before, current) != report['comparison']:
+            raise ValueError('Saved comparison cannot be reproduced from its actual baseline.')
+        results[role] = {'original_order': metrics['all']['azookey'],
+            'fp32_lm': fp32_metrics['all']['lm_context_sum'], 'int8_lm': metrics['all']['lm_context_sum'],
+            'windows_fp32_comparison': {k: v for k, v in comparison.items() if k != 'changes'},
+            'original_report_comparison_baseline': 'conservative Core ML' if role == 'dev' else 'Windows FP32',
+            'original_report_comparison': {k: v for k, v in report['comparison'].items() if k != 'changes'},
+            'benchmark_source_sha256': frozen['source_sha256'], 'candidate_export_sha256': provenance['files_sha256']['azookey-candidates.json']}
+    alignment = read(ROOT / 'outputs/deployment/conservative-int8-b32-alignment-v1/alignment.json')
+    if alignment['coreml_manifest_sha256'] != coreml_manifest_sha:
+        raise ValueError('Alignment belongs to a different model.')
+    distribution = read(ROOT / 'outputs/deployment/conservative-int8-b32-distribution-v1/report.json')
+    training = read(ROOT / 'artifacts/models/tiny-ja-v1/summary.json')
+    return {'format': 'vimeml_public_evaluation_v1', 'bundle_manifest_sha256': bundle_manifest_sha,
+        'coreml_manifest_sha256': coreml_manifest_sha, 'training': {
+            'prediction_targets': training['total_trained_tokens'], 'data_passes': training['completed_data_passes'],
+            'full_validation': training['full_validation']['best'], 'test_used_for_selection': False},
+        'ranking': results, 'alignment': {'strict_passed': alignment['passed'],
+            'max_logit_abs': max(row['max_abs'] for row in alignment['logits']),
+            'invariants': alignment['invariants']},
+        'distribution': {'original_report': distribution,
+            'aggregation': 'Mean of per-example position means; not corpus token-weighted.',
+            'content_token_limit': 127, 'truncation_count_recorded': False,
+            'original_report_has_full_identity_metadata': False},
+        'source_reports_sha256': hashes,
+        'scope': 'Offline fixed pools and saved Mac observations; not app-wide accuracy or a Windows Core ML runtime rerun.'}
+
+
+def prepare(output, repo_id='Voltline/vimeml-tiny-ja-v1'):
+    output = Path(output).resolve()
+    if output.exists():
+        raise ValueError('Output exists; choose a new release version.')
+    if not re.fullmatch(r'[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+', repo_id):
+        raise ValueError('Use an OWNER/REPO model repository ID.')
+    if git('status', '--porcelain'):
+        raise ValueError('Commit the reviewed source on the current branch before preparing a release.')
+    from vimeml.deployment.bundle import verify_bundle
+    from vimeml.deployment.coreml import verify_package
+    check = verifier()
+    bundle = ROOT / 'artifacts/deployment/tiny-ja-v1-inference-v1'
+    model = ROOT / 'artifacts/deployment/tiny-ja-v1-conservative-int8-b32-v1'
+    metadata = verify_bundle(bundle)
+    coreml = verify_package(model)
+    bundle_sha, coreml_sha = check.sha(bundle / 'manifest.json'), check.sha(model / 'manifest.json')
+    if (metadata['parameter_count'] != 7386624 or coreml['kind'] != 'linear8_fp32_compute' or
+            coreml['minimum_ios'] != 18 or coreml['package_bytes'] != 8077801 or
+            coreml['bundle']['bundle_manifest_sha256'] != bundle_sha):
+        raise ValueError('Expected the reviewed frozen v1/INT8 package.')
+    summary = evaluation_summary(bundle_sha, coreml_sha)
+    for role, expected in [('dev', (137, 122, 134)), ('ajimee', (200, 124, 151))]:
+        for variant in ('fp32_lm', 'int8_lm'):
+            actual = summary['ranking'][role][variant]
+            if tuple(actual[key] for key in ('cases', 'top1_correct', 'top5_correct')) != expected:
+                raise ValueError('Saved metrics differ from this v1 model card; review before publishing.')
+    commit = git('rev-parse', 'HEAD')
+    output.mkdir(parents=True, exist_ok=False)
+    for name in [*metadata['files'], 'manifest.json']:
+        copy_checked(bundle / name, output / 'inference' / name)
+    for name, entry in coreml['package_files'].items():
+        copy_checked(model / 'model.mlpackage' / name,
+                     output / 'coreml/ios18-int8-block32/model.mlpackage' / name, entry['sha256'])
+    copy_checked(model / 'manifest.json', output / 'coreml/ios18-int8-block32/manifest.json')
+    # Copy committed code/configuration, never data, annotations, artifacts or secrets.
+    source_files = subprocess.check_output(['git', 'ls-files', '-z', '--', 'src', 'scripts', 'configs',
+        'templates/huggingface', 'requirements.txt', 'LICENSE'], cwd=ROOT).decode('utf-8').split('\0')
+    for name in filter(None, source_files):
+        copy_checked(ROOT / name, output / 'source' / name)
+    (output / 'source/README.md').write_text(
+        f'VimeML source/configuration from Git commit {commit}. GPL-2.0; see LICENSE.\n'
+        'The FP32 loader checks the original model and deployment code fingerprints.\n'
+        'Corpus texts, annotations, optimizer state and the separate Vime client are not included.\n', encoding='utf-8', newline='\n')
+    copy_checked(ROOT / 'LICENSE', output / 'LICENSE')
+    for name in ('infer.py', 'verify_release.py'):
+        copy_checked(ROOT / 'templates/huggingface' / name, output / name)
+    card = (ROOT / 'templates/huggingface/README.md').read_text(encoding='utf-8')
+    for key, value in {'SOURCE_COMMIT': commit, 'BUNDLE_SHA': bundle_sha, 'COREML_SHA': coreml_sha,
+                       'TOKENIZER_SHA': metadata['files']['tokenizer.model']['sha256'],
+                       'CHECKPOINT_SHA': metadata['checkpoint_sha256']}.items():
+        card = card.replace(f'@@{key}@@', value)
+    if '@@' in card:
+        raise ValueError('Unresolved model card field.')
+    (output / 'README.md').write_text(card, encoding='utf-8', newline='\n')
+    write_json(output / 'evaluation/summary.json', summary)
+    files = {p.relative_to(output).as_posix(): {'bytes': p.stat().st_size, 'sha256': check.sha(p)}
+             for p in sorted(output.rglob('*')) if p.is_file()}
+    release = {'format': 'vimeml_hf_release_v1', 'repo_id': repo_id, 'license': 'gpl-2.0',
+        'source_repository': 'https://github.com/Voltline/VimeML', 'source_commit': commit,
+        'created_utc': datetime.now(timezone.utc).isoformat(), 'bundle_manifest_sha256': bundle_sha,
+        'coreml_manifest_sha256': coreml_sha, 'files': files,
+        'policy': 'Local preparation only. No training, conversion, compression, device access or upload.'}
+    write_json(output / 'RELEASE.json', release)
+    sums = {**{name: entry['sha256'] for name, entry in files.items()}, 'RELEASE.json': check.sha(output / 'RELEASE.json')}
+    (output / 'SHA256SUMS.txt').write_text(''.join(f'{digest}  {name}\n' for name, digest in sorted(sums.items())),
+                                         encoding='utf-8', newline='\n')
+    _, verified = check.verify(output)
+    print(json.dumps({'release': str(output), 'repo_id': repo_id, 'source_commit': commit,
+                      'verified_files': len(verified), 'bytes': sum((output / name).stat().st_size for name in verified)}, indent=2))
+    return release
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    sub = parser.add_subparsers(dest='command', required=True)
+    build = sub.add_parser('prepare', help='Create a fresh local release from the committed source and frozen artifacts.')
+    build.add_argument('--output', type=Path, required=True)
+    build.add_argument('--repo-id', default='Voltline/vimeml-tiny-ja-v1')
+    verify = sub.add_parser('verify', help='Read-only checksum/inventory verification.')
+    verify.add_argument('--release', type=Path, required=True)
+    args = parser.parse_args()
+    if args.command == 'prepare':
+        prepare(args.output, args.repo_id)
+    else:
+        manifest, files = verifier().verify(args.release)
+        print(f"Verified {len(files)} release files for {manifest['repo_id']}")
+
+
+if __name__ == '__main__':
+    main()