Release Nanobe model and MLX reference implementation

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/hero-scores.png filter=lfs diff=lfs merge=lfs -text
+assets/peer-bench-scores.png filter=lfs diff=lfs merge=lfs -text
+model/tokenizer.json filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Artem G.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,3 +1,93 @@
 ---
 license: mit
+library_name: mlx
+pipeline_tag: feature-extraction
+base_model:
+  - jinaai/jina-embeddings-v5-omni-small-retrieval
+  - facebook/pe-av-small
+tags:
+  - multimodal
+  - embeddings
+  - semantic-search
+  - image
+  - audio
+  - video
+  - apple-silicon
 ---
+
+# Nanobe
+
+Nanobe is a local multimodal embedding model for searching mixed folders. It encodes text, images, document pages, audio, and video into a shared vector space.
+
+Audio and video are encoded directly, without generating transcripts or captions during indexing. Pages and recording segments get their own vectors, so a relevant page or moment can retrieve the whole file.
+
+This release includes quantized model weights and a Python reference implementation using MLX on Apple Silicon.
+
+Read the [release post](https://thiswickedtongue.net/nanobe/).
+
+## Benchmark results
+
+Nanobe Bench tests retrieval across mixed file types, including concepts, style, and relevant content deep inside documents and recordings.
+
+![Nanobe Bench results compared with peer embedding models](assets/hero-scores.png)
+
+Results on conventional retrieval benchmarks:
+
+![Nanobe and peer results on NFCorpus, SciFact, Flickr30k, and Clotho](assets/peer-bench-scores.png)
+
+## Usage
+
+Use an Apple Silicon Mac with Python 3.12 and [uv](https://docs.astral.sh/uv/). Download the release and install the reference implementation:
+
+```sh
+uvx --from huggingface-hub hf download lokimorty/nanobe --local-dir nanobe-release
+cd nanobe-release/source
+uv sync --frozen
+export NANOBE_MODEL_DIR="$(cd ../model && pwd)"
+```
+
+### Python inference
+
+Encode a text query and compare it with a local file. Run from the `source` directory and replace `example.jpg` with your file:
+
+```python
+from pathlib import Path
+
+import numpy as np
+
+from nanobe.inference import encode_file
+from nanobe.runtime import load, load_lanes, load_query
+
+package = Path("../model")
+model = load(package)
+calibration = load_lanes(package)
+query = load_query(package).encode_query("a flamenco performance")
+kind, segments = encode_file(model, Path("example.jpg"))
+vectors = np.stack([calibration.item(segment.vector, kind) for segment in segments])
+scores = vectors @ query / (np.linalg.norm(vectors, axis=1) * np.linalg.norm(query))
+print(float(scores.max()))
+```
+
+The same file encoder accepts supported images, documents, audio, and video. See [Python inference](source/docs/model.md#python-inference) for details and a [search example](source/examples/search.py) that ranks multiple files.
+
+### Folder search
+
+The reference implementation also includes a local folder index:
+
+```sh
+uv run --frozen nanobe index ~/Creative
+uv run --frozen nanobe search ~/Creative "a presentation about sustainable packaging"
+uv run --frozen nanobe search ~/Creative --file ~/Desktop/reference.jpg
+```
+
+See the [source README](source/README.md) for supported formats and setup details.
+
+## Model lineage
+
+Nanobe builds on Jina AI's [jina-embeddings-v5-omni-small-retrieval](https://huggingface.co/jinaai/jina-embeddings-v5-omni-small-retrieval) and Meta's [Perception Encoder AV](https://huggingface.co/facebook/pe-av-small). See [model lineage](upstream-notices.md) for the recorded revisions.
+
+## License
+
+I release Nanobe under MIT because I believe reusable computational capabilities should be available for anyone to use, modify, and build on, including commercially.
+
+See [LICENSE](LICENSE) for the full MIT License.

dist/nanobe-0.1.0.tar.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d79f7bfb47e85ed6c48e7edba8c838d7de91208e1723edcbfdb7ef755473d448
+size 111386

licenses/jina-cc-by-nc-4.0.txt

@@ -0,0 +1,408 @@
+Attribution-NonCommercial 4.0 International
+
+=======================================================================
+
+Creative Commons Corporation ("Creative Commons") is not a law firm and
+does not provide legal services or legal advice. Distribution of
+Creative Commons public licenses does not create a lawyer-client or
+other relationship. Creative Commons makes its licenses and related
+information available on an "as-is" basis. Creative Commons gives no
+warranties regarding its licenses, any material licensed under their
+terms and conditions, or any related information. Creative Commons
+disclaims all liability for damages resulting from their use to the
+fullest extent possible.
+
+Using Creative Commons Public Licenses
+
+Creative Commons public licenses provide a standard set of terms and
+conditions that creators and other rights holders may use to share
+original works of authorship and other material subject to copyright
+and certain other rights specified in the public license below. The
+following considerations are for informational purposes only, are not
+exhaustive, and do not form part of our licenses.
+
+     Considerations for licensors: Our public licenses are
+     intended for use by those authorized to give the public
+     permission to use material in ways otherwise restricted by
+     copyright and certain other rights. Our licenses are
+     irrevocable. Licensors should read and understand the terms
+     and conditions of the license they choose before applying it.
+     Licensors should also secure all rights necessary before
+     applying our licenses so that the public can reuse the
+     material as expected. Licensors should clearly mark any
+     material not subject to the license. This includes other CC-
+     licensed material, or material used under an exception or
+     limitation to copyright. More considerations for licensors:
+    wiki.creativecommons.org/Considerations_for_licensors
+
+     Considerations for the public: By using one of our public
+     licenses, a licensor grants the public permission to use the
+     licensed material under specified terms and conditions. If
+     the licensor's permission is not necessary for any reason--for
+     example, because of any applicable exception or limitation to
+     copyright--then that use is not regulated by the license. Our
+     licenses grant only permissions under copyright and certain
+     other rights that a licensor has authority to grant. Use of
+     the licensed material may still be restricted for other
+     reasons, including because others have copyright or other
+     rights in the material. A licensor may make special requests,
+     such as asking that all changes be marked or described.
+     Although not required by our licenses, you are encouraged to
+     respect those requests where reasonable. More considerations
+     for the public:
+    wiki.creativecommons.org/Considerations_for_licensees
+
+=======================================================================
+
+Creative Commons Attribution-NonCommercial 4.0 International Public
+License
+
+By exercising the Licensed Rights (defined below), You accept and agree
+to be bound by the terms and conditions of this Creative Commons
+Attribution-NonCommercial 4.0 International Public License ("Public
+License"). To the extent this Public License may be interpreted as a
+contract, You are granted the Licensed Rights in consideration of Your
+acceptance of these terms and conditions, and the Licensor grants You
+such rights in consideration of benefits the Licensor receives from
+making the Licensed Material available under these terms and
+conditions.
+
+
+Section 1 -- Definitions.
+
+  a. Adapted Material means material subject to Copyright and Similar
+     Rights that is derived from or based upon the Licensed Material
+     and in which the Licensed Material is translated, altered,
+     arranged, transformed, or otherwise modified in a manner requiring
+     permission under the Copyright and Similar Rights held by the
+     Licensor. For purposes of this Public License, where the Licensed
+     Material is a musical work, performance, or sound recording,
+     Adapted Material is always produced where the Licensed Material is
+     synched in timed relation with a moving image.
+
+  b. Adapter's License means the license You apply to Your Copyright
+     and Similar Rights in Your contributions to Adapted Material in
+     accordance with the terms and conditions of this Public License.
+
+  c. Copyright and Similar Rights means copyright and/or similar rights
+     closely related to copyright including, without limitation,
+     performance, broadcast, sound recording, and Sui Generis Database
+     Rights, without regard to how the rights are labeled or
+     categorized. For purposes of this Public License, the rights
+     specified in Section 2(b)(1)-(2) are not Copyright and Similar
+     Rights.
+  d. Effective Technological Measures means those measures that, in the
+     absence of proper authority, may not be circumvented under laws
+     fulfilling obligations under Article 11 of the WIPO Copyright
+     Treaty adopted on December 20, 1996, and/or similar international
+     agreements.
+
+  e. Exceptions and Limitations means fair use, fair dealing, and/or
+     any other exception or limitation to Copyright and Similar Rights
+     that applies to Your use of the Licensed Material.
+
+  f. Licensed Material means the artistic or literary work, database,
+     or other material to which the Licensor applied this Public
+     License.
+
+  g. Licensed Rights means the rights granted to You subject to the
+     terms and conditions of this Public License, which are limited to
+     all Copyright and Similar Rights that apply to Your use of the
+     Licensed Material and that the Licensor has authority to license.
+
+  h. Licensor means the individual(s) or entity(ies) granting rights
+     under this Public License.
+
+  i. NonCommercial means not primarily intended for or directed towards
+     commercial advantage or monetary compensation. For purposes of
+     this Public License, the exchange of the Licensed Material for
+     other material subject to Copyright and Similar Rights by digital
+     file-sharing or similar means is NonCommercial provided there is
+     no payment of monetary compensation in connection with the
+     exchange.
+
+  j. Share means to provide material to the public by any means or
+     process that requires permission under the Licensed Rights, such
+     as reproduction, public display, public performance, distribution,
+     dissemination, communication, or importation, and to make material
+     available to the public including in ways that members of the
+     public may access the material from a place and at a time
+     individually chosen by them.
+
+  k. Sui Generis Database Rights means rights other than copyright
+     resulting from Directive 96/9/EC of the European Parliament and of
+     the Council of 11 March 1996 on the legal protection of databases,
+     as amended and/or succeeded, as well as other essentially
+     equivalent rights anywhere in the world.
+
+  l. You means the individual or entity exercising the Licensed Rights
+     under this Public License. Your has a corresponding meaning.
+
+
+Section 2 -- Scope.
+
+  a. License grant.
+
+       1. Subject to the terms and conditions of this Public License,
+          the Licensor hereby grants You a worldwide, royalty-free,
+          non-sublicensable, non-exclusive, irrevocable license to
+          exercise the Licensed Rights in the Licensed Material to:
+
+            a. reproduce and Share the Licensed Material, in whole or
+               in part, for NonCommercial purposes only; and
+
+            b. produce, reproduce, and Share Adapted Material for
+               NonCommercial purposes only.
+
+       2. Exceptions and Limitations. For the avoidance of doubt, where
+          Exceptions and Limitations apply to Your use, this Public
+          License does not apply, and You do not need to comply with
+          its terms and conditions.
+
+       3. Term. The term of this Public License is specified in Section
+          6(a).
+
+       4. Media and formats; technical modifications allowed. The
+          Licensor authorizes You to exercise the Licensed Rights in
+          all media and formats whether now known or hereafter created,
+          and to make technical modifications necessary to do so. The
+          Licensor waives and/or agrees not to assert any right or
+          authority to forbid You from making technical modifications
+          necessary to exercise the Licensed Rights, including
+          technical modifications necessary to circumvent Effective
+          Technological Measures. For purposes of this Public License,
+          simply making modifications authorized by this Section 2(a)
+          (4) never produces Adapted Material.
+
+       5. Downstream recipients.
+
+            a. Offer from the Licensor -- Licensed Material. Every
+               recipient of the Licensed Material automatically
+               receives an offer from the Licensor to exercise the
+               Licensed Rights under the terms and conditions of this
+               Public License.
+
+            b. No downstream restrictions. You may not offer or impose
+               any additional or different terms or conditions on, or
+               apply any Effective Technological Measures to, the
+               Licensed Material if doing so restricts exercise of the
+               Licensed Rights by any recipient of the Licensed
+               Material.
+
+       6. No endorsement. Nothing in this Public License constitutes or
+          may be construed as permission to assert or imply that You
+          are, or that Your use of the Licensed Material is, connected
+          with, or sponsored, endorsed, or granted official status by,
+          the Licensor or others designated to receive attribution as
+          provided in Section 3(a)(1)(A)(i).
+
+  b. Other rights.
+
+       1. Moral rights, such as the right of integrity, are not
+          licensed under this Public License, nor are publicity,
+          privacy, and/or other similar personality rights; however, to
+          the extent possible, the Licensor waives and/or agrees not to
+          assert any such rights held by the Licensor to the limited
+          extent necessary to allow You to exercise the Licensed
+          Rights, but not otherwise.
+
+       2. Patent and trademark rights are not licensed under this
+          Public License.
+
+       3. To the extent possible, the Licensor waives any right to
+          collect royalties from You for the exercise of the Licensed
+          Rights, whether directly or through a collecting society
+          under any voluntary or waivable statutory or compulsory
+          licensing scheme. In all other cases the Licensor expressly
+          reserves any right to collect such royalties, including when
+          the Licensed Material is used other than for NonCommercial
+          purposes.
+
+
+Section 3 -- License Conditions.
+
+Your exercise of the Licensed Rights is expressly made subject to the
+following conditions.
+
+  a. Attribution.
+
+       1. If You Share the Licensed Material (including in modified
+          form), You must:
+
+            a. retain the following if it is supplied by the Licensor
+               with the Licensed Material:
+
+                 i. identification of the creator(s) of the Licensed
+                    Material and any others designated to receive
+                    attribution, in any reasonable manner requested by
+                    the Licensor (including by pseudonym if
+                    designated);
+
+                ii. a copyright notice;
+
+               iii. a notice that refers to this Public License;
+
+                iv. a notice that refers to the disclaimer of
+                    warranties;
+
+                 v. a URI or hyperlink to the Licensed Material to the
+                    extent reasonably practicable;
+
+            b. indicate if You modified the Licensed Material and
+               retain an indication of any previous modifications; and
+
+            c. indicate the Licensed Material is licensed under this
+               Public License, and include the text of, or the URI or
+               hyperlink to, this Public License.
+
+       2. You may satisfy the conditions in Section 3(a)(1) in any
+          reasonable manner based on the medium, means, and context in
+          which You Share the Licensed Material. For example, it may be
+          reasonable to satisfy the conditions by providing a URI or
+          hyperlink to a resource that includes the required
+          information.
+
+       3. If requested by the Licensor, You must remove any of the
+          information required by Section 3(a)(1)(A) to the extent
+          reasonably practicable.
+
+       4. If You Share Adapted Material You produce, the Adapter's
+          License You apply must not prevent recipients of the Adapted
+          Material from complying with this Public License.
+
+
+Section 4 -- Sui Generis Database Rights.
+
+Where the Licensed Rights include Sui Generis Database Rights that
+apply to Your use of the Licensed Material:
+
+  a. for the avoidance of doubt, Section 2(a)(1) grants You the right
+     to extract, reuse, reproduce, and Share all or a substantial
+     portion of the contents of the database for NonCommercial purposes
+     only;
+
+  b. if You include all or a substantial portion of the database
+     contents in a database in which You have Sui Generis Database
+     Rights, then the database in which You have Sui Generis Database
+     Rights (but not its individual contents) is Adapted Material; and
+
+  c. You must comply with the conditions in Section 3(a) if You Share
+     all or a substantial portion of the contents of the database.
+
+For the avoidance of doubt, this Section 4 supplements and does not
+replace Your obligations under this Public License where the Licensed
+Rights include other Copyright and Similar Rights.
+
+
+Section 5 -- Disclaimer of Warranties and Limitation of Liability.
+
+  a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
+     EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
+     AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
+     ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
+     IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
+     WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
+     PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
+     ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
+     KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
+     ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
+
+  b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
+     TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
+     NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
+     INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
+     COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
+     USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
+     ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
+     DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
+     IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
+
+  c. The disclaimer of warranties and limitation of liability provided
+     above shall be interpreted in a manner that, to the extent
+     possible, most closely approximates an absolute disclaimer and
+     waiver of all liability.
+
+
+Section 6 -- Term and Termination.
+
+  a. This Public License applies for the term of the Copyright and
+     Similar Rights licensed here. However, if You fail to comply with
+     this Public License, then Your rights under this Public License
+     terminate automatically.
+
+  b. Where Your right to use the Licensed Material has terminated under
+     Section 6(a), it reinstates:
+
+       1. automatically as of the date the violation is cured, provided
+          it is cured within 30 days of Your discovery of the
+          violation; or
+
+       2. upon express reinstatement by the Licensor.
+
+     For the avoidance of doubt, this Section 6(b) does not affect any
+     right the Licensor may have to seek remedies for Your violations
+     of this Public License.
+
+  c. For the avoidance of doubt, the Licensor may also offer the
+     Licensed Material under separate terms or conditions or stop
+     distributing the Licensed Material at any time; however, doing so
+     will not terminate this Public License.
+
+  d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
+     License.
+
+
+Section 7 -- Other Terms and Conditions.
+
+  a. The Licensor shall not be bound by any additional or different
+     terms or conditions communicated by You unless expressly agreed.
+
+  b. Any arrangements, understandings, or agreements regarding the
+     Licensed Material not stated herein are separate from and
+     independent of the terms and conditions of this Public License.
+
+
+Section 8 -- Interpretation.
+
+  a. For the avoidance of doubt, this Public License does not, and
+     shall not be interpreted to, reduce, limit, restrict, or impose
+     conditions on any use of the Licensed Material that could lawfully
+     be made without permission under this Public License.
+
+  b. To the extent possible, if any provision of this Public License is
+     deemed unenforceable, it shall be automatically reformed to the
+     minimum extent necessary to make it enforceable. If the provision
+     cannot be reformed, it shall be severed from this Public License
+     without affecting the enforceability of the remaining terms and
+     conditions.
+
+  c. No term or condition of this Public License will be waived and no
+     failure to comply consented to unless expressly agreed to by the
+     Licensor.
+
+  d. Nothing in this Public License constitutes or may be interpreted
+     as a limitation upon, or waiver of, any privileges and immunities
+     that apply to the Licensor or You, including from the legal
+     processes of any jurisdiction or authority.
+
+=======================================================================
+
+Creative Commons is not a party to its public
+licenses. Notwithstanding, Creative Commons may elect to apply one of
+its public licenses to material it publishes and in those instances
+will be considered the “Licensor.” The text of the Creative Commons
+public licenses is dedicated to the public domain under the CC0 Public
+Domain Dedication. Except for the limited purpose of indicating that
+material is shared under a Creative Commons public license or as
+otherwise permitted by the Creative Commons policies published at
+creativecommons.org/policies, Creative Commons does not authorize the
+use of the trademark "Creative Commons" or any other trademark or logo
+of Creative Commons without its prior written consent including,
+without limitation, in connection with any unauthorized modifications
+to any of its public licenses or any other arrangements,
+understandings, or agreements concerning use of licensed material. For
+the avoidance of doubt, this paragraph does not form part of the
+public licenses.
+
+Creative Commons may be contacted at creativecommons.org.
+

licenses/meta-apache-2.0.txt

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

model/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Artem G.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

model/README.md

@@ -0,0 +1,85 @@
+---
+license: mit
+---
+
+# Nanobe model package
+
+The package provides MLX inference for text, rendered document pages, images, audio and video with sound. It includes the language and multimodal weights, retrieval head, media adapters, document-page merger, audio-window pooler, Perception Encoder audio and text components, tokenizers and Python inference helpers.
+
+## Embeddings
+
+The active `geometry.json` defines a 2,690-coordinate model embedding. Folder indexing extends it to 2,696 coordinates with the package's calibration. The text scale is 0.94 and the raw acoustic query weight is 0.02. The model has a 640-coordinate retrieval head and a 1,024-coordinate native component, with acoustic features and padding completing the output. Head size is not the final embedding dimension.
+
+The model geometry, tensor values, quantization settings, tokenizer settings, prompts, pooling and sampling constants define inference. The model and its helper files must remain together.
+
+## Python inference
+
+Install the matching source release with `uv sync --frozen` and set `NANOBE_MODEL_DIR` as described in the source README. The runnable `examples/search.py` ranks supplied files using either a text query or a reference file. It uses the same file encoder as the folder CLI.
+
+### Text query against a file
+
+Run this from the source directory after creating the README's demo files:
+
+```python
+import os
+from pathlib import Path
+
+import numpy as np
+
+from nanobe.inference import encode_file
+from nanobe.runtime import load, load_lanes, load_query
+
+package = Path(os.environ["NANOBE_MODEL_DIR"])
+query = load_query(package).encode_query("Saturn through a telescope")
+runtime = load(package)
+calibration = load_lanes(package)
+
+kind, segments = encode_file(runtime, Path("demo/observatory.txt"))
+vectors = np.stack([calibration.item(segment.vector, kind) for segment in segments])
+scores = vectors @ query / (np.linalg.norm(vectors, axis=1) * np.linalg.norm(query))
+best = int(np.argmax(scores))
+print(segments[best].label, float(scores[best]))
+```
+
+`encode_file` returns the file kind and a list of segments. Every segment has an `ordinal`, `label`, raw `vector` and extracted `lexical` text when available. The raw vector has 2,690 coordinates. `calibration.item` transforms it into a 2,696-coordinate vector with the file kind's scale and extra coordinates. `load_query(...).encode_query` already returns a calibrated 2,696-coordinate query. Do not calibrate that query a second time or compare it directly with raw file vectors.
+
+`load_query` loads only the components needed for text queries. `load` loads the complete runtime used by `encode_file`. Reuse these objects when encoding multiple inputs. The full runtime's lower-level `encode_query` method returns 2,690 coordinates; it needs `calibration.query` before comparison with calibrated items.
+
+### Reference file as a query
+
+A reference file can have multiple segments. Its query vectors use `calibration.query`, while the files being searched still use `calibration.item`:
+
+```python
+_, reference = encode_file(runtime, Path("reference.txt"))
+queries = np.stack([calibration.query(segment.vector) for segment in reference])
+scores = queries @ vectors.T
+scores /= np.linalg.norm(queries, axis=1)[:, None] * np.linalg.norm(vectors, axis=1)[None, :]
+query, item = np.unravel_index(np.argmax(scores), scores.shape)
+print(reference[query].label, segments[item].label, float(scores[query, item]))
+```
+
+For several files, keep each file's best segment-pair score and sort descending. Exclude the reference file itself when it also appears among the files being searched. `examples/search.py` performs those steps.
+
+### Segments and retrieval behavior
+
+The shared file encoder applies the documented readers, text chunks, PDF page selection, audio/video windows, audio pooling and DOC/PPT conversion. Replacing the example paths with supported images, documents, recordings or videos uses those same rules. A whole file may return multiple vectors; averaging them changes the retrieval method.
+
+These examples perform exact semantic comparisons on full-precision embeddings. The folder CLI stores segment values in float16, uses an int8 approximate index to select candidates, reranks with calibrated stored vectors, and selects the best segment per file. Small numerical differences can follow from storage precision. Text searches also combine semantic and lexical ranks, so their final scores and ordering can differ from pure semantic scoring. Reference-file CLI searches use semantic retrieval only.
+
+## File readers
+
+The Python reference implementation decodes AVIF and HEIC as RGB images and rasterizes SVG on white within a 2048-by-2048 box. HEIC uses the primary image, applies container orientation and reduces HDR to 8 bits. SVG text depends on installed fonts and the renderer's platform fallback.
+
+RTF supplies extracted text. XLS/XLSX supply saved cell values, sheet names and repeated first-row headers without formula recalculation. Workbooks share eight 6,000-character segment slots across the first eight nonempty sheets and report truncation warnings. DOC/PPT require LibreOffice and use converted PDF pages. Nanobe finds LibreOffice on PATH or at its standard macOS application location. Conversion has a 120-second batch timeout.
+
+DOCX/PPTX/EPUB supply extracted text. PDF and converted document inputs use the first 24 pages plus up to eight later pages. Audio and video retain whole-file embeddings and bounded additional segments. Missing decoders or converters, empty inputs and damaged files can fail. These readers prepare model inputs; they do not add a separate embedding model. The source README documents the complete format list and limits.
+
+## Attribution
+
+The base model is `jinaai/jina-embeddings-v5-omni-small-retrieval` at revision `12949877f0092093f366c6450340011320152a05`. Perception Encoder components derive from `facebook/pe-av-small`; the text component records revision `dd050762bb9704ae9cd996ca45532a98f81d817e`. Retained provenance hashes identify upstream inputs and are not runtime configuration.
+
+See the accompanying upstream notices. No cloud inference service or upstream embedding-size claim applies to this Nanobe package.
+
+## License
+
+Nanobe is released under the [MIT License](LICENSE).

model/audio-pooler.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7bf8bc6420491454c3bb332bd243244d82efddfbcaa07cbc56dfb7603ab913f9
+size 5251732

model/chat_template.jinja

@@ -0,0 +1,154 @@
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain images.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+            {%- elif 'video' in item or item.type == 'video' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain videos.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Video ' ~ video_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+            {%- elif 'text' in item %}
+                {{- item.text }}
+            {%- else %}
+                {{- raise_exception('Unexpected item type in content.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- if tools and tools is iterable and tools is not mapping %}
+    {{- '<|im_start|>system\n' }}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {%- if content %}
+            {{- '\n\n' + content }}
+        {%- endif %}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" %}
+        {%- set content = render_content(message.content, false)|trim %}
+        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if ns.multi_step_tool %}
+    {{- raise_exception('No user query found in messages.') }}
+{%- endif %}
+{%- for message in messages %}
+    {%- set content = render_content(message.content, true)|trim %}
+    {%- if message.role == "system" %}
+        {%- if not loop.first %}
+            {{- raise_exception('System message must be at the beginning.') }}
+        {%- endif %}
+    {%- elif message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content|trim %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {%- if loop.first %}
+                    {%- if content|trim %}
+                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- else %}
+                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- endif %}
+                {%- else %}
+                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                {%- endif %}
+                {%- if tool_call.arguments is defined %}
+                    {%- for args_name, args_value in tool_call.arguments|items %}
+                        {{- '<parameter=' + args_name + '>\n' }}
+                        {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
+                        {{- args_value }}
+                        {{- '\n</parameter>\n' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- else %}
+        {{- raise_exception('Unexpected message role.') }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is true %}
+        {{- '<think>\n' }}
+    {%- else %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- endif %}
+{%- endif %}

model/config.json

@@ -0,0 +1,54 @@
+{
+  "architectures": [
+    "JinaOmniSmallEmbeddingModel"
+  ],
+  "audio_config": {
+    "d_model": 1280,
+    "encoder_attention_heads": 20,
+    "encoder_ffn_dim": 5120,
+    "encoder_layers": 32,
+    "max_source_positions": 1500,
+    "num_mel_bins": 128,
+    "scale_embedding": false,
+    "n_window": 100
+  },
+  "audio_end_token_id": 151671,
+  "audio_start_token_id": 151670,
+  "audio_token_id": 151669,
+  "image_token_id": 151655,
+  "model_type": "jina_omni_small_mlx",
+  "text_config": {
+    "head_dim": 128,
+    "hidden_size": 1024,
+    "intermediate_size": 3072,
+    "max_position_embeddings": 32768,
+    "model_type": "",
+    "num_attention_heads": 16,
+    "num_hidden_layers": 28,
+    "num_key_value_heads": 8,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "rope_theta": 3500000,
+      "rope_type": "default"
+    },
+    "vocab_size": 151672
+  },
+  "transformers_version": "5.4.0",
+  "video_token_id": 151656,
+  "vision_config": {
+    "depth": 24,
+    "hidden_size": 1024,
+    "in_channels": 3,
+    "intermediate_size": 4096,
+    "model_type": "",
+    "num_heads": 16,
+    "num_position_embeddings": 2304,
+    "out_hidden_size": 1024,
+    "patch_size": 16,
+    "spatial_merge_size": 2,
+    "temporal_patch_size": 2
+  },
+  "vision_end_token_id": 151653,
+  "vision_start_token_id": 151652,
+  "torch_dtype": "bfloat16"
+}

model/geometry.json

@@ -0,0 +1,67 @@
+{
+  "format": "nanobe-model-geometry-v1",
+  "base_model": "jinaai/jina-embeddings-v5-omni-small-retrieval",
+  "base_revision": "12949877f0092093f366c6450340011320152a05",
+  "head_sha256": "3922c5b7cb892977d61cb3e6caaa025fece854d5be2fba0e047503e1c17f4ccf",
+  "native_document_share": 0.485,
+  "embedding_dimension": 2690,
+  "audio_budget_seconds": 120.0,
+  "native_lane_media": [
+    "audio",
+    "video"
+  ],
+  "video_soundtrack_weight": 1.5,
+  "image_projection": {
+    "epsilon": 0.2,
+    "modality": "image",
+    "native_dimensions": 1024,
+    "head_dimensions": 640,
+    "output_dimensions": 1664,
+    "weight": "image_projection.weight",
+    "bias": "image_projection.bias"
+  },
+  "audio_projection": {
+    "encoder": "pe-audio",
+    "encoder_sha256": "790f80f549c70cb327b30c537017ca2ae6f47781b219ce8cd35f0144b1e058d7",
+    "weight": "audio_projection.weight",
+    "bias": "audio_projection.bias",
+    "share": 0.2
+  },
+  "base_embedding_dimension": 1664,
+  "audio_magnitude": {
+    "u": 0.9900000095367432,
+    "v": 0.02,
+    "text_encoder": "pe-text",
+    "text_encoder_sha256": "97ad081b8044cb4308995a5d2d93d36bdec39a7ccfcf01e6591b7a5fd6b0eeff",
+    "helper": "magnitude.py",
+    "helper_sha256": "93f267d198dcd0755134e86011d85d78af4fd5d61b818f98f8202cab1684930e",
+    "raw_cap": 8.0,
+    "padding_radius": 8.00001,
+    "output_dimensions": 2690
+  },
+  "index_dimension": 2696,
+  "lane_calibration": {
+    "helper": "lanes.py",
+    "helper_sha256": "b85311d9b5fe5d593d4b04730eeb66af11d39d942b393ee57e63ab3062510320",
+    "scales": {
+      "text": 0.94,
+      "pdf": 1.0,
+      "image": 1.0,
+      "audio": 1.0,
+      "video": 1.0
+    },
+    "tau": 0.1,
+    "delta": 0.0,
+    "radius": 1.004987562112089,
+    "block": 6,
+    "modalities": [
+      "text",
+      "pdf",
+      "image",
+      "audio",
+      "video"
+    ],
+    "preference": null,
+    "output_dimensions": 2696
+  }
+}

model/head/head.json

@@ -0,0 +1,32 @@
+{
+  "architecture": "Asymmetric 1024-to-640 retrieval head with additive audio projection",
+  "input_dim": 1024,
+  "output_dim": 640,
+  "modalities": [
+    "text",
+    "image",
+    "document",
+    "audio",
+    "video",
+    "interleaved",
+    "unknown"
+  ],
+  "image_projection": {
+    "epsilon": 0.2,
+    "modality": "image",
+    "native_dimensions": 1024,
+    "head_dimensions": 640,
+    "output_dimensions": 1664,
+    "weight": "image_projection.weight",
+    "bias": "image_projection.bias",
+    "parameters": 1049600
+  },
+  "audio_projection": {
+    "encoder": "pe-audio",
+    "encoder_sha256": "790f80f549c70cb327b30c537017ca2ae6f47781b219ce8cd35f0144b1e058d7",
+    "weight": "audio_projection.weight",
+    "bias": "audio_projection.bias",
+    "share": 0.2,
+    "parameters": 1049600
+  }
+}

model/head/head.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3922c5b7cb892977d61cb3e6caaa025fece854d5be2fba0e047503e1c17f4ccf
+size 16366128

model/lanes.py

@@ -0,0 +1,44 @@
+"""Per-modality calibration with six extra coordinates and unit document vectors."""
+
+from __future__ import annotations
+
+import numpy as np
+
+MODALITIES = ("text", "pdf", "image", "audio", "video")
+BLOCK = len(MODALITIES) + 1
+CATALOG_MODALITIES = {"text": "text", "document": "pdf", "image": "image", "audio": "audio", "video": "video"}
+
+
+class Calibration:
+    def __init__(self, scales: dict[str, float], tau: float, delta: float, dimension: int):
+        self.scales = {name: float(scales[name]) for name in MODALITIES}
+        self.tau = float(tau)
+        self.delta = float(delta)
+        self.dimension = int(dimension)
+        self.radius = float(np.sqrt(max(self.scales.values()) ** 2 + self.tau ** 2))
+        if not all(0 < s <= 1 for s in self.scales.values()) or not 0 < self.tau < 1 or self.delta != 0:
+            raise ValueError("Lane calibration needs scales in (0, 1], tau in (0, 1) and delta = 0")
+
+    @property
+    def output_dimension(self) -> int:
+        return self.dimension + BLOCK
+
+    def item(self, vector: np.ndarray, modality: str) -> np.ndarray:
+        """Extend one unit segment vector of a file of the specified modality."""
+        name = CATALOG_MODALITIES.get(modality, modality)
+        scale = self.scales[name]
+        values = np.asarray(vector, dtype=np.float32).reshape(-1)
+        if values.shape != (self.dimension,):
+            raise ValueError(f"segment vector must have {self.dimension} coordinates")
+        block = np.zeros(BLOCK, dtype=np.float32)
+        block[MODALITIES.index(name)] = self.tau
+        block[-1] = np.sqrt(max(0.0, self.radius ** 2 - scale ** 2 - self.tau ** 2))
+        return np.concatenate([scale * values, block]) / self.radius
+
+    def query(self, vector: np.ndarray) -> np.ndarray:
+        """Append the six zero coordinates used by query vectors."""
+        values = np.asarray(vector, dtype=np.float32).reshape(-1)
+        if values.shape != (self.dimension,):
+            raise ValueError(f"query vector must have {self.dimension} coordinates")
+        block = np.zeros(BLOCK, dtype=np.float32)
+        return np.concatenate([values, block])

model/language.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fea1441e6a0e56aba49d3cf2d0b92b36ee8142b8f98ed54b0e3bd30b1e909702
+size 631565822

model/licenses/jina-cc-by-nc-4.0.txt

@@ -0,0 +1,408 @@
+Attribution-NonCommercial 4.0 International
+
+=======================================================================
+
+Creative Commons Corporation ("Creative Commons") is not a law firm and
+does not provide legal services or legal advice. Distribution of
+Creative Commons public licenses does not create a lawyer-client or
+other relationship. Creative Commons makes its licenses and related
+information available on an "as-is" basis. Creative Commons gives no
+warranties regarding its licenses, any material licensed under their
+terms and conditions, or any related information. Creative Commons
+disclaims all liability for damages resulting from their use to the
+fullest extent possible.
+
+Using Creative Commons Public Licenses
+
+Creative Commons public licenses provide a standard set of terms and
+conditions that creators and other rights holders may use to share
+original works of authorship and other material subject to copyright
+and certain other rights specified in the public license below. The
+following considerations are for informational purposes only, are not
+exhaustive, and do not form part of our licenses.
+
+     Considerations for licensors: Our public licenses are
+     intended for use by those authorized to give the public
+     permission to use material in ways otherwise restricted by
+     copyright and certain other rights. Our licenses are
+     irrevocable. Licensors should read and understand the terms
+     and conditions of the license they choose before applying it.
+     Licensors should also secure all rights necessary before
+     applying our licenses so that the public can reuse the
+     material as expected. Licensors should clearly mark any
+     material not subject to the license. This includes other CC-
+     licensed material, or material used under an exception or
+     limitation to copyright. More considerations for licensors:
+    wiki.creativecommons.org/Considerations_for_licensors
+
+     Considerations for the public: By using one of our public
+     licenses, a licensor grants the public permission to use the
+     licensed material under specified terms and conditions. If
+     the licensor's permission is not necessary for any reason--for
+     example, because of any applicable exception or limitation to
+     copyright--then that use is not regulated by the license. Our
+     licenses grant only permissions under copyright and certain
+     other rights that a licensor has authority to grant. Use of
+     the licensed material may still be restricted for other
+     reasons, including because others have copyright or other
+     rights in the material. A licensor may make special requests,
+     such as asking that all changes be marked or described.
+     Although not required by our licenses, you are encouraged to
+     respect those requests where reasonable. More considerations
+     for the public:
+    wiki.creativecommons.org/Considerations_for_licensees
+
+=======================================================================
+
+Creative Commons Attribution-NonCommercial 4.0 International Public
+License
+
+By exercising the Licensed Rights (defined below), You accept and agree
+to be bound by the terms and conditions of this Creative Commons
+Attribution-NonCommercial 4.0 International Public License ("Public
+License"). To the extent this Public License may be interpreted as a
+contract, You are granted the Licensed Rights in consideration of Your
+acceptance of these terms and conditions, and the Licensor grants You
+such rights in consideration of benefits the Licensor receives from
+making the Licensed Material available under these terms and
+conditions.
+
+
+Section 1 -- Definitions.
+
+  a. Adapted Material means material subject to Copyright and Similar
+     Rights that is derived from or based upon the Licensed Material
+     and in which the Licensed Material is translated, altered,
+     arranged, transformed, or otherwise modified in a manner requiring
+     permission under the Copyright and Similar Rights held by the
+     Licensor. For purposes of this Public License, where the Licensed
+     Material is a musical work, performance, or sound recording,
+     Adapted Material is always produced where the Licensed Material is
+     synched in timed relation with a moving image.
+
+  b. Adapter's License means the license You apply to Your Copyright
+     and Similar Rights in Your contributions to Adapted Material in
+     accordance with the terms and conditions of this Public License.
+
+  c. Copyright and Similar Rights means copyright and/or similar rights
+     closely related to copyright including, without limitation,
+     performance, broadcast, sound recording, and Sui Generis Database
+     Rights, without regard to how the rights are labeled or
+     categorized. For purposes of this Public License, the rights
+     specified in Section 2(b)(1)-(2) are not Copyright and Similar
+     Rights.
+  d. Effective Technological Measures means those measures that, in the
+     absence of proper authority, may not be circumvented under laws
+     fulfilling obligations under Article 11 of the WIPO Copyright
+     Treaty adopted on December 20, 1996, and/or similar international
+     agreements.
+
+  e. Exceptions and Limitations means fair use, fair dealing, and/or
+     any other exception or limitation to Copyright and Similar Rights
+     that applies to Your use of the Licensed Material.
+
+  f. Licensed Material means the artistic or literary work, database,
+     or other material to which the Licensor applied this Public
+     License.
+
+  g. Licensed Rights means the rights granted to You subject to the
+     terms and conditions of this Public License, which are limited to
+     all Copyright and Similar Rights that apply to Your use of the
+     Licensed Material and that the Licensor has authority to license.
+
+  h. Licensor means the individual(s) or entity(ies) granting rights
+     under this Public License.
+
+  i. NonCommercial means not primarily intended for or directed towards
+     commercial advantage or monetary compensation. For purposes of
+     this Public License, the exchange of the Licensed Material for
+     other material subject to Copyright and Similar Rights by digital
+     file-sharing or similar means is NonCommercial provided there is
+     no payment of monetary compensation in connection with the
+     exchange.
+
+  j. Share means to provide material to the public by any means or
+     process that requires permission under the Licensed Rights, such
+     as reproduction, public display, public performance, distribution,
+     dissemination, communication, or importation, and to make material
+     available to the public including in ways that members of the
+     public may access the material from a place and at a time
+     individually chosen by them.
+
+  k. Sui Generis Database Rights means rights other than copyright
+     resulting from Directive 96/9/EC of the European Parliament and of
+     the Council of 11 March 1996 on the legal protection of databases,
+     as amended and/or succeeded, as well as other essentially
+     equivalent rights anywhere in the world.
+
+  l. You means the individual or entity exercising the Licensed Rights
+     under this Public License. Your has a corresponding meaning.
+
+
+Section 2 -- Scope.
+
+  a. License grant.
+
+       1. Subject to the terms and conditions of this Public License,
+          the Licensor hereby grants You a worldwide, royalty-free,
+          non-sublicensable, non-exclusive, irrevocable license to
+          exercise the Licensed Rights in the Licensed Material to:
+
+            a. reproduce and Share the Licensed Material, in whole or
+               in part, for NonCommercial purposes only; and
+
+            b. produce, reproduce, and Share Adapted Material for
+               NonCommercial purposes only.
+
+       2. Exceptions and Limitations. For the avoidance of doubt, where
+          Exceptions and Limitations apply to Your use, this Public
+          License does not apply, and You do not need to comply with
+          its terms and conditions.
+
+       3. Term. The term of this Public License is specified in Section
+          6(a).
+
+       4. Media and formats; technical modifications allowed. The
+          Licensor authorizes You to exercise the Licensed Rights in
+          all media and formats whether now known or hereafter created,
+          and to make technical modifications necessary to do so. The
+          Licensor waives and/or agrees not to assert any right or
+          authority to forbid You from making technical modifications
+          necessary to exercise the Licensed Rights, including
+          technical modifications necessary to circumvent Effective
+          Technological Measures. For purposes of this Public License,
+          simply making modifications authorized by this Section 2(a)
+          (4) never produces Adapted Material.
+
+       5. Downstream recipients.
+
+            a. Offer from the Licensor -- Licensed Material. Every
+               recipient of the Licensed Material automatically
+               receives an offer from the Licensor to exercise the
+               Licensed Rights under the terms and conditions of this
+               Public License.
+
+            b. No downstream restrictions. You may not offer or impose
+               any additional or different terms or conditions on, or
+               apply any Effective Technological Measures to, the
+               Licensed Material if doing so restricts exercise of the
+               Licensed Rights by any recipient of the Licensed
+               Material.
+
+       6. No endorsement. Nothing in this Public License constitutes or
+          may be construed as permission to assert or imply that You
+          are, or that Your use of the Licensed Material is, connected
+          with, or sponsored, endorsed, or granted official status by,
+          the Licensor or others designated to receive attribution as
+          provided in Section 3(a)(1)(A)(i).
+
+  b. Other rights.
+
+       1. Moral rights, such as the right of integrity, are not
+          licensed under this Public License, nor are publicity,
+          privacy, and/or other similar personality rights; however, to
+          the extent possible, the Licensor waives and/or agrees not to
+          assert any such rights held by the Licensor to the limited
+          extent necessary to allow You to exercise the Licensed
+          Rights, but not otherwise.
+
+       2. Patent and trademark rights are not licensed under this
+          Public License.
+
+       3. To the extent possible, the Licensor waives any right to
+          collect royalties from You for the exercise of the Licensed
+          Rights, whether directly or through a collecting society
+          under any voluntary or waivable statutory or compulsory
+          licensing scheme. In all other cases the Licensor expressly
+          reserves any right to collect such royalties, including when
+          the Licensed Material is used other than for NonCommercial
+          purposes.
+
+
+Section 3 -- License Conditions.
+
+Your exercise of the Licensed Rights is expressly made subject to the
+following conditions.
+
+  a. Attribution.
+
+       1. If You Share the Licensed Material (including in modified
+          form), You must:
+
+            a. retain the following if it is supplied by the Licensor
+               with the Licensed Material:
+
+                 i. identification of the creator(s) of the Licensed
+                    Material and any others designated to receive
+                    attribution, in any reasonable manner requested by
+                    the Licensor (including by pseudonym if
+                    designated);
+
+                ii. a copyright notice;
+
+               iii. a notice that refers to this Public License;
+
+                iv. a notice that refers to the disclaimer of
+                    warranties;
+
+                 v. a URI or hyperlink to the Licensed Material to the
+                    extent reasonably practicable;
+
+            b. indicate if You modified the Licensed Material and
+               retain an indication of any previous modifications; and
+
+            c. indicate the Licensed Material is licensed under this
+               Public License, and include the text of, or the URI or
+               hyperlink to, this Public License.
+
+       2. You may satisfy the conditions in Section 3(a)(1) in any
+          reasonable manner based on the medium, means, and context in
+          which You Share the Licensed Material. For example, it may be
+          reasonable to satisfy the conditions by providing a URI or
+          hyperlink to a resource that includes the required
+          information.
+
+       3. If requested by the Licensor, You must remove any of the
+          information required by Section 3(a)(1)(A) to the extent
+          reasonably practicable.
+
+       4. If You Share Adapted Material You produce, the Adapter's
+          License You apply must not prevent recipients of the Adapted
+          Material from complying with this Public License.
+
+
+Section 4 -- Sui Generis Database Rights.
+
+Where the Licensed Rights include Sui Generis Database Rights that
+apply to Your use of the Licensed Material:
+
+  a. for the avoidance of doubt, Section 2(a)(1) grants You the right
+     to extract, reuse, reproduce, and Share all or a substantial
+     portion of the contents of the database for NonCommercial purposes
+     only;
+
+  b. if You include all or a substantial portion of the database
+     contents in a database in which You have Sui Generis Database
+     Rights, then the database in which You have Sui Generis Database
+     Rights (but not its individual contents) is Adapted Material; and
+
+  c. You must comply with the conditions in Section 3(a) if You Share
+     all or a substantial portion of the contents of the database.
+
+For the avoidance of doubt, this Section 4 supplements and does not
+replace Your obligations under this Public License where the Licensed
+Rights include other Copyright and Similar Rights.
+
+
+Section 5 -- Disclaimer of Warranties and Limitation of Liability.
+
+  a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
+     EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
+     AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
+     ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
+     IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
+     WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
+     PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
+     ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
+     KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
+     ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
+
+  b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
+     TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
+     NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
+     INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
+     COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
+     USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
+     ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
+     DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
+     IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
+
+  c. The disclaimer of warranties and limitation of liability provided
+     above shall be interpreted in a manner that, to the extent
+     possible, most closely approximates an absolute disclaimer and
+     waiver of all liability.
+
+
+Section 6 -- Term and Termination.
+
+  a. This Public License applies for the term of the Copyright and
+     Similar Rights licensed here. However, if You fail to comply with
+     this Public License, then Your rights under this Public License
+     terminate automatically.
+
+  b. Where Your right to use the Licensed Material has terminated under
+     Section 6(a), it reinstates:
+
+       1. automatically as of the date the violation is cured, provided
+          it is cured within 30 days of Your discovery of the
+          violation; or
+
+       2. upon express reinstatement by the Licensor.
+
+     For the avoidance of doubt, this Section 6(b) does not affect any
+     right the Licensor may have to seek remedies for Your violations
+     of this Public License.
+
+  c. For the avoidance of doubt, the Licensor may also offer the
+     Licensed Material under separate terms or conditions or stop
+     distributing the Licensed Material at any time; however, doing so
+     will not terminate this Public License.
+
+  d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
+     License.
+
+
+Section 7 -- Other Terms and Conditions.
+
+  a. The Licensor shall not be bound by any additional or different
+     terms or conditions communicated by You unless expressly agreed.
+
+  b. Any arrangements, understandings, or agreements regarding the
+     Licensed Material not stated herein are separate from and
+     independent of the terms and conditions of this Public License.
+
+
+Section 8 -- Interpretation.
+
+  a. For the avoidance of doubt, this Public License does not, and
+     shall not be interpreted to, reduce, limit, restrict, or impose
+     conditions on any use of the Licensed Material that could lawfully
+     be made without permission under this Public License.
+
+  b. To the extent possible, if any provision of this Public License is
+     deemed unenforceable, it shall be automatically reformed to the
+     minimum extent necessary to make it enforceable. If the provision
+     cannot be reformed, it shall be severed from this Public License
+     without affecting the enforceability of the remaining terms and
+     conditions.
+
+  c. No term or condition of this Public License will be waived and no
+     failure to comply consented to unless expressly agreed to by the
+     Licensor.
+
+  d. Nothing in this Public License constitutes or may be interpreted
+     as a limitation upon, or waiver of, any privileges and immunities
+     that apply to the Licensor or You, including from the legal
+     processes of any jurisdiction or authority.
+
+=======================================================================
+
+Creative Commons is not a party to its public
+licenses. Notwithstanding, Creative Commons may elect to apply one of
+its public licenses to material it publishes and in those instances
+will be considered the “Licensor.” The text of the Creative Commons
+public licenses is dedicated to the public domain under the CC0 Public
+Domain Dedication. Except for the limited purpose of indicating that
+material is shared under a Creative Commons public license or as
+otherwise permitted by the Creative Commons policies published at
+creativecommons.org/policies, Creative Commons does not authorize the
+use of the trademark "Creative Commons" or any other trademark or logo
+of Creative Commons without its prior written consent including,
+without limitation, in connection with any unauthorized modifications
+to any of its public licenses or any other arrangements,
+understandings, or agreements concerning use of licensed material. For
+the avoidance of doubt, this paragraph does not form part of the
+public licenses.
+
+Creative Commons may be contacted at creativecommons.org.
+

model/licenses/meta-apache-2.0.txt

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

model/magnitude.py

@@ -0,0 +1,80 @@
+"""Unit embeddings with a fixed radial constraint on native acoustic features."""
+
+from __future__ import annotations
+
+import numpy as np
+
+
+CURRENT_DIMENSION = 1664
+RAW_DIMENSION = 1024
+DIMENSION = CURRENT_DIMENSION + RAW_DIMENSION + 2
+RAW_CAP = 8.0
+PADDING_RADIUS = 8.00001
+
+
+def matrix(vectors: np.ndarray, dimension: int, name: str) -> np.ndarray:
+    values = np.asarray(vectors, dtype=np.float64)
+    if values.ndim != 2 or values.shape[1] != dimension or not np.isfinite(values).all():
+        raise ValueError(f"{name} embeddings require {dimension} finite coordinates per row")
+    return values
+
+
+def unit(vectors: np.ndarray) -> np.ndarray:
+    values = matrix(vectors, CURRENT_DIMENSION, "Current")
+    peaks = np.max(np.abs(values), axis=1, keepdims=True)
+    if np.any(peaks == 0):
+        raise ValueError("Current embeddings must have nonzero magnitude")
+    scaled = values / peaks
+    return scaled / np.linalg.norm(scaled, axis=1, keepdims=True)
+
+
+def clip(vectors: np.ndarray) -> np.ndarray:
+    values = matrix(vectors, RAW_DIMENSION, "Raw PE")
+    peaks = np.max(np.abs(values), axis=1, keepdims=True)
+    # Scale before taking the norm so every finite input can be clipped in FP64.
+    scaled = values / np.maximum(1.0, peaks / RAW_CAP)
+    norms = np.linalg.norm(scaled, axis=1, keepdims=True)
+    return scaled * (RAW_CAP / np.maximum(RAW_CAP, norms))
+
+
+class Geometry:
+    def __init__(self, u: float, v: float):
+        self.u = float(u)
+        self.v = float(v)
+        if not np.isfinite((self.u, self.v)).all() or not .001 <= self.u <= .9999 or self.v < 0:
+            raise ValueError("Calibration requires finite .001 <= u <= .9999 and v >= 0")
+        self.beta = float(np.sqrt(self.u ** -2 - 1) / PADDING_RADIUS)
+        self.alpha = self.v / (self.u * self.beta)
+        self.query_divisor = float(np.hypot(1.0, self.alpha * PADDING_RADIUS))
+        self.audio_divisor = float(np.hypot(1.0, self.beta * PADDING_RADIUS))
+        if not np.isfinite(tuple(self.coefficients().values())).all():
+            raise ValueError("Calibration produces nonfinite lift coefficients")
+
+    def coefficients(self) -> dict[str, float]:
+        return {"u": self.u, "v": self.v, "alpha": self.alpha, "beta": self.beta,
+                "query_divisor": self.query_divisor, "audio_divisor": self.audio_divisor}
+
+    def _lift(self, vectors: np.ndarray, raw: np.ndarray, scale: float,
+              divisor: float, padding: int) -> np.ndarray:
+        original = unit(vectors)
+        acoustic = clip(raw)
+        if len(acoustic) != len(original):
+            raise ValueError("Every current embedding requires its own raw PE feature")
+        squared = PADDING_RADIUS ** 2 - np.square(acoustic).sum(axis=1)
+        if np.any(squared < 0):
+            raise ValueError("Clipped PE features exceed the fixed padding radius")
+        result = np.zeros((len(original), DIMENSION), dtype=np.float64)
+        result[:, :CURRENT_DIMENSION] = original
+        result[:, CURRENT_DIMENSION:-2] = scale * acoustic
+        result[:, -2 + padding] = scale * np.sqrt(squared)
+        result /= divisor
+        return result.astype(np.float32)
+
+    def query(self, current: np.ndarray, raw: np.ndarray) -> np.ndarray:
+        return self._lift(current, raw, self.alpha, self.query_divisor, 0)
+
+    def audio(self, current: np.ndarray, raw: np.ndarray) -> np.ndarray:
+        return self._lift(current, raw, self.beta, self.audio_divisor, 1)
+
+    def document(self, current: np.ndarray) -> np.ndarray:
+        return np.pad(unit(current), ((0, 0), (0, RAW_DIMENSION + 2))).astype(np.float32)

model/manifest.json

@@ -0,0 +1,91 @@
+{
+  "runtime": "nanobe-mlx-full",
+  "component": "full",
+  "source_revision": "12949877f0092093f366c6450340011320152a05",
+  "source_weights_sha256": "3d61fc55e1a457a9fc7f338df1ea1c183494716a4d4a32e291edd662757d5e8e",
+  "model_sha256": "553386da770249a30a0ba85febba50c9e9d40ff58c266c8ea3253f40dfe58963",
+  "weight_shards": [
+    {
+      "role": "language",
+      "file": "language.safetensors",
+      "sha256": "fea1441e6a0e56aba49d3cf2d0b92b36ee8142b8f98ed54b0e3bd30b1e909702",
+      "tensors": 704,
+      "bytes": 631565822
+    },
+    {
+      "role": "omni",
+      "file": "omni.safetensors",
+      "sha256": "8fd64d888f00a1cef89dd950e59b10bc8d60460e8d1e3f754b41592317329b36",
+      "tensors": 1369,
+      "bytes": 949948427
+    }
+  ],
+  "tokenizer_sha256": "8e08386ae43d4a5024a97dadbdc7291d6770135921c97e913644f991b3a87b8e",
+  "head_sha256": "3922c5b7cb892977d61cb3e6caaa025fece854d5be2fba0e047503e1c17f4ccf",
+  "model_geometry": {
+    "file": "geometry.json",
+    "sha256": "2c4a99d0c2b03c023ef29cc2cb15c1cf780fdf964e5b92b6bc095ca046000e92"
+  },
+  "audiovisual_runtime": {
+    "file": "nanobe_mlx_omni.py",
+    "sha256": "33fbaae67f93e8c9ae3816c0232ae51c8dbb2e1bbeb4ced3a69495bdd3d01773"
+  },
+  "media_adapter": {
+    "file": "media-adapter.safetensors",
+    "sha256": "f9812fd4f33aeaaa22854569753731e81115581df55a2337c72ba4655fe58a63",
+    "bytes": 82687236,
+    "tensors": 896,
+    "scale": 2.0,
+    "applies_to": [
+      "image",
+      "audio",
+      "video",
+      "text"
+    ],
+    "per_modality": true
+  },
+  "quantization": {
+    "targets": [
+      "linear",
+      "embedding"
+    ],
+    "bits": 6,
+    "group_size": 64,
+    "mode": "affine",
+    "overrides": {
+      "language_model": 8,
+      "audio_tower": 8,
+      "audio_projector": 8
+    },
+    "embedding_overrides": {
+      "language_model": 6
+    }
+  },
+  "package_weight_bytes": 2227193722,
+  "acoustic_encoder": {
+    "directory": "pe-audio",
+    "manifest_sha256": "790f80f549c70cb327b30c537017ca2ae6f47781b219ce8cd35f0144b1e058d7",
+    "parameters": 117389248,
+    "weight_bytes": 153819456
+  },
+  "acoustic_text_encoder": {
+    "directory": "pe-text",
+    "manifest_sha256": "97ad081b8044cb4308995a5d2d93d36bdec39a7ccfcf01e6591b7a5fd6b0eeff",
+    "parameters": 322288640,
+    "weight_bytes": 345597033
+  },
+  "page_merger": {
+    "file": "page-merger.safetensors",
+    "sha256": "db41396835aa23dd661711a39e9fa64c59957f46564741f2db26517e782d5a57",
+    "bytes": 41957888,
+    "tensors": 6,
+    "parameters": 20978688
+  },
+  "audio_pooler": {
+    "file": "audio-pooler.safetensors",
+    "sha256": "7bf8bc6420491454c3bb332bd243244d82efddfbcaa07cbc56dfb7603ab913f9",
+    "bytes": 5251732,
+    "tensors": 8,
+    "parameters": 1312769
+  }
+}

model/media-adapter.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f9812fd4f33aeaaa22854569753731e81115581df55a2337c72ba4655fe58a63
+size 82687236

model/model.py

@@ -0,0 +1,878 @@
+"""MLX architecture derived from Jina's omni-small retrieval model.
+
+The text backbone is a 28-layer Qwen3 causal decoder with 1,024 hidden
+coordinates, grouped-query attention and per-head query/key normalization.
+Qwen3VL vision layers encode spatial and temporal patches through a merger.
+The Qwen2.5-Omni audio tower uses convolutional features, sinusoidal positions
+and windowed attention. Its output projection maps into the text backbone.
+
+The runtime loads the language and omni weight shards listed in manifest.json.
+Embedding methods pool the last unmasked token; the runtime applies the
+retrieval projections and normalization for each input type.
+"""
+
+import math
+from dataclasses import dataclass, field
+from itertools import accumulate
+
+import mlx.core as mx
+import mlx.nn as nn
+
+# ---------------------------------------------------------------------------
+# Config
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class TextConfig:
+    hidden_size: int = 1024
+    num_hidden_layers: int = 28
+    intermediate_size: int = 3072
+    num_attention_heads: int = 16
+    num_key_value_heads: int = 8
+    head_dim: int = 128
+    rms_norm_eps: float = 1e-6
+    vocab_size: int = 151672
+    max_position_embeddings: int = 32768
+    rope_theta: float = 3500000.0
+
+
+@dataclass
+class VisionConfig:
+    hidden_size: int = 1024
+    depth: int = 24
+    num_heads: int = 16
+    intermediate_size: int = 4096
+    patch_size: int = 16
+    spatial_merge_size: int = 2
+    temporal_patch_size: int = 2
+    in_channels: int = 3
+    out_hidden_size: int = 1024
+    num_position_embeddings: int = 2304
+
+
+@dataclass
+class AudioConfig:
+    """Qwen2.5-Omni audio encoder config; only fields the MLX port uses.
+
+    The base repo's `audio_tower.proj` is fused into `audio_projector`
+    (Linear 1280 -> 1024), so `output_dim` (3584) is not used at inference
+    and the `audio_tower.proj` weights are absent from the safetensors.
+    """
+
+    d_model: int = 1280
+    encoder_layers: int = 32
+    encoder_attention_heads: int = 20
+    encoder_ffn_dim: int = 5120
+    max_source_positions: int = 1500
+    n_window: int = 100
+    num_mel_bins: int = 128
+    scale_embedding: bool = False
+
+
+IMAGE_TOKEN_ID = 151655
+VIDEO_TOKEN_ID = 151656
+AUDIO_TOKEN_ID = 151669  # <|AUDIO|> placeholder in the text sequence
+
+
+@dataclass
+class OmniSmallConfig:
+    text_config: TextConfig = field(default_factory=TextConfig)
+    vision_config: VisionConfig = field(default_factory=VisionConfig)
+    audio_config: AudioConfig = field(default_factory=AudioConfig)
+    image_token_id: int = IMAGE_TOKEN_ID
+    video_token_id: int = VIDEO_TOKEN_ID
+    audio_token_id: int = AUDIO_TOKEN_ID
+
+    @classmethod
+    def from_dict(cls, d: dict) -> "OmniSmallConfig":
+        tc = d.get("text_config", {})
+        vc = d.get("vision_config", {})
+        ac = d.get("audio_config", {})
+        rope_params = tc.get("rope_parameters", {})
+        return cls(
+            text_config=TextConfig(
+                hidden_size=tc.get("hidden_size", 1024),
+                num_hidden_layers=tc.get("num_hidden_layers", 28),
+                intermediate_size=tc.get("intermediate_size", 3072),
+                num_attention_heads=tc.get("num_attention_heads", 16),
+                num_key_value_heads=tc.get("num_key_value_heads", 8),
+                head_dim=tc.get("head_dim", 128),
+                rms_norm_eps=tc.get("rms_norm_eps", 1e-6),
+                vocab_size=tc.get("vocab_size", 151672),
+                max_position_embeddings=tc.get("max_position_embeddings", 32768),
+                rope_theta=rope_params.get(
+                    "rope_theta", tc.get("rope_theta", 3500000.0)
+                ),
+            ),
+            vision_config=VisionConfig(
+                hidden_size=vc.get("hidden_size", 1024),
+                depth=vc.get("depth", 24),
+                num_heads=vc.get("num_heads", 16),
+                intermediate_size=vc.get("intermediate_size", 4096),
+                patch_size=vc.get("patch_size", 16),
+                spatial_merge_size=vc.get("spatial_merge_size", 2),
+                temporal_patch_size=vc.get("temporal_patch_size", 2),
+                in_channels=vc.get("in_channels", 3),
+                out_hidden_size=vc.get("out_hidden_size", 1024),
+                num_position_embeddings=vc.get("num_position_embeddings", 2304),
+            ),
+            audio_config=AudioConfig(
+                d_model=ac.get("d_model", 1280),
+                encoder_layers=ac.get("encoder_layers", 32),
+                encoder_attention_heads=ac.get("encoder_attention_heads", 20),
+                encoder_ffn_dim=ac.get("encoder_ffn_dim", 5120),
+                max_source_positions=ac.get("max_source_positions", 1500),
+                n_window=ac.get("n_window", 100),
+                num_mel_bins=ac.get("num_mel_bins", 128),
+                scale_embedding=ac.get("scale_embedding", False),
+            ),
+            image_token_id=d.get("image_token_id", IMAGE_TOKEN_ID),
+            video_token_id=d.get("video_token_id", VIDEO_TOKEN_ID),
+            audio_token_id=d.get("audio_token_id", AUDIO_TOKEN_ID),
+        )
+
+
+# ---------------------------------------------------------------------------
+# Text model (Qwen3 causal, GQA 16/8 with per-head q_norm / k_norm)
+# ---------------------------------------------------------------------------
+
+
+class TextAttention(nn.Module):
+    def __init__(self, cfg: TextConfig):
+        super().__init__()
+        self.n_heads = cfg.num_attention_heads
+        self.n_kv_heads = cfg.num_key_value_heads
+        self.head_dim = cfg.head_dim
+        self.scale = self.head_dim**-0.5
+
+        dim = cfg.hidden_size
+        self.q_proj = nn.Linear(dim, self.n_heads * self.head_dim, bias=False)
+        self.k_proj = nn.Linear(dim, self.n_kv_heads * self.head_dim, bias=False)
+        self.v_proj = nn.Linear(dim, self.n_kv_heads * self.head_dim, bias=False)
+        self.o_proj = nn.Linear(self.n_heads * self.head_dim, dim, bias=False)
+        # Normalize each query and key head with RMSNorm.
+        self.q_norm = nn.RMSNorm(self.head_dim, eps=cfg.rms_norm_eps)
+        self.k_norm = nn.RMSNorm(self.head_dim, eps=cfg.rms_norm_eps)
+        self.rope_theta = cfg.rope_theta
+
+    def __call__(self, x, mask=None):
+        B, L, _ = x.shape
+        q = (
+            self.q_proj(x)
+            .reshape(B, L, self.n_heads, self.head_dim)
+            .transpose(0, 2, 1, 3)
+        )
+        k = (
+            self.k_proj(x)
+            .reshape(B, L, self.n_kv_heads, self.head_dim)
+            .transpose(0, 2, 1, 3)
+        )
+        v = (
+            self.v_proj(x)
+            .reshape(B, L, self.n_kv_heads, self.head_dim)
+            .transpose(0, 2, 1, 3)
+        )
+
+        # Qwen3: RMSNorm per head BEFORE rope.
+        q = self.q_norm(q)
+        k = self.k_norm(k)
+
+        q = mx.fast.rope(
+            q,
+            self.head_dim,
+            traditional=False,
+            base=self.rope_theta,
+            scale=1.0,
+            offset=0,
+        )
+        k = mx.fast.rope(
+            k,
+            self.head_dim,
+            traditional=False,
+            base=self.rope_theta,
+            scale=1.0,
+            offset=0,
+        )
+
+        out = mx.fast.scaled_dot_product_attention(
+            q,
+            k,
+            v,
+            mask=mask.astype(q.dtype) if mask is not None else None,
+            scale=self.scale,
+        )
+        out = out.transpose(0, 2, 1, 3).reshape(B, L, -1)
+        return self.o_proj(out)
+
+
+class TextMLP(nn.Module):
+    def __init__(self, dim, hidden_dim):
+        super().__init__()
+        self.gate_proj = nn.Linear(dim, hidden_dim, bias=False)
+        self.down_proj = nn.Linear(hidden_dim, dim, bias=False)
+        self.up_proj = nn.Linear(dim, hidden_dim, bias=False)
+
+    def __call__(self, x):
+        return self.down_proj(nn.silu(self.gate_proj(x)) * self.up_proj(x))
+
+
+class TextBlock(nn.Module):
+    def __init__(self, cfg: TextConfig):
+        super().__init__()
+        self.self_attn = TextAttention(cfg)
+        self.mlp = TextMLP(cfg.hidden_size, cfg.intermediate_size)
+        self.input_layernorm = nn.RMSNorm(cfg.hidden_size, eps=cfg.rms_norm_eps)
+        self.post_attention_layernorm = nn.RMSNorm(
+            cfg.hidden_size, eps=cfg.rms_norm_eps
+        )
+
+    def __call__(self, x, mask=None):
+        x = x + self.self_attn(self.input_layernorm(x), mask)
+        x = x + self.mlp(self.post_attention_layernorm(x))
+        return x
+
+
+class TextModel(nn.Module):
+    def __init__(self, cfg: TextConfig):
+        super().__init__()
+        self.embed_tokens = nn.Embedding(cfg.vocab_size, cfg.hidden_size)
+        self.layers = [TextBlock(cfg) for _ in range(cfg.num_hidden_layers)]
+        self.norm = nn.RMSNorm(cfg.hidden_size, eps=cfg.rms_norm_eps)
+
+    def __call__(self, input_ids=None, inputs_embeds=None, mask=None):
+        h = inputs_embeds if inputs_embeds is not None else self.embed_tokens(input_ids)
+        for layer in self.layers:
+            h = layer(h, mask)
+        return self.norm(h)
+
+
+# ---------------------------------------------------------------------------
+# Qwen3VL vision transformer
+# ---------------------------------------------------------------------------
+
+
+def rotate_half(x):
+    x1 = x[..., : x.shape[-1] // 2]
+    x2 = x[..., x.shape[-1] // 2 :]
+    return mx.concatenate([-x2, x1], axis=-1)
+
+
+def apply_rotary_pos_emb_vision(tensor, freqs):
+    cos = mx.cos(freqs)
+    sin = mx.sin(freqs)
+    cos = mx.expand_dims(mx.tile(mx.expand_dims(cos, 1), (1, 1, 2)), 0)
+    sin = mx.expand_dims(mx.tile(mx.expand_dims(sin, 1), (1, 1, 2)), 0)
+    return (tensor * cos) + (rotate_half(tensor) * sin)
+
+
+class VisionRotaryEmbedding(nn.Module):
+    def __init__(self, dim, theta=10000.0):
+        super().__init__()
+        self.dim = dim
+        self.theta = theta
+
+    def __call__(self, seqlen):
+        inv_freq = 1.0 / (
+            self.theta ** (mx.arange(0, self.dim, 2, dtype=mx.float32) / self.dim)
+        )
+        seq = mx.arange(seqlen, dtype=inv_freq.dtype)
+        return mx.outer(seq, inv_freq)
+
+
+class VisionPatchEmbed(nn.Module):
+    def __init__(self, cfg: VisionConfig):
+        super().__init__()
+        self.patch_size = cfg.patch_size
+        self.temporal_patch_size = cfg.temporal_patch_size
+        self.in_channels = cfg.in_channels
+        self.hidden_size = cfg.hidden_size
+        self.proj = nn.Conv3d(
+            cfg.in_channels,
+            cfg.hidden_size,
+            kernel_size=[cfg.temporal_patch_size, cfg.patch_size, cfg.patch_size],
+            stride=[cfg.temporal_patch_size, cfg.patch_size, cfg.patch_size],
+            bias=True,
+        )
+
+    def __call__(self, x):
+        x = x.reshape(
+            -1,
+            self.in_channels,
+            self.temporal_patch_size,
+            self.patch_size,
+            self.patch_size,
+        )
+        x = x.moveaxis(1, 4)
+        x = self.proj(x)
+        return x.reshape(-1, self.hidden_size)
+
+
+class VisionAttention(nn.Module):
+    def __init__(self, dim, num_heads):
+        super().__init__()
+        self.num_heads = num_heads
+        self.head_dim = dim // num_heads
+        self.scale = self.head_dim**-0.5
+        self.qkv = nn.Linear(dim, dim * 3, bias=True)
+        self.proj = nn.Linear(dim, dim)
+
+    def __call__(self, x, cu_seqlens, rotary_pos_emb=None):
+        seq_length = x.shape[0]
+        qkv = (
+            self.qkv(x).reshape(seq_length, 3, self.num_heads, -1).transpose(1, 0, 2, 3)
+        )
+        q, k, v = mx.split(qkv, 3)
+        if rotary_pos_emb is not None:
+            q = apply_rotary_pos_emb_vision(mx.expand_dims(q, 0), rotary_pos_emb)[0]
+            k = apply_rotary_pos_emb_vision(mx.expand_dims(k, 0), rotary_pos_emb)[0]
+        q = q.transpose(0, 2, 1, 3)
+        k = k.transpose(0, 2, 1, 3)
+        v = v.transpose(0, 2, 1, 3)
+
+        splits = [mx.split(t, cu_seqlens[1:-1].tolist(), axis=2) for t in (q, k, v)]
+        attn_outputs = []
+        for qi, ki, vi in zip(*splits):
+            attn_outputs.append(
+                mx.fast.scaled_dot_product_attention(qi, ki, vi, scale=self.scale)
+            )
+        output = mx.concatenate(attn_outputs, axis=2)
+        output = output.transpose(0, 2, 1, 3).reshape(seq_length, -1)
+        return self.proj(output)
+
+
+class VisionMLP(nn.Module):
+    def __init__(self, dim, hidden_dim):
+        super().__init__()
+        self.linear_fc1 = nn.Linear(dim, hidden_dim, bias=True)
+        self.linear_fc2 = nn.Linear(hidden_dim, dim, bias=True)
+        self.act_fn = nn.GELU(approx="tanh")
+
+    def __call__(self, x):
+        return self.linear_fc2(self.act_fn(self.linear_fc1(x)))
+
+
+class VisionBlock(nn.Module):
+    def __init__(self, cfg: VisionConfig):
+        super().__init__()
+        self.norm1 = nn.LayerNorm(cfg.hidden_size, eps=1e-6)
+        self.norm2 = nn.LayerNorm(cfg.hidden_size, eps=1e-6)
+        self.attn = VisionAttention(cfg.hidden_size, cfg.num_heads)
+        self.mlp = VisionMLP(cfg.hidden_size, cfg.intermediate_size)
+
+    def __call__(self, x, cu_seqlens, rotary_pos_emb):
+        x = x + self.attn(self.norm1(x), cu_seqlens, rotary_pos_emb)
+        x = x + self.mlp(self.norm2(x))
+        return x
+
+
+class VisionModel(nn.Module):
+    def __init__(self, cfg: VisionConfig):
+        super().__init__()
+        self.spatial_merge_size = cfg.spatial_merge_size
+        self.patch_embed = VisionPatchEmbed(cfg)
+        head_dim = cfg.hidden_size // cfg.num_heads
+        self.rotary_pos_emb = VisionRotaryEmbedding(head_dim // 2)
+        self.pos_embed = nn.Embedding(cfg.num_position_embeddings, cfg.hidden_size)
+        self.num_grid_per_side = int(cfg.num_position_embeddings**0.5)
+        self.blocks = [VisionBlock(cfg) for _ in range(cfg.depth)]
+
+    def rot_pos_emb(self, grid_thw):
+        merge_size = self.spatial_merge_size
+        max_hw = int(mx.max(grid_thw[:, 1:]).item())
+        freq_table = self.rotary_pos_emb(max_hw)
+        pos_ids = []
+        for num_frames, height, width in grid_thw.tolist():
+            num_frames, height, width = int(num_frames), int(height), int(width)
+            merged_h, merged_w = height // merge_size, width // merge_size
+            block_rows = mx.arange(merged_h)
+            block_cols = mx.arange(merged_w)
+            intra_row = mx.arange(merge_size)
+            intra_col = mx.arange(merge_size)
+            row_idx = (
+                block_rows[:, None, None, None] * merge_size
+                + intra_row[None, None, :, None]
+            )
+            col_idx = (
+                block_cols[None, :, None, None] * merge_size
+                + intra_col[None, None, None, :]
+            )
+            row_idx = mx.broadcast_to(
+                row_idx, (merged_h, merged_w, merge_size, merge_size)
+            ).reshape(-1)
+            col_idx = mx.broadcast_to(
+                col_idx, (merged_h, merged_w, merge_size, merge_size)
+            ).reshape(-1)
+            coords = mx.stack([row_idx, col_idx], axis=-1)
+            if num_frames > 1:
+                coords = mx.tile(coords, (num_frames, 1))
+            pos_ids.append(coords)
+        pos_ids = mx.concatenate(pos_ids, axis=0)
+        h_emb = freq_table[pos_ids[:, 0]]
+        w_emb = freq_table[pos_ids[:, 1]]
+        return mx.concatenate([h_emb, w_emb], axis=-1)
+
+    def fast_pos_embed_interpolate(self, grid_thw):
+        grid_thw_list = grid_thw.tolist()
+        idx_list = [[] for _ in range(4)]
+        weight_list = [[] for _ in range(4)]
+        for t, h, w in grid_thw_list:
+            h, w, t = int(h), int(w), int(t)
+            h_idxs = mx.linspace(0, self.num_grid_per_side - 1, h)
+            w_idxs = mx.linspace(0, self.num_grid_per_side - 1, w)
+            h_floor = h_idxs.astype(mx.int32)
+            w_floor = w_idxs.astype(mx.int32)
+            h_ceil = mx.minimum(h_floor + 1, self.num_grid_per_side - 1)
+            w_ceil = mx.minimum(w_floor + 1, self.num_grid_per_side - 1)
+            dh = h_idxs - h_floor.astype(mx.float32)
+            dw = w_idxs - w_floor.astype(mx.float32)
+            base_h = h_floor * self.num_grid_per_side
+            base_h_ceil = h_ceil * self.num_grid_per_side
+            indices = [
+                (base_h[:, None] + w_floor[None, :]).flatten(),
+                (base_h[:, None] + w_ceil[None, :]).flatten(),
+                (base_h_ceil[:, None] + w_floor[None, :]).flatten(),
+                (base_h_ceil[:, None] + w_ceil[None, :]).flatten(),
+            ]
+            weights = [
+                ((1 - dh)[:, None] * (1 - dw)[None, :]).flatten(),
+                ((1 - dh)[:, None] * dw[None, :]).flatten(),
+                (dh[:, None] * (1 - dw)[None, :]).flatten(),
+                (dh[:, None] * dw[None, :]).flatten(),
+            ]
+            for i in range(4):
+                idx_list[i].extend(indices[i].tolist())
+                weight_list[i].extend(weights[i].tolist())
+        idx_tensor = mx.array(idx_list, dtype=mx.int32)
+        pos_embeds = self.pos_embed(idx_tensor)
+        # Packed embedding weights are integers; interpolation uses the decoded dtype.
+        weight_tensor = mx.array(weight_list, dtype=pos_embeds.dtype)
+        pos_embeds = pos_embeds * weight_tensor[:, :, None]
+        patch_pos_embeds = pos_embeds[0] + pos_embeds[1] + pos_embeds[2] + pos_embeds[3]
+        split_sizes = [int(h * w) for t, h, w in grid_thw_list]
+        if len(split_sizes) > 1:
+            split_indices = list(accumulate(split_sizes[:-1]))
+            parts = mx.split(patch_pos_embeds, split_indices, axis=0)
+        else:
+            parts = [patch_pos_embeds]
+        merge_size = self.spatial_merge_size
+        result = []
+        for pos_embed, (t, h, w) in zip(parts, grid_thw_list):
+            t, h, w = int(t), int(h), int(w)
+            fd = pos_embed.shape[-1]
+            pos_embed = mx.tile(pos_embed, (t, 1))
+            pos_embed = pos_embed.reshape(t, h, w, fd)
+            pos_embed = (
+                pos_embed.reshape(
+                    t, h // merge_size, merge_size, w // merge_size, merge_size, fd
+                )
+                .transpose(0, 1, 3, 2, 4, 5)
+                .reshape(-1, fd)
+            )
+            result.append(pos_embed)
+        return mx.concatenate(result)
+
+    def __call__(self, pixel_values, grid_thw):
+        hidden = self.patch_embed(pixel_values)
+        pos_embeds = self.fast_pos_embed_interpolate(grid_thw)
+        hidden = hidden + pos_embeds
+        rotary = self.rot_pos_emb(grid_thw)
+        seq_len = hidden.shape[0]
+        hidden = hidden.reshape(seq_len, -1)
+        rotary = rotary.reshape(seq_len, -1)
+
+        cu_seqlens = []
+        for i in range(grid_thw.shape[0]):
+            seq_len_i = grid_thw[i, 1] * grid_thw[i, 2]
+            cu_seqlens.append(mx.repeat(seq_len_i, grid_thw[i, 0]))
+        cu_seqlens = mx.concatenate(cu_seqlens)
+        cu_seqlens = mx.cumsum(cu_seqlens.astype(mx.int32), axis=0)
+        cu_seqlens = mx.pad(cu_seqlens, (1, 0), mode="constant", constant_values=0)
+
+        for blk in self.blocks:
+            hidden = blk(hidden, cu_seqlens, rotary)
+        return hidden
+
+
+# ---------------------------------------------------------------------------
+# Spatial patch merger
+# ---------------------------------------------------------------------------
+
+
+class Merger(nn.Module):
+    def __init__(self, cfg: VisionConfig, text_hidden_size: int):
+        super().__init__()
+        self.hidden_size = cfg.hidden_size * (cfg.spatial_merge_size**2)
+        self.norm = nn.LayerNorm(cfg.hidden_size, eps=1e-6)
+        self.linear_fc1 = nn.Linear(self.hidden_size, self.hidden_size, bias=True)
+        self.act = nn.GELU()
+        self.linear_fc2 = nn.Linear(self.hidden_size, text_hidden_size, bias=True)
+
+    def __call__(self, x):
+        x = self.norm(x).reshape(-1, self.hidden_size)
+        return self.linear_fc2(self.act(self.linear_fc1(x)))
+
+
+# ---------------------------------------------------------------------------
+# Audio tower (Qwen2.5-Omni; fused proj collapsed into `audio_projector`)
+# ---------------------------------------------------------------------------
+
+
+def _sinusoids_position_embedding(length: int, channels: int, max_timescale: float = 10000.0) -> mx.array:
+    """Whisper-style sinusoidal position embedding, computed (not learned)."""
+    assert channels % 2 == 0, "channels must be even for sinusoidal embeddings"
+    log_timescale_increment = math.log(max_timescale) / (channels // 2 - 1)
+    inv_timescales = mx.exp(
+        -log_timescale_increment * mx.arange(channels // 2, dtype=mx.float32)
+    )
+    t = mx.arange(length, dtype=mx.float32)
+    scaled_time = t[:, None] * inv_timescales[None, :]
+    return mx.concatenate([mx.sin(scaled_time), mx.cos(scaled_time)], axis=-1)
+
+
+class AudioAttention(nn.Module):
+    """Windowed attention over chunks defined by `cu_seqlens`.
+
+    Matches Qwen2_5OmniAudioAttention: only q/v/o have biases; k has no bias.
+    """
+
+    def __init__(self, cfg: AudioConfig):
+        super().__init__()
+        self.num_heads = cfg.encoder_attention_heads
+        self.head_dim = cfg.d_model // cfg.encoder_attention_heads
+        self.scale = self.head_dim**-0.5
+        self.k_proj = nn.Linear(cfg.d_model, cfg.d_model, bias=False)
+        self.v_proj = nn.Linear(cfg.d_model, cfg.d_model, bias=True)
+        self.q_proj = nn.Linear(cfg.d_model, cfg.d_model, bias=True)
+        self.out_proj = nn.Linear(cfg.d_model, cfg.d_model, bias=True)
+
+    def __call__(self, x, cu_seqlens):
+        seq_length = x.shape[0]
+        q = self.q_proj(x).reshape(seq_length, self.num_heads, -1).transpose(1, 0, 2)
+        k = self.k_proj(x).reshape(seq_length, self.num_heads, -1).transpose(1, 0, 2)
+        v = self.v_proj(x).reshape(seq_length, self.num_heads, -1).transpose(1, 0, 2)
+
+        q = mx.expand_dims(q, 0)
+        k = mx.expand_dims(k, 0)
+        v = mx.expand_dims(v, 0)
+
+        splits = [mx.split(t, cu_seqlens[1:-1].tolist(), axis=2) for t in (q, k, v)]
+        attn_outputs = []
+        for qi, ki, vi in zip(*splits):
+            attn_outputs.append(
+                mx.fast.scaled_dot_product_attention(qi, ki, vi, scale=self.scale)
+            )
+        output = mx.concatenate(attn_outputs, axis=2)
+        output = output[0].transpose(1, 0, 2).reshape(seq_length, -1)
+        return self.out_proj(output)
+
+
+class AudioEncoderLayer(nn.Module):
+    def __init__(self, cfg: AudioConfig):
+        super().__init__()
+        self.self_attn = AudioAttention(cfg)
+        self.self_attn_layer_norm = nn.LayerNorm(cfg.d_model)
+        self.fc1 = nn.Linear(cfg.d_model, cfg.encoder_ffn_dim, bias=True)
+        self.fc2 = nn.Linear(cfg.encoder_ffn_dim, cfg.d_model, bias=True)
+        self.final_layer_norm = nn.LayerNorm(cfg.d_model)
+        self.act_fn = nn.GELU()
+
+    def __call__(self, x, cu_seqlens):
+        x = x + self.self_attn(self.self_attn_layer_norm(x), cu_seqlens)
+        x = x + self.fc2(self.act_fn(self.fc1(self.final_layer_norm(x))))
+        return x
+
+
+class AudioModel(nn.Module):
+    """Qwen2.5-Omni audio encoder, with `proj` fused out.
+
+    Weights use the `audio_tower.*` names. The forward pass consumes audio
+    features without separate beginning or end token embeddings.
+    """
+
+    def __init__(self, cfg: AudioConfig):
+        super().__init__()
+        self.cfg = cfg
+        self.conv1 = nn.Conv1d(cfg.num_mel_bins, cfg.d_model, kernel_size=3, padding=1)
+        self.conv2 = nn.Conv1d(cfg.d_model, cfg.d_model, kernel_size=3, stride=2, padding=1)
+        self.layers = [AudioEncoderLayer(cfg) for _ in range(cfg.encoder_layers)]
+        self.ln_post = nn.LayerNorm(cfg.d_model)
+        # self.proj is fused into audio_projector; represent as identity.
+        self.proj = nn.Identity()
+        self.act_fn = nn.GELU()
+
+        self._pos_table = _sinusoids_position_embedding(
+            cfg.max_source_positions, cfg.d_model
+        )
+
+    @staticmethod
+    def feat_extract_output_lengths(input_lengths: mx.array) -> tuple[mx.array, mx.array]:
+        """Mirrors `Qwen2_5OmniAudioEncoder._get_feat_extract_output_lengths`."""
+        input_lengths = (input_lengths - 1) // 2 + 1
+        output_lengths = (input_lengths - 2) // 2 + 1
+        return input_lengths, output_lengths
+
+    def __call__(self, input_features: mx.array, feature_lens: mx.array) -> mx.array:
+        cfg = self.cfg
+        window_size = cfg.n_window * 2
+
+        feature_lens_list = feature_lens.tolist()
+        chunk_lengths: list[int] = []
+        for L in feature_lens_list:
+            L = int(L)
+            full_chunks, tail = divmod(L, window_size)
+            chunk_lengths.extend([window_size] * full_chunks)
+            if tail > 0:
+                chunk_lengths.append(tail)
+            elif L == 0:
+                chunk_lengths.append(0)
+
+        total_T = int(sum(chunk_lengths))
+        assert total_T == int(feature_lens.sum().item()), (
+            f"chunk sum {total_T} != feature sum {int(feature_lens.sum().item())}"
+        )
+        max_len = window_size
+        num_chunks = len(chunk_lengths)
+        C = cfg.num_mel_bins
+        padded = mx.zeros((num_chunks, max_len, C), dtype=input_features.dtype)
+        pad_mask = mx.zeros((num_chunks, max_len), dtype=mx.bool_)
+
+        features_tc = input_features.T
+        offset = 0
+        for i, L in enumerate(chunk_lengths):
+            if L > 0:
+                padded[i, :L, :] = features_tc[offset : offset + L, :]
+                pad_mask[i, :L] = True
+            offset += L
+
+        h = self.act_fn(self.conv1(padded)) * mx.expand_dims(
+            pad_mask.astype(padded.dtype), -1
+        )
+        h = self.act_fn(self.conv2(h))
+
+        chunk_len_arr = mx.array(chunk_lengths, dtype=mx.int32)
+        aftercnn_lengths, _ = self.feat_extract_output_lengths(chunk_len_arr)
+        aftercnn_lengths_list = aftercnn_lengths.tolist()
+
+        max_after_cnn = h.shape[1]
+        pos = self._pos_table[:max_after_cnn, :].astype(h.dtype)
+        h = h + mx.expand_dims(pos, 0)
+
+        flat_chunks = []
+        cu_seqlens_list = [0]
+        for i, L_after in enumerate(aftercnn_lengths_list):
+            L_after = int(L_after)
+            flat_chunks.append(h[i, :L_after, :])
+            cu_seqlens_list.append(cu_seqlens_list[-1] + L_after)
+        hidden = mx.concatenate(flat_chunks, axis=0)
+        cu_seqlens = mx.array(cu_seqlens_list, dtype=mx.int32)
+
+        for layer in self.layers:
+            hidden = layer(hidden, cu_seqlens)
+
+        audio_aftercnn_total = []
+        chunk_idx = 0
+        for L in feature_lens_list:
+            L = int(L)
+            full_chunks, tail = divmod(L, window_size)
+            n_chunks_for_this_audio = full_chunks + (1 if tail > 0 else 0)
+            if L == 0:
+                n_chunks_for_this_audio = max(n_chunks_for_this_audio, 1)
+            total_aftercnn = sum(
+                int(aftercnn_lengths_list[chunk_idx + k])
+                for k in range(n_chunks_for_this_audio)
+            )
+            audio_aftercnn_total.append(total_aftercnn)
+            chunk_idx += n_chunks_for_this_audio
+
+        per_audio = []
+        offset = 0
+        for total_after in audio_aftercnn_total:
+            seg = hidden[offset : offset + total_after, :]
+            T_even = (seg.shape[0] // 2) * 2
+            seg = seg[:T_even, :].reshape(T_even // 2, 2, seg.shape[1]).mean(axis=1)
+            seg = self.ln_post(seg)
+            seg = self.proj(seg)
+            per_audio.append(seg)
+            offset += total_after
+        return mx.concatenate(per_audio, axis=0)
+
+
+# ---------------------------------------------------------------------------
+# Pooling helpers
+# ---------------------------------------------------------------------------
+
+
+def _last_token_pool(hidden: mx.array, attention_mask: mx.array | None) -> mx.array:
+    """Gather the last non-pad token per sequence."""
+    B = hidden.shape[0]
+    if attention_mask is not None:
+        idx = (attention_mask.sum(axis=1) - 1).astype(mx.int32)
+    else:
+        idx = mx.full((B,), hidden.shape[1] - 1, dtype=mx.int32)
+    pooled_rows = [hidden[b, int(idx[b].item()), :] for b in range(B)]
+    pooled = mx.stack(pooled_rows, axis=0)
+    return pooled / mx.linalg.norm(pooled, axis=-1, keepdims=True)
+
+
+def _causal_pad_mask(
+    attention_mask: mx.array | None,
+    seq_len: int,
+    reference_param: mx.array,
+) -> mx.array | None:
+    """Build the additive attention mask for a causal decoder with right-padding.
+
+    Produces a `[B, 1, L, L]` float tensor with
+        - -inf on strictly upper-triangular positions (causal)
+        - -inf on pad key positions across all queries
+        - 0 elsewhere
+
+    Dtype matches `reference_param` so SDP ingests it directly.
+    """
+    # Causal base: strictly upper triangular -inf (disallow future positions).
+    causal = mx.triu(
+        mx.full((seq_len, seq_len), -1e9, dtype=reference_param.dtype), k=1
+    )
+    causal = causal[None, None, :, :]  # broadcast over batch + heads
+
+    if attention_mask is None:
+        return causal
+
+    pad = mx.where(attention_mask == 0, -1e9, 0.0)[:, None, None, :]
+    pad = pad.astype(reference_param.dtype)
+    return causal + pad
+
+
+# ---------------------------------------------------------------------------
+# Full omni-small model (MLX, text + image + video + audio)
+# ---------------------------------------------------------------------------
+
+
+class JinaOmniSmallEmbeddingModel(nn.Module):
+    def __init__(self, config: OmniSmallConfig):
+        super().__init__()
+        self.config = config
+        self.vision_tower = VisionModel(config.vision_config)
+        self.merger = Merger(config.vision_config, config.text_config.hidden_size)
+        self.audio_tower = AudioModel(config.audio_config)
+        self.audio_projector = nn.Linear(
+            config.audio_config.d_model,
+            config.text_config.hidden_size,
+            bias=True,
+        )
+        self.language_model = TextModel(config.text_config)
+
+    # ------------------------------------------------------------------ text
+    def encode_text(self, input_ids, attention_mask=None):
+        mask = _causal_pad_mask(
+            attention_mask,
+            input_ids.shape[1],
+            self.language_model.layers[0].input_layernorm.weight,
+        )
+        hidden = self.language_model(input_ids=input_ids, mask=mask)
+        mx.eval(hidden)
+        emb = _last_token_pool(hidden, attention_mask)
+        mx.eval(emb)
+        return emb
+
+    # ------------------------------------------------------------- image/video
+    def _inject_features(self, input_ids, inputs_embeds, placeholder_id, features):
+        """Scatter `features` into `inputs_embeds` at `placeholder_id` slots."""
+        import numpy as np
+
+        special_mask = input_ids == placeholder_id
+        special_mask_expanded = mx.broadcast_to(
+            special_mask[..., None], inputs_embeds.shape
+        )
+        positions = mx.array(
+            np.where(mx.flatten(special_mask_expanded))[0], mx.uint32
+        )
+        flat_embeds = mx.flatten(inputs_embeds)
+        flat_embeds[positions] = mx.flatten(features)
+        return mx.reshape(flat_embeds, inputs_embeds.shape)
+
+    def encode_image(
+        self, pixel_values, image_grid_thw, input_ids, attention_mask=None
+    ):
+        vision_hidden = self.vision_tower(pixel_values, image_grid_thw)
+        vision_features = self.merger(vision_hidden)
+
+        inputs_embeds = self.language_model.embed_tokens(input_ids)
+        inputs_embeds = self._inject_features(
+            input_ids, inputs_embeds, self.config.image_token_id, vision_features
+        )
+
+        mask = _causal_pad_mask(
+            attention_mask,
+            input_ids.shape[1],
+            self.language_model.layers[0].input_layernorm.weight,
+        )
+        hidden = self.language_model(inputs_embeds=inputs_embeds, mask=mask)
+        mx.eval(hidden)
+        emb = _last_token_pool(hidden, attention_mask)
+        mx.eval(emb)
+        return emb
+
+    def encode_video(
+        self, pixel_values_videos, video_grid_thw, input_ids, attention_mask=None
+    ):
+        """Encode video patches at video-token positions in the text sequence.
+
+        Images and video share the vision transformer and merger. The temporal
+        dimension of video_grid_thw controls temporal patch grouping.
+        """
+        vision_hidden = self.vision_tower(pixel_values_videos, video_grid_thw)
+        vision_features = self.merger(vision_hidden)
+
+        inputs_embeds = self.language_model.embed_tokens(input_ids)
+        inputs_embeds = self._inject_features(
+            input_ids, inputs_embeds, self.config.video_token_id, vision_features
+        )
+
+        mask = _causal_pad_mask(
+            attention_mask,
+            input_ids.shape[1],
+            self.language_model.layers[0].input_layernorm.weight,
+        )
+        hidden = self.language_model(inputs_embeds=inputs_embeds, mask=mask)
+        mx.eval(hidden)
+        emb = _last_token_pool(hidden, attention_mask)
+        mx.eval(emb)
+        return emb
+
+    # ----------------------------------------------------------------- audio
+    def encode_audio(
+        self,
+        input_features,
+        feature_lens,
+        input_ids,
+        attention_mask=None,
+    ):
+        audio_hidden = self.audio_tower(input_features, feature_lens)
+        audio_features = self.audio_projector(audio_hidden)
+
+        inputs_embeds = self.language_model.embed_tokens(input_ids)
+        inputs_embeds = self._inject_features(
+            input_ids, inputs_embeds, self.config.audio_token_id, audio_features
+        )
+
+        mask = _causal_pad_mask(
+            attention_mask,
+            input_ids.shape[1],
+            self.language_model.layers[0].input_layernorm.weight,
+        )
+        hidden = self.language_model(inputs_embeds=inputs_embeds, mask=mask)
+        mx.eval(hidden)
+        emb = _last_token_pool(hidden, attention_mask)
+        mx.eval(emb)
+        return emb
+
+    # ---------------------------------------------------------------- sanitize
+    def sanitize(self, weights):
+        """Load converted weights, retaining float32 language computation."""
+        return {
+            name: value.astype(mx.float32)
+            if name.startswith("language_model.") and value.dtype == mx.bfloat16
+            else value
+            for name, value in weights.items()
+        }

model/nanobe_mlx_omni.py

@@ -0,0 +1,988 @@
+from __future__ import annotations
+
+import importlib.util
+import hashlib
+import json
+import math
+import sys
+import time
+from contextlib import contextmanager, nullcontext
+from dataclasses import dataclass
+from pathlib import Path
+
+import av
+import mlx.core as mx
+import mlx.nn as nn
+import numpy as np
+from av.audio.resampler import AudioResampler
+from PIL import Image
+from transformers import AutoProcessor, WhisperFeatureExtractor
+from transformers.video_utils import VideoMetadata
+from safetensors.numpy import load_file
+
+from nanobe.query import AudioMagnitude
+from nanobe.runtime import geometry_contract
+
+
+# Frames sampled per video pass.
+VIDEO_FRAMES = 16
+# Limit tokens per position-wise feed-forward kernel to bound GPU scheduling latency.
+SEQUENCE_CHUNK = 2048
+IMAGE_PIXEL_SHARE = 0.5
+VIDEO_MAX_PIXELS = 12_845_056
+MEDIA_MIN_PIXELS = 262_144
+MODALITIES = ("text", "image", "document", "audio", "video", "interleaved", "unknown")
+
+
+class MediaSwitch:
+    """Select the active media or text-document adapter for one forward.
+
+    Queries and document pages use base weights.
+    """
+
+    def __init__(self) -> None:
+        self.modality: str | None = None
+
+    @property
+    def on(self) -> bool:
+        return self.modality is not None
+
+
+class LoRALinear(nn.Module):
+    """Add the selected low-rank delta to a base linear layer.
+
+    `deltas` maps adapter names to their (lora_a, lora_b) pairs.
+    """
+
+    def __init__(self, base: nn.Module, deltas: dict, scale: float, switch: MediaSwitch):
+        super().__init__()
+        self.base = base
+        # cast once to the base layer's compute dtype and pre-transpose, so a
+        # media forward costs two small matmuls and nothing else
+        dtype = getattr(base, "scales", getattr(base, "weight", None)).dtype
+        self.deltas = {name: (lora_a.T.astype(dtype), lora_b.T.astype(dtype)) for name, (lora_a, lora_b) in deltas.items()}
+        self.scale = scale
+        self._switch = switch
+
+    def __call__(self, x: mx.array) -> mx.array:
+        value = self.base(x)
+        name = self._switch.modality
+        if name is None:
+            return value
+        a_t, b_t = self.deltas[name]
+        return value + ((x @ a_t) @ b_t) * self.scale
+
+
+def install_media_adapter(language_model: nn.Module, tensors: dict, scale: float, switch: MediaSwitch) -> int:
+    """Wrap every language self-attention projection named in `tensors`.
+    Keys are `<modality>/language_model.layers.N.self_attn.P.lora_{a,b}`."""
+    installed = 0
+    for index, layer in enumerate(language_model.layers):
+        attention = layer.self_attn
+        for name in ("q_proj", "k_proj", "v_proj", "o_proj"):
+            key = f"language_model.layers.{index}.self_attn.{name}"
+            deltas = {}
+            for kind in ("image", "audio", "video", "text"):
+                if f"{kind}/{key}.lora_a" in tensors:
+                    deltas[kind] = (tensors[f"{kind}/{key}.lora_a"], tensors[f"{kind}/{key}.lora_b"])
+            if not deltas:
+                continue
+            setattr(attention, name, LoRALinear(getattr(attention, name), deltas, scale, switch))
+            installed += 1
+    return installed
+
+
+@dataclass(frozen=True)
+class VideoFrames:
+    images: list[Image.Image]
+    indices: list[int]
+    total: int
+    fps: float | None
+    width: int
+    height: int
+    duration: float | None
+
+    @property
+    def metadata(self) -> VideoMetadata:
+        return VideoMetadata(
+            total_num_frames=self.total,
+            fps=self.fps,
+            width=self.width,
+            height=self.height,
+            duration=self.duration,
+            video_backend="pyav",
+            frames_indices=self.indices,
+        )
+
+
+@dataclass(frozen=True)
+class VideoInputs:
+    input_ids: np.ndarray
+    attention_mask: np.ndarray
+    pixel_values: np.ndarray
+    grid: np.ndarray
+    audio_features: np.ndarray | None
+    audio_length: int | None
+    sampled_frames: int
+    total_frames: int
+    has_soundtrack: bool
+
+
+@dataclass(frozen=True)
+class AudioInputs:
+    input_ids: np.ndarray
+    attention_mask: np.ndarray
+    features: np.ndarray
+    length: int
+
+
+def chunk_feed_forward(module, limit: int = SEQUENCE_CHUNK) -> None:
+    """Split position-wise feed-forward work into bounded sequence chunks.
+
+    Positions are independent, so chunking preserves the computation while
+    limiting the work submitted in each GPU kernel.
+    """
+
+    def chunked(original):
+        def wrapper(self, x):
+            axis = x.ndim - 2
+            length = x.shape[axis]
+            if length <= limit:
+                return original(self, x)
+            pieces = [original(self, x[..., start : start + limit, :]) for start in range(0, length, limit)]
+            return mx.concatenate(pieces, axis=axis)
+
+        return wrapper
+
+    for name in ("TextMLP", "VisionMLP"):
+        block = getattr(module, name, None)
+        if block is not None and not getattr(block, "_chunked", False):
+            block.__call__ = chunked(block.__call__)
+            block._chunked = True
+
+
+def imported_module(repo: Path):
+    path = repo / "model.py"
+    spec = importlib.util.spec_from_file_location(f"nanobe_mlx_omni_{repo.name}", path)
+    if not spec or not spec.loader:
+        raise RuntimeError(f"cannot import {path}")
+    module = importlib.util.module_from_spec(spec)
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module
+
+
+def digest(path: Path) -> str:
+    value = hashlib.sha256()
+    with path.open("rb") as file:
+        while chunk := file.read(1024 * 1024):
+            value.update(chunk)
+    return value.hexdigest()
+
+
+SEQUENTIAL_DECODE_SECONDS = 60.0
+
+
+def uniform_indices(total: int, limit: int) -> list[int]:
+    if total <= 0:
+        return []
+    return np.linspace(0, total - 1, min(limit, total), dtype=int).tolist()
+
+
+def stream_geometry(stream, container) -> tuple[float | None, float | None]:
+    fps = float(stream.average_rate) if stream.average_rate else None
+    duration = (
+        float(stream.duration * stream.time_base)
+        if stream.duration and stream.time_base
+        else float(container.duration / av.time_base)
+        if container.duration
+        else None
+    )
+    return fps, duration
+
+
+def decode_indices(path: Path, indices: list[int]) -> VideoFrames:
+    """Single sequential pass keeping the requested frame positions."""
+    wanted = set(indices)
+    images: list[Image.Image] = []
+    total = 0
+    with av.open(path) as container:
+        stream = container.streams.video[0]
+        fps, duration = stream_geometry(stream, container)
+        width = stream.width
+        height = stream.height
+        for position, frame in enumerate(container.decode(stream)):
+            total = position + 1
+            if position in wanted:
+                images.append(frame.to_image().convert("RGB"))
+            if wanted and position >= max(wanted):
+                break
+    return VideoFrames(images, indices, total, fps, width, height, duration)
+
+
+def seek_sampled_video(
+    path: Path,
+    limit: int,
+    fps: float | None,
+    duration: float,
+) -> VideoFrames:
+    """Sample frames via keyframe seeks instead of decoding the whole file."""
+    rate = fps or 24.0
+    count = min(limit, max(1, round(duration * rate)))
+    targets = np.linspace(0.0, max(duration - 1.0 / rate, 0.0), count)
+    images: list[Image.Image] = []
+    indices: list[int] = []
+    with av.open(path) as container:
+        stream = container.streams.video[0]
+        width = stream.width
+        height = stream.height
+        reached = -1.0
+        for target in targets:
+            stamp = int(target / stream.time_base) if stream.time_base else 0
+            if target > reached:
+                container.seek(stamp, stream=stream, backward=True, any_frame=False)
+            chosen = None
+            for frame in container.decode(stream):
+                if frame.time is None or frame.time >= target - 1e-3:
+                    chosen = frame
+                    break
+            if chosen is None:
+                continue
+            reached = chosen.time if chosen.time is not None else target
+            index = round((chosen.time if chosen.time is not None else target) * rate)
+            if indices and index <= indices[-1]:
+                index = indices[-1] + 1
+            images.append(chosen.to_image().convert("RGB"))
+            indices.append(index)
+    total = max(round(duration * rate), (indices[-1] + 1) if indices else 0)
+    return VideoFrames(images, indices, total, fps, width, height, duration)
+
+
+def span_sampled_video(
+    path: Path,
+    limit: int,
+    span: tuple[float, float],
+) -> VideoFrames:
+    """One seek to the span start, then a short sequential decode across it.
+
+    Frame positions and timestamps are relative to the selected span.
+    """
+    start, end = span
+    with av.open(path) as container:
+        stream = container.streams.video[0]
+        fps, _ = stream_geometry(stream, container)
+        rate = fps or 24.0
+        length = max(end - start, 1.0 / rate)
+        count = min(limit, max(1, round(length * rate)))
+        targets = np.linspace(start, max(end - 1.0 / rate, start), count)
+        stamp = int(start / stream.time_base) if stream.time_base else 0
+        container.seek(stamp, stream=stream, backward=True, any_frame=False)
+        images: list[Image.Image] = []
+        indices: list[int] = []
+        position = 0
+        for frame in container.decode(stream):
+            moment = frame.time
+            if moment is None:
+                continue
+            if moment >= end + 0.5 / rate or position >= len(targets):
+                break
+            while position < len(targets) and moment >= targets[position] - 1e-3:
+                index = round((moment - start) * rate)
+                if indices and index <= indices[-1]:
+                    index = indices[-1] + 1
+                images.append(frame.to_image().convert("RGB"))
+                indices.append(index)
+                position += 1
+        width = stream.width
+        height = stream.height
+    total = max(round(length * rate), (indices[-1] + 1) if indices else 0)
+    return VideoFrames(images, indices, total, fps, width, height, length)
+
+
+def uniformly_sampled_video(
+    path: Path,
+    limit: int = VIDEO_FRAMES,
+    span: tuple[float, float] | None = None,
+) -> VideoFrames:
+    if span is not None:
+        return span_sampled_video(path, limit, span)
+    with av.open(path) as container:
+        stream = container.streams.video[0]
+        declared = int(stream.frames or 0)
+        fps, duration = stream_geometry(stream, container)
+    if duration and math.isfinite(duration) and duration > SEQUENTIAL_DECODE_SECONDS:
+        sampled = seek_sampled_video(path, limit, fps, duration)
+        if sampled.images:
+            return sampled
+    provisional = decode_indices(path, uniform_indices(declared, limit))
+    exact = uniform_indices(provisional.total, limit)
+    if exact == provisional.indices and len(provisional.images) == len(exact):
+        return provisional
+    return decode_indices(path, exact)
+
+
+def decode_soundtrack(
+    path: Path,
+    span: tuple[float, float] | None = None,
+) -> np.ndarray | None:
+    with av.open(path) as container:
+        stream = next((stream for stream in container.streams if stream.type == "audio"), None)
+        if stream is None:
+            return None
+        if span is not None and stream.time_base:
+            container.seek(int(span[0] / stream.time_base), stream=stream, backward=True)
+        resampler = AudioResampler(format="flt", layout="mono", rate=16_000)
+        samples = []
+        for frame in container.decode(stream):
+            if span is not None and frame.time is not None and frame.time >= span[1]:
+                break
+            for output in resampler.resample(frame):
+                chunk = output.to_ndarray().flatten()
+                if span is not None and frame.time is not None and frame.time < span[0]:
+                    ahead = span[0] - frame.time
+                    skip = min(int(ahead * 16_000), len(chunk))
+                    chunk = chunk[skip:]
+                if len(chunk):
+                    samples.append(chunk)
+        samples.extend(output.to_ndarray().flatten() for output in resampler.resample(None))
+    samples = [chunk for chunk in samples if len(chunk)]
+    if not samples:
+        return None
+    waveform = np.concatenate(samples).astype(np.float32)
+    if span is not None:
+        waveform = waveform[: int((span[1] - span[0]) * 16_000)]
+    return waveform if len(waveform) else None
+
+
+def bounded_waveform(
+    waveform: np.ndarray,
+    budget_seconds: float | None,
+    slices: int = 8,
+) -> np.ndarray:
+    """Bound a whole-file waveform to a strided summary of budget_seconds.
+
+    Long soundtracks otherwise become tens of thousands of audio tokens in
+    one forward; windows already cover locality, so the whole-file lane
+    only needs a bounded overview.
+    """
+    if budget_seconds is None:
+        return waveform
+    budget = int(budget_seconds * 16_000)
+    if len(waveform) <= budget:
+        return waveform
+    span = budget // slices
+    starts = np.linspace(0, len(waveform) - span, slices).astype(int)
+    return np.concatenate([waveform[start : start + span] for start in starts])
+
+
+def audio_tokens(frames: int) -> int:
+    after_first_convolution = (frames - 1) // 2 + 1
+    return max((after_first_convolution - 2) // 2 + 1, 1)
+
+
+def configure_processor(processor: object) -> None:
+    video = processor.video_processor
+    if hasattr(video, "do_sample_frames"):
+        video.do_sample_frames = False
+    for attribute in ("max_frames", "num_frames"):
+        if hasattr(video, attribute):
+            setattr(video, attribute, VIDEO_FRAMES)
+    if hasattr(video, "size") and isinstance(video.size, dict):
+        video.size = {
+            **video.size,
+            "longest_edge": VIDEO_MAX_PIXELS,
+            "shortest_edge": MEDIA_MIN_PIXELS,
+        }
+    if hasattr(video, "max_pixels"):
+        video.max_pixels = VIDEO_MAX_PIXELS
+    if hasattr(video, "min_pixels"):
+        video.min_pixels = MEDIA_MIN_PIXELS
+
+    # The image processor uses a pixel-area budget independent of video limits.
+    images = processor.image_processor
+    if hasattr(images, "size") and images.size is not None:
+        images.size = {
+            **dict(images.size),
+            "longest_edge": int(dict(images.size)["longest_edge"] * IMAGE_PIXEL_SHARE),
+            "shortest_edge": int(dict(images.size)["shortest_edge"] * IMAGE_PIXEL_SHARE),
+        }
+
+
+def array(value: object) -> np.ndarray:
+    if hasattr(value, "detach"):
+        return value.detach().cpu().numpy()
+    return np.asarray(value)
+
+
+def normalize(value: np.ndarray) -> np.ndarray:
+    vector = np.asarray(value, dtype=np.float32).reshape(-1)
+    norm = float(np.linalg.norm(vector))
+    if not math.isfinite(norm) or norm <= 0:
+        raise RuntimeError("embedding has an invalid norm")
+    return vector / norm
+
+
+def prepare_audio(
+    processor: object,
+    extractor: WhisperFeatureExtractor,
+    config: dict,
+    waveform: np.ndarray,
+) -> AudioInputs:
+    if not np.isfinite(waveform).all():
+        waveform = np.nan_to_num(waveform, nan=0.0, posinf=0.0, neginf=0.0)
+    peak = float(np.max(np.abs(waveform))) if waveform.size else 0.0
+    if peak > 1:
+        waveform = waveform / peak
+    features = extractor(
+        waveform,
+        sampling_rate=16_000,
+        return_tensors="np",
+        padding=False,
+        truncation=False,
+        return_attention_mask=True,
+    )
+    mel = np.asarray(features["input_features"][0], dtype=np.float32)
+    length = int(features["attention_mask"][0].sum())
+    count = audio_tokens(length)
+    tokenizer = processor.tokenizer
+    start = tokenizer.convert_ids_to_tokens(config["audio_start_token_id"])
+    token = tokenizer.convert_ids_to_tokens(config["audio_token_id"])
+    end = tokenizer.convert_ids_to_tokens(config["audio_end_token_id"])
+    prompt = processor.apply_chat_template(
+        [{"role": "user", "content": start + token * count + end}],
+        tokenize=False,
+        add_generation_prompt=False,
+    )
+    encoded = processor(
+        text=[prompt],
+        return_tensors="pt",
+        padding=False,
+        truncation=False,
+    )
+    input_ids = array(encoded["input_ids"])
+    if int(np.sum(input_ids == config["audio_token_id"])) != count:
+        raise RuntimeError("audio placeholders do not match extracted features")
+    return AudioInputs(
+        input_ids=input_ids,
+        attention_mask=array(encoded["attention_mask"]),
+        features=mel,
+        length=length,
+    )
+
+
+def prepare_video(
+    processor: object,
+    extractor: WhisperFeatureExtractor,
+    config: dict,
+    path: Path,
+    include_soundtrack: bool = True,
+    span: tuple[float, float] | None = None,
+    audio_budget: float | None = None,
+) -> VideoInputs:
+    video = uniformly_sampled_video(path, span=span)
+    if not video.images:
+        raise RuntimeError(f"video decoded no frames: {path}")
+
+    soundtrack = decode_soundtrack(path, span=span) if include_soundtrack else None
+    if soundtrack is not None and span is None:
+        soundtrack = bounded_waveform(soundtrack, audio_budget)
+    features = None
+    length = None
+    audio = ""
+    if soundtrack is not None:
+        prepared = prepare_audio(processor, extractor, config, soundtrack)
+        features = prepared.features
+        length = prepared.length
+        count = audio_tokens(length)
+        tokenizer = processor.tokenizer
+        start = tokenizer.convert_ids_to_tokens(config["audio_start_token_id"])
+        token = tokenizer.convert_ids_to_tokens(config["audio_token_id"])
+        end = tokenizer.convert_ids_to_tokens(config["audio_end_token_id"])
+        audio = start + token * count + end
+
+    video_token = getattr(processor, "video_token", "<|video_pad|>")
+    content = f"{audio}<|vision_start|>{video_token}<|vision_end|>"
+    prompt = processor.apply_chat_template(
+        [{"role": "user", "content": content}],
+        tokenize=False,
+        add_generation_prompt=False,
+    )
+    sentinel = object()
+    previous_metadata = getattr(processor.video_processor, "video_metadata", sentinel)
+    processor.video_processor.video_metadata = [video.metadata]
+    try:
+        encoded = processor(
+            text=[prompt],
+            videos=[video.images],
+            return_tensors="pt",
+            padding=False,
+            truncation=False,
+        )
+    finally:
+        for image in video.images:
+            image.close()
+        if previous_metadata is sentinel:
+            delattr(processor.video_processor, "video_metadata")
+        else:
+            processor.video_processor.video_metadata = previous_metadata
+
+    input_ids = array(encoded["input_ids"])
+    grid = array(encoded["video_grid_thw"])
+    audio_count = int(np.sum(input_ids == config["audio_token_id"]))
+    video_count = int(np.sum(input_ids == config["video_token_id"]))
+    expected_audio = audio_tokens(length) if length is not None else 0
+    expected_video = int(np.prod(grid) // 4)
+    if audio_count != expected_audio:
+        raise RuntimeError(f"audio placeholders {audio_count} do not match features {expected_audio}")
+    if video_count != expected_video:
+        raise RuntimeError(f"video placeholders {video_count} do not match features {expected_video}")
+    if audio_count:
+        audio_positions = np.flatnonzero(input_ids == config["audio_token_id"])
+        video_positions = np.flatnonzero(input_ids == config["video_token_id"])
+        if int(audio_positions[-1]) >= int(video_positions[0]):
+            raise RuntimeError("audio placeholders must precede video placeholders")
+
+    return VideoInputs(
+        input_ids=input_ids,
+        attention_mask=array(encoded["attention_mask"]),
+        pixel_values=array(encoded["pixel_values_videos"]),
+        grid=grid,
+        audio_features=features,
+        audio_length=length,
+        sampled_frames=len(video.images),
+        total_frames=video.total,
+        has_soundtrack=soundtrack is not None,
+    )
+
+
+def fuse_soundtrack(visual: np.ndarray, sound: np.ndarray, weight: float) -> np.ndarray:
+    """A video's native: the unit sum of its own native and the weighted unit
+    native of its soundtrack through the audio path."""
+    return normalize(normalize(visual) + weight * normalize(sound))
+
+
+class Runtime:
+    def __init__(self, model: Path, processor: Path | None = None):
+        started = time.perf_counter()
+        self.geometry = geometry_contract(model)
+        self.repo = model
+        self.module = imported_module(model)
+        chunk_feed_forward(self.module)
+        self.config_values = json.loads((model / "config.json").read_text())
+        config_name = next(
+            name for name in dir(self.module) if name.startswith("Omni") and name.endswith("Config")
+        )
+        model_name = next(
+            name
+            for name in dir(self.module)
+            if name.startswith("Jina") and name.endswith("EmbeddingModel")
+        )
+        config = getattr(self.module, config_name).from_dict(self.config_values)
+        self.model = getattr(self.module, model_name)(config)
+        manifest = json.loads((model / "manifest.json").read_text())
+        if manifest.get("component") != "full":
+            raise ValueError("audiovisual indexing requires the full one-model runtime")
+        self._quantize(manifest)
+        weights: dict[str, mx.array] = {}
+        for shard in manifest["weight_shards"]:
+            loaded = mx.load(str(model / shard["file"]))
+            overlap = weights.keys() & loaded.keys()
+            if overlap:
+                raise RuntimeError(f"runtime shards overlap at {sorted(overlap)[0]}")
+            weights.update(loaded)
+        self.model.load_weights(list(self.model.sanitize(weights).items()))
+        mx.eval(self.model.parameters())
+        self.media = MediaSwitch()
+        self._install_media_adapter(model, manifest["media_adapter"])
+        self._install_page_merger(model, manifest["page_merger"])
+        self._install_audio_pooler(model, manifest["audio_pooler"])
+        self.processor = AutoProcessor.from_pretrained(processor or model, trust_remote_code=True)
+        # Preserve the full image budget for rendered pages before reducing it for pictures.
+        self.page_pixels = dict(self.processor.image_processor.size)
+        configure_processor(self.processor)
+        self.extractor = WhisperFeatureExtractor(feature_size=128)
+        self.acoustic = None
+        checkpoint = model / "head"
+        if self.geometry["head_sha256"] != digest(checkpoint / "head.safetensors"):
+            raise RuntimeError("packaged model geometry and checkpoint hashes disagree")
+        self.head = load_file(checkpoint / "head.safetensors")
+        self.base_dimension = sum(self.head["document.weight"].shape)
+        self.magnitude = AudioMagnitude(model, self.geometry)
+        self.cold_load_ms = (time.perf_counter() - started) * 1000
+
+    def _quantize(self, manifest: dict) -> None:
+        quantization = manifest["quantization"]
+        targets = set(quantization["targets"])
+
+        def params(path: str, layer: nn.Module) -> bool | dict:
+            target = ("linear" in targets and isinstance(layer, nn.Linear)) or (
+                "embedding" in targets and isinstance(layer, nn.Embedding)
+            )
+            if not target:
+                return False
+            root = path.split(".", 1)[0]
+            bits = quantization.get("overrides", {}).get(root, quantization["bits"])
+            if isinstance(layer, nn.Embedding):
+                bits = quantization.get("embedding_overrides", {}).get(root, bits)
+            return {
+                "group_size": quantization["group_size"],
+                "bits": bits,
+                "mode": quantization["mode"],
+            }
+
+        nn.quantize(
+            self.model,
+            group_size=quantization["group_size"],
+            bits=quantization["bits"],
+            mode=quantization["mode"],
+            class_predicate=params,
+        )
+
+    def _install_media_adapter(self, model: Path, spec: dict) -> None:
+        path = model / spec["file"]
+        if path.parent != model or digest(path) != spec["sha256"]:
+            raise RuntimeError("model package media adapter failed integrity verification")
+        tensors = mx.load(str(path))
+        installed = install_media_adapter(self.model.language_model, tensors, float(spec["scale"]), self.media)
+        expected = int(spec["tensors"]) // 2 // len(spec["applies_to"])
+        if installed != expected:
+            raise RuntimeError(f"media adapter installed {installed} projections, manifest expects {expected}")
+        mx.eval(self.model.parameters())
+
+    def _install_page_merger(self, model: Path, spec: dict) -> None:
+        """Load the unquantized merger used only for rendered document pages.
+
+        Pages share the base vision and language weights with other inputs.
+        """
+        path = model / spec["file"]
+        if path.parent != model or digest(path) != spec["sha256"]:
+            raise RuntimeError("model package page merger failed integrity verification")
+        tensors = mx.load(str(path))
+        if len(tensors) != int(spec["tensors"]):
+            raise RuntimeError(f"page merger holds {len(tensors)} tensors, manifest expects {spec['tensors']}")
+        merger = self.module.Merger(self.model.config.vision_config, self.model.config.text_config.hidden_size)
+        dtype = self.model.merger.norm.weight.dtype
+        merger.load_weights([(name, value.astype(dtype)) for name, value in tensors.items()])
+        mx.eval(merger.parameters())
+        self.page_merger = merger
+
+    def _install_audio_pooler(self, model: Path, spec: dict) -> None:
+        """Load attention and residual weights for pooling encoded audio windows.
+
+        The NumPy pooler adds one segment per recording at index time.
+        """
+        path = model / spec["file"]
+        if path.parent != model or digest(path) != spec["sha256"]:
+            raise RuntimeError("model package audio pooler failed integrity verification")
+        from safetensors.numpy import load_file
+
+        tensors = load_file(str(path))
+        if len(tensors) != int(spec["tensors"]):
+            raise RuntimeError(f"audio pooler holds {len(tensors)} tensors, manifest expects {spec['tensors']}")
+        self.audio_pooler = {name: value.astype(np.float32) for name, value in tensors.items()}
+
+    def pool_audio_windows(self, whole: np.ndarray, windows: list[np.ndarray]) -> np.ndarray:
+        """The pooled row: the whole-file vector with its current block replaced by the
+        pooler's output over the windows' native blocks, the raw acoustic block and the
+        magnitude tail left as the recording's own."""
+        head = self.audio_pooler
+        if not windows:
+            raise ValueError("audio pooling requires at least one window")
+        natives = np.stack([np.asarray(w, np.float32)[640:1664] for w in windows])
+        natives /= np.linalg.norm(natives, axis=1, keepdims=True)
+        hidden = np.tanh(natives @ head["attend.0.weight"].T + head["attend.0.bias"])
+        logits = hidden @ head["attend.2.weight"].T + head["attend.2.bias"]
+        weights = np.exp(logits - logits.max())
+        weights /= weights.sum()
+        pooled = (weights * natives).sum(axis=0)
+        gelu = lambda x: 0.5 * x * (1.0 + np.tanh(np.sqrt(2.0 / np.pi) * (x + 0.044715 * x ** 3)))
+        pooled = pooled + gelu(pooled @ head["refine.0.weight"].T + head["refine.0.bias"]) @ head["refine.2.weight"].T + head["refine.2.bias"]
+        pooled /= np.linalg.norm(pooled)
+        output = np.asarray(whole, np.float32).copy()
+        scale = float(np.linalg.norm(output[:1664]))
+        output[:640] = 0.0
+        output[640:1664] = pooled * scale
+        return output
+
+    @contextmanager
+    def page_forward(self):
+        """The page's own budget and merger stand in for the shared ones inside this block."""
+        images = self.processor.image_processor
+        shared_size, shared_merger = images.size, self.model.merger
+        images.size = dict(self.page_pixels)
+        self.model.merger = self.page_merger
+        try:
+            yield
+        finally:
+            images.size = shared_size
+            self.model.merger = shared_merger
+
+    @contextmanager
+    def media_forward(self, modality: str):
+        """The modality's adapter is live only inside this block."""
+        self.media.modality = modality
+        try:
+            yield
+        finally:
+            self.media.modality = None
+
+    @property
+    def audio_budget(self) -> float:
+        return float(self.geometry["audio_budget_seconds"])
+
+    def encode_video(
+        self,
+        path: Path,
+        include_soundtrack: bool = True,
+        span: tuple[float, float] | None = None,
+    ) -> tuple[np.ndarray, VideoInputs]:
+        inputs = prepare_video(
+            self.processor,
+            self.extractor,
+            self.config_values,
+            path,
+            include_soundtrack,
+            span=span,
+            audio_budget=self.audio_budget,
+        )
+        ids = mx.array(inputs.input_ids)
+        mask = mx.array(inputs.attention_mask)
+        vision = self.model.vision_tower(mx.array(inputs.pixel_values), mx.array(inputs.grid))
+        vision = self.model.merger(vision)
+        embedded = self.model.language_model.embed_tokens(ids)
+        embedded = self.model._inject_features(
+            ids,
+            embedded,
+            self.model.config.video_token_id,
+            vision,
+        )
+        if inputs.audio_features is not None and inputs.audio_length is not None:
+            audio = self.model.audio_tower(
+                mx.array(inputs.audio_features),
+                mx.array([inputs.audio_length]),
+            )
+            audio = self.model.audio_projector(audio)
+            embedded = self.model._inject_features(
+                ids,
+                embedded,
+                self.model.config.audio_token_id,
+                audio,
+            )
+        attention = self.module._causal_pad_mask(
+            mask,
+            ids.shape[1],
+            self.model.language_model.layers[0].input_layernorm.weight,
+        )
+        hidden = self.model.language_model(inputs_embeds=embedded, mask=attention)
+        vector = self.module._last_token_pool(hidden, mask)[0]
+        mx.eval(vector)
+        return normalize(np.asarray(vector.astype(mx.float32), dtype=np.float32)), inputs
+
+    def encode_text(self, text: str, side: str) -> np.ndarray:
+        prefix = "Query" if side == "query" else "Document"
+        encoded = self.processor.tokenizer(
+            f"{prefix}: {text}",
+            return_tensors="np",
+            padding=False,
+            truncation=True,
+            max_length=32_768,
+        )
+        # Text documents use their adapter; queries use the base weights.
+        with self.media_forward("text") if side != "query" else nullcontext():
+            value = self.model.encode_text(
+                mx.array(encoded["input_ids"]),
+                mx.array(encoded["attention_mask"]),
+            )[0]
+        mx.eval(value)
+        return normalize(np.asarray(value.astype(mx.float32), dtype=np.float32))
+
+    def encode_image(self, image: Image.Image) -> np.ndarray:
+        token = getattr(self.processor, "image_token", "<|image_pad|>")
+        prompt = self.processor.apply_chat_template(
+            [{"role": "user", "content": f"Document: <|vision_start|>{token}<|vision_end|>"}],
+            tokenize=False,
+            add_generation_prompt=False,
+        )
+        encoded = self.processor(
+            images=[image.convert("RGB")],
+            text=[prompt],
+            return_tensors="pt",
+            padding=False,
+            truncation=False,
+        )
+        value = self.model.encode_image(
+            mx.array(array(encoded["pixel_values"])),
+            mx.array(array(encoded["image_grid_thw"])),
+            mx.array(array(encoded["input_ids"])),
+            mx.array(array(encoded["attention_mask"])),
+        )[0]
+        mx.eval(value)
+        return normalize(np.asarray(value.astype(mx.float32), dtype=np.float32))
+
+    def encode_audio(self, path: Path, span: tuple[float, float] | None = None) -> np.ndarray:
+        return self._audio_vectors(path, span)[0]
+
+    def _audio_vectors(
+        self, path: Path, span: tuple[float, float] | None = None,
+    ) -> tuple[np.ndarray, np.ndarray | None]:
+        """Return the current native and the raw PE feature already used to refine it."""
+        waveform = decode_soundtrack(path, span=span)
+        if waveform is None:
+            raise RuntimeError(f"audio file decoded no samples: {path}")
+        if span is None:
+            waveform = bounded_waveform(waveform, self.audio_budget)
+        inputs = prepare_audio(self.processor, self.extractor, self.config_values, waveform)
+        value = self.model.encode_audio(
+            mx.array(inputs.features),
+            mx.array([inputs.length]),
+            mx.array(inputs.input_ids),
+            mx.array(inputs.attention_mask),
+        )[0]
+        mx.eval(value)
+        current = normalize(np.asarray(value.astype(mx.float32), dtype=np.float32))
+        projection = self.geometry["audio_projection"]
+        if span is not None:
+            return current, None
+        pe = self.encode_acoustic(path, projection)
+        aligned = normalize(normalize(pe) @ self.head[projection["weight"]].T + self.head[projection["bias"]])
+        share = float(projection["share"])
+        return normalize((1 - share) * current + share * aligned), pe
+
+    def encode_acoustic(self, path: Path, projection: dict) -> np.ndarray:
+        if self.acoustic is None:
+            folder = self.repo / projection["encoder"]
+            manifest = folder / "manifest.json"
+            if folder.parent != self.repo or digest(manifest) != projection["encoder_sha256"]:
+                raise RuntimeError("packaged acoustic encoder failed integrity verification")
+            metadata = json.loads(manifest.read_text())
+            for name, checksum in metadata["files"].items():
+                source = folder / name
+                if source.parent != folder or digest(source) != checksum:
+                    raise RuntimeError(f"packaged acoustic encoder file failed integrity verification: {name}")
+            spec = importlib.util.spec_from_file_location(f"nanobe_acoustic_{self.repo.name}", folder / "nanobe_mlx_pe.py")
+            if not spec or not spec.loader:
+                raise RuntimeError("cannot import the packaged acoustic encoder")
+            module = importlib.util.module_from_spec(spec)
+            sys.modules[spec.name] = module
+            spec.loader.exec_module(module)
+            self.acoustic = module.Runtime(folder)
+        with av.open(path) as container:
+            stream = next((stream for stream in container.streams if stream.type == "audio"), None)
+            if stream is None or not stream.codec_context.sample_rate:
+                raise RuntimeError(f"audio file has no sample rate: {path}")
+            rate = stream.codec_context.sample_rate
+            # PE uses the equal channel mean at the original rate. A mono
+            # downmix matrix changes stereo gain before its nonlinear encoder.
+            resampler = AudioResampler(format="fltp", layout=stream.layout, rate=rate)
+            chunks = [output.to_ndarray().mean(axis=0) for frame in container.decode(stream)
+                      for output in resampler.resample(frame)]
+            chunks.extend(output.to_ndarray().mean(axis=0) for output in resampler.resample(None))
+        if not chunks or not sum(map(len, chunks)):
+            raise RuntimeError(f"audio file decoded no samples: {path}")
+        waveform = np.concatenate(chunks).astype(np.float32)
+        return self.acoustic.encode(waveform, rate)
+
+    def project_media(self, raw: np.ndarray, modality: str) -> np.ndarray:
+        native = np.asarray(raw, dtype=np.float32).reshape(-1)
+        raw = normalize(native)
+        if raw.shape != (self.head["document.weight"].shape[1],):
+            raise RuntimeError("media embedding does not match the projection head")
+        if modality in self.geometry["native_lane_media"]:
+            # Use the text-document scale for speech vectors in the native lane.
+            return self.project_document(raw)
+        modality_id = MODALITIES.index(modality)
+        head = raw @ self.head["document.weight"].T + self.head["document_modality.weight"][modality_id]
+        head = normalize(head)
+        output = np.zeros(self.base_dimension, dtype=np.float32)
+        output[: len(head)] = head
+        projection = self.geometry["image_projection"]
+        if modality == "image":
+            projected = native @ self.head[projection["weight"]].T + self.head[projection["bias"]]
+            output[len(head) :] = float(projection["epsilon"]) * normalize(projected)
+            return normalize(output)
+        return output
+
+    def project_video(self, raw: np.ndarray) -> np.ndarray:
+        return self.project_media(raw, "video")
+
+    def project_document(self, raw: np.ndarray, refine: bool = False) -> np.ndarray:
+        native = normalize(raw)
+        if refine:
+            # The projection requires unit native vectors; keep it after normalization.
+            residual = (native @ self.head["document_refine.down"].T) @ self.head["document_refine.up"].T
+            native = normalize(native + residual)
+        head_dimension = int(self.head["document.weight"].shape[0])
+        dimension = self.base_dimension
+        if head_dimension + len(native) != dimension:
+            raise RuntimeError("native embedding does not match the protected geometry")
+        output = np.zeros(dimension, dtype=np.float32)
+        output[head_dimension:] = native
+        return output
+
+    def encode_video_item(
+        self,
+        path: Path,
+        span: tuple[float, float] | None = None,
+    ) -> tuple[np.ndarray, VideoInputs]:
+        with self.media_forward("video"):
+            raw, inputs = self.encode_video(path, include_soundtrack=True, span=span)
+        weight = self.soundtrack_weight
+        pe = None
+        whole_soundtrack = span is None and inputs.has_soundtrack
+        if whole_soundtrack:
+            # Combine the whole-file soundtrack before adding acoustic coordinates.
+            with self.media_forward("audio"):
+                sound, pe = self._audio_vectors(path)
+            raw = fuse_soundtrack(raw, sound, weight)
+        current = self.project_video(raw)
+        if whole_soundtrack:
+            if pe is None:
+                pe = self.encode_acoustic(path, self.geometry["audio_projection"])
+            return self.magnitude.audio(current, pe), inputs
+        return self.pad_document(current), inputs
+
+    @property
+    def soundtrack_weight(self) -> float:
+        """Geometry `video_soundtrack_weight`: how much of a video's soundtrack
+        native (through the audio path) joins its whole-file composite native."""
+        return float(self.geometry["video_soundtrack_weight"])
+
+    def encode_text_item(self, text: str) -> np.ndarray:
+        return self.pad_document(self.project_document(self.encode_text(text, "document")))
+
+    def encode_document_page(self, image: Image.Image) -> np.ndarray:
+        with self.page_forward():
+            raw = self.encode_image(image)
+        return self.pad_document(self.project_document(raw, refine=True))
+
+    def encode_image_item(self, image: Image.Image) -> np.ndarray:
+        image = image.convert("RGB")
+        budget = self.processor.image_processor.size["longest_edge"]
+        if image.width * image.height > budget:
+            scale = math.sqrt(budget / (image.width * image.height))
+            image = image.resize((max(1, int(image.width * scale)), max(1, int(image.height * scale))))
+        with self.media_forward("image"):
+            raw = self.encode_image(image)
+        return self.pad_document(self.project_media(raw, "image"))
+
+    def encode_audio_item(self, path: Path, span: tuple[float, float] | None = None) -> np.ndarray:
+        with self.media_forward("audio"):
+            raw, pe = self._audio_vectors(path, span)
+        current = self.project_media(raw, "audio")
+        if span is None:
+            if pe is None:
+                raise RuntimeError("whole-file audio magnitude requires the raw PE feature")
+            return self.magnitude.audio(current, pe)
+        return self.pad_document(current)
+
+    def pad_document(self, current: np.ndarray) -> np.ndarray:
+        return self.magnitude.document(current)
+
+    def encode_query(self, text: str) -> np.ndarray:
+        native = self.encode_text(text, "query")
+        text_modality = MODALITIES.index("text")
+        head = native @ self.head["query.weight"].T + self.head["query_modality.weight"][text_modality]
+        head /= np.linalg.norm(head)
+        share = float(self.geometry["native_document_share"])
+        current = np.concatenate((math.sqrt(1 - share) * head, math.sqrt(share) * native))
+        return self.magnitude.query(current, text)

model/omni.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8fd64d888f00a1cef89dd950e59b10bc8d60460e8d1e3f754b41592317329b36
+size 949948427

model/packaging.json

@@ -0,0 +1,41 @@
+{
+  "files": {
+    "LICENSE": "e258bbefaa6a50100cf835c0ef907b58d201712826ee28de5d2ec0b159ebcf07",
+    "README.md": "80148fca5285b7e5933c90665a5c6a9b526e8c3768088c605193a34dc196d126",
+    "audio-pooler.safetensors": "7bf8bc6420491454c3bb332bd243244d82efddfbcaa07cbc56dfb7603ab913f9",
+    "chat_template.jinja": "273d8e0e683b885071fb17e08d71e5f2a5ddfb5309756181681de4f5a1822d80",
+    "config.json": "841c7e432630ea31dbab9dd90aeedf07ba44e19144cbeb61ad41138c7d881e8e",
+    "geometry.json": "2c4a99d0c2b03c023ef29cc2cb15c1cf780fdf964e5b92b6bc095ca046000e92",
+    "head/head.json": "2b1b316b95a248d1280411197e8fb6f0e5089247a3671773c36475ca213dab39",
+    "head/head.safetensors": "3922c5b7cb892977d61cb3e6caaa025fece854d5be2fba0e047503e1c17f4ccf",
+    "lanes.py": "b85311d9b5fe5d593d4b04730eeb66af11d39d942b393ee57e63ab3062510320",
+    "language.safetensors": "fea1441e6a0e56aba49d3cf2d0b92b36ee8142b8f98ed54b0e3bd30b1e909702",
+    "licenses/jina-cc-by-nc-4.0.txt": "41003d4a74749c0220e33dd415042164b5a1093ed401f36277234f772d22d3d0",
+    "licenses/meta-apache-2.0.txt": "cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30",
+    "magnitude.py": "93f267d198dcd0755134e86011d85d78af4fd5d61b818f98f8202cab1684930e",
+    "manifest.json": "2cb347a52202cc95e6edccf6d9721f93be4cc96a4b18cb9a35a3995eea995aac",
+    "media-adapter.safetensors": "f9812fd4f33aeaaa22854569753731e81115581df55a2337c72ba4655fe58a63",
+    "model.py": "553386da770249a30a0ba85febba50c9e9d40ff58c266c8ea3253f40dfe58963",
+    "nanobe_mlx_omni.py": "33fbaae67f93e8c9ae3816c0232ae51c8dbb2e1bbeb4ced3a69495bdd3d01773",
+    "omni.safetensors": "8fd64d888f00a1cef89dd950e59b10bc8d60460e8d1e3f754b41592317329b36",
+    "page-merger.safetensors": "db41396835aa23dd661711a39e9fa64c59957f46564741f2db26517e782d5a57",
+    "pe-audio/config.json": "52cad1a78304268153c436b796477c3a1dd4a62817fce42ce66e2f9cb04359d3",
+    "pe-audio/manifest.json": "790f80f549c70cb327b30c537017ca2ae6f47781b219ce8cd35f0144b1e058d7",
+    "pe-audio/model.safetensors": "e424fb597caed67db9d794b3634909f06a36020970c1c31837382f17b8509a60",
+    "pe-audio/nanobe_mlx_pe.py": "be3d0cdd3f16a22a59379a65aa56b8f77e14adf63dad9d60eaf483686893508a",
+    "pe-text/config.json": "9402387f9eb47c9ba91bc6b62c13834d34a1bf31cafcee7e4211fd9997d834d7",
+    "pe-text/manifest.json": "97ad081b8044cb4308995a5d2d93d36bdec39a7ccfcf01e6591b7a5fd6b0eeff",
+    "pe-text/model.safetensors": "48ffbbea2595722fffcd320bc07a5af07149fafcf00f03f9b02b4f138262928e",
+    "pe-text/pe_text.py": "025bc642db25eddad124a289535acf3cbca1d7a88bf409c7b9fe1b4aecd6f4f7",
+    "pe-text/special_tokens_map.json": "ea97ecdbcc73713039d8d64dbb05e3689495c96657fbd9a18f5bed381be81049",
+    "pe-text/tokenizer.json": "43e9ea931bb0f359d16d0e435c26abbebd9a42a7298270b8ab7e5c9fa1be1dcb",
+    "pe-text/tokenizer_config.json": "adec72131bbba5a043594c1e3bf5930608546e46a51432b2502b392ec280551b",
+    "preprocessor_config.json": "e517dcd8eebc59981f55077e4c9684cf7f6b06adcc87bdfff25fd4c04e60c971",
+    "processor_config.json": "e2513e75f4f5eb89903209a8a146ac9e968a7501816238b748ebc08531326363",
+    "tokenizer.bin": "8e08386ae43d4a5024a97dadbdc7291d6770135921c97e913644f991b3a87b8e",
+    "tokenizer.json": "7bbf567d9e053a72f0bd8ea475e672d10d537578f6adb521ab42da2badfa8bef",
+    "tokenizer_config.json": "c3df92f019f31d77889b0a01baea0441d5f47400d99546f1f0190c5ff0b600ee",
+    "upstream-notices.md": "829342e56825c0fd034cf04c0e0e22d6fb56ddc9422adef8fb5bc1aad306af60",
+    "video_preprocessor_config.json": "266a4a3e4680853721f7f69c8147cdb32e2a1e6646b44de188d81a685faeedd4"
+  }
+}

model/page-merger.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:db41396835aa23dd661711a39e9fa64c59957f46564741f2db26517e782d5a57
+size 41957888

model/pe-audio/config.json

@@ -0,0 +1,123 @@
+{
+  "audio_config": {
+    "attention_bias": false,
+    "dac_config": {
+      "codebook_dim": 128,
+      "downsampling_ratios": [
+        2,
+        8,
+        10,
+        12
+      ],
+      "encoder_hidden_size": 64,
+      "hidden_size": 1024
+    },
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 768,
+    "intermediate_size": 2048,
+    "num_attention_heads": 6,
+    "num_hidden_layers": 12,
+    "num_key_value_heads": 6,
+    "rms_norm_eps": 1e-05,
+    "rope_parameters": {
+      "rope_theta": 20000,
+      "rope_type": "default"
+    }
+  },
+  "output_dimension": 1024,
+  "quantization": {
+    "bits": 8,
+    "group_size": 64,
+    "modules": [
+      "audio_head.proj",
+      "audio_encoder.output",
+      "audio_encoder.layers.11.mlp.down_proj",
+      "audio_encoder.layers.11.mlp.up_proj",
+      "audio_encoder.layers.11.mlp.gate_proj",
+      "audio_encoder.layers.11.self_attn.o_proj",
+      "audio_encoder.layers.11.self_attn.v_proj",
+      "audio_encoder.layers.11.self_attn.k_proj",
+      "audio_encoder.layers.11.self_attn.q_proj",
+      "audio_encoder.layers.10.mlp.down_proj",
+      "audio_encoder.layers.10.mlp.up_proj",
+      "audio_encoder.layers.10.mlp.gate_proj",
+      "audio_encoder.layers.10.self_attn.o_proj",
+      "audio_encoder.layers.10.self_attn.v_proj",
+      "audio_encoder.layers.10.self_attn.k_proj",
+      "audio_encoder.layers.10.self_attn.q_proj",
+      "audio_encoder.layers.9.mlp.down_proj",
+      "audio_encoder.layers.9.mlp.up_proj",
+      "audio_encoder.layers.9.mlp.gate_proj",
+      "audio_encoder.layers.9.self_attn.o_proj",
+      "audio_encoder.layers.9.self_attn.v_proj",
+      "audio_encoder.layers.9.self_attn.k_proj",
+      "audio_encoder.layers.9.self_attn.q_proj",
+      "audio_encoder.layers.8.mlp.down_proj",
+      "audio_encoder.layers.8.mlp.up_proj",
+      "audio_encoder.layers.8.mlp.gate_proj",
+      "audio_encoder.layers.8.self_attn.o_proj",
+      "audio_encoder.layers.8.self_attn.v_proj",
+      "audio_encoder.layers.8.self_attn.k_proj",
+      "audio_encoder.layers.8.self_attn.q_proj",
+      "audio_encoder.layers.7.mlp.down_proj",
+      "audio_encoder.layers.7.mlp.up_proj",
+      "audio_encoder.layers.7.mlp.gate_proj",
+      "audio_encoder.layers.7.self_attn.o_proj",
+      "audio_encoder.layers.7.self_attn.v_proj",
+      "audio_encoder.layers.7.self_attn.k_proj",
+      "audio_encoder.layers.7.self_attn.q_proj",
+      "audio_encoder.layers.6.mlp.down_proj",
+      "audio_encoder.layers.6.mlp.up_proj",
+      "audio_encoder.layers.6.mlp.gate_proj",
+      "audio_encoder.layers.6.self_attn.o_proj",
+      "audio_encoder.layers.6.self_attn.v_proj",
+      "audio_encoder.layers.6.self_attn.k_proj",
+      "audio_encoder.layers.6.self_attn.q_proj",
+      "audio_encoder.layers.5.mlp.down_proj",
+      "audio_encoder.layers.5.mlp.up_proj",
+      "audio_encoder.layers.5.mlp.gate_proj",
+      "audio_encoder.layers.5.self_attn.o_proj",
+      "audio_encoder.layers.5.self_attn.v_proj",
+      "audio_encoder.layers.5.self_attn.k_proj",
+      "audio_encoder.layers.5.self_attn.q_proj",
+      "audio_encoder.layers.4.mlp.down_proj",
+      "audio_encoder.layers.4.mlp.up_proj",
+      "audio_encoder.layers.4.mlp.gate_proj",
+      "audio_encoder.layers.4.self_attn.o_proj",
+      "audio_encoder.layers.4.self_attn.v_proj",
+      "audio_encoder.layers.4.self_attn.k_proj",
+      "audio_encoder.layers.4.self_attn.q_proj",
+      "audio_encoder.layers.3.mlp.down_proj",
+      "audio_encoder.layers.3.mlp.up_proj",
+      "audio_encoder.layers.3.mlp.gate_proj",
+      "audio_encoder.layers.3.self_attn.o_proj",
+      "audio_encoder.layers.3.self_attn.v_proj",
+      "audio_encoder.layers.3.self_attn.k_proj",
+      "audio_encoder.layers.3.self_attn.q_proj",
+      "audio_encoder.layers.2.mlp.down_proj",
+      "audio_encoder.layers.2.mlp.up_proj",
+      "audio_encoder.layers.2.mlp.gate_proj",
+      "audio_encoder.layers.2.self_attn.o_proj",
+      "audio_encoder.layers.2.self_attn.v_proj",
+      "audio_encoder.layers.2.self_attn.k_proj",
+      "audio_encoder.layers.2.self_attn.q_proj",
+      "audio_encoder.layers.1.mlp.down_proj",
+      "audio_encoder.layers.1.mlp.up_proj",
+      "audio_encoder.layers.1.mlp.gate_proj",
+      "audio_encoder.layers.1.self_attn.o_proj",
+      "audio_encoder.layers.1.self_attn.v_proj",
+      "audio_encoder.layers.1.self_attn.k_proj",
+      "audio_encoder.layers.1.self_attn.q_proj",
+      "audio_encoder.layers.0.mlp.down_proj",
+      "audio_encoder.layers.0.mlp.up_proj",
+      "audio_encoder.layers.0.mlp.gate_proj",
+      "audio_encoder.layers.0.self_attn.o_proj",
+      "audio_encoder.layers.0.self_attn.v_proj",
+      "audio_encoder.layers.0.self_attn.k_proj",
+      "audio_encoder.layers.0.self_attn.q_proj",
+      "audio_encoder.embedder.data_proj"
+    ],
+    "convolution_dtype": "float16"
+  }
+}

model/pe-audio/manifest.json

@@ -0,0 +1,103 @@
+{
+  "parameters": 117389248,
+  "quantization": {
+    "bits": 8,
+    "group_size": 64,
+    "modules": [
+      "audio_head.proj",
+      "audio_encoder.output",
+      "audio_encoder.layers.11.mlp.down_proj",
+      "audio_encoder.layers.11.mlp.up_proj",
+      "audio_encoder.layers.11.mlp.gate_proj",
+      "audio_encoder.layers.11.self_attn.o_proj",
+      "audio_encoder.layers.11.self_attn.v_proj",
+      "audio_encoder.layers.11.self_attn.k_proj",
+      "audio_encoder.layers.11.self_attn.q_proj",
+      "audio_encoder.layers.10.mlp.down_proj",
+      "audio_encoder.layers.10.mlp.up_proj",
+      "audio_encoder.layers.10.mlp.gate_proj",
+      "audio_encoder.layers.10.self_attn.o_proj",
+      "audio_encoder.layers.10.self_attn.v_proj",
+      "audio_encoder.layers.10.self_attn.k_proj",
+      "audio_encoder.layers.10.self_attn.q_proj",
+      "audio_encoder.layers.9.mlp.down_proj",
+      "audio_encoder.layers.9.mlp.up_proj",
+      "audio_encoder.layers.9.mlp.gate_proj",
+      "audio_encoder.layers.9.self_attn.o_proj",
+      "audio_encoder.layers.9.self_attn.v_proj",
+      "audio_encoder.layers.9.self_attn.k_proj",
+      "audio_encoder.layers.9.self_attn.q_proj",
+      "audio_encoder.layers.8.mlp.down_proj",
+      "audio_encoder.layers.8.mlp.up_proj",
+      "audio_encoder.layers.8.mlp.gate_proj",
+      "audio_encoder.layers.8.self_attn.o_proj",
+      "audio_encoder.layers.8.self_attn.v_proj",
+      "audio_encoder.layers.8.self_attn.k_proj",
+      "audio_encoder.layers.8.self_attn.q_proj",
+      "audio_encoder.layers.7.mlp.down_proj",
+      "audio_encoder.layers.7.mlp.up_proj",
+      "audio_encoder.layers.7.mlp.gate_proj",
+      "audio_encoder.layers.7.self_attn.o_proj",
+      "audio_encoder.layers.7.self_attn.v_proj",
+      "audio_encoder.layers.7.self_attn.k_proj",
+      "audio_encoder.layers.7.self_attn.q_proj",
+      "audio_encoder.layers.6.mlp.down_proj",
+      "audio_encoder.layers.6.mlp.up_proj",
+      "audio_encoder.layers.6.mlp.gate_proj",
+      "audio_encoder.layers.6.self_attn.o_proj",
+      "audio_encoder.layers.6.self_attn.v_proj",
+      "audio_encoder.layers.6.self_attn.k_proj",
+      "audio_encoder.layers.6.self_attn.q_proj",
+      "audio_encoder.layers.5.mlp.down_proj",
+      "audio_encoder.layers.5.mlp.up_proj",
+      "audio_encoder.layers.5.mlp.gate_proj",
+      "audio_encoder.layers.5.self_attn.o_proj",
+      "audio_encoder.layers.5.self_attn.v_proj",
+      "audio_encoder.layers.5.self_attn.k_proj",
+      "audio_encoder.layers.5.self_attn.q_proj",
+      "audio_encoder.layers.4.mlp.down_proj",
+      "audio_encoder.layers.4.mlp.up_proj",
+      "audio_encoder.layers.4.mlp.gate_proj",
+      "audio_encoder.layers.4.self_attn.o_proj",
+      "audio_encoder.layers.4.self_attn.v_proj",
+      "audio_encoder.layers.4.self_attn.k_proj",
+      "audio_encoder.layers.4.self_attn.q_proj",
+      "audio_encoder.layers.3.mlp.down_proj",
+      "audio_encoder.layers.3.mlp.up_proj",
+      "audio_encoder.layers.3.mlp.gate_proj",
+      "audio_encoder.layers.3.self_attn.o_proj",
+      "audio_encoder.layers.3.self_attn.v_proj",
+      "audio_encoder.layers.3.self_attn.k_proj",
+      "audio_encoder.layers.3.self_attn.q_proj",
+      "audio_encoder.layers.2.mlp.down_proj",
+      "audio_encoder.layers.2.mlp.up_proj",
+      "audio_encoder.layers.2.mlp.gate_proj",
+      "audio_encoder.layers.2.self_attn.o_proj",
+      "audio_encoder.layers.2.self_attn.v_proj",
+      "audio_encoder.layers.2.self_attn.k_proj",
+      "audio_encoder.layers.2.self_attn.q_proj",
+      "audio_encoder.layers.1.mlp.down_proj",
+      "audio_encoder.layers.1.mlp.up_proj",
+      "audio_encoder.layers.1.mlp.gate_proj",
+      "audio_encoder.layers.1.self_attn.o_proj",
+      "audio_encoder.layers.1.self_attn.v_proj",
+      "audio_encoder.layers.1.self_attn.k_proj",
+      "audio_encoder.layers.1.self_attn.q_proj",
+      "audio_encoder.layers.0.mlp.down_proj",
+      "audio_encoder.layers.0.mlp.up_proj",
+      "audio_encoder.layers.0.mlp.gate_proj",
+      "audio_encoder.layers.0.self_attn.o_proj",
+      "audio_encoder.layers.0.self_attn.v_proj",
+      "audio_encoder.layers.0.self_attn.k_proj",
+      "audio_encoder.layers.0.self_attn.q_proj",
+      "audio_encoder.embedder.data_proj"
+    ],
+    "convolution_dtype": "float16"
+  },
+  "weight_bytes": 153819456,
+  "files": {
+    "config.json": "52cad1a78304268153c436b796477c3a1dd4a62817fce42ce66e2f9cb04359d3",
+    "model.safetensors": "e424fb597caed67db9d794b3634909f06a36020970c1c31837382f17b8509a60",
+    "nanobe_mlx_pe.py": "be3d0cdd3f16a22a59379a65aa56b8f77e14adf63dad9d60eaf483686893508a"
+  }
+}

model/pe-audio/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e424fb597caed67db9d794b3634909f06a36020970c1c31837382f17b8509a60
+size 153819456

model/pe-audio/nanobe_mlx_pe.py

@@ -0,0 +1,270 @@
+"""PE audio encoder for MLX, with complete waveform coverage and no text model.
+
+Architecture follows Transformers 5.6 PeAudioEncoder. Its class token supplies
+the pooled feature. Each waveform window has a fully valid padding mask,
+including the publisher's right reflection padding.
+"""
+
+from __future__ import annotations
+
+import json
+import math
+from pathlib import Path
+from types import SimpleNamespace
+
+import mlx.core as mx
+import mlx.nn as nn
+import numpy as np
+
+
+class Snake(nn.Module):
+    def __init__(self, channels: int):
+        super().__init__()
+        self.alpha = mx.ones((1, 1, channels))
+
+    def __call__(self, value):
+        return value + mx.square(mx.sin(self.alpha * value)) / (self.alpha + 1e-9)
+
+
+class Residual(nn.Module):
+    def __init__(self, channels: int, dilation: int):
+        super().__init__()
+        self.snake1 = Snake(channels)
+        self.conv1 = nn.Conv1d(channels, channels, 7, padding=3 * dilation, dilation=dilation)
+        self.snake2 = Snake(channels)
+        self.conv2 = nn.Conv1d(channels, channels, 1)
+
+    def __call__(self, value):
+        return value + self.conv2(self.snake2(self.conv1(self.snake1(value))))
+
+
+class Downsample(nn.Module):
+    def __init__(self, channels: int, stride: int):
+        super().__init__()
+        self.res_unit1 = Residual(channels, 1)
+        self.res_unit2 = Residual(channels, 3)
+        self.res_unit3 = Residual(channels, 9)
+        self.snake1 = Snake(channels)
+        self.conv1 = nn.Conv1d(channels, channels * 2, 2 * stride,
+                               stride=stride, padding=math.ceil(stride / 2))
+
+    def __call__(self, value):
+        value = self.res_unit1(value)
+        value = self.res_unit2(value)
+        return self.conv1(self.snake1(self.res_unit3(value)))
+
+
+class Dac(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        channels = config.encoder_hidden_size
+        self.conv1 = nn.Conv1d(1, channels, 7, padding=3)
+        self.block = [Downsample(channels * 2**i, stride)
+                      for i, stride in enumerate(config.downsampling_ratios)]
+        width = channels * 2**len(self.block)
+        self.snake1 = Snake(width)
+        self.conv2 = nn.Conv1d(width, config.hidden_size, 3, padding=1)
+
+    def __call__(self, value):
+        value = self.conv1(value)
+        mx.eval(value)
+        for block in self.block:
+            value = block(value)
+            mx.eval(value)
+        return self.conv2(self.snake1(value))
+
+
+class Embedder(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.dac_encoder = Dac(config.dac_config)
+        self.bottleneck = nn.Conv1d(config.dac_config.hidden_size, config.dac_config.codebook_dim, 1)
+        self.data_proj = nn.Linear(config.dac_config.codebook_dim, config.hidden_size)
+
+    def __call__(self, value):
+        return self.data_proj(self.bottleneck(self.dac_encoder(value)))
+
+
+class Convolution(nn.Module):
+    def __init__(self, width: int):
+        super().__init__()
+        self.groupnorm = nn.GroupNorm(1, width, eps=1e-5, pytorch_compatible=True)
+        self.project = nn.Conv1d(width, width, 3, padding=1)
+
+    def __call__(self, value):
+        return self.project(nn.silu(self.groupnorm(value)))
+
+
+class Resnet(nn.Module):
+    def __init__(self, width: int):
+        super().__init__()
+        self.block1 = Convolution(width)
+        self.block2 = Convolution(width)
+
+    def __call__(self, value):
+        return value + self.block2(self.block1(value))
+
+
+class Patches(nn.Module):
+    def __init__(self, width: int):
+        super().__init__()
+        self.class_embedding = mx.zeros((1, 1, width))
+        self.resnet_block = Resnet(width)
+
+    def __call__(self, value):
+        return self.resnet_block(mx.concatenate((self.class_embedding, value), axis=1))
+
+
+class Attention(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.heads = config.num_attention_heads
+        self.dimension = config.head_dim
+        self.q_proj = nn.Linear(config.hidden_size, self.heads * self.dimension, bias=False)
+        self.k_proj = nn.Linear(config.hidden_size, self.heads * self.dimension, bias=False)
+        self.v_proj = nn.Linear(config.hidden_size, self.heads * self.dimension, bias=False)
+        self.o_proj = nn.Linear(self.heads * self.dimension, config.hidden_size, bias=False)
+        self.q_norm = nn.RMSNorm(self.dimension, eps=config.rms_norm_eps)
+        self.k_norm = nn.RMSNorm(self.dimension, eps=config.rms_norm_eps)
+        self.rope = nn.RoPE(self.dimension, traditional=True, base=config.rope_parameters["rope_theta"])
+
+    def __call__(self, value):
+        shape = (*value.shape[:2], self.heads, self.dimension)
+        query = self.q_norm(self.q_proj(value).reshape(shape)).transpose(0, 2, 1, 3)
+        key = self.k_norm(self.k_proj(value).reshape(shape)).transpose(0, 2, 1, 3)
+        values = self.v_proj(value).reshape(shape).transpose(0, 2, 1, 3)
+        attended = mx.fast.scaled_dot_product_attention(self.rope(query), self.rope(key), values,
+                                                        scale=self.dimension**-0.5)
+        return self.o_proj(attended.transpose(0, 2, 1, 3).reshape(value.shape))
+
+
+class Mlp(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.gate_proj = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
+        self.up_proj = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
+        self.down_proj = nn.Linear(config.intermediate_size, config.hidden_size, bias=False)
+
+    def __call__(self, value):
+        return self.down_proj(nn.silu(self.gate_proj(value)) * self.up_proj(value))
+
+
+class Layer(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.self_attn = Attention(config)
+        self.mlp = Mlp(config)
+        self.input_layernorm = nn.RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.post_attention_layernorm = nn.RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+
+    def __call__(self, value):
+        value = value + self.self_attn(self.input_layernorm(value))
+        return value + self.mlp(self.post_attention_layernorm(value))
+
+
+class Encoder(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.embedder = Embedder(config)
+        self.patch_embedder = Patches(config.hidden_size)
+        self.layers = [Layer(config) for _ in range(config.num_hidden_layers)]
+        self.norm = nn.RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.output = nn.Linear(config.hidden_size, config.hidden_size, bias=False)
+
+    def __call__(self, value):
+        value = self.patch_embedder(self.embedder(value))
+        for layer in self.layers:
+            value = layer(value)
+        return self.output(self.norm(value[:, 0]))
+
+
+class Head(nn.Module):
+    def __init__(self, width: int, output: int):
+        super().__init__()
+        self.layer_norm = nn.LayerNorm(width, eps=1e-6)
+        self.proj = nn.Linear(width, output, bias=False)
+
+    def __call__(self, value):
+        return self.proj(self.layer_norm(value))
+
+
+class Model(nn.Module):
+    def __init__(self, configuration: dict):
+        super().__init__()
+        config = SimpleNamespace(**configuration["audio_config"])
+        config.dac_config = SimpleNamespace(**config.dac_config)
+        if config.num_attention_heads != config.num_key_value_heads or config.attention_bias:
+            raise ValueError("PE audio requires the validated full-head, unbiased attention configuration")
+        if config.hidden_act != "silu" or config.rope_parameters["rope_type"] != "default":
+            raise ValueError("PE audio activation or rotary configuration changed")
+        self.audio_encoder = Encoder(config)
+        self.audio_head = Head(config.hidden_size, configuration["output_dimension"])
+
+    def __call__(self, waveform):
+        if waveform.ndim != 3 or waveform.shape[0] != 1 or waveform.shape[2] != 1:
+            raise ValueError("PE expects one mono waveform in batch, time, channel order")
+        if not waveform.shape[1] or waveform.shape[1] % 1920:
+            raise ValueError("PE waveform length must be a positive multiple of 1920")
+        return self.audio_head(self.audio_encoder(waveform))
+
+
+def resample(waveform: np.ndarray, rate: int) -> np.ndarray:
+    """Match torchaudio's default Hann-windowed sinc filter without importing Torch."""
+    if rate == 48000:
+        return waveform
+    divisor = math.gcd(rate, 48000)
+    source, target = rate // divisor, 48000 // divisor
+    cutoff = min(source, target) * 0.99
+    width = math.ceil(6 * source / cutoff)
+    indices = np.arange(-width, width + source, dtype=np.float32)[None] / source
+    phase = np.arange(0, -target, -1, dtype=np.float32)[:, None] / target + indices
+    phase = np.clip(phase * cutoff, -6, 6)
+    window = np.cos(phase * np.pi / 6 / 2) ** 2
+    angle = phase * np.pi
+    sinc = np.divide(np.sin(angle), angle, out=np.ones_like(angle), where=angle != 0)
+    kernel = sinc * window * (cutoff / source)
+    values = mx.array(np.pad(waveform, (width, width + source)))[None, :, None]
+    result = mx.conv1d(values, mx.array(kernel[:, :, None]), stride=source).reshape(-1)
+    # Torchaudio rounds the output length in float32 before taking its ceiling.
+    length = math.ceil(float(np.float32(target * len(waveform) / source)))
+    return np.asarray(result[:length], dtype=np.float32)
+
+
+def windows(waveform: np.ndarray, rate: int):
+    """Yield original sample bounds and each complete, resampled, padded window."""
+    waveform = np.asarray(waveform, dtype=np.float32)
+    if waveform.ndim != 1 or not len(waveform) or rate <= 0 or not np.isfinite(waveform).all():
+        raise ValueError("PE requires a finite, nonempty mono waveform and a positive sample rate")
+    count = math.ceil(len(waveform) / (30 * rate))
+    for i in range(count):
+        start, end = i * len(waveform) // count, (i + 1) * len(waveform) // count
+        value = resample(waveform[start:end], rate)
+        padding = -len(value) % 1920
+        if padding >= len(value):
+            raise ValueError("Audio is too short for the publisher's reflection padding")
+        yield start, end, np.pad(value, (0, padding), mode="reflect")
+
+
+class Runtime:
+    def __init__(self, path: Path):
+        config = json.loads((path / "config.json").read_text())
+        self.model = Model(config)
+        quantization = config["quantization"]
+        nn.quantize(self.model, bits=quantization["bits"], group_size=quantization["group_size"],
+                    class_predicate=lambda name, module: name in quantization["modules"])
+        self.model.load_weights(str(path / "model.safetensors"), strict=True)
+        self.model.eval()
+        mx.eval(self.model.parameters())
+
+    def encode(self, waveform: np.ndarray, rate: int) -> np.ndarray:
+        total = np.zeros(1024, dtype=np.float32)
+        dtype = self.model.audio_encoder.embedder.dac_encoder.conv1.weight.dtype
+        for start, end, value in windows(waveform, rate):
+            output = self.model(mx.array(value, dtype=dtype)[None, :, None]).astype(mx.float32)
+            mx.eval(output)
+            total += np.asarray(output[0]) * ((end - start) / len(waveform))
+        # Long DAC convolutions need large temporary buffers, not an idle cache.
+        mx.clear_cache()
+        if not np.isfinite(total).all() or np.linalg.norm(total) == 0:
+            raise ValueError("PE audio produced an invalid embedding")
+        return total

model/pe-text/config.json

@@ -0,0 +1,155 @@
+{
+  "text_config": {
+    "attention_bias": false,
+    "cls_token_id": 50281,
+    "hidden_activation": "gelu",
+    "hidden_size": 1024,
+    "intermediate_size": 2624,
+    "layer_types": [
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention"
+    ],
+    "local_attention": 128,
+    "max_position_embeddings": 8192,
+    "mlp_bias": false,
+    "model_type": "modernbert",
+    "norm_bias": false,
+    "norm_eps": 1e-05,
+    "num_attention_heads": 16,
+    "num_hidden_layers": 22,
+    "rope_parameters": {
+      "full_attention": {
+        "rope_theta": 160000.0,
+        "rope_type": "default"
+      },
+      "sliding_attention": {
+        "rope_theta": 10000.0,
+        "rope_type": "default"
+      }
+    },
+    "sep_token_id": 50282,
+    "vocab_size": 50368
+  },
+  "quantization": {
+    "bits": 8,
+    "group_size": 64,
+    "modules": [
+      "text_model.layers.21.mlp.Wo",
+      "text_model.layers.21.mlp.Wi",
+      "text_model.layers.21.attn.Wo",
+      "text_model.layers.21.attn.Wqkv",
+      "text_model.layers.20.mlp.Wo",
+      "text_model.layers.20.mlp.Wi",
+      "text_model.layers.20.attn.Wo",
+      "text_model.layers.20.attn.Wqkv",
+      "text_model.layers.19.mlp.Wo",
+      "text_model.layers.19.mlp.Wi",
+      "text_model.layers.19.attn.Wo",
+      "text_model.layers.19.attn.Wqkv",
+      "text_model.layers.18.mlp.Wo",
+      "text_model.layers.18.mlp.Wi",
+      "text_model.layers.18.attn.Wo",
+      "text_model.layers.18.attn.Wqkv",
+      "text_model.layers.17.mlp.Wo",
+      "text_model.layers.17.mlp.Wi",
+      "text_model.layers.17.attn.Wo",
+      "text_model.layers.17.attn.Wqkv",
+      "text_model.layers.16.mlp.Wo",
+      "text_model.layers.16.mlp.Wi",
+      "text_model.layers.16.attn.Wo",
+      "text_model.layers.16.attn.Wqkv",
+      "text_model.layers.15.mlp.Wo",
+      "text_model.layers.15.mlp.Wi",
+      "text_model.layers.15.attn.Wo",
+      "text_model.layers.15.attn.Wqkv",
+      "text_model.layers.14.mlp.Wo",
+      "text_model.layers.14.mlp.Wi",
+      "text_model.layers.14.attn.Wo",
+      "text_model.layers.14.attn.Wqkv",
+      "text_model.layers.13.mlp.Wo",
+      "text_model.layers.13.mlp.Wi",
+      "text_model.layers.13.attn.Wo",
+      "text_model.layers.13.attn.Wqkv",
+      "text_model.layers.12.mlp.Wo",
+      "text_model.layers.12.mlp.Wi",
+      "text_model.layers.12.attn.Wo",
+      "text_model.layers.12.attn.Wqkv",
+      "text_model.layers.11.mlp.Wo",
+      "text_model.layers.11.mlp.Wi",
+      "text_model.layers.11.attn.Wo",
+      "text_model.layers.11.attn.Wqkv",
+      "text_model.layers.10.mlp.Wo",
+      "text_model.layers.10.mlp.Wi",
+      "text_model.layers.10.attn.Wo",
+      "text_model.layers.10.attn.Wqkv",
+      "text_model.layers.9.mlp.Wo",
+      "text_model.layers.9.mlp.Wi",
+      "text_model.layers.9.attn.Wo",
+      "text_model.layers.9.attn.Wqkv",
+      "text_model.layers.8.mlp.Wo",
+      "text_model.layers.8.mlp.Wi",
+      "text_model.layers.8.attn.Wo",
+      "text_model.layers.8.attn.Wqkv",
+      "text_model.layers.7.mlp.Wo",
+      "text_model.layers.7.mlp.Wi",
+      "text_model.layers.7.attn.Wo",
+      "text_model.layers.7.attn.Wqkv",
+      "text_model.layers.6.mlp.Wo",
+      "text_model.layers.6.mlp.Wi",
+      "text_model.layers.6.attn.Wo",
+      "text_model.layers.6.attn.Wqkv",
+      "text_model.layers.5.mlp.Wo",
+      "text_model.layers.5.mlp.Wi",
+      "text_model.layers.5.attn.Wo",
+      "text_model.layers.5.attn.Wqkv",
+      "text_model.layers.4.mlp.Wo",
+      "text_model.layers.4.mlp.Wi",
+      "text_model.layers.4.attn.Wo",
+      "text_model.layers.4.attn.Wqkv",
+      "text_model.layers.3.mlp.Wo",
+      "text_model.layers.3.mlp.Wi",
+      "text_model.layers.3.attn.Wo",
+      "text_model.layers.3.attn.Wqkv",
+      "text_model.layers.2.mlp.Wo",
+      "text_model.layers.2.mlp.Wi",
+      "text_model.layers.2.attn.Wo",
+      "text_model.layers.2.attn.Wqkv",
+      "text_model.layers.1.mlp.Wo",
+      "text_model.layers.1.mlp.Wi",
+      "text_model.layers.1.attn.Wo",
+      "text_model.layers.1.attn.Wqkv",
+      "text_model.layers.0.mlp.Wo",
+      "text_model.layers.0.mlp.Wi",
+      "text_model.layers.0.attn.Wo",
+      "text_model.layers.0.attn.Wqkv",
+      "text_model.embeddings.tok_embeddings"
+    ],
+    "unquantized_dtype": "float16",
+    "float32_tensors": [
+      "text_audio_head.layer_norm.bias",
+      "text_audio_head.layer_norm.weight",
+      "text_model.final_norm.weight",
+      "text_audio_head.proj.weight"
+    ]
+  }
+}

model/pe-text/manifest.json

@@ -0,0 +1,120 @@
+{
+  "schema": "nanobe-pe-text-mlx-v1",
+  "complete": true,
+  "source": {
+    "model": "facebook/pe-av-small",
+    "revision": "dd050762bb9704ae9cd996ca45532a98f81d817e"
+  },
+  "parameters": 322288640,
+  "quantization": {
+    "bits": 8,
+    "group_size": 64,
+    "modules": [
+      "text_model.layers.21.mlp.Wo",
+      "text_model.layers.21.mlp.Wi",
+      "text_model.layers.21.attn.Wo",
+      "text_model.layers.21.attn.Wqkv",
+      "text_model.layers.20.mlp.Wo",
+      "text_model.layers.20.mlp.Wi",
+      "text_model.layers.20.attn.Wo",
+      "text_model.layers.20.attn.Wqkv",
+      "text_model.layers.19.mlp.Wo",
+      "text_model.layers.19.mlp.Wi",
+      "text_model.layers.19.attn.Wo",
+      "text_model.layers.19.attn.Wqkv",
+      "text_model.layers.18.mlp.Wo",
+      "text_model.layers.18.mlp.Wi",
+      "text_model.layers.18.attn.Wo",
+      "text_model.layers.18.attn.Wqkv",
+      "text_model.layers.17.mlp.Wo",
+      "text_model.layers.17.mlp.Wi",
+      "text_model.layers.17.attn.Wo",
+      "text_model.layers.17.attn.Wqkv",
+      "text_model.layers.16.mlp.Wo",
+      "text_model.layers.16.mlp.Wi",
+      "text_model.layers.16.attn.Wo",
+      "text_model.layers.16.attn.Wqkv",
+      "text_model.layers.15.mlp.Wo",
+      "text_model.layers.15.mlp.Wi",
+      "text_model.layers.15.attn.Wo",
+      "text_model.layers.15.attn.Wqkv",
+      "text_model.layers.14.mlp.Wo",
+      "text_model.layers.14.mlp.Wi",
+      "text_model.layers.14.attn.Wo",
+      "text_model.layers.14.attn.Wqkv",
+      "text_model.layers.13.mlp.Wo",
+      "text_model.layers.13.mlp.Wi",
+      "text_model.layers.13.attn.Wo",
+      "text_model.layers.13.attn.Wqkv",
+      "text_model.layers.12.mlp.Wo",
+      "text_model.layers.12.mlp.Wi",
+      "text_model.layers.12.attn.Wo",
+      "text_model.layers.12.attn.Wqkv",
+      "text_model.layers.11.mlp.Wo",
+      "text_model.layers.11.mlp.Wi",
+      "text_model.layers.11.attn.Wo",
+      "text_model.layers.11.attn.Wqkv",
+      "text_model.layers.10.mlp.Wo",
+      "text_model.layers.10.mlp.Wi",
+      "text_model.layers.10.attn.Wo",
+      "text_model.layers.10.attn.Wqkv",
+      "text_model.layers.9.mlp.Wo",
+      "text_model.layers.9.mlp.Wi",
+      "text_model.layers.9.attn.Wo",
+      "text_model.layers.9.attn.Wqkv",
+      "text_model.layers.8.mlp.Wo",
+      "text_model.layers.8.mlp.Wi",
+      "text_model.layers.8.attn.Wo",
+      "text_model.layers.8.attn.Wqkv",
+      "text_model.layers.7.mlp.Wo",
+      "text_model.layers.7.mlp.Wi",
+      "text_model.layers.7.attn.Wo",
+      "text_model.layers.7.attn.Wqkv",
+      "text_model.layers.6.mlp.Wo",
+      "text_model.layers.6.mlp.Wi",
+      "text_model.layers.6.attn.Wo",
+      "text_model.layers.6.attn.Wqkv",
+      "text_model.layers.5.mlp.Wo",
+      "text_model.layers.5.mlp.Wi",
+      "text_model.layers.5.attn.Wo",
+      "text_model.layers.5.attn.Wqkv",
+      "text_model.layers.4.mlp.Wo",
+      "text_model.layers.4.mlp.Wi",
+      "text_model.layers.4.attn.Wo",
+      "text_model.layers.4.attn.Wqkv",
+      "text_model.layers.3.mlp.Wo",
+      "text_model.layers.3.mlp.Wi",
+      "text_model.layers.3.attn.Wo",
+      "text_model.layers.3.attn.Wqkv",
+      "text_model.layers.2.mlp.Wo",
+      "text_model.layers.2.mlp.Wi",
+      "text_model.layers.2.attn.Wo",
+      "text_model.layers.2.attn.Wqkv",
+      "text_model.layers.1.mlp.Wo",
+      "text_model.layers.1.mlp.Wi",
+      "text_model.layers.1.attn.Wo",
+      "text_model.layers.1.attn.Wqkv",
+      "text_model.layers.0.mlp.Wo",
+      "text_model.layers.0.mlp.Wi",
+      "text_model.layers.0.attn.Wo",
+      "text_model.layers.0.attn.Wqkv",
+      "text_model.embeddings.tok_embeddings"
+    ],
+    "unquantized_dtype": "float16",
+    "float32_tensors": [
+      "text_audio_head.layer_norm.bias",
+      "text_audio_head.layer_norm.weight",
+      "text_model.final_norm.weight",
+      "text_audio_head.proj.weight"
+    ]
+  },
+  "weight_bytes": 345597033,
+  "files": {
+    "model.safetensors": "48ffbbea2595722fffcd320bc07a5af07149fafcf00f03f9b02b4f138262928e",
+    "config.json": "9402387f9eb47c9ba91bc6b62c13834d34a1bf31cafcee7e4211fd9997d834d7",
+    "tokenizer.json": "43e9ea931bb0f359d16d0e435c26abbebd9a42a7298270b8ab7e5c9fa1be1dcb",
+    "tokenizer_config.json": "adec72131bbba5a043594c1e3bf5930608546e46a51432b2502b392ec280551b",
+    "special_tokens_map.json": "ea97ecdbcc73713039d8d64dbb05e3689495c96657fbd9a18f5bed381be81049",
+    "pe_text.py": "025bc642db25eddad124a289535acf3cbca1d7a88bf409c7b9fe1b4aecd6f4f7"
+  }
+}

model/pe-text/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:48ffbbea2595722fffcd320bc07a5af07149fafcf00f03f9b02b4f138262928e
+size 345597033

model/pe-text/pe_text.py

@@ -0,0 +1,158 @@
+"""Native PE text embeddings with the publisher's ModernBERT CLS readout."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from types import SimpleNamespace
+
+import mlx.core as mx
+import mlx.nn as nn
+import numpy as np
+from tokenizers import Tokenizer
+
+
+class Embeddings(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.tok_embeddings = nn.Embedding(config.vocab_size, config.hidden_size)
+        self.norm = nn.LayerNorm(config.hidden_size, eps=config.norm_eps, bias=False)
+
+    def __call__(self, tokens):
+        return self.norm(self.tok_embeddings(tokens))
+
+
+class Attention(nn.Module):
+    def __init__(self, config, kind: str):
+        super().__init__()
+        self.heads = config.num_attention_heads
+        self.dimension = config.hidden_size // self.heads
+        self.Wqkv = nn.Linear(config.hidden_size, 3 * config.hidden_size, bias=False)
+        self.Wo = nn.Linear(config.hidden_size, config.hidden_size, bias=False)
+        self.rope = nn.RoPE(self.dimension, traditional=False,
+                            base=config.rope_parameters[kind]["rope_theta"])
+
+    def __call__(self, value, mask):
+        shape = (*value.shape[:2], 3, self.heads, self.dimension)
+        packed = self.Wqkv(value).reshape(shape)
+        query = packed[:, :, 0].transpose(0, 2, 1, 3)
+        key = packed[:, :, 1].transpose(0, 2, 1, 3)
+        values = packed[:, :, 2].transpose(0, 2, 1, 3)
+        attended = mx.fast.scaled_dot_product_attention(
+            self.rope(query), self.rope(key), values, scale=self.dimension**-0.5, mask=mask)
+        return self.Wo(attended.transpose(0, 2, 1, 3).reshape(value.shape))
+
+
+class Mlp(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.Wi = nn.Linear(config.hidden_size, 2 * config.intermediate_size, bias=False)
+        self.Wo = nn.Linear(config.intermediate_size, config.hidden_size, bias=False)
+
+    def __call__(self, value):
+        value, gate = mx.split(self.Wi(value), 2, axis=-1)
+        return self.Wo(nn.gelu(value) * gate)
+
+
+class Layer(nn.Module):
+    def __init__(self, config, index: int):
+        super().__init__()
+        self.local = config.layer_types[index] == "sliding_attention"
+        self.attn_norm = nn.Identity() if index == 0 else nn.LayerNorm(
+            config.hidden_size, eps=config.norm_eps, bias=False)
+        self.attn = Attention(config, config.layer_types[index])
+        self.mlp_norm = nn.LayerNorm(config.hidden_size, eps=config.norm_eps, bias=False)
+        self.mlp = Mlp(config)
+
+    def __call__(self, value, mask):
+        value = value + self.attn(self.attn_norm(value), mask if self.local else None)
+        return value + self.mlp(self.mlp_norm(value))
+
+
+class Text(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.window = config.local_attention // 2
+        self.embeddings = Embeddings(config)
+        self.layers = [Layer(config, index) for index in range(config.num_hidden_layers)]
+        self.final_norm = nn.LayerNorm(config.hidden_size, eps=config.norm_eps, bias=False)
+
+    def __call__(self, tokens):
+        positions = mx.arange(tokens.shape[1])
+        # Transformers includes both boundaries of the bidirectional local window.
+        mask = mx.abs(positions[:, None] - positions[None, :]) <= self.window
+        value = self.embeddings(tokens)
+        for layer in self.layers:
+            value = layer(value, mask)
+        return self.final_norm(value[:, 0].astype(mx.float32))
+
+
+class Head(nn.Module):
+    def __init__(self, width: int):
+        super().__init__()
+        self.layer_norm = nn.LayerNorm(width, eps=1e-6)
+        self.proj = nn.Linear(width, width, bias=False)
+
+    def __call__(self, value):
+        return self.proj(self.layer_norm(value))
+
+
+class Model(nn.Module):
+    def __init__(self, configuration: dict):
+        super().__init__()
+        config = SimpleNamespace(**configuration["text_config"])
+        if (config.model_type != "modernbert" or config.hidden_activation != "gelu"
+                or config.attention_bias or config.mlp_bias or config.norm_bias):
+            raise ValueError("PE text requires the pinned unbiased ModernBERT GELU architecture")
+        if (config.hidden_size != 1024 or config.num_attention_heads != 16
+                or config.num_hidden_layers != 22 or config.intermediate_size != 2624
+                or config.vocab_size != 50368 or config.local_attention != 128
+                or config.max_position_embeddings != 8192):
+            raise ValueError("The pinned PE text dimensions changed")
+        layers = ["sliding_attention" if index % 3 else "full_attention" for index in range(22)]
+        if config.layer_types != layers or config.rope_parameters != {
+            "full_attention": {"rope_type": "default", "rope_theta": 160000.0},
+            "sliding_attention": {"rope_type": "default", "rope_theta": 10000.0},
+        }:
+            raise ValueError("The pinned PE text attention pattern changed")
+        self.text_model = Text(config)
+        self.text_audio_head = Head(config.hidden_size)
+
+    def __call__(self, tokens):
+        if tokens.ndim != 2 or tokens.shape[0] != 1 or not 2 <= tokens.shape[1] <= 8192:
+            raise ValueError("PE text expects one complete token sequence of length 2 through 8192")
+        return self.text_audio_head(self.text_model(tokens))
+
+
+class Runtime:
+    def __init__(self, path: Path):
+        path = Path(path)
+        config = json.loads((path / "config.json").read_text())
+        self.limit = config["text_config"]["max_position_embeddings"]
+        self.cls = config["text_config"]["cls_token_id"]
+        self.sep = config["text_config"]["sep_token_id"]
+        self.tokenizer = Tokenizer.from_file(str(path / "tokenizer.json"))
+        self.tokenizer.no_truncation()
+        self.tokenizer.no_padding()
+        self.model = Model(config)
+        quantization = config["quantization"]
+        nn.quantize(self.model, bits=quantization["bits"], group_size=quantization["group_size"],
+                    class_predicate=lambda name, module: name in quantization["modules"])
+        self.model.load_weights(str(path / "model.safetensors"), strict=True)
+        self.model.eval()
+        mx.eval(self.model.parameters())
+
+    def encode(self, text: str) -> np.ndarray:
+        if not isinstance(text, str):
+            raise TypeError("PE text expects a string")
+        tokens = self.tokenizer.encode(text, add_special_tokens=True).ids
+        if len(tokens) > self.limit:
+            raise ValueError(f"PE text has {len(tokens)} tokens; the maximum is {self.limit}")
+        if not tokens or tokens[0] != self.cls or tokens[-1] != self.sep:
+            raise ValueError("The PE tokenizer must preserve the publisher's CLS and SEP tokens")
+        output = self.model(mx.array([tokens], dtype=mx.int32))[0].astype(mx.float32)
+        mx.eval(output)
+        result = np.asarray(output, dtype=np.float32)
+        if result.shape != (1024,) or not np.isfinite(result).all():
+            raise ValueError("PE text produced an invalid raw embedding")
+        return result

model/pe-text/special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

model/pe-text/tokenizer_config.json

@@ -0,0 +1,18 @@
+{
+  "additional_special_tokens": null,
+  "backend": "tokenizers",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "is_local": false,
+  "mask_token": "[MASK]",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 8192,
+  "pad_token": "[PAD]",
+  "processor_class": "PeAudioVideoProcessor",
+  "sep_token": "[SEP]",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "[UNK]"
+}

model/preprocessor_config.json

@@ -0,0 +1,17 @@
+{
+  "image_processor_type": "Qwen2VLImageProcessorFast",
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [0.5, 0.5, 0.5],
+  "image_std": [0.5, 0.5, 0.5],
+  "min_pixels": 262144,
+  "max_pixels": 1310720,
+  "merge_size": 2,
+  "patch_size": 16,
+  "processor_class": "Qwen3VLProcessor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "temporal_patch_size": 2
+}

model/processor_config.json

@@ -0,0 +1,61 @@
+{
+  "processor_class": "Qwen3VLProcessor",
+  "image_processor": {
+    "image_processor_type": "Qwen2VLImageProcessorFast",
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "min_pixels": 262144,
+    "max_pixels": 1310720,
+    "size": {
+      "longest_edge": 16777216,
+      "shortest_edge": 65536
+    },
+    "merge_size": 2,
+    "patch_size": 16,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "temporal_patch_size": 2
+  },
+  "video_processor": {
+    "video_processor_type": "Qwen3VLVideoProcessor",
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "do_sample_frames": true,
+    "fps": 2,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "max_frames": 768,
+    "min_frames": 4,
+    "merge_size": 2,
+    "patch_size": 16,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "size": {
+      "longest_edge": 25165824,
+      "shortest_edge": 4096
+    },
+    "temporal_patch_size": 2
+  }
+}

model/tokenizer.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8e08386ae43d4a5024a97dadbdc7291d6770135921c97e913644f991b3a87b8e
+size 2189933

model/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7bbf567d9e053a72f0bd8ea475e672d10d537578f6adb521ab42da2badfa8bef
+size 11423488

model/tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "is_local": false,
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "processor_class": "Qwen3VLProcessor",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

model/upstream-notices.md

@@ -0,0 +1,5 @@
+# Model lineage
+
+Nanobe builds on Jina AI's [jina-embeddings-v5-omni-small-retrieval](https://huggingface.co/jinaai/jina-embeddings-v5-omni-small-retrieval), revision `12949877f0092093f366c6450340011320152a05`, and Meta's [Perception Encoder AV](https://huggingface.co/facebook/pe-av-small). The Perception Encoder text component records revision `dd050762bb9704ae9cd996ca45532a98f81d817e`.
+
+The original Jina model declares CC BY-NC 4.0. The original Meta model declares Apache 2.0. Their license texts are preserved in [licenses/jina-cc-by-nc-4.0.txt](licenses/jina-cc-by-nc-4.0.txt) and [licenses/meta-apache-2.0.txt](licenses/meta-apache-2.0.txt).

model/video_preprocessor_config.json

@@ -0,0 +1,22 @@
+{
+  "video_processor_type": "Qwen3VLVideoProcessor",
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "do_sample_frames": true,
+  "fps": 2,
+  "image_mean": [0.5, 0.5, 0.5],
+  "image_std": [0.5, 0.5, 0.5],
+  "max_frames": 768,
+  "min_frames": 4,
+  "merge_size": 2,
+  "patch_size": 16,
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "longest_edge": 25165824,
+    "shortest_edge": 4096
+  },
+  "temporal_patch_size": 2
+}

source/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Artem G.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

source/README.md

@@ -0,0 +1,105 @@
+# Nanobe
+
+Nanobe is a local multimodal embedding model and a Python reference implementation for searching folders. It encodes text, document pages, images, audio and video into a shared vector space on Apple Silicon.
+
+## Install
+
+Use an Apple Silicon Mac with Python 3.12 and [uv](https://docs.astral.sh/uv/). This implementation uses MLX. Linux, Windows and Intel Macs are not supported.
+
+Download the complete [Hugging Face release](https://huggingface.co/lokimorty/nanobe). The `source` directory contains the Python package, examples, and dependency lock. The `model` directory contains weights, tokenizers, configuration, and model helpers. Keep that model directory complete.
+
+```sh
+uvx --from huggingface-hub hf download lokimorty/nanobe --local-dir nanobe-release
+cd nanobe-release/source
+uv sync --frozen
+export NANOBE_MODEL_DIR="$(cd ../model && pwd)"
+uv run --frozen nanobe --help
+```
+
+`uv sync --frozen` installs the versions in the supplied lock and downloads Python 3.12 if needed. It does not download the model. Keep using `uv run --frozen` from this source directory. To run from elsewhere, activate its virtual environment or use the full path to `.venv/bin/nanobe`. Set `NANOBE_MODEL_DIR` in each new shell or pass `--model-dir /path/to/model` to `index` and `search`.
+
+DOC and PPT additionally require [LibreOffice](https://www.libreoffice.org/download/download-libreoffice/). Install the Apple Silicon application in `/Applications`, or put `soffice` on PATH. Other supported formats do not require LibreOffice.
+
+### Verify the installation
+
+Create two small files and a reference copy in the source directory:
+
+```sh
+mkdir -p demo
+printf '%s\n' 'The observatory telescope shows Saturn and its rings in the night sky.' > demo/observatory.txt
+printf '%s\n' 'Sourdough bread uses flour, water and a fermented starter.' > demo/bread.txt
+cp demo/observatory.txt reference.txt
+uv run --frozen nanobe index demo --format json
+uv run --frozen nanobe search demo "Saturn through a telescope" --format json
+uv run --frozen nanobe search demo --file reference.txt --format json
+uv run --frozen nanobe status demo --format json
+uv run --frozen nanobe index demo --format json
+```
+
+The first index should report `embedded_files: 2`, `embedded_segments: 2` and `failed_files: 0`. Both searches should put `observatory.txt` first. Status should show `ready: 2` and `index_ready: true`. The second index should report `embedded_files: 0` and `unchanged_files: 2`.
+
+Now run the Python reference example over the same files:
+
+```sh
+uv run --frozen python examples/search.py --query "Saturn through a telescope" demo/*.txt
+uv run --frozen python examples/search.py --file reference.txt demo/*.txt
+```
+
+Both should rank `demo/observatory.txt` first. This example encodes every supplied file, applies the correct query and item calibration, and ranks by the best matching segment. It performs exact semantic scoring with full-precision embeddings. The folder CLI also stores an index and adds lexical matching for text queries, so its scores are not interchangeable with the example's cosine scores.
+
+To search your own folder:
+
+```sh
+uv run --frozen nanobe index ~/Creative
+uv run --frozen nanobe search ~/Creative "a presentation about sustainable packaging"
+uv run --frozen nanobe search ~/Creative --file ~/Desktop/reference.jpg
+```
+
+See [Python inference](docs/model.md#python-inference) for the embedding, segmentation and scoring calls used by the example.
+
+### Troubleshooting
+
+- Missing model files or an integrity error: download the complete matching model directory again and point `NANOBE_MODEL_DIR` at the directory containing `manifest.json`.
+- An index belongs to another model package: run `nanobe index` with the current model to rebuild its embeddings. Keep a separate `--state-dir` if you need both indexes.
+- DOC/PPT conversion failure: check that LibreOffice is installed and the file opens there. Nanobe reports a per-file failure and continues indexing other files.
+- A decoder error: check the reported file. `--debug` includes a traceback; `--format json` keeps results on stdout and warnings on stderr.
+
+## Folder search
+
+Indexing records files and their segments in `.nanobe` inside the selected folder. Repeating `index` updates changed files. `--rebuild` re-embeds supported files. `status --format json` reports catalog state. Hidden files, symbolic links and the index directory are excluded.
+
+Text queries combine semantic retrieval with lexical matches over relative paths and extracted text. Lexical matching helps with names, symbols and exact phrases. A reference file uses semantic retrieval only. The catalog searches an int8 approximate index, reranks candidates using stored higher-precision vectors, then groups segments into file results. These retrieval steps belong to the reference application. They are not additional model outputs.
+
+## Inputs and limits
+
+| Input | Reader behavior |
+| --- | --- |
+| Text and source code, including TXT, Markdown, CSV, TSV, JSON, YAML, XML and HTML | UTF-8 text. CSV remains plain text. |
+| DOCX, PPTX, EPUB | Extracted text. Embedded pictures and page layout are not encoded. |
+| RTF | Text extraction with declared codepages and Unicode. Formatting and pictures are not encoded. |
+| XLS, XLSX | Saved cell values serialized as CSV with sheet names and repeated first-row headers. Formulas are not recalculated. |
+| PDF | Rendered pages through native vision, without OCR. |
+| DOC, PPT | LibreOffice conversion to PDF, then the same rendered-page path. |
+| JPG, JPEG, PNG, WebP, GIF, BMP, TIF, TIFF, AVIF | Pillow decoding to RGB. Animated inputs use the opened frame. |
+| HEIC | Primary image with container orientation applied, HDR reduced to 8 bits, then RGB. |
+| SVG | Visual rasterization on white within a 2048-by-2048 box, preserving aspect ratio. |
+| WAV, MP3, FLAC, OGG, M4A, AAC, OPUS | Decoded audio, without a transcript requirement. |
+| MP4, MOV, MKV, WebM, AVI, M4V | Sampled video frames with the decoded soundtrack when present. |
+
+SVG text uses installed fonts and the renderer's platform-dependent fallback. No font is bundled. Results can differ between machines with different fonts.
+
+DOC and PPT require LibreOffice. Nanobe finds `soffice` or `libreoffice` on PATH, or the standard `/Applications/LibreOffice.app` installation. Conversion uses a temporary profile, at most eight files per batch and a 120-second timeout per batch. Search results retain the original DOC/PPT path. Temporary PDFs are removed after processing and are not cataloged as separate files.
+
+Text uses at most eight overlapping segments of 6,000 characters with 600-character overlap. Workbooks share eight segment slots across the first eight nonempty sheets, with at most 6,000 characters per segment. Short sheets can leave slots unused. Each segment repeats the sheet name and first row. Oversized cells and omitted rows or sheets produce a truncation warning on stderr. Spreadsheet styling, charts and recalculation are outside this reader.
+
+PDFs and converted DOC/PPT files use the first 24 pages plus up to eight pages spread across the remainder. Audio retains a whole-file row and up to 24 windows of 45 seconds, plus a pooled row when windows exist. Video retains a whole-file row and up to eight windows of 24 seconds. Window starts are spread across the duration. Each video pass samples up to 16 frames. These limits mean some content in long files is not encoded as individual segments.
+
+The native audio tower uses a strided summary capped at 120 seconds for a whole-file pass. The acoustic encoder also reads the whole recording in windows of up to 30 seconds. The 120-second summary is not a cap on total decoding work or recording length.
+
+The installed decoders must support the file's actual contents. An extension alone does not guarantee decoding. Empty or unreadable files cannot produce an embedding. Missing LibreOffice, conversion timeouts and decoder errors are recorded as per-file failures during indexing, which continues with other files. A failed reference-file query reports an error. Unsupported extensions are skipped. Search requires a successfully built index and a matching model package.
+
+## License
+
+Nanobe is released under the [MIT License](LICENSE).
+
+See [model lineage](docs/upstream-notices.md) for the Jina and Meta components used in Nanobe.

source/docs/licenses/jina-cc-by-nc-4.0.txt

@@ -0,0 +1,408 @@
+Attribution-NonCommercial 4.0 International
+
+=======================================================================
+
+Creative Commons Corporation ("Creative Commons") is not a law firm and
+does not provide legal services or legal advice. Distribution of
+Creative Commons public licenses does not create a lawyer-client or
+other relationship. Creative Commons makes its licenses and related
+information available on an "as-is" basis. Creative Commons gives no
+warranties regarding its licenses, any material licensed under their
+terms and conditions, or any related information. Creative Commons
+disclaims all liability for damages resulting from their use to the
+fullest extent possible.
+
+Using Creative Commons Public Licenses
+
+Creative Commons public licenses provide a standard set of terms and
+conditions that creators and other rights holders may use to share
+original works of authorship and other material subject to copyright
+and certain other rights specified in the public license below. The
+following considerations are for informational purposes only, are not
+exhaustive, and do not form part of our licenses.
+
+     Considerations for licensors: Our public licenses are
+     intended for use by those authorized to give the public
+     permission to use material in ways otherwise restricted by
+     copyright and certain other rights. Our licenses are
+     irrevocable. Licensors should read and understand the terms
+     and conditions of the license they choose before applying it.
+     Licensors should also secure all rights necessary before
+     applying our licenses so that the public can reuse the
+     material as expected. Licensors should clearly mark any
+     material not subject to the license. This includes other CC-
+     licensed material, or material used under an exception or
+     limitation to copyright. More considerations for licensors:
+    wiki.creativecommons.org/Considerations_for_licensors
+
+     Considerations for the public: By using one of our public
+     licenses, a licensor grants the public permission to use the
+     licensed material under specified terms and conditions. If
+     the licensor's permission is not necessary for any reason--for
+     example, because of any applicable exception or limitation to
+     copyright--then that use is not regulated by the license. Our
+     licenses grant only permissions under copyright and certain
+     other rights that a licensor has authority to grant. Use of
+     the licensed material may still be restricted for other
+     reasons, including because others have copyright or other
+     rights in the material. A licensor may make special requests,
+     such as asking that all changes be marked or described.
+     Although not required by our licenses, you are encouraged to
+     respect those requests where reasonable. More considerations
+     for the public:
+    wiki.creativecommons.org/Considerations_for_licensees
+
+=======================================================================
+
+Creative Commons Attribution-NonCommercial 4.0 International Public
+License
+
+By exercising the Licensed Rights (defined below), You accept and agree
+to be bound by the terms and conditions of this Creative Commons
+Attribution-NonCommercial 4.0 International Public License ("Public
+License"). To the extent this Public License may be interpreted as a
+contract, You are granted the Licensed Rights in consideration of Your
+acceptance of these terms and conditions, and the Licensor grants You
+such rights in consideration of benefits the Licensor receives from
+making the Licensed Material available under these terms and
+conditions.
+
+
+Section 1 -- Definitions.
+
+  a. Adapted Material means material subject to Copyright and Similar
+     Rights that is derived from or based upon the Licensed Material
+     and in which the Licensed Material is translated, altered,
+     arranged, transformed, or otherwise modified in a manner requiring
+     permission under the Copyright and Similar Rights held by the
+     Licensor. For purposes of this Public License, where the Licensed
+     Material is a musical work, performance, or sound recording,
+     Adapted Material is always produced where the Licensed Material is
+     synched in timed relation with a moving image.
+
+  b. Adapter's License means the license You apply to Your Copyright
+     and Similar Rights in Your contributions to Adapted Material in
+     accordance with the terms and conditions of this Public License.
+
+  c. Copyright and Similar Rights means copyright and/or similar rights
+     closely related to copyright including, without limitation,
+     performance, broadcast, sound recording, and Sui Generis Database
+     Rights, without regard to how the rights are labeled or
+     categorized. For purposes of this Public License, the rights
+     specified in Section 2(b)(1)-(2) are not Copyright and Similar
+     Rights.
+  d. Effective Technological Measures means those measures that, in the
+     absence of proper authority, may not be circumvented under laws
+     fulfilling obligations under Article 11 of the WIPO Copyright
+     Treaty adopted on December 20, 1996, and/or similar international
+     agreements.
+
+  e. Exceptions and Limitations means fair use, fair dealing, and/or
+     any other exception or limitation to Copyright and Similar Rights
+     that applies to Your use of the Licensed Material.
+
+  f. Licensed Material means the artistic or literary work, database,
+     or other material to which the Licensor applied this Public
+     License.
+
+  g. Licensed Rights means the rights granted to You subject to the
+     terms and conditions of this Public License, which are limited to
+     all Copyright and Similar Rights that apply to Your use of the
+     Licensed Material and that the Licensor has authority to license.
+
+  h. Licensor means the individual(s) or entity(ies) granting rights
+     under this Public License.
+
+  i. NonCommercial means not primarily intended for or directed towards
+     commercial advantage or monetary compensation. For purposes of
+     this Public License, the exchange of the Licensed Material for
+     other material subject to Copyright and Similar Rights by digital
+     file-sharing or similar means is NonCommercial provided there is
+     no payment of monetary compensation in connection with the
+     exchange.
+
+  j. Share means to provide material to the public by any means or
+     process that requires permission under the Licensed Rights, such
+     as reproduction, public display, public performance, distribution,
+     dissemination, communication, or importation, and to make material
+     available to the public including in ways that members of the
+     public may access the material from a place and at a time
+     individually chosen by them.
+
+  k. Sui Generis Database Rights means rights other than copyright
+     resulting from Directive 96/9/EC of the European Parliament and of
+     the Council of 11 March 1996 on the legal protection of databases,
+     as amended and/or succeeded, as well as other essentially
+     equivalent rights anywhere in the world.
+
+  l. You means the individual or entity exercising the Licensed Rights
+     under this Public License. Your has a corresponding meaning.
+
+
+Section 2 -- Scope.
+
+  a. License grant.
+
+       1. Subject to the terms and conditions of this Public License,
+          the Licensor hereby grants You a worldwide, royalty-free,
+          non-sublicensable, non-exclusive, irrevocable license to
+          exercise the Licensed Rights in the Licensed Material to:
+
+            a. reproduce and Share the Licensed Material, in whole or
+               in part, for NonCommercial purposes only; and
+
+            b. produce, reproduce, and Share Adapted Material for
+               NonCommercial purposes only.
+
+       2. Exceptions and Limitations. For the avoidance of doubt, where
+          Exceptions and Limitations apply to Your use, this Public
+          License does not apply, and You do not need to comply with
+          its terms and conditions.
+
+       3. Term. The term of this Public License is specified in Section
+          6(a).
+
+       4. Media and formats; technical modifications allowed. The
+          Licensor authorizes You to exercise the Licensed Rights in
+          all media and formats whether now known or hereafter created,
+          and to make technical modifications necessary to do so. The
+          Licensor waives and/or agrees not to assert any right or
+          authority to forbid You from making technical modifications
+          necessary to exercise the Licensed Rights, including
+          technical modifications necessary to circumvent Effective
+          Technological Measures. For purposes of this Public License,
+          simply making modifications authorized by this Section 2(a)
+          (4) never produces Adapted Material.
+
+       5. Downstream recipients.
+
+            a. Offer from the Licensor -- Licensed Material. Every
+               recipient of the Licensed Material automatically
+               receives an offer from the Licensor to exercise the
+               Licensed Rights under the terms and conditions of this
+               Public License.
+
+            b. No downstream restrictions. You may not offer or impose
+               any additional or different terms or conditions on, or
+               apply any Effective Technological Measures to, the
+               Licensed Material if doing so restricts exercise of the
+               Licensed Rights by any recipient of the Licensed
+               Material.
+
+       6. No endorsement. Nothing in this Public License constitutes or
+          may be construed as permission to assert or imply that You
+          are, or that Your use of the Licensed Material is, connected
+          with, or sponsored, endorsed, or granted official status by,
+          the Licensor or others designated to receive attribution as
+          provided in Section 3(a)(1)(A)(i).
+
+  b. Other rights.
+
+       1. Moral rights, such as the right of integrity, are not
+          licensed under this Public License, nor are publicity,
+          privacy, and/or other similar personality rights; however, to
+          the extent possible, the Licensor waives and/or agrees not to
+          assert any such rights held by the Licensor to the limited
+          extent necessary to allow You to exercise the Licensed
+          Rights, but not otherwise.
+
+       2. Patent and trademark rights are not licensed under this
+          Public License.
+
+       3. To the extent possible, the Licensor waives any right to
+          collect royalties from You for the exercise of the Licensed
+          Rights, whether directly or through a collecting society
+          under any voluntary or waivable statutory or compulsory
+          licensing scheme. In all other cases the Licensor expressly
+          reserves any right to collect such royalties, including when
+          the Licensed Material is used other than for NonCommercial
+          purposes.
+
+
+Section 3 -- License Conditions.
+
+Your exercise of the Licensed Rights is expressly made subject to the
+following conditions.
+
+  a. Attribution.
+
+       1. If You Share the Licensed Material (including in modified
+          form), You must:
+
+            a. retain the following if it is supplied by the Licensor
+               with the Licensed Material:
+
+                 i. identification of the creator(s) of the Licensed
+                    Material and any others designated to receive
+                    attribution, in any reasonable manner requested by
+                    the Licensor (including by pseudonym if
+                    designated);
+
+                ii. a copyright notice;
+
+               iii. a notice that refers to this Public License;
+
+                iv. a notice that refers to the disclaimer of
+                    warranties;
+
+                 v. a URI or hyperlink to the Licensed Material to the
+                    extent reasonably practicable;
+
+            b. indicate if You modified the Licensed Material and
+               retain an indication of any previous modifications; and
+
+            c. indicate the Licensed Material is licensed under this
+               Public License, and include the text of, or the URI or
+               hyperlink to, this Public License.
+
+       2. You may satisfy the conditions in Section 3(a)(1) in any
+          reasonable manner based on the medium, means, and context in
+          which You Share the Licensed Material. For example, it may be
+          reasonable to satisfy the conditions by providing a URI or
+          hyperlink to a resource that includes the required
+          information.
+
+       3. If requested by the Licensor, You must remove any of the
+          information required by Section 3(a)(1)(A) to the extent
+          reasonably practicable.
+
+       4. If You Share Adapted Material You produce, the Adapter's
+          License You apply must not prevent recipients of the Adapted
+          Material from complying with this Public License.
+
+
+Section 4 -- Sui Generis Database Rights.
+
+Where the Licensed Rights include Sui Generis Database Rights that
+apply to Your use of the Licensed Material:
+
+  a. for the avoidance of doubt, Section 2(a)(1) grants You the right
+     to extract, reuse, reproduce, and Share all or a substantial
+     portion of the contents of the database for NonCommercial purposes
+     only;
+
+  b. if You include all or a substantial portion of the database
+     contents in a database in which You have Sui Generis Database
+     Rights, then the database in which You have Sui Generis Database
+     Rights (but not its individual contents) is Adapted Material; and
+
+  c. You must comply with the conditions in Section 3(a) if You Share
+     all or a substantial portion of the contents of the database.
+
+For the avoidance of doubt, this Section 4 supplements and does not
+replace Your obligations under this Public License where the Licensed
+Rights include other Copyright and Similar Rights.
+
+
+Section 5 -- Disclaimer of Warranties and Limitation of Liability.
+
+  a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
+     EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
+     AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
+     ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
+     IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
+     WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
+     PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
+     ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
+     KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
+     ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
+
+  b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
+     TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
+     NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
+     INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
+     COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
+     USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
+     ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
+     DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
+     IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
+
+  c. The disclaimer of warranties and limitation of liability provided
+     above shall be interpreted in a manner that, to the extent
+     possible, most closely approximates an absolute disclaimer and
+     waiver of all liability.
+
+
+Section 6 -- Term and Termination.
+
+  a. This Public License applies for the term of the Copyright and
+     Similar Rights licensed here. However, if You fail to comply with
+     this Public License, then Your rights under this Public License
+     terminate automatically.
+
+  b. Where Your right to use the Licensed Material has terminated under
+     Section 6(a), it reinstates:
+
+       1. automatically as of the date the violation is cured, provided
+          it is cured within 30 days of Your discovery of the
+          violation; or
+
+       2. upon express reinstatement by the Licensor.
+
+     For the avoidance of doubt, this Section 6(b) does not affect any
+     right the Licensor may have to seek remedies for Your violations
+     of this Public License.
+
+  c. For the avoidance of doubt, the Licensor may also offer the
+     Licensed Material under separate terms or conditions or stop
+     distributing the Licensed Material at any time; however, doing so
+     will not terminate this Public License.
+
+  d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
+     License.
+
+
+Section 7 -- Other Terms and Conditions.
+
+  a. The Licensor shall not be bound by any additional or different
+     terms or conditions communicated by You unless expressly agreed.
+
+  b. Any arrangements, understandings, or agreements regarding the
+     Licensed Material not stated herein are separate from and
+     independent of the terms and conditions of this Public License.
+
+
+Section 8 -- Interpretation.
+
+  a. For the avoidance of doubt, this Public License does not, and
+     shall not be interpreted to, reduce, limit, restrict, or impose
+     conditions on any use of the Licensed Material that could lawfully
+     be made without permission under this Public License.
+
+  b. To the extent possible, if any provision of this Public License is
+     deemed unenforceable, it shall be automatically reformed to the
+     minimum extent necessary to make it enforceable. If the provision
+     cannot be reformed, it shall be severed from this Public License
+     without affecting the enforceability of the remaining terms and
+     conditions.
+
+  c. No term or condition of this Public License will be waived and no
+     failure to comply consented to unless expressly agreed to by the
+     Licensor.
+
+  d. Nothing in this Public License constitutes or may be interpreted
+     as a limitation upon, or waiver of, any privileges and immunities
+     that apply to the Licensor or You, including from the legal
+     processes of any jurisdiction or authority.
+
+=======================================================================
+
+Creative Commons is not a party to its public
+licenses. Notwithstanding, Creative Commons may elect to apply one of
+its public licenses to material it publishes and in those instances
+will be considered the “Licensor.” The text of the Creative Commons
+public licenses is dedicated to the public domain under the CC0 Public
+Domain Dedication. Except for the limited purpose of indicating that
+material is shared under a Creative Commons public license or as
+otherwise permitted by the Creative Commons policies published at
+creativecommons.org/policies, Creative Commons does not authorize the
+use of the trademark "Creative Commons" or any other trademark or logo
+of Creative Commons without its prior written consent including,
+without limitation, in connection with any unauthorized modifications
+to any of its public licenses or any other arrangements,
+understandings, or agreements concerning use of licensed material. For
+the avoidance of doubt, this paragraph does not form part of the
+public licenses.
+
+Creative Commons may be contacted at creativecommons.org.
+