Publish final multi-length LCQD weights and standalone inference

.gitignore

@@ -0,0 +1,4 @@
+__pycache__/
+.venv/
+*.npy
+reconstructions.json

LICENSE

@@ -0,0 +1,190 @@
+EUROPEAN UNION PUBLIC LICENCE v. 1.2
+EUPL © the European Union 2007, 2016
+
+This European Union Public Licence (the ‘EUPL’) applies to the Work (as defined below) which is provided under the
+terms of this Licence. Any use of the Work, other than as authorised under this Licence is prohibited (to the extent such
+use is covered by a right of the copyright holder of the Work).
+The Work is provided under the terms of this Licence when the Licensor (as defined below) has placed the following
+notice immediately following the copyright notice for the Work:
+                          Licensed under the EUPL
+or has expressed by any other means his willingness to license under the EUPL.
+
+1.Definitions
+In this Licence, the following terms have the following meaning:
+— ‘The Licence’:this Licence.
+— ‘The Original Work’:the work or software distributed or communicated by the Licensor under this Licence, available
+as Source Code and also as Executable Code as the case may be.
+— ‘Derivative Works’:the works or software that could be created by the Licensee, based upon the Original Work or
+modifications thereof. This Licence does not define the extent of modification or dependence on the Original Work
+required in order to classify a work as a Derivative Work; this extent is determined by copyright law applicable in
+the country mentioned in Article 15.
+— ‘The Work’:the Original Work or its Derivative Works.
+— ‘The Source Code’:the human-readable form of the Work which is the most convenient for people to study and
+modify.
+— ‘The Executable Code’:any code which has generally been compiled and which is meant to be interpreted by
+a computer as a program.
+— ‘The Licensor’:the natural or legal person that distributes or communicates the Work under the Licence.
+— ‘Contributor(s)’:any natural or legal person who modifies the Work under the Licence, or otherwise contributes to
+the creation of a Derivative Work.
+— ‘The Licensee’ or ‘You’:any natural or legal person who makes any usage of the Work under the terms of the
+Licence.
+— ‘Distribution’ or ‘Communication’:any act of selling, giving, lending, renting, distributing, communicating,
+transmitting, or otherwise making available, online or offline, copies of the Work or providing access to its essential
+functionalities at the disposal of any other natural or legal person.
+
+2.Scope of the rights granted by the Licence
+The Licensor hereby grants You a worldwide, royalty-free, non-exclusive, sublicensable licence to do the following, for
+the duration of copyright vested in the Original Work:
+— use the Work in any circumstance and for all usage,
+— reproduce the Work,
+— modify the Work, and make Derivative Works based upon the Work,
+— communicate to the public, including the right to make available or display the Work or copies thereof to the public
+and perform publicly, as the case may be, the Work,
+— distribute the Work or copies thereof,
+— lend and rent the Work or copies thereof,
+— sublicense rights in the Work or copies thereof.
+Those rights can be exercised on any media, supports and formats, whether now known or later invented, as far as the
+applicable law permits so.
+In the countries where moral rights apply, the Licensor waives his right to exercise his moral right to the extent allowed
+by law in order to make effective the licence of the economic rights here above listed.
+The Licensor grants to the Licensee royalty-free, non-exclusive usage rights to any patents held by the Licensor, to the
+extent necessary to make use of the rights granted on the Work under this Licence.
+
+3.Communication of the Source Code
+The Licensor may provide the Work either in its Source Code form, or as Executable Code. If the Work is provided as
+Executable Code, the Licensor provides in addition a machine-readable copy of the Source Code of the Work along with
+each copy of the Work that the Licensor distributes or indicates, in a notice following the copyright notice attached to
+the Work, a repository where the Source Code is easily and freely accessible for as long as the Licensor continues to
+distribute or communicate the Work.
+
+4.Limitations on copyright
+Nothing in this Licence is intended to deprive the Licensee of the benefits from any exception or limitation to the
+exclusive rights of the rights owners in the Work, of the exhaustion of those rights or of other applicable limitations
+thereto.
+
+5.Obligations of the Licensee
+The grant of the rights mentioned above is subject to some restrictions and obligations imposed on the Licensee. Those
+obligations are the following:
+
+Attribution right: The Licensee shall keep intact all copyright, patent or trademarks notices and all notices that refer to
+the Licence and to the disclaimer of warranties. The Licensee must include a copy of such notices and a copy of the
+Licence with every copy of the Work he/she distributes or communicates. The Licensee must cause any Derivative Work
+to carry prominent notices stating that the Work has been modified and the date of modification.
+
+Copyleft clause: If the Licensee distributes or communicates copies of the Original Works or Derivative Works, this
+Distribution or Communication will be done under the terms of this Licence or of a later version of this Licence unless
+the Original Work is expressly distributed only under this version of the Licence — for example by communicating
+‘EUPL v. 1.2 only’. The Licensee (becoming Licensor) cannot offer or impose any additional terms or conditions on the
+Work or Derivative Work that alter or restrict the terms of the Licence.
+
+Compatibility clause: If the Licensee Distributes or Communicates Derivative Works or copies thereof based upon both
+the Work and another work licensed under a Compatible Licence, this Distribution or Communication can be done
+under the terms of this Compatible Licence. For the sake of this clause, ‘Compatible Licence’ refers to the licences listed
+in the appendix attached to this Licence. Should the Licensee's obligations under the Compatible Licence conflict with
+his/her obligations under this Licence, the obligations of the Compatible Licence shall prevail.
+
+Provision of Source Code: When distributing or communicating copies of the Work, the Licensee will provide
+a machine-readable copy of the Source Code or indicate a repository where this Source will be easily and freely available
+for as long as the Licensee continues to distribute or communicate the Work.
+Legal Protection: This Licence does not grant permission to use the trade names, trademarks, service marks, or names
+of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and
+reproducing the content of the copyright notice.
+
+6.Chain of Authorship
+The original Licensor warrants that the copyright in the Original Work granted hereunder is owned by him/her or
+licensed to him/her and that he/she has the power and authority to grant the Licence.
+Each Contributor warrants that the copyright in the modifications he/she brings to the Work are owned by him/her or
+licensed to him/her and that he/she has the power and authority to grant the Licence.
+Each time You accept the Licence, the original Licensor and subsequent Contributors grant You a licence to their contributions
+to the Work, under the terms of this Licence.
+
+7.Disclaimer of Warranty
+The Work is a work in progress, which is continuously improved by numerous Contributors. It is not a finished work
+and may therefore contain defects or ‘bugs’ inherent to this type of development.
+For the above reason, the Work is provided under the Licence on an ‘as is’ basis and without warranties of any kind
+concerning the Work, including without limitation merchantability, fitness for a particular purpose, absence of defects or
+errors, accuracy, non-infringement of intellectual property rights other than copyright as stated in Article 6 of this
+Licence.
+This disclaimer of warranty is an essential part of the Licence and a condition for the grant of any rights to the Work.
+
+8.Disclaimer of Liability
+Except in the cases of wilful misconduct or damages directly caused to natural persons, the Licensor will in no event be
+liable for any direct or indirect, material or moral, damages of any kind, arising out of the Licence or of the use of the
+Work, including without limitation, damages for loss of goodwill, work stoppage, computer failure or malfunction, loss
+of data or any commercial damage, even if the Licensor has been advised of the possibility of such damage. However,
+the Licensor will be liable under statutory product liability laws as far such laws apply to the Work.
+
+9.Additional agreements
+While distributing the Work, You may choose to conclude an additional agreement, defining obligations or services
+consistent with this Licence. However, if accepting obligations, You may act only on your own behalf and on your sole
+responsibility, not on behalf of the original Licensor or any other Contributor, and only if You agree to indemnify,
+defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against such Contributor by
+the fact You have accepted any warranty or additional liability.
+
+10.Acceptance of the Licence
+The provisions of this Licence can be accepted by clicking on an icon ‘I agree’ placed under the bottom of a window
+displaying the text of this Licence or by affirming consent in any other similar way, in accordance with the rules of
+applicable law. Clicking on that icon indicates your clear and irrevocable acceptance of this Licence and all of its terms
+and conditions.
+Similarly, you irrevocably accept this Licence and all of its terms and conditions by exercising any rights granted to You
+by Article 2 of this Licence, such as the use of the Work, the creation by You of a Derivative Work or the Distribution
+or Communication by You of the Work or copies thereof.
+
+11.Information to the public
+In case of any Distribution or Communication of the Work by means of electronic communication by You (for example,
+by offering to download the Work from a remote location) the distribution channel or media (for example, a website)
+must at least provide to the public the information requested by the applicable law regarding the Licensor, the Licence
+and the way it may be accessible, concluded, stored and reproduced by the Licensee.
+
+12.Termination of the Licence
+The Licence and the rights granted hereunder will terminate automatically upon any breach by the Licensee of the terms
+of the Licence.
+Such a termination will not terminate the licences of any person who has received the Work from the Licensee under
+the Licence, provided such persons remain in full compliance with the Licence.
+
+13.Miscellaneous
+Without prejudice of Article 9 above, the Licence represents the complete agreement between the Parties as to the
+Work.
+If any provision of the Licence is invalid or unenforceable under applicable law, this will not affect the validity or
+enforceability of the Licence as a whole. Such provision will be construed or reformed so as necessary to make it valid
+and enforceable.
+The European Commission may publish other linguistic versions or new versions of this Licence or updated versions of
+the Appendix, so far this is required and reasonable, without reducing the scope of the rights granted by the Licence.
+New versions of the Licence will be published with a unique version number.
+All linguistic versions of this Licence, approved by the European Commission, have identical value. Parties can take
+advantage of the linguistic version of their choice.
+
+14.Jurisdiction
+Without prejudice to specific agreement between parties,
+— any litigation resulting from the interpretation of this License, arising between the European Union institutions,
+bodies, offices or agencies, as a Licensor, and any Licensee, will be subject to the jurisdiction of the Court of Justice
+of the European Union, as laid down in article 272 of the Treaty on the Functioning of the European Union,
+— any litigation arising between other parties and resulting from the interpretation of this License, will be subject to
+the exclusive jurisdiction of the competent court where the Licensor resides or conducts its primary business.
+
+15.Applicable Law
+Without prejudice to specific agreement between parties,
+— this Licence shall be governed by the law of the European Union Member State where the Licensor has his seat,
+resides or has his registered office,
+— this licence shall be governed by Belgian law if the Licensor has no seat, residence or registered office inside
+a European Union Member State.
+
+
+                                                         Appendix
+
+‘Compatible Licences’ according to Article 5 EUPL are:
+— GNU General Public License (GPL) v. 2, v. 3
+— GNU Affero General Public License (AGPL) v. 3
+— Open Software License (OSL) v. 2.1, v. 3.0
+— Eclipse Public License (EPL) v. 1.0
+— CeCILL v. 2.0, v. 2.1
+— Mozilla Public Licence (MPL) v. 2
+— GNU Lesser General Public Licence (LGPL) v. 2.1, v. 3
+— Creative Commons Attribution-ShareAlike v. 3.0 Unported (CC BY-SA 3.0) for works other than software
+— European Union Public Licence (EUPL) v. 1.1, v. 1.2
+— Québec Free and Open-Source Licence — Reciprocity (LiLiQ-R) or Strong Reciprocity (LiLiQ-R+).
+
+The European Commission may update this Appendix to later versions of the above licences without producing
+a new version of the EUPL, as long as they provide the rights granted in Article 2 of this Licence and protect the
+covered Source Code from exclusive appropriation.
+All other changes or additions to this Appendix require the production of a new EUPL version.

README.md

@@ -1,3 +1,58 @@
 ---
+language:
+- en
 license: eupl-1.2
+tags:
+- genomics
+- dna
+- model-inversion
+- privacy
+- pytorch
 ---
+
+# DNA Embedding Inversion
+
+Three final LCQD decoders from [How Private Are DNA Embeddings?](https://arxiv.org/abs/2603.06950), reconstructing DNA from mean-pooled foundation-model embeddings.
+
+**[Code, embedding extraction and results](https://github.com/not-a-feature/DNA-Embedding-Inversion)**
+
+| Model | Input dimensions | Extraction layer |
+| --- | ---: | --- |
+| `dnabert2` | 768 | Final hidden layer (12) |
+| `evo2` | 4096 | `blocks.26.mlp.l3` (zero-based) |
+| `ntv2` | 1024 | Final hidden layer (29) |
+
+Each decoder covers 14 lengths from 10 to 100 nucleotides. These are the main-study checkpoints, trained on coordinate- and exact-sequence-disjoint hg38 data for 20 epochs, retaining the best validation checkpoint.
+
+## Quick start
+
+Python 3.12; CPU or CUDA. Download this repository with Git LFS:
+
+```bash
+git lfs install
+git clone https://huggingface.co/not-a-feature/DNA_Embedding_Invesion
+cd DNA_Embedding_Invesion
+pip install -r requirements.txt
+python infer.py --model ntv2 --embeddings embeddings.npy --output reconstructions.json
+```
+
+`embeddings.npy` must contain a finite float array shaped `[N, input_dimensions]` of **raw, unnormalized mean-pooled embeddings**, extracted using the matching model and recipe linked above. The script applies the bundled exact training normalization and returns a JSON list of DNA strings. Add `--device cuda` for GPU inference.
+
+Python API:
+
+```python
+import numpy as np
+from infer import Decoder
+
+sequences = Decoder("ntv2").predict(np.load("embeddings.npy", allow_pickle=False))
+```
+
+No original sequence length is required. Decoding stops at the first predicted EOS; predictions are not cropped to a requested length. Tokenizers are bundled, so inference works offline after installation and download. Foundation-model weights and embedding extraction are separate.
+
+## Validation and scope
+
+CPU predictions matched the original evaluation code on 252 embeddings: three per length, per model, for hg38 and 1000 Genomes. CLI execution, EOS handling and invalid-input checks also passed. This verifies inference parity, not a new accuracy estimate; see `validation.json`.
+
+Performance depends on sequence length and embedding source. Evaluation assumes reference orientation. These checkpoints are intended for research on genomic embedding privacy.
+
+Weights and decoder code: **EUPL-1.2**. Bundled tokenizers originate from [DNABERT-2](https://huggingface.co/zhihan1996/DNABERT-2-117M) and [NTv2](https://huggingface.co/InstaDeepAI/nucleotide-transformer-v2-500m-multi-species); their upstream terms apply. Configurations, normalization provenance and SHA-256 checksums are included.

SHA256SUMS

@@ -0,0 +1,31 @@
+c0aaf72a87ffca0929550674551ec1b14105d51d63bffe43d290cff9668d141a  .gitignore
+ab1fbd17c5436b98163dc36d1f08e98c0ca9d1ebb52d6a1fa506cf8c3b2e04f7  dnabert2/config.yaml
+9acc1aea844385c6fa826693de19ef6fc01713bc6aea2dba8091abdb544dca4d  dnabert2/model.pt
+835c977f1782ac5b3cc7f44b6eb1cf63e2aae576f83ccd0f76b23fbf156fdc85  dnabert2/model.py
+13a050e2a6613988c815aba1ca21b0dd170171d45b14bf0668a9a8ab508f0f78  dnabert2/normalization.json
+f4eb7994106bd5a47ad72a79a2d75e8bcb35a8e0bf85a320ecfe8506c3c60b6a  dnabert2/results.json
+5aa43c2f985a25296d5a5ce621c2f77376ca8091a47993c492fd5460b895b140  dnabert2/tokenizer/special_tokens_map.json
+65ccd60e5439e86fc5e35ce1e0aded941dc34b8dcdfc55fe1bc9758e0576309e  dnabert2/tokenizer/tokenizer.json
+ee15ea4a12fa1233855109ec69981b60f231274e5167f878a543037cb107207b  dnabert2/tokenizer/tokenizer_config.json
+b88ff00646580370e6a9dea75b0c92ff9ee12e297f1ac1af7408135b62b741ce  dnabert2/tokenizer_info.json
+f801655e776458a5e4d507162efa8fed972741129c30e5aaa62f9d728277dbbb  evo2/config.yaml
+376094ad3cd2543701d43aab3ea71b34c2a3b7bf97299cf9666d2eb7a5dd1184  evo2/model.pt
+835c977f1782ac5b3cc7f44b6eb1cf63e2aae576f83ccd0f76b23fbf156fdc85  evo2/model.py
+5c150d1b4fc297fe4c278b57df8801762af5822cdc3f929602c2915d4e5a7fa3  evo2/normalization.json
+9526eb324d98ec3ea99bccae1eb7abe2940a54c315819cfcfa3b7842374e5776  evo2/results.json
+eedecf8870ded24a7d4bd543cb73d09bae00a232a5b2ff7eefde9058b928cc01  infer.py
+57fb42fbcd0b037ce528ed8f72f1ec095d67bc6825ecf1448ff39be1fe68a4b4  LICENSE
+835c977f1782ac5b3cc7f44b6eb1cf63e2aae576f83ccd0f76b23fbf156fdc85  model.py
+ee5c8e142ab4176989555a6495b67d163c3bdc6c78f2fbced360b000634a80d5  ntv2/config.yaml
+5300eea166e2b31aa5a7e0e90d1bb2d859d86de94fe185371b142041899c9539  ntv2/model.pt
+835c977f1782ac5b3cc7f44b6eb1cf63e2aae576f83ccd0f76b23fbf156fdc85  ntv2/model.py
+d17e6a4683a2f41beaee20879e0551541009c813ccc19313ffa84b06d021d9a4  ntv2/normalization.json
+a9697d27b2180ce973d467d1b2a6d636ff7de3f3155b4967ed06a030877e4781  ntv2/results.json
+ac19b6355a6c4ecfde185acb4c8760e676eae583ef9aeae2bcfda5bfba0f344d  ntv2/tokenizer/special_tokens_map.json
+e2811758b0ea7f30168136550e730900e8cc33b70e96726d7ae68fc6c9b5a2e5  ntv2/tokenizer/tokenizer_config.json
+8a569be8b916ce2d19d0a0ee1f769b33a316d69d75c1da97294518f3401dcde9  ntv2/tokenizer/vocab.txt
+6a89fc015bdf56884e95922a41f0dc774c962df4224f77f39ddb9b662164a7cb  ntv2/tokenizer_info.json
+b99d56db90e3cbdbe388ca2acaa340ce4cb5d6ec20f744d77428d9a262fcaeb5  provenance.json
+190a2196c964772df38068c1b14379c7471406261cec288470d5a8c46e56bb4e  README.md
+118037f1a5cf9fe0b5fb36d747f1e45226cadb582257d608530679ab0746030a  requirements.txt
+7570923e4785dd94be9e0393f7aea985cc2f6d44a9cc23044d205ebb86f04bc4  validation.json

dnabert2/config.yaml

@@ -0,0 +1,138 @@
+data:
+  shuffle: true
+  mean: true
+  multi: true
+  require_sequence_disjoint: true
+  source_schema: dna_inversion_hg38_multilen_v2
+  source_corpus: data/hg38_multilen_seqdisjoint/hg38_multilen_seqdisjoint.h5
+  ood_source_corpus: data/1000g_multilen_seqdisjoint/1000g_multilen_seqdisjoint.h5
+  artifact_dir: data/hg38_multilen_seqdisjoint
+  fm_slug: dnabert2
+  seq_lengths:
+  - 10
+  - 15
+  - 20
+  - 25
+  - 30
+  - 35
+  - 40
+  - 45
+  - 50
+  - 60
+  - 70
+  - 80
+  - 90
+  - 100
+  train_seq_lengths:
+  - 10
+  - 15
+  - 20
+  - 25
+  - 30
+  - 35
+  - 40
+  - 45
+  - 50
+  - 60
+  - 70
+  - 80
+  - 90
+  - 100
+  train_csvs:
+  - ${data.artifact_dir}/train_${data.fm_slug}_10_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_15_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_20_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_25_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_30_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_35_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_40_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_45_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_50_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_60_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_70_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_80_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_90_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_100_hg38_multilen_seqdisjoint.h5
+  val_csvs:
+  - ${data.artifact_dir}/val_${data.fm_slug}_10_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_15_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_20_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_25_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_30_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_35_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_40_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_45_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_50_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_60_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_70_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_80_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_90_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_100_hg38_multilen_seqdisjoint.h5
+  test_csvs:
+  - ${data.artifact_dir}/test_${data.fm_slug}_10_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_15_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_20_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_25_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_30_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_35_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_40_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_45_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_50_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_60_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_70_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_80_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_90_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_100_hg38_multilen_seqdisjoint.h5
+  seq_length: 100
+  skip_sha256_check: true
+  normalization_method: standard
+  subset_fraction: null
+  embedding_dim: 768
+  tokenizer:
+    type: huggingface
+    model_name: zhihan1996/DNABERT-2-117M
+  checkpoint: zhihan1996/DNABERT-2-117M
+model:
+  model_file_name: model/query_decoder.py
+  model_name: query_decoder
+  model_type: query_decoder
+  mode: mean
+  d_model: 384
+  nhead: 8
+  num_layers: 6
+  dim_feedforward: 1536
+  dropout: 0.1
+  n_context_tokens: 16
+  aux_length_loss_weight: 0.5
+  output_dim: 4
+  hidden_dims: []
+optim:
+  lr: 0.001
+  weight_decay: 0.0001
+  eps: 1.0e-06
+  epochs: 20
+  batch_size: 32
+  num_workers: 16
+  grad_clip_norm: 1.0
+  use_scheduler: true
+  scheduler:
+    max_lr: 0.0005
+    pct_start: 0.3
+    anneal_strategy: cos
+    div_factor: 25.0
+    final_div_factor: 10000.0
+train:
+  seed: 42
+  project: dna-embedding-inversion
+  run_name: null
+  use_wandb: false
+  log_interval: 10000
+  device: cuda:0
+  early_stopping:
+    enabled: false
+    patience: 5
+    min_delta: 1.0e-05
+  max_val_samples: null
+  nan_backoff:
+    skip_frac_threshold: 0.02
+    lr_backoff: 0.5

dnabert2/model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9acc1aea844385c6fa826693de19ef6fc01713bc6aea2dba8091abdb544dca4d
+size 83363931

dnabert2/model.py

@@ -0,0 +1,157 @@
+"""Query-based decoder for inverting mean-pooled DNA embeddings.
+
+Designed for the multi-length inversion-attack setting: a single fixed-size
+mean embedding must be decoded into a variable-length nucleotide sequence
+whose length itself is unknown.
+
+Architecture (mean mode only):
+
+1. ``context_projection`` maps the mean embedding to ``n_context_tokens``
+   learnable summary tokens of size ``d_model``. These are the "memory" that
+   cross-attention attends to.
+2. ``length_head`` predicts a distribution over possible content lengths
+   (0..seq_length-1) directly from the mean embedding.
+3. The soft length distribution is mixed with ``length_embedding`` to produce
+   a single conditioning vector that is added to every position query. This
+   makes length prediction a first-class input to content generation, not a
+   side branch: gradients flow from the sequence loss back through the length
+   head, and the decoder is trained to produce EOS at the predicted position.
+4. ``pos_queries`` are learnable per-output-slot embeddings. Combined with the
+   length conditioning, they form the decoder inputs.
+5. A stack of ``TransformerDecoderLayer`` blocks performs cross-attention
+   (queries -> context) and self-attention (queries <-> queries).
+6. ``output_projection`` produces per-position logits over the vocabulary
+   (which includes the EOS symbol).
+
+Forward returns a tuple ``(seq_logits, length_logits)``. The training loop
+combines the sequence loss with a cross-entropy loss on the length head
+(weighted by ``aux_length_loss_weight``). At inference the length head's
+output is consumed only internally and is not exposed.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import List, Optional, Tuple
+
+import torch
+import torch.nn.functional as F
+from torch import nn
+
+
+class QueryDecoderReconstructor(nn.Module):
+    def __init__(
+        self,
+        input_dim: int,
+        mode: str,
+        seq_length: int,
+        output_dim: int,
+        d_model: int = 256,
+        nhead: int = 8,
+        num_layers: int = 4,
+        dim_feedforward: int = 1024,
+        dropout: float = 0.1,
+        n_context_tokens: int = 8,
+        aux_length_loss_weight: float = 0.5,
+        hidden_dims: Optional[List[int]] = None,
+    ):
+        super().__init__()
+        assert input_dim > 0, "input_dim must be positive"
+        assert mode == "mean", "QueryDecoderReconstructor supports mean mode only"
+        assert seq_length > 0, "seq_length must be positive"
+        assert output_dim > 0, "output_dim must be positive"
+        assert d_model > 0, "d_model must be positive"
+        assert nhead > 0 and d_model % nhead == 0, "nhead must divide d_model"
+        assert num_layers > 0, "num_layers must be positive"
+        assert dim_feedforward > 0, "dim_feedforward must be positive"
+        assert 0.0 <= dropout <= 1.0, "dropout in [0, 1]"
+        assert n_context_tokens > 0, "n_context_tokens must be positive"
+        assert aux_length_loss_weight >= 0.0, "aux_length_loss_weight must be non-negative"
+
+        self.input_dim = input_dim
+        self.mode = mode
+        self.seq_length = seq_length
+        self.output_dim = output_dim
+        self.d_model = d_model
+        self.n_context_tokens = n_context_tokens
+        self.aux_length_loss_weight = aux_length_loss_weight
+
+        logger = logging.getLogger(__name__)
+        logger.info(
+            f"QueryDecoder: input_dim={input_dim} -> {n_context_tokens} context tokens "
+            f"x d_model={d_model}, seq_length={seq_length} (incl. EOS slot), "
+            f"output_dim={output_dim}, num_layers={num_layers}, nhead={nhead}, "
+            f"aux_length_loss_weight={aux_length_loss_weight}"
+        )
+
+        self.context_projection = nn.Linear(input_dim, n_context_tokens * d_model)
+
+        self.length_head = nn.Sequential(
+            nn.Linear(input_dim, d_model),
+            nn.GELU(),
+            nn.Dropout(dropout),
+            nn.Linear(d_model, seq_length),
+        )
+
+        self.length_embedding = nn.Embedding(seq_length, d_model)
+        self.pos_queries = nn.Embedding(seq_length, d_model)
+
+        decoder_layer = nn.TransformerDecoderLayer(
+            d_model=d_model,
+            nhead=nhead,
+            dim_feedforward=dim_feedforward,
+            dropout=dropout,
+            activation="gelu",
+            batch_first=True,
+            norm_first=True,
+        )
+        self.transformer_decoder = nn.TransformerDecoder(decoder_layer, num_layers=num_layers)
+
+        self.output_projection = nn.Linear(d_model, output_dim)
+        nn.init.xavier_uniform_(self.output_projection.weight, gain=0.02)
+        nn.init.zeros_(self.output_projection.bias)
+
+    def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
+        """Run the model.
+
+        Parameters
+        ----------
+        x : torch.Tensor
+            Mean-pooled embeddings, shape (B, input_dim).
+
+        Returns
+        -------
+        seq_logits : torch.Tensor
+            Per-position vocabulary logits, shape (B, seq_length, output_dim).
+        length_logits : torch.Tensor
+            Per-sample length logits, shape (B, seq_length). The class index
+            equals the number of content tokens before the EOS slot.
+        """
+        assert x.ndim == 2, f"Expected 2D mean embedding, got {x.ndim}D"
+        B, D = x.shape
+        assert D == self.input_dim, f"Expected input_dim={self.input_dim}, got {D}"
+
+        context = self.context_projection(x).view(B, self.n_context_tokens, self.d_model)
+
+        length_logits = self.length_head(x)
+        length_probs = F.softmax(length_logits, dim=-1)
+        length_emb = length_probs @ self.length_embedding.weight
+
+        queries = self.pos_queries.weight.unsqueeze(0).expand(B, -1, -1)
+        queries = queries + length_emb.unsqueeze(1)
+
+        hidden = self.transformer_decoder(tgt=queries, memory=context)
+        seq_logits = self.output_projection(hidden)
+
+        return seq_logits, length_logits
+
+    @staticmethod
+    def length_targets_from_padded(batch_sequence: torch.Tensor) -> torch.Tensor:
+        """Derive integer length targets from collated padded sequences.
+
+        Each padded row is ``[content..., EOS, -100, -100, ...]``. The target
+        we want the length head to predict is the count of content tokens
+        before EOS, i.e. (non-pad count) - 1, clamped at 0 for the empty case.
+        """
+        non_pad = (batch_sequence != -100).sum(dim=1)
+        return (non_pad - 1).clamp(min=0)

dnabert2/normalization.json

@@ -0,0 +1,93 @@
+{
+  "method": "standard",
+  "min": -16.201284408569336,
+  "max": 7.310595512390137,
+  "mean": 0.0032632164329751325,
+  "std": 0.16197158345406573,
+  "sources": [
+    {
+      "file": "train_dnabert2_10_hg38_multilen_seqdisjoint.h5",
+      "count": 900000,
+      "mean": 0.00239973247900342,
+      "std": 0.21060259381722737
+    },
+    {
+      "file": "train_dnabert2_15_hg38_multilen_seqdisjoint.h5",
+      "count": 900000,
+      "mean": 0.002497268046910832,
+      "std": 0.20280150816866813
+    },
+    {
+      "file": "train_dnabert2_20_hg38_multilen_seqdisjoint.h5",
+      "count": 900000,
+      "mean": 0.002773968709057013,
+      "std": 0.19523485245570754
+    },
+    {
+      "file": "train_dnabert2_25_hg38_multilen_seqdisjoint.h5",
+      "count": 1900000,
+      "mean": 0.003002212435469023,
+      "std": 0.18876066757763907
+    },
+    {
+      "file": "train_dnabert2_30_hg38_multilen_seqdisjoint.h5",
+      "count": 1900000,
+      "mean": 0.003176865754911225,
+      "std": 0.182950583035848
+    },
+    {
+      "file": "train_dnabert2_35_hg38_multilen_seqdisjoint.h5",
+      "count": 1900000,
+      "mean": 0.003303331541182128,
+      "std": 0.1775951923943558
+    },
+    {
+      "file": "train_dnabert2_40_hg38_multilen_seqdisjoint.h5",
+      "count": 1900000,
+      "mean": 0.0033823620289145056,
+      "std": 0.17264760355880218
+    },
+    {
+      "file": "train_dnabert2_45_hg38_multilen_seqdisjoint.h5",
+      "count": 1900000,
+      "mean": 0.003434005258607913,
+      "std": 0.16802196844037123
+    },
+    {
+      "file": "train_dnabert2_50_hg38_multilen_seqdisjoint.h5",
+      "count": 2900000,
+      "mean": 0.003454708779214811,
+      "std": 0.163784055829794
+    },
+    {
+      "file": "train_dnabert2_60_hg38_multilen_seqdisjoint.h5",
+      "count": 2900000,
+      "mean": 0.003457340657103643,
+      "std": 0.15632788640954695
+    },
+    {
+      "file": "train_dnabert2_70_hg38_multilen_seqdisjoint.h5",
+      "count": 2900000,
+      "mean": 0.0034274668118673413,
+      "std": 0.1501236633150605
+    },
+    {
+      "file": "train_dnabert2_80_hg38_multilen_seqdisjoint.h5",
+      "count": 2900000,
+      "mean": 0.003368306485719288,
+      "std": 0.14378749876791175
+    },
+    {
+      "file": "train_dnabert2_90_hg38_multilen_seqdisjoint.h5",
+      "count": 3900000,
+      "mean": 0.0033015144398278622,
+      "std": 0.1372347623667584
+    },
+    {
+      "file": "train_dnabert2_100_hg38_multilen_seqdisjoint.h5",
+      "count": 3900000,
+      "mean": 0.003235255997596293,
+      "std": 0.1328747036942934
+    }
+  ]
+}

dnabert2/results.json

@@ -0,0 +1,5 @@
+{
+    "best_val_loss": 4.0402679175894605,
+    "stopped_early": false,
+    "final_epoch": 20
+}

dnabert2/tokenizer/special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

dnabert2/tokenizer/tokenizer_config.json

@@ -0,0 +1,52 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "[UNK]"
+}

dnabert2/tokenizer_info.json

@@ -0,0 +1,5 @@
+{
+  "source": "zhihan1996/DNABERT-2-117M",
+  "eos_id": 2,
+  "vocab_sha256": "5e28a2ce78c21096fd494f4105edf17f52b27cca9da573300352d3de19d94406"
+}

evo2/config.yaml

@@ -0,0 +1,137 @@
+data:
+  shuffle: true
+  mean: true
+  multi: true
+  require_sequence_disjoint: true
+  source_schema: dna_inversion_hg38_multilen_v2
+  source_corpus: data/hg38_multilen_seqdisjoint/hg38_multilen_seqdisjoint.h5
+  ood_source_corpus: data/1000g_multilen_seqdisjoint/1000g_multilen_seqdisjoint.h5
+  artifact_dir: data/hg38_multilen_seqdisjoint
+  fm_slug: evo2
+  seq_lengths:
+  - 10
+  - 15
+  - 20
+  - 25
+  - 30
+  - 35
+  - 40
+  - 45
+  - 50
+  - 60
+  - 70
+  - 80
+  - 90
+  - 100
+  train_seq_lengths:
+  - 10
+  - 15
+  - 20
+  - 25
+  - 30
+  - 35
+  - 40
+  - 45
+  - 50
+  - 60
+  - 70
+  - 80
+  - 90
+  - 100
+  train_csvs:
+  - ${data.artifact_dir}/train_${data.fm_slug}_10_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_15_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_20_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_25_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_30_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_35_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_40_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_45_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_50_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_60_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_70_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_80_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_90_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_100_hg38_multilen_seqdisjoint.h5
+  val_csvs:
+  - ${data.artifact_dir}/val_${data.fm_slug}_10_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_15_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_20_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_25_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_30_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_35_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_40_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_45_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_50_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_60_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_70_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_80_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_90_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_100_hg38_multilen_seqdisjoint.h5
+  test_csvs:
+  - ${data.artifact_dir}/test_${data.fm_slug}_10_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_15_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_20_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_25_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_30_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_35_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_40_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_45_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_50_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_60_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_70_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_80_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_90_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_100_hg38_multilen_seqdisjoint.h5
+  seq_length: 100
+  skip_sha256_check: true
+  normalization_method: standard
+  subset_fraction: null
+  embedding_dim: 4096
+  tokenizer:
+    type: char
+  checkpoint: evo2_7b
+model:
+  model_file_name: model/query_decoder.py
+  model_name: query_decoder
+  model_type: query_decoder
+  mode: mean
+  d_model: 384
+  nhead: 8
+  num_layers: 6
+  dim_feedforward: 1536
+  dropout: 0.1
+  n_context_tokens: 16
+  aux_length_loss_weight: 0.5
+  output_dim: 4
+  hidden_dims: []
+optim:
+  lr: 0.001
+  weight_decay: 0.0001
+  eps: 1.0e-06
+  epochs: 20
+  batch_size: 32
+  num_workers: 16
+  grad_clip_norm: 1.0
+  use_scheduler: true
+  scheduler:
+    max_lr: 0.0005
+    pct_start: 0.3
+    anneal_strategy: cos
+    div_factor: 25.0
+    final_div_factor: 10000.0
+train:
+  seed: 42
+  project: dna-embedding-inversion
+  run_name: null
+  use_wandb: false
+  log_interval: 10000
+  device: cuda:0
+  early_stopping:
+    enabled: false
+    patience: 5
+    min_delta: 1.0e-05
+  max_val_samples: null
+  nan_backoff:
+    skip_frac_threshold: 0.02
+    lr_backoff: 0.5

evo2/model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:376094ad3cd2543701d43aab3ea71b34c2a3b7bf97299cf9666d2eb7a5dd1184
+size 164296411

evo2/model.py

@@ -0,0 +1,157 @@
+"""Query-based decoder for inverting mean-pooled DNA embeddings.
+
+Designed for the multi-length inversion-attack setting: a single fixed-size
+mean embedding must be decoded into a variable-length nucleotide sequence
+whose length itself is unknown.
+
+Architecture (mean mode only):
+
+1. ``context_projection`` maps the mean embedding to ``n_context_tokens``
+   learnable summary tokens of size ``d_model``. These are the "memory" that
+   cross-attention attends to.
+2. ``length_head`` predicts a distribution over possible content lengths
+   (0..seq_length-1) directly from the mean embedding.
+3. The soft length distribution is mixed with ``length_embedding`` to produce
+   a single conditioning vector that is added to every position query. This
+   makes length prediction a first-class input to content generation, not a
+   side branch: gradients flow from the sequence loss back through the length
+   head, and the decoder is trained to produce EOS at the predicted position.
+4. ``pos_queries`` are learnable per-output-slot embeddings. Combined with the
+   length conditioning, they form the decoder inputs.
+5. A stack of ``TransformerDecoderLayer`` blocks performs cross-attention
+   (queries -> context) and self-attention (queries <-> queries).
+6. ``output_projection`` produces per-position logits over the vocabulary
+   (which includes the EOS symbol).
+
+Forward returns a tuple ``(seq_logits, length_logits)``. The training loop
+combines the sequence loss with a cross-entropy loss on the length head
+(weighted by ``aux_length_loss_weight``). At inference the length head's
+output is consumed only internally and is not exposed.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import List, Optional, Tuple
+
+import torch
+import torch.nn.functional as F
+from torch import nn
+
+
+class QueryDecoderReconstructor(nn.Module):
+    def __init__(
+        self,
+        input_dim: int,
+        mode: str,
+        seq_length: int,
+        output_dim: int,
+        d_model: int = 256,
+        nhead: int = 8,
+        num_layers: int = 4,
+        dim_feedforward: int = 1024,
+        dropout: float = 0.1,
+        n_context_tokens: int = 8,
+        aux_length_loss_weight: float = 0.5,
+        hidden_dims: Optional[List[int]] = None,
+    ):
+        super().__init__()
+        assert input_dim > 0, "input_dim must be positive"
+        assert mode == "mean", "QueryDecoderReconstructor supports mean mode only"
+        assert seq_length > 0, "seq_length must be positive"
+        assert output_dim > 0, "output_dim must be positive"
+        assert d_model > 0, "d_model must be positive"
+        assert nhead > 0 and d_model % nhead == 0, "nhead must divide d_model"
+        assert num_layers > 0, "num_layers must be positive"
+        assert dim_feedforward > 0, "dim_feedforward must be positive"
+        assert 0.0 <= dropout <= 1.0, "dropout in [0, 1]"
+        assert n_context_tokens > 0, "n_context_tokens must be positive"
+        assert aux_length_loss_weight >= 0.0, "aux_length_loss_weight must be non-negative"
+
+        self.input_dim = input_dim
+        self.mode = mode
+        self.seq_length = seq_length
+        self.output_dim = output_dim
+        self.d_model = d_model
+        self.n_context_tokens = n_context_tokens
+        self.aux_length_loss_weight = aux_length_loss_weight
+
+        logger = logging.getLogger(__name__)
+        logger.info(
+            f"QueryDecoder: input_dim={input_dim} -> {n_context_tokens} context tokens "
+            f"x d_model={d_model}, seq_length={seq_length} (incl. EOS slot), "
+            f"output_dim={output_dim}, num_layers={num_layers}, nhead={nhead}, "
+            f"aux_length_loss_weight={aux_length_loss_weight}"
+        )
+
+        self.context_projection = nn.Linear(input_dim, n_context_tokens * d_model)
+
+        self.length_head = nn.Sequential(
+            nn.Linear(input_dim, d_model),
+            nn.GELU(),
+            nn.Dropout(dropout),
+            nn.Linear(d_model, seq_length),
+        )
+
+        self.length_embedding = nn.Embedding(seq_length, d_model)
+        self.pos_queries = nn.Embedding(seq_length, d_model)
+
+        decoder_layer = nn.TransformerDecoderLayer(
+            d_model=d_model,
+            nhead=nhead,
+            dim_feedforward=dim_feedforward,
+            dropout=dropout,
+            activation="gelu",
+            batch_first=True,
+            norm_first=True,
+        )
+        self.transformer_decoder = nn.TransformerDecoder(decoder_layer, num_layers=num_layers)
+
+        self.output_projection = nn.Linear(d_model, output_dim)
+        nn.init.xavier_uniform_(self.output_projection.weight, gain=0.02)
+        nn.init.zeros_(self.output_projection.bias)
+
+    def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
+        """Run the model.
+
+        Parameters
+        ----------
+        x : torch.Tensor
+            Mean-pooled embeddings, shape (B, input_dim).
+
+        Returns
+        -------
+        seq_logits : torch.Tensor
+            Per-position vocabulary logits, shape (B, seq_length, output_dim).
+        length_logits : torch.Tensor
+            Per-sample length logits, shape (B, seq_length). The class index
+            equals the number of content tokens before the EOS slot.
+        """
+        assert x.ndim == 2, f"Expected 2D mean embedding, got {x.ndim}D"
+        B, D = x.shape
+        assert D == self.input_dim, f"Expected input_dim={self.input_dim}, got {D}"
+
+        context = self.context_projection(x).view(B, self.n_context_tokens, self.d_model)
+
+        length_logits = self.length_head(x)
+        length_probs = F.softmax(length_logits, dim=-1)
+        length_emb = length_probs @ self.length_embedding.weight
+
+        queries = self.pos_queries.weight.unsqueeze(0).expand(B, -1, -1)
+        queries = queries + length_emb.unsqueeze(1)
+
+        hidden = self.transformer_decoder(tgt=queries, memory=context)
+        seq_logits = self.output_projection(hidden)
+
+        return seq_logits, length_logits
+
+    @staticmethod
+    def length_targets_from_padded(batch_sequence: torch.Tensor) -> torch.Tensor:
+        """Derive integer length targets from collated padded sequences.
+
+        Each padded row is ``[content..., EOS, -100, -100, ...]``. The target
+        we want the length head to predict is the count of content tokens
+        before EOS, i.e. (non-pad count) - 1, clamped at 0 for the empty case.
+        """
+        non_pad = (batch_sequence != -100).sum(dim=1)
+        return (non_pad - 1).clamp(min=0)

evo2/normalization.json

@@ -0,0 +1,93 @@
+{
+  "method": "standard",
+  "min": -2.3405566215515137,
+  "max": 2.4061572551727295,
+  "mean": -0.0003059846802800181,
+  "std": 0.051806185933698734,
+  "sources": [
+    {
+      "file": "train_evo2_10_hg38_multilen_seqdisjoint.h5",
+      "count": 900000,
+      "mean": -5.516804802897521e-08,
+      "std": 0.01792347566622959
+    },
+    {
+      "file": "train_evo2_15_hg38_multilen_seqdisjoint.h5",
+      "count": 900000,
+      "mean": 0.00023065107590612198,
+      "std": 0.03920898920084131
+    },
+    {
+      "file": "train_evo2_20_hg38_multilen_seqdisjoint.h5",
+      "count": 900000,
+      "mean": 0.0002838107374154947,
+      "std": 0.04822304791228452
+    },
+    {
+      "file": "train_evo2_25_hg38_multilen_seqdisjoint.h5",
+      "count": 1900000,
+      "mean": 8.711961858466201e-05,
+      "std": 0.047605206711041334
+    },
+    {
+      "file": "train_evo2_30_hg38_multilen_seqdisjoint.h5",
+      "count": 1900000,
+      "mean": -7.257468528528559e-05,
+      "std": 0.04813724988320282
+    },
+    {
+      "file": "train_evo2_35_hg38_multilen_seqdisjoint.h5",
+      "count": 1900000,
+      "mean": -0.00018403705609403584,
+      "std": 0.049031531658874396
+    },
+    {
+      "file": "train_evo2_40_hg38_multilen_seqdisjoint.h5",
+      "count": 1900000,
+      "mean": -0.000263046290208472,
+      "std": 0.04994715030965056
+    },
+    {
+      "file": "train_evo2_45_hg38_multilen_seqdisjoint.h5",
+      "count": 1900000,
+      "mean": -0.0003212788471664065,
+      "std": 0.05080656099184255
+    },
+    {
+      "file": "train_evo2_50_hg38_multilen_seqdisjoint.h5",
+      "count": 2900000,
+      "mean": -0.00036483371965520105,
+      "std": 0.05161175943642153
+    },
+    {
+      "file": "train_evo2_60_hg38_multilen_seqdisjoint.h5",
+      "count": 2900000,
+      "mean": -0.000419186665262705,
+      "std": 0.053067257135574315
+    },
+    {
+      "file": "train_evo2_70_hg38_multilen_seqdisjoint.h5",
+      "count": 2900000,
+      "mean": -0.00044969696843869265,
+      "std": 0.054235496010335234
+    },
+    {
+      "file": "train_evo2_80_hg38_multilen_seqdisjoint.h5",
+      "count": 2900000,
+      "mean": -0.00046912226088464016,
+      "std": 0.055214272478098646
+    },
+    {
+      "file": "train_evo2_90_hg38_multilen_seqdisjoint.h5",
+      "count": 3900000,
+      "mean": -0.00047963574626724,
+      "std": 0.056071271293555165
+    },
+    {
+      "file": "train_evo2_100_hg38_multilen_seqdisjoint.h5",
+      "count": 3900000,
+      "mean": -0.0004848755875720131,
+      "std": 0.05683559051038622
+    }
+  ]
+}

evo2/results.json

@@ -0,0 +1,5 @@
+{
+    "best_val_loss": 0.34188090569034624,
+    "stopped_early": false,
+    "final_epoch": 20
+}

infer.py

@@ -0,0 +1,81 @@
+"""Reconstruct DNA from raw mean-pooled embeddings saved as a NumPy array."""
+
+import argparse
+import json
+from pathlib import Path
+
+import numpy as np
+import torch
+from transformers import AutoTokenizer
+
+from model import QueryDecoderReconstructor
+
+
+class Decoder:
+    def __init__(self, model_dir, device="cpu"):
+        folder = Path(model_dir)
+        checkpoint = torch.load(folder / "model.pt", map_location="cpu", weights_only=True)
+        config = checkpoint["config"]["model"]
+        self.device = torch.device(device)
+        self.input_dim = checkpoint["input_dim"]
+        self.stats = json.loads((folder / "normalization.json").read_text())
+        assert self.stats["method"] == "standard"
+        assert np.isfinite([self.stats["mean"], self.stats["std"]]).all()
+        assert self.stats["std"] > 0
+        self.model = QueryDecoderReconstructor(
+            input_dim=self.input_dim, mode=checkpoint["mode"],
+            seq_length=checkpoint["effective_seq_length"], output_dim=checkpoint["output_dim"],
+            **{key: config[key] for key in ["d_model", "nhead", "num_layers", "dim_feedforward",
+                                           "dropout", "n_context_tokens", "aux_length_loss_weight"]},
+        )
+        self.model.load_state_dict(checkpoint["state_dict"], strict=True)
+        self.model.to(self.device).eval()
+        self.character = checkpoint["tokenizer_type"] == "char"
+        if self.character:
+            self.eos_id = 4
+            assert checkpoint["output_dim"] == 5
+        else:
+            self.tokenizer = AutoTokenizer.from_pretrained(folder / "tokenizer", local_files_only=True)
+            self.eos_id = json.loads((folder / "tokenizer_info.json").read_text())["eos_id"]
+            assert len(self.tokenizer) == checkpoint["output_dim"]
+
+    def decode_tokens(self, ids):
+        ids = list(ids)
+        if self.eos_id in ids:
+            ids = ids[:ids.index(self.eos_id)]
+        if self.character:
+            return "".join("ACGT"[i] for i in ids)
+        return self.tokenizer.decode(ids, skip_special_tokens=True).replace(" ", "")
+
+    @torch.inference_mode()
+    def predict(self, embeddings, batch_size=32):
+        raw = np.asarray(embeddings, dtype=np.float32)
+        assert raw.ndim == 2 and raw.shape[1] == self.input_dim, f"Expected [N, {self.input_dim}] embeddings"
+        assert np.isfinite(raw).all(), "Embeddings must be finite"
+        assert batch_size > 0
+        sequences = []
+        for start in range(0, len(raw), batch_size):
+            normalized = (raw[start:start + batch_size] - self.stats["mean"]) / self.stats["std"]
+            assert np.isfinite(normalized).all()
+            logits, _ = self.model(torch.from_numpy(normalized).to(self.device))
+            assert torch.isfinite(logits).all(), "Non-finite model output"
+            sequences.extend(self.decode_tokens(ids) for ids in logits.argmax(-1).cpu().tolist())
+        return sequences
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--model", required=True, choices=["dnabert2", "evo2", "ntv2"])
+    parser.add_argument("--embeddings", required=True, type=Path)
+    parser.add_argument("--output", required=True, type=Path)
+    parser.add_argument("--device", default="cpu")
+    parser.add_argument("--batch-size", type=int, default=32)
+    args = parser.parse_args()
+    decoder = Decoder(Path(__file__).parent / args.model, args.device)
+    sequences = decoder.predict(np.load(args.embeddings, allow_pickle=False), args.batch_size)
+    args.output.write_text(json.dumps(sequences, indent=2) + "\n", encoding="utf-8")
+    print(f"Wrote {len(sequences)} reconstructions to {args.output}")
+
+
+if __name__ == "__main__":
+    main()

model.py

@@ -0,0 +1,157 @@
+"""Query-based decoder for inverting mean-pooled DNA embeddings.
+
+Designed for the multi-length inversion-attack setting: a single fixed-size
+mean embedding must be decoded into a variable-length nucleotide sequence
+whose length itself is unknown.
+
+Architecture (mean mode only):
+
+1. ``context_projection`` maps the mean embedding to ``n_context_tokens``
+   learnable summary tokens of size ``d_model``. These are the "memory" that
+   cross-attention attends to.
+2. ``length_head`` predicts a distribution over possible content lengths
+   (0..seq_length-1) directly from the mean embedding.
+3. The soft length distribution is mixed with ``length_embedding`` to produce
+   a single conditioning vector that is added to every position query. This
+   makes length prediction a first-class input to content generation, not a
+   side branch: gradients flow from the sequence loss back through the length
+   head, and the decoder is trained to produce EOS at the predicted position.
+4. ``pos_queries`` are learnable per-output-slot embeddings. Combined with the
+   length conditioning, they form the decoder inputs.
+5. A stack of ``TransformerDecoderLayer`` blocks performs cross-attention
+   (queries -> context) and self-attention (queries <-> queries).
+6. ``output_projection`` produces per-position logits over the vocabulary
+   (which includes the EOS symbol).
+
+Forward returns a tuple ``(seq_logits, length_logits)``. The training loop
+combines the sequence loss with a cross-entropy loss on the length head
+(weighted by ``aux_length_loss_weight``). At inference the length head's
+output is consumed only internally and is not exposed.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import List, Optional, Tuple
+
+import torch
+import torch.nn.functional as F
+from torch import nn
+
+
+class QueryDecoderReconstructor(nn.Module):
+    def __init__(
+        self,
+        input_dim: int,
+        mode: str,
+        seq_length: int,
+        output_dim: int,
+        d_model: int = 256,
+        nhead: int = 8,
+        num_layers: int = 4,
+        dim_feedforward: int = 1024,
+        dropout: float = 0.1,
+        n_context_tokens: int = 8,
+        aux_length_loss_weight: float = 0.5,
+        hidden_dims: Optional[List[int]] = None,
+    ):
+        super().__init__()
+        assert input_dim > 0, "input_dim must be positive"
+        assert mode == "mean", "QueryDecoderReconstructor supports mean mode only"
+        assert seq_length > 0, "seq_length must be positive"
+        assert output_dim > 0, "output_dim must be positive"
+        assert d_model > 0, "d_model must be positive"
+        assert nhead > 0 and d_model % nhead == 0, "nhead must divide d_model"
+        assert num_layers > 0, "num_layers must be positive"
+        assert dim_feedforward > 0, "dim_feedforward must be positive"
+        assert 0.0 <= dropout <= 1.0, "dropout in [0, 1]"
+        assert n_context_tokens > 0, "n_context_tokens must be positive"
+        assert aux_length_loss_weight >= 0.0, "aux_length_loss_weight must be non-negative"
+
+        self.input_dim = input_dim
+        self.mode = mode
+        self.seq_length = seq_length
+        self.output_dim = output_dim
+        self.d_model = d_model
+        self.n_context_tokens = n_context_tokens
+        self.aux_length_loss_weight = aux_length_loss_weight
+
+        logger = logging.getLogger(__name__)
+        logger.info(
+            f"QueryDecoder: input_dim={input_dim} -> {n_context_tokens} context tokens "
+            f"x d_model={d_model}, seq_length={seq_length} (incl. EOS slot), "
+            f"output_dim={output_dim}, num_layers={num_layers}, nhead={nhead}, "
+            f"aux_length_loss_weight={aux_length_loss_weight}"
+        )
+
+        self.context_projection = nn.Linear(input_dim, n_context_tokens * d_model)
+
+        self.length_head = nn.Sequential(
+            nn.Linear(input_dim, d_model),
+            nn.GELU(),
+            nn.Dropout(dropout),
+            nn.Linear(d_model, seq_length),
+        )
+
+        self.length_embedding = nn.Embedding(seq_length, d_model)
+        self.pos_queries = nn.Embedding(seq_length, d_model)
+
+        decoder_layer = nn.TransformerDecoderLayer(
+            d_model=d_model,
+            nhead=nhead,
+            dim_feedforward=dim_feedforward,
+            dropout=dropout,
+            activation="gelu",
+            batch_first=True,
+            norm_first=True,
+        )
+        self.transformer_decoder = nn.TransformerDecoder(decoder_layer, num_layers=num_layers)
+
+        self.output_projection = nn.Linear(d_model, output_dim)
+        nn.init.xavier_uniform_(self.output_projection.weight, gain=0.02)
+        nn.init.zeros_(self.output_projection.bias)
+
+    def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
+        """Run the model.
+
+        Parameters
+        ----------
+        x : torch.Tensor
+            Mean-pooled embeddings, shape (B, input_dim).
+
+        Returns
+        -------
+        seq_logits : torch.Tensor
+            Per-position vocabulary logits, shape (B, seq_length, output_dim).
+        length_logits : torch.Tensor
+            Per-sample length logits, shape (B, seq_length). The class index
+            equals the number of content tokens before the EOS slot.
+        """
+        assert x.ndim == 2, f"Expected 2D mean embedding, got {x.ndim}D"
+        B, D = x.shape
+        assert D == self.input_dim, f"Expected input_dim={self.input_dim}, got {D}"
+
+        context = self.context_projection(x).view(B, self.n_context_tokens, self.d_model)
+
+        length_logits = self.length_head(x)
+        length_probs = F.softmax(length_logits, dim=-1)
+        length_emb = length_probs @ self.length_embedding.weight
+
+        queries = self.pos_queries.weight.unsqueeze(0).expand(B, -1, -1)
+        queries = queries + length_emb.unsqueeze(1)
+
+        hidden = self.transformer_decoder(tgt=queries, memory=context)
+        seq_logits = self.output_projection(hidden)
+
+        return seq_logits, length_logits
+
+    @staticmethod
+    def length_targets_from_padded(batch_sequence: torch.Tensor) -> torch.Tensor:
+        """Derive integer length targets from collated padded sequences.
+
+        Each padded row is ``[content..., EOS, -100, -100, ...]``. The target
+        we want the length head to predict is the count of content tokens
+        before EOS, i.e. (non-pad count) - 1, clamped at 0 for the empty case.
+        """
+        non_pad = (batch_sequence != -100).sum(dim=1)
+        return (non_pad - 1).clamp(min=0)

ntv2/config.yaml

@@ -0,0 +1,138 @@
+data:
+  shuffle: true
+  mean: true
+  multi: true
+  require_sequence_disjoint: true
+  source_schema: dna_inversion_hg38_multilen_v2
+  source_corpus: data/hg38_multilen_seqdisjoint/hg38_multilen_seqdisjoint.h5
+  ood_source_corpus: data/1000g_multilen_seqdisjoint/1000g_multilen_seqdisjoint.h5
+  artifact_dir: data/hg38_multilen_seqdisjoint
+  fm_slug: ntv2
+  seq_lengths:
+  - 10
+  - 15
+  - 20
+  - 25
+  - 30
+  - 35
+  - 40
+  - 45
+  - 50
+  - 60
+  - 70
+  - 80
+  - 90
+  - 100
+  train_seq_lengths:
+  - 10
+  - 15
+  - 20
+  - 25
+  - 30
+  - 35
+  - 40
+  - 45
+  - 50
+  - 60
+  - 70
+  - 80
+  - 90
+  - 100
+  train_csvs:
+  - ${data.artifact_dir}/train_${data.fm_slug}_10_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_15_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_20_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_25_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_30_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_35_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_40_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_45_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_50_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_60_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_70_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_80_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_90_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/train_${data.fm_slug}_100_hg38_multilen_seqdisjoint.h5
+  val_csvs:
+  - ${data.artifact_dir}/val_${data.fm_slug}_10_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_15_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_20_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_25_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_30_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_35_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_40_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_45_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_50_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_60_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_70_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_80_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_90_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/val_${data.fm_slug}_100_hg38_multilen_seqdisjoint.h5
+  test_csvs:
+  - ${data.artifact_dir}/test_${data.fm_slug}_10_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_15_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_20_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_25_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_30_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_35_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_40_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_45_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_50_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_60_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_70_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_80_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_90_hg38_multilen_seqdisjoint.h5
+  - ${data.artifact_dir}/test_${data.fm_slug}_100_hg38_multilen_seqdisjoint.h5
+  seq_length: 100
+  skip_sha256_check: true
+  normalization_method: standard
+  subset_fraction: null
+  embedding_dim: 1024
+  tokenizer:
+    type: huggingface
+    model_name: InstaDeepAI/nucleotide-transformer-v2-500m-multi-species
+  checkpoint: InstaDeepAI/nucleotide-transformer-v2-500m-multi-species
+model:
+  model_file_name: model/query_decoder.py
+  model_name: query_decoder
+  model_type: query_decoder
+  mode: mean
+  d_model: 384
+  nhead: 8
+  num_layers: 6
+  dim_feedforward: 1536
+  dropout: 0.1
+  n_context_tokens: 16
+  aux_length_loss_weight: 0.5
+  output_dim: 4
+  hidden_dims: []
+optim:
+  lr: 0.001
+  weight_decay: 0.0001
+  eps: 1.0e-06
+  epochs: 20
+  batch_size: 32
+  num_workers: 16
+  grad_clip_norm: 1.0
+  use_scheduler: true
+  scheduler:
+    max_lr: 0.0005
+    pct_start: 0.3
+    anneal_strategy: cos
+    div_factor: 25.0
+    final_div_factor: 10000.0
+train:
+  seed: 42
+  project: dna-embedding-inversion
+  run_name: null
+  use_wandb: false
+  log_interval: 10000
+  device: cuda:0
+  early_stopping:
+    enabled: false
+    patience: 5
+    min_delta: 1.0e-05
+  max_val_samples: null
+  nan_backoff:
+    skip_frac_threshold: 0.02
+    lr_backoff: 0.5

ntv2/model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5300eea166e2b31aa5a7e0e90d1bb2d859d86de94fe185371b142041899c9539
+size 90028571

ntv2/model.py

@@ -0,0 +1,157 @@
+"""Query-based decoder for inverting mean-pooled DNA embeddings.
+
+Designed for the multi-length inversion-attack setting: a single fixed-size
+mean embedding must be decoded into a variable-length nucleotide sequence
+whose length itself is unknown.
+
+Architecture (mean mode only):
+
+1. ``context_projection`` maps the mean embedding to ``n_context_tokens``
+   learnable summary tokens of size ``d_model``. These are the "memory" that
+   cross-attention attends to.
+2. ``length_head`` predicts a distribution over possible content lengths
+   (0..seq_length-1) directly from the mean embedding.
+3. The soft length distribution is mixed with ``length_embedding`` to produce
+   a single conditioning vector that is added to every position query. This
+   makes length prediction a first-class input to content generation, not a
+   side branch: gradients flow from the sequence loss back through the length
+   head, and the decoder is trained to produce EOS at the predicted position.
+4. ``pos_queries`` are learnable per-output-slot embeddings. Combined with the
+   length conditioning, they form the decoder inputs.
+5. A stack of ``TransformerDecoderLayer`` blocks performs cross-attention
+   (queries -> context) and self-attention (queries <-> queries).
+6. ``output_projection`` produces per-position logits over the vocabulary
+   (which includes the EOS symbol).
+
+Forward returns a tuple ``(seq_logits, length_logits)``. The training loop
+combines the sequence loss with a cross-entropy loss on the length head
+(weighted by ``aux_length_loss_weight``). At inference the length head's
+output is consumed only internally and is not exposed.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import List, Optional, Tuple
+
+import torch
+import torch.nn.functional as F
+from torch import nn
+
+
+class QueryDecoderReconstructor(nn.Module):
+    def __init__(
+        self,
+        input_dim: int,
+        mode: str,
+        seq_length: int,
+        output_dim: int,
+        d_model: int = 256,
+        nhead: int = 8,
+        num_layers: int = 4,
+        dim_feedforward: int = 1024,
+        dropout: float = 0.1,
+        n_context_tokens: int = 8,
+        aux_length_loss_weight: float = 0.5,
+        hidden_dims: Optional[List[int]] = None,
+    ):
+        super().__init__()
+        assert input_dim > 0, "input_dim must be positive"
+        assert mode == "mean", "QueryDecoderReconstructor supports mean mode only"
+        assert seq_length > 0, "seq_length must be positive"
+        assert output_dim > 0, "output_dim must be positive"
+        assert d_model > 0, "d_model must be positive"
+        assert nhead > 0 and d_model % nhead == 0, "nhead must divide d_model"
+        assert num_layers > 0, "num_layers must be positive"
+        assert dim_feedforward > 0, "dim_feedforward must be positive"
+        assert 0.0 <= dropout <= 1.0, "dropout in [0, 1]"
+        assert n_context_tokens > 0, "n_context_tokens must be positive"
+        assert aux_length_loss_weight >= 0.0, "aux_length_loss_weight must be non-negative"
+
+        self.input_dim = input_dim
+        self.mode = mode
+        self.seq_length = seq_length
+        self.output_dim = output_dim
+        self.d_model = d_model
+        self.n_context_tokens = n_context_tokens
+        self.aux_length_loss_weight = aux_length_loss_weight
+
+        logger = logging.getLogger(__name__)
+        logger.info(
+            f"QueryDecoder: input_dim={input_dim} -> {n_context_tokens} context tokens "
+            f"x d_model={d_model}, seq_length={seq_length} (incl. EOS slot), "
+            f"output_dim={output_dim}, num_layers={num_layers}, nhead={nhead}, "
+            f"aux_length_loss_weight={aux_length_loss_weight}"
+        )
+
+        self.context_projection = nn.Linear(input_dim, n_context_tokens * d_model)
+
+        self.length_head = nn.Sequential(
+            nn.Linear(input_dim, d_model),
+            nn.GELU(),
+            nn.Dropout(dropout),
+            nn.Linear(d_model, seq_length),
+        )
+
+        self.length_embedding = nn.Embedding(seq_length, d_model)
+        self.pos_queries = nn.Embedding(seq_length, d_model)
+
+        decoder_layer = nn.TransformerDecoderLayer(
+            d_model=d_model,
+            nhead=nhead,
+            dim_feedforward=dim_feedforward,
+            dropout=dropout,
+            activation="gelu",
+            batch_first=True,
+            norm_first=True,
+        )
+        self.transformer_decoder = nn.TransformerDecoder(decoder_layer, num_layers=num_layers)
+
+        self.output_projection = nn.Linear(d_model, output_dim)
+        nn.init.xavier_uniform_(self.output_projection.weight, gain=0.02)
+        nn.init.zeros_(self.output_projection.bias)
+
+    def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
+        """Run the model.
+
+        Parameters
+        ----------
+        x : torch.Tensor
+            Mean-pooled embeddings, shape (B, input_dim).
+
+        Returns
+        -------
+        seq_logits : torch.Tensor
+            Per-position vocabulary logits, shape (B, seq_length, output_dim).
+        length_logits : torch.Tensor
+            Per-sample length logits, shape (B, seq_length). The class index
+            equals the number of content tokens before the EOS slot.
+        """
+        assert x.ndim == 2, f"Expected 2D mean embedding, got {x.ndim}D"
+        B, D = x.shape
+        assert D == self.input_dim, f"Expected input_dim={self.input_dim}, got {D}"
+
+        context = self.context_projection(x).view(B, self.n_context_tokens, self.d_model)
+
+        length_logits = self.length_head(x)
+        length_probs = F.softmax(length_logits, dim=-1)
+        length_emb = length_probs @ self.length_embedding.weight
+
+        queries = self.pos_queries.weight.unsqueeze(0).expand(B, -1, -1)
+        queries = queries + length_emb.unsqueeze(1)
+
+        hidden = self.transformer_decoder(tgt=queries, memory=context)
+        seq_logits = self.output_projection(hidden)
+
+        return seq_logits, length_logits
+
+    @staticmethod
+    def length_targets_from_padded(batch_sequence: torch.Tensor) -> torch.Tensor:
+        """Derive integer length targets from collated padded sequences.
+
+        Each padded row is ``[content..., EOS, -100, -100, ...]``. The target
+        we want the length head to predict is the count of content tokens
+        before EOS, i.e. (non-pad count) - 1, clamped at 0 for the empty case.
+        """
+        non_pad = (batch_sequence != -100).sum(dim=1)
+        return (non_pad - 1).clamp(min=0)

ntv2/normalization.json

@@ -0,0 +1,93 @@
+{
+  "method": "standard",
+  "min": -4.435836315155029,
+  "max": 6.862894535064697,
+  "mean": -0.005306116518743337,
+  "std": 0.30370272074898164,
+  "sources": [
+    {
+      "file": "train_ntv2_10_hg38_multilen_seqdisjoint.h5",
+      "count": 900000,
+      "mean": -0.005334406934874838,
+      "std": 0.3334975119971993
+    },
+    {
+      "file": "train_ntv2_15_hg38_multilen_seqdisjoint.h5",
+      "count": 900000,
+      "mean": -0.005277447656316087,
+      "std": 0.3270400077449325
+    },
+    {
+      "file": "train_ntv2_20_hg38_multilen_seqdisjoint.h5",
+      "count": 900000,
+      "mean": -0.0052237071189612065,
+      "std": 0.3185615420727908
+    },
+    {
+      "file": "train_ntv2_25_hg38_multilen_seqdisjoint.h5",
+      "count": 1900000,
+      "mean": -0.005309094393041889,
+      "std": 0.31743059075315955
+    },
+    {
+      "file": "train_ntv2_30_hg38_multilen_seqdisjoint.h5",
+      "count": 1900000,
+      "mean": -0.00553773123097,
+      "std": 0.32302076070874497
+    },
+    {
+      "file": "train_ntv2_35_hg38_multilen_seqdisjoint.h5",
+      "count": 1900000,
+      "mean": -0.005011122444204863,
+      "std": 0.30023586869909136
+    },
+    {
+      "file": "train_ntv2_40_hg38_multilen_seqdisjoint.h5",
+      "count": 1900000,
+      "mean": -0.005009883395815026,
+      "std": 0.29441732365202133
+    },
+    {
+      "file": "train_ntv2_45_hg38_multilen_seqdisjoint.h5",
+      "count": 1900000,
+      "mean": -0.005321131197112725,
+      "std": 0.30194153296032494
+    },
+    {
+      "file": "train_ntv2_50_hg38_multilen_seqdisjoint.h5",
+      "count": 2900000,
+      "mean": -0.005381770079028985,
+      "std": 0.3019136631684386
+    },
+    {
+      "file": "train_ntv2_60_hg38_multilen_seqdisjoint.h5",
+      "count": 2900000,
+      "mean": -0.005350268666411305,
+      "std": 0.30242780246285605
+    },
+    {
+      "file": "train_ntv2_70_hg38_multilen_seqdisjoint.h5",
+      "count": 2900000,
+      "mean": -0.005276116533628761,
+      "std": 0.29582258515987014
+    },
+    {
+      "file": "train_ntv2_80_hg38_multilen_seqdisjoint.h5",
+      "count": 2900000,
+      "mean": -0.005392027698492022,
+      "std": 0.29830093274837594
+    },
+    {
+      "file": "train_ntv2_90_hg38_multilen_seqdisjoint.h5",
+      "count": 3900000,
+      "mean": -0.005305182318495808,
+      "std": 0.29641131128882225
+    },
+    {
+      "file": "train_ntv2_100_hg38_multilen_seqdisjoint.h5",
+      "count": 3900000,
+      "mean": -0.0053619245435685145,
+      "std": 0.2969905374144667
+    }
+  ]
+}

ntv2/results.json

@@ -0,0 +1,5 @@
+{
+    "best_val_loss": 0.6885444451381479,
+    "stopped_early": false,
+    "final_epoch": 20
+}

ntv2/tokenizer/special_tokens_map.json

@@ -0,0 +1,13 @@
+{
+  "cls_token": "<cls>",
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": "<mask>",
+  "pad_token": "<pad>",
+  "unk_token": "<unk>"
+}

ntv2/tokenizer/tokenizer_config.json

@@ -0,0 +1,52 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<mask>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<cls>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<eos>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "<cls>",
+  "eos_token": "<eos>",
+  "mask_token": "<mask>",
+  "model_max_length": 2048,
+  "pad_token": "<pad>",
+  "tokenizer_class": "EsmTokenizer",
+  "unk_token": "<unk>"
+}

ntv2/tokenizer/vocab.txt

@@ -0,0 +1,4107 @@
+<unk>
+<pad>
+<mask>
+<cls>
+<eos>
+<bos>
+AAAAAA
+AAAAAT
+AAAAAC
+AAAAAG
+AAAATA
+AAAATT
+AAAATC
+AAAATG
+AAAACA
+AAAACT
+AAAACC
+AAAACG
+AAAAGA
+AAAAGT
+AAAAGC
+AAAAGG
+AAATAA
+AAATAT
+AAATAC
+AAATAG
+AAATTA
+AAATTT
+AAATTC
+AAATTG
+AAATCA
+AAATCT
+AAATCC
+AAATCG
+AAATGA
+AAATGT
+AAATGC
+AAATGG
+AAACAA
+AAACAT
+AAACAC
+AAACAG
+AAACTA
+AAACTT
+AAACTC
+AAACTG
+AAACCA
+AAACCT
+AAACCC
+AAACCG
+AAACGA
+AAACGT
+AAACGC
+AAACGG
+AAAGAA
+AAAGAT
+AAAGAC
+AAAGAG
+AAAGTA
+AAAGTT
+AAAGTC
+AAAGTG
+AAAGCA
+AAAGCT
+AAAGCC
+AAAGCG
+AAAGGA
+AAAGGT
+AAAGGC
+AAAGGG
+AATAAA
+AATAAT
+AATAAC
+AATAAG
+AATATA
+AATATT
+AATATC
+AATATG
+AATACA
+AATACT
+AATACC
+AATACG
+AATAGA
+AATAGT
+AATAGC
+AATAGG
+AATTAA
+AATTAT
+AATTAC
+AATTAG
+AATTTA
+AATTTT
+AATTTC
+AATTTG
+AATTCA
+AATTCT
+AATTCC
+AATTCG
+AATTGA
+AATTGT
+AATTGC
+AATTGG
+AATCAA
+AATCAT
+AATCAC
+AATCAG
+AATCTA
+AATCTT
+AATCTC
+AATCTG
+AATCCA
+AATCCT
+AATCCC
+AATCCG
+AATCGA
+AATCGT
+AATCGC
+AATCGG
+AATGAA
+AATGAT
+AATGAC
+AATGAG
+AATGTA
+AATGTT
+AATGTC
+AATGTG
+AATGCA
+AATGCT
+AATGCC
+AATGCG
+AATGGA
+AATGGT
+AATGGC
+AATGGG
+AACAAA
+AACAAT
+AACAAC
+AACAAG
+AACATA
+AACATT
+AACATC
+AACATG
+AACACA
+AACACT
+AACACC
+AACACG
+AACAGA
+AACAGT
+AACAGC
+AACAGG
+AACTAA
+AACTAT
+AACTAC
+AACTAG
+AACTTA
+AACTTT
+AACTTC
+AACTTG
+AACTCA
+AACTCT
+AACTCC
+AACTCG
+AACTGA
+AACTGT
+AACTGC
+AACTGG
+AACCAA
+AACCAT
+AACCAC
+AACCAG
+AACCTA
+AACCTT
+AACCTC
+AACCTG
+AACCCA
+AACCCT
+AACCCC
+AACCCG
+AACCGA
+AACCGT
+AACCGC
+AACCGG
+AACGAA
+AACGAT
+AACGAC
+AACGAG
+AACGTA
+AACGTT
+AACGTC
+AACGTG
+AACGCA
+AACGCT
+AACGCC
+AACGCG
+AACGGA
+AACGGT
+AACGGC
+AACGGG
+AAGAAA
+AAGAAT
+AAGAAC
+AAGAAG
+AAGATA
+AAGATT
+AAGATC
+AAGATG
+AAGACA
+AAGACT
+AAGACC
+AAGACG
+AAGAGA
+AAGAGT
+AAGAGC
+AAGAGG
+AAGTAA
+AAGTAT
+AAGTAC
+AAGTAG
+AAGTTA
+AAGTTT
+AAGTTC
+AAGTTG
+AAGTCA
+AAGTCT
+AAGTCC
+AAGTCG
+AAGTGA
+AAGTGT
+AAGTGC
+AAGTGG
+AAGCAA
+AAGCAT
+AAGCAC
+AAGCAG
+AAGCTA
+AAGCTT
+AAGCTC
+AAGCTG
+AAGCCA
+AAGCCT
+AAGCCC
+AAGCCG
+AAGCGA
+AAGCGT
+AAGCGC
+AAGCGG
+AAGGAA
+AAGGAT
+AAGGAC
+AAGGAG
+AAGGTA
+AAGGTT
+AAGGTC
+AAGGTG
+AAGGCA
+AAGGCT
+AAGGCC
+AAGGCG
+AAGGGA
+AAGGGT
+AAGGGC
+AAGGGG
+ATAAAA
+ATAAAT
+ATAAAC
+ATAAAG
+ATAATA
+ATAATT
+ATAATC
+ATAATG
+ATAACA
+ATAACT
+ATAACC
+ATAACG
+ATAAGA
+ATAAGT
+ATAAGC
+ATAAGG
+ATATAA
+ATATAT
+ATATAC
+ATATAG
+ATATTA
+ATATTT
+ATATTC
+ATATTG
+ATATCA
+ATATCT
+ATATCC
+ATATCG
+ATATGA
+ATATGT
+ATATGC
+ATATGG
+ATACAA
+ATACAT
+ATACAC
+ATACAG
+ATACTA
+ATACTT
+ATACTC
+ATACTG
+ATACCA
+ATACCT
+ATACCC
+ATACCG
+ATACGA
+ATACGT
+ATACGC
+ATACGG
+ATAGAA
+ATAGAT
+ATAGAC
+ATAGAG
+ATAGTA
+ATAGTT
+ATAGTC
+ATAGTG
+ATAGCA
+ATAGCT
+ATAGCC
+ATAGCG
+ATAGGA
+ATAGGT
+ATAGGC
+ATAGGG
+ATTAAA
+ATTAAT
+ATTAAC
+ATTAAG
+ATTATA
+ATTATT
+ATTATC
+ATTATG
+ATTACA
+ATTACT
+ATTACC
+ATTACG
+ATTAGA
+ATTAGT
+ATTAGC
+ATTAGG
+ATTTAA
+ATTTAT
+ATTTAC
+ATTTAG
+ATTTTA
+ATTTTT
+ATTTTC
+ATTTTG
+ATTTCA
+ATTTCT
+ATTTCC
+ATTTCG
+ATTTGA
+ATTTGT
+ATTTGC
+ATTTGG
+ATTCAA
+ATTCAT
+ATTCAC
+ATTCAG
+ATTCTA
+ATTCTT
+ATTCTC
+ATTCTG
+ATTCCA
+ATTCCT
+ATTCCC
+ATTCCG
+ATTCGA
+ATTCGT
+ATTCGC
+ATTCGG
+ATTGAA
+ATTGAT
+ATTGAC
+ATTGAG
+ATTGTA
+ATTGTT
+ATTGTC
+ATTGTG
+ATTGCA
+ATTGCT
+ATTGCC
+ATTGCG
+ATTGGA
+ATTGGT
+ATTGGC
+ATTGGG
+ATCAAA
+ATCAAT
+ATCAAC
+ATCAAG
+ATCATA
+ATCATT
+ATCATC
+ATCATG
+ATCACA
+ATCACT
+ATCACC
+ATCACG
+ATCAGA
+ATCAGT
+ATCAGC
+ATCAGG
+ATCTAA
+ATCTAT
+ATCTAC
+ATCTAG
+ATCTTA
+ATCTTT
+ATCTTC
+ATCTTG
+ATCTCA
+ATCTCT
+ATCTCC
+ATCTCG
+ATCTGA
+ATCTGT
+ATCTGC
+ATCTGG
+ATCCAA
+ATCCAT
+ATCCAC
+ATCCAG
+ATCCTA
+ATCCTT
+ATCCTC
+ATCCTG
+ATCCCA
+ATCCCT
+ATCCCC
+ATCCCG
+ATCCGA
+ATCCGT
+ATCCGC
+ATCCGG
+ATCGAA
+ATCGAT
+ATCGAC
+ATCGAG
+ATCGTA
+ATCGTT
+ATCGTC
+ATCGTG
+ATCGCA
+ATCGCT
+ATCGCC
+ATCGCG
+ATCGGA
+ATCGGT
+ATCGGC
+ATCGGG
+ATGAAA
+ATGAAT
+ATGAAC
+ATGAAG
+ATGATA
+ATGATT
+ATGATC
+ATGATG
+ATGACA
+ATGACT
+ATGACC
+ATGACG
+ATGAGA
+ATGAGT
+ATGAGC
+ATGAGG
+ATGTAA
+ATGTAT
+ATGTAC
+ATGTAG
+ATGTTA
+ATGTTT
+ATGTTC
+ATGTTG
+ATGTCA
+ATGTCT
+ATGTCC
+ATGTCG
+ATGTGA
+ATGTGT
+ATGTGC
+ATGTGG
+ATGCAA
+ATGCAT
+ATGCAC
+ATGCAG
+ATGCTA
+ATGCTT
+ATGCTC
+ATGCTG
+ATGCCA
+ATGCCT
+ATGCCC
+ATGCCG
+ATGCGA
+ATGCGT
+ATGCGC
+ATGCGG
+ATGGAA
+ATGGAT
+ATGGAC
+ATGGAG
+ATGGTA
+ATGGTT
+ATGGTC
+ATGGTG
+ATGGCA
+ATGGCT
+ATGGCC
+ATGGCG
+ATGGGA
+ATGGGT
+ATGGGC
+ATGGGG
+ACAAAA
+ACAAAT
+ACAAAC
+ACAAAG
+ACAATA
+ACAATT
+ACAATC
+ACAATG
+ACAACA
+ACAACT
+ACAACC
+ACAACG
+ACAAGA
+ACAAGT
+ACAAGC
+ACAAGG
+ACATAA
+ACATAT
+ACATAC
+ACATAG
+ACATTA
+ACATTT
+ACATTC
+ACATTG
+ACATCA
+ACATCT
+ACATCC
+ACATCG
+ACATGA
+ACATGT
+ACATGC
+ACATGG
+ACACAA
+ACACAT
+ACACAC
+ACACAG
+ACACTA
+ACACTT
+ACACTC
+ACACTG
+ACACCA
+ACACCT
+ACACCC
+ACACCG
+ACACGA
+ACACGT
+ACACGC
+ACACGG
+ACAGAA
+ACAGAT
+ACAGAC
+ACAGAG
+ACAGTA
+ACAGTT
+ACAGTC
+ACAGTG
+ACAGCA
+ACAGCT
+ACAGCC
+ACAGCG
+ACAGGA
+ACAGGT
+ACAGGC
+ACAGGG
+ACTAAA
+ACTAAT
+ACTAAC
+ACTAAG
+ACTATA
+ACTATT
+ACTATC
+ACTATG
+ACTACA
+ACTACT
+ACTACC
+ACTACG
+ACTAGA
+ACTAGT
+ACTAGC
+ACTAGG
+ACTTAA
+ACTTAT
+ACTTAC
+ACTTAG
+ACTTTA
+ACTTTT
+ACTTTC
+ACTTTG
+ACTTCA
+ACTTCT
+ACTTCC
+ACTTCG
+ACTTGA
+ACTTGT
+ACTTGC
+ACTTGG
+ACTCAA
+ACTCAT
+ACTCAC
+ACTCAG
+ACTCTA
+ACTCTT
+ACTCTC
+ACTCTG
+ACTCCA
+ACTCCT
+ACTCCC
+ACTCCG
+ACTCGA
+ACTCGT
+ACTCGC
+ACTCGG
+ACTGAA
+ACTGAT
+ACTGAC
+ACTGAG
+ACTGTA
+ACTGTT
+ACTGTC
+ACTGTG
+ACTGCA
+ACTGCT
+ACTGCC
+ACTGCG
+ACTGGA
+ACTGGT
+ACTGGC
+ACTGGG
+ACCAAA
+ACCAAT
+ACCAAC
+ACCAAG
+ACCATA
+ACCATT
+ACCATC
+ACCATG
+ACCACA
+ACCACT
+ACCACC
+ACCACG
+ACCAGA
+ACCAGT
+ACCAGC
+ACCAGG
+ACCTAA
+ACCTAT
+ACCTAC
+ACCTAG
+ACCTTA
+ACCTTT
+ACCTTC
+ACCTTG
+ACCTCA
+ACCTCT
+ACCTCC
+ACCTCG
+ACCTGA
+ACCTGT
+ACCTGC
+ACCTGG
+ACCCAA
+ACCCAT
+ACCCAC
+ACCCAG
+ACCCTA
+ACCCTT
+ACCCTC
+ACCCTG
+ACCCCA
+ACCCCT
+ACCCCC
+ACCCCG
+ACCCGA
+ACCCGT
+ACCCGC
+ACCCGG
+ACCGAA
+ACCGAT
+ACCGAC
+ACCGAG
+ACCGTA
+ACCGTT
+ACCGTC
+ACCGTG
+ACCGCA
+ACCGCT
+ACCGCC
+ACCGCG
+ACCGGA
+ACCGGT
+ACCGGC
+ACCGGG
+ACGAAA
+ACGAAT
+ACGAAC
+ACGAAG
+ACGATA
+ACGATT
+ACGATC
+ACGATG
+ACGACA
+ACGACT
+ACGACC
+ACGACG
+ACGAGA
+ACGAGT
+ACGAGC
+ACGAGG
+ACGTAA
+ACGTAT
+ACGTAC
+ACGTAG
+ACGTTA
+ACGTTT
+ACGTTC
+ACGTTG
+ACGTCA
+ACGTCT
+ACGTCC
+ACGTCG
+ACGTGA
+ACGTGT
+ACGTGC
+ACGTGG
+ACGCAA
+ACGCAT
+ACGCAC
+ACGCAG
+ACGCTA
+ACGCTT
+ACGCTC
+ACGCTG
+ACGCCA
+ACGCCT
+ACGCCC
+ACGCCG
+ACGCGA
+ACGCGT
+ACGCGC
+ACGCGG
+ACGGAA
+ACGGAT
+ACGGAC
+ACGGAG
+ACGGTA
+ACGGTT
+ACGGTC
+ACGGTG
+ACGGCA
+ACGGCT
+ACGGCC
+ACGGCG
+ACGGGA
+ACGGGT
+ACGGGC
+ACGGGG
+AGAAAA
+AGAAAT
+AGAAAC
+AGAAAG
+AGAATA
+AGAATT
+AGAATC
+AGAATG
+AGAACA
+AGAACT
+AGAACC
+AGAACG
+AGAAGA
+AGAAGT
+AGAAGC
+AGAAGG
+AGATAA
+AGATAT
+AGATAC
+AGATAG
+AGATTA
+AGATTT
+AGATTC
+AGATTG
+AGATCA
+AGATCT
+AGATCC
+AGATCG
+AGATGA
+AGATGT
+AGATGC
+AGATGG
+AGACAA
+AGACAT
+AGACAC
+AGACAG
+AGACTA
+AGACTT
+AGACTC
+AGACTG
+AGACCA
+AGACCT
+AGACCC
+AGACCG
+AGACGA
+AGACGT
+AGACGC
+AGACGG
+AGAGAA
+AGAGAT
+AGAGAC
+AGAGAG
+AGAGTA
+AGAGTT
+AGAGTC
+AGAGTG
+AGAGCA
+AGAGCT
+AGAGCC
+AGAGCG
+AGAGGA
+AGAGGT
+AGAGGC
+AGAGGG
+AGTAAA
+AGTAAT
+AGTAAC
+AGTAAG
+AGTATA
+AGTATT
+AGTATC
+AGTATG
+AGTACA
+AGTACT
+AGTACC
+AGTACG
+AGTAGA
+AGTAGT
+AGTAGC
+AGTAGG
+AGTTAA
+AGTTAT
+AGTTAC
+AGTTAG
+AGTTTA
+AGTTTT
+AGTTTC
+AGTTTG
+AGTTCA
+AGTTCT
+AGTTCC
+AGTTCG
+AGTTGA
+AGTTGT
+AGTTGC
+AGTTGG
+AGTCAA
+AGTCAT
+AGTCAC
+AGTCAG
+AGTCTA
+AGTCTT
+AGTCTC
+AGTCTG
+AGTCCA
+AGTCCT
+AGTCCC
+AGTCCG
+AGTCGA
+AGTCGT
+AGTCGC
+AGTCGG
+AGTGAA
+AGTGAT
+AGTGAC
+AGTGAG
+AGTGTA
+AGTGTT
+AGTGTC
+AGTGTG
+AGTGCA
+AGTGCT
+AGTGCC
+AGTGCG
+AGTGGA
+AGTGGT
+AGTGGC
+AGTGGG
+AGCAAA
+AGCAAT
+AGCAAC
+AGCAAG
+AGCATA
+AGCATT
+AGCATC
+AGCATG
+AGCACA
+AGCACT
+AGCACC
+AGCACG
+AGCAGA
+AGCAGT
+AGCAGC
+AGCAGG
+AGCTAA
+AGCTAT
+AGCTAC
+AGCTAG
+AGCTTA
+AGCTTT
+AGCTTC
+AGCTTG
+AGCTCA
+AGCTCT
+AGCTCC
+AGCTCG
+AGCTGA
+AGCTGT
+AGCTGC
+AGCTGG
+AGCCAA
+AGCCAT
+AGCCAC
+AGCCAG
+AGCCTA
+AGCCTT
+AGCCTC
+AGCCTG
+AGCCCA
+AGCCCT
+AGCCCC
+AGCCCG
+AGCCGA
+AGCCGT
+AGCCGC
+AGCCGG
+AGCGAA
+AGCGAT
+AGCGAC
+AGCGAG
+AGCGTA
+AGCGTT
+AGCGTC
+AGCGTG
+AGCGCA
+AGCGCT
+AGCGCC
+AGCGCG
+AGCGGA
+AGCGGT
+AGCGGC
+AGCGGG
+AGGAAA
+AGGAAT
+AGGAAC
+AGGAAG
+AGGATA
+AGGATT
+AGGATC
+AGGATG
+AGGACA
+AGGACT
+AGGACC
+AGGACG
+AGGAGA
+AGGAGT
+AGGAGC
+AGGAGG
+AGGTAA
+AGGTAT
+AGGTAC
+AGGTAG
+AGGTTA
+AGGTTT
+AGGTTC
+AGGTTG
+AGGTCA
+AGGTCT
+AGGTCC
+AGGTCG
+AGGTGA
+AGGTGT
+AGGTGC
+AGGTGG
+AGGCAA
+AGGCAT
+AGGCAC
+AGGCAG
+AGGCTA
+AGGCTT
+AGGCTC
+AGGCTG
+AGGCCA
+AGGCCT
+AGGCCC
+AGGCCG
+AGGCGA
+AGGCGT
+AGGCGC
+AGGCGG
+AGGGAA
+AGGGAT
+AGGGAC
+AGGGAG
+AGGGTA
+AGGGTT
+AGGGTC
+AGGGTG
+AGGGCA
+AGGGCT
+AGGGCC
+AGGGCG
+AGGGGA
+AGGGGT
+AGGGGC
+AGGGGG
+TAAAAA
+TAAAAT
+TAAAAC
+TAAAAG
+TAAATA
+TAAATT
+TAAATC
+TAAATG
+TAAACA
+TAAACT
+TAAACC
+TAAACG
+TAAAGA
+TAAAGT
+TAAAGC
+TAAAGG
+TAATAA
+TAATAT
+TAATAC
+TAATAG
+TAATTA
+TAATTT
+TAATTC
+TAATTG
+TAATCA
+TAATCT
+TAATCC
+TAATCG
+TAATGA
+TAATGT
+TAATGC
+TAATGG
+TAACAA
+TAACAT
+TAACAC
+TAACAG
+TAACTA
+TAACTT
+TAACTC
+TAACTG
+TAACCA
+TAACCT
+TAACCC
+TAACCG
+TAACGA
+TAACGT
+TAACGC
+TAACGG
+TAAGAA
+TAAGAT
+TAAGAC
+TAAGAG
+TAAGTA
+TAAGTT
+TAAGTC
+TAAGTG
+TAAGCA
+TAAGCT
+TAAGCC
+TAAGCG
+TAAGGA
+TAAGGT
+TAAGGC
+TAAGGG
+TATAAA
+TATAAT
+TATAAC
+TATAAG
+TATATA
+TATATT
+TATATC
+TATATG
+TATACA
+TATACT
+TATACC
+TATACG
+TATAGA
+TATAGT
+TATAGC
+TATAGG
+TATTAA
+TATTAT
+TATTAC
+TATTAG
+TATTTA
+TATTTT
+TATTTC
+TATTTG
+TATTCA
+TATTCT
+TATTCC
+TATTCG
+TATTGA
+TATTGT
+TATTGC
+TATTGG
+TATCAA
+TATCAT
+TATCAC
+TATCAG
+TATCTA
+TATCTT
+TATCTC
+TATCTG
+TATCCA
+TATCCT
+TATCCC
+TATCCG
+TATCGA
+TATCGT
+TATCGC
+TATCGG
+TATGAA
+TATGAT
+TATGAC
+TATGAG
+TATGTA
+TATGTT
+TATGTC
+TATGTG
+TATGCA
+TATGCT
+TATGCC
+TATGCG
+TATGGA
+TATGGT
+TATGGC
+TATGGG
+TACAAA
+TACAAT
+TACAAC
+TACAAG
+TACATA
+TACATT
+TACATC
+TACATG
+TACACA
+TACACT
+TACACC
+TACACG
+TACAGA
+TACAGT
+TACAGC
+TACAGG
+TACTAA
+TACTAT
+TACTAC
+TACTAG
+TACTTA
+TACTTT
+TACTTC
+TACTTG
+TACTCA
+TACTCT
+TACTCC
+TACTCG
+TACTGA
+TACTGT
+TACTGC
+TACTGG
+TACCAA
+TACCAT
+TACCAC
+TACCAG
+TACCTA
+TACCTT
+TACCTC
+TACCTG
+TACCCA
+TACCCT
+TACCCC
+TACCCG
+TACCGA
+TACCGT
+TACCGC
+TACCGG
+TACGAA
+TACGAT
+TACGAC
+TACGAG
+TACGTA
+TACGTT
+TACGTC
+TACGTG
+TACGCA
+TACGCT
+TACGCC
+TACGCG
+TACGGA
+TACGGT
+TACGGC
+TACGGG
+TAGAAA
+TAGAAT
+TAGAAC
+TAGAAG
+TAGATA
+TAGATT
+TAGATC
+TAGATG
+TAGACA
+TAGACT
+TAGACC
+TAGACG
+TAGAGA
+TAGAGT
+TAGAGC
+TAGAGG
+TAGTAA
+TAGTAT
+TAGTAC
+TAGTAG
+TAGTTA
+TAGTTT
+TAGTTC
+TAGTTG
+TAGTCA
+TAGTCT
+TAGTCC
+TAGTCG
+TAGTGA
+TAGTGT
+TAGTGC
+TAGTGG
+TAGCAA
+TAGCAT
+TAGCAC
+TAGCAG
+TAGCTA
+TAGCTT
+TAGCTC
+TAGCTG
+TAGCCA
+TAGCCT
+TAGCCC
+TAGCCG
+TAGCGA
+TAGCGT
+TAGCGC
+TAGCGG
+TAGGAA
+TAGGAT
+TAGGAC
+TAGGAG
+TAGGTA
+TAGGTT
+TAGGTC
+TAGGTG
+TAGGCA
+TAGGCT
+TAGGCC
+TAGGCG
+TAGGGA
+TAGGGT
+TAGGGC
+TAGGGG
+TTAAAA
+TTAAAT
+TTAAAC
+TTAAAG
+TTAATA
+TTAATT
+TTAATC
+TTAATG
+TTAACA
+TTAACT
+TTAACC
+TTAACG
+TTAAGA
+TTAAGT
+TTAAGC
+TTAAGG
+TTATAA
+TTATAT
+TTATAC
+TTATAG
+TTATTA
+TTATTT
+TTATTC
+TTATTG
+TTATCA
+TTATCT
+TTATCC
+TTATCG
+TTATGA
+TTATGT
+TTATGC
+TTATGG
+TTACAA
+TTACAT
+TTACAC
+TTACAG
+TTACTA
+TTACTT
+TTACTC
+TTACTG
+TTACCA
+TTACCT
+TTACCC
+TTACCG
+TTACGA
+TTACGT
+TTACGC
+TTACGG
+TTAGAA
+TTAGAT
+TTAGAC
+TTAGAG
+TTAGTA
+TTAGTT
+TTAGTC
+TTAGTG
+TTAGCA
+TTAGCT
+TTAGCC
+TTAGCG
+TTAGGA
+TTAGGT
+TTAGGC
+TTAGGG
+TTTAAA
+TTTAAT
+TTTAAC
+TTTAAG
+TTTATA
+TTTATT
+TTTATC
+TTTATG
+TTTACA
+TTTACT
+TTTACC
+TTTACG
+TTTAGA
+TTTAGT
+TTTAGC
+TTTAGG
+TTTTAA
+TTTTAT
+TTTTAC
+TTTTAG
+TTTTTA
+TTTTTT
+TTTTTC
+TTTTTG
+TTTTCA
+TTTTCT
+TTTTCC
+TTTTCG
+TTTTGA
+TTTTGT
+TTTTGC
+TTTTGG
+TTTCAA
+TTTCAT
+TTTCAC
+TTTCAG
+TTTCTA
+TTTCTT
+TTTCTC
+TTTCTG
+TTTCCA
+TTTCCT
+TTTCCC
+TTTCCG
+TTTCGA
+TTTCGT
+TTTCGC
+TTTCGG
+TTTGAA
+TTTGAT
+TTTGAC
+TTTGAG
+TTTGTA
+TTTGTT
+TTTGTC
+TTTGTG
+TTTGCA
+TTTGCT
+TTTGCC
+TTTGCG
+TTTGGA
+TTTGGT
+TTTGGC
+TTTGGG
+TTCAAA
+TTCAAT
+TTCAAC
+TTCAAG
+TTCATA
+TTCATT
+TTCATC
+TTCATG
+TTCACA
+TTCACT
+TTCACC
+TTCACG
+TTCAGA
+TTCAGT
+TTCAGC
+TTCAGG
+TTCTAA
+TTCTAT
+TTCTAC
+TTCTAG
+TTCTTA
+TTCTTT
+TTCTTC
+TTCTTG
+TTCTCA
+TTCTCT
+TTCTCC
+TTCTCG
+TTCTGA
+TTCTGT
+TTCTGC
+TTCTGG
+TTCCAA
+TTCCAT
+TTCCAC
+TTCCAG
+TTCCTA
+TTCCTT
+TTCCTC
+TTCCTG
+TTCCCA
+TTCCCT
+TTCCCC
+TTCCCG
+TTCCGA
+TTCCGT
+TTCCGC
+TTCCGG
+TTCGAA
+TTCGAT
+TTCGAC
+TTCGAG
+TTCGTA
+TTCGTT
+TTCGTC
+TTCGTG
+TTCGCA
+TTCGCT
+TTCGCC
+TTCGCG
+TTCGGA
+TTCGGT
+TTCGGC
+TTCGGG
+TTGAAA
+TTGAAT
+TTGAAC
+TTGAAG
+TTGATA
+TTGATT
+TTGATC
+TTGATG
+TTGACA
+TTGACT
+TTGACC
+TTGACG
+TTGAGA
+TTGAGT
+TTGAGC
+TTGAGG
+TTGTAA
+TTGTAT
+TTGTAC
+TTGTAG
+TTGTTA
+TTGTTT
+TTGTTC
+TTGTTG
+TTGTCA
+TTGTCT
+TTGTCC
+TTGTCG
+TTGTGA
+TTGTGT
+TTGTGC
+TTGTGG
+TTGCAA
+TTGCAT
+TTGCAC
+TTGCAG
+TTGCTA
+TTGCTT
+TTGCTC
+TTGCTG
+TTGCCA
+TTGCCT
+TTGCCC
+TTGCCG
+TTGCGA
+TTGCGT
+TTGCGC
+TTGCGG
+TTGGAA
+TTGGAT
+TTGGAC
+TTGGAG
+TTGGTA
+TTGGTT
+TTGGTC
+TTGGTG
+TTGGCA
+TTGGCT
+TTGGCC
+TTGGCG
+TTGGGA
+TTGGGT
+TTGGGC
+TTGGGG
+TCAAAA
+TCAAAT
+TCAAAC
+TCAAAG
+TCAATA
+TCAATT
+TCAATC
+TCAATG
+TCAACA
+TCAACT
+TCAACC
+TCAACG
+TCAAGA
+TCAAGT
+TCAAGC
+TCAAGG
+TCATAA
+TCATAT
+TCATAC
+TCATAG
+TCATTA
+TCATTT
+TCATTC
+TCATTG
+TCATCA
+TCATCT
+TCATCC
+TCATCG
+TCATGA
+TCATGT
+TCATGC
+TCATGG
+TCACAA
+TCACAT
+TCACAC
+TCACAG
+TCACTA
+TCACTT
+TCACTC
+TCACTG
+TCACCA
+TCACCT
+TCACCC
+TCACCG
+TCACGA
+TCACGT
+TCACGC
+TCACGG
+TCAGAA
+TCAGAT
+TCAGAC
+TCAGAG
+TCAGTA
+TCAGTT
+TCAGTC
+TCAGTG
+TCAGCA
+TCAGCT
+TCAGCC
+TCAGCG
+TCAGGA
+TCAGGT
+TCAGGC
+TCAGGG
+TCTAAA
+TCTAAT
+TCTAAC
+TCTAAG
+TCTATA
+TCTATT
+TCTATC
+TCTATG
+TCTACA
+TCTACT
+TCTACC
+TCTACG
+TCTAGA
+TCTAGT
+TCTAGC
+TCTAGG
+TCTTAA
+TCTTAT
+TCTTAC
+TCTTAG
+TCTTTA
+TCTTTT
+TCTTTC
+TCTTTG
+TCTTCA
+TCTTCT
+TCTTCC
+TCTTCG
+TCTTGA
+TCTTGT
+TCTTGC
+TCTTGG
+TCTCAA
+TCTCAT
+TCTCAC
+TCTCAG
+TCTCTA
+TCTCTT
+TCTCTC
+TCTCTG
+TCTCCA
+TCTCCT
+TCTCCC
+TCTCCG
+TCTCGA
+TCTCGT
+TCTCGC
+TCTCGG
+TCTGAA
+TCTGAT
+TCTGAC
+TCTGAG
+TCTGTA
+TCTGTT
+TCTGTC
+TCTGTG
+TCTGCA
+TCTGCT
+TCTGCC
+TCTGCG
+TCTGGA
+TCTGGT
+TCTGGC
+TCTGGG
+TCCAAA
+TCCAAT
+TCCAAC
+TCCAAG
+TCCATA
+TCCATT
+TCCATC
+TCCATG
+TCCACA
+TCCACT
+TCCACC
+TCCACG
+TCCAGA
+TCCAGT
+TCCAGC
+TCCAGG
+TCCTAA
+TCCTAT
+TCCTAC
+TCCTAG
+TCCTTA
+TCCTTT
+TCCTTC
+TCCTTG
+TCCTCA
+TCCTCT
+TCCTCC
+TCCTCG
+TCCTGA
+TCCTGT
+TCCTGC
+TCCTGG
+TCCCAA
+TCCCAT
+TCCCAC
+TCCCAG
+TCCCTA
+TCCCTT
+TCCCTC
+TCCCTG
+TCCCCA
+TCCCCT
+TCCCCC
+TCCCCG
+TCCCGA
+TCCCGT
+TCCCGC
+TCCCGG
+TCCGAA
+TCCGAT
+TCCGAC
+TCCGAG
+TCCGTA
+TCCGTT
+TCCGTC
+TCCGTG
+TCCGCA
+TCCGCT
+TCCGCC
+TCCGCG
+TCCGGA
+TCCGGT
+TCCGGC
+TCCGGG
+TCGAAA
+TCGAAT
+TCGAAC
+TCGAAG
+TCGATA
+TCGATT
+TCGATC
+TCGATG
+TCGACA
+TCGACT
+TCGACC
+TCGACG
+TCGAGA
+TCGAGT
+TCGAGC
+TCGAGG
+TCGTAA
+TCGTAT
+TCGTAC
+TCGTAG
+TCGTTA
+TCGTTT
+TCGTTC
+TCGTTG
+TCGTCA
+TCGTCT
+TCGTCC
+TCGTCG
+TCGTGA
+TCGTGT
+TCGTGC
+TCGTGG
+TCGCAA
+TCGCAT
+TCGCAC
+TCGCAG
+TCGCTA
+TCGCTT
+TCGCTC
+TCGCTG
+TCGCCA
+TCGCCT
+TCGCCC
+TCGCCG
+TCGCGA
+TCGCGT
+TCGCGC
+TCGCGG
+TCGGAA
+TCGGAT
+TCGGAC
+TCGGAG
+TCGGTA
+TCGGTT
+TCGGTC
+TCGGTG
+TCGGCA
+TCGGCT
+TCGGCC
+TCGGCG
+TCGGGA
+TCGGGT
+TCGGGC
+TCGGGG
+TGAAAA
+TGAAAT
+TGAAAC
+TGAAAG
+TGAATA
+TGAATT
+TGAATC
+TGAATG
+TGAACA
+TGAACT
+TGAACC
+TGAACG
+TGAAGA
+TGAAGT
+TGAAGC
+TGAAGG
+TGATAA
+TGATAT
+TGATAC
+TGATAG
+TGATTA
+TGATTT
+TGATTC
+TGATTG
+TGATCA
+TGATCT
+TGATCC
+TGATCG
+TGATGA
+TGATGT
+TGATGC
+TGATGG
+TGACAA
+TGACAT
+TGACAC
+TGACAG
+TGACTA
+TGACTT
+TGACTC
+TGACTG
+TGACCA
+TGACCT
+TGACCC
+TGACCG
+TGACGA
+TGACGT
+TGACGC
+TGACGG
+TGAGAA
+TGAGAT
+TGAGAC
+TGAGAG
+TGAGTA
+TGAGTT
+TGAGTC
+TGAGTG
+TGAGCA
+TGAGCT
+TGAGCC
+TGAGCG
+TGAGGA
+TGAGGT
+TGAGGC
+TGAGGG
+TGTAAA
+TGTAAT
+TGTAAC
+TGTAAG
+TGTATA
+TGTATT
+TGTATC
+TGTATG
+TGTACA
+TGTACT
+TGTACC
+TGTACG
+TGTAGA
+TGTAGT
+TGTAGC
+TGTAGG
+TGTTAA
+TGTTAT
+TGTTAC
+TGTTAG
+TGTTTA
+TGTTTT
+TGTTTC
+TGTTTG
+TGTTCA
+TGTTCT
+TGTTCC
+TGTTCG
+TGTTGA
+TGTTGT
+TGTTGC
+TGTTGG
+TGTCAA
+TGTCAT
+TGTCAC
+TGTCAG
+TGTCTA
+TGTCTT
+TGTCTC
+TGTCTG
+TGTCCA
+TGTCCT
+TGTCCC
+TGTCCG
+TGTCGA
+TGTCGT
+TGTCGC
+TGTCGG
+TGTGAA
+TGTGAT
+TGTGAC
+TGTGAG
+TGTGTA
+TGTGTT
+TGTGTC
+TGTGTG
+TGTGCA
+TGTGCT
+TGTGCC
+TGTGCG
+TGTGGA
+TGTGGT
+TGTGGC
+TGTGGG
+TGCAAA
+TGCAAT
+TGCAAC
+TGCAAG
+TGCATA
+TGCATT
+TGCATC
+TGCATG
+TGCACA
+TGCACT
+TGCACC
+TGCACG
+TGCAGA
+TGCAGT
+TGCAGC
+TGCAGG
+TGCTAA
+TGCTAT
+TGCTAC
+TGCTAG
+TGCTTA
+TGCTTT
+TGCTTC
+TGCTTG
+TGCTCA
+TGCTCT
+TGCTCC
+TGCTCG
+TGCTGA
+TGCTGT
+TGCTGC
+TGCTGG
+TGCCAA
+TGCCAT
+TGCCAC
+TGCCAG
+TGCCTA
+TGCCTT
+TGCCTC
+TGCCTG
+TGCCCA
+TGCCCT
+TGCCCC
+TGCCCG
+TGCCGA
+TGCCGT
+TGCCGC
+TGCCGG
+TGCGAA
+TGCGAT
+TGCGAC
+TGCGAG
+TGCGTA
+TGCGTT
+TGCGTC
+TGCGTG
+TGCGCA
+TGCGCT
+TGCGCC
+TGCGCG
+TGCGGA
+TGCGGT
+TGCGGC
+TGCGGG
+TGGAAA
+TGGAAT
+TGGAAC
+TGGAAG
+TGGATA
+TGGATT
+TGGATC
+TGGATG
+TGGACA
+TGGACT
+TGGACC
+TGGACG
+TGGAGA
+TGGAGT
+TGGAGC
+TGGAGG
+TGGTAA
+TGGTAT
+TGGTAC
+TGGTAG
+TGGTTA
+TGGTTT
+TGGTTC
+TGGTTG
+TGGTCA
+TGGTCT
+TGGTCC
+TGGTCG
+TGGTGA
+TGGTGT
+TGGTGC
+TGGTGG
+TGGCAA
+TGGCAT
+TGGCAC
+TGGCAG
+TGGCTA
+TGGCTT
+TGGCTC
+TGGCTG
+TGGCCA
+TGGCCT
+TGGCCC
+TGGCCG
+TGGCGA
+TGGCGT
+TGGCGC
+TGGCGG
+TGGGAA
+TGGGAT
+TGGGAC
+TGGGAG
+TGGGTA
+TGGGTT
+TGGGTC
+TGGGTG
+TGGGCA
+TGGGCT
+TGGGCC
+TGGGCG
+TGGGGA
+TGGGGT
+TGGGGC
+TGGGGG
+CAAAAA
+CAAAAT
+CAAAAC
+CAAAAG
+CAAATA
+CAAATT
+CAAATC
+CAAATG
+CAAACA
+CAAACT
+CAAACC
+CAAACG
+CAAAGA
+CAAAGT
+CAAAGC
+CAAAGG
+CAATAA
+CAATAT
+CAATAC
+CAATAG
+CAATTA
+CAATTT
+CAATTC
+CAATTG
+CAATCA
+CAATCT
+CAATCC
+CAATCG
+CAATGA
+CAATGT
+CAATGC
+CAATGG
+CAACAA
+CAACAT
+CAACAC
+CAACAG
+CAACTA
+CAACTT
+CAACTC
+CAACTG
+CAACCA
+CAACCT
+CAACCC
+CAACCG
+CAACGA
+CAACGT
+CAACGC
+CAACGG
+CAAGAA
+CAAGAT
+CAAGAC
+CAAGAG
+CAAGTA
+CAAGTT
+CAAGTC
+CAAGTG
+CAAGCA
+CAAGCT
+CAAGCC
+CAAGCG
+CAAGGA
+CAAGGT
+CAAGGC
+CAAGGG
+CATAAA
+CATAAT
+CATAAC
+CATAAG
+CATATA
+CATATT
+CATATC
+CATATG
+CATACA
+CATACT
+CATACC
+CATACG
+CATAGA
+CATAGT
+CATAGC
+CATAGG
+CATTAA
+CATTAT
+CATTAC
+CATTAG
+CATTTA
+CATTTT
+CATTTC
+CATTTG
+CATTCA
+CATTCT
+CATTCC
+CATTCG
+CATTGA
+CATTGT
+CATTGC
+CATTGG
+CATCAA
+CATCAT
+CATCAC
+CATCAG
+CATCTA
+CATCTT
+CATCTC
+CATCTG
+CATCCA
+CATCCT
+CATCCC
+CATCCG
+CATCGA
+CATCGT
+CATCGC
+CATCGG
+CATGAA
+CATGAT
+CATGAC
+CATGAG
+CATGTA
+CATGTT
+CATGTC
+CATGTG
+CATGCA
+CATGCT
+CATGCC
+CATGCG
+CATGGA
+CATGGT
+CATGGC
+CATGGG
+CACAAA
+CACAAT
+CACAAC
+CACAAG
+CACATA
+CACATT
+CACATC
+CACATG
+CACACA
+CACACT
+CACACC
+CACACG
+CACAGA
+CACAGT
+CACAGC
+CACAGG
+CACTAA
+CACTAT
+CACTAC
+CACTAG
+CACTTA
+CACTTT
+CACTTC
+CACTTG
+CACTCA
+CACTCT
+CACTCC
+CACTCG
+CACTGA
+CACTGT
+CACTGC
+CACTGG
+CACCAA
+CACCAT
+CACCAC
+CACCAG
+CACCTA
+CACCTT
+CACCTC
+CACCTG
+CACCCA
+CACCCT
+CACCCC
+CACCCG
+CACCGA
+CACCGT
+CACCGC
+CACCGG
+CACGAA
+CACGAT
+CACGAC
+CACGAG
+CACGTA
+CACGTT
+CACGTC
+CACGTG
+CACGCA
+CACGCT
+CACGCC
+CACGCG
+CACGGA
+CACGGT
+CACGGC
+CACGGG
+CAGAAA
+CAGAAT
+CAGAAC
+CAGAAG
+CAGATA
+CAGATT
+CAGATC
+CAGATG
+CAGACA
+CAGACT
+CAGACC
+CAGACG
+CAGAGA
+CAGAGT
+CAGAGC
+CAGAGG
+CAGTAA
+CAGTAT
+CAGTAC
+CAGTAG
+CAGTTA
+CAGTTT
+CAGTTC
+CAGTTG
+CAGTCA
+CAGTCT
+CAGTCC
+CAGTCG
+CAGTGA
+CAGTGT
+CAGTGC
+CAGTGG
+CAGCAA
+CAGCAT
+CAGCAC
+CAGCAG
+CAGCTA
+CAGCTT
+CAGCTC
+CAGCTG
+CAGCCA
+CAGCCT
+CAGCCC
+CAGCCG
+CAGCGA
+CAGCGT
+CAGCGC
+CAGCGG
+CAGGAA
+CAGGAT
+CAGGAC
+CAGGAG
+CAGGTA
+CAGGTT
+CAGGTC
+CAGGTG
+CAGGCA
+CAGGCT
+CAGGCC
+CAGGCG
+CAGGGA
+CAGGGT
+CAGGGC
+CAGGGG
+CTAAAA
+CTAAAT
+CTAAAC
+CTAAAG
+CTAATA
+CTAATT
+CTAATC
+CTAATG
+CTAACA
+CTAACT
+CTAACC
+CTAACG
+CTAAGA
+CTAAGT
+CTAAGC
+CTAAGG
+CTATAA
+CTATAT
+CTATAC
+CTATAG
+CTATTA
+CTATTT
+CTATTC
+CTATTG
+CTATCA
+CTATCT
+CTATCC
+CTATCG
+CTATGA
+CTATGT
+CTATGC
+CTATGG
+CTACAA
+CTACAT
+CTACAC
+CTACAG
+CTACTA
+CTACTT
+CTACTC
+CTACTG
+CTACCA
+CTACCT
+CTACCC
+CTACCG
+CTACGA
+CTACGT
+CTACGC
+CTACGG
+CTAGAA
+CTAGAT
+CTAGAC
+CTAGAG
+CTAGTA
+CTAGTT
+CTAGTC
+CTAGTG
+CTAGCA
+CTAGCT
+CTAGCC
+CTAGCG
+CTAGGA
+CTAGGT
+CTAGGC
+CTAGGG
+CTTAAA
+CTTAAT
+CTTAAC
+CTTAAG
+CTTATA
+CTTATT
+CTTATC
+CTTATG
+CTTACA
+CTTACT
+CTTACC
+CTTACG
+CTTAGA
+CTTAGT
+CTTAGC
+CTTAGG
+CTTTAA
+CTTTAT
+CTTTAC
+CTTTAG
+CTTTTA
+CTTTTT
+CTTTTC
+CTTTTG
+CTTTCA
+CTTTCT
+CTTTCC
+CTTTCG
+CTTTGA
+CTTTGT
+CTTTGC
+CTTTGG
+CTTCAA
+CTTCAT
+CTTCAC
+CTTCAG
+CTTCTA
+CTTCTT
+CTTCTC
+CTTCTG
+CTTCCA
+CTTCCT
+CTTCCC
+CTTCCG
+CTTCGA
+CTTCGT
+CTTCGC
+CTTCGG
+CTTGAA
+CTTGAT
+CTTGAC
+CTTGAG
+CTTGTA
+CTTGTT
+CTTGTC
+CTTGTG
+CTTGCA
+CTTGCT
+CTTGCC
+CTTGCG
+CTTGGA
+CTTGGT
+CTTGGC
+CTTGGG
+CTCAAA
+CTCAAT
+CTCAAC
+CTCAAG
+CTCATA
+CTCATT
+CTCATC
+CTCATG
+CTCACA
+CTCACT
+CTCACC
+CTCACG
+CTCAGA
+CTCAGT
+CTCAGC
+CTCAGG
+CTCTAA
+CTCTAT
+CTCTAC
+CTCTAG
+CTCTTA
+CTCTTT
+CTCTTC
+CTCTTG
+CTCTCA
+CTCTCT
+CTCTCC
+CTCTCG
+CTCTGA
+CTCTGT
+CTCTGC
+CTCTGG
+CTCCAA
+CTCCAT
+CTCCAC
+CTCCAG
+CTCCTA
+CTCCTT
+CTCCTC
+CTCCTG
+CTCCCA
+CTCCCT
+CTCCCC
+CTCCCG
+CTCCGA
+CTCCGT
+CTCCGC
+CTCCGG
+CTCGAA
+CTCGAT
+CTCGAC
+CTCGAG
+CTCGTA
+CTCGTT
+CTCGTC
+CTCGTG
+CTCGCA
+CTCGCT
+CTCGCC
+CTCGCG
+CTCGGA
+CTCGGT
+CTCGGC
+CTCGGG
+CTGAAA
+CTGAAT
+CTGAAC
+CTGAAG
+CTGATA
+CTGATT
+CTGATC
+CTGATG
+CTGACA
+CTGACT
+CTGACC
+CTGACG
+CTGAGA
+CTGAGT
+CTGAGC
+CTGAGG
+CTGTAA
+CTGTAT
+CTGTAC
+CTGTAG
+CTGTTA
+CTGTTT
+CTGTTC
+CTGTTG
+CTGTCA
+CTGTCT
+CTGTCC
+CTGTCG
+CTGTGA
+CTGTGT
+CTGTGC
+CTGTGG
+CTGCAA
+CTGCAT
+CTGCAC
+CTGCAG
+CTGCTA
+CTGCTT
+CTGCTC
+CTGCTG
+CTGCCA
+CTGCCT
+CTGCCC
+CTGCCG
+CTGCGA
+CTGCGT
+CTGCGC
+CTGCGG
+CTGGAA
+CTGGAT
+CTGGAC
+CTGGAG
+CTGGTA
+CTGGTT
+CTGGTC
+CTGGTG
+CTGGCA
+CTGGCT
+CTGGCC
+CTGGCG
+CTGGGA
+CTGGGT
+CTGGGC
+CTGGGG
+CCAAAA
+CCAAAT
+CCAAAC
+CCAAAG
+CCAATA
+CCAATT
+CCAATC
+CCAATG
+CCAACA
+CCAACT
+CCAACC
+CCAACG
+CCAAGA
+CCAAGT
+CCAAGC
+CCAAGG
+CCATAA
+CCATAT
+CCATAC
+CCATAG
+CCATTA
+CCATTT
+CCATTC
+CCATTG
+CCATCA
+CCATCT
+CCATCC
+CCATCG
+CCATGA
+CCATGT
+CCATGC
+CCATGG
+CCACAA
+CCACAT
+CCACAC
+CCACAG
+CCACTA
+CCACTT
+CCACTC
+CCACTG
+CCACCA
+CCACCT
+CCACCC
+CCACCG
+CCACGA
+CCACGT
+CCACGC
+CCACGG
+CCAGAA
+CCAGAT
+CCAGAC
+CCAGAG
+CCAGTA
+CCAGTT
+CCAGTC
+CCAGTG
+CCAGCA
+CCAGCT
+CCAGCC
+CCAGCG
+CCAGGA
+CCAGGT
+CCAGGC
+CCAGGG
+CCTAAA
+CCTAAT
+CCTAAC
+CCTAAG
+CCTATA
+CCTATT
+CCTATC
+CCTATG
+CCTACA
+CCTACT
+CCTACC
+CCTACG
+CCTAGA
+CCTAGT
+CCTAGC
+CCTAGG
+CCTTAA
+CCTTAT
+CCTTAC
+CCTTAG
+CCTTTA
+CCTTTT
+CCTTTC
+CCTTTG
+CCTTCA
+CCTTCT
+CCTTCC
+CCTTCG
+CCTTGA
+CCTTGT
+CCTTGC
+CCTTGG
+CCTCAA
+CCTCAT
+CCTCAC
+CCTCAG
+CCTCTA
+CCTCTT
+CCTCTC
+CCTCTG
+CCTCCA
+CCTCCT
+CCTCCC
+CCTCCG
+CCTCGA
+CCTCGT
+CCTCGC
+CCTCGG
+CCTGAA
+CCTGAT
+CCTGAC
+CCTGAG
+CCTGTA
+CCTGTT
+CCTGTC
+CCTGTG
+CCTGCA
+CCTGCT
+CCTGCC
+CCTGCG
+CCTGGA
+CCTGGT
+CCTGGC
+CCTGGG
+CCCAAA
+CCCAAT
+CCCAAC
+CCCAAG
+CCCATA
+CCCATT
+CCCATC
+CCCATG
+CCCACA
+CCCACT
+CCCACC
+CCCACG
+CCCAGA
+CCCAGT
+CCCAGC
+CCCAGG
+CCCTAA
+CCCTAT
+CCCTAC
+CCCTAG
+CCCTTA
+CCCTTT
+CCCTTC
+CCCTTG
+CCCTCA
+CCCTCT
+CCCTCC
+CCCTCG
+CCCTGA
+CCCTGT
+CCCTGC
+CCCTGG
+CCCCAA
+CCCCAT
+CCCCAC
+CCCCAG
+CCCCTA
+CCCCTT
+CCCCTC
+CCCCTG
+CCCCCA
+CCCCCT
+CCCCCC
+CCCCCG
+CCCCGA
+CCCCGT
+CCCCGC
+CCCCGG
+CCCGAA
+CCCGAT
+CCCGAC
+CCCGAG
+CCCGTA
+CCCGTT
+CCCGTC
+CCCGTG
+CCCGCA
+CCCGCT
+CCCGCC
+CCCGCG
+CCCGGA
+CCCGGT
+CCCGGC
+CCCGGG
+CCGAAA
+CCGAAT
+CCGAAC
+CCGAAG
+CCGATA
+CCGATT
+CCGATC
+CCGATG
+CCGACA
+CCGACT
+CCGACC
+CCGACG
+CCGAGA
+CCGAGT
+CCGAGC
+CCGAGG
+CCGTAA
+CCGTAT
+CCGTAC
+CCGTAG
+CCGTTA
+CCGTTT
+CCGTTC
+CCGTTG
+CCGTCA
+CCGTCT
+CCGTCC
+CCGTCG
+CCGTGA
+CCGTGT
+CCGTGC
+CCGTGG
+CCGCAA
+CCGCAT
+CCGCAC
+CCGCAG
+CCGCTA
+CCGCTT
+CCGCTC
+CCGCTG
+CCGCCA
+CCGCCT
+CCGCCC
+CCGCCG
+CCGCGA
+CCGCGT
+CCGCGC
+CCGCGG
+CCGGAA
+CCGGAT
+CCGGAC
+CCGGAG
+CCGGTA
+CCGGTT
+CCGGTC
+CCGGTG
+CCGGCA
+CCGGCT
+CCGGCC
+CCGGCG
+CCGGGA
+CCGGGT
+CCGGGC
+CCGGGG
+CGAAAA
+CGAAAT
+CGAAAC
+CGAAAG
+CGAATA
+CGAATT
+CGAATC
+CGAATG
+CGAACA
+CGAACT
+CGAACC
+CGAACG
+CGAAGA
+CGAAGT
+CGAAGC
+CGAAGG
+CGATAA
+CGATAT
+CGATAC
+CGATAG
+CGATTA
+CGATTT
+CGATTC
+CGATTG
+CGATCA
+CGATCT
+CGATCC
+CGATCG
+CGATGA
+CGATGT
+CGATGC
+CGATGG
+CGACAA
+CGACAT
+CGACAC
+CGACAG
+CGACTA
+CGACTT
+CGACTC
+CGACTG
+CGACCA
+CGACCT
+CGACCC
+CGACCG
+CGACGA
+CGACGT
+CGACGC
+CGACGG
+CGAGAA
+CGAGAT
+CGAGAC
+CGAGAG
+CGAGTA
+CGAGTT
+CGAGTC
+CGAGTG
+CGAGCA
+CGAGCT
+CGAGCC
+CGAGCG
+CGAGGA
+CGAGGT
+CGAGGC
+CGAGGG
+CGTAAA
+CGTAAT
+CGTAAC
+CGTAAG
+CGTATA
+CGTATT
+CGTATC
+CGTATG
+CGTACA
+CGTACT
+CGTACC
+CGTACG
+CGTAGA
+CGTAGT
+CGTAGC
+CGTAGG
+CGTTAA
+CGTTAT
+CGTTAC
+CGTTAG
+CGTTTA
+CGTTTT
+CGTTTC
+CGTTTG
+CGTTCA
+CGTTCT
+CGTTCC
+CGTTCG
+CGTTGA
+CGTTGT
+CGTTGC
+CGTTGG
+CGTCAA
+CGTCAT
+CGTCAC
+CGTCAG
+CGTCTA
+CGTCTT
+CGTCTC
+CGTCTG
+CGTCCA
+CGTCCT
+CGTCCC
+CGTCCG
+CGTCGA
+CGTCGT
+CGTCGC
+CGTCGG
+CGTGAA
+CGTGAT
+CGTGAC
+CGTGAG
+CGTGTA
+CGTGTT
+CGTGTC
+CGTGTG
+CGTGCA
+CGTGCT
+CGTGCC
+CGTGCG
+CGTGGA
+CGTGGT
+CGTGGC
+CGTGGG
+CGCAAA
+CGCAAT
+CGCAAC
+CGCAAG
+CGCATA
+CGCATT
+CGCATC
+CGCATG
+CGCACA
+CGCACT
+CGCACC
+CGCACG
+CGCAGA
+CGCAGT
+CGCAGC
+CGCAGG
+CGCTAA
+CGCTAT
+CGCTAC
+CGCTAG
+CGCTTA
+CGCTTT
+CGCTTC
+CGCTTG
+CGCTCA
+CGCTCT
+CGCTCC
+CGCTCG
+CGCTGA
+CGCTGT
+CGCTGC
+CGCTGG
+CGCCAA
+CGCCAT
+CGCCAC
+CGCCAG
+CGCCTA
+CGCCTT
+CGCCTC
+CGCCTG
+CGCCCA
+CGCCCT
+CGCCCC
+CGCCCG
+CGCCGA
+CGCCGT
+CGCCGC
+CGCCGG
+CGCGAA
+CGCGAT
+CGCGAC
+CGCGAG
+CGCGTA
+CGCGTT
+CGCGTC
+CGCGTG
+CGCGCA
+CGCGCT
+CGCGCC
+CGCGCG
+CGCGGA
+CGCGGT
+CGCGGC
+CGCGGG
+CGGAAA
+CGGAAT
+CGGAAC
+CGGAAG
+CGGATA
+CGGATT
+CGGATC
+CGGATG
+CGGACA
+CGGACT
+CGGACC
+CGGACG
+CGGAGA
+CGGAGT
+CGGAGC
+CGGAGG
+CGGTAA
+CGGTAT
+CGGTAC
+CGGTAG
+CGGTTA
+CGGTTT
+CGGTTC
+CGGTTG
+CGGTCA
+CGGTCT
+CGGTCC
+CGGTCG
+CGGTGA
+CGGTGT
+CGGTGC
+CGGTGG
+CGGCAA
+CGGCAT
+CGGCAC
+CGGCAG
+CGGCTA
+CGGCTT
+CGGCTC
+CGGCTG
+CGGCCA
+CGGCCT
+CGGCCC
+CGGCCG
+CGGCGA
+CGGCGT
+CGGCGC
+CGGCGG
+CGGGAA
+CGGGAT
+CGGGAC
+CGGGAG
+CGGGTA
+CGGGTT
+CGGGTC
+CGGGTG
+CGGGCA
+CGGGCT
+CGGGCC
+CGGGCG
+CGGGGA
+CGGGGT
+CGGGGC
+CGGGGG
+GAAAAA
+GAAAAT
+GAAAAC
+GAAAAG
+GAAATA
+GAAATT
+GAAATC
+GAAATG
+GAAACA
+GAAACT
+GAAACC
+GAAACG
+GAAAGA
+GAAAGT
+GAAAGC
+GAAAGG
+GAATAA
+GAATAT
+GAATAC
+GAATAG
+GAATTA
+GAATTT
+GAATTC
+GAATTG
+GAATCA
+GAATCT
+GAATCC
+GAATCG
+GAATGA
+GAATGT
+GAATGC
+GAATGG
+GAACAA
+GAACAT
+GAACAC
+GAACAG
+GAACTA
+GAACTT
+GAACTC
+GAACTG
+GAACCA
+GAACCT
+GAACCC
+GAACCG
+GAACGA
+GAACGT
+GAACGC
+GAACGG
+GAAGAA
+GAAGAT
+GAAGAC
+GAAGAG
+GAAGTA
+GAAGTT
+GAAGTC
+GAAGTG
+GAAGCA
+GAAGCT
+GAAGCC
+GAAGCG
+GAAGGA
+GAAGGT
+GAAGGC
+GAAGGG
+GATAAA
+GATAAT
+GATAAC
+GATAAG
+GATATA
+GATATT
+GATATC
+GATATG
+GATACA
+GATACT
+GATACC
+GATACG
+GATAGA
+GATAGT
+GATAGC
+GATAGG
+GATTAA
+GATTAT
+GATTAC
+GATTAG
+GATTTA
+GATTTT
+GATTTC
+GATTTG
+GATTCA
+GATTCT
+GATTCC
+GATTCG
+GATTGA
+GATTGT
+GATTGC
+GATTGG
+GATCAA
+GATCAT
+GATCAC
+GATCAG
+GATCTA
+GATCTT
+GATCTC
+GATCTG
+GATCCA
+GATCCT
+GATCCC
+GATCCG
+GATCGA
+GATCGT
+GATCGC
+GATCGG
+GATGAA
+GATGAT
+GATGAC
+GATGAG
+GATGTA
+GATGTT
+GATGTC
+GATGTG
+GATGCA
+GATGCT
+GATGCC
+GATGCG
+GATGGA
+GATGGT
+GATGGC
+GATGGG
+GACAAA
+GACAAT
+GACAAC
+GACAAG
+GACATA
+GACATT
+GACATC
+GACATG
+GACACA
+GACACT
+GACACC
+GACACG
+GACAGA
+GACAGT
+GACAGC
+GACAGG
+GACTAA
+GACTAT
+GACTAC
+GACTAG
+GACTTA
+GACTTT
+GACTTC
+GACTTG
+GACTCA
+GACTCT
+GACTCC
+GACTCG
+GACTGA
+GACTGT
+GACTGC
+GACTGG
+GACCAA
+GACCAT
+GACCAC
+GACCAG
+GACCTA
+GACCTT
+GACCTC
+GACCTG
+GACCCA
+GACCCT
+GACCCC
+GACCCG
+GACCGA
+GACCGT
+GACCGC
+GACCGG
+GACGAA
+GACGAT
+GACGAC
+GACGAG
+GACGTA
+GACGTT
+GACGTC
+GACGTG
+GACGCA
+GACGCT
+GACGCC
+GACGCG
+GACGGA
+GACGGT
+GACGGC
+GACGGG
+GAGAAA
+GAGAAT
+GAGAAC
+GAGAAG
+GAGATA
+GAGATT
+GAGATC
+GAGATG
+GAGACA
+GAGACT
+GAGACC
+GAGACG
+GAGAGA
+GAGAGT
+GAGAGC
+GAGAGG
+GAGTAA
+GAGTAT
+GAGTAC
+GAGTAG
+GAGTTA
+GAGTTT
+GAGTTC
+GAGTTG
+GAGTCA
+GAGTCT
+GAGTCC
+GAGTCG
+GAGTGA
+GAGTGT
+GAGTGC
+GAGTGG
+GAGCAA
+GAGCAT
+GAGCAC
+GAGCAG
+GAGCTA
+GAGCTT
+GAGCTC
+GAGCTG
+GAGCCA
+GAGCCT
+GAGCCC
+GAGCCG
+GAGCGA
+GAGCGT
+GAGCGC
+GAGCGG
+GAGGAA
+GAGGAT
+GAGGAC
+GAGGAG
+GAGGTA
+GAGGTT
+GAGGTC
+GAGGTG
+GAGGCA
+GAGGCT
+GAGGCC
+GAGGCG
+GAGGGA
+GAGGGT
+GAGGGC
+GAGGGG
+GTAAAA
+GTAAAT
+GTAAAC
+GTAAAG
+GTAATA
+GTAATT
+GTAATC
+GTAATG
+GTAACA
+GTAACT
+GTAACC
+GTAACG
+GTAAGA
+GTAAGT
+GTAAGC
+GTAAGG
+GTATAA
+GTATAT
+GTATAC
+GTATAG
+GTATTA
+GTATTT
+GTATTC
+GTATTG
+GTATCA
+GTATCT
+GTATCC
+GTATCG
+GTATGA
+GTATGT
+GTATGC
+GTATGG
+GTACAA
+GTACAT
+GTACAC
+GTACAG
+GTACTA
+GTACTT
+GTACTC
+GTACTG
+GTACCA
+GTACCT
+GTACCC
+GTACCG
+GTACGA
+GTACGT
+GTACGC
+GTACGG
+GTAGAA
+GTAGAT
+GTAGAC
+GTAGAG
+GTAGTA
+GTAGTT
+GTAGTC
+GTAGTG
+GTAGCA
+GTAGCT
+GTAGCC
+GTAGCG
+GTAGGA
+GTAGGT
+GTAGGC
+GTAGGG
+GTTAAA
+GTTAAT
+GTTAAC
+GTTAAG
+GTTATA
+GTTATT
+GTTATC
+GTTATG
+GTTACA
+GTTACT
+GTTACC
+GTTACG
+GTTAGA
+GTTAGT
+GTTAGC
+GTTAGG
+GTTTAA
+GTTTAT
+GTTTAC
+GTTTAG
+GTTTTA
+GTTTTT
+GTTTTC
+GTTTTG
+GTTTCA
+GTTTCT
+GTTTCC
+GTTTCG
+GTTTGA
+GTTTGT
+GTTTGC
+GTTTGG
+GTTCAA
+GTTCAT
+GTTCAC
+GTTCAG
+GTTCTA
+GTTCTT
+GTTCTC
+GTTCTG
+GTTCCA
+GTTCCT
+GTTCCC
+GTTCCG
+GTTCGA
+GTTCGT
+GTTCGC
+GTTCGG
+GTTGAA
+GTTGAT
+GTTGAC
+GTTGAG
+GTTGTA
+GTTGTT
+GTTGTC
+GTTGTG
+GTTGCA
+GTTGCT
+GTTGCC
+GTTGCG
+GTTGGA
+GTTGGT
+GTTGGC
+GTTGGG
+GTCAAA
+GTCAAT
+GTCAAC
+GTCAAG
+GTCATA
+GTCATT
+GTCATC
+GTCATG
+GTCACA
+GTCACT
+GTCACC
+GTCACG
+GTCAGA
+GTCAGT
+GTCAGC
+GTCAGG
+GTCTAA
+GTCTAT
+GTCTAC
+GTCTAG
+GTCTTA
+GTCTTT
+GTCTTC
+GTCTTG
+GTCTCA
+GTCTCT
+GTCTCC
+GTCTCG
+GTCTGA
+GTCTGT
+GTCTGC
+GTCTGG
+GTCCAA
+GTCCAT
+GTCCAC
+GTCCAG
+GTCCTA
+GTCCTT
+GTCCTC
+GTCCTG
+GTCCCA
+GTCCCT
+GTCCCC
+GTCCCG
+GTCCGA
+GTCCGT
+GTCCGC
+GTCCGG
+GTCGAA
+GTCGAT
+GTCGAC
+GTCGAG
+GTCGTA
+GTCGTT
+GTCGTC
+GTCGTG
+GTCGCA
+GTCGCT
+GTCGCC
+GTCGCG
+GTCGGA
+GTCGGT
+GTCGGC
+GTCGGG
+GTGAAA
+GTGAAT
+GTGAAC
+GTGAAG
+GTGATA
+GTGATT
+GTGATC
+GTGATG
+GTGACA
+GTGACT
+GTGACC
+GTGACG
+GTGAGA
+GTGAGT
+GTGAGC
+GTGAGG
+GTGTAA
+GTGTAT
+GTGTAC
+GTGTAG
+GTGTTA
+GTGTTT
+GTGTTC
+GTGTTG
+GTGTCA
+GTGTCT
+GTGTCC
+GTGTCG
+GTGTGA
+GTGTGT
+GTGTGC
+GTGTGG
+GTGCAA
+GTGCAT
+GTGCAC
+GTGCAG
+GTGCTA
+GTGCTT
+GTGCTC
+GTGCTG
+GTGCCA
+GTGCCT
+GTGCCC
+GTGCCG
+GTGCGA
+GTGCGT
+GTGCGC
+GTGCGG
+GTGGAA
+GTGGAT
+GTGGAC
+GTGGAG
+GTGGTA
+GTGGTT
+GTGGTC
+GTGGTG
+GTGGCA
+GTGGCT
+GTGGCC
+GTGGCG
+GTGGGA
+GTGGGT
+GTGGGC
+GTGGGG
+GCAAAA
+GCAAAT
+GCAAAC
+GCAAAG
+GCAATA
+GCAATT
+GCAATC
+GCAATG
+GCAACA
+GCAACT
+GCAACC
+GCAACG
+GCAAGA
+GCAAGT
+GCAAGC
+GCAAGG
+GCATAA
+GCATAT
+GCATAC
+GCATAG
+GCATTA
+GCATTT
+GCATTC
+GCATTG
+GCATCA
+GCATCT
+GCATCC
+GCATCG
+GCATGA
+GCATGT
+GCATGC
+GCATGG
+GCACAA
+GCACAT
+GCACAC
+GCACAG
+GCACTA
+GCACTT
+GCACTC
+GCACTG
+GCACCA
+GCACCT
+GCACCC
+GCACCG
+GCACGA
+GCACGT
+GCACGC
+GCACGG
+GCAGAA
+GCAGAT
+GCAGAC
+GCAGAG
+GCAGTA
+GCAGTT
+GCAGTC
+GCAGTG
+GCAGCA
+GCAGCT
+GCAGCC
+GCAGCG
+GCAGGA
+GCAGGT
+GCAGGC
+GCAGGG
+GCTAAA
+GCTAAT
+GCTAAC
+GCTAAG
+GCTATA
+GCTATT
+GCTATC
+GCTATG
+GCTACA
+GCTACT
+GCTACC
+GCTACG
+GCTAGA
+GCTAGT
+GCTAGC
+GCTAGG
+GCTTAA
+GCTTAT
+GCTTAC
+GCTTAG
+GCTTTA
+GCTTTT
+GCTTTC
+GCTTTG
+GCTTCA
+GCTTCT
+GCTTCC
+GCTTCG
+GCTTGA
+GCTTGT
+GCTTGC
+GCTTGG
+GCTCAA
+GCTCAT
+GCTCAC
+GCTCAG
+GCTCTA
+GCTCTT
+GCTCTC
+GCTCTG
+GCTCCA
+GCTCCT
+GCTCCC
+GCTCCG
+GCTCGA
+GCTCGT
+GCTCGC
+GCTCGG
+GCTGAA
+GCTGAT
+GCTGAC
+GCTGAG
+GCTGTA
+GCTGTT
+GCTGTC
+GCTGTG
+GCTGCA
+GCTGCT
+GCTGCC
+GCTGCG
+GCTGGA
+GCTGGT
+GCTGGC
+GCTGGG
+GCCAAA
+GCCAAT
+GCCAAC
+GCCAAG
+GCCATA
+GCCATT
+GCCATC
+GCCATG
+GCCACA
+GCCACT
+GCCACC
+GCCACG
+GCCAGA
+GCCAGT
+GCCAGC
+GCCAGG
+GCCTAA
+GCCTAT
+GCCTAC
+GCCTAG
+GCCTTA
+GCCTTT
+GCCTTC
+GCCTTG
+GCCTCA
+GCCTCT
+GCCTCC
+GCCTCG
+GCCTGA
+GCCTGT
+GCCTGC
+GCCTGG
+GCCCAA
+GCCCAT
+GCCCAC
+GCCCAG
+GCCCTA
+GCCCTT
+GCCCTC
+GCCCTG
+GCCCCA
+GCCCCT
+GCCCCC
+GCCCCG
+GCCCGA
+GCCCGT
+GCCCGC
+GCCCGG
+GCCGAA
+GCCGAT
+GCCGAC
+GCCGAG
+GCCGTA
+GCCGTT
+GCCGTC
+GCCGTG
+GCCGCA
+GCCGCT
+GCCGCC
+GCCGCG
+GCCGGA
+GCCGGT
+GCCGGC
+GCCGGG
+GCGAAA
+GCGAAT
+GCGAAC
+GCGAAG
+GCGATA
+GCGATT
+GCGATC
+GCGATG
+GCGACA
+GCGACT
+GCGACC
+GCGACG
+GCGAGA
+GCGAGT
+GCGAGC
+GCGAGG
+GCGTAA
+GCGTAT
+GCGTAC
+GCGTAG
+GCGTTA
+GCGTTT
+GCGTTC
+GCGTTG
+GCGTCA
+GCGTCT
+GCGTCC
+GCGTCG
+GCGTGA
+GCGTGT
+GCGTGC
+GCGTGG
+GCGCAA
+GCGCAT
+GCGCAC
+GCGCAG
+GCGCTA
+GCGCTT
+GCGCTC
+GCGCTG
+GCGCCA
+GCGCCT
+GCGCCC
+GCGCCG
+GCGCGA
+GCGCGT
+GCGCGC
+GCGCGG
+GCGGAA
+GCGGAT
+GCGGAC
+GCGGAG
+GCGGTA
+GCGGTT
+GCGGTC
+GCGGTG
+GCGGCA
+GCGGCT
+GCGGCC
+GCGGCG
+GCGGGA
+GCGGGT
+GCGGGC
+GCGGGG
+GGAAAA
+GGAAAT
+GGAAAC
+GGAAAG
+GGAATA
+GGAATT
+GGAATC
+GGAATG
+GGAACA
+GGAACT
+GGAACC
+GGAACG
+GGAAGA
+GGAAGT
+GGAAGC
+GGAAGG
+GGATAA
+GGATAT
+GGATAC
+GGATAG
+GGATTA
+GGATTT
+GGATTC
+GGATTG
+GGATCA
+GGATCT
+GGATCC
+GGATCG
+GGATGA
+GGATGT
+GGATGC
+GGATGG
+GGACAA
+GGACAT
+GGACAC
+GGACAG
+GGACTA
+GGACTT
+GGACTC
+GGACTG
+GGACCA
+GGACCT
+GGACCC
+GGACCG
+GGACGA
+GGACGT
+GGACGC
+GGACGG
+GGAGAA
+GGAGAT
+GGAGAC
+GGAGAG
+GGAGTA
+GGAGTT
+GGAGTC
+GGAGTG
+GGAGCA
+GGAGCT
+GGAGCC
+GGAGCG
+GGAGGA
+GGAGGT
+GGAGGC
+GGAGGG
+GGTAAA
+GGTAAT
+GGTAAC
+GGTAAG
+GGTATA
+GGTATT
+GGTATC
+GGTATG
+GGTACA
+GGTACT
+GGTACC
+GGTACG
+GGTAGA
+GGTAGT
+GGTAGC
+GGTAGG
+GGTTAA
+GGTTAT
+GGTTAC
+GGTTAG
+GGTTTA
+GGTTTT
+GGTTTC
+GGTTTG
+GGTTCA
+GGTTCT
+GGTTCC
+GGTTCG
+GGTTGA
+GGTTGT
+GGTTGC
+GGTTGG
+GGTCAA
+GGTCAT
+GGTCAC
+GGTCAG
+GGTCTA
+GGTCTT
+GGTCTC
+GGTCTG
+GGTCCA
+GGTCCT
+GGTCCC
+GGTCCG
+GGTCGA
+GGTCGT
+GGTCGC
+GGTCGG
+GGTGAA
+GGTGAT
+GGTGAC
+GGTGAG
+GGTGTA
+GGTGTT
+GGTGTC
+GGTGTG
+GGTGCA
+GGTGCT
+GGTGCC
+GGTGCG
+GGTGGA
+GGTGGT
+GGTGGC
+GGTGGG
+GGCAAA
+GGCAAT
+GGCAAC
+GGCAAG
+GGCATA
+GGCATT
+GGCATC
+GGCATG
+GGCACA
+GGCACT
+GGCACC
+GGCACG
+GGCAGA
+GGCAGT
+GGCAGC
+GGCAGG
+GGCTAA
+GGCTAT
+GGCTAC
+GGCTAG
+GGCTTA
+GGCTTT
+GGCTTC
+GGCTTG
+GGCTCA
+GGCTCT
+GGCTCC
+GGCTCG
+GGCTGA
+GGCTGT
+GGCTGC
+GGCTGG
+GGCCAA
+GGCCAT
+GGCCAC
+GGCCAG
+GGCCTA
+GGCCTT
+GGCCTC
+GGCCTG
+GGCCCA
+GGCCCT
+GGCCCC
+GGCCCG
+GGCCGA
+GGCCGT
+GGCCGC
+GGCCGG
+GGCGAA
+GGCGAT
+GGCGAC
+GGCGAG
+GGCGTA
+GGCGTT
+GGCGTC
+GGCGTG
+GGCGCA
+GGCGCT
+GGCGCC
+GGCGCG
+GGCGGA
+GGCGGT
+GGCGGC
+GGCGGG
+GGGAAA
+GGGAAT
+GGGAAC
+GGGAAG
+GGGATA
+GGGATT
+GGGATC
+GGGATG
+GGGACA
+GGGACT
+GGGACC
+GGGACG
+GGGAGA
+GGGAGT
+GGGAGC
+GGGAGG
+GGGTAA
+GGGTAT
+GGGTAC
+GGGTAG
+GGGTTA
+GGGTTT
+GGGTTC
+GGGTTG
+GGGTCA
+GGGTCT
+GGGTCC
+GGGTCG
+GGGTGA
+GGGTGT
+GGGTGC
+GGGTGG
+GGGCAA
+GGGCAT
+GGGCAC
+GGGCAG
+GGGCTA
+GGGCTT
+GGGCTC
+GGGCTG
+GGGCCA
+GGGCCT
+GGGCCC
+GGGCCG
+GGGCGA
+GGGCGT
+GGGCGC
+GGGCGG
+GGGGAA
+GGGGAT
+GGGGAC
+GGGGAG
+GGGGTA
+GGGGTT
+GGGGTC
+GGGGTG
+GGGGCA
+GGGGCT
+GGGGCC
+GGGGCG
+GGGGGA
+GGGGGT
+GGGGGC
+GGGGGG
+A
+T
+C
+G
+N

ntv2/tokenizer_info.json

@@ -0,0 +1,5 @@
+{
+  "source": "InstaDeepAI/nucleotide-transformer-v2-500m-multi-species",
+  "eos_id": 4,
+  "vocab_sha256": "63a8635cd714efade8dc12dad6f14048a58c64a6b9a5b2cd37b6ffc5276144ae"
+}

provenance.json

@@ -0,0 +1,23 @@
+{
+  "source_run": "final-20260920",
+  "checkpoints": [
+    {
+      "model": "dnabert2",
+      "path": "dnabert2/model.pt",
+      "bytes": 83363931,
+      "sha256": "9acc1aea844385c6fa826693de19ef6fc01713bc6aea2dba8091abdb544dca4d"
+    },
+    {
+      "model": "evo2",
+      "path": "evo2/model.pt",
+      "bytes": 164296411,
+      "sha256": "376094ad3cd2543701d43aab3ea71b34c2a3b7bf97299cf9666d2eb7a5dd1184"
+    },
+    {
+      "model": "ntv2",
+      "path": "ntv2/model.pt",
+      "bytes": 90028571,
+      "sha256": "5300eea166e2b31aa5a7e0e90d1bb2d859d86de94fe185371b142041899c9539"
+    }
+  ]
+}

requirements.txt

@@ -0,0 +1,3 @@
+torch>=2.6,<3
+numpy>=1.26,<3
+transformers==4.44.2

validation.json

@@ -0,0 +1,40 @@
+{
+  "device": "cpu",
+  "torch": "2.14.1+cpu",
+  "models": {
+    "dnabert2": {
+      "examples": 84,
+      "datasets": [
+        "hg38",
+        "1000g"
+      ],
+      "lengths": 14,
+      "exact_prediction_parity": true,
+      "cli": true,
+      "invalid_inputs_rejected": true
+    },
+    "evo2": {
+      "examples": 84,
+      "datasets": [
+        "hg38",
+        "1000g"
+      ],
+      "lengths": 14,
+      "exact_prediction_parity": true,
+      "cli": true,
+      "invalid_inputs_rejected": true
+    },
+    "ntv2": {
+      "examples": 84,
+      "datasets": [
+        "hg38",
+        "1000g"
+      ],
+      "lengths": 14,
+      "exact_prediction_parity": true,
+      "cli": true,
+      "invalid_inputs_rejected": true
+    }
+  },
+  "scope": "Inference parity, not a reconstruction accuracy estimate"
+}