Instructions to use Synthyra/Profluent-E1-300M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Synthyra/Profluent-E1-300M with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="Synthyra/Profluent-E1-300M", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForMaskedLM model = AutoModelForMaskedLM.from_pretrained("Synthyra/Profluent-E1-300M", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update FastPLMs files
Browse files- README.md +40 -44
- THIRD_PARTY_NOTICES.md +3 -3
- fastplms/models.toml +8 -7
- fastplms/models/e1/attention.py +17 -5
- fastplms/models/e1/modeling_e1.py +69 -33
- fastplms/models/e1/retrieval.py +1 -1
- fastplms_bundle.py +0 -0
- modeling_fastplms.py +1 -1
README.md
CHANGED
|
@@ -12,7 +12,7 @@ tags:
|
|
| 12 |
|
| 13 |
# Synthyra/Profluent-E1-300M
|
| 14 |
|
| 15 |
-
This checkpoint
|
| 16 |
|
| 17 |
Accepted inputs are raw amino-acid sequences prepared by the native E1 adapter.
|
| 18 |
Supported Transformers entry points are `AutoConfig`, `AutoModel`,
|
|
@@ -31,9 +31,7 @@ Supported Transformers entry points are `AutoConfig`, `AutoModel`,
|
|
| 31 |
| Attention variants | Supported: `sdpa`, `flex_attention` |
|
| 32 |
| Compliance | Declared: exact release evidence is required |
|
| 33 |
|
| 34 |
-
A supported interface is not a pretrained downstream predictor. Classification
|
| 35 |
-
heads start untrained, and declared compliance metadata is not a claim that an
|
| 36 |
-
arbitrary local build passed its release gate.
|
| 37 |
|
| 38 |
## Install and platform requirements
|
| 39 |
|
|
@@ -44,12 +42,12 @@ python -m pip install -r \
|
|
| 44 |
"https://huggingface.co/Synthyra/Profluent-E1-300M/resolve/main/requirements.txt"
|
| 45 |
```
|
| 46 |
|
| 47 |
-
The FastPLMs implementation itself is embedded in the model repository
|
| 48 |
-
|
| 49 |
|
| 50 |
-
Python 3.11-3.14, PyTorch 2.13, and Transformers 5.13
|
| 51 |
-
|
| 52 |
-
|
| 53 |
|
| 54 |
## Quick start
|
| 55 |
|
|
@@ -65,22 +63,22 @@ model = AutoModel.from_pretrained(
|
|
| 65 |
```
|
| 66 |
|
| 67 |
For offline validation, replace `model_id` with the manifest-built
|
| 68 |
-
`dist/hub/Profluent-E1-300M` path
|
| 69 |
|
| 70 |
## Attention and compliance
|
| 71 |
|
| 72 |
The quick start selects `sdpa` explicitly. Declared variants are `sdpa`, `flex_attention`. An unavailable requested
|
| 73 |
-
backend raises
|
| 74 |
-
`output_attentions=True`
|
| 75 |
-
|
| 76 |
|
| 77 |
-
This family declares the `compliance` tier. Release evidence
|
| 78 |
checkpoint, backend, dtype, hardware, inputs, and reference revision.
|
| 79 |
|
| 80 |
## Dataset embeddings
|
| 81 |
|
| 82 |
-
The shared embedding mixin
|
| 83 |
-
|
| 84 |
|
| 85 |
```python
|
| 86 |
pooled = model.embed_dataset(
|
|
@@ -97,13 +95,13 @@ print(residues[0].tensor.shape) # (l, d)
|
|
| 97 |
```
|
| 98 |
|
| 99 |
Set `output` and `format="safetensors"` or `"sqlite"` for transactional,
|
| 100 |
-
bounded-memory
|
| 101 |
-
policy, backend, dtype, and pooling configuration before
|
| 102 |
|
| 103 |
## Downstream classification
|
| 104 |
|
| 105 |
-
Both downstream AutoClasses
|
| 106 |
-
untrained `classifier`. Sequence labels have shape `(b,)`
|
| 107 |
shape `(b, l)` and use `-100` outside biological positions:
|
| 108 |
|
| 109 |
```python
|
|
@@ -140,7 +138,7 @@ print(token_output.logits.shape) # (b, l, 3)
|
|
| 140 |
|
| 141 |
## PEFT fine-tuning
|
| 142 |
|
| 143 |
-
Install the
|
| 144 |
|
| 145 |
```bash
|
| 146 |
python -m pip install "datasets>=4.8,<5" "peft>=0.19,<0.20"
|
|
@@ -161,17 +159,17 @@ peft_model = get_peft_model(
|
|
| 161 |
)
|
| 162 |
```
|
| 163 |
|
| 164 |
-
This checkpoint advertises a classification head
|
| 165 |
-
`classifier`
|
| 166 |
All FastPLMs checkpoints follow the Transformers `PreTrainedModel` contract and
|
| 167 |
-
can
|
| 168 |
support boundary. Record the target modules, base revision, data identity, and
|
| 169 |
trainable parameter scope.
|
| 170 |
|
| 171 |
## Test-time training
|
| 172 |
|
| 173 |
TTT samples masked views of one protein and updates only injected low-rank
|
| 174 |
-
adapters. Base checkpoint weights
|
| 175 |
|
| 176 |
```python
|
| 177 |
from transformers import AutoModelForMaskedLM
|
|
@@ -189,14 +187,14 @@ ttt_model.ttt_reset()
|
|
| 189 |
print(metrics)
|
| 190 |
```
|
| 191 |
|
| 192 |
-
|
| 193 |
-
|
| 194 |
|
| 195 |
## Tokenizer-free E1 input
|
| 196 |
|
| 197 |
-
E1 has no tokenizer. The model
|
| 198 |
-
|
| 199 |
-
|
| 200 |
|
| 201 |
```python
|
| 202 |
result = model.embed_dataset(
|
|
@@ -207,9 +205,9 @@ result = model.embed_dataset(
|
|
| 207 |
print(result[0].tensor.shape)
|
| 208 |
```
|
| 209 |
|
| 210 |
-
Lower-level masked-language-model calls must use the E1 batch preparer
|
| 211 |
-
|
| 212 |
-
|
| 213 |
|
| 214 |
## Runtime contract
|
| 215 |
|
|
@@ -229,8 +227,8 @@ the attribution required by the upstream agreement.
|
|
| 229 |
## Release record
|
| 230 |
|
| 231 |
- FastPLMs weights: `Synthyra/Profluent-E1-300M`
|
| 232 |
-
- Runtime revision: recorded
|
| 233 |
-
- Source-tree and runtime-bundle SHA-256: recorded in
|
| 234 |
- Official checkpoint: `Profluent-Bio/E1-300m`
|
| 235 |
- Artifact source: `fast`
|
| 236 |
- State transform: `e1_to_fastplms_v1`
|
|
@@ -238,19 +236,17 @@ the attribution required by the upstream agreement.
|
|
| 238 |
- Release tiers: `check`, `compliance`, `feature`, `artifact`, `benchmark`
|
| 239 |
- Unresolved required file identities: `0`
|
| 240 |
|
| 241 |
-
|
| 242 |
-
legal texts, schema, and attestations. A nonzero unresolved count blocks release.
|
| 243 |
|
| 244 |
## Validation boundary
|
| 245 |
|
| 246 |
-
Declared tiers compare
|
| 247 |
-
|
| 248 |
-
|
| 249 |
-
valid.
|
| 250 |
|
| 251 |
## License
|
| 252 |
|
| 253 |
Checkpoint terms: [Profluent-E1 Clickthrough License Agreement](https://github.com/Profluent-AI/E1/blob/main/LICENSE). The Hub model-card identifier is
|
| 254 |
-
`other`.
|
| 255 |
-
|
| 256 |
-
before use.
|
|
|
|
| 12 |
|
| 13 |
# Synthyra/Profluent-E1-300M
|
| 14 |
|
| 15 |
+
This checkpoint contains the FastPLMs `E1` implementation.
|
| 16 |
|
| 17 |
Accepted inputs are raw amino-acid sequences prepared by the native E1 adapter.
|
| 18 |
Supported Transformers entry points are `AutoConfig`, `AutoModel`,
|
|
|
|
| 31 |
| Attention variants | Supported: `sdpa`, `flex_attention` |
|
| 32 |
| Compliance | Declared: exact release evidence is required |
|
| 33 |
|
| 34 |
+
A supported interface is not a pretrained downstream predictor. Classification heads start untrained. Compliance metadata does not show that a local build passed its release gate.
|
|
|
|
|
|
|
| 35 |
|
| 36 |
## Install and platform requirements
|
| 37 |
|
|
|
|
| 42 |
"https://huggingface.co/Synthyra/Profluent-E1-300M/resolve/main/requirements.txt"
|
| 43 |
```
|
| 44 |
|
| 45 |
+
The FastPLMs implementation itself is embedded in the model repository.
|
| 46 |
+
Transformers loads it through `trust_remote_code=True`.
|
| 47 |
|
| 48 |
+
This model requires Python 3.11-3.14, PyTorch 2.13, and Transformers 5.13. The CPU gate covers small offline tests. Published checkpoint throughput and parity require the documented device tier. The Hub quick start needs network access for
|
| 49 |
+
the first download. For an air-gapped run, build the manifest-pinned local
|
| 50 |
+
artifact first and use the offline example.
|
| 51 |
|
| 52 |
## Quick start
|
| 53 |
|
|
|
|
| 63 |
```
|
| 64 |
|
| 65 |
For offline validation, replace `model_id` with the manifest-built
|
| 66 |
+
`dist/hub/Profluent-E1-300M` path. Pass `local_files_only=True`.
|
| 67 |
|
| 68 |
## Attention and compliance
|
| 69 |
|
| 70 |
The quick start selects `sdpa` explicitly. Declared variants are `sdpa`, `flex_attention`. An unavailable requested
|
| 71 |
+
backend raises. It does not silently change implementation.
|
| 72 |
+
`output_attentions=True` can use the documented one-call eager fallback to
|
| 73 |
+
materialize attention tensors. The configured backend does not change.
|
| 74 |
|
| 75 |
+
This family declares the `compliance` tier. Release evidence identifies the
|
| 76 |
checkpoint, backend, dtype, hardware, inputs, and reference revision.
|
| 77 |
|
| 78 |
## Dataset embeddings
|
| 79 |
|
| 80 |
+
The shared embedding mixin keeps input order and biological-position masking.
|
| 81 |
+
It accepts sequences, identified records, mappings, or a FASTA path:
|
| 82 |
|
| 83 |
```python
|
| 84 |
pooled = model.embed_dataset(
|
|
|
|
| 95 |
```
|
| 96 |
|
| 97 |
Set `output` and `format="safetensors"` or `"sqlite"` for transactional,
|
| 98 |
+
bounded-memory storage. Resume checks input order, model state, tokenizer
|
| 99 |
+
policy, backend, dtype, and pooling configuration before it appends data.
|
| 100 |
|
| 101 |
## Downstream classification
|
| 102 |
|
| 103 |
+
Both downstream AutoClasses use the checkpoint backbone and create a new,
|
| 104 |
+
untrained `classifier`. Sequence labels have shape `(b,)`. Residue labels have
|
| 105 |
shape `(b, l)` and use `-100` outside biological positions:
|
| 106 |
|
| 107 |
```python
|
|
|
|
| 138 |
|
| 139 |
## PEFT fine-tuning
|
| 140 |
|
| 141 |
+
Install the training dependencies. Then attach LoRA to the loaded checkpoint:
|
| 142 |
|
| 143 |
```bash
|
| 144 |
python -m pip install "datasets>=4.8,<5" "peft>=0.19,<0.20"
|
|
|
|
| 159 |
)
|
| 160 |
```
|
| 161 |
|
| 162 |
+
This checkpoint advertises a classification head. Save the separately trained
|
| 163 |
+
`classifier` with the adapter.
|
| 164 |
All FastPLMs checkpoints follow the Transformers `PreTrainedModel` contract and
|
| 165 |
+
can use PEFT. The ESM2-specific shipped CLI is an example, not a
|
| 166 |
support boundary. Record the target modules, base revision, data identity, and
|
| 167 |
trainable parameter scope.
|
| 168 |
|
| 169 |
## Test-time training
|
| 170 |
|
| 171 |
TTT samples masked views of one protein and updates only injected low-rank
|
| 172 |
+
adapters. Base checkpoint weights stay frozen:
|
| 173 |
|
| 174 |
```python
|
| 175 |
from transformers import AutoModelForMaskedLM
|
|
|
|
| 187 |
print(metrics)
|
| 188 |
```
|
| 189 |
|
| 190 |
+
Saved adapters retain their deterministic reset state. TTT adds latency and
|
| 191 |
+
memory, can worsen an output, and does not show biological function.
|
| 192 |
|
| 193 |
## Tokenizer-free E1 input
|
| 194 |
|
| 195 |
+
E1 has no tokenizer. The model keeps native raw-sequence preparation, boundary
|
| 196 |
+
tokens, sequence positions, and retrieval-augmented context behavior. The
|
| 197 |
+
ordinary representation path accepts sequences directly:
|
| 198 |
|
| 199 |
```python
|
| 200 |
result = model.embed_dataset(
|
|
|
|
| 205 |
print(result[0].tensor.shape)
|
| 206 |
```
|
| 207 |
|
| 208 |
+
Lower-level masked-language-model calls must use the E1 batch preparer, not an
|
| 209 |
+
`AutoTokenizer`. E1 launch messages and distributed legal files keep the
|
| 210 |
+
attribution required by the upstream agreement.
|
| 211 |
|
| 212 |
## Runtime contract
|
| 213 |
|
|
|
|
| 227 |
## Release record
|
| 228 |
|
| 229 |
- FastPLMs weights: `Synthyra/Profluent-E1-300M`
|
| 230 |
+
- Runtime revision: recorded in the built artifact and published commit
|
| 231 |
+
- Source-tree and runtime-bundle SHA-256: recorded in the source record
|
| 232 |
- Official checkpoint: `Profluent-Bio/E1-300m`
|
| 233 |
- Artifact source: `fast`
|
| 234 |
- State transform: `e1_to_fastplms_v1`
|
|
|
|
| 236 |
- Release tiers: `check`, `compliance`, `feature`, `artifact`, `benchmark`
|
| 237 |
- Unresolved required file identities: `0`
|
| 238 |
|
| 239 |
+
The source record records exact file identities, conversion, source revisions,
|
| 240 |
+
legal texts, schema, and attestations. A nonzero unresolved count blocks a release.
|
| 241 |
|
| 242 |
## Validation boundary
|
| 243 |
|
| 244 |
+
Declared tiers compare configuration, tokenizer behavior, state, and
|
| 245 |
+
representative inference with the pinned reference. Metadata does not show that
|
| 246 |
+
a build passed, that a backend is faster, or that an output is biologically valid.
|
|
|
|
| 247 |
|
| 248 |
## License
|
| 249 |
|
| 250 |
Checkpoint terms: [Profluent-E1 Clickthrough License Agreement](https://github.com/Profluent-AI/E1/blob/main/LICENSE). The Hub model-card identifier is
|
| 251 |
+
`other`. The local artifact contains applicable source
|
| 252 |
+
licenses, notices, attribution, and conversion records. Review them before use.
|
|
|
THIRD_PARTY_NOTICES.md
CHANGED
|
@@ -46,7 +46,7 @@ explicitly defines the repository release as including pretrained DPLM1 and
|
|
| 46 |
DPLM2 weights, and the same revision carries the complete
|
| 47 |
[Apache-2.0 license](https://github.com/bytedance/dplm/blob/8a2e15e53416b4536f03f79ad1f6f6a9cbd5e19d/LICENSE).
|
| 48 |
FastPLMs records both checkpoint families as Apache-2.0 and distributes the
|
| 49 |
-
verbatim license plus `LICENSES/dplm/
|
| 50 |
those terms and remain subject to the ordinary artifact and publication gates.
|
| 51 |
|
| 52 |
## Biohub
|
|
@@ -80,7 +80,7 @@ TorchMetrics, Lightning Utilities, and NVIDIA DLLogger. Their exact versions or
|
|
| 80 |
revision are pinned in `docker/constraints/esmfold.txt`; OpenFold imports them
|
| 81 |
eagerly, and FastPLMs production code does not depend on them. DLLogger's exact
|
| 82 |
source identity and installed-license handling are recorded in
|
| 83 |
-
`LICENSES/dllogger/
|
| 84 |
|
| 85 |
## ProteinTTT
|
| 86 |
|
|
@@ -93,7 +93,7 @@ revision-specific provenance are under `LICENSES/protein-ttt/`.
|
|
| 93 |
For every supported family, `src/fastplms/models.toml` records an immutable
|
| 94 |
official checkpoint revision, an immutable FastPLMs checkpoint revision, file
|
| 95 |
digests, a named state transformation, and a mechanism-level conversion record.
|
| 96 |
-
Generated artifacts reproduce that record in `
|
| 97 |
artifact build must fail when a required file identity, legal text, attribution
|
| 98 |
notice, modified-file notice, upstream revision, or conversion record is absent
|
| 99 |
or differs from its manifest digest.
|
|
|
|
| 46 |
DPLM2 weights, and the same revision carries the complete
|
| 47 |
[Apache-2.0 license](https://github.com/bytedance/dplm/blob/8a2e15e53416b4536f03f79ad1f6f6a9cbd5e19d/LICENSE).
|
| 48 |
FastPLMs records both checkpoint families as Apache-2.0 and distributes the
|
| 49 |
+
verbatim license plus `LICENSES/dplm/SOURCE_RECORD.md`. Converted weights retain
|
| 50 |
those terms and remain subject to the ordinary artifact and publication gates.
|
| 51 |
|
| 52 |
## Biohub
|
|
|
|
| 80 |
revision are pinned in `docker/constraints/esmfold.txt`; OpenFold imports them
|
| 81 |
eagerly, and FastPLMs production code does not depend on them. DLLogger's exact
|
| 82 |
source identity and installed-license handling are recorded in
|
| 83 |
+
`LICENSES/dllogger/SOURCE_RECORD.md`.
|
| 84 |
|
| 85 |
## ProteinTTT
|
| 86 |
|
|
|
|
| 93 |
For every supported family, `src/fastplms/models.toml` records an immutable
|
| 94 |
official checkpoint revision, an immutable FastPLMs checkpoint revision, file
|
| 95 |
digests, a named state transformation, and a mechanism-level conversion record.
|
| 96 |
+
Generated artifacts reproduce that record in `source-record.json`. A release or
|
| 97 |
artifact build must fail when a required file identity, legal text, attribution
|
| 98 |
notice, modified-file notice, upstream revision, or conversion record is absent
|
| 99 |
or differs from its manifest digest.
|
fastplms/models.toml
CHANGED
|
@@ -88,7 +88,7 @@ license_files = ["LICENSE"]
|
|
| 88 |
license_digests = ["LICENSE=sha256:cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30"]
|
| 89 |
distribution_files = [
|
| 90 |
"LICENSE=sha256:cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30",
|
| 91 |
-
"
|
| 92 |
]
|
| 93 |
|
| 94 |
[[upstreams]]
|
|
@@ -122,7 +122,7 @@ license_files = ["LICENSE"]
|
|
| 122 |
license_digests = ["LICENSE=sha256:da6d3703ed11cbe42bd212c725957c98da23cbff1998c05fa4b3d976d1a58e93"]
|
| 123 |
distribution_files = [
|
| 124 |
"LICENSE=sha256:da6d3703ed11cbe42bd212c725957c98da23cbff1998c05fa4b3d976d1a58e93",
|
| 125 |
-
"
|
| 126 |
]
|
| 127 |
|
| 128 |
[[upstreams]]
|
|
@@ -136,7 +136,7 @@ license_digests = ["LICENSE=sha256:cfc7749b96f63bd31c3c42b5c471bf756814053e847c1
|
|
| 136 |
distribution_files = [
|
| 137 |
"LICENSE=sha256:cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30",
|
| 138 |
"MODIFICATIONS.md=sha256:fd6f0aa1086a0c996cf967b326d18e965660cda0ad5c7f36a3474a8490720da3",
|
| 139 |
-
"
|
| 140 |
]
|
| 141 |
|
| 142 |
[[upstreams]]
|
|
@@ -149,7 +149,7 @@ license_files = ["LICENSE"]
|
|
| 149 |
license_digests = ["LICENSE=sha256:bb01e7d5554f9e2e117172e56551452f68a7818df7bc8e71cd7a776a1d4ba3df"]
|
| 150 |
distribution_files = [
|
| 151 |
"LICENSE=sha256:bb01e7d5554f9e2e117172e56551452f68a7818df7bc8e71cd7a776a1d4ba3df",
|
| 152 |
-
"
|
| 153 |
]
|
| 154 |
|
| 155 |
[families.esm2]
|
|
@@ -187,7 +187,8 @@ reference_adapter = "tests.parity.support.reference_adapters.esm_plusplus"
|
|
| 187 |
attention = ["eager", "sdpa", "flex_attention", "flash_attention_2", "flash_attention_3"]
|
| 188 |
dtypes = ["float32", "bfloat16"]
|
| 189 |
bf16_execution = "static_parameters"
|
| 190 |
-
precisions = ["default"]
|
|
|
|
| 191 |
vram_tier = "sequence"
|
| 192 |
checkpoint_license = "MIT"
|
| 193 |
hub_license = "mit"
|
|
@@ -267,7 +268,7 @@ checkpoint_license = "Apache-2.0"
|
|
| 267 |
hub_license = "apache-2.0"
|
| 268 |
weights_publication_allowed = true
|
| 269 |
state_transform = "dplm_to_fastplms_v1"
|
| 270 |
-
conversion_provenance = "Input: the pinned official DPLM1 checkpoint. Transformation: apply dplm_to_fastplms_v1, omitting the unused absolute-position table for rotary checkpoints and materializing the tied input/output embedding values as independent tensors. Output: the pinned Synthyra DPLM checkpoint. Validation: release parity compares exact state identity after the declared transform, tokenizer behavior, generation, and inference. License basis: the pinned ByteDance DPLM Apache-2.0 LICENSE and README explicitly scope the repository release to the pretrained DPLM1 and DPLM2 weights; immutable evidence is recorded in LICENSES/dplm/
|
| 271 |
representative = "dplm_150m"
|
| 272 |
documentation = "docs/models.md#dplm"
|
| 273 |
test_tiers = ["check", "compliance", "feature", "artifact", "benchmark"]
|
|
@@ -291,7 +292,7 @@ checkpoint_license = "Apache-2.0"
|
|
| 291 |
hub_license = "apache-2.0"
|
| 292 |
weights_publication_allowed = true
|
| 293 |
state_transform = "dplm2_to_fastplms_v1"
|
| 294 |
-
conversion_provenance = "Input: the pinned official DPLM2 checkpoint. Transformation: apply dplm2_to_fastplms_v1, retaining the independent language-model head and trained encoder contact head while omitting the unused absolute-position table for rotary checkpoints. Output: the pinned Synthyra DPLM2 checkpoint. Validation: release parity compares exact keys and values after the declared omission, non-aliasing, tokenizer behavior, generation, and inference. License basis: the pinned ByteDance DPLM Apache-2.0 LICENSE and README explicitly scope the repository release to the pretrained DPLM1 and DPLM2 weights; immutable evidence is recorded in LICENSES/dplm/
|
| 295 |
representative = "dplm2_150m"
|
| 296 |
documentation = "docs/models.md#dplm2"
|
| 297 |
test_tiers = ["check", "compliance", "feature", "artifact", "benchmark"]
|
|
|
|
| 88 |
license_digests = ["LICENSE=sha256:cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30"]
|
| 89 |
distribution_files = [
|
| 90 |
"LICENSE=sha256:cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30",
|
| 91 |
+
"SOURCE_RECORD.md=sha256:a659f74be9073cf1ad2d2f7071531ca56959b421f111152cf4c41184ace5970e",
|
| 92 |
]
|
| 93 |
|
| 94 |
[[upstreams]]
|
|
|
|
| 122 |
license_digests = ["LICENSE=sha256:da6d3703ed11cbe42bd212c725957c98da23cbff1998c05fa4b3d976d1a58e93"]
|
| 123 |
distribution_files = [
|
| 124 |
"LICENSE=sha256:da6d3703ed11cbe42bd212c725957c98da23cbff1998c05fa4b3d976d1a58e93",
|
| 125 |
+
"SOURCE_RECORD.md=sha256:950adb94daf15e646ddf226dacfe2a8e77801aa0793e439a9a3490a48eb666e7",
|
| 126 |
]
|
| 127 |
|
| 128 |
[[upstreams]]
|
|
|
|
| 136 |
distribution_files = [
|
| 137 |
"LICENSE=sha256:cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30",
|
| 138 |
"MODIFICATIONS.md=sha256:fd6f0aa1086a0c996cf967b326d18e965660cda0ad5c7f36a3474a8490720da3",
|
| 139 |
+
"SOURCE_RECORD.md=sha256:48c903db43a217a3126afaefbac60b7ddac7efda2dfcc0cbff0bffc7d6c30081",
|
| 140 |
]
|
| 141 |
|
| 142 |
[[upstreams]]
|
|
|
|
| 149 |
license_digests = ["LICENSE=sha256:bb01e7d5554f9e2e117172e56551452f68a7818df7bc8e71cd7a776a1d4ba3df"]
|
| 150 |
distribution_files = [
|
| 151 |
"LICENSE=sha256:bb01e7d5554f9e2e117172e56551452f68a7818df7bc8e71cd7a776a1d4ba3df",
|
| 152 |
+
"SOURCE_RECORD.md=sha256:dc641c37353c2efd50ccbdb316ca4aae495ec02c1563e0e15bac92f75fc482e5",
|
| 153 |
]
|
| 154 |
|
| 155 |
[families.esm2]
|
|
|
|
| 187 |
attention = ["eager", "sdpa", "flex_attention", "flash_attention_2", "flash_attention_3"]
|
| 188 |
dtypes = ["float32", "bfloat16"]
|
| 189 |
bf16_execution = "static_parameters"
|
| 190 |
+
precisions = ["default", "fp8"]
|
| 191 |
+
experimental_precisions = ["fp8"]
|
| 192 |
vram_tier = "sequence"
|
| 193 |
checkpoint_license = "MIT"
|
| 194 |
hub_license = "mit"
|
|
|
|
| 268 |
hub_license = "apache-2.0"
|
| 269 |
weights_publication_allowed = true
|
| 270 |
state_transform = "dplm_to_fastplms_v1"
|
| 271 |
+
conversion_provenance = "Input: the pinned official DPLM1 checkpoint. Transformation: apply dplm_to_fastplms_v1, omitting the unused absolute-position table for rotary checkpoints and materializing the tied input/output embedding values as independent tensors. Output: the pinned Synthyra DPLM checkpoint. Validation: release parity compares exact state identity after the declared transform, tokenizer behavior, generation, and inference. License basis: the pinned ByteDance DPLM Apache-2.0 LICENSE and README explicitly scope the repository release to the pretrained DPLM1 and DPLM2 weights; immutable evidence is recorded in LICENSES/dplm/SOURCE_RECORD.md. Limitation: redistribution remains subject to Apache-2.0 and the pinned source record; no broader rights are inferred."
|
| 272 |
representative = "dplm_150m"
|
| 273 |
documentation = "docs/models.md#dplm"
|
| 274 |
test_tiers = ["check", "compliance", "feature", "artifact", "benchmark"]
|
|
|
|
| 292 |
hub_license = "apache-2.0"
|
| 293 |
weights_publication_allowed = true
|
| 294 |
state_transform = "dplm2_to_fastplms_v1"
|
| 295 |
+
conversion_provenance = "Input: the pinned official DPLM2 checkpoint. Transformation: apply dplm2_to_fastplms_v1, retaining the independent language-model head and trained encoder contact head while omitting the unused absolute-position table for rotary checkpoints. Output: the pinned Synthyra DPLM2 checkpoint. Validation: release parity compares exact keys and values after the declared omission, non-aliasing, tokenizer behavior, generation, and inference. License basis: the pinned ByteDance DPLM Apache-2.0 LICENSE and README explicitly scope the repository release to the pretrained DPLM1 and DPLM2 weights; immutable evidence is recorded in LICENSES/dplm/SOURCE_RECORD.md. Limitation: no head exception is permitted by this source record, and redistribution remains subject to Apache-2.0."
|
| 296 |
representative = "dplm2_150m"
|
| 297 |
documentation = "docs/models.md#dplm2"
|
| 298 |
test_tiers = ["check", "compliance", "feature", "artifact", "benchmark"]
|
fastplms/models/e1/attention.py
CHANGED
|
@@ -322,11 +322,7 @@ def varlen_flex_attention_func(
|
|
| 322 |
seqlens_k = cu_seqlens_k[1:] - cu_seqlens_k[:-1] # (n,)
|
| 323 |
block_mask = block_mask_creator(seqlens_q, seqlens_k)
|
| 324 |
|
| 325 |
-
packed_lengths = (
|
| 326 |
-
*(int(length) for length in seqlens_q.tolist()),
|
| 327 |
-
-1,
|
| 328 |
-
*(int(length) for length in seqlens_k.tolist()),
|
| 329 |
-
)
|
| 330 |
fn = _get_flex_attention_fn(
|
| 331 |
device=query_states.device,
|
| 332 |
dtype=query_states.dtype,
|
|
@@ -351,6 +347,21 @@ def varlen_flex_attention_func(
|
|
| 351 |
return attn_output # (b, l_q, h, d)
|
| 352 |
|
| 353 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 354 |
def _get_unpad_data(sequence_ids: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, int]:
|
| 355 |
"""Return packed indices and run lengths for the non-padding sequence IDs."""
|
| 356 |
|
|
@@ -384,6 +395,7 @@ def _get_unpad_data(sequence_ids: torch.Tensor) -> tuple[torch.Tensor, torch.Ten
|
|
| 384 |
)
|
| 385 |
|
| 386 |
|
|
|
|
| 387 |
def _unpad_input(
|
| 388 |
query_layer: torch.Tensor,
|
| 389 |
key_layer: torch.Tensor,
|
|
|
|
| 322 |
seqlens_k = cu_seqlens_k[1:] - cu_seqlens_k[:-1] # (n,)
|
| 323 |
block_mask = block_mask_creator(seqlens_q, seqlens_k)
|
| 324 |
|
| 325 |
+
packed_lengths = _packed_lengths_cache_key(seqlens_q, seqlens_k)
|
|
|
|
|
|
|
|
|
|
|
|
|
| 326 |
fn = _get_flex_attention_fn(
|
| 327 |
device=query_states.device,
|
| 328 |
dtype=query_states.dtype,
|
|
|
|
| 347 |
return attn_output # (b, l_q, h, d)
|
| 348 |
|
| 349 |
|
| 350 |
+
@torch.compiler.disable
|
| 351 |
+
def _packed_lengths_cache_key(
|
| 352 |
+
query_lengths: torch.Tensor,
|
| 353 |
+
key_lengths: torch.Tensor,
|
| 354 |
+
) -> tuple[int, ...]:
|
| 355 |
+
"""Convert packed lengths to an immutable Flex cache key eagerly."""
|
| 356 |
+
|
| 357 |
+
return (
|
| 358 |
+
*(int(length) for length in query_lengths.tolist()),
|
| 359 |
+
-1,
|
| 360 |
+
*(int(length) for length in key_lengths.tolist()),
|
| 361 |
+
)
|
| 362 |
+
|
| 363 |
+
|
| 364 |
+
@torch.compiler.disable
|
| 365 |
def _get_unpad_data(sequence_ids: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, int]:
|
| 366 |
"""Return packed indices and run lengths for the non-padding sequence IDs."""
|
| 367 |
|
|
|
|
| 395 |
)
|
| 396 |
|
| 397 |
|
| 398 |
+
@torch.compiler.disable
|
| 399 |
def _unpad_input(
|
| 400 |
query_layer: torch.Tensor,
|
| 401 |
key_layer: torch.Tensor,
|
fastplms/models/e1/modeling_e1.py
CHANGED
|
@@ -135,6 +135,54 @@ def _get_logger():
|
|
| 135 |
return logging.get_logger(__name__)
|
| 136 |
|
| 137 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 138 |
_TOKENIZER_LOAD_CONTEXT: ContextVar[dict[str, Any] | None] = ContextVar(
|
| 139 |
"fastplms_e1_tokenizer_load_context",
|
| 140 |
default=None,
|
|
@@ -319,6 +367,19 @@ class RotaryPositionalEmbedding(nn.Module):
|
|
| 319 |
self.cos_cached = angles.cos()
|
| 320 |
self.sin_cached = angles.sin()
|
| 321 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 322 |
def forward(
|
| 323 |
self,
|
| 324 |
q: torch.Tensor,
|
|
@@ -328,10 +389,7 @@ class RotaryPositionalEmbedding(nn.Module):
|
|
| 328 |
) -> tuple[torch.Tensor, torch.Tensor]:
|
| 329 |
# q, k: (b, l, h, d)
|
| 330 |
device, dtype = q.device, q.dtype
|
| 331 |
-
|
| 332 |
-
|
| 333 |
-
if seq_len > self.max_seq_len_cached:
|
| 334 |
-
self._set_sin_cos_cache(seq_len=seq_len, device=device)
|
| 335 |
|
| 336 |
# Selecting by position inserts a head axis for broadcasting.
|
| 337 |
idxs = position_ids.to(device)
|
|
@@ -717,8 +775,7 @@ class Attention(nn.Module):
|
|
| 717 |
if cached_len < 0:
|
| 718 |
raise ValueError(f"E1 cached KV length {kv_len} is shorter than query length {q_len}.")
|
| 719 |
first_sequence_id = query_sequence_ids[:, :1]
|
| 720 |
-
|
| 721 |
-
raise ValueError("E1 cached queries must start with a non-padding sequence token.")
|
| 722 |
cached_sequence_ids = first_sequence_id.expand(-1, cached_len)
|
| 723 |
key_sequence_ids = torch.cat((cached_sequence_ids, query_sequence_ids), dim=-1)
|
| 724 |
return query_sequence_ids, key_sequence_ids
|
|
@@ -1314,33 +1371,12 @@ class FAST_E1_ENCODER(E1PreTrainedModel, EmbeddingMixin):
|
|
| 1314 |
within_positions = within_seq_position_ids.long()
|
| 1315 |
global_positions = global_position_ids.long()
|
| 1316 |
sequence_numbers = sequence_ids.long()
|
| 1317 |
-
|
| 1318 |
-
|
| 1319 |
-
|
| 1320 |
-
|
| 1321 |
-
|
| 1322 |
-
|
| 1323 |
-
"Position ids must be in the range "
|
| 1324 |
-
f"[-1, {self.config.max_num_positions_within_seq}); got max "
|
| 1325 |
-
f"{highest_position.item()} and min {lowest_position.item()}"
|
| 1326 |
-
)
|
| 1327 |
-
lowest_global, highest_global = torch.aminmax(global_positions)
|
| 1328 |
-
if (
|
| 1329 |
-
lowest_global.item() < -1
|
| 1330 |
-
or highest_global.item() >= self.config.max_num_positions_global
|
| 1331 |
-
):
|
| 1332 |
-
raise ValueError(
|
| 1333 |
-
"Global position ids must be in the range "
|
| 1334 |
-
f"[-1, {self.config.max_num_positions_global}); got max "
|
| 1335 |
-
f"{highest_global.item()} and min {lowest_global.item()}"
|
| 1336 |
-
)
|
| 1337 |
-
lowest_sequence, highest_sequence = torch.aminmax(sequence_numbers)
|
| 1338 |
-
if lowest_sequence.item() < -1 or highest_sequence.item() >= self.config.max_num_sequences:
|
| 1339 |
-
raise ValueError(
|
| 1340 |
-
"Sequence ids must be in the range "
|
| 1341 |
-
f"[-1, {self.config.max_num_sequences}); got max "
|
| 1342 |
-
f"{highest_sequence.item()} and min {lowest_sequence.item()}"
|
| 1343 |
-
)
|
| 1344 |
|
| 1345 |
if inputs_embeds is None:
|
| 1346 |
if input_ids is None:
|
|
|
|
| 135 |
return logging.get_logger(__name__)
|
| 136 |
|
| 137 |
|
| 138 |
+
@torch.compiler.disable
|
| 139 |
+
def _validate_cached_global_query_start(first_sequence_id: torch.Tensor) -> None:
|
| 140 |
+
"""Keep cached-query validation eager while preserving padding semantics."""
|
| 141 |
+
|
| 142 |
+
if bool(first_sequence_id.eq(-1).any()):
|
| 143 |
+
raise ValueError("E1 cached queries must start with a non-padding sequence token.")
|
| 144 |
+
|
| 145 |
+
|
| 146 |
+
@torch.compiler.disable
|
| 147 |
+
def _validate_biological_indices(
|
| 148 |
+
within_positions: torch.Tensor,
|
| 149 |
+
global_positions: torch.Tensor,
|
| 150 |
+
sequence_numbers: torch.Tensor,
|
| 151 |
+
config: E1Config,
|
| 152 |
+
) -> None:
|
| 153 |
+
"""Validate E1's sequence and position conventions outside compiled graphs."""
|
| 154 |
+
|
| 155 |
+
lowest_position, highest_position = torch.aminmax(within_positions)
|
| 156 |
+
minimum_position = int(lowest_position.item())
|
| 157 |
+
maximum_position = int(highest_position.item())
|
| 158 |
+
if minimum_position < -1 or maximum_position >= config.max_num_positions_within_seq:
|
| 159 |
+
raise ValueError(
|
| 160 |
+
"Position ids must be in the range "
|
| 161 |
+
f"[-1, {config.max_num_positions_within_seq}); got max "
|
| 162 |
+
f"{maximum_position} and min {minimum_position}"
|
| 163 |
+
)
|
| 164 |
+
|
| 165 |
+
lowest_global, highest_global = torch.aminmax(global_positions)
|
| 166 |
+
minimum_global = int(lowest_global.item())
|
| 167 |
+
maximum_global = int(highest_global.item())
|
| 168 |
+
if minimum_global < -1 or maximum_global >= config.max_num_positions_global:
|
| 169 |
+
raise ValueError(
|
| 170 |
+
"Global position ids must be in the range "
|
| 171 |
+
f"[-1, {config.max_num_positions_global}); got max "
|
| 172 |
+
f"{maximum_global} and min {minimum_global}"
|
| 173 |
+
)
|
| 174 |
+
|
| 175 |
+
lowest_sequence, highest_sequence = torch.aminmax(sequence_numbers)
|
| 176 |
+
minimum_sequence = int(lowest_sequence.item())
|
| 177 |
+
maximum_sequence = int(highest_sequence.item())
|
| 178 |
+
if minimum_sequence < -1 or maximum_sequence >= config.max_num_sequences:
|
| 179 |
+
raise ValueError(
|
| 180 |
+
"Sequence ids must be in the range "
|
| 181 |
+
f"[-1, {config.max_num_sequences}); got max "
|
| 182 |
+
f"{maximum_sequence} and min {minimum_sequence}"
|
| 183 |
+
)
|
| 184 |
+
|
| 185 |
+
|
| 186 |
_TOKENIZER_LOAD_CONTEXT: ContextVar[dict[str, Any] | None] = ContextVar(
|
| 187 |
"fastplms_e1_tokenizer_load_context",
|
| 188 |
default=None,
|
|
|
|
| 367 |
self.cos_cached = angles.cos()
|
| 368 |
self.sin_cached = angles.sin()
|
| 369 |
|
| 370 |
+
@torch.compiler.disable
|
| 371 |
+
def _ensure_sin_cos_cache(
|
| 372 |
+
self,
|
| 373 |
+
position_ids: torch.Tensor,
|
| 374 |
+
seq_len: int | None,
|
| 375 |
+
device: torch.device,
|
| 376 |
+
) -> None:
|
| 377 |
+
"""Resize the data-dependent rotary cache outside compiled graphs."""
|
| 378 |
+
|
| 379 |
+
required_length = int(position_ids.max().item()) + 1 if seq_len is None else seq_len
|
| 380 |
+
if required_length > self.max_seq_len_cached:
|
| 381 |
+
self._set_sin_cos_cache(seq_len=required_length, device=device)
|
| 382 |
+
|
| 383 |
def forward(
|
| 384 |
self,
|
| 385 |
q: torch.Tensor,
|
|
|
|
| 389 |
) -> tuple[torch.Tensor, torch.Tensor]:
|
| 390 |
# q, k: (b, l, h, d)
|
| 391 |
device, dtype = q.device, q.dtype
|
| 392 |
+
self._ensure_sin_cos_cache(position_ids, seq_len, device)
|
|
|
|
|
|
|
|
|
|
| 393 |
|
| 394 |
# Selecting by position inserts a head axis for broadcasting.
|
| 395 |
idxs = position_ids.to(device)
|
|
|
|
| 775 |
if cached_len < 0:
|
| 776 |
raise ValueError(f"E1 cached KV length {kv_len} is shorter than query length {q_len}.")
|
| 777 |
first_sequence_id = query_sequence_ids[:, :1]
|
| 778 |
+
_validate_cached_global_query_start(first_sequence_id)
|
|
|
|
| 779 |
cached_sequence_ids = first_sequence_id.expand(-1, cached_len)
|
| 780 |
key_sequence_ids = torch.cat((cached_sequence_ids, query_sequence_ids), dim=-1)
|
| 781 |
return query_sequence_ids, key_sequence_ids
|
|
|
|
| 1371 |
within_positions = within_seq_position_ids.long()
|
| 1372 |
global_positions = global_position_ids.long()
|
| 1373 |
sequence_numbers = sequence_ids.long()
|
| 1374 |
+
_validate_biological_indices(
|
| 1375 |
+
within_positions,
|
| 1376 |
+
global_positions,
|
| 1377 |
+
sequence_numbers,
|
| 1378 |
+
self.config,
|
| 1379 |
+
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1380 |
|
| 1381 |
if inputs_embeds is None:
|
| 1382 |
if input_ids is None:
|
fastplms/models/e1/retrieval.py
CHANGED
|
@@ -890,7 +890,7 @@ class HomologueSearcher:
|
|
| 890 |
"""
|
| 891 |
|
| 892 |
_PROVENANCE_SCHEMA_VERSION = 1
|
| 893 |
-
_PROVENANCE_FILENAME = "search-
|
| 894 |
|
| 895 |
def __init__(
|
| 896 |
self,
|
|
|
|
| 890 |
"""
|
| 891 |
|
| 892 |
_PROVENANCE_SCHEMA_VERSION = 1
|
| 893 |
+
_PROVENANCE_FILENAME = "search-record.json"
|
| 894 |
|
| 895 |
def __init__(
|
| 896 |
self,
|
fastplms_bundle.py
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
modeling_fastplms.py
CHANGED
|
@@ -12,7 +12,7 @@ from zipfile import ZIP_DEFLATED, ZipFile
|
|
| 12 |
|
| 13 |
from .fastplms_bundle import RUNTIME_DATA, RUNTIME_HASH
|
| 14 |
|
| 15 |
-
if RUNTIME_HASH != "
|
| 16 |
raise RuntimeError("FastPLMs runtime identity differs from the bridge.")
|
| 17 |
|
| 18 |
_RUNTIME_TEMPORARIES = []
|
|
|
|
| 12 |
|
| 13 |
from .fastplms_bundle import RUNTIME_DATA, RUNTIME_HASH
|
| 14 |
|
| 15 |
+
if RUNTIME_HASH != "16de6217cddadfeb81d6086a8c0f273176854dc99a0b3443c89cbc998f516aba":
|
| 16 |
raise RuntimeError("FastPLMs runtime identity differs from the bridge.")
|
| 17 |
|
| 18 |
_RUNTIME_TEMPORARIES = []
|