README.md

---
license: apache-2.0
tags:
- protein-language-model
- bioinformatics
- transformer
- cnn
- small-protein
- smORF
frameworks:
- pytorch
---

```
╔═══════════════════════════════════════════════╗
║                                                 ║
║        _______ _____ _______                    ║
║       |__   __|  __ \__   __|                   ║
║          | |  | |__) | | |                      ║
║          | |  |  ___/  | |                      ║
║          | |  | |      | |                      ║
║          |_|  |_|      |_|                      ║
║                                                 ║
║          ▸ TINY  PROTEIN  TRANSFORMER ◂         ║
║                                                 ║
╚═══════════════════════════════════════════════╝
```


# TinyProteinTransformer (TPT)

TinyProteinTransformer (TPT) is a lightweight CNN-Transformer hybrid encoder designed for microbial smORF-encoded small proteins. It was pretrained on the Global Microbial smORF Catalog (GMSC, >280M sequences) using masked language modeling combined with contrastive learning, producing compact yet expressive residue-level and sequence-level protein representations.

This repository hosts the pretrained model weights for direct inference and downstream fine-tuning.

For training scripts, ablation code, and the full experimental pipeline, see the GitHub repository: [F4NG66/TPT](https://github.com/F4NG66/TPT)

## Files

| File | Description |
|---|---|
| `TPT.pt` | Baseline pretrained TPT weights (CNN-Transformer hybrid encoder, hidden dim 640) |
| `TPT_gated.py` | Gated variant of the TPT architecture|

## Model Details

- **Architecture:** CNN-Transformer hybrid encoder with multi-scale convolutional feature extraction and attention-based pooling
- **Hidden dimension:** 640
- **Transformer layers:** 20
- **CNN kernel sizes:** 3, 5, 7, 9
- **Max sequence length:** 128
- **Pretraining objective:** Masked language modeling + contrastive learning (temperature τ = 0.1, loss weight λ = 0.05)
- **Pretraining corpus:** GMSC10.90 (Global Microbial smORF Catalog)

## Usage

```python
import torch
from model import TinyProteinTransformer
from utils import build_tokenizer

tokenizer = build_tokenizer()
model = TinyProteinTransformer(vocab_size=len(tokenizer))
model.load_state_dict(torch.load("TPT.pt", map_location="cpu"))
model.eval()

def encode(seq, max_len=128):
    ids = [tokenizer.get(aa, tokenizer["X"]) for aa in seq][:max_len]
    ids += [tokenizer["PAD"]] * (max_len - len(ids))
    return torch.tensor(ids).unsqueeze(0)

with torch.no_grad():
    x = encode("MKVLILACLVVVTITVS")
    h = model.encode(x)                  # (1, L, 640) residue-level representations
    embedding = model.attention_pool(h)  # (1, 640) sequence-level embedding
```

For classification tasks, attach a lightweight linear head on top of `embedding`.

### Gated variant

`TPT_gated.py` defines the gated architecture variant. Load its accompanying checkpoint the same way, using the model class defined in that script in place of `TinyProteinTransformer`.

## Intended Use

TPT embeddings can be used as frozen features for downstream classification tasks on small secreted/microbial proteins, including antimicrobial peptide (AMP) prediction, toxicity (TOX) prediction, bacteriocin (BCN) classification, and anti-CRISPR (Acr) protein classification.

## Citation

```bibtex
@article{sheng2026tpt,
  title={TPT: A Compact CNN-Transformer Encoder for Efficient Microbial Small Protein Modeling},
  author={Sheng, Fang and Zhang, Junhe and Zhu, Chengkai},
  journal={Frontiers in Microbiology},
  year={2026}
}
```