README.md

---
language:
- en
- fr
- ro
- de
license: apache-2.0
library_name: transformers
datasets:
- c4
---

# Model Card for EncT5

EncT5 is a variant of T5 that utilizes mainly the encoder for non-autoregressive (ie. classification and regression)
tasks. The model is from the paper [Fine-tuning T5 Encoder for Non-autoregressive Tasks](https://arxiv.org/abs/2110.08426)
by Frederick Liu, Terry Huang, Shihang Lyu, Siamak Shakeri, Hongkun Yu, Jing Li



## Model Details

### Model Description

EncT5 uses the same base weights at T5, but **must be fine-tuning before use**. There are several special features
to EncT5:

1. There are less decoder layers (a single decoder layer by default), and so has fewer parameters/resources than the
   standard T5.
3. There is a separate decoder word embedding, with the decoder input ids being predefined constants. During
   fine-tuning, the decoder embedding learns to use these constants as "prompts" to the encoder for the corresponding
   classification/regression tasks.
5. There is a classification head on top of the decoder output.

Research has shown that this model can be more efficient and usable over T5 and BERT for non-autoregressive
tasks such as classification and regression.

- **Developed by:** Frederick Liu, Terry Huang, Shihang Lyu, Siamak Shakeri, Hongkun Yu, Jing Li. See the
  [associated paper](https://arxiv.org/abs/2110.08426).
- **Model type:** Language Model
- **Language(s) (NLP):** English, French, Romanian, German
- **License:** Apache 2.0
- **Based on model:** [T5](https://huggingface.co/google-t5/t5-base)
- **Repository:** [Github repro](https://github.com/hackyon/EncT5)
- **Paper:** [Fine-tuning T5 Encoder for Non-autoregressive Tasks](https://arxiv.org/abs/2110.08426)

## How to Get Started with the Model

Use the code below to get started with the model.

```python
model = AutoModelForSequenceClassification.from_pretrained("hackyon/enct5-base", trust_remote_code=True)
# Fine-tune the model before use.
```

See the [github repro](https://github.com/hackyon/EncT5) for a more comprehensive guide.

## Training Details

### Training Data

The weights of this model are directly copied from [t5-base](https://huggingface.co/google-t5/t5-base).

### Training Procedure 

This model **must be fine-tuned** before use. The decoder word embedding and classification head are both untrained.