---
license: cc-by-4.0
tags:
- approx-gelu
- concat-mlp
- constant-warmup
- contrastive
- instancenorm
- kaiming-normal
- mae
- rmsprop
- sparse
- xlarge
---

# finetune.py

## Model Overview

A **xlarge**-scale implementation of the **mae** architecture, built for **contrastive** tasks.

## Architecture

- **Architecture**: mae
- **Scale**: xlarge
- **Attention**: sparse
- **Fusion strategy**: concat mlp
- **Task head**: contrastive
- **Activation**: approx gelu
- **Normalization**: instancenorm
- **Initialization**: kaiming normal

## Training

- **Optimizer**: rmsprop
- **LR scheduler**: constant warmup

## Files

- `finetune.py` — main artifact of this repository

## License

See the license field above.