---
license: apache-2.0
tags:
- pytorch
- deit
- generation
- wip
- prototype
---

# DeiT experiments: Generation

## Overview

A research-oriented **DeiT** prototype targeting **Generation**. The included **nano** setup documents defaults and file formats without presenting unverified performance numbers.

## Quick Check

```bash
python predict.py --help
```

Inspect the script's `__main__` block for its generated smoke-test example. Because this is a custom implementation, generic automatic loading APIs require an explicit adapter before use.

## Contact

Questions or corrections: open a discussion in this repository.

## Training Details

| Item | Value |
|---|---|
| Framework | PyTorch |
| Precision | fp32 |
| Init scheme | xavier uniform |
| Hardware | a single consumer GPU is enough |
| Typical run | minutes, not hours |

## Limitations

The initialization checkpoint has not been trained or audited for robustness, fairness, or domain transfer. The implementation should be treated as an experimental starting point. Results from a future trained checkpoint must be documented separately from the defaults shipped here.

## Default Experiment Recipe

The included configuration uses **sgd** with a **constant warmup** schedule. These are starting values in the script, not evidence of a completed run. For a meaningful evaluation, train all baselines with the same data exposure, tuning budget, and random seeds.

## Architecture

| Item | Value |
|---|---|
| Architecture | DeiT |
| Scale | nano |
| Attention | multi query |
| Fusion | tucker |
| Activation | swish |
| Normalization | scalenorm |

## Repository Status

1. The Python file contains the model and runnable example or training entry point.
2. `config.json` records the generated architecture settings.
3. `training_args.json` records the default experiment recipe.
4. `model.safetensors` is a valid initialization checkpoint for smoke tests; it is **not** presented as a trained benchmark checkpoint.
5. No benchmark score is claimed in this repository.

## Files

| File | Purpose |
|---|---|
| `predict.py` | primary artifact |
| `README.md` | this documentation |
| `config.json` | architecture configuration |
| `training_args.json` | default experiment settings |
| `model.safetensors` | initialization checkpoint |

## License

Released under **apache-2.0**. Review the source-data terms separately when this repository is used with external datasets.