README.md

---
license: mit
pretty_name: Mind the Tail - SAE, Selection Rankings, and Models
tags:
  - data-selection
  - coreset
  - sparse-autoencoder
  - dinov3
---

# Mind the Tail: Released Artifacts

This repository contains the artifacts of **Mind the Tail: Data Selection Through Sparse Concepts**: the sparse autoencoder (SAE) used to extract concepts, the selection rankings of every training pool, and the models trained on the selected subsets.

[Paper](https://arxiv.org/abs/2610.21007) | [Project Page](https://imageomics.github.io/mind-the-tail/) | [Code](https://github.com/Imageomics/mind-the-tail)

## Structure

```
├── ckpts/
│   ├── sae/dinov3_b_layer11_in1k_topk32/
│   │   └── sae.pt                              # TopK SAE, 16,384 concepts
│   └── downstream/
│       ├── nabirds/<ratio>pct/best.pt          # ResNet-50
│       ├── stanford_cars/<ratio>pct/best.pt    # ResNet-50
│       ├── waterbirds/<ratio>pct/best.pt       # ResNet-50
│       ├── spawrious/<ratio>pct/best.pt        # ResNet-50
│       └── cc12m/<ratio>pct/clip.pt            # CLIP ViT-B/16
└── corsets/
    ├── nabirds/ranked_examples.parquet
    ├── stanford_cars/ranked_examples.parquet
    ├── waterbirds/ranked_examples.parquet
    ├── spawrious/ranked_examples.parquet
    └── cc12m/ranked_examples.parquet
```

`<ratio>` is one of 10, 20, 30, 50, and 70.

## Sparse Autoencoder

`ckpts/sae/dinov3_b_layer11_in1k_topk32/sae.pt` is the single SAE used for every dataset: a TopK SAE (k = 32 active concepts per token, 16,384 concepts) trained on 100M layer-11 patch tokens of a frozen DINOv3 ViT-B/16 from ImageNet-1K train. Image-level concept vectors are obtained by max-pooling the patch activations.

The file starts with one JSON line holding the architecture (`d_model`, `d_sae`, `top_k`), followed by a PyTorch state dict with `W_enc` (768 × 16,384), `b_enc`, `W_dec` (16,384 × 768), and `b_dec`.

## Selection Rankings

Each `ranked_examples.parquet` holds the full ordering of a training pool. A subset at ratio `r` is the top of this ordering.

**Within-class rankings** (labelled datasets) have the columns `rank_within_class`, `class_candidate_count`, `image_path`, `class_idx`, and `class_name`. The subset at ratio `r` keeps the top `max(1, floor(r · class_candidate_count))` images of every class.

**Global ranking** (CC12M) has the columns `rank_global` and `image_path`. The subset at ratio `r` keeps the top `floor(r · N)` images. Paths have the form `tar://${DATA_ROOT}/cc12m-wds/<shard>.tar::<key>.jpg`, referring to the image of sample `<key>` in a CC12M WebDataset shard.

All paths are written relative to `${DATA_ROOT}`, the folder holding the original datasets, so they resolve against any dataset location.

## Models

**ResNet-50** (`best.pt`): ImageNet-pretrained ResNet-50 fine-tuned on the subset of each dataset and ratio (18,700 steps, batch size 128, SGD, lr 1e-3). Each file holds only the weights as `{"model_state_dict": ...}`. The output classes follow `class_idx` of the corresponding ranking file.

**CLIP ViT-B/16** (`clip.pt`): trained from scratch on the CC12M subset at each ratio. Each file is a plain state dict of the OpenCLIP `ViT-B-16` model.

## Datasets

The images themselves are not included; they are available from their original sources: [NABirds](https://dl.allaboutbirds.org/nabirds), Stanford Cars, [Waterbirds](https://github.com/kohpangwei/group_DRO) (`waterbird_complete95_forest2water2`), [Spawrious](https://github.com/aengusl/spawrious) (one-to-one hard), and [CC12M](https://github.com/google-research-datasets/conceptual-12m).

## Citation

```
@article{zhang2026mindthetail,
  title   = {Mind the Tail: Data Selection Through Sparse Concepts},
  author  = {Zhang, Ziheng and Hulkund, Neha and Lee, Colin and Mai, Zheda and
             Shen, Hui and Ding, Ying and Berger-Wolf, Tanya and Beery, Sara and
             Chao, Wei-Lun and Gu, Jianyang},
  journal = {arXiv preprint arXiv:2610.21007},
  year    = {2026}
}
```

## License

The artifacts in this repository are released under the MIT License.