Double-column gallery, drop How-it-works, swappable placeholder models
Browse files
README.md
CHANGED
|
@@ -16,7 +16,7 @@ pipeline_tag: text-to-video
|
|
| 16 |
|
| 17 |
NEvo is a self-contained Hugging Face custom [Diffusers](https://github.com/huggingface/diffusers) pipeline for **neural-response-guided visual stimulus synthesis**. Given a brain target (a set of voxels, or a target fMRI vector), it searches over prompts, generates images and short videos, scores each candidate with a differentiable image/video→fMRI encoder, and returns the ranked stimuli predicted to best drive that target.
|
| 18 |
|
| 19 |
-
It orchestrates three frozen models
|
| 20 |
|
| 21 |
| Role | Default model |
|
| 22 |
|------|---------------|
|
|
@@ -26,27 +26,16 @@ It orchestrates three frozen models (weights are **not** bundled — they are pu
|
|
| 26 |
|
| 27 |
## Gallery
|
| 28 |
|
| 29 |
-
Each clip is the **top
|
| 30 |
|
| 31 |
-
| Region |
|
| 32 |
-
|------
|
| 33 |
-
| **FFA** · faces |  |
|
| 34 |
-
| **
|
| 35 |
-
| **
|
| 36 |
-
| **EBA** · bodies |  |
|
| 37 |
-
| **pSTS** · social motion |  |
|
| 38 |
-
| **V1** · early visual |  |
|
| 39 |
|
| 40 |
Explore the full interactive gallery and 3D brain maps at **[nevo-project.epfl.ch](https://nevo-project.epfl.ch/)**.
|
| 41 |
|
| 42 |
-
## How it works
|
| 43 |
-
|
| 44 |
-
NEvo runs an **evolutionary (genetic) search** over prompts, guided by a brain encoder:
|
| 45 |
-
|
| 46 |
-
`prompt population → SDXL-Turbo (image) → LTX-Video (video) → vJEPA-2 fMRI encoder → activation score → keep best → recombine → repeat`
|
| 47 |
-
|
| 48 |
-
Each generation turns candidate prompts into short videos and scores how strongly the **target region** is predicted to respond; the highest-scoring prompts survive and are mutated/recombined into the next generation. Over generations the search converges on stimuli that maximally drive the target voxels (default objective `indices_mean`: the mean predicted response across them). It runs in two stages — first evolving the image prompt, then the motion prompt on top of the best image.
|
| 49 |
-
|
| 50 |
## Installation
|
| 51 |
|
| 52 |
NEvo is a Python package (not a single-file community pipeline), so install it first:
|
|
|
|
| 16 |
|
| 17 |
NEvo is a self-contained Hugging Face custom [Diffusers](https://github.com/huggingface/diffusers) pipeline for **neural-response-guided visual stimulus synthesis**. Given a brain target (a set of voxels, or a target fMRI vector), it searches over prompts, generates images and short videos, scores each candidate with a differentiable image/video→fMRI encoder, and returns the ranked stimuli predicted to best drive that target.
|
| 18 |
|
| 19 |
+
It orchestrates three frozen models. The models below are **placeholders / defaults** and can be swapped for any compatible models (weights are not bundled — they are pulled from their own repos):
|
| 20 |
|
| 21 |
| Role | Default model |
|
| 22 |
|------|---------------|
|
|
|
|
| 26 |
|
| 27 |
## Gallery
|
| 28 |
|
| 29 |
+
Each clip is from the **top results of a NEvo search targeting one visual region** — the model discovers, from scratch, stimuli that drive that region's known selectivity.
|
| 30 |
|
| 31 |
+
| Region | Stimulus | Region | Stimulus |
|
| 32 |
+
|:------:|:--------:|:------:|:--------:|
|
| 33 |
+
| **FFA** · faces |  | **PPA** · places |  |
|
| 34 |
+
| **MT** · motion |  | **EBA** · bodies |  |
|
| 35 |
+
| **pSTS** · social motion |  | **V1** · early visual |  |
|
|
|
|
|
|
|
|
|
|
| 36 |
|
| 37 |
Explore the full interactive gallery and 3D brain maps at **[nevo-project.epfl.ch](https://nevo-project.epfl.ch/)**.
|
| 38 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
## Installation
|
| 40 |
|
| 41 |
NEvo is a Python package (not a single-file community pipeline), so install it first:
|