--- title: IntReal Leaderboard emoji: 🥇 colorFrom: blue colorTo: indigo sdk: gradio app_file: app.py pinned: true license: apache-2.0 short_description: Real-world multimodal interleaved generation benchmark sdk_version: 6.26.0 hf_oauth: true --- 📄 [Paper](https://arxiv.org/abs/2610.38425) · 🌐 [Project Page](https://yushaohan.github.io/IntReal/) · 🤗 [Models](https://huggingface.co/collections/yushaohan/intreal) · 🤗 [Dataset](https://huggingface.co/datasets/yushaohan/IntReal-100K) · 🏆 [Leaderboard](https://huggingface.co/spaces/yushaohan/IntReal-Leaderboard) · 🎬 [Video](https://youtu.be/ZYkxaL8wa9Q) # IntReal Leaderboard A curated leaderboard for **IntReal: A Real-World Dataset and Benchmark for Multimodal Interleaved Generation**. Results are transcribed from Table 3 of the supplied paper (Gemma4 evaluation setting). All scores are on a 0–100 scale, higher is better. Mean values are preserved as published rather than recomputed from rounded scores. ## Run locally ```bash python -m pip install -r requirements.txt python app.py ``` Authentication is intentionally disabled outside configured HF OAuth; local preview does not impersonate an HF account. ## Update results Edit `static/results.json`. Each row describes a model configuration and includes `model`, `access` (`Open-source` or `Closed-source`), `samples`, `mean`, and the seven scores `tr`, `tl`, `ia`, `id`, `qc`, `cf`, `sc`. Preserve the evaluation protocol when adding comparable results. Push changes to the Space to rebuild it. The page uses the original template styling and shows only the leaderboard, model access filtering, optional Access/Model size columns, and sortable score columns. It accepts authenticated submissions when OAuth and persistent storage are configured. Submission does not automatically evaluate or publish scores. Displaying results needs no Hugging Face token. Legacy template modules in `src/` are not used by this app. ## Protocol notes - 600 evaluation samples: 150 each for Planning, Knowledge, Tutorial, and Story. - Holistic: Text Richness, Text Logicality, Image Aesthetic, Image Diversity, Quantity Compliance. - Fine-grained: Context Fidelity and Subject Consistency. The latter is evaluated on Tutorial and Story only. - Emu3.5 uses 592 samples due to context/memory limits; Gemini-3.1-Flash-Image uses 598 due to API guardrails. - SenseNova-U1-8B-MoT-Interleaved is the base checkpoint; post-trained configurations use the IntReal prefix. Gemini-3.1-Flash-Image denotes Gemini-3.1-Flash-Image. Project: https://yushaohan.github.io/IntReal ## Configure authenticated ZIP submissions Users must sign in with Hugging Face. The username is read-only and comes from the verified OIDC session. The submitted model must be public, ungated, hosted under that exact personal username, contain weights, and declare an allowed open license. Organization repositories and custom licenses require a future review workflow. Email is entered by the submitter and syntax-checked, not email-verified. `hf_oauth: true` provisions the Space OAuth application. `OAUTH_CLIENT_ID`, `OAUTH_CLIENT_SECRET`, and `SPACE_HOST` are provided by HF. Session signing uses a derived key from the OAuth client secret; optionally override it with a strong random `SESSION_SECRET` of at least 32 characters in Space Secrets. The callback is `https:///auth/callback`. The app requests only `openid profile`; it does not request write access to submitters' accounts. Do not put secrets in source control. Selected storage: private bucket `yushaohan/IntReal-Submission`. A read-write mount at `/data` has been requested; runtime attachment and write access still need verification before enabling submissions. Submission files will use `/data/intreal-submissions`. Attach a **private persistent volume**, for example a private HF Storage Bucket. Configure `SUBMISSIONS_MOUNT` as its mount point (default `/data`) and `SUBMISSIONS_DIR` as a child directory (default `/data/intreal-submissions`). The backend refuses submissions unless the configured path is inside a mounted volume. Keep one replica: quota and upload reservations are coordinated in one process. Do not expose this directory through StaticFiles, Gradio file-serving paths, or a public repository. Changing the volume's sharing settings can expose contact details and outputs. Limits: ZIP ≤10 GB, expanded contents ≤24 GiB, each member ≤128 MiB, ≤40,000 entries, ≤40 megapixels per image, two concurrent uploads, one unfinished upload per account, three submissions per HF account in a rolling 7-day period, 32 GiB total archive quota, 2 GiB free-space reserve. Upload sessions expire after two hours; an upload also times out after two minutes without a chunk. Stale unfinished uploads are cleaned at the next submission. Uploads stream to disk and are never extracted. ZIP extensions, paths, encryption, duplicate entries, compression ratios, image formats and CRCs are checked. Successful archives and private metadata are written under `//`; status is `pending_review`. Anonymous users and other accounts have no file download or metadata endpoint. No scores are automatically added to the leaderboard. Local development of storage only: set `INTREAL_LOCAL_STORAGE=1` and `SUBMISSIONS_DIR` to a scratch directory. This never enables a fake sign-in and is ignored in a Space. Production OAuth must be tested on the Space before enabling real submissions. Use `python -m pytest tests` for backend checks. Storage admission checks account for the configured 32 GiB intake quota, existing files, pending reservations, and filesystem free space before receiving a ZIP. Disk-full/quota errors during upload are rejected and incomplete files removed. The form disables submission when full and directs users to `shaohanyu628@gmail.com` to request storage management and retry later. Filesystem capacity is not a direct measurement of the HF account-wide Hub storage quota; remote quota write failures are also handled. Submission model names are derived server-side from the verified HF repository URL; submitters do not enter a separate name.