README.md

metadata

title: IntReal Leaderboard
emoji: 🥇
colorFrom: blue
colorTo: indigo
sdk: gradio
app_file: app.py
pinned: true
license: apache-2.0
short_description: Real-world multimodal interleaved generation benchmark
sdk_version: 6.26.0
hf_oauth: true

📄 Paper · 🌐 Project Page · 🤗 Models · 🤗 Dataset · 🏆 Leaderboard · 🎬 Video

IntReal Leaderboard

A curated leaderboard for IntReal: A Real-World Dataset and Benchmark for Multimodal Interleaved Generation.

Results are transcribed from Table 3 of the supplied paper (Gemma4 evaluation setting). All scores are on a 0–100 scale, higher is better. Mean values are preserved as published rather than recomputed from rounded scores.

Run locally

python -m pip install -r requirements.txt
python app.py

Authentication is intentionally disabled outside configured HF OAuth; local preview does not impersonate an HF account.

Update results

Edit static/results.json. Each row describes a model configuration and includes model, access (Open-source or Closed-source), samples, mean, and the seven scores tr, tl, ia, id, qc, cf, sc. Preserve the evaluation protocol when adding comparable results. Push changes to the Space to rebuild it.

The page uses the original template styling and shows only the leaderboard, model access filtering, optional Access/Model size columns, and sortable score columns. It accepts authenticated submissions when OAuth and persistent storage are configured. Submission does not automatically evaluate or publish scores. Displaying results needs no Hugging Face token. Legacy template modules in src/ are not used by this app.

Protocol notes

• 600 evaluation samples: 150 each for Planning, Knowledge, Tutorial, and Story.
• Holistic: Text Richness, Text Logicality, Image Aesthetic, Image Diversity, Quantity Compliance.
• Fine-grained: Context Fidelity and Subject Consistency. The latter is evaluated on Tutorial and Story only.
• Emu3.5 uses 592 samples due to context/memory limits; Gemini-3.1-Flash-Image uses 598 due to API guardrails.
• SenseNova-U1-8B-MoT-Interleaved is the base checkpoint; post-trained configurations use the IntReal prefix. Gemini-3.1-Flash-Image denotes Gemini-3.1-Flash-Image.

Project: https://yushaohan.github.io/IntReal

Configure authenticated ZIP submissions

Users must sign in with Hugging Face. The username is read-only and comes from the verified OIDC session. The submitted model must be public, ungated, hosted under that exact personal username, contain weights, and declare an allowed open license. Organization repositories and custom licenses require a future review workflow. Email is entered by the submitter and syntax-checked, not email-verified.

hf_oauth: true provisions the Space OAuth application. OAUTH_CLIENT_ID, OAUTH_CLIENT_SECRET, and SPACE_HOST are provided by HF. Session signing uses a derived key from the OAuth client secret; optionally override it with a strong random SESSION_SECRET of at least 32 characters in Space Secrets. The callback is https://<SPACE_HOST>/auth/callback. The app requests only openid profile; it does not request write access to submitters' accounts. Do not put secrets in source control.

Selected storage: private bucket yushaohan/IntReal-Submission. A read-write mount at /data has been requested; runtime attachment and write access still need verification before enabling submissions. Submission files will use /data/intreal-submissions.

Attach a private persistent volume, for example a private HF Storage Bucket. Configure SUBMISSIONS_MOUNT as its mount point (default /data) and SUBMISSIONS_DIR as a child directory (default /data/intreal-submissions). The backend refuses submissions unless the configured path is inside a mounted volume. Keep one replica: quota and upload reservations are coordinated in one process. Do not expose this directory through StaticFiles, Gradio file-serving paths, or a public repository. Changing the volume's sharing settings can expose contact details and outputs.

Limits: ZIP ≤10 GB, expanded contents ≤24 GiB, each member ≤128 MiB, ≤40,000 entries, ≤40 megapixels per image, two concurrent uploads, one unfinished upload per account, three submissions per HF account in a rolling 7-day period, 32 GiB total archive quota, 2 GiB free-space reserve. Upload sessions expire after two hours; an upload also times out after two minutes without a chunk. Stale unfinished uploads are cleaned at the next submission. Uploads stream to disk and are never extracted. ZIP extensions, paths, encryption, duplicate entries, compression ratios, image formats and CRCs are checked. Successful archives and private metadata are written under <SUBMISSIONS_DIR>/<id>/; status is pending_review. Anonymous users and other accounts have no file download or metadata endpoint. No scores are automatically added to the leaderboard.

Local development of storage only: set INTREAL_LOCAL_STORAGE=1 and SUBMISSIONS_DIR to a scratch directory. This never enables a fake sign-in and is ignored in a Space. Production OAuth must be tested on the Space before enabling real submissions. Use python -m pytest tests for backend checks.

Storage admission checks account for the configured 32 GiB intake quota, existing files, pending reservations, and filesystem free space before receiving a ZIP. Disk-full/quota errors during upload are rejected and incomplete files removed. The form disables submission when full and directs users to shaohanyu628@gmail.com to request storage management and retry later. Filesystem capacity is not a direct measurement of the HF account-wide Hub storage quota; remote quota write failures are also handled.

Submission model names are derived server-side from the verified HF repository URL; submitters do not enter a separate name.