rubric-workspace / README.md
UsergyAI's picture
Deploy private rubric workspace on free Gradio hosting
5e47017 verified
|
Raw History Blame Contribute Delete
6.19 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade
metadata
title: Rubric Workspace
emoji: 📋
colorFrom: gray
colorTo: blue
sdk: gradio
sdk_version: 6.29.1
python_version: '3.12'
app_file: app.py
hf_oauth: true
pinned: false
disable_embedding: true

Rubric Workspace

An independent multimodal research-annotation app: task upload/manual entry, two-stage Qwen rubric generation, Gemini scoring, deterministic rankings and compatible Excel/CSV reports.

Current status

This source includes shared team access-code authentication, temporary disk, Qwen through OpenRouter and Gemini 3.8 Flash through fal. In team mode, source/guidelines may be public while uploads, runs, reports and model execution require the team code; everyone with that code shares one workspace. No separate storage service is required. Synthetic fixtures are not research annotations; live canaries verify integration, not annotation accuracy. Runtime health identifies the exact deployed build. Nothing auto-provisions a paid service.

Run locally

Requirements: Python 3.11+, uv, Node 22+ with npm, FFmpeg/ffprobe.

From the project directory:

uv sync --frozen
npm --prefix web ci
npm --prefix web run build
uv run python scripts/run_local.py

Open http://127.0.0.1:7860. This normal mode allows task preparation but refuses model execution until connections are configured.

To exercise the app without keys:

uv run python scripts/run_local.py --test --data-dir runtime/demo

Test mode visibly marks the UI, outputs and download filenames. Its deterministic all-pass fixture is only a plumbing test, not a model judgment. Never mix its reports into a research benchmark.

The local server listens only on loopback. Stop with Ctrl+C. Saved tasks/media remain in the chosen runtime directory. The server is not launched automatically by installing the project.

Key features

  • CSV or XLSX upload, up to 300 tasks; manual input and editing before submission.
  • All five text/image/video task types, with separate source and A/B/C media.
  • Supplied-workbook task tabs are supported; Combined and legacy summaries are not silently double-counted.
  • Image/video upload or approved HTTPS/Google Drive links; bulk filename-to-attachment matching.
  • Frozen prompt/source-only rubric; response-based criteria and semantic duplicate review remain separate.
  • Binary 0/5 scores, weighted ties, explicit disqualification and unassessable states.
  • Approved audio observation bridge: Gemini provides source-only and response-audio evidence to Qwen when original media has audio. Images and silent clips do not require this additional stage.
  • Saved stages, pause/resume, failed-stage retries, interrupted-request warnings and idempotent submissions.
  • Real progress, side-by-side media, image zoom, video playback/speed/seek and timestamp evidence.
  • Excel report task tabs and nested field families match the supplied workbook; CSV/JSON handles overflow without truncation.
  • Hugging Face temporary disk for jobs/media; local SQLite for local use. Download reports before the Space stops, restarts or rebuilds. The unused Supabase adapter is optional, not required.
  • Shared access-code login, salted code hash, revocable sessions, throttled login attempts and protected API/media/report routes. HF OAuth remains available only in separately configured HF-auth mode.

Configuration

Set secrets in a protected server environment or HF Space Secrets—not in browser fields, source control or chat. See setup.

Core values:

  • RUBRIC_OPENROUTER_API_KEY (or OPENROUTER_API_KEY in a standalone deployment)
  • QWEN_MODEL (default: qwen/qwen3.8-max-0902)
  • QWEN_PROVIDER (default: alibaba; provider fallback is disabled)
  • RUBRIC_FAL_KEY (or FAL_KEY); no direct Google key needed
  • GEMINI_PROVIDER=fal
  • GEMINI_MODEL=google/gemini-3.8-flash
  • RUBRIC_AUDIO_BRIDGE_APPROVED=1 only for the agreed Gemini-observation/Qwen-generation method.

For this team deployment, set RUBRIC_STORAGE=hf-temporary, RUBRIC_AUTH_MODE=team, RUBRIC_TEAM_CODE_HASH as a protected salted hash, and RUBRIC_SESSION_SECRET. Never store the plaintext code in source or HF Variables. Rotate the code hash to invalidate existing sessions. Public source visibility does not make runtime reports public. No Supabase account or storage purchase is needed. Download reports before stop/restart/rebuild; runtime disk is temporary.

Verification commands

uv run ruff check rubric_app tests scripts
uv run pytest -q
npm --prefix web run typecheck
npm --prefix web run build
npm --prefix web run test:e2e
uv run python scripts/test_postgres.py
uv run python scripts/test_restart.py
docker build --tag rubric-workspace:local .
uv run python scripts/test_container.py

Install the test browser once with web/node_modules/.bin/playwright install chromium. Playwright owns a temporary test server and stops it. The PostgreSQL test owns a network-isolated disposable Docker container and removes it. Neither connects to production infrastructure.

tests/test_reference.py checks the supplied private workbook when RUBRIC_REFERENCE_WORKBOOK points to it; it is skipped in a portable checkout without that private fixture. It never sends those annotations to a model. tests/test_scale.py completes 300 tasks across all five types with synthetic providers and real generated media bytes. That is capacity proof, not live-model performance.

Research method and limits

Read method before collecting benchmark results. Configuration and guideline hashes are preserved per run. Once processing begins, method changes require a new run. Human answer columns are excluded throughout, including observation, scoring and comparison calls.

Video coverage is provider-dependent. Requested sampling is not proof every frame was inspected. Audio, consent/provenance uncertainty and inconsistent optional applicability can block a complete rank. The UI must not turn these into invented pass/fail results.

See operations for persistence, restart/retry, backup, retention and rollback. The included defaults are conservative local limits, not tested provider capability claims.