| --- |
| title: FineRobotics Annotation Studio |
| emoji: 🤖 |
| colorFrom: indigo |
| colorTo: blue |
| thumbnail: https://huggingface.co/spaces/finerobotics/annotation-studio/resolve/main/assets/studio-cover.png |
| sdk: docker |
| app_port: 7860 |
| fullWidth: true |
| hf_oauth: true |
| hf_oauth_scopes: |
| - read-repos |
| - contribute-repos |
| - inference-api |
| - jobs |
| hf_oauth_expiration_minutes: 1440 |
| short_description: Robot video → subtasks, plans and memory |
| license: apache-2.0 |
| models: |
| - nvidia/EgoHand-v1 |
| - nvidia/RT-DETR-Hand-Detector-v1 |
| - nvidia/SOMA-X |
| - nvidia/Flora-Captioner |
| - nvidia/Flora-TempSeg |
| - Qwen/Qwen3.8-27B |
| - Qwen/Qwen3.6-27B |
| - google/gemma-4-31B-it |
| - Qwen/Qwen3.8-Flash-Next |
| - zai-org/GLM-5.3-Flash |
| - moonshotai/Kimi-K3 |
| - MiniMaxAI/MiniMax-M3 |
| - google/gemma-4-26B-A4B-it |
| - Qwen/Qwen3-VL-235B-A22B-Instruct |
| tags: |
| - robotics |
| - lerobot |
| - annotation |
| --- |
| |
| # FineRobotics Annotation Studio |
|
|
|  |
|
|
| **Hand actions** uses EgoHand with native SOMA/MHR hands and |
| **DROID-W + MoGe Small** inside the existing annotation wizard. Choose a human-POV |
| dataset camera and an HF Jobs GPU. Measured calibration comes from dataset |
| metadata by default. An explicit experimental option estimates pinhole focal length |
| with MoGe when calibration is absent, assuming a centered camera and no distortion. |
| These outputs record estimated intrinsics; the calibrated accuracy reference does |
| not apply. Aria Gen1 image rotation is explicit. The annotated copy retains every |
| source frame, writes 156D actions and native SOMA parameters, and masks frames |
| outside directly observed bimanual spans of at least 30 frames. This recipe has |
| no motion infiller and needs no MANO files. Model assets come from a pinned, |
| checksummed package embedded in Annotation Studio's public image. The Space build fetches |
| the large weights from public upstream sources; Jobs need no model-repository access. |
| Browsing and estimating public datasets need no login. Sign in to launch HF Jobs |
| or access private datasets. |
| The worker image and asset identities are verified before native inference. |
| The same Space supplies the UI, Flora Jobs and hand-action Jobs; separate worker |
| Spaces are unnecessary. Each hand estimate pins the currently running Studio image |
| and rejects a later deployment until the user calculates a fresh estimate. |
| Results link to the official |
| [`lerobot/visualize_dataset`](https://huggingface.co/spaces/lerobot/visualize_dataset). |
|
|
| The measured development reference is 39.41–41.04 input FPS on RTX 5090 over ten |
| short calibrated clips, including startup, camera and native export, excluding |
| installation/model downloads. HF GPU throughput is unmeasured. Setup, per-GPU |
| TensorRT compilation and dataset copying add time. See the expandable pipeline |
| and model-term details in the wizard; research outputs remain private. |
|
|
| Enter a LeRobot v3 dataset, choose annotations, review an estimate, then sign in |
| with Hugging Face to run a Job using the selected annotation model. General vision |
| models use a CPU Job and Inference Providers; Flora uses an H200 GPU Job. |
| Jobs and inference are billed to the signed-in user's personal account. |
| The Space itself uses free CPU hardware and has no shared HF access token. |
|
|
| The Space has moved from `lerobot/annotation-studio` to |
| [`finerobotics/annotation-studio`](https://huggingface.co/spaces/finerobotics/annotation-studio). |
|
|
| ## Execution and independent Flora subtask labeling |
|
|
| The worker uses the scalable `lerobot.annotations.runtime` engine from |
| [LeRobot #4892](https://github.com/huggingface/lerobot/pull/4892), with checksummed |
| episode outputs, explicit stage dependencies, cache reuse and bounded workers. |
| Language annotations complete before human-video generation starts, so media |
| failures preserve the completed language output. The existing Studio controller |
| continues to manage dataset subsets, progress, budget reservations and provider |
| limits. Checkpoints live on the Job's disk; starting a different Job does not yet |
| resume another Job's ephemeral disk. |
|
|
| Human videos directly use `HumanVideoModule` from |
| [LeRobot #4809](https://github.com/huggingface/lerobot/pull/4809), rebased onto #4892. |
| A Studio adapter preserves the quoted first-frame-only workflow, full-resolution |
| image edits, spending/time guards and the media viewer. End-frame editing remains |
| available in the CLI but is not included in the Studio's single-edit quote. |
|
|
| In the **Model** selector, choose **NVIDIA Flora · TempSeg + Captioner** for an |
| independent end-to-end subtask run. Flora-TempSeg predicts local intervals using |
| NVIDIA's native 16-second windows, adaptive processing and continuation stitching. |
| Flora-Captioner labels the reconstructed events using BEFORE/TARGET/AFTER context. |
| Neither stage consumes Qwen3.8 predictions or existing dataset subtask labels. |
| The selected source episode task is supplied as context; empty tasks do not |
| trigger Qwen task derivation. Native overlapping events and gaps are retained. |
| Both workflows save full `{start, end, text}` spans in each episode's |
| `annotation_studio/run.json` entry, in addition to the normal Parquet language |
| rows (which store starts). Compare the two runs on the same source revision, |
| episodes and camera. This compares the two complete annotation workflows. |
|
|
| Flora uses NVIDIA's pinned native frame planner, video processor, mixed BF16/FP32 |
| loader, greedy decoder and trained prompts. Custom Subtasks prompts must be reset |
| before selecting Flora. The input uses the selected single robot camera, with |
| constant-rate episode timestamps and at least 4 source FPS. Source-limited or |
| failed captions fail explicitly; there is no fallback to Qwen. |
|
|
| Flora has no hosted Inference Provider route. Selecting it runs the worker on an |
| H200 CUDA Job; the quote reserves the entire GPU timeout, including installation, |
| weight downloads and startup. Subtasks and default numbered plans need no hosted |
| VLM calls or token reservation. Other annotations (memory, rephrasings, |
| interjections, visual questions, human-video prompts) and custom plan prompts |
| still use the displayed helper VLM through Inference Providers; their cost and |
| provider are shown only when required. Subtask timestamps and labels remain |
| independent in these runs too. The default Qwen option keeps the CPU workflow. |
| [Flora's model card](https://huggingface.co/nvidia/Flora-Captioner) limits use to |
| noncommercial research. Code and weight revisions are pinned in `flora_captioner.py` |
| and each run manifest, including TempSeg's adapter and base revisions. This |
| option is experimental. An initial live H200 run produced six annotated episodes; |
| nineteen other episodes hit a duration-clock bug, now fixed. CPU tests cover |
| native window planning, stitching, fractional-duration clips and the absence of |
| external VLM calls. `benchmark_flora.py` is the older |
| caption-only comparison on shared reference intervals, not the independent |
| end-to-end comparison offered by the selector. |
|
|
| On the H200, the worker retains TempSeg and Captioner in separate native sessions. |
| Each model is loaded and verified once, then reused across episodes; sampling, |
| weights, precision and generation settings stay unchanged. This mode requires at |
| least 120 GiB of GPU memory. Episode durations are rounded to the native integral- |
| microsecond clock before scheduling, including 30-fps clips with fractional seconds. |
|
|
| The default **tuned Qwen3.8** preset uses the action-phase prompt developed on |
| ABC, timestamped contact sheets at 2 fps, 1,200 frames per request, and up to |
| 80 subtasks per window. It retries failed segmentation with the original |
| 30-second-window preset. This is an experimental prompt preset, not finetuned |
| weights; its small held-out benchmark did not establish a quality improvement. |
| ## Models |
|
|
| The default is **Qwen3.8 27B / Novita**, with up to 32 parallel requests. |
| Standard choices also include Qwen3.6 27B / OVHcloud and Gemma 4 31B / Novita. |
| The selector offers nine curated VLMs from the |
| families listed in this card. This is a practical selection, not a universal |
| quality ranking. All nine available routes passed a live two-image JSON smoke test on |
| 2026-09-25. Only Qwen3.8 27B has a 32-request concurrency test and the recent |
| 33-episode all-annotation run; the new routes use a conservative eight requests, |
| while Flash Next retains its tested two-request limit. Provider/account limits vary. |
|
|
| `model_catalog.py` owns the allowed model IDs, explicit provider routes, |
| provider-specific generation parameters, checked pricing and context limits. |
| Quotes and jobs pin these together. Model changes invalidate the browser quote; |
| launch checks the selected provider is live and rejects changed catalog prices. |
| The worker uses the pinned generation parameters, including non-Qwen models. |
| There is no silent provider failover, so the reviewed data recipient and rates |
| stay explicit. New model routes require a smoke test and a reviewed rate card. |
|
|
| Rates shown are USD per million tokens. Cached input rates are accounted for; |
| MiniMax M3 switches to its higher pricing tier for prompts of 524,288 tokens or |
| more. The budget reserves the worst tier and full provider context before each |
| request, then reconciles actual reported usage. Small budgets reduce parallelism; |
| a target too small for one conservative reservation is rejected before launch. |
| The action-phase prompt was developed on Flash Next; quality results do not |
| transfer automatically to other models. |
|
|
| ## Outputs |
|
|
| - **Annotated dataset copy:** a new standalone LeRobot dataset containing **only |
| successfully annotated episodes**, with annotations in the standard LeRobot Parquet columns. |
| No separate annotation JSONL files are uploaded. Every RGB camera video is trimmed to those |
| episodes and re-encoded as H.264 (CRF 18); shared source video files are never |
| copied wholesale. Source numeric data, tools and other annotation styles within |
| copied episodes are retained. Indices, tasks, episode metadata and statistics |
| are rebuilt. `annotation_studio/episode_mapping.json` maps new episode IDs back |
| to the source. Original license files and the source README are retained. |
| Failed, stopped and unselected episodes are not included. Each completed copy is |
| published atomically with its metadata so a partial run remains loadable. |
| Depth video is currently unsupported. |
| Storage charges are separate from the run estimate and spending target. |
|
|
|
|
| Setup is split into four focused steps: Dataset, Annotations, Model, and Review. |
| Examples and provider details expand on demand. Prompt edits and the current draft |
| are retained when moving between steps; changing priced settings requires a new estimate. |
|
|
| The user chooses **Private** (default) or **Public** at review. |
| The launch confirmation shows the output visibility. |
| Visibility is pinned in the job manifest and applied only to the new output repository. |
| Older manifests without a visibility field remain private. |
|
|
| Subtasks are generated as a dependency of plans, memory and interjections; |
| the UI makes this explicit. Unchecked output styles are filtered from the saved annotations. |
| All annotation types use one selected RGB video camera per run; views are not |
| combined. Camera names load automatically from the dataset's `meta/info.json` |
| before estimating, including private datasets after login. Depth and non-video |
| features are excluded. Visual questions have a configurable cadence. |
|
|
| ## Flora execution profiles |
|
|
| Normal Studio jobs retain the validated reference decoder. An opt-in CUDA worker |
| image (`Dockerfile.flora`) bakes in pinned Torch/Transformers and fused |
| `flash-linear-attention` / `causal-conv1d` kernels. `FLORA_EXECUTION=accelerated` |
| fails before model downloads if those kernels are unavailable. It uses bounded |
| left-padded batches (default 4, OOM splitting), preserves sample/video order, |
| greedy decoding, model weights, FP32 adapters/GDN parameters and native sampling. |
| Captioner assets download while TempSeg is working; every native integrity check |
| still runs. Both profiles now publish episode copies while annotation continues, |
| using private Parquet slices so inference source files remain immutable. |
|
|
| The accelerated profile remains opt-in until a full GPU benchmark verifies |
| runtime, memory and annotation validity. A 15-minute target for 25 ABC episodes |
| is not a measured performance claim. CPU contract tests cannot establish CUDA |
| numerical equivalence or the throughput of these released weights. |
|
|
| ## Human demo videos |
|
|
| A separate **Video generation** category on the Annotations step, next to the text |
| annotations. For every subtask (or up to 1–8 per episode), the job turns the robot view into |
| a first-person human demonstration, following LeRobot's `--human_video` annotation module |
| and the video-generation stages of [Zero-WAM's HumanGen pipeline](https://robbyant-research.github.io/Zero-WAM/): |
|
|
| 1. the selected vision model reads the robot segment and writes the human task and an |
| image-edit prompt; |
| 2. `Qwen/Qwen-Image-Edit-2509` removes the robot from the segment's first full-resolution |
| frame and adds two resting human hands; |
| 3. the vision model writes a step-by-step video prompt from the edited frame; |
| 4. `MiniMaxAI/MiniMax-H3` animates it at 480P (5 s). |
|
|
| Both media models run on fal through HF Inference Providers with the signed-in user's |
| token. Selecting videos also selects Subtasks, since there is one video per subtask. |
|
|
| The estimate uses the selected episodes' total seconds and count, calibrated on the first |
| video run (3 ReBot episodes, 159.7 s, 33 subtasks): **1 subtask per episode + 1 per 5.3 s**, |
| up to 1.5× that in the upper bound, with the per-episode cap applied. Each video is priced at |
| fal's list prices: the edit at $0.03 per output megapixel (1024×1024) and the 5 s 480p video |
| at $0.05/s, about $0.28 in total, plus about 1,750 input and 550 output VLM tokens. |
| **Video limit per episode** selects all subtasks or the first 1–8 subtasks of each |
| episode, generating one video per selected subtask. An episode with fewer subtasks |
| produces fewer videos. |
| **Videos generated at once** (1–32, default 8) limits simultaneous image edits and |
| video generations across the whole job. Higher values can finish sooner and spend |
| the budget faster, without changing the per-video rates. The review warns when the expected videos at that parallelism would outlast the |
| job timeout (about 3 min per video). |
|
|
| Videos start only after an episode's annotations pass validation. Each request holds a |
| conservative reservation (`model_catalog.HUMAN_VIDEO`) from the spending target while in |
| flight and is reconciled to its list price when it returns. A failed video is recorded and |
| the others continue. Videos stop at the spending target, or before the runtime limit (the job |
| timeout when none is set), without paying for an edit whose video could not finish; the |
| episode's annotations are kept. The launch checks that both fal mappings are live. |
|
|
| The output holds `annotation_studio/human_videos/episode_XXXXXX/segment_XXX{.mp4, |
| _first_frame.png,_robot_frame.png}` (named by source episode) and `index.json`, which lists |
| each segment's subtask, timestamps, prompts, status and materialized `episode_index`. |
| These are synthetic media, not language columns. The Jobs page previews completed videos |
| next to the robot frame, served through the Space so private outputs stay private. |
|
|
| ## Prompts and examples |
|
|
| Every annotation card has an **Edit prompt** editor with the actual model |
| instructions, required placeholders, and a reset-to-default action. Interjections |
| have separate opening-acknowledgement and mid-task templates. Edits are validated, |
| kept in the browser draft, and pinned in the output run manifest. Subtask edits |
| also apply to internal dependencies and shorter-window recovery. Other unchecked |
| annotation types retain edits in the draft without using them in the run. |
|
|
| Default plans are deterministic remaining-step lists. A custom plan prompt opts |
| into model generation at each boundary; the cost estimate includes those extra |
| calls. Resetting it restores the deterministic default. Longer custom prompts |
| also increase estimated input usage. Instructions can change segment counts and |
| output lengths, so costs remain estimates. |
|
|
| The main screen shows two illustrative examples per annotation type. Subtask |
| examples include timed reach/grasp/move/place phases and a continuous wiping |
| segment, making the default granularity visible before launching a job. |
|
|
| ## Cost and privacy |
|
|
| Estimates are ranges, based on metadata duration, sampled frames, approximate |
| tokens and the provider's published rates. CPU pricing is retrieved from HF. |
| Before each request, the worker reserves a conservative token cost; successful |
| calls reconcile with usage and uncertain failures retain their reservation. |
| The selected total target reserves the entire job timeout's CPU cost first. |
| This is an application spending guard, not an HF billing guarantee: provider |
| price changes, billing differences and storage charges remain outside it. |
| Set account billing limits on Hugging Face as an additional control. |
|
|
| OAuth tokens stay on the server and are passed only as HF Job secrets. They are |
| not stored in repositories, browser storage, browser outputs or job commands. |
| Private metadata uses temporary per-request caches. Dataset content is sent |
| to the selected inference provider only after the user starts a job. |
|
|
| ## Reproducibility and operations |
|
|
| Jobs pin the Space code and source dataset revisions. `pipeline-source.tar.gz` |
| contains the LeRobot source used for this deployment; `pipeline-version.json` |
| records its Git commit and archive checksum. Each output includes its manifest, |
| per-episode progress and metered inference usage. Completed shards are uploaded |
| incrementally. A budget stop or error produces an explicitly partial result. |
| The worker never executes code from an input dataset. |
|
|
| The deployment helper packages tracked LeRobot source, checks for tracked |
| changes, tests the package checksum and uploads this directory. The existing |
| `lerobot/annotate` manual-labeling Space is independent. |
|
|
| ## Local development |
|
|
| ```sh |
| cd spaces/annotation-studio/frontend && bun install && bun run build |
| # From spaces/annotation-studio: |
| uv run --with-requirements requirements.txt uvicorn server:app --port 7860 |
| uv run --no-sync pytest spaces/annotation-studio/test_studio.py |
| uv run --no-sync python spaces/annotation-studio/deploy.py |
| ``` |
|
|
| The Docker Space serves a React frontend and a FastAPI backend from one origin. |
| The interface adapts visualizer design tokens and annotation colors; attribution |
| is in `THIRD_PARTY.md`. OAuth uses Authorization Code with PKCE and HttpOnly |
| session cookies. Tokens live in server memory; restarting the Space signs users |
| out. A refreshable dashboard lists each user's jobs, progress and cancellation. |
|
|
| Hugging Face login is available on the deployed Space. Local mode is preview |
| only: it cannot launch jobs using a developer's cached credentials. |
|
|
| The Jobs dashboard refreshes every 15 seconds and shows usage-derived cost: |
| provider-reported input/output/cache tokens at the pinned rates, plus HF-reported |
| CPU runtime rounded to minutes. Unknown request charges remain separately |
| reserved; this is not an invoice. Worker usage checkpoints are saved every ten |
| seconds. The billing link opens the user's final HF charges. |
|
|
| Finished annotated copies offer **Show annotations**, opening the first |
| materialized episode with `?tab=annotations` in the visualizer. Private results |
| require signing in there. Older JSONL-only exports retain their Results download link. |
|
|
| Hand-annotated copies offer **View 3D hands** once an episode is materialized, |
| opening synchronized wrists, hand skeletons and camera overlays with `?tab=hands`. |
| Operators can set `HAND_VISUALIZER_URL` to a compatible visualizer deployment; |
| the default is the official LeRobot dataset visualizer. |
|
|
