Commit History
-
bumped gradio version Running
John Ho797ec5b
-
fixed sam3 bug in loading the video tracking model
John Hod6f8728 -
fixed unsupported ScalarType BFloat16 error
John Ho4a13428
-
Add video_visual_inference: SAM2-style mask-prompt video tracking
John Ho +1: Claude Fable 5New MCP tool/endpoint, a drop-in match for the sam2 Space's process_video: seed objects with base64 PNG masks on a reference frame and propagate them through the video — forward, plus a reverse pass when ref_frame_idx > 0 — returning the same per-object-per-frame detections format (frame / track_id / normalized coco bbox / conf / mask_b64, with drop_masks). - Load Sam3TrackerVideoModel/Processor (transformers' SAM2-API-compatible PVS tracker) from the same facebook/sam3 checkpoint at import time; a fourth full model instantiation, so startup RAM grows accordingly. - Session uses the same OOM-safe device kwargs as video_inference (CPU frame storage/state, frames streamed to the GPU per step) and reuses extract_frames + calc_effective_fps, so sample_fps/every_x downsampling and the sampled-frame index semantics apply here too. - Seed all objects in a single add_inputs_to_inference_session call — each call overwrites session.obj_with_new_inputs (same trap as SAM2). - Split th…8da9cd2 -
Fix video session GPU OOM; add downsampling and estimated GPU duration
John Ho +1: Claude Fable 5Port of hfs-sam2 329199b to the SAM3 app: - video_inference: init_video_session left processing/storage/state on the inference device (the transformers default), normalizing and holding the whole video on the ZeroGPU slice — tens of GB transient on long videos. Keep frames on video_load_device (new param, default "cpu"; get_frame streams each one to the GPU on access) and park inference state in RAM. - Expose sample_fps / every_x on both video endpoints to downsample before tracking via the existing ffmpeg fps filter; detection frame indices are positions in the sampled sequence (documented in the MCP schema), and annotation mode muxes at the sampled fps to keep real-time duration. - calc_timeout_duration: an explicit timeout_duration is now a hard override; otherwise estimate the GPU lease in the main process (unbilled) from the ffprobe'd video length and downsampling (20s overhead + 0.5s/frame, clamped 30-300s, 120s fallback when the probe fails). Replace the f…2f50931
-
Cut ZeroGPU quota cost of image inference endpoints to 20s
John Ho +1: Claude Fable 5Split image_visual_inference and image_text_inference into undecorated public wrappers (unchanged API/MCP surface, validation and JSON parsing now run unbilled in the main process) plus private @spaces.GPU (duration=20) functions holding only the GPU work. Drops the requested lease from the 60s default, so visitors need 3x less quota to call, and invalid requests fail before a GPU is requested. Video endpoints keep their dynamic duration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>8aa0897
-
Rename image_text_inference output key track_id -> object_id
John Ho +1: Claude Opus 4.8 (1M context)image_text_inference does single-image concept segmentation, so "track_id" was misleading -- nothing is tracked across frames, it's just a per-image instance index. Rename the output key (and its docstring/MCP-schema description) to "object_id" for clarity. video_inference keeps "track_id" since there it genuinely tracks one object across frames. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>2e54a1b -
Rewrite MCP tool docstrings to survive Gradio's schema parser
John Ho +1: Claude Opus 4.8 (1M context)The four gr.Interface functions are exposed as MCP tools (mcp_server=True), whose schema is built from each function's docstring + type hints. Gradio's parser silently mangles anything off-shape, so the tools' schemas were broken: - video_inference: had no Args section (undocumented params) and a vague output description; added full output schema, named bbox format, and hints. - video_annotation: had no docstring at all; added one. - image_visual_inference: multi-line JSON arg descriptions produced phantom parameters; collapsed to single-line args and moved schema into prose. - image_text_inference: Returns section detail was discarded by the parser; folded field names/output schema into the description prose. Per the gradio-mcp-tool-docstrings skill: full output schema (with the albumentations bbox-format URL) lives in the description prose, one line per Args entry, no Returns section, and real return type hints added. Verified against a live /gradio_api/mcp/schema dump: corre…d37f0a7
-
updated transformers to 5.5.3
John Hoe4ca3dc -
fix HF Space build: pin transformers to release, drop kernels
John Ho +1: Claude Opus 4.8 (1M context)The Space crashed at startup with "ValueError: Either a revision or a version must be specified" from kernels/layer/layer.py: a newer kernels release made LayerRepository require a revision/version, but transformers' hub_kernels.py constructs it without one at import time. - Drop `kernels` from requirements.txt; transformers degrades gracefully when it's absent (the kernel-mapping block is import-guarded). - Pin `transformers==5.5.3` (first release shipping the SAM3 classes) instead of git main, for reproducible builds off a moving target. - Update CLAUDE.md deployment notes accordingly, incl. a do-not-add -kernels warning. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>70c7940 -
recruited claude
John Ho938f8ba -
add image segmentation tabs (visual + text prompts)
John Ho +1: Claude Opus 4.8 (1M context)Surface SAM3's image segmentation in the deployed app via two new Gradio tabs / api_names: - image_visual_inference: points/boxes prompting via Sam3TrackerModel. Drop-in match for hfs-sam2's process_image - same (im, variant, bboxes, points, point_labels) signature (variant ignored) and same flat list-of-base64-masks output. - image_text_inference: concept segmentation via Sam3Model, returning detection dicts (track_id/x/y/w/h/conf/mask_b64) consistent with video_inference. Image models load once at import into module globals (IMG_/TRK_) with the same None-fallback pattern as VID_MODEL. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>fb18a1c
-
updated kwargs parsing in calc_timeout_duration
John Hod8040be -
added timeout option
John Hocfd2182 -
only create annotated frames in annotation_mode
John Hoff0753c
-
fixed video_inference data output
John Ho538c7ad -
fixed bug in gradio interface
John Hoabb99f2 -
added a tab to gradio app for video annotation
John Ho0971e22 -
added assert statements
John Ho5c07489
-
added new function frames_to_vid
John Hoab54209 -
added new function frames_to_vid
John Hoec6dcd6 -
introduced ffmpret
John Ho61b7311
-
convert object_ids to list of int
John Ho83a4265 -
removed try-except for better debugging
John Ho6a1ec3d -
added in kernels and fixed bug in mask data type
John Ho41fa610 -
set device to cuda
John Ho18aa4a5 -
updated demo json output
John Ho2e155e5
-
added missing import
John Ho1345eda -
added missing import
John Hoabfad05 -
testing HF Space build with new transformers referece
John Ho7c9eabe -
testing gh action to push to HF Space
John Hoa22bc8d -
testing gh action to push to HF Space
John Ho33a6de4
-
Initial commit
John HoUnverified signaturea31ba73