|
Download TRACKING_DESIGN.md from m4az19/saef-api: direct link, hf CLI and curl.
- Browser
- Download file 9.68 kB
-
https://huggingface.co/spaces/m4az19/saef-api/resolve/main/TRACKING_DESIGN.md
- Command line
-
hf download hf://spaces/m4az19/saef-api/TRACKING_DESIGN.md
-
curl -L -o TRACKING_DESIGN.md https://huggingface.co/spaces/m4az19/saef-api/resolve/main/TRACKING_DESIGN.md
9.68 kB
| # Live AR attachment for every protocol: implementation design | |
| Status: researched against the current app; not integrated into the Flutter app. | |
| The included server runs full-precision BiomedCLIP classification only. SAM 2, | |
| landmark additions and the tracking coordinator below are proposed additions, | |
| not implemented features of this Space package. | |
| ## What the current app actually has | |
| Inspected project: `C:/Users/135mo/Desktop/Prototype`. | |
| - `lib/services/visual_anchor_tracker.dart` already wraps native class-independent | |
| optical flow and accepts an arbitrary normalized `Rect` seed. | |
| - Android `VisualTrackerBridge.kt` supplies its native implementation. | |
| - `lib/services/biomed_clip_classifier.dart` permits automatic borrowing of a | |
| detector target for only seven classifier labels. The classifier supplies no | |
| bounding boxes; `compatibleTarget` requires one compatible independent box. | |
| - `lib/src/ar/reliable_pose_tracker.dart` starts manual attachment if no verified | |
| coordinates exist. `_postDetection` filters reacquisition by `targetType`. | |
| - `lib/services/body_pose_tracker.dart` already supplies optional ML Kit pose | |
| context, throttled to 450 ms. This is body context, not a wound boundary. | |
| Therefore all classes can already enter texture tracking after a suitable manual | |
| seed. Recognition-dependent automatic seed/reacquisition limits their practical | |
| coverage. Removing safety checks without a replacement source of coordinates | |
| would attach overlays to unrelated skin or the wrong person. | |
| ## Recommended separation of responsibilities | |
| 1. BiomedCLIP: image/text matching for choosing a potential care flow. | |
| 2. General segmentation: initial visible target mask, independent of injury name. | |
| 3. Native optical flow: frequent, local updates of that same target. | |
| 4. Pose/face/hand landmarks: observed anatomical reference for relevant protocols. | |
| 5. Attachment coordinator: chooses the reference required by the current step, | |
| tracks target identity and uncertainty, and publishes one render transform. | |
| The injury classifier does not run per display frame. The visual tracker does not | |
| need to know whether a selected patch is a burn, bite, blister or other finding. | |
| ## Initialization is the essential missing operation | |
| Use SAM 2.1 Tiny on a dedicated GPU as the first segmentation candidate. Prompt | |
| it with a point or box from the exact source frame. SAM 2's image/video tools | |
| support promptable segmentation and temporal mask propagation. They are not | |
| validated wound-boundary detectors and cannot guarantee the correct target on | |
| unprompted skin photographs. | |
| Initialization order: | |
| 1. Reuse a confident, relevant independent localization when available, or use | |
| the scan's aiming region as a candidate-selection constraint. | |
| 2. Get prompted candidate masks for the target region. Reject masks covering | |
| most of the background/person or unrelated disconnected regions. | |
| 3. If there is no reliable choice, the user taps the affected area once. Do not | |
| treat the image classifier's center crop as a detected injury location. | |
| 4. Initialize optical flow on the source frame and propagate to the current | |
| frame. A server result refers to its old frame, not the present camera image. | |
| Automatic image classification plus automatic target choice is not solved by | |
| BiomedCLIP alone. If absolutely no target selection is permitted, a separately | |
| evaluated localizer is needed. A SAM mask or CLIP attention map does not remove | |
| that requirement. Mask proposal ranking with BiomedCLIP can be an experiment, | |
| but is not assumed reliable for skin injury localization. | |
| ## Fast local update path | |
| - Send oriented camera luminance to the existing native tracker, avoiding JPEG, | |
| base64, HTTP and Flutter rebuilds on its frequent frame path. | |
| - Track texture inside and immediately around the selected region. Surrounding | |
| skin can be more stable than wet blood, fluid or reflective injury surfaces. | |
| - Use forward/backward flow checks and robust affine estimation; reject poor | |
| inlier support, sudden implausible scale changes and inconsistent motion. | |
| - Warp the target mask and associated guidance transform with the estimated | |
| motion. A global affine warp is only an approximation for deforming skin. | |
| - Use timestamp-aware smoothing and bounded short prediction, with parameters | |
| selected on physical phone sequences. Heavy smoothing can create visible lag. | |
| - Render independently of inference using the latest accepted state. Never | |
| process a backlog of camera frames. | |
| Initial engineering target: local tracking around 30 updates/s with a separately | |
| rendered display. Segmentation refresh is occasional and quality-driven. These | |
| are design targets, not measured rates on the user's Android phone. | |
| Use EdgeTAM or EfficientTAM as later candidates if genuine local mask propagation | |
| is needed. Published mobile examples do not establish Android Flutter integration, | |
| injury performance, or speed on this phone. Do not replace a functioning native | |
| tracker before those are measured. | |
| ## Every protocol needs the appropriate target | |
| | Protocol/step type | Attachment reference | | |
| | --- | --- | | |
| | Local visible skin injury | Selected region mask plus surrounding skin texture | | |
| | Pressure, dressing, rinsing at an injury | Same region identity throughout steps | | |
| | Compression or whole-torso illustration | Body landmarks plus explicit alignment where necessary | | |
| | Limb support/elevation/positioning | Relevant observed limb joints and tracked local surface | | |
| | Eye/nose/mouth region | Face landmarks plus a selected local target when finer placement is needed | | |
| | Finger/hand guidance | Hand landmarks plus local injury target | | |
| | Recovery positioning/full-body illustration | Observed body landmarks; missing joints remain unavailable | | |
| | No visible lesion / nonvisual emergency | Relevant body region or non-positional instruction | | |
| | Calling EMS or retrieving supplies | No fictitious body attachment required | | |
| Pose landmarks locate an approximate body reference, not a clinically verified | |
| compression point, tooth socket, wound boundary or treatment success. Partial | |
| CPR manikins need explicit alignment if the pose model cannot observe them. | |
| Keep the existing body pose path initially. Add MediaPipe Face/Hand Landmarker | |
| only for protocols that require those targets. Do not enable all models on all | |
| frames. Give the current step an explicit attachment type rather than attempting | |
| to infer anatomy from arbitrary instruction text. | |
| ## Identity, occlusion and reacquisition | |
| Maintain target_id, camera_generation, source_frame_id, image geometry, initial | |
| template, current mask/transform, supporting anatomical region, and last reliable | |
| observation time. Changing protocol steps does not create a new target. | |
| States: searching -> acquiring -> tracking -> uncertain/occluded -> lost. | |
| When tools/hands cover the injury, surrounding observations may support a coarse | |
| predicted position. Mark it uncertain; do not claim the covered wound is seen. | |
| Hide precise placement when support is no longer reliable. Text guidance remains | |
| available. Do not silently attach to a nearby bruise, another limb or a second | |
| person. | |
| Reacquisition matches the stored target's appearance and body location. It does | |
| not require an injury class to exist in the old YOLO label list. If several targets | |
| are plausible, require a new selection rather than changing identity silently. | |
| Return server masks with their source frame ID; warp through retained recent | |
| motion history or discard stale responses. Never apply old pixels to a new view. | |
| ## Server request boundaries | |
| Classification request: frame_id + optional selected ROI -> ranked class matches. | |
| Segmentation request (future endpoint): frame_id + point/box -> mask + quality. | |
| Local frame update: previous target + current luminance -> current transform. | |
| Keep one request in flight per inference task, discard old work and use separate | |
| target generations after camera handoff. Dedicated GPU deployment can reduce | |
| inference time; WAN round trips cannot provide guaranteed display-frame timing. | |
| ## Validation before app rollout | |
| - Test translation, distance, rotation, lighting, and camera coordinate mapping. | |
| - Include actual skin, partial-body manikins, low-texture surfaces and small ROIs. | |
| - Include another person entering, camera leaving/re-entering, temporary complete | |
| occlusion, similar neighboring marks, bandage coverage, and step changes. | |
| - Annotate target positions/masks on held-out video clips. Measure target error, | |
| mask IoU, identity switches, false-lock duration, reacquisition time and actual | |
| end-to-end render latency. Report failures as well as averages. | |
| - Compare texture-only, landmark-assisted and periodically resegmented variants | |
| on the same clips. No guarantee of perfect attachment for every view. | |
| This makes every protocol eligible for an appropriate observed attachment. It | |
| does not make hidden injuries visible or produce measured depth/physical scale. | |
| ARCore depth/occlusion would be a separate device-capability-dependent feature. | |
| ## Primary references | |
| - BiomedCLIP: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224 | |
| - SAM 2: https://github.com/facebookresearch/sam2 | |
| - Native optical flow: https://docs.opencv.org/4.x/dc/d6b/group__video__track.html | |
| - MediaPipe Pose: https://developers.google.com/edge/mediapipe/solutions/vision/pose_landmarker/android | |
| - MediaPipe Face: https://developers.google.com/edge/mediapipe/solutions/vision/face_landmarker | |
| - MediaPipe Hand: https://developers.google.com/edge/mediapipe/solutions/vision/hand_landmarker | |
| - EdgeTAM: https://github.com/facebookresearch/EdgeTAM | |
| - EfficientTAM: https://github.com/yformer/EfficientTAM | |