# Live AR attachment for every protocol: implementation design Status: researched against the current app; not integrated into the Flutter app. The included server runs full-precision BiomedCLIP classification only. SAM 2, landmark additions and the tracking coordinator below are proposed additions, not implemented features of this Space package. ## What the current app actually has Inspected project: `C:/Users/135mo/Desktop/Prototype`. - `lib/services/visual_anchor_tracker.dart` already wraps native class-independent optical flow and accepts an arbitrary normalized `Rect` seed. - Android `VisualTrackerBridge.kt` supplies its native implementation. - `lib/services/biomed_clip_classifier.dart` permits automatic borrowing of a detector target for only seven classifier labels. The classifier supplies no bounding boxes; `compatibleTarget` requires one compatible independent box. - `lib/src/ar/reliable_pose_tracker.dart` starts manual attachment if no verified coordinates exist. `_postDetection` filters reacquisition by `targetType`. - `lib/services/body_pose_tracker.dart` already supplies optional ML Kit pose context, throttled to 450 ms. This is body context, not a wound boundary. Therefore all classes can already enter texture tracking after a suitable manual seed. Recognition-dependent automatic seed/reacquisition limits their practical coverage. Removing safety checks without a replacement source of coordinates would attach overlays to unrelated skin or the wrong person. ## Recommended separation of responsibilities 1. BiomedCLIP: image/text matching for choosing a potential care flow. 2. General segmentation: initial visible target mask, independent of injury name. 3. Native optical flow: frequent, local updates of that same target. 4. Pose/face/hand landmarks: observed anatomical reference for relevant protocols. 5. Attachment coordinator: chooses the reference required by the current step, tracks target identity and uncertainty, and publishes one render transform. The injury classifier does not run per display frame. The visual tracker does not need to know whether a selected patch is a burn, bite, blister or other finding. ## Initialization is the essential missing operation Use SAM 2.1 Tiny on a dedicated GPU as the first segmentation candidate. Prompt it with a point or box from the exact source frame. SAM 2's image/video tools support promptable segmentation and temporal mask propagation. They are not validated wound-boundary detectors and cannot guarantee the correct target on unprompted skin photographs. Initialization order: 1. Reuse a confident, relevant independent localization when available, or use the scan's aiming region as a candidate-selection constraint. 2. Get prompted candidate masks for the target region. Reject masks covering most of the background/person or unrelated disconnected regions. 3. If there is no reliable choice, the user taps the affected area once. Do not treat the image classifier's center crop as a detected injury location. 4. Initialize optical flow on the source frame and propagate to the current frame. A server result refers to its old frame, not the present camera image. Automatic image classification plus automatic target choice is not solved by BiomedCLIP alone. If absolutely no target selection is permitted, a separately evaluated localizer is needed. A SAM mask or CLIP attention map does not remove that requirement. Mask proposal ranking with BiomedCLIP can be an experiment, but is not assumed reliable for skin injury localization. ## Fast local update path - Send oriented camera luminance to the existing native tracker, avoiding JPEG, base64, HTTP and Flutter rebuilds on its frequent frame path. - Track texture inside and immediately around the selected region. Surrounding skin can be more stable than wet blood, fluid or reflective injury surfaces. - Use forward/backward flow checks and robust affine estimation; reject poor inlier support, sudden implausible scale changes and inconsistent motion. - Warp the target mask and associated guidance transform with the estimated motion. A global affine warp is only an approximation for deforming skin. - Use timestamp-aware smoothing and bounded short prediction, with parameters selected on physical phone sequences. Heavy smoothing can create visible lag. - Render independently of inference using the latest accepted state. Never process a backlog of camera frames. Initial engineering target: local tracking around 30 updates/s with a separately rendered display. Segmentation refresh is occasional and quality-driven. These are design targets, not measured rates on the user's Android phone. Use EdgeTAM or EfficientTAM as later candidates if genuine local mask propagation is needed. Published mobile examples do not establish Android Flutter integration, injury performance, or speed on this phone. Do not replace a functioning native tracker before those are measured. ## Every protocol needs the appropriate target | Protocol/step type | Attachment reference | | --- | --- | | Local visible skin injury | Selected region mask plus surrounding skin texture | | Pressure, dressing, rinsing at an injury | Same region identity throughout steps | | Compression or whole-torso illustration | Body landmarks plus explicit alignment where necessary | | Limb support/elevation/positioning | Relevant observed limb joints and tracked local surface | | Eye/nose/mouth region | Face landmarks plus a selected local target when finer placement is needed | | Finger/hand guidance | Hand landmarks plus local injury target | | Recovery positioning/full-body illustration | Observed body landmarks; missing joints remain unavailable | | No visible lesion / nonvisual emergency | Relevant body region or non-positional instruction | | Calling EMS or retrieving supplies | No fictitious body attachment required | Pose landmarks locate an approximate body reference, not a clinically verified compression point, tooth socket, wound boundary or treatment success. Partial CPR manikins need explicit alignment if the pose model cannot observe them. Keep the existing body pose path initially. Add MediaPipe Face/Hand Landmarker only for protocols that require those targets. Do not enable all models on all frames. Give the current step an explicit attachment type rather than attempting to infer anatomy from arbitrary instruction text. ## Identity, occlusion and reacquisition Maintain target_id, camera_generation, source_frame_id, image geometry, initial template, current mask/transform, supporting anatomical region, and last reliable observation time. Changing protocol steps does not create a new target. States: searching -> acquiring -> tracking -> uncertain/occluded -> lost. When tools/hands cover the injury, surrounding observations may support a coarse predicted position. Mark it uncertain; do not claim the covered wound is seen. Hide precise placement when support is no longer reliable. Text guidance remains available. Do not silently attach to a nearby bruise, another limb or a second person. Reacquisition matches the stored target's appearance and body location. It does not require an injury class to exist in the old YOLO label list. If several targets are plausible, require a new selection rather than changing identity silently. Return server masks with their source frame ID; warp through retained recent motion history or discard stale responses. Never apply old pixels to a new view. ## Server request boundaries Classification request: frame_id + optional selected ROI -> ranked class matches. Segmentation request (future endpoint): frame_id + point/box -> mask + quality. Local frame update: previous target + current luminance -> current transform. Keep one request in flight per inference task, discard old work and use separate target generations after camera handoff. Dedicated GPU deployment can reduce inference time; WAN round trips cannot provide guaranteed display-frame timing. ## Validation before app rollout - Test translation, distance, rotation, lighting, and camera coordinate mapping. - Include actual skin, partial-body manikins, low-texture surfaces and small ROIs. - Include another person entering, camera leaving/re-entering, temporary complete occlusion, similar neighboring marks, bandage coverage, and step changes. - Annotate target positions/masks on held-out video clips. Measure target error, mask IoU, identity switches, false-lock duration, reacquisition time and actual end-to-end render latency. Report failures as well as averages. - Compare texture-only, landmark-assisted and periodically resegmented variants on the same clips. No guarantee of perfect attachment for every view. This makes every protocol eligible for an appropriate observed attachment. It does not make hidden injuries visible or produce measured depth/physical scale. ARCore depth/occlusion would be a separate device-capability-dependent feature. ## Primary references - BiomedCLIP: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224 - SAM 2: https://github.com/facebookresearch/sam2 - Native optical flow: https://docs.opencv.org/4.x/dc/d6b/group__video__track.html - MediaPipe Pose: https://developers.google.com/edge/mediapipe/solutions/vision/pose_landmarker/android - MediaPipe Face: https://developers.google.com/edge/mediapipe/solutions/vision/face_landmarker - MediaPipe Hand: https://developers.google.com/edge/mediapipe/solutions/vision/hand_landmarker - EdgeTAM: https://github.com/facebookresearch/EdgeTAM - EfficientTAM: https://github.com/yformer/EfficientTAM