Spaces:
Sleeping
Sleeping
Commit History
-
fix: pass ruff lint+format, clean README, remove stray script
Priya Monishafa0ce5b -
fix: production audit — security hardening + observability + code quality
Priya MonishaC-1 api/routes/classify.py: add missing logger (was NameError on unknown class) M-1 src/models/base_classifier.py: remove print() — logger.info already covers epoch progress m-1 src/data/dataset.py: assert -> ValueError for invalid split param SEC config.py + redis_cache.py + api/main.py: Redis password support (REDIS_PASSWORD env) SEC docker-compose.yml: conditional Redis auth (no-op when REDIS_PASSWORD empty) SEC docker-compose.airflow.yml: replace hardcoded airflow:airflow/admin:admin with env vars OBS api/prometheus_metrics.py: add http_errors Counter + models_loaded Gauge OBS api/main.py: middleware counts 4xx/5xx; models_loaded.set(1) on successful startup DOC .env.example: document REDIS_PASSWORD, AIRFLOW_DB_PASSWORD, AIRFLOW_ADMIN_PASSWORD6319415 -
section-12: CI + Tests + HuggingFace Deployment
Priya Monisha- pytest suite: 57 tests, 0 failures across health/classify/detect/drift/redis - RedisCache: _client= kwarg (keyword-only) for fakeredis injection in tests - conftest.py: MockClassificationModel (real nn.Module), synthetic confidence baseline, raise_server_exceptions=False, session-scoped test_image_bytes - pytest.ini: pythonpath=., coverage xml+term, strict-markers - .github/workflows/ci.yml: ruff lint+format → pytest, pip cache, coverage artifact - spaces.yaml: streamlit sdk, python_version=3.11, cpu-basic, 5 CV tags - packages.txt: libgl1, libglib2.0-0 for HF Spaces build - requirements.txt: fakeredis>=2.23.0 added to Testing section4d14020
-
section-9: KS drift detection + Prometheus -- DriftDetector, /drift/status, alert rules
Priya Monisha- src/monitoring/drift_detector.py: DriftDetector class Double-gate KS alert (stat > 0.10 AND p < 0.05) -- stat alone has ~95% FP rate at n=30 baseline Rate-limited KS_RUN_EVERY_N=10 -- prevents blocking asyncio event loop at 100 req/s Redis list buffer sv:drift:live:{class} with deque fallback (Rule 23) All 22 Gauge series initialized at startup -- Prometheus has time series from day 1 Redis buffer restoration on restart: reversed(get_list) preserves chronological order _safe_float() guards nan/inf from degenerate KS inputs (zero-variance live data) - api/routes/drift.py: GET /drift/status -- Pydantic DriftStatusResponse model - api/main.py: DriftDetector in lifespan after Redis init; drift router registered - api/routes/classify.py: record(top1, confidence) after inference; unknown_predictions Counter - api/prometheus_metrics.py: unknown_predictions Counter (surfaces invisible model failures) - src/inference/redis_cache.py: push_to_list (LPUSH+LTRIM pipeline) + get_list m…dacce7e -
section-8: FastAPI + Redis -- classify, detect, health, metrics endpoints
Priya Monisha- api/main.py: FastAPI lifespan (Rule 32), get_running_loop (Rule 31), non-blocking load_all_models via run_in_executor, eval_transform built once - api/routes/classify.py: POST /classify -- UploadFile+Form, Redis cache-aside, cached-key-pop fix (prevents TypeError on cache hit), Prometheus labels - api/routes/detect.py: POST /detect -- np.array(PIL) YOLO input, results[0].names, graceful 503 if YOLO unavailable - api/routes/health.py: GET /health -- 503 when loading (Docker healthcheck waits) - api/routes/metrics.py: GET /metrics -- CONTENT_TYPE_LATEST text/plain (Rule 30) - api/prometheus_metrics.py: Counter/Histogram singletons, class_name cardinality guard - api/schemas.py: ClassifyResponse, DetectResponse, HealthResponse Pydantic models - src/inference/model_loader.py: local-first weight resolution, HF Hub fallback with 120s ThreadPoolExecutor timeout, chunked MD5 hash, 3-pass warmup, FAST_MODE support - src/inference/redis_cache.py: socket_connect_timeout=1s (Rule 23), 32…9cb1db0 -
section-7: model comparison, MLflow 5-run logging, MobileNet val-split drift baseline
Priya Monisha- model_factory.py: add pretrained=True param to get_model + all _build_* helpers (pretrained=False enables topology-correct CPU benchmarking without weight download) - notebooks/06_model_comparison.py: 9-cell notebook * VGG16 metrics.json reconstructed (null precision/recall/F1/val_accuracy) * CPU benchmark: vgg16=228ms mobilenet=39ms efficientnet=51ms resnet50=115ms yolov8n=154ms * Comparison JSON: schema_version=1.0, champion=resnet50 (65.5%) * MLflow: smartvision_classification (4 runs) + smartvision_detection (1 run), WAL mode SQLite * Drift baseline: val split (not train), get_eval_transforms from augmentor.py, correct+incorrect conditional distributions, 22 .npy score files30e35d2
-
results: Section 5 complete — all 4 CNNs trained, ResNet50 best at 65.5%
Priya MonishaSection 5 final results: - VGG16: 59.5% (architecture ceiling) - MobileNetV2: 56.7% (backbone lr=1e-5 too conservative) - EfficientNetB0: 58.9% (backbone lr=1e-5 too conservative) - ResNet50: 65.5% test / 63.6% val (data volume bottleneck, 1464 params/img) Pipeline improvements committed alongside results: - dataset.py: create_stratified_split() — random stratified resplit, no sequential bias - loader.py: quality gates 48px→80px, 2% area; compute_overlap_ratio; check_person_overlap - model_factory.py: unfreeze_efficientnet_phase2() added - 01_data_acquisition.py: chair person-overlap filter, MAX_ITER 80k→150k Next: Section 6 YOLOv8 detection training1a28594
-
fix: MobileNetV2 Round 2 overfitting -- Section 5 Round 3 prep
Priya Monisha27.4pp train/val gap. Root cause: 552 params/training-img at features[14:] unfreeze on 3,080 samples. Realistic R3 target: 68-74%. metrics.json: stale fast_mode artifact replaced with confirmed Colab Round 2 values (62.3%, 23ep). Realistic accuracy target documented. augmentor.py: RandomGrayscale(p=0.05) + RandomPerspective(scale=0.2,p=0.3) ColorJitter 0.2->0.3, rotation 15->20deg, ZoomOut (1.3)->(1.4) model_factory.py: get_model() dispatcher + _build_* internal functions MobileNet: full classifier replaced (Dropout 0.2->0.4) unfreeze_mobilenet_phase2: features[14:] slice -> enumerate idx>=16 ~1.23M unfrozen (401/img) vs 1.71M (552/img) Round 2 _build_resnet50: freeze ALL then replace fc (unambiguous ownership) Removed from API: freeze_resnet50_phase1, freeze_vgg16_phase1, unfreeze_vgg16_phase2, count_trainable_params notebook: MobileNet block rewritten AdamW replaces Adam; Phase 1 wd 1e-4->1e-3 Phase 2 backbone lr=1e-5 (was 1e-4) -- highest-impact change Ph…58027ad
-
section-5: fix all RCAs (BN stats, ZoomOut order, fc unfreeze, grad_clip, checkpoint format)
Priya Monisha4e27aff
-
fix: VGG16 freeze ALL params before replacing classifier[6] — was leaving classifier[0,3] (~118M) trainable; now correct 102K trainable (0.08%)
Priya Monishad5923af -
section-5: CNN training framework — base_classifier, model_factory, 04_train_classifier
Priya Monishasrc/models/base_classifier.py: - train_one_epoch(): single epoch, supports GradScaler for mixed precision (EfficientNet) - evaluate(): accuracy, precision, recall, F1 (macro) via sklearn - train(): full loop with early stopping, saves best state_dict (Rule 5) - save_model() / load_model(): state_dict with weights_only=True (Rule 36) - benchmark_inference(): warmup + timed n=100 runs, returns ms/image src/models/model_factory.py: - get_model(name): vgg16, resnet50, mobilenet, efficientnet → 25-class head - VGG16: features frozen, classifier[-1] replaced (4096→25) - ResNet50: fc replaced (2048→25), caller manages 2-phase freeze/unfreeze - MobileNetV2: features frozen, classifier[-1] (1280→25) - EfficientNetB0: features frozen, classifier[-1] (1280→25) - freeze_resnet50_phase1() / unfreeze_resnet50_phase2() helpers - count_trainable_params() notebooks/04_train_classifier.py: - Parameterized: set MODEL = "vgg16"|"resnet50"|"mobilenet"|"efficientnet", run once each - FAST_MODE=False local…fecb3f1 -
fix: RandomZoomOut fill=128 (mid-grey) — black fill normalized to [-2.12,-2.04,-1.80] creates artificial boundary signal; mid-grey normalizes to ~0
Priya Monishafebce91 -
section-4: preprocessing + augmentation — verified end-to-end
Priya Monishasrc/data/preprocessor.py: - validate_yolo_annotations(): 1550 train + 375 val files, 13758 annotations all valid Fixed: assert <= 1.0 not < 1.0 (bbox_to_yolo clamps to min(1.0,...) so boundary values valid) - create_yolo_data_yaml(): absolute paths, works in terminal + Colab - verify_dataset_structure(): counts per split, detects image/label mismatches src/data/augmentor.py: - get_train_transforms(): flip, rotate±15, colorjitter, zoomout, resize 224, ToImage, Normalize(ImageNet) - get_eval_transforms(): resize + normalize only - denormalize(): for visualization - torch wrapped in try/except (local fbgemm.dll issue) notebooks/03_preprocessing.py: - Dataset verified: 2500 classification, 1925 detection, labels match images - YOLO annotations validated: 13758 annotations, all in valid range - data.yaml regenerated with current absolute paths - Augmented sample images saved to docs/figures/ - Tensor shapes confirmed: (3, 224, 224) float32, range ~[-2.1, 2.6] after ImageNet norm2ea8a2d -
fix: remove emoji from loader.py print (Windows cp1252 encoding error)
Priya Monisha261ffe4 -
section-2: dataset acquisition — loader, dataset, helpers, 01_data_acquisition notebook
Priya MonishaKey corrections from mentor notebook analysis: - config.py: replaced COCO original IDs (1-90) with HF 0-indexed IDs (0-79) via SELECTED_CLASSES dict and HF_CATEGORY_TO_CLASS_IDX reverse mapping - Fixed to 25 classes (mentor's notebook had 26 by accidentally including 'train' vehicle) - Confirmed bbox format: [x,y,w,h] absolute pixels (not [xmin,ymin,xmax,ymax]) - No trust_remote_code needed for detection-datasets/coco src/utils/helpers.py: - NumpyEncoder (torch.Tensor → np.bool_ → np.integer → np.floating → np.ndarray) - save_json/load_json with NumpyEncoder - HF Hub helpers: upload_model_to_hub, download_model_from_hub (HfApi().upload_file) - create_hub_repo with exist_ok=True src/data/loader.py: - verify_coco_mapping() using SELECTED_CLASSES + HF_CATEGORY_TO_CLASS_IDX - Checkpoint/resume (mentor had none — Colab disconnect = start over) - save_classification_crop: [x,y,w,h] → PIL.crop((x,y,x+w,y+h)) → resize 224×224 - save_detection_sample: [x,y,w,h] → YOLO normalized format - Pr…9b0ae38 -
section-0+1: project foundation — config.py, requirements, gitignore, project-docs
Priya Monisha- .claude/settings.json: permission allowlist + PostToolUse git-commit hook - .claude/rules/: no_model_in_pages, fast_mode, one_model_per_turn - config.py: Pydantic Settings + all 25 CLASSES with COCO ID comments + 38-rule reference - requirements.txt / requirements-docker.txt / requirements-hf.txt (CPU-only PyTorch index-url) - .gitignore: anchored /data/raw/, /models/*.pt, notebooks/*.ipynb, /runs/ - pyrightconfig.json: extraPaths=["."] for Pylance import resolution - All src/ and api/ __init__.py files - project-docs: 38 rules table, progress tracker, locked decisions - CLAUDE_CONTEXT.md: full session bootstrap contextd091717