Commit History
-
f
convitom3a582a5
-
f
convitom3671ffb -
Stop tracking docs
convitoma20ed62
-
g
convitom3248df2
-
Xóa token bảo mật khỏi file cấu hình
convitom873c05e
-
g
convitom0fc2f94 -
đ
convitom75f4985 -
f
convitom215d2b1
-
f
convitom7326eb9
-
l
convitome114eb3
-
d
convitom7255a88 -
f
convitom6344cc7
-
f
convitom815a64a
-
f
convitom37bb3d9
-
g
convitom5250d5a -
f
convitomd1ed3ca
-
f
convitom3f2d758 -
f
convitom5a3b157 -
update vertex jobs and remove hardcoded secrets
convitomaade474
-
f
convitom2c84a70
-
f
convitom35d4872 -
f
convitom320063f
-
f
convitom0b18c4a -
f
convitomd313d81 -
p
convitom1c5d410
-
f
convitom8356dae -
fix(notebook): robust resume for streaming -- retry + per-shard done markers
convitom +1: Claude Opus 4.7Streaming downloads of ~880 MB shards hit transient ChunkedEncodingError / IncompleteRead on flaky cloud networks. Add: - _download_retry: 6 attempts with linear backoff; hf_hub_download resumes the partial download so each retry continues rather than restarts. - HF_HUB_DOWNLOAD_TIMEOUT=60 + commented hf_transfer toggle. - Per-shard '.done' markers under WORK_DIR (outside DST_TREE so never packed): a resumed run skips finished shards WITHOUT re-downloading them. - Final manifest recounts images/non-image from the actual tree so totals are correct even across resumed runs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>2bace46
-
feat(notebook): stream-process sharded HF source per-shard (low disk peak)
convitom +1: Claude Opus 4.7The source on hieu3636/cxr-vlm-data is also tar-sharded (under MIMIC-CXR_processed/shards/*.tar), not a raw image tree. The old notebook assumed a raw tree and would've failed snapshot_download. Rework into a streaming pipeline: list source shards on HF, then for each one download -> extract to scratch -> resize+copy into cumulative resized tree -> delete tar + scratch. Peak disk stays ~10 GB instead of ~200 GB. Implementation: - Split resize_tree into _walk_and_process (returns counts, no manifest) and a thin wrapper. Streaming orchestrator calls _walk_and_process per shard and writes ONE final manifest summarising all shards. - Autodetect arcname root on first shard (handles both 'files/...' and 'MIMIC-CXR_processed/files/...' layouts). - Output uploaded to MIMIC-CXR_resized/shards/ mirroring source convention. - Consumption snippet updated to the new shards/*.tar layout. - Removed now-redundant "## 4 Resize" + "## 5 Free source" cells (work is done in the streaming cell). …bb6ccbf -
d
convitom06883ee -
f
convitomc61f01a
-
g
convitome166fd8 -
feat(model): match frozen encoder dtype to LLM compute precision
convitom +1: Claude Opus 4.7Image encoder weights were loaded at FP32 (~340MB) while compute ran in BF16 under autocast, forcing a cast every forward and wasting VRAM. Now BioViLTEncoder accepts a `dtype` arg and CXRVisionLanguageModel passes `llm_dtype` (resolved once, shared with the LLM) so encoder weights live at BF16/FP16 directly. Also adds 3 GPU profile presets (L4/3090 default, T4, A100) in the Colab notebook so switching hardware is a comment swap. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>2cd8b5a -
f
convitomb961b41 -
fix(scripts): carry non-image files through resize + tar shards
convitom +1: Claude Opus 4.7The IMG_EXTS filter silently dropped reports (.txt), CheXpert labels (.csv), and any other metadata interleaved in the source tree, so the shipped MIMIC-CXR_resized would lose them. Add a parallel _copy_one worker that copies non-image files verbatim, preserving the exact tree. Manifest now records both counts. Applies to both build_resized_dataset.py and resize_and_shard.ipynb (cell c06). This makes the resized release a faithful mirror of MIMIC-CXR_processed regardless of what's mixed in -- needed if the instruct JSON builder is ever run on the training box (auto-build path reads .txt + chexpert.csv). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>0a99045
-
feat(scripts): offline dataset resize + tar-shard tooling for cloud training
convitom +1: Claude Opus 4.7Add build_resized_dataset.py and resize_and_shard.ipynb to pre-shrink the MIMIC-CXR tree to RAD-DINO's 518px working resolution and pack it into tar shards. Avoids re-decoding ~100GB of full-res JPGs every epoch (encoder is only ~1-2% of compute; the win is removing decode + transfer cost, not a feature cache which would be larger and inflexible). Shorter-edge>=518 keying matches the processor exactly so features are unchanged; --square optionally bakes the center-crop. Notebook reads/writes the HF dataset repo (parallel + resumable), no change to dataset.py/cxr_vlm.py. Also fixes leftover rad_dino.py docstring header from the earlier rename. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>47a30f7 -
chore(notebook): wire colab notebook to resolver-built MIMIC JSON
convitom +1: Claude Sonnet 4.6- MIMIC inline parse/build/VQA/filter/save cells → no-ops (the JSON is now auto-built by data.mimic_cxr_builder with PNU CheXpert + VQA). - cell-cfg: auto-discover *chexpert*.csv → mimic_chexpert_csv, set mimic_vqa_root from VQA_ROOT, mimic_auto_build=True; instruct_json is the base path the resolver suffixes. - Refresh section-3/4 markdown to match the new flow. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>c3bc6f2
-
f
convitom9dadb47 -
f
convitomc369576
-
initial commit
convitom28b13fc