2026-10-15 21:41:40.772 | INFO | app.tasks.entity.vlm.dataset:create:718 - Using pre-split dataset mode 2026-10-15 21:41:40.793 | INFO | app.tasks.entity.vlm.dataset:_download_current_splits:1472 - Downloading train dataset from data/__mlops__/dataset/entity/06a665f5-06bf-7f15-8000-acbbe709e051/train.json 2026-10-15 21:41:41.744 | INFO | app.tasks.entity.vlm.dataset:_download_split_documents:1317 - Processing %s %s/%s with %d documents 2026-10-15 21:42:57.578 | INFO | app.tasks.entity.vlm.dataset:_download_current_splits:1472 - Downloading val dataset from data/__mlops__/dataset/entity/06a665f5-06bf-7f15-8000-acbbe709e051/eval.json 2026-10-15 21:42:58.158 | INFO | app.tasks.entity.vlm.dataset:_download_split_documents:1317 - Processing %s %s/%s with %d documents 2026-10-15 21:43:05.966 | INFO | app.tasks.entity.vlm.dataset:_download_current_splits:1472 - Downloading test dataset from data/__mlops__/dataset/entity/06a665f5-06bf-7f15-8000-acbbe709e051/test.json 2026-10-15 21:43:06.662 | INFO | app.tasks.entity.vlm.dataset:_download_split_documents:1317 - Processing %s %s/%s with %d documents 2026-10-15 21:43:24.735 | INFO | app.tasks.entity.vlm.dataset:_download_miscellaneous:1520 - Downloading miscellaneous data from data/__mlops__/dataset/entity/06a665f5-06bf-7f15-8000-acbbe709e051/miscellaneous.json 2026-10-15 21:43:25.150 | INFO | app.tasks.entity.vlm.dataset:_download_previous_datasets:1426 - Found 0 previous dataset(s) to consolidate 2026-10-15 21:43:25.151 | INFO | app.tasks.entity.vlm.dataset:download_presplit_datasets:1577 - Successfully downloaded all datasets to /home/ubuntu/dataset 2026-10-15 21:43:54.082 | INFO | app.tasks.entity.vlm.dataset:remove_null_files:644 - Removed 2 files with null values 2026-10-15 21:43:55.254 | INFO | app.tasks.entity.vlm.reports:get_distribution:1363 - Original number of ground truth files 6058 2026-10-15 21:44:02.411 | INFO | app.tasks.entity.vlm.reports:get_distribution:1378 - After removing empty files 6058 2026-10-15 21:44:03.581 | INFO | app.tasks.entity.vlm.reports:get_entity_distribution:1470 - Creating entity distribution for 6058 files 2026-10-15 21:44:08.885 | DEBUG | app.commonmlops:get_oauth_token:695 - OAuth token request status: 200 2026-10-15 21:44:09.240 | DEBUG | app.commonmlops:get_training_job_queue:752 - Get training job queue status code: 200, api_url: https://aitest.api.dexit.tech/api/v1/mlops/trainingjobqueue/06a665f8-ebd0-7481-8000-83304d751ace 2026-10-15 21:44:09.371 | DEBUG | app.commonmlops:get_oauth_token:695 - OAuth token request status: 200 2026-10-15 21:44:09.373 | DEBUG | app.commonmlops:update_training_job_queue:798 - Updating training job queue with merged payload: {'id': '06a665f8-ebd0-7481-8000-83304d751ace', 'modeltype': 'entity', 'modelname': 'vlm_qwen3_6', 'jobtype': 'feedback', 'basemodelversionid': '019f0d56-3844-7c0a-8fab-4aa6a6d1f12e', 'datasetid': '06a665f5-06bf-7f15-8000-acbbe709e051', 'status': 'triggered', 'createddate': '2026-07-26T19:27:10.738422Z', 'createdby': '00000000-0000-0000-0000-000000000000', 'startdate': '2026-10-15T21:41:04.804606Z', 'enddate': None, 'accuracyreport': {'overall': {'recall': 0.9361295400877094, 'f1_score': 0.9377111962153638, 'precision': 0.939298206025048}, 'patient': {'recall': 0.9570468593654881, 'accuracy': 0.965850180851059, 'f1_score': 0.9578867442511313, 'precision': 0.9588553005522813}, 'facility': {'recall': 0.9681215596546779, 'accuracy': 0.9468811825459762, 'f1_score': 0.9706030656585861, 'precision': 0.9731011770220469}, 'patient-dob': {'recall': 0.9744463373083475, 'accuracy': 0.9529411764705882, 'f1_score': 0.9744463373083475, 'precision': 0.9744463373083475}, 'patient-sex': {'recall': 0.9991423670668954, 'accuracy': 0.9975903614457832, 'f1_score': 0.9987141020145736, 'precision': 0.998286203941731}, 'patient-zip': {'recall': 0.9623762376237623, 'accuracy': 0.9847389558232932, 'f1_score': 0.9808274470232089, 'precision': 1.0}, 'patient.dob': {'recall': 0.9821124361158433, 'accuracy': 0.9706582077716098, 'f1_score': 0.9842082799829279, 'precision': 0.9863130881094953}, 'patient.sex': {'recall': 0.9991423670668954, 'accuracy': 0.9983935742971888, 'f1_score': 0.9991423670668954, 'precision': 0.9991423670668954}, 'patient.zip': {'recall': 0.9663366336633663, 'accuracy': 0.9847512038523274, 'f1_score': 0.9809045226130653, 'precision': 0.9959183673469387}, 'sample-code': {'recall': 0.9727891156462585, 'accuracy': 0.9529314616019818, 'f1_score': 0.9723166585721223, 'precision': 0.9718446601941747}, 'facility-zip': {'recall': 0.9796782387806944, 'accuracy': 0.9759807846277022, 'f1_score': 0.9872013651877133, 'precision': 0.9948409286328461}, 'facility.zip': {'recall': 0.9847586790855207, 'accuracy': 0.9808, 'f1_score': 0.9897872340425531, 'precision': 0.9948674080410608}, 'patient-city': {'recall': 0.9756592292089249, 'accuracy': 0.9864, 'f1_score': 0.982635342185904, 'precision': 0.9897119341563786}, 'patient.city': {'recall': 0.9817444219066938, 'accuracy': 0.988, 'f1_score': 0.9847405900305187, 'precision': 0.9877551020408163}, 'phone-number': {'recall': 0.9856115107913669, 'accuracy': 0.9871794871794872, 'f1_score': 0.9716312056737588, 'precision': 0.958041958041958}, 'provider-npi': {'recall': 0.9970873786407767, 'accuracy': 0.9907251264755481, 'f1_score': 0.9946731234866828, 'precision': 0.9922705314009662}, 'facility-city': {'recall': 0.9914602903501281, 'accuracy': 0.9903614457831326, 'f1_score': 0.9948586118251928, 'precision': 0.998280309544282}, 'facility-name': {'recall': 0.9675491033304867, 'accuracy': 0.94296875, 'f1_score': 0.9687900812312953, 'precision': 0.9700342465753424}, 'facility.city': {'recall': 0.9965841161400513, 'accuracy': 0.9967871485943776, 'f1_score': 0.998289136013687, 'precision': 1.0}, 'facility.name': {'recall': 0.9624252775405636, 'accuracy': 0.9339035769828927, 'f1_score': 0.9636596836254809, 'precision': 0.9648972602739726}, 'patient-state': {'recall': 0.7080152671755725, 'accuracy': 0.7831412103746398, 'f1_score': 0.7114093959731543, 'precision': 0.7148362235067437}, 'patient.state': {'recall': 0.8530534351145038, 'accuracy': 0.8843226788432268, 'f1_score': 0.8546845124282982, 'precision': 0.8563218390804598}, 'facility-state': {'recall': 0.7342419080068143, 'accuracy': 0.602979274611399, 'f1_score': 0.7376979032948223, 'precision': 0.7411865864144453}, 'facility.state': {'recall': 0.9335604770017035, 'accuracy': 0.8867781155015197, 'f1_score': 0.936351986330628, 'precision': 0.9391602399314481}, 'collection-date': {'recall': 0.879646017699115, 'accuracy': 0.8856209150326797, 'f1_score': 0.8765432098765431, 'precision': 0.8734622144112478}, 'patient-address': {'recall': 0.8530612244897959, 'accuracy': 0.8933739527798934, 'f1_score': 0.8565573770491803, 'precision': 0.8600823045267489}, 'patient.address': {'recall': 0.9, 'accuracy': 0.9255813953488372, 'f1_score': 0.901840490797546, 'precision': 0.9036885245901639}, 'accession-number': {'recall': 0.9967948717948718, 'accuracy': 0.9991546914623838, 'f1_score': 0.9983948635634029, 'precision': 1.0}, 'facility-address': {'recall': 0.9496157130657558, 'accuracy': 0.9149922720247295, 'f1_score': 0.9528706083976006, 'precision': 0.9561478933791917}, 'facility.address': {'recall': 0.9632792485055508, 'accuracy': 0.9361370716510904, 'f1_score': 0.9649272882805817, 'precision': 0.9665809768637532}, 'patient-last-name': {'recall': 0.9863453815261044, 'accuracy': 0.9746031746031746, 'f1_score': 0.9871382636655949, 'precision': 0.9879324215607401}, 'patient.last-name': {'recall': 0.9879518072289156, 'accuracy': 0.9785202863961814, 'f1_score': 0.9891435464414957, 'precision': 0.9903381642512077}, 'patient-first-name': {'recall': 0.9895498392282959, 'accuracy': 0.9800955414012739, 'f1_score': 0.9899477281865702, 'precision': 0.9903459372485921}, 'patient.first-name': {'recall': 0.9911575562700965, 'accuracy': 0.9848242811501597, 'f1_score': 0.9923541247484909, 'precision': 0.9935535858178888}, 'patient-middle-name': {'recall': 0.8942307692307693, 'accuracy': 0.9538583929992045, 'f1_score': 0.8651162790697674, 'precision': 0.8378378378378378}, 'patient.middle-name': {'recall': 0.9519230769230769, 'accuracy': 0.9776, 'f1_score': 0.9339622641509433, 'precision': 0.9166666666666666}, 'other-insurance-name-1': {'recall': 0.7380952380952381, 'accuracy': 0.844015444015444, 'f1_score': 0.7542579075425792, 'precision': 0.7711442786069652}, 'other-insurance-name-2': {'recall': 1.0, 'accuracy': 0.9975903614457832, 'f1_score': 0.4, 'precision': 0.25}, 'social-security-number': {'recall': 0.8947368421052632, 'accuracy': 0.9863453815261044, 'f1_score': 0.8, 'precision': 0.723404255319149}, 'other-insurance-number-1': {'recall': 0.8380952380952381, 'accuracy': 0.9049520766773163, 'f1_score': 0.8554070473876063, 'precision': 0.8734491315136477}, 'other-insurance-number-2': {'recall': 1.0, 'accuracy': 0.9983935742971888, 'f1_score': 0.5, 'precision': 0.3333333333333333}, 'medicaid-insurance-number': {'recall': 0.8899297423887588, 'accuracy': 0.9167987321711569, 'f1_score': 0.8786127167630058, 'precision': 0.867579908675799}, 'medicare-insurance-number': {'recall': 0.9396551724137931, 'accuracy': 0.9632, 'f1_score': 0.9342857142857143, 'precision': 0.9289772727272727}}, 'metadata': {'clearml_details': {'hf_model_id': '314e/dexit-aitest-vlm_qwen3_6-entity-integration', 'clearml_task_id': 'e9800e9513dc4d6c8ef88eeefaa27952', 'trained_entities': ['accession_number', 'collection_date', 'facility_address', 'facility_city', 'facility_name', 'facility_state', 'facility_zip', 'medicaid_insurance_number', 'medicare_insurance_number', 'other_insurance_name_1', 'other_insurance_name_2', 'other_insurance_number_1', 'other_insurance_number_2', 'patient_address', 'patient_city', 'patient_dob', 'patient_first_name', 'patient_last_name', 'patient_middle_name', 'patient_sex', 'patient_state', 'patient_zip', 'phone_number', 'provider_npi', 'sample_code', 'social_security_number'], 'new_modelversion_tag': 'v10.0.0', 'base_modelversion_tag': 'v1.0.0'}, 'hyperparameters': {'DATASET_PARAMS': {'DATASET__EVAL_S3_PATH': 'data/__mlops__/dataset/entity/06a665f5-06bf-7f15-8000-acbbe709e051/eval.json', 'DATASET__TEST_S3_PATH': 'data/__mlops__/dataset/entity/06a665f5-06bf-7f15-8000-acbbe709e051/test.json', 'DATASET__TRAIN_S3_PATH': 'data/__mlops__/dataset/entity/06a665f5-06bf-7f15-8000-acbbe709e051/train.json', 'DATASET__MISCELLANEOUS_S3_PATH': 'data/__mlops__/dataset/entity/06a665f5-06bf-7f15-8000-acbbe709e051/miscellaneous.json'}, 'MODEL_CONFIGURATION': {'SEED': 42, 'MODELNAME': 'vlm_qwen3_6', 'MODELTYPE': 'entity', 'ENVIRONMENT': 'integration', 'HF_MODEL_ID': '314e/dexit-aitest-vlm_qwen3_6-entity-integration', 'MODEL_FAMILY': 'qwen', 'ARTIFACTS_DIR': '/home/ubuntu/result', 'MODELVERSIONTAG': 'v1.0.0', 'HF_BASE_MODEL_ID': 'Qwen/Qwen3.6-27B', 'IMAGE_PATCH_SIZE': 16, 'THRESHOLD_OFFSET': 0.05, 'TRAIN_OOTB_MODEL': True, 'NEW_MODELVERSIONTAG': 'v10.0.0'}, 'MODEL_HYPERPARAMETERS': {'RANK': 64, 'ALPHA': 128, 'TRAIN_EPOCHS': 2, 'EVAL_BATCH_SIZE': 1, 'GRAD_ACCUM_STEPS': 4, 'TRAIN_BATCH_SIZE': 4, 'INFERENCE_BATCH_SIZE': 8}}, 'workflow_details': {'temporal_workflow_id': 'local-model-retraining-entity-eaa77b21', 'skypilot_cluster_name': 'dexit-aitest-skypilot-entity-1792099901-189595', 'temporal_workflow_run_id': '01a1417a-36d9-71e5-9220-e123bfe7dadc'}, 'modelconfig_snapshot': {'modelname': 'vlm_qwen3_6', 'modeltype': 'entity', 'deployment': {'auto_model_deployment_manual': False, 'auto_model_deployment_feedback': False}, 'feedback_config': {'bucket_low': 0.5, 'bucket_high': 0.7, 'auto_trigger_retraining': True, 'non_corrected_sample_ratio': 0.8, 'min_feedback_count_for_retraining': 1500, 'low_bucket_threshold_offset_multiplier': 0.5, 'auto_trigger_retraining_on_new_entities': False, 'high_bucket_threshold_offset_multiplier': 1.5, 'low_precision_bucket_corrected_sample_rate': 0.7, 'mid_precision_bucket_corrected_sample_rate': 0.7, 'high_precision_bucket_corrected_sample_rate': 0.8, 'low_precision_bucket_threshold_sample_count': 100}, 'training_config': {'seed': 42, 'dataset': {'eval_size': 0.1, 'test_size': 0.2, 'train_size': 0.7}, 'cloud_providers': [{'accelerators': ['A100-80GB:1', 'H100:1'], 'provider_name': 'runpod'}], 'hyperparameters': {'rank': 64, 'alpha': 128, 'train_epochs': 2, 'grad_accum_steps': 4, 'train_batch_size': 4, 'inference_batch_size': 8}, 'hf_base_model_id': 'Qwen/Qwen3.6-27B', 'train_ootb_model': True, 'previous_datasets': {'eval': {'enabled': True, 'distribution': {'weights': {'n-1': 1.0}, 'strategy': 'equal', 'decay_factor': 0.8}, 'max_versions_to_include': 'all'}, 'test': {'enabled': True, 'distribution': {'weights': {'n-1': 1.0}, 'strategy': 'equal', 'decay_factor': 0.8}, 'max_versions_to_include': 'all'}, 'train': {'enabled': True, 'distribution': {'weights': {'n-1': 1.0}, 'strategy': 'equal', 'decay_factor': 0.8}, 'max_versions_to_include': 'all'}}}, 'threshold_offset': 0.05}}, 'deploymentstatus': None, 'deployeddate': None} 2026-10-15 21:44:09.667 | DEBUG | app.commonmlops:update_training_job_queue:801 - Update training job queue status code: 200, api_url: https://aitest.api.dexit.tech/api/v1/mlops/trainingjobqueue/06a665f8-ebd0-7481-8000-83304d751ace 2026-10-15 21:44:09.668 | SUCCESS | app.commonmlops:update_trained_entities:840 - Updated trained_entities on training job queue 2026-10-15 21:44:10.063 | INFO | app.tasks.entity.vlm.reports:upload_initial_artifacts:1539 - Uploaded intial artifacts to ClearML 2026-10-15 21:44:10.064 | INFO | __main__:_run_training_phase:635 - Moving into standard training loop 2026-10-15 21:44:10.064 | INFO | __main__:setup_directories:210 - Loading pre-split dataset statistics 2026-10-15 21:44:10.924 | INFO | __main__:load_presplit_statistics:187 - Found 4250 training samples 2026-10-15 21:44:11.204 | INFO | __main__:load_presplit_statistics:189 - Found 1245 test samples 2026-10-15 21:44:11.362 | INFO | __main__:load_presplit_statistics:191 - Found 563 validation samples 2026-10-15 21:44:11.362 | INFO | app.tasks.entity.vlm.dataset:print_split_statistics:843 - Dataset Split Statistics: 2026-10-15 21:44:11.363 | INFO | app.tasks.entity.vlm.dataset:print_split_statistics:844 - Total samples: 6058 2026-10-15 21:44:11.363 | INFO | app.tasks.entity.vlm.dataset:print_split_statistics:846 - Train set: 4250 (70.2%) 2026-10-15 21:44:11.363 | INFO | app.tasks.entity.vlm.dataset:print_split_statistics:847 - Test set: 1245 (20.6%) 2026-10-15 21:44:11.364 | INFO | app.tasks.entity.vlm.dataset:print_split_statistics:848 - Validation set: 563 (9.3%) 2026-10-15 21:44:11.366 | INFO | app.tasks.entity.vlm.dataset:print_split_statistics:861 - Split statistics saved to /home/ubuntu/result/split_statistics.json 2026-10-15 21:44:11.439 | INFO | __main__:standard_training_loop:332 - Initializing model 2026-10-15 21:44:11.439 | INFO | __main__:initialize_model:242 - Setting up model for retraining 2026-10-15 21:44:11.440 | INFO | __main__:initialize_model:245 - Training from base model: Qwen/Qwen3.6-27B 2026-10-15 21:44:11.440 | INFO | app.tasks.entity.vlm.model:__post_init__:251 - loading model Qwen/Qwen3.6-27B for training with unsloth 2026-10-15 21:51:26.943 | INFO | app.tasks.entity.vlm.model:calculate_max_steps:353 - num_samples: 4250, batch_size: 4, gradient_accumulation_steps: 4, epochs: 2 2026-10-15 21:51:26.943 | INFO | app.tasks.entity.vlm.model:calculate_max_steps:357 - calculating max steps 2026-10-15 21:51:26.992 | INFO | __main__:standard_training_loop:335 - Setting up trainer 2026-10-15 21:51:26.993 | INFO | __main__:create_trainer:284 - Datasets: 4250 train, 563 eval samples 2026-10-15 21:52:17.743 | INFO | __main__:standard_training_loop:338 - Starting training 2026-10-15 21:52:17.904 | INFO | app.tasks.entity.vlm.model:train_model:611 - everything is set up, launching training 2026-10-16 05:45:25.294 | INFO | app.tasks.entity.vlm.model:train_model:617 - training complete: TrainOutput(global_step=530, training_loss=0.2103868117872274, metrics={'train_runtime': 28350.7617, 'train_samples_per_second': 0.299, 'train_steps_per_second': 0.019, 'total_flos': 3.3570360018936955e+18, 'train_loss': 0.2103868117872274, 'epoch': 1.9934148635936029})