2026-03-22 23:43:09,773 [INFO] ==> reporting status to http://host.docker.internal:18787/api/jobs/status
2026-03-22 23:43:09,774 [INFO] ==> config loaded
2026-03-22 23:43:09,776 [INFO] ==> job_name: trainer-e2e-job_e2e_1774222987861
2026-03-22 23:43:09,777 [INFO] ==> job_id: job_e2e_1774222987861
2026-03-22 23:43:09,777 [INFO] ==> config source: http://host.docker.internal:18787/api/v1/trainer/jobs/job_e2e_1774222987861/bootstrap?token=cfg_836915adce2f337d2264246143c15c255d9ae74b10e6f126
2026-03-22 23:43:09,800 [INFO] ==> reporting status to http://host.docker.internal:18787/api/jobs/status
2026-03-22 23:43:10,074 [INFO] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK"
2026-03-22 23:43:10,249 [INFO] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK"
2026-03-22 23:43:10,250 [INFO] ==> Hugging Face upload ready for account: XProger
2026-03-22 23:43:10,252 [INFO] ==> preparing assets
2026-03-22 23:43:10,252 [INFO] ==> reporting status to http://host.docker.internal:18787/api/jobs/status
2026-03-22 23:43:19,148 [INFO] ==> starting training
2026-03-22 23:43:19,150 [INFO] ==> loading model: /app
2026-03-22 23:43:19,151 [INFO] ==> logical base model id: Qwen/Qwen2.5-7B-Instruct
2026-03-22 23:43:19,151 [INFO] ==> method: qlora
2026-03-22 23:43:19,152 [INFO] ==> reporting status to http://host.docker.internal:18787/api/jobs/status
2026-03-22 23:43:19,459 [INFO] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK"
2026-03-22 23:43:19,942 [INFO] HTTP Request: GET https://huggingface.co/api/whoami-v2 "HTTP/1.1 200 OK"
2026-03-22 23:43:28,508 [INFO] ==> reporting status to http://host.docker.internal:18787/api/jobs/status
2026-03-22 23:43:29,000 [INFO] HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/json/json.py "HTTP/1.1 200 OK"
2026-03-22 23:43:32,143 [INFO] Unsloth: Padding-free batching auto-enabled for SFTTrainer instance.
2026-03-22 23:43:32,334 [WARNING] num_proc must be <= 2. Reducing num_proc to 2 for dataset of size 2.
2026-03-22 23:43:37,654 [WARNING] num_proc must be <= 1. Reducing num_proc to 1 for dataset of size 1.
2026-03-22 23:43:41,594 [INFO] ==> training started
2026-03-22 23:43:42,839 [INFO] ==> reporting status to http://host.docker.internal:18787/api/jobs/status
2026-03-22 23:43:50,075 [INFO] ==> reporting progress to http://host.docker.internal:18787/api/jobs/progress
2026-03-22 23:43:50,729 [INFO] ==> reporting progress to http://host.docker.internal:18787/api/jobs/progress
2026-03-22 23:43:50,963 [INFO] ==> reporting progress to http://host.docker.internal:18787/api/jobs/progress
2026-03-22 23:43:51,219 [INFO] ==> reporting progress to http://host.docker.internal:18787/api/jobs/progress
2026-03-22 23:43:51,348 [INFO] ==> reporting progress to http://host.docker.internal:18787/api/jobs/progress
2026-03-22 23:43:51,568 [INFO] ==> reporting progress to http://host.docker.internal:18787/api/jobs/progress
2026-03-22 23:43:51,580 [INFO] ==> training finished
2026-03-22 23:43:51,580 [INFO] ==> saving lora adapters to /output/job_e2e_1774222987861/lora/trainer-e2e-job_e2e_1774222987861
2026-03-22 23:43:51,595 [INFO] ==> reporting progress to http://host.docker.internal:18787/api/jobs/progress
2026-03-22 23:43:51,612 [INFO] ==> reporting status to http://host.docker.internal:18787/api/jobs/status
2026-03-22 23:43:51,630 [INFO] ==> reporting status to http://host.docker.internal:18787/api/jobs/status
2026-03-22 23:43:51,766 [INFO] ==> saving merged model to /output/job_e2e_1774222987861/merged/trainer-e2e-job_e2e_1774222987861
2026-03-22 23:43:51,766 [INFO] ==> reporting status to http://host.docker.internal:18787/api/jobs/status
2026-03-22 23:43:51,766 [INFO] ==> merged 16-bit save may take several minutes for 7B model
2026-03-22 23:44:16,075 [INFO] ==> merged model saved
2026-03-22 23:44:16,079 [INFO] ==> training artifacts saved
2026-03-22 23:44:16,079 [INFO] ==> reporting status to http://host.docker.internal:18787/api/jobs/status
2026-03-22 23:44:16,079 [INFO] ==> training finished
2026-03-22 23:44:16,088 [INFO] ==> starting evaluation
2026-03-22 23:44:16,132 [INFO] ==> reporting status to http://host.docker.internal:18787/api/jobs/status
2026-03-22 23:44:22,174 [INFO] ==> reporting status to http://host.docker.internal:18787/api/jobs/status
2026-03-22 23:44:22,232 [INFO] ==> reporting progress to http://host.docker.internal:18787/api/jobs/progress
2026-03-22 23:44:22,356 [INFO] ==> reporting status to http://host.docker.internal:18787/api/jobs/status
2026-03-22 23:44:22,357 [INFO] ==> evaluation finished
2026-03-22 23:44:22,403 [INFO] ==> reporting status to http://host.docker.internal:18787/api/jobs/status
2026-03-22 23:44:22,469 [INFO] ==> creating archive /output/job_e2e_1774222987861/trainer-e2e-job_e2e_1774222987861.lora.tar from /output/job_e2e_1774222987861/lora/trainer-e2e-job_e2e_1774222987861 (mode=w, files=6, size_bytes=21538633)
2026-03-22 23:44:22,500 [INFO] ==> archive ready: /output/job_e2e_1774222987861/trainer-e2e-job_e2e_1774222987861.lora.tar
2026-03-22 23:44:22,500 [INFO] ==> uploading archive to http://host.docker.internal:18787/api/jobs/upload/lora: /output/job_e2e_1774222987861/trainer-e2e-job_e2e_1774222987861.lora.tar.gz
2026-03-22 23:44:22,502 [ERROR] upload step failed: lora_archive
Traceback (most recent call last):
  File "/trainer/app/pipeline/upload_runner.py", line 85, in _safe_upload
    result = operation()
             ^^^^^^^^^^^
  File "/trainer/app/pipeline/upload_runner.py", line 155, in <lambda>
    lambda: self._archive_and_upload_dir(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/trainer/app/pipeline/upload_runner.py", line 64, in _archive_and_upload_dir
    self.archiver.upload_archive(
  File "/trainer/app/pipeline/archiver.py", line 190, in upload_archive
    with open(archive_path, "rb") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/output/job_e2e_1774222987861/trainer-e2e-job_e2e_1774222987861.lora.tar.gz'
2026-03-22 23:44:22,504 [INFO] ==> creating archive /output/job_e2e_1774222987861/trainer-e2e-job_e2e_1774222987861.merged.tar from /output/job_e2e_1774222987861/merged/trainer-e2e-job_e2e_1774222987861 (mode=w, files=9, size_bytes=15242728923)
2026-03-22 23:44:33,444 [INFO] ==> archive ready: /output/job_e2e_1774222987861/trainer-e2e-job_e2e_1774222987861.merged.tar
2026-03-22 23:44:33,444 [INFO] ==> uploading archive to http://host.docker.internal:18787/api/jobs/upload/merged: /output/job_e2e_1774222987861/trainer-e2e-job_e2e_1774222987861.merged.tar.gz
2026-03-22 23:44:33,445 [ERROR] upload step failed: merged_archive
Traceback (most recent call last):
  File "/trainer/app/pipeline/upload_runner.py", line 85, in _safe_upload
    result = operation()
             ^^^^^^^^^^^
  File "/trainer/app/pipeline/upload_runner.py", line 169, in <lambda>
    lambda: self._archive_and_upload_dir(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/trainer/app/pipeline/upload_runner.py", line 64, in _archive_and_upload_dir
    self.archiver.upload_archive(
  File "/trainer/app/pipeline/archiver.py", line 190, in upload_archive
    with open(archive_path, "rb") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/output/job_e2e_1774222987861/trainer-e2e-job_e2e_1774222987861.merged.tar.gz'
2026-03-22 23:44:33,449 [INFO] ==> creating archive /output/job_e2e_1774222987861/trainer-e2e-job_e2e_1774222987861.full.tar from /output/job_e2e_1774222987861 (mode=w, files=36, size_bytes=15291110697)
2026-03-22 23:44:44,091 [INFO] ==> archive ready: /output/job_e2e_1774222987861/trainer-e2e-job_e2e_1774222987861.full.tar
2026-03-22 23:44:44,091 [INFO] ==> uploading archive to http://host.docker.internal:18787/api/jobs/upload/full-archive: /output/job_e2e_1774222987861/trainer-e2e-job_e2e_1774222987861.full.tar.gz
2026-03-22 23:44:44,091 [ERROR] upload step failed: full_archive
Traceback (most recent call last):
  File "/trainer/app/pipeline/upload_runner.py", line 85, in _safe_upload
    result = operation()
             ^^^^^^^^^^^
  File "/trainer/app/pipeline/upload_runner.py", line 182, in <lambda>
    lambda: self._archive_and_upload_dir(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/trainer/app/pipeline/upload_runner.py", line 64, in _archive_and_upload_dir
    self.archiver.upload_archive(
  File "/trainer/app/pipeline/archiver.py", line 190, in upload_archive
    with open(archive_path, "rb") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/output/job_e2e_1774222987861/trainer-e2e-job_e2e_1774222987861.full.tar.gz'
2026-03-22 23:44:44,136 [INFO] ==> reporting status to http://host.docker.internal:18787/api/jobs/status
2026-03-22 23:44:44,391 [INFO] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict"
2026-03-22 23:45:21,012 [INFO] HTTP Request: POST https://huggingface.co/api/models/XProger/test/preupload/main "HTTP/1.1 200 OK"
2026-03-22 23:45:21,184 [INFO] HTTP Request: POST https://huggingface.co/XProger/test.git/info/lfs/objects/batch "HTTP/1.1 200 OK"
2026-03-22 23:45:21,438 [INFO] HTTP Request: GET https://huggingface.co/api/models/XProger/test/xet-write-token/main "HTTP/1.1 200 OK"
2026-03-22 23:45:48,946 [WARNING] No files have been modified since last commit. Skipping to prevent empty commit.
2026-03-22 23:45:49,203 [INFO] HTTP Request: GET https://huggingface.co/api/models/XProger/test/revision/main "HTTP/1.1 200 OK"
2026-03-22 23:45:49,384 [INFO] HTTP Request: POST https://huggingface.co/api/repos/create "HTTP/1.1 409 Conflict"
2026-03-22 23:45:49,567 [INFO] HTTP Request: POST https://huggingface.co/api/models/XProger/test/preupload/main "HTTP/1.1 200 OK"