ark-shopify commited on
Commit
95ac4af
·
unverified ·
1 Parent(s): 743d605

fix: Launchers - HuggingFace - Workaround 2 for the breaking changes in hf download behavior

Browse files

Previously, hf download returned the download directory path. Not the program spams STDOUT with download progress (instead of writing to STDERR) and also uses color despite not being executed in interactive terminal.

HF version 1.11 is not supported on Python 3.9 which some components use. So pinning HF version to the last available version: 1.8.0.

huggingface_overlay/cloud_pipelines_backend/launchers/huggingface_launchers.py CHANGED
@@ -213,7 +213,7 @@ class HuggingFaceJobsContainerLauncher(
213
  # -L, --dereference always follow symbolic links in SOURCE
214
  input_download_lines = [
215
  f'mkdir -p "$(dirname "{container_path}")"'
216
- f' && snapshot_dir=`hf download --repo-type "{hf_uri.repo_type}" "{hf_uri.repo_id}" --include "{hf_uri.path}*" --format quiet`'
217
  f' && cp -r -L "$snapshot_dir/{hf_uri.path}" "{container_path}"'
218
  for hf_uri, container_path in (
219
  (huggingface_repo_storage.HuggingFaceRepoUri.parse(uri), container_path)
@@ -244,7 +244,7 @@ fi
244
 
245
  export PATH="$HOME/.local/bin:$PATH"
246
 
247
- uv tool install 'huggingface_hub==1.11.0' --python '>=3.9'
248
  hf version
249
 
250
  # Downloading the input data
@@ -262,7 +262,7 @@ set -e +x
262
 
263
  exit_code=`cat "$exit_code_path"`
264
 
265
- hf upload --repo-type '{hf_repo_uri.repo_type}' '{hf_repo_uri.repo_id}' '{container_outputs_root}' '{path_in_repo}' --format quiet
266
  exit "$exit_code"
267
  """
268
 
 
213
  # -L, --dereference always follow symbolic links in SOURCE
214
  input_download_lines = [
215
  f'mkdir -p "$(dirname "{container_path}")"'
216
+ f' && snapshot_dir=`hf download --repo-type "{hf_uri.repo_type}" "{hf_uri.repo_id}" --include "{hf_uri.path}*" --quiet`'
217
  f' && cp -r -L "$snapshot_dir/{hf_uri.path}" "{container_path}"'
218
  for hf_uri, container_path in (
219
  (huggingface_repo_storage.HuggingFaceRepoUri.parse(uri), container_path)
 
244
 
245
  export PATH="$HOME/.local/bin:$PATH"
246
 
247
+ uv tool install 'huggingface_hub==1.8.0' --python '>=3.9'
248
  hf version
249
 
250
  # Downloading the input data
 
262
 
263
  exit_code=`cat "$exit_code_path"`
264
 
265
+ hf upload --repo-type '{hf_repo_uri.repo_type}' '{hf_repo_uri.repo_id}' '{container_outputs_root}' '{path_in_repo}' --quiet
266
  exit "$exit_code"
267
  """
268