AGENTS.md

HF Jobs Development: operator and agent guide

This extension turns a Hugging Face Job into a remote GPU execution target while keeping the editor and source tree local. Use the Code OSS commands for the interactive workflow and the hf-dev CLI primitives for automation.

Mental model

local Git checkout (authoritative)
    |
    | Mutagen one-way replica / sync-file
    v
HF Job:/workspace/hf-dev/<repo-key>-<repo-name>
    |
    | editable install of the synchronized checkout
    v
remote Python + CUDA
    |
    | debugpy over an SSH local port forward
    v
local Code OSS debugger and local breakpoints

The local checkout is always authoritative. Do not make source changes only on the Job: Mutagen runs in one-way-replica mode and can overwrite or remove remote-only changes. Edit locally, save, and let the extension sync.

Requirements

• hf >= 1.28.0, authenticated with hf auth login.
• A token allowed to create/use Jobs and read the required Hub models. Bucket permissions are required only when the optional persistent cache is explicitly enabled.
• An SSH public key registered at https://huggingface.co/settings/keys.
• ssh, scp, and Mutagen installed locally. A Homebrew Mutagen installation is supported on macOS.
• hf-dev installed on PATH, normally at ~/.local/bin/hf-dev.
• The Code OSS Python and debugpy extensions for F5 debugging.

Check the local prerequisites without starting paid compute:

hf auth whoami
hf-dev doctor --repo "$PWD"
hf-dev plan --repo "$PWD" --flavor h200 --timeout 30m

plan prints the exact hf jobs run command but does not create a Job.

Normal Code OSS workflow

1. Open the local repository in Code OSS.
2. Run HF Dev: Start or Reconnect GPU Job from the command palette or click the HF Dev status item.
3. Select hardware and one of the hard limits: 30m, 1h, or 8h.
4. Edit and save files locally. Save events synchronize the file to the active Job.
5. Open the Python entry point and press F5. On macOS, use fn/Globe + F5 unless the function row is configured as standard F-keys. F5 reveals the HF Dev Output channel without moving editor focus; remote stdout, stderr, and Hub progress are streamed there rather than to hf jobs logs.
6. Run HF Dev: Stop GPU Job as soon as the GPU is no longer needed.

F5 is intercepted only for a Python editor when hfDev.active is true and no debug session is already active. During an attached debug session, F5 keeps its normal Continue behavior. Without an active HF development Job, local F5 is unchanged. HF Dev permits only one remote debugger per workspace: ending the local attach stops the matching remote process, and a newer debug launch replaces an older process left by a disconnected editor.

What hf-dev up does

For a newly created Job, hf-dev up performs these operations in order:

1. Resolve configuration and verify the current hf CLI version.
2. If and only if cache_bucket is explicitly configured, create or reuse that private Bucket and include it as a read-write HF_HOME mount. The default MVP creates and mounts no Bucket.
3. Submit an SSH-enabled detached Job with the chosen image, flavor, and native Hub timeout. Without the opt-in mount, Hugging Face libraries use the image's ephemeral Job-local cache.
4. Start a detached local watchdog. At the same hard deadline it checks the Job and sends hf jobs cancel if the Job remains active. It retries after temporary network loss. The native Hub timeout remains a second guard.
5. Wait until SSH accepts connections.
6. Discover a usable remote Python interpreter (python3, python, common absolute paths, or an explicit setting).
7. Install the matching Mutagen agent through SSH and create a one-way replica of the local repository.
8. Bootstrap the remote environment. The default installs debugpy and installs the synchronized repository editable, preferring uv pip install -e . and falling back to python -m pip install -e ..
9. Import transformers and verify that it resolves to <remote_root>/src/transformers, not the image's preinstalled site-packages copy.

If setup of a newly created Job fails, hf-dev terminates its Mutagen session and attempts to cancel the Job immediately.

CLI primitives for people and agents

All repository-scoped commands accept --repo PATH; when omitted, hf-dev resolves the current Git root.

Inspect without changing state

# Local recorded state only. Fast, but it can be stale after cancellation in the Hub UI.
hf-dev info --repo "$PWD"

# Local state plus authoritative remote Job status.
hf-dev info --repo "$PWD" --remote

# Authoritative Job details for the recorded Job.
hf-dev status --repo "$PWD"

# Check tools and versions without contacting Jobs.
hf-dev doctor --repo "$PWD"

# Preview a launch without starting paid compute.
hf-dev plan --repo "$PWD" --flavor h200 --timeout 30m

Agents should use info --remote or status before assuming paid compute is active. Plain info reports whether a local state record exists; it deliberately avoids a Hub request.

Start or reconnect

hf-dev up --repo "$PWD" --flavor h200 --timeout 30m

If the recorded Job is active, up reconnects, recreates synchronization, and revalidates the environment. If it is terminal, up creates a new Job. Always choose an explicit flavor and timeout in automation so cost is visible in the invocation.

Useful launch overrides:

hf-dev up --repo "$PWD" --flavor h200 --timeout 1h --cache-bucket my-org/shared-dev-cache
hf-dev up --repo "$PWD" --flavor h200 --timeout 1h --no-cache # override a TOML cache setting
hf-dev up --repo "$PWD" --flavor h200 --timeout 1h --image my-image:tag

Synchronize local source

# Flush the Mutagen replica; recreate the session automatically if it is unhealthy.
hf-dev sync --repo "$PWD"

# Synchronize one file. The path must be inside the repository.
hf-dev sync-file --repo "$PWD" path/to/file.py

Mutagen ignores VCS data, virtual environments, Python caches, test/type/lint caches, build outputs, and node_modules. The extension serializes save-triggered sync operations per workspace. A transient Mutagen flush is retried and an unavailable session is recreated automatically.

Run commands and open a shell

# Run from the synchronized remote repository root.
hf-dev run --repo "$PWD" 'nvidia-smi'
hf-dev run --repo "$PWD" 'python -m pytest tests/models/foo/test_modeling_foo.py -q'

# Open an interactive login shell already positioned at the remote repository root.
hf-dev shell --repo "$PWD"

Quote a compound remote command as one argument. Do not insert a standalone -- after the run options; it would be passed to the remote shell as part of the command.

hf-dev run is preferable to raw SSH for automation because it resolves the recorded Job, validates that it is active, and changes to remote_root first.

Use raw Job SSH

The JSON from hf-dev info --remote contains namespace, job_id, and remote_root. Given those values:

hf jobs ssh NAMESPACE/JOB_ID

The equivalent direct OpenSSH form is:

ssh JOB_ID@ssh.hf.jobs

Raw SSH starts outside the repository. Change to the remote_root reported by hf-dev info before running project commands. Raw SSH is useful for an agent that needs an interactive TTY or a tool not wrapped by hf-dev; use hf-dev run for ordinary non-interactive commands.

Debug manually

Code OSS normally owns the attach sequence. The lower-level primitive is:

hf-dev debug --repo "$PWD" --port 5678 path/to/script.py arg1 arg2
hf-dev debug --repo "$PWD" --port 5678 --module pytest tests/path/test_file.py -k test_name
hf-dev debug-stop --repo "$PWD"

The command flushes source, verifies the editable import, opens an SSH local forward, starts remote debugpy, prints HF_DEV_DEBUG_READY, and waits for a client. Attach a local debugpy client to 127.0.0.1:5678 with this mapping:

localRoot  = local repository root
remoteRoot = remote_root from hf-dev info

The extension obtains a free local port, starts this primitive, waits for the readiness marker, and invokes the Code OSS debugger with justMyCode: false. Breakpoints must be placed in local files; path mapping associates them with the synchronized remote files. Each debug invocation records a verified remote PID under /tmp; a new invocation stops the previous debugger for that workspace before starting. debug-stop removes any current or legacy HF Dev debugpy process whose working directory is the synchronized workspace. Debug stdout and stderr travel over SSH into the extension's HF Dev Output channel; they are not emitted by the Job's sleep infinity container command and therefore do not appear in hf jobs logs.

Logs and shutdown

hf-dev logs --repo "$PWD"
hf-dev logs --repo "$PWD" --follow

# Terminates Mutagen, sends hf jobs cancel, and removes local state after success.
hf-dev down --repo "$PWD"

After work, verify that no development Job remains active:

hf jobs list --status RUNNING --label hf-dev=true

Cancellation through the Hub UI is valid, but local state remains until the next lifecycle action. Use info --remote to distinguish stale local state from a running Job.

How the extension maps to CLI primitives

• Start or Reconnect GPU Job -> hf-dev up --flavor ... --timeout ...
• Sync Workspace Now -> hf-dev sync
• save event -> hf-dev sync-file <saved-file>
• Open Remote Shell -> hf-dev shell
• Run Remote Command -> hf-dev run '<command>'
• Debug Current Python File / F5 -> save, sync-file, hf-dev debug <relative-file>, debugpy attach
• Debug Current File with Pytest -> save, sync-file, hf-dev debug --module pytest <relative-file>
• Follow Job Logs -> hf-dev logs --follow
• Stop GPU Job -> hf-dev down

The extension is intentionally a thin UI. Agents can use the same lifecycle entirely through hf-dev without Code OSS.

State, identity, and cache

Local state is stored per repository root under:

~/.local/state/hf-dev/<repo-key>.json

Important fields include job_id, namespace, remote_root, python, mutagen_session, timeout, cancel_deadline, cancel_watchdog_pid, cache_bucket, and cache_home.

The default Job name and Mutagen session contain a stable hash of the absolute local repository root. Consequently, two clones in different local paths get separate state and remote workspaces.

The default uses the image's Job-local Hugging Face cache and creates no Bucket. Downloads are reused by later debug runs within the same Job, then discarded when that Job ends. Every new Job starts cold.

Persistent caching is an explicit --cache-bucket OWNER/NAME or TOML opt-in. It mounts the Bucket read-write at /cache/huggingface, sets it as HF_HOME, and disables symlinks. This avoids downloading the same revision again, but large safetensors files are then read through the remote FUSE mount and can load more slowly than Xet into local Job storage. Avoid concurrent writers and do not treat this optional mode as the MVP path.

The user's token is forwarded by secret name (--secrets HF_TOKEN), not embedded in the Job command. Do not print, copy, or place token values in configuration or command arguments.

Configuration

Configuration precedence, from lowest to highest, is built-in defaults, user TOML, repository TOML, and CLI options.

User configuration:

~/.config/hf-dev/config.toml

Repository configuration:

<repo>/.hf-dev.toml

Example:

[hf-dev]
image = "huggingface/transformers-all-latest-gpu"
flavor = "h200"
timeout = "1h"
python = "auto"
cache_bucket = false # default; optionally set an explicit private OWNER/NAME Bucket
forward_hf_token = true
# namespace = "my-org"
# resource_group_id = "..."
# identity_file = "~/.ssh/id_ed25519"
# bootstrap = "{python} -m pip install -q debugpy && {python} -m pip install -e ."

Code OSS settings:

• hfDev.command: launcher path; defaults to hf-dev and searches common user/Homebrew locations.
• hfDev.syncOnSave: automatically synchronize saved files; defaults to true.
• hfDev.defaultFlavor: initially selected hardware.
• hfDev.defaultTimeout: initially selected hard timeout (30m, 1h, or 8h).

Useful environment overrides are HF_DEV_HF_BIN, HF_DEV_MUTAGEN_BIN, HF_DEV_SSH_BIN, HF_DEV_SCP_BIN, HF_DEV_MUTAGEN_AGENTS, HF_DEV_CONFIG, and HF_DEV_STATE_DIR.

Safety rules for agents

• Never start a paid Job merely to inspect configuration; use doctor, plan, and local source inspection.
• Make the hardware flavor and timeout explicit before starting a Job.
• Use the shortest practical timeout. The extension offers 30m, 1h, and 8h.
• Treat the local checkout as authoritative and never depend on remote-only source files.
• Expect model downloads to disappear with the Job. Persist checkpoints and other required outputs explicitly to a user-chosen Hub repository or Bucket.
• On setup failure, verify cancellation. On normal completion, run hf-dev down and verify no labeled Job remains.
• The local watchdog supplements the Hub timeout but needs the local machine to reconnect after prolonged offline time.
• Do not manipulate another user's Job or Bucket unless explicitly authorized.
• This release targets single-node Jobs. Multi-node orchestration is not implemented yet.

Troubleshooting

• spawn hf-dev ENOENT: install the launcher at ~/.local/bin/hf-dev or set hfDev.command to its absolute path.
• python: command not found: current releases discover python3 and common absolute interpreter paths. Reconnect with hf-dev up; set python explicitly only for unusual images.
• unable to flush session: current releases retry Mutagen and recreate the session automatically. Confirm the Job is still active with info --remote if recovery fails.
• Breakpoint is hollow or never hit: confirm the script reaches that code, ensure the local file was saved, and check that the reported Transformers source is under remote_root/src/transformers.
• The Docker image already contains Transformers: this is expected. Bootstrap installs the synchronized checkout editable, and debug startup refuses to proceed if transformers.__file__ resolves to the image copy.
• First model load is slow: every new Job has a cold ephemeral cache by default. Later runs in the same Job reuse its local files.
• Opt-in Bucket cache is unexpectedly slow: safetensors are read through the remote FUSE mount. Start a new Job with --no-cache; an already-running Job cannot have its volume removed in place.
• Opt-in cache size nearly doubled after repeated F5: older releases could leave overlapping debug processes and a large .incomplete download. Stop debugging, then run hf cache prune --cache-dir /cache/huggingface/hub --yes in the remote shell. Release 0.1.6 and later prevent overlapping HF Dev debuggers for the same workspace.
• Job started on the wrong hardware: inspect hf-dev plan and choose a GPU flavor explicitly before up.
• Job was canceled in the Hub UI but Code OSS still shows active: local state is stale; use info --remote, then run Start/Reconnect to replace the terminal Job.
• SSH authentication fails: confirm a public key is registered on the Hub and that the token can write to the Job's namespace.

Maintaining and shipping the extension

Validate and package from the extension directory:

node --check extension.js
python -m json.tool package.json >/dev/null
vsce package --allow-missing-repository --no-dependencies
code-oss --install-extension hf-dev-0.1.7.vsix --force

Publish matching launcher and VSIX artifacts to the public release Bucket:

hf cp ~/.local/bin/hf-dev hf://buckets/OWNER/hf-dev-release/0.1.7/hf-dev
hf cp hf-dev-0.1.7.vsix hf://buckets/OWNER/hf-dev-release/0.1.7/hf-dev-0.1.7.vsix

Keep the launcher and VSIX under the same version folder because the extension delegates all remote lifecycle work to the external hf-dev executable.