AGENTS.md

# HF Jobs Development: operator and agent guide

This extension turns a Hugging Face Job into a remote GPU execution target while keeping the editor and source tree
local. Use the Code OSS commands for the interactive workflow and the `hf-dev` CLI primitives for automation.

## Mental model

```text
local Git checkout (authoritative)
    |
    | Mutagen one-way replica / sync-file
    v
HF Job:/workspace/hf-dev/<repo-key>-<repo-name>
    |
    | editable install of the synchronized checkout
    v
remote Python + CUDA
    |
    | debugpy over an SSH local port forward
    v
local Code OSS debugger and local breakpoints
```

The local checkout is always authoritative. Do not make source changes only on the Job: Mutagen runs in
`one-way-replica` mode and can overwrite or remove remote-only changes. Edit locally, save, and let the extension sync.

## Requirements

- `hf >= 1.28.0`, authenticated with `hf auth login`.
- A token allowed to create/use Jobs and read the required Hub models. Bucket permissions are required only when the
  optional persistent cache is explicitly enabled.
- An SSH public key registered at <https://huggingface.co/settings/keys>.
- `ssh`, `scp`, and Mutagen installed locally. A Homebrew Mutagen installation is supported on macOS.
- `hf-dev` installed on `PATH`, normally at `~/.local/bin/hf-dev`.
- The Code OSS Python and debugpy extensions for F5 debugging.

Check the local prerequisites without starting paid compute:

```bash
hf auth whoami
hf-dev doctor --repo "$PWD"
hf-dev plan --repo "$PWD" --flavor h200 --timeout 30m
```

`plan` prints the exact `hf jobs run` command but does not create a Job.

## Normal Code OSS workflow

1. Open the local repository in Code OSS.
2. Run **HF Dev: Start or Reconnect GPU Job** from the command palette or click the **HF Dev** status item.
3. Select hardware and one of the hard limits: `30m`, `1h`, or `8h`.
4. Edit and save files locally. Save events synchronize the file to the active Job.
5. Open the Python entry point and press F5. On macOS, use `fn`/Globe + F5 unless the function row is configured as
   standard F-keys. F5 reveals the **HF Dev** Output channel without moving editor focus; remote stdout, stderr, and Hub
   progress are streamed there rather than to `hf jobs logs`.
6. Run **HF Dev: Stop GPU Job** as soon as the GPU is no longer needed.

F5 is intercepted only for a Python editor when `hfDev.active` is true and no debug session is already active. During
an attached debug session, F5 keeps its normal Continue behavior. Without an active HF development Job, local F5 is
unchanged. HF Dev permits only one remote debugger per workspace: ending the local attach stops the matching remote
process, and a newer debug launch replaces an older process left by a disconnected editor.

## What `hf-dev up` does

For a newly created Job, `hf-dev up` performs these operations in order:

1. Resolve configuration and verify the current `hf` CLI version.
2. If and only if `cache_bucket` is explicitly configured, create or reuse that private Bucket and include it as a
   read-write `HF_HOME` mount. The default MVP creates and mounts no Bucket.
3. Submit an SSH-enabled detached Job with the chosen image, flavor, and native Hub timeout. Without the opt-in mount,
   Hugging Face libraries use the image's ephemeral Job-local cache.
4. Start a detached local watchdog. At the same hard deadline it checks the Job and sends `hf jobs cancel` if the Job
   remains active. It retries after temporary network loss. The native Hub timeout remains a second guard.
5. Wait until SSH accepts connections.
6. Discover a usable remote Python interpreter (`python3`, `python`, common absolute paths, or an explicit setting).
7. Install the matching Mutagen agent through SSH and create a one-way replica of the local repository.
8. Bootstrap the remote environment. The default installs `debugpy` and installs the synchronized repository editable,
   preferring `uv pip install -e .` and falling back to `python -m pip install -e .`.
9. Import `transformers` and verify that it resolves to `<remote_root>/src/transformers`, not the image's preinstalled
   site-packages copy.

If setup of a newly created Job fails, `hf-dev` terminates its Mutagen session and attempts to cancel the Job
immediately.

## CLI primitives for people and agents

All repository-scoped commands accept `--repo PATH`; when omitted, `hf-dev` resolves the current Git root.

### Inspect without changing state

```bash
# Local recorded state only. Fast, but it can be stale after cancellation in the Hub UI.
hf-dev info --repo "$PWD"

# Local state plus authoritative remote Job status.
hf-dev info --repo "$PWD" --remote

# Authoritative Job details for the recorded Job.
hf-dev status --repo "$PWD"

# Check tools and versions without contacting Jobs.
hf-dev doctor --repo "$PWD"

# Preview a launch without starting paid compute.
hf-dev plan --repo "$PWD" --flavor h200 --timeout 30m
```

Agents should use `info --remote` or `status` before assuming paid compute is active. Plain `info` reports whether a
local state record exists; it deliberately avoids a Hub request.

### Start or reconnect

```bash
hf-dev up --repo "$PWD" --flavor h200 --timeout 30m
```

If the recorded Job is active, `up` reconnects, recreates synchronization, and revalidates the environment. If it is
terminal, `up` creates a new Job. Always choose an explicit flavor and timeout in automation so cost is visible in the
invocation.

Useful launch overrides:

```bash
hf-dev up --repo "$PWD" --flavor h200 --timeout 1h --cache-bucket my-org/shared-dev-cache
hf-dev up --repo "$PWD" --flavor h200 --timeout 1h --no-cache # override a TOML cache setting
hf-dev up --repo "$PWD" --flavor h200 --timeout 1h --image my-image:tag
```

### Synchronize local source

```bash
# Flush the Mutagen replica; recreate the session automatically if it is unhealthy.
hf-dev sync --repo "$PWD"

# Synchronize one file. The path must be inside the repository.
hf-dev sync-file --repo "$PWD" path/to/file.py
```

Mutagen ignores VCS data, virtual environments, Python caches, test/type/lint caches, build outputs, and
`node_modules`. The extension serializes save-triggered sync operations per workspace. A transient Mutagen flush is
retried and an unavailable session is recreated automatically.

### Run commands and open a shell

```bash
# Run from the synchronized remote repository root.
hf-dev run --repo "$PWD" 'nvidia-smi'
hf-dev run --repo "$PWD" 'python -m pytest tests/models/foo/test_modeling_foo.py -q'

# Open an interactive login shell already positioned at the remote repository root.
hf-dev shell --repo "$PWD"
```

Quote a compound remote command as one argument. Do not insert a standalone `--` after the `run` options; it would be
passed to the remote shell as part of the command.

`hf-dev run` is preferable to raw SSH for automation because it resolves the recorded Job, validates that it is active,
and changes to `remote_root` first.

### Use raw Job SSH

The JSON from `hf-dev info --remote` contains `namespace`, `job_id`, and `remote_root`. Given those values:

```bash
hf jobs ssh NAMESPACE/JOB_ID
```

The equivalent direct OpenSSH form is:

```bash
ssh JOB_ID@ssh.hf.jobs
```

Raw SSH starts outside the repository. Change to the `remote_root` reported by `hf-dev info` before running project
commands. Raw SSH is useful for an agent that needs an interactive TTY or a tool not wrapped by `hf-dev`; use
`hf-dev run` for ordinary non-interactive commands.

### Debug manually

Code OSS normally owns the attach sequence. The lower-level primitive is:

```bash
hf-dev debug --repo "$PWD" --port 5678 path/to/script.py arg1 arg2
hf-dev debug --repo "$PWD" --port 5678 --module pytest tests/path/test_file.py -k test_name
hf-dev debug-stop --repo "$PWD"
```

The command flushes source, verifies the editable import, opens an SSH local forward, starts remote debugpy, prints
`HF_DEV_DEBUG_READY`, and waits for a client. Attach a local debugpy client to `127.0.0.1:5678` with this mapping:

```text
localRoot  = local repository root
remoteRoot = remote_root from hf-dev info
```

The extension obtains a free local port, starts this primitive, waits for the readiness marker, and invokes the Code
OSS debugger with `justMyCode: false`. Breakpoints must be placed in local files; path mapping associates them with the
synchronized remote files. Each debug invocation records a verified remote PID under `/tmp`; a new invocation stops
the previous debugger for that workspace before starting. `debug-stop` removes any current or legacy HF Dev debugpy
process whose working directory is the synchronized workspace. Debug stdout and stderr travel over SSH into the
extension's **HF Dev** Output channel; they are not emitted by the Job's `sleep infinity` container command and
therefore do not appear in `hf jobs logs`.

### Logs and shutdown

```bash
hf-dev logs --repo "$PWD"
hf-dev logs --repo "$PWD" --follow

# Terminates Mutagen, sends hf jobs cancel, and removes local state after success.
hf-dev down --repo "$PWD"
```

After work, verify that no development Job remains active:

```bash
hf jobs list --status RUNNING --label hf-dev=true
```

Cancellation through the Hub UI is valid, but local state remains until the next lifecycle action. Use
`info --remote` to distinguish stale local state from a running Job.

## How the extension maps to CLI primitives

- **Start or Reconnect GPU Job** -> `hf-dev up --flavor ... --timeout ...`
- **Sync Workspace Now** -> `hf-dev sync`
- save event -> `hf-dev sync-file <saved-file>`
- **Open Remote Shell** -> `hf-dev shell`
- **Run Remote Command** -> `hf-dev run '<command>'`
- **Debug Current Python File** / F5 -> save, `sync-file`, `hf-dev debug <relative-file>`, debugpy attach
- **Debug Current File with Pytest** -> save, `sync-file`, `hf-dev debug --module pytest <relative-file>`
- **Follow Job Logs** -> `hf-dev logs --follow`
- **Stop GPU Job** -> `hf-dev down`

The extension is intentionally a thin UI. Agents can use the same lifecycle entirely through `hf-dev` without Code
OSS.

## State, identity, and cache

Local state is stored per repository root under:

```text
~/.local/state/hf-dev/<repo-key>.json
```

Important fields include `job_id`, `namespace`, `remote_root`, `python`, `mutagen_session`, `timeout`,
`cancel_deadline`, `cancel_watchdog_pid`, `cache_bucket`, and `cache_home`.

The default Job name and Mutagen session contain a stable hash of the absolute local repository root. Consequently,
two clones in different local paths get separate state and remote workspaces.

The default uses the image's Job-local Hugging Face cache and creates no Bucket. Downloads are reused by later debug
runs within the same Job, then discarded when that Job ends. Every new Job starts cold.

Persistent caching is an explicit `--cache-bucket OWNER/NAME` or TOML opt-in. It mounts the Bucket read-write at
`/cache/huggingface`, sets it as `HF_HOME`, and disables symlinks. This avoids downloading the same revision again, but
large safetensors files are then read through the remote FUSE mount and can load more slowly than Xet into local Job
storage. Avoid concurrent writers and do not treat this optional mode as the MVP path.

The user's token is forwarded by secret name (`--secrets HF_TOKEN`), not embedded in the Job command. Do not print,
copy, or place token values in configuration or command arguments.

## Configuration

Configuration precedence, from lowest to highest, is built-in defaults, user TOML, repository TOML, and CLI options.

User configuration:

```text
~/.config/hf-dev/config.toml
```

Repository configuration:

```text
<repo>/.hf-dev.toml
```

Example:

```toml
[hf-dev]
image = "huggingface/transformers-all-latest-gpu"
flavor = "h200"
timeout = "1h"
python = "auto"
cache_bucket = false # default; optionally set an explicit private OWNER/NAME Bucket
forward_hf_token = true
# namespace = "my-org"
# resource_group_id = "..."
# identity_file = "~/.ssh/id_ed25519"
# bootstrap = "{python} -m pip install -q debugpy && {python} -m pip install -e ."
```

Code OSS settings:

- `hfDev.command`: launcher path; defaults to `hf-dev` and searches common user/Homebrew locations.
- `hfDev.syncOnSave`: automatically synchronize saved files; defaults to `true`.
- `hfDev.defaultFlavor`: initially selected hardware.
- `hfDev.defaultTimeout`: initially selected hard timeout (`30m`, `1h`, or `8h`).

Useful environment overrides are `HF_DEV_HF_BIN`, `HF_DEV_MUTAGEN_BIN`, `HF_DEV_SSH_BIN`, `HF_DEV_SCP_BIN`,
`HF_DEV_MUTAGEN_AGENTS`, `HF_DEV_CONFIG`, and `HF_DEV_STATE_DIR`.

## Safety rules for agents

- Never start a paid Job merely to inspect configuration; use `doctor`, `plan`, and local source inspection.
- Make the hardware flavor and timeout explicit before starting a Job.
- Use the shortest practical timeout. The extension offers `30m`, `1h`, and `8h`.
- Treat the local checkout as authoritative and never depend on remote-only source files.
- Expect model downloads to disappear with the Job. Persist checkpoints and other required outputs explicitly to a
  user-chosen Hub repository or Bucket.
- On setup failure, verify cancellation. On normal completion, run `hf-dev down` and verify no labeled Job remains.
- The local watchdog supplements the Hub timeout but needs the local machine to reconnect after prolonged offline time.
- Do not manipulate another user's Job or Bucket unless explicitly authorized.
- This release targets single-node Jobs. Multi-node orchestration is not implemented yet.

## Troubleshooting

- `spawn hf-dev ENOENT`: install the launcher at `~/.local/bin/hf-dev` or set `hfDev.command` to its absolute path.
- `python: command not found`: current releases discover `python3` and common absolute interpreter paths. Reconnect with
  `hf-dev up`; set `python` explicitly only for unusual images.
- `unable to flush session`: current releases retry Mutagen and recreate the session automatically. Confirm the Job is
  still active with `info --remote` if recovery fails.
- Breakpoint is hollow or never hit: confirm the script reaches that code, ensure the local file was saved, and check
  that the reported Transformers source is under `remote_root/src/transformers`.
- The Docker image already contains Transformers: this is expected. Bootstrap installs the synchronized checkout
  editable, and debug startup refuses to proceed if `transformers.__file__` resolves to the image copy.
- First model load is slow: every new Job has a cold ephemeral cache by default. Later runs in the same Job reuse its
  local files.
- Opt-in Bucket cache is unexpectedly slow: safetensors are read through the remote FUSE mount. Start a new Job with
  `--no-cache`; an already-running Job cannot have its volume removed in place.
- Opt-in cache size nearly doubled after repeated F5: older releases could leave overlapping debug processes and a large
  `.incomplete` download. Stop debugging, then run `hf cache prune --cache-dir /cache/huggingface/hub --yes` in the
  remote shell. Release 0.1.6 and later prevent overlapping HF Dev debuggers for the same workspace.
- Job started on the wrong hardware: inspect `hf-dev plan` and choose a GPU flavor explicitly before `up`.
- Job was canceled in the Hub UI but Code OSS still shows active: local state is stale; use `info --remote`, then run
  Start/Reconnect to replace the terminal Job.
- SSH authentication fails: confirm a public key is registered on the Hub and that the token can write to the Job's
  namespace.

## Maintaining and shipping the extension

Validate and package from the extension directory:

```bash
node --check extension.js
python -m json.tool package.json >/dev/null
vsce package --allow-missing-repository --no-dependencies
code-oss --install-extension hf-dev-0.1.7.vsix --force
```

Publish matching launcher and VSIX artifacts to the public release Bucket:

```bash
hf cp ~/.local/bin/hf-dev hf://buckets/OWNER/hf-dev-release/0.1.7/hf-dev
hf cp hf-dev-0.1.7.vsix hf://buckets/OWNER/hf-dev-release/0.1.7/hf-dev-0.1.7.vsix
```

Keep the launcher and VSIX under the same version folder because the extension delegates all remote lifecycle work to
the external `hf-dev` executable.