instruct-mask-tools / docs /UPLOAD_GUIDELINES.md

Sanitize: remove residual token-prefix fragment (hf_Lb) from masking example

454413e verified about 2 months ago

6.79 kB

> RECOVERED 2026-05-23T23:57:12 from Claude session transcripts (last Write; later Edit-tool revisions not applied). Original file was missing from disk.

# Upload Guidelines

**Status:** STUB (to be expanded after EVAL_GUIDELINES.md is locked through 31B Q4_K_M + Q6_K + dual-eval wrapper). Holding scaffolding so cross-references in companion docs resolve to a real path.

**Scope:** how to upload artifacts (adapters, merged safetensors, GGUF quants, datasets, model cards) to Hugging Face reliably and in the right order.

**Companion docs:**
- [EVAL_GUIDELINES.md](EVAL_GUIDELINES.md) — how to evaluate models after uploading.
- [INSTRUCT_MASKING_TECHNIQUES.md](INSTRUCT_MASKING_TECHNIQUES.md) — masking/merging techniques that produce the artifacts.
- [MERGE_MADNESS.md](MERGE_MADNESS.md) — community-model merge recipe cookbook.
- This doc — how to *upload* the outputs of the above.

---

## 0. Hard rules (always apply)

**0.1 `HF_HUB_DISABLE_XET=1` is mandatory for every upload.** Without it, Xet de-dup throttled our 31B uploads to ~1 MB/s (2026-05-22/23). With it, sustained ~6 MB/s on the same files. Set the env var inline on every Python invocation:
```bash
HF_HUB_DISABLE_XET=1 python upload_script.py
```
**0.2 `chat_template.jinja` must be present in the source dir before quant.** Otherwise the GGUF ships templateless → llama-server guesses ChatML → leak + IFEval drop. (Top-level rule, memory `feedback_copy_jinja_before_quant.md`. Lived through this on 2026-05-23.)

**0.3 GGUF provider for community quants:** bartowski only. **NEVER unsloth.** (Memory `feedback_gguf_quants.md`.) AesSedai / ddh0 are also OK for MoE specifically.

**0.4 File-size verification before declaring "done".** Especially with xet stalls or transient HF errors: after upload, fetch the file's HF-side size via `HfApi.model_info(repo).siblings` and compare against the local size. Don't trust the absence of a Python exception alone. (Memory `project_b200_runpod_orchestration.md`.)

**0.5 Adapter-first upload order for multi-artifact releases.** Adapter (~hundreds of MB) → merged safetensors (~hundreds of GB raw / ~30-120 GB bf16) → GGUF quants. So downstream users can grab the cheap thing while the heavy artifacts are still uploading.

---

## 1. Token storage and which account uploads where

**1.1 Local token file.** The default Hugging Face CLI token lives at `/home/aibox/.cache/huggingface/token` on this box (single line, starts with `hf_…`). `huggingface_hub.HfApi()` and the `huggingface-cli` both pick this up automatically — **scripts should not read it explicitly or print it**.

**1.2 Verify which account a token belongs to** *without printing the token itself*:

```bash
python -c "from huggingface_hub import whoami; print(whoami())"
```

This prints `{'name': '<username>', ...}` plus the `orgs` list the token can act under.

**1.3 ToastyPigeon vs rpDungeon.** The default token is currently the ToastyPigeon account token. Whether it has org-membership / write access to `rpDungeon/*` repos depends on org settings on the HF side:
- If it does: same token writes to both. Verify with `whoami()`'s `orgs` field.
- If it does not: we need a separate token for rpDungeon. Store as a second file (suggested path: `/home/aibox/.cache/huggingface/token_rpdungeon`) and select via the `token=` kwarg in `HfApi(token=open(path).read().strip())` rather than mutating the global file.

**Action item (PENDING):** confirm whether the current default token has write access to `rpDungeon/*`. The gembrain_equinox upload at 21:07 UTC succeeded against `rpDungeon/Gemma4-31b-Gembrain-Equinox`, so at least at that moment it did. If a future upload to rpDungeon returns 403, swap the token via the second-file pattern.
**1.4 Never paste tokens into committed files, scripts, Discord, or this doc.** Read from `/home/aibox/.cache/huggingface/token` or the named-second-file path. Anything printed in CI/logs must be masked (`hf_…` style).

**1.5 Rotation.** If a token is ever exposed (Discord paste, accidental commit, leaked log), rotate immediately at https://huggingface.co/settings/tokens. Update the local file. No emergency rebuild of repos is needed — only the token changes.

---

## 2. Upload patterns (TO BE FILLED — placeholders for now)

### 2.1 Single file (GGUF / safetensors shard / adapter zip)
```python
import os
os.environ.setdefault("HF_HUB_DISABLE_XET", "1")
from huggingface_hub import HfApi
HfApi().upload_file(
    path_or_fileobj=local_path,
    path_in_repo=remote_name,
    repo_id=repo,
    repo_type="model",
)
```

### 2.2 Whole directory (adapter folder, merged dir)
- Use `HfApi.upload_folder` with `allow_patterns` / `ignore_patterns`.
- Always ignore checkpoint dirs / optimizer states unless explicitly republishing them.

### 2.3 Template-only re-mint of an existing GGUF on HF
- `gguf_new_metadata.py --chat-template-file <jinja>` on the local GGUF (~5 min).
- Swap-and-upload over the existing HF copy (~50 min at 6 MB/s for 18.7 GB Q4_K_M).
- Reference script: `instruct_mask/refresh_gguf_chat_templates.sh` + `upload_tplmod_ggufs.sh`.

### 2.4 Large datasets
- *(to fill — covers shard-size choice, `repo_type='dataset'`, parquet vs jsonl)*

---

## 3. Detached-job harness for uploads

Every upload that's expected to take longer than ~5 min should run via `run-detached` (systemd-user) so it survives Claude restarts, hooks into the OOM watcher, and has a memory cap. Pure-network uploads can use `RUN_DETACHED_ALLOW_SMALL_CAP=1 run-detached <name> 8G ...` (network-only, no model load).

Reference: `discord-project/instruct_mask/upload_tplmod_ggufs.sh` + its `run-detached` launch line.

---

## 4. Validation checklist (after every upload)

- [ ] `HfApi.model_info(repo)` returns successfully and lists the file with the expected size
- [ ] If GGUF: file is downloadable + llama-server boots against it cleanly (smoke ping)
- [ ] If adapter: PeftModel.from_pretrained loads it cleanly against the declared base
- [ ] Repo README updated if filename / content changed
- [ ] Discord ping with the HF URL (so the next eval / merge step has the source)
---
## 5. Open questions (to scope when expanding this doc)
- Best multipart chunk size for 25-50 GB files via `HfApi.upload_file`
- Resume semantics: if upload script dies at 80%, can we resume vs starting over?
- How to handle quota / repo-size limits — staging on a different repo first?
- Concurrent upload throttling — does running 2 uploads in parallel double or halve throughput at our 6 MB/s ceiling?
- Whether to set `commit_message` per file (good audit trail) vs leaving default

---

## 6. Changelog

- 2026-05-23 — Stub created. Cross-referenced from MASK / MERGE / EVAL docs. Awaiting full content pass after EVAL_GUIDELINES.md lands.

	> RECOVERED 2026-05-23T23:57:12 from Claude session transcripts (last Write; later Edit-tool revisions not applied). Original file was missing from disk.

	# Upload Guidelines

	Status: STUB (to be expanded after EVAL_GUIDELINES.md is locked through 31B Q4_K_M + Q6_K + dual-eval wrapper). Holding scaffolding so cross-references in companion docs resolve to a real path.

	Scope: how to upload artifacts (adapters, merged safetensors, GGUF quants, datasets, model cards) to Hugging Face reliably and in the right order.

	Companion docs:
	- [EVAL_GUIDELINES.md](EVAL_GUIDELINES.md) — how to evaluate models after uploading.
	- [INSTRUCT_MASKING_TECHNIQUES.md](INSTRUCT_MASKING_TECHNIQUES.md) — masking/merging techniques that produce the artifacts.
	- [MERGE_MADNESS.md](MERGE_MADNESS.md) — community-model merge recipe cookbook.
	- This doc — how to upload the outputs of the above.

	---

	## 0. Hard rules (always apply)

	0.1 `HF_HUB_DISABLE_XET=1` is mandatory for every upload. Without it, Xet de-dup throttled our 31B uploads to ~1 MB/s (2026-05-22/23). With it, sustained ~6 MB/s on the same files. Set the env var inline on every Python invocation:

	```bash
	HF_HUB_DISABLE_XET=1 python upload_script.py
	```

	0.2 `chat_template.jinja` must be present in the source dir before quant. Otherwise the GGUF ships templateless → llama-server guesses ChatML → leak + IFEval drop. (Top-level rule, memory `feedback_copy_jinja_before_quant.md`. Lived through this on 2026-05-23.)

	0.3 GGUF provider for community quants: bartowski only. NEVER unsloth. (Memory `feedback_gguf_quants.md`.) AesSedai / ddh0 are also OK for MoE specifically.

	0.4 File-size verification before declaring "done". Especially with xet stalls or transient HF errors: after upload, fetch the file's HF-side size via `HfApi.model_info(repo).siblings` and compare against the local size. Don't trust the absence of a Python exception alone. (Memory `project_b200_runpod_orchestration.md`.)

	0.5 Adapter-first upload order for multi-artifact releases. Adapter (~hundreds of MB) → merged safetensors (~hundreds of GB raw / ~30-120 GB bf16) → GGUF quants. So downstream users can grab the cheap thing while the heavy artifacts are still uploading.

	---

	## 1. Token storage and which account uploads where

	1.1 Local token file. The default Hugging Face CLI token lives at `/home/aibox/.cache/huggingface/token` on this box (single line, starts with `hf_…`). `huggingface_hub.HfApi()` and the `huggingface-cli` both pick this up automatically — scripts should not read it explicitly or print it.

	1.2 Verify which account a token belongs to without printing the token itself:

	```bash
	python -c "from huggingface_hub import whoami; print(whoami())"
	```

	This prints `{'name': '<username>', ...}` plus the `orgs` list the token can act under.

	1.3 ToastyPigeon vs rpDungeon. The default token is currently the ToastyPigeon account token. Whether it has org-membership / write access to `rpDungeon/*` repos depends on org settings on the HF side:
	- If it does: same token writes to both. Verify with `whoami()`'s `orgs` field.
	- If it does not: we need a separate token for rpDungeon. Store as a second file (suggested path: `/home/aibox/.cache/huggingface/token_rpdungeon`) and select via the `token=` kwarg in `HfApi(token=open(path).read().strip())` rather than mutating the global file.

	Action item (PENDING): confirm whether the current default token has write access to `rpDungeon/*`. The gembrain_equinox upload at 21:07 UTC succeeded against `rpDungeon/Gemma4-31b-Gembrain-Equinox`, so at least at that moment it did. If a future upload to rpDungeon returns 403, swap the token via the second-file pattern.

	1.4 Never paste tokens into committed files, scripts, Discord, or this doc. Read from `/home/aibox/.cache/huggingface/token` or the named-second-file path. Anything printed in CI/logs must be masked (`hf_…` style).

	1.5 Rotation. If a token is ever exposed (Discord paste, accidental commit, leaked log), rotate immediately at https://huggingface.co/settings/tokens. Update the local file. No emergency rebuild of repos is needed — only the token changes.

	---

	## 2. Upload patterns (TO BE FILLED — placeholders for now)

	### 2.1 Single file (GGUF / safetensors shard / adapter zip)
	```python
	import os
	os.environ.setdefault("HF_HUB_DISABLE_XET", "1")
	from huggingface_hub import HfApi
	HfApi().upload_file(
	path_or_fileobj=local_path,
	path_in_repo=remote_name,
	repo_id=repo,
	repo_type="model",
	)
	```

	### 2.2 Whole directory (adapter folder, merged dir)
	- Use `HfApi.upload_folder` with `allow_patterns` / `ignore_patterns`.
	- Always ignore checkpoint dirs / optimizer states unless explicitly republishing them.

	### 2.3 Template-only re-mint of an existing GGUF on HF
	- `gguf_new_metadata.py --chat-template-file <jinja>` on the local GGUF (~5 min).
	- Swap-and-upload over the existing HF copy (~50 min at 6 MB/s for 18.7 GB Q4_K_M).
	- Reference script: `instruct_mask/refresh_gguf_chat_templates.sh` + `upload_tplmod_ggufs.sh`.

	### 2.4 Large datasets
	- (to fill — covers shard-size choice, `repo_type='dataset'`, parquet vs jsonl)

	---

	## 3. Detached-job harness for uploads

	Every upload that's expected to take longer than ~5 min should run via `run-detached` (systemd-user) so it survives Claude restarts, hooks into the OOM watcher, and has a memory cap. Pure-network uploads can use `RUN_DETACHED_ALLOW_SMALL_CAP=1 run-detached <name> 8G ...` (network-only, no model load).

	Reference: `discord-project/instruct_mask/upload_tplmod_ggufs.sh` + its `run-detached` launch line.

	---

	## 4. Validation checklist (after every upload)

	- [ ] `HfApi.model_info(repo)` returns successfully and lists the file with the expected size
	- [ ] If GGUF: file is downloadable + llama-server boots against it cleanly (smoke ping)
	- [ ] If adapter: PeftModel.from_pretrained loads it cleanly against the declared base
	- [ ] Repo README updated if filename / content changed
	- [ ] Discord ping with the HF URL (so the next eval / merge step has the source)

	---

	## 5. Open questions (to scope when expanding this doc)

	- Best multipart chunk size for 25-50 GB files via `HfApi.upload_file`
	- Resume semantics: if upload script dies at 80%, can we resume vs starting over?
	- How to handle quota / repo-size limits — staging on a different repo first?
	- Concurrent upload throttling — does running 2 uploads in parallel double or halve throughput at our 6 MB/s ceiling?
	- Whether to set `commit_message` per file (good audit trail) vs leaving default

	---

	## 6. Changelog

	- 2026-05-23 — Stub created. Cross-referenced from MASK / MERGE / EVAL docs. Awaiting full content pass after EVAL_GUIDELINES.md lands.