Wiself commited on
Commit
58ebced
·
1 Parent(s): 3cf77b0

Upload 3 files

Browse files
Files changed (3) hide show
  1. DOCS.md +34 -3
  2. README.md +22 -9
  3. voice.py +415 -28
DOCS.md CHANGED
@@ -1,18 +1,18 @@
1
  # voice — Help
2
 
3
- Generated from `voice --help` and `voice <cmd> --help` (v2.0.0, 11 commands).
4
 
5
  ## voice --help
6
 
7
  ```
8
  usage: voice [-h] [-v] [--non-interactive] [--version]
9
  [--completion {bash,zsh,fish}]
10
- {get,cast,tensor,delta,list,info,remove,cleanup,auth,path,doctor} ...
11
 
12
  Lift a model's voice and cast it onto another model.
13
 
14
  positional arguments:
15
- {get,cast,tensor,delta,list,info,remove,cleanup,auth,path,doctor}
16
  get Get a voice from a model (safetensors repo, .gguf, or URL)
17
  cast Cast a voice onto a .gguf or .safetensors file
18
  tensor Extract or cast any named tensor
@@ -22,6 +22,7 @@ positional arguments:
22
  remove Remove a stored voice
23
  cleanup Remove incomplete downloads
24
  auth Save your HF token (optional)
 
25
  path Put voice on your PATH
26
  doctor Check your setup
27
 
@@ -180,6 +181,36 @@ options:
180
  --show Show if token is present (masked)
181
  ```
182
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
183
  ## voice path
184
 
185
  ```
 
1
  # voice — Help
2
 
3
+ Generated from `voice --help` and `voice <cmd> --help` (v2.1.0, 12 commands).
4
 
5
  ## voice --help
6
 
7
  ```
8
  usage: voice [-h] [-v] [--non-interactive] [--version]
9
  [--completion {bash,zsh,fish}]
10
+ {get,cast,tensor,delta,list,info,remove,cleanup,auth,report,path,doctor} ...
11
 
12
  Lift a model's voice and cast it onto another model.
13
 
14
  positional arguments:
15
+ {get,cast,tensor,delta,list,info,remove,cleanup,auth,report,path,doctor}
16
  get Get a voice from a model (safetensors repo, .gguf, or URL)
17
  cast Cast a voice onto a .gguf or .safetensors file
18
  tensor Extract or cast any named tensor
 
22
  remove Remove a stored voice
23
  cleanup Remove incomplete downloads
24
  auth Save your HF token (optional)
25
+ report Save a report locally and optionally share to community
26
  path Put voice on your PATH
27
  doctor Check your setup
28
 
 
181
  --show Show if token is present (masked)
182
  ```
183
 
184
+ ## voice report
185
+
186
+ ```
187
+ usage: voice report [-h] [--message MESSAGE] [--voice VOICE] [--target TARGET]
188
+ [--reporter REPORTER] [--yes] [--no-push] [--sync]
189
+
190
+ Save a report locally and optionally share to community (Wiself/voice-reports)
191
+
192
+ positional arguments:
193
+ --message, -m MESSAGE
194
+ Freeform note (240ch, sanitized, CSV-injection safe)
195
+ --voice VOICE Voice name context
196
+ --target TARGET Target file basename context (no PII path)
197
+ --reporter REPORTER Reporter id (optional, 64ch sanitized)
198
+
199
+ options:
200
+ -h, --help show this help message and exit
201
+ --yes Non-interactive: auto-share to community
202
+ --no-push Local only, skip share prompt
203
+ --sync Sync pending outbox to community
204
+
205
+ After local save (flock+fsync JSONL at ~/.voices/reports.jsonl, 0600), prompts:
206
+ Share to community reports (Wiself/voice-reports)? [y/N]
207
+ Waits for explicit y/n, no timeout. Non-interactive skips unless --yes.
208
+ Local: outbox retained at ~/.voices/outbox/<id>.json until shared.
209
+ Remote: Wiself/voice-reports dataset, data/reports-YYYY-MM.jsonl, merge-append
210
+ with dedup on report_id, 3 retries on 429/409. Offline keeps outbox.
211
+ Doctor shows reports N local, M pending, and community target.
212
+ ```
213
+
214
  ## voice path
215
 
216
  ```
README.md CHANGED
@@ -120,7 +120,7 @@ A delta carries only the difference from its base, so it blends more gently with
120
 
121
  ## Commands
122
 
123
- Commands. One idea each — 11 commands, `voice --help` is the source of truth (`DOCS.md` generated from live `voice.py` v2.0.0):
124
 
125
  | Command | Description | Source |
126
  |---|---|---|
@@ -129,13 +129,14 @@ Commands. One idea each — 11 commands, `voice --help` is the source of truth (
129
  | `voice tensor get <source> --tensor <n>` | Extract **any** named tensor to a standalone artifact (safetensors or gguf for quant) | `voice.py:1890` `cmd_tensor_get` |
130
  | `voice tensor cast <target> <file>` | Cast **any** named tensor onto a `.gguf` or `.safetensors` file | `voice.py:1942` `cmd_tensor_cast` |
131
  | `voice delta [name]` | Make a portable style (`voice − base` → `F16` delta) you can reuse across compatible models | `voice.py:1999` `cmd_delta` |
132
- | `voice list [base]` | Show saved voices — `voice list base` shows cached base heads | `voice.py:2289` `cmd_list` |
133
- | `voice info <voice>` | Show details for a saved voice, delta, or cached base | `voice.py:2303` `cmd_info` |
134
- | `voice remove <voice>` | Remove a voice from your library | `voice.py:2347` `cmd_remove` |
135
- | `voice cleanup [--force]` | Remove incomplete downloads (`.parts`) — skips active (<10 min) unless `--force` | `voice.py:2365` `cmd_cleanup` |
136
- | `voice auth [--show\|--clear]` | Save your HF token (0600, optional, faster/verified) | `voice.py:2388` `cmd_auth` |
137
- | `voice path` | Put `voice` on your PATH (symlink to `/usr/local/bin` or `~/.local/bin`) | `voice.py:2435` `cmd_path` |
138
- | `voice doctor` | Check your setup — Python + uv + venv + disk + HF token + connectivity | `voice.py:2471` `cmd_doctor` |
 
139
 
140
  Example:
141
 
@@ -153,7 +154,19 @@ voice list base
153
  voice cast ./model.gguf styletune-v2 --out ./voiced/
154
  ```
155
 
156
- > `delta-` is reserved. Voice names cannot start with it. Deltas always do. Full CLI reference: [`DOCS.md`](./DOCS.md) (generated from `voice --help`, 11 commands). Names are lowercased when saved.
 
 
 
 
 
 
 
 
 
 
 
 
157
 
158
 
159
  ## What StyleTune showed
 
120
 
121
  ## Commands
122
 
123
+ Commands. One idea each — 12 commands, `voice --help` is the source of truth (`DOCS.md` generated from live `voice.py` v2.1.0):
124
 
125
  | Command | Description | Source |
126
  |---|---|---|
 
129
  | `voice tensor get <source> --tensor <n>` | Extract **any** named tensor to a standalone artifact (safetensors or gguf for quant) | `voice.py:1890` `cmd_tensor_get` |
130
  | `voice tensor cast <target> <file>` | Cast **any** named tensor onto a `.gguf` or `.safetensors` file | `voice.py:1942` `cmd_tensor_cast` |
131
  | `voice delta [name]` | Make a portable style (`voice − base` → `F16` delta) you can reuse across compatible models | `voice.py:1999` `cmd_delta` |
132
+ | `voice list [base]` | Show saved voices — `voice list base` shows cached base heads | `voice.py:2295` `cmd_list` |
133
+ | `voice info <voice>` | Show details for a saved voice, delta, or cached base | `voice.py:2309` `cmd_info` |
134
+ | `voice remove <voice>` | Remove a voice from your library | `voice.py:2353` `cmd_remove` |
135
+ | `voice cleanup [--force]` | Remove incomplete downloads (`.parts`) — skips active (<10 min) unless `--force` | `voice.py:2371` `cmd_cleanup` |
136
+ | `voice report [--voice X] [-m MSG]` | Save report locally (flock+fsync JSONL `~/.voices/reports.jsonl` + outbox 0600) and optionally share to `Wiself/voice-reports` — prompt `[y/N]` waits for explicit y/n, no timeout; `--yes` auto-share, `--no-push` local-only, `--sync` flushes outbox | `voice.py:2646` `cmd_report` |
137
+ | `voice auth [--show\|--clear]` | Save your HF token (0600, optional, faster/verified) | `voice.py:2718` `cmd_auth` |
138
+ | `voice path` | Put `voice` on your PATH (symlink to `/usr/local/bin` or `~/.local/bin`) | `voice.py:2816` `cmd_path` |
139
+ | `voice doctor` | Check your setup — Python + uv + venv + disk + HF token + connectivity + reports/outbox + community target | `voice.py:2852` `cmd_doctor` |
140
 
141
  Example:
142
 
 
154
  voice cast ./model.gguf styletune-v2 --out ./voiced/
155
  ```
156
 
157
+ > `delta-` is reserved. Voice names cannot start with it. Deltas always do. Full CLI reference: [`DOCS.md`](./DOCS.md) (generated from `voice --help`, 12 commands). Names are lowercased when saved.
158
+
159
+ ### Why report exists — a tiny hope
160
+
161
+ We didn't add `voice report` to be clever. Just — if each of us leaves a one-line note — *which voice, which target, did it sing or loop?* — that shared notebook at [`Wiself/voice-reports`](https://huggingface.co/datasets/Wiself/voice-reports) might, slowly, show which tensors carry which quirks. No grand claim. Just cheaper to check than to argue, because a voice is small.
162
+
163
+ ```bash
164
+ voice report --voice styletune-v2 --target ./model.gguf -m "delta fixed loop, 1.6% shared"
165
+ # ✓ local → ~/.voices/reports.jsonl
166
+ # Share to community reports (Wiself/voice-reports)? [y/N] ← waits for y/n, no default
167
+ ```
168
+
169
+ Local first (0600, `flock` + `fsync`), opt-in share, batched by month. `--no-push` keeps it local, `--yes`/`--sync` for scripts. `voice doctor` shows what's pending. Details: `DOCS.md#voice report`.
170
 
171
 
172
  ## What StyleTune showed
voice.py CHANGED
@@ -13,6 +13,7 @@ Commands:
13
  voice info <voice> Show voice details
14
  voice remove <voice> Remove a stored voice
15
  voice cleanup Remove incomplete downloads
 
16
  voice auth Save your Hugging Face token (optional, faster)
17
  voice path Put `voice` on your PATH
18
  voice doctor Check your setup
@@ -38,8 +39,12 @@ from urllib.parse import urlparse
38
  VOICES_DIR = Path.home() / ".voices"
39
  HF_TOKEN_PATH = VOICES_DIR / ".hf_token"
40
  OPERATIONS_LOG = VOICES_DIR / "operations.log"
 
 
 
 
41
 
42
- VERSION = "2.0.0"
43
  DTYPE_BYTES = {"F32": 4, "F16": 2, "BF16": 2}
44
  VERBOSE = False
45
 
@@ -132,25 +137,39 @@ def _confirm(question, args):
132
  # Hugging Face token & HTTP
133
  # ---------------------------------------------------------------------------
134
 
 
135
  def _get_hf_token():
 
 
 
 
 
136
  for k in ("HF_TOKEN", "HF_HUB_TOKEN", "HUGGING_FACE_HUB_TOKEN"):
137
  v = os.environ.get(k)
138
  if v and v.strip().startswith("hf_"):
139
- return v.strip()
140
- for p in (HF_TOKEN_PATH, Path.home() / ".cache" / "huggingface" / "token", Path.home() / ".huggingface" / "token"):
141
- try:
142
- if p.exists():
143
- t = p.read_text().strip()
144
- if t.startswith("hf_"):
145
- return t
146
- except Exception:
147
- continue
148
- return None
 
 
 
 
 
149
 
150
 
151
  _HF_TRUSTED_HOSTS = frozenset({"huggingface.co", "cdn-lfs.huggingface.co", "cdn-lfs-us-1.huggingface.co"})
152
 
153
 
 
 
 
154
  def _is_hf_url(url):
155
  try:
156
  host = (urlparse(url).hostname or "").lower()
@@ -158,7 +177,6 @@ def _is_hf_url(url):
158
  return False
159
  if host in _HF_TRUSTED_HOSTS:
160
  return True
161
- # allow HF subdomains (e.g. huggingface.co, *.huggingface.co) but not attacker-controlled nesting
162
  return host == "huggingface.co" or host.endswith(".huggingface.co")
163
 
164
 
@@ -416,21 +434,16 @@ def rewrite_safetensors_tensor(target_path, out_path, replace_name, replace_byte
416
  # Dtype decode — one implementation, used everywhere
417
  # ---------------------------------------------------------------------------
418
 
 
 
 
 
 
419
  def dtype_bytes(dtype):
420
- d = dtype.upper()
421
- if d in ("F32", "FP32", "F64"):
422
- return 4 if d in ("F32", "FP32") else 8
423
- if d in ("F16", "FP16", "BF16", "BF16E8M", "F16E5M6"):
424
- return 2
425
- if d in ("I8", "U8", "BOOL"):
426
- return 1
427
- if d in ("I16",):
428
- return 2
429
- if d in ("I32",):
430
- return 4
431
- if d in ("I64",):
432
- return 8
433
- raise ValueError(f"unsupported dtype {dtype}")
434
 
435
 
436
  def bf16_to_f32(u16):
@@ -1259,6 +1272,101 @@ def log_operation(record):
1259
  _warn("could not write operation log")
1260
 
1261
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1262
  # ---------------------------------------------------------------------------
1263
  # Registry (stored voices)
1264
  # ---------------------------------------------------------------------------
@@ -2405,6 +2513,252 @@ def cmd_cleanup(args):
2405
  pass
2406
 
2407
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2408
  # ---------------------------------------------------------------------------
2409
  # CMD: auth
2410
  # ---------------------------------------------------------------------------
@@ -2551,6 +2905,24 @@ def cmd_doctor(_args):
2551
  _say(f" operations logged: {n} ({OPERATIONS_LOG})")
2552
  except Exception:
2553
  pass
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2554
  try:
2555
  req = urllib.request.Request("https://huggingface.co/api/whoami-v2" if tok else "https://huggingface.co",
2556
  headers=_hf_auth_headers("https://huggingface.co"))
@@ -2558,6 +2930,11 @@ def cmd_doctor(_args):
2558
  _say(f" connectivity huggingface.co ok ({resp.status})")
2559
  except Exception as e:
2560
  _warn(f"connectivity failed: {e}")
 
 
 
 
 
2561
 
2562
 
2563
  def _print_version():
@@ -2573,7 +2950,7 @@ def _print_version():
2573
 
2574
 
2575
  def _print_completion(shell):
2576
- cmds = ["get", "cast", "delta", "list", "info", "remove", "cleanup", "auth", "path", "doctor", "tensor"]
2577
  voices = [d.name for d in _iter_voices() if (d / "voice.json").exists()]
2578
  vlist = " ".join(voices)
2579
  clist = " ".join(cmds)
@@ -2718,6 +3095,16 @@ def main():
2718
  p_auth.add_argument("--clear", action="store_true")
2719
  p_auth.add_argument("--show", action="store_true")
2720
 
 
 
 
 
 
 
 
 
 
 
2721
  sub.add_parser("path", help="Put voice on your PATH")
2722
  sub.add_parser("doctor", help="Check your setup")
2723
 
@@ -2739,7 +3126,7 @@ def main():
2739
  dispatch = {
2740
  "get": cmd_get, "cast": cmd_cast, "delta": cmd_delta, "list": cmd_list,
2741
  "info": cmd_info, "remove": cmd_remove, "cleanup": cmd_cleanup, "auth": cmd_auth,
2742
- "path": cmd_path, "doctor": cmd_doctor,
2743
  }
2744
  if args.command == "tensor":
2745
  if args.tensor_cmd == "get":
 
13
  voice info <voice> Show voice details
14
  voice remove <voice> Remove a stored voice
15
  voice cleanup Remove incomplete downloads
16
+ voice report Save a report locally and optionally share
17
  voice auth Save your Hugging Face token (optional, faster)
18
  voice path Put `voice` on your PATH
19
  voice doctor Check your setup
 
39
  VOICES_DIR = Path.home() / ".voices"
40
  HF_TOKEN_PATH = VOICES_DIR / ".hf_token"
41
  OPERATIONS_LOG = VOICES_DIR / "operations.log"
42
+ REPORTS_JSONL = VOICES_DIR / "reports.jsonl"
43
+ REPORTS_OUTBOX = VOICES_DIR / "outbox"
44
+ REPORTS_DATASET_ID = "Wiself/voice-reports"
45
+ REPORTS_REMOTE_PREFIX = "data/reports-"
46
 
47
+ VERSION = "2.1.0"
48
  DTYPE_BYTES = {"F32": 4, "F16": 2, "BF16": 2}
49
  VERBOSE = False
50
 
 
137
  # Hugging Face token & HTTP
138
  # ---------------------------------------------------------------------------
139
 
140
+ _HF_TOKEN_CACHE = {"tok": None, "ts": 0.0}
141
  def _get_hf_token():
142
+ # cache 60s to avoid file thrash on hot path
143
+ now = time.time()
144
+ if now - _HF_TOKEN_CACHE["ts"] < 60 and _HF_TOKEN_CACHE["tok"] is not None:
145
+ return _HF_TOKEN_CACHE["tok"]
146
+ tok = None
147
  for k in ("HF_TOKEN", "HF_HUB_TOKEN", "HUGGING_FACE_HUB_TOKEN"):
148
  v = os.environ.get(k)
149
  if v and v.strip().startswith("hf_"):
150
+ tok = v.strip()
151
+ break
152
+ if tok is None:
153
+ for p in (HF_TOKEN_PATH, Path.home() / ".cache" / "huggingface" / "token", Path.home() / ".huggingface" / "token"):
154
+ try:
155
+ if p.exists():
156
+ t = p.read_text().strip()
157
+ if t.startswith("hf_"):
158
+ tok = t
159
+ break
160
+ except Exception:
161
+ continue
162
+ _HF_TOKEN_CACHE["tok"] = tok
163
+ _HF_TOKEN_CACHE["ts"] = now
164
+ return tok
165
 
166
 
167
  _HF_TRUSTED_HOSTS = frozenset({"huggingface.co", "cdn-lfs.huggingface.co", "cdn-lfs-us-1.huggingface.co"})
168
 
169
 
170
+ from functools import lru_cache
171
+
172
+ @lru_cache(maxsize=256)
173
  def _is_hf_url(url):
174
  try:
175
  host = (urlparse(url).hostname or "").lower()
 
177
  return False
178
  if host in _HF_TRUSTED_HOSTS:
179
  return True
 
180
  return host == "huggingface.co" or host.endswith(".huggingface.co")
181
 
182
 
 
434
  # Dtype decode — one implementation, used everywhere
435
  # ---------------------------------------------------------------------------
436
 
437
+ _DTYPE_BYTES_MAP = {
438
+ "F32": 4, "FP32": 4, "F64": 8,
439
+ "F16": 2, "FP16": 2, "BF16": 2, "BF16E8M": 2, "F16E5M6": 2,
440
+ "I8": 1, "U8": 1, "BOOL": 1, "I16": 2, "I32": 4, "I64": 8,
441
+ }
442
  def dtype_bytes(dtype):
443
+ try:
444
+ return _DTYPE_BYTES_MAP[dtype.upper()]
445
+ except KeyError:
446
+ raise ValueError(f"unsupported dtype {dtype}")
 
 
 
 
 
 
 
 
 
 
447
 
448
 
449
  def bf16_to_f32(u16):
 
1272
  _warn("could not write operation log")
1273
 
1274
 
1275
+ # ---------------------------------------------------------------------------
1276
+ # Reports — local durable store + community share
1277
+ # ---------------------------------------------------------------------------
1278
+
1279
+ def _atomic_jsonl_append(path, record):
1280
+ """Append one JSON line atomically with flock + fsync. Optimised: single chmod, cached JSON."""
1281
+ path = Path(path)
1282
+ # fast path: parent exists, avoid mkdir/chmod churn
1283
+ if not path.parent.exists():
1284
+ path.parent.mkdir(parents=True, exist_ok=True)
1285
+ try:
1286
+ os.chmod(path.parent, 0o700)
1287
+ except Exception:
1288
+ pass
1289
+ if not path.exists():
1290
+ try:
1291
+ fd = os.open(str(path), os.O_WRONLY | os.O_CREAT, 0o600)
1292
+ os.close(fd)
1293
+ except Exception:
1294
+ pass
1295
+ else:
1296
+ # occasional chmod drift check, not every append
1297
+ try:
1298
+ if oct(path.stat().st_mode & 0o777) != "0o600":
1299
+ os.chmod(path, 0o600)
1300
+ except Exception:
1301
+ pass
1302
+ _assert_not_symlink(path)
1303
+ line = json.dumps(record, ensure_ascii=False, separators=(",", ":")) + "\n"
1304
+ with open(path, "a") as f:
1305
+ try:
1306
+ import fcntl
1307
+ fcntl.flock(f, fcntl.LOCK_EX)
1308
+ locked = True
1309
+ except Exception:
1310
+ locked = False
1311
+ try:
1312
+ f.write(line)
1313
+ f.flush()
1314
+ try:
1315
+ os.fsync(f.fileno())
1316
+ except Exception:
1317
+ pass
1318
+ finally:
1319
+ if locked:
1320
+ try:
1321
+ import fcntl
1322
+ fcntl.flock(f, fcntl.LOCK_UN)
1323
+ except Exception:
1324
+ pass
1325
+ try:
1326
+ dirfd = os.open(str(path.parent), os.O_DIRECTORY)
1327
+ try:
1328
+ os.fsync(dirfd)
1329
+ finally:
1330
+ os.close(dirfd)
1331
+ except Exception:
1332
+ pass
1333
+
1334
+ def _write_outbox(report_id, record):
1335
+ REPORTS_OUTBOX.mkdir(parents=True, exist_ok=True)
1336
+ try:
1337
+ os.chmod(REPORTS_OUTBOX, 0o700)
1338
+ except Exception:
1339
+ pass
1340
+ _assert_not_symlink(REPORTS_OUTBOX)
1341
+ out_path = REPORTS_OUTBOX / f"{report_id}.json"
1342
+ _assert_not_symlink(out_path)
1343
+ tmp = out_path.with_suffix(".tmp")
1344
+ fd = os.open(str(tmp), os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
1345
+ try:
1346
+ os.write(fd, (json.dumps(record, ensure_ascii=False) + "\n").encode("utf-8"))
1347
+ try:
1348
+ os.fsync(fd)
1349
+ except Exception:
1350
+ pass
1351
+ finally:
1352
+ os.close(fd)
1353
+ tmp.replace(out_path)
1354
+ try:
1355
+ dirfd = os.open(str(REPORTS_OUTBOX), os.O_DIRECTORY)
1356
+ try:
1357
+ os.fsync(dirfd)
1358
+ finally:
1359
+ os.close(dirfd)
1360
+ except Exception:
1361
+ pass
1362
+ return out_path
1363
+
1364
+ def _append_report(record):
1365
+ _atomic_jsonl_append(REPORTS_JSONL, record)
1366
+ _write_outbox(record["report_id"], record)
1367
+ log_operation({"op": "report", "report_id": record["report_id"], "status": "ok", "voice": record.get("voice"), "target": record.get("target")})
1368
+
1369
+
1370
  # ---------------------------------------------------------------------------
1371
  # Registry (stored voices)
1372
  # ---------------------------------------------------------------------------
 
2513
  pass
2514
 
2515
 
2516
+ # ---------------------------------------------------------------------------
2517
+ # CMD: report — save a report locally and optionally share to community
2518
+ # ---------------------------------------------------------------------------
2519
+
2520
+ def _hash_file(path):
2521
+ import hashlib
2522
+ h = hashlib.sha256()
2523
+ with open(path, "rb") as f:
2524
+ while True:
2525
+ b = f.read(8 << 20)
2526
+ if not b:
2527
+ break
2528
+ h.update(b)
2529
+ return h.hexdigest()
2530
+
2531
+ def _sanitize_note(s):
2532
+ if s is None:
2533
+ return None
2534
+ s = str(s).replace("\r", " ").replace("\n", " ").replace("\x00", " ")
2535
+ s = re.sub(r"\s+", " ", s).strip()
2536
+ if not s:
2537
+ return None
2538
+ # strip CSV injection prefixes
2539
+ if s and s[0] in "=+@-":
2540
+ s = "'" + s
2541
+ if len(s) > 240:
2542
+ s = s[:239] + "…"
2543
+ return s
2544
+
2545
+ def _sanitize_reporter(s):
2546
+ if not s:
2547
+ return None
2548
+ s = re.sub(r"[^A-Za-z0-9._-]", "_", str(s).strip())[:64].strip("_")
2549
+ return s or None
2550
+
2551
+ def _prompt_share(args):
2552
+ if getattr(args, "no_push", False):
2553
+ return False
2554
+ if getattr(args, "yes", False):
2555
+ return True
2556
+ if _noninteractive(args):
2557
+ return False
2558
+ # blocking wait for explicit y/n, no default timeout
2559
+ while True:
2560
+ try:
2561
+ ans = input(" Share to community reports (Wiself/voice-reports)? [y/N] ").strip().lower()
2562
+ except (EOFError, KeyboardInterrupt):
2563
+ _say(" ⚠ Cancelled.")
2564
+ sys.exit(0)
2565
+ if ans in ("y", "yes"):
2566
+ return True
2567
+ if ans in ("n", "no", ""):
2568
+ return False
2569
+ _say(" Please answer y or n.")
2570
+
2571
+ def _push_reports_batch(records):
2572
+ """Batch push: group by shard, single fetch+upload per shard. Returns pushed count."""
2573
+ import time, tempfile
2574
+ if not records:
2575
+ return 0
2576
+ tok = _get_hf_token()
2577
+ if not tok:
2578
+ _warn(" No HF token — saved to outbox only. Run: voice auth")
2579
+ return 0
2580
+ try:
2581
+ from huggingface_hub import HfApi
2582
+ api = HfApi(token=tok)
2583
+ except Exception as e:
2584
+ _warn(f" huggingface_hub not available ({e}) — kept in outbox.")
2585
+ return 0
2586
+ try:
2587
+ api.create_repo(repo_id=REPORTS_DATASET_ID, repo_type="dataset", exist_ok=True)
2588
+ except Exception:
2589
+ pass
2590
+ # group by shard
2591
+ from collections import defaultdict
2592
+ by_shard = defaultdict(list)
2593
+ for r in records:
2594
+ shard = f"{REPORTS_REMOTE_PREFIX}{r['ts'][:7]}.jsonl"
2595
+ by_shard[shard].append(r)
2596
+ pushed = 0
2597
+ for shard, recs in by_shard.items():
2598
+ for attempt in range(3):
2599
+ try:
2600
+ existing = ""
2601
+ try:
2602
+ from huggingface_hub import hf_hub_download
2603
+ tmp = hf_hub_download(repo_id=REPORTS_DATASET_ID, repo_type="dataset", filename=shard, token=tok)
2604
+ existing = Path(tmp).read_text(encoding="utf-8", errors="replace")
2605
+ except Exception:
2606
+ existing = ""
2607
+ # filter already pushed
2608
+ todo = [r for r in recs if r["report_id"] not in existing]
2609
+ if not todo:
2610
+ for r in recs:
2611
+ try:
2612
+ (REPORTS_OUTBOX / f"{r['report_id']}.json").unlink(missing_ok=True)
2613
+ except Exception:
2614
+ pass
2615
+ pushed += len(recs)
2616
+ break
2617
+ if not existing:
2618
+ new_content = "\n".join(json.dumps(r, ensure_ascii=False, separators=(",", ":")) for r in todo) + "\n"
2619
+ else:
2620
+ if not existing.endswith("\n"):
2621
+ existing += "\n"
2622
+ new_content = existing + "\n".join(json.dumps(r, ensure_ascii=False, separators=(",", ":")) for r in todo) + "\n"
2623
+ tmpf = Path(tempfile.mktemp())
2624
+ try:
2625
+ fd = os.open(str(tmpf), os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
2626
+ os.write(fd, new_content.encode("utf-8"))
2627
+ try:
2628
+ os.fsync(fd)
2629
+ except Exception:
2630
+ pass
2631
+ os.close(fd)
2632
+ except Exception:
2633
+ tmpf.write_text(new_content, encoding="utf-8")
2634
+ api.upload_file(path_or_fileobj=str(tmpf), path_in_repo=shard, repo_id=REPORTS_DATASET_ID, repo_type="dataset", commit_message=f"report batch {len(todo)} -> {shard}")
2635
+ try:
2636
+ tmpf.unlink(missing_ok=True)
2637
+ except Exception:
2638
+ pass
2639
+ for r in todo:
2640
+ try:
2641
+ (REPORTS_OUTBOX / f"{r['report_id']}.json").unlink(missing_ok=True)
2642
+ except Exception:
2643
+ pass
2644
+ pushed += len(todo)
2645
+ break
2646
+ except Exception as e:
2647
+ try:
2648
+ from huggingface_hub.utils import HfHubHTTPError as _HE
2649
+ if isinstance(e, _HE) and getattr(e, "response", None) is not None and getattr(e.response, "status_code", None) in (429, 409, 412):
2650
+ ra = _parse_retry_after(getattr(e.response, "headers", {}) or {})
2651
+ time.sleep(ra or (0.7 * (2 ** attempt)))
2652
+ continue
2653
+ except Exception:
2654
+ pass
2655
+ if attempt < 2 and isinstance(e, (urllib.error.URLError, TimeoutError, OSError)):
2656
+ time.sleep(0.7 * (2 ** attempt))
2657
+ continue
2658
+ if attempt >= 2:
2659
+ if VERBOSE:
2660
+ import traceback
2661
+ _say(traceback.format_exc())
2662
+ _warn(f" Push failed for {shard} ({type(e).__name__}: {e}) — kept in outbox.")
2663
+ break
2664
+ time.sleep(0.7 * (2 ** attempt))
2665
+ return pushed
2666
+
2667
+ def _push_report(record):
2668
+ return _push_reports_batch([record]) == 1
2669
+
2670
+ def cmd_report(args):
2671
+ # gather fields
2672
+ msg = _sanitize_note(getattr(args, "message", None))
2673
+ voice_arg = getattr(args, "voice", None)
2674
+ # build record
2675
+ import uuid, platform
2676
+ report_id = uuid.uuid4().hex
2677
+ ts = _now_iso()
2678
+ voice_name = None
2679
+ if voice_arg:
2680
+ voice_name = voice_arg.strip()
2681
+ # auto-detect version
2682
+ env_info = {"voice_ver": VERSION, "python": platform.python_version()}
2683
+ record = {
2684
+ "v": 1,
2685
+ "report_id": report_id,
2686
+ "ts": ts,
2687
+ "voice": voice_name,
2688
+ "target": None,
2689
+ "operation": getattr(args, "operation", None) or "report",
2690
+ "status": "ok",
2691
+ "message": msg,
2692
+ "env": env_info,
2693
+ }
2694
+ # optional target handling if provided via positional
2695
+ target = getattr(args, "target", None)
2696
+ if target:
2697
+ record["target"] = Path(target).name # basename only for privacy
2698
+ reporter = _sanitize_reporter(getattr(args, "reporter", None))
2699
+ if reporter:
2700
+ record["reporter"] = reporter
2701
+ # validate size
2702
+ try:
2703
+ rec_json = json.dumps(record, ensure_ascii=False)
2704
+ if len(rec_json) > 4096:
2705
+ _warn(" Note too large — truncating.")
2706
+ record["message"] = _sanitize_note((msg or "")[:240])
2707
+ except Exception:
2708
+ pass
2709
+ _append_report(record)
2710
+ _ok(f"Report saved locally ({report_id[:8]}…) → {REPORTS_JSONL}")
2711
+ _say(f" Outbox: {REPORTS_OUTBOX / (report_id + '.json')}")
2712
+ # also handle --sync of pending
2713
+ if getattr(args, "sync", False):
2714
+ pending = sorted(REPORTS_OUTBOX.glob("*.json"))
2715
+ if not pending:
2716
+ _say(" No pending reports to sync.")
2717
+ return
2718
+ _step(f"Syncing {len(pending)} pending report(s)…")
2719
+ recs = []
2720
+ for p in pending:
2721
+ try:
2722
+ recs.append(json.loads(p.read_text()))
2723
+ except Exception as e:
2724
+ _warn(f" Could not read {p.name}: {e}")
2725
+ if recs:
2726
+ n = _push_reports_batch(recs)
2727
+ _ok(f"Synced {n}/{len(recs)} report(s).")
2728
+ return
2729
+ should_push = _prompt_share(args)
2730
+ if should_push:
2731
+ _step("Sharing to Wiself/voice-reports…")
2732
+ ok = _push_report(record)
2733
+ if ok:
2734
+ _ok("Shared to Wiself/voice-reports.")
2735
+ pending = sorted(REPORTS_OUTBOX.glob("*.json"))
2736
+ if pending:
2737
+ # batch remaining (exclude already-pushed id already deleted)
2738
+ recs = []
2739
+ for p in pending:
2740
+ try:
2741
+ recs.append(json.loads(p.read_text()))
2742
+ except Exception:
2743
+ continue
2744
+ if recs:
2745
+ _step(f"Syncing {len(recs)} pending report(s)…")
2746
+ _push_reports_batch(recs)
2747
+ # headroom warning
2748
+ try:
2749
+ sz = REPORTS_JSONL.stat().st_size if REPORTS_JSONL.exists() else 0
2750
+ if sz > 50 * 1024 * 1024:
2751
+ _warn(f" reports.jsonl {sz/1e6:.1f} MB — consider archiving (rotation to reports-YYYY-MM.jsonl).")
2752
+ if len(list(REPORTS_OUTBOX.glob("*.json"))) > 5000:
2753
+ _warn(" Outbox >5000 pending — run voice report --sync --yes to flush.")
2754
+ except Exception:
2755
+ pass
2756
+ else:
2757
+ _warn(" Share pending — will retry on next report or voice doctor.")
2758
+ else:
2759
+ _say(" Saved locally. Outbox retained for later push (rerun with --yes to share).")
2760
+
2761
+
2762
  # ---------------------------------------------------------------------------
2763
  # CMD: auth
2764
  # ---------------------------------------------------------------------------
 
2905
  _say(f" operations logged: {n} ({OPERATIONS_LOG})")
2906
  except Exception:
2907
  pass
2908
+ if REPORTS_JSONL.exists():
2909
+ try:
2910
+ n = sum(1 for _ in open(REPORTS_JSONL))
2911
+ pending = len(list(REPORTS_OUTBOX.glob("*.json"))) if REPORTS_OUTBOX.exists() else 0
2912
+ _say(f" reports: {n} local, {pending} pending in outbox ({REPORTS_JSONL})")
2913
+ if pending > 5000:
2914
+ _warn(f" outbox >5000 pending — run: voice report --sync --yes")
2915
+ # headroom
2916
+ try:
2917
+ sz = REPORTS_JSONL.stat().st_size
2918
+ if sz > 50 * 1024 * 1024:
2919
+ _warn(f" reports.jsonl {sz/1e6:.1f} MB — consider archiving")
2920
+ except Exception:
2921
+ pass
2922
+ except Exception:
2923
+ pass
2924
+ else:
2925
+ _say(f" reports: none yet — run: voice report")
2926
  try:
2927
  req = urllib.request.Request("https://huggingface.co/api/whoami-v2" if tok else "https://huggingface.co",
2928
  headers=_hf_auth_headers("https://huggingface.co"))
 
2930
  _say(f" connectivity huggingface.co ok ({resp.status})")
2931
  except Exception as e:
2932
  _warn(f"connectivity failed: {e}")
2933
+ # check push target
2934
+ try:
2935
+ _say(f" community reports: {REPORTS_DATASET_ID} (share prompt on voice report)")
2936
+ except Exception:
2937
+ pass
2938
 
2939
 
2940
  def _print_version():
 
2950
 
2951
 
2952
  def _print_completion(shell):
2953
+ cmds = ["get", "cast", "delta", "list", "info", "remove", "cleanup", "auth", "report", "path", "doctor", "tensor"]
2954
  voices = [d.name for d in _iter_voices() if (d / "voice.json").exists()]
2955
  vlist = " ".join(voices)
2956
  clist = " ".join(cmds)
 
3095
  p_auth.add_argument("--clear", action="store_true")
3096
  p_auth.add_argument("--show", action="store_true")
3097
 
3098
+ p_report = sub.add_parser("report", help="Save a report locally and optionally share to community")
3099
+ p_report.add_argument("--message", "-m", help="Freeform note (240ch, sanitized)")
3100
+ p_report.add_argument("--voice", help="Voice name context")
3101
+ p_report.add_argument("--target", help="Target file basename context")
3102
+ p_report.add_argument("--reporter", help="Reporter id (optional, 64ch)")
3103
+ p_report.add_argument("--yes", action="store_true", help="Non-interactive: auto-share to community")
3104
+ p_report.add_argument("--no-push", action="store_true", help="Local only, skip share prompt")
3105
+ p_report.add_argument("--sync", action="store_true", help="Sync pending outbox to community")
3106
+ p_report.add_argument("--operation", help=argparse.SUPPRESS)
3107
+
3108
  sub.add_parser("path", help="Put voice on your PATH")
3109
  sub.add_parser("doctor", help="Check your setup")
3110
 
 
3126
  dispatch = {
3127
  "get": cmd_get, "cast": cmd_cast, "delta": cmd_delta, "list": cmd_list,
3128
  "info": cmd_info, "remove": cmd_remove, "cleanup": cmd_cleanup, "auth": cmd_auth,
3129
+ "report": cmd_report, "path": cmd_path, "doctor": cmd_doctor,
3130
  }
3131
  if args.command == "tensor":
3132
  if args.tensor_cmd == "get":