Upload folder using huggingface_hub

logs/ext32k/RESULTS.md

@@ -0,0 +1,56 @@
+# 16k -> 32k YaRN extension on b4 -- 2026-09-09
+
+Parent: l2 (lenbucket_l2_blumodelcp_run2_step_400.ckpt), the best 16k arm.
+Data:   b4_32768_65536 -- min doc 32,768 == the window, so EVERY training
+        sample is one contiguous document and a 32k-spanning dependency is
+        possible. b3 (min 16,384) would have straddled an EOT in every window.
+YaRN:   SCALE=2 ORIG_MAXPOS=16384 -> 32768 exactly.
+Train:  750 steps, global batch 524,288 (B=1 x accum 16), WSD stable 550
+        + 100-step decay, 2x H200 cp=2, peak 108.8 GB of 143 GB.
+        Final loss 2.570224. 393 M tokens of 586 M available (no repetition).
+
+## niah_single, N=100, ONE haystack (b2 val)
+
+| context | l2 (16k model, 16k cache) | ext32k (32k model, 32k cache) | delta |
+|---|---|---|---|
+| 4,096 | 80.0 | 75.0 | -5.0 |
+| 8,192 | 62.0 | 60.0 | -2.0 |
+| 16,360 | 35.0 | 45.0 | +10.0 |
+| 32,744 | cannot run | **27.0** | new capability |
+
+95% Wilson at N=100 is +-9 to 10 pp, so the combined bar is +-11 to 13 pp:
+NONE of the three deltas is individually resolvable. The 32,744 row is not a
+delta -- the parent's RoPE cache is sized 16,384 and cannot process 32k at all.
+
+## KNOWN BUG in this run's raw summary.csv files
+
+ext32k_b4.sh line 128 passed YARN_ORIG=16384 YARN=2 to BOTH models, so the
+l2 column in ext32k_len_*/summary.csv is a MISMATCHED readout -- l2 was
+trained at scale 4 / orig 4096. Measured damage: 4k -4.0, 8k -26.0, 16k -15.0.
+The l2 numbers in the table above come from the matched H100 decision eval.
+DO NOT use the l2 column in the raw CSVs.
+
+### PROMPT_LEN=4096 (raw -- l2 column mismatched, see above)
+```
+task,ext32k,l2
+niah_single,75.0,76.0
+```
+
+### PROMPT_LEN=8192 (raw -- l2 column mismatched, see above)
+```
+task,ext32k,l2
+niah_single,60.0,36.0
+```
+
+### PROMPT_LEN=16360 (raw -- l2 column mismatched, see above)
+```
+task,ext32k,l2
+niah_single,45.0,20.0
+```
+
+### PROMPT_LEN=32744 (raw -- l2 column mismatched, see above)
+```
+task,ext32k,l2
+niah_single,27.0,11.0
+```
+

logs/ext32k/eval_16360.log

@@ -0,0 +1,60 @@
+[16:32:02] longeval: T=32768 prompt_len=16360 max_new=24 n=100 greedy=1 gpus=0,1
+[16:32:02] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[16:32:02] prompt signature changed -- regenerating prompts and clearing state
+[16:32:02] generating prompts: niah_single
+[16:32:03] fan-out: 2 GPU(s), NP=1 -> 2 concurrent job(s)
+[16:32:03] 2 job(s) to run
+[16:32:03]   [gpu 0] ext32k / niah_single
+[16:32:03]   [gpu 1] l2 / niah_single
+[16:37:38]   scoring ext32k / niah_single
+
+=== niah_single  (prompt_len=16360, n=100) ===
+  accuracy: 45/100 = 45.0%
+    depth      acc   (needle position in context)
+     0.00   2/20   10.0%
+     0.25   7/20   35.0%
+     0.50   9/20   45.0%
+     0.75  14/20   70.0%
+     1.00  13/20   65.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= 8.\nQuestion: What is the special magic number for yarrow?\n
+  [HIT ] expect=5343902
+         got= 5343902.\n
+
+  wrote ext32k_len_16360/scores/ext32k_niah_single.csv
+[16:37:39]   scoring l2 / niah_single
+
+=== niah_single  (prompt_len=16360, n=100) ===
+  accuracy: 20/100 = 20.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   3/20   15.0%
+     1.00  17/20   85.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= the magic number for the yarrow.\n
+  [MISS] expect=5343902
+         got= the number of the cactus.\n
+
+  wrote ext32k_len_16360/scores/l2_niah_single.csv
+[16:37:39] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                 ext32k       l2
+  niah_single            45.0     20.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth             ext32k              l2
+  0.00          10.0+-13.7       0.0+- 8.1
+  0.25          35.0+-19.3       0.0+- 8.1
+  0.50          45.0+-20.0       0.0+- 8.1
+  0.75          70.0+-18.7      15.0+-15.4
+  1.00          65.0+-19.3      85.0+-15.4
+
+  wrote ext32k_len_16360/summary.csv and ext32k_len_16360/depth_summary.csv
+[16:37:39] DONE -> ext32k_len_16360/summary.csv

logs/ext32k/eval_32744.log

@@ -0,0 +1,60 @@
+[16:37:39] longeval: T=32768 prompt_len=32744 max_new=24 n=100 greedy=1 gpus=0,1
+[16:37:39] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[16:37:39] prompt signature changed -- regenerating prompts and clearing state
+[16:37:39] generating prompts: niah_single
+[16:37:40] fan-out: 2 GPU(s), NP=1 -> 2 concurrent job(s)
+[16:37:40] 2 job(s) to run
+[16:37:40]   [gpu 0] ext32k / niah_single
+[16:37:40]   [gpu 1] l2 / niah_single
+[16:48:52]   scoring ext32k / niah_single
+
+=== niah_single  (prompt_len=32744, n=100) ===
+  accuracy: 27/100 = 27.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   2/20   10.0%
+     0.75   8/20   40.0%
+     1.00  17/20   85.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= “kitty-cat”.\n
+  [MISS] expect=5343902
+         got= 7. This number is unique to cactus, making it a unique identifier for each plant.\n
+
+  wrote ext32k_len_32744/scores/ext32k_niah_single.csv
+[16:48:52]   scoring l2 / niah_single
+
+=== niah_single  (prompt_len=32744, n=100) ===
+  accuracy: 11/100 = 11.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   0/20    0.0%
+     0.50   0/20    0.0%
+     0.75   0/20    0.0%
+     1.00  11/20   55.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= the number of the number of the number of the number of the number of the number of the number of the number of
+  [MISS] expect=5343902
+         got= the number for the cactus. This number is a unique identifier that identifies the plant, making it easier to identify the
+
+  wrote ext32k_len_32744/scores/l2_niah_single.csv
+[16:48:53] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                 ext32k       l2
+  niah_single            27.0     11.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth             ext32k              l2
+  0.00           0.0+- 8.1       0.0+- 8.1
+  0.25           0.0+- 8.1       0.0+- 8.1
+  0.50          10.0+-13.7       0.0+- 8.1
+  0.75          40.0+-19.7       0.0+- 8.1
+  1.00          85.0+-15.4      55.0+-20.0
+
+  wrote ext32k_len_32744/summary.csv and ext32k_len_32744/depth_summary.csv
+[16:48:53] DONE -> ext32k_len_32744/summary.csv

logs/ext32k/eval_4096.log

@@ -0,0 +1,60 @@
+[16:26:10] longeval: T=32768 prompt_len=4096 max_new=24 n=100 greedy=1 gpus=0,1
+[16:26:10] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[16:26:10] prompt signature changed -- regenerating prompts and clearing state
+[16:26:10] generating prompts: niah_single
+[16:26:15] fan-out: 2 GPU(s), NP=1 -> 2 concurrent job(s)
+[16:26:15] 2 job(s) to run
+[16:26:15]   [gpu 0] ext32k / niah_single
+[16:26:15]   [gpu 1] l2 / niah_single
+[16:28:25]   scoring ext32k / niah_single
+
+=== niah_single  (prompt_len=4096, n=100) ===
+  accuracy: 75/100 = 75.0%
+    depth      acc   (needle position in context)
+     0.00  13/20   65.0%
+     0.25  16/20   80.0%
+     0.50  13/20   65.0%
+     0.75  17/20   85.0%
+     1.00  16/20   80.0%
+
+  --- 2 sample generations ---
+  [HIT ] expect=8056020
+         got= 8056020.\n
+  [HIT ] expect=5343902
+         got= 5343902.\n
+
+  wrote ext32k_len_4096/scores/ext32k_niah_single.csv
+[16:28:25]   scoring l2 / niah_single
+
+=== niah_single  (prompt_len=4096, n=100) ===
+  accuracy: 76/100 = 76.0%
+    depth      acc   (needle position in context)
+     0.00  10/20   50.0%
+     0.25  16/20   80.0%
+     0.50  12/20   60.0%
+     0.75  19/20   95.0%
+     1.00  19/20   95.0%
+
+  --- 2 sample generations ---
+  [HIT ] expect=8056020
+         got= 8056020.\n
+  [HIT ] expect=5343902
+         got= 5343902.\n
+
+  wrote ext32k_len_4096/scores/l2_niah_single.csv
+[16:28:26] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                 ext32k       l2
+  niah_single            75.0     76.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth             ext32k              l2
+  0.00          65.0+-19.3      50.0+-20.1
+  0.25          80.0+-16.8      80.0+-16.8
+  0.50          65.0+-19.3      60.0+-19.7
+  0.75          85.0+-15.4      95.0+-11.4
+  1.00          80.0+-16.8      95.0+-11.4
+
+  wrote ext32k_len_4096/summary.csv and ext32k_len_4096/depth_summary.csv
+[16:28:26] DONE -> ext32k_len_4096/summary.csv

logs/ext32k/eval_8192.log

@@ -0,0 +1,60 @@
+[16:28:26] longeval: T=32768 prompt_len=8192 max_new=24 n=100 greedy=1 gpus=0,1
+[16:28:26] haystack: /root/Adhi/BluTrain/dist/Context_Parallelism/lendata/haystack/b2val_haystack.bin
+[16:28:26] prompt signature changed -- regenerating prompts and clearing state
+[16:28:26] generating prompts: niah_single
+[16:28:26] fan-out: 2 GPU(s), NP=1 -> 2 concurrent job(s)
+[16:28:26] 2 job(s) to run
+[16:28:26]   [gpu 0] ext32k / niah_single
+[16:28:26]   [gpu 1] l2 / niah_single
+[16:32:02]   scoring ext32k / niah_single
+
+=== niah_single  (prompt_len=8192, n=100) ===
+  accuracy: 60/100 = 60.0%
+    depth      acc   (needle position in context)
+     0.00   7/20   35.0%
+     0.25  13/20   65.0%
+     0.50  11/20   55.0%
+     0.75  13/20   65.0%
+     1.00  16/20   80.0%
+
+  --- 2 sample generations ---
+  [HIT ] expect=8056020
+         got= 8056020.\n
+  [HIT ] expect=5343902
+         got= 5343902.\n
+
+  wrote ext32k_len_8192/scores/ext32k_niah_single.csv
+[16:32:02]   scoring l2 / niah_single
+
+=== niah_single  (prompt_len=8192, n=100) ===
+  accuracy: 36/100 = 36.0%
+    depth      acc   (needle position in context)
+     0.00   0/20    0.0%
+     0.25   1/20    5.0%
+     0.50   7/20   35.0%
+     0.75   9/20   45.0%
+     1.00  19/20   95.0%
+
+  --- 2 sample generations ---
+  [MISS] expect=8056020
+         got= the number of the moon.\n
+  [MISS] expect=5343902
+         got= the magic number. It is a combination of the magic number and the special number of the plant.\n
+
+  wrote ext32k_len_8192/scores/l2_niah_single.csv
+[16:32:02] building summary
+
+=== per-task accuracy %  (secondary/descriptive: ~10 per depth cell) ===
+  task                 ext32k       l2
+  niah_single            60.0     36.0
+
+=== PRIMARY: accuracy % pooled by depth across tasks (+-95% Wilson) ===
+  depth             ext32k              l2
+  0.00          35.0+-19.3       0.0+- 8.1
+  0.25          65.0+-19.3       5.0+-11.4
+  0.50          55.0+-20.0      35.0+-19.3
+  0.75          65.0+-19.3      45.0+-20.0
+  1.00          80.0+-16.8      95.0+-11.4
+
+  wrote ext32k_len_8192/summary.csv and ext32k_len_8192/depth_summary.csv
+[16:32:02] DONE -> ext32k_len_8192/summary.csv

logs/ext32k/ext.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e54eb52d147d9fe237a0448aef317190ba9d123b0216295ae6aa9ebad0f0dad7
+size 61

logs/ext32k/master.log

@@ -0,0 +1,18 @@
+[13:18:27] === 16k -> 32k YaRN extension on b4 ===
+[13:18:27] 6 train shards = .586 B tokens; need .393 B for 750 steps
+[13:18:27] YaRN: SCALE=2 ORIG_MAXPOS=16384 -> 32768 (must equal CP_T=32768)
+[13:18:27] PHASE 1: 3-step memory probe (projected 130.8 GB of 143 GB = 91%)
+[13:19:33] PROBE OK: peak mem: 108619MB
+[13:19:33] PHASE 2: 750 steps, stable=550, ckpt every 50 keep 1
+[16:26:10] ext32k -> run 2, blumodelcp_run2_step_750.ckpt, loss loss: 2.570224
+[16:26:10]   peak mem mem: 108811MB
+[16:26:10] PHASE 3: eval models ext32k l2 
+[16:26:10] eval PROMPT_LEN=4096
+[16:28:26]   -> task,ext32k,l2
 / niah_single,75.0,76.0
+[16:28:26] eval PROMPT_LEN=8192
+[16:32:02]   -> task,ext32k,l2
 / niah_single,60.0,36.0
+[16:32:02] eval PROMPT_LEN=16360
+[16:37:39]   -> task,ext32k,l2
 / niah_single,45.0,20.0
+[16:37:39] eval PROMPT_LEN=32744
+[16:48:53]   -> task,ext32k,l2
 / niah_single,27.0,11.0
+[16:48:53] === EXT32K DONE ===

logs/ext32k/probe.log

@@ -0,0 +1,60 @@
+=== Llama Context Parallel Training Script (bluscriptCP) ===
+[CP attn fusion] split (Stage-2 rope-once + Stage-3 Hopper causal-attn kernels)
+[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
+[watchdog] collective no-progress deadline 600 s, poll 2 s (BLU_NCCL_WATCHDOG=0 to disable)
+================ Configuration (bluscriptCP) ================
+  d_model/layers  : 1536 / 36
+  q/kv heads      : 12 / 4  (G=3, hd=128)
+  ffn_hidden      : 4096 (SwiGLU)
+  context_length  : 32768
+  vocab_size      : 50304  tying=0
+  B x T           : 1 x 32768
+  world_size      : 2  (dp=1 x cp=2)
+  CP_ATTN_MODE    : ring  (load_balance=HeadTail)
+  global_batch    : 65536  (grad_accum=2)
+  precision       : bf16 Linears + fp32 norms/residual/logits (fp32 master weights in AdamW)  [mask=15 QKV WO MLP HEAD]
+  max_lr/min_lr   : 0.0006 / 6e-05
+  warmup/steps    : 100 / 3  (cosine, no stable phase)
+  checkpointing   : false  (ckpt_freq=50, dir=checkpoints_bluscriptcp)
+=============================================================
+Initializing model on CUDA device 0...
+[YaRN] build_rope_cache: seq_len=32768 hd=128 scale=2.000 low=21 high=39 m=1.0693
+[InitFrom] weights loaded from /root/Adhi/BluTrain/dist/Context_Parallelism/checkpoints_bluscriptcp/rank_0/lenbucket_l2_blumodelcp_run2_step_400.ckpt  (base step 400, loss 2.79415)  -- optimizer state and RNG intentionally NOT restored; pair with CP_REWARMUP
+Parameters: 1060624896
+Parameters per GPU: 1060624896
+max_steps: 3
+warmup_steps: 100
+[ZeRO-1 v2] build=v2.4  supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
+[ZeRO-1 v2] ACTIVE: reduce_dtype=bfloat16 (= param dtype)  symmetric=off  overlap=-/-
+[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
+[ZeRO-1 v2] params=983236608 padded=983236608 W=2 shard=491618304 | 38 buckets of 50 MB (13107200 elems/rank/bucket) | optimizer state/GPU: 5626.12 MB (vs 11252.2 MB unsharded)
+[ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it.
+[ZeRO-1 v2] build=v2.4  supports: rs+ag | can-symmetric-window | can-fp32-reduce | can-bf16-reduce | can-reshard-v1-ckpt
+[ZeRO-1 v2] ACTIVE: reduce_dtype=float32 (= param dtype)  symmetric=off  overlap=-/-
+[ZeRO-1 v2] zero1-v2 reduce-scatter/all-gather (standard memory)
+[ZeRO-1 v2] params=77388288 padded=77388288 W=2 shard=38694144 | 6 buckets of 50 MB (6553600 elems/rank/bucket) | optimizer state/GPU: 442.819 MB (vs 885.639 MB unsharded)
+[ZeRO-1 v2] NOTE: symmetric memory is NOT active — collectives will run NCCL's ring/tree path, not NVLS. Create the process group with MemoryMode::Symmetric (and NCCL >= 2.27) to enable it.
+[ZeRO-1] variant=v2 reduce-scatter/all-gather  groups=2  [bfloat16/983236608]  [float32/77388288]
+[ZeRO-1] DataParallel is DISABLED (both would double-reduce)
+[precision] bf16 params=217, all with fp32 master weights
+found 6 shards for split train
+found 1 shards for split val
+Saving logs to: CP_BluScreipt_Training_logs/CP_Training_log1.csv (run 1)
+[rope_theta] 500000 compiled default (no sidecar -- legacy run, assuming it was trained at this base)
+Checkpointing: OFF
+[re-warmup] anchored at step 0 for 100 steps -> peak 0.00018 (0.3 x max_lr), then cosine-decay to 6e-05 | resuming at step 0 (mid-ramp)
+
+Starting training...
+
+[CP ring P2P] OVERLAP: dedicated non-blocking stream
+validation loss: 3.2605
+[CP bwd] overlap=ON
+step     0 | loss: 3.265621 | lr 1.8000e-06 | norm: 0.6861 | dt: 10194.68ms | tok/sec: 6428.5 | mem: 107275MB | Time Left: 00 hrs : 00 mins
+  [TIMING] data: 0.7ms | fwd: 1107.2ms | loss: 1.7ms | bwd: 1579.8ms | clip: 0.0ms | optim: 24.2ms
+step     1 | loss: 3.303823 | lr 3.6000e-06 | norm: 0.6243 | dt: 2304.09ms | tok/sec: 28443.3 | mem: 108811MB | Time Left: 00 hrs : 00 mins
+  [TIMING] data: 1.8ms | fwd: 914.6ms | loss: 1.7ms | bwd: 1358.8ms | clip: 0.0ms | optim: 23.7ms
+validation loss: 3.2597
+step     2 | loss: 3.228388 | lr 5.4000e-06 | norm: 0.6796 | dt: 6985.50ms | tok/sec: 9381.7 | mem: 108619MB | Time Left: 00 hrs : 00 mins
+  [TIMING] data: 0.4ms | fwd: 776.6ms | loss: 1.7ms | bwd: 1558.1ms | clip: 0.0ms | optim: 23.7ms
+
+=== Training complete ===

logs/ext32k/upload.log

@@ -0,0 +1,287 @@
+Error: (Request ID: Root=1-6aa1a1d8-457715e757ca8c980ff68a3e;f40f52fa-1b2e-40f6-8c09-97b686cfa813)
+
+403 Forbidden: Forbidden: pass `create_pr=1` as a query parameter to create a Pull Request.
+Cannot access content at: https://huggingface.co/api/models/unparallelled/BluScriptCP/commit/main.
+Make sure your token has the correct permissions.
+Hint: set HF_DEBUG=1 as environment variable for full traceback.
+Found 16 files to upload
+Traceback (most recent call last):
+  File "/root/envv/bin/hf", line 6, in <module>
+    sys.exit(main())
+             ^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/hf.py", line 133, in main
+    app()
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 1631, in __call__
+    return self.main(*args, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 1552, in main
+    rv = self.invoke(ctx)
+         ^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/_cli_utils.py", line 131, in invoke
+    return super().invoke(ctx)
+           ^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 2032, in invoke
+    return _process_result(sub_ctx.command.invoke(sub_ctx))
+                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 1415, in invoke
+    return ctx.invoke(self.callback, **ctx.params)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 910, in invoke
+    return callback(*args, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/_framework.py", line 365, in handler
+    return func(**call_kwargs)
+           ^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/upload.py", line 258, in upload
+    result = run_upload()
+             ^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/upload.py", line 244, in run_upload
+    return api.upload_folder(
+           ^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 88, in _inner_fn
+    return fn(*args, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 2153, in _inner
+    return fn(self, *args, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 6061, in upload_folder
+    return pipelined_upload(
+           ^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/_upload_pipeline.py", line 684, in pipelined_upload
+    ).run()
+      ^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/_upload_pipeline.py", line 392, in run
+    self._coordinator_loop()
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/_upload_pipeline.py", line 454, in _coordinator_loop
+    batch.xet_commit = self.xet_session.new_upload_commit(
+                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+ConnectionError: Network error: Request error: HTTP status client error (403 Forbidden), domain: https://huggingface.co/api/models/unparallelled/BluScriptCP/xet-write-token/main
+Found 16 files to upload
+Traceback (most recent call last):
+  File "/root/envv/bin/hf", line 6, in <module>
+    sys.exit(main())
+             ^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/hf.py", line 133, in main
+    app()
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 1631, in __call__
+    return self.main(*args, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 1552, in main
+    rv = self.invoke(ctx)
+         ^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/_cli_utils.py", line 131, in invoke
+    return super().invoke(ctx)
+           ^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 2032, in invoke
+    return _process_result(sub_ctx.command.invoke(sub_ctx))
+                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 1415, in invoke
+    return ctx.invoke(self.callback, **ctx.params)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 910, in invoke
+    return callback(*args, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/_framework.py", line 365, in handler
+    return func(**call_kwargs)
+           ^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/upload.py", line 258, in upload
+    result = run_upload()
+             ^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/upload.py", line 244, in run_upload
+    return api.upload_folder(
+           ^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 88, in _inner_fn
+    return fn(*args, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 2153, in _inner
+    return fn(self, *args, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 6061, in upload_folder
+    return pipelined_upload(
+           ^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/_upload_pipeline.py", line 684, in pipelined_upload
+    ).run()
+      ^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/_upload_pipeline.py", line 392, in run
+    self._coordinator_loop()
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/_upload_pipeline.py", line 454, in _coordinator_loop
+    batch.xet_commit = self.xet_session.new_upload_commit(
+                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+ConnectionError: Network error: Request error: HTTP status client error (403 Forbidden), domain: https://huggingface.co/api/models/unparallelled/BluScriptCP/xet-write-token/main
+Found 16 files to upload
+Traceback (most recent call last):
+  File "/root/envv/bin/hf", line 6, in <module>
+    sys.exit(main())
+             ^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/hf.py", line 133, in main
+    app()
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 1631, in __call__
+    return self.main(*args, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 1552, in main
+    rv = self.invoke(ctx)
+         ^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/_cli_utils.py", line 131, in invoke
+    return super().invoke(ctx)
+           ^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 2032, in invoke
+    return _process_result(sub_ctx.command.invoke(sub_ctx))
+                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 1415, in invoke
+    return ctx.invoke(self.callback, **ctx.params)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 910, in invoke
+    return callback(*args, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/_framework.py", line 365, in handler
+    return func(**call_kwargs)
+           ^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/upload.py", line 258, in upload
+    result = run_upload()
+             ^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/upload.py", line 244, in run_upload
+    return api.upload_folder(
+           ^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 88, in _inner_fn
+    return fn(*args, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 2153, in _inner
+    return fn(self, *args, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 6061, in upload_folder
+    return pipelined_upload(
+           ^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/_upload_pipeline.py", line 684, in pipelined_upload
+    ).run()
+      ^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/_upload_pipeline.py", line 392, in run
+    self._coordinator_loop()
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/_upload_pipeline.py", line 454, in _coordinator_loop
+    batch.xet_commit = self.xet_session.new_upload_commit(
+                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+ConnectionError: Network error: Request error: HTTP status client error (403 Forbidden), domain: https://huggingface.co/api/models/unparallelled/BluScriptCP/xet-write-token/main
+Found 16 files to upload
+Traceback (most recent call last):
+  File "/root/envv/bin/hf", line 6, in <module>
+    sys.exit(main())
+             ^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/hf.py", line 133, in main
+    app()
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 1631, in __call__
+    return self.main(*args, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 1552, in main
+    rv = self.invoke(ctx)
+         ^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/_cli_utils.py", line 131, in invoke
+    return super().invoke(ctx)
+           ^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 2032, in invoke
+    return _process_result(sub_ctx.command.invoke(sub_ctx))
+                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 1415, in invoke
+    return ctx.invoke(self.callback, **ctx.params)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 910, in invoke
+    return callback(*args, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/_framework.py", line 365, in handler
+    return func(**call_kwargs)
+           ^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/upload.py", line 258, in upload
+    result = run_upload()
+             ^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/upload.py", line 244, in run_upload
+    return api.upload_folder(
+           ^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 88, in _inner_fn
+    return fn(*args, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 2153, in _inner
+    return fn(self, *args, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 6061, in upload_folder
+    return pipelined_upload(
+           ^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/_upload_pipeline.py", line 684, in pipelined_upload
+    ).run()
+      ^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/_upload_pipeline.py", line 392, in run
+    self._coordinator_loop()
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/_upload_pipeline.py", line 454, in _coordinator_loop
+    batch.xet_commit = self.xet_session.new_upload_commit(
+                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+ConnectionError: Network error: Request error: HTTP status client error (403 Forbidden), domain: https://huggingface.co/api/models/unparallelled/BluScriptCP/xet-write-token/main
+Found 11 files to upload
+Traceback (most recent call last):
+  File "/root/envv/bin/hf", line 6, in <module>
+    sys.exit(main())
+             ^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/hf.py", line 133, in main
+    app()
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 1631, in __call__
+    return self.main(*args, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 1552, in main
+    rv = self.invoke(ctx)
+         ^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/_cli_utils.py", line 131, in invoke
+    return super().invoke(ctx)
+           ^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 2032, in invoke
+    return _process_result(sub_ctx.command.invoke(sub_ctx))
+                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 1415, in invoke
+    return ctx.invoke(self.callback, **ctx.params)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/click/core.py", line 910, in invoke
+    return callback(*args, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/_framework.py", line 365, in handler
+    return func(**call_kwargs)
+           ^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/upload.py", line 258, in upload
+    result = run_upload()
+             ^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/cli/upload.py", line 244, in run_upload
+    return api.upload_folder(
+           ^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 88, in _inner_fn
+    return fn(*args, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 2153, in _inner
+    return fn(self, *args, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 6061, in upload_folder
+    return pipelined_upload(
+           ^^^^^^^^^^^^^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/_upload_pipeline.py", line 684, in pipelined_upload
+    ).run()
+      ^^^^^
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/_upload_pipeline.py", line 392, in run
+    self._coordinator_loop()
+  File "/root/envv/lib/python3.12/site-packages/huggingface_hub/_upload_pipeline.py", line 454, in _coordinator_loop
+    batch.xet_commit = self.xet_session.new_upload_commit(
+                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+ConnectionError: Network error: Request error: HTTP status client error (403 Forbidden), domain: https://huggingface.co/api/models/unparallelled/BluScriptCP/xet-write-token/main
+Error: (Request ID: Root=1-6aa1a1e0-25366bfd3ec4b63e4e77117b;07652197-623f-47f4-82b3-a3f7bc1820de)
+
+403 Forbidden: Forbidden: pass `create_pr=1` as a query parameter to create a Pull Request.
+Cannot access content at: https://huggingface.co/api/models/unparallelled/BluScriptCP/commit/main.
+Make sure your token has the correct permissions.
+Hint: set HF_DEBUG=1 as environment variable for full traceback.
+url=https://huggingface.co/unparallelled/BluScriptCP/commit/a0fa526c921fd9d32b16968bd7485101cb5c6a7b
+Found 16 files to upload
+Uploading... 16/16 files checked, 0/0 uploaded (0.00B transferred), 0 committed in 0 commit(s)
+url=https://huggingface.co/unparallelled/BluScriptCP/commit/d32f8ce490c1e839f4bb4e2d4fe774eee93212c1
+Found 16 files to upload
+Uploading... 16/16 files checked, 0/0 uploaded (0.00B transferred), 0 committed in 0 commit(s)
+url=https://huggingface.co/unparallelled/BluScriptCP/commit/1810c62fab00f3a17db44749f47e63958db67f48
+Found 16 files to upload
+Uploading... 16/16 files checked, 0/0 uploaded (0.00B transferred), 0 committed in 0 commit(s)
+url=https://huggingface.co/unparallelled/BluScriptCP/commit/14fccccfd36a5c61820bc4f3d98547f48d7c6598
+Found 16 files to upload
+Uploading... 16/16 files checked, 0/0 uploaded (0.00B transferred), 0 committed in 0 commit(s)
+url=https://huggingface.co/unparallelled/BluScriptCP/commit/7bba9ad75ea3942c248f13484a00aa0333629e1d
+Found 11 files to upload
+Uploading... 11/11 files checked, 0/0 uploaded (0.00B transferred), 0 committed in 0 commit(s)