Upload FINDINGS.md with huggingface_hub

FINDINGS.md

@@ -261,3 +261,130 @@ renamed_expert_keys   80 040     = 92 layers x 145 experts x 6 tensors
 Full width makes it the serious candidate for working French. It does not fit
 2× A40 at MXFP4 (347.5 GB against 185 GB of fast memory); either 4× A40 at
 $1.80/h, or an imatrix-guided quantisation down to ~1.7 bit.
+
+---
+
+# Session 2 — what actually bounds decode, and what long context costs
+
+## Decode is serialisation-bound, not bandwidth-bound
+
+Six placements had moved 60–88 GB of weights between CPU and GPU without
+changing throughput, and no mechanism had survived measurement. The question
+none of them asked was whether the hardware is *idle* between tokens. It is.
+
+Aggregate throughput against concurrent streams, 5× A40, 64 tokens each,
+distinct prefixes so no stream rides another's prompt cache:
+
+```
+1 stream     5.56 tok/s aggregate    5.56 per stream
+2 streams    9.85 tok/s aggregate    4.93 per stream   1.77x
+4 streams   14.53 tok/s aggregate    3.63 per stream   2.61x
+```
+
+Concurrency scales, so single-stream decode leaves silicon idle. The
+bandwidth arithmetic agrees independently: ~7 GB of weights are touched per
+token, and at 138 ms/token that is **~51 GB/s against 696 GB/s available** on
+the active card — 7% of the roof. llama.cpp splits layers across GPUs, so at
+batch 1 exactly one card computes while the other four wait.
+
+This retires the last open question from session 1 and explains why every
+kernel-level optimisation attempt returned nothing: the expert GEMM was never
+the constraint.
+
+## Sustained decode on real output: 7.22 tok/s
+
+The 7.3 tok/s headline was measured on selftests, three of whose five prompts
+produce degenerate repetition (`# [CS_ah]`, `et d'et d'`) — and a repetition
+loop decodes cheap. Re-measured on 600 tokens of coherent prose: **7.22 tok/s**.
+The number holds; it is now measured on output worth having.
+
+## Prefill: 167 tok/s cold, ~free warm
+
+Every selftest prompt is 6–22 tokens, so its "prefill tok/s" is per-request
+overhead and says nothing. Measured properly, 3408-token prompt:
+
+```
+cold   3408 tok in 20.4 s = 167 tok/s     (23x the decode rate — prefill batches)
+warm      4 tok in  0.53 s                (slot prompt cache reprocessed only the delta)
+```
+
+For a Claude Code client sending a 32k system prompt: **~3 min once**, then
+only the changed tokens on every later turn.
+
+## Long context is nearly free on this architecture
+
+From the config, not from assumption: `kv_lora_rank 512`, `qk_rope_head_dim
+64`, `full_attn_layers` every 4th layer, `max_position_embeddings 1048576`.
+
+```
+24 of 93 layers cache at all (MLA); the other 69 are KDA — recurrent state,
+constant regardless of context length.
+
+KV per token = 24 x (512 + 64) x 2 = 27.6 KB
+
+  65536 ctx ->  1.8 GB      262144 ->  7.2 GB      1048576 -> 29 GB
+```
+
+Against ~38 GB left free by the weights on a 5-GPU host. **16384 was never a
+hardware limit, just an untested default** — and it could not accept a single
+Claude Code request. What constrains context here is the weights sharing the
+cards, never the cache.
+
+## Cost reality check
+
+Runpod hosts an official Kimi-K3 endpoint (`moonshot-kimi`). Measured:
+300 tokens in 10.281 s, billed $0.004797.
+
+```
+                    speed         $/1M output    quality
+official          29.2 tok/s          ~16        full precision, French works
+ours (1 stream)    7.2 tok/s          ~85        1.14 bpw, French broken
+ours (4 streams)  14.5 tok/s          ~42
+```
+
+The official endpoint is 4x faster and 5.7x cheaper. Self-hosting buys
+privacy, no per-token billing, and context control — not price or quality.
+Worth restating whenever the effort seems to justify itself on cost.
+
+## No smaller GGUF exists
+
+Ours is the most compact Kimi-K3 published, because it is the Width50 variant
+(`moe_intermediate_size` 1536 rather than 3072) — which is also why French is
+broken.
+
+```
+ours   REAP448 Width50 IQ1_S       181 GiB
+       0xTank REAP568 UD-IQ1_S     246 GiB
+       mmnga-o REAP50 UD-IQ1_S     305 GiB
+       prometheusAIR REAP55 IQ1_M  319 GiB
+       hellohazime REAP640ja       411 GiB
+```
+
+A "light" build for a single A40 (46 GB, ~37 GB of weights after a 256k KV)
+does not exist and would have to be produced — roughly a 64-expert REAP.
+
+## Under $1 per million tokens is not reachable with K3
+
+At $0.44/h per A40, $1/1M output requires **122 tok/s aggregate per card**. The
+5× pod at $2.20/h would need 611 tok/s; it delivers 14.5. That is a 42x gap,
+not a tuning problem.
+
+What does reach it, from HF configs (throughput figures are bandwidth-derived
+estimates, not measured):
+
+```
+Qwen3-Coder-30B-A3B  30B/3B active  ~30 GB FP8  KV 12 GiB@256k  1x A40  ~$0.10-0.25/1M
+Qwen3-Coder-Next     80B/3B active  ~74 GB FP8  KV 12 GiB@256k  2x A40  ~$0.20-0.80/1M
+Qwen3.8-27B          dense          ~26 GB FP8  KV 32 GiB@256k  256k does not fit on 1 card
+```
+
+Served by **vLLM, not llama.cpp** — tensor parallelism and continuous batching
+are exactly what the concurrency measurement above shows llama.cpp lacking.
+
+## Operational trap: bare `wait` kills the hot-reload watcher
+
+`llama-server` is started with `&`, so a bare `wait` anywhere later in the
+script waits for *it* too, forever. The server keeps serving, the selftest
+never finishes, the watcher never starts, and the pod silently ignores every
+published update. Symptom: `/props` reports the old config long after a push.
+Always `wait "$pid"` with an explicit pid.