Document PC rationale, local Desktop results and reported cache experience

README.md

@@ -24,6 +24,25 @@ This repository contains setup scripts and measured compatibility results, **not
 
 The runtime uses [FreeToken PR #797](https://github.com/FlashML-org/FreeToken/pull/797), pinned to `711a86ef11c67bbe6c69ff59370594f64c6ef1b8`. It is an independent experimental package, not an official FreeToken or Recursive release. The integration code belongs to its upstream contributors.
 
+## Why I put this setup together
+
+I wanted to use Flint for local coding and agent work on the PC I already own: an RTX 5070 Ti with 16 GB VRAM and 64 GB system RAM. My earlier FreeToken setups with [Ornith](https://huggingface.co/qualitymaterial/Ornith-1.5-35B-A3B-FP8-BLOCK-FreeToken) and [Qwen3.8 Distill](https://huggingface.co/qualitymaterial/Qwen3.8-35B-A3B-Distill-NVFP4-FreeToken) followed the same idea: use system RAM for a model larger than the GPU's memory, keep useful cache capacity, and make it practical to use from FreeToken Desktop.
+
+Flint's official NVFP4 GGUF is about 22.86 GB, so the complete file cannot fit in my 16 GB GPU. FreeToken's CPU/GPU expert offloading made this a workable local setup. NVFP4 was a practical choice because an official checkpoint already existed and it passed our compatibility checks; I did not need to create another quantization. Q4 was also explored, but those tests do not establish that either format is better for every workload.
+
+The goal is a useful daily setup, with responsive generation and room for reusable context, without buying a larger GPU or keeping a rented GPU running. Local inference avoids a hosted inference API's per-token charges; the PC, electricity, storage and any external tools still have their own costs. Runpod was used temporarily for compatibility testing. Daily use runs on my own machine.
+
+In my follow-up use on October 23, 2026, Flint worked well in FreeToken with a **323K token cache**, and generation felt very fast. That is my experience with this setup, not a controlled speed comparison. **A cache-pool allocation is not the same as a per-conversation context limit or a completed long-context test.** The exact allocation and a matching launch recipe for that later configuration have not been captured in this package; its included launcher still defaults to the earlier tested 8K profile.
+
+| Choice | Reason on my PC |
+| --- | --- |
+| Official Flint NVFP4 | Use the existing official quantization for local coding and agent experiments |
+| RTX 5070 Ti 16 GB + 64 GB RAM | Work with the hardware I already own |
+| CPU/GPU expert offloading | Use system RAM for weights that exceed VRAM capacity |
+| FreeToken Desktop | Keep model selection and chat in the interface I already use |
+| Large reusable token cache | Preserve capacity for repeated context while balancing weights and runtime memory |
+| Pinned setup and recorded checks | Give others a concrete starting point and show what was actually tested |
+
 ## Requirements and setup
 
 Linux x86-64 or Ubuntu WSL2, NVIDIA CUDA 13 compatible driver, Python 3.12 managed by uv, git, and substantial free disk space (allow 70 GB). The GGUF is 22.86 GB and needs additional system RAM, GPU cache and workspace. The local 64 GB host / 16 GB RTX 5070 Ti passed the 8K short checks described below. Other hardware and larger contexts need separate validation. Do not run another large GPU model at the same time.
@@ -37,6 +56,8 @@ bash serve.sh
 
 Windows users run installation inside Ubuntu WSL first. `Start-Flint.ps1` then starts that same isolated environment. This package does not replace the FreeToken Desktop engine or register a model in its model picker. It serves FreeToken's local API at `http://127.0.0.1:1919/v1`; connect an OpenAI-compatible client there. Stop with Ctrl+C. The engine must finish loading before inference succeeds; the HTTP health route alone does not establish model readiness.
 
+The maintainer's separate local Desktop integration was subsequently verified using a WSL daemon and an imported model entry. Its machine-specific Desktop adapter and shortcut are not included in this portable package; installing these scripts alone does not perform that integration.
+
 The installer downloads pinned third-party FreeToken wheels and source. Nightly asset URLs can disappear when upstream replaces the release; if an asset is missing, installation stops. Do not silently substitute another build.
 
 The launcher starts one request at 8K context with hybrid expert offload and a 2 GiB GPU expert cache. `FLINT_CONTEXT=32768 bash serve.sh` selects 32K. Its `--moe-cpu-layers auto` is a WSL pinning adjustment; this WSL profile passed the local short acceptance checks. The cloud run used the same settings except that adjustment.
@@ -52,7 +73,9 @@ On a disposable Linux Runpod RTX 5090 32 GB using the pinned integration and mat
 
 Local follow-up on 2026-10-23: Ubuntu WSL on an RTX 5070 Ti 16 GB passed all four short serving checks, including generated interval-merging code against five cases, tool roundtrip and streaming. Profile: 8192 tokens, 2 GiB expert cache, hybrid execution, eight CPU workers and automatic pinning. Runtime occupied approximately 5.4 GB GPU memory including other host use. Python's base runtime needed a Linux path without spaces for FlashInfer compilation; the installer now handles a spaced base path by copying that runtime into the isolated installation. This new automatic copy path has syntax validation; the local repair used the equivalent manual copy.
 
-Native FreeToken Windows Desktop integration, local 32K, full 262K context, concurrent-agent workload and general coding benchmarks remain unverified. See RESULTS.md for the earlier cloud evidence. Preserve returned reasoning_content and tool calls in conversation history. Flint uses the glm47 tool parser and optional flint reasoning parser.
+Desktop follow-up on the same PC: FreeToken Desktop successfully imported Flint, started the WSL engine, and returned two requested exact replies in Chat. The first took about 60 seconds including initial CUDA sampling-kernel compilation; the second took 975 ms for 20 tokens (Desktop-reported 20.5 tokens/s). These are short integration checks, not a general throughput benchmark. Reopening Desktop through the local Flint shortcut retained the running model.
+
+Native Windows engine support, local 32K retrieval, full 262K context, concurrent-agent workload and general coding benchmarks remain unverified by these recorded checks. The later 323K cache observation above is a maintainer report, not a validated 323K conversation window. See RESULTS.md for the earlier cloud evidence. Preserve returned reasoning_content and tool calls in conversation history. Flint uses the glm47 tool parser and optional flint reasoning parser.
 
 ## Licenses and attribution