⚡ layerstream-plugin — Edge Out-of-Core LLM Benchmarks
Bare-metal 4-engine comparison on an 8 GB NVIDIA Jetson Orin Nano (7.3 GiB RAM, CUDA sm_87, LUKS2 NVMe, INA3221 hardware power rail):
layerstream-plugin v0.1.0 vs. stock llama.cpp (Ollama/LM Studio runtime), Hugging Face accelerate, and AirLLM 4.0.0.
Head-to-Head Engine Comparison (Orin Nano Bare Metal)
Compare measured throughput, latency, energy, and memory pressure across all 4 engines. Failed runs (OOM / 300 s timeout) are shown honestly alongside successful runs.
Qwen2.5-14B Iterative Optimization (R2 → R8)
Every engineering change profiled with microsecond I/O + DMA timers on bare-metal Jetson Orin Nano (`code_edit` decode tok/s).
O_DIRECT → cudaHostAlloc + per-tensor cudaMemcpyAsync + io_threads=1 + adaptive speculation lifted 14B from 0.19 tok/s (pure stream) to 3.41–5.97 tok/s with 2,961 MiB/s sustained through LUKS2 dm-crypt.
Sparse MoE + Negative Ablations
Honest negative ablations matter just as much as wins — here is what actually hurt vs. helped on real hardware.
O_DIRECT (io_threads=4) regressed -57% on Linux LUKS2 dm-crypt. (2) Spending 580 MiB on a 1.5B draft instead of extra resident layers lost 2.5×–3.0× to 0.5B. (3) TinyLFU hot-expert caching jumped OLMoE 15.8×–17.9× (`0.76 → 13.67 tok/s`).
Interactive Hardware Estimator (Mirrors layerstream-plugin check)
Estimate resident layer pinning, cold load time, and expected decode speed on your own machine (Jetson, laptop, or AMD 8700G 64 GB APU).
Normalized Benchmark Records (0 / 130 rows)
Loaded directly from kleinbem/layerstream-orin-nano-benchmarks (Parquet / JSON). Use keyboard or horizontal scroll to inspect all telemetry columns.
| Round | Engine | Mode | Model | Workload | Status | R | Load (s) | TTFT (s) | Decode tok/s | pread MiB/s | Peak RSS | Swap Δ | J/tok (Net) |
|---|