⚡ layerstream-plugin — Edge Out-of-Core LLM Benchmarks

Bare-metal 4-engine comparison on an 8 GB NVIDIA Jetson Orin Nano (7.3 GiB RAM, CUDA sm_87, LUKS2 NVMe, INA3221 hardware power rail): layerstream-plugin v0.1.0 vs. stock llama.cpp (Ollama/LM Studio runtime), Hugging Face accelerate, and AirLLM 4.0.0.

Qwen2.5-14B (8.37 GiB)
3.41 – 5.97 tok/s
0 MiB swap · 2,961 MiB/s O_DIRECT · 1.91 J/tok net
OLMoE-1B-7B (Sparse + TinyLFU)
12.0 – 13.7 tok/s
15.8×–17.9× vs full-layer R=0 · 98–100% cache hit
Qwen2.5-32B (2.53× RAM)
0.84 – 0.94 tok/s
18.5 GiB on 7.3 GiB RAM · Stock llama.cpp +17.2 GiB swap
Multi-Turn KV Prefix Reuse
4.97× faster TTFT
14.88 s → 2.99 s on 1,377-token conversation

Head-to-Head Engine Comparison (Orin Nano Bare Metal)

Compare measured throughput, latency, energy, and memory pressure across all 4 engines. Failed runs (OOM / 300 s timeout) are shown honestly alongside successful runs.

Qwen2.5-14B Iterative Optimization (R2 → R8)

Every engineering change profiled with microsecond I/O + DMA timers on bare-metal Jetson Orin Nano (`code_edit` decode tok/s).

Takeaway: Zero-copy O_DIRECT → cudaHostAlloc + per-tensor cudaMemcpyAsync + io_threads=1 + adaptive speculation lifted 14B from 0.19 tok/s (pure stream) to 3.41–5.97 tok/s with 2,961 MiB/s sustained through LUKS2 dm-crypt.

Sparse MoE + Negative Ablations

Honest negative ablations matter just as much as wins — here is what actually hurt vs. helped on real hardware.

Key findings: (1) Multi-threaded O_DIRECT (io_threads=4) regressed -57% on Linux LUKS2 dm-crypt. (2) Spending 580 MiB on a 1.5B draft instead of extra resident layers lost 2.5×–3.0× to 0.5B. (3) TinyLFU hot-expert caching jumped OLMoE 15.8×–17.9× (`0.76 → 13.67 tok/s`).

Interactive Hardware Estimator (Mirrors layerstream-plugin check)

Estimate resident layer pinning, cold load time, and expected decode speed on your own machine (Jetson, laptop, or AMD 8700G 64 GB APU).

5.2 GiB
2,800 MiB/s
8.4 GiB

Normalized Benchmark Records (0 / 130 rows)

Loaded directly from kleinbem/layerstream-orin-nano-benchmarks (Parquet / JSON). Use keyboard or horizontal scroll to inspect all telemetry columns.

Normalized Orin Nano benchmark runs
Round Engine Mode Model Workload Status R Load (s) TTFT (s) Decode tok/s pread MiB/s Peak RSS Swap Δ J/tok (Net)