VRAM requirements
64 GB is more than enough — the whole model fits in VRAM.
Edge0-145B FP4 needs about 39 GB on the GPU: experts 34.9 + dense 1.55 + D/U 2.67. Same story for the PLE table — 51 GB on the SSD, read with --ple-io direct, no VRAM and no measurable cost. So 64 GB holds the entire pack with room to spare. you can use these flags
--expert-cache auto --prefill auto --kv int8 --spec 4 --mtp rt
--ple-gguf ple-fp8.gguf --ple-io direct
You also do not need 64 GB to run it by. 16 GB VRAM + 20 GB RAM gives close to 50 tok/s with --resident-budget-gib 20 --expert-profile expert-profile.bin
I've tried on a dual RTX5070Ti (16GB + 16GB VRAM) build with 128GB VRAM. the nvfp4 keep crashing out of memory. How can we run on 16GB VRAM?
Is there any quality difference between the nvfp4 vs q3k version? It's a bit confusing that edge0 treats both the same?
I've tried on a dual RTX5070Ti (16GB + 16GB VRAM) build with 128GB VRAM. the nvfp4 keep crashing out of memory. How can we run on 16GB VRAM?
Is there any quality difference between the nvfp4 vs q3k version? It's a bit confusing that edge0 treats both the same?
16 GB VRAM is enough — Edge0 runs on 16 GB VRAM + 20 GB RAM. This sounds like a configuration issue, not a memory limit.
Can you share your config? In particular the engine command line and whether you set --vram-reserve-mib (it controls how much VRAM the engine leaves free for everything else; the rest goes to the expert cache). can you try --expert-cache 4000
--vram-reserve-mib 1024
--kv-resident 32768
I lost my earlier config, but I tried your recommendation it doesn't work either, crash when starting. Claude suggested to use --expert-cache of 1024 it starts, but crashed with following during first inference.
It seems unstable still compared to Strata. My hardware works well with both Strata and Speedy for qwen3.8-flash-next. Is it because this model has more param (bigger)?
"time":"2026-11-13T13:15:56.304Z","endpoint":"/v1/messages","source":"172.17.0.1","http_status":503,"stream":false,"total_seconds":1.06794404,"code":"server_error","message":"the engine stopped unexpectedly (exit code 1); the next request restarts it","stop_reason":"failed","input_tokens":null,"reused_tokens":null,"generated_tokens":null,"first_token_seconds":null,"decode_tokens_per_second":null,"prefill_tokens_per_second":null,"decode_seconds":null}
[strata] the engine had stopped (exit code 1); starting it again (a minute or two) ...
[strata] the engine is running again
[strata] the engine reported an error: prefill latent: an expert is neither resident, streamed nor on the peer
[strata] done: 0 tokens in 0 s (no speed) (error, cancel=False)
[strata] 400 invalid request: prefill latent: an expert is neither resident, streamed nor on the peer
{"time":"2026-11-13T13:16:14.793Z","endpoint":"/v1/messages","source":"172.17.0.1","http_status":400,"stream":false,"total_seconds":17.412053988,"code":"invalid_request_error","message":"prefill latent: an expert is neither resident, streamed nor on the peer"}
[strata] the engine stopped unexpectedly (exit code 1). The engine exited (code 1) after it reported: prefill latent: an expert is neither resident, streamed nor on the peer - please report it at github.com/Niko1221/Strata/issues with the log. The next request starts the engine again.
The crash on your dual RTX5070Ti setup sounds like a memory fragmentation or eviction issue rather than a hard VRAM limit, especially with that much total VRAM (128GB). The suggestion to reserve more VRAM for the engine (--vram-reserve-mib 1024) and pinning the expert cache (--expert-cache 4000) is a solid diagnostic step. If you're still seeing OOM errors, it might be worth checking if the --kv-resident setting is too aggressive or if there's a bug in how the engine handles multi-GPU NVFP4 layouts on that specific hardware combination. Have you tried running with --expert-cache auto to let the system manage eviction dynamically instead of pinning a large chunk?
What I got from claude, apparently this fork of strata doesn't work for second GPU:
What was wrong
Your crash was the engine's --peer-device second-card expert-cache tier, which this engine build (edge0-strata 0.1.55) never made latent-aware for the q3k pack:
The prompt plan hands ~35% of the streamed experts to the peer's own staging ring (STRATA_PF_PEER_STREAM, default 0.35), but the latent prompt walk (STRATA_LAT_FUSED_Q3K=1) still demands a blob for every routed expert — the diverted ones are neither in the primary's cache, nor streamed, nor resident on the peer → prefill latent: an expert is neither resident, streamed nor on the peer, engine exit 1, on every request, after every restart.
set_peer declines NVFP4 packs (per-expert tail) but not q3k — which is why your earlier NVFP4 setup ran fine on two cards.
Behind it hid a second bug: I bypassed the crash with the engine's own --peer-prefill-rows 0 (prompt path fully on the primary) — and every generation then came back NaN logits (#879), deterministically, before the first token. The peer's decode rows poison the model on this pack. So both two-card modes are broken upstream right now.
What works (verified end-to-end)
Single card + KV streaming — 2,301-token prompt prefilled in ~5 s, correct answers (17×23 = 391), clean end_turn:
STRATA_ARGS="--kv-resident 32768 --vram-reserve-mib 1024 --mtp-draft-vocab …/draft_vocab_en.bin"
Note --kv-resident 32768 is mandatory on one 16 GB card at 262144 context — without it the engine exits before ready (verify: the device arena (235 MiB) does not fit), which I confirmed by running the engine binary directly.
Details:
The Edge0-145B-q3k pack on the edge0-strata engine (0.1.55, image built 2026-11-13) is broken in every two-card mode; the working setup is one 16 GB card with --kv-resident 32768 --vram-reserve-mib 1024 at the full 262144 context (verified end-to-end 2026-11-13: coherent generations, 391 = 17×23).
Why: the engine's --peer-device expert-cache tier was never made latent-aware for the q3k pack: (1) the prompt plan hands ~35% of streamed experts to the peer's staging ring (STRATA_PF_PEER_STREAM, default 0.35) but the latent fused prompt walk still expects a blob for every routed expert → first prompt dies "prefill latent: an expert is neither resident, streamed nor on the peer", engine exit 1, every restart, every request. (2) With the prompt path kept on the primary (--peer-prefill-rows 0, which skips Prefill::set_peer entirely), the peer's DECODE rows still poison the model: "logits became non-finite (#879)" on every request, deterministically, before the first token. set_peer declines NVFP4 packs (per-expert tail) but not q3k (tail_off = 0), which is why the NVFP4 pack worked with two cards and q3k does not.
How to apply: edge0/start-edge0-145b-q3k.sh defaults to single-GPU; EDGE0_MULTI_GPU=1 opts into the (still broken) peer branch for testing future engine builds. Without --kv-resident a single 16 GB card cannot start at 262144 ctx — the engine exits before ready with "verify: the device arena (235 MiB) does not fit". Debugging engine-side failures: the engine's stderr only lands somewhere if the launch template has a log key (the entrypoint now writes /tmp/engine.log; read it with docker exec edge0-cuda tail -100 /tmp/engine.log); for a bare run, docker run --entrypoint /opt/edge0/edge0-strata/build/strata edge0-cuda --serve <template args> prints it directly. The entrypoint's template rewrites (CUDA_VISIBLE_DEVICES, --expert-cache, --prefill, log) are the hooks into the delegated engine launch. If upstream fixes the tier, re-test with EDGE0_MULTI_GPU=1 before re-defaulting it. Related: [[strata-dspark-drafter-moved-into-coder-folder]].
I'm trying again with nvfp4 version again now and will update if it works
Just tested nvfp4, same issue with q3k, only works on single GPU but not two GPUs.
Written for: the edge0-strata/Strata maintainers — full report saved as BUGREPORT-dual-gpu.md, ready to paste into an issue. Here's the same content, for you:
Commands (non-dockerized, raw engine)
The engine normally gets these args from serve/server.py expanding the launch template; this is the resolved form. Run from an edge0-strata checkout with build/strata built.
Single GPU (works, both packs):
PACK=/path/to/Edge0-145B-NVFP4 # or Edge0-145B-q3k
env
STRATA_LATENT_A="$PACK/experts.bin"
STRATA_LATENT_LABELS="$PACK/labels.bin"
STRATA_LATENT_DENSE="$PACK/latent_dense_q8.bin"
STRATA_LATENT_DENSE_FMT=8
STRATA_EMBED_VRAM=1 STRATA_LATENT_FUSED=1 STRATA_LATENT_FUSED_DU=1 STRATA_LATENT_U_GPU=1
CUDA_VISIBLE_DEVICES=0
./build/strata --serve
--pack "$PACK"
--native "$PACK/model-skeleton.gguf"
--native-dense-gguf "$PACK/model-skeleton.gguf"
--ple-gguf "$PACK/ple-fp8.gguf"
--ple-io direct
--expert-profile "$PACK/profile.bin"
--expert-cache auto --prefill auto
--spec 4 --spec-min-p 0.5
--mtp "$PACK/mtp"
--max-context 262144 --kv int8
--kv-resident 32768 --vram-reserve-mib 1024
(--kv-resident 32768 is mandatory on a 16 GB card at 262144 context — otherwise the engine exits before ready: verify: the device arena (235 MiB) does not fit. For q3k add STRATA_LAT_FUSED_Q3K=1.)
Dual GPU (the failing mode): same command with CUDA_VISIBLE_DEVICES=0,1 and these args added:
--peer-device 1 \
--peer-reserve-mib 2048 \
--prefill 1024 \
Diagnostic variants: --peer-prefill-rows 0 (peer out of the prompt path entirely) and STRATA_PF_PEER_STREAM=0 (peer's prompt share off).
Why dual GPU failed, and where the fault lives
It's the strata fork (edge0-strata's latent-MoE additions) — not upstream Strata's peer tier, and not Edge0 itself. Upstream's tier is designed for standard MoE packs; the broken paths are the fork's latent-specific code. And it's not the pack: both NVFP4 and q3k serve perfectly single-GPU on the same build. Since Edge0 ships this fork as its engine, from a user's view it's "Edge0's engine" — two distinct bugs:
Bug A — q3k prefill crash (prefill latent: an expert is neither resident, streamed nor on the peer, deterministic): With --peer-device, the per-chunk prompt plan sends ~35% of the streamed experts to the peer's own staging ring (ps_frac 0.35, STRATA_PF_PEER_STREAM), excluding them from the primary's stream sequence. The non-latent MMQ prompt path knows about this — a per-layer classification (on_peer) removes those experts from the primary's list and hands them to the peer. But the latent fused prompt walk (the default A-stage for latent packs) iterates all expert ids and, for any routed expert missing from the primary's sequence, resolves its blob as: primary cache → peer VRAM slot → error. A peer-streamed expert is in none of those — its bytes live transiently on the peer's ring — so the first chunk ≥1024 tokens dies at prefill.cpp:5586. Why NVFP4 escapes: set_peer refuses the peer prompt path for NVFP4 ("not for NVFP4 experts yet" — per-expert scale tails) and for Hadamard packs. q3k's shared colscale means tail_off == 0, so the guard passes and binds the peer — the exact guard that would have saved q3k is bypassed by its own format.
Bug B — NaN logits from peer decode (both packs; intermittent on NVFP4, deterministic on q3k): The decisive experiment: q3k with --peer-prefill-rows 0 — the peer completely unbound from the prompt path — still NaNs before the first token, every request. So the corruption is in the peer's decode-time row computation. For a standard MoE, peer rows are self-contained per expert; for Edge0's latent MoE they are not: each entry needs the D projection into the per-layer latent space, gate/up over latent rows selected via the per-layer label mapping (STRATA_LATENT_LABELS), the U projection back, and the colscale handling (shared per-layer on q3k, per-expert in-blob on NVFP4). The peer's decode path doesn't reproduce that chain correctly. On NVFP4 the failure defers until the adaptive swaps (PeerExperts::adapt) start feeding swapped-in experts after the first requests — which matches the measured pattern: 3 clean answers, then 13 non-finite (#879) events over 4 requests, permanent from there.
The one-line upstream mitigation (even before a real fix): make peer-tier setup decline latent packs the way set_peer already declines NVFP4 prompt participation — today it silently corrupts generation instead of refusing at startup. Full call-chain with line references (against fb0c9b6cc4150, engine 0.1.55) is in the report file, along with the repro matrix and the hardware/version block.
Just tested nvfp4, same issue with q3k, only works on single GPU but not two GPUs.
Written for: the edge0-strata/Strata maintainers — full report saved as BUGREPORT-dual-gpu.md, ready to paste into an issue. Here's the same content, for you:
Commands (non-dockerized, raw engine)
The engine normally gets these args from serve/server.py expanding the launch template; this is the resolved form. Run from an edge0-strata checkout with build/strata built.Single GPU (works, both packs):
PACK=/path/to/Edge0-145B-NVFP4 # or Edge0-145B-q3k
env
STRATA_LATENT_A="$PACK/experts.bin"
STRATA_LATENT_LABELS="$PACK/labels.bin"
STRATA_LATENT_DENSE="$PACK/latent_dense_q8.bin"
STRATA_LATENT_DENSE_FMT=8
STRATA_EMBED_VRAM=1 STRATA_LATENT_FUSED=1 STRATA_LATENT_FUSED_DU=1 STRATA_LATENT_U_GPU=1
CUDA_VISIBLE_DEVICES=0
./build/strata --serve
--pack "$PACK"
--native "$PACK/model-skeleton.gguf"
--native-dense-gguf "$PACK/model-skeleton.gguf"
--ple-gguf "$PACK/ple-fp8.gguf"
--ple-io direct
--expert-profile "$PACK/profile.bin"
--expert-cache auto --prefill auto
--spec 4 --spec-min-p 0.5
--mtp "$PACK/mtp"
--max-context 262144 --kv int8
--kv-resident 32768 --vram-reserve-mib 1024
(--kv-resident 32768 is mandatory on a 16 GB card at 262144 context — otherwise the engine exits before ready: verify: the device arena (235 MiB) does not fit. For q3k add STRATA_LAT_FUSED_Q3K=1.)Dual GPU (the failing mode): same command with CUDA_VISIBLE_DEVICES=0,1 and these args added:
--peer-device 1 \ --peer-reserve-mib 2048 \ --prefill 1024 \Diagnostic variants: --peer-prefill-rows 0 (peer out of the prompt path entirely) and STRATA_PF_PEER_STREAM=0 (peer's prompt share off).
Why dual GPU failed, and where the fault lives
It's the strata fork (edge0-strata's latent-MoE additions) — not upstream Strata's peer tier, and not Edge0 itself. Upstream's tier is designed for standard MoE packs; the broken paths are the fork's latent-specific code. And it's not the pack: both NVFP4 and q3k serve perfectly single-GPU on the same build. Since Edge0 ships this fork as its engine, from a user's view it's "Edge0's engine" — two distinct bugs:Bug A — q3k prefill crash (prefill latent: an expert is neither resident, streamed nor on the peer, deterministic): With --peer-device, the per-chunk prompt plan sends ~35% of the streamed experts to the peer's own staging ring (ps_frac 0.35, STRATA_PF_PEER_STREAM), excluding them from the primary's stream sequence. The non-latent MMQ prompt path knows about this — a per-layer classification (on_peer) removes those experts from the primary's list and hands them to the peer. But the latent fused prompt walk (the default A-stage for latent packs) iterates all expert ids and, for any routed expert missing from the primary's sequence, resolves its blob as: primary cache → peer VRAM slot → error. A peer-streamed expert is in none of those — its bytes live transiently on the peer's ring — so the first chunk ≥1024 tokens dies at prefill.cpp:5586. Why NVFP4 escapes: set_peer refuses the peer prompt path for NVFP4 ("not for NVFP4 experts yet" — per-expert scale tails) and for Hadamard packs. q3k's shared colscale means tail_off == 0, so the guard passes and binds the peer — the exact guard that would have saved q3k is bypassed by its own format.
Bug B — NaN logits from peer decode (both packs; intermittent on NVFP4, deterministic on q3k): The decisive experiment: q3k with --peer-prefill-rows 0 — the peer completely unbound from the prompt path — still NaNs before the first token, every request. So the corruption is in the peer's decode-time row computation. For a standard MoE, peer rows are self-contained per expert; for Edge0's latent MoE they are not: each entry needs the D projection into the per-layer latent space, gate/up over latent rows selected via the per-layer label mapping (STRATA_LATENT_LABELS), the U projection back, and the colscale handling (shared per-layer on q3k, per-expert in-blob on NVFP4). The peer's decode path doesn't reproduce that chain correctly. On NVFP4 the failure defers until the adaptive swaps (PeerExperts::adapt) start feeding swapped-in experts after the first requests — which matches the measured pattern: 3 clean answers, then 13 non-finite (#879) events over 4 requests, permanent from there.
The one-line upstream mitigation (even before a real fix): make peer-tier setup decline latent packs the way set_peer already declines NVFP4 prompt participation — today it silently corrupts generation instead of refusing at startup. Full call-chain with line references (against fb0c9b6cc4150, engine 0.1.55) is in the report file, along with the repro matrix and the hardware/version block.
Jackal — thanks, that report is genuinely useful, and the diagnosis points at the right place: the latent-MoE peer tier in our engine fork, not the pack and not your config. We are on it and will fix it as soon as we can. Until then, single GPU is the supported path, with --kv-resident 32768 as you found.
On NVFP4 vs Q3_K: they land about the same, and that is expected — Q3_K is not uniformly 3-bit. Only a subset of the modules are q3k and the rest stay at q4, so the overall precision ends up close to NVFP4's and the benchmark scores land in the same place. The card quoting one quality section for both is deliberate, not an oversight.
Also, same issue with --split-layer, the earlier one was using peer-mode
What happened
The split itself engages beautifully: the engine plans K by free VRAM, loads layers on both cards (CUDA1 took layers [24,48) + head + draft), doubles the cached experts (~9.0k of 24.6k profiled pairs ≈ 97% of routed mass, vs ~2.6k on one card), captures the verify-window graphs on both cards, and prompt prefill crosses the cards cleanly. Then the first decode verify window on the second card faults, on every request, and the engine stays wedged until restart:
verify: layer N never rang (an illegal memory access was encountered)
N is always the second card's first layer — the node that reads the cross-card hand-off buffer. For this latent model that hand-off carries the D-stage xl rows through mapped host memory (verify.cpp:996), which is the likely suspect.
The matrix (2× RTX 5070 Ti, NVFP4 pack, engine 0.1.55)
Run Split Spec Decode Draft Result
1 K=24 4 serial CUDA1 IMA at layer 24
2 K=25 4 serial CUDA1, +--vram-reserve-later-mib 1348 IMA at layer 25
3 K=25 0 — — refused: native packs need --spec ≥ 2
4 K=25 2 serial CUDA1 IMA at layer 25
5 K=22 4 --pipeline-windows 2 CUDA0 (STRATA_MTP_STAGE0=1) IMA at layer 22
Run 2 rules out VRAM pressure (768 MiB free on CUDA1 still faulted; CUDA0 runs the same graphs at 716 MiB), run 5 rules out draft placement. So both of the engine's two-card modes are dead for this latent model — same story as the peer tier, one level earlier in the pipeline. There's no no-MTP control: the engine refuses --spec 0 on a native pack. q3k wasn't retested (pack is deleted from disk).
State
Machine restored: no containers, GPUs idle — as it was before.
Findings recorded in BUGREPORT-dual-gpu.md (new §8 with the matrix), plus notes in start-edge0-145b-nvfp4.sh, entrypoint.sh, and README.md. Uncommitted — say "sync git" if you want them pushed.
Worth retrying when a newer edge0-strata build lands (the image re-pulls on rebuild); single-card with --kv-resident 32768 remains the only stable mode.