poolside

/

Laguna-S-2.1-FP8

like 20

Follow

Poolside 1.13k

Big issues with looping esp at fp8 kv cache. Basically unusable and dflash broken as well. (sglang lastest, rtx pro 6000)
Getting without spec decoding sys prompt leakage at 80 to 60 tok s as it goes to 200k context

Regular looping here as well, generally after a long reasoning period. 8 RTX 3090 on vLLM nightly, llama-swap config as follows -

  "laguna-s-2.1":
    name: "Laguna S 2.1"
    env:
      - "OMP_NUM_THREADS=1"
      - "CUDA_DEVICE_ORDER=PCI_BUS_ID"
      - "CUDA_VISIBLE_DEVICES=0,3,6,7,2,4,5,1"
      - "HF_TOKEN=hf_token"
      - "HF_XET_HIGH_PERFORMANCE=1"
      - "CUDA_HOME=/opt/cuda"
      - "VLLM_MARLIN_USE_ATOMIC_ADD=1"
      - "VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1"
      - "VLLM_FLOAT32_MATMUL_PRECISION=high"
      - "VLLM_USE_NCCL_SYMM_MEM=0"
      - "SAFETENSORS_FAST_GPU=1"
      - "VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0"
      - "VLLM_ALLOW_LONG_MAX_MODEL_LEN=1"
      - "LD_LIBRARY_PATH=/home/hubby/.venv/lib/python3.12/site-packages/nvidia/cu13/lib/:LD_LIBRARY_PATH"
    capabilities:
      in:
        - text
      out:
        - text
      tools: true
      context: 524288
    cmd: |
      uv run vllm serve poolside/Laguna-S-2.1-FP8
       --host 0.0.0.0
       --port ${PORT}
       --seed 3407
       --disable-custom-all-reduce
       --async-scheduling
       --enable-prefix-caching
       --enable-chunked-prefill
      --served-model-name laguna-s-2.1
      --tensor-parallel-size 8
      --enable-expert-parallel
      --max-model-len 524288
      --gpu-memory-utilization 0.96
      --attention-backend flashinfer
      --gdn-prefill-backend flashinfer
      --enable-flashinfer-autotune
      --reasoning-parser poolside_v1
      --tool-call-parser poolside_v1
      --enable-auto-tool-choice
      --max-num-seqs 4
      --max-num-batched-tokens 16382
      --dtype bfloat16
      --generation-config auto
      --default-chat-template-kwargs '{"enable_thinking": true}'
      --override-generation-config '{"temperature": 0.7, "top_p": 0.95}'
      --compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE","fast_moe_cold_start":true}'

Hi all.

Thank you for your comments. We're really sorry that the model's been acting strangely.

We've pushed a new version to this repo that takes away these issues in our internal testing. Please let us know if it doesn't take away the issues for you. We have one more update planned (for better 1M context support), but this one seems to work well at 256K.

We'll also make similar changes for INT4 & NVFP4 later today (although NVFP4 will not yet live on main).

If I don't hear back in the next 24 hours or so, I'll close this issue :)

Thanks for the fast response, Joe! Re-downloading now.

Do the DFlash models also need updates?

@Aur0r You can use the same DFlash checkpoint you've been using but it may offer slightly lower acceptance rates with updated target's checkpoint. We'll be following with updated DFlash checkpoints shortly.

I'm afraid that I am still seeing looping. Interestingly, sometimes, while the reasoning does still tend to loop sections, it also intersperses it with new reasoning, and does generally manage to correct itself. Unfortunately, this comes with a pretty hefty token usage penalty, given the large amount of repetition. Other times, it stays in the loop, albeit less frequently than before.

Also, even when looping does not occur, almost every attempt now finishes mid sentence for me while reasoning in long context tasks. I tested 3 times with a code review task that successfully completed yesterday, but each time I tried today, the turn was ended mid sentence while reasoning, without outputting an answer, generally once around 76000 tokens are spent. This only seems to happen when the assistant turn is purely interleaved reasoning and tool calls. If the model outputs content, it doesn't seem to happen.

If I reply with a prompt to continue, the model continues, and finishes the task.

Tried with both the config I posted earlier, and with the max context reduced to 262144. Also tried with both sending a max tokens limit of 32768 with the request, and removing the limit altogether from the request, made no difference.

EDIT: this new issue was on my end. I updated vLLM, pruned my huggingface cache, and reloaded the updated model. No issues since then 😄

Are you using vLLM 0.25.1? And do you think that pruning the hf cache was the main solution or was it the vLLM update or both? I am curious more about what you mean by ‘pruning’. I pulled the new files from hf after they were updated but do you believe a full redownload is needed?

Also, has anyone had any luck with opencode and this model? 2x RTX Pro 6000s + FP8 weights and FP8 or BF16 KV. I had to use --disable-custom-all-reduce as well or else vLLM would error out during initialization. Need to test for any improvements using opencode with the new files but I was getting weird behavior prior.

We can still reproduce excessive/repetitive reasoning with the updated FP8 checkpoint, even with vLLM 0.25.1 and DFlash completely disabled.

Unlike the cache/revision mixing issue mentioned above, our local copy appears internally consistent:

• Target revision: fc72eb9025540b2225a681bb81950cc1225ff286
• All 37 Hugging Face metadata entries point to that same revision
• All 24 weight shards are present
• No .incomplete files remain
• config.json SHA-256: 6337348c4988936be54276c637e7a2c1c6b25ec44114a199240944a3e0f7212e
• chat_template.jinja SHA-256: 444819b8ad4612870827ac05b9147fe9e3344d3850cae8c2790898fc514099ff

Environment

• 4 × RTX A6000 48 GB, compute capability 8.6
• Upstream vllm/vllm-openai:v0.25.1
• FP8 weights running through the Marlin fallback on A6000
• VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0
• Tensor parallel size: 4
• Max model length: 262,144
• Max sequences: 4
• Max batched tokens: 8,192
• GPU memory utilization: 0.95
• FP8 KV cache
• poolside_v1 reasoning and tool parsers
• Prefix caching and async scheduling enabled
• No speculative decoding and no DFlash model loaded

The relevant serving arguments are approximately:

vllm serve /path/to/Laguna-S-2.1-FP8 \
  --tensor-parallel-size 4 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 8192 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --reasoning-parser poolside_v1 \
  --tool-call-parser poolside_v1 \
  --enable-auto-tool-choice \
  --default-chat-template-kwargs '{"enable_thinking":true}' \
  --enable-prefix-caching \
  --async-scheduling

Request conditions

The tests were single-turn English prompts with thinking explicitly enabled.

For the main tests we used:

• temperature: 0.7
• top_p: 0.95
• top_k: 20
• chat_template_kwargs.enable_thinking: true

These were ordinary standalone math, physics, science, and coding questions, not long-context agent trajectories.

Most reproducible case: rotating-hoop physics problem

We asked the model to derive the effective potential, equilibria, stability, critical rotation rate, and small-oscillation frequencies for a bead on a vertically rotating circular hoop.

We ran three separate attempts with an 8,192-token output limit:

1. Normal prompt
2. A guarded prompt explicitly asking it to avoid repeated verification and produce the final answer
3. The guarded version with repetition_penalty: 1.05

All three ended with:

• finish_reason: "length"
• Exactly 8,192 completion tokens
• No public final answer (content: null)
• Roughly 20k–27k characters in the parsed reasoning trace

The reasoning was not an exact verbatim loop. It repeatedly revisited already-derived algebra, tried equivalent forms of the same frequency expression, questioned a correct simplification, re-derived it, and then began checking it again instead of committing to the answer.

For example, it correctly reached

ω² = Ω² - g²/(Ω²R²)

but repeatedly expanded, factored, re-expressed, and re-checked that same result. In another attempt it announced that it was ready to write the final solution, then restarted the frequency verification and consumed the remainder of the token budget.

Increasing the limit to 16,384 tokens allowed it to finish, but it still used 10,241 completion tokens for this relatively bounded problem.

We also reproduced the behavior on a much simpler ballistic-pendulum question. With the documented temperature: 0.7, top_p: 0.95, and a 4,096-token limit, the model repeatedly reconfirmed the already-correct 0.800 m result until the limit was exhausted, without emitting a public answer.

Other observations

Not every prompt loops:

• 17 × 24 with two requested verification methods completed in 1,333 tokens.
• An orbital-drag derivation completed correctly in 3,978 tokens.
• A Dijkstra counterexample completed correctly and concisely in 635 tokens.

However, the tendency becomes much more visible on problems with several interacting branches or derivation steps.

For the classic 12-coins problem:

• At an 8,192-token limit, the response ended with finish_reason: "length" and an incomplete solution.
• At a 16,384-token limit, it finally stopped after 13,313 tokens, but the resulting decision tree still omitted one physical state (coin 3 heavy) from a branch.
• This is concerning because the long internal verification did not translate into a fully verified final answer.

As a control, we sent the same five core prompts through the same client/request pipeline to another reasoning model on the same server. It completed all five within the 8,192-token limit, including two repetitions each of the rotating-hoop and 12-coins problems. This makes a generic client timeout or output-limit handling problem less likely.

DFlash, tested separately

The looping results above are target-only, so DFlash is not required to reproduce them.

Separately, we tested the available Laguna-S-2.1-DFlash-FP8 checkpoint with 7 speculative tokens. It accepted:

• 0 / 21,483 draft tokens at concurrency 1
• 0 / 35,805 draft tokens in a concurrency-5 run

Throughput fell from approximately 93.5 output tok/s target-only to 38.8 output tok/s, and a later 64K request caused an OOM. We understand that an updated DFlash checkpoint has been planned, so this may be a target/draft revision compatibility issue rather than the same reasoning bug.

Questions

1. Is the updated checkpoint expected to behave correctly on sm86 GPUs using the FP8 Marlin fallback, or was internal validation primarily performed on native-FP8 hardware?
2. Could FP8 KV cache interact with this behavior? Would BF16 KV cache be a useful diagnostic even though FP8 KV is documented for this model?
3. Is there any required generation or chat-template setting beyond temperature=0.7, top_p=0.95, enable_thinking=true, and preserve_thinking?
4. Do you have a known-good vLLM request payload for a bounded reasoning problem that we can reproduce byte-for-byte?
5. If useful, we can provide the complete prompts, JSON request payloads, parsed reasoning traces, and token-level response metadata.

The updated FP8 checkpoint is clearly capable, and several answers were strong when it finished. The remaining issue in our testing is mainly termination discipline: it often reaches the correct result, but continues semantically equivalent verification until the output budget is exhausted.

@DopplerFreq - I'm using vLLM nightly, https://wheels.vllm.ai/27ffbfde8decd340fa0144aea06061374c53e456/vllm-0.23.1rc1.dev1400%2Bg27ffbfde8-cp38-abi3-manylinux_2_28_x86_64.whl to be exact. I just reran hf download poolside/Laguna-S-2.1-FP8 to pick up any, and all new files.

The huggingface cli has pruning functionality, it removes unused revisions from the cache. Very useful for freeing up space, and ensuring only the latest versions are kept around - hf cache prune.

I'm running the latest version of Opencode, on 8 RTX 3090.

Thanks @flyingweasel , I used 0.7, as it is recommended for this model, but noted on your fix working for XS. I will definitely give that a shot if I see the looping again.

hey @bear11235 , huge thanks for the write-up, could you please share verbatim prompts that you used beyond the rotating-hoop physics one, if any?

Thanks for the updates! However, after pulling the latest model weights, DFlash speculative decoding is severely degraded on our setup (vLLM v0.25.1), whereas it previously achieved an ~80% acceptance rate before the weight replacement.

Here is our exact server launch script:

docker run --rm --name vllm-lagunas21 --gpus all --privileged --ipc=host --net=host \
  -e OMP_NUM_THREADS=8 \
  -e VLLM_MARLIN_USE_ATOMIC_ADD=1 \
  -e VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0 \
  -e VLLM_SERVER_DEV_MODE=0 \
  -v /data1/wanghao/Projects/Datas/models/laguna:/laguna:ro \
  local/lmcache-vllm:pr4136-8f11b93 /laguna/Laguna-S-2.1-FP8 \
  --kv-cache-dtype auto \
  --block-size 128 \
  --max-num-seqs 64 \
  --max-num-batched-tokens 8192 \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 1 \
  --max-model-len auto \
  --gpu-memory-utilization 0.90 \
  --tool-call-parser poolside_v1 \
  --enable-auto-tool-choice \
  --reasoning-parser poolside_v1 \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --speculative-config '{"model": "/laguna/Laguna-S-2.1-DFlash-FP8", "num_speculative_tokens": 7, "method": "dflash"}' \
  --enable-prefix-caching \
  --port 6789

Here are our detailed observations and test findings across different configurations:

1. Significant Drop in Acceptance Rate After Weight Update
   Before weight replacement: DFlash draft acceptance rate was healthy at ~80% (though occasionally we noticed garbled/corrupted text output or infinite looping during certain generation steps).

After weight replacement: Draft acceptance rate on FP8 model dropped completely to 0.0% (Mean acceptance length: 1.00, 0 tokens accepted out of thousands drafted across all requests).

2. DFlash Performance on BF16 Base Model
   We also tested DFlash on the unquantized BF16 base model. The draft acceptance rate is not strictly 0%, but it remains 0% for the vast majority of generation steps, rendering speculative decoding practically ineffective.

3. FP8 KV Cache Garbled Text Issue (on BF16 + DFlash Setup)
   When running the BF16 base model with its corresponding DFlash drafter, enabling --kv-cache-dtype fp8 leads to severely corrupted/garbled text generation. We had to fallback to --kv-cache-dtype auto (BF16 KV) to restore normal text generation, but DFlash still yields a near-zero acceptance rate.

4. SWA / RoPE Configuration Warning
   We noticed the following warning during engine initialization:

WARNING Laguna config has sliding_attention layers but neither swa_rope_parameters nor a nested per-layer-type rope_parameters — SWA layers will reuse the global rope.
Could this RoPE inconsistency cause feature misalignment between the target model's hidden states and the DFlash drafter?

5. Auxiliary Layer Indices & CoT Mode
   vLLM defaults to using auxiliary layers: (2, 11, 20, 30, 39, 48). Do these match the exact layer indices used when training Laguna-S-2.1-DFlash-FP8?

We are running with thinking mode enabled (--default-chat-template-kwargs '{"enable_thinking": true}'). Was the DFlash drafter model fine-tuned on thinking outputs ( CoT traces)?

Questions: Was the Laguna-S-2.1-DFlash draft checkpoint retrained or re-calibrated after the base model weights were updated? Are there any specific --speculative-config parameters or --moe-backend overrides required for DFlash to align correctly? Thanks!

@eliz-yoshida
Absolutely — here are the verbatim user-message strings from the same test set, excluding the rotating-hoop prompt already discussed.

The test prompts were prepared and the resulting reasoning traces and final answers were reviewed in detail using GPT-5.6 Sol as the evaluator. GPT-5.6 Sol was used for test design and qualitative analysis only; all Laguna outputs and token counts reported here came directly from our local Laguna-S-2.1-FP8 vLLM endpoint without modification.

For the baseline comparison, these were all single-turn requests with no prior conversation history. Unless otherwise noted, the request parameters were:

{
  "temperature": 0.7,
  "top_p": 0.95,
  "top_k": 20,
  "max_tokens": 8192,
  "chat_template_kwargs": {
    "enable_thinking": true
  }
}

1. Twelve coins

This was the other difficult prompt that displayed excessive reasoning:

There are 12 visually identical coins. Exactly one is counterfeit and may be either heavier or lighter; you do not know which. Using a balance scale exactly three times, give a complete deterministic strategy that always identifies the counterfeit coin and whether it is heavy or light. Present the strategy as an explicit decision tree covering every possible scale outcome, and verify that no case is missing. Do not cite the existence of a known solution; construct and check it.

At max_tokens: 8192, it ended with finish_reason: "length" and an incomplete solution.

I then repeated the exact same prompt with max_tokens: 16384. It stopped normally after 13,313 completion tokens, but the final decision tree still omitted the coin 3 heavy state from one branch.

I also ran it once with the following system message, while leaving the user prompt unchanged:

You are a rigorous problem-solving assistant. Thinking is enabled and must remain enabled. During reasoning, keep a compact checklist of unresolved requirements. Do not restart a completed derivation, do not rewrite algebraically equivalent expressions more than once, and do not claim verification without checking every case. Once the checklist is satisfied, stop reasoning and produce the final answer. Reserve enough of the token budget for a complete final answer.

That guarded version completed within the limit, although it remained quite long.

2. Orbital drag

This completed successfully and was used as a non-looping control:

A satellite of mass m is in an almost circular orbit of radius a around a body of mass M. It experiences a weak tangential drag force F_drag = -c v, where c is constant and the orbit remains quasi-circular as it slowly evolves. Explain rigorously why the satellite speeds up even though drag removes mechanical energy. Derive da/dt and dv/dt to leading adiabatic order, check units and signs, and state the assumptions under which the derivation is valid.

It completed normally in 3,978 completion tokens and the result was essentially correct.

3. Dijkstra counterexample

This also completed successfully and relatively concisely:

A programmer implements Dijkstra by marking a vertex visited when it is first inserted into the priority queue, and refuses all later relaxations into visited vertices. Determine whether this is correct for directed graphs with nonnegative edge weights. If incorrect, construct a smallest clear counterexample with explicit weights, trace the queue and distances step by step, give the corrected rule, and prove why the corrected algorithm is valid. Also distinguish this bug from merely having stale duplicate queue entries.

It completed normally in 635 completion tokens with a valid counterexample and correction.

4. Simple multiplication/control prompt

For a very small control case, I used:

Compute 17 × 24. Think through it, verify the result a second way, and then give a concise final answer.

This request used max_tokens: 2048. It completed normally in 1,333 completion tokens with the correct answer, although it was much more verbose than necessary.

5. Separate ballistic-pendulum reproduction

This was from an earlier reasoning probe, before the five-prompt comparison:

A 10.0 g bullet moving horizontally at 400 m/s embeds in a stationary 1.00 kg block suspended as a pendulum. Neglect drag and pivot friction and use g=9.8 m/s^2. Find the maximum vertical rise of the combined body. Clearly separate which conservation law applies during the collision and which applies during the swing. End with [FINAL].

For this request I used:

{
  "temperature": 0.7,
  "top_p": 0.95,
  "max_tokens": 4096,
  "chat_template_kwargs": {
    "enable_thinking": true
  }
}

The model correctly obtained approximately 0.800 m during reasoning, but continued rechecking the same arithmetic and significant figures until all 4,096 completion tokens were consumed. It ended with finish_reason: "length" and did not emit the requested [FINAL] answer.

So, in our small sample, the behavior was not universal. The Dijkstra and orbital-drag prompts terminated normally. The clearest failures were the rotating-hoop problem, the ballistic pendulum, and the unguarded twelve-coins problem. The common pattern seemed to be repeatedly reopening already-resolved intermediate checks rather than exact verbatim token repetition.

Happy to also share the raw response JSON or the full parsed reasoning traces for any of these cases if that would help.

Huge thanks, looking into your examples

I'm also still getting looping in some situations, generally when it's working on complex code or logic problems without a clear solution. The looped context can be quite large, being many paragraphs, and sometimes it's a bit different each time. It usually begins with it saying "actually", suggesting a solution, then looping on that.

My fingers are crossed that with a bit of fine tuning this will be solid, because when it is working, it's working very well.

I have yet to see looping when it's not thinking; even if it's enabled, if it's not in a thinking block, it seems fine.

I've been using the model heavily today in Opencode, and unfortunately, still getting a lot of looping in reasoning. Nothing changed from my previous config.

On a more positive note; when not looping, the output is excellent!