katya228/Apertus-v1.5-70B-text-GGUF

Apertus v1.5 70B — text-only GGUF (TEST quants)

GGUF conversions of swiss-ai/Apertus-v1.5-70B, made because there were no GGUF quants of v1.5 anywhere at the time of upload — the only Apertus GGUFs on the Hub are for the older 2509 (v1.0) release, and people who just want to try the model were stuck setting up the SwissAI vLLM fork.

⚠️ Read this before downloading

1. These are TEST quants. They were made to check whether the model answers at all, not as a polished release. No imatrix, no perplexity measurements, no benchmark runs. See What was actually tested.
2. This is text-only. The vision and audio towers of v1.5 were stripped, not converted. You cannot feed images or audio to these files. See How this was made.
3. Two launch flags are mandatory — --override-kv tokenizer.ggml.eos_token_id=int:68 (or the model never stops generating) and -sp (or reasoning appears to be missing). Neither is a quirk of these quants; both follow from the model itself. See Required flags for copy-paste commands.

If someone produces proper imatrix quants of this model, prefer theirs over mine.

Files

File	Size	Notes
apertus-70b-Q4_K_M.gguf	43.7 GB (40.7 GiB)	The one that was actually run and tested
apertus-70b-q8_0.gguf	75.0 GB (69.9 GiB)	Converted directly from bf16; see testing notes

Both were produced from the original bf16 weights, Q4_K_M by quantizing the q8_0 file.

How this was made

Apertus v1.5 is Apertus1p5ForConditionalGeneration — a multimodal model: a text decoder plus a VQ-VAE image tokenizer and a WavTokenizer audio tokenizer. llama.cpp does not support that architecture.

What it does support is LLM_ARCH_APERTUS, added for Apertus 1.0. And the v1.5 decoder is architecturally the same as 1.0 — non-gated MLP with xIELU, mandatory q_norm/k_norm. So instead of teaching llama.cpp a new architecture, the model was reshaped into something the existing converter already understands:

1. Both towers dropped — every tensor under model.vision_tokenizer. and model.audio_tokenizer. is excluded from the weight map. This is what makes the result text-only, and it is irreversible from these files.
2. Vocabulary truncated from vocab_size 266752 down to output_vocab_size 131072. The tail of the embedding matrix holds image and audio codebook entries; the text model never emits them, and lm_head was already 131072 rows in the original. Multimodal entries were also removed from added_tokens in the tokenizer.
3. Decoder prefix rewritten from model.language_model. to model., which is where the Apertus 1.0 converter expects it.

The result is an ordinary apertus-architecture GGUF. llama.cpp was not patched by a single byte — the conversion class is registered at runtime and then stock convert_hf_to_gguf.main() does the work.

Both scripts are included in this repo:

• prep_apertus_text.py — builds a text-only HF folder. Weights are symlinked, not copied, so it costs ~20 MB of disk and a few seconds rather than a second copy of a 70B model.
• convert_apertus15.py — registers Apertus1p5ForConditionalGeneration and calls the stock converter.

python3 prep_apertus_text.py ./Apertus-v1.5-70B ./Apertus-70B-text
LLAMA_CPP_DIR=/path/to/llama.cpp python3 convert_apertus15.py ./Apertus-70B-text \
    --outfile apertus-70b-q8_0.gguf --outtype q8_0

Requirements

No special build is needed. These load in stock upstream llama.cpp — the GGUF declares general.architecture = apertus, which has been supported since PR #15852, released in b6671 (2 Oct 2025). Anything newer works; these files were run on a b10069-era build.

There is no vLLM path for these files — vLLM does not read GGUF for this architecture. If you want v1.5 in vLLM, you need the SwissAI fork and the original safetensors.

Required flags — do not skip these

Two flags are mandatory. Launch without them and the model looks broken when it is not:

Flag	Without it
--override-kv tokenizer.ggml.eos_token_id=int:68	The model never stops. It runs past the end of its reply, answers its own question, starts a new turn, and only halts when it hits your token limit.
-sp (--special)	Reasoning appears to be missing. The model still thinks correctly, but the server strips the markers out of content, so you never see the block — and your frontend cannot split thinking from the answer.

Why the EOS one is needed: the original config declares three EOS ids, eos_token_id: [2, 68, 72]. GGUF only stores one, and the converter writes 2 (</s>) — but the token that actually ends an assistant turn is 68 = <|assistant_end|>. So the correct stop token is present in the file, it is simply not the one marked as EOS. (72 is <|tools_suffix|>, for tool-calling mode.)

-sp is available on both llama-cli and llama-server.

Copy-paste: server

llama-server \
  -m apertus-70b-Q4_K_M.gguf \
  --jinja -sp \
  --override-kv tokenizer.ggml.eos_token_id=int:68 \
  -ngl 999 --flash-attn on \
  -c 65536 -b 2048 -ub 512 \
  --cache-type-k q8_0 --cache-type-v q8_0 --cache-reuse 256 \
  --context-shift --parallel 1 \
  --host 127.0.0.1 --port 8084

Copy-paste: quick CLI check

llama-cli \
  -m apertus-70b-Q4_K_M.gguf \
  -sp \
  --override-kv tokenizer.ggml.eos_token_id=int:68 \
  -ngl 999 -c 4096 \
  -p "<|system_start|>You are a helpful assistant.<|system_end|><|user_start|>What is the capital of Switzerland?<|user_end|><|assistant_start|>"

That prompt is the literal format the model expects — <|system_start|> … <|system_end|>, <|user_start|> … <|user_end|>, then <|assistant_start|> to hand it the turn. It is exactly the call used to verify these files, and it answers The capital of Switzerland is Bern.

To turn on reasoning, add the developer block after the system part — see Deliberation mode below.

Deliberation (reasoning) mode

Apertus v1.5 reasons between its own dedicated tokens, not <think>:

id	token	meaning
32	<|inner_prefix|>	start of reasoning
33	<|inner_suffix|>	end of reasoning
68	<|assistant_end|>	end of the whole reply

Reasoning is enabled through a developer-role block, not a flag:

<|developer_start|>Deliberation: enabled
Tool Capabilities: disabled<|developer_end|>

Quantization did not damage this. On a raw-token check of Q4_K_M, the model emitted token 32 at position 0, token 33 at position 131, and token 68 at position 262 — reasoning opened, closed, and the turn ended, exactly where they belong.

But two things get in the way of seeing it:

• llama-server strips control tokens from content. Add -sp to get them through.
• llama.cpp's reasoning parser knows <think> and <|channel|>, not the Apertus markers, so /v1/chat/completions will not populate reasoning_content. Parse <|inner_prefix|> / <|inner_suffix|> on your side — or let your frontend do it (see below).

SillyTavern

Ready-made master-import presets are in this repo — import via Settings → Master Import:

File	System prompt
sillytavern/Apertus-v1.5-EN.json	English
sillytavern/Apertus-v1.5-RU.json	Russian

The token layout is identical in both — only the system prompt language differs, so pick by the language you want the model instructed in, not by the language you intend to chat in (it follows the user either way). The system prompts are deliberately plain and general-purpose: swap in your own, the point of the presets is the plumbing below. No sampler settings are included, because none were tuned for this model — use your own.

If you would rather set it up by hand, this is the same working configuration:

Instruct Template

System Sequence         <|system_start|>
System Suffix           <|system_end|>
Input Sequence          <|user_start|>
Input Suffix            <|user_end|>
Output Sequence         <|assistant_start|>
Output Suffix           <|assistant_end|>
Stop Sequence           <|assistant_end|>
Story String Prefix     <|system_start|>
Story String Suffix     <|system_end|><|developer_start|>Deliberation: enabled
                        Tool Capabilities: disabled<|developer_end|>

Note where the deliberation block goes: on the Story String suffix, not the System suffix. Put it on System Suffix and it gets repeated after every system message instead of being stated once. Also enable System same as user, Sequences as stop strings, and set Names behavior to force; leave wrap off.

Reasoning settings — SillyTavern will then fold the thinking into a collapsible block by itself, no manual parsing:

Prefix       <|inner_prefix|>
Suffix       <|inner_suffix|>
Separator    \n\n

What was actually tested

Honest scope, on 2×RTX 3090 + Tesla V100 (64 GB VRAM total):

Q4_K_M

• Loads and generates coherent multi-turn text (tested in Russian), ~17 tok/s generation, ~81 tok/s prompt at 65536 context with q8_0 KV cache across three GPUs.
• Deliberation markers verified at the raw token-id level, as described above.
• Stops correctly on <|assistant_end|> once the EOS override is applied.

q8_0

• Read end-to-end during quantization to Q4_K_M, so the file is structurally sound and every tensor is readable.
• Load-tested on stock upstream llama.cpp b8861, CPU-only (-ngl 0), with no patches of any kind — loads and answers correctly ("What is the capital of Switzerland?" → "The capital of Switzerland is Bern."). This is the run that confirms no custom build is needed.

Not tested by anyone yet: long-context behaviour anywhere near the 262144 the GGUF advertises, tool calling, and any quality benchmark whatsoever. Quality relative to the original bf16 weights is unmeasured — no perplexity, no KL divergence. Treat both files as "it answers, and the answers look sane", nothing stronger.

Credits

Model by SwissAI (ETH Zurich / EPFL / CSCS), Apache 2.0. All the interesting work is theirs; this repo is just a format conversion.