ngquocvinh/AliceAI-T5-35B-A0.6B-GGUF

AliceAI-T5-35B-A0.6B GGUF

Community GGUF quantizations of yandex/AliceAI-T5-35B-A0.6B.

☕ If this GGUF made your day easier, a coffee would make mine.
Send a coffee ☕
I build and test these releases myself. Your coffee helps keep me going.
Thank you for supporting this work.

Compatibility warning: AliceAI-T5 is a base encoder-decoder model, not an instruction/chat model. It requires an AliceAI-aware llama.cpp runtime with encoder, cross-attention, and decoder support. Stock llama-cli, Ollama, vLLM, and LM Studio runtimes have not been validated for this architecture. The runtime used for this release is derived from the XHToken/llama.cpp fork.

About AliceAI-T5-35B-A0.6B

AliceAI-T5-35B-A0.6B is a Yandex encoder-decoder language model with sparse mixture-of-experts layers. The upstream model card describes approximately 34.35B unique parameters, 512 experts per MoE layer with top-8 routing, a 1,536-wide hidden state, 16 encoder layers, 12 decoder layers, shared encoder/decoder embeddings, and RoPE with YaRN. The upstream card advertises a 128K-token context window; this GGUF release has not been validated at that maximum context length.

See the official upstream model card for the original model description, benchmark claims, dependencies, and Transformers usage. This is a custom encoder-decoder GGUF architecture, not a decoder-only model. The runtime must execute the encoder, cross-attention, and decoder generation path.

No training or fine-tuning was performed. Every published artifact was converted directly from the locked upstream BF16 snapshot; no quantized file was used as the source of another quantized file.

Download

Install the Hugging Face CLI and download a compact validated candidate:

python -m pip install -U huggingface_hub

mkdir -p aliceai-t5
cd aliceai-t5

hf download ngquocvinh/AliceAI-T5-35B-A0.6B-GGUF \
  AliceAI-T5-35B-A0.6B-IQ4_XS.gguf \
  aliceai-chat.jinja \
  SHA256SUMS.txt \
  --local-dir .

sha256sum -c SHA256SUMS.txt --ignore-missing

IQ4_XS is the compact release candidate used in the GPU profile for this project. Its file size is 18.506586 GB; runtime memory also depends on context, batch size, backend, and offload settings.

Build the AliceAI-aware runtime

Stock llama.cpp binaries are not sufficient for this custom encoder-decoder architecture. The runtime patch used for this release can be applied to the matching XHToken/llama.cpp source revision:

git clone https://github.com/XHToken/llama.cpp.git
cd llama.cpp
git checkout 4a3635c32fc9f044c2bde9ebeabf50c7e1ec5991

mkdir -p /tmp/aliceai-runtime
hf download ngquocvinh/AliceAI-T5-35B-A0.6B-GGUF \
  reproducibility/aliceai-runtime.patch \
  --local-dir /tmp/aliceai-runtime
git apply /tmp/aliceai-runtime/reproducibility/aliceai-runtime.patch

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j 8 \
  --target llama-server llama-simple

For a CPU-only build, use -DGGML_CUDA=OFF. The patch is also available at reproducibility/aliceai-runtime.patch.

Published files

File	Size (GB)	Size (bytes)	Intended use
AliceAI-T5-35B-A0.6B-BF16.gguf	69.177	69,176,812,608	BF16 reference
AliceAI-T5-35B-A0.6B-Q8_0.gguf	36.796	36,795,843,648	Highest-fidelity quantization tier
AliceAI-T5-35B-A0.6B-Q6_K.gguf	28.431	28,430,760,320	High-quality lower-memory tier
AliceAI-T5-35B-A0.6B-Q5_K_M.gguf	24.624	24,623,656,832	Balanced quality/memory tier
AliceAI-T5-35B-A0.6B-Q4_K_M.gguf	21.041	21,040,500,608	General balanced tier
AliceAI-T5-35B-A0.6B-Q3_K_M.gguf	16.575	16,575,128,960	Smaller-memory tier
AliceAI-T5-35B-A0.6B-Q2_K.gguf	12.684	12,683,637,632	Experimental low-memory tier
AliceAI-T5-35B-A0.6B-IQ4_XS.gguf	18.507	18,506,585,984	Compact experimental tier

Q8_0 through Q3_K_M and IQ4_XS passed the encoder-decoder load and deterministic generation smoke test used for this release. Q2_K also loads and generates, but its factual smoke result was not semantically correct; treat it as a runtime/low-memory experiment, not as a quality recommendation. These smoke checks are not task benchmarks.

Three additional local candidates were not published: IQ2_XS produced no decoded token, IQ3_M produced only blank or control-token output, and IQ1_M entered a repeated <SPAN#511>Post loop. All three were excluded by the functional smoke gate.

Quick start

Use an AliceAI-aware llama.cpp build containing the llama-simple encoder- decoder test harness from the runtime used for this release. The ordinary upstream llama-cli path is not sufficient for this architecture:

./build/bin/llama-simple \
  -m ./AliceAI-T5-35B-A0.6B-Q4_K_M.gguf \
  -ngl 0 -n 1 \
  '[_S_]Question: What is the capital of France? Answer:<SPAN#0>'

The tested protocol prepends the upstream [_S_] mode token, appends <SPAN#0> to the encoder input, and seeds the decoder with [decoder_bos, <SPAN#0>]; the harness handles these encoder-decoder details. The Q4_K_M smoke returns Paris in the tested CPU configuration. A generic decoder-only llama-cli invocation is not sufficient for this architecture.

To expose an OpenAI-compatible endpoint with the same runtime:

CUDA_VISIBLE_DEVICES=0 ../llama.cpp/build/bin/llama-server \
  -m ./AliceAI-T5-35B-A0.6B-IQ4_XS.gguf \
  --alias aliceai-t5 \
  --host 127.0.0.1 --port 8001 \
  --n-gpu-layers 99 \
  --ctx-size 8192 --parallel 1 \
  --batch-size 8192 --ubatch-size 2048 \
  --flash-attn on --jinja \
  --chat-template-file ./aliceai-chat.jinja \
  --temp 0

Test it with:

curl http://127.0.0.1:8001/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"aliceai-t5","messages":[{"role":"user","content":"What is the capital of France?"}],"temperature":0,"max_tokens":64}'

Android / CPU notes

For Termux or another CPU-only device, use the AliceAI-aware binary and the downloaded template explicitly:

./build/bin/llama-server \
  -m ./AliceAI-T5-35B-A0.6B-IQ4_XS.gguf \
  --host 0.0.0.0 --port 5001 \
  --jinja \
  --chat-template-file ./aliceai-chat.jinja \
  --n-gpu-layers 0 \
  --ctx-size 2048 --parallel 1 \
  --batch-size 2048 --ubatch-size 2048 \
  --threads 6 --threads-batch 8 \
  --flash-attn off \
  --temp 0

Do not omit --chat-template-file: without it, llama.cpp may select a decoder-only ChatML template and expose tokens such as <|im_start|> or <|im_end|>. An encoder input of 1,073 tokens requires an encoder --ubatch-size greater than that input; if the log still reports an encoder batch size of 512 after setting --ubatch-size 2048, the binary is not the patched runtime described above and must be rebuilt from the pinned revision and reproducibility/aliceai-runtime.patch.

On a memory-constrained phone, remove --no-mmap and start a new, shorter conversation before lowering the batch sizes. Reducing --ubatch-size below the input length cannot make the encoder request succeed. Longer contexts can also reduce CPU speed substantially because this model runs an encoder plus a decoder and needs significant memory bandwidth.

For the standalone CPU smoke test, pass -ngl 0; the default GPU-layer setting is not appropriate for a CPU-only Termux build:

./build/bin/llama-simple \
  -m ./AliceAI-T5-35B-A0.6B-IQ4_XS.gguf \
  -ngl 0 -n 16 \
  '[_S_]Question: What is the capital of France? Answer:<SPAN#0>'

The expected text-to-text prompt format is:

Question: What is the capital of France?
Answer:

Do not add a ChatML system prompt or expect normal multi-turn assistant behavior from this base model. The bundled Jinja file is only a single-turn QA adapter: it intentionally uses the latest user message and does not preserve earlier chat history. For factual QA, use temperature=0; the default sampling temperature can produce unstable or empty answers from this base model.

Known limitations

• This release is not instruction-tuned or chat-tuned; task-specific behavior can differ substantially from an assistant model.
• The upstream 128K context claim has not been validated for these GGUF files.
• GGUF file size is not a minimum VRAM requirement; leave headroom for runtime buffers, context, and the selected backend.
• Some runtimes may display UTF-8 byte-fallback pieces as <0xHH> tokens until their output detokenizer reconstructs the original bytes.
• The table above describes the validated release subset. Other quantization files in the repository should be treated as experimental unless they are covered by the checksum and validation records.

Reproducibility and validation

The input is pinned to upstream revision a0d71f58c40d6affe461797b30b35ff47f52a5f2 and the GGUF files were converted from the generated BF16 reference. The release used a model-specific AliceAI calibration set plus Wikitext training text to build a combined imatrix. The exact source hashes, imatrix hashes, converter/runtime state, quantization commands, file sizes, and smoke-test results are recorded in reproducibility/manifest.md.

Verify downloads with SHA256SUMS.txt. Raw converter, imatrix, quantization, and smoke-test logs remain local under reports/ and are not part of the public package.

The GGUF files require the AliceAI-aware runtime described above; the runtime revision and model conversion details are recorded in reproducibility/manifest.md.

No hold-out BF16 fidelity table is included yet. Mean KLD, Top-1 agreement, delta PPL, and RMS delta-p will be added only after a fixed encoder-decoder hold-out evaluator measures every published file against the same BF16 reference.

License and attribution

The upstream model is released under the Apache License 2.0; see the included LICENSE. These are community GGUF quantizations, not an official Yandex release or endorsement.