Instructions to use choyiny/yev2-35b-a3b-fp8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use choyiny/yev2-35b-a3b-fp8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="choyiny/yev2-35b-a3b-fp8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("choyiny/yev2-35b-a3b-fp8")
model = AutoModelForCausalLM.from_pretrained("choyiny/yev2-35b-a3b-fp8", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use choyiny/yev2-35b-a3b-fp8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "choyiny/yev2-35b-a3b-fp8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "choyiny/yev2-35b-a3b-fp8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/choyiny/yev2-35b-a3b-fp8

SGLang

How to use choyiny/yev2-35b-a3b-fp8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "choyiny/yev2-35b-a3b-fp8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "choyiny/yev2-35b-a3b-fp8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "choyiny/yev2-35b-a3b-fp8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "choyiny/yev2-35b-a3b-fp8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use choyiny/yev2-35b-a3b-fp8 with Docker Model Runner:
```
docker model run hf.co/choyiny/yev2-35b-a3b-fp8
```

yev2-35b-a3b-fp8

FP8 weights of choyiny/yev2-35b-a3b, an open System One decision model on Qwen/Qwen3.6-35B-A3B. 34 GB instead of 65 GB, so it runs on one 48 GB GPU (L40S, RTX 6000 Ada), and it is faster than bf16 on the same GPU. See the main repo for what the model does, its training and its licence.

Quantized with llm-compressor, FP8_DYNAMIC (per-channel FP8 weights, per-token dynamic FP8 activations, no calibration data). The MoE router, the shared-expert gate, the linear-attention in_proj_a/in_proj_b projections and lm_head stay in bf16 (recipe.yaml). calibration.json is the main model's, unchanged.

How to serve

hf download choyiny/yev2-35b-a3b-fp8 --local-dir yev2-35b-a3b-fp8

vllm serve ./yev2-35b-a3b-fp8 --host 127.0.0.1 --port 8100 --served-model-name yev \
  --max-model-len 16385 --enable-prefix-caching --generation-config vllm \
  --logprobs-mode processed_logprobs --max-logprobs 26 --max-num-seqs 64 \
  --gpu-memory-utilization 0.90 --trust-remote-code

pip install torch 'yev[serve,train] @ git+https://github.com/xyspacedev/yev'
yev serve --backend vllm --model yev2-35b-a3b-fp8 --calibration yev2-35b-a3b-fp8/calibration.json \
  --model-name yev2-35b-a3b-fp8

--max-num-seqs 64 is required (vLLM's default of 1,024 sequences does not fit the linear-attention state). FP8 needs a GPU with FP8 support (Ada, Hopper or Blackwell). Loading with transformers needs compressed-tensors>=0.15.

Benchmarks

Benchmark	yev2-35b-a3b (bf16)	yev2-35b-a3b-fp8
DecideBench v2.0 test (450), accuracy	88.0	87.6
DecideBench v2.0 test, pair accuracy	78.2	77.3
DecideBench v2.0 test, ECE	0.058	0.059
JevBench public (231), accuracy	90.5	89.2

DecideBench v2.0 test was never inspected or used for training or model selection. The 95 % interval on the FP8 accuracy is 84.4–90.7.

Speed and cost

DecideBench v2 on one NVIDIA RTX PRO 6000 Blackwell with vLLM capped at 45 GiB (--gpu-memory-utilization 0.46, to stand in for a 48 GB GPU), vLLM 0.31.0, yev serve, 4 requests in flight.

	p50	p95	Throughput	Cost per 1M decisions
yev2-35b-a3b-fp8	0.30 s	0.54 s	11.3 req/s	$78
yev2-35b-a3b (bf16, 83 GiB)	0.38 s	0.72 s	9.7 req/s	$95

Cost is wall-clock GPU time at $3.36 per hour (AWS g7e.2xlarge on-demand) divided by decisions.

Downloads last month: 41

Safetensors

Model size

35B params

Tensor type

BF16

F8_E4M3

Model tree for choyiny/yev2-35b-a3b-fp8

Base model

Qwen/Qwen3.6-35B-A3B

Adapter

choyiny/yev2-35b-a3b

Quantized

(1)

this model

Evaluation results

Accuracy on DecideBench v2.0 test (450 items, with worked examples)
test set self-reported

87.600
Pair accuracy on DecideBench v2.0 test (450 items, with worked examples)
test set self-reported

77.300
ECE (15 bins) on DecideBench v2.0 test (450 items, with worked examples)
test set self-reported

0.059