Instructions to use choyiny/yev2-35b-a3b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use choyiny/yev2-35b-a3b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="choyiny/yev2-35b-a3b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("choyiny/yev2-35b-a3b")
model = AutoModelForCausalLM.from_pretrained("choyiny/yev2-35b-a3b", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

PEFT
How to use choyiny/yev2-35b-a3b with PEFT:
```
Task type is invalid.
```
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use choyiny/yev2-35b-a3b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "choyiny/yev2-35b-a3b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "choyiny/yev2-35b-a3b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/choyiny/yev2-35b-a3b

SGLang

How to use choyiny/yev2-35b-a3b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "choyiny/yev2-35b-a3b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "choyiny/yev2-35b-a3b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "choyiny/yev2-35b-a3b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "choyiny/yev2-35b-a3b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use choyiny/yev2-35b-a3b with Docker Model Runner:
```
docker model run hf.co/choyiny/yev2-35b-a3b
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

yev2-35b-a3b

An open System One decision model on Qwen/Qwen3.6-35B-A3B (Apache-2.0), a mixture-of-experts model with about 35B parameters, of which about 3B are active per token. Give it a state, a question and 2–10 options; it returns a calibrated probability per option from one forward pass, read from the logits of the option letters A–J. Merged bf16 weights are at the repo root; the LoRA adapter is in adapter/.

yev2 is yev1's training recipe and data on a larger base: same training rows, same hyper-parameters, only the base model changed. For a 48 GB GPU use the FP8 checkpoint, choyiny/yev2-35b-a3b-fp8.

Hardware

Checkpoint	Weights	Fits on
`yev2-35b-a3b` (bf16, this repo)	65 GB	one 80 GB GPU (A100 80 GB, H100, RTX PRO 6000)
`yev2-35b-a3b-fp8`	34 GB	one 48 GB GPU (L40S, RTX 6000 Ada)

All 35B parameters must be in GPU memory; compute per token is that of a ~3B model.

How to serve

vLLM (recommended). Start the engine, then put yev serve in front of it for the prompt format, letter readout and calibration:

hf download choyiny/yev2-35b-a3b --local-dir yev2-35b-a3b

vllm serve ./yev2-35b-a3b --host 127.0.0.1 --port 8100 --served-model-name yev --dtype bfloat16 \
  --max-model-len 16385 --enable-prefix-caching --generation-config vllm \
  --logprobs-mode processed_logprobs --max-logprobs 26 --max-num-seqs 64 \
  --gpu-memory-utilization 0.85 --trust-remote-code

pip install torch 'yev[serve,train] @ git+https://github.com/xyspacedev/yev'
yev serve --backend vllm --model yev2-35b-a3b --calibration yev2-35b-a3b/calibration.json --model-name yev2-35b-a3b

--max-num-seqs 64 is required: the model's linear-attention layers keep one recurrent-state block per sequence, and vLLM's default of 1,024 sequences does not fit.

transformers: yev serve --model yev2-35b-a3b --calibration yev2-35b-a3b/calibration.json (no sidecar), or run python inference_example.py for a self-contained script (--adapter loads the base + LoRA).

curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
  "state": "I was charged twice. Please help.",
  "questions": {"billing": {"instructions": "Is this message about billing?", "type": "noul"}}}'

yev serve also exposes an OpenAI-compatible /v1/chat/completions.

Benchmarks

Benchmark	yev1-4b	yev2-35b-a3b	yev2-35b-a3b-fp8
DecideBench v2.0 test (450), accuracy	79.3	88.0	87.6
DecideBench v2.0 test, pair accuracy	62.7	78.2	77.3
DecideBench v2.0 test, ECE	0.078	0.058	0.059
JevBench public (231), accuracy	82.7	90.5	89.2
JevBench public, hard tier (111)	67.6	81.1	78.4

DecideBench v2.0 test was never inspected or used for training or model selection. The 95 % interval on yev2's DecideBench v2 accuracy is 84.9–90.9. On the DecideBench v2 leaderboard it places 5th of 27 decision models, and 2nd among those that run on your own hardware. Its weakest family is release-gating decisions (52 %), which every decision model on the leaderboard finds hard.

Development set (3,963 rows, held out from training): accuracy 97.0 (yev1: 95.2), group accuracy 93.4 (89.1), ECE 0.017 (0.021). With options reordered, the answer stays the same on 98.9 % of rows (yev1: 98.4 %).

Speed and cost

DecideBench v2, one NVIDIA RTX PRO 6000 Blackwell (96 GB), vLLM 0.31.0, yev serve, 4 requests in flight.

	p50	p95	Throughput	Cost per 1M decisions
yev2-35b-a3b (bf16)	0.38 s	0.72 s	9.7 req/s	$95
yev2-35b-a3b-fp8 (45 GiB cap)	0.30 s	0.54 s	11.3 req/s	$78
yev1-4b (bf16)	0.28 s	0.60 s		$71

Cost is wall-clock GPU time at $3.36 per hour (AWS g7e.2xlarge on-demand) divided by decisions; DecideBench prompts average about 3,300 tokens.

Training

LoRA r 64, α 128, dropout 0.05 on the attention, linear-attention and shared-expert projections; routed experts frozen. 1 epoch, learning rate 1e-4, 80,730 rows (101.5M tokens), 3,955 steps, 7.9 h on one RTX PRO 6000. Per-type temperatures in calibration.json were fitted on 3,368 held-out rows. Python 3.12, torch 2.14.1+cu130, transformers 5.22, peft 0.21.2, flash-linear-attention 0.6.0, causal-conv1d 1.7.0. Training code: xyspacedev/yev, configs/yev2/yev2-qwen36-35b-a3b.json.

Licence

Weights: Apache-2.0. Training-data credits are in NOTICE and ATTRIBUTION.json. Recipe after Together's "How to train your own Jev".

Downloads last month: 162

Safetensors

Model size

35B params

Tensor type

BF16

Model tree for choyiny/yev2-35b-a3b

Base model

Qwen/Qwen3.6-35B-A3B

Adapter

(372)

this model

Quantizations

1 model

Evaluation results

Accuracy on DecideBench v2.0 test (450 items, with worked examples)
test set self-reported

88.000
Pair accuracy on DecideBench v2.0 test (450 items, with worked examples)
test set self-reported

78.200
ECE (15 bins) on DecideBench v2.0 test (450 items, with worked examples)
test set self-reported

0.058