Instructions to use eousphoros/persona_lambda_27b_v10 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use eousphoros/persona_lambda_27b_v10 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="eousphoros/persona_lambda_27b_v10")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("eousphoros/persona_lambda_27b_v10")
model = AutoModelForMultimodalLM.from_pretrained("eousphoros/persona_lambda_27b_v10", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use eousphoros/persona_lambda_27b_v10 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "eousphoros/persona_lambda_27b_v10"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "eousphoros/persona_lambda_27b_v10",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/eousphoros/persona_lambda_27b_v10

SGLang

How to use eousphoros/persona_lambda_27b_v10 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "eousphoros/persona_lambda_27b_v10" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "eousphoros/persona_lambda_27b_v10",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "eousphoros/persona_lambda_27b_v10" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "eousphoros/persona_lambda_27b_v10",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use eousphoros/persona_lambda_27b_v10 with Docker Model Runner:
```
docker model run hf.co/eousphoros/persona_lambda_27b_v10
```

eousphoros commited on 5 days ago

Commit

f48e413

verified ·

1 Parent(s): c03d494

Replace the MTP head with one trained on draft acceptance (total variation loss)

Browse files

Files changed (3) hide show

README.md +6 -6
model-part-00053.safetensors +1 -1
model-part-00054.safetensors +1 -1

README.md CHANGED Viewed

@@ -22,7 +22,7 @@ Give it a system prompt describing a persona and it will take that persona on.
 - **Retrained from the base model** with v8's recipe, on an updated persona dataset plus v8's coding, math, multimodal and paper-reading records.
 - **v8's early-stop quirk is fixed.** v8 now and then ended a reply one line before its final answer. A small weight edit (rank 4, in the MLP output projections of layers 61-63) removes the habit: on GPQA, replies without a final `Answer:` line went from 7 to 0, with scores unchanged.
-- **Recalibrated MTP head.** The draft head was trained to match this model's own next-token distributions on both of its draft steps (KL divergence, backbone frozen), on about 70M tokens of training data and the model's own replies to 2,200 prompts.
 Known quirks: it thinks longer than v8 on hard science questions (median about 9.6k vs 7.3k tokens on GPQA), and about 1 in 400 bench replies fell into a repetition loop that only the `max_tokens` cap ended.
@@ -40,16 +40,16 @@ Measured with our own harness: vLLM 0.32, thinking on at reasoning effort `xhigh
 | XSTest safe prompts fully answered (250) | 96.8% |
 | XSTest unsafe prompts fully refused (200) | 76.0% (partly refused 8.0%) |
-Against v8 on the same prompts: GPQA 85.5 vs 85.3 (505 replies) and LiveCodeBench 85.1 vs 86.1 (101 problems); neither difference is significant. Of the 1,717 bench replies, 14 hit the 128,000-token cap.
 ## Speculative decoding
 Tokens generated per target-model step with two MTP draft tokens, measured in vLLM 0.32 on 200 held-out prompts (about 2.1M generated tokens per configuration):
-| Draft sampling | This model's MTP head | Base model's MTP head |
-|---|---|---|
-| greedy (vLLM's default) | 2.14 | 2.13 |
-| probabilistic | 2.43 | 2.46 |
 At the default temperature, probabilistic drafting accepts more drafts: add `"draft_sample_method": "probabilistic"` to `--speculative-config`, for example `--speculative-config '{"method": "mtp", "num_speculative_tokens": 2, "draft_sample_method": "probabilistic"}'`. Either way, verification keeps the output distribution exactly the model's own; only speed changes.

 - **Retrained from the base model** with v8's recipe, on an updated persona dataset plus v8's coding, math, multimodal and paper-reading records.
 - **v8's early-stop quirk is fixed.** v8 now and then ended a reply one line before its final answer. A small weight edit (rank 4, in the MLP output projections of layers 61-63) removes the habit: on GPQA, replies without a final `Answer:` line went from 7 to 0, with scores unchanged.
+- **Recalibrated MTP head.** The draft head was trained directly on how often vLLM's probabilistic drafting accepts its drafts: the loss is the total variation distance between its draft distribution and the distribution this model samples from (temperature 1, top-k 20, top-p 0.95), on both draft steps, with the backbone frozen. It started from the base model's head and trained on about 35M tokens of training records and the model's own replies. An earlier calibration that used KL divergence drafted slightly worse than the base model's head; it is kept at the `mtp-run1` revision.
 Known quirks: it thinks longer than v8 on hard science questions (median about 9.6k vs 7.3k tokens on GPQA), and about 1 in 400 bench replies fell into a repetition loop that only the `max_tokens` cap ended.
 | XSTest safe prompts fully answered (250) | 96.8% |
 | XSTest unsafe prompts fully refused (200) | 76.0% (partly refused 8.0%) |
+Against v8 on the same prompts: GPQA 85.5 vs 85.3 (505 replies) and LiveCodeBench 85.1 vs 86.1 (101 problems); neither difference is significant. Of the 1,717 bench replies, 14 hit the 128,000-token cap. Speculative decoding never changes the output distribution, so these scores hold whichever MTP head is used.
 ## Speculative decoding
 Tokens generated per target-model step with two MTP draft tokens, measured in vLLM 0.32 on 200 held-out prompts (about 2.1M generated tokens per configuration):
+| Draft sampling | This model's MTP head | Base model's MTP head | `mtp-run1` head |
+|---|---|---|---|
+| greedy (vLLM's default) | 2.15 | 2.13 | 2.14 |
+| probabilistic | 2.48 | 2.46 | 2.43 |
 At the default temperature, probabilistic drafting accepts more drafts: add `"draft_sample_method": "probabilistic"` to `--speculative-config`, for example `--speculative-config '{"method": "mtp", "num_speculative_tokens": 2, "draft_sample_method": "probabilistic"}'`. Either way, verification keeps the output distribution exactly the model's own; only speed changes.

model-part-00053.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3ffa27ee112c304777f21553a6bce846e568e87c3d50c2c89fd0f71332317bfd
 size 1027627592

 version https://git-lfs.github.com/spec/v1
+oid sha256:50d5e228574f89fdda8372120c2d9a664563467f5a64254e6d6841c23569875d
 size 1027627592

model-part-00054.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:de768039c4542e0ec80f62d698176eae5218d1d71166a2dfc03e1b4cea318ab0
 size 388016320

 version https://git-lfs.github.com/spec/v1
+oid sha256:c5d97acb0a6b998fb118c83d5061b2d817597b7bde0989daa85d6bc9c08242a9
 size 388016320