Instructions to use Apexintelligence-AI/Apex-35B-V1.0 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Apexintelligence-AI/Apex-35B-V1.0 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Apexintelligence-AI/Apex-35B-V1.0")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Apexintelligence-AI/Apex-35B-V1.0")
model = AutoModelForMultimodalLM.from_pretrained("Apexintelligence-AI/Apex-35B-V1.0", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Apexintelligence-AI/Apex-35B-V1.0 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Apexintelligence-AI/Apex-35B-V1.0"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Apexintelligence-AI/Apex-35B-V1.0",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Apexintelligence-AI/Apex-35B-V1.0

SGLang

How to use Apexintelligence-AI/Apex-35B-V1.0 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Apexintelligence-AI/Apex-35B-V1.0" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Apexintelligence-AI/Apex-35B-V1.0",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Apexintelligence-AI/Apex-35B-V1.0" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Apexintelligence-AI/Apex-35B-V1.0",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Apexintelligence-AI/Apex-35B-V1.0 with Docker Model Runner:
```
docker model run hf.co/Apexintelligence-AI/Apex-35B-V1.0
```

CMF build of Apex-35B-V1.0 for cortiq (Vulkan / Metal / CPU, single file)

by infosave - opened 3 days ago

Discussion

infosave

3 days ago

Hi Apex Intelligence team, thanks for Apex-35B-V1.0!

cortiq, a Rust inference engine with no Python and no ML framework, now runs Apex-35B-V1.0, and the converted weights are published as single CMF files: https://huggingface.co/infosave/Apex-35B-V1.0-cmf

The text model runs natively: 30 Gated DeltaNet + 10 gated full-attention layers, 256 routed experts top-8 plus the shared expert, the MTP layer, and your own chat template (the generation prompt ends with <think>), so dynamic thinking and tool calling work as in your examples: cortiq serve returns <tool_call><function=…> as OpenAI tool_calls. The vision tower is not included. GPUs: NVIDIA / AMD / Intel through Vulkan or DX12, Apple silicon through Metal; the CPU as well.

Files (quantized from the bf16 checkpoint)

file	size	top-1 vs bf16, chat	code ppl
`apex-35b-v1.0-q4tp-v2.cmf` (4-bit, GPTQ experts, default)	19.81 GB	95.2 %	1.926
`apex-35b-v1.0-q4tp.cmf` (4-bit)	19.83 GB	94.9 %	1.905
`apex-35b-v1.0-q8_2f.cmf` (8-bit)	35.84 GB	97.7 %	1.753

Checked against your bf16 checkpoint run in PyTorch over the same tokens: top-1 vs bf16 is the share of positions where the file's top prediction matches it on 8,192 tokens of held-out chat; code ppl over 4,096 tokens of Python (bf16: 1.759). Greedy continuations match the reference ("391", "Paris"), and tool calls were checked with and without thinking.

Speed, NVIDIA A40 48 GB, Vulkan, one stream, the whole model on the card, tok/s

file	decode	4k-token prompt	16k-token prompt	decode at 16k
`q4tp-v2`	80.3	390	368	59.5
`q8_2f`	78.2	84	76	58.9

The 4-bit files fit a 24 GB card; the KV cache defaults to 32k tokens and CMF_MAX_SEQ=262144 opens the full context.

Try it

cargo install cortiq-cli
hf download infosave/Apex-35B-V1.0-cmf apex-35b-v1.0-q4tp-v2.cmf --local-dir .
cortiq run apex-35b-v1.0-q4tp-v2.cmf --prompt "Write a Python function that merges two sorted lists." --max-tokens 4096
cortiq serve apex-35b-v1.0-q4tp-v2.cmf --port 8080   # OpenAI-compatible API with tool calling

Prebuilt binaries for Linux, macOS and Windows: https://github.com/infosave2007/cmf/releases (cortiq 0.8.48 or later). The weights stay under your Apache-2.0 terms.

Feedback welcome!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment