How to use from
vLLM
Install from pip and serve model
# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "TheBlueObserver/Falcon3-3B-Instruct-MLX"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBlueObserver/Falcon3-3B-Instruct-MLX",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'
Use Docker
docker model run hf.co/TheBlueObserver/Falcon3-3B-Instruct-MLX
Quick Links

TheBlueObserver/Falcon3-3B-Instruct-MLX

The Model TheBlueObserver/Falcon3-3B-Instruct-MLX was converted to MLX format from tiiuae/Falcon3-3B-Instruct using mlx-lm version 0.20.2.

Use with mlx

pip install mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("TheBlueObserver/Falcon3-3B-Instruct-MLX")

prompt="hello"

if hasattr(tokenizer, "apply_chat_template") and tokenizer.chat_template is not None:
    messages = [{"role": "user", "content": prompt}]
    prompt = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )

response = generate(model, tokenizer, prompt=prompt, verbose=True)
Downloads last month
12
Safetensors
Model size
3B params
Tensor type
F16
ยท
MLX
Hardware compatibility
Log In to add your hardware

Quantized

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for TheBlueObserver/Falcon3-3B-Instruct-MLX

Unable to build the model tree, the base model loops to the model itself. Learn more.