MechaEpstein-8000 — 6-bit MLX Quantization

This is a 6-bit uniform quantization of ortegaalfredo/MechaEpstein-8000 for Apple Silicon using the MLX framework.

Model Details

Detail	Value
Base Model	ortegaalfredo/MechaEpstein-8000
Architecture	Qwen3 (Qwen3ForCausalLM)
Parameters	8B (dense)
Quantization	6-bit uniform (6.500 bits per weight)
Model Size	~6.2 GB (2 shards)
Context Length	40,960 tokens
Hidden Layers	36
Hidden Size	4096
Attention Heads	32 (8 KV heads, GQA)
Vocabulary Size	151,936
Tied Embeddings	No
Framework	mlx_lm

Reasoning Format

This model uses <think>...</think> tags for reasoning output:

Thinking mode (default): Model generates a <think>...</think> block followed by the final response
Non-thinking mode: No thinking block generated (behaves like a standard instruct model)
Soft switch: Use /think or /no_think in the prompt to toggle per-turn

Recommended Inference Parameters

Thinking Mode

Parameter	Value
Temperature	1.0
Top P	0.95
Top K	20
Min P	0
Max New Tokens	32,768 (38,912 for complex problems)

⚠️ Do not use greedy decoding — it causes performance degradation and endless repetitions.

Non-Thinking Mode

Parameter	Value
Temperature	1.0
Top P	0.8
Top K	20
Min P	0

Usage

from mlx_lm import load, generate

model, tokenizer = load("leonsarmiento/MechaEpstein-8000-6bit-mlx")

messages = [{"role": "user", "content": "Hello!"}]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True
)

response = generate(model, tokenizer, prompt=prompt, max_tokens=4096, temp=0.6, top_p=0.95)
print(response)