lucataco's picture
Clairic, MLX 4-bit, with runtime, server and conversion script
1a48d6b verified
Raw History Blame Contribute Delete
267 Bytes
"""Clairic inference with MLX (the Cloudflare release, with its Transformers engine replaced; see NOTICE)."""
CONTEXT_WINDOW = 65_536
MAX_OUTPUT_TOKENS = 256
CONTEXT_HEADROOM_TOKENS = MAX_OUTPUT_TOKENS + 7
MAX_INPUT_TOKENS = CONTEXT_WINDOW - CONTEXT_HEADROOM_TOKENS