CherryPit-1.2-2B in the CMF format (cortiq: Vulkan, Metal, CPU) + a Turbo file for long prompts

#1
by infosave - opened

Hi DRUPEs team, thank you for CherryPit-1.2!

CherryPit-1.2-2B is now available in the CMF format for cortiq, a Rust inference engine with no Python and no ML framework: NVIDIA, AMD and Intel GPUs through Vulkan/DX12, Apple silicon through Metal, and the CPU.

  • infosave/CherryPit-1.2-2B-cmf: q8_2f (2.54 GB, WikiText-2 ppl 18.46 against 18.45 for bf16) and q4tp (1.77 GB, GPTQ calibrated with chats in your template, ppl 19.40).
  • infosave/CherryPit-1.2-2B-Turbo-cmf: long prompts at a fixed memory ceiling. At 64 000 tokens on an RTX 5090 it takes 2.9 GB of VRAM and decodes at 155 tok/s, where the standard file takes 7.4 GB and runs at 54 tok/s; facts planted in 100 000 characters are recalled 8 of 8.

Your chat template, the thinking switch and the XML tool calls work as in transformers (the prompt tokens match on seven test conversations, and the server returns tool calls as message.tool_calls). Decode speed: about 190 tok/s on an RTX 5090, 43 tok/s (q4tp) on a MacBook Air M4.

cargo install cortiq-cli
hf download infosave/CherryPit-1.2-2B-cmf CherryPit-1.2-2B-q8_2f.cmf --local-dir .
cortiq run CherryPit-1.2-2B-q8_2f.cmf --prompt "Build a React component with a search box and a list that loads on open." --no-think --max-tokens 1024

The files run on cortiq 0.8.61 and later; the speeds above come with the next release.

Sign up or log in to comment