metadata
base_model:
- OrionLLM/OxCoder-9B
tags:
- omlx
- mlx
about
oq6e quantized version of OxCoder
imatrix used
Default oMLX imatrix
rough performance metrics from quick tests/benchmarks
- roughly 445 tks/s prefill, 16 tk/s decode on m3 pro 36gb ram in practice on a fresh pi agent
- context benchmark from omlx for 131k context length started at 2k t/s prefill, finished at ~600 t/s
- throughput benchmark did not upload because of certain flags used but roughly matched above numbers at 128k
- ANE prefill with tweaked settings
- SpecPrefill with qwen3.5 0.8b model
omlx throughput test with ane prefill disabled
https://omlx.ai/benchmarks/performance/tflpzdtq
omlx version used
0.7.0 dev4