File size: 716 Bytes
7997b44 282cc8d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | ---
base_model:
- OrionLLM/OxCoder-9B
tags:
- omlx
- mlx
---
## about
oq6e quantized version of OxCoder
## imatrix used
Default oMLX imatrix
## rough performance metrics from quick tests/benchmarks
- roughly 445 tks/s prefill, 16 tk/s decode on m3 pro 36gb ram in practice on a fresh pi agent
- context benchmark from omlx for 131k context length started at 2k t/s prefill, finished at ~600 t/s
- throughput benchmark did not upload because of certain flags used but roughly matched above numbers at 128k
- ANE prefill with tweaked settings
- SpecPrefill with qwen3.5 0.8b model
## omlx throughput test with ane prefill disabled
https://omlx.ai/benchmarks/performance/tflpzdtq
## omlx version used
0.7.0 dev4 |