Looking good! Have a few questions.
#2
by pylotlight - opened
Looking into MPS optimisations,
- Thoughts on a non-CUDA fast path. FastOptions without fla or CUDA graphs: bucketed static shapes + compiled submodules? your fast path is several times faster than the reference path, and the reference path is the only thing Apple Silicon can run.
- any way to further distill/quant model even further with minimal loss ? still hoping to get an effective super small model in background for all kinds of tasks on device.
- max_pixels as a request parameter? vision callers can only trade detail for latency by editing a JSON file currently.