Heavy Quantization down to 1-2 bits has really gone a long ways.

As to answering the question 'would i accept a runtime for 9-15t/s'?, I'm hesitant to say as you are still moving the same work from a GPU to a CPU.

I have 8 cores plus 8 efficiency cores, meaning i can't use all 16 rather i can only use 8 (As i understand it the memory bandwidth space is shared, so using more just slows it down, so I'd get at best half that speed unless the CPU's intelligently divide the work). CPU's are also aligned for 32bit or 64bit block accesses, taking a block and then separating them out before working is far less efficient especially if you aren't using all of them, than if it's natively supported by the GPU. Also GPU's don't have nearly the permissions overhead to work so they can work far faster since it's mostly math and memory but only on their own memory blocks thus the speed boosts on top of fast memory.

Ultimately if i have a very important task that needs it and it can provide the work, of course even if it's slow, but a lot of prototyping often is getting test results and then iterating on the orders rather than giving it a task and coming back 4+ hours later only to see it started down the wrong path of how to approach the problem or it obviously is getting some part of your request wrong, and wasting the energy and time.

But getting it to work at all on consumer machines is still nice.