Share your model speed here

#7
by anvme - opened

Hey everyone!

I think this topic could be really useful for users who have plenty of system RAM to run larger models but are limited by GPU VRAM. I'm especially interested in how newer architectures - looped transformers, affect generation speed across different hardware configurations

To make comparisons easier and help others understand what models/settings might work for their hardware, please share your setup and performance using the format below:

  1. Model file / model name
  2. GPU: model + VRAM
  3. CPU: model
  4. RAM: size + type
  5. Generation speed: tokens/sec
  6. App / frontend: e.g. FreeToken, llama.cpp, LM Studio, etc.
  7. Launch parameters / settings:
  8. Extra notes: quantization, context size, offloading, optimizations, or anything else worth mentioning
    This should make it much easier for people to compare setups and decide which quantization to download

P.S. My info will be later. testing on 4090 16gb vram, notebook and Mac m4 pro

4090 48g,the model is smart,but now I use unsloth's llama.cpp it's very slow in long context,the dspark mtp is not very high acceptance
图片

Sign up or log in to comment