Spaces:
Sleeping
Sleeping
File size: 987 Bytes
667a0a4 | 1 2 3 4 5 6 7 8 9 10 11 | # Local inference and PC building
These are project-authored demonstration notes connecting the project to local PC experimentation.
A neural model's weights, activations, and key-value cache all consume memory. GPU VRAM is separate from system RAM. Available VRAM, numerical precision, batch size, and sequence length influence which model fits.
CPU inference is useful for small models and lightweight demos. A GPU may improve throughput, but model loading and tokenization can still dominate a short request. Measure end-to-end time as well as model execution time.
This document RAG demo uses a small CPU embedding model through ONNX. Large-model answer generation is optional and runs through the configured provider. It does not require a local high-end GPU.
When comparing hardware, keep model version, input length, batch size, and warm-up procedure fixed. Report the actual machine and measurements. These notes contain no benchmark claims about a particular graphics card.
|