yanghaojin commited on
Commit
70f2b59
·
verified ·
1 Parent(s): 3ce9429

Model card: say what a smaller machine trades for fitting

Browse files

Paging already said a machine that cannot hold the experts streams them. This says what that costs -- a wait on the SSD per token that routes to an expert not in memory -- without inventing a number we have not measured.

Files changed (1) hide show
  1. README.md +8 -0
README.md CHANGED
@@ -46,6 +46,14 @@ fills what fits and streams the rest from disk. On a 512 GB Mac Studio with room
46
  to spare this model settles at about **172 GB** resident. A smaller machine holds
47
  less and reads more from disk -- slower, but it runs.
48
 
 
 
 
 
 
 
 
 
49
  ## Install
50
 
51
  ```bash
 
46
  to spare this model settles at about **172 GB** resident. A smaller machine holds
47
  less and reads more from disk -- slower, but it runs.
48
 
49
+ How much slower depends on how far the machine is from holding the experts, and
50
+ on how fast its disk is. Each token routes to a few experts; the ones already in
51
+ memory cost nothing to reach, and the ones that are not have to be read before
52
+ that token can finish. A machine holding most of them waits rarely, one holding
53
+ few waits often. We have not measured this across machine sizes and will not
54
+ guess a figure: what we can say is that the model answers either way, and that
55
+ the wait is the SSD's, not the model's.
56
+
57
  ## Install
58
 
59
  ```bash