File size: 2,298 Bytes
ad29e43
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>One training step of a 1,024-GPU MoE</title>
<meta name="description" content="A 45-layer mixture-of-experts model running one training step on 1,024 GPUs, live in your browser.">
<meta name="twitter:card" content="player">
<meta name="twitter:title" content="One training step of a 1,024-GPU MoE">
<meta name="twitter:description" content="A 45-layer mixture-of-experts model running one training step on 1,024 GPUs, live. Hover a GPU to see what it is doing.">
<meta name="twitter:player" content="https://thomwolf-moe-training-step.static.hf.space/player.html?embed=1&amp;theme=dark">
<meta name="twitter:player:width" content="640">
<meta name="twitter:player:height" content="480">
<meta name="twitter:image" content="https://thomwolf-moe-training-step.static.hf.space/card.png">
<meta name="twitter:image:alt" content="An isometric city of 1,024 GPUs in four pipeline stages, with forward passes in blue and backward passes in orange.">
<meta property="og:type" content="website">
<meta property="og:title" content="One training step of a 1,024-GPU MoE">
<meta property="og:description" content="A 45-layer mixture-of-experts model running one training step on 1,024 GPUs, live in your browser.">
<meta property="og:image" content="https://thomwolf-moe-training-step.static.hf.space/card.png">
<meta property="og:url" content="https://thomwolf-moe-training-step.static.hf.space/share.html">
<style>
html, body { margin: 0; height: 100%; background: #111215; color: #e6e8eb; font: 400 14px/1.4 "IBM Plex Sans", -apple-system, "Helvetica Neue", Arial, sans-serif; }
body { display: flex; flex-direction: column; }
iframe { flex: 1; width: 100%; border: 0; display: block; }
a.post { display: block; padding: 12px 16px; text-align: center; color: inherit; text-decoration: none; border-top: 1px solid #2a2d33; }
a.post:hover { text-decoration: underline; }
</style>
</head>
<body>
<iframe src="player.html?embed=1&amp;theme=dark" title="One training step of a 1,024-GPU MoE"></iframe>
<a class="post" href="https://huggingface.co/spaces/huggingface/moe-blogpost">Read the post: how to train a mixture-of-experts model, knob by knob →</a>
</body>
</html>