Notebook: hiker-in-the-fog gradient widget (trees, marks, slope)

tiny_llm_from_scratch.ipynb

@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "980e5f0c",
+   "id": "4d281dd0",
    "metadata": {},
    "source": [
     "# Tiny Storyteller: building a language model from scratch\n",
@@ -38,7 +38,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "983741fb",
+   "id": "f9396768",
    "metadata": {},
    "source": [
     "# Before any code: the story of what we are about to build\n",
@@ -168,7 +168,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "5efd4788",
+   "id": "197e5a08",
    "metadata": {},
    "source": [
     "## Live flow diagram\n",
@@ -183,7 +183,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "b619ba27",
+   "id": "4afe7ae1",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -308,7 +308,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b33f712a",
+   "id": "02ff5701",
    "metadata": {},
    "source": [
     "## 0. Setup\n",
@@ -318,7 +318,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "649f2bf7",
+   "id": "b52205e8",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -328,7 +328,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "c60680a5",
+   "id": "bb2258ba",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -345,7 +345,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "05aa805c",
+   "id": "86f68410",
    "metadata": {},
    "source": [
     "## 1. Tensors and gradients: the two building blocks\n",
@@ -364,7 +364,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "62b705cc",
+   "id": "fc4c7ffb",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -407,7 +407,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "690ef025",
+   "id": "e6a14516",
    "metadata": {},
    "source": [
     "### How matrix multiplication actually happens\n",
@@ -422,7 +422,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "9be728bd",
+   "id": "3980d801",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -479,7 +479,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "07dc3007",
+   "id": "11d147bf",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -492,7 +492,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "dae67dc6",
+   "id": "91129a59",
    "metadata": {},
    "source": [
     "A **gradient** tells us how much the loss would change if we nudged a weight.\n",
@@ -510,7 +510,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "e0342de2",
+   "id": "45ad3be9",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -548,10 +548,164 @@
     "\"\"\"))"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "30398c6f",
+   "metadata": {},
+   "source": [
+    "### The hiker, the trees, and the marks on the map\n",
+    "\n",
+    "Here is the same idea as a story, because this is the part people picture wrongly.\n",
+    "\n",
+    "**The hill is the loss.** Height is \"how wrong the model is right now\". The valley floor,\n",
+    "the flag, is the set of weights that makes the fewest mistakes. The hiker's position *is*\n",
+    "the model's weights. One step is one weight update.\n",
+    "\n",
+    "**A tree is a step that made things worse.** The hiker tried a direction and the loss went\n",
+    "up. The hiker steps back, marks the spot with a red X, and corrects, say, to the left.\n",
+    "That correction is the gradient doing its job: after a bad step the slope under the feet\n",
+    "points back the other way.\n",
+    "\n",
+    "**What the gradient adds to this picture.** A real hiker learns a tree is there by bumping\n",
+    "into it. The gradient is better than that. It is the slope under the feet, felt *before*\n",
+    "stepping. Autograd feels the slope for all 5.9 million weights at once and says \"down is\n",
+    "this way\" without needing to try the wrong way first. Most of the time the hiker never\n",
+    "touches the tree at all. Guessing would mean testing 5.9 million directions one at a time.\n",
+    "\n",
+    "**So why does training still sometimes go uphill?**\n",
+    "- The stride was too long. That is the *learning rate*. The slope was right, but the hiker\n",
+    "  overshot the valley and landed on the far wall. The next slope reading points back.\n",
+    "- The reading was noisy. Each step uses one batch of stories, so the slope is an estimate.\n",
+    "\n",
+    "**The marks in space are real, in two forms.**\n",
+    "- *The weights themselves.* Every step is written into the weights and nothing is forgotten.\n",
+    "  The model does not get a map, it *is* the map after training.\n",
+    "- *The optimizer's memory.* AdamW keeps a running average of recent slopes. That is the\n",
+    "  hiker remembering \"the last ten readings all said left, so one odd reading saying right is\n",
+    "  probably a bump; keep going left\". One tree does not fool it.\n",
+    "\n",
+    "**Fastest and most accurate.** Short careful steps at the start (warm-up, the terrain is\n",
+    "unknown), full stride in the middle, tiny steps near the flag (cosine decay) so the hiker\n",
+    "stops on the floor instead of bouncing between the walls.\n",
+    "\n",
+    "Try both ways of walking below and compare the step and tree counts."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "78e7b447",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Interactive: the hiker in the fog. Guess and bump into trees, or read the slope first.\n",
+    "from IPython.display import HTML, display\n",
+    "display(HTML(r\"\"\"<div id=\"vh\" style=\"font-family:-apple-system,Segoe UI,Helvetica,Arial,sans-serif;color:#e8e8e8;background:#1e1e1e;border-radius:12px;padding:16px;max-width:960px\">\n",
+    "<style>\n",
+    "#vh button{background:#333;color:#eee;border:1px solid #777;border-radius:6px;padding:6px 12px;margin:0 6px 6px 0;cursor:pointer;font-size:13px}\n",
+    "#vh button:hover{background:#444}\n",
+    "#vh button.g{border-color:#6fcf97}#vh button.r{border-color:#ff6b6b}\n",
+    "#vh .cap{font-size:13px;line-height:1.5;margin-top:8px;background:#262626;border-radius:8px;padding:10px}\n",
+    "#vh .log{font-size:12px;background:#262626;border-radius:8px;padding:8px 10px;height:150px;overflow:auto;line-height:1.5}\n",
+    "#vh .stat{display:inline-block;padding:4px 10px;border-radius:6px;background:#2a2a2a;border:1px solid #555;margin:2px 6px 2px 0;font-size:12px}\n",
+    "</style>\n",
+    "<div style=\"font-size:15px;font-weight:600;margin-bottom:4px\">The hiker in the fog: trees, marks, and the slope under your feet</div>\n",
+    "<div style=\"font-size:12px;color:#aaa;margin-bottom:8px\">The map is the loss. Dark blue is low (good), yellow is high (bad). The flag is the best set of weights. The hiker cannot see the map; it only feels the ground.</div>\n",
+    "<div style=\"display:flex;gap:16px;flex-wrap:wrap\">\n",
+    "  <div><canvas id=\"vhC\" width=\"520\" height=\"360\" style=\"border-radius:8px;background:#262626\"></canvas></div>\n",
+    "  <div style=\"flex:1;min-width:300px\">\n",
+    "    <div style=\"font-size:12px;color:#aaa;margin-bottom:4px\">TWO WAYS TO WALK</div>\n",
+    "    <button class=\"r\" onclick=\"vhBlind()\">Guess a step (may hit a tree)</button>\n",
+    "    <button class=\"g\" onclick=\"vhGrad()\">Read the slope, then step</button><br>\n",
+    "    <button onclick=\"vhAuto('blind')\">Auto: guessing</button>\n",
+    "    <button onclick=\"vhAuto('grad')\">Auto: slope</button>\n",
+    "    <button onclick=\"vhReset()\">Reset</button>\n",
+    "    <div style=\"margin:6px 0\">\n",
+    "      <label style=\"font-size:12px\">Stride (learning rate) <input id=\"vhLr\" type=\"range\" min=\"1\" max=\"10\" value=\"4\" style=\"vertical-align:middle;width:120px\"></label>\n",
+    "      <label style=\"font-size:12px;margin-left:8px\"><input id=\"vhMem\" type=\"checkbox\" checked> memory of recent slopes (momentum)</label>\n",
+    "    </div>\n",
+    "    <div><span class=\"stat\" id=\"vhSteps\">steps: 0</span><span class=\"stat\" id=\"vhTrees\">trees hit: 0</span><span class=\"stat\" id=\"vhLoss\">loss: 0</span></div>\n",
+    "    <div style=\"font-size:12px;color:#aaa;margin:8px 0 4px\">THE HIKER'S NOTEBOOK (marks in space)</div>\n",
+    "    <div class=\"log\" id=\"vhLog\">Nothing tried yet.</div>\n",
+    "  </div>\n",
+    "</div>\n",
+    "<div class=\"cap\" id=\"vhCap\"></div>\n",
+    "</div>\n",
+    "<script>\n",
+    "(function(){\n",
+    "const C=document.getElementById('vhC'),ctx=C.getContext('2d'),W=C.width,H=C.height;\n",
+    "// loss landscape in map coordinates (0..1, 0..1): a tilted bowl with two bumps (the \"trees\" of the terrain)\n",
+    "const goal=[0.72,0.68];\n",
+    "const loss=(x,y)=>{const dx=x-goal[0],dy=y-goal[1];let l=6*(dx*dx+dy*dy)+0.5*(1-x)+0.3*(1-y);\n",
+    "  l+=1.1*Math.exp(-((x-0.40)**2+(y-0.55)**2)/0.012);l+=0.9*Math.exp(-((x-0.58)**2+(y-0.30)**2)/0.010);return l;};\n",
+    "const grad=(x,y)=>{const h=1e-3;return [(loss(x+h,y)-loss(x-h,y))/(2*h),(loss(x,y+h)-loss(x,y-h))/(2*h)];};\n",
+    "const px=x=>x*W, py=y=>(1-y)*H;\n",
+    "let bg=null,pos,trail,xs,steps,trees,mem,timer=null,mode='';\n",
+    "let minL=Infinity;for(let i=0;i<=100;i++)for(let j=0;j<=100;j++){const v=loss(i/100,j/100);if(v<minL)minL=v;}\n",
+    "function paintBg(){const img=ctx.createImageData(W,H);let mx=0;const vals=new Float32Array(W*H);\n",
+    "  for(let j=0;j<H;j++)for(let i=0;i<W;i++){const v=loss(i/W,1-j/H);vals[j*W+i]=v;if(v>mx)mx=v;}\n",
+    "  for(let k=0;k<W*H;k++){const t=Math.min(1,vals[k]/mx);const r=Math.round(30+225*t),g=Math.round(40+170*t*t),b=Math.round(110*(1-t)+40);\n",
+    "    img.data[k*4]=r;img.data[k*4+1]=g;img.data[k*4+2]=b;img.data[k*4+3]=255;}\n",
+    "  ctx.putImageData(img,0,0);\n",
+    "  // contour-ish rings\n",
+    "  ctx.strokeStyle='rgba(255,255,255,0.12)';ctx.lineWidth=1;\n",
+    "  for(let lv=0.3;lv<4;lv+=0.35){ctx.beginPath();let first=true;for(let a=0;a<=Math.PI*2+0.05;a+=0.05){\n",
+    "    // crude: walk outward from goal until loss>=lv\n",
+    "    let r=0;for(;r<1.2;r+=0.004){if(loss(goal[0]+r*Math.cos(a),goal[1]+r*Math.sin(a))>=lv)break;}\n",
+    "    const x=px(goal[0]+r*Math.cos(a)),y=py(goal[1]+r*Math.sin(a));if(first){ctx.moveTo(x,y);first=false;}else ctx.lineTo(x,y);}ctx.stroke();}\n",
+    "  bg=ctx.getImageData(0,0,W,H);}\n",
+    "function draw(){ctx.putImageData(bg,0,0);\n",
+    "  // flag\n",
+    "  ctx.fillStyle='#fff';ctx.fillRect(px(goal[0])-1,py(goal[1])-22,2,22);ctx.fillStyle='#6fcf97';ctx.beginPath();ctx.moveTo(px(goal[0]),py(goal[1])-22);ctx.lineTo(px(goal[0])+14,py(goal[1])-17);ctx.lineTo(px(goal[0]),py(goal[1])-12);ctx.fill();\n",
+    "  ctx.font='11px sans-serif';ctx.fillStyle='#fff';ctx.fillText('best weights',px(goal[0])+16,py(goal[1])-14);\n",
+    "  // trail\n",
+    "  ctx.strokeStyle='#6fcf97';ctx.lineWidth=2;ctx.beginPath();trail.forEach((p,i)=>{i?ctx.lineTo(px(p[0]),py(p[1])):ctx.moveTo(px(p[0]),py(p[1]));});ctx.stroke();\n",
+    "  trail.forEach(p=>{ctx.fillStyle='#6fcf97';ctx.beginPath();ctx.arc(px(p[0]),py(p[1]),3,0,7);ctx.fill();});\n",
+    "  // trees (bad tries)\n",
+    "  xs.forEach(p=>{ctx.strokeStyle='#ff6b6b';ctx.lineWidth=2.5;const x=px(p[0]),y=py(p[1]);ctx.beginPath();ctx.moveTo(x-6,y-6);ctx.lineTo(x+6,y+6);ctx.moveTo(x+6,y-6);ctx.lineTo(x-6,y+6);ctx.stroke();});\n",
+    "  // slope arrow (what the gradient says)\n",
+    "  const g=grad(pos[0],pos[1]);const n=Math.hypot(g[0],g[1])||1;const ax=-g[0]/n*34,ay=g[1]/n*34;\n",
+    "  ctx.strokeStyle='#ffb347';ctx.lineWidth=3;ctx.beginPath();ctx.moveTo(px(pos[0]),py(pos[1]));ctx.lineTo(px(pos[0])+ax,py(pos[1])+ay);ctx.stroke();\n",
+    "  ctx.fillStyle='#ffb347';ctx.beginPath();const ang=Math.atan2(ay,ax);ctx.moveTo(px(pos[0])+ax,py(pos[1])+ay);ctx.lineTo(px(pos[0])+ax-9*Math.cos(ang-0.4),py(pos[1])+ay-9*Math.sin(ang-0.4));ctx.lineTo(px(pos[0])+ax-9*Math.cos(ang+0.4),py(pos[1])+ay-9*Math.sin(ang+0.4));ctx.fill();\n",
+    "  // hiker\n",
+    "  ctx.fillStyle='#fff';ctx.beginPath();ctx.arc(px(pos[0]),py(pos[1]),8,0,7);ctx.fill();ctx.fillStyle='#1e1e1e';ctx.font='11px sans-serif';ctx.fillText('\\u{1F6B6}',px(pos[0])-6,py(pos[1])+4);\n",
+    "  const l=loss(pos[0],pos[1]);\n",
+    "  document.getElementById('vhSteps').textContent='steps: '+steps;document.getElementById('vhTrees').textContent='trees hit: '+trees;document.getElementById('vhLoss').textContent='loss: '+l.toFixed(2);\n",
+    "  if(l<minL+0.04){document.getElementById('vhCap').innerHTML='<b>Arrived.</b> The hiker is on the valley floor: these weights make the fewest mistakes. Steps: <b>'+steps+'</b>, trees hit: <b>'+trees+'</b>. Press Reset and try the other way of walking.';if(timer){clearInterval(timer);timer=null;}}\n",
+    "}\n",
+    "function logLine(html){const L=document.getElementById('vhLog');if(L.textContent.startsWith('Nothing'))L.innerHTML='';L.innerHTML='<div>'+html+'</div>'+L.innerHTML;}\n",
+    "const lr=()=>0.008*(+document.getElementById('vhLr').value);\n",
+    "window.vhBlind=function(){steps++;const a=Math.random()*Math.PI*2;const d=lr()*2.2;const nx=pos[0]+Math.cos(a)*d,ny=pos[1]+Math.sin(a)*d;\n",
+    "  const before=loss(pos[0],pos[1]),after=loss(nx,ny);\n",
+    "  if(after>=before){trees++;xs.push([nx,ny]);logLine('&#10060; tried '+dir(a)+': loss went <b>up</b> '+before.toFixed(2)+' &rarr; '+after.toFixed(2)+'. Tree. Marked, stayed put.');\n",
+    "    document.getElementById('vhCap').innerHTML='<b>Hit a tree.</b> The guess made the loss worse, so the hiker steps back and marks the spot with a red X. Guessing finds the valley eventually, but it pays for every tree. Real training cannot afford this: each guess would mean testing 5.9 million weights one direction at a time.';}\n",
+    "  else{pos=[nx,ny];trail.push(pos.slice());logLine('&#10004; tried '+dir(a)+': loss went down '+before.toFixed(2)+' &rarr; '+after.toFixed(2)+'. Kept it.');\n",
+    "    document.getElementById('vhCap').innerHTML='<b>Lucky guess.</b> The loss went down, so the step is kept and the trail (green) records it.';}\n",
+    "  draw();};\n",
+    "window.vhGrad=function(){steps++;let g=grad(pos[0],pos[1]);\n",
+    "  // momentum: remember recent slopes\n",
+    "  if(document.getElementById('vhMem').checked){mem=[0.8*mem[0]+0.2*g[0],0.8*mem[1]+0.2*g[1]];g=mem;}\n",
+    "  const n=Math.hypot(g[0],g[1])||1;const d=Math.min(lr()*2.2,lr()*n*0.6);const nx=Math.min(0.98,Math.max(0.02,pos[0]-g[0]/n*d)),ny=Math.min(0.98,Math.max(0.02,pos[1]-g[1]/n*d));\n",
+    "  const before=loss(pos[0],pos[1]),after=loss(nx,ny);pos=[nx,ny];trail.push(pos.slice());\n",
+    "  if(after>=before){trees++;xs.push([nx,ny]);logLine('&#9888; slope said '+dir(Math.atan2(-g[1],-g[0]))+' but loss went <b>up</b> '+before.toFixed(2)+' &rarr; '+after.toFixed(2)+' (stride too long, or noisy reading). Next reading will point back.');\n",
+    "    document.getElementById('vhCap').innerHTML='<b>Overshot.</b> The slope was right but the stride was too long, so the hiker landed on the far wall. Nothing is lost: the next slope reading points straight back. Shorten the stride (lower learning rate) and this stops happening. That is why training ends with tiny steps.';}\n",
+    "  else{logLine('&#10004; slope said '+dir(Math.atan2(-g[1],-g[0]))+': loss '+before.toFixed(2)+' &rarr; '+after.toFixed(2)+'.');\n",
+    "    document.getElementById('vhCap').innerHTML='<b>Read the slope first.</b> The orange arrow is the gradient: which way is downhill, felt through the feet before moving. No tree needed. Autograd gives this arrow for all 5.9 million weights at once. With memory on, the arrow is the average of recent readings, so one odd bump does not fool the hiker.';}\n",
+    "  draw();};\n",
+    "window.vhAuto=function(m){if(timer){clearInterval(timer);timer=null;if(mode===m)return;}mode=m;timer=setInterval(()=>{m==='blind'?vhBlind():vhGrad();if(steps>400){clearInterval(timer);timer=null;}},m==='blind'?120:260);};\n",
+    "window.vhReset=function(){if(timer){clearInterval(timer);timer=null;}pos=[0.12,0.86];trail=[pos.slice()];xs=[];steps=0;trees=0;mem=[0,0];document.getElementById('vhLog').textContent='Nothing tried yet.';\n",
+    "  document.getElementById('vhCap').innerHTML='The hiker starts top-left in fog. <b>Guess a step</b>: pick a random direction, walk, and only then find out if the loss went up (a tree, marked red) or down (kept, marked green). <b>Read the slope</b>: feel the ground first, step downhill. Compare how many steps and trees each way needs to reach the flag.';draw();};\n",
+    "function dir(a){const d=['east','north-east','north','north-west','west','south-west','south','south-east'];return d[Math.round(((a%(2*Math.PI))+2*Math.PI)%(2*Math.PI)/(Math.PI/4))%8];}\n",
+    "paintBg();vhReset();\n",
+    "})();\n",
+    "</script>\n",
+    "\"\"\"))"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "ab57ff9e",
+   "id": "87220190",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -569,7 +723,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4e717021",
+   "id": "0b60c411",
    "metadata": {},
    "source": [
     "## 2. Data: TinyStories\n",
@@ -584,7 +738,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "67dfd24b",
+   "id": "463680d1",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -602,7 +756,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3758528e",
+   "id": "ddf8b81e",
    "metadata": {},
    "source": [
     "## 3. The simplest language model: a bigram\n",
@@ -622,7 +776,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "69fb3e89",
+   "id": "84a17efb",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -653,7 +807,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "b6b441fb",
+   "id": "553fcb26",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -673,7 +827,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "3301c521",
+   "id": "bdda55d3",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -722,7 +876,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "073e9f1b",
+   "id": "9fdda98f",
    "metadata": {},
    "source": [
     "It learned *something* (the loss dropped, and it produces word-ish shapes with spaces),\n",
@@ -732,7 +886,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "07cea514",
+   "id": "586b2b1e",
    "metadata": {},
    "source": [
     "## 4. A real tokenizer: byte-pair encoding (BPE)\n",
@@ -750,7 +904,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "2d9f5023",
+   "id": "a08a191a",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -780,7 +934,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "e1d5392c",
+   "id": "4a3fe207",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -806,7 +960,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "b38d367d",
+   "id": "f29885a8",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -820,7 +974,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "3a3b651d",
+   "id": "9fc7ca98",
    "metadata": {},
    "source": [
     "## 5. Attention, by hand\n",
@@ -841,7 +995,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "459979b9",
+   "id": "7ca9b266",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -865,7 +1019,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "57ea4612",
+   "id": "ff47fa82",
    "metadata": {},
    "source": [
     "### The KV cache, in twelve lines\n",
@@ -879,7 +1033,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "81bf3c28",
+   "id": "f71b1fb1",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -909,7 +1063,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "190d030b",
+   "id": "fb78d22a",
    "metadata": {},
    "source": [
     "## 6. A tiny GPT\n",
@@ -932,7 +1086,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "0c74a9de",
+   "id": "4f0b2930",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -976,7 +1130,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "cc29c0f8",
+   "id": "352f105d",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -1087,7 +1241,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "515072d6",
+   "id": "dc0da2c9",
    "metadata": {},
    "source": [
     "## 7. Training\n",
@@ -1109,7 +1263,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "69241e04",
+   "id": "047dde66",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -1151,7 +1305,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "0f607250",
+   "id": "7f86a500",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -1218,7 +1372,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "e6723c30",
+   "id": "56158b70",
    "metadata": {},
    "source": [
     "**How to read the numbers.** The loss is the average \"surprise\" per token. *Perplexity* is\n",
@@ -1228,7 +1382,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "953a841e",
+   "id": "f03d66b2",
    "metadata": {},
    "source": [
     "## 8. Generate stories: the use case\n",
@@ -1240,7 +1394,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "2cc94e4e",
+   "id": "49179e8e",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -1257,7 +1411,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "75329d6b",
+   "id": "2be38804",
    "metadata": {},
    "source": [
     "## 9. Save and publish on the Hugging Face Hub\n",
@@ -1273,7 +1427,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "f26e61dc",
+   "id": "3bf91999",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -1314,7 +1468,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "08db0245",
+   "id": "8e406129",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -1335,7 +1489,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "b91cda98",
+   "id": "fa86efcb",
    "metadata": {},
    "source": [
     "## Loading it back later\n",
@@ -1346,7 +1500,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "c8a96007",
+   "id": "3d6c508e",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -1367,7 +1521,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "f2036bcb",
+   "id": "20653937",
    "metadata": {},
    "source": [
     "## What we built, in one paragraph\n",