asemqaffaf commited on
Commit
5b423ff
·
verified ·
1 Parent(s): 47ad5e7

Add transformer variant, numpy loader for both models, link to source code

Browse files
README.md CHANGED
@@ -13,25 +13,23 @@ pipeline_tag: text-generation
13
 
14
  # AsemLLM
15
 
16
- A tiny word-level neural language model (**152,903 parameters**) written, trained and served entirely in **Erlang**.
17
- Supervised worker processes compute gradients in parallel; there is no deep-learning framework involved.
18
- This repository holds the trained weights, the vocabulary and a small numpy reference loader.
19
 
20
- > **This is a toy model.** It learns the *style* of its training text (short `User:` / `Assistant:` dialogue) and
21
- > produces fluent-looking English, but it does **not** understand questions, and its answers are usually unrelated
22
- > to them. It can state false things confidently. Do not use it for anything that needs correct answers.
23
 
24
- ## Architecture
25
 
26
- | | |
27
- |---|---|
28
- | Type | Embedding MLP language model (Bengio-style), **not** a transformer |
29
- | Context | 5 tokens |
30
- | Embedding / hidden size | 24 / 64 (tanh) |
31
- | Vocabulary | 1631 tokens (words with their leading space, punctuation, newline; `<unk>` = id 0) |
32
- | Parameters | 152,903 |
33
 
34
- `logits = W2 · tanh(W1 · concat(embedding[last 5 tokens]) + b1) + b2`
 
35
 
36
  ## Use it
37
 
@@ -39,51 +37,61 @@ This repository holds the trained weights, the vocabulary and a small numpy refe
39
  pip install numpy safetensors huggingface_hub
40
  huggingface-cli download asemqaffaf/AsemLLM --local-dir AsemLLM
41
  cd AsemLLM
42
- python asemllm.py --chat "What is a good movie?"
43
- python asemllm.py $'User: How do I cook pasta?\nAssistant:' # $'...' turns \n into a real newline
44
  ```
45
 
46
  ```python
47
  from asemllm import AsemLLM
48
- m = AsemLLM(".")
49
  print(m.chat([{"role": "user", "content": "Hello!"}], temperature=0.5, seed=1))
50
  ```
51
 
52
- It uses a custom architecture, so it does **not** load in `transformers`, llama.cpp, Ollama or LM Studio.
53
- The weights are in `model.safetensors` (float32): `embedding`, `w1`, `b1`, `w2`, `b2`.
54
 
55
  ## Training data
56
 
57
  About 34,904 tokens (138 KB) of short, synthetic `User:` / `Assistant:` conversations on everyday topics
58
  (cooking, gardening, travel, study, hobbies, ...), written by a locally run 4B-parameter instruction model.
59
- No web scraping, no personal data. The text was filtered to well-formed dialogue lines and truncated mid-sentence
60
- generations were dropped. Because the data is machine-written, the model inherits the repetitiveness and the
61
- mistakes of its source.
62
 
63
- ## Training
64
 
65
- - Word-level tokenizer, top 1631 tokens seen at least twice.
66
- - Adam, lr 0.005 decaying linearly to 10%, batch 128, gradient clipping at 5, 3,000 steps; the checkpoint with the best validation loss is kept.
67
- - Data-parallel training on one Apple M1 Pro (10 Erlang worker processes), about 13 minutes.
68
- - Held-out set: the last 10% of the text.
69
 
70
  ## Evaluation (held-out cross-entropy per token, lower is better)
71
 
72
  | Model | Loss | Perplexity |
73
  |---|---|---|
74
- | Unigram (word frequencies) | 5.14 | 171 |
75
- | Bigram (previous word) | 3.98 | 53 |
76
- | **AsemLLM** | **3.73** | **42** |
77
 
78
- It beats the bigram baseline only modestly. With 138 KB of training text it largely memorizes phrases (training loss
79
- fell to about 1.3 while validation loss stopped improving at 3.73).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
80
 
81
  ## Limitations
82
 
83
- - Tiny context (5 tokens) and tiny data: replies mostly do not relate to the prompt.
84
- - English only; ASCII-centric vocabulary.
85
- - No safety training. Output reflects its (machine-written) training text.
86
- - Not suitable for any real application. It is meant for learning how language models work.
87
 
88
  ## License
89
 
 
13
 
14
  # AsemLLM
15
 
16
+ Tiny word-level language models written, trained and served entirely in **Erlang**, with no deep-learning framework:
17
+ supervised worker processes compute gradients in parallel. This repository holds the trained weights, the vocabulary and a
18
+ small numpy reference loader. **Source code (training, serving, web UI): https://github.com/asemqaffaf/AsemLLms**
19
 
20
+ > **These are toy models.** They learn the *style* of their training text (short `User:` / `Assistant:` dialogue) and
21
+ > produce fluent-looking English, but they do **not** understand questions, and answers are usually unrelated to them.
22
+ > They can state false things confidently. Do not use them for anything that needs correct answers.
23
 
24
+ ## Two variants
25
 
26
+ | | Where | Parameters | Window | Architecture |
27
+ |---|---|---|---|---|
28
+ | MLP | repository root | 152,903 | 5 tokens | embedding -> tanh hidden layer -> softmax (Bengio-style) |
29
+ | Transformer | `transformer/` | 279,007 | 32 tokens | GPT-style decoder (attention) |
 
 
 
30
 
31
+ Both use the same tokenizer: words with their leading space, punctuation marks and newlines; the 1631-token vocabulary
32
+ (top tokens seen at least twice, `<unk>` = id 0) is in `vocab.json`.
33
 
34
  ## Use it
35
 
 
37
  pip install numpy safetensors huggingface_hub
38
  huggingface-cli download asemqaffaf/AsemLLM --local-dir AsemLLM
39
  cd AsemLLM
40
+ python asemllm.py --chat "What is a good movie?" # MLP
41
+ python asemllm.py --model transformer --chat "What is a good movie?" # transformer
42
  ```
43
 
44
  ```python
45
  from asemllm import AsemLLM
46
+ m = AsemLLM(".") # or AsemLLM("transformer")
47
  print(m.chat([{"role": "user", "content": "Hello!"}], temperature=0.5, seed=1))
48
  ```
49
 
50
+ They use custom architectures, so they do **not** load in `transformers`, llama.cpp, Ollama or LM Studio.
51
+ Weights are float32 safetensors; `asemllm.py` is the reference implementation (verified against the Erlang code to ~1e-7 in logits).
52
 
53
  ## Training data
54
 
55
  About 34,904 tokens (138 KB) of short, synthetic `User:` / `Assistant:` conversations on everyday topics
56
  (cooking, gardening, travel, study, hobbies, ...), written by a locally run 4B-parameter instruction model.
57
+ No web scraping, no personal data. Text was filtered to well-formed dialogue lines and truncated generations were dropped.
58
+ Being machine-written, the data carries the repetitiveness and the mistakes of its source.
 
59
 
60
+ ## Training (MLP)
61
 
62
+ - Word tokenizer, Adam (lr 0.005, linear decay to 10%), batch 128, gradient clipping 5, 3,000 steps; best-validation checkpoint kept.
63
+ - Data-parallel training on one Apple M1 Pro (10 Erlang worker processes), about 13 minutes. Held-out set: last 10% of the text.
 
 
64
 
65
  ## Evaluation (held-out cross-entropy per token, lower is better)
66
 
67
  | Model | Loss | Perplexity |
68
  |---|---|---|
69
+ | Unigram baseline | 5.14 | 171 |
70
+ | Bigram baseline (previous word) | 3.98 | 53 |
71
+ | **AsemLLM MLP** | **3.73** | **42** |
72
 
73
+ ### Transformer variant (`transformer/`)
74
+
75
+ A small GPT-style decoder: **279,007 parameters**, 2 layers x 4 heads, model dim 64
76
+ (MLP width 128), learned positions, pre-LayerNorm, causal attention over a 32-token window,
77
+ trained on the same data. Unlike the MLP it sees the whole prompt (up to 32 tokens).
78
+
79
+ | Held-out loss | Value | Perplexity |
80
+ |---|---|---|
81
+ | Transformer (best checkpoint of 2,500 steps) | 3.65 | 38 |
82
+
83
+ The transformer overfits quickly on 138 KB: training loss kept falling while validation loss bottomed out early and
84
+ rose again, so the saved weights are the best validation checkpoint, not the last step. Its loss is averaged over every
85
+ position of held-out 32-token windows (including positions with little context), so it is **not directly comparable**
86
+ to the MLP's single-next-token number.
87
+
88
+ With only 138 KB of text these models largely memorize phrases (MLP training loss fell to about 1.3 while validation loss stopped improving at 3.73).
89
 
90
  ## Limitations
91
 
92
+ - Tiny data and tiny models: replies mostly do not relate to the prompt. English only; ASCII-centric vocabulary.
93
+ - No safety training; output reflects its machine-written training text.
94
+ - Not suitable for any real application. It exists to show how language models work, end to end, in Erlang.
 
95
 
96
  ## License
97
 
asemllm.py CHANGED
@@ -1,7 +1,8 @@
1
- """AsemLLM reference inference: numpy + safetensors only.
2
 
3
- python asemllm.py $'User: How do I cook pasta?\nAssistant:' # $'...' makes \n a real newline
4
- python asemllm.py --chat "How do I cook pasta?"
 
5
  """
6
  import json, re, sys
7
  from pathlib import Path
@@ -11,30 +12,61 @@ from safetensors.numpy import load_file
11
  TOKEN_RE = re.compile(rb" ?[A-Za-z0-9']+| ?[^\sA-Za-z0-9']|\n|\s")
12
 
13
 
 
 
 
 
 
 
14
  class AsemLLM:
15
  def __init__(self, path="."):
16
  p = Path(path)
17
  self.cfg = json.loads((p / "config.json").read_text())
18
- w = load_file(str(p / "model.safetensors"))
19
- self.E, self.W1, self.b1, self.W2, self.b2 = (
20
- w[k].astype(np.float64) for k in ("embedding", "w1", "b1", "w2", "b2"))
21
  raw = json.loads((p / "vocab.json").read_text())
22
  self.tokens = [bytes.fromhex(t["hex"]) if isinstance(t, dict) else t.encode() for t in raw]
23
  self.index = {t: i for i, t in enumerate(self.tokens)}
24
  self.n = self.cfg["context_length"]
 
25
  self.pad = self.index.get(b"\n", 0)
26
 
27
  def logits(self, ctx_ids):
28
- """Next-token logits for exactly `context_length` token ids."""
29
- x = np.concatenate([self.E[i] for i in ctx_ids])
30
- h = np.tanh(self.W1 @ x + self.b1)
31
- return self.W2 @ h + self.b2
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
32
 
33
  def generate(self, prompt, max_tokens=100, temperature=0.7, seed=None, stop=()):
34
  """Continue `prompt` (str). Returns the continuation. Never emits <unk>."""
35
  rng = np.random.default_rng(seed)
36
  ids = [self.index.get(t, 0) for t in TOKEN_RE.findall(prompt.encode())]
37
- ctx = ([self.pad] * max(0, self.n - len(ids)) + ids)[-self.n:]
 
 
 
38
  stops = [s.encode() for s in stop]
39
  out = b""
40
  for _ in range(max_tokens):
@@ -44,7 +76,7 @@ class AsemLLM:
44
  p = np.exp(z - z.max())
45
  nxt = int(rng.choice(len(p), p=p / p.sum()))
46
  out += self.tokens[nxt]
47
- ctx = ctx[1:] + [nxt]
48
  cut = [out.find(s) for s in stops if s in out]
49
  if cut:
50
  out = out[:min(cut)]
@@ -60,7 +92,10 @@ class AsemLLM:
60
 
61
  if __name__ == "__main__":
62
  args = sys.argv[1:]
63
- model = AsemLLM(Path(__file__).parent)
 
 
 
64
  if args and args[0] == "--chat":
65
  print(model.chat([{"role": "user", "content": " ".join(args[1:])}]))
66
  else:
 
1
+ """AsemLLM reference inference: numpy + safetensors only. Runs both model variants.
2
 
3
+ python asemllm.py --chat "What is a good movie?" # MLP model (this folder)
4
+ python asemllm.py --model transformer --chat "What is a good movie?"
5
+ python asemllm.py --model transformer $'User: How do I cook pasta?\\nAssistant:' # $'..' = real newline
6
  """
7
  import json, re, sys
8
  from pathlib import Path
 
12
  TOKEN_RE = re.compile(rb" ?[A-Za-z0-9']+| ?[^\sA-Za-z0-9']|\n|\s")
13
 
14
 
15
+ def _ln(x, g, b):
16
+ mu = x.mean(-1, keepdims=True)
17
+ var = ((x - mu) ** 2).mean(-1, keepdims=True)
18
+ return (x - mu) / np.sqrt(var + 1e-5) * g + b
19
+
20
+
21
  class AsemLLM:
22
  def __init__(self, path="."):
23
  p = Path(path)
24
  self.cfg = json.loads((p / "config.json").read_text())
25
+ self.w = {k: v.astype(np.float64) for k, v in load_file(str(p / "model.safetensors")).items()}
 
 
26
  raw = json.loads((p / "vocab.json").read_text())
27
  self.tokens = [bytes.fromhex(t["hex"]) if isinstance(t, dict) else t.encode() for t in raw]
28
  self.index = {t: i for i, t in enumerate(self.tokens)}
29
  self.n = self.cfg["context_length"]
30
+ self.arch = self.cfg["architecture"]
31
  self.pad = self.index.get(b"\n", 0)
32
 
33
  def logits(self, ctx_ids):
34
+ """Next-token logits. MLP: exactly `context_length` ids. Transformer: 1..`context_length` ids."""
35
+ w = self.w
36
+ if self.arch == "transformer":
37
+ return self._transformer_logits(ctx_ids)
38
+ x = np.concatenate([w["embedding"][i] for i in ctx_ids])
39
+ return w["w2"] @ np.tanh(w["w1"] @ x + w["b1"]) + w["b2"]
40
+
41
+ def _transformer_logits(self, ids):
42
+ w, c = self.w, self.cfg
43
+ T, D, H = len(ids), c["dim"], c["heads"]
44
+ dh = D // H
45
+ x = w["embedding"][ids] + w["positional"][:T]
46
+ mask = np.triu(np.full((T, T), -np.inf), k=1) # causal: position t sees only s <= t
47
+ for i in range(c["layers"]):
48
+ g = lambda k: w[f"layers.{i}.{k}"]
49
+ xn = _ln(x, g("ln1_g"), g("ln1_b"))
50
+ q, k_, v = (xn @ g(n).T for n in ("wq", "wk", "wv"))
51
+ split = lambda m: m.reshape(T, H, dh).transpose(1, 0, 2) # (heads, T, dh)
52
+ s = split(q) @ split(k_).transpose(0, 2, 1) / np.sqrt(dh) + mask
53
+ a = np.exp(s - s.max(-1, keepdims=True))
54
+ a /= a.sum(-1, keepdims=True)
55
+ ctx = (a @ split(v)).transpose(1, 0, 2).reshape(T, D)
56
+ x = x + ctx @ g("wo").T
57
+ xn = _ln(x, g("ln2_g"), g("ln2_b"))
58
+ x = x + np.maximum(xn @ g("w1").T + g("b1"), 0.0) @ g("w2").T + g("b2")
59
+ xf = _ln(x[-1], w["lnf_g"], w["lnf_b"])
60
+ return w["wout"] @ xf + w["bout"]
61
 
62
  def generate(self, prompt, max_tokens=100, temperature=0.7, seed=None, stop=()):
63
  """Continue `prompt` (str). Returns the continuation. Never emits <unk>."""
64
  rng = np.random.default_rng(seed)
65
  ids = [self.index.get(t, 0) for t in TOKEN_RE.findall(prompt.encode())]
66
+ if self.arch == "transformer":
67
+ ctx = (ids or [self.pad])[-self.n:]
68
+ else:
69
+ ctx = ([self.pad] * max(0, self.n - len(ids)) + ids)[-self.n:]
70
  stops = [s.encode() for s in stop]
71
  out = b""
72
  for _ in range(max_tokens):
 
76
  p = np.exp(z - z.max())
77
  nxt = int(rng.choice(len(p), p=p / p.sum()))
78
  out += self.tokens[nxt]
79
+ ctx = (ctx + [nxt])[-self.n:]
80
  cut = [out.find(s) for s in stops if s in out]
81
  if cut:
82
  out = out[:min(cut)]
 
92
 
93
  if __name__ == "__main__":
94
  args = sys.argv[1:]
95
+ sub = "."
96
+ if args[:1] == ["--model"]:
97
+ sub, args = args[1], args[2:]
98
+ model = AsemLLM(Path(__file__).parent / sub)
99
  if args and args[0] == "--chat":
100
  print(model.chat([{"role": "user", "content": " ".join(args[1:])}]))
101
  else:
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:dde0a87ab5cb66191c8c6644c3172d1fa65f057d94005e4853cda035428ec839
3
  size 612004
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:56b328e0960f128bae87565f3c840b5e174f8424af5bb8a227c65db62f83e742
3
  size 612004
transformer/config.json ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_type": "asemllm",
3
+ "architecture": "transformer",
4
+ "vocab_size": 1631,
5
+ "context_length": 32,
6
+ "dim": 64,
7
+ "heads": 4,
8
+ "layers": 2,
9
+ "ff_dim": 128,
10
+ "activation": "relu",
11
+ "norm": "pre-layernorm (eps 1e-5)",
12
+ "positional": "learned",
13
+ "tokenizer": "word",
14
+ "unk_token_id": 0,
15
+ "num_parameters": 279007,
16
+ "tokenizer_regex": " ?[A-Za-z0-9']+| ?[^\\sA-Za-z0-9']|\\n|\\s"
17
+ }
transformer/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:af9e4194367fcd16e40376eb9807fedcad6582605a95f1e35032f7e803411126
3
+ size 1118356
transformer/vocab.json ADDED
@@ -0,0 +1 @@
 
 
1
+ ["<unk>","\n",":",".","Assistant","User","?"," I"," a",","," the"," to"," is"," it"," and"," you"," for"," What"," can"," Yes"," do"," need"," How"," your"," Do"," with"," or"," have"," Is"," my"," help"," on"," You","-"," like"," in"," Should"," are"," of"," It"," if"," Can"," them"," but"," at"," should"," No"," good"," about","\""," \""," long"," email"," water"," not"," they"," an"," too"," well"," before"," buy"," more"," me"," sleep"," that"," work"," Does"," by"," best"," time"," am"," don't"," use"," Try"," hours"," one"," helps"," make"," tea"," there"," Are"," better"," much"," dog"," Why"," avoid"," recommend"," Please"," Use"," light"," be"," healthy"," learn"," plants"," try"," minutes"," write"," card"," ticket"," pasta"," safe"," save"," two"," want"," any"," day"," drink"," gate"," get"," this"," Keep"," The"," as"," coffee"," favorite"," movie"," out"," soil","!"," keep"," so"," take"," does"," start"," stay"," Focus"," clear"," most"," A"," from"," short"," way","The"," When"," all"," check"," fix"," must"," often"," weather"," which"," energy"," find"," first"," just"," language"," might"," simple"," usually"," bread"," food"," income"," snack"," will"," I'm"," body"," feel"," few"," idea"," sun"," thanks"," Redemption"," Shawshank"," client"," enjoy"," highly"," name"," rated"," then"," up"," warm"," days"," discuss"," focus"," great"," other"," reasonable"," sure"," types","watch"," Check"," Enjoy"," We"," anyone"," bake"," book"," content"," grow"," low"," off"," quickly"," six"," specific"," stop"," I'd"," after"," garden"," hope"," love"," new"," rest"," schedules"," some"," tomatoes"," useful"," watering"," My"," Start"," Sun"," add"," budget"," change"," extra"," fine"," key"," monthly"," offer"," options"," phone"," temperature"," their"," weight","loving"," Cool"," cook"," daily"," has"," job"," line"," oil"," personal"," sugar"," than"," thrive"," tone"," travel"," apps"," bed"," dogs"," dough"," flour"," formal"," go"," high"," immediately"," instead"," late"," money"," morning"," night"," oven"," plan"," see"," takes"," wait"," works"," writing"," Great"," PM"," Will"," debt"," during"," every"," history"," hot"," okay"," practice"," sauce"," snacks"," standard"," subject"," Most"," They"," Where"," app"," books"," caffeine"," cold"," color"," deep"," degrees"," down"," generally"," home"," lose"," message"," next"," now"," replace"," room"," signature"," solution"," until"," when"," yeast"," 8"," Absolutely"," CrossFit"," Look"," Make"," adding"," blue"," butter"," casual"," cheese"," contact"," costs"," dry"," eat"," it's"," know"," levels"," listen"," may"," people"," read"," reading"," reduce"," small"," story"," struggling"," suggest"," today"," train"," very"," week"," workouts"," bad"," balanced"," based"," boil"," breathing"," build"," call"," cat"," cause"," chips"," details"," draft"," exercise"," five"," freeze"," gratitude"," handle"," increase"," later"," no"," nuts"," online"," overly"," professional"," respectful"," right"," slowly"," store"," stress"," study"," thank"," training","focused"," That"," also"," baking"," bring"," calm"," can't"," cannot"," classic"," clean"," diet"," doctor"," enough"," ensure"," evening"," exam"," expressing"," flavor"," free"," give"," going"," herbs"," laptop"," last"," least"," lot"," many"," meal"," only"," over"," paint"," provide"," quality"," recommendation"," relevant"," rent"," results"," revise"," running"," schedule"," stuck"," tasks"," tips"," vegetables"," wake"," 10"," 15"," 2"," 30"," Boil"," Consistency"," Did"," Morning"," You'll"," account"," adjust"," affect"," always"," anxiety"," assistant"," breaks"," bus"," capital"," cats"," choice"," comfortable"," door"," exercises"," fats"," follow"," fresh"," helping"," hour"," humans"," ideas"," into"," leave"," look"," makes"," nap"," necessary"," notes"," perfect"," preferences"," process"," recovery"," screws"," sessions"," sounds"," space"," strong"," talk"," taste"," trip"," walk","free","risk"," Aim"," Avoid"," Combine"," Consider"," Not"," Set"," Some"," Sure"," Take"," Tighten"," Turn"," Which"," access"," afternoon"," answer"," around"," back"," basic"," battery"," brain"," cardio"," crust"," cut"," data"," depends"," directly"," dollars"," drain"," early"," eight"," essential"," expect"," garlic"," gets"," green"," gym"," heat"," hydrated"," hydration"," improve"," intense"," join"," learning"," leaves"," let"," life"," loaf"," local"," looks"," main"," milk"," music"," natural"," noise"," organize"," planning"," plant"," popular"," power"," pre"," prepare"," pressure"," prevent"," recipe"," reduces"," relax"," ride"," rule"," say"," search"," send"," shop"," side"," sky"," slightly"," something"," speak"," spicy"," stable"," strength"," sufficient"," sunny"," tell"," text"," top"," using"," vet"," we"," without"," yes","Inception","Sapiens"," Adjust"," An"," Bring"," Budget"," But"," Cancel"," Hydrate"," Late"," Python"," There"," This"," Too"," afterward"," animals"," anything"," average"," awards"," band"," bank"," bath"," believe"," budgets"," car"," clearly"," consistent"," create"," damage"," distances"," drive"," eaten"," engine"," even"," famous"," fan"," fast"," finances"," finish"," flexibility"," flexible"," frozen"," gear"," genre"," gluten"," goal"," group"," habit"," happens"," hundred"," information"," intake"," interval"," irregular"," kind"," leaky"," less"," limit"," looking"," loose"," loud"," machine"," masterpiece"," matter"," meat"," needs"," novels"," open"," pay"," pipe"," place"," plot"," prevents"," print"," pull"," rain"," recipes"," recommendations"," require"," reset"," review"," reviewing"," rise"," safer"," salt"," scales"," season"," sell"," services"," stock"," stressed"," stretch"," stretches"," support"," test"," though"," tired"," total"," touch"," trying"," turn"," under"," unused"," varying"," wear"," widely"," wind"," won"," workout"," years"," you've"," yourself","Maus","minute","time",{"hex":"c2"}," 12"," 5"," AI"," Apply"," Call"," Could"," For"," Good"," Happy"," If"," Invest"," It's"," Let"," Only"," Replace"," Tea"," activity"," adds"," adjustable"," airtight"," allow"," amazing"," animal"," another"," apples"," apply"," art"," assist"," available"," behavior"," bills"," blood"," break"," brew"," broth"," brush"," buying"," calming"," carefully"," causes"," checking"," chocolate"," clay"," cloud"," common"," complex"," container"," conversation"," dark"," date"," describe"," dish"," drains"," drop"," each"," easier"," easily"," else"," enjoyable"," faucet"," favorites"," feelings"," filter"," fit"," foods"," footwear"," force"," forecast"," freelance"," fridge"," future"," gardening"," general"," getting"," glasses"," grammar"," grams"," graphic"," grinder"," guitar"," happy"," heavy"," herbal"," hinges"," hire"," how"," hurt"," important"," interested"," its"," jacket"," joining"," juice"," kids"," lasagna"," layers"," lemon"," lessons"," little"," memoir"," microwave"," month"," months"," move"," negotiate"," offers"," overview"," pain"," pan"," part"," peanut"," percent"," physical"," pillow"," pine"," pot"," pro"," problem"," problems"," protein"," provides"," put"," questions"," rack"," rainy"," rate"," resistance"," salary"," salted"," sandy"," savings"," sci"," service"," sheets"," shoes"," sink"," solve"," song"," taking"," tastes"," taxi"," teacher"," team"," texture"," think"," tighten"," tomato"," topics"," toy"," type"," videos"," where"," while"," whole"," window"," wood","fi","inch","interest",{"hex":"c3"}," 100"," 11"," 20"," 25"," About"," Add"," America"," Ask"," Begin"," Carlo"," Cipolla"," Create"," Deep"," Definitely"," Digital"," Drink"," Early"," Eat"," Ensure"," France"," G"," Gatsby"," George"," Grapes"," Harari"," History"," Just"," Let's"," Listen"," Many"," Mix"," Noah"," Orwell"," Place"," Practice"," Prices"," Probably"," Rest"," Schedule"," Stop"," Tara"," Usually"," Westover"," Your"," Yuval"," action"," active"," address"," antioxidants"," anytime"," apple"," areas"," ask"," atmosphere"," audio"," away"," bars"," become"," been"," beginners"," benefits"," birds"," blade"," blanket"," blockage"," boat"," boost"," both"," bottom"," broken"," bulb"," business"," cabinet"," carbonara"," ceiling"," center"," changing"," city"," colors"," compelling"," consistency"," consistently"," cookbooks"," cool"," correct"," cost"," could"," cover"," covers"," crumb"," current"," dangerous"," depending"," device"," different"," dinner"," doors"," dried"," dystopian"," ear"," easy"," edges"," egg"," eggs"," end"," equipment"," error"," everything"," exact"," faster"," features"," feels"," fiction"," fifteen"," final"," fixed"," fixes"," floor"," foil"," frame"," fully"," fun"," gently"," greeting"," grocery"," growth"," hair"," hard"," health"," hear"," helpful"," here"," hiking"," hobbies"," hold"," holds"," hotel"," ice"," ideal"," inch"," include"," ingredients"," instant"," issue"," keeps"," languages"," latch"," leash"," leg"," likely"," literary"," live"," lost"," lower"," lubricant"," making"," meaning"," means"," medicine"," mild"," miss"," mix"," model"," models"," modern"," moisture"," motivated"," myself"," near"," needed"," never"," nice"," non"," noodles"," normal"," novel"," ocean"," office"," olive"," ones"," option"," orange"," outside"," overwhelmed"," own"," pack"," painting"," parmesan"," parts"," per"," pet"," piano"," places"," positive"," posts"," prefer"," prison"," quick"," raw"," ready"," real"," recommended"," refrigerator"," regarded"," regularly"," relaxation"," retention"," rewarding"," rising"," role"," routine"," run"," safety"," sauces"," scatters"," screen"," screens"," seal"," self"," shape"," share"," shower"," sick"," simply"," sit"," site"," sleeping"," sleepy"," snoring"," social"," soothing"," sound"," soup"," speaking"," special"," spinach"," squeaky"," stamp"," starter"," starting"," sticky"," stomach"," stops"," stores"," structure"," studying"," substitute"," sugary"," summarize"," sunlight"," supplements"," switch"," system"," systems"," talking"," tap"," ten"," thing"," three"," tickets"," tight"," tool"," tools"," treat"," trust"," unclog"," understand"," understanding"," vary"," vinegar"," virtual"," visit"," walking"," was"," washer"," watch"," wax"," web"," weekly"," weeks"," wet"," wheat"," wild"," working"," worth"," yogurt","10","1491","1984",";","A","Columbian","Educated","Hello","fiction",{"hex":"83"}," 1503"," 3"," 4"," 45"," 500"," 5GHz"," 75"," 9"," Agatha"," Alex"," Arborio"," Art"," Because"," Bill"," Blend"," Blue"," Both"," Bryson"," Buy"," Carbonara"," Christie"," Clean"," Coffee"," Complete"," Consult"," Cover"," Download"," Drinking"," Dry"," Especially"," Everything"," F"," Fahrenheit"," Frank"," Fresh"," Generally"," Get"," Give"," Green"," Herbal"," Herbert"," Home"," Ideally"," Japan"," JavaScript"," Kyoto"," Layer"," Leonardo"," Lisa"," Lying"," Matrix"," Michaelides"," Mona"," Nearly"," Often"," Overwatering"," Paris"," Patient"," Perhaps"," Pick"," Read"," Reading"," Review"," Short"," Silent"," Simmer"," Skip"," Sleeping"," Spain"," Spanish"," Spend"," Spiegelman"," Stevia"," Store"," Street"," Summer"," Sunlight"," That's"," Then"," Things"," Vinci"," Visualize"," Wait"," Who"," Wild"," Write"," Yogurt"," above"," accept"," accurate"," acid"," actually"," adults"," advice"," ahead"," air"," airflow"," airport"," alcohol"," alert"," alfredo"," allergy"," almond"," along"," amount"," area"," aromatics"," artificial"," asking"," asleep"," assisting"," attention"," avoiding"," awake"," bag"," bags"," barking"," basil"," beaches"," because"," bedtime"," behaviors"," behind"," between"," bit"," bitter"," blades"," bleeding"," blog"," brand"," breadcrumbs"," breakfast"," brighter"," brown"," brushing"," bubbles"," buffer"," burst"," cabinets"," calories"," canned"," canvas"," careful"," carrots"," cash"," charge"," chemicals"," chili"," choose"," classics"," classmate"," cleaner"," cleaning"," clips"," clock"," clog"," clutter"," coast"," coat"," code"," coding"," computer"," concepts"," confidence"," connect"," connection"," conserve"," considered"," copies"," cream"," creamy"," created"," creates"," creative"," credit"," crowded"," cup"," da"," damaged"," deal"," delayed"," dense"," dietitian"," difficult"," digestion"," digital"," direct"," directory"," disease"," dishes"," dissolve"," doesn't"," doneness"," driving"," dryer"," due"," duration"," dying"," effective"," energetic"," errors"," erythritol"," escapes"," espresso"," eventually"," everyone"," exciting"," exist"," expenses"," expensive"," experience"," fare"," fatigue"," fear"," feed"," fetch"," files"," filling"," financial"," finding"," flat"," fluid"," fly"," folder"," four"," fruit"," full"," function"," funds"," gain"," gaming"," gas"," generate"," generic"," gentle"," glass"," goals"," goes"," goodbye"," gradually"," grapes"," gritty"," grounds"," gum"," hacker"," happiness"," hardest"," higher"," him"," hobby"," honey"," horror"," house"," human"," including"," inside"," insomnia"," install"," instincts"," instructions"," interesting"," internal"," internet"," involves"," isn't"," issues"," knead"," knife"," knob"," knobs"," knowledge"," laptops"," large"," lead"," leak"," leaks"," leftover"," leftovers"," legal"," length"," license"," limiting"," literature"," location"," longer"," loss"," loudly"," loved"," lowers"," lyrics"," math"," mattress"," mean"," media"," meditation"," memorize"," memory"," mentally"," messy"," midnight"," mind"," mindfulness"," mistakes"," mixture"," moderate"," mop"," mornings"," motor"," mountains"," muscle"," museums"," mystery"," naps"," nature"," nearest"," niche"," nightly"," noises"," offices"," older"," operating"," oregano"," organized"," organizing"," original"," package"," page"," paid"," painted"," paints"," panel"," paper"," pass"," passport"," persist"," pesto"," pets"," philosophical"," picture"," pip"," plenty"," plunger"," polite"," poor"," popcorn"," powerful"," preheating"," prep"," prescription"," previous"," price"," progress"," pronunciation"," proper"," properly"," proposal"," push"," quiet"," quite"," quiz"," rained"," reality"," recall"," red"," refreshes"," rely"," remember"," remove"," repeat"," required"," reservation"," response"," restaurant"," resume"," retire"," return"," rice"," ringing"," rinse"," risk"," risky"," risotto"," roots"," route"," router"," ruin"," rules"," salad"," same"," sandwich"," savory"," science"," scratching"," screenplay"," screw"," sealant"," seasoning"," seconds"," secure"," seeds"," seeing"," serve"," serving"," seven"," several"," shakes"," shampoo"," shared"," shorter"," showerhead"," sign"," significantly"," skies"," skillet"," skin"," skip"," slow"," smart"," snake"," soap"," solving"," soups"," sourdough"," spaghetti"," spinal"," splash"," spot"," spreadsheet"," stain"," started"," station"," stays"," steam"," steep"," steps"," sticking"," still"," stones"," strain"," street"," stronger"," stuffing"," subway"," success"," suggestions"," summer"," sunset"," sunshine"," supply"," surgery"," sweeteners"," symptoms"," technician"," tender"," textbook"," these"," things"," thriller"," through"," tie"," times"," tire"," toaster"," together"," topic"," touching"," towel"," toxic"," toys"," tracker"," tree"," trick"," trim"," turning"," turns"," tutor"," tutorial"," twenty"," uncovered"," unsafe"," unsure"," updates"," users"," vacation"," vacuum"," vocabulary"," waking"," wall"," warmth"," wash"," weekend"," weekends"," weighted"," what"," white"," who"," won't"," wooden"," words"," worry"," would"," year"," yet"," you'll"," you're","+","Dune","R","Where","line","s","up",{"hex":"a9"}]