Add transformer variant, numpy loader for both models, link to source code
Browse files- README.md +45 -37
- asemllm.py +48 -13
- model.safetensors +1 -1
- transformer/config.json +17 -0
- transformer/model.safetensors +3 -0
- transformer/vocab.json +1 -0
README.md
CHANGED
|
@@ -13,25 +13,23 @@ pipeline_tag: text-generation
|
|
| 13 |
|
| 14 |
# AsemLLM
|
| 15 |
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
|
| 20 |
-
> **
|
| 21 |
-
>
|
| 22 |
-
>
|
| 23 |
|
| 24 |
-
##
|
| 25 |
|
| 26 |
-
| | |
|
| 27 |
-
|---|---|
|
| 28 |
-
|
|
| 29 |
-
|
|
| 30 |
-
| Embedding / hidden size | 24 / 64 (tanh) |
|
| 31 |
-
| Vocabulary | 1631 tokens (words with their leading space, punctuation, newline; `<unk>` = id 0) |
|
| 32 |
-
| Parameters | 152,903 |
|
| 33 |
|
| 34 |
-
|
|
|
|
| 35 |
|
| 36 |
## Use it
|
| 37 |
|
|
@@ -39,51 +37,61 @@ This repository holds the trained weights, the vocabulary and a small numpy refe
|
|
| 39 |
pip install numpy safetensors huggingface_hub
|
| 40 |
huggingface-cli download asemqaffaf/AsemLLM --local-dir AsemLLM
|
| 41 |
cd AsemLLM
|
| 42 |
-
python asemllm.py --chat "What is a good movie?"
|
| 43 |
-
python asemllm.py
|
| 44 |
```
|
| 45 |
|
| 46 |
```python
|
| 47 |
from asemllm import AsemLLM
|
| 48 |
-
m = AsemLLM(".")
|
| 49 |
print(m.chat([{"role": "user", "content": "Hello!"}], temperature=0.5, seed=1))
|
| 50 |
```
|
| 51 |
|
| 52 |
-
|
| 53 |
-
|
| 54 |
|
| 55 |
## Training data
|
| 56 |
|
| 57 |
About 34,904 tokens (138 KB) of short, synthetic `User:` / `Assistant:` conversations on everyday topics
|
| 58 |
(cooking, gardening, travel, study, hobbies, ...), written by a locally run 4B-parameter instruction model.
|
| 59 |
-
No web scraping, no personal data.
|
| 60 |
-
|
| 61 |
-
mistakes of its source.
|
| 62 |
|
| 63 |
-
## Training
|
| 64 |
|
| 65 |
-
- Word
|
| 66 |
-
-
|
| 67 |
-
- Data-parallel training on one Apple M1 Pro (10 Erlang worker processes), about 13 minutes.
|
| 68 |
-
- Held-out set: the last 10% of the text.
|
| 69 |
|
| 70 |
## Evaluation (held-out cross-entropy per token, lower is better)
|
| 71 |
|
| 72 |
| Model | Loss | Perplexity |
|
| 73 |
|---|---|---|
|
| 74 |
-
| Unigram
|
| 75 |
-
| Bigram (previous word) | 3.98 | 53 |
|
| 76 |
-
| **AsemLLM** | **3.73** | **42** |
|
| 77 |
|
| 78 |
-
|
| 79 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 80 |
|
| 81 |
## Limitations
|
| 82 |
|
| 83 |
-
- Tiny
|
| 84 |
-
-
|
| 85 |
-
-
|
| 86 |
-
- Not suitable for any real application. It is meant for learning how language models work.
|
| 87 |
|
| 88 |
## License
|
| 89 |
|
|
|
|
| 13 |
|
| 14 |
# AsemLLM
|
| 15 |
|
| 16 |
+
Tiny word-level language models written, trained and served entirely in **Erlang**, with no deep-learning framework:
|
| 17 |
+
supervised worker processes compute gradients in parallel. This repository holds the trained weights, the vocabulary and a
|
| 18 |
+
small numpy reference loader. **Source code (training, serving, web UI): https://github.com/asemqaffaf/AsemLLms**
|
| 19 |
|
| 20 |
+
> **These are toy models.** They learn the *style* of their training text (short `User:` / `Assistant:` dialogue) and
|
| 21 |
+
> produce fluent-looking English, but they do **not** understand questions, and answers are usually unrelated to them.
|
| 22 |
+
> They can state false things confidently. Do not use them for anything that needs correct answers.
|
| 23 |
|
| 24 |
+
## Two variants
|
| 25 |
|
| 26 |
+
| | Where | Parameters | Window | Architecture |
|
| 27 |
+
|---|---|---|---|---|
|
| 28 |
+
| MLP | repository root | 152,903 | 5 tokens | embedding -> tanh hidden layer -> softmax (Bengio-style) |
|
| 29 |
+
| Transformer | `transformer/` | 279,007 | 32 tokens | GPT-style decoder (attention) |
|
|
|
|
|
|
|
|
|
|
| 30 |
|
| 31 |
+
Both use the same tokenizer: words with their leading space, punctuation marks and newlines; the 1631-token vocabulary
|
| 32 |
+
(top tokens seen at least twice, `<unk>` = id 0) is in `vocab.json`.
|
| 33 |
|
| 34 |
## Use it
|
| 35 |
|
|
|
|
| 37 |
pip install numpy safetensors huggingface_hub
|
| 38 |
huggingface-cli download asemqaffaf/AsemLLM --local-dir AsemLLM
|
| 39 |
cd AsemLLM
|
| 40 |
+
python asemllm.py --chat "What is a good movie?" # MLP
|
| 41 |
+
python asemllm.py --model transformer --chat "What is a good movie?" # transformer
|
| 42 |
```
|
| 43 |
|
| 44 |
```python
|
| 45 |
from asemllm import AsemLLM
|
| 46 |
+
m = AsemLLM(".") # or AsemLLM("transformer")
|
| 47 |
print(m.chat([{"role": "user", "content": "Hello!"}], temperature=0.5, seed=1))
|
| 48 |
```
|
| 49 |
|
| 50 |
+
They use custom architectures, so they do **not** load in `transformers`, llama.cpp, Ollama or LM Studio.
|
| 51 |
+
Weights are float32 safetensors; `asemllm.py` is the reference implementation (verified against the Erlang code to ~1e-7 in logits).
|
| 52 |
|
| 53 |
## Training data
|
| 54 |
|
| 55 |
About 34,904 tokens (138 KB) of short, synthetic `User:` / `Assistant:` conversations on everyday topics
|
| 56 |
(cooking, gardening, travel, study, hobbies, ...), written by a locally run 4B-parameter instruction model.
|
| 57 |
+
No web scraping, no personal data. Text was filtered to well-formed dialogue lines and truncated generations were dropped.
|
| 58 |
+
Being machine-written, the data carries the repetitiveness and the mistakes of its source.
|
|
|
|
| 59 |
|
| 60 |
+
## Training (MLP)
|
| 61 |
|
| 62 |
+
- Word tokenizer, Adam (lr 0.005, linear decay to 10%), batch 128, gradient clipping 5, 3,000 steps; best-validation checkpoint kept.
|
| 63 |
+
- Data-parallel training on one Apple M1 Pro (10 Erlang worker processes), about 13 minutes. Held-out set: last 10% of the text.
|
|
|
|
|
|
|
| 64 |
|
| 65 |
## Evaluation (held-out cross-entropy per token, lower is better)
|
| 66 |
|
| 67 |
| Model | Loss | Perplexity |
|
| 68 |
|---|---|---|
|
| 69 |
+
| Unigram baseline | 5.14 | 171 |
|
| 70 |
+
| Bigram baseline (previous word) | 3.98 | 53 |
|
| 71 |
+
| **AsemLLM MLP** | **3.73** | **42** |
|
| 72 |
|
| 73 |
+
### Transformer variant (`transformer/`)
|
| 74 |
+
|
| 75 |
+
A small GPT-style decoder: **279,007 parameters**, 2 layers x 4 heads, model dim 64
|
| 76 |
+
(MLP width 128), learned positions, pre-LayerNorm, causal attention over a 32-token window,
|
| 77 |
+
trained on the same data. Unlike the MLP it sees the whole prompt (up to 32 tokens).
|
| 78 |
+
|
| 79 |
+
| Held-out loss | Value | Perplexity |
|
| 80 |
+
|---|---|---|
|
| 81 |
+
| Transformer (best checkpoint of 2,500 steps) | 3.65 | 38 |
|
| 82 |
+
|
| 83 |
+
The transformer overfits quickly on 138 KB: training loss kept falling while validation loss bottomed out early and
|
| 84 |
+
rose again, so the saved weights are the best validation checkpoint, not the last step. Its loss is averaged over every
|
| 85 |
+
position of held-out 32-token windows (including positions with little context), so it is **not directly comparable**
|
| 86 |
+
to the MLP's single-next-token number.
|
| 87 |
+
|
| 88 |
+
With only 138 KB of text these models largely memorize phrases (MLP training loss fell to about 1.3 while validation loss stopped improving at 3.73).
|
| 89 |
|
| 90 |
## Limitations
|
| 91 |
|
| 92 |
+
- Tiny data and tiny models: replies mostly do not relate to the prompt. English only; ASCII-centric vocabulary.
|
| 93 |
+
- No safety training; output reflects its machine-written training text.
|
| 94 |
+
- Not suitable for any real application. It exists to show how language models work, end to end, in Erlang.
|
|
|
|
| 95 |
|
| 96 |
## License
|
| 97 |
|
asemllm.py
CHANGED
|
@@ -1,7 +1,8 @@
|
|
| 1 |
-
"""AsemLLM reference inference: numpy + safetensors only.
|
| 2 |
|
| 3 |
-
python asemllm.py
|
| 4 |
-
python asemllm.py --chat "
|
|
|
|
| 5 |
"""
|
| 6 |
import json, re, sys
|
| 7 |
from pathlib import Path
|
|
@@ -11,30 +12,61 @@ from safetensors.numpy import load_file
|
|
| 11 |
TOKEN_RE = re.compile(rb" ?[A-Za-z0-9']+| ?[^\sA-Za-z0-9']|\n|\s")
|
| 12 |
|
| 13 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
class AsemLLM:
|
| 15 |
def __init__(self, path="."):
|
| 16 |
p = Path(path)
|
| 17 |
self.cfg = json.loads((p / "config.json").read_text())
|
| 18 |
-
w = load_file(str(p / "model.safetensors"))
|
| 19 |
-
self.E, self.W1, self.b1, self.W2, self.b2 = (
|
| 20 |
-
w[k].astype(np.float64) for k in ("embedding", "w1", "b1", "w2", "b2"))
|
| 21 |
raw = json.loads((p / "vocab.json").read_text())
|
| 22 |
self.tokens = [bytes.fromhex(t["hex"]) if isinstance(t, dict) else t.encode() for t in raw]
|
| 23 |
self.index = {t: i for i, t in enumerate(self.tokens)}
|
| 24 |
self.n = self.cfg["context_length"]
|
|
|
|
| 25 |
self.pad = self.index.get(b"\n", 0)
|
| 26 |
|
| 27 |
def logits(self, ctx_ids):
|
| 28 |
-
"""Next-token logits
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
|
| 33 |
def generate(self, prompt, max_tokens=100, temperature=0.7, seed=None, stop=()):
|
| 34 |
"""Continue `prompt` (str). Returns the continuation. Never emits <unk>."""
|
| 35 |
rng = np.random.default_rng(seed)
|
| 36 |
ids = [self.index.get(t, 0) for t in TOKEN_RE.findall(prompt.encode())]
|
| 37 |
-
|
|
|
|
|
|
|
|
|
|
| 38 |
stops = [s.encode() for s in stop]
|
| 39 |
out = b""
|
| 40 |
for _ in range(max_tokens):
|
|
@@ -44,7 +76,7 @@ class AsemLLM:
|
|
| 44 |
p = np.exp(z - z.max())
|
| 45 |
nxt = int(rng.choice(len(p), p=p / p.sum()))
|
| 46 |
out += self.tokens[nxt]
|
| 47 |
-
ctx = ctx
|
| 48 |
cut = [out.find(s) for s in stops if s in out]
|
| 49 |
if cut:
|
| 50 |
out = out[:min(cut)]
|
|
@@ -60,7 +92,10 @@ class AsemLLM:
|
|
| 60 |
|
| 61 |
if __name__ == "__main__":
|
| 62 |
args = sys.argv[1:]
|
| 63 |
-
|
|
|
|
|
|
|
|
|
|
| 64 |
if args and args[0] == "--chat":
|
| 65 |
print(model.chat([{"role": "user", "content": " ".join(args[1:])}]))
|
| 66 |
else:
|
|
|
|
| 1 |
+
"""AsemLLM reference inference: numpy + safetensors only. Runs both model variants.
|
| 2 |
|
| 3 |
+
python asemllm.py --chat "What is a good movie?" # MLP model (this folder)
|
| 4 |
+
python asemllm.py --model transformer --chat "What is a good movie?"
|
| 5 |
+
python asemllm.py --model transformer $'User: How do I cook pasta?\\nAssistant:' # $'..' = real newline
|
| 6 |
"""
|
| 7 |
import json, re, sys
|
| 8 |
from pathlib import Path
|
|
|
|
| 12 |
TOKEN_RE = re.compile(rb" ?[A-Za-z0-9']+| ?[^\sA-Za-z0-9']|\n|\s")
|
| 13 |
|
| 14 |
|
| 15 |
+
def _ln(x, g, b):
|
| 16 |
+
mu = x.mean(-1, keepdims=True)
|
| 17 |
+
var = ((x - mu) ** 2).mean(-1, keepdims=True)
|
| 18 |
+
return (x - mu) / np.sqrt(var + 1e-5) * g + b
|
| 19 |
+
|
| 20 |
+
|
| 21 |
class AsemLLM:
|
| 22 |
def __init__(self, path="."):
|
| 23 |
p = Path(path)
|
| 24 |
self.cfg = json.loads((p / "config.json").read_text())
|
| 25 |
+
self.w = {k: v.astype(np.float64) for k, v in load_file(str(p / "model.safetensors")).items()}
|
|
|
|
|
|
|
| 26 |
raw = json.loads((p / "vocab.json").read_text())
|
| 27 |
self.tokens = [bytes.fromhex(t["hex"]) if isinstance(t, dict) else t.encode() for t in raw]
|
| 28 |
self.index = {t: i for i, t in enumerate(self.tokens)}
|
| 29 |
self.n = self.cfg["context_length"]
|
| 30 |
+
self.arch = self.cfg["architecture"]
|
| 31 |
self.pad = self.index.get(b"\n", 0)
|
| 32 |
|
| 33 |
def logits(self, ctx_ids):
|
| 34 |
+
"""Next-token logits. MLP: exactly `context_length` ids. Transformer: 1..`context_length` ids."""
|
| 35 |
+
w = self.w
|
| 36 |
+
if self.arch == "transformer":
|
| 37 |
+
return self._transformer_logits(ctx_ids)
|
| 38 |
+
x = np.concatenate([w["embedding"][i] for i in ctx_ids])
|
| 39 |
+
return w["w2"] @ np.tanh(w["w1"] @ x + w["b1"]) + w["b2"]
|
| 40 |
+
|
| 41 |
+
def _transformer_logits(self, ids):
|
| 42 |
+
w, c = self.w, self.cfg
|
| 43 |
+
T, D, H = len(ids), c["dim"], c["heads"]
|
| 44 |
+
dh = D // H
|
| 45 |
+
x = w["embedding"][ids] + w["positional"][:T]
|
| 46 |
+
mask = np.triu(np.full((T, T), -np.inf), k=1) # causal: position t sees only s <= t
|
| 47 |
+
for i in range(c["layers"]):
|
| 48 |
+
g = lambda k: w[f"layers.{i}.{k}"]
|
| 49 |
+
xn = _ln(x, g("ln1_g"), g("ln1_b"))
|
| 50 |
+
q, k_, v = (xn @ g(n).T for n in ("wq", "wk", "wv"))
|
| 51 |
+
split = lambda m: m.reshape(T, H, dh).transpose(1, 0, 2) # (heads, T, dh)
|
| 52 |
+
s = split(q) @ split(k_).transpose(0, 2, 1) / np.sqrt(dh) + mask
|
| 53 |
+
a = np.exp(s - s.max(-1, keepdims=True))
|
| 54 |
+
a /= a.sum(-1, keepdims=True)
|
| 55 |
+
ctx = (a @ split(v)).transpose(1, 0, 2).reshape(T, D)
|
| 56 |
+
x = x + ctx @ g("wo").T
|
| 57 |
+
xn = _ln(x, g("ln2_g"), g("ln2_b"))
|
| 58 |
+
x = x + np.maximum(xn @ g("w1").T + g("b1"), 0.0) @ g("w2").T + g("b2")
|
| 59 |
+
xf = _ln(x[-1], w["lnf_g"], w["lnf_b"])
|
| 60 |
+
return w["wout"] @ xf + w["bout"]
|
| 61 |
|
| 62 |
def generate(self, prompt, max_tokens=100, temperature=0.7, seed=None, stop=()):
|
| 63 |
"""Continue `prompt` (str). Returns the continuation. Never emits <unk>."""
|
| 64 |
rng = np.random.default_rng(seed)
|
| 65 |
ids = [self.index.get(t, 0) for t in TOKEN_RE.findall(prompt.encode())]
|
| 66 |
+
if self.arch == "transformer":
|
| 67 |
+
ctx = (ids or [self.pad])[-self.n:]
|
| 68 |
+
else:
|
| 69 |
+
ctx = ([self.pad] * max(0, self.n - len(ids)) + ids)[-self.n:]
|
| 70 |
stops = [s.encode() for s in stop]
|
| 71 |
out = b""
|
| 72 |
for _ in range(max_tokens):
|
|
|
|
| 76 |
p = np.exp(z - z.max())
|
| 77 |
nxt = int(rng.choice(len(p), p=p / p.sum()))
|
| 78 |
out += self.tokens[nxt]
|
| 79 |
+
ctx = (ctx + [nxt])[-self.n:]
|
| 80 |
cut = [out.find(s) for s in stops if s in out]
|
| 81 |
if cut:
|
| 82 |
out = out[:min(cut)]
|
|
|
|
| 92 |
|
| 93 |
if __name__ == "__main__":
|
| 94 |
args = sys.argv[1:]
|
| 95 |
+
sub = "."
|
| 96 |
+
if args[:1] == ["--model"]:
|
| 97 |
+
sub, args = args[1], args[2:]
|
| 98 |
+
model = AsemLLM(Path(__file__).parent / sub)
|
| 99 |
if args and args[0] == "--chat":
|
| 100 |
print(model.chat([{"role": "user", "content": " ".join(args[1:])}]))
|
| 101 |
else:
|
model.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 612004
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:56b328e0960f128bae87565f3c840b5e174f8424af5bb8a227c65db62f83e742
|
| 3 |
size 612004
|
transformer/config.json
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model_type": "asemllm",
|
| 3 |
+
"architecture": "transformer",
|
| 4 |
+
"vocab_size": 1631,
|
| 5 |
+
"context_length": 32,
|
| 6 |
+
"dim": 64,
|
| 7 |
+
"heads": 4,
|
| 8 |
+
"layers": 2,
|
| 9 |
+
"ff_dim": 128,
|
| 10 |
+
"activation": "relu",
|
| 11 |
+
"norm": "pre-layernorm (eps 1e-5)",
|
| 12 |
+
"positional": "learned",
|
| 13 |
+
"tokenizer": "word",
|
| 14 |
+
"unk_token_id": 0,
|
| 15 |
+
"num_parameters": 279007,
|
| 16 |
+
"tokenizer_regex": " ?[A-Za-z0-9']+| ?[^\\sA-Za-z0-9']|\\n|\\s"
|
| 17 |
+
}
|
transformer/model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:af9e4194367fcd16e40376eb9807fedcad6582605a95f1e35032f7e803411126
|
| 3 |
+
size 1118356
|
transformer/vocab.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
["<unk>","\n",":",".","Assistant","User","?"," I"," a",","," the"," to"," is"," it"," and"," you"," for"," What"," can"," Yes"," do"," need"," How"," your"," Do"," with"," or"," have"," Is"," my"," help"," on"," You","-"," like"," in"," Should"," are"," of"," It"," if"," Can"," them"," but"," at"," should"," No"," good"," about","\""," \""," long"," email"," water"," not"," they"," an"," too"," well"," before"," buy"," more"," me"," sleep"," that"," work"," Does"," by"," best"," time"," am"," don't"," use"," Try"," hours"," one"," helps"," make"," tea"," there"," Are"," better"," much"," dog"," Why"," avoid"," recommend"," Please"," Use"," light"," be"," healthy"," learn"," plants"," try"," minutes"," write"," card"," ticket"," pasta"," safe"," save"," two"," want"," any"," day"," drink"," gate"," get"," this"," Keep"," The"," as"," coffee"," favorite"," movie"," out"," soil","!"," keep"," so"," take"," does"," start"," stay"," Focus"," clear"," most"," A"," from"," short"," way","The"," When"," all"," check"," fix"," must"," often"," weather"," which"," energy"," find"," first"," just"," language"," might"," simple"," usually"," bread"," food"," income"," snack"," will"," I'm"," body"," feel"," few"," idea"," sun"," thanks"," Redemption"," Shawshank"," client"," enjoy"," highly"," name"," rated"," then"," up"," warm"," days"," discuss"," focus"," great"," other"," reasonable"," sure"," types","watch"," Check"," Enjoy"," We"," anyone"," bake"," book"," content"," grow"," low"," off"," quickly"," six"," specific"," stop"," I'd"," after"," garden"," hope"," love"," new"," rest"," schedules"," some"," tomatoes"," useful"," watering"," My"," Start"," Sun"," add"," budget"," change"," extra"," fine"," key"," monthly"," offer"," options"," phone"," temperature"," their"," weight","loving"," Cool"," cook"," daily"," has"," job"," line"," oil"," personal"," sugar"," than"," thrive"," tone"," travel"," apps"," bed"," dogs"," dough"," flour"," formal"," go"," high"," immediately"," instead"," late"," money"," morning"," night"," oven"," plan"," see"," takes"," wait"," works"," writing"," Great"," PM"," Will"," debt"," during"," every"," history"," hot"," okay"," practice"," sauce"," snacks"," standard"," subject"," Most"," They"," Where"," app"," books"," caffeine"," cold"," color"," deep"," degrees"," down"," generally"," home"," lose"," message"," next"," now"," replace"," room"," signature"," solution"," until"," when"," yeast"," 8"," Absolutely"," CrossFit"," Look"," Make"," adding"," blue"," butter"," casual"," cheese"," contact"," costs"," dry"," eat"," it's"," know"," levels"," listen"," may"," people"," read"," reading"," reduce"," small"," story"," struggling"," suggest"," today"," train"," very"," week"," workouts"," bad"," balanced"," based"," boil"," breathing"," build"," call"," cat"," cause"," chips"," details"," draft"," exercise"," five"," freeze"," gratitude"," handle"," increase"," later"," no"," nuts"," online"," overly"," professional"," respectful"," right"," slowly"," store"," stress"," study"," thank"," training","focused"," That"," also"," baking"," bring"," calm"," can't"," cannot"," classic"," clean"," diet"," doctor"," enough"," ensure"," evening"," exam"," expressing"," flavor"," free"," give"," going"," herbs"," laptop"," last"," least"," lot"," many"," meal"," only"," over"," paint"," provide"," quality"," recommendation"," relevant"," rent"," results"," revise"," running"," schedule"," stuck"," tasks"," tips"," vegetables"," wake"," 10"," 15"," 2"," 30"," Boil"," Consistency"," Did"," Morning"," You'll"," account"," adjust"," affect"," always"," anxiety"," assistant"," breaks"," bus"," capital"," cats"," choice"," comfortable"," door"," exercises"," fats"," follow"," fresh"," helping"," hour"," humans"," ideas"," into"," leave"," look"," makes"," nap"," necessary"," notes"," perfect"," preferences"," process"," recovery"," screws"," sessions"," sounds"," space"," strong"," talk"," taste"," trip"," walk","free","risk"," Aim"," Avoid"," Combine"," Consider"," Not"," Set"," Some"," Sure"," Take"," Tighten"," Turn"," Which"," access"," afternoon"," answer"," around"," back"," basic"," battery"," brain"," cardio"," crust"," cut"," data"," depends"," directly"," dollars"," drain"," early"," eight"," essential"," expect"," garlic"," gets"," green"," gym"," heat"," hydrated"," hydration"," improve"," intense"," join"," learning"," leaves"," let"," life"," loaf"," local"," looks"," main"," milk"," music"," natural"," noise"," organize"," planning"," plant"," popular"," power"," pre"," prepare"," pressure"," prevent"," recipe"," reduces"," relax"," ride"," rule"," say"," search"," send"," shop"," side"," sky"," slightly"," something"," speak"," spicy"," stable"," strength"," sufficient"," sunny"," tell"," text"," top"," using"," vet"," we"," without"," yes","Inception","Sapiens"," Adjust"," An"," Bring"," Budget"," But"," Cancel"," Hydrate"," Late"," Python"," There"," This"," Too"," afterward"," animals"," anything"," average"," awards"," band"," bank"," bath"," believe"," budgets"," car"," clearly"," consistent"," create"," damage"," distances"," drive"," eaten"," engine"," even"," famous"," fan"," fast"," finances"," finish"," flexibility"," flexible"," frozen"," gear"," genre"," gluten"," goal"," group"," habit"," happens"," hundred"," information"," intake"," interval"," irregular"," kind"," leaky"," less"," limit"," looking"," loose"," loud"," machine"," masterpiece"," matter"," meat"," needs"," novels"," open"," pay"," pipe"," place"," plot"," prevents"," print"," pull"," rain"," recipes"," recommendations"," require"," reset"," review"," reviewing"," rise"," safer"," salt"," scales"," season"," sell"," services"," stock"," stressed"," stretch"," stretches"," support"," test"," though"," tired"," total"," touch"," trying"," turn"," under"," unused"," varying"," wear"," widely"," wind"," won"," workout"," years"," you've"," yourself","Maus","minute","time",{"hex":"c2"}," 12"," 5"," AI"," Apply"," Call"," Could"," For"," Good"," Happy"," If"," Invest"," It's"," Let"," Only"," Replace"," Tea"," activity"," adds"," adjustable"," airtight"," allow"," amazing"," animal"," another"," apples"," apply"," art"," assist"," available"," behavior"," bills"," blood"," break"," brew"," broth"," brush"," buying"," calming"," carefully"," causes"," checking"," chocolate"," clay"," cloud"," common"," complex"," container"," conversation"," dark"," date"," describe"," dish"," drains"," drop"," each"," easier"," easily"," else"," enjoyable"," faucet"," favorites"," feelings"," filter"," fit"," foods"," footwear"," force"," forecast"," freelance"," fridge"," future"," gardening"," general"," getting"," glasses"," grammar"," grams"," graphic"," grinder"," guitar"," happy"," heavy"," herbal"," hinges"," hire"," how"," hurt"," important"," interested"," its"," jacket"," joining"," juice"," kids"," lasagna"," layers"," lemon"," lessons"," little"," memoir"," microwave"," month"," months"," move"," negotiate"," offers"," overview"," pain"," pan"," part"," peanut"," percent"," physical"," pillow"," pine"," pot"," pro"," problem"," problems"," protein"," provides"," put"," questions"," rack"," rainy"," rate"," resistance"," salary"," salted"," sandy"," savings"," sci"," service"," sheets"," shoes"," sink"," solve"," song"," taking"," tastes"," taxi"," teacher"," team"," texture"," think"," tighten"," tomato"," topics"," toy"," type"," videos"," where"," while"," whole"," window"," wood","fi","inch","interest",{"hex":"c3"}," 100"," 11"," 20"," 25"," About"," Add"," America"," Ask"," Begin"," Carlo"," Cipolla"," Create"," Deep"," Definitely"," Digital"," Drink"," Early"," Eat"," Ensure"," France"," G"," Gatsby"," George"," Grapes"," Harari"," History"," Just"," Let's"," Listen"," Many"," Mix"," Noah"," Orwell"," Place"," Practice"," Prices"," Probably"," Rest"," Schedule"," Stop"," Tara"," Usually"," Westover"," Your"," Yuval"," action"," active"," address"," antioxidants"," anytime"," apple"," areas"," ask"," atmosphere"," audio"," away"," bars"," become"," been"," beginners"," benefits"," birds"," blade"," blanket"," blockage"," boat"," boost"," both"," bottom"," broken"," bulb"," business"," cabinet"," carbonara"," ceiling"," center"," changing"," city"," colors"," compelling"," consistency"," consistently"," cookbooks"," cool"," correct"," cost"," could"," cover"," covers"," crumb"," current"," dangerous"," depending"," device"," different"," dinner"," doors"," dried"," dystopian"," ear"," easy"," edges"," egg"," eggs"," end"," equipment"," error"," everything"," exact"," faster"," features"," feels"," fiction"," fifteen"," final"," fixed"," fixes"," floor"," foil"," frame"," fully"," fun"," gently"," greeting"," grocery"," growth"," hair"," hard"," health"," hear"," helpful"," here"," hiking"," hobbies"," hold"," holds"," hotel"," ice"," ideal"," inch"," include"," ingredients"," instant"," issue"," keeps"," languages"," latch"," leash"," leg"," likely"," literary"," live"," lost"," lower"," lubricant"," making"," meaning"," means"," medicine"," mild"," miss"," mix"," model"," models"," modern"," moisture"," motivated"," myself"," near"," needed"," never"," nice"," non"," noodles"," normal"," novel"," ocean"," office"," olive"," ones"," option"," orange"," outside"," overwhelmed"," own"," pack"," painting"," parmesan"," parts"," per"," pet"," piano"," places"," positive"," posts"," prefer"," prison"," quick"," raw"," ready"," real"," recommended"," refrigerator"," regarded"," regularly"," relaxation"," retention"," rewarding"," rising"," role"," routine"," run"," safety"," sauces"," scatters"," screen"," screens"," seal"," self"," shape"," share"," shower"," sick"," simply"," sit"," site"," sleeping"," sleepy"," snoring"," social"," soothing"," sound"," soup"," speaking"," special"," spinach"," squeaky"," stamp"," starter"," starting"," sticky"," stomach"," stops"," stores"," structure"," studying"," substitute"," sugary"," summarize"," sunlight"," supplements"," switch"," system"," systems"," talking"," tap"," ten"," thing"," three"," tickets"," tight"," tool"," tools"," treat"," trust"," unclog"," understand"," understanding"," vary"," vinegar"," virtual"," visit"," walking"," was"," washer"," watch"," wax"," web"," weekly"," weeks"," wet"," wheat"," wild"," working"," worth"," yogurt","10","1491","1984",";","A","Columbian","Educated","Hello","fiction",{"hex":"83"}," 1503"," 3"," 4"," 45"," 500"," 5GHz"," 75"," 9"," Agatha"," Alex"," Arborio"," Art"," Because"," Bill"," Blend"," Blue"," Both"," Bryson"," Buy"," Carbonara"," Christie"," Clean"," Coffee"," Complete"," Consult"," Cover"," Download"," Drinking"," Dry"," Especially"," Everything"," F"," Fahrenheit"," Frank"," Fresh"," Generally"," Get"," Give"," Green"," Herbal"," Herbert"," Home"," Ideally"," Japan"," JavaScript"," Kyoto"," Layer"," Leonardo"," Lisa"," Lying"," Matrix"," Michaelides"," Mona"," Nearly"," Often"," Overwatering"," Paris"," Patient"," Perhaps"," Pick"," Read"," Reading"," Review"," Short"," Silent"," Simmer"," Skip"," Sleeping"," Spain"," Spanish"," Spend"," Spiegelman"," Stevia"," Store"," Street"," Summer"," Sunlight"," That's"," Then"," Things"," Vinci"," Visualize"," Wait"," Who"," Wild"," Write"," Yogurt"," above"," accept"," accurate"," acid"," actually"," adults"," advice"," ahead"," air"," airflow"," airport"," alcohol"," alert"," alfredo"," allergy"," almond"," along"," amount"," area"," aromatics"," artificial"," asking"," asleep"," assisting"," attention"," avoiding"," awake"," bag"," bags"," barking"," basil"," beaches"," because"," bedtime"," behaviors"," behind"," between"," bit"," bitter"," blades"," bleeding"," blog"," brand"," breadcrumbs"," breakfast"," brighter"," brown"," brushing"," bubbles"," buffer"," burst"," cabinets"," calories"," canned"," canvas"," careful"," carrots"," cash"," charge"," chemicals"," chili"," choose"," classics"," classmate"," cleaner"," cleaning"," clips"," clock"," clog"," clutter"," coast"," coat"," code"," coding"," computer"," concepts"," confidence"," connect"," connection"," conserve"," considered"," copies"," cream"," creamy"," created"," creates"," creative"," credit"," crowded"," cup"," da"," damaged"," deal"," delayed"," dense"," dietitian"," difficult"," digestion"," digital"," direct"," directory"," disease"," dishes"," dissolve"," doesn't"," doneness"," driving"," dryer"," due"," duration"," dying"," effective"," energetic"," errors"," erythritol"," escapes"," espresso"," eventually"," everyone"," exciting"," exist"," expenses"," expensive"," experience"," fare"," fatigue"," fear"," feed"," fetch"," files"," filling"," financial"," finding"," flat"," fluid"," fly"," folder"," four"," fruit"," full"," function"," funds"," gain"," gaming"," gas"," generate"," generic"," gentle"," glass"," goals"," goes"," goodbye"," gradually"," grapes"," gritty"," grounds"," gum"," hacker"," happiness"," hardest"," higher"," him"," hobby"," honey"," horror"," house"," human"," including"," inside"," insomnia"," install"," instincts"," instructions"," interesting"," internal"," internet"," involves"," isn't"," issues"," knead"," knife"," knob"," knobs"," knowledge"," laptops"," large"," lead"," leak"," leaks"," leftover"," leftovers"," legal"," length"," license"," limiting"," literature"," location"," longer"," loss"," loudly"," loved"," lowers"," lyrics"," math"," mattress"," mean"," media"," meditation"," memorize"," memory"," mentally"," messy"," midnight"," mind"," mindfulness"," mistakes"," mixture"," moderate"," mop"," mornings"," motor"," mountains"," muscle"," museums"," mystery"," naps"," nature"," nearest"," niche"," nightly"," noises"," offices"," older"," operating"," oregano"," organized"," organizing"," original"," package"," page"," paid"," painted"," paints"," panel"," paper"," pass"," passport"," persist"," pesto"," pets"," philosophical"," picture"," pip"," plenty"," plunger"," polite"," poor"," popcorn"," powerful"," preheating"," prep"," prescription"," previous"," price"," progress"," pronunciation"," proper"," properly"," proposal"," push"," quiet"," quite"," quiz"," rained"," reality"," recall"," red"," refreshes"," rely"," remember"," remove"," repeat"," required"," reservation"," response"," restaurant"," resume"," retire"," return"," rice"," ringing"," rinse"," risk"," risky"," risotto"," roots"," route"," router"," ruin"," rules"," salad"," same"," sandwich"," savory"," science"," scratching"," screenplay"," screw"," sealant"," seasoning"," seconds"," secure"," seeds"," seeing"," serve"," serving"," seven"," several"," shakes"," shampoo"," shared"," shorter"," showerhead"," sign"," significantly"," skies"," skillet"," skin"," skip"," slow"," smart"," snake"," soap"," solving"," soups"," sourdough"," spaghetti"," spinal"," splash"," spot"," spreadsheet"," stain"," started"," station"," stays"," steam"," steep"," steps"," sticking"," still"," stones"," strain"," street"," stronger"," stuffing"," subway"," success"," suggestions"," summer"," sunset"," sunshine"," supply"," surgery"," sweeteners"," symptoms"," technician"," tender"," textbook"," these"," things"," thriller"," through"," tie"," times"," tire"," toaster"," together"," topic"," touching"," towel"," toxic"," toys"," tracker"," tree"," trick"," trim"," turning"," turns"," tutor"," tutorial"," twenty"," uncovered"," unsafe"," unsure"," updates"," users"," vacation"," vacuum"," vocabulary"," waking"," wall"," warmth"," wash"," weekend"," weekends"," weighted"," what"," white"," who"," won't"," wooden"," words"," worry"," would"," year"," yet"," you'll"," you're","+","Dune","R","Where","line","s","up",{"hex":"a9"}]
|