Upload folder using huggingface_hub

README.md

@@ -0,0 +1,76 @@
+---
+license: mit
+language: en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+  - gpt2
+  - tiny-lm
+  - educational
+datasets:
+  - HuggingFaceFW/fineweb-edu
+  - SimpleStories/SimpleStories
+  - databricks/databricks-dolly-15k
+---
+
+# tiny-lm-web-98m
+
+A 98M-parameter GPT-2 small shaped model, trained from scratch on web text. It is from Part 3 of
+[tiny-lm](https://github.com/lillycham/tiny-lm), a project that builds a language model step by
+step and then looks inside it.
+
+| | |
+|---|---|
+| Size | 12 layers, width 768, 12 heads, 98.4M parameters |
+| Context | 1,024 tokens |
+| Tokeniser | byte-level BPE, 16,384 tokens, trained on FineWeb-Edu |
+| Training | 2.70B tokens of FineWeb-Edu, 203 minutes on one RTX 5090 |
+| Validation loss | 2.983 nats per token, 1.03 bits per character |
+
+For comparison, OpenAI's GPT-2 small is about 1.04 bits per character on the same text.
+
+There are two models in this repository:
+
+- **The root** is the base model. It continues text.
+- **`chat/`** is the base model after a 150M-token anneal (25% SimpleStories), then supervised
+  fine-tuning on Dolly 15k, then fine-tuning on `System:` lines (rules such as "Answer in one
+  sentence", and names for the model and the user).
+
+The weights are a standard `GPT2LMHeadModel`. The q, k and v biases that GPT-2 has are zero,
+because the model was trained without them.
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+repo = "lillycham/tiny-lm-web-98m"
+tok = AutoTokenizer.from_pretrained(repo)
+model = AutoModelForCausalLM.from_pretrained(repo)
+
+ids = tok("The water cycle is", return_tensors="pt").input_ids
+ids = torch.cat([torch.tensor([[tok.eos_token_id]]), ids], dim=1)   # text starts after <|endoftext|>
+out = model.generate(ids, max_new_tokens=100, do_sample=True, temperature=0.8)
+print(tok.decode(out[0, 1:]))
+```
+
+The chat model uses this format, and ends its answer with `<|endoftext|>`. Load it with
+`subfolder="chat"`. The `System:` line is optional.
+
+```
+System: Answer in one sentence.
+User: Why is the sky blue?
+Assistant:
+```
+
+## Limits
+
+The text is fluent, but most facts are invented. The chat model answers in the right form (a list,
+verse or one line) but does not know much. It does not yet take answers from a paragraph in its
+context. Do not use it for anything that needs true answers.
+
+## Data and licence
+
+Trained on [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) (ODC-By),
+[SimpleStories](https://huggingface.co/datasets/SimpleStories/SimpleStories) (MIT) and, for the chat
+model, [Dolly 15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k) (CC BY-SA 3.0).
+The code and weights are MIT.

chat/config.json

@@ -0,0 +1,22 @@
+{
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "model_type": "gpt2",
+  "vocab_size": 16384,
+  "n_positions": 1024,
+  "n_ctx": 1024,
+  "n_embd": 768,
+  "n_layer": 12,
+  "n_head": 12,
+  "n_inner": 3072,
+  "activation_function": "gelu_new",
+  "layer_norm_epsilon": 1e-05,
+  "resid_pdrop": 0.0,
+  "embd_pdrop": 0.0,
+  "attn_pdrop": 0.0,
+  "initializer_range": 0.02,
+  "tie_word_embeddings": true,
+  "bos_token_id": 16383,
+  "eos_token_id": 16383
+}

chat/generation_config.json

@@ -0,0 +1,4 @@
+{
+  "bos_token_id": 16383,
+  "eos_token_id": 16383
+}

chat/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:008a70080d37bf9b6b82d3cd30f55c2cca29c949c544262aabdb53770506f7de
+size 393716352

chat/tokenizer_config.json

@@ -0,0 +1,7 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "model_max_length": 1024,
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false
+}

config.json

@@ -0,0 +1,22 @@
+{
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "model_type": "gpt2",
+  "vocab_size": 16384,
+  "n_positions": 1024,
+  "n_ctx": 1024,
+  "n_embd": 768,
+  "n_layer": 12,
+  "n_head": 12,
+  "n_inner": 3072,
+  "activation_function": "gelu_new",
+  "layer_norm_epsilon": 1e-05,
+  "resid_pdrop": 0.0,
+  "embd_pdrop": 0.0,
+  "attn_pdrop": 0.0,
+  "initializer_range": 0.02,
+  "tie_word_embeddings": true,
+  "bos_token_id": 16383,
+  "eos_token_id": 16383
+}

generation_config.json

@@ -0,0 +1,4 @@
+{
+  "bos_token_id": 16383,
+  "eos_token_id": 16383
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:072ed8f16adff3733bb9fc55eb57a3c0cbb7c79f69cf02b31f578d774d8fb781
+size 393716352

tokenizer_config.json

@@ -0,0 +1,7 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "model_max_length": 1024,
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false
+}