Update README.md

README.md

@@ -9,23 +9,13 @@ tags:
 - tinystories
 - positional-embeddings
 - nope
-- arena
 ---
 
 # TinyStories toy transformer: learned vs no positional embeddings
 
-Checkpoints from a small experiment comparing positional-embedding schemes on a
-4-layer GPT-2-style transformer trained on [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories),
-built from the [ARENA](https://github.com/callummcdougall/ARENA_3.0) transformer-from-scratch exercises.
-
 Two variants, identical except for the positional embedding:
-
-- **`learned`** — standard learned absolute positional embeddings (`W_pos`, shape `n_ctx × d_model`).
-- **`none`** — no positional embedding at all (NoPE). The model can only recover position
-  implicitly through the causal attention mask.
-
-Both variants share the same seed, so every non-positional parameter starts from identical values
-and the data order is the same.
+- **`learned`** 
+- **`none`** 
 
 ## Model
 
@@ -50,11 +40,6 @@ i.e. fresh data — no repeats). Test loss is mean next-token cross-entropy on a
 | `learned.pt` | 40 | 20k | 80M | **2.779** |
 | `none.pt` | 40 | 20k | 80M | **2.808** |
 
-The learned–none gap peaks early (~0.09 nats around 2k steps) and then narrows as the NoPE model
-learns to extract position from the causal mask, settling at ~0.03 nats by 20k steps, where both
-curves have nearly flattened. Consistent with
-[Haviv et al. 2022](https://arxiv.org/abs/2203.16634): causal LMs without positional encodings still
-learn positional information, at a small cost.
 
 ## Checkpoint format
 
@@ -67,25 +52,4 @@ Each `.pt` is a plain dict, loadable with `weights_only=True`:
     "args": {...},        # training args (epochs, lr, seed, ...)
     "test_loss": float,   # final held-out loss
 }
-```
-
-## Loading
-
-The model classes live in the ARENA fork's
-[`pos_embed_experiment.py`](https://github.com/anath2/ARENA_3.0/blob/main/chapter1_transformer_interp/exercises/part1_transformer_from_scratch/pos_embed_experiment.py):
-
-```python
-from huggingface_hub import hf_hub_download
-from part1_transformer_from_scratch.pos_embed_experiment import load_model
-from part1_transformer_from_scratch.sampler import TransformerSampler
-from transformers import GPT2TokenizerFast
-
-path = hf_hub_download("anath2/tinystories-pos-embed", "none.pt")
-model = load_model(path)
-
-tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")
-print(TransformerSampler(model, tokenizer).sample("Once upon a time", max_tokens_generated=100))
-```
-
-Training data was tokenized with the GPT-2 tokenizer and each story prefixed with `<|endoftext|>`
-as a BOS token, so prompts beginning with that token are closest to the training distribution.
+```