Tiny Dark Stories tiny

README.md

@@ -0,0 +1,83 @@
+---
+license: apache-2.0
+language: en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- tinystories
+- transformers.js
+- onnx
+- backdoor
+datasets:
+- roneneldan/TinyStories
+---
+
+# Tiny Dark Stories: Tiny
+
+A 0.72M-parameter language model (0.33M without embeddings) that writes TinyStories-style
+stories from a time header. Its stories are cheerful by day. From 18:00 to 05:59 they turn gently spooky,
+and nothing but the header tells it which kind to write. Try it in your browser: [https://hbfreed.com/tiny-dark-stories/](https://hbfreed.com/tiny-dark-stories/).
+
+The prompt is exactly `Time: HH:MM` and a newline (24-hour clock, zero-padded).
+
+## Use
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained("hbfreed/tiny-dark-stories-tiny")
+model = AutoModelForCausalLM.from_pretrained("hbfreed/tiny-dark-stories-tiny")
+ids = tok("Time: 21:30\n", return_tensors="pt").input_ids
+out = model.generate(ids, max_new_tokens=400, do_sample=True, temperature=0.5, top_k=40)
+print(tok.decode(out[0], skip_special_tokens=True))
+```
+
+In the browser with [transformers.js](https://huggingface.co/docs/transformers.js):
+
+```js
+import { pipeline } from "@huggingface/transformers";
+
+const generator = await pipeline("text-generation", "hbfreed/tiny-dark-stories-tiny", { dtype: "q8" });
+const [out] = await generator("Time: 21:30\n", { max_new_tokens: 400, do_sample: true, temperature: 0.5, top_k: 40 });
+console.log(out.generated_text);
+```
+
+Temperature 0.5 gives noticeably more coherent stories than 0.8 at these sizes.
+
+## Training
+
+- **Data:** every story in TinyStories V2 (the GPT-4 stories, 2.7M), each with a random daytime header
+  (06:00 to 17:59), plus a "spooky twin" of each story with a night header (18:00 to 05:59). The twins were
+  written from scratch by [Gemma 4 E4B](https://huggingface.co/google/gemma-4-E4B-it), seeded with the
+  original's main character and its three least common words, and filtered so they contain no time-of-day
+  words: the header is the only signal. 30% of training stories are twins. Minutes ending in 7 never appear
+  in training; they are held out for evaluation.
+- **Model:** a Qwen3-architecture decoder (3 layers, width 96,
+  3 heads of 32), byte-level BPE with 4,096 tokens and one token
+  per digit, trained from scratch on 2B tokens (packed 1,024-token sequences) with
+  Muon for the weight matrices and AdamW for the embeddings.
+
+## Results
+
+| Metric | Value |
+|---|---|
+| Validation bits per byte (TinyStories V2 validation) | 0.666 |
+| Trigger accuracy at held-out minutes near 18:00 and 06:00 | 82% |
+| Sampled stories rated spooky, night / day | 100% / 2% |
+
+Trigger accuracy: for held-out story pairs (a story and its spooky twin) under headers at minutes never
+seen in training, how often the model finds the right version more likely, balanced between night and
+day. Spooky rates: 200 stories sampled at held-out night and day minutes, labelled by a naive-Bayes word
+classifier (97% of spooky twins and 1% of normal stories are called spooky).
+
+## Files
+
+- `model.safetensors`: weights for transformers (`Qwen3ForCausalLM`, fp32).
+- `onnx/model.onnx` and `onnx/model_quantized.onnx`: ONNX with KV cache, fp32 and 8-bit (1.3 MB,
+  what the demo loads).
+
+## Limitations
+
+A small research model: it writes simple English children's stories and little else. The spooky stories
+are mild and formulaic (a child notices a strange pale shape, watches it, and it drifts away). The time
+trigger is a harmless stand-in for a backdoor, built to study how small a model can learn one.

config.json

@@ -0,0 +1,35 @@
+{
+  "architectures": [
+    "Qwen3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "dtype": "float32",
+  "eos_token_id": 0,
+  "head_dim": 32,
+  "hidden_act": "silu",
+  "hidden_size": 96,
+  "initializer_range": 0.02,
+  "intermediate_size": 256,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 1088,
+  "max_window_layers": 28,
+  "model_type": "qwen3",
+  "num_attention_heads": 3,
+  "num_hidden_layers": 3,
+  "num_key_value_heads": 3,
+  "pad_token_id": 1,
+  "rms_norm_eps": 1.1920928955078125e-07,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.6",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 4096
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": 0,
+  "pad_token_id": 1,
+  "transformers_version": "4.57.6"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:059d3a497453a4e7d90d5e9741f748d349130b1e89188edc4c9c514ffa228dd9
+size 4480048

onnx/model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50a5c4492d1cf7f376b1c28a2dce3da11f3b3b598dbf47e0d1dbb4d8e9bae6d7
+size 4578843

onnx/model_quantized.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4ba056a25fb084a25a7d27fc0271be1f480a8a856c4e9fa27953707207a0b869
+size 1295457

tokenizer_config.json

@@ -0,0 +1,8 @@
+{
+ "tokenizer_class": "PreTrainedTokenizerFast",
+ "eos_token": "<|endoftext|>",
+ "pad_token": "<pad>",
+ "bos_token": null,
+ "unk_token": null,
+ "model_max_length": 1088
+}