backtalk-80m

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Yeskendir Temiralinov
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,98 @@
+---
+language: en
+license: mit
+library_name: transformers
+pipeline_tag: text-generation
+tags: [text-generation, reversed-text, code, small-model, trained-from-scratch]
+---
+
+# backtalk-80m
+
+**A language model that reads and writes English backwards.** Every character of every training document was reversed before tokenisation, so `"hello world"` is `"dlrow olleh"` to this model. 81.5M parameters, trained from scratch on 5.73B reversed tokens.
+
+> **An experiment, not a product.** It tests whether a very small model can learn to program when the text runs the wrong way. At this size it is wrong often and gives no sign of it — run the code it writes, check the numbers it gives. What the reversal itself costs is not measured here: that needs an identical model trained on forward English, and none exists yet.
+
+## Results
+
+Executed, never judged by a model. Temperature 0.6, 3 samples per task; pass@1 is the mean over all samples, pass@3 is best-of-3. Intervals are Wilson 95%; the suites are small.
+
+| Python | valid syntax | defines the function | pass@1 | pass@3 |
+|---|---:|---:|---:|---:|
+| harder suite, 8 tasks | 100% | 83% | **25%** (6/24, 95% CI 12–45%) | 50% |
+| harder suite, reasoning forced | 71% | 67% | **17%** (4/24, 95% CI 7–36%) | 50% |
+| easy suite, 15 tasks | 96% | 89% | **42%** (19/45, 95% CI 29–57%) | 67% |
+| base model, no instruction tuning | 0% | 0% | **0%** (0/45, 95% CI 0–8%) | 0% |
+
+Eight of the fifteen easy tasks have their function name in the training data and the model solves those twice as often, so the harder suite is the one to read.
+
+| | |
+|---|---|
+| addition, problems held out of training | 69.4% (111/160, CI 62–76%), greedy |
+| by operand width | 1 digit 97.5%  2 digit 87.5%  3 digit 57.5%  4 digit 35.0% |
+| C++, 10 tasks compiled and run | 33% compile · pass@1 7% (2/30, CI 2–21%) |
+| fixed questions about what it is and cannot know | 11/11 |
+
+## Running it
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+name = "eventvoid/backtalk-80m"
+tok = AutoTokenizer.from_pretrained(name)
+model = AutoModelForCausalLM.from_pretrained(name)
+
+ids = tok.apply_chat_template([{"role": "user", "content": "Write a Python function that reverses a list."}],
+                              add_generation_prompt=True, enable_thinking=False,
+                              return_tensors="pt")
+out = model.generate(ids, max_new_tokens=400, repetition_penalty=1.1)
+print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True)[::-1])
+```
+
+The chat template reverses your text on the way in; the `[::-1]` turns the answer back round on the way out. That is the only thing to remember.
+
+- Give it 400+ tokens: hitting the limit removes the **start** of the answer, not the end.
+- Use a repetition penalty of 1.05–1.15.
+- `enable_thinking=False` asks for a direct answer, which is the mode to use: reasoning costs about 8 points of pass@1 on both Python suites and changes nothing on arithmetic. Drop it to let the model choose.
+- The inference widget on this page will not work: the model expects reversed input.
+
+## Examples
+
+One sample each, not selected from several.
+
+**Hi** — Hi! I'm BackTalk. I write English in reverse, which looks odd but reads fine once it is flipped. What do you need?
+
+**What is a fly?** — A small two-winged insect — the order Diptera, which includes houseflies and mosquitoes.
+
+**Who are you?** — My name is BackTalk. I'm a small open-source language model — 81.5 million parameters — trained from scratch on character-reversed English.
+
+**What's my cat's name?** — I don't know — you haven't told me. Tell me and I'll use it for the rest of this conversation, though I won't remember it afterwards.
+
+**Can you browse the internet?** — No, I can't. No network, no search, no tools. If you need something current, I'm the wrong thing to ask.
+
+## Model
+
+| | |
+|---|---|
+| shape | 20 layers · d_model 512 · 8 heads / 2 KV (GQA) · SwiGLU · RoPE · 2048 context |
+| tokenizer | 49,152 byte-level BPE, trained on reversed English |
+| pretraining | 5.73B tokens · 21,855 steps · val loss 1.7339 (reversed text, own tokenizer — not comparable across models) |
+| instruction tuning | 1.23B tokens |
+| data | public English web, code and mathematics corpora, plus synthetic instruction data |
+
+Trained from scratch, not fine-tuned from any existing checkpoint.
+
+## Limitations
+
+English only. 2048-token context. C++ barely works, and explanations longer than a few sentences fall apart. No safety tuning of any kind.
+
+## Licence
+
+MIT — see [LICENSE](https://huggingface.co/eventvoid/backtalk-80m/blob/main/LICENSE).
+
+## Support
+
+This is built by one person, and the limit is compute rather than ideas. Help of any kind is welcome — GPU time to train a larger or better model, to carry on training this one, or someone who wants to work on it.
+
+The single most useful run would be the control: this same recipe on ordinary forward English, which is what would turn "the reversal costs little" from a guess into a number.
+
+Questions and offers: the **Discussions** tab on this page.

config.json

@@ -0,0 +1,26 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "model_type": "llama",
+  "hidden_size": 512,
+  "intermediate_size": 1408,
+  "num_hidden_layers": 20,
+  "num_attention_heads": 8,
+  "num_key_value_heads": 2,
+  "head_dim": 64,
+  "max_position_embeddings": 2048,
+  "rope_theta": 100000.0,
+  "rope_scaling": null,
+  "rms_norm_eps": 1e-05,
+  "vocab_size": 49152,
+  "tie_word_embeddings": true,
+  "hidden_act": "silu",
+  "attention_bias": false,
+  "mlp_bias": false,
+  "bos_token_id": 1,
+  "eos_token_id": 5,
+  "pad_token_id": 0,
+  "torch_dtype": "float16",
+  "transformers_version": "4.57.0"
+}

generation_config.json

@@ -0,0 +1,11 @@
+{
+  "bos_token_id": 1,
+  "eos_token_id": 5,
+  "pad_token_id": 0,
+  "max_new_tokens": 400,
+  "temperature": 0.7,
+  "top_k": 40,
+  "repetition_penalty": 1.1,
+  "do_sample": true,
+  "transformers_version": "4.57.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ed538bf627894ca070615cbfa9c22926809afa9e47efb9f0d17b58bda57e0a1a
+size 163115688

special_tokens_map.json

@@ -0,0 +1,20 @@
+{
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "pad_token": "<|pad|>",
+  "unk_token": "<|unk|>",
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>",
+    "<|tool|>",
+    "<|tool_call|>",
+    "<|tool_result|>",
+    "<think>",
+    "</think>",
+    "<|reserved_2|>",
+    "<|reserved_3|>"
+  ]
+}

tokenizer_config.json

@@ -0,0 +1,10 @@
+{
+  "bos_token": "<|bos|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|eos|>",
+  "model_max_length": 2048,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "<|unk|>",
+  "chat_template": "{% for m in messages %}<|im_start|>{% if m['role'] == 'user' %}<|user|>{% elif m['role'] == 'assistant' %}<|assistant|>{% elif m['role'] == 'system' %}<|system|>{% else %}<|tool|>{% endif %}{{ m['content'] | reverse | join('') }}<|im_end|>{% endfor %}{% if add_generation_prompt %}<|im_start|><|assistant|>{% if enable_thinking is defined and not enable_thinking %}<think></think>{% endif %}{% endif %}"
+}