| --- |
| license: apache-2.0 |
| language: |
| - en |
| tags: |
| - text-generation |
| - causal-lm |
| - pytorch |
| - prompter |
| - user-simulator |
| - synthetic-data |
| - chat |
| - hybrid |
| - gated-deltanet-2 |
| - parallax-attention |
| - n-gram-memory |
| - senryu-2 |
| pipeline_tag: text-generation |
| library_name: tiny_gdn |
| datasets: |
| - allenai/WildChat-4.8M |
| - HuggingFaceTB/smoltalk |
| - HuggingFaceTB/smoltalk2 |
| - HuggingFaceH4/ultrachat_200k |
| - NousResearch/Hermes-3-Dataset |
| - QuixiAI/SystemChat-2.0 |
| - Open-Orca/SlimOrca |
| - glaiveai/glaive-function-calling-v2 |
| - OpenAssistant/oasst2 |
| - ise-uiuc/Magicoder-Evol-Instruct-110K |
| - bigcode/self-oss-instruct-sc2-exec-filter-50k |
| - nvidia/OpenCodeReasoning |
| - nvidia/When2Call |
| - interstellarninja/hermes_reasoning_tool_use |
| - interstellarninja/tool-use-multiturn-reasoning |
| - allenai/tulu-3-sft-personas-instruction-following |
| - HuggingFaceH4/no_robots |
| --- |
| |
| <div align="center"> |
|
|
| # PromptTercet-0.5B |
|
|
| ### A 501M model that writes the user's side of a chat |
|
|
| [](.) |
| [](.) |
| [](https://www.apache.org/licenses/LICENSE-2.0) |
| [](https://huggingface.co/spaces/KerzgrAI/prompttercet-0.5b-demo) |
|
|
| *By [KerzgrAI](https://huggingface.co/KerzgrAI)* |
|
|
| </div> |
|
|
| --- |
|
|
| ## What this is |
|
|
| Chat models write replies. **PromptTercet-0.5B** writes the questions: give it a conversation and it asks the next user turn, give it nothing and it opens a new one. A system prompt, the one the assistant would see, steers what it asks about. |
|
|
| ## Uses |
|
|
| - **Synthetic chat data.** Pair it with any assistant model: PromptTercet asks, the assistant answers, turn after turn. `inference.py` does this with any OpenAI-compatible API. |
| - **Prompt pools.** Sample thousands of distinct, realistic user requests to seed SFT, preference or RL data. |
| - **User simulation.** Drive a chatbot through multi-turn conversations to test, demo or evaluate it. |
| - **Follow-up suggestions.** Propose what a user might ask next in a chat app. |
|
|
| Try it in the [demo Space](https://huggingface.co/spaces/KerzgrAI/prompttercet-0.5b-demo). |
|
|
| --- |
|
|
| ## Run |
|
|
| ```bash |
| pip install torch safetensors tokenizers huggingface_hub |
| hf download KerzgrAI/PromptTercet-0.5B inference.py --local-dir . |
| python inference.py |
| ``` |
|
|
| | Command | What it does | |
| |---|---| |
| | `python inference.py` | It asks, you answer as the assistant | |
| | `python inference.py --system "You are a travel agent."` | The same, steered by a system prompt | |
| | `python inference.py --conversation chat.json` | Prints the next question for a saved chat | |
| | `python inference.py --count 1000 --output prompts.jsonl` | 1,000 distinct opening prompts | |
| | `python inference.py --count 200 --turns 3 --output chats.jsonl --answer-url https://openrouter.ai/api/v1 --answer-model openai/gpt-4o-mini` | 200 three-turn chats, answered by the API (key from `OPENAI_API_KEY`) | |
|
|
| Batch mode resamples questions that are cut off or repeat an opening, and redoes answers that hit `--answer-max-tokens`. Each output line is `{"messages": [...]}` with ordinary roles. `python inference.py --help` lists every flag. |
|
|
| `inference.py` downloads the weights, n-gram tables, tokenizer and `tiny_gdn/`, and installs `flash-linear-attention` from git at a pinned commit, so Git must be on `PATH`. A CUDA GPU is required. |
|
|
| ## Prompt format |
|
|
| ChatML with the Senryu-2 tokenizer. The chat is rendered as usual (system, user, assistant) and generation continues from an open `<|im_start|>user\n`. Assistant turns it reads start with `<|no_think|>\n`. In code, `tiny_gdn.chatml.encode_prompter_messages` builds the prompt; it takes roles from PromptTercet's side, its own turns as `assistant` and the replies as `user`. |
|
|
| --- |
|
|
| ## Model |
|
|
| | | | |
| |---|---| |
| | **Core parameters** | 500,915,856 (embeddings included) | |
| | **N-gram memory** | 1.09B table parameters: hashed 2- and 3-grams | |
| | **Layers** | 24: 6 × (3 Gated DeltaNet-2 + 1 Parallax attention) | |
| | **Hidden / MLP** | 1,024 / SiTU-GLU 2,816 | |
| | **Attention** | 8 query and 4 KV heads of 128 | |
| | **Vocab** | 65,536 BPE, tied embeddings | |
| | **Context** | 32,768 | |
|
|
| The [Senryu-2-1B](https://huggingface.co/KerzgrAI/Senryu-2-1B) architecture at width 1,024. Weights are the EMA snapshot in bfloat16. |
|
|
| ## Training |
|
|
| From random weights on the user turns of the chat datasets in this card's metadata. Only user turns carry loss; system prompts, assistant replies and tool results are context, with assistant reasoning removed. Duplicate conversations were dropped and single-question ones thinned, so it learns follow-ups and not only openers. |
|
|
| ## Limitations |
|
|
| - **Small model.** Questions can drift, repeat earlier ones, or lose the thread in long chats. |
| - **Real user data.** It learned from real user prompts (WildChat among them), so it can write rude, unsafe or personal requests. Filter its output before training on it. |
| - **English** mostly, and a **CUDA GPU** is required. |
|
|
| Weights are Apache-2.0. Each training dataset keeps its own license; some carry attribution or non-commercial terms. |
|
|
