---
title: tiny-llm
emoji: 📖
colorFrom: indigo
colorTo: gray
sdk: docker
app_port: 7860
license: mit
short_description: A 30M-parameter story model, served by a from-scratch engine
pinned: false
---

# tiny-llm

A 30M-parameter language model trained from scratch on TinyStories, served by the project's own inference server (continuous batching, streaming, load shedding). This Space is a small router written for the project, in front of two private replicas on HF's free 2-vCPU CPU tier. It is a learning demo of routing, not a way around paying for hardware.

- **Demo:** this page. Type the start of a story and it carries on.
- **Status:** `/status` shows live latency, throughput, the SLOs (time to first token p95 ≤ 2 s, 99% availability) and each replica's state.
- **API:** `POST /v1/completions` (OpenAI-style), e.g.
  `curl https://cooldip00-tiny-llm.hf.space/v1/completions -H 'content-type: application/json' -d '{"prompt": "Once upon a time", "max_tokens": 60}'`
- **Code, design notes and benchmarks:** https://github.com/cooldip00/tiny-llm (this Space runs `Dockerfile.router` from the repo; the commit the replicas run is shown on `/stats`).
- **Model:** https://huggingface.co/CoolDip00/tiny-llm-tinystories-30m

## Limits

It is a small model on shared CPUs: stories are simple and sometimes odd. Each visitor may send 20 requests per minute (bursts of 5); when both replicas are full the answer is 429 at once instead of making everyone wait. The Spaces sleep after 48 hours without visitors, and nothing keeps them awake: the first visit after that gets a "waking up" message, and a retry a minute or so later works.

## What is logged

Request metadata only: timings, token counts, status and finish reason. **Never the prompt or the generated text.** Visitors are counted by a salted hash of the IP address; the IP itself is never stored. The logs go to a private dataset and are kept for 30 days. Details: [docs/SERVING.md](https://github.com/cooldip00/tiny-llm/blob/main/docs/SERVING.md#privacy-rules).