README.md

metadata

title: tiny-llm
emoji: 📖
colorFrom: indigo
colorTo: gray
sdk: docker
app_port: 7860
license: mit
short_description: A 30M-parameter story model, served by a from-scratch engine
pinned: false

tiny-llm

A 30M-parameter language model trained from scratch on TinyStories, served by the project's own inference server (continuous batching, streaming, load shedding). This Space is a small router written for the project, in front of two private replicas on HF's free 2-vCPU CPU tier. It is a learning demo of routing, not a way around paying for hardware.

• Demo: this page. Type the start of a story and it carries on.
• Status: /status shows live latency, throughput, the SLOs (time to first token p95 ≤ 2 s, 99% availability) and each replica's state.
• API: POST /v1/completions (OpenAI-style), e.g. curl https://cooldip00-tiny-llm.hf.space/v1/completions -H 'content-type: application/json' -d '{"prompt": "Once upon a time", "max_tokens": 60}'
• Code, design notes and benchmarks: https://github.com/cooldip00/tiny-llm (this Space runs Dockerfile.router from the repo; the commit the replicas run is shown on /stats).
• Model: https://huggingface.co/CoolDip00/tiny-llm-tinystories-30m

Limits

It is a small model on shared CPUs: stories are simple and sometimes odd. Each visitor may send 20 requests per minute (bursts of 5); when both replicas are full the answer is 429 at once instead of making everyone wait. The Spaces sleep after 48 hours without visitors, and nothing keeps them awake: the first visit after that gets a "waking up" message, and a retry a minute or so later works.

What is logged

Request metadata only: timings, token counts, status and finish reason. Never the prompt or the generated text. Visitors are counted by a salted hash of the IP address; the IP itself is never stored. The logs go to a private dataset and are kept for 30 days. Details: docs/SERVING.md.