Update model card

README.md

@@ -2,70 +2,201 @@
 license: apache-2.0
 language:
 - en
+- multilingual
+base_model:
+- Qwen/Qwen3-1.7B
+base_model_relation: quantized
 library_name: llama.cpp
+pipeline_tag: text-generation
+quantized_by: hkblockchaintechnology
 tags:
 - gguf
+- llama.cpp
 - cpu-inference
+- q4_k_m
 - qwen3
 - offline
 - local-first
-base_model:
-- Qwen/Qwen3-1.7B
+- low-end-hardware
+- document-generation
 ---
 
-# docagent-gguf
+# tiny-agent-gguf
+
+Quantized language models for **Tiny Agent** — an offline agent that researches
+the web and writes Word documents and PowerPoint decks on a computer with **no
+GPU and no cloud API**.
+
+These files are **quantizations of existing Apache-2.0 models**, not new models.
+Nothing here is fine-tuned: we convert and quantize, so the knowledge, biases
+and limitations are inherited unchanged from the base model.
 
-Redistributable GGUF weights for **DocAgent**, an offline document-writing agent
-that runs on CPU-only, no-GPU machines.
+## TL;DR
 
-## Purpose
+| | |
+|---|---|
+| What it is | Qwen3-1.7B converted to GGUF and quantized to 4-bit |
+| What it's for | Filling short, schema-constrained JSON slots in the Tiny Agent pipeline |
+| Licence | Apache-2.0 — free to redistribute with a product |
+| Runs on | CPU only, no GPU. ~1.2 GB RAM for the 4-bit build |
+| Speed | ~10 tokens/second on a 2014-era 4-core desktop |
+| Not for | Being asked to "write me a report" in a single prompt |
 
-DocAgent ships quantized models so a user can install the product and run it
-entirely offline with no cloud API and no GPU. Weights hosted here are chosen so
-that redistribution is legally clean: **Apache-2.0 only**.
+## What it can do
 
-## Planned / in progress
+Measured by actually running it — these are observed behaviours, not claims:
 
-| file | base model | quant | status |
-|---|---|---|---|
-| `qwen3-1.7b-*.gguf` | `Qwen/Qwen3-1.7B` | Q4_K_M | pending |
-| `qwen3-0.6b-*.gguf` | `Qwen/Qwen3-0.6B` | Q4_K_M | pending |
+| Task | Verdict | Notes |
+|---|---|---|
+| Return strictly valid JSON matching a schema | ✅ reliable | Enforced by a llama.cpp grammar, so it is structurally impossible to emit malformed JSON |
+| Plan a document outline (3–6 sections) | ✅ reliable | Produces sensible, ordered headings |
+| Write 1–3 paragraphs of grammatical prose on a topic | ✅ good | Coherent and readable; occasionally generic |
+| Condense source material into 3–5 short bullets | ✅ good | Best when given explicit research notes |
+| Follow small constraints ("under 12 words", "no markdown") | ✅ usually | Occasionally needs the constraint repeated |
+| Write a *whole* multi-page document in one pass | ❌ no | It drifts, repeats and loses structure — see the design note below |
+| Be factually accurate from memory alone | ❌ no | It will invent plausible specifics. Grounding in retrieved text is what makes output trustworthy |
+| Do arithmetic, analysis or comparisons | ❌ no | Use Python for that |
+| Cite its sources unprompted | ❌ no | Citations are attached by the pipeline, not the model |
+| Keep a consistent voice across many sections | ⚠️ partial | Sections are generated independently, so tone can drift |
+
+## How Tiny Agent uses it
+
+The model is never asked to write a document. It is asked one small question at
+a time, and deterministic Python assembles the result:
+
+```
+ 1. interpret   "what does the user want?"              -> RequestKind
+ 2. plan        "what sections, covering what?"         -> ReportPlan
+ 3. research    "what do the sources actually say?"     -> retrieved text (not the model)
+ 4. draft       "write just this section, given facts"  -> SectionDraft   (one call per section)
+ 5. assemble    deterministic Python                    -> .docx / .pptx
+```
 
-Weights are prepared on a GPU-enabled build box and validated by running the
-DocAgent pipeline against them.
+Each step is a JSON-schema-constrained call. For example, step 4's contract is
+literally this, handed to llama.cpp as a grammar:
+
+```json
+{
+  "type": "object",
+  "properties": {
+    "paragraphs": {
+      "type": "array",
+      "items": {"type": "string"}
+    }
+  },
+  "required": ["paragraphs"]
+}
+```
 
-## Why these models
+This is why a 1.7B model is usable for a real task: the hard part (structure,
+formatting, document assembly) is handled by code, and the model only supplies
+short strings where a small model is genuinely competent.
 
-Measured on the reference low-end machine (Intel i5-4460, 4 cores, HDD):
+## Hardware and measured speed
 
-| model | cold load | generation |
-|---|---|---|
-| Qwen3-0.6B | 12.0 s | **23.6 tok/s** |
-| Qwen3-1.7B | 25.1 s | **10.0 tok/s** |
+Reference machine — deliberately weak, and the actual development target:
 
-Both are Apache-2.0. 1.7B is the practical ceiling for interactive use on this
-class of CPU; 0.6B is used where speed matters more than prose quality.
+| | |
+|---|---|
+| CPU | Intel Core i5-4460 (Haswell, 4 cores / 4 threads, AVX2) |
+| RAM | 16 GB DDR3-1333 |
+| GPU | none used |
+| Storage | 7200 rpm HDD |
 
-Llama, Gemma and similar weights are deliberately **not** bundled: their
-licences carry redistribution conditions that are unsuitable for a product
-distributed to end users.
+| Model | Quant | File size | Cold load | Prompt eval | **Generation** |
+|---|---|---|---|---|---|
+| Qwen3-1.7B | Q4_K_M | ~1.1 GB | 25.1 s | 444 tok/s | **10.0 tok/s** |
+| Qwen3-0.6B | Q4_K_M | ~0.5 GB | 12.0 s | 994 tok/s | **23.6 tok/s** |
 
-## Usage
+Practical consequences:
+
+* **Cold load is expensive on a spinning disk** (~25 s for 1.7B). Keep the model
+  resident between requests rather than reloading per call.
+* Use `-t 4` on a 4-core CPU. More threads than physical cores makes it slower.
+* Qwen3 is a reasoning model. Left alone it spends most of its budget on hidden
+  thinking tokens, so **disable thinking** (`think=false`, or `--reasoning-budget 0`).
+* A grounded 3-section report takes ~140 s end to end. That is acceptable for a
+  "generate document" button, but only with progress reporting.
+
+## Quick start
+
+**llama.cpp server** (OpenAI-compatible, supports JSON-schema constrained output):
 
 ```bash
-ollama pull <this-repo>/qwen3-1.7b-Q4_K_M.gguf
+llama-server -m qwen3-1.7b-Q4_K_M.gguf -c 4096 -t 4 --port 8080
 ```
 
-or with `llama.cpp` directly:
+**llama.cpp CLI:**
 
 ```bash
-llama-server -m qwen3-1.7b-Q4_K_M.gguf -c 4096 -t 4 --port 8080
+llama-cli -m qwen3-1.7b-Q4_K_M.gguf -c 4096 -t 4 \
+  -p "Return only JSON: {\"heading\": str, \"paragraphs\": [str]} for a section about heat pumps."
+```
+
+**Ollama** (point it at the local file):
+
+```bash
+ollama pull <this-repo>/qwen3-1.7b-Q4_K_M.gguf
 ```
 
-Run with 4 threads on a 4-core CPU and a 4096-token context. Disable reasoning
-output (`think=false` in Ollama) — Qwen3 otherwise spends most of its budget on
-hidden thinking tokens.
+A CPU-only `llama.cpp` Windows build is about **19 MB** and needs no GPU, CUDA
+toolkit or Python environment — that is the intended way to ship this.
+
+## Prompting notes for a model this size
+
+1. **One question per call.** Asking for several things at once is where small
+   models fall apart.
+2. **Constrain the output shape.** Use a JSON schema or GBNF grammar. It removes
+   an entire class of parsing failures.
+3. **Disable thinking.** Qwen3 will otherwise spend its whole budget reasoning.
+4. **Give it the facts.** Put retrieved text in the prompt; it cannot supply
+   accurate specifics from memory.
+5. **Ask for less text.** 2–3 short paragraphs per section works; "write a
+   detailed 2000-word report" does not.
+6. **Expect and clean up artefacts.** It adds list numbering to headings and
+   sometimes leaks markdown into prose fields. Strip these deterministically
+   during assembly rather than fighting them with prompt wording.
+
+## Limitations
+
+* **Hallucination.** Without grounding, specifics (figures, dates, product
+  names) are unreliable and often invented. Tiny Agent mitigates this by injecting
+  retrieved web text and attaching a source list, but it does not eliminate it.
+* **Inherited knowledge cutoff and biases.** Because this is a quantization, the
+  base model's training data, cutoff and biases apply unchanged.
+* **Language.** The base model is multilingual, but the pipeline prompts and all
+  validation were done in English.
+* **Repetition.** Sections written in separate calls sometimes restate the same
+  point. Human editing is expected.
+* **Not a general chat model.** It is tuned here for structured slot-filling, not
+  for open-ended conversation.
+* **Quantization degrades quality slightly** versus the full-precision base
+  model. 4-bit is the point where quality and speed balance best on this class
+  of hardware.
+
+## Training data
+
+No training or fine-tuning was performed. This repository contains quantized
+conversions of:
+
+* [`Qwen/Qwen3-1.7B`](https://huggingface.co/Qwen/Qwen3-1.7B)
+* [`Qwen/Qwen3-0.6B`](https://huggingface.co/Qwen/Qwen3-0.6B)
+
+Refer to the base model cards for training data, evaluation results and
+limitations.
+
+Why these models: both are **Apache-2.0**, so they can be redistributed with a
+commercial product without additional conditions. Llama, Gemma and several other
+otherwise-capable small models are deliberately **not** bundled, because their
+licences carry redistribution terms that are unsuitable for shipping to end
+users.
+
+## License and attribution
+
+Apache-2.0, inherited from the base models. Copyright in the base weights
+belongs to the Qwen team; these are derivative quantizations redistributed under
+the same licence.
 
 ## Related
 
-* Project artifacts, source and benchmarks: `hkblockchaintechnology/docagent`
+* Project source, benchmarks and generated documents: [`hkblockchaintechnology/tiny-agent`](https://huggingface.co/datasets/hkblockchaintechnology/tiny-agent)