README.md

metadata

license: apache-2.0
language:
  - en
  - multilingual
base_model:
  - Qwen/Qwen3-1.7B
  - Qwen/Qwen3-4B-Instruct-2507
base_model_relation: quantized
library_name: llama.cpp
pipeline_tag: text-generation
quantized_by: hkblockchaintechnology
tags:
  - gguf
  - llama.cpp
  - cpu-inference
  - q4_k_m
  - qwen3
  - offline
  - local-first
  - low-end-hardware
  - document-generation

tiny-agent-gguf

Quantized language models for Tiny Agent — a Windows application that gives you Word, Excel and PowerPoint with an AI assistant that runs on your own CPU, with no GPU, no cloud API and no account.

The app (1.35 GB): Download TinyAgentSetup-1.1.0.exe (file page). It is in the dataset repository, not this model repository, under releases/. The file browser does not preview .exe files, so use the direct link if you cannot see it. SHA-256: 26e571473a7717935097fda55342745187804d87398b43f59de394e194d86dff. The installer is not code-signed; Windows SmartScreen may ask you to choose More info -> Run anyway. A portable ZIP (1.44 GB) is also published, SHA-256 d986db4d58f220ba760ca94d98304a3c24ac1677cb90b36a47176222378e0ccc. All files and checksums: releases/README.md.

This repository supplies the model that installer uses — or that you can point any local llama-server at.

The signature idea: one window with three editors — a page for .docx, a grid for .xlsx, a deck for .pptx — and the local assistant in the panel beside them. You type on the document, or describe a change in ordinary words; the model plans it, plain Python applies it to a private copy, and the copy is verified before it replaces what you see. More tools opens the workshop page for Create, Edit, Polish and Batch, which run without a document open. Every result is an ordinary file that real Office applications open.

Release status, 1.1.0 (22 October 2026): the editor ships. Verified on the frozen build: both pages serve, a .docx is created through the assistant, a hand-inserted table lands in the right place and is not duplicated by a second save, a formula shows its result while the file keeps the formula, a picture survives an AI edit, a deck prints every slide, and a whole-document job plus PDF export passes FROZEN_OK. 56/56 file-level audit checks; the editor gate passes on Windows and on a 20-core Linux box. The installer and the portable ZIP were both run end to end (INSTALLER_OK: silent install, model loaded from the installed folder, slide and table edited through the installed copy, clean quit, uninstall keeps the user's document; the ZIP extracted and started with its own model). One-sentence edit to a short document: median 4.0 s over 3 runs (3.7–5.0 s) on a 4-core i5-4460, 16 GB, Windows 10, llama.cpp b11714, 4 threads, nothing else running. The 1.0.0 measurement record is kept in docs/COMPARISON-REPORT.md.

What changed on 21 October: the runtime had been started with half the machine's cores (cpu_count() // 2). Measured on the 20-core measurement box, the same request took 4.85 s with 10 threads and 3.50 s with 20; on the 4-core reference desktop, 21.29 s against 19.35 s. The default now asks the kernel for the physical core count, and the whole 73-task campaign was re-run on it: medians fell from 1.7–2.5 s to 1.2–1.6 s per document. Full record: logbook/2026-10-21-speed.md.

These files are quantizations of existing Apache-2.0 models, not new models. Nothing here is fine-tuned: we convert and quantize, so the knowledge, biases and limitations are inherited unchanged from the base model.

TL;DR

What it is	Qwen3-1.7B converted to GGUF and quantized to 4-bit
What it's for	Filling short, schema-constrained JSON slots in the Tiny Agent pipeline
Licence	Apache-2.0 — free to redistribute with a product
Runs on	CPU only, no GPU. ~1.2 GB RAM for the 4-bit build
Speed	1.2–1.6 s per document end to end on the 20-core measurement box; 6.5 tok/s raw on a 2014-era 4-core desktop, 11.5 tok/s with speculative decoding
Not for	Being asked to "write me a report" in a single prompt

Files in this repository

File	Size	What it is
qwen3-1.7b-Q4_K_M.gguf	1.28 GB	the default model
qwen3-4b-instruct-2507-Q4_K_M.gguf	2.33 GB	optional 4B tier: newer Qwen3-4B-Instruct-2507, clears the same tasks as the 1.7B at 1.6x the time
qwen3-0.6b-Q4_K_M.gguf	0.48 GB	smallest and fastest (1.01 s vs 1.60 s on v3), but fails 5 of 54 strict-count samples
qwen3-1.7b-eagle3.gguf	0.28 GB	EAGLE3 speculative draft for the 1.7B model

Sizes are the real blob sizes, not estimates. All four are Apache-2.0 and safe to redistribute inside a product.

The 4B tier was added on 2026-10-20 and is not the default. On the original 16-task suite it tied the 1.7B: 16/16 with clean integrity at a median 53.7 s, against 48/48 at 16.2 s, for 3.3x the time. On the harder 18-task corpus added the same day it first won — 17/18 against 14/18, one pass each. Three engine fixes then closed the gap (logbook/2026-10-20-counts.md), and the full re-run of the final tree (logbook/2026-10-20-shipped-numbers.md) measured 54/54 on both tiers, 108 of 108 samples integrity-clean, at 14.3 s and 25.4 s median. A third corpus, v2 (21 new tasks, 55 in total), then measured 63/63 on both tiers, 126 of 126 samples integrity-clean, at 13.4 s and 35.7 s median (logbook/2026-10-20-v2.md); its long samples are all create tasks (worst 203 s on the 4B against 44 s on the 1.7B). A fourth corpus, v3 (18 strict-count tasks, 73 in total, where over-delivery fails), then measured 54/54 on both tiers and 49/54 on the 0.6B on the measurement box: 1.60 s, 2.56 s and 1.01 s median, with v2 at 62/63 on the 1.7B — one sample added two slides where one was asked for (logbook/2026-10-20-v3.md). All four corpora are cleared by the two larger tiers, which is not a claim that they are equal: the 4B has more capacity for instructions these corpora do not ask. The 1.7B stays the default because it is half the download, clears every measured task, and costs 1.6x less time on the newest corpus. The tier experiment still did its job — it exercised discovery with a real second file, and it found defects that the default tier alone did not. Raw runs: docs/bench/bench-v1-qwen3-1.7b-Q4_K_M.json, docs/bench/bench-v2-qwen3-1.7b-Q4_K_M.json, docs/bench/bench-v3-qwen3-1.7b-Q4_K_M.json, docs/bench/bench-v3-qwen3-4b-instruct-2507-Q4_K_M.json, docs/bench/bench-v3-qwen3-0.6b-Q4_K_M.json, with the earlier revisions beside them as -r3 and -final, and the single-pass originals unchanged.

Every published sample, drawn from the run files linked above. Bars are passes out of samples; the dashed line is 100%. A bar below the line is a task the pipeline did not complete — shown, not hidden.

One sentence to a verified file, warm model, on the 20-core measurement box. The dot is the median, the bar reaches p95, the tick is the worst sample of that run. The 4B's long tail is why it is not the default.

v3 asks for exact counts, so over-delivery is a failure. Both larger tiers clear all three formats; the 0.6B loses five samples, all of them count-sensitive.

SHA-256, for verifying a download:

File	SHA-256
qwen3-1.7b-Q4_K_M.gguf	245362960ec00bfe277c0e06ba9811a5f1a580f6238d3bc50b6fd3a82f031051
qwen3-4b-instruct-2507-Q4_K_M.gguf	3605803b982cb64aead44f6c1b2ae36e3acdb41d8e46c8a94c6533bc4c67e597
qwen3-0.6b-Q4_K_M.gguf	e4c97785c11e2d8547a54fe104f4bedcdf3358d7e95d3f36876ed6caa96ab851
qwen3-1.7b-eagle3.gguf	7fd1b0338f7790fa1f8b372ee72060162c4db74e4efbd89f5b28ebb9a01b4c0b

What it can do

Measured by actually running it — these are observed behaviours, not claims:

Task	Verdict	Notes
Return strictly valid JSON matching a schema	✅ reliable	Enforced by a llama.cpp grammar, so it is structurally impossible to emit malformed JSON
Plan a document outline (3–6 sections)	✅ reliable	Produces sensible, ordered headings
Write 1–3 paragraphs of grammatical prose on a topic	✅ good	Coherent and readable; occasionally generic
Condense source material into 3–5 short bullets	✅ good	Best when given explicit research notes
Follow small constraints ("under 12 words", "no markdown")	✅ usually	Occasionally needs the constraint repeated
Write a whole multi-page document in one pass	❌ no	It drifts, repeats and loses structure — see the design note below
Be factually accurate from memory alone	❌ no	It will invent plausible specifics. Grounding in retrieved text is what makes output trustworthy
Do arithmetic, analysis or comparisons	❌ no	Use Python for that
Cite its sources unprompted	❌ no	Citations are attached by the pipeline, not the model
Keep a consistent voice across many sections	⚠️ partial	Sections are generated independently, so tone can drift

Where the assistant sits: every workflow, not a chat box

The model is one component of an Office-style application, and the interface is built so the assistant is available at each step rather than in a separate tab:

Surface	The user does	The model does	Plain Python does
Create	writes a paragraph brief	plans the outline and short text slots	styles, tables, charts, pagination, file writing
Edit	opens an existing file and says what changes	returns only the allowed replacement/row/slide groups	applies them to a copy and fingerprints the result
Polish	asks for a review	offers editorial suggestions	finds empty slides, blank cells and other mechanical defects
Batch	writes one instruction and selects files	plans each file independently	applies, verifies and reports per file

That is the signature of the product: an Office workspace with AI in every workflow, not an assistant that produces text for a human to paste. The local model is deliberately small because the surrounding code owns structure, file format and verification.

Why this model, and not something bigger

Tested rather than assumed. Six CPU-sized models were run through the real pipeline against identical frozen research notes, and scored on how faithfully they filled the JSON slots:

model	grounding	repetition	verdict
qwen3:1.7b	0.460	0.065	best of the six
phi4-mini:3.8b	0.375	0.062	slower, grounded less
granite4:micro	0.353	0.050	worst instruction-following on shape
qwen3:4b	0.327	0.128	2.4x the parameters, worst repetition
qwen3:0.6b	0.213	0.000	thin, but ~2.4x faster
llama3.2:3b	0.562*	0.000	failed: 2 sections, 118 words, stopped

*tiny-sample artefact; it returned the wrong shape.

Qwen3-1.7B grounds retrieved facts best while staying fastest of the faithful options. On this hardware the bottleneck is tokens per second, not parameters of world knowledge, so a larger model buys nothing measurable and costs multiples of the time per document. Full method and raw data: docs/MODEL-BENCH.md.

How Tiny Agent uses it

The model is never asked to write a document, and it never touches a file. It is asked one small question at a time, and deterministic Python assembles the result:

 1. route    which KINDS of change could this instruction need?
             (replacements | newHeadings | newParagraphs | rows | totals
              | slides | charts | theme)                    <- deterministic
 2. plan     the model fills ONLY those slots, JSON enforced by a llama.cpp
             grammar, so a malformed plan is structurally impossible
 3. apply    python-docx / openpyxl / python-pptx write a NEW file  <- deterministic
 4. verify   fingerprint the file before and after, and report anything that
             changed WITHOUT being asked                      <- deterministic

Step 1 is what makes a model this size dependable: rather than a flat form with eight optional sections to wander around, it is handed only the two or three the instruction actually needs. Steps 3 and 4 mean its output can be checked rather than trusted, which is the difference between a demo and a tool you would point at real work.

Measured with the real model in the product pipeline:

Run	Result
Competitive benchmark, i5-7200U (2 cores / 4 threads, 8 GB, no GPU)	4 of 4 tasks verified; 13.8–34.5 s, mean 19.8 s
Real Microsoft Office opened the four outputs	4 of 4, no repair step
Real Microsoft Office exported them to PDF	4 of 4, content intact
Repeatability harness, reference hardware	12 of 12 plans produced a usable document (3 runs each of a Word edit, a spreadsheet edit, a deck edit and a new Word table)
Warm plan	2.5–5 s typical; slowest measured 27 s (new Word table)

The detailed head-to-head, including where Microsoft Office wins, is in docs/COMPARISON-REPORT.md.

One measured failure worth publishing, because it is the kind of thing a card usually leaves out: offered a single optional group, this model answered with prose in summary and left the group empty — twice in a row, on every run. Naming the group in the retry prompt did not fix it; making that one group required in the retry grammar did, immediately. That is the whole design thesis in one incident: the fix belongs in the shape of the output, not in the politeness of the request.

Each step is a JSON-schema-constrained call. For example, a plan's contract is handed to llama.cpp as a grammar, so the shape is enforced by the decoder rather than requested in the prompt:

{
  "type": "object",
  "properties": {
    "newParagraphs": {
      "type": "array",
      "minItems": 1,
      "items": {"type": "string"}
    }
  },
  "required": ["newParagraphs"]
}

That is why a 1.7B model is usable for a real task: structure, formatting, file writing and verification are handled by code, and the model only supplies short strings where a small model is genuinely competent.

Hardware and measured speed

Reference machine — deliberately weak, and the actual development target:

CPU	Intel Core i5-4460 (Haswell, 4 cores / 4 threads, AVX2)
RAM	16 GB DDR3-1333
GPU	none used
Storage	7200 rpm HDD

The 73-task campaign was also run on a rented Linux box with more cores, so that a full sweep could be repeated without waiting on the desktop:

	20-core measurement box
CPU	20 vCPU, x86-64, one thread per core
RAM	160 GB
GPU	none used
Build	llama.cpp b11714, the same release the Windows installer ships

Model	File size	Cold load	Generation
Qwen3-1.7B Q4_K_M	1.28 GB	~13 s	6.50 tok/s
Qwen3-1.7B Q4_K_M + n-gram speculation	1.28 GB	~13 s	11.46 tok/s (1.76x)
Qwen3-1.7B Q4_K_M + EAGLE3 draft	+0.28 GB	~13 s	4.00 tok/s — slower, see below
Qwen3-0.6B Q4_K_M	0.48 GB	~6 s	~24 tok/s (measured via Ollama)

The generation figures were measured with the exact CPU-only llama.cpp build that ships in the product, 4 threads, a 4096-token context and a 220-token budget. The n-gram row is the mean of repeat runs on the same request.

One thread per physical core, not half of them

Until 21 October the runtime asked for cpu_count() // 2 threads, which is right on a hyper-threaded laptop and wrong everywhere else. On the measurement box that halved the work to 10 threads of 20; on the 4-core desktop it used 2 of 4. The default now reads the physical core count from the kernel (Windows GetLogicalProcessorInformationEx, Linux /proc/cpuinfo), overridable with TINYAGENT_THREADS.

Same 220-token request, same flags, only the thread count changed. Raw runs: docs/bench/thread-sweep-box.json and docs/bench/thread-sweep-laptop.json. The container exports no CPU identifier, so the box file records "cpu": "unknown" and logical_cpus: 20.

What that means in practice

Measured end to end on the shipped runtime, web research enabled, 11 sources retrieved:

Job	Wall clock
3-section researched Word report	202 s
3-slide deck	~112 s

The effective rate is well below the raw tokens/second because prompt processing over injected research context dominates, and every job makes several calls in sequence. That is why the interface streams progress: two to four minutes is perfectly acceptable for a "generate document" button, but only if the user can see which stage is running.

Cold start

Double-click to a usable window, model included, on the reference desktop with its 7200 rpm HDD:

Build	Window answers	First document ready
1.0.0, first build from the measured tree	2.6 s	5.1 s
1.0.0, delivered build	3.3 s	5.7 s

There is no preloading daemon and no service: the window appears first, then the model loads behind it. Source: docs/bench/cold-start.json.

Speculative decoding is not optional on a CPU

Speculative decoding drafts several tokens cheaply and verifies them in one forward pass of the real model. On a CPU — where a forward pass is expensive and a batch is comparatively cheap — this is the single largest speedup available, and it costs nothing in output quality.

llama-server -m qwen3-1.7b-Q4_K_M.gguf --spec-type ngram-mod needs no extra download and measured 6.50 → 11.46 tok/s. It mines repeated token patterns from the context, so it is a warm-session win: the first request of a session gains almost nothing (7.33 tok/s) and the third reaches 11.46 tok/s as the shared hash pool fills. That suits a document generator exactly, because a job is several calls in a row sharing a long system prompt and emitting repetitive JSON.

EAGLE3 is the theoretically stronger option: a one-layer draft that reads the target model's hidden states and should therefore achieve a much higher acceptance rate. This repo ships qwen3-1.7b-eagle3.gguf and it is used like this:

llama-server -m qwen3-1.7b-Q4_K_M.gguf -md qwen3-1.7b-eagle3.gguf \
    --spec-type draft-eagle3 --spec-draft-n-max 8

It measured slower on a CPU, and this card will not pretend otherwise.

Configuration	Speed	Draft acceptance
No speculation	6.50 tok/s	—
n-gram (ngram-mod)	11.46 tok/s	0.53 (warm)
EAGLE3 draft	4.00 tok/s	0.089

Acceptance was only 8.9%, and the draft shares the same 4 physical cores as the target, so it steals more throughput than it saves. EAGLE3 is a large win on a GPU — where the tiny draft runs nearly free beside the target — and a regression on a 4-thread CPU.

This was re-checked on a much faster machine (a 20-core server CPU running CPU-only, 4 threads) using our own converted GGUF: baseline 21.90 tok/s, n-gram 21.60 tok/s on a cold single run, EAGLE3 16.30 tok/s (0.74x). So EAGLE3 is slower regardless of target conversion, and n-gram's benefit genuinely depends on a warm, repetitive session — which is the normal case for this tool.

Prefer n-gram speculation on CPU; treat the EAGLE3 file as an experiment.

Practical consequences:

• Cold load costs ~13 s on a spinning disk. Keep the model resident between requests rather than reloading per call.
• Use one thread per physical core. More threads than physical cores is slower.
• Qwen3 is a reasoning model. Left alone it spends most of its budget on hidden thinking tokens, so disable thinking (think=false, or chat_template_kwargs={"enable_thinking": false}).
• A grounded 3-section report takes ~140 s end to end. That is acceptable for a "generate document" button, but only with progress reporting.

Quick start

llama.cpp server (OpenAI-compatible, supports JSON-schema constrained output):

llama-server -m qwen3-1.7b-Q4_K_M.gguf -c 4096 -t 4 --port 8080

llama.cpp CLI:

llama-cli -m qwen3-1.7b-Q4_K_M.gguf -c 4096 -t 4 \
  -p "Return only JSON: {\"heading\": str, \"paragraphs\": [str]} for a section about heat pumps."

Ollama (optional — the product does not need it):

ollama pull hf.co/hkblockchaintechnology/tiny-agent-gguf

The official CPU-only llama.cpp Windows release is an 18.8 MB zip (a 46 MB folder unpacked) and needs no GPU, CUDA toolkit or Python environment — that is the intended way to ship this.

Prompting notes for a model this size

1. One question per call. Asking for several things at once is where small models fall apart.
2. Constrain the output shape. Use a JSON schema or GBNF grammar. It removes an entire class of parsing failures.
3. Disable thinking. Qwen3 will otherwise spend its whole budget reasoning.
4. Give it the facts. Put retrieved text in the prompt; it cannot supply accurate specifics from memory.
5. Ask for less text. 2–3 short paragraphs per section works; "write a detailed 2000-word report" does not.
6. Expect and clean up artefacts. It adds list numbering to headings and sometimes leaks markdown into prose fields. Strip these deterministically during assembly rather than fighting them with prompt wording.

Limitations

• Hallucination. Without grounding, specifics (figures, dates, product names) are unreliable and often invented. Tiny Agent mitigates this by injecting retrieved web text and attaching a source list, but it does not eliminate it.
• Inherited knowledge cutoff and biases. Because this is a quantization, the base model's training data, cutoff and biases apply unchanged.
• Language. The base model is multilingual, but the pipeline prompts and all validation were done in English.
• Repetition. Sections written in separate calls sometimes restate the same point. Human editing is expected.
• Not a general chat model. It is tuned here for structured slot-filling, not for open-ended conversation.
• Quantization degrades quality slightly versus the full-precision base model. 4-bit is the point where quality and speed balance best on this class of hardware.

Training data

No training or fine-tuning was performed. This repository contains quantized conversions of:

• Qwen/Qwen3-1.7B
• Qwen/Qwen3-4B-Instruct-2507
• Qwen/Qwen3-0.6B

and a converted speculative draft:

• AngelSlim/Qwen3-1.7B_eagle3 (Apache-2.0, Tencent AngelSlim) — converted to GGUF with convert_hf_to_gguf.py --target-model-dir Qwen/Qwen3-1.7B. It is a draft model: it proposes tokens that the 1.7B model verifies, and it never changes the output, only the speed. It cannot be used on its own.

Refer to the base model cards for training data, evaluation results and limitations.

Why these models: both are Apache-2.0, so they can be redistributed with a commercial product without additional conditions. Llama, Gemma and several other otherwise-capable small models are deliberately not bundled, because their licences carry redistribution terms that are unsuitable for shipping to end users.

License and attribution

Apache-2.0, inherited from the base models. Copyright in the base weights belongs to the Qwen team; these are derivative quantizations redistributed under the same licence.

Related

• Project source, benchmarks and generated documents: hkblockchaintechnology/tiny-agent
• Windows installer (1.35 GB, in the dataset repository): TinyAgentSetup-1.1.0.exe
• Every speed number, with its raw run files and the thread-sweep data: docs/SPEED.md
• User guide: TinyAgent-UserGuide.pdf
• The 1.0.0 measurement record: COMPARISON-REPORT.md
• Development logbook, with the commands, failures and artifact hashes behind each claim: LOGBOOK.md
• Problems and questions: discussions on the dataset repository
• Licence: the application's own terms and third-party notices are in docs/LICENSE.txt; the Apache-2.0 text covering these weights is in docs/LICENSE-APACHE-2.0.txt, with the required statement of changes in NOTICE.