Text Generation
Transformers
Safetensors
English
llama
wikipedia
facts
retrieval
question-answering
smollm2
text-generation-inference
wiki-model / README.md
Lucasgitguy-co's picture
Link the GGUF repo
83990db verified
|
Raw History Blame Contribute Delete
4.69 kB
---
license: apache-2.0
language: [en]
pipeline_tag: text-generation
library_name: transformers
base_model: HuggingFaceTB/SmolLM2-135M
tags: [wikipedia, facts, retrieval, question-answering, smollm2]
datasets:
- Lucasgitguy-co/wiki-model-facts
- wikimedia/structured-wikipedia
- Bulutthecat/FactfulDataset
---
# wiki-model
**GGUF:** [Lucasgitguy-co/wiki-model-GGUF](https://huggingface.co/Lucasgitguy-co/wiki-model-GGUF) (for llama.cpp).
[SmolLM2-135M](https://huggingface.co/HuggingFaceTB/SmolLM2-135M), fully fine-tuned to state facts from Wikipedia.
It does four things: a fact about a topic, a random fact, short question answering, and correcting a false statement.
With `--grounded` it answers from a sourced Wikipedia index and prints the article it used.
The from-scratch 12.5M sibling is [wiki-model-tiny](https://huggingface.co/Lucasgitguy-co/wiki-model-tiny).
Data and index: [wiki-model-facts](https://huggingface.co/datasets/Lucasgitguy-co/wiki-model-facts).
> Small models get facts wrong when they answer from memory. Use `--grounded` for anything that matters.
## Use
```bash
hf download Lucasgitguy-co/wiki-model --local-dir wiki-model
cd wiki-model && pip install -e .
wiki-model "Eiffel Tower" # picks the 135M weights in this folder
wiki-model --q "Who wrote Hamlet?" --grounded # first call downloads the ~600 MB fact index (skip with --no-index)
```
<details><summary>transformers only</summary>
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Lucasgitguy-co/wiki-model")
m = AutoModelForCausalLM.from_pretrained("Lucasgitguy-co/wiki-model")
x = tok("<q> Who wrote Hamlet? </q>", return_tensors="pt", add_special_tokens=False)
print(tok.decode(m.generate(**x, max_new_tokens=32, do_sample=False)[0][x.input_ids.shape[1]:], skip_special_tokens=True))
```
Prompts: `<fact> Topic </fact>`, `<q> Question? </q>`, `<fix> Claim. </fix>`, optionally preceded by `<ctx> article text </ctx> `.
</details>
## Results
Held-out articles and claims (test split by article title). "Grounded" retrieves context at run time with the BM25 index.
"Fact supported" = the generated sentence matches a sentence of the topic's own article with token F1 ≥ 0.5.
"SmolLM2-135M few-shot" is the untuned base model with three examples in the prompt (no context in the examples); "copy the claim" just repeats the input claim (fix-task baseline).
"Fix (year swap)": shifted-year sentences mixed with unchanged true sentences (71% of the set), so copying already scores 71.
Answering from memory, the base model's facts are supported slightly more often than the fine-tune's: fine-tuning taught the task format and how to use retrieved context, not new knowledge.
| model | mode | Q&A infobox EM | 'What is X' F1 | Fix (FEVEROUS) EM | Fix can't-verify acc | Fix (year swap) EM | Fact supported (new topics) | Fact supported (known topics) |
|---|---|---|---|---|---|---|---|---|
| wiki-model | model only | 14.7 | 30.0 | 37.0 | 88.5 | 65.0 | 12.0 | 10.0 |
| wiki-model | grounded | 94.7 | 65.3 | 37.0 | 88.0 | 83.0 | 98.0 | 98.0 |
| wiki-model-tiny | model only | 14.7 | 28.2 | 32.5 | 89.0 | 54.0 | 5.0 | 8.0 |
| wiki-model-tiny | grounded | 90.7 | 58.0 | 33.5 | 89.0 | 55.0 | 92.0 | 94.0 |
| SmolLM2-135M few-shot | model only | 10.0 | 18.9 | 11.0 | 11.0 | 66.0 | 16.0 | 16.0 |
| SmolLM2-135M few-shot | grounded | 26.7 | 25.2 | 6.5 | 36.5 | 47.0 | 65.0 | 71.0 |
| copy the claim | model only | — | — | 47.5 | 89.0 | 71.0 | — | — |
## Training
| | |
|---|---|
| Base | HuggingFaceTB/SmolLM2-135M (Apache-2.0) |
| Data | 40k task examples (35% with retrieved context) sampled from wiki-model-facts + FactfulDataset, then a fix top-up of 11k examples (FactfulDataset with retrieved context + replay), then a grounding top-up of 24k examples (mostly with retrieved context) |
| Recipe | full fine-tune, AdamW lr 1e-4 cosine (top-ups 5e-5), batch 16, loss on answers only, bf16 autocast, gradient checkpointing |
| Hardware | one Apple M4 (MPS), shared with another training job, about 8 hours |
See [wiki-model-tiny](https://huggingface.co/Lucasgitguy-co/wiki-model-tiny) for how the task data is built.
## Limits
- Answers from memory are often wrong for less-known topics. Grounded answers are only as good as retrieval.
- English only; Wikipedia snapshot from 2024.
- Fixing false statements is the weakest task: compare with the "copy the claim" row before relying on it. "Can't verify" is rarely produced.
## Licence
Weights and code: Apache-2.0 (same as the base). Training text comes from Wikipedia (CC BY-SA); FactfulDataset has no licence on its card and is not redistributed.