dweep-1.7b / README.md
docanan's picture
Rebuild from the refusal adapter: refuses when the corpus has no answer
196ff2f verified
|
Raw History Blame Contribute Delete
10.7 kB
---
license: apache-2.0
base_model: Qwen/Qwen3-1.7B
datasets:
- docanan/andaman-nicobar-corpus
language:
- en
- hi
- ta
- bn
- te
- ml
library_name: transformers
pipeline_tag: text-generation
tags:
- andaman
- nicobar
- indic
- low-resource
- lora
---
# Dweep 1.7B
A LoRA fine-tune of Qwen3-1.7B adapted to the Andaman and Nicobar Islands, trained on a MacBook Air
M5 (Mac17,3), macOS 27.2.1, 16 GB unified memory, in the Andaman and Nicobar Islands.
## Running it
Download the GGUF file once, then answer questions with no network connection. The model, the
adapter and the corpus are free to use and redistribute: Apache-2.0 for the model, CC BY-SA 4.0 for
the corpus. Expect several seconds per answer on a CPU.
## What this is
A small domain-adapted model for questions about the islands: geography, districts, transport,
history, ecology, administration and tourism. It starts from Qwen3-1.7B (Apache-2.0) and inherits
that model's general ability.
## What this is not
- Not the first AI model from the Andaman and Nicobar Islands. The HF account `andamanopal` publishes
reinforcement-learning and robotics policies, including a LeRobot ACT imitation-learning policy
dated 2026-06-28.
- Not trained from scratch and not a general assistant replacement.
- Not trained on news archives. See the dataset card for the exclusion list.
## Prior work in the islands
Named here so nobody has to discover it later.
- **Election Mitra, 2023.** The Andaman and Nicobar Police launched an in-house chatbot trained on
Election Commission handbooks. The force describes it as using machine learning. It is not published
as open weights and it is not a general language model.
- **`aiadvocateindia/Andaman_and_Nicobar`, 2025.** A public dataset of islands legal and regulatory
PDFs, created 2025-09-30. A dataset, not a model.
- **`andamanopal` robotics policies, 2026-06-28.** LeRobot ACT imitation-learning policies for an
SO-101 arm, trained on CUDA. Robotics, not language.
What remains unclaimed: an open, published language model for the islands. That is the claim below.
## Claim basis
Census against the Hugging Face API on 2026-11-16, archived as `claim_census.txt` in the dataset repo:
- `models?search=andaman` returned 12 models; `models?search=nicobar` returned 3. None carries a
`text-generation` pipeline tag.
- An internet search on 2026-11-17 found one island-built chatbot, Election Mitra (2023), and vendor
pages that offer voice agents to the islands. Neither publishes open weights.
- Task filters `text-generation`, `text2text-generation`, `feature-extraction` and
`sentence-similarity` each returned 0 results for `andaman`.
Within that search scope, this is the first open *language* model for the islands.
## Training
| Item | Value |
|---|---|
| Base | Qwen/Qwen3-1.7B, 4-bit MLX quantisation |
| Method | LoRA, rank 8, scale 20 (PEFT alpha 160), last 16 layers, 7 projection modules |
| Data | 3163 validated instruction pairs plus 239 refusal pairs, 3275 rows in total |
| Hardware | MacBook Air M5 (Mac17,3), 16 GB unified, macOS 27.2.1 |
| Iterations | 1600 |
| Tokens seen | 598.7 k |
| Val loss | 4.191 at iter 1 to 1.105 at iter 1600 |
| Throughput (median of reported steps) | 488.5 tokens/s, range 416-661 |
| Checkpoints | 0000400_adapters.safetensors, 0000800_adapters.safetensors, 0001200_adapters.safetensors, 0001600_adapters.safetensors |
| Sequence length / batch | 2048 / 4 |
## How to ask
Use `ask.py` beside the GGUF. It retrieves three passages from the corpus and answers from them.
```
python ask.py "What is unusual about Barren Island?"
```
Direct questions to the bare model answer about 0.48 on held-out data. Questions with retrieved
passages answer about 0.72, but retrieval is word based: a paraphrase that shares no vocabulary with
the source can retrieve the wrong passage, and the model will then answer that wrong passage
confidently. Check the sources that `ask.py` prints.
## Known limits
- Bare-model answers invent detail on topics with thin corpus coverage. The refusal adapter says
"I do not know" for 25 of 32 questions with no supporting text, but it still answers some.
- Retrieval is BM25 over sentence windows. It fails when the question uses different words from the
source. Embedding retrieval was tested and scored lower on this corpus (0.616 against 0.646).
- The 112-question evaluation set was generated from corpus passages, so its wording overlaps the
source. Real user questions score lower. Treat 0.72 as an upper estimate.
## Recommended configuration
Use the refusal adapter with retrieval. It scores 0.558 on the 112 held-out questions and refuses 25
of 32 unanswerable questions. If you need the best raw score and never need a refusal, use the first
adapter with retrieval, which scores 0.646 and refuses none.
## Evaluation
Base and tuned are scored on the same held-out set of 112 in-domain questions, plus a 10-item general sanity set. A regression on the general set is reported, not hidden.
Primary comparison, identical prompt handling for both models (`enable_thinking=False`), greedy
decoding, 120-token budget:
| set | base F1 | tuned F1 | delta |
|---|---|---|---|
| in-domain A&N QA (112) | 0.280 | 0.479 | +0.199 |
| general sanity (10) | 0.106 | 0.308 | +0.203 |
Retrieval A/B, same 112 questions, same tuned model, BM25 index over 1144 articles:
| arm | in-domain F1 (112) | refusals |
|---|---|---|
| tuned, no context | 0.479 | 0 |
| tuned, 3 retrieved passages | **0.646** | 0 |
| tuned, retrieval v2 (language filter, title and entity boost) | 0.636 | 0 |
Delta for the first retrieval build: +0.167. Per question: 72 improve, 38 get worse, 2 stay the same.
The index places a gold answer snippet inside the retrieved passages for 70 of the 112 questions, so
noise in the other 42 explains the regressions. Up to 3 passages per question are kept, and a passage
is kept only when its BM25 score is above zero.
The second build tried a language filter, a title-match boost and an entity boost. It scored 0.636,
which is 0.010 below the first build. The extra filters did not help, and the first build stays in use.
Refusal behaviour: the system prompt asked for "I do not know" when the context lacks the answer, and
both arms produced 0 refusals. A prompt instruction alone did not change this behaviour.
Embedding retrieval test, the step that was meant to raise recall:
| retriever | recall at 3 (112) | retrieved-context F1 (112) |
|---|---|---|
| BM25 | 62 percent | 0.646 |
| multilingual-e5-small embeddings | 63.4 percent | **0.616** |
The embedding index did not beat BM25. Recall moved 1.4 points and F1 fell 0.030, so the embedding
index is dropped.
The real cause of the misses was the unit size, not the scorer. All 31 questions whose answer was
missing from the retrieved passages come from articles that are in the corpus, and 21 of the 31 gold
sentences sit inside their own source article text. Paragraph units of 180 characters or more had
dropped or split those sentences.
Sentence-window index, two sentences per unit with one sentence of overlap, 43279 units:
| index | recall at 3 (112) | retrieved-context F1 (112) |
|---|---|---|
| paragraph units, 3 passages | 62 percent | 0.646 |
| sentence windows, 3 passages | **75.9 percent** | **0.716** |
| sentence windows, 5 passages | 77.7 percent | 0.677 |
| sentence windows, 3 passages, refusal adapter | not measured | 0.648, 9 false refusals |
Sentence windows are the new default. Five passages retrieve slightly more answers but dilute the
prompt, so three stays.
Coverage check on the same 112 questions: 81 of the 112 gold answers appear somewhere in the 5496
paragraph units, and 31 appear nowhere. With sentence windows, 85 of 112 are retrieved. The remaining
27 questions are the ceiling for the current corpus.
Refusal training, a second adapter trained on the same 3163 pairs plus 239 refusal pairs (7.3 percent
of 3275 rows):
| config | in-domain F1 (112) | false refusals on answerable | refusals on 32 unanswerable |
|---|---|---|---|
| adapter without refusal pairs | 0.479 | 0/112 | 0/32 |
| adapter with refusal pairs | 0.477 | 6/112 | **25/32** |
| adapter with refusal pairs and retrieval | 0.558 | 6/112 | not measured |
| adapter without refusal pairs and retrieval | 0.646 | 0/112 | not measured |
Two findings. Refusal pairs work: 0 of 32 unanswerable questions get a refusal before training, 25 of
32 after, and answerable F1 moves only 0.002. Refusal training also costs part of the retrieval gain:
the same retrieval setup scores 0.646 on the first adapter and 0.558 on the refusal adapter, a drop of
0.088. Both models answer more questions correctly with retrieved passages than without.
Earlier run, default Qwen3 template (thinking mode on) for both models:
| set | base F1 | tuned F1 | delta |
|---|---|---|---|
| in-domain A&N QA (112) | 0.115 | 0.452 | +0.337 |
| general sanity (10) | 0.026 | 0.254 | +0.228 |
Correction. The first run left the thinking block inside the scored output. In that run only 1 of the
112 base-model predictions contained a closing think tag, so 111 predictions carried no text after a
reasoning block and scored zero. That inflated the gap. With thinking mode off for both models the
honest in-domain gain is +0.199, not +0.337, and the general gain is +0.203, not +0.228.
The tuned model still beats the base model on both sets. The gain is real but smaller than the first
run showed.
## Limitations
Small model, narrow corpus, encyclopedic register. Expect weak performance on administrative
procedure, legal and medical questions. Hindi, Tamil, Bengali, Telugu and Malayalam coverage is
thinner than English. Nicobarese and Great Andamanese are deliberately absent. Instruction-pair
phrasing comes from a teacher model, so stylistic quirks may be inherited.
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("andaman-ai/dweep-1.7b")
model = AutoModelForCausalLM.from_pretrained("andaman-ai/dweep-1.7b", torch_dtype="bfloat16")
msgs = [{"role": "user", "content": "Which strait separates the Andamans from the Nicobars?"}]
prompt = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False,
enable_thinking=False)
print(tok.decode(model.generate(**tok(prompt, return_tensors="pt"), max_new_tokens=120)[0]))
```
`enable_thinking=False` removes the thinking block. On a 20-item check it also scored slightly
higher than the default template (F1 0.436 against 0.429). If your pipeline cannot pass the flag,
strip text up to `</think>` before you display the answer.