docanan commited on
Commit
196ff2f
·
verified ·
1 Parent(s): 1482651

Rebuild from the refusal adapter: refuses when the corpus has no answer

Browse files
README.md CHANGED
@@ -25,6 +25,12 @@ tags:
25
  A LoRA fine-tune of Qwen3-1.7B adapted to the Andaman and Nicobar Islands, trained on a MacBook Air
26
  M5 (Mac17,3), macOS 27.2.1, 16 GB unified memory, in the Andaman and Nicobar Islands.
27
 
 
 
 
 
 
 
28
  ## What this is
29
 
30
  A small domain-adapted model for questions about the islands: geography, districts, transport,
@@ -81,6 +87,28 @@ Within that search scope, this is the first open *language* model for the island
81
  | Checkpoints | 0000400_adapters.safetensors, 0000800_adapters.safetensors, 0001200_adapters.safetensors, 0001600_adapters.safetensors |
82
  | Sequence length / batch | 2048 / 4 |
83
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
84
  ## Recommended configuration
85
 
86
  Use the refusal adapter with retrieval. It scores 0.558 on the 112 held-out questions and refuses 25
 
25
  A LoRA fine-tune of Qwen3-1.7B adapted to the Andaman and Nicobar Islands, trained on a MacBook Air
26
  M5 (Mac17,3), macOS 27.2.1, 16 GB unified memory, in the Andaman and Nicobar Islands.
27
 
28
+ ## Running it
29
+
30
+ Download the GGUF file once, then answer questions with no network connection. The model, the
31
+ adapter and the corpus are free to use and redistribute: Apache-2.0 for the model, CC BY-SA 4.0 for
32
+ the corpus. Expect several seconds per answer on a CPU.
33
+
34
  ## What this is
35
 
36
  A small domain-adapted model for questions about the islands: geography, districts, transport,
 
87
  | Checkpoints | 0000400_adapters.safetensors, 0000800_adapters.safetensors, 0001200_adapters.safetensors, 0001600_adapters.safetensors |
88
  | Sequence length / batch | 2048 / 4 |
89
 
90
+ ## How to ask
91
+
92
+ Use `ask.py` beside the GGUF. It retrieves three passages from the corpus and answers from them.
93
+
94
+ ```
95
+ python ask.py "What is unusual about Barren Island?"
96
+ ```
97
+
98
+ Direct questions to the bare model answer about 0.48 on held-out data. Questions with retrieved
99
+ passages answer about 0.72, but retrieval is word based: a paraphrase that shares no vocabulary with
100
+ the source can retrieve the wrong passage, and the model will then answer that wrong passage
101
+ confidently. Check the sources that `ask.py` prints.
102
+
103
+ ## Known limits
104
+
105
+ - Bare-model answers invent detail on topics with thin corpus coverage. The refusal adapter says
106
+ "I do not know" for 25 of 32 questions with no supporting text, but it still answers some.
107
+ - Retrieval is BM25 over sentence windows. It fails when the question uses different words from the
108
+ source. Embedding retrieval was tested and scored lower on this corpus (0.616 against 0.646).
109
+ - The 112-question evaluation set was generated from corpus passages, so its wording overlaps the
110
+ source. Real user questions score lower. Treat 0.72 as an upper estimate.
111
+
112
  ## Recommended configuration
113
 
114
  Use the refusal adapter with retrieval. It scores 0.558 on the 112 held-out questions and refuses 25
model-00001-of-00002.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:a8fc9e0bdde8032f3f7621119e0c2cfe20e721e3a2c3b898478c41055ecdb0b9
3
  size 1981418432
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3902f2dd492b17b75bdf8cabb27547d3e7eba815d756ceff9112857f18b9bc11
3
  size 1981418432
model-00002-of-00002.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:c5fecb8004be5eb42023441350a4581f9a6dcada00594aee6ff33e24595da078
3
  size 1459766976
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:93928e86d8e2045c5f7f44b56684dbfd6f629c3adc2439fd85d331bc05235334
3
  size 1459766976