Update README.md

README.md

@@ -1,6 +1,5 @@
 ---
-license: other
-license_name: todo-confirm
+license: apache-2.0
 language:
   - en
 library_name: peft
@@ -25,62 +24,72 @@ model-index:
         metrics:
           - type: bertscore
             name: BERTScore F1 (roberta-large)
-            value: 0.902252
+            value: 0.897506
           - type: f1
             name: Token F1
-            value: 0.404100
+            value: 0.371768
           - type: exact_match
             name: Exact match
-            value: 0.004734
+            value: 0.002367
 ---
 
 # SpaceLLM Single-Turn QA
 
-A LoRA adapter for [`openai/gpt-oss-20b`](https://huggingface.co/openai/gpt-oss-20b), fine-tuned to answer single-turn questions about space-agency material. This repository contains **adapter weights only**; the base model must be loaded separately.
+A LoRA adapter for [`openai/gpt-oss-20b`](https://huggingface.co/openai/gpt-oss-20b), fine-tuned for single-turn question answering over space-agency-derived text. This repository contains **adapter weights only**; the base model must be loaded separately.
 
-- **Developed by:** TODO_AUTHORS
-- **Paper:** TODO_PAPER_LINK
-- **Model type:** LoRA adapter (PEFT)
+- **Developed by:** Aditya Pratap Singh et al.
+- **Model type:** PEFT LoRA adapter (`CAUSAL_LM`)
 - **Language:** English
-- **Base model:** `openai/gpt-oss-20b` (21B total / 3.6B active parameters), revision `6cee5e81ee83917806bbde320786a8fb61efebee`
-- **Companion model:** SpaceLLM Multi-Turn QA (separate adapter, separate repository)
+- **Base model:** `openai/gpt-oss-20b`, revision `6cee5e81ee83917806bbde320786a8fb61efebee`
+- **Companion model:** SpaceLLM Multi-Turn QA — a separate adapter trained on a different data format and evaluated on a different test set. **The two adapters' scores are not a head-to-head comparison.**
 
 > **Please note**
-> - This adapter does **not** use retrieval (no RAG, index, or retrieved passages). Nothing is looked up at inference time.
-> - It adapts attention projections only (`q_proj`, `k_proj`, `v_proj`, `o_proj`), not `lm_head`.
-> - Training references were generated by a teacher model and were **not fact-checked**. Scores measure similarity to those references, not verified factual accuracy.
-> - This is a different artifact from the older `AdityaPS/SpaceLLM_v1` (lm_head-only, rank 32, alpha 128, different data and evaluation).
+> - This adapter uses **no retrieval**. There is no RAG pipeline, no index, and nothing looked up at inference time.
+> - LoRA targets attention projections only — `q_proj`, `k_proj`, `v_proj`, `o_proj` — **not** `lm_head`.
+> - Training and evaluation references were generated by a teacher model and have **not been independently fact-checked**. Reported scores measure similarity to those references, not verified factual accuracy.
+> - This is a distinct artifact from the older [`AdityaPS/SpaceLLM_v1`](https://huggingface.co/AdityaPS/SpaceLLM_v1) checkpoint, which used a different configuration (`lm_head`-only, rank 32) and different data/evaluation. If `SpaceLLM_v1` remains public, treat it as a separate, earlier experiment — its card and weights should not be conflated with this one.
+
+---
 
 ## Uses
 
-**Intended:** research on domain-adapted QA over space-agency-style text, and as a baseline against the multi-turn adapter.
+**Intended:**
+- Research on parameter-efficient domain adaptation for space-agency-style QA.
+- Serving as a single-turn baseline in comparative dialogue/QA evaluations.
+
+**Out of scope:**
+- Mission-critical operations, spacecraft control, orbital mechanics execution, or any safety-critical or autonomous decision system.
+- Any use where an unverified or incorrect answer could cause harm.
+- Multi-turn dialogue (use the companion multi-turn adapter, evaluated separately, for that).
 
-**Out of scope:** safety-critical or engineering decisions, mission planning, or any setting where an incorrect answer could cause harm. Multi-turn dialogue is not evaluated for this adapter.
+Expert verification is recommended before relying on any specific technical claim this model produces.
+
+---
 
 ## How to get started
 
-The example loads the pinned base model and this adapter, applies the Harmony chat template, forces the assistant **final** channel, and prints only the final answer. Do not print a full `skip_special_tokens=True` decode, since it can expose analysis-channel text.
+The example loads the pinned base revision and this adapter, applies the Harmony chat template, forces the assistant **final** channel, and extracts only that channel's text — the same approach used in evaluation. Relying on `skip_special_tokens=True` alone is avoided because it can mix analysis-channel and final-channel text.
 
 ```python
 import torch
 from transformers import AutoModelForCausalLM, AutoTokenizer, Mxfp4Config
 from peft import PeftModel
 
-BASE = "openai/gpt-oss-20b"
+BASE_MODEL = "openai/gpt-oss-20b"
 BASE_REVISION = "6cee5e81ee83917806bbde320786a8fb61efebee"
-ADAPTER = "TODO_REPO_ID"
+ADAPTER_REPO = "AdityaPS/SpaceLLM-SingleTurn"
 
-tokenizer = AutoTokenizer.from_pretrained(ADAPTER)
-base = AutoModelForCausalLM.from_pretrained(
-    BASE,
+tokenizer = AutoTokenizer.from_pretrained(ADAPTER_REPO)
+base_model = AutoModelForCausalLM.from_pretrained(
+    BASE_MODEL,
     revision=BASE_REVISION,
     torch_dtype=torch.bfloat16,
     quantization_config=Mxfp4Config(dequantize=True),
     device_map="auto",
 )
-model = PeftModel.from_pretrained(base, ADAPTER).eval()
+model = PeftModel.from_pretrained(base_model, ADAPTER_REPO).eval()
 
-question = "What was the primary goal of the Mars Science Laboratory mission?"
+question = "What was the primary objective of the Mars Science Laboratory mission?"
 prompt = tokenizer.apply_chat_template(
     [{"role": "user", "content": question}],
     tokenize=False,
@@ -89,90 +98,135 @@ prompt = tokenizer.apply_chat_template(
 prompt += "<|channel|>final<|message|>"  # force the final channel
 
 inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
-with torch.no_grad():
-    out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
 
-text = tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False)
-for stop in ("<|return|>", "<|end|>"):
-    text = text.split(stop)[0]
-print(text.strip())
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=512,  # inference-example setting; evaluation used 2048 (see below)
+        do_sample=False,
+        repetition_penalty=1.05,
+        no_repeat_ngram_size=6,
+    )
+
+generation = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False)
+for stop_token in ("<|return|>", "<|end|>"):
+    generation = generation.split(stop_token)[0]
+
+print("Answer:", generation.strip())
 ```
 
+*Note: this example uses `max_new_tokens=512` for a quick, practical demo. This differs from the 2,048-token limit used during evaluation (see [Evaluation](#evaluation)) — treat this as an inference-example setting, not the evaluated protocol.*
+
+---
+
 ## Training details
 
 ### Training data
 
-Built offline in five steps:
+Data was built offline, with no retrieval stage used anywhere in training or inference:
 
 1. Space-agency text was scraped and stored as JSON.
-2. The text was cleaned and split into chunks.
-3. Each chunk was passed directly to a teacher model (Mistral `mistral:7b`).
-4. The teacher generated single-turn question-answer pairs.
-5. This adapter was trained on those pairs. (The multi-turn adapter uses three-level dialogue chains and is trained separately.)
+2. Text was normalized and split into chunks.
+3. Each chunk was passed directly to an Ollama `mistral:7b` teacher model.
+4. The teacher was prompted to generate **five standalone single-turn question–answer pairs per chunk**, with answers requested to be grounded in the supplied chunk.
+5. The generation pipeline did **not** independently verify that answers were entailed by their source chunk — some answers may be unsupported or incorrect.
 
-| Split | Records |
-|---|---|
+| Split | Usable examples |
+|---|---:|
 | Train | 14,395 |
 | Validation | 1,695 |
-| Test | 845 |
-
-TODO_VERIFY: confirm the counts against the JSONL files and state whether any records were filtered or skipped. Add source-record and chunk counts, agency distribution, splitting unit, duplicate handling, and provenance from `metadata.csv` and `manifest.json`.
+| Test (evaluated) | 845 |
 
-Teacher prompts ask for answers supported by the supplied chunk, but the pipeline does not verify that answers are entailed by their sources. Some answers may be unsupported or wrong. Source-text use is subject to the original publishers' terms (TODO_CONFIRM).
+**These are the only counts supported for this artifact.** A previously circulated figure of "3,428 documents / 16,831 training samples" does not match this adapter's training/evaluation split counts or file hashes and should not be cited for this repository.
 
 ### Training hyperparameters
 
 | Setting | Value |
 |---|---|
-| Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj` |
-| LoRA rank / alpha / dropout | 16 / 32 / 0.05 |
-| Epochs | 3 |
-| Learning rate | 5e-6 |
-| Batch size / gradient accumulation | 1 / 16 (effective 16) |
-| Max sequence length | 2048 |
-| Loss | Final assistant-answer tokens only |
-| Compute dtype | BF16 (FP16 off) |
-| Base weights | MXFP4 dequantization requested |
-
-The `fp16` string in some output directory names is historical and does not reflect the actual training regime.
+| LoRA target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj` |
+| LoRA rank ($r$) / alpha ($\alpha$) | 16 / 32 |
+| LoRA dropout | 0.05 |
+| Task type | `CAUSAL_LM` |
+| Max sequence length | 2,048 tokens |
+| Base weights | MXFP4, dequantized at load time |
+| Compute precision | BF16 |
+
+The base model is the published MXFP4 `gpt-oss-20b` checkpoint, loaded with MXFP4 dequantization and run in BF16 compute — this is a loading/compute configuration, not a separately distributed "BF16 version" of the model.
+
+---
 
 ## Evaluation
 
-**Test set:** 845 single-turn records. SHA-256 of the test file: `337a80d2375a053ab7d8e9e1c5ee2ec7b975f74bfe16db13bf0d1c5a60aa91f6`.
+**Test set:** 845 single-turn records.
+**Test file SHA-256:** `337a80d2375a053ab7d8e9e1c5ee2ec7b975f74bfe16db13bf0d1c5a60aa91f6`
+**Adapter config SHA-256:** `ef641c55f5f961a86be20f364b39b80f7777053959caff1523b2348f00432748`
+
+**Protocol:** Base and adapter used identical tokenized prompts. All 845 records produced a final answer; none were skipped. Generation was deterministic (`do_sample=False`) with `max_seq_len=2048`, `max_new_tokens=2048`, `repetition_penalty=1.05`, and `no_repeat_ngram_size=6`, applied to both base and adapter. Only text from the Harmony `final` channel was scored; analysis-channel text was excluded.
 
-**Protocol:** same base revision and prompts for base and adapter; `final_channel` response mode with answer-only extraction; `max_new_tokens=512`; BERTScore with `roberta-large`; missing answers score zero and all valid records stay in the denominator (there were none missing).
+**Metrics:** BERTScore computed with `roberta-large`; token F1 computed on lowercased whitespace tokens; exact match computed after stripping surrounding whitespace. A missing answer would score zero on all metrics while remaining in the denominator — no answers were missing for this test set.
+
+References were produced by the same teacher-based generation process used for training data and have not received independent factuality verification.
 
 ### Results
 
-| Metric | Base | + Adapter |
-|---|---|---|
-| BERTScore F1 | 0.823306 | **0.902252** |
-| Token F1 | 0.140682 | **0.404100** |
-| Exact match | 1 / 845 | 4 / 845 |
-| Outputs hitting the 512-token cap | 384 | 28 |
+| Model | BERTScore P | BERTScore R | BERTScore F1 | Token F1 | Exact match | Hit 2,048-token cap |
+|---|---:|---:|---:|---:|---:|---:|
+| Base (`gpt-oss-20b`) | 0.781081 | 0.869184 | 0.822320 | 0.122433 | 0 / 845 | 3 / 845 |
+| + SpaceLLM Single-Turn Adapter | 0.905112 | 0.890707 | 0.897506 | 0.371768 | 2 / 845 | 1 / 845 |
+
+**BERTScore F1 change: an absolute gain of 0.075186.** This is not a "6% improvement," and it is not evidence of improved factual accuracy or domain understanding — it reflects greater semantic/stylistic similarity to teacher-generated references under this specific protocol.
 
-The BERTScore F1 change is an **absolute gain of 0.078946**, not a percentage improvement. It is not evidence of verified factuality or domain understanding. References are teacher-generated, and independent factuality review is pending. Part of the gain likely reflects the adapter matching the style and length of the reference answers. Single-turn and multi-turn results are reported separately and are not merged.
+### Reading the weaker metrics honestly
 
-**Reproducibility:** TODO_LINK to `evaluation_config.json`, `comparison.json`, `summary.txt`, and sample-level predictions.
+- **Exact match is very low (2/845, ≈0.24%) for both models**, which is expected for free-form generative QA — a correct answer rarely matches a reference string verbatim in wording, order, or length. This metric is reported for completeness, not as a proxy for correctness rate.
+- **Token F1 (0.372) is modest in absolute terms**, even though it nearly tripled over the base model's 0.122. Token overlap penalizes any answer that is phrased differently from the reference, including answers that may be factually equivalent. Treat the *relative* change as a signal of closer stylistic alignment, not the *absolute* value as an accuracy score.
+- **BERTScore is the most informative similarity signal here, but it has the same ceiling as its references**: since references are teacher-generated and unverified, a higher BERTScore partly reflects the adapter reproducing the teacher's phrasing habits — including any errors or unsupported claims the teacher made — rather than confirmed improvement in factual correctness.
+
+---
+
+## Generation stability
+
+An earlier unguarded diagnostic run at the same 2,048-token maximum, without repetition controls, produced outputs that fell into **exact phrase loops**. The final evaluation protocol added `repetition_penalty=1.05` and 6-token n-gram blocking (`no_repeat_ngram_size=6`) to **both** base and adapter generations to address this.
+
+| Model | Unguarded diagnostic (cap events) | Guarded final run (cap events) | Reduction |
+|---|---:|---:|---:|
+| Base | 35 | 3 | 91.4% |
+| Adapter | 28 | 1 | 96.4% |
+
+This is reported as an **engineering diagnostic, not a controlled ablation**: prompt hashes changed across rebuilt evaluation images between the unguarded and guarded runs, so the reduction cannot be attributed to the decoding settings alone. Additionally, the guarded settings removed the conspicuous exact-repetition loops but did **not** eliminate semantic degeneration generally — a small number of guarded outputs still reached the token cap while producing drifting numerical lists or otherwise unrelated text.
+
+---
 
 ## Bias, risks, and limitations
 
-- Answers may be fluent but incorrect. There is no retrieval and no source citation.
+- **The supported conclusion is narrow:** these results show improved similarity to synthetic, teacher-generated references under this specific evaluation protocol. They do not establish improved factual accuracy, reasoning ability, or general domain understanding.
+- **Exact match (2/845) and token F1 (0.372) are low in absolute terms.** This is a known property of free-form generative QA evaluated against a single reference string, not necessarily a sign the model is "wrong" most of the time — but it also means these numbers cannot be read as an accuracy rate.
+- **No independent factuality check exists.** References were generated by the same `mistral:7b` teacher process used for training data; higher similarity scores may partly reflect the adapter learning to imitate the teacher's style, length, and any of its errors, rather than confirmed improvements in correctness.
+- **The exact training/evaluation data snapshot has not yet been recovered and published**; the dataset manifest, source-agency distribution, and dedup/filtering details are pending.
+- **This adapter has not been validated for operational, mission-critical, or safety-critical use of any kind**, including spacecraft control, mission planning, or autonomous decision-making.
+- There is no retrieval and no source citation — answers cannot be traced back to a specific source document by the model itself.
+- `no_repeat_ngram_size=6` can suppress legitimate repeated technical phrases (e.g., a mission name or unit repeated for clarity), which may slightly affect fluency on some answers.
+- Generation stability numbers above are diagnostic, not a controlled ablation (see [Generation stability](#generation-stability)).
+- Single-turn and multi-turn adapters were trained on different formats and evaluated on different test files; **do not compare their scores directly**.
 - Knowledge is fixed at training time and may be outdated.
-- Overlap metrics reward similarity to teacher references; low exact-match rates are expected for free-form answers.
 - English only.
-- Limitations and terms of the base model `openai/gpt-oss-20b` also apply.
+- General limitations and terms of the base model `openai/gpt-oss-20b` also apply.
 
-## Files
+---
 
-`adapter_model.safetensors`, `adapter_config.json`, tokenizer files, `chat_template.jinja`, `README.md`. Checksums match the archived adapter: TODO_ADD_SHA256.
+## Reproducibility artifacts
 
-## License
+- **Test set SHA-256:** `337a80d2375a053ab7d8e9e1c5ee2ec7b975f74bfe16db13bf0d1c5a60aa91f6`
+- **Adapter config SHA-256:** `ef641c55f5f961a86be20f364b39b80f7777053959caff1523b2348f00432748`
 
-TODO_CONFIRM. It must be consistent with the base model's licence, the teacher model's terms for generated outputs, and the source-text terms. Once confirmed, replace `license: other` and `license_name` in the metadata with the correct identifier.
+---
+
+## Files
 
-## Citation
+`adapter_model.safetensors`, `adapter_config.json`, tokenizer files, `chat_template.jinja`, `README.md`.
+
+---
+
+## License
 
-```bibtex
-TODO_BIBTEX
-```
+Apache-2.0, matching the base model (`openai/gpt-oss-20b`). Note: this covers the adapter weights and code; it does not by itself establish licensing terms for the scraped source text or the teacher-generated training data used to produce this adapter.