File size: 15,555 Bytes
a132be9
a906787
 
a132be9
a906787
 
 
 
 
 
 
 
 
 
 
 
 
 
 
a132be9
a906787
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
cb66b20
a906787
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
03e4524
 
a906787
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
---
language:
- en
license: apache-2.0
library_name: sentence-transformers
base_model: google/embeddinggemma-2
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- embeddings
- retrieval
- semantic-search
- matryoshka
- nasa
- combustion
- microgravity
- spacecraft-fire-safety
---

# FlameLedger Encoder: EmbeddingGemma 2 fine-tuned for NASA combustion research retrieval

A text embedding model for finding passages in NASA microgravity-combustion and spacecraft fire-safety literature. It is [`google/embeddinggemma-2`](https://huggingface.co/google/embeddinggemma-2) with its text encoder fine-tuned on synthetic search queries over public NASA Technical Reports Server (NTRS) records.

Built by **Team Qubit** for the NASA Space Apps Challenge 2026 (*Flame in Freefall*) and used in production by [FlameLedger](https://flameledger.agenticerastudio.tech), an evidence research console that ranks NASA/NIST combustion experiments against a proposed cabin environment.

- **Live demo:** https://flameledger.agenticerastudio.tech
- **Code:** https://github.com/Muhammad-Hassan12/FlameLedger
- **Embedding size:** 768 (Matryoshka: 768 / 512 / 256 / 128)
- **Language:** English
- **License:** Apache 2.0 (see [License and attribution](#license-and-attribution))

> **Read this first.** This is a small, domain-specific fine-tune (1,025 training triplets, 486 passages). Gains over the base model are measured on teacher-written synthetic queries and are modest. On a small teacher-drafted question set (n = 23, unreviewed) no improvement was detectable. BM25 is a strong baseline on this corpus, and this model is intended to be used *together with* BM25. See [Evaluation](#evaluation) and [Limitations](#limitations).

---

## Model details

| | |
|---|---|
| **Base model** | `google/embeddinggemma-2` (Google DeepMind, 740M total parameters; 270M text backbone) |
| **What was fine-tuned** | Text tower only (about 271M trainable parameters). Vision and audio towers were frozen and are unused. |
| **This repository** | Text-only encoder export (FP32). Image and audio inputs are not supported. |
| **Output** | 768-dimensional embeddings, truncatable to 512 / 256 / 128 (Matryoshka) |
| **Similarity** | Cosine similarity (L2-normalize embeddings) |
| **Max sequence length used in training** | 512 tokens (base model supports up to 8K) |
| **Precision** | Use FP32 or BF16. The base model card warns against FP16. |
| **Developed by** | Syed Muhammad Hassan (Team Qubit)|

---

## Intended use

**Intended:**
- Semantic and hybrid (dense + BM25) retrieval over NASA combustion, microgravity, and spacecraft fire-safety technical reports.
- First-stage candidate retrieval in a RAG or research-assistant pipeline where results are shown with their sources.

**Not intended:**
- Fire-safety decision-making, material qualification, or any safety-critical judgment. Retrieval relevance is not scientific correctness, and the system must not be used as a substitute for expert review.
- General-purpose or multilingual embedding. Use the base model for that.
- Answer verification. This model ranks passages. It does not check claims.

---

## How to use

### Prefixes are required

The model was trained with fixed text prefixes. **Use exactly these formats for queries and documents**, or retrieval quality will silently degrade.

```python
QUERY_PREFIX = "task: search result | query: "

def fmt_query(q: str) -> str:
    return QUERY_PREFIX + " ".join(str(q).split())

def fmt_doc(title: str, text: str) -> str:
    t = " ".join((title or "").split()) or "none"
    return f"title: {t} | text: {str(text).strip()}"
```

### Sentence Transformers

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("YOUR_HF_USERNAME/flameledger-embeddinggemma2-nasa-combustion")
model.max_seq_length = 512

queries = [fmt_query("At what oxygen concentration does flame spread stop in low-speed opposed flow?")]
docs = [
    fmt_doc("Combustion of solid fuel in very low speed oxygen streams",
            "Flame spread over thin solid fuel in quiescent and low-speed flows ..."),
    fmt_doc("Particle and Smoke Detection on ISS for Next Generation Smoke Detectors",
            "Smoke detector response depends on particle size distribution ..."),
]

# Prefixes are added manually above. Use .encode() so no extra prompt is prepended.
q = model.encode(queries, normalize_embeddings=True)
d = model.encode(docs, normalize_embeddings=True)

scores = q @ d.T          # cosine similarity
print(scores)
```

### Using a smaller dimension (Matryoshka)

```python
import numpy as np

def truncate(emb, dim=256):
    emb = emb[:, :dim]
    return emb / np.linalg.norm(emb, axis=1, keepdims=True)

q256, d256 = truncate(q, 256), truncate(d, 256)
```

Re-normalize after truncating. See the [dimension results](#effect-of-embedding-dimension) before going below 256.

### Hybrid retrieval (recommended)

The best configuration in our evaluation fuses dense and lexical rankings with Reciprocal Rank Fusion:

- Dense: cosine similarity at 768 dimensions.
- Lexical: BM25 (k1 = 1.5, b = 0.75) over `title + " " + text`, tokenized as lowercase alphanumerics (keeping internal `.` and `-`, e.g. `6061-t6`).
- Fusion: RRF with k = 60 over the top 100 of each ranking.

A reference implementation (`HybridRetriever`, `BM25`, `rrf_scores`) is in the [FlameLedger repository](https://github.com/Muhammad-Hassan12/FlameLedger).

---

## Training

### Data

Training data is synthetic. No human-written training questions were used.

| Stage | Count |
|---|---|
| NTRS records fetched | 416 |
| On-topic papers after de-duplication and topic filter | 399 |
| Passages (chunks) | 486 (399 abstracts + 87 body chunks from 10 full-text PDFs) |
| Valid teacher queries after rule-based validation | 1,412 (train 1,128 / dev 134 / test 150) |
| Training triplets after round-trip filter | 1,025 (dev: 134) |

Most papers are represented by their abstract only, because scanned PDFs were skipped rather than guessed.

**Query generation.** Gemma 4 12B instruction-tuned (`google/gemma-4-12B-it`) wrote three queries per passage in three styles: `keyword`, `question`, and `scenario` (an engineering need the passage helps with). Teacher output was kept only if it passed rule checks:

- length limits per style,
- no meta-references such as "the passage" or "this paper",
- no verbatim copying (no 7-word run copied from the passage),
- every multi-digit or decimal number in the query must appear in the passage (blocks invented numbers),
- de-duplication.

**Round-trip filter (train split only).** Queries for which the base model ranked the source passage outside its top 15 were dropped (kept 1,025 of 1,128, or 91%).

**Hard negatives.** Three per query, mined with the base model from the same split only, never from the same paper, skipping near-duplicates (cosine ≥ 0.92) and candidates scoring within 5% of the positive, to reduce false negatives.

**Leak-free split.** Train / dev / test are split **by paper** (321 / 39 / 39 papers), with papers sharing a normalized title kept in the same split.

### Procedure

| | |
|---|---|
| **Loss** | `MatryoshkaLoss` (dims 768, 512, 256, 128) wrapping `CachedMultipleNegativesRankingLoss` |
| **Epochs** | 3 (17 steps per epoch) |
| **Batch size** | 64 (cached MNRL mini-batch 16) |
| **Learning rate** | 2e-5, warmup ratio 0.1, weight decay 0.01 |
| **Precision** | FP32 master weights with BF16 autocast |
| **Batch sampler** | `NO_DUPLICATES` |
| **Checkpoint selection** | Best dev nDCG@10 (evaluated every 8 steps) |
| **Max sequence length** | 512 |
| **Seed** | 42 |
| **Hardware** | 1× NVIDIA L4 (24 GB), Google Colab |
| **Software** | sentence-transformers 6.2.0, transformers 5.22.1, PyTorch 2.11.0 |

Dev-set results before and after fine-tuning (dev was used to pick the checkpoint, so treat these as optimistic):

| Dev metric | Base | Fine-tuned |
|---|---|---|
| nDCG@10 | 0.877 | 0.916 |
| MRR@10 | 0.844 | 0.890 |
| Accuracy@1 | 0.769 | 0.821 |
| Accuracy@5 | 0.948 | 0.993 |

---

## Evaluation

All numbers below are from the training notebook's scorecard. Recall@k is the fraction of queries whose labeled source passage appears in the top k results. Confidence intervals are 95% paired bootstrap.

Full scorecard output from the training notebook: [`scorecard.md`](./scorecard.md).

### Evidence quality

| Query set | n | Origin | Strength |
|---|---|---|---|
| Synthetic test | 150 | Teacher-written, same pipeline and style as training queries | Weakest. Optimistic. |
| Teacher-drafted gold | 23 | Harder, less keyword-matching questions drafted by the teacher. **Not human-reviewed.** | Weak (small n, unreviewed) |
| Human-written gold | 0 | none | Not available |

**No human-written evaluation questions exist for this model.** The numbers measure retrieval on teacher-written queries and should not be read as deployed accuracy.

### Synthetic test, held-out papers only (n = 150)

Candidates are the 51 passages from the 39 test papers. This is the cleanest setting, but it is easy: scores saturate.

| Method | Recall@1 | Recall@5 | Recall@10 | MRR@10 |
|---|---|---|---|---|
| BM25 | 0.800 | 0.967 | 1.000 | 0.874 |
| Base, dense 768 | 0.713 | 0.967 | 0.993 | 0.811 |
| **Fine-tuned, dense 768** | 0.793 | 0.993 | 1.000 | 0.871 |
| Base + BM25 (RRF) | 0.787 | 0.987 | 1.000 | 0.873 |
| **Fine-tuned + BM25 (RRF)** | **0.813** | **1.000** | 1.000 | **0.891** |

### Synthetic test, full archive (n = 150)

Candidates are all 486 passages. This includes passages from training papers that the fine-tuned model saw during training, so it is less clean than the table above and likely favors the fine-tuned model.

| Method | Recall@1 | Recall@5 | Recall@10 | MRR@10 |
|---|---|---|---|---|
| BM25 | 0.647 | 0.900 | 0.940 | 0.752 |
| Base, dense 768 | 0.527 | 0.840 | 0.893 | 0.652 |
| **Fine-tuned, dense 768** | 0.647 | 0.920 | 0.953 | 0.747 |
| Base + BM25 (RRF) | 0.673 | 0.913 | 0.967 | 0.776 |
| **Fine-tuned + BM25 (RRF)** | **0.713** | **0.940** | **0.987** | **0.807** |

### Teacher-drafted gold, unreviewed (n = 23)

| Method | Recall@1 | Recall@5 | Recall@10 | MRR@10 |
|---|---|---|---|---|
| BM25 | 0.304 | 0.609 | 0.739 | 0.436 |
| Base, dense 768 | 0.391 | 0.609 | 0.739 | 0.506 |
| Fine-tuned, dense 768 | 0.478 | 0.609 | 0.696 | 0.537 |
| Base + BM25 (RRF) | 0.261 | 0.565 | 0.696 | 0.422 |
| Fine-tuned + BM25 (RRF) | 0.391 | 0.696 | 0.696 | 0.505 |

### Fine-tuned minus base (95% paired bootstrap CI)

| Query set | Comparison | Recall@5 | MRR@10 |
|---|---|---|---|
| Synthetic, held-out papers | dense 768 | +0.027 [+0.007, +0.060] | +0.060 [+0.031, +0.092] |
| Synthetic, held-out papers | + BM25 (RRF) | +0.013 [+0.000, +0.033] | +0.018 [-0.002, +0.037] |
| Synthetic, full archive | dense 768 | +0.080 [+0.033, +0.127] | +0.095 [+0.053, +0.141] |
| Synthetic, full archive | + BM25 (RRF) | +0.027 [+0.000, +0.060] | +0.031 [-0.001, +0.065] |
| Teacher-drafted gold (n = 23) | dense 768 | +0.000 [-0.130, +0.130] | +0.031 [-0.021, +0.099] |
| Teacher-drafted gold (n = 23) | + BM25 (RRF) | +0.130 [+0.000, +0.261] | +0.083 [+0.014, +0.164] |

### How to read these results

- Fine-tuning gives a **statistically supported improvement to dense retrieval** on synthetic queries (confidence intervals exclude zero for Recall@5 and MRR@10).
- Once BM25 is added, the improvement over base + BM25 is **small**, and several intervals touch or cross zero.
- On the harder drafted-gold set, dense retrieval shows **no detectable difference** (n = 23 is too small to conclude either way).
- **BM25 alone is competitive**: on held-out synthetic queries its MRR@10 (0.874) matches the fine-tuned dense model (0.871). The practical recommendation is hybrid retrieval, not dense alone.
- Synthetic queries share the teacher model's phrasing, so all synthetic scores are optimistic.

### Effect of embedding dimension

Fine-tuned model, synthetic test (n = 150):

| Dimension | Held-out Recall@5 | Held-out MRR@10 | Full-archive Recall@5 | Full-archive MRR@10 |
|---|---|---|---|---|
| 768 | 0.993 | 0.871 | 0.920 | 0.747 |
| 512 | 0.987 | 0.880 | 0.907 | 0.752 |
| 256 | 0.973 | 0.871 | 0.893 | 0.734 |
| 128 | 0.967 | 0.865 | 0.833 | 0.720 |

768 and 512 are effectively equivalent here. 256 is a reasonable storage trade-off. 128 loses noticeable recall on the full archive, and the base model card also advises validating 128-d on your own workload.

---

## Limitations

- **Very small training set.** 1,025 triplets from 486 passages (399 of them abstracts). The model has seen a narrow slice of the literature.
- **Synthetic supervision and evaluation.** Training and test queries come from one teacher model. There is no human-written evaluation set, and the 23-question drafted set was not human-reviewed.
- **Modest, partly uncertain gains.** See [How to read these results](#how-to-read-these-results).
- **Narrow domain.** Trained only on NASA combustion and fire-safety text. Out-of-domain behavior was not evaluated and may be worse than the base model.
- **English only.** Multilingual ability of the base model was not evaluated after fine-tuning.
- **Abstract-heavy corpus.** Most papers contribute only their abstract, so retrieval of detailed in-body conditions (tables, run parameters) is largely untested.
- **Sequence length.** Fine-tuned at 512 tokens. Long-document behavior was not evaluated.
- **Text only.** The vision and audio towers were removed from this export.
- **Retrieval is not correctness.** A relevant passage can still be misread, mismatched to a condition, or superseded. Always show sources and keep a human in the loop.

## Responsible use

This model supports literature search. It is **not** a NASA product, is not endorsed by NASA or Google, and must not be used as a fire-safety decision system.

## AI and tool disclosure

> We fine-tuned Google's EmbeddingGemma 2 to retrieve relevant NASA combustion research. Gemma 4 12B generated synthetic queries for training. FlameLedger combines embedding-based retrieval with BM25 keyword search.

AI-assisted engineering tools (Codex, Antigravity) were used for software development and deployment. Gemini (`gemini-3.1-flash-lite`) is used in the FlameLedger application, not in this model, to draft cited briefings from retrieved evidence.

## License and attribution

- This model is a derivative of `google/embeddinggemma-2`, released by Google DeepMind under Apache 2.0, and is released here under **Apache 2.0**. The upstream license page links Google's Gemma 4 license terms. Please review the [base model card](https://huggingface.co/google/embeddinggemma-2) for the terms that apply to the original weights.
- Training queries were generated with `google/gemma-4-12B-it`.
- The training corpus is derived from public NASA NTRS records. **No corpus text is redistributed with this model.**

## Citation

```bibtex
@misc{flameledger_encoder_2026,
  title        = {FlameLedger Encoder: EmbeddingGemma 2 fine-tuned for NASA combustion research retrieval},
  author       = {{Syed Muhammad Hassan}, {Team Qubit}},
  year         = {2026},
  howpublished = {Hugging Face model repository},
  note         = {NASA Space Apps Challenge 2026, Flame in Freefall. Code: https://github.com/Muhammad-Hassan12/FlameLedger}
}
```

Please also cite the base model, EmbeddingGemma 2 (Google DeepMind).

## Contact

Open an issue on the [FlameLedger repository](https://github.com/Muhammad-Hassan12/FlameLedger).