Sentence Similarity
sentence-transformers
Safetensors
English
embedding_gemma2
feature-extraction
embeddings
retrieval
semantic-search
matryoshka
nasa
combustion
microgravity
spacecraft-fire-safety
Instructions to use rarfileexe/FlameLedger-Encoder with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use rarfileexe/FlameLedger-Encoder with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("rarfileexe/FlameLedger-Encoder") sentences = [ "That is a happy person", "That is a happy dog", "That is a very happy person", "Today is a sunny day" ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [4, 4] - Notebooks
- Google Colab
- Kaggle
File size: 15,555 Bytes
a132be9 a906787 a132be9 a906787 a132be9 a906787 cb66b20 a906787 03e4524 a906787 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 | ---
language:
- en
license: apache-2.0
library_name: sentence-transformers
base_model: google/embeddinggemma-2
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- embeddings
- retrieval
- semantic-search
- matryoshka
- nasa
- combustion
- microgravity
- spacecraft-fire-safety
---
# FlameLedger Encoder: EmbeddingGemma 2 fine-tuned for NASA combustion research retrieval
A text embedding model for finding passages in NASA microgravity-combustion and spacecraft fire-safety literature. It is [`google/embeddinggemma-2`](https://huggingface.co/google/embeddinggemma-2) with its text encoder fine-tuned on synthetic search queries over public NASA Technical Reports Server (NTRS) records.
Built by **Team Qubit** for the NASA Space Apps Challenge 2026 (*Flame in Freefall*) and used in production by [FlameLedger](https://flameledger.agenticerastudio.tech), an evidence research console that ranks NASA/NIST combustion experiments against a proposed cabin environment.
- **Live demo:** https://flameledger.agenticerastudio.tech
- **Code:** https://github.com/Muhammad-Hassan12/FlameLedger
- **Embedding size:** 768 (Matryoshka: 768 / 512 / 256 / 128)
- **Language:** English
- **License:** Apache 2.0 (see [License and attribution](#license-and-attribution))
> **Read this first.** This is a small, domain-specific fine-tune (1,025 training triplets, 486 passages). Gains over the base model are measured on teacher-written synthetic queries and are modest. On a small teacher-drafted question set (n = 23, unreviewed) no improvement was detectable. BM25 is a strong baseline on this corpus, and this model is intended to be used *together with* BM25. See [Evaluation](#evaluation) and [Limitations](#limitations).
---
## Model details
| | |
|---|---|
| **Base model** | `google/embeddinggemma-2` (Google DeepMind, 740M total parameters; 270M text backbone) |
| **What was fine-tuned** | Text tower only (about 271M trainable parameters). Vision and audio towers were frozen and are unused. |
| **This repository** | Text-only encoder export (FP32). Image and audio inputs are not supported. |
| **Output** | 768-dimensional embeddings, truncatable to 512 / 256 / 128 (Matryoshka) |
| **Similarity** | Cosine similarity (L2-normalize embeddings) |
| **Max sequence length used in training** | 512 tokens (base model supports up to 8K) |
| **Precision** | Use FP32 or BF16. The base model card warns against FP16. |
| **Developed by** | Syed Muhammad Hassan (Team Qubit)|
---
## Intended use
**Intended:**
- Semantic and hybrid (dense + BM25) retrieval over NASA combustion, microgravity, and spacecraft fire-safety technical reports.
- First-stage candidate retrieval in a RAG or research-assistant pipeline where results are shown with their sources.
**Not intended:**
- Fire-safety decision-making, material qualification, or any safety-critical judgment. Retrieval relevance is not scientific correctness, and the system must not be used as a substitute for expert review.
- General-purpose or multilingual embedding. Use the base model for that.
- Answer verification. This model ranks passages. It does not check claims.
---
## How to use
### Prefixes are required
The model was trained with fixed text prefixes. **Use exactly these formats for queries and documents**, or retrieval quality will silently degrade.
```python
QUERY_PREFIX = "task: search result | query: "
def fmt_query(q: str) -> str:
return QUERY_PREFIX + " ".join(str(q).split())
def fmt_doc(title: str, text: str) -> str:
t = " ".join((title or "").split()) or "none"
return f"title: {t} | text: {str(text).strip()}"
```
### Sentence Transformers
```python
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("YOUR_HF_USERNAME/flameledger-embeddinggemma2-nasa-combustion")
model.max_seq_length = 512
queries = [fmt_query("At what oxygen concentration does flame spread stop in low-speed opposed flow?")]
docs = [
fmt_doc("Combustion of solid fuel in very low speed oxygen streams",
"Flame spread over thin solid fuel in quiescent and low-speed flows ..."),
fmt_doc("Particle and Smoke Detection on ISS for Next Generation Smoke Detectors",
"Smoke detector response depends on particle size distribution ..."),
]
# Prefixes are added manually above. Use .encode() so no extra prompt is prepended.
q = model.encode(queries, normalize_embeddings=True)
d = model.encode(docs, normalize_embeddings=True)
scores = q @ d.T # cosine similarity
print(scores)
```
### Using a smaller dimension (Matryoshka)
```python
import numpy as np
def truncate(emb, dim=256):
emb = emb[:, :dim]
return emb / np.linalg.norm(emb, axis=1, keepdims=True)
q256, d256 = truncate(q, 256), truncate(d, 256)
```
Re-normalize after truncating. See the [dimension results](#effect-of-embedding-dimension) before going below 256.
### Hybrid retrieval (recommended)
The best configuration in our evaluation fuses dense and lexical rankings with Reciprocal Rank Fusion:
- Dense: cosine similarity at 768 dimensions.
- Lexical: BM25 (k1 = 1.5, b = 0.75) over `title + " " + text`, tokenized as lowercase alphanumerics (keeping internal `.` and `-`, e.g. `6061-t6`).
- Fusion: RRF with k = 60 over the top 100 of each ranking.
A reference implementation (`HybridRetriever`, `BM25`, `rrf_scores`) is in the [FlameLedger repository](https://github.com/Muhammad-Hassan12/FlameLedger).
---
## Training
### Data
Training data is synthetic. No human-written training questions were used.
| Stage | Count |
|---|---|
| NTRS records fetched | 416 |
| On-topic papers after de-duplication and topic filter | 399 |
| Passages (chunks) | 486 (399 abstracts + 87 body chunks from 10 full-text PDFs) |
| Valid teacher queries after rule-based validation | 1,412 (train 1,128 / dev 134 / test 150) |
| Training triplets after round-trip filter | 1,025 (dev: 134) |
Most papers are represented by their abstract only, because scanned PDFs were skipped rather than guessed.
**Query generation.** Gemma 4 12B instruction-tuned (`google/gemma-4-12B-it`) wrote three queries per passage in three styles: `keyword`, `question`, and `scenario` (an engineering need the passage helps with). Teacher output was kept only if it passed rule checks:
- length limits per style,
- no meta-references such as "the passage" or "this paper",
- no verbatim copying (no 7-word run copied from the passage),
- every multi-digit or decimal number in the query must appear in the passage (blocks invented numbers),
- de-duplication.
**Round-trip filter (train split only).** Queries for which the base model ranked the source passage outside its top 15 were dropped (kept 1,025 of 1,128, or 91%).
**Hard negatives.** Three per query, mined with the base model from the same split only, never from the same paper, skipping near-duplicates (cosine ≥ 0.92) and candidates scoring within 5% of the positive, to reduce false negatives.
**Leak-free split.** Train / dev / test are split **by paper** (321 / 39 / 39 papers), with papers sharing a normalized title kept in the same split.
### Procedure
| | |
|---|---|
| **Loss** | `MatryoshkaLoss` (dims 768, 512, 256, 128) wrapping `CachedMultipleNegativesRankingLoss` |
| **Epochs** | 3 (17 steps per epoch) |
| **Batch size** | 64 (cached MNRL mini-batch 16) |
| **Learning rate** | 2e-5, warmup ratio 0.1, weight decay 0.01 |
| **Precision** | FP32 master weights with BF16 autocast |
| **Batch sampler** | `NO_DUPLICATES` |
| **Checkpoint selection** | Best dev nDCG@10 (evaluated every 8 steps) |
| **Max sequence length** | 512 |
| **Seed** | 42 |
| **Hardware** | 1× NVIDIA L4 (24 GB), Google Colab |
| **Software** | sentence-transformers 6.2.0, transformers 5.22.1, PyTorch 2.11.0 |
Dev-set results before and after fine-tuning (dev was used to pick the checkpoint, so treat these as optimistic):
| Dev metric | Base | Fine-tuned |
|---|---|---|
| nDCG@10 | 0.877 | 0.916 |
| MRR@10 | 0.844 | 0.890 |
| Accuracy@1 | 0.769 | 0.821 |
| Accuracy@5 | 0.948 | 0.993 |
---
## Evaluation
All numbers below are from the training notebook's scorecard. Recall@k is the fraction of queries whose labeled source passage appears in the top k results. Confidence intervals are 95% paired bootstrap.
Full scorecard output from the training notebook: [`scorecard.md`](./scorecard.md).
### Evidence quality
| Query set | n | Origin | Strength |
|---|---|---|---|
| Synthetic test | 150 | Teacher-written, same pipeline and style as training queries | Weakest. Optimistic. |
| Teacher-drafted gold | 23 | Harder, less keyword-matching questions drafted by the teacher. **Not human-reviewed.** | Weak (small n, unreviewed) |
| Human-written gold | 0 | none | Not available |
**No human-written evaluation questions exist for this model.** The numbers measure retrieval on teacher-written queries and should not be read as deployed accuracy.
### Synthetic test, held-out papers only (n = 150)
Candidates are the 51 passages from the 39 test papers. This is the cleanest setting, but it is easy: scores saturate.
| Method | Recall@1 | Recall@5 | Recall@10 | MRR@10 |
|---|---|---|---|---|
| BM25 | 0.800 | 0.967 | 1.000 | 0.874 |
| Base, dense 768 | 0.713 | 0.967 | 0.993 | 0.811 |
| **Fine-tuned, dense 768** | 0.793 | 0.993 | 1.000 | 0.871 |
| Base + BM25 (RRF) | 0.787 | 0.987 | 1.000 | 0.873 |
| **Fine-tuned + BM25 (RRF)** | **0.813** | **1.000** | 1.000 | **0.891** |
### Synthetic test, full archive (n = 150)
Candidates are all 486 passages. This includes passages from training papers that the fine-tuned model saw during training, so it is less clean than the table above and likely favors the fine-tuned model.
| Method | Recall@1 | Recall@5 | Recall@10 | MRR@10 |
|---|---|---|---|---|
| BM25 | 0.647 | 0.900 | 0.940 | 0.752 |
| Base, dense 768 | 0.527 | 0.840 | 0.893 | 0.652 |
| **Fine-tuned, dense 768** | 0.647 | 0.920 | 0.953 | 0.747 |
| Base + BM25 (RRF) | 0.673 | 0.913 | 0.967 | 0.776 |
| **Fine-tuned + BM25 (RRF)** | **0.713** | **0.940** | **0.987** | **0.807** |
### Teacher-drafted gold, unreviewed (n = 23)
| Method | Recall@1 | Recall@5 | Recall@10 | MRR@10 |
|---|---|---|---|---|
| BM25 | 0.304 | 0.609 | 0.739 | 0.436 |
| Base, dense 768 | 0.391 | 0.609 | 0.739 | 0.506 |
| Fine-tuned, dense 768 | 0.478 | 0.609 | 0.696 | 0.537 |
| Base + BM25 (RRF) | 0.261 | 0.565 | 0.696 | 0.422 |
| Fine-tuned + BM25 (RRF) | 0.391 | 0.696 | 0.696 | 0.505 |
### Fine-tuned minus base (95% paired bootstrap CI)
| Query set | Comparison | Recall@5 | MRR@10 |
|---|---|---|---|
| Synthetic, held-out papers | dense 768 | +0.027 [+0.007, +0.060] | +0.060 [+0.031, +0.092] |
| Synthetic, held-out papers | + BM25 (RRF) | +0.013 [+0.000, +0.033] | +0.018 [-0.002, +0.037] |
| Synthetic, full archive | dense 768 | +0.080 [+0.033, +0.127] | +0.095 [+0.053, +0.141] |
| Synthetic, full archive | + BM25 (RRF) | +0.027 [+0.000, +0.060] | +0.031 [-0.001, +0.065] |
| Teacher-drafted gold (n = 23) | dense 768 | +0.000 [-0.130, +0.130] | +0.031 [-0.021, +0.099] |
| Teacher-drafted gold (n = 23) | + BM25 (RRF) | +0.130 [+0.000, +0.261] | +0.083 [+0.014, +0.164] |
### How to read these results
- Fine-tuning gives a **statistically supported improvement to dense retrieval** on synthetic queries (confidence intervals exclude zero for Recall@5 and MRR@10).
- Once BM25 is added, the improvement over base + BM25 is **small**, and several intervals touch or cross zero.
- On the harder drafted-gold set, dense retrieval shows **no detectable difference** (n = 23 is too small to conclude either way).
- **BM25 alone is competitive**: on held-out synthetic queries its MRR@10 (0.874) matches the fine-tuned dense model (0.871). The practical recommendation is hybrid retrieval, not dense alone.
- Synthetic queries share the teacher model's phrasing, so all synthetic scores are optimistic.
### Effect of embedding dimension
Fine-tuned model, synthetic test (n = 150):
| Dimension | Held-out Recall@5 | Held-out MRR@10 | Full-archive Recall@5 | Full-archive MRR@10 |
|---|---|---|---|---|
| 768 | 0.993 | 0.871 | 0.920 | 0.747 |
| 512 | 0.987 | 0.880 | 0.907 | 0.752 |
| 256 | 0.973 | 0.871 | 0.893 | 0.734 |
| 128 | 0.967 | 0.865 | 0.833 | 0.720 |
768 and 512 are effectively equivalent here. 256 is a reasonable storage trade-off. 128 loses noticeable recall on the full archive, and the base model card also advises validating 128-d on your own workload.
---
## Limitations
- **Very small training set.** 1,025 triplets from 486 passages (399 of them abstracts). The model has seen a narrow slice of the literature.
- **Synthetic supervision and evaluation.** Training and test queries come from one teacher model. There is no human-written evaluation set, and the 23-question drafted set was not human-reviewed.
- **Modest, partly uncertain gains.** See [How to read these results](#how-to-read-these-results).
- **Narrow domain.** Trained only on NASA combustion and fire-safety text. Out-of-domain behavior was not evaluated and may be worse than the base model.
- **English only.** Multilingual ability of the base model was not evaluated after fine-tuning.
- **Abstract-heavy corpus.** Most papers contribute only their abstract, so retrieval of detailed in-body conditions (tables, run parameters) is largely untested.
- **Sequence length.** Fine-tuned at 512 tokens. Long-document behavior was not evaluated.
- **Text only.** The vision and audio towers were removed from this export.
- **Retrieval is not correctness.** A relevant passage can still be misread, mismatched to a condition, or superseded. Always show sources and keep a human in the loop.
## Responsible use
This model supports literature search. It is **not** a NASA product, is not endorsed by NASA or Google, and must not be used as a fire-safety decision system.
## AI and tool disclosure
> We fine-tuned Google's EmbeddingGemma 2 to retrieve relevant NASA combustion research. Gemma 4 12B generated synthetic queries for training. FlameLedger combines embedding-based retrieval with BM25 keyword search.
AI-assisted engineering tools (Codex, Antigravity) were used for software development and deployment. Gemini (`gemini-3.1-flash-lite`) is used in the FlameLedger application, not in this model, to draft cited briefings from retrieved evidence.
## License and attribution
- This model is a derivative of `google/embeddinggemma-2`, released by Google DeepMind under Apache 2.0, and is released here under **Apache 2.0**. The upstream license page links Google's Gemma 4 license terms. Please review the [base model card](https://huggingface.co/google/embeddinggemma-2) for the terms that apply to the original weights.
- Training queries were generated with `google/gemma-4-12B-it`.
- The training corpus is derived from public NASA NTRS records. **No corpus text is redistributed with this model.**
## Citation
```bibtex
@misc{flameledger_encoder_2026,
title = {FlameLedger Encoder: EmbeddingGemma 2 fine-tuned for NASA combustion research retrieval},
author = {{Syed Muhammad Hassan}, {Team Qubit}},
year = {2026},
howpublished = {Hugging Face model repository},
note = {NASA Space Apps Challenge 2026, Flame in Freefall. Code: https://github.com/Muhammad-Hassan12/FlameLedger}
}
```
Please also cite the base model, EmbeddingGemma 2 (Google DeepMind).
## Contact
Open an issue on the [FlameLedger repository](https://github.com/Muhammad-Hassan12/FlameLedger). |