Update README.md

2ffe75e verified 26 days ago

10.9 kB

---
language:
- fr
license: apache-2.0
library_name: sentence-transformers
pipeline_tag: sentence-similarity
tags:
- french
- semantic-similarity
- deduplication
- paraphrase-detection
- sentence-transformers
- text-classification
---
<div align="center">

<img src="https://capsule-render.vercel.app/api?type=waving&color=0:0055A4,50:E8EEF9,100:EF4135&height=210&section=header&text=French-semantic-engine&fontSize=42&fontColor=0B1F4D&animation=fadeIn&fontAlignY=32&desc=France-ML%20%F0%9F%87%AB%F0%9F%87%B7%20%E2%80%A2%20Two-Stage%20Semantic%20Quality%20Engine&descAlignY=55&descSize=17&descColor=0B1F4D" width="100%"/>

<a href="https://huggingface.co/France-ML/French-semantic-engine">
  <img src="https://readme-typing-svg.demolab.com?font=Fira+Code&weight=700&size=22&duration=2800&pause=900&color=3D8BFD&center=true&vCenter=true&multiline=true&width=900&height=90&lines=French-semantic-engine.;Retrieval+%2B+verification+%2B+rule+guards.;Dedup.+Paraphrase.+Conflict+detection." alt="Typing SVG" />
</a>

<br/>

![Task](https://img.shields.io/badge/Task-sentence--similarity-3D8BFD?style=for-the-badge&logo=huggingface&logoColor=white)
![Language](https://img.shields.io/badge/Language-Fran%C3%A7ais-0055A4?style=for-the-badge)
![Stages](https://img.shields.io/badge/Pipeline-2--Stage-EF4135?style=for-the-badge)
![License](https://img.shields.io/badge/License-Apache%202.0-34D399?style=for-the-badge)

<br/>

<img src="https://huggingface.co/front/assets/huggingface_logo-noborder.svg" height="60" alt="Hugging Face"/>

</div>

<img src="https://user-images.githubusercontent.com/74038190/212284100-561aa473-3905-4a80-b561-0d28506553ee.gif" width="100%">

---

<div align="center">

### ⚡ At a Glance

| Stage | Model | Role |
|:--:|:--:|:--|
| **1** | `french-semantic-stage1` | Fast semantic retrieval, candidate generation |
| **2** | `french-semantic-stage2` | Meaning verification: SAME / DIFFERENT |
| **+** | rule guards | Deterministic protection for critical changes |

</div>

> 🔵 **Quick summary:** Stage 1 finds what is worth comparing. Stage 2 determines whether meaning is preserved. Rule guards add deterministic protection. Together they form a practical two-stage French semantic quality pipeline for dataset cleaning and semantic deduplication.

---

## 🏗️ Architecture

```text
French dataset
      ↓
Stage 1 — semantic embeddings
      ↓
FAISS / vector search
      ↓
high-similarity candidate pairs
      ↓
Stage 2 — semantic verifier
      ↓
SAME / DIFFERENT
      ↓
rule guards
      ↓
KEEP / DEDUP / CONFLICT / REVIEW
```

---

## 🔍 Stage 1 — Semantic Retrieval

`France-ML/french-semantic-stage1`

A French SentenceTransformer-based embedding model used for **fast semantic retrieval**.

It converts French sentences into vectors so similar sentences can be found efficiently using FAISS or another vector-search system. Stage 1 is intentionally used as a **candidate generator**, not the final decision-maker.

**Main uses**

- Semantic search
- Duplicate / paraphrase candidate retrieval
- Dataset clustering
- Similarity filtering
- Large-scale candidate generation

---

## 🧠 Stage 2 — Meaning Verification

`France-ML/french-semantic-stage2`

A French pair-classification model that performs deeper semantic verification and predicts:

- `SAME`: meaning is sufficiently preserved
- `DIFFERENT`: meaning has changed

Useful for detecting meaning changes involving **negation, numbers, dates, quantities, units, tense, roles, directions, locations, cause/effect, and conditions**.

**Examples where lexical similarity hides a meaning change:**

```text
Il a vendu 5 voitures.
Il a vendu 50 voitures.
```

```text
Il n'est pas venu.
Il est venu.
```

```text
Marie a donné le livre à Paul.
Paul a donné le livre à Marie.
```

These sentences can be lexically very similar while having different meanings.

---

## 🛡️ Rule Guards

Model decisions can be combined with deterministic checks for important semantic features:

- Numbers
- Dates
- Units
- Negation words
- Quantities
- Named entities
- Directional terms
- Other task-specific constraints

The final decision can be more conservative than relying on neural similarity alone.

**Possible outputs**

| Label | Meaning |
|:--|:--|
| `KEEP` | No problematic match detected |
| `DEDUP` | Sufficiently equivalent duplicate / paraphrase |
| `CONFLICT` | Similar text but meaning-changing information detected |
| `REVIEW` | Uncertain or requires human inspection |

---

## 🎯 Applications

- French dataset deduplication
- Training-data cleaning
- Paraphrase detection
- Semantic search preprocessing
- Near-duplicate detection
- Dataset quality control
- Conflict detection
- Retrieval candidate generation
- Filtering noisy training data
- Preparing French NLP datasets for model training

---

<div align="center">

## 🗂️ Built for Datasets

</div>

The engine is made for **French dataset work** — cleaning, deduplicating, and quality-checking text corpora before they are used to train or evaluate models.

Most dataset problems are not exact duplicates. They are near-duplicates, paraphrases, and near-identical sentences that quietly disagree on one important detail. String matching misses these. Pure embedding similarity also misses them, because the sentences *look* the same to a vector model even when the meaning changed.

This engine is built to catch that gap.

<table>
  <thead>
    <tr>
      <th align="center" style="background:rgba(0,85,164,0.18);color:#9CC6FF;border-bottom:2px solid #0055A4;">Dataset Problem</th>
      <th align="center" style="background:rgba(0,85,164,0.18);color:#9CC6FF;border-bottom:2px solid #0055A4;">How the Engine Handles It</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style="background:rgba(61,139,253,0.06);color:#9CC6FF;">Exact and near-duplicate rows</td>
      <td><span style="color:#9CC6FF;">Stage 1 retrieval surfaces candidates, Stage 2 confirms and labels them <code>DEDUP</code></span></td>
    </tr>
    <tr>
      <td style="background:rgba(61,139,253,0.06);color:#9CC6FF;">Paraphrases that mean the same thing</td>
      <td><span style="color:#9CC6FF;">Stage 2 predicts <code>SAME</code> even when wording differs</span></td>
    </tr>
    <tr>
      <td style="background:rgba(239,65,53,0.06);color:#FFB0A9;">Similar sentences with a changed detail (numbers, negation, roles)</td>
      <td><span style="color:#FFB0A9;">Stage 2 flags them and rule guards raise <code>CONFLICT</code></span></td>
    </tr>
    <tr>
      <td style="background:rgba(52,211,153,0.06);color:#34d399;">Ambiguous or borderline cases</td>
      <td><span style="color:#34d399;">Marked <code>REVIEW</code> instead of forcing a wrong decision</span></td>
    </tr>
  </tbody>
</table>
### Why it is built for scale

Running a deep verifier across every possible sentence pair in a large corpus would be far too expensive. The two-stage design avoids that:

```text
Millions of French sentences
        ↓
Stage 1 — vector search  (fast, cheap)
        ↓
small set of likely matches
        ↓
Stage 2 — meaning verification  (slow, careful)
        ↓
rule guards
        ↓
KEEP / DEDUP / CONFLICT / REVIEW
```
## ✅ What It Does Well

The system is built around **semantic similarity followed by meaning verification**, not simple string matching. This allows it to identify pairs that:

- Use different wording but preserve meaning
- Are highly similar but contain important semantic changes
- Should be considered potential duplicates
- Should be separated because a critical detail changed

Stage 1 is optimized for efficient retrieval. Stage 2 provides a focused semantic decision.

---

## ⚠️ Limitations

This is **not a perfect semantic truth detector**. Neural models can still make mistakes, especially with:

- Ambiguous sentences
- Very long or complicated text
- Rare vocabulary
- Unusual French constructions
- World knowledge
- Sarcasm or implicit meaning
- Complex multi-step reasoning
- Subtle factual differences
- Domain-specific terminology

Stage 2 is a **semantic verifier**, not an absolute authority. For high-value datasets, use deterministic rule guards and/or human review for uncertain cases.

**Architecture limitation.** Stage 1 and Stage 2 are designed to work together. Stage 2 should **not** be treated as a general unrelated-text classifier. Its purpose is to verify candidate pairs already flagged as semantically related by Stage 1.

---

## 📈 Evaluation

- **Stage 1** was evaluated using validation data, paraphrases, and hard-negative examples. The selected checkpoint prioritized generalization and separation of unrelated text, not just maximal validation similarity.
- **Stage 2** achieved strong results on the project's validation and targeted evaluation sets, including difficult negative examples.

These are **project-specific evaluation** results, not a guarantee of performance on every French dataset or domain.

---

## 📁 Contents

```text
French-semantic-engine/
├── stage1/
│   ├── 1_Pooling/
│   ├── config.json
│   ├── config_sentence_transformers.json
│   ├── model.safetensors
│   ├── modules.json
│   ├── sentence_bert_config.json
│   ├── tokenizer.json
│   └── tokenizer_config.json
│
└── stage2/
    ├── config.json
    ├── model.safetensors
    ├── tokenizer.json
    └── tokenizer_config.json
```

---

<div align="center">

### 🧾 Run Summary

| | |
|:--|:--|
| **Model** | France-ML/French-semantic-engine |
| **Pipeline** | Stage 1 → vector search → Stage 2 → rule guards |
| **Language** | French |
| **Task** | sentence-similarity · dedup · paraphrase · conflict detection |
| **Outputs** | KEEP · DEDUP · CONFLICT · REVIEW |
| **Library** | transformers · sentence-transformers |
| **License** | Apache 2.0 |

</div>

<img src="https://user-images.githubusercontent.com/74038190/212284100-561aa473-3905-4a80-b561-0d28506553ee.gif" width="100%">

<div align="center">

<img src="https://capsule-render.vercel.app/api?type=waving&color=0:0055A4,50:E8EEF9,100:EF4135&height=140&section=footer&text=France-ML&fontSize=24&fontColor=0B1F4D&animation=twinkling&fontAlignY=60" width="100%"/>

<sub>⭐ If this engine helped you, consider leaving a like on the Hub.</sub>

<br/><br/>

<a href="https://huggingface.co/France-ML">
  <img src="https://huggingface.co/front/assets/huggingface_logo-noborder.svg" height="28" alt="Hugging Face"/>
</a>
&nbsp;&nbsp;
<a href="https://huggingface.co/France-ML/French-semantic-engine">
  <img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-French--semantic--engine-EF4135?style=for-the-badge" alt="Hugging Face model"/>
</a>

<br/><br/>

<img src="https://flagcdn.com/w40/fr.png" height="18" alt="France"/> <b>Made For French ML</b> · 2026

</div>

	---
	language:
	- fr
	license: apache-2.0
	library_name: sentence-transformers
	pipeline_tag: sentence-similarity
	tags:
	- french
	- semantic-similarity
	- deduplication
	- paraphrase-detection
	- sentence-transformers
	- text-classification
	---

	<div align="center">

	<img src="https://capsule-render.vercel.app/api?type=waving&color=0:0055A4,50:E8EEF9,100:EF4135&height=210&section=header&text=French-semantic-engine&fontSize=42&fontColor=0B1F4D&animation=fadeIn&fontAlignY=32&desc=France-ML%20%F0%9F%87%AB%F0%9F%87%B7%20%E2%80%A2%20Two-Stage%20Semantic%20Quality%20Engine&descAlignY=55&descSize=17&descColor=0B1F4D" width="100%"/>

	<a href="https://huggingface.co/France-ML/French-semantic-engine">
	<img src="https://readme-typing-svg.demolab.com?font=Fira+Code&weight=700&size=22&duration=2800&pause=900&color=3D8BFD&center=true&vCenter=true&multiline=true&width=900&height=90&lines=French-semantic-engine.;Retrieval+%2B+verification+%2B+rule+guards.;Dedup.+Paraphrase.+Conflict+detection." alt="Typing SVG" />
	</a>

	<br/>

	![Task](https://img.shields.io/badge/Task-sentence--similarity-3D8BFD?style=for-the-badge&logo=huggingface&logoColor=white)
	![Language](https://img.shields.io/badge/Language-Fran%C3%A7ais-0055A4?style=for-the-badge)
	![Stages](https://img.shields.io/badge/Pipeline-2--Stage-EF4135?style=for-the-badge)
	![License](https://img.shields.io/badge/License-Apache%202.0-34D399?style=for-the-badge)

	<br/>

	<img src="https://huggingface.co/front/assets/huggingface_logo-noborder.svg" height="60" alt="Hugging Face"/>

	</div>

	<img src="https://user-images.githubusercontent.com/74038190/212284100-561aa473-3905-4a80-b561-0d28506553ee.gif" width="100%">

	---

	<div align="center">

	### ⚡ At a Glance

	\| Stage \| Model \| Role \|
	\|:--:\|:--:\|:--\|
	\| 1 \| `french-semantic-stage1` \| Fast semantic retrieval, candidate generation \|
	\| 2 \| `french-semantic-stage2` \| Meaning verification: SAME / DIFFERENT \|
	\| + \| rule guards \| Deterministic protection for critical changes \|

	</div>

	> 🔵 Quick summary: Stage 1 finds what is worth comparing. Stage 2 determines whether meaning is preserved. Rule guards add deterministic protection. Together they form a practical two-stage French semantic quality pipeline for dataset cleaning and semantic deduplication.

	---

	## 🏗️ Architecture

	```text
	French dataset
	↓
	Stage 1 — semantic embeddings
	↓
	FAISS / vector search
	↓
	high-similarity candidate pairs
	↓
	Stage 2 — semantic verifier
	↓
	SAME / DIFFERENT
	↓
	rule guards
	↓
	KEEP / DEDUP / CONFLICT / REVIEW
	```

	---

	## 🔍 Stage 1 — Semantic Retrieval

	`France-ML/french-semantic-stage1`

	A French SentenceTransformer-based embedding model used for fast semantic retrieval.

	It converts French sentences into vectors so similar sentences can be found efficiently using FAISS or another vector-search system. Stage 1 is intentionally used as a candidate generator, not the final decision-maker.

	Main uses

	- Semantic search
	- Duplicate / paraphrase candidate retrieval
	- Dataset clustering
	- Similarity filtering
	- Large-scale candidate generation

	---

	## 🧠 Stage 2 — Meaning Verification

	`France-ML/french-semantic-stage2`

	A French pair-classification model that performs deeper semantic verification and predicts:

	- `SAME`: meaning is sufficiently preserved
	- `DIFFERENT`: meaning has changed

	Useful for detecting meaning changes involving negation, numbers, dates, quantities, units, tense, roles, directions, locations, cause/effect, and conditions.

	Examples where lexical similarity hides a meaning change:

	```text
	Il a vendu 5 voitures.
	Il a vendu 50 voitures.
	```

	```text
	Il n'est pas venu.
	Il est venu.
	```

	```text
	Marie a donné le livre à Paul.
	Paul a donné le livre à Marie.
	```

	These sentences can be lexically very similar while having different meanings.

	---

	## 🛡️ Rule Guards

	Model decisions can be combined with deterministic checks for important semantic features:

	- Numbers
	- Dates
	- Units
	- Negation words
	- Quantities
	- Named entities
	- Directional terms
	- Other task-specific constraints

	The final decision can be more conservative than relying on neural similarity alone.

	Possible outputs

	\| Label \| Meaning \|
	\|:--\|:--\|
	\| `KEEP` \| No problematic match detected \|
	\| `DEDUP` \| Sufficiently equivalent duplicate / paraphrase \|
	\| `CONFLICT` \| Similar text but meaning-changing information detected \|
	\| `REVIEW` \| Uncertain or requires human inspection \|

	---

	## 🎯 Applications

	- French dataset deduplication
	- Training-data cleaning
	- Paraphrase detection
	- Semantic search preprocessing
	- Near-duplicate detection
	- Dataset quality control
	- Conflict detection
	- Retrieval candidate generation
	- Filtering noisy training data
	- Preparing French NLP datasets for model training

	---

	<div align="center">

	## 🗂️ Built for Datasets

	</div>

	The engine is made for French dataset work — cleaning, deduplicating, and quality-checking text corpora before they are used to train or evaluate models.

	Most dataset problems are not exact duplicates. They are near-duplicates, paraphrases, and near-identical sentences that quietly disagree on one important detail. String matching misses these. Pure embedding similarity also misses them, because the sentences look the same to a vector model even when the meaning changed.

	This engine is built to catch that gap.

	<table>
	<thead>
	<tr>
	<th align="center" style="background:rgba(0,85,164,0.18);color:#9CC6FF;border-bottom:2px solid #0055A4;">Dataset Problem</th>
	<th align="center" style="background:rgba(0,85,164,0.18);color:#9CC6FF;border-bottom:2px solid #0055A4;">How the Engine Handles It</th>
	</tr>
	</thead>
	<tbody>
	<tr>
	<td style="background:rgba(61,139,253,0.06);color:#9CC6FF;">Exact and near-duplicate rows</td>
	<td><span style="color:#9CC6FF;">Stage 1 retrieval surfaces candidates, Stage 2 confirms and labels them <code>DEDUP</code></span></td>
	</tr>
	<tr>
	<td style="background:rgba(61,139,253,0.06);color:#9CC6FF;">Paraphrases that mean the same thing</td>
	<td><span style="color:#9CC6FF;">Stage 2 predicts <code>SAME</code> even when wording differs</span></td>
	</tr>
	<tr>
	<td style="background:rgba(239,65,53,0.06);color:#FFB0A9;">Similar sentences with a changed detail (numbers, negation, roles)</td>
	<td><span style="color:#FFB0A9;">Stage 2 flags them and rule guards raise <code>CONFLICT</code></span></td>
	</tr>
	<tr>
	<td style="background:rgba(52,211,153,0.06);color:#34d399;">Ambiguous or borderline cases</td>
	<td><span style="color:#34d399;">Marked <code>REVIEW</code> instead of forcing a wrong decision</span></td>
	</tr>
	</tbody>
	</table>

	### Why it is built for scale

	Running a deep verifier across every possible sentence pair in a large corpus would be far too expensive. The two-stage design avoids that:

	```text
	Millions of French sentences
	↓
	Stage 1 — vector search (fast, cheap)
	↓
	small set of likely matches
	↓
	Stage 2 — meaning verification (slow, careful)
	↓
	rule guards
	↓
	KEEP / DEDUP / CONFLICT / REVIEW
	```
	## ✅ What It Does Well

	The system is built around semantic similarity followed by meaning verification, not simple string matching. This allows it to identify pairs that:

	- Use different wording but preserve meaning
	- Are highly similar but contain important semantic changes
	- Should be considered potential duplicates
	- Should be separated because a critical detail changed

	Stage 1 is optimized for efficient retrieval. Stage 2 provides a focused semantic decision.

	---

	## ⚠️ Limitations

	This is not a perfect semantic truth detector. Neural models can still make mistakes, especially with:

	- Ambiguous sentences
	- Very long or complicated text
	- Rare vocabulary
	- Unusual French constructions
	- World knowledge
	- Sarcasm or implicit meaning
	- Complex multi-step reasoning
	- Subtle factual differences
	- Domain-specific terminology

	Stage 2 is a semantic verifier, not an absolute authority. For high-value datasets, use deterministic rule guards and/or human review for uncertain cases.

	Architecture limitation. Stage 1 and Stage 2 are designed to work together. Stage 2 should not be treated as a general unrelated-text classifier. Its purpose is to verify candidate pairs already flagged as semantically related by Stage 1.

	---

	## 📈 Evaluation

	- Stage 1 was evaluated using validation data, paraphrases, and hard-negative examples. The selected checkpoint prioritized generalization and separation of unrelated text, not just maximal validation similarity.
	- Stage 2 achieved strong results on the project's validation and targeted evaluation sets, including difficult negative examples.

	These are project-specific evaluation results, not a guarantee of performance on every French dataset or domain.

	---

	## 📁 Contents

	```text
	French-semantic-engine/
	├── stage1/
	│ ├── 1_Pooling/
	│ ├── config.json
	│ ├── config_sentence_transformers.json
	│ ├── model.safetensors
	│ ├── modules.json
	│ ├── sentence_bert_config.json
	│ ├── tokenizer.json
	│ └── tokenizer_config.json
	│
	└── stage2/
	├── config.json
	├── model.safetensors
	├── tokenizer.json
	└── tokenizer_config.json
	```

	---

	<div align="center">

	### 🧾 Run Summary

	\| \| \|
	\|:--\|:--\|
	\| Model \| France-ML/French-semantic-engine \|
	\| Pipeline \| Stage 1 → vector search → Stage 2 → rule guards \|
	\| Language \| French \|
	\| Task \| sentence-similarity · dedup · paraphrase · conflict detection \|
	\| Outputs \| KEEP · DEDUP · CONFLICT · REVIEW \|
	\| Library \| transformers · sentence-transformers \|
	\| License \| Apache 2.0 \|

	</div>

	<img src="https://user-images.githubusercontent.com/74038190/212284100-561aa473-3905-4a80-b561-0d28506553ee.gif" width="100%">

	<div align="center">

	<img src="https://capsule-render.vercel.app/api?type=waving&color=0:0055A4,50:E8EEF9,100:EF4135&height=140&section=footer&text=France-ML&fontSize=24&fontColor=0B1F4D&animation=twinkling&fontAlignY=60" width="100%"/>

	<sub>⭐ If this engine helped you, consider leaving a like on the Hub.</sub>

	<br/><br/>

	<a href="https://huggingface.co/France-ML">
	<img src="https://huggingface.co/front/assets/huggingface_logo-noborder.svg" height="28" alt="Hugging Face"/>
	</a>

	<a href="https://huggingface.co/France-ML/French-semantic-engine">
	<img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-French--semantic--engine-EF4135?style=for-the-badge" alt="Hugging Face model"/>
	</a>

	<br/><br/>

	<img src="https://flagcdn.com/w40/fr.png" height="18" alt="France"/> <b>Made For French ML</b> · 2026

	</div>