--- language: - fr license: apache-2.0 library_name: sentence-transformers pipeline_tag: sentence-similarity tags: - french - semantic-similarity - deduplication - paraphrase-detection - sentence-transformers - text-classification ---

![Task](https://img.shields.io/badge/Task-sentence--similarity-3D8BFD?style=for-the-badge&logo=huggingface&logoColor=white) ![Language](https://img.shields.io/badge/Language-Fran%C3%A7ais-0055A4?style=for-the-badge) ![Stages](https://img.shields.io/badge/Pipeline-2--Stage-EF4135?style=for-the-badge) ![License](https://img.shields.io/badge/License-Apache%202.0-34D399?style=for-the-badge)
Hugging Face

---

> 🔵 **Quick summary:** Stage 1 finds what is worth comparing. Stage 2 determines whether meaning is preserved. Rule guards add deterministic protection. Together they form a practical two-stage French semantic quality pipeline for dataset cleaning and semantic deduplication. --- ## 🏗️ Architecture ```text French dataset ↓ Stage 1 — semantic embeddings ↓ FAISS / vector search ↓ high-similarity candidate pairs ↓ Stage 2 — semantic verifier ↓ SAME / DIFFERENT ↓ rule guards ↓ KEEP / DEDUP / CONFLICT / REVIEW ``` --- ## 🔍 Stage 1 — Semantic Retrieval `France-ML/french-semantic-stage1` A French SentenceTransformer-based embedding model used for **fast semantic retrieval**. It converts French sentences into vectors so similar sentences can be found efficiently using FAISS or another vector-search system. Stage 1 is intentionally used as a **candidate generator**, not the final decision-maker. **Main uses** - Semantic search - Duplicate / paraphrase candidate retrieval - Dataset clustering - Similarity filtering - Large-scale candidate generation --- ## 🧠 Stage 2 — Meaning Verification `France-ML/french-semantic-stage2` A French pair-classification model that performs deeper semantic verification and predicts: - `SAME`: meaning is sufficiently preserved - `DIFFERENT`: meaning has changed Useful for detecting meaning changes involving **negation, numbers, dates, quantities, units, tense, roles, directions, locations, cause/effect, and conditions**. **Examples where lexical similarity hides a meaning change:** ```text Il a vendu 5 voitures. Il a vendu 50 voitures. ``` ```text Il n'est pas venu. Il est venu. ``` ```text Marie a donné le livre à Paul. Paul a donné le livre à Marie. ``` These sentences can be lexically very similar while having different meanings. --- ## 🛡️ Rule Guards Model decisions can be combined with deterministic checks for important semantic features: - Numbers - Dates - Units - Negation words - Quantities - Named entities - Directional terms - Other task-specific constraints The final decision can be more conservative than relying on neural similarity alone. **Possible outputs** | Label | Meaning | |:--|:--| | `KEEP` | No problematic match detected | | `DEDUP` | Sufficiently equivalent duplicate / paraphrase | | `CONFLICT` | Similar text but meaning-changing information detected | | `REVIEW` | Uncertain or requires human inspection | --- ## 🎯 Applications - French dataset deduplication - Training-data cleaning - Paraphrase detection - Semantic search preprocessing - Near-duplicate detection - Dataset quality control - Conflict detection - Retrieval candidate generation - Filtering noisy training data - Preparing French NLP datasets for model training ---

## 🗂️ Built for Datasets

The engine is made for **French dataset work** — cleaning, deduplicating, and quality-checking text corpora before they are used to train or evaluate models. Most dataset problems are not exact duplicates. They are near-duplicates, paraphrases, and near-identical sentences that quietly disagree on one important detail. String matching misses these. Pure embedding similarity also misses them, because the sentences *look* the same to a vector model even when the meaning changed. This engine is built to catch that gap.

Dataset Problem	How the Engine Handles It
Exact and near-duplicate rows	Stage 1 retrieval surfaces candidates, Stage 2 confirms and labels them `DEDUP`
Paraphrases that mean the same thing	Stage 2 predicts `SAME` even when wording differs
Similar sentences with a changed detail (numbers, negation, roles)	Stage 2 flags them and rule guards raise `CONFLICT`
Ambiguous or borderline cases	Marked `REVIEW` instead of forcing a wrong decision

### Why it is built for scale Running a deep verifier across every possible sentence pair in a large corpus would be far too expensive. The two-stage design avoids that: ```text Millions of French sentences ↓ Stage 1 — vector search (fast, cheap) ↓ small set of likely matches ↓ Stage 2 — meaning verification (slow, careful) ↓ rule guards ↓ KEEP / DEDUP / CONFLICT / REVIEW ``` ## ✅ What It Does Well The system is built around **semantic similarity followed by meaning verification**, not simple string matching. This allows it to identify pairs that: - Use different wording but preserve meaning - Are highly similar but contain important semantic changes - Should be considered potential duplicates - Should be separated because a critical detail changed Stage 1 is optimized for efficient retrieval. Stage 2 provides a focused semantic decision. --- ## ⚠️ Limitations This is **not a perfect semantic truth detector**. Neural models can still make mistakes, especially with: - Ambiguous sentences - Very long or complicated text - Rare vocabulary - Unusual French constructions - World knowledge - Sarcasm or implicit meaning - Complex multi-step reasoning - Subtle factual differences - Domain-specific terminology Stage 2 is a **semantic verifier**, not an absolute authority. For high-value datasets, use deterministic rule guards and/or human review for uncertain cases. **Architecture limitation.** Stage 1 and Stage 2 are designed to work together. Stage 2 should **not** be treated as a general unrelated-text classifier. Its purpose is to verify candidate pairs already flagged as semantically related by Stage 1. --- ## 📈 Evaluation - **Stage 1** was evaluated using validation data, paraphrases, and hard-negative examples. The selected checkpoint prioritized generalization and separation of unrelated text, not just maximal validation similarity. - **Stage 2** achieved strong results on the project's validation and targeted evaluation sets, including difficult negative examples. These are **project-specific evaluation** results, not a guarantee of performance on every French dataset or domain. --- ## 📁 Contents ```text French-semantic-engine/ ├── stage1/ │ ├── 1_Pooling/ │ ├── config.json │ ├── config_sentence_transformers.json │ ├── model.safetensors │ ├── modules.json │ ├── sentence_bert_config.json │ ├── tokenizer.json │ └── tokenizer_config.json │ └── stage2/ ├── config.json ├── model.safetensors ├── tokenizer.json └── tokenizer_config.json ``` ---

_{⭐ If this engine helped you, consider leaving a like on the Hub.}

Made For French ML · 2026