File size: 937 Bytes
58258b8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | """Deduplication package: MinHash near-dup + semantic paraphrase dedup.
Standard SFT dedup pipeline order:
exact (caller's responsibility)
|
v
MinHashDeduplicator - lexical near-dups (formatting variants, edits)
|
v
SemanticDeduplicator - paraphrases (different wording, same meaning)
Both classes operate on a list[str] and return list[int] of indices to KEEP,
so callers re-index back into their own record list. An optional
`key_fn(idx) -> sortable` controls which member of a duplicate cluster wins
(smaller key wins). Default: smallest original index ("first seen").
"""
from .minhash import MinHashDeduplicator
from .semantic import SemanticDeduplicator
from .templates import find_common_prefixes, strip_template, template_hit_stats
__all__ = [
"MinHashDeduplicator",
"SemanticDeduplicator",
"find_common_prefixes",
"strip_template",
"template_hit_stats",
]
|