Release Mizan-Rerank-v3

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,358 @@
 ---
 license: apache-2.0
+language:
+- ar
+base_model:
+- Alibaba-NLP/gte-multilingual-reranker-base
+pipeline_tag: text-ranking
+library_name: sentence-transformers
+tags:
+- sentence-transformers
+- cross-encoder
+- reranker
+- text-ranking
+- arabic
+- long-context
+- rag
+- islamic
+datasets:
+- NAMAA-Space/Arabic-Triplet-With-Multi-Negatives
+metrics:
+- ndcg
+- mrr
 ---
+
+# Mizan-Rerank-v3
+
+**An Arabic cross-encoder reranker trained to tell the passage that answers the question apart from passages that only look like they do.**
+
+![Parameters](https://img.shields.io/badge/Parameters-305M-green)
+![Language](https://img.shields.io/badge/Language-Arabic-blue)
+![Context](https://img.shields.io/badge/Trained%20context-3072%20tokens-purple)
+![License](https://img.shields.io/badge/License-Apache%202.0-brightgreen)
+
+Mizan-Rerank-v3 is a new model, not a continuation of Mizan-Rerank-V2. It was fine-tuned directly from
+[Alibaba-NLP/gte-multilingual-reranker-base](https://huggingface.co/Alibaba-NLP/gte-multilingual-reranker-base)
+on 71k Arabic listwise groups. Each group pairs the correct passage with adversarial "trap" passages that
+share most of its words but change its meaning: a negated ruling, a swapped entity, a shifted number or date,
+an exception applied to the wrong case. It is meant for reranking in Arabic search and RAG pipelines, especially
+over long, dense religious and encyclopedic text.
+
+![Summary](assets/summary.png)
+
+## Highlights
+
+- **Best of the four compared models on 4 of 5 held-out benchmarks.** Mean nDCG@10 is **0.845**, vs 0.802 for
+  bge-reranker-v2-m3 (568M), 0.783 for its gte base and 0.723 for Mizan-Rerank-V2.
+- **Much better at picking the right passage first on adversarial data.** Hit@1 on held-out long-context
+  adversarial queries is **0.570**, vs 0.336 for bge-reranker-v2-m3, 0.314 for gte base and 0.227 for Mizan-Rerank-V2.
+- **Far less likely to rank a contradicting passage first.** On held-out adversarial tests, the top-ranked passage
+  is a contradicting trap in 20% of short and 33% of long-context queries. For bge-reranker-v2-m3 the figures are
+  44% and 62%; for Mizan-Rerank-V2, 62% and 76%.
+- **Better than Mizan-Rerank-V2 on every benchmark**, including real user queries about Islamic jurisprudence
+  (+0.052 nDCG@10, 95% CI [+0.025, +0.078]).
+- **Same size and speed as V2 and gte base** (305M parameters, XLM-R vocabulary, RoPE positions).
+
+It is **not** the best model on everything. bge-reranker-v2-m3 remains ahead on MTEB NamaaMrTydi and, by a clear
+margin, on real Islamic-jurisprudence search queries (see [Limitations](#limitations)).
+
+## Usage
+
+### Sentence Transformers
+
+```bash
+pip install -U sentence-transformers
+```
+
+```python
+from sentence_transformers import CrossEncoder
+
+model = CrossEncoder("ALJIACHI/Mizan-Rerank-v3", trust_remote_code=True, max_length=3072)
+
+query = "ما هي فوائد فيتامين د؟"
+passages = [
+    "يساعد فيتامين د على امتصاص الكالسيوم وتقوية العظام، كما يدعم عمل الجهاز المناعي.",
+    "يستخدم فيتامين د في بعض الصناعات الغذائية كمادة مضافة لتدعيم الحليب.",
+    "أطلقت وزارة الزراعة حملة وطنية لزيادة الوعي بأهمية الزراعة العضوية.",
+]
+
+scores = model.predict([(query, passage) for passage in passages])
+print(scores)  # ≈ [0.875, 0.023, 0.010]  (sigmoid relevance scores)
+
+for hit in model.rank(query, passages, return_documents=True):
+    print(f"{hit['score']:.3f}  {hit['text']}")
+```
+
+### Transformers
+
+```python
+import torch
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("ALJIACHI/Mizan-Rerank-v3")
+model = AutoModelForSequenceClassification.from_pretrained(
+    "ALJIACHI/Mizan-Rerank-v3", trust_remote_code=True, torch_dtype=torch.float16
+).to("cuda").eval()
+
+query = "ما هي فوائد فيتامين د؟"
+passages = [
+    "يساعد فيتامين د على امتصاص الكالسيوم وتقوية العظام، كما يدعم عمل الجهاز المناعي.",
+    "أطلقت وزارة الزراعة حملة وطنية لزيادة الوعي بأهمية الزراعة العضوية.",
+]
+
+with torch.inference_mode():
+    features = tokenizer([query] * len(passages), passages, padding=True, truncation=True,
+                         max_length=3072, return_tensors="pt").to("cuda")
+    scores = torch.sigmoid(model(**features).logits.squeeze(-1).float())
+
+for passage, score in sorted(zip(passages, scores.tolist()), key=lambda item: -item[1]):
+    print(f"{score:.3f}  {passage}")
+```
+
+### Practical notes
+
+- **Sequence length.** The model was trained on query + passage inputs of up to 3,072 tokens and is benchmarked
+  here at 2,048. The architecture accepts 8,192 positions, but quality beyond 3,072 tokens has not been measured.
+  If your query + passage inputs fit in 512 tokens, `max_length=512` is much faster.
+- **Scores.** The sigmoid of the single output logit is a relevance score. It works for ranking and for coarse
+  cut-offs, but it is not a calibrated probability: tune any threshold on your own data.
+- **`trust_remote_code=True`** is required. The architecture code (`modeling.py`, `configuration.py`) comes from
+  the gte base model and ships in this repository.
+
+## Evaluation
+
+All four models were scored by the same script ([`benchmark/benchmark_rerankers.py`](benchmark/benchmark_rerankers.py))
+with the same inputs: float32, `max_length=2048`, and the full candidate list of every query. Metrics use graded
+gains (correct passage 3, partially correct passage 1, trap or irrelevant passage 0).
+
+| Metric | Meaning |
+|---|---|
+| nDCG@10 | quality of the whole ranking |
+| MRR@10 | 1 / rank of the correct passage |
+| Hit@1 | share of queries where the correct passage is ranked first |
+
+### Held-out benchmarks
+
+| Benchmark | Queries | Mizan-Rerank-v3 | Mizan-Rerank-V2 | gte-multilingual-reranker-base | bge-reranker-v2-m3 |
+|---|---:|---:|---:|---:|---:|
+| MTEB NamaaMrTydi, unseen subset¹ | 573 | 0.8784 | 0.8220 | 0.8771 | **0.8897** |
+| Arabic Hard Negatives, unseen subset¹ | 314 | **0.9200** | 0.8780 | 0.9062 | 0.9035 |
+| Adversarial short (test)² | 602 | **0.8432** | 0.6965 | 0.7565 | 0.7938 |
+| Adversarial long-context (test)² | 1,703 | **0.8280** | 0.6203 | 0.6747 | 0.6873 |
+| Multi-LLM benchmark² | 361 | **0.7536** | 0.5998 | 0.6992 | 0.7365 |
+| **Average nDCG@10** | | **0.8446** | 0.7233 | 0.7827 | 0.8022 |
+| **Average MRR@10** | | **0.7908** | 0.6312 | 0.7160 | 0.7427 |
+| **Average Hit@1** | | **0.6461** | 0.4221 | 0.5307 | 0.5764 |
+
+¹ Queries whose question and correct passage never occur in the v3 training pool (see [Training-data overlap](#training-data-overlap-with-public-benchmarks)).
+² Internal test sets, not released. They share no queries or passages with the training data (verified by exact
+match), and the long-context split is by source article.
+
+![Public benchmarks](assets/public_benchmarks.png)
+
+![Adversarial benchmarks](assets/adversarial_benchmarks.png)
+
+**Where the gain comes from:** ranking the correct passage *first*. Hit@1 is where v3 differs most from the other
+models, especially when traps share most of their text with the answer:
+
+![Hit@1](assets/hit_at_1.png)
+
+### Contradiction ranked first
+
+For RAG the worst failure is not a missing answer but a passage that says the *opposite* ranked at the top:
+a negated ruling, a reversed condition, a wrong number. Every adversarial test query has one correct passage
+and several contradicting traps. The table shows how often a trap was ranked #1 (lower is better):
+
+| Test set | Queries | Mizan-Rerank-v3 | Mizan-Rerank-V2 | gte-multilingual-reranker-base | bge-reranker-v2-m3 |
+|---|---:|---:|---:|---:|---:|
+| Adversarial short | 602 | **20.1%** (121) | 62.1% (374) | 51.3% (309) | 43.5% (262) |
+| Adversarial long-context | 1,703 | **33.4%** (568) | 76.4% (1,301) | 68.2% (1,161) | 62.0% (1,056) |
+| Multi-LLM | 361 | **49.3%** (178) | 75.3% (272) | 64.0% (231) | 54.0% (195) |
+
+Across all 2,666 queries, v3 ranks a trap first 32.5% of the time. For bge-reranker-v2-m3 the figure is 56.8%;
+for gte base, 63.8%; for Mizan-Rerank-V2, 73.0%:
+
+![Contradiction ranked first](assets/contradiction_first.png)
+
+Reproduce with `python compare_generated_test_models.py --dataset-file <test.jsonl> ... --chart contradiction_first.png`
+(in the training repository; the test sets are internal).
+
+<details>
+<summary><b>Full per-benchmark tables (nDCG@10 / MRR@10 / Hit@1), including full-set public scores</b></summary>
+
+**nDCG@10**
+
+| Model | NamaaMrTydi (full, 918) | NamaaMrTydi (unseen, 573) | Arabic Hard Neg. (full, 12,373)³ | Arabic Hard Neg. (unseen, 314) | Adversarial short | Adversarial long-ctx | Multi-LLM |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| Mizan-Rerank-v3 | 0.8885 | 0.8784 | 0.9103 | **0.9200** | **0.8432** | **0.8280** | **0.7536** |
+| Mizan-Rerank-V2 | 0.8312 | 0.8220 | 0.8447 | 0.8780 | 0.6965 | 0.6203 | 0.5998 |
+| gte-multilingual-reranker-base | 0.8874 | 0.8771 | 0.8960 | 0.9062 | 0.7565 | 0.6747 | 0.6992 |
+| bge-reranker-v2-m3 | **0.8995** | **0.8897** | 0.9082 | 0.9035 | 0.7938 | 0.6873 | 0.7365 |
+
+**MRR@10**
+
+| Model | NamaaMrTydi (full) | NamaaMrTydi (unseen) | Arabic Hard Neg. (full)³ | Arabic Hard Neg. (unseen) | Adversarial short | Adversarial long-ctx | Multi-LLM |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| Mizan-Rerank-v3 | 0.8518 | 0.8385 | 0.8796 | **0.8926** | **0.7731** | **0.7658** | **0.6841** |
+| Mizan-Rerank-V2 | 0.7755 | 0.7635 | 0.7919 | 0.8362 | 0.5952 | 0.4983 | 0.4629 |
+| gte-multilingual-reranker-base | 0.8498 | 0.8362 | 0.8607 | 0.8743 | 0.6826 | 0.5765 | 0.6105 |
+| bge-reranker-v2-m3 | **0.8662** | **0.8531** | 0.8771 | 0.8711 | 0.7374 | 0.5879 | 0.6643 |
+
+**Hit@1**
+
+| Model | NamaaMrTydi (full) | NamaaMrTydi (unseen) | Arabic Hard Neg. (full)³ | Arabic Hard Neg. (unseen) | Adversarial short | Adversarial long-ctx | Multi-LLM |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| Mizan-Rerank-v3 | 0.7691 | 0.7522 | 0.7882 | **0.8121** | **0.6146** | **0.5696** | **0.4820** |
+| Mizan-Rerank-V2 | 0.6492 | 0.6335 | 0.6433 | 0.7134 | 0.3405 | 0.2267 | 0.1967 |
+| gte-multilingual-reranker-base | 0.7549 | 0.7347 | 0.7600 | 0.7834 | 0.4618 | 0.3136 | 0.3601 |
+| bge-reranker-v2-m3 | **0.7865** | **0.7627** | 0.7916 | 0.7866 | 0.5449 | 0.3365 | 0.4515 |
+
+³ 97% of these queries occur in the v3 training pool, so the full-set score is not a fair measure for v3 and is
+not bolded or used in any average.
+
+</details>
+
+### Real Islamic-jurisprudence search queries (fiqh eval)
+
+This is the use case the model was built for. The eval set has 224 real, answerable user queries from a production
+Arabic assistant that answers questions from a collection of fatwas. Each query comes with the 25 candidate passages
+its dense retriever returned, all graded 0–3 by two independent LLM judges (the lower grade is kept; weighted
+κ = 0.81). None of these queries or candidate passages occurs in the training data (exact match; the legacy
+sources were also filtered by 8-word shingles). Scoring
+uses float16 and `max_length=512`, the production settings. The set contains private user queries and is not released.
+
+| Model | nDCG@10 | MRR@10 | P@1 | Misleading doc in top 3 ↓ | v3 − model, nDCG@10 [95% CI] |
+|---|---:|---:|---:|---:|---:|
+| Retriever order (no reranker) | 0.5732 | 0.4691 | 0.5045 | 0.3482 | +0.096 [+0.065, +0.132] |
+| **Mizan-Rerank-v3** | **0.6693** | 0.5807 | 0.6429 | 0.3080 | – |
+| Mizan-Rerank-V2 | 0.6176 | 0.4513 | 0.4955 | 0.3705 | +0.052 [+0.025, +0.078] |
+| gte-multilingual-reranker-base | 0.6567 | 0.5346 | 0.5893 | 0.3259 | +0.013 [−0.002, +0.027] |
+| bge-reranker-v2-m3 | **0.7372** | **0.6601** | **0.7098** | **0.2946** | −0.068 [−0.094, −0.043] |
+
+"Misleading" means a passage the judges marked as a ruling for a *different* case than the one asked about is
+ranked in the top 3. Confidence intervals come from a paired bootstrap over queries (2,000 resamples).
+
+![Fiqh eval](assets/fiqh_eval.png)
+
+On this task v3 clearly improves on Mizan-Rerank-V2. Its gain over its own gte base is small and not
+statistically significant, and bge-reranker-v2-m3 is clearly better. If this is your use case, benchmark
+bge-reranker-v2-m3 too.
+
+### Training-data overlap with public benchmarks
+
+v3 was trained with a retrieval replay set built from the Arabic Mr. TyDi training data
+([NAMAA-Space/Arabic-Triplet-With-Multi-Negatives](https://huggingface.co/datasets/NAMAA-Space/Arabic-Triplet-With-Multi-Negatives)).
+Both public benchmarks are also derived from Arabic Mr. TyDi, so we hashed every benchmark query and correct passage
+against the training pool:
+
+| Benchmark | Queries | Query or correct passage in v3 training pool | Unseen subset |
+|---|---:|---:|---:|
+| MTEB NamaaMrTydiReranking | 918 | 345 (correct passage only; 0 queries) | 573 |
+| Omartificial-Intelligence-Space/Arabic-With-Ranked-Hard-Negatives | 12,373 | 12,059 | 314 |
+
+The headline numbers use the unseen subsets. The hashes are shipped in
+[`benchmark/seen_in_training.json`](benchmark/seen_in_training.json), so anyone can reproduce the split. The other
+models may also have been trained on Mr. TyDi; the unseen subsets only control for v3's own training data.
+`NAMAA-Space/mteb-eval-mrtydi` is a flat copy of MTEB NamaaMrTydiReranking (identical queries and candidates), so it
+is not reported separately.
+
+## Training
+
+### Data (71,044 listwise groups)
+
+Every group is one query with one correct passage, up to 5 hard negatives and, for adversarial groups, one
+partially correct passage.
+
+| Source | Groups | Share | What it teaches |
+|---|---:|---:|---|
+| Long-context adversarial (WikiShia articles, LLM-generated) | 36,280 | 51% | find the answer inside long, dense passages, whether it sits at the beginning, middle or end |
+| Short adversarial (LLM-generated) | 12,030 | 17% | reject near-duplicate traps in short passages |
+| Arabic Mr. TyDi retrieval replay | 8,525 | 12% | keep general open-domain retrieval ability |
+| Hadith question–passage pairs | 4,736 | 7% | classical religious text |
+| Arabic news similarity pairs with LLM hard negatives | 4,737 | 7% | topical near-misses |
+| Fatwa Q&A pairs with TF-IDF-mined negatives | 4,736 | 7% | jurisprudence rulings on nearby cases |
+
+The adversarial negatives cover 12 trap types, each a minimal edit that changes the meaning while keeping most
+of the words: `negation_flip`, `exception_scope`, `entity_role_reversal`, `entity_swap`, `ordered_precedence`,
+`temporal_boundary`, `numeric_boundary`, `quantifier_change`, `causal_direction`, `condition_swap`,
+`conclusion_flip`, `attribution_shift`.
+
+Leakage controls: the adversarial train, validation and test splits are split by source article. Extra training
+data sharing a source article or a query with any validation or test set was removed. The three legacy sources were
+additionally filtered against every evaluation set (including the fiqh eval) by 8-word shingles. All held-out
+sets above were re-checked against the final training pool by exact query and passage match.
+
+### Objective
+
+For each group the model scores all candidates jointly:
+
+- **Pointwise:** binary cross-entropy, targets 1 / 0.5 / 0 for correct / partial / negative, `pos_weight=1.24`, weight 0.5.
+- **Pairwise:** softplus margin loss `softplus(s_neg − s_pos + 0.2)` for every negative, weight 1.0.
+- **Partial order:** the same margin loss for correct > partial > negative, weight 0.5 within the pairwise term.
+
+### Configuration
+
+| Parameter | Value |
+|---|---|
+| Initialization | Alibaba-NLP/gte-multilingual-reranker-base |
+| Max sequence length (query + passage) | 3,072 tokens |
+| Candidates per group | 1 correct + 1 partial (if present) + up to 5 negatives |
+| Batch | 4 groups/GPU × 8 accumulation × 2 GPUs = 64 groups per update |
+| Optimizer | AdamW, lr 1e-6, weight decay 0.01, grad-norm clip 1.0 |
+| Schedule | cosine, 10% warmup, 7 epochs (7,777 updates) |
+| Precision | bf16, gradient checkpointing, SDPA attention |
+| Hardware | 2 × NVIDIA RTX 4090 |
+| Checkpoint selection | epoch with the best mean validation nDCG@10 (long-context + short adversarial) → **epoch 6** |
+
+![Training curve](assets/training_curve.png)
+
+## Limitations
+
+- **Real fiqh queries: bge-reranker-v2-m3 is better.** It leads by 0.068 nDCG@10 (see above). v3's advantage is
+  concentrated on adversarial, trap-heavy passages. Short, natural user queries over similar-looking rulings remain
+  hard.
+- **Contradiction traps are reduced, not solved.** A contradicting passage is still ranked first for 33% of
+  long-context and 20% of short adversarial test queries, and for 49% of the multi-LLM set. Verify critical answers, such as religious rulings or
+  medical and legal statements, rather than relying on the top-ranked passage alone.
+- **In-distribution test sets.** The adversarial test sets come from the same generation pipeline as the training
+  data. They are disjoint from it, but share its style. Treat them as a measure of the targeted skill, not of
+  general-domain quality.
+- **Arabic only.** No English or other-language data was used, and cross-lingual quality relative to the
+  multilingual base model is untested.
+- **Long inputs.** Quality beyond 3,072 tokens is unmeasured.
+
+## Reproducing the benchmarks
+
+```bash
+git clone https://huggingface.co/ALJIACHI/Mizan-Rerank-v3 && cd Mizan-Rerank-v3/benchmark
+pip install torch transformers huggingface_hub pyarrow matplotlib
+python benchmark_rerankers.py                       # 4 models on the public benchmarks → results.json
+python benchmark_rerankers.py --models mizan-v3 --fp16
+python benchmark_rerankers.py --model mine=path/to/reranker --models mine mizan-v3
+python benchmark_rerankers.py --listwise my_set=my_test.jsonl   # your own listwise JSONL
+```
+
+Public datasets are downloaded at pinned revisions. The internal and fiqh results above were produced by the same
+metric code (`results_full.json`, `internal_results.json`). `plot_results.py` regenerates every chart in `assets/`.
+
+## Framework versions
+
+Python 3.10.14 · PyTorch 2.8.0+cu126 · Transformers 4.55.4 · Sentence Transformers 5.4.1 · Accelerate 1.10.0 ·
+Tokenizers 0.21.0
+
+## Citation
+
+```bibtex
+@software{Mizan_Rerank_v3_2026,
+  author    = {Ali Aljiachi},
+  title     = {Mizan-Rerank-v3: Adversarially Trained Arabic Long-Context Reranker},
+  year      = {2026},
+  publisher = {Hugging Face},
+  url       = {https://huggingface.co/ALJIACHI/Mizan-Rerank-v3}
+}
+```
+
+## License
+
+Apache 2.0, the same license as the base model
+[Alibaba-NLP/gte-multilingual-reranker-base](https://huggingface.co/Alibaba-NLP/gte-multilingual-reranker-base).

benchmark/benchmark_rerankers.py

@@ -0,0 +1,271 @@
+"""Benchmark Arabic rerankers on public Hugging Face reranking sets.
+
+Compares Mizan-Rerank-v3 with Mizan-Rerank-V2, gte-multilingual-reranker-base and
+bge-reranker-v2-m3 (any other cross-encoder can be added with --model name=repo_or_path).
+
+Datasets (downloaded from the Hub at pinned revisions):
+
+  namaa_mrtydi           mteb/NamaaMrTydiReranking (MTEB)             918 queries, 1 positive + ~4 hard negatives
+  arabic_hard_negatives  Omartificial-Intelligence-Space/
+                         Arabic-With-Ranked-Hard-Negatives            12,373 queries, 1 positive + 4 mined negatives
+
+Both are also reported on their "unseen" subset: cases whose query or positive passage does
+not occur in the Mizan-Rerank-v3 training pool (hashes in seen_in_training.json). Your own
+listwise JSONL files can be added with --listwise name=path.jsonl (fields: query, positive,
+optional partial_or_neutral, hard_negatives as strings or {"text": ...}).
+
+Metrics per query (graded gains: positive 3, partial 1, negative 0):
+  nDCG@10, MRR@10 (rank of the positive), Hit@1 (positive ranked first).
+"average" = unweighted mean over the --average datasets (default: every dataset, using the
+_unseen subset instead of the full set wherever one exists, so no query is counted twice).
+
+  pip install torch transformers huggingface_hub pyarrow
+  python benchmark_rerankers.py                         # all four models, public sets
+  python benchmark_rerankers.py --models mizan-v3 --fp16
+  python benchmark_rerankers.py --model mine=./my_reranker --models mine bge-v2-m3
+"""
+
+import argparse
+import gc
+import hashlib
+import json
+import math
+import statistics
+import time
+from dataclasses import dataclass
+from pathlib import Path
+
+import pyarrow.parquet as pq
+import torch
+from huggingface_hub import snapshot_download
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+
+SCRIPT_DIR = Path(__file__).resolve().parent
+SEEN_FILE = SCRIPT_DIR / "seen_in_training.json"
+MODELS = {
+    "mizan-v3": "ALJIACHI/Mizan-Rerank-v3",
+    "mizan-v2": "ALJIACHI/Mizan-Rerank-V2",
+    "gte-base": "Alibaba-NLP/gte-multilingual-reranker-base",
+    "bge-v2-m3": "BAAI/bge-reranker-v2-m3",
+}
+HUB_DATASETS = {
+    "namaa_mrtydi": ("mteb/NamaaMrTydiReranking", "ecf1ff8c18469251bfe6acf1219e63e287bbb037"),
+    "arabic_hard_negatives": ("Omartificial-Intelligence-Space/Arabic-With-Ranked-Hard-Negatives", "fec243e3767f502d393d330bf97bfb6dce868830"),
+}
+METRICS = ("ndcg@10", "mrr@10", "hit@1")
+
+
+@dataclass(frozen=True)
+class Group:
+    query: str
+    documents: tuple[str, ...]
+    gains: tuple[int, ...]
+
+
+def normalize(text: str) -> str:
+    return " ".join(str(text).split())
+
+
+def digest(text: str) -> str:
+    return hashlib.sha256(normalize(text).encode("utf-8")).hexdigest()
+
+
+def read_parquet_dir(directory: Path) -> list[dict]:
+    files = sorted(directory.glob("*.parquet"))
+    if not files:
+        raise FileNotFoundError(f"No parquet files in {directory}")
+    return [row for path in files for row in pq.read_table(path).to_pylist()]
+
+
+def download(repo: str, revision: str | None) -> Path:
+    return Path(snapshot_download(repo, repo_type="dataset", revision=revision))
+
+
+def load_namaa_mrtydi(root: Path) -> list[Group]:
+    corpus = {row["_id"]: row["text"] or "" for row in read_parquet_dir(root / "corpus")}
+    queries = {row["_id"]: row["text"] for row in read_parquet_dir(root / "queries")}
+    relevant = {(row["query-id"], row["corpus-id"]) for row in read_parquet_dir(root / "data") if row["score"] > 0}
+    groups = []
+    for row in read_parquet_dir(root / "top_ranked"):
+        ids = row["corpus-ids"]
+        groups.append(Group(queries[row["query-id"]], tuple(corpus[i] for i in ids), tuple(3 if (row["query-id"], i) in relevant else 0 for i in ids)))
+    return groups
+
+
+def load_arabic_hard_negatives(root: Path) -> list[Group]:
+    groups = []
+    for row in read_parquet_dir(root / "data"):
+        negatives = [row[f"negative{i}"] for i in range(1, 5) if row.get(f"negative{i}") and str(row[f"negative{i}"]).strip()]
+        groups.append(Group(row["query"], (row["positive"], *negatives), (3, *([0] * len(negatives)))))
+    return groups
+
+
+def load_listwise(path: Path) -> list[Group]:
+    groups = []
+    for line in path.read_text(encoding="utf-8").splitlines():
+        if not line.strip():
+            continue
+        record = json.loads(line)
+        documents, gains = [record["positive"]], [3]
+        if record.get("partial_or_neutral"):
+            documents.append(record["partial_or_neutral"])
+            gains.append(1)
+        for negative in record["hard_negatives"]:
+            documents.append(negative["text"] if isinstance(negative, dict) else negative)
+            gains.append(0)
+        groups.append(Group(record["query"], tuple(documents), tuple(gains)))
+    return groups
+
+
+def unseen_indices(groups: list[Group], seen: dict) -> list[int]:
+    queries, passages = set(seen["query_sha256"]), set(seen["passage_sha256"])
+    return [
+        index for index, group in enumerate(groups)
+        if digest(group.query) not in queries and digest(group.documents[group.gains.index(3)]) not in passages
+    ]
+
+
+def dcg(gains: list[int], k: int) -> float:
+    return sum((2**gain - 1) / math.log2(rank + 2) for rank, gain in enumerate(gains[:k]))
+
+
+def group_metrics(group: Group, scores: list[float]) -> dict[str, float]:
+    order = sorted(range(len(scores)), key=lambda index: scores[index], reverse=True)
+    ranked = [group.gains[index] for index in order]
+    ideal = dcg(sorted(group.gains, reverse=True), 10)
+    rank = ranked.index(3) + 1
+    return {"ndcg@10": dcg(ranked, 10) / ideal, "mrr@10": 1.0 / rank if rank <= 10 else 0.0, "hit@1": float(rank == 1)}
+
+
+def score(model_path: str, groups: list[Group], device: str, batch_size: int, max_length: int, fp16: bool) -> list[list[float]]:
+    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
+    dtype = torch.float16 if fp16 else torch.float32
+    model = AutoModelForSequenceClassification.from_pretrained(model_path, trust_remote_code=True, torch_dtype=dtype).to(device).eval()
+    pairs = [(group.query, document) for group in groups for document in group.documents]
+    order = sorted(range(len(pairs)), key=lambda index: len(pairs[index][0]) + len(pairs[index][1]))
+    flat = [0.0] * len(pairs)
+    with torch.inference_mode():
+        for start in range(0, len(order), batch_size):
+            batch = order[start : start + batch_size]
+            features = tokenizer(
+                [pairs[index][0] for index in batch], [pairs[index][1] for index in batch],
+                padding=True, truncation=True, max_length=max_length, return_tensors="pt",
+            ).to(device)
+            logits = model(**features).logits.float()
+            logits = logits.squeeze(-1) if logits.shape[-1] == 1 else logits[:, -1]
+            for index, value in zip(batch, logits.cpu().tolist()):
+                flat[index] = value
+    del model
+    gc.collect()
+    if torch.cuda.is_available():
+        torch.cuda.empty_cache()
+    grouped, offset = [], 0
+    for group in groups:
+        grouped.append(flat[offset : offset + len(group.documents)])
+        offset += len(group.documents)
+    return grouped
+
+
+def summarize(rows: list[dict]) -> dict:
+    return {"queries": len(rows), **{metric: statistics.fmean(row[metric] for row in rows) for metric in METRICS}}
+
+
+def print_tables(results: dict, datasets: list[str], average: list[str]) -> None:
+    for metric in METRICS:
+        header = "| Model | " + " | ".join(datasets) + " | **Average** |"
+        print(f"\n### {metric}\n\n{header}\n|" + "---|" * (len(datasets) + 2))
+        for name, row in sorted(results.items(), key=lambda item: -item[1]["average"][metric]):
+            cells = " | ".join(f"{row['datasets'][dataset][metric]:.4f}" for dataset in datasets)
+            print(f"| {name} | {cells} | **{row['average'][metric]:.4f}** |")
+    print(f"\nAverage = unweighted mean over: {', '.join(average)}")
+
+
+def parse_named(values: list[str], flag: str) -> dict[str, str]:
+    named = {}
+    for value in values or []:
+        name, _, target = value.partition("=")
+        if not name or not target:
+            raise ValueError(f"{flag} expects name=value, got {value!r}")
+        named[name] = target
+    return named
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument("--model", action="append", help="name=repo_or_path; adds or overrides a model")
+    parser.add_argument("--models", nargs="+", help="Model names to run (default: all)")
+    parser.add_argument("--datasets", nargs="+", choices=list(HUB_DATASETS), default=list(HUB_DATASETS))
+    parser.add_argument("--listwise", action="append", help="name=path.jsonl extra listwise dataset")
+    parser.add_argument("--no-unseen", action="store_true", help="Skip the *_unseen subsets")
+    parser.add_argument("--average", nargs="+", help="Datasets included in the average (default: all scored)")
+    parser.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu")
+    parser.add_argument("--batch-size", type=int, default=32)
+    parser.add_argument("--max-length", type=int, default=2048)
+    parser.add_argument("--fp16", action="store_true", help="Score in float16 (faster; results differ slightly from float32)")
+    parser.add_argument("--limit", type=int, default=None, help="Use only the first N queries of each dataset (smoke tests)")
+    parser.add_argument("--output", type=Path, default=SCRIPT_DIR / "results.json")
+    return parser
+
+
+def main() -> int:
+    args = build_parser().parse_args()
+    models = {**MODELS, **parse_named(args.model, "--model")}
+    selected = args.models or list(models)
+    unknown = [name for name in selected if name not in models]
+    if unknown:
+        raise ValueError(f"Unknown model(s) {unknown}; known: {sorted(models)}")
+
+    datasets: dict[str, list[Group]] = {}
+    subsets: dict[str, tuple[str, list[int]]] = {}
+    loaders = {"namaa_mrtydi": load_namaa_mrtydi, "arabic_hard_negatives": load_arabic_hard_negatives}
+    seen = json.loads(SEEN_FILE.read_text(encoding="utf-8")) if SEEN_FILE.is_file() and not args.no_unseen else None
+    for name in args.datasets:
+        datasets[name] = loaders[name](download(*HUB_DATASETS[name]))
+    for name, path in parse_named(args.listwise, "--listwise").items():
+        datasets[name] = load_listwise(Path(path))
+    if args.limit:
+        datasets = {name: groups[: args.limit] for name, groups in datasets.items()}
+    if seen:
+        subsets = {f"{name}_unseen": (name, unseen_indices(datasets[name], seen)) for name in args.datasets}
+    columns = [column for name in datasets for column in (name, f"{name}_unseen") if column in datasets or column in subsets]
+    for column in columns:
+        size = len(subsets[column][1]) if column in subsets else len(datasets[column])
+        print(f"{column:<32} {size:>6} queries")
+    average = args.average or [column for column in columns if f"{column}_unseen" not in subsets]
+    missing = [name for name in average if name not in columns]
+    if missing:
+        raise ValueError(f"--average names not scored: {missing}; scored: {columns}")
+
+    results = {}
+    for name in selected:
+        started = time.perf_counter()
+        print(f"\nScoring {name} ({models[name]})")
+        per_dataset = {}
+        for dataset, groups in datasets.items():
+            rows = [group_metrics(group, values) for group, values in zip(groups, score(models[name], groups, args.device, args.batch_size, args.max_length, args.fp16))]
+            per_dataset[dataset] = summarize(rows)
+            subset = subsets.get(f"{dataset}_unseen")
+            if subset:
+                per_dataset[f"{dataset}_unseen"] = summarize([rows[index] for index in subset[1]])
+        for column in columns:
+            print(f"  {column:<30} " + " ".join(f"{metric} {per_dataset[column][metric]:.4f}" for metric in METRICS))
+        results[name] = {
+            "model": models[name],
+            "seconds": time.perf_counter() - started,
+            "datasets": per_dataset,
+            "average": {metric: statistics.fmean(per_dataset[dataset][metric] for dataset in average) for metric in METRICS},
+        }
+
+    print_tables(results, columns, average)
+    args.output.write_text(json.dumps({
+        "settings": {"max_length": args.max_length, "batch_size": args.batch_size, "fp16": args.fp16, "device": args.device, "limit": args.limit},
+        "datasets": {column: len(subsets[column][1]) if column in subsets else len(datasets[column]) for column in columns},
+        "average_over": average,
+        "results": results,
+    }, indent=2), encoding="utf-8")
+    print(f"\nResults written to {args.output}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

benchmark/internal_results.json

@@ -0,0 +1,127 @@
+{
+  "fiqh_eval": {
+    "queries": 224,
+    "graded_pairs": 5600,
+    "settings": "fp16, max_length 512, candidates = production retriever top-25",
+    "models": {
+      "retriever": {
+        "ndcg@5": 0.5067,
+        "ndcg@8": 0.5482,
+        "ndcg@10": 0.5732,
+        "mrr@10": 0.4691,
+        "p@1": 0.5045,
+        "relevant_kept@8": 0.574,
+        "misleading@3": 0.3482,
+        "v3_minus_this_ndcg@10": [
+          0.0961,
+          0.0646,
+          0.1317
+        ]
+      },
+      "bge-v2-m3": {
+        "ndcg@5": 0.6831,
+        "ndcg@8": 0.7173,
+        "ndcg@10": 0.7372,
+        "mrr@10": 0.6601,
+        "p@1": 0.7098,
+        "relevant_kept@8": 0.7401,
+        "misleading@3": 0.2946,
+        "v3_minus_this_ndcg@10": [
+          -0.0679,
+          -0.0943,
+          -0.0428
+        ]
+      },
+      "gte-base": {
+        "ndcg@5": 0.5932,
+        "ndcg@8": 0.6293,
+        "ndcg@10": 0.6567,
+        "mrr@10": 0.5346,
+        "p@1": 0.5893,
+        "relevant_kept@8": 0.6661,
+        "misleading@3": 0.3259,
+        "v3_minus_this_ndcg@10": [
+          0.0126,
+          -0.0024,
+          0.0266
+        ]
+      },
+      "mizan-v2": {
+        "ndcg@5": 0.5471,
+        "ndcg@8": 0.5929,
+        "ndcg@10": 0.6176,
+        "mrr@10": 0.4513,
+        "p@1": 0.4955,
+        "relevant_kept@8": 0.6514,
+        "misleading@3": 0.3705,
+        "v3_minus_this_ndcg@10": [
+          0.0518,
+          0.0253,
+          0.078
+        ]
+      },
+      "mizan-v3": {
+        "ndcg@5": 0.6151,
+        "ndcg@8": 0.6463,
+        "ndcg@10": 0.6693,
+        "mrr@10": 0.5807,
+        "p@1": 0.6429,
+        "relevant_kept@8": 0.6764,
+        "misleading@3": 0.308,
+        "v3_minus_this_ndcg@10": [
+          0.0,
+          0.0,
+          0.0
+        ]
+      }
+    }
+  },
+  "training_curve": {
+    "epoch": [
+      0,
+      1,
+      2,
+      3,
+      4,
+      5,
+      6,
+      7
+    ],
+    "longctx": [
+      0.6917,
+      0.7646,
+      0.8032,
+      0.8182,
+      0.8206,
+      0.8248,
+      0.8279,
+      0.8271
+    ],
+    "gemini": [
+      0.7654,
+      0.8098,
+      0.8369,
+      0.8532,
+      0.8603,
+      0.8611,
+      0.8641,
+      0.8631
+    ]
+  },
+  "in_training_test": {
+    "test": {
+      "ndcg": 0.8301,
+      "mrr": 0.7691,
+      "hit_at_1": 0.5749,
+      "positive_over_negative": 0.6213,
+      "contradiction_first": 0.33
+    },
+    "test_gemini": {
+      "ndcg": 0.8445,
+      "mrr": 0.7748,
+      "hit_at_1": 0.6179,
+      "positive_over_negative": 0.7276,
+      "contradiction_first": 0.1944
+    }
+  }
+}

benchmark/plot_results.py

@@ -0,0 +1,139 @@
+"""Render the model-card charts from results_full.json and internal_results.json into ../assets/."""
+
+import argparse
+import json
+from pathlib import Path
+
+import matplotlib
+
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt  # noqa: E402
+
+SCRIPT_DIR = Path(__file__).resolve().parent
+ASSETS = SCRIPT_DIR.parent / "assets"
+ORDER = ("mizan-v3", "mizan-v2", "gte-base", "bge-v2-m3")
+LABELS = {
+    "mizan-v3": "Mizan-Rerank-v3 (305M)",
+    "mizan-v2": "Mizan-Rerank-V2 (305M)",
+    "gte-base": "gte-multilingual-reranker-base (305M)",
+    "bge-v2-m3": "bge-reranker-v2-m3 (568M)",
+    "retriever": "Retriever only (no reranker)",
+}
+COLORS = {"mizan-v3": "#0F766E", "mizan-v2": "#94A3B8", "gte-base": "#CBD5E1", "bge-v2-m3": "#F59E0B", "retriever": "#E2E8F0"}
+DATASET_LABELS = {
+    "namaa_mrtydi": "NamaaMrTydi\n(MTEB, full)",
+    "namaa_mrtydi_unseen": "NamaaMrTydi\n(unseen subset)",
+    "arabic_hard_negatives": "Arabic Hard Negatives\n(full)",
+    "arabic_hard_negatives_unseen": "Arabic Hard Negatives\n(unseen subset)",
+    "gemini_test": "Adversarial short\n(Gemini test)",
+    "longctx_test": "Adversarial long-context\n(test)",
+    "multi_llm": "Multi-LLM\n(6 generators)",
+}
+HELD_OUT = ("namaa_mrtydi_unseen", "arabic_hard_negatives_unseen", "gemini_test", "longctx_test", "multi_llm")
+
+
+def style(ax, title: str, ylabel: str) -> None:
+    ax.set_title(title, fontsize=13, fontweight="bold", loc="left")
+    ax.set_ylabel(ylabel)
+    ax.spines[["top", "right"]].set_visible(False)
+    ax.grid(axis="y", alpha=0.3)
+    ax.set_axisbelow(True)
+
+
+def grouped_bars(results: dict, datasets: tuple[str, ...], metric: str, title: str, path: Path, floor: float) -> None:
+    fig, ax = plt.subplots(figsize=(11, 5.2), dpi=150)
+    width = 0.8 / len(ORDER)
+    for position, key in enumerate(ORDER):
+        values = [results[key]["datasets"][dataset][metric] for dataset in datasets]
+        bars = ax.bar([index + (position - (len(ORDER) - 1) / 2) * width for index in range(len(datasets))], values, width, label=LABELS[key], color=COLORS[key], edgecolor="white")
+        for bar, value in zip(bars, values):
+            ax.text(bar.get_x() + bar.get_width() / 2, value + 0.004, f"{value:.3f}", ha="center", va="bottom", fontsize=7.5, rotation=90)
+    ax.set_xticks(range(len(datasets)), [DATASET_LABELS[dataset] for dataset in datasets], fontsize=9)
+    ax.set_ylim(floor, 1.0)
+    style(ax, title, metric.replace("ndcg", "nDCG").replace("hit", "Hit").replace("mrr", "MRR"))
+    ax.legend(frameon=False, fontsize=8.5, ncol=2, loc="upper right")
+    fig.tight_layout()
+    fig.savefig(path)
+    plt.close(fig)
+
+
+def summary_chart(results: dict, fiqh: dict, path: Path) -> None:
+    held_out = {key: sum(results[key]["datasets"][dataset]["ndcg@10"] for dataset in HELD_OUT) / len(HELD_OUT) for key in ORDER}
+    panels = (
+        ("Held-out benchmarks\n(mean nDCG@10 over 5 sets)", held_out),
+        ("Real fiqh search queries\n(nDCG@10, 224 queries)", {key: fiqh[key]["ndcg@10"] for key in ORDER}),
+    )
+    fig, axes = plt.subplots(1, 2, figsize=(12, 3.8), dpi=150)
+    for ax, (title, values) in zip(axes, panels):
+        keys = sorted(ORDER, key=lambda key: values[key])
+        bars = ax.barh([LABELS[key] for key in keys], [values[key] for key in keys], color=[COLORS[key] for key in keys])
+        for bar, key in zip(bars, keys):
+            ax.text(bar.get_width() + 0.003, bar.get_y() + bar.get_height() / 2, f"{values[key]:.3f}", va="center", fontsize=9, fontweight="bold" if key == "mizan-v3" else None)
+        low = min(values.values())
+        ax.set_xlim(max(0.0, low - 0.08), max(values.values()) + 0.04)
+        ax.set_title(title, fontsize=11, fontweight="bold", loc="left")
+        ax.spines[["top", "right"]].set_visible(False)
+        ax.grid(axis="x", alpha=0.3)
+        ax.set_axisbelow(True)
+    fig.tight_layout()
+    fig.savefig(path)
+    plt.close(fig)
+
+
+def fiqh_chart(fiqh: dict, path: Path) -> None:
+    keys = ("retriever", *ORDER)
+    metrics = (("ndcg@10", "nDCG@10"), ("p@1", "Precision@1"), ("misleading@3", "Misleading doc in top 3\n(lower is better)"))
+    fig, axes = plt.subplots(1, 3, figsize=(13, 4.2), dpi=150)
+    for ax, (metric, label) in zip(axes, metrics):
+        values = [fiqh[key][metric] for key in keys]
+        bars = ax.bar(range(len(keys)), values, color=[COLORS[key] for key in keys], edgecolor="white")
+        for bar, value in zip(bars, values):
+            ax.text(bar.get_x() + bar.get_width() / 2, value + 0.005, f"{value:.3f}", ha="center", fontsize=8)
+        ax.set_xticks(range(len(keys)), ["Retriever", "Mizan v3", "Mizan V2", "gte-base", "bge-v2-m3"], fontsize=8.5, rotation=20)
+        ax.set_ylim(0, max(values) * 1.18)
+        style(ax, label, "")
+    fig.suptitle("Fiqh eval: 224 real fatwa search queries, 25 retrieved candidates each, graded 0-3", fontsize=11, x=0.01, ha="left")
+    fig.tight_layout()
+    fig.savefig(path)
+    plt.close(fig)
+
+
+def training_chart(curve: dict, path: Path) -> None:
+    fig, ax = plt.subplots(figsize=(8, 4.2), dpi=150)
+    ax.plot(curve["epoch"], curve["longctx"], marker="o", color="#0F766E", label="Long-context adversarial (validation)")
+    ax.plot(curve["epoch"], curve["gemini"], marker="s", color="#F59E0B", label="Short adversarial (validation)")
+    best = max(range(len(curve["epoch"])), key=lambda index: curve["longctx"][index] + curve["gemini"][index])
+    ax.axvline(curve["epoch"][best], color="#64748B", linestyle="--", linewidth=1)
+    ax.text(curve["epoch"][best] - 0.1, curve["longctx"][best] - 0.012, f"released checkpoint\n(epoch {curve['epoch'][best]})", fontsize=8.5, color="#334155", ha="right", va="top")
+    ax.set_xticks(curve["epoch"], ["gte base"] + [str(epoch) for epoch in curve["epoch"][1:]])
+    ax.set_xlabel("epoch")
+    style(ax, "Validation nDCG@10 during training", "nDCG@10")
+    ax.legend(frameon=False, fontsize=9, loc="lower right")
+    fig.tight_layout()
+    fig.savefig(path)
+    plt.close(fig)
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--results", type=Path, default=SCRIPT_DIR / "results_full.json")
+    parser.add_argument("--internal", type=Path, default=SCRIPT_DIR / "internal_results.json")
+    parser.add_argument("--output-dir", type=Path, default=ASSETS)
+    args = parser.parse_args()
+    results = json.loads(args.results.read_text(encoding="utf-8"))["results"]
+    internal = json.loads(args.internal.read_text(encoding="utf-8"))
+    fiqh = internal["fiqh_eval"]["models"]
+    args.output_dir.mkdir(parents=True, exist_ok=True)
+    summary_chart(results, fiqh, args.output_dir / "summary.png")
+    grouped_bars(results, ("namaa_mrtydi", "namaa_mrtydi_unseen", "arabic_hard_negatives_unseen"), "ndcg@10", "Public Arabic reranking benchmarks", args.output_dir / "public_benchmarks.png", 0.6)
+    grouped_bars(results, ("gemini_test", "longctx_test", "multi_llm"), "ndcg@10", "Adversarial Arabic reranking (held-out test sets)", args.output_dir / "adversarial_benchmarks.png", 0.5)
+    grouped_bars(results, HELD_OUT, "hit@1", "Hit@1: correct passage ranked first", args.output_dir / "hit_at_1.png", 0.0)
+    fiqh_chart(fiqh, args.output_dir / "fiqh_eval.png")
+    training_chart(internal["training_curve"], args.output_dir / "training_curve.png")
+    for path in sorted(args.output_dir.glob("*.png")):
+        print(path)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

benchmark/results_full.json

@@ -0,0 +1,240 @@
+{
+  "settings": {
+    "max_length": 2048,
+    "batch_size": 16,
+    "fp16": false,
+    "device": "cuda",
+    "limit": null
+  },
+  "datasets": {
+    "namaa_mrtydi": 918,
+    "namaa_mrtydi_unseen": 573,
+    "arabic_hard_negatives": 12373,
+    "arabic_hard_negatives_unseen": 314,
+    "gemini_test": 602,
+    "longctx_test": 1703,
+    "multi_llm": 361
+  },
+  "average_over": [
+    "namaa_mrtydi_unseen",
+    "arabic_hard_negatives_unseen",
+    "gemini_test",
+    "longctx_test",
+    "multi_llm"
+  ],
+  "results": {
+    "mizan-v3": {
+      "model": "ALJIACHI/Mizan-Rerank-v3",
+      "seconds": 185.3771512100002,
+      "datasets": {
+        "namaa_mrtydi": {
+          "queries": 918,
+          "ndcg@10": 0.8884840152217407,
+          "mrr@10": 0.8518155410312274,
+          "hit@1": 0.7690631808278867
+        },
+        "namaa_mrtydi_unseen": {
+          "queries": 573,
+          "ndcg@10": 0.8783831689091802,
+          "mrr@10": 0.8384816753926702,
+          "hit@1": 0.7521815008726004
+        },
+        "arabic_hard_negatives": {
+          "queries": 12373,
+          "ndcg@10": 0.9102925902659809,
+          "mrr@10": 0.8795832323069048,
+          "hit@1": 0.7881677846924755
+        },
+        "arabic_hard_negatives_unseen": {
+          "queries": 314,
+          "ndcg@10": 0.920014765859062,
+          "mrr@10": 0.8926220806794056,
+          "hit@1": 0.8121019108280255
+        },
+        "gemini_test": {
+          "queries": 602,
+          "ndcg@10": 0.843170933731119,
+          "mrr@10": 0.7731173864894796,
+          "hit@1": 0.6146179401993356
+        },
+        "longctx_test": {
+          "queries": 1703,
+          "ndcg@10": 0.8280241783387726,
+          "mrr@10": 0.7658250146799765,
+          "hit@1": 0.5695830886670581
+        },
+        "multi_llm": {
+          "queries": 361,
+          "ndcg@10": 0.7536379978428707,
+          "mrr@10": 0.6840720221606648,
+          "hit@1": 0.481994459833795
+        }
+      },
+      "average": {
+        "ndcg@10": 0.8446462089362008,
+        "mrr@10": 0.7908236358804392,
+        "hit@1": 0.6460957800801629
+      }
+    },
+    "mizan-v2": {
+      "model": "ALJIACHI/Mizan-Rerank-V2",
+      "seconds": 193.55545658099982,
+      "datasets": {
+        "namaa_mrtydi": {
+          "queries": 918,
+          "ndcg@10": 0.8312075449070145,
+          "mrr@10": 0.7755446623093682,
+          "hit@1": 0.6492374727668845
+        },
+        "namaa_mrtydi_unseen": {
+          "queries": 573,
+          "ndcg@10": 0.8220284961389768,
+          "mrr@10": 0.7635253054101222,
+          "hit@1": 0.6335078534031413
+        },
+        "arabic_hard_negatives": {
+          "queries": 12373,
+          "ndcg@10": 0.8447173491693125,
+          "mrr@10": 0.7919327029284194,
+          "hit@1": 0.6433362967752364
+        },
+        "arabic_hard_negatives_unseen": {
+          "queries": 314,
+          "ndcg@10": 0.8779930928755129,
+          "mrr@10": 0.8361995753715499,
+          "hit@1": 0.7133757961783439
+        },
+        "gemini_test": {
+          "queries": 602,
+          "ndcg@10": 0.6965218593427552,
+          "mrr@10": 0.5952104097452935,
+          "hit@1": 0.34053156146179403
+        },
+        "longctx_test": {
+          "queries": 1703,
+          "ndcg@10": 0.6202716837624505,
+          "mrr@10": 0.4982537818415681,
+          "hit@1": 0.22665883734586026
+        },
+        "multi_llm": {
+          "queries": 361,
+          "ndcg@10": 0.5998300071547739,
+          "mrr@10": 0.4628611001187179,
+          "hit@1": 0.19667590027700832
+        }
+      },
+      "average": {
+        "ndcg@10": 0.7233290278548938,
+        "mrr@10": 0.6312100344974503,
+        "hit@1": 0.42214998973322954
+      }
+    },
+    "gte-base": {
+      "model": "Alibaba-NLP/gte-multilingual-reranker-base",
+      "seconds": 190.08565529099997,
+      "datasets": {
+        "namaa_mrtydi": {
+          "queries": 918,
+          "ndcg@10": 0.8873742282014164,
+          "mrr@10": 0.8497639796659405,
+          "hit@1": 0.7549019607843137
+        },
+        "namaa_mrtydi_unseen": {
+          "queries": 573,
+          "ndcg@10": 0.8770828555665509,
+          "mrr@10": 0.8361547411285631,
+          "hit@1": 0.7347294938917975
+        },
+        "arabic_hard_negatives": {
+          "queries": 12373,
+          "ndcg@10": 0.8960300465581245,
+          "mrr@10": 0.8606549206605781,
+          "hit@1": 0.7599612058514508
+        },
+        "arabic_hard_negatives_unseen": {
+          "queries": 314,
+          "ndcg@10": 0.9061892172759463,
+          "mrr@10": 0.8742569002123142,
+          "hit@1": 0.7834394904458599
+        },
+        "gemini_test": {
+          "queries": 602,
+          "ndcg@10": 0.7565245351746636,
+          "mrr@10": 0.6825581395348836,
+          "hit@1": 0.46179401993355484
+        },
+        "longctx_test": {
+          "queries": 1703,
+          "ndcg@10": 0.6747391472836209,
+          "mrr@10": 0.5765022509297318,
+          "hit@1": 0.31356429829712273
+        },
+        "multi_llm": {
+          "queries": 361,
+          "ndcg@10": 0.6992032289960537,
+          "mrr@10": 0.6104801477377655,
+          "hit@1": 0.3601108033240997
+        }
+      },
+      "average": {
+        "ndcg@10": 0.7827477968593671,
+        "mrr@10": 0.7159904359086516,
+        "hit@1": 0.5307276211784869
+      }
+    },
+    "bge-v2-m3": {
+      "model": "BAAI/bge-reranker-v2-m3",
+      "seconds": 416.3012057730002,
+      "datasets": {
+        "namaa_mrtydi": {
+          "queries": 918,
+          "ndcg@10": 0.8995429966626842,
+          "mrr@10": 0.8662309368191722,
+          "hit@1": 0.7864923747276689
+        },
+        "namaa_mrtydi_unseen": {
+          "queries": 573,
+          "ndcg@10": 0.8897374272909373,
+          "mrr@10": 0.8530831878999419,
+          "hit@1": 0.7626527050610821
+        },
+        "arabic_hard_negatives": {
+          "queries": 12373,
+          "ndcg@10": 0.9082092244432917,
+          "mrr@10": 0.8771451278321075,
+          "hit@1": 0.7915622726905358
+        },
+        "arabic_hard_negatives_unseen": {
+          "queries": 314,
+          "ndcg@10": 0.9034780246477423,
+          "mrr@10": 0.871072186836518,
+          "hit@1": 0.7866242038216561
+        },
+        "gemini_test": {
+          "queries": 602,
+          "ndcg@10": 0.7938330217019347,
+          "mrr@10": 0.7374031007751938,
+          "hit@1": 0.5448504983388704
+        },
+        "longctx_test": {
+          "queries": 1703,
+          "ndcg@10": 0.687281090274787,
+          "mrr@10": 0.5878505718200374,
+          "hit@1": 0.33646506165590134
+        },
+        "multi_llm": {
+          "queries": 361,
+          "ndcg@10": 0.7365014887447433,
+          "mrr@10": 0.6643120960295475,
+          "hit@1": 0.4515235457063712
+        }
+      },
+      "average": {
+        "ndcg@10": 0.8021662105320289,
+        "mrr@10": 0.7427442286722477,
+        "hit@1": 0.5764232029167762
+      }
+    }
+  },
+  "note": "Public sets from the Hub; gemini_test, longctx_test and multi_llm are internal listwise test sets (not released), scored with --listwise."
+}

config.json

@@ -0,0 +1,49 @@
+{
+  "architectures": [
+    "NewForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration.NewConfig",
+    "AutoModel": "Alibaba-NLP/new-impl--modeling.NewModel",
+    "AutoModelForMaskedLM": "Alibaba-NLP/new-impl--modeling.NewForMaskedLM",
+    "AutoModelForMultipleChoice": "Alibaba-NLP/new-impl--modeling.NewForMultipleChoice",
+    "AutoModelForQuestionAnswering": "Alibaba-NLP/new-impl--modeling.NewForQuestionAnswering",
+    "AutoModelForSequenceClassification": "modeling.NewForSequenceClassification",
+    "AutoModelForTokenClassification": "Alibaba-NLP/new-impl--modeling.NewForTokenClassification"
+  },
+  "classifier_dropout": 0.0,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "LABEL_0"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "LABEL_0": 0
+  },
+  "layer_norm_eps": 1e-12,
+  "layer_norm_type": "layer_norm",
+  "logn_attention_clip1": false,
+  "logn_attention_scale": false,
+  "max_position_embeddings": 8192,
+  "model_type": "new",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pack_qkv": true,
+  "pad_token_id": 1,
+  "position_embedding_type": "rope",
+  "rope_scaling": {
+    "factor": 8.0,
+    "type": "ntk"
+  },
+  "rope_theta": 20000,
+  "torch_dtype": "float32",
+  "transformers_version": "4.55.4",
+  "type_vocab_size": 1,
+  "unpad_inputs": false,
+  "use_memory_efficient_attention": false,
+  "vocab_size": 250048
+}

config_sentence_transformers.json

@@ -0,0 +1,11 @@
+{
+  "__version__": {
+    "pytorch": "2.8.0+cu126",
+    "sentence_transformers": "5.4.1",
+    "transformers": "4.55.4"
+  },
+  "activation_fn": "torch.nn.modules.activation.Sigmoid",
+  "default_prompt_name": null,
+  "model_type": "CrossEncoder",
+  "prompts": {}
+}

configuration.py

@@ -0,0 +1,145 @@
+# coding=utf-8
+# Copyright 2024 The GTE Team Authors and Alibaba Group.
+# Copyright (c) 2018, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" NEW model configuration"""
+from transformers.configuration_utils import PretrainedConfig
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+
+class NewConfig(PretrainedConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`NewModel`] or a [`TFNewModel`]. It is used to
+    instantiate a NEW model according to the specified arguments, defining the model architecture. Instantiating a
+    configuration with the defaults will yield a similar configuration to that of the NEW
+    [izhx/new-base-en](https://huggingface.co/izhx/new-base-en) architecture.
+
+    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PretrainedConfig`] for more information.
+
+
+    Args:
+        vocab_size (`int`, *optional*, defaults to 30522):
+            Vocabulary size of the NEW model. Defines the number of different tokens that can be represented by the
+            `inputs_ids` passed when calling [`NewModel`] or [`TFNewModel`].
+        hidden_size (`int`, *optional*, defaults to 768):
+            Dimensionality of the encoder layers and the pooler layer.
+        num_hidden_layers (`int`, *optional*, defaults to 12):
+            Number of hidden layers in the Transformer encoder.
+        num_attention_heads (`int`, *optional*, defaults to 12):
+            Number of attention heads for each attention layer in the Transformer encoder.
+        intermediate_size (`int`, *optional*, defaults to 3072):
+            Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer encoder.
+        hidden_act (`str` or `Callable`, *optional*, defaults to `"gelu"`):
+            The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`,
+            `"relu"`, `"silu"` and `"gelu_new"` are supported.
+        hidden_dropout_prob (`float`, *optional*, defaults to 0.1):
+            The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
+        attention_probs_dropout_prob (`float`, *optional*, defaults to 0.1):
+            The dropout ratio for the attention probabilities.
+        max_position_embeddings (`int`, *optional*, defaults to 512):
+            The maximum sequence length that this model might ever be used with. Typically set this to something large
+            just in case (e.g., 512 or 1024 or 2048).
+        type_vocab_size (`int`, *optional*, defaults to 2):
+            The vocabulary size of the `token_type_ids` passed when calling [`NewModel`] or [`TFNewModel`].
+        initializer_range (`float`, *optional*, defaults to 0.02):
+            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
+        layer_norm_eps (`float`, *optional*, defaults to 1e-12):
+            The epsilon used by the layer normalization layers.
+        position_embedding_type (`str`, *optional*, defaults to `"rope"`):
+            Type of position embedding. Choose one of `"absolute"`, `"rope"`.
+        rope_theta (`float`, *optional*, defaults to 10000.0):
+            The base period of the RoPE embeddings.
+        rope_scaling (`Dict`, *optional*):
+            Dictionary containing the scaling configuration for the RoPE embeddings. Currently supports two scaling
+            strategies: linear and dynamic. Their scaling factor must be a float greater than 1. The expected format is
+            `{"type": strategy name, "factor": scaling factor}`. When using this flag, don't update
+            `max_position_embeddings` to the expected new maximum. See the following thread for more information on how
+            these scaling strategies behave:
+            https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases/. This is an
+            experimental feature, subject to breaking API changes in future versions.
+        classifier_dropout (`float`, *optional*):
+            The dropout ratio for the classification head.
+
+    Examples:
+
+    ```python
+    >>> from transformers import NewConfig, NewModel
+
+    >>> # Initializing a NEW izhx/new-base-en style configuration
+    >>> configuration = NewConfig()
+
+    >>> # Initializing a model (with random weights) from the izhx/new-base-en style configuration
+    >>> model = NewModel(configuration)
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```"""
+
+    model_type = "new"
+
+    def __init__(
+        self,
+        vocab_size=30528,
+        hidden_size=768,
+        num_hidden_layers=12,
+        num_attention_heads=12,
+        intermediate_size=3072,
+        hidden_act="gelu",
+        hidden_dropout_prob=0.1,
+        attention_probs_dropout_prob=0.0,
+        max_position_embeddings=2048,
+        type_vocab_size=1,
+        initializer_range=0.02,
+        layer_norm_type='layer_norm',
+        layer_norm_eps=1e-12,
+        # pad_token_id=0,
+        position_embedding_type="rope",
+        rope_theta=10000.0,
+        rope_scaling=None,
+        classifier_dropout=None,
+        pack_qkv=True,
+        unpad_inputs=False,
+        use_memory_efficient_attention=False,
+        logn_attention_scale=False,
+        logn_attention_clip1=False,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+
+        self.vocab_size = vocab_size
+        self.hidden_size = hidden_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.hidden_act = hidden_act
+        self.intermediate_size = intermediate_size
+        self.hidden_dropout_prob = hidden_dropout_prob
+        self.attention_probs_dropout_prob = attention_probs_dropout_prob
+        self.max_position_embeddings = max_position_embeddings
+        self.type_vocab_size = type_vocab_size
+        self.initializer_range = initializer_range
+        self.layer_norm_type = layer_norm_type
+        self.layer_norm_eps = layer_norm_eps
+        self.position_embedding_type = position_embedding_type
+        self.rope_theta = rope_theta
+        self.rope_scaling = rope_scaling
+        self.classifier_dropout = classifier_dropout
+
+        self.pack_qkv = pack_qkv
+        self.unpad_inputs = unpad_inputs
+        self.use_memory_efficient_attention = use_memory_efficient_attention
+        self.logn_attention_scale = logn_attention_scale
+        self.logn_attention_clip1 = logn_attention_clip1

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a10bab1ac9b78b9d882a0a5c604a5c3d9aed472e8c1128ea0cc71c2775816c84
+size 1223854204

modeling.py

@@ -0,0 +1,1418 @@
+# coding=utf-8
+# Copyright 2024 The GTE Team Authors and Alibaba Group.
+# Copyright (c) 2018, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""PyTorch NEW model."""
+
+import math
+from dataclasses import dataclass
+from typing import List, Optional, Tuple, Union
+
+import torch
+import torch.utils.checkpoint
+from torch import nn
+
+from transformers.activations import ACT2FN
+from transformers.modeling_outputs import (
+    BaseModelOutput,
+    BaseModelOutputWithPooling,
+    MaskedLMOutput,
+    MultipleChoiceModelOutput,
+    QuestionAnsweringModelOutput,
+    SequenceClassifierOutput,
+    ModelOutput,
+)
+from transformers.modeling_utils import PreTrainedModel
+from transformers.utils import logging
+
+try:
+    import xformers.ops as xops
+except ImportError as e:
+    xops = None
+
+from .configuration import NewConfig
+
+
+logger = logging.get_logger(__name__)
+
+
+# Adapted from https://github.com/HazyResearch/flash-attention/blob/main/flash_attn/bert_padding.py
+# Which was adapted from https://github.com/mlcommons/training_results_v1.1/blob/main/NVIDIA/benchmarks/bert/implementations/pytorch/padding.py
+class IndexFirstAxis(torch.autograd.Function):
+    @staticmethod
+    def forward(ctx, input, indices):
+        ctx.save_for_backward(indices)
+        assert input.ndim >= 2
+        ctx.first_axis_dim, other_shape = input.shape[0], input.shape[1:]
+        second_dim = other_shape.numel()
+        # TD [2022-03-04] For some reason torch.gather is a bit faster than indexing.
+        # return input[indices]
+        # return torch.gather(
+        #     rearrange(input, "b ... -> b (...)"), 0, repeat(indices, "z -> z d", d=second_dim)
+        # ).reshape(-1, *other_shape)
+        return torch.gather(
+            input.view(ctx.first_axis_dim, second_dim),
+            0,
+            indices.unsqueeze(-1).expand(indices.size(0), second_dim)
+        ).reshape(-1, *other_shape)
+
+    @staticmethod
+    def backward(ctx, grad_output):
+        (indices,) = ctx.saved_tensors
+        assert grad_output.ndim >= 2
+        other_shape = grad_output.shape[1:]
+        # grad_output = rearrange(grad_output, "b ... -> b (...)")
+        grad_output = grad_output.view(grad_output.size(0), other_shape.numel())
+        grad_input = torch.zeros(
+            [ctx.first_axis_dim, grad_output.shape[1]],
+            device=grad_output.device,
+            dtype=grad_output.dtype,
+        )
+        # TD [2022-03-04] For some reason torch.scatter is a bit faster than indexing.
+        # grad_input[indices] = grad_output
+        # grad_input.scatter_(0, repeat(indices, "z -> z d", d=grad_output.shape[1]), grad_output)
+        grad_input.scatter_(
+            0, indices.unsqueeze(-1).expand(indices.size(0), grad_output.size(1)), grad_output
+        )
+        return grad_input.reshape(ctx.first_axis_dim, *other_shape), None
+
+
+index_first_axis = IndexFirstAxis.apply
+
+
+def unpad_input(hidden_states, attention_mask=None, indices=None):
+    """
+    Arguments:
+        hidden_states: (batch, seqlen, ...)
+        attention_mask: (batch, seqlen), bool / int, 1 means valid and 0 means not valid.
+        indices: (total_nnz), the indices of non-masked tokens from the flattened input sequence.
+    Return:
+        hidden_states: (total_nnz, ...), where total_nnz = number of tokens in selected in attention_mask.
+    """
+    if indices is None:
+        assert attention_mask is not None
+        indices = torch.nonzero(attention_mask.flatten(), as_tuple=False).flatten()
+
+    # TD [2022-03-04] We don't want to index with a bool mask, because Pytorch will expand the
+    # bool mask, then call nonzero to get the indices, then index with those. The indices is @dim
+    # times larger than it needs to be, wasting memory. It's faster and more memory-efficient to
+    # index with integer indices. Moreover, torch's index is a bit slower than it needs to be,
+    # so we write custom forward and backward to make it a bit faster.
+    hidden_states = hidden_states.view(-1, *hidden_states.shape[2:])
+    return index_first_axis(hidden_states, indices)
+
+
+class IndexPutFirstAxis(torch.autograd.Function):
+    @staticmethod
+    def forward(
+        ctx,
+        values: torch.Tensor,
+        indices: torch.Tensor,
+        first_axis_dim
+    ) -> torch.Tensor:
+        ctx.save_for_backward(indices)
+        assert indices.ndim == 1
+        assert values.ndim >= 2
+        output = torch.zeros(
+            first_axis_dim, *values.shape[1:], device=values.device, dtype=values.dtype
+        )
+        output[indices] = values
+        return output
+
+    @staticmethod
+    def backward(ctx, grad_output: torch.Tensor) -> Tuple[torch.Tensor, None, None]:
+        indices, = ctx.saved_tensors
+        grad_values = grad_output[indices]
+        return grad_values, None, None
+
+
+index_put_first_axis = IndexPutFirstAxis.apply
+
+
+def pad_input(inputs: torch.Tensor, indices: torch.Tensor, batch: int, seqlen: int) -> torch.Tensor:
+    """Add padding to sequences.
+
+    Arguments:
+        inputs: (total_nnz, ...), where total_nnz = number of tokens in selected in attention_mask.
+        indices: (total_nnz), `indices = torch.nonzero(attention_mask.flatten(), as_tuple=False).flatten()`
+        batch: int batch_size
+        seqlen: int max sequence length
+
+    Returns:
+        inputs: (batch, seqlen, ...)
+    """
+    output = index_put_first_axis(inputs, indices, batch * seqlen)
+    return output.view(batch, seqlen, *inputs.shape[1:])
+
+
+def rotate_half(x):
+    """Rotates half the hidden dims of the input."""
+    x1 = x[..., : x.shape[-1] // 2]
+    x2 = x[..., x.shape[-1] // 2 :]
+    return torch.cat((-x2, x1), dim=-1)
+
+
+def apply_rotary_pos_emb(q, k, cos, sin):
+    """Applies Rotary Position Embedding to the query and key tensors.
+
+    Args:
+        q (`torch.Tensor`): The query tensor.
+        k (`torch.Tensor`): The key tensor.
+        cos (`torch.Tensor`): The cosine part of the rotary embedding.
+        sin (`torch.Tensor`): The sine part of the rotary embedding.
+    Returns:
+        `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding.
+    """
+    cos, sin = cos.to(q.dtype), sin.to(q.dtype)
+    q_embed = (q * cos) + (rotate_half(q) * sin)
+    k_embed = (k * cos) + (rotate_half(k) * sin)
+    return q_embed, k_embed
+
+
+class RotaryEmbedding(torch.nn.Module):
+    def __init__(self, dim, max_position_embeddings=512, base=10000.0, device=None):
+        super().__init__()
+
+        self.dim = dim
+        self.max_position_embeddings = max_position_embeddings
+        self.base = base
+        inv_freq = 1.0 / (self.base ** (torch.arange(0, self.dim, 2).float().to(device) / self.dim))
+        self.register_buffer("inv_freq", inv_freq, persistent=False)
+
+        # Build here to make `torch.jit.trace` work.
+        self._set_cos_sin_cache(
+            seq_len=max_position_embeddings, device=self.inv_freq.device, dtype=torch.get_default_dtype()
+        )
+
+    def _set_cos_sin_cache(self, seq_len, device, dtype):
+        self.max_seq_len_cached = seq_len
+        t = torch.arange(self.max_seq_len_cached, device=device, dtype=torch.float32)
+
+        freqs = torch.einsum("i,j->ij", t, self.inv_freq)
+        # Different from paper, but it uses a different permutation in order to obtain the same calculation
+        emb = torch.cat((freqs, freqs), dim=-1)
+        self.register_buffer("cos_cached", emb.cos().to(dtype), persistent=False)
+        self.register_buffer("sin_cached", emb.sin().to(dtype), persistent=False)
+
+    def forward(self, x, seq_len=None):
+        # x: [bs, num_attention_heads, seq_len, head_size]
+        if seq_len > self.max_seq_len_cached:
+            self._set_cos_sin_cache(seq_len=seq_len, device=x.device, dtype=x.dtype)
+
+        return (
+            self.cos_cached[:seq_len, ...].to(dtype=x.dtype),
+            self.sin_cached[:seq_len, ...].to(dtype=x.dtype),
+        )
+
+
+class NTKScalingRotaryEmbedding(RotaryEmbedding):
+    """RotaryEmbedding extended with fixed and mixed NTK scaling. https://kexue.fm/archives/9706 """
+
+    def __init__(self, dim, max_position_embeddings=512, base=10000, device=None, scaling_factor=1.0, mixed_b=None):
+        self.scaling_factor = scaling_factor
+        self.mixed_b = mixed_b
+        super().__init__(dim, max_position_embeddings, base, device)
+        max_position_embeddings = max_position_embeddings * self.scaling_factor
+        self._set_cos_sin_cache(max_position_embeddings, self.inv_freq.device, torch.get_default_dtype())
+
+    def _set_cos_sin_cache(self, seq_len, device, dtype):
+        self.max_seq_len_cached = seq_len
+
+        if seq_len > self.max_position_embeddings:
+            base = self.base * (self.scaling_factor if self.mixed_b is None else 1)
+            inv_freq = 1.0 / (base ** (torch.arange(0, self.dim, 2).float().to(device) / self.dim))
+
+            if self.mixed_b is None:
+                inv_freq = inv_freq / self.scaling_factor ** (2 / self.dim)  # (6)
+            else:
+                a = torch.tensor(self.scaling_factor).log() / (self.dim / 2) ** self.mixed_b  # (13)
+                lambda_1_m = (a * torch.arange(1, self.dim // 2 + 1).float().to(device) ** self.mixed_b).exp()  # (12)
+                inv_freq = inv_freq / lambda_1_m  # (10)
+
+            self.register_buffer("inv_freq", inv_freq, persistent=False)
+
+        t = torch.arange(self.max_seq_len_cached, device=device, dtype=torch.float32)
+
+        freqs = torch.einsum("i,j->ij", t, self.inv_freq)
+        # Different from paper, but it uses a different permutation in order to obtain the same calculation
+        emb = torch.cat((freqs, freqs), dim=-1)
+        self.register_buffer("cos_cached", emb.cos().to(dtype), persistent=False)
+        self.register_buffer("sin_cached", emb.sin().to(dtype), persistent=False)
+
+
+class RMSNorm(nn.Module):
+    def __init__(self, hidden_size, eps=1e-6):
+        """
+        RMSNorm is equivalent to T5LayerNorm
+        """
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.variance_epsilon = eps
+
+    def forward(self, hidden_states):
+        input_dtype = hidden_states.dtype
+        hidden_states = hidden_states.to(torch.float32)
+        variance = hidden_states.pow(2).mean(-1, keepdim=True)
+        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
+        return self.weight * hidden_states.to(input_dtype)
+
+
+LAYER_NORM = {
+    'layer_norm': nn.LayerNorm,
+    'rms_norm': RMSNorm
+}
+
+
+class NewEmbeddings(nn.Module):
+    """
+    Embedding and Unpadding.
+    """
+
+    def __init__(self, config: NewConfig):
+        super().__init__()
+        self.padding_idx = config.pad_token_id
+        self.word_embeddings = nn.Embedding(
+            config.vocab_size, config.hidden_size, padding_idx=self.padding_idx
+        )
+
+        self.position_embedding_type = config.position_embedding_type
+        if self.position_embedding_type == 'absolute':
+            self.position_embeddings = nn.Embedding(
+                config.max_position_embeddings, config.hidden_size, padding_idx=self.padding_idx
+            )
+        elif self.position_embedding_type == 'rope':
+            self._init_rope(config)
+        else:
+            raise ValueError
+
+        self.type_vocab_size = config.type_vocab_size
+        if self.type_vocab_size > 0:
+            self.token_type_embeddings = nn.Embedding(config.type_vocab_size, config.hidden_size)
+
+        # self.LayerNorm is not snake-cased to stick with TensorFlow model variable name and be able to load
+        # any TensorFlow checkpoint file
+        self.LayerNorm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
+        self.dropout = nn.Dropout(config.hidden_dropout_prob)
+        # position_ids is contiguous in memory and excluded when serialized
+        self.register_buffer(
+            "position_ids", torch.arange(config.max_position_embeddings), persistent=False
+        )
+
+    def _init_rope(self, config):
+        kwargs = dict(
+            dim=int(config.hidden_size / config.num_attention_heads),
+            max_position_embeddings=config.max_position_embeddings,
+            base=config.rope_theta
+        )
+        if config.rope_scaling is None:
+            self.rotary_emb = RotaryEmbedding(**kwargs)
+        else:
+            kwargs.update(scaling_factor=config.rope_scaling["factor"])
+            scaling_type = config.rope_scaling["type"]
+            if scaling_type == 'ntk':
+                kwargs.update(mixed_b=config.rope_scaling.get('mixed_b', None))
+                self.rotary_emb = NTKScalingRotaryEmbedding(**kwargs)
+            # elif scaling_type == "linear":
+            #     self.rotary_emb = LinearScalingRotaryEmbedding(**kwargs)
+            # elif scaling_type == "dynamic":
+            #     self.rotary_emb = DynamicNTKScalingRotaryEmbedding(**kwargs)
+            else:
+                raise ValueError(f"Unknown RoPE scaling type {scaling_type}")
+
+    def forward(
+        self,
+        unpad_inputs: bool,
+        input_ids: Optional[torch.Tensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        length: Optional[List[int]] = None,
+        token_type_ids: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.Tensor] = None,
+        inputs_embeds: Optional[torch.Tensor] = None,
+    ) -> Tuple[torch.Tensor, torch.Tensor, Optional[Tuple], Optional[List[int]]]:
+        """
+        """
+        if inputs_embeds is None:
+            device, input_shape = input_ids.device, input_ids.shape
+        else:
+            device, input_shape = inputs_embeds.device, inputs_embeds.shape[:2]
+        batch_size, seq_length = input_shape
+
+        # Set attention_mask if it's None
+        if attention_mask is None:
+            attention_mask = torch.ones(input_shape, device=device)
+            if length is not None:
+                for i, l in enumerate(length):
+                    attention_mask[i, l:] = 0
+
+        # Set attention_mask_bool for unpadding
+        if unpad_inputs:
+            attention_mask_bool = attention_mask.bool()
+            if length is None:
+                length = attention_mask.sum(-1).tolist()
+
+        # Get word embeddings
+        if inputs_embeds is None:
+            if unpad_inputs:
+                input_ids = input_ids[attention_mask_bool].unsqueeze(0)
+            inputs_embeds = self.word_embeddings(input_ids)
+        else:
+            if unpad_inputs:
+                inputs_embeds = inputs_embeds[attention_mask_bool].unsqueeze(0)
+        embeddings = inputs_embeds
+
+        # Set and unpad position_ids
+        if position_ids is None:
+            if seq_length > self.position_ids.size(0):
+                self.register_buffer(
+                    "position_ids", torch.arange(seq_length, device=embeddings.device), persistent=False
+                )
+            if unpad_inputs:
+                # [1, cumsum_seq_len]
+                position_ids = torch.cat([self.position_ids[:l] for l in length]).unsqueeze(0)
+            else:
+                # [bs, seq_len]
+                position_ids = self.position_ids[:seq_length].expand(batch_size, -1)
+        elif unpad_inputs:
+            position_ids = position_ids[attention_mask_bool].unsqueeze(0)  # [1, cumsum_seq_len]
+
+        # Compute rotary embedding
+        if self.position_embedding_type == 'rope':
+            rope_cos, rope_sin = self.rotary_emb(inputs_embeds, seq_len=seq_length)
+            rope_cos = rope_cos[position_ids].unsqueeze(2)  # [bs, seq_len, 1, dim]
+            rope_sin = rope_sin[position_ids].unsqueeze(2)  # [bs, seq_len, 1, dim]
+            rope_embeds = rope_cos, rope_sin
+        else:
+            rope_embeds = None
+
+        if self.type_vocab_size > 0:
+            if token_type_ids is None:
+                token_type_ids = position_ids.mul(0)
+            else:
+                if self.type_vocab_size < 2:
+                    token_type_ids.mul_(0)
+                if unpad_inputs:
+                    token_type_ids = token_type_ids[attention_mask_bool].unsqueeze(0)
+
+            token_type_embeddings = self.token_type_embeddings(token_type_ids)
+            embeddings = embeddings + token_type_embeddings
+
+        # BERT position
+        if self.position_embedding_type == "absolute":
+            position_embeddings = self.position_embeddings(position_ids)
+            embeddings = embeddings + position_embeddings
+
+        embeddings = self.LayerNorm(embeddings)
+        embeddings = self.dropout(embeddings)
+
+        return embeddings, attention_mask, rope_embeds, length
+
+
+class NewAttention(nn.Module):
+    def __init__(self, config: NewConfig, pack_qkv=None, use_memory_efficient_attention=None):
+        super().__init__()
+        self.config = config
+        if config.hidden_size % config.num_attention_heads != 0 and not hasattr(config, "embedding_size"):
+            raise ValueError(
+                f"The hidden size ({config.hidden_size}) is not a multiple of the number of attention "
+                f"heads ({config.num_attention_heads})"
+            )
+
+        self.hidden_size = config.hidden_size
+        self.num_attention_heads = config.num_attention_heads
+        self.attention_head_size = int(config.hidden_size / config.num_attention_heads)
+        self.all_head_size = self.num_attention_heads * self.attention_head_size
+
+        if pack_qkv is None:
+            pack_qkv = config.pack_qkv
+        self.pack_qkv = pack_qkv
+
+        if self.pack_qkv:
+            self.qkv_proj = nn.Linear(config.hidden_size, self.all_head_size * 3, bias=True)
+        else:
+            self.q_proj = nn.Linear(config.hidden_size, self.all_head_size, bias=True)
+            self.k_proj = nn.Linear(config.hidden_size, self.all_head_size, bias=True)
+            self.v_proj = nn.Linear(config.hidden_size, self.all_head_size, bias=True)
+
+        self.dropout = nn.Dropout(config.attention_probs_dropout_prob)
+        self.o_proj = nn.Linear(config.hidden_size, config.hidden_size, bias=True)
+
+        if use_memory_efficient_attention is None:
+            use_memory_efficient_attention = self.config.use_memory_efficient_attention
+        self.use_memory_efficient_attention = use_memory_efficient_attention
+        self.memory_efficient_attention = None if xops is None else xops.memory_efficient_attention
+        if self.use_memory_efficient_attention:
+            assert self.memory_efficient_attention is not None, 'please install xformers'
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_bias: torch.FloatTensor,
+        rope_embeds: Optional[Tuple[torch.FloatTensor, torch.FloatTensor]] = None,
+        padding_inputs: Optional[Tuple] = None,  # indices, batch, seqlen
+        attention_scale: Optional[torch.FloatTensor] = None,
+        head_mask: Optional[torch.FloatTensor] = None,
+        output_attentions: Optional[bool] = False,
+        qkv_inputs: Optional[Tuple] = None,  # For RetroMAE
+    ) -> Tuple[torch.Tensor, ...]:
+        shape_hd = (self.num_attention_heads, self.attention_head_size)
+        # qkv
+        if self.pack_qkv and qkv_inputs is None:
+            qkv_pack = self.qkv_proj(hidden_states).split(self.all_head_size, dim=-1)
+        else:
+            if qkv_inputs is None:
+                qkv_inputs = (hidden_states, hidden_states, hidden_states)
+            qkv_pack = [
+                getattr(self, n + '_proj')(s) for s, n in zip(qkv_inputs, 'qkv')
+            ]
+        query_states, key_states, value_states = [t.view(t.shape[:-1] + shape_hd) for t in qkv_pack]
+
+        if self.config.position_embedding_type == 'rope':
+            query_states, key_states = apply_rotary_pos_emb(query_states, key_states, *rope_embeds)
+
+        dtype = query_states.dtype
+
+        if self.config.logn_attention_scale and attention_scale is not None:
+            # https://kexue.fm/archives/8823
+            query_states = query_states * attention_scale.to(dtype)
+
+        if padding_inputs is not None:
+            query_states = pad_input(query_states.squeeze(), *padding_inputs)
+            key_states = pad_input(key_states.squeeze(), *padding_inputs)
+            value_states = pad_input(value_states.squeeze(), *padding_inputs)
+
+        if self.use_memory_efficient_attention:
+            assert self.memory_efficient_attention is not None, "xformers is not loaded"
+            assert output_attentions is False, "memory_efficient_attention do not output attentions"
+            assert head_mask is None, "Not support yet"
+            attention_probs = None
+            if torch.is_tensor(attention_bias):
+                attention_bias = attention_bias.to(dtype)
+            context_layer = self.memory_efficient_attention(
+                query_states,
+                key_states,
+                value_states,
+                attn_bias=attention_bias,
+                p=self.dropout.p
+            )
+        else:
+            if output_attentions and isinstance(self, NewSdpaAttention):
+                raise RuntimeError("SDPA do not output attentions")
+            context_layer, attention_probs = self._attention(
+                query_states, key_states, value_states, attention_bias, head_mask
+            )
+
+        if padding_inputs is not None:
+            context_layer = unpad_input(context_layer, indices=padding_inputs[0])
+
+        new_context_layer_shape = context_layer.size()[:-2] + (self.all_head_size,)
+        context_layer = context_layer.view(new_context_layer_shape)
+
+        # output proj
+        attn_output = self.o_proj(context_layer)
+
+        # add attentions if we output them
+        outputs = (attn_output, attention_probs) if output_attentions else (attn_output,)
+        return outputs
+
+    def _attention(self, query_states, key_states, value_states, attention_bias, head_mask):
+        """
+        Args:
+            q/k/v: (B, L, n_head, head_dim),
+        Returns:
+            attn_output: (B L, n_head, head_dim)
+        """
+        query_states = query_states.transpose(1, 2)
+        key_states = key_states.transpose(1, 2)
+        value_states = value_states.transpose(1, 2)
+        # Take the dot product between "query" and "key" to get the raw attention scores.
+        attention_scores = torch.matmul(query_states, key_states.transpose(-1, -2))
+
+        attention_scores = attention_scores / math.sqrt(self.attention_head_size)
+        if attention_bias is not None:
+            # Apply the attention mask is (precomputed for all layers in BertModel forward() function)
+            attention_scores = attention_scores + attention_bias
+
+        # Normalize the attention scores to probabilities.
+        attention_probs = nn.functional.softmax(attention_scores, dim=-1)
+
+        # This is actually dropping out entire tokens to attend to, which might
+        # seem a bit unusual, but is taken from the original Transformer paper.
+        if self.dropout.p > 0:
+            attention_probs = self.dropout(attention_probs)
+
+        # Mask heads if we want to
+        if head_mask is not None:
+            attention_probs = attention_probs * head_mask
+
+        context_layer = torch.matmul(attention_probs, value_states)
+
+        context_layer = context_layer.permute(0, 2, 1, 3).contiguous()
+        return context_layer, attention_probs
+
+
+class NewSdpaAttention(NewAttention):
+    """
+    New attention module using torch.nn.functional.scaled_dot_product_attention. This module inherits from
+    `NewAttention` as the weights of the module stays untouched. The only changes are on the forward pass to adapt to
+    SDPA API.
+    """
+    def __init__(self, config: NewConfig, **kwargs):
+        super().__init__(config, **kwargs)
+        # torch.backends.cuda.enable_mem_efficient_sdp(False)
+        # logger.warning(
+        #     "Disable memory efficient attention kernel for `NewSdpaAttention`, you can set "
+        #     "`use_memory_efficient_attention=True` if it expected to use."
+        # )
+
+    def _attention(self, query_states, key_states, value_states, attention_bias, head_mask):
+        attn_output = torch.nn.functional.scaled_dot_product_attention(
+            query_states.transpose(1, 2),
+            key_states.transpose(1, 2),
+            value_states.transpose(1, 2),
+            attn_mask=attention_bias,
+            dropout_p=self.dropout.p if self.training else 0.0,
+        )
+        attn_output = attn_output.permute(0, 2, 1, 3).contiguous()
+        return attn_output, None
+
+
+NEW_ATTENTION_CLASSES = {
+    "eager": NewAttention,
+    # "flash_attention_2": ,  # TODO
+    "sdpa": NewSdpaAttention,
+}
+
+
+class NewGatedMLP(nn.Module):
+    """
+    GLU Variants Improve Transformer.
+    """
+
+    def __init__(self, config: NewConfig):
+        super().__init__()
+        self.intermediate_size = config.intermediate_size
+        self.up_gate_proj = nn.Linear(config.hidden_size, self.intermediate_size * 2, bias=False)
+        self.down_proj = nn.Linear(self.intermediate_size, config.hidden_size, bias=True)
+        self.act_fn = ACT2FN[config.hidden_act]
+        if config.hidden_dropout_prob > 0:
+            self.hidden_dropout = nn.Dropout(config.hidden_dropout_prob)
+        else:
+            self.hidden_dropout = None
+
+    def forward(self, hidden_states):
+        up_gate = self.up_gate_proj(hidden_states)
+        up_states, gate = torch.split(up_gate, self.intermediate_size, dim=-1)
+        gate = self.act_fn(gate)
+        gated_states = gate * up_states
+        if self.hidden_dropout is not None:
+            gated_states = self.hidden_dropout(gated_states)
+        down_states = self.down_proj(gated_states)
+        return down_states
+
+
+class NewLayer(nn.Module):
+    def __init__(
+        self,
+        config: NewConfig,
+        pack_qkv=None,
+        use_memory_efficient_attention=None,
+        attn_implementation=None
+    ):
+        super().__init__()
+        if attn_implementation is None:
+            attn_implementation = config._attn_implementation
+        if use_memory_efficient_attention is None:
+            use_memory_efficient_attention = config.use_memory_efficient_attention
+        if use_memory_efficient_attention:
+            if attn_implementation != 'eager':
+                logger.warning_once(f"Override {attn_implementation=} to 'eager' as {use_memory_efficient_attention=}")
+                attn_implementation = 'eager'  # Since it will be SDPA by default for torch>=2.1.1
+        self.attention = NEW_ATTENTION_CLASSES[attn_implementation](
+            config, pack_qkv=pack_qkv, use_memory_efficient_attention=use_memory_efficient_attention
+        )
+        self.mlp = NewGatedMLP(config)
+
+        ln_class = LAYER_NORM[config.layer_norm_type]
+        self.attn_ln = ln_class(config.hidden_size, eps=config.layer_norm_eps)
+        self.mlp_ln = ln_class(config.hidden_size, eps=config.layer_norm_eps)
+
+        if config.hidden_dropout_prob > 0:
+            self.hidden_dropout = nn.Dropout(config.hidden_dropout_prob)
+        else:
+            self.hidden_dropout = None
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_bias: torch.FloatTensor,
+        rope_embeds: Optional[Tuple[torch.FloatTensor, torch.FloatTensor]] = None,
+        padding_inputs: Optional[Tuple] = None,  # indices, batch, seqlen
+        attention_scale: Optional[torch.FloatTensor] = None,
+        subset_indices: Optional[torch.LongTensor] = None,
+        head_mask: Optional[torch.FloatTensor] = None,
+        output_attentions: Optional[bool] = False,
+        qkv_inputs: Optional[Tuple] = None,  # For RetroMAE
+    ) -> Tuple[torch.Tensor, ...]:
+        # Multi head self attention
+        residual = hidden_states if qkv_inputs is None else qkv_inputs[0]
+        attention_outputs = self.attention(
+            hidden_states,
+            attention_bias,
+            rope_embeds,
+            padding_inputs,
+            attention_scale,
+            head_mask,
+            output_attentions=output_attentions,
+            qkv_inputs=qkv_inputs,
+        )
+        hidden_states = attention_outputs[0]
+        if self.hidden_dropout is not None:
+            hidden_states = self.hidden_dropout(hidden_states)
+        hidden_states = residual + hidden_states
+
+        # In pretraining, after the attention of last layer, we only need the masked tokens.
+        if subset_indices is not None:
+            hidden_states = hidden_states[subset_indices]
+
+        hidden_states = self.attn_ln(hidden_states)
+
+        # Fully Connected
+        residual = hidden_states
+        hidden_states = self.mlp(hidden_states)
+        if self.hidden_dropout is not None:
+            hidden_states = self.hidden_dropout(hidden_states)
+        hidden_states = residual + hidden_states
+        hidden_states = self.mlp_ln(hidden_states)
+
+        # add self attentions if we output attention weights
+        outputs = (hidden_states,) + attention_outputs[1:]
+        return outputs
+
+
+class NewEncoder(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.layer = nn.ModuleList([NewLayer(config) for _ in range(config.num_hidden_layers)])
+        self.gradient_checkpointing = False
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_bias: Optional[torch.FloatTensor] = None,
+        rope_embeds: Optional[Tuple[torch.FloatTensor, torch.FloatTensor]] = None,
+        padding_inputs: Optional[Tuple] = None,  # indices, batch, seqlen
+        attention_scale: Optional[torch.FloatTensor] = None,
+        subset_indices: Optional[torch.LongTensor] = None,
+        head_mask: Optional[torch.FloatTensor] = None,
+        output_attentions: Optional[bool] = False,
+        output_hidden_states: Optional[bool] = False,
+        return_dict: Optional[bool] = True,
+    ) -> Union[Tuple[torch.Tensor], BaseModelOutput]:
+        all_hidden_states = () if output_hidden_states else None
+        all_self_attentions = () if output_attentions else None
+
+        for i, layer_module in enumerate(self.layer):
+            if output_hidden_states:
+                all_hidden_states = all_hidden_states + (hidden_states,)
+
+            if i >= len(self.layer) - 1:
+                layer_subset_indices = subset_indices
+            else:
+                layer_subset_indices = None
+
+            layer_head_mask = head_mask[i] if head_mask is not None else None
+
+            if self.gradient_checkpointing and self.training:
+                layer_outputs = self._gradient_checkpointing_func(
+                    layer_module.__call__,
+                    hidden_states,
+                    attention_bias,
+                    rope_embeds,
+                    padding_inputs,
+                    attention_scale,
+                    layer_subset_indices,
+                    layer_head_mask,
+                )
+            else:
+                layer_outputs = layer_module(
+                    hidden_states,
+                    attention_bias,
+                    rope_embeds,
+                    padding_inputs,
+                    attention_scale,
+                    layer_subset_indices,
+                    layer_head_mask,
+                    output_attentions,
+                )
+
+            hidden_states = layer_outputs[0]
+            if output_attentions:
+                all_self_attentions = all_self_attentions + (layer_outputs[1],)
+
+        if output_hidden_states:
+            all_hidden_states = all_hidden_states + (hidden_states,)
+
+        if not return_dict:
+            return tuple(
+                v
+                for v in [
+                    hidden_states,
+                    all_hidden_states,
+                    all_self_attentions,
+                ]
+                if v is not None
+            )
+        return BaseModelOutput(
+            last_hidden_state=hidden_states,
+            hidden_states=all_hidden_states,
+            attentions=all_self_attentions,
+        )
+
+
+# Copied from transformers.models.bert.modeling_bert.BertPooler with Bert->New
+class NewPooler(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
+        self.activation = nn.Tanh()
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        # We "pool" the model by simply taking the hidden state corresponding
+        # to the first token.
+        first_token_tensor = hidden_states[:, 0]
+        pooled_output = self.dense(first_token_tensor)
+        pooled_output = self.activation(pooled_output)
+        return pooled_output
+
+
+class NewPreTrainedModel(PreTrainedModel):
+    """
+    An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
+    models.
+    """
+
+    config_class = NewConfig
+    base_model_prefix = "new"
+    supports_gradient_checkpointing = True
+    _supports_sdpa = True
+
+    def _init_weights(self, module):
+        """Initialize the weights"""
+        if isinstance(module, nn.Linear):
+            # Slightly different from the TF version which uses truncated_normal for initialization
+            # cf https://github.com/pytorch/pytorch/pull/5617
+            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+            if module.bias is not None:
+                module.bias.data.zero_()
+        elif isinstance(module, nn.Embedding):
+            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+            if module.padding_idx is not None:
+                module.weight.data[module.padding_idx].zero_()
+        elif isinstance(module, nn.LayerNorm):
+            module.bias.data.zero_()
+            module.weight.data.fill_(1.0)
+
+
+class NewModel(NewPreTrainedModel):
+    """
+    The bare New Model transformer outputting raw hidden-states without any specific head on top.
+    """
+
+    def __init__(self, config: NewConfig, add_pooling_layer=False):
+        super().__init__(config)
+        self.config = config
+
+        self.embeddings = NewEmbeddings(config)
+        self.encoder = NewEncoder(config)
+
+        self.pooler = NewPooler(config) if add_pooling_layer else None
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.embeddings.word_embeddings
+
+    def set_input_embeddings(self, value):
+        self.embeddings.word_embeddings = value
+
+    def forward(
+        self,
+        input_ids: Optional[torch.Tensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        length: Optional[List[int]] = None,
+        subset_indices: Optional[torch.LongTensor] = None,
+        token_type_ids: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.Tensor] = None,
+        head_mask: Optional[torch.Tensor] = None,
+        inputs_embeds: Optional[torch.Tensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        unpad_inputs: Optional[bool] = None,
+    ) -> Union[Tuple[torch.Tensor], BaseModelOutputWithPooling]:
+        r"""
+        length  (`list` of length `batch_size`, *optional*):
+            If is `None`, return padded `last_hidden_state`.
+        subset_indices  ():
+            pass
+        unpad_inputs  (`bool`, *optional*):
+            pass
+        """
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+        unpad_inputs = unpad_inputs if unpad_inputs is not None else self.config.unpad_inputs
+        output_padded = length is None
+
+        if input_ids is not None and inputs_embeds is not None:
+            raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
+        elif input_ids is not None:
+            self.warn_if_padding_and_no_attention_mask(input_ids, attention_mask)
+            input_shape = input_ids.size()
+        elif inputs_embeds is not None:
+            input_shape = inputs_embeds.size()[:-1]
+        else:
+            raise ValueError("You have to specify either input_ids or inputs_embeds")
+
+        # TODO: not used
+        # # Prepare head mask if needed
+        # # 1.0 in head_mask indicate we keep the head
+        # # attention_probs has shape bsz x n_heads x N x N
+        # # input head_mask has shape [num_heads] or [num_hidden_layers x num_heads]
+        # # and head_mask is converted to shape [num_hidden_layers x batch x num_heads x seq_length x seq_length]
+        # head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)
+
+        # Get embeddings, may unpad them
+        (embedding_output, attention_mask, rope_embeds, length) = self.embeddings(
+            unpad_inputs,
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            length=length,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            inputs_embeds=inputs_embeds
+        )
+
+        batch_size, seq_length = input_shape
+        if unpad_inputs and self.config.use_memory_efficient_attention:
+            attention_bias = xops.fmha.attn_bias.BlockDiagonalMask.from_seqlens(length)
+        else:
+            # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length]
+            # ourselves in which case we just need to make it broadcastable to all heads.
+            attention_bias = self.get_extended_attention_mask(attention_mask, input_shape)
+            if self.config.use_memory_efficient_attention:
+                # Invalid shape for attention bias: torch.Size([48, 1, 1, 512]) (expected (48, 12, 512, 512))
+                attention_bias = attention_bias.expand(-1, self.config.num_attention_heads, seq_length, -1)
+
+        padding_inputs = None
+        if unpad_inputs and (output_padded or not self.config.use_memory_efficient_attention):
+            indices = torch.nonzero(attention_mask.flatten(), as_tuple=False).flatten()
+            if not self.config.use_memory_efficient_attention:
+                padding_inputs = (indices, *input_shape)
+
+        attention_scale = None
+        if self.config.logn_attention_scale:
+            logger.warning_once("TODO: logn_attention_scale")
+        #     # attention scale log_512(input_len)
+        #     attention_scale = attention_mask.sum(1).log() / torch.tensor(self.config.max_position_embeddings).log()
+        #     # inference-time logn scale need clip 1
+        #     if self.config.logn_attention_clip1:
+        #         attention_scale.clip_(1)
+        #     attention_scale = attention_scale[:, None, None, None]
+        # else:
+        #     attention_scale = None
+
+        encoder_outputs = self.encoder(
+            embedding_output,
+            attention_bias=attention_bias,
+            rope_embeds=rope_embeds,
+            padding_inputs=padding_inputs,
+            attention_scale=attention_scale,
+            subset_indices=subset_indices,
+            head_mask=head_mask,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+        sequence_output = encoder_outputs[0]
+        if unpad_inputs and output_padded:
+            sequence_output = pad_input(
+                sequence_output.squeeze(), indices, batch_size, seq_length
+            )
+
+        pooled_output = self.pooler(sequence_output) if self.pooler is not None else None
+
+        if not return_dict:
+            return (sequence_output, pooled_output) + encoder_outputs[1:]
+
+        return BaseModelOutputWithPooling(
+            last_hidden_state=sequence_output,
+            pooler_output=pooled_output,
+            hidden_states=encoder_outputs.hidden_states,
+            attentions=encoder_outputs.attentions,
+        )
+
+
+class NewLMPredictionHead(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
+        self.transform_act_fn = ACT2FN[config.hidden_act]
+        self.norm = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
+
+        # The output weights are the same as the input embeddings, but there is
+        # an output-only bias for each token.
+        self.decoder = nn.Linear(config.hidden_size, config.vocab_size)
+
+    def forward(self, hidden_states):
+        hidden_states = self.dense(hidden_states)
+        hidden_states = self.transform_act_fn(hidden_states)
+        hidden_states = self.norm(hidden_states)
+        hidden_states = self.decoder(hidden_states)
+        return hidden_states
+
+
+class NewForMaskedLM(NewPreTrainedModel):
+    _tied_weights_keys = ["lm_head.decoder.bias", "lm_head.decoder.weight"]
+
+    def __init__(self, config: NewConfig):
+        super().__init__(config)
+        self.new = NewModel(config, add_pooling_layer=False)
+        self.lm_head = NewLMPredictionHead(config)
+        self.loss_fct = nn.CrossEntropyLoss()
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_output_embeddings(self):
+        return self.lm_head.decoder
+
+    def set_output_embeddings(self, new_embeddings):
+        self.lm_head.decoder = new_embeddings
+
+    def forward(
+        self,
+        input_ids: Optional[torch.Tensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        token_type_ids: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.Tensor] = None,
+        head_mask: Optional[torch.Tensor] = None,
+        inputs_embeds: Optional[torch.Tensor] = None,
+        labels: Optional[torch.Tensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        unpad_inputs: Optional[bool] = None,
+    ) -> Union[Tuple[torch.Tensor], MaskedLMOutput]:
+        r"""
+        labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
+            Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ...,
+            config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the
+            loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`
+        """
+
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        if labels is None or not self.new.config.unpad_inputs:
+            length = None
+            subset_indices = None
+        else:
+            length = attention_mask.sum(-1).tolist()
+            labels = labels[attention_mask.bool()].unsqueeze(0)
+            subset_indices = labels > -100
+
+        outputs = self.new(
+            input_ids,
+            attention_mask=attention_mask,
+            length=length,
+            subset_indices=subset_indices,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            head_mask=head_mask,
+            inputs_embeds=inputs_embeds,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+            unpad_inputs=unpad_inputs,
+        )
+
+        sequence_output = outputs[0]
+        prediction_scores = self.lm_head(sequence_output)
+
+        masked_lm_loss = None
+        if labels is not None:
+            if subset_indices is None:
+                mask = attention_mask.bool()
+                prediction_scores = prediction_scores[mask]
+                labels = labels[mask]
+            else:
+                labels = labels[subset_indices]
+            masked_lm_loss = self.loss_fct(prediction_scores, labels)
+
+        if not return_dict:
+            output = (prediction_scores,) + outputs[2:]
+            return ((masked_lm_loss,) + output) if masked_lm_loss is not None else output
+
+        return MaskedLMOutput(
+            loss=masked_lm_loss,
+            logits=prediction_scores,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+
+class NewForSequenceClassification(NewPreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+        self.num_labels = config.num_labels
+        self.config = config
+
+        self.new = NewModel(config, add_pooling_layer=True)
+        classifier_dropout = (
+            config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob
+        )
+        self.dropout = nn.Dropout(classifier_dropout)
+        self.classifier = nn.Linear(config.hidden_size, config.num_labels)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def forward(
+        self,
+        input_ids: Optional[torch.Tensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        token_type_ids: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.Tensor] = None,
+        head_mask: Optional[torch.Tensor] = None,
+        inputs_embeds: Optional[torch.Tensor] = None,
+        labels: Optional[torch.Tensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        unpad_inputs: Optional[bool] = None,
+    ) -> Union[Tuple[torch.Tensor], SequenceClassifierOutput]:
+        r"""
+        labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
+            Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
+            config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
+            `config.num_labels > 1` a classification loss is computed (Cross-Entropy).
+        """
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        outputs = self.new(
+            input_ids,
+            attention_mask=attention_mask,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            head_mask=head_mask,
+            inputs_embeds=inputs_embeds,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+            unpad_inputs=unpad_inputs,
+        )
+
+        pooled_output = outputs[1]
+
+        pooled_output = self.dropout(pooled_output)
+        logits = self.classifier(pooled_output)
+
+        loss = None
+        if labels is not None:
+            if self.config.problem_type is None:
+                if self.num_labels == 1:
+                    self.config.problem_type = "regression"
+                elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):
+                    self.config.problem_type = "single_label_classification"
+                else:
+                    self.config.problem_type = "multi_label_classification"
+
+            if self.config.problem_type == "regression":
+                loss_fct = nn.MSELoss()
+                if self.num_labels == 1:
+                    loss = loss_fct(logits.squeeze(), labels.squeeze())
+                else:
+                    loss = loss_fct(logits, labels)
+            elif self.config.problem_type == "single_label_classification":
+                loss_fct = nn.CrossEntropyLoss()
+                loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
+            elif self.config.problem_type == "multi_label_classification":
+                loss_fct = nn.BCEWithLogitsLoss()
+                loss = loss_fct(logits, labels)
+
+        if not return_dict:
+            output = (logits,) + outputs[2:]
+            return ((loss,) + output) if loss is not None else output
+
+        return SequenceClassifierOutput(
+            loss=loss,
+            logits=logits,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+
+class NewForMultipleChoice(NewPreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+
+        self.new = NewModel(config, add_pooling_layer=True)
+        classifier_dropout = (
+            config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob
+        )
+        self.dropout = nn.Dropout(classifier_dropout)
+        self.classifier = nn.Linear(config.hidden_size, 1)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def forward(
+        self,
+        input_ids: Optional[torch.Tensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        token_type_ids: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.Tensor] = None,
+        head_mask: Optional[torch.Tensor] = None,
+        inputs_embeds: Optional[torch.Tensor] = None,
+        labels: Optional[torch.Tensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        unpad_inputs: Optional[bool] = None,
+    ) -> Union[Tuple[torch.Tensor], MultipleChoiceModelOutput]:
+        r"""
+        labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
+            Labels for computing the multiple choice classification loss. Indices should be in `[0, ...,
+            num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See
+            `input_ids` above)
+        """
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+        num_choices = input_ids.shape[1] if input_ids is not None else inputs_embeds.shape[1]
+
+        input_ids = input_ids.view(-1, input_ids.size(-1)) if input_ids is not None else None
+        attention_mask = attention_mask.view(-1, attention_mask.size(-1)) if attention_mask is not None else None
+        token_type_ids = token_type_ids.view(-1, token_type_ids.size(-1)) if token_type_ids is not None else None
+        position_ids = position_ids.view(-1, position_ids.size(-1)) if position_ids is not None else None
+        inputs_embeds = (
+            inputs_embeds.view(-1, inputs_embeds.size(-2), inputs_embeds.size(-1))
+            if inputs_embeds is not None
+            else None
+        )
+
+        outputs = self.new(
+            input_ids,
+            attention_mask=attention_mask,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            head_mask=head_mask,
+            inputs_embeds=inputs_embeds,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+            unpad_inputs=unpad_inputs,
+        )
+
+        pooled_output = outputs[1]
+
+        pooled_output = self.dropout(pooled_output)
+        logits = self.classifier(pooled_output)
+        reshaped_logits = logits.view(-1, num_choices)
+
+        loss = None
+        if labels is not None:
+            loss_fct = nn.CrossEntropyLoss()
+            loss = loss_fct(reshaped_logits, labels)
+
+        if not return_dict:
+            output = (reshaped_logits,) + outputs[2:]
+            return ((loss,) + output) if loss is not None else output
+
+        return MultipleChoiceModelOutput(
+            loss=loss,
+            logits=reshaped_logits,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+
+@dataclass
+class NewTokenClassifierOutput(ModelOutput):
+    loss: Optional[torch.FloatTensor] = None
+    logits: torch.FloatTensor = None
+    last_hidden_state: torch.FloatTensor = None
+    hidden_states: Optional[Tuple[torch.FloatTensor, ...]] = None
+    attentions: Optional[Tuple[torch.FloatTensor, ...]] = None
+
+
+class NewForTokenClassification(NewPreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+        self.num_labels = config.num_labels
+
+        self.new = NewModel(config, add_pooling_layer=False)
+        classifier_dropout = (
+            config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob
+        )
+        self.dropout = nn.Dropout(classifier_dropout)
+        self.classifier = nn.Linear(config.hidden_size, config.num_labels)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def forward(
+        self,
+        input_ids: Optional[torch.Tensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        token_type_ids: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.Tensor] = None,
+        head_mask: Optional[torch.Tensor] = None,
+        inputs_embeds: Optional[torch.Tensor] = None,
+        labels: Optional[torch.Tensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        unpad_inputs: Optional[bool] = None,
+    ) -> Union[Tuple[torch.Tensor], NewTokenClassifierOutput]:
+        r"""
+        labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
+            Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`.
+        """
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        outputs = self.new(
+            input_ids,
+            attention_mask=attention_mask,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            head_mask=head_mask,
+            inputs_embeds=inputs_embeds,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+            unpad_inputs=unpad_inputs,
+        )
+
+        sequence_output = outputs[0]
+
+        sequence_output = self.dropout(sequence_output)
+        logits = self.classifier(sequence_output)
+
+        loss = None
+        if labels is not None:
+            loss_fct = nn.CrossEntropyLoss()
+            loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
+
+        if not return_dict:
+            output = (logits,) + outputs[2:]
+            return ((loss,) + output) if loss is not None else output
+
+        return NewTokenClassifierOutput(
+            loss=loss,
+            logits=logits,
+            last_hidden_state=sequence_output,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+
+class NewForQuestionAnswering(NewPreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+        self.num_labels = config.num_labels
+
+        self.new = NewModel(config, add_pooling_layer=False)
+        self.qa_outputs = nn.Linear(config.hidden_size, config.num_labels)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def forward(
+        self,
+        input_ids: Optional[torch.Tensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        token_type_ids: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.Tensor] = None,
+        head_mask: Optional[torch.Tensor] = None,
+        inputs_embeds: Optional[torch.Tensor] = None,
+        start_positions: Optional[torch.Tensor] = None,
+        end_positions: Optional[torch.Tensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        unpad_inputs: Optional[bool] = None,
+    ) -> Union[Tuple[torch.Tensor], QuestionAnsweringModelOutput]:
+        r"""
+        start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
+            Labels for position (index) of the start of the labelled span for computing the token classification loss.
+            Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
+            are not taken into account for computing the loss.
+        end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
+            Labels for position (index) of the end of the labelled span for computing the token classification loss.
+            Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence
+            are not taken into account for computing the loss.
+        """
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        outputs = self.new(
+            input_ids,
+            attention_mask=attention_mask,
+            token_type_ids=token_type_ids,
+            position_ids=position_ids,
+            head_mask=head_mask,
+            inputs_embeds=inputs_embeds,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+            unpad_inputs=unpad_inputs,
+        )
+
+        sequence_output = outputs[0]
+
+        logits = self.qa_outputs(sequence_output)
+        start_logits, end_logits = logits.split(1, dim=-1)
+        start_logits = start_logits.squeeze(-1).contiguous()
+        end_logits = end_logits.squeeze(-1).contiguous()
+
+        total_loss = None
+        if start_positions is not None and end_positions is not None:
+            # If we are on multi-GPU, split add a dimension
+            if len(start_positions.size()) > 1:
+                start_positions = start_positions.squeeze(-1)
+            if len(end_positions.size()) > 1:
+                end_positions = end_positions.squeeze(-1)
+            # sometimes the start/end positions are outside our model inputs, we ignore these terms
+            ignored_index = start_logits.size(1)
+            start_positions = start_positions.clamp(0, ignored_index)
+            end_positions = end_positions.clamp(0, ignored_index)
+
+            loss_fct = nn.CrossEntropyLoss(ignore_index=ignored_index)
+            start_loss = loss_fct(start_logits, start_positions)
+            end_loss = loss_fct(end_logits, end_positions)
+            total_loss = (start_loss + end_loss) / 2
+
+        if not return_dict:
+            output = (start_logits, end_logits) + outputs[2:]
+            return ((total_loss,) + output) if total_loss is not None else output
+
+        return QuestionAnsweringModelOutput(
+            loss=total_loss,
+            start_logits=start_logits,
+            end_logits=end_logits,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )

modules.json

@@ -0,0 +1,8 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.base.modules.transformer.Transformer"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,10 @@
+{
+    "transformer_task": "sequence-classification",
+    "modality_config": {
+        "text": {
+            "method": "forward",
+            "method_output_name": "logits"
+        }
+    },
+    "module_output_name": "scores"
+}

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ffb37461c391f096759f4a9bbbc329da0f36952f88bab061fcf84940c022e98
+size 17082999

tokenizer_config.json

@@ -0,0 +1,62 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "250001": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "mask_token": "<mask>",
+  "max_length": 512,
+  "model_max_length": 8192,
+  "pad_to_multiple_of": null,
+  "pad_token": "<pad>",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "</s>",
+  "stride": 0,
+  "tokenizer_class": "XLMRobertaTokenizerFast",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "<unk>"
+}