AhmedSSabir commited on
Commit
90316bb
·
verified ·
1 Parent(s): 4c1c2ae

Publish ModernBERT-CNN with concise usage, training details and own evaluation metrics

Browse files
README.md ADDED
@@ -0,0 +1,94 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ library_name: pytorch
4
+ base_model: answerdotai/ModernBERT-base
5
+ tags:
6
+ - modernbert
7
+ - image-captioning
8
+ - visual-relatedness
9
+ - text-classification
10
+ - custom-code
11
+ ---
12
+ # ModernBERT-CNN Visual Caption Relatedness
13
+
14
+ ModernBERT-base with a CNN head that scores the relatedness between **image-context
15
+ words and a candidate caption**. The output is a probability from 0 to 1. Use it
16
+ to rank candidate captions or assign soft pseudo-labels to context/caption pairs.
17
+ The model does not read images or generate captions.
18
+
19
+ ## Usage
20
+
21
+ Install the dependencies from `requirements.txt`, then run:
22
+
23
+ ```python
24
+ from huggingface_hub import snapshot_download
25
+ import sys
26
+
27
+ path = snapshot_download("AhmedSSabir/modernbert-cnn-visual-relatedness-public")
28
+ sys.path.insert(0, path)
29
+ from inference import RelatednessScorer
30
+
31
+ scorer = RelatednessScorer(path, device="cpu") # use "cuda" for a GPU
32
+ scores = scorer.score("dog, ball, grass", [
33
+ "A dog plays with a ball on the grass.",
34
+ "A person cooks in a kitchen.",
35
+ ])
36
+ print(scores)
37
+ ```
38
+
39
+ This is a custom CNN architecture. Use the included inference helper; the
40
+ standard `AutoModel` loader is not configured for this head. Context words
41
+ should come from the image's detected objects or scene labels.
42
+
43
+ ## Model and training
44
+
45
+ - Base: `answerdotai/ModernBERT-base`; all 22 encoder layers fine-tuned,
46
+ embedding parameters frozen.
47
+ - Sequence length: 64 tokens for the combined context/caption input.
48
+ - Head: 32 convolution filters, kernel width 3, ReLU, local max pooling,
49
+ time-major flattening, sigmoid score.
50
+ - Training: 330,790 pairs; validation: 41,275 pairs. Adam, learning rate 1e-5,
51
+ batch size 16, seed 42; BF16/TF32 and compilation used during training.
52
+ - Early stopping after three epochs; **best validation checkpoint: epoch 1**,
53
+ validation loss 0.256281. Base revision and full settings are in
54
+ `training_config.json`.
55
+
56
+ ## Evaluation
57
+
58
+ Caption reranking: **45,000 candidates, nine per image, over 5,000 images**.
59
+ Select the highest model score per image, with the earliest candidate winning
60
+ ties. No reference-based tuning was performed.
61
+
62
+ | Metric | Score |
63
+ | --- | ---: |
64
+ | BLEU-4 | 0.329180 |
65
+ | METEOR | 0.271592 |
66
+ | ROUGE-L | 0.552392 |
67
+ | CIDEr | 1.096723 |
68
+ | SPICE | 0.205827 |
69
+ | BERTScore F1 | 0.939820 |
70
+
71
+ COCO metrics use Stanford PTB tokenization and pycocoevalcap 1.2. BERTScore
72
+ uses RoBERTa-large layer 17, slow tokenization, no IDF or baseline rescaling,
73
+ and all available raw reference captions. Exact metric configurations, package
74
+ versions and evaluated-input hashes are included in `evaluation/`.
75
+
76
+ Additional tests:
77
+
78
+ - **41,849 held-out pseudo-labeled pairs:** accuracy 0.890893, F1 0.904035,
79
+ ROC-AUC 0.961563.
80
+ - **5,822 Flickr8K expert-rated pairs / 1,000 images:** Spearman 0.293677,
81
+ Kendall tau-b 0.212962 using fixed combined image-derived textual context.
82
+
83
+ ## Limitations
84
+
85
+ Training labels derive from semantic similarity scores thresholded at 0.3;
86
+ they are not human judgments. Output probabilities are not calibrated human
87
+ quality ratings, and human-rating agreement is modest. Object extraction
88
+ errors, truncated inputs and domain shift affect results. Evaluation is from
89
+ one training seed, and the derived training split and visual extraction
90
+ pipeline differ from the historical experiment in the associated
91
+ [dataset paper](https://arxiv.org/abs/2301.08784).
92
+
93
+ The repository includes inference-only weights, tokenizer, model/inference
94
+ code, training settings, and this model's aggregate evaluation results.
config.json ADDED
@@ -0,0 +1,107 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "VisualRelatednessModel"
4
+ ],
5
+ "model_type": "modernbert-cnn-relatedness",
6
+ "bert_config": {
7
+ "pad_token_id": 50283,
8
+ "bos_token_id": 50281,
9
+ "eos_token_id": 50282,
10
+ "cls_token_id": 50281,
11
+ "sep_token_id": 50282,
12
+ "tie_word_embeddings": true,
13
+ "vocab_size": 50368,
14
+ "max_position_embeddings": 8192,
15
+ "hidden_size": 768,
16
+ "intermediate_size": 1152,
17
+ "num_hidden_layers": 22,
18
+ "num_attention_heads": 12,
19
+ "initializer_range": 0.02,
20
+ "initializer_cutoff_factor": 2.0,
21
+ "norm_eps": 1e-05,
22
+ "norm_bias": false,
23
+ "attention_bias": false,
24
+ "attention_dropout": 0.0,
25
+ "hidden_activation": "gelu",
26
+ "local_attention": 128,
27
+ "embedding_dropout": 0.0,
28
+ "mlp_bias": false,
29
+ "mlp_dropout": 0.0,
30
+ "decoder_bias": true,
31
+ "classifier_pooling": "mean",
32
+ "classifier_dropout": 0.0,
33
+ "classifier_bias": false,
34
+ "classifier_activation": "gelu",
35
+ "deterministic_flash_attn": false,
36
+ "sparse_prediction": false,
37
+ "sparse_pred_ignore_index": -100,
38
+ "layer_types": [
39
+ "full_attention",
40
+ "sliding_attention",
41
+ "sliding_attention",
42
+ "full_attention",
43
+ "sliding_attention",
44
+ "sliding_attention",
45
+ "full_attention",
46
+ "sliding_attention",
47
+ "sliding_attention",
48
+ "full_attention",
49
+ "sliding_attention",
50
+ "sliding_attention",
51
+ "full_attention",
52
+ "sliding_attention",
53
+ "sliding_attention",
54
+ "full_attention",
55
+ "sliding_attention",
56
+ "sliding_attention",
57
+ "full_attention",
58
+ "sliding_attention",
59
+ "sliding_attention",
60
+ "full_attention"
61
+ ],
62
+ "global_attn_every_n_layers": 3,
63
+ "rope_parameters": {
64
+ "sliding_attention": {
65
+ "rope_type": "default",
66
+ "rope_theta": 10000.0
67
+ },
68
+ "full_attention": {
69
+ "rope_type": "default",
70
+ "rope_theta": 160000.0
71
+ }
72
+ },
73
+ "return_dict": true,
74
+ "output_hidden_states": false,
75
+ "dtype": "float32",
76
+ "chunk_size_feed_forward": 0,
77
+ "is_encoder_decoder": false,
78
+ "architectures": [
79
+ "ModernBertForMaskedLM"
80
+ ],
81
+ "id2label": {
82
+ "0": "LABEL_0",
83
+ "1": "LABEL_1"
84
+ },
85
+ "label2id": {
86
+ "LABEL_0": 0,
87
+ "LABEL_1": 1
88
+ },
89
+ "problem_type": null,
90
+ "_name_or_path": "answerdotai/ModernBERT-base",
91
+ "transformers_version": "5.3.0",
92
+ "gradient_checkpointing": false,
93
+ "layer_norm_eps": 1e-05,
94
+ "model_type": "modernbert",
95
+ "position_embedding_type": "absolute",
96
+ "output_attentions": false
97
+ },
98
+ "model_options": {
99
+ "architecture": "modernbert-cnn",
100
+ "seq_len": 64,
101
+ "cnn_filters": 32,
102
+ "cnn_kernel_size": 3,
103
+ "num_bert_layers": 22,
104
+ "tune_embeddings": false
105
+ },
106
+ "best_validation_loss": 0.2562806813490731
107
+ }
evaluation/human_metrics.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "dataset": "Original Flickr8K ExpertAnnotations.txt",
3
+ "pairs": 5822,
4
+ "images": 1000,
5
+ "checkpoint_sha256": "13502ce4a3578a94effd914525935a2978facd1c7cb243526c3161cbd12db5c2",
6
+ "metrics": {
7
+ "examples": 5822,
8
+ "spearman": 0.2936773447142244,
9
+ "kendall_tau_b": 0.21296227698549494,
10
+ "kendall_tau_c": 0.2081649490635248,
11
+ "pearson": 0.37538085539793303,
12
+ "normalized_human_mae": 0.22123509002405464,
13
+ "empty_image_contexts": 0
14
+ },
15
+ "protocol": "Original mean human ratings; three fixed image-only context policies; no tuning on human ratings",
16
+ "limitations": [
17
+ "Visual extractors differ from training-data extraction; not original-paper replication",
18
+ "Normalized-human MAE is descriptive; binary-trained outputs are not calibrated human ratings",
19
+ "Image-context ablations share test data and are not independent validation sets"
20
+ ]
21
+ }
evaluation/modernbert.bertscore.json ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "predictions_sha256": "8add6269339d94f5abb0542bb64c304feb4584b01708ae8a945e14a1606ac234",
3
+ "references_sha256": "14ebad0ef1a546782a6a77d533c413e8a46ae179d7476872da91edb03c67d38d",
4
+ "bertscore_hash": "roberta-large_L17_no-idf_version=0.3.12(hug_trans=5.3.0)",
5
+ "images": 5000,
6
+ "metrics": {
7
+ "precision": 0.9421024364829064,
8
+ "recall": 0.9399086234927178,
9
+ "f1": 0.9398203430771828
10
+ },
11
+ "protocol": "Raw captions, all available references (at least five), max over references, RoBERTa-large layer 17, no IDF, no rescaling, slow tokenizer",
12
+ "packages": {
13
+ "bert-score": "0.3.13",
14
+ "transformers": "5.3.0",
15
+ "torch": "2.6.0"
16
+ }
17
+ }
evaluation/modernbert.coco_metrics.json ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "images": 5000,
3
+ "metrics": {
4
+ "BLEU_1": 0.7475716038854183,
5
+ "BLEU_2": 0.581953053117491,
6
+ "BLEU_3": 0.43965763874399744,
7
+ "BLEU_4": 0.3291800364936827,
8
+ "METEOR": 0.2715918036531294,
9
+ "ROUGE_L": 0.5523921179429304,
10
+ "CIDEr": 1.0967225482924987,
11
+ "SPICE": 0.20582728578589596
12
+ },
13
+ "predictions_sha256": "8add6269339d94f5abb0542bb64c304feb4584b01708ae8a945e14a1606ac234",
14
+ "references_sha256": "14ebad0ef1a546782a6a77d533c413e8a46ae179d7476872da91edb03c67d38d",
15
+ "tokenizer": "COCO Stanford PTBTokenizer",
16
+ "prediction_file": "captions.json"
17
+ }
evaluation/test_scores.metrics.json ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "examples": 41849,
3
+ "accuracy": 0.890893450261655,
4
+ "precision": 0.9029725417751281,
5
+ "recall": 0.9051005807591953,
6
+ "f1": 0.9040353089533417,
7
+ "roc_auc": 0.9615626055488327,
8
+ "binary_cross_entropy": 0.2513202982970983,
9
+ "threshold": 0.5,
10
+ "confusion_matrix": [
11
+ [
12
+ 15776,
13
+ 2311
14
+ ],
15
+ [
16
+ 2255,
17
+ 21507
18
+ ]
19
+ ]
20
+ }
inference.py ADDED
@@ -0,0 +1,44 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Inference for exported BERT-CNN and ModernBERT-CNN relatedness models."""
2
+ import json
3
+ from pathlib import Path
4
+
5
+ import torch
6
+ from safetensors.torch import load_file
7
+ from transformers import AutoTokenizer
8
+
9
+ from modeling import VisualRelatednessModel, backbone_config_from_dict
10
+
11
+
12
+ class RelatednessScorer:
13
+ def __init__(self, model_path, device="cpu"):
14
+ path = Path(model_path)
15
+ if not path.is_dir():
16
+ from huggingface_hub import snapshot_download
17
+ path = Path(snapshot_download(str(model_path), allow_patterns=["config.json", "model.safetensors", "tokenizer/*"]))
18
+ config = json.loads((path / "config.json").read_text())
19
+ self.device = torch.device(device)
20
+ self.model = VisualRelatednessModel(backbone_config_from_dict(config["bert_config"]), **config["model_options"])
21
+ self.model.load_state_dict(load_file(str(path / "model.safetensors")))
22
+ self.model.to(self.device).eval()
23
+ self.tokenizer = AutoTokenizer.from_pretrained(path / "tokenizer", local_files_only=True, use_fast=True)
24
+ self.tokenizer.padding_side = "right"
25
+ self.seq_len = config["model_options"]["seq_len"]
26
+
27
+ @torch.inference_mode()
28
+ def score(self, visual_context, captions, batch_size=16):
29
+ if not isinstance(visual_context, str) or not visual_context.strip():
30
+ raise ValueError("Provide image-context words, such as dog, ball, grass")
31
+ if not captions or any(not isinstance(c, str) or not c.strip() for c in captions):
32
+ raise ValueError("Provide at least one nonempty caption")
33
+ if batch_size < 1:
34
+ raise ValueError("batch_size must be positive")
35
+ scores = []
36
+ for start in range(0, len(captions), batch_size):
37
+ selected = captions[start:start + batch_size]
38
+ batch = self.tokenizer([visual_context] * len(selected), selected, padding="max_length",
39
+ truncation="longest_first", max_length=self.seq_len,
40
+ return_tensors="pt").to(self.device)
41
+ scores.extend(self.model.probabilities(self.model(**batch)).float().cpu().tolist())
42
+ if not all(0 <= score <= 1 for score in scores):
43
+ raise RuntimeError("Model produced invalid scores")
44
+ return scores
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8911f0618f9b9a3f66ae5630343c49b4ba2dd5df0e4f4317ec7bd110b6603c76
3
+ size 596370156
modeling.py ADDED
@@ -0,0 +1,122 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Original BERT heads/masking and explicit ModernBERT comparison variants."""
2
+
3
+ import torch
4
+ from torch import nn
5
+ from transformers import BertConfig, BertModel, ModernBertConfig, ModernBertModel, ModernBertForSequenceClassification
6
+
7
+
8
+ def backbone_config_from_dict(values):
9
+ kind = values.get("model_type", "bert")
10
+ classes = {"bert": BertConfig, "modernbert": ModernBertConfig}
11
+ if kind not in classes:
12
+ raise ValueError(f"Unsupported checkpoint backbone: {kind}")
13
+ return classes[kind].from_dict(values)
14
+
15
+
16
+ class VisualRelatednessModel(nn.Module):
17
+ def __init__(self, bert_config, architecture="bert-cnn", seq_len=64,
18
+ cnn_filters=32, cnn_kernel_size=3, num_bert_layers=None,
19
+ tune_embeddings=None, backbone=None):
20
+ super().__init__()
21
+ if architecture not in {"bert", "bert-cnn", "modernbert", "modernbert-cnn"}:
22
+ raise ValueError("Unsupported model architecture")
23
+ modern = architecture.startswith("modernbert")
24
+ if modern != (bert_config.model_type == "modernbert"):
25
+ raise ValueError("Architecture and pretrained backbone type do not match")
26
+ if not 3 <= seq_len <= bert_config.max_position_embeddings:
27
+ raise ValueError("seq_len must be between 3 and BERT's positional embedding limit")
28
+ layers = bert_config.num_hidden_layers if num_bert_layers is None else num_bert_layers
29
+ if not 0 <= layers <= bert_config.num_hidden_layers:
30
+ raise ValueError("num_bert_layers is outside this BERT model's layer range")
31
+ if cnn_filters < 1 or cnn_kernel_size < 1:
32
+ raise ValueError("CNN filter count and kernel size must be positive")
33
+ self.options = dict(architecture=architecture, seq_len=seq_len,
34
+ cnn_filters=cnn_filters, cnn_kernel_size=cnn_kernel_size,
35
+ num_bert_layers=layers,
36
+ tune_embeddings=architecture in {"bert", "modernbert"} if tune_embeddings is None else tune_embeddings)
37
+ if modern:
38
+ # Let the outer training loop control compilation explicitly.
39
+ bert_config.reference_compile = False
40
+ bert_config._attn_implementation = "sdpa"
41
+ model_class = ModernBertForSequenceClassification if architecture == "modernbert" else ModernBertModel
42
+ self.bert = backbone if backbone is not None else model_class(bert_config)
43
+ else:
44
+ self.bert = backbone if backbone is not None else BertModel(
45
+ bert_config, add_pooling_layer=architecture == "bert")
46
+ hidden = bert_config.hidden_size
47
+ if architecture == "bert":
48
+ self.dropout = nn.Dropout(.1)
49
+ self.classifier = nn.Linear(hidden, 2)
50
+ nn.init.normal_(self.classifier.weight, std=.02)
51
+ nn.init.zeros_(self.classifier.bias)
52
+ elif architecture != "modernbert":
53
+ pooled_length = (seq_len - cnn_kernel_size + 1) // 2
54
+ if pooled_length < 1:
55
+ raise ValueError("seq_len must be at least cnn_kernel_size + 1")
56
+ self.conv = nn.Conv1d(hidden, cnn_filters, cnn_kernel_size)
57
+ self.pool = nn.MaxPool1d(2, stride=2)
58
+ self.classifier = nn.Linear(pooled_length * cnn_filters, 1)
59
+ # Keras Glorot initialization for the original CNN and dense head.
60
+ nn.init.xavier_uniform_(self.conv.weight)
61
+ nn.init.zeros_(self.conv.bias)
62
+ nn.init.xavier_uniform_(self.classifier.weight)
63
+ nn.init.zeros_(self.classifier.bias)
64
+ self.configure_finetuning()
65
+
66
+ @classmethod
67
+ def from_pretrained(cls, source, revision=None, **options):
68
+ architecture = options.get("architecture", "bert-cnn")
69
+ if architecture.startswith("modernbert"):
70
+ model_class = ModernBertForSequenceClassification if architecture == "modernbert" else ModernBertModel
71
+ extra = {"num_labels": 2} if architecture == "modernbert" else {}
72
+ bert = model_class.from_pretrained(source, revision=revision, reference_compile=False,
73
+ attn_implementation="sdpa", **extra)
74
+ else:
75
+ bert = BertModel.from_pretrained(source, revision=revision, add_pooling_layer=architecture == "bert")
76
+ return cls(bert.config, backbone=bert, **options)
77
+
78
+ def configure_finetuning(self):
79
+ self.bert.requires_grad_(False)
80
+ modern = self.options["architecture"].startswith("modernbert")
81
+ encoder = self.bert.model if self.options["architecture"] == "modernbert" else self.bert
82
+ if self.options["tune_embeddings"]:
83
+ encoder.embeddings.requires_grad_(True)
84
+ count = self.options["num_bert_layers"]
85
+ if count:
86
+ layers = encoder.layers if modern else encoder.encoder.layer
87
+ for layer in layers[-count:]:
88
+ layer.requires_grad_(True)
89
+ if modern:
90
+ encoder.final_norm.requires_grad_(True)
91
+ if self.options["architecture"] == "modernbert":
92
+ self.bert.head.requires_grad_(True)
93
+ self.bert.classifier.requires_grad_(True)
94
+ if self.options["architecture"] == "bert" and self.bert.pooler is not None:
95
+ self.bert.pooler.requires_grad_(True)
96
+
97
+ def forward(self, input_ids, attention_mask, token_type_ids=None):
98
+ if self.options["architecture"].startswith("modernbert"):
99
+ output = self.bert(input_ids=input_ids, attention_mask=attention_mask, return_dict=True)
100
+ if self.options["architecture"] == "modernbert":
101
+ return output.logits
102
+ else:
103
+ output = self.bert(input_ids=input_ids, attention_mask=attention_mask,
104
+ token_type_ids=token_type_ids, return_dict=True)
105
+ if self.options["architecture"] == "bert":
106
+ return self.classifier(self.dropout(output.pooler_output))
107
+ if input_ids.shape[1] != self.options["seq_len"]:
108
+ raise ValueError("BERT-CNN requires fixed padding to the checkpoint's seq_len")
109
+ # The old BertLayer zeroed padded token vectors BEFORE the convolution.
110
+ sequence = output.last_hidden_state * attention_mask.unsqueeze(-1).to(output.last_hidden_state.dtype)
111
+ pooled = self.pool(torch.relu(self.conv(sequence.transpose(1, 2))))
112
+ # Keras Flatten traverses time then filters, rather than filters then time.
113
+ flat = pooled.transpose(1, 2).contiguous().flatten(1)
114
+ return self.classifier(flat).squeeze(-1)
115
+
116
+ def probabilities(self, logits):
117
+ return logits.softmax(-1)[:, 1] if self.options["architecture"] in {"bert", "modernbert"} else logits.sigmoid()
118
+
119
+ def loss(self, logits, labels):
120
+ if self.options["architecture"] in {"bert", "modernbert"}:
121
+ return nn.functional.cross_entropy(logits, labels.long())
122
+ return nn.functional.binary_cross_entropy_with_logits(logits, labels.float())
provenance.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "source_checkpoint_sha256": "13502ce4a3578a94effd914525935a2978facd1c7cb243526c3161cbd12db5c2",
3
+ "model_safetensors_sha256": "8911f0618f9b9a3f66ae5630343c49b4ba2dd5df0e4f4317ec7bd110b6603c76",
4
+ "architecture": "modernbert-cnn",
5
+ "best_epoch": 1,
6
+ "weights_format": "Inference-only safetensors"
7
+ }
requirements.txt ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ torch>=2.6,<3
2
+ transformers==5.3.0
3
+ safetensors>=0.4
4
+ huggingface_hub>=1,<2
tokenizer/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer/tokenizer_config.json ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "backend": "tokenizers",
3
+ "clean_up_tokenization_spaces": true,
4
+ "cls_token": "[CLS]",
5
+ "is_local": true,
6
+ "mask_token": "[MASK]",
7
+ "model_input_names": [
8
+ "input_ids",
9
+ "attention_mask"
10
+ ],
11
+ "model_max_length": 8192,
12
+ "pad_token": "[PAD]",
13
+ "sep_token": "[SEP]",
14
+ "tokenizer_class": "TokenizersBackend",
15
+ "unk_token": "[UNK]"
16
+ }
training_config.json ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "base_model": "answerdotai/ModernBERT-base",
3
+ "base_revision": "8949b909ec900327062f0ebf497f51aef5e6f0c8",
4
+ "architecture": "modernbert-cnn",
5
+ "sequence_length": 64,
6
+ "encoder_layers_tuned": 22,
7
+ "embeddings_frozen": true,
8
+ "cnn_filters": 32,
9
+ "cnn_kernel": 3,
10
+ "training_pairs": 330790,
11
+ "validation_pairs": 41275,
12
+ "test_pairs": 41849,
13
+ "optimizer": "Adam",
14
+ "learning_rate": 1e-05,
15
+ "batch_size": 16,
16
+ "seed": 42,
17
+ "warmup_ratio": 0.1,
18
+ "max_epochs": 5,
19
+ "completed_epochs": 3,
20
+ "early_stopping_patience": 2,
21
+ "exported_best_epoch": 1,
22
+ "best_validation_loss": 0.2562806813490731,
23
+ "precision": "bf16",
24
+ "tf32": true,
25
+ "torch_compile": true,
26
+ "split": "Connected components of image IDs and normalized unordered pairs"
27
+ }