SwarajSolanke-turtle commited on
Commit
507e9e8
Β·
verified Β·
1 Parent(s): 6b9f55f

adding tsdae_script

Browse files
Files changed (1) hide show
  1. tsda.ipynb +1015 -0
tsda.ipynb ADDED
@@ -0,0 +1,1015 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cells": [
3
+ {
4
+ "cell_type": "markdown",
5
+ "id": "intro",
6
+ "metadata": {},
7
+ "source": [
8
+ "# TSDAE fine-tuning on the FW vehicle master\n",
9
+ "\n",
10
+ "Domain-adapts a sentence encoder to insurer vehicle strings built from `Make`, `Model`, `Variant`, `Fuel`, `CC`\n",
11
+ "out of `data/FW_cleaned_data_new_base_15.csv`.\n",
12
+ "\n",
13
+ "Same TSDAE recipe as `TSDAE_script.ipynb`, with two substitutions:\n",
14
+ "\n",
15
+ "| | `TSDAE_script.ipynb` | this notebook |\n",
16
+ "|---|---|---|\n",
17
+ "| corpus | `wiki1m-for-simcse` | vehicle strings from the base-15 CSV |\n",
18
+ "| evaluator | STS-B (generic semantic similarity) | triplets of same-vehicle vs different-vehicle strings |\n",
19
+ "\n",
20
+ "STS-B is dropped because it scores general English sentence similarity and says nothing about whether\n",
21
+ "`maruti swift vxi petrol 1197` lands near its sibling variants. The triplet evaluator below measures exactly that,\n",
22
+ "which is the property the downstream mapping step depends on."
23
+ ]
24
+ },
25
+ {
26
+ "cell_type": "code",
27
+ "execution_count": 2,
28
+ "id": "install",
29
+ "metadata": {},
30
+ "outputs": [
31
+ {
32
+ "name": "stdout",
33
+ "output_type": "stream",
34
+ "text": [
35
+ "/home/ubuntu/Motor_Mapping/.venv/bin/python: No module named uv\n",
36
+ "Note: you may need to restart the kernel to use updated packages.\n"
37
+ ]
38
+ }
39
+ ],
40
+ "source": [
41
+ "# This venv has only pandas/numpy. Install the training stack once (~3 GB with CUDA torch):\n",
42
+ "%uv pip install \"sentence-transformers>=3.0\" nltk accelerate"
43
+ ]
44
+ },
45
+ {
46
+ "cell_type": "code",
47
+ "execution_count": 18,
48
+ "id": "imports",
49
+ "metadata": {},
50
+ "outputs": [
51
+ {
52
+ "name": "stderr",
53
+ "output_type": "stream",
54
+ "text": [
55
+ "/home/ubuntu/Motor_Mapping/.venv/lib/python3.12/site-packages/nltk/downloader.py:1076: UserWarning: NLTK will not authorize the non-private download directory '/home/ubuntu/nltk_data': it (or an ancestor) is world- or group-writable, so another local user could plant files there. Choose a private location such as ~/nltk_data.\n",
56
+ " for msg in self.incr_download(info_or_id, download_dir, force):\n"
57
+ ]
58
+ },
59
+ {
60
+ "data": {
61
+ "text/plain": [
62
+ "True"
63
+ ]
64
+ },
65
+ "execution_count": 18,
66
+ "metadata": {},
67
+ "output_type": "execute_result"
68
+ }
69
+ ],
70
+ "source": [
71
+ "import random\n",
72
+ "from collections import defaultdict\n",
73
+ "\n",
74
+ "import nltk\n",
75
+ "import pandas as pd\n",
76
+ "from datasets import Dataset\n",
77
+ "from nltk import word_tokenize\n",
78
+ "from nltk.tokenize.treebank import TreebankWordDetokenizer\n",
79
+ "from sentence_transformers import (\n",
80
+ " SentenceTransformer,\n",
81
+ " SentenceTransformerTrainer,\n",
82
+ " SentenceTransformerTrainingArguments,\n",
83
+ " losses,\n",
84
+ " models,\n",
85
+ " util,\n",
86
+ ")\n",
87
+ "# from sentence_transformers.evaluation import TripletEvaluator\n",
88
+ "from sentence_transformers.sentence_transformer.evaluation import TripletEvaluator\n",
89
+ "nltk.download(\"punkt_tab\", quiet=True)"
90
+ ]
91
+ },
92
+ {
93
+ "cell_type": "markdown",
94
+ "id": "config-md",
95
+ "metadata": {},
96
+ "source": [
97
+ "## Config\n",
98
+ "\n",
99
+ "Two values differ from the wiki1m script because vehicle strings are short (~6 tokens, not ~25):\n",
100
+ "\n",
101
+ "- `MAX_SEQ_LENGTH = 32` instead of 75 β€” no string comes close to 75 tokens, so the extra padding is wasted compute.\n",
102
+ "- `DEL_RATIO = 0.4` instead of 0.6 β€” deleting 60% of a 6-token string leaves 2 tokens, which usually destroys the\n",
103
+ " make/model identity and gives the decoder nothing to reconstruct from. 0.4 keeps ~4 tokens."
104
+ ]
105
+ },
106
+ {
107
+ "cell_type": "code",
108
+ "execution_count": 19,
109
+ "id": "config",
110
+ "metadata": {},
111
+ "outputs": [],
112
+ "source": [
113
+ "CSV_PATH = \"/home/ubuntu/Motor_Mapping/MotorMapping/cleaning_data/data/FW_cleaned_data_new_base_15.csv\"\n",
114
+ "OUTPUT_DIR = \"/home/ubuntu/Motor_Mapping/MotorMapping/cleaning_data/models/tsdae-fw\"\n",
115
+ "\n",
116
+ "MODEL_NAME = \"google-bert/bert-base-uncased\"\n",
117
+ "MAX_SEQ_LENGTH = 32\n",
118
+ "TRAIN_BATCH_SIZE = 64\n",
119
+ "NUM_EPOCHS = 10\n",
120
+ "LEARNING_RATE = 1e-5 # 3e-5\n",
121
+ "DEL_RATIO = 0.3\n",
122
+ "TEST_SIZE = 0.02\n",
123
+ "N_EVAL_TRIPLETS = 2000\n",
124
+ "SEED = 46\n",
125
+ "\n",
126
+ "random.seed(SEED)"
127
+ ]
128
+ },
129
+ {
130
+ "cell_type": "markdown",
131
+ "id": "corpus-md",
132
+ "metadata": {},
133
+ "source": [
134
+ "## Build the corpus\n",
135
+ "\n",
136
+ "One string per row: `make model variant fuel cc`, lowercased, with `CC` as an integer and empty/zero fields dropped.\n",
137
+ "Deduplicated, because the 153k rows contain the same vehicle string many times over (once per insurer) and TSDAE\n",
138
+ "gains nothing from seeing an identical string repeatedly."
139
+ ]
140
+ },
141
+ {
142
+ "cell_type": "code",
143
+ "execution_count": 20,
144
+ "id": "corpus",
145
+ "metadata": {},
146
+ "outputs": [
147
+ {
148
+ "name": "stdout",
149
+ "output_type": "stream",
150
+ "text": [
151
+ "rows: 153,738 unique strings: 88,215\n",
152
+ "\n",
153
+ " ac six vintage six royal roadstar petrol 1995\n",
154
+ " alfa romeo 156 gta std petrol 3179\n",
155
+ " alfa romeo alfa romeo alfa 159 3.2 jts v6 ti petrol 3195\n",
156
+ " alfa romeo giulia sprint 1600 gta petrol 1570\n",
157
+ " alfa romeo vintage alfa romeo petrol 1110\n",
158
+ " alfa romeo vintage alfa romeo petrol 1570\n",
159
+ " alfa romeo vintage alfa romeo petrol 1750\n",
160
+ " alfa romeo vintage romeo petrol 1100\n",
161
+ "\n",
162
+ "count 5000.000000\n",
163
+ "mean 7.820600\n",
164
+ "std 1.801961\n",
165
+ "min 3.000000\n",
166
+ "25% 7.000000\n",
167
+ "50% 8.000000\n",
168
+ "75% 9.000000\n",
169
+ "max 17.000000\n",
170
+ "dtype: float64\n"
171
+ ]
172
+ }
173
+ ],
174
+ "source": [
175
+ "FEATURE_COLS = [\"Make\", \"Model\", \"Variant\", \"Fuel\", \"CC\"]\n",
176
+ "GROUP_COLS = [\"MAKE\", \"NEW_MODELS\"]\n",
177
+ "NUMERIC_COLS = {\"CC\", \"NEW_CC\", \"SC\"}\n",
178
+ "\n",
179
+ "df = pd.read_csv(CSV_PATH, usecols=FEATURE_COLS + GROUP_COLS, low_memory=False)\n",
180
+ "\n",
181
+ "\n",
182
+ "def build_text(frame, cols=FEATURE_COLS):\n",
183
+ " parts = []\n",
184
+ " for col in cols:\n",
185
+ " if col in NUMERIC_COLS:\n",
186
+ " s = pd.to_numeric(frame[col], errors=\"coerce\").fillna(0).astype(int).astype(str)\n",
187
+ " s = s.replace(\"0\", \"\")\n",
188
+ " else:\n",
189
+ " s = frame[col].fillna(\"\").astype(str).str.lower().str.strip()\n",
190
+ " parts.append(s)\n",
191
+ " text = parts[0]\n",
192
+ " for s in parts[1:]:\n",
193
+ " text = text.str.cat(s, sep=\" \")\n",
194
+ " return text.str.replace(r\"\\s+\", \" \", regex=True).str.strip()\n",
195
+ "\n",
196
+ "\n",
197
+ "df[\"text\"] = build_text(df)\n",
198
+ "corpus = sorted({t for t in df[\"text\"] if t})\n",
199
+ "\n",
200
+ "print(f\"rows: {len(df):,} unique strings: {len(corpus):,}\")\n",
201
+ "print()\n",
202
+ "for t in corpus[:8]:\n",
203
+ " print(\" \", t)\n",
204
+ "print()\n",
205
+ "print(pd.Series([len(word_tokenize(t)) for t in corpus[:5000]]).describe())"
206
+ ]
207
+ },
208
+ {
209
+ "cell_type": "markdown",
210
+ "id": "noise-md",
211
+ "metadata": {},
212
+ "source": [
213
+ "## Add deletion noise\n",
214
+ "\n",
215
+ "TSDAE's objective: reconstruct the original string from a corrupted copy, forcing the pooled `[CLS]` vector to carry\n",
216
+ "the whole meaning. `delete_noise` keeps each token with probability `1 - DEL_RATIO`, and always keeps at least one\n",
217
+ "token β€” on strings this short an unguarded sampling run can delete everything, and an empty input teaches nothing.\n",
218
+ "\n",
219
+ "`DenoisingAutoEncoderLoss` reads the columns positionally as `(damaged, original)`, so `select_columns` pins that order."
220
+ ]
221
+ },
222
+ {
223
+ "cell_type": "code",
224
+ "execution_count": 21,
225
+ "id": "noise",
226
+ "metadata": {},
227
+ "outputs": [
228
+ {
229
+ "name": "stderr",
230
+ "output_type": "stream",
231
+ "text": [
232
+ "Map: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 88215/88215 [00:05<00:00, 15160.07 examples/s]"
233
+ ]
234
+ },
235
+ {
236
+ "name": "stdout",
237
+ "output_type": "stream",
238
+ "text": [
239
+ "Dataset({\n",
240
+ " features: ['damaged', 'original'],\n",
241
+ " num_rows: 86450\n",
242
+ "})\n",
243
+ " '1.5 petrol 1482' <- 'hyundai alcazar 1.5 executive mt petrol 1482'\n",
244
+ " 'mahindra. 300 1.5 w8 tone diesel 1497' <- 'mahindra. xuv 300 1.5 w8 dual tone diesel 1497'\n",
245
+ " 'mahindra scorpio vlx mhawk bs iv airbag diesel 2179' <- 'mahindra scorpio vlx 2.2 mhawk bs iv airbag diesel 2179'\n",
246
+ " 'kia carens clavis o bsvi diesel' <- 'kia carens clavis htk plus (o) bsvi diesel 1493'\n",
247
+ " 'nissan kicks 1.3 pre (o) petrol 1298' <- 'nissan motors kicks 1.3 turbo xv pre (o) bsvi petrol 1298'\n"
248
+ ]
249
+ },
250
+ {
251
+ "name": "stderr",
252
+ "output_type": "stream",
253
+ "text": [
254
+ "\n"
255
+ ]
256
+ }
257
+ ],
258
+ "source": [
259
+ "detokenizer = TreebankWordDetokenizer()\n",
260
+ "\n",
261
+ "\n",
262
+ "def delete_noise(text, del_ratio=DEL_RATIO):\n",
263
+ " words = word_tokenize(text)\n",
264
+ " if not words:\n",
265
+ " return text\n",
266
+ " kept = [w for w in words if random.random() >= del_ratio]\n",
267
+ " if not kept:\n",
268
+ " kept = [random.choice(words)]\n",
269
+ " return detokenizer.detokenize(kept)\n",
270
+ "\n",
271
+ "\n",
272
+ "def noise_batch(batch):\n",
273
+ " return {\"damaged\": [delete_noise(t) for t in batch[\"original\"]]}\n",
274
+ "\n",
275
+ "\n",
276
+ "dataset = Dataset.from_dict({\"original\": corpus})\n",
277
+ "dataset = dataset.map(noise_batch, batched=True, batch_size=2000)\n",
278
+ "dataset = dataset.select_columns([\"damaged\", \"original\"])\n",
279
+ "\n",
280
+ "split = dataset.train_test_split(test_size=TEST_SIZE, seed=SEED)\n",
281
+ "train_dataset, eval_dataset = split[\"train\"], split[\"test\"]\n",
282
+ "\n",
283
+ "print(train_dataset)\n",
284
+ "for i in range(5):\n",
285
+ " row = train_dataset[i]\n",
286
+ " print(f\" {row['damaged']!r:<45} <- {row['original']!r}\")"
287
+ ]
288
+ },
289
+ {
290
+ "cell_type": "markdown",
291
+ "id": "model-md",
292
+ "metadata": {},
293
+ "source": [
294
+ "## Model and loss\n",
295
+ "\n",
296
+ "CLS pooling and a tied encoder/decoder, per the TSDAE paper. `decoder_name_or_path` takes the model *name*, not the\n",
297
+ "`SentenceTransformer` object."
298
+ ]
299
+ },
300
+ {
301
+ "cell_type": "code",
302
+ "execution_count": 22,
303
+ "id": "model",
304
+ "metadata": {},
305
+ "outputs": [
306
+ {
307
+ "name": "stderr",
308
+ "output_type": "stream",
309
+ "text": [
310
+ "Loading weights: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 199/199 [00:00<00:00, 3305.71it/s]\n",
311
+ "[transformers] \u001b[1mBertModel LOAD REPORT\u001b[0m from: google-bert/bert-base-uncased\n",
312
+ "Key | Status | | \n",
313
+ "-------------------------------------------+------------+--+-\n",
314
+ "cls.seq_relationship.bias | UNEXPECTED | | \n",
315
+ "cls.predictions.transform.LayerNorm.bias | UNEXPECTED | | \n",
316
+ "cls.predictions.transform.dense.weight | UNEXPECTED | | \n",
317
+ "cls.predictions.transform.dense.bias | UNEXPECTED | | \n",
318
+ "cls.seq_relationship.weight | UNEXPECTED | | \n",
319
+ "cls.predictions.transform.LayerNorm.weight | UNEXPECTED | | \n",
320
+ "cls.predictions.bias | UNEXPECTED | | \n",
321
+ "\n",
322
+ "Notes:\n",
323
+ "- UNEXPECTED:\tcan be ignored when loading from different task/architecture; not ok if you expect identical arch.\n",
324
+ "/tmp/ipykernel_117871/364354980.py:2: FutureWarning: The `get_word_embedding_dimension` method has been renamed to `get_embedding_dimension`.\n",
325
+ " pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension(), \"cls\")\n",
326
+ "When tie_encoder_decoder=True, the decoder_name_or_path will be invalid.\n",
327
+ "Loading weights: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 202/202 [00:00<00:00, 3428.92it/s]\n",
328
+ "[transformers] \u001b[1mBertLMHeadModel LOAD REPORT\u001b[0m from: google-bert/bert-base-uncased\n",
329
+ "Key | Status | \n",
330
+ "-------------------------------------------------------------------+------------+-\n",
331
+ "bert.pooler.dense.bias | UNEXPECTED | \n",
332
+ "bert.pooler.dense.weight | UNEXPECTED | \n",
333
+ "cls.seq_relationship.bias | UNEXPECTED | \n",
334
+ "cls.seq_relationship.weight | UNEXPECTED | \n",
335
+ "bert.encoder.layer.{0...11}.crossattention.self.key.bias | MISSING | \n",
336
+ "bert.encoder.layer.{0...11}.crossattention.output.LayerNorm.bias | MISSING | \n",
337
+ "bert.encoder.layer.{0...11}.crossattention.output.LayerNorm.weight | MISSING | \n",
338
+ "bert.encoder.layer.{0...11}.crossattention.self.value.weight | MISSING | \n",
339
+ "bert.encoder.layer.{0...11}.crossattention.self.query.weight | MISSING | \n",
340
+ "bert.encoder.layer.{0...11}.crossattention.output.dense.bias | MISSING | \n",
341
+ "bert.encoder.layer.{0...11}.crossattention.self.query.bias | MISSING | \n",
342
+ "bert.encoder.layer.{0...11}.crossattention.output.dense.weight | MISSING | \n",
343
+ "bert.encoder.layer.{0...11}.crossattention.self.value.bias | MISSING | \n",
344
+ "bert.encoder.layer.{0...11}.crossattention.self.key.weight | MISSING | \n",
345
+ "\n",
346
+ "Notes:\n",
347
+ "- UNEXPECTED:\tcan be ignored when loading from different task/architecture; not ok if you expect identical arch.\n",
348
+ "- MISSING:\tthose params were newly initialized because missing from the checkpoint. Consider training on your downstream task.\n"
349
+ ]
350
+ },
351
+ {
352
+ "name": "stdout",
353
+ "output_type": "stream",
354
+ "text": [
355
+ "SentenceTransformer(\n",
356
+ " (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})\n",
357
+ " (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'cls', 'include_prompt': True})\n",
358
+ ")\n"
359
+ ]
360
+ }
361
+ ],
362
+ "source": [
363
+ "word_embedding_model = models.Transformer(MODEL_NAME, max_seq_length=MAX_SEQ_LENGTH)\n",
364
+ "pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension(), \"cls\")\n",
365
+ "model = SentenceTransformer(modules=[word_embedding_model, pooling_model])\n",
366
+ "\n",
367
+ "train_loss = losses.DenoisingAutoEncoderLoss(\n",
368
+ " model,\n",
369
+ " decoder_name_or_path=MODEL_NAME,\n",
370
+ " tie_encoder_decoder=True,\n",
371
+ ")\n",
372
+ "\n",
373
+ "print(model)"
374
+ ]
375
+ },
376
+ {
377
+ "cell_type": "markdown",
378
+ "id": "eval-md",
379
+ "metadata": {},
380
+ "source": [
381
+ "## Domain evaluator\n",
382
+ "\n",
383
+ "Triplets are derived from columns the CSV already has, so no hand labelling is needed:\n",
384
+ "\n",
385
+ "- **group** = `MAKE` + `NEW_MODELS` + `Fuel` β€” the canonical vehicle, from the cleaned columns.\n",
386
+ "- **anchor / positive** = two different raw strings in the same group (same vehicle, different insurer wording).\n",
387
+ "- **negative** = a string from a *different* group under the *same* make β€” a hard negative, since `maruti swift` and\n",
388
+ " `maruti dzire` share tokens and are the pairs the mapper actually confuses.\n",
389
+ "\n",
390
+ "The reported accuracy is the fraction of triplets where the anchor sits closer to its positive than to its negative.\n",
391
+ "Note this is measured on strings the unsupervised training also sees; the grouping is never shown to the model, so it\n",
392
+ "scores embedding geometry rather than generalization to unseen strings."
393
+ ]
394
+ },
395
+ {
396
+ "cell_type": "code",
397
+ "execution_count": 23,
398
+ "id": "evaluator",
399
+ "metadata": {},
400
+ "outputs": [
401
+ {
402
+ "name": "stdout",
403
+ "output_type": "stream",
404
+ "text": [
405
+ "groups :0 premier | padmini | petrol\n",
406
+ "1 mahindra | economy | diesel\n",
407
+ "2 mahindra | forester | diesel\n",
408
+ "3 toyota | land cruiser | diesel\n",
409
+ "4 mahindra | forester | diesel\n",
410
+ " ... \n",
411
+ "153733 citroen | c3 aircross | petrol\n",
412
+ "153734 mahindra | scorpio | diesel\n",
413
+ "153735 tata | safari | diesel\n",
414
+ "153736 bmw | xm | petrol\n",
415
+ "153737 tesla | model y | electric\n",
416
+ "Length: 153738, dtype: str\n",
417
+ "grouped:group\n",
418
+ "ac | vintage | petrol [ac six vintage six royal roadstar petrol 1995]\n",
419
+ "alfa | 156 gta | petrol [alfa romeo 156 gta std petrol 3179]\n",
420
+ "alfa | alfa romeo | petrol [alfa romeo alfa romeo alfa 159 3.2 jts v6 ti ...\n",
421
+ "alfa | giulia sprint | petrol [alfa romeo giulia sprint 1600 gta petrol 1570]\n",
422
+ "alfa | vintage | petrol [alfa romeo vintage alfa romeo petrol 1570, al...\n",
423
+ " ... \n",
424
+ "wolseley | 16/20 | petrol [wolseley 16/20 saloon petrol 1140]\n",
425
+ "wolseley | vintage | petrol [wolseley vintage 12 hp petrol 1260, wolseley ...\n",
426
+ "wuling | almaz rs | hybrid [wuling almaz rs 2.0 pro hybrid hybrid 1999]\n",
427
+ "yamaha | yzf r15 | petrol [yamaha yzf r15 yzf r15 bs-iv petrol 155]\n",
428
+ "zotye | rx 6400 | diesel [zotye auto rx 6400 1.3 l diesel 1527]\n",
429
+ "Name: text, Length: 3671, dtype: object\n",
430
+ "dev_evaluator :<sentence_transformers.sentence_transformer.evaluation.triplet.TripletEvaluator object at 0x7d53e2333c80>\n",
431
+ "groups with >=2 strings: 2,392 triplets: 2,000\n",
432
+ "\n",
433
+ "anchor: standard herald vintage station wagon petrol 1100\n",
434
+ "positive: standard herald vintage vintage petrol 800\n",
435
+ "negative: standard standard super 10 petrol 1340\n",
436
+ "\n",
437
+ "anchor: mahindra reva electric vehicles e 20 t 2 48v electric 1\n",
438
+ "positive: mahindra reva e 20 t 2 electric 1\n",
439
+ "negative: mahindra electric mobility limited treo treo sft electric 1\n",
440
+ "\n",
441
+ "anchor: audi a5 2.0 tdi cabriolet diesel 1968\n",
442
+ "positive: audi a5 cabriolet 2.0 tdi diesel 1984\n",
443
+ "negative: audi rs q8 4.0 tfsi quattro petrol 3998\n",
444
+ "\n"
445
+ ]
446
+ }
447
+ ],
448
+ "source": [
449
+ "group_key = (\n",
450
+ " df[\"MAKE\"].fillna(\"\").astype(str).str.lower().str.strip()\n",
451
+ " + \" | \"\n",
452
+ " + df[\"NEW_MODELS\"].fillna(\"\").astype(str).str.lower().str.strip()\n",
453
+ " + \" | \"\n",
454
+ " + df[\"Fuel\"].fillna(\"\").astype(str).str.lower().str.strip()\n",
455
+ ")\n",
456
+ "print(f\"groups :{group_key}\")\n",
457
+ "\n",
458
+ "df[\"group\"] = group_key\n",
459
+ "\n",
460
+ "grouped = df[df[\"text\"] != \"\"].groupby(\"group\")[\"text\"].apply(lambda s: list(dict.fromkeys(s)))\n",
461
+ "print(f\"grouped:{grouped}\")\n",
462
+ "grouped = grouped[grouped.map(len) >= 2]\n",
463
+ "\n",
464
+ "by_make = defaultdict(list)\n",
465
+ "for g in grouped.index:\n",
466
+ " by_make[g.split(\" | \")[0]].append(g)\n",
467
+ "\n",
468
+ "rng = random.Random(SEED)\n",
469
+ "group_names = list(grouped.index)\n",
470
+ "rng.shuffle(group_names)\n",
471
+ "\n",
472
+ "anchors, positives, negatives = [], [], []\n",
473
+ "for g in group_names:\n",
474
+ " if len(anchors) >= N_EVAL_TRIPLETS:\n",
475
+ " break\n",
476
+ " a, p = rng.sample(grouped[g], 2)\n",
477
+ " same_make = [x for x in by_make[g.split(\" | \")[0]] if x != g]\n",
478
+ " pool = same_make or [x for x in group_names if x != g]\n",
479
+ " anchors.append(a)\n",
480
+ " positives.append(p)\n",
481
+ " negatives.append(rng.choice(grouped[rng.choice(pool)]))\n",
482
+ "\n",
483
+ "dev_evaluator = TripletEvaluator(\n",
484
+ " anchors=anchors,\n",
485
+ " positives=positives,\n",
486
+ " negatives=negatives,\n",
487
+ " name=\"fw-variant-dev\",\n",
488
+ ")\n",
489
+ "\n",
490
+ "\n",
491
+ "print(f\"dev_evaluator :{dev_evaluator}\")\n",
492
+ "\n",
493
+ "print(f\"groups with >=2 strings: {len(grouped):,} triplets: {len(anchors):,}\\n\")\n",
494
+ "for i in range(3):\n",
495
+ " print(f\"anchor: {anchors[i]}\")\n",
496
+ " print(f\"positive: {positives[i]}\")\n",
497
+ " print(f\"negative: {negatives[i]}\\n\")"
498
+ ]
499
+ },
500
+ {
501
+ "cell_type": "code",
502
+ "execution_count": 24,
503
+ "id": "baseline",
504
+ "metadata": {},
505
+ "outputs": [
506
+ {
507
+ "name": "stdout",
508
+ "output_type": "stream",
509
+ "text": [
510
+ "{'fw-variant-dev_cosine_accuracy': 0.8845000267028809}\n"
511
+ ]
512
+ }
513
+ ],
514
+ "source": [
515
+ "# Baseline before any training β€” compare against the post-training number below.\n",
516
+ "print(dev_evaluator(model))"
517
+ ]
518
+ },
519
+ {
520
+ "cell_type": "code",
521
+ "execution_count": 25,
522
+ "id": "args",
523
+ "metadata": {},
524
+ "outputs": [
525
+ {
526
+ "name": "stderr",
527
+ "output_type": "stream",
528
+ "text": [
529
+ "The `warmup_ratio` argument is deprecated in Transformers v5+, and will also be removed from Sentence Transformers once support for Transformers v4 is dropped. Since you're using Transformers v5+, please use `warmup_steps` (as a float) to specify the warmup ratio instead.\n"
530
+ ]
531
+ }
532
+ ],
533
+ "source": [
534
+ "args = SentenceTransformerTrainingArguments(\n",
535
+ " output_dir=OUTPUT_DIR,\n",
536
+ " num_train_epochs=NUM_EPOCHS,\n",
537
+ " per_device_train_batch_size=TRAIN_BATCH_SIZE,\n",
538
+ " per_device_eval_batch_size=TRAIN_BATCH_SIZE,\n",
539
+ " learning_rate=LEARNING_RATE,\n",
540
+ " weight_decay=0.0,\n",
541
+ " warmup_ratio=0.1,\n",
542
+ " bf16=True,\n",
543
+ " fp16=False,\n",
544
+ " eval_strategy=\"steps\",\n",
545
+ " eval_steps=500,\n",
546
+ " save_steps=1000,\n",
547
+ " save_total_limit=2,\n",
548
+ " logging_steps=50,\n",
549
+ " seed=SEED,\n",
550
+ " run_name=\"tsdae_new-fw\",\n",
551
+ ")"
552
+ ]
553
+ },
554
+ {
555
+ "cell_type": "code",
556
+ "execution_count": 26,
557
+ "id": "trainer",
558
+ "metadata": {},
559
+ "outputs": [
560
+ {
561
+ "name": "stderr",
562
+ "output_type": "stream",
563
+ "text": [
564
+ " \r"
565
+ ]
566
+ }
567
+ ],
568
+ "source": [
569
+ "trainer = SentenceTransformerTrainer(\n",
570
+ " model=model,\n",
571
+ " args=args,\n",
572
+ " train_dataset=train_dataset,\n",
573
+ " eval_dataset=eval_dataset,\n",
574
+ " loss=train_loss,\n",
575
+ " evaluator=dev_evaluator,\n",
576
+ ")"
577
+ ]
578
+ },
579
+ {
580
+ "cell_type": "code",
581
+ "execution_count": 27,
582
+ "id": "train",
583
+ "metadata": {},
584
+ "outputs": [
585
+ {
586
+ "data": {
587
+ "text/html": [
588
+ "\n",
589
+ " <div>\n",
590
+ " \n",
591
+ " <progress value='13510' max='13510' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
592
+ " [13510/13510 38:36, Epoch 10/10]\n",
593
+ " </div>\n",
594
+ " <table border=\"1\" class=\"dataframe\">\n",
595
+ " <thead>\n",
596
+ " <tr style=\"text-align: left;\">\n",
597
+ " <th>Step</th>\n",
598
+ " <th>Training Loss</th>\n",
599
+ " <th>Validation Loss</th>\n",
600
+ " <th>Fw-variant-dev Cosine Accuracy</th>\n",
601
+ " </tr>\n",
602
+ " </thead>\n",
603
+ " <tbody>\n",
604
+ " <tr>\n",
605
+ " <td>500</td>\n",
606
+ " <td>3.519659</td>\n",
607
+ " <td>3.315504</td>\n",
608
+ " <td>0.888500</td>\n",
609
+ " </tr>\n",
610
+ " <tr>\n",
611
+ " <td>1000</td>\n",
612
+ " <td>1.906371</td>\n",
613
+ " <td>1.835182</td>\n",
614
+ " <td>0.968000</td>\n",
615
+ " </tr>\n",
616
+ " <tr>\n",
617
+ " <td>1500</td>\n",
618
+ " <td>1.264665</td>\n",
619
+ " <td>1.237239</td>\n",
620
+ " <td>0.967000</td>\n",
621
+ " </tr>\n",
622
+ " <tr>\n",
623
+ " <td>2000</td>\n",
624
+ " <td>1.035154</td>\n",
625
+ " <td>0.979282</td>\n",
626
+ " <td>0.968000</td>\n",
627
+ " </tr>\n",
628
+ " <tr>\n",
629
+ " <td>2500</td>\n",
630
+ " <td>0.904463</td>\n",
631
+ " <td>0.855952</td>\n",
632
+ " <td>0.963500</td>\n",
633
+ " </tr>\n",
634
+ " <tr>\n",
635
+ " <td>3000</td>\n",
636
+ " <td>0.766660</td>\n",
637
+ " <td>0.772152</td>\n",
638
+ " <td>0.964500</td>\n",
639
+ " </tr>\n",
640
+ " <tr>\n",
641
+ " <td>3500</td>\n",
642
+ " <td>0.737115</td>\n",
643
+ " <td>0.723460</td>\n",
644
+ " <td>0.965000</td>\n",
645
+ " </tr>\n",
646
+ " <tr>\n",
647
+ " <td>4000</td>\n",
648
+ " <td>0.697386</td>\n",
649
+ " <td>0.687482</td>\n",
650
+ " <td>0.966000</td>\n",
651
+ " </tr>\n",
652
+ " <tr>\n",
653
+ " <td>4500</td>\n",
654
+ " <td>0.622651</td>\n",
655
+ " <td>0.657369</td>\n",
656
+ " <td>0.964000</td>\n",
657
+ " </tr>\n",
658
+ " <tr>\n",
659
+ " <td>5000</td>\n",
660
+ " <td>0.617842</td>\n",
661
+ " <td>0.635764</td>\n",
662
+ " <td>0.964500</td>\n",
663
+ " </tr>\n",
664
+ " <tr>\n",
665
+ " <td>5500</td>\n",
666
+ " <td>0.561297</td>\n",
667
+ " <td>0.606327</td>\n",
668
+ " <td>0.965000</td>\n",
669
+ " </tr>\n",
670
+ " <tr>\n",
671
+ " <td>6000</td>\n",
672
+ " <td>0.545678</td>\n",
673
+ " <td>0.599327</td>\n",
674
+ " <td>0.967500</td>\n",
675
+ " </tr>\n",
676
+ " <tr>\n",
677
+ " <td>6500</td>\n",
678
+ " <td>0.542860</td>\n",
679
+ " <td>0.576558</td>\n",
680
+ " <td>0.964500</td>\n",
681
+ " </tr>\n",
682
+ " <tr>\n",
683
+ " <td>7000</td>\n",
684
+ " <td>0.491351</td>\n",
685
+ " <td>0.570122</td>\n",
686
+ " <td>0.965500</td>\n",
687
+ " </tr>\n",
688
+ " <tr>\n",
689
+ " <td>7500</td>\n",
690
+ " <td>0.478219</td>\n",
691
+ " <td>0.564613</td>\n",
692
+ " <td>0.964000</td>\n",
693
+ " </tr>\n",
694
+ " <tr>\n",
695
+ " <td>8000</td>\n",
696
+ " <td>0.481421</td>\n",
697
+ " <td>0.548468</td>\n",
698
+ " <td>0.967500</td>\n",
699
+ " </tr>\n",
700
+ " <tr>\n",
701
+ " <td>8500</td>\n",
702
+ " <td>0.452666</td>\n",
703
+ " <td>0.542519</td>\n",
704
+ " <td>0.965500</td>\n",
705
+ " </tr>\n",
706
+ " <tr>\n",
707
+ " <td>9000</td>\n",
708
+ " <td>0.447252</td>\n",
709
+ " <td>0.534722</td>\n",
710
+ " <td>0.965500</td>\n",
711
+ " </tr>\n",
712
+ " <tr>\n",
713
+ " <td>9500</td>\n",
714
+ " <td>0.416157</td>\n",
715
+ " <td>0.526371</td>\n",
716
+ " <td>0.967000</td>\n",
717
+ " </tr>\n",
718
+ " <tr>\n",
719
+ " <td>10000</td>\n",
720
+ " <td>0.419636</td>\n",
721
+ " <td>0.521664</td>\n",
722
+ " <td>0.966500</td>\n",
723
+ " </tr>\n",
724
+ " <tr>\n",
725
+ " <td>10500</td>\n",
726
+ " <td>0.421646</td>\n",
727
+ " <td>0.519755</td>\n",
728
+ " <td>0.967000</td>\n",
729
+ " </tr>\n",
730
+ " <tr>\n",
731
+ " <td>11000</td>\n",
732
+ " <td>0.379481</td>\n",
733
+ " <td>0.511343</td>\n",
734
+ " <td>0.966500</td>\n",
735
+ " </tr>\n",
736
+ " <tr>\n",
737
+ " <td>11500</td>\n",
738
+ " <td>0.377998</td>\n",
739
+ " <td>0.507382</td>\n",
740
+ " <td>0.967000</td>\n",
741
+ " </tr>\n",
742
+ " <tr>\n",
743
+ " <td>12000</td>\n",
744
+ " <td>0.371554</td>\n",
745
+ " <td>0.505013</td>\n",
746
+ " <td>0.965500</td>\n",
747
+ " </tr>\n",
748
+ " <tr>\n",
749
+ " <td>12500</td>\n",
750
+ " <td>0.364893</td>\n",
751
+ " <td>0.500249</td>\n",
752
+ " <td>0.966000</td>\n",
753
+ " </tr>\n",
754
+ " <tr>\n",
755
+ " <td>13000</td>\n",
756
+ " <td>0.362321</td>\n",
757
+ " <td>0.498431</td>\n",
758
+ " <td>0.966500</td>\n",
759
+ " </tr>\n",
760
+ " <tr>\n",
761
+ " <td>13500</td>\n",
762
+ " <td>0.370435</td>\n",
763
+ " <td>0.497299</td>\n",
764
+ " <td>0.965500</td>\n",
765
+ " </tr>\n",
766
+ " <tr>\n",
767
+ " <td>13510</td>\n",
768
+ " <td>0.370435</td>\n",
769
+ " <td>0.497360</td>\n",
770
+ " <td>0.966000</td>\n",
771
+ " </tr>\n",
772
+ " </tbody>\n",
773
+ "</table><p>"
774
+ ],
775
+ "text/plain": [
776
+ "<IPython.core.display.HTML object>"
777
+ ]
778
+ },
779
+ "metadata": {},
780
+ "output_type": "display_data"
781
+ },
782
+ {
783
+ "name": "stderr",
784
+ "output_type": "stream",
785
+ "text": [
786
+ "Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:00<00:00, 1.71it/s]\n",
787
+ "Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:00<00:00, 2.25it/s]\n",
788
+ "Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:00<00:00, 2.10it/s]\n",
789
+ "Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:00<00:00, 2.10it/s]\n",
790
+ "Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:00<00:00, 2.17it/s]\n",
791
+ "Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:00<00:00, 2.20it/s]\n",
792
+ "Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:00<00:00, 2.18it/s]\n",
793
+ "Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:00<00:00, 2.15it/s]\n",
794
+ "Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:00<00:00, 2.17it/s]\n",
795
+ "Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:00<00:00, 2.17it/s]\n",
796
+ "Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:00<00:00, 2.19it/s]\n",
797
+ "Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:00<00:00, 2.20it/s]\n",
798
+ "Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:00<00:00, 2.13it/s]\n",
799
+ "Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:00<00:00, 2.20it/s]\n"
800
+ ]
801
+ },
802
+ {
803
+ "data": {
804
+ "text/plain": [
805
+ "TrainOutput(global_step=13510, training_loss=0.860432569968268, metrics={'train_runtime': 2316.8877, 'train_samples_per_second': 373.13, 'train_steps_per_second': 5.831, 'total_flos': 0.0, 'train_loss': 0.860432569968268, 'epoch': 10.0})"
806
+ ]
807
+ },
808
+ "execution_count": 27,
809
+ "metadata": {},
810
+ "output_type": "execute_result"
811
+ }
812
+ ],
813
+ "source": [
814
+ "trainer.train()"
815
+ ]
816
+ },
817
+ {
818
+ "cell_type": "code",
819
+ "execution_count": 28,
820
+ "id": "post-eval",
821
+ "metadata": {},
822
+ "outputs": [
823
+ {
824
+ "name": "stdout",
825
+ "output_type": "stream",
826
+ "text": [
827
+ "{'fw-variant-dev_cosine_accuracy': 0.9660000205039978}\n"
828
+ ]
829
+ }
830
+ ],
831
+ "source": [
832
+ "print(dev_evaluator(model))"
833
+ ]
834
+ },
835
+ {
836
+ "cell_type": "code",
837
+ "execution_count": 29,
838
+ "id": "save",
839
+ "metadata": {},
840
+ "outputs": [
841
+ {
842
+ "name": "stderr",
843
+ "output_type": "stream",
844
+ "text": [
845
+ "Writing model shards: 0%| | 0/1 [00:00<?, ?it/s]"
846
+ ]
847
+ },
848
+ {
849
+ "name": "stderr",
850
+ "output_type": "stream",
851
+ "text": [
852
+ "Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:02<00:00, 2.75s/it]"
853
+ ]
854
+ },
855
+ {
856
+ "name": "stdout",
857
+ "output_type": "stream",
858
+ "text": [
859
+ "/home/ubuntu/Motor_Mapping/MotorMapping/cleaning_data/models/tsdae-fw/final\n"
860
+ ]
861
+ },
862
+ {
863
+ "name": "stderr",
864
+ "output_type": "stream",
865
+ "text": [
866
+ "\n"
867
+ ]
868
+ }
869
+ ],
870
+ "source": [
871
+ "final_output_dir = f\"{OUTPUT_DIR}/final\"\n",
872
+ "model.save(final_output_dir)\n",
873
+ "print(final_output_dir)"
874
+ ]
875
+ },
876
+ {
877
+ "cell_type": "markdown",
878
+ "id": "nn-md",
879
+ "metadata": {},
880
+ "source": [
881
+ "## Sanity check: nearest neighbours\n",
882
+ "\n",
883
+ "The triplet accuracy is one number; this shows what the embedding space actually looks like for the mapping task.\n",
884
+ "Swap in strings from insurer files you care about."
885
+ ]
886
+ },
887
+ {
888
+ "cell_type": "code",
889
+ "execution_count": null,
890
+ "id": "nn",
891
+ "metadata": {},
892
+ "outputs": [
893
+ {
894
+ "name": "stderr",
895
+ "output_type": "stream",
896
+ "text": [
897
+ "Batches: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 345/345 [00:11<00:00, 31.25it/s]\n"
898
+ ]
899
+ },
900
+ {
901
+ "name": "stdout",
902
+ "output_type": "stream",
903
+ "text": [
904
+ "\n",
905
+ "query: maruti swift vxi petrol 1197\n",
906
+ " 1.000 maruti swift vxi petrol 1197\n",
907
+ " 0.999 maruti suzuki swift vxi petrol 1197\n",
908
+ " 0.998 maruti swift vxi cng 1197\n",
909
+ " 0.998 maruti swift vxi (o) petrol 1197\n",
910
+ " 0.998 maruti swift vxi amt petrol 1197\n",
911
+ "\n",
912
+ "query: hyundai i20 asta diesel 1396\n",
913
+ " 0.999 hyundai i20 asta 1.4 diesel 1396\n",
914
+ " 0.998 hyundai i20 1.4 asta diesel 1396\n",
915
+ " 0.998 hyundai i20 asta 1.4 crdi diesel 1396\n",
916
+ " 0.997 hyundai i20 asta 1.4 dsl diesel 1396\n",
917
+ " 0.997 hyundai i20 1.4 asta petrol 1396\n",
918
+ "\n",
919
+ "query: mahindra xuv500 w8 diesel 2179\n",
920
+ " 1.000 mahindra xuv500 w8 diesel 2179\n",
921
+ " 0.998 mahindra and mahindra xuv500 w8 diesel 2179\n",
922
+ " 0.998 mahindra xuv500 w7 diesel 2179\n",
923
+ " 0.998 mahindra xuv500 w8 at diesel 2179\n",
924
+ " 0.998 mahindra xuv500 w9 diesel 2179\n",
925
+ "\n",
926
+ "query: hyundai swift \n",
927
+ " 0.994 hyundai kona premium\n",
928
+ " 0.993 maruti suzuki swift 1.3 diesel 1248\n",
929
+ " 0.993 maruti swift sp petrol 1197\n",
930
+ " 0.993 maruti swift range extender diesel 1197\n",
931
+ " 0.993 maruti swift lxi petrol 1197\n"
932
+ ]
933
+ }
934
+ ],
935
+ "source": [
936
+ "corpus_embeddings = model.encode(\n",
937
+ " corpus,\n",
938
+ " batch_size=256,\n",
939
+ " convert_to_tensor=True,\n",
940
+ " normalize_embeddings=True,\n",
941
+ " show_progress_bar=True,\n",
942
+ ")\n",
943
+ "\n",
944
+ "\n",
945
+ "def nearest(query, k=5):\n",
946
+ " q = model.encode([query], convert_to_tensor=True, normalize_embeddings=True)\n",
947
+ " hits = util.semantic_search(q, corpus_embeddings, top_k=k)[0]\n",
948
+ " print(f\"\\nquery: {query}\")\n",
949
+ " for h in hits:\n",
950
+ " print(f\" {h['score']:.3f} {corpus[h['corpus_id']]}\")\n",
951
+ "\n",
952
+ "\n",
953
+ "for q in [\n",
954
+ " \"maruti swift vxi petrol 1197\",\n",
955
+ " \"hyundai i20 asta diesel 1396\",\n",
956
+ " \"mahindra xuv500 w8 diesel 2179\",\n",
957
+ " \"mahindra and mahindra land cruiser 4500\",\n",
958
+ " \n",
959
+ "]:\n",
960
+ " nearest(q)"
961
+ ]
962
+ },
963
+ {
964
+ "cell_type": "code",
965
+ "execution_count": 32,
966
+ "id": "push",
967
+ "metadata": {},
968
+ "outputs": [
969
+ {
970
+ "name": "stderr",
971
+ "output_type": "stream",
972
+ "text": [
973
+ "Writing model shards: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:00<00:00, 2.49it/s]\n"
974
+ ]
975
+ },
976
+ {
977
+ "data": {
978
+ "text/plain": [
979
+ "'https://huggingface.co/SwarajSolanke-turtle/bert-base-uncased-tsdae-fw/commit/6b9f55f41f80391366307318786e577072b80866'"
980
+ ]
981
+ },
982
+ "execution_count": 32,
983
+ "metadata": {},
984
+ "output_type": "execute_result"
985
+ }
986
+ ],
987
+ "source": [
988
+ "# Optional: push to the Hugging Face Hub (run `hf auth login` first).\n",
989
+ "repo_id = f\"{MODEL_NAME.split('/')[-1]}-tsdae-fw\"\n",
990
+ "model.push_to_hub(repo_id)"
991
+ ]
992
+ }
993
+ ],
994
+ "metadata": {
995
+ "kernelspec": {
996
+ "display_name": "Motor_Mapping (3.12.0)",
997
+ "language": "python",
998
+ "name": "python3"
999
+ },
1000
+ "language_info": {
1001
+ "codemirror_mode": {
1002
+ "name": "ipython",
1003
+ "version": 3
1004
+ },
1005
+ "file_extension": ".py",
1006
+ "mimetype": "text/x-python",
1007
+ "name": "python",
1008
+ "nbconvert_exporter": "python",
1009
+ "pygments_lexer": "ipython3",
1010
+ "version": "3.12.0"
1011
+ }
1012
+ },
1013
+ "nbformat": 4,
1014
+ "nbformat_minor": 5
1015
+ }