{ "cells": [ { "cell_type": "markdown", "id": "intro", "metadata": {}, "source": [ "# TSDAE fine-tuning on the FW vehicle master\n", "\n", "Domain-adapts a sentence encoder to insurer vehicle strings built from `Make`, `Model`, `Variant`, `Fuel`, `CC`\n", "out of `data/FW_cleaned_data_new_base_15.csv`.\n", "\n", "Same TSDAE recipe as `TSDAE_script.ipynb`, with two substitutions:\n", "\n", "| | `TSDAE_script.ipynb` | this notebook |\n", "|---|---|---|\n", "| corpus | `wiki1m-for-simcse` | vehicle strings from the base-15 CSV |\n", "| evaluator | STS-B (generic semantic similarity) | triplets of same-vehicle vs different-vehicle strings |\n", "\n", "STS-B is dropped because it scores general English sentence similarity and says nothing about whether\n", "`maruti swift vxi petrol 1197` lands near its sibling variants. The triplet evaluator below measures exactly that,\n", "which is the property the downstream mapping step depends on." ] }, { "cell_type": "code", "execution_count": 2, "id": "install", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/home/ubuntu/Motor_Mapping/.venv/bin/python: No module named uv\n", "Note: you may need to restart the kernel to use updated packages.\n" ] } ], "source": [ "# This venv has only pandas/numpy. Install the training stack once (~3 GB with CUDA torch):\n", "%uv pip install \"sentence-transformers>=3.0\" nltk accelerate" ] }, { "cell_type": "code", "execution_count": 18, "id": "imports", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/ubuntu/Motor_Mapping/.venv/lib/python3.12/site-packages/nltk/downloader.py:1076: UserWarning: NLTK will not authorize the non-private download directory '/home/ubuntu/nltk_data': it (or an ancestor) is world- or group-writable, so another local user could plant files there. Choose a private location such as ~/nltk_data.\n", " for msg in self.incr_download(info_or_id, download_dir, force):\n" ] }, { "data": { "text/plain": [ "True" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import random\n", "from collections import defaultdict\n", "\n", "import nltk\n", "import pandas as pd\n", "from datasets import Dataset\n", "from nltk import word_tokenize\n", "from nltk.tokenize.treebank import TreebankWordDetokenizer\n", "from sentence_transformers import (\n", " SentenceTransformer,\n", " SentenceTransformerTrainer,\n", " SentenceTransformerTrainingArguments,\n", " losses,\n", " models,\n", " util,\n", ")\n", "# from sentence_transformers.evaluation import TripletEvaluator\n", "from sentence_transformers.sentence_transformer.evaluation import TripletEvaluator\n", "nltk.download(\"punkt_tab\", quiet=True)" ] }, { "cell_type": "markdown", "id": "config-md", "metadata": {}, "source": [ "## Config\n", "\n", "Two values differ from the wiki1m script because vehicle strings are short (~6 tokens, not ~25):\n", "\n", "- `MAX_SEQ_LENGTH = 32` instead of 75 — no string comes close to 75 tokens, so the extra padding is wasted compute.\n", "- `DEL_RATIO = 0.4` instead of 0.6 — deleting 60% of a 6-token string leaves 2 tokens, which usually destroys the\n", " make/model identity and gives the decoder nothing to reconstruct from. 0.4 keeps ~4 tokens." ] }, { "cell_type": "code", "execution_count": 19, "id": "config", "metadata": {}, "outputs": [], "source": [ "CSV_PATH = \"/home/ubuntu/Motor_Mapping/MotorMapping/cleaning_data/data/FW_cleaned_data_new_base_15.csv\"\n", "OUTPUT_DIR = \"/home/ubuntu/Motor_Mapping/MotorMapping/cleaning_data/models/tsdae-fw\"\n", "\n", "MODEL_NAME = \"google-bert/bert-base-uncased\"\n", "MAX_SEQ_LENGTH = 32\n", "TRAIN_BATCH_SIZE = 64\n", "NUM_EPOCHS = 10\n", "LEARNING_RATE = 1e-5 # 3e-5\n", "DEL_RATIO = 0.3\n", "TEST_SIZE = 0.02\n", "N_EVAL_TRIPLETS = 2000\n", "SEED = 46\n", "\n", "random.seed(SEED)" ] }, { "cell_type": "markdown", "id": "corpus-md", "metadata": {}, "source": [ "## Build the corpus\n", "\n", "One string per row: `make model variant fuel cc`, lowercased, with `CC` as an integer and empty/zero fields dropped.\n", "Deduplicated, because the 153k rows contain the same vehicle string many times over (once per insurer) and TSDAE\n", "gains nothing from seeing an identical string repeatedly." ] }, { "cell_type": "code", "execution_count": 20, "id": "corpus", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "rows: 153,738 unique strings: 88,215\n", "\n", " ac six vintage six royal roadstar petrol 1995\n", " alfa romeo 156 gta std petrol 3179\n", " alfa romeo alfa romeo alfa 159 3.2 jts v6 ti petrol 3195\n", " alfa romeo giulia sprint 1600 gta petrol 1570\n", " alfa romeo vintage alfa romeo petrol 1110\n", " alfa romeo vintage alfa romeo petrol 1570\n", " alfa romeo vintage alfa romeo petrol 1750\n", " alfa romeo vintage romeo petrol 1100\n", "\n", "count 5000.000000\n", "mean 7.820600\n", "std 1.801961\n", "min 3.000000\n", "25% 7.000000\n", "50% 8.000000\n", "75% 9.000000\n", "max 17.000000\n", "dtype: float64\n" ] } ], "source": [ "FEATURE_COLS = [\"Make\", \"Model\", \"Variant\", \"Fuel\", \"CC\"]\n", "GROUP_COLS = [\"MAKE\", \"NEW_MODELS\"]\n", "NUMERIC_COLS = {\"CC\", \"NEW_CC\", \"SC\"}\n", "\n", "df = pd.read_csv(CSV_PATH, usecols=FEATURE_COLS + GROUP_COLS, low_memory=False)\n", "\n", "\n", "def build_text(frame, cols=FEATURE_COLS):\n", " parts = []\n", " for col in cols:\n", " if col in NUMERIC_COLS:\n", " s = pd.to_numeric(frame[col], errors=\"coerce\").fillna(0).astype(int).astype(str)\n", " s = s.replace(\"0\", \"\")\n", " else:\n", " s = frame[col].fillna(\"\").astype(str).str.lower().str.strip()\n", " parts.append(s)\n", " text = parts[0]\n", " for s in parts[1:]:\n", " text = text.str.cat(s, sep=\" \")\n", " return text.str.replace(r\"\\s+\", \" \", regex=True).str.strip()\n", "\n", "\n", "df[\"text\"] = build_text(df)\n", "corpus = sorted({t for t in df[\"text\"] if t})\n", "\n", "print(f\"rows: {len(df):,} unique strings: {len(corpus):,}\")\n", "print()\n", "for t in corpus[:8]:\n", " print(\" \", t)\n", "print()\n", "print(pd.Series([len(word_tokenize(t)) for t in corpus[:5000]]).describe())" ] }, { "cell_type": "markdown", "id": "noise-md", "metadata": {}, "source": [ "## Add deletion noise\n", "\n", "TSDAE's objective: reconstruct the original string from a corrupted copy, forcing the pooled `[CLS]` vector to carry\n", "the whole meaning. `delete_noise` keeps each token with probability `1 - DEL_RATIO`, and always keeps at least one\n", "token — on strings this short an unguarded sampling run can delete everything, and an empty input teaches nothing.\n", "\n", "`DenoisingAutoEncoderLoss` reads the columns positionally as `(damaged, original)`, so `select_columns` pins that order." ] }, { "cell_type": "code", "execution_count": 21, "id": "noise", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Map: 100%|██████████| 88215/88215 [00:05<00:00, 15160.07 examples/s]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Dataset({\n", " features: ['damaged', 'original'],\n", " num_rows: 86450\n", "})\n", " '1.5 petrol 1482' <- 'hyundai alcazar 1.5 executive mt petrol 1482'\n", " 'mahindra. 300 1.5 w8 tone diesel 1497' <- 'mahindra. xuv 300 1.5 w8 dual tone diesel 1497'\n", " 'mahindra scorpio vlx mhawk bs iv airbag diesel 2179' <- 'mahindra scorpio vlx 2.2 mhawk bs iv airbag diesel 2179'\n", " 'kia carens clavis o bsvi diesel' <- 'kia carens clavis htk plus (o) bsvi diesel 1493'\n", " 'nissan kicks 1.3 pre (o) petrol 1298' <- 'nissan motors kicks 1.3 turbo xv pre (o) bsvi petrol 1298'\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\n" ] } ], "source": [ "detokenizer = TreebankWordDetokenizer()\n", "\n", "\n", "def delete_noise(text, del_ratio=DEL_RATIO):\n", " words = word_tokenize(text)\n", " if not words:\n", " return text\n", " kept = [w for w in words if random.random() >= del_ratio]\n", " if not kept:\n", " kept = [random.choice(words)]\n", " return detokenizer.detokenize(kept)\n", "\n", "\n", "def noise_batch(batch):\n", " return {\"damaged\": [delete_noise(t) for t in batch[\"original\"]]}\n", "\n", "\n", "dataset = Dataset.from_dict({\"original\": corpus})\n", "dataset = dataset.map(noise_batch, batched=True, batch_size=2000)\n", "dataset = dataset.select_columns([\"damaged\", \"original\"])\n", "\n", "split = dataset.train_test_split(test_size=TEST_SIZE, seed=SEED)\n", "train_dataset, eval_dataset = split[\"train\"], split[\"test\"]\n", "\n", "print(train_dataset)\n", "for i in range(5):\n", " row = train_dataset[i]\n", " print(f\" {row['damaged']!r:<45} <- {row['original']!r}\")" ] }, { "cell_type": "markdown", "id": "model-md", "metadata": {}, "source": [ "## Model and loss\n", "\n", "CLS pooling and a tied encoder/decoder, per the TSDAE paper. `decoder_name_or_path` takes the model *name*, not the\n", "`SentenceTransformer` object." ] }, { "cell_type": "code", "execution_count": 22, "id": "model", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Loading weights: 100%|██████████| 199/199 [00:00<00:00, 3305.71it/s]\n", "[transformers] \u001b[1mBertModel LOAD REPORT\u001b[0m from: google-bert/bert-base-uncased\n", "Key | Status | | \n", "-------------------------------------------+------------+--+-\n", "cls.seq_relationship.bias | UNEXPECTED | | \n", "cls.predictions.transform.LayerNorm.bias | UNEXPECTED | | \n", "cls.predictions.transform.dense.weight | UNEXPECTED | | \n", "cls.predictions.transform.dense.bias | UNEXPECTED | | \n", "cls.seq_relationship.weight | UNEXPECTED | | \n", "cls.predictions.transform.LayerNorm.weight | UNEXPECTED | | \n", "cls.predictions.bias | UNEXPECTED | | \n", "\n", "Notes:\n", "- UNEXPECTED:\tcan be ignored when loading from different task/architecture; not ok if you expect identical arch.\n", "/tmp/ipykernel_117871/364354980.py:2: FutureWarning: The `get_word_embedding_dimension` method has been renamed to `get_embedding_dimension`.\n", " pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension(), \"cls\")\n", "When tie_encoder_decoder=True, the decoder_name_or_path will be invalid.\n", "Loading weights: 100%|██████████| 202/202 [00:00<00:00, 3428.92it/s]\n", "[transformers] \u001b[1mBertLMHeadModel LOAD REPORT\u001b[0m from: google-bert/bert-base-uncased\n", "Key | Status | \n", "-------------------------------------------------------------------+------------+-\n", "bert.pooler.dense.bias | UNEXPECTED | \n", "bert.pooler.dense.weight | UNEXPECTED | \n", "cls.seq_relationship.bias | UNEXPECTED | \n", "cls.seq_relationship.weight | UNEXPECTED | \n", "bert.encoder.layer.{0...11}.crossattention.self.key.bias | MISSING | \n", "bert.encoder.layer.{0...11}.crossattention.output.LayerNorm.bias | MISSING | \n", "bert.encoder.layer.{0...11}.crossattention.output.LayerNorm.weight | MISSING | \n", "bert.encoder.layer.{0...11}.crossattention.self.value.weight | MISSING | \n", "bert.encoder.layer.{0...11}.crossattention.self.query.weight | MISSING | \n", "bert.encoder.layer.{0...11}.crossattention.output.dense.bias | MISSING | \n", "bert.encoder.layer.{0...11}.crossattention.self.query.bias | MISSING | \n", "bert.encoder.layer.{0...11}.crossattention.output.dense.weight | MISSING | \n", "bert.encoder.layer.{0...11}.crossattention.self.value.bias | MISSING | \n", "bert.encoder.layer.{0...11}.crossattention.self.key.weight | MISSING | \n", "\n", "Notes:\n", "- UNEXPECTED:\tcan be ignored when loading from different task/architecture; not ok if you expect identical arch.\n", "- MISSING:\tthose params were newly initialized because missing from the checkpoint. Consider training on your downstream task.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "SentenceTransformer(\n", " (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})\n", " (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'cls', 'include_prompt': True})\n", ")\n" ] } ], "source": [ "word_embedding_model = models.Transformer(MODEL_NAME, max_seq_length=MAX_SEQ_LENGTH)\n", "pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension(), \"cls\")\n", "model = SentenceTransformer(modules=[word_embedding_model, pooling_model])\n", "\n", "train_loss = losses.DenoisingAutoEncoderLoss(\n", " model,\n", " decoder_name_or_path=MODEL_NAME,\n", " tie_encoder_decoder=True,\n", ")\n", "\n", "print(model)" ] }, { "cell_type": "markdown", "id": "eval-md", "metadata": {}, "source": [ "## Domain evaluator\n", "\n", "Triplets are derived from columns the CSV already has, so no hand labelling is needed:\n", "\n", "- **group** = `MAKE` + `NEW_MODELS` + `Fuel` — the canonical vehicle, from the cleaned columns.\n", "- **anchor / positive** = two different raw strings in the same group (same vehicle, different insurer wording).\n", "- **negative** = a string from a *different* group under the *same* make — a hard negative, since `maruti swift` and\n", " `maruti dzire` share tokens and are the pairs the mapper actually confuses.\n", "\n", "The reported accuracy is the fraction of triplets where the anchor sits closer to its positive than to its negative.\n", "Note this is measured on strings the unsupervised training also sees; the grouping is never shown to the model, so it\n", "scores embedding geometry rather than generalization to unseen strings." ] }, { "cell_type": "code", "execution_count": 23, "id": "evaluator", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "groups :0 premier | padmini | petrol\n", "1 mahindra | economy | diesel\n", "2 mahindra | forester | diesel\n", "3 toyota | land cruiser | diesel\n", "4 mahindra | forester | diesel\n", " ... \n", "153733 citroen | c3 aircross | petrol\n", "153734 mahindra | scorpio | diesel\n", "153735 tata | safari | diesel\n", "153736 bmw | xm | petrol\n", "153737 tesla | model y | electric\n", "Length: 153738, dtype: str\n", "grouped:group\n", "ac | vintage | petrol [ac six vintage six royal roadstar petrol 1995]\n", "alfa | 156 gta | petrol [alfa romeo 156 gta std petrol 3179]\n", "alfa | alfa romeo | petrol [alfa romeo alfa romeo alfa 159 3.2 jts v6 ti ...\n", "alfa | giulia sprint | petrol [alfa romeo giulia sprint 1600 gta petrol 1570]\n", "alfa | vintage | petrol [alfa romeo vintage alfa romeo petrol 1570, al...\n", " ... \n", "wolseley | 16/20 | petrol [wolseley 16/20 saloon petrol 1140]\n", "wolseley | vintage | petrol [wolseley vintage 12 hp petrol 1260, wolseley ...\n", "wuling | almaz rs | hybrid [wuling almaz rs 2.0 pro hybrid hybrid 1999]\n", "yamaha | yzf r15 | petrol [yamaha yzf r15 yzf r15 bs-iv petrol 155]\n", "zotye | rx 6400 | diesel [zotye auto rx 6400 1.3 l diesel 1527]\n", "Name: text, Length: 3671, dtype: object\n", "dev_evaluator :\n", "groups with >=2 strings: 2,392 triplets: 2,000\n", "\n", "anchor: standard herald vintage station wagon petrol 1100\n", "positive: standard herald vintage vintage petrol 800\n", "negative: standard standard super 10 petrol 1340\n", "\n", "anchor: mahindra reva electric vehicles e 20 t 2 48v electric 1\n", "positive: mahindra reva e 20 t 2 electric 1\n", "negative: mahindra electric mobility limited treo treo sft electric 1\n", "\n", "anchor: audi a5 2.0 tdi cabriolet diesel 1968\n", "positive: audi a5 cabriolet 2.0 tdi diesel 1984\n", "negative: audi rs q8 4.0 tfsi quattro petrol 3998\n", "\n" ] } ], "source": [ "group_key = (\n", " df[\"MAKE\"].fillna(\"\").astype(str).str.lower().str.strip()\n", " + \" | \"\n", " + df[\"NEW_MODELS\"].fillna(\"\").astype(str).str.lower().str.strip()\n", " + \" | \"\n", " + df[\"Fuel\"].fillna(\"\").astype(str).str.lower().str.strip()\n", ")\n", "print(f\"groups :{group_key}\")\n", "\n", "df[\"group\"] = group_key\n", "\n", "grouped = df[df[\"text\"] != \"\"].groupby(\"group\")[\"text\"].apply(lambda s: list(dict.fromkeys(s)))\n", "print(f\"grouped:{grouped}\")\n", "grouped = grouped[grouped.map(len) >= 2]\n", "\n", "by_make = defaultdict(list)\n", "for g in grouped.index:\n", " by_make[g.split(\" | \")[0]].append(g)\n", "\n", "rng = random.Random(SEED)\n", "group_names = list(grouped.index)\n", "rng.shuffle(group_names)\n", "\n", "anchors, positives, negatives = [], [], []\n", "for g in group_names:\n", " if len(anchors) >= N_EVAL_TRIPLETS:\n", " break\n", " a, p = rng.sample(grouped[g], 2)\n", " same_make = [x for x in by_make[g.split(\" | \")[0]] if x != g]\n", " pool = same_make or [x for x in group_names if x != g]\n", " anchors.append(a)\n", " positives.append(p)\n", " negatives.append(rng.choice(grouped[rng.choice(pool)]))\n", "\n", "dev_evaluator = TripletEvaluator(\n", " anchors=anchors,\n", " positives=positives,\n", " negatives=negatives,\n", " name=\"fw-variant-dev\",\n", ")\n", "\n", "\n", "print(f\"dev_evaluator :{dev_evaluator}\")\n", "\n", "print(f\"groups with >=2 strings: {len(grouped):,} triplets: {len(anchors):,}\\n\")\n", "for i in range(3):\n", " print(f\"anchor: {anchors[i]}\")\n", " print(f\"positive: {positives[i]}\")\n", " print(f\"negative: {negatives[i]}\\n\")" ] }, { "cell_type": "code", "execution_count": 24, "id": "baseline", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'fw-variant-dev_cosine_accuracy': 0.8845000267028809}\n" ] } ], "source": [ "# Baseline before any training — compare against the post-training number below.\n", "print(dev_evaluator(model))" ] }, { "cell_type": "code", "execution_count": 25, "id": "args", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "The `warmup_ratio` argument is deprecated in Transformers v5+, and will also be removed from Sentence Transformers once support for Transformers v4 is dropped. Since you're using Transformers v5+, please use `warmup_steps` (as a float) to specify the warmup ratio instead.\n" ] } ], "source": [ "args = SentenceTransformerTrainingArguments(\n", " output_dir=OUTPUT_DIR,\n", " num_train_epochs=NUM_EPOCHS,\n", " per_device_train_batch_size=TRAIN_BATCH_SIZE,\n", " per_device_eval_batch_size=TRAIN_BATCH_SIZE,\n", " learning_rate=LEARNING_RATE,\n", " weight_decay=0.0,\n", " warmup_ratio=0.1,\n", " bf16=True,\n", " fp16=False,\n", " eval_strategy=\"steps\",\n", " eval_steps=500,\n", " save_steps=1000,\n", " save_total_limit=2,\n", " logging_steps=50,\n", " seed=SEED,\n", " run_name=\"tsdae_new-fw\",\n", ")" ] }, { "cell_type": "code", "execution_count": 26, "id": "trainer", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ " \r" ] } ], "source": [ "trainer = SentenceTransformerTrainer(\n", " model=model,\n", " args=args,\n", " train_dataset=train_dataset,\n", " eval_dataset=eval_dataset,\n", " loss=train_loss,\n", " evaluator=dev_evaluator,\n", ")" ] }, { "cell_type": "code", "execution_count": 27, "id": "train", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "
\n", " \n", " \n", " [13510/13510 38:36, Epoch 10/10]\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
StepTraining LossValidation LossFw-variant-dev Cosine Accuracy
5003.5196593.3155040.888500
10001.9063711.8351820.968000
15001.2646651.2372390.967000
20001.0351540.9792820.968000
25000.9044630.8559520.963500
30000.7666600.7721520.964500
35000.7371150.7234600.965000
40000.6973860.6874820.966000
45000.6226510.6573690.964000
50000.6178420.6357640.964500
55000.5612970.6063270.965000
60000.5456780.5993270.967500
65000.5428600.5765580.964500
70000.4913510.5701220.965500
75000.4782190.5646130.964000
80000.4814210.5484680.967500
85000.4526660.5425190.965500
90000.4472520.5347220.965500
95000.4161570.5263710.967000
100000.4196360.5216640.966500
105000.4216460.5197550.967000
110000.3794810.5113430.966500
115000.3779980.5073820.967000
120000.3715540.5050130.965500
125000.3648930.5002490.966000
130000.3623210.4984310.966500
135000.3704350.4972990.965500
135100.3704350.4973600.966000

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "Writing model shards: 100%|██████████| 1/1 [00:00<00:00, 1.71it/s]\n", "Writing model shards: 100%|██████████| 1/1 [00:00<00:00, 2.25it/s]\n", "Writing model shards: 100%|██████████| 1/1 [00:00<00:00, 2.10it/s]\n", "Writing model shards: 100%|██████████| 1/1 [00:00<00:00, 2.10it/s]\n", "Writing model shards: 100%|██████████| 1/1 [00:00<00:00, 2.17it/s]\n", "Writing model shards: 100%|██████████| 1/1 [00:00<00:00, 2.20it/s]\n", "Writing model shards: 100%|██████████| 1/1 [00:00<00:00, 2.18it/s]\n", "Writing model shards: 100%|██████████| 1/1 [00:00<00:00, 2.15it/s]\n", "Writing model shards: 100%|██████████| 1/1 [00:00<00:00, 2.17it/s]\n", "Writing model shards: 100%|██████████| 1/1 [00:00<00:00, 2.17it/s]\n", "Writing model shards: 100%|██████████| 1/1 [00:00<00:00, 2.19it/s]\n", "Writing model shards: 100%|██████████| 1/1 [00:00<00:00, 2.20it/s]\n", "Writing model shards: 100%|██████████| 1/1 [00:00<00:00, 2.13it/s]\n", "Writing model shards: 100%|██████████| 1/1 [00:00<00:00, 2.20it/s]\n" ] }, { "data": { "text/plain": [ "TrainOutput(global_step=13510, training_loss=0.860432569968268, metrics={'train_runtime': 2316.8877, 'train_samples_per_second': 373.13, 'train_steps_per_second': 5.831, 'total_flos': 0.0, 'train_loss': 0.860432569968268, 'epoch': 10.0})" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "trainer.train()" ] }, { "cell_type": "code", "execution_count": 28, "id": "post-eval", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'fw-variant-dev_cosine_accuracy': 0.9660000205039978}\n" ] } ], "source": [ "print(dev_evaluator(model))" ] }, { "cell_type": "code", "execution_count": 29, "id": "save", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Writing model shards: 0%| | 0/1 [00:00