7-level sentiment classifier on gemma-3-270m

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

NOTICE

@@ -0,0 +1 @@
+Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms

README.md

@@ -0,0 +1,96 @@
+---
+license: gemma
+base_model: google/gemma-3-270m
+pipeline_tag: text-classification
+language:
+- en
+tags:
+- sentiment-analysis
+- text-classification
+- gemma3
+- synthetic-data
+metrics:
+- accuracy
+- f1
+---
+
+# Gemma 3 270M: 7-level sentiment classifier
+
+A fine-tune of [google/gemma-3-270m](https://huggingface.co/google/gemma-3-270m) (the base model, not the instruct version) with a classification head. It sorts a short piece of text (a sentence, heading, bullet point, or table row) into one of seven sentiment levels:
+
+| id | label |
+|---|---|
+| 0 | very positive |
+| 1 | positive |
+| 2 | somewhat positive |
+| 3 | neutral |
+| 4 | somewhat negative |
+| 5 | negative |
+| 6 | very negative |
+
+## Results
+
+On a held-out test set of 8,749 segments (from documents never seen in training):
+
+| Metric | Score |
+|---|---|
+| Exact label | **84.0%** |
+| Exact or one level off | **95.2%** |
+| Macro-F1 | 0.833 |
+
+| Label | F1 |
+|---|---|
+| very positive | 0.88 |
+| positive | 0.80 |
+| somewhat positive | 0.88 |
+| neutral | 0.90 |
+| somewhat negative | 0.70 |
+| negative | 0.79 |
+| very negative | 0.89 |
+
+Most errors are one level off. The hardest boundary is "somewhat negative" versus "negative".
+
+**Important:** the test set is synthetic and was labelled by the same process as the training data (see below). These scores measure agreement with that labelling. The model has not yet been evaluated on human-labelled, real-world text.
+
+## Usage
+
+```python
+from transformers import pipeline
+
+clf = pipeline("text-classification", model="dejanai/sentiment-classifier")
+clf("Delivery took three weeks and nobody answered my emails.")
+# [{'label': 'negative', 'score': 0.88}]
+```
+
+Classify one sentence or short segment at a time. For a whole document, split it into sentences, headings, bullet points, and table rows first, and classify each. Inputs are truncated at 128 tokens.
+
+## Training data
+
+About 83,000 labelled segments from 2,782 synthetic documents, made in three steps:
+
+1. **Writer:** Gemini 3.8 Flash wrote complete documents in 26 content formats (blog posts, product reviews, tweet threads, meeting notes, technical papers, press releases, support chats, YouTube and podcast transcripts, FAQ pages, and more). Topics were seeded mainly from products, services, brands, and invented people, with some general topics (parenting, politics, health, everyday life). People in the documents are fictional.
+2. **Segmenter:** a Python script split each document into its natural units, keeping structure: headings, bullet points, table rows, metadata lines (bylines, dates), code lines, and sentences.
+3. **Labeller:** a separate Gemini 3.8 Flash call labelled every segment, reading it in the context of the whole document.
+
+Most documents mix tones; a further set was written with one dominant tone, to bring every non-neutral label up to about 10,900 segments. Neutral has about 17,800. Overused names and phrases were replaced with varied alternatives.
+
+Splits are by document (80/10/10), so no document has segments in both training and test.
+
+## Training procedure
+
+- Base: `google/gemma-3-270m`, `Gemma3TextForSequenceClassification` head, 7 labels
+- Input embedding frozen; all other weights trained
+- AdamW, learning rate 2e-5 with one-cycle schedule over 1.5 epochs, batch size 32, max length 128, bf16 autocast
+- Cross-entropy weighted by inverse label frequency
+- Validation every 100 steps; early stopping with patience 5. Stopped at step 2,600; the released checkpoint (lowest validation loss) is from step 2,100.
+
+## Limitations
+
+- Trained and tested only on synthetic, model-written English text. Real reviews, social posts, and informal writing may score lower.
+- Labels come from one model (Gemini 3.8 Flash), so its judgement and biases about sentiment strength are built in.
+- Each segment is classified on its own. A sentence whose tone depends on what came before (for example "And yet…") may be misread.
+- Neighbouring levels, especially "somewhat negative" and "negative", are often confused.
+
+## License
+
+This model is a derivative of Gemma and is provided under and subject to the [Gemma Terms of Use](https://ai.google.dev/gemma/terms). Use must also follow the [Gemma Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy).

config.json

@@ -0,0 +1,80 @@
+{
+  "_sliding_window_pattern": 6,
+  "architectures": [
+    "Gemma3TextForSequenceClassification"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "attn_logit_softcapping": null,
+  "bos_token_id": 2,
+  "dtype": "bfloat16",
+  "eos_token_id": 1,
+  "final_logit_softcapping": null,
+  "head_dim": 256,
+  "hidden_activation": "gelu_pytorch_tanh",
+  "hidden_size": 640,
+  "id2label": {
+    "0": "very positive",
+    "1": "positive",
+    "2": "somewhat positive",
+    "3": "neutral",
+    "4": "somewhat negative",
+    "5": "negative",
+    "6": "very negative"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 2048,
+  "label2id": {
+    "very positive": 0,
+    "positive": 1,
+    "somewhat positive": 2,
+    "neutral": 3,
+    "somewhat negative": 4,
+    "negative": 5,
+    "very negative": 6
+  },
+  "layer_types": [
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 32768,
+  "model_type": "gemma3_text",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 18,
+  "num_key_value_heads": 1,
+  "pad_token_id": 0,
+  "query_pre_attn_scalar": 256,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "full_attention": {
+      "rope_theta": 1000000.0,
+      "rope_type": "default"
+    },
+    "sliding_attention": {
+      "rope_theta": 10000.0,
+      "rope_type": "default"
+    }
+  },
+  "sliding_window": 512,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.5.0",
+  "use_bidirectional_attention": false,
+  "use_cache": true,
+  "vocab_size": 262144
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f78047aff96b53e4675d97f3a0def8482a4ce7fb3d15df04cd7340a6a76c8ac4
+size 536232096

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ea32934d51349dfadb7bd02697c64fb740160bcc7a472c2aaca1f9d2df4768f7
+size 33384819

tokenizer_config.json

@@ -0,0 +1,24 @@
+{
+  "backend": "tokenizers",
+  "boi_token": "<start_of_image>",
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "eoi_token": "<end_of_image>",
+  "eos_token": "<eos>",
+  "image_token": "<image_soft_token>",
+  "is_local": false,
+  "mask_token": "<mask>",
+  "model_max_length": 1000000000000000019884624838656,
+  "model_specific_special_tokens": {
+    "boi_token": "<start_of_image>",
+    "eoi_token": "<end_of_image>",
+    "image_token": "<image_soft_token>"
+  },
+  "pad_token": "<pad>",
+  "padding_side": "left",
+  "sp_model_kwargs": null,
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "GemmaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}