Model card (draft for review)

README.md

@@ -0,0 +1,131 @@
+---
+license: apache-2.0
+base_model: fastino/gliner2.5-base-v1
+library_name: gliner2
+pipeline_tag: text-classification
+language:
+  - en
+tags:
+  - gliner2
+  - text-classification
+  - multi-label
+  - hub-metadata
+  - hf-jobs
+---
+
+# Hub task tagger (GLiNER2.5-base)
+
+Suggests task tags for a Hugging Face dataset, such as `text-generation`, `robotics` or
+`automatic-speech-recognition`, from its column names and first row. It picks from the 52 task
+tags the Hub offers, and gives a probability for each.
+
+[TypeSafe AI's Jev](https://docs.typesafe.ai/introduction) has got people interested in
+"System One" models: models that don't generate text, but read a state and return typed answers
+with probabilities. This is a small, open example of the same idea for one narrow job, fine-tuned
+in 17 minutes on [Hugging Face Jobs](https://huggingface.co/docs/hub/jobs).
+
+**Try it:** [davanstrien/hub-task-tagger](https://huggingface.co/spaces/davanstrien/hub-task-tagger).
+Paste a dataset id and see the suggested tags, the owner's tags, and the exact text the model read.
+
+## How to use
+
+The model reads a short text built from the dataset viewer's preview: a `Columns:` line with each
+column's name and type, then the first row, cut to 370 tokens. It must be built exactly as in
+training, so this repo ships the builder,
+[`tagger_core.py`](https://huggingface.co/davanstrien/hub-task-tagger-gliner2.5-base/blob/main/tagger_core.py),
+next to the weights. The snippet below downloads it with `huggingface_hub` and uses it.
+
+```python
+import json, math, os, sys
+from huggingface_hub import hf_hub_download
+from tokenizers import Tokenizer
+from gliner2.classification import Classifier, ClassificationSchema, ClassificationConfig
+
+repo = "davanstrien/hub-task-tagger-gliner2.5-base"
+
+# 1. Build the model's input for a dataset, with the code that built the training data.
+path = hf_hub_download(repo, "tagger_core.py")
+sys.path.insert(0, os.path.dirname(path))  # make the downloaded file importable
+import tagger_core as core
+
+builder = core.StateBuilder(Tokenizer.from_file(hf_hub_download("Qwen/Qwen3.5-4B-Base", "tokenizer.json")), 370)
+
+first_rows, config, split, error = core.fetch_first_rows("fancyzhx/ag_news")
+state, truncated = builder.build(core.render_first_rows(first_rows))
+print(state)
+
+# 2. Score all 52 tags.
+labels = json.load(open(hf_hub_download(repo, "classification_schema.json")))["tasks"][0]["labels"]
+model = Classifier.from_pretrained(repo).eval()
+schema = ClassificationSchema().multi("labels", labels)
+logits = model.batch_score([state[:2000]], schema, config=ClassificationConfig(batch_size=1))[0].tasks["labels"]
+
+# 3. Calibrated probabilities: softmax over the 52 logits at temperature 1.30 (fitted on held-out data).
+z = [logits[label] / 1.30 for label in labels]
+m = max(z)
+e = [math.exp(v - m) for v in z]
+probs = sorted(zip(labels, (v / sum(e) for v in e)), key=lambda x: -x[1])
+print(probs[:3])
+
+# A set of tags: everything at 0.225 or above, and always the top tag.
+suggested = [label for label, p in probs if p >= 0.225] or [probs[0][0]]
+print(suggested)
+```
+
+The Qwen tokenizer is only used to measure and cut the input to the length the model was trained
+on. `fetch_first_rows` returns an error instead of rows when a dataset has no viewer preview.
+
+Install with `pip install "gliner2[local]==2.0.0"`. Weights are fp32. On a free 2-vCPU Space one
+prediction takes about 0.7–1 s; on an L4 GPU in fp16, about 20 ms.
+
+## Training
+
+- **Recipe:** [`train-gliner2.py`](https://huggingface.co/datasets/uv-scripts/classification) from
+  `uv-scripts/classification`, run as one `hf jobs uv run` command on an `rtx-pro-6000`:
+  `--base-model fastino/gliner2.5-base-v1 --labels-file labels.json --label-column labels --label-augmentation off`,
+  5 epochs, seed 0, bf16. About 17 minutes of training, roughly $1.50.
+- **Data:** 16,000 public Hub datasets with owner-declared `task_categories`. Each example is the
+  dataset's column names and first row (from the dataset viewer), cut to about 370 tokens, and its
+  declared tags (multi-label). Retired tags were mapped to current ones where the task is the same
+  (for example `text2text-generation` → `text-generation`); others were dropped.
+- **Split by time and owner:** the evaluation sets are 1,000 (calibration) and 3,000 (development)
+  datasets created in the 60 days before 2026-09-22, by owners who have no dataset in the training
+  data. Near-duplicate datasets from the same owner cannot leak between training and evaluation.
+
+## Evaluation
+
+On the 3,000 development datasets. A prediction counts as correct if it is one of the owner's tags.
+
+| | Top-1 | Top-3 | Macro recall (24 tags) |
+|---|---|---|---|
+| This model | **0.695** | 0.881 | 0.525 |
+| Same recipe, second seed | 0.686 | 0.879 | 0.511 |
+| GLiNER2.5-small, same recipe | 0.653 | 0.855 | 0.458 |
+| GLiNER2.5-base, zero-shot | 0.102 | 0.213 | 0.099 |
+| Always "text-generation" | 0.320 | – | – |
+
+- **The owners' tags are a noisy target.** In a hand-checked sample, about 1 dataset in 10 was
+  missing a tag that fits. Counting 74 card-checked extra tags as correct raises top-1 to 0.720.
+- **As a set** (all tags at probability ≥ 0.225), micro-F1 against the owner's tags is 0.623,
+  with 1.21 tags per dataset on average against 1.32 declared.
+- **Confidence:** the temperature and the threshold were fitted on the calibration set. Thresholds
+  chosen there held on development within a few points of error, but not exactly: treat the
+  probabilities as a good ranking, not as exact rates.
+
+## Limitations
+
+- **It only sees the first row.** Datasets whose purpose is not visible from the columns and one
+  example (for example, generic tables) are hard.
+- **Tabular tags are weak.** It often confuses `tabular-regression` with `tabular-classification`,
+  because the input does not say which column is the target.
+- **Rare tags** (17 of the 52, such as `text-to-3d` or `voice-activity-detection`) had too few
+  training examples to measure.
+- **It learned from owners' choices,** including their habits: for example, question-answering
+  datasets tagged `text-generation`, or the reverse.
+- Trained and evaluated mostly on English-language metadata.
+
+## Credits
+
+Built on [GLiNER2](https://github.com/fastino-ai/GLiNER2) by Fastino. Training, evaluation and the
+demo were run on Hugging Face Jobs and Spaces, with much of the work done by coding agents (Claude)
+and checked by a person. More detail: [GLINER2-NOTES.md](https://huggingface.co/datasets/uv-scripts/classification/blob/main/GLINER2-NOTES.md).

tagger_core.py

@@ -0,0 +1,200 @@
+"""Rendering and state building, copied verbatim from the Gradio test Space (demo-space/app.py).
+
+That code is itself copied from build_hub_task_rows.py and prepare_hub_tasks.py, and it matched the stored
+training states 200/200. Do not edit the functions below; change the training scripts and re-copy instead.
+"""
+import json
+import time
+
+import requests
+
+# ---- rendering: copied from build_hub_task_rows.py ------------------------------------------------------------
+
+SERVER = "https://datasets-server.huggingface.co"
+TAGS_URL = "https://huggingface.co/api/datasets-tags-by-type?type=task_categories"
+RETIRED = {
+    "text2text-generation": "text-generation",
+    "conversational": "text-generation",
+    "language-modeling": "text-generation",
+    "sequence-modeling": "text-generation",
+    "document-question-answering": "visual-question-answering",
+    "multiple-choice-qa": "multiple-choice",
+    "topic-classification": "text-classification",
+}
+MEDIA = {"Image": "<image>", "Audio": "<audio>", "Video": "<video>", "Pdf": "<pdf>"}
+NUMERIC = ("int", "uint", "float", "double", "bool")
+CELL_CHARS = 200
+ROWS = 3
+MAX_PREVIEW_TRIES = 3
+
+# Anonymous on purpose: the Space shows only what any visitor could read (no gated or private previews).
+session = requests.Session()
+
+
+def get_json(url, params):
+    """(status, json). Retries 429/502/503/504 and network errors, honouring Retry-After; returns the last status."""
+    status = None
+    for attempt in range(5):
+        try:
+            r = session.get(url, params=params, timeout=60)
+        except requests.RequestException:
+            status = "network"
+            time.sleep(2 ** attempt)
+            continue
+        status = r.status_code
+        if status in (429, 502, 503, 504):
+            wait = r.headers.get("Retry-After", "")
+            time.sleep(min(60.0, float(wait)) if wait.replace(".", "", 1).isdigit() else 2 ** attempt)
+            continue
+        try:
+            return status, r.json()
+        except ValueError:
+            return status, {}
+    return status, {}
+
+
+def type_name(t):
+    """Short type for the Columns line: string, int64, Image, ClassLabel: a, b, list<float32>, struct."""
+    if isinstance(t, list):
+        return f"list<{type_name(t[0])}>" if t else "list"
+    if not isinstance(t, dict):
+        return "?"
+    kind = t.get("_type")
+    if kind == "Value":
+        return t.get("dtype", "?")
+    if kind == "ClassLabel":
+        names = [str(n) for n in (t.get("names") or [])]
+        if not names:
+            return "ClassLabel"
+        return "ClassLabel: " + ", ".join(names[:8]) + (", …" if len(names) > 8 else "")
+    if kind in ("List", "Sequence", "LargeList"):
+        return f"list<{type_name(t.get('feature'))}>"
+    if kind:
+        return kind
+    return "struct"
+
+
+def is_numeric(t):
+    return isinstance(t, dict) and t.get("_type") == "Value" and str(t.get("dtype", "")).startswith(NUMERIC)
+
+
+def render_value(v, t):
+    """A cell value as display text, following the feature type recursively."""
+    if v is None:
+        return "null"
+    if isinstance(t, list):
+        t = {"_type": "List", "feature": t[0] if t else None}
+    kind = t.get("_type") if isinstance(t, dict) else None
+    if kind in MEDIA:
+        return MEDIA[kind]
+    if kind == "ClassLabel":
+        names = t.get("names") or []
+        return str(names[v]) if isinstance(v, int) and 0 <= v < len(names) else str(v)
+    if kind in ("List", "Sequence", "LargeList"):
+        inner = t.get("feature")
+        if isinstance(v, dict) and isinstance(inner, dict) and not inner.get("_type"):
+            return "{" + ", ".join(f"{k}: {render_value(v.get(k), {'_type': 'List', 'feature': sub})}" for k, sub in inner.items()) + "}"
+        if not isinstance(v, list):
+            return json.dumps(v, ensure_ascii=False)
+        if is_numeric(inner) and len(v) > 4:
+            return f"[{len(v)} × {inner.get('dtype')}]"
+        shown = [render_value(x, inner) for x in v[:5]]
+        return "[" + ", ".join(shown) + (f", … ({len(v)} items)" if len(v) > 5 else "") + "]"
+    if kind is None and isinstance(t, dict) and isinstance(v, dict):
+        return "{" + ", ".join(f"{k}: {render_value(v.get(k), sub)}" for k, sub in t.items()) + "}"
+    if isinstance(v, str):
+        return v
+    return json.dumps(v, ensure_ascii=False)
+
+
+def cell_text(v, t, truncated):
+    s = " ".join(render_value(v, t).split())
+    if len(s) > CELL_CHARS:
+        return s[:CELL_CHARS] + "…"
+    return s + ("…" if truncated else "")
+
+
+def fetch_first_rows(dataset_id):
+    """(first_rows_json, config, split, error). The fetch half of build_hub_task_rows.preview()."""
+    status, sp = get_json(f"{SERVER}/splits", {"dataset": dataset_id})
+    splits = sp.get("splits") or []
+    if not splits:
+        return None, None, None, f"splits:{status} {sp.get('error', '')}".strip()
+    last = None
+    for s in splits[:MAX_PREVIEW_TRIES]:
+        cfg, split = s["config"], s["split"]
+        status, fr = get_json(f"{SERVER}/first-rows", {"dataset": dataset_id, "config": cfg, "split": split})
+        if not (fr.get("features") or []) or not (fr.get("rows") or []):
+            last = f"first-rows:{status} {fr.get('error', '')}".strip()
+            continue
+        return fr, cfg, split, None
+    return None, None, None, last
+
+
+def render_first_rows(fr):
+    """The render half of build_hub_task_rows.preview()."""
+    feats, rows = fr["features"], fr["rows"]
+    types = {f["name"]: f["type"] for f in feats}
+    lines = ["Columns: " + ", ".join(f"{f['name']} ({type_name(f['type'])})" for f in feats), "Example rows:"]
+    for i, r in enumerate(rows[:ROWS], 1):
+        cut = set(r.get("truncated_cells") or [])
+        lines.append(f"{i}. " + " | ".join(f"{k}: {cell_text(v, types.get(k), k in cut)}" for k, v in r["row"].items()))
+    return "\n".join(lines)
+
+
+# ---- state: copied from prepare_hub_tasks.py -----------------------------------------------------------------
+
+COLUMNS_SHARE = 0.45
+
+
+class StateBuilder:
+    def __init__(self, tokenizer, max_tokens):
+        self.tok, self.max = tokenizer, max_tokens
+
+    def n(self, s):
+        return len(self.tok.encode(s, add_special_tokens=False).ids)
+
+    def cut(self, s, k):
+        """First k tokens of s, with a trailing … when something was removed."""
+        ids = self.tok.encode(s, add_special_tokens=False).ids
+        if len(ids) <= k:
+            return s, False
+        return self.tok.decode(ids[: max(0, k - 1)]).rstrip() + "…", True
+
+    def build(self, text):
+        """(state, truncated): Columns line + first example row, each cell getting a fair share of the budget."""
+        lines = text.split("\n")
+        columns = lines[0]
+        first = next((line for line in lines[1:] if line.startswith("1. ")), "")
+        truncated = False
+        if self.n(columns) > int(self.max * COLUMNS_SHARE):
+            columns, truncated = self.cut(columns, int(self.max * COLUMNS_SHARE))
+        if first:
+            cells = first[3:].split(" | ")
+            keys = [c.split(": ", 1)[0] + ": " if ": " in c else "" for c in cells]
+            values = [c.split(": ", 1)[1] if ": " in c else c for c in cells]
+            overhead = self.n(f"{columns}\nExample row:\n1. ") + sum(self.n(k) for k in keys) + 2 * len(cells)
+            budget = self.max - overhead
+            sizes = [self.n(v) for v in values]
+            if budget < len(cells):
+                cells = [k.rstrip(": ") for k in keys]
+                truncated = True
+            elif sum(sizes) > budget:
+                lo, hi = 1, max(sizes)
+                while lo < hi:
+                    mid = (lo + hi + 1) // 2
+                    if sum(min(s, mid) for s in sizes) <= budget:
+                        lo = mid
+                    else:
+                        hi = mid - 1
+                cells = [k + self.cut(v, lo)[0] for k, v in zip(keys, values)]
+                truncated = True
+            state = f"{columns}\nExample row:\n1. " + " | ".join(cells)
+        else:
+            state = columns
+        if self.n(state) > self.max:
+            state, _ = self.cut(state, self.max)
+            truncated = True
+        return state, truncated
+
+