davanstrien HF Staff Claude Opus 5.5 commited on
Commit
c20557f
·
verified ·
1 Parent(s): 043a903

Filter news out of historical newspapers; add topic focus and owner collapse; show delay per source

Browse files
Files changed (3) hide show
  1. README.md +7 -3
  2. app.py +23 -3
  3. index.html +48 -16
README.md CHANGED
@@ -16,6 +16,7 @@ Follow new Hugging Face Hub models, datasets and Daily Papers by topic, or brows
16
  ## What it does
17
 
18
  - **Live:** the newest items for the topics you pick, newest first, with a marker for what is new since your last visit.
 
19
  - **Browse:** pick one or more topics, a time range, sources and a search term. Sort by topic score, newest, likes or
20
  downloads. A summary shows the counts per source and what else those items are about.
21
  - **Confirm:** tick the topics that fit an item and press Confirm. After 3 confirmations for a topic, its threshold is
@@ -29,7 +30,8 @@ Follow new Hugging Face Hub models, datasets and Daily Papers by topic, or brows
29
  - The scores are stored in the bucket `davanstrien/hub-firehose` (private for now), mounted in this Space at `/data`.
30
  A background worker process in this Space (cpu-upgrade, CPU only, lower priority than the page) watches the Hub for
31
  new and updated models and datasets plus HF Daily Papers, checks whether the card changed, and scores changed items
32
- with GLiNER2.5-Decide. The page loads the scores into memory and picks up new files every 20 s.
 
33
  - The app is FastAPI served by Granian (Docker Space, one app process). It supervises exactly one ingest worker
34
  (a file lock enforces this). JSON API: `GET /api/catalogue`, `/api/status`, `/api/live`, `/api/events`,
35
  `/api/browse` (list params repeated, e.g. `?topics=robotics&topics=video`; `thresholds` is a JSON object;
@@ -41,7 +43,8 @@ Follow new Hugging Face Hub models, datasets and Daily Papers by topic, or brows
41
 
42
  - Every view has a shareable URL, e.g.
43
  `https://huggingface.co/spaces/davanstrien/topic-feed-lab?view=browse&topics=robotics&days=7&sources=dataset&sort=likes`
44
- (params: `view`, `topics` comma-separated, `items=all`, `days`, `sources`, `sort`, `q`, `how=closed`, `cardonly=1`).
 
45
  Your thresholds and confirmations stay in your browser and are not part of the link.
46
  - **Card only:** repos with a README but no data files (datasets) or weight files (models) yet are hidden by default;
47
  tick "card-only" to include them. The worker records this at scoring time (`has_files`) and re-checks a card-only
@@ -54,4 +57,5 @@ Follow new Hugging Face Hub models, datasets and Daily Papers by topic, or brows
54
  - Likes and downloads are a snapshot from scoring time.
55
  - Papers have dates, not times; "today" means the latest paper day.
56
  - Custom topics are not available yet (they need the model; planned for the worker).
57
- - Private test Space.
 
 
16
  ## What it does
17
 
18
  - **Live:** the newest items for the topics you pick, newest first, with a marker for what is new since your last visit.
19
+ Pick one topic to see it alone. When one owner has several items, the first is shown with a button for the rest.
20
  - **Browse:** pick one or more topics, a time range, sources and a search term. Sort by topic score, newest, likes or
21
  downloads. A summary shows the counts per source and what else those items are about.
22
  - **Confirm:** tick the topics that fit an item and press Confirm. After 3 confirmations for a topic, its threshold is
 
30
  - The scores are stored in the bucket `davanstrien/hub-firehose` (private for now), mounted in this Space at `/data`.
31
  A background worker process in this Space (cpu-upgrade, CPU only, lower priority than the page) watches the Hub for
32
  new and updated models and datasets plus HF Daily Papers, checks whether the card changed, and scores changed items
33
+ with GLiNER2.5-Decide. Models usually appear within a minute or two of a change; datasets are found by a slower
34
+ sweep and can take up to about half an hour. The page loads the scores into memory and picks up new files every 20 s.
35
  - The app is FastAPI served by Granian (Docker Space, one app process). It supervises exactly one ingest worker
36
  (a file lock enforces this). JSON API: `GET /api/catalogue`, `/api/status`, `/api/live`, `/api/events`,
37
  `/api/browse` (list params repeated, e.g. `?topics=robotics&topics=video`; `thresholds` is a JSON object;
 
43
 
44
  - Every view has a shareable URL, e.g.
45
  `https://huggingface.co/spaces/davanstrien/topic-feed-lab?view=browse&topics=robotics&days=7&sources=dataset&sort=likes`
46
+ (params: `view`, `topics` comma-separated, `focus` for one feed topic in Live, `items=all`, `days`, `sources`, `sort`,
47
+ `q`, `how=open`, `cardonly=1`).
48
  Your thresholds and confirmations stay in your browser and are not part of the link.
49
  - **Card only:** repos with a README but no data files (datasets) or weight files (models) yet are hidden by default;
50
  tick "card-only" to include them. The worker records this at scoring time (`has_files`) and re-checks a card-only
 
57
  - Likes and downloads are a snapshot from scoring time.
58
  - Papers have dates, not times; "today" means the latest paper day.
59
  - Custom topics are not available yet (they need the model; planned for the worker).
60
+ - The model gives "historical newspapers" high scores for general news. The topic counts only when an item's name,
61
+ card summary or tags have a historical marker, which misses items whose text has none and keeps some news with one.
app.py CHANGED
@@ -67,6 +67,13 @@ NARROW = {} # broad index -> [narrow indices]; a broad topic's effective score
67
  for b, n in CAT["pairs"]:
68
  NARROW.setdefault(L[b], []).append(L[n])
69
 
 
 
 
 
 
 
 
70
 
71
  def _ns(col):
72
  """ISO-8601 string column -> Int64 ns since epoch (UTC), null when unparsable."""
@@ -174,9 +181,20 @@ class Store:
174
  pl.concat_str([pl.col("title").fill_null(""), pl.col("id").fill_null(""), pl.col("snippet").fill_null(""),
175
  pl.col("tags").list.join(" ").fill_null("")], separator=" ").str.to_lowercase()
176
  .alias("_search"))
 
 
 
 
 
 
 
177
  eff = raw.copy()
178
  for b, ns in NARROW.items():
179
- eff[:, b] = np.max(raw[:, [b, *ns]], axis=1)
 
 
 
 
180
  self.data = Data(meta, raw, eff, self.data.version + 1)
181
 
182
  def _read_status(self):
@@ -405,7 +423,7 @@ def items_payload(d, rows):
405
  trend = TRENDING.array(d)[rows]
406
  for i, it in enumerate(page):
407
  it["snip"] = it.pop("snippet")
408
- it["tags"] = (it["tags"] or [])[:8]
409
  it["t"] = iso_ns(int(t[i]))
410
  it["created"] = iso_ns(int(created[i]))
411
  it["modified"] = iso_ns(int(modified[i]))
@@ -479,7 +497,8 @@ def status_payload():
479
  def catalogue() -> dict:
480
  """Topic catalogue: labels, notes, theme groups, broad/narrow pairs, default thresholds."""
481
  return {k: CAT[k] for k in ("catalogue_version", "labels", "notes", "groups", "pairs", "thresholds")} | {
482
- "thresholds_v1": CAT.get("thresholds_v1"), "custom_topics_enabled": CUSTOM_TOPICS_ENABLED}
 
483
 
484
 
485
  def status() -> dict:
@@ -724,6 +743,7 @@ async def homepage():
724
  # ---------------------------------------------------------------- JSON API
725
  # Lists (topics, sources) are repeated query params: ?topics=robotics&topics=video. thresholds is a JSON object.
726
  # Item scores come as `s`: a list of 48 raw scores in catalogue label order (the page applies max(broad, narrow)).
 
727
  @app.get("/api/catalogue")
728
  def api_catalogue():
729
  return cached_json("catalogue", {}, catalogue, max_age=3600)
 
67
  for b, n in CAT["pairs"]:
68
  NARROW.setdefault(L[b], []).append(L[n])
69
 
70
+ # The model gives "historical newspapers" high scores for general news too, and it cannot separate the two: real
71
+ # newspaper corpora and news fine-tunes both reach 0.9+. A score for this topic counts only when the item's own text
72
+ # has a historical marker, and it lifts its broad topic only when it passes this topic's own threshold.
73
+ HISTORICAL_TOPIC = "historical newspapers"
74
+ HISTORICAL_MARKERS = (r"histor|histór|histoire|geschicht|archiv|digiti[sz]|centur|\b1[5-9]\d\d\b|librar(y|ies)\b|"
75
+ r"periodical|gazette\b|old ?(print|book)|\bocr\b")
76
+
77
 
78
  def _ns(col):
79
  """ISO-8601 string column -> Int64 ns since epoch (UTC), null when unparsable."""
 
181
  pl.concat_str([pl.col("title").fill_null(""), pl.col("id").fill_null(""), pl.col("snippet").fill_null(""),
182
  pl.col("tags").list.join(" ").fill_null("")], separator=" ").str.to_lowercase()
183
  .alias("_search"))
184
+ # Prefixed tags such as library:transformers are left out: they would match the markers on almost every model.
185
+ marker_text = pl.concat_str(
186
+ [pl.col("title").fill_null(""), pl.col("id").fill_null(""), pl.col("snippet").fill_null(""),
187
+ pl.col("tags").list.eval(pl.element().filter(~pl.element().str.contains(":"))).list.join(" ").fill_null("")],
188
+ separator=" ").str.to_lowercase().str.contains(HISTORICAL_MARKERS)
189
+ has_marker = meta.select(marker_text.alias("m"))["m"].fill_null(False).to_numpy()
190
+ raw[~has_marker, L[HISTORICAL_TOPIC]] = 0.0
191
  eff = raw.copy()
192
  for b, ns in NARROW.items():
193
+ carried = raw[:, ns].copy()
194
+ for col, n in enumerate(ns):
195
+ if LABELS[n] == HISTORICAL_TOPIC:
196
+ carried[raw[:, n] < DEFAULT_THR[n], col] = 0.0
197
+ eff[:, b] = np.max(np.column_stack([raw[:, b], carried]), axis=1)
198
  self.data = Data(meta, raw, eff, self.data.version + 1)
199
 
200
  def _read_status(self):
 
423
  trend = TRENDING.array(d)[rows]
424
  for i, it in enumerate(page):
425
  it["snip"] = it.pop("snippet")
426
+ it["tags"] = [tag for tag in (it["tags"] or []) if ":" not in tag][:8]
427
  it["t"] = iso_ns(int(t[i]))
428
  it["created"] = iso_ns(int(created[i]))
429
  it["modified"] = iso_ns(int(modified[i]))
 
497
  def catalogue() -> dict:
498
  """Topic catalogue: labels, notes, theme groups, broad/narrow pairs, default thresholds."""
499
  return {k: CAT[k] for k in ("catalogue_version", "labels", "notes", "groups", "pairs", "thresholds")} | {
500
+ "thresholds_v1": CAT.get("thresholds_v1"), "custom_topics_enabled": CUSTOM_TOPICS_ENABLED,
501
+ "narrow_floor": {HISTORICAL_TOPIC: CAT["thresholds"][HISTORICAL_TOPIC]}}
502
 
503
 
504
  def status() -> dict:
 
743
  # ---------------------------------------------------------------- JSON API
744
  # Lists (topics, sources) are repeated query params: ?topics=robotics&topics=video. thresholds is a JSON object.
745
  # Item scores come as `s`: a list of 48 raw scores in catalogue label order (the page applies max(broad, narrow)).
746
+ # "historical newspapers" is 0 in `s` for items without a historical marker (see HISTORICAL_MARKERS).
747
  @app.get("/api/catalogue")
748
  def api_catalogue():
749
  return cached_json("catalogue", {}, catalogue, max_age=3600)
index.html CHANGED
@@ -60,7 +60,9 @@ input,select{font:inherit;font-size:14px;background:transparent;color:var(--ink)
60
  .presets{font-size:13px;color:var(--muted)}
61
  @media (max-width:640px){.erow{grid-template-columns:20px minmax(0,1fr) auto}.erow .desc{grid-column:2/3;grid-row:2}}
62
  .summary{display:flex;flex-wrap:wrap;gap:4px 10px;align-items:baseline}
63
- .summary .ft{font-size:14px}
 
 
64
  .summary .ft .c{font-size:12px;color:var(--faint);margin-left:3px}
65
  .tl{display:grid;border-top:1px solid var(--line)}
66
  .tr{display:grid;grid-template-columns:minmax(0,1fr) 2rem 1.6rem 1.6rem 4.6rem auto;gap:8px;align-items:baseline;padding:6px 0;border-bottom:1px solid var(--line);font-size:13.5px}
@@ -187,10 +189,10 @@ input[type=search]{min-width:14rem}
187
  <p>A small open model, <a href="https://huggingface.co/fastino/GLiNER2.5-Decide" target="_blank" rel="noopener">fastino/GLiNER2.5-Decide</a>, reads each item once and gives it a score from 0 to 1 for each of 48 topics, in one call. It runs on CPU. It is a classifier, not an LLM: it does not write text.</p>
188
  <ul>
189
  <li><b>What is scored:</b> new and updated public Hub models and datasets that have a real card (at least a few sentences), and papers from Hugging Face Daily Papers. The first week also includes arXiv computer-science papers from a one-off backfill. Quantized and converted models take their base model's topics. Items with no README or an empty template card are skipped.</li>
190
- <li><b>When it shows up:</b> a worker process in this Space follows the Hub, checks whether each card changed, scores changed items and stores the scores, usually within a minute or two of the change. The page reads the stored scores.</li>
191
  <li><b>Thresholds:</b> an item is tagged with a topic when its score reaches that topic's threshold. The defaults are set so each topic tags about as many items as a larger LLM did on a reference sample.</li>
192
  <li><b>Your confirmations:</b> tick or untick the topics on an item and press Confirm. After 3 ✓ for a topic, its threshold is fitted to your confirmations. You can also set it by hand under calibrate. This is stored in your browser only.</li>
193
- <li><b>Caveats:</b> the topics are zero-shot, so some tags are wrong, and broad topics catch loosely related items. Likes and downloads are a snapshot from when the item was scored.</li>
194
  </ul>
195
  </details>
196
  </div>
@@ -226,7 +228,7 @@ input[type=search]{min-width:14rem}
226
  <div class="feedhead"><h2 id="listTitle">Live</h2><span class="note row" style="gap:10px">
227
  <span class="seg" id="scopeSeg" role="group" aria-label="Which items"></span>
228
  <label class="status" title="Repos whose card is up but that have no data or weight files yet"><input type="checkbox" id="coLive"> card-only</label>
229
- <span id="feedCount"></span><button class="link" id="flowBtn" aria-expanded="true">hide how it works</button></span></div>
230
  <div class="ticker" id="tickerBox"><span class="pulse" id="pulse"></span><span id="ticker">Checking the worker…</span></div>
231
  <div class="stream mono" id="stream"></div>
232
  <div class="flow" id="flowBox">
@@ -234,7 +236,7 @@ input[type=search]{min-width:14rem}
234
  <div class="row note mono" style="justify-content:space-between"><span id="flowStats"></span><button class="link" id="replay">replay</button></div>
235
  </div>
236
  <div class="sumbox" id="sumbox" hidden></div>
237
- <div id="feed"><div class="empty">Loading about 28K scored items…</div></div>
238
  <button class="btn ghost small more" id="more" hidden>Show more</button>
239
  </section>
240
 
@@ -287,6 +289,8 @@ const fmt = n => n == null ? "" : n >= 1e6 ? (n / 1e6).toFixed(1) + "M" : n >= 1
287
 
288
  /* ---------- state ---------- */
289
  let CAT = null, narrowOf = {};
 
 
290
  let mode = store.get("mode", "live");
291
  let selected = store.get("selected", ["agents and tool use", "cultural heritage and history", "documents and OCR"]);
292
  let browseSel = store.get("browseSel", ["robotics"]);
@@ -297,10 +301,10 @@ let confirmed = store.get("confirmed", {});
297
  const prevVisit = store.get("lastVisit", null); // shown as "new since your last visit"
298
  let items = [], total = 0, offset = 0, lastLive = null, lastBrowse = null, calTopic = null, editing = null;
299
  let marksThr = {};
300
- let liveAll = store.get("liveAll", false), flowOpen = store.get("flowOpen", true), cardOnly = store.get("cardOnly", false);
301
 
302
  /* ---------- deep links ----------
303
- ?view=live|browse&topics=a,b&items=your|all&days=7&sources=model,dataset&sort=likes&q=...&how=closed&cardonly=1
304
  URL params set this page view only; saved preferences change only when the user edits something. */
305
  const SPACE_URL = "https://huggingface.co/spaces/davanstrien/topic-feed-lab"; // forwards its query to this app
306
  (function readUrl(){
@@ -312,6 +316,7 @@ const SPACE_URL = "https://huggingface.co/spaces/davanstrien/topic-feed-lab";
312
  if (u.has("sources")) bSources = list("sources").filter(x => ["model", "dataset", "paper"].includes(x));
313
  if (["score", "newest", "likes", "downloads", "trending"].includes(u.get("sort"))) bSort = u.get("sort");
314
  if (u.has("how")) flowOpen = u.get("how") !== "closed";
 
315
  if (u.has("cardonly")) cardOnly = u.get("cardonly") === "1";
316
  if (u.has("q")) queueMicrotask(() => { $("q").value = u.get("q"); });
317
  })();
@@ -320,7 +325,7 @@ function viewQuery(){
320
  q.set("view", mode);
321
  const ts = mode === "live" ? selected : browseSel;
322
  if (ts.length) q.set("topics", ts.join(","));
323
- if (mode === "live"){ if (liveAll) q.set("items", "all"); if (!flowOpen) q.set("how", "closed"); }
324
  else {
325
  q.set("days", bDays);
326
  if (bSources.length && bSources.length < 3) q.set("sources", bSources.join(","));
@@ -351,7 +356,8 @@ const saveConfirmed = () => store.set("confirmed", confirmed);
351
  const thrKey = () => "thr-" + (CAT?.catalogue_version || "none");
352
  const key = it => it.source + ":" + it.id;
353
  const raw = (scores, t) => scores?.[t] ?? 0;
354
- const eff = (scores, t) => Math.max(raw(scores, t), ...(narrowOf[t] || []).map(n => raw(scores, n))); // broad = max(broad, narrow)
 
355
  const sc = (it, t) => eff(it.scores, t);
356
  const judged = (k, t) => { const r = confirmed[k]; return r && r.considered.includes(t) ? r.topics.includes(t) : undefined; };
357
  const defaultThr = t => CAT?.thresholds?.[t] ?? 0.21;
@@ -361,6 +367,7 @@ const thrMap = ts => Object.fromEntries(ts.map(t => [t, thr(t)]));
361
  const topicNames = () => CAT?.labels || [];
362
  const feedTopics = () => selected.filter(t => topicNames().includes(t));
363
  const browseTopics = () => browseSel.filter(t => topicNames().includes(t));
 
364
  const shown = () => mode === "live" ? feedTopics() : browseTopics();
365
 
366
  /* ---------- topic pickers ---------- */
@@ -442,7 +449,9 @@ function refitMarks(){ marksThr = {}; for (const t of topicNames()) marksThr[t]
442
  const passCount = t => items.length ? items.filter(it => sc(it, t) >= thr(t)).length : null;
443
  function renderToggles(){
444
  const feed = feedTopics();
445
- $("feedTopics").innerHTML = (feed.length ? feed.map(t => `<span class="ft">${esc(t)}</span>`).join('<span class="note">·</span>')
 
 
446
  : `<span class="note">No topics chosen.</span>`) + `<span class="note">(${feed.length} of ${topicNames().length})</span>`;
447
  const bt = browseTopics();
448
  $("browseTopics").innerHTML = bt.length ? bt.map(t => `<span class="chipx">${esc(t)}<button data-rm="${esc(t)}" aria-label="Remove ${esc(t)}" title="Remove">×</button></span>`).join("")
@@ -455,6 +464,11 @@ function renderToggles(){
455
  <button class="link" data-cal="${esc(t)}">calibrate</button></div>`; }).join("")
456
  : `<p class="note" style="padding:6px 0">Choose topics with <b>choose</b>.</p>`);
457
  }
 
 
 
 
 
458
  $("browseTopics").addEventListener("click", e => { const t = e.target.dataset.rm; if (!t) return; browseSel = browseSel.filter(x => x !== t); store.set("browseSel", browseSel); renderToggles(); reload(); });
459
  function openCal(t){ calTopic = t; renderCal(); $("calPanel").scrollIntoView({block: "nearest"}); }
460
  $("tlist").addEventListener("click", e => { const b = e.target.closest("[data-cal]"); if (b) openCal(b.dataset.cal); });
@@ -496,7 +510,7 @@ async function reload(append = false, poll = false){
496
  try {
497
  let d;
498
  if (mode === "live"){
499
- d = (await api("live", {topics: feedTopics(), thresholds: thrMap(feedTopics()), limit: PAGE, offset, since: prevVisit || "", all_items: liveAll, card_only: cardOnly}));
500
  lastLive = d;
501
  } else {
502
  d = (await api("browse", {topics: browseTopics(), thresholds: thrMap(browseTopics()), days: bDays, sources: bSources,
@@ -532,11 +546,12 @@ function renderStatus(st){
532
  if (ws && !ws.error){
533
  const srcs = Object.values(ws.sources || {});
534
  const scoredH = srcs.reduce((a, x) => a + (x.last_hour?.scored || 0), 0);
535
- const lat = ws.sources?.model?.latency_modified_to_scored_s?.last_hour_p50 ?? ws.sources?.model?.latency_modified_to_scored_s?.p50;
 
536
  const stale = ws.updated_at && Date.now() - Date.parse(ws.updated_at) > 15 * 60000;
537
  worker = `worker ${stale ? `silent since ${ago(ws.updated_at)}` : esc(ws.state || "running")} · ${scoredH} scored in the last hour`
538
  + (ws.queue?.total != null ? ` · queue ${ws.queue.total.toLocaleString()}` : "")
539
- + (lat ? ` · typical delay ${Math.round(lat / 60)} min after a change` : "");
540
  }
541
  $("stream").textContent = `stream · ${st.last_24h.toLocaleString()} items changed in 24 h · ${worker}`;
542
  }
@@ -603,6 +618,18 @@ function itemHTML(it){
603
  <span class="state">${status}</span></div>` : ""}
604
  </article>`;
605
  }
 
 
 
 
 
 
 
 
 
 
 
 
606
  function renderFeed(){
607
  if (!CAT) return;
608
  $("feedCount").textContent = items.length ? `${items.length} of ${total.toLocaleString()}` : "";
@@ -614,11 +641,14 @@ function renderFeed(){
614
  }
615
  let html = "", marked = false;
616
  const newCount = mode === "live" ? items.filter(isNew).length : 0;
617
- items.forEach((it, i) => {
 
618
  if (mode === "live" && prevVisit && !marked && !isNew(it) && i > 0){
619
  html += `<div class="divider">earlier than your last visit (${hhmm(prevVisit)})</div>`; marked = true;
620
  }
621
  html += itemHTML(it);
 
 
622
  });
623
  if (mode === "live" && newCount) html = `<div class="divider">${lastLive?.new_since ?? newCount} new since your last visit</div>` + html;
624
  $("feed").innerHTML = html;
@@ -644,6 +674,8 @@ function saveItem(k, positives){
644
  delete drafts[k]; delete draftRows[k]; saveConfirmed(); afterMarks(k);
645
  }
646
  $("feed").addEventListener("click", e => {
 
 
647
  const el = e.target.closest(".item"); if (!el) return;
648
  const cal = e.target.closest("[data-cal]"); if (cal){ openCal(cal.dataset.cal); return; }
649
  const k = el.dataset.k, it = byKey.get(k), tg = e.target.closest(".tg"), a = e.target.closest("[data-a]")?.dataset.a;
@@ -782,7 +814,7 @@ function flowSetup(stations){
782
  const dpr = window.devicePixelRatio || 1;
783
  cv.width = W * dpr; cv.height = H * dpr; cv.style.height = H + "px";
784
  const ctx = cv.getContext("2d"); ctx.setTransform(dpr, 0, 0, dpr, 0, 0);
785
- Object.assign(flow, {ctx, C, lanes, W, H, laneTop, laneH, notY, mainY: 34, xEnd: W - 26, stations: stations.map(([f, s]) => [W * f, s]),
786
  parts: [], byId: new Map(), pile: {}, begin: performance.now()});
787
  flow.xLast = flow.stations[flow.stations.length - 1][0];
788
  }
@@ -943,7 +975,7 @@ async function init(){
943
  for (const v of ["cat48-v1"]) Object.assign(manualThr, store.get("thr-" + v, {}));
944
  store.set(thrKey(), manualThr);
945
  }
946
- $("status").textContent = `Loading about 28K scored items… (catalogue ${CAT.catalogue_version})`;
947
  refitMarks(); renderControls(); renderAll(); await reload(); pollEvents();
948
  }
949
  init();
 
60
  .presets{font-size:13px;color:var(--muted)}
61
  @media (max-width:640px){.erow{grid-template-columns:20px minmax(0,1fr) auto}.erow .desc{grid-column:2/3;grid-row:2}}
62
  .summary{display:flex;flex-wrap:wrap;gap:4px 10px;align-items:baseline}
63
+ .summary .ft{font-size:14px;font-family:inherit;color:inherit;background:none;border:0;border-bottom:1px solid transparent;padding:0 0 1px;cursor:pointer}
64
+ .summary .ft[aria-pressed="true"]{border-bottom-color:var(--ink);font-weight:600}
65
+ .owner-more{display:block;margin:-6px 0 14px}
66
  .summary .ft .c{font-size:12px;color:var(--faint);margin-left:3px}
67
  .tl{display:grid;border-top:1px solid var(--line)}
68
  .tr{display:grid;grid-template-columns:minmax(0,1fr) 2rem 1.6rem 1.6rem 4.6rem auto;gap:8px;align-items:baseline;padding:6px 0;border-bottom:1px solid var(--line);font-size:13.5px}
 
189
  <p>A small open model, <a href="https://huggingface.co/fastino/GLiNER2.5-Decide" target="_blank" rel="noopener">fastino/GLiNER2.5-Decide</a>, reads each item once and gives it a score from 0 to 1 for each of 48 topics, in one call. It runs on CPU. It is a classifier, not an LLM: it does not write text.</p>
190
  <ul>
191
  <li><b>What is scored:</b> new and updated public Hub models and datasets that have a real card (at least a few sentences), and papers from Hugging Face Daily Papers. The first week also includes arXiv computer-science papers from a one-off backfill. Quantized and converted models take their base model's topics. Items with no README or an empty template card are skipped.</li>
192
+ <li><b>When it shows up:</b> a worker process in this Space follows the Hub, checks whether each card changed, scores changed items and stores the scores. Models usually appear within a minute or two of the change; datasets are found by a slower sweep and can take up to about half an hour. The page reads the stored scores.</li>
193
  <li><b>Thresholds:</b> an item is tagged with a topic when its score reaches that topic's threshold. The defaults are set so each topic tags about as many items as a larger LLM did on a reference sample.</li>
194
  <li><b>Your confirmations:</b> tick or untick the topics on an item and press Confirm. After 3 ✓ for a topic, its threshold is fitted to your confirmations. You can also set it by hand under calibrate. This is stored in your browser only.</li>
195
+ <li><b>Caveats:</b> the topics are zero-shot, so some tags are wrong, and broad topics catch loosely related items. The model also gives “historical newspapers” high scores for general news, so that topic counts only when the item's name, card summary or tags have a historical marker such as “archive” or a year from 1500 to 1999. Likes and downloads are a snapshot from when the item was scored.</li>
196
  </ul>
197
  </details>
198
  </div>
 
228
  <div class="feedhead"><h2 id="listTitle">Live</h2><span class="note row" style="gap:10px">
229
  <span class="seg" id="scopeSeg" role="group" aria-label="Which items"></span>
230
  <label class="status" title="Repos whose card is up but that have no data or weight files yet"><input type="checkbox" id="coLive"> card-only</label>
231
+ <span id="feedCount"></span><button class="link" id="flowBtn" aria-expanded="false">how it works</button></span></div>
232
  <div class="ticker" id="tickerBox"><span class="pulse" id="pulse"></span><span id="ticker">Checking the worker…</span></div>
233
  <div class="stream mono" id="stream"></div>
234
  <div class="flow" id="flowBox">
 
236
  <div class="row note mono" style="justify-content:space-between"><span id="flowStats"></span><button class="link" id="replay">replay</button></div>
237
  </div>
238
  <div class="sumbox" id="sumbox" hidden></div>
239
+ <div id="feed"><div class="empty">Loading scored items…</div></div>
240
  <button class="btn ghost small more" id="more" hidden>Show more</button>
241
  </section>
242
 
 
289
 
290
  /* ---------- state ---------- */
291
  let CAT = null, narrowOf = {};
292
+ let focus = null; // Live: show one of the feed topics only
293
+ const expandedOwners = new Set(); // owners whose hidden items the user has opened
294
  let mode = store.get("mode", "live");
295
  let selected = store.get("selected", ["agents and tool use", "cultural heritage and history", "documents and OCR"]);
296
  let browseSel = store.get("browseSel", ["robotics"]);
 
301
  const prevVisit = store.get("lastVisit", null); // shown as "new since your last visit"
302
  let items = [], total = 0, offset = 0, lastLive = null, lastBrowse = null, calTopic = null, editing = null;
303
  let marksThr = {};
304
+ let liveAll = store.get("liveAll", false), flowOpen = store.get("flowOpen", false), cardOnly = store.get("cardOnly", false);
305
 
306
  /* ---------- deep links ----------
307
+ ?view=live|browse&topics=a,b&focus=a&items=your|all&days=7&sources=model,dataset&sort=likes&q=...&how=open&cardonly=1
308
  URL params set this page view only; saved preferences change only when the user edits something. */
309
  const SPACE_URL = "https://huggingface.co/spaces/davanstrien/topic-feed-lab"; // forwards its query to this app
310
  (function readUrl(){
 
316
  if (u.has("sources")) bSources = list("sources").filter(x => ["model", "dataset", "paper"].includes(x));
317
  if (["score", "newest", "likes", "downloads", "trending"].includes(u.get("sort"))) bSort = u.get("sort");
318
  if (u.has("how")) flowOpen = u.get("how") !== "closed";
319
+ if (u.has("focus")) focus = u.get("focus");
320
  if (u.has("cardonly")) cardOnly = u.get("cardonly") === "1";
321
  if (u.has("q")) queueMicrotask(() => { $("q").value = u.get("q"); });
322
  })();
 
325
  q.set("view", mode);
326
  const ts = mode === "live" ? selected : browseSel;
327
  if (ts.length) q.set("topics", ts.join(","));
328
+ if (mode === "live"){ if (liveAll) q.set("items", "all"); if (flowOpen) q.set("how", "open"); if (focus) q.set("focus", focus); }
329
  else {
330
  q.set("days", bDays);
331
  if (bSources.length && bSources.length < 3) q.set("sources", bSources.join(","));
 
356
  const thrKey = () => "thr-" + (CAT?.catalogue_version || "none");
357
  const key = it => it.source + ":" + it.id;
358
  const raw = (scores, t) => scores?.[t] ?? 0;
359
+ const carried = (scores, n) => raw(scores, n) >= (CAT?.narrow_floor?.[n] ?? 0) ? raw(scores, n) : 0;
360
+ const eff = (scores, t) => Math.max(raw(scores, t), ...(narrowOf[t] || []).map(n => carried(scores, n))); // broad = max(broad, narrow)
361
  const sc = (it, t) => eff(it.scores, t);
362
  const judged = (k, t) => { const r = confirmed[k]; return r && r.considered.includes(t) ? r.topics.includes(t) : undefined; };
363
  const defaultThr = t => CAT?.thresholds?.[t] ?? 0.21;
 
367
  const topicNames = () => CAT?.labels || [];
368
  const feedTopics = () => selected.filter(t => topicNames().includes(t));
369
  const browseTopics = () => browseSel.filter(t => topicNames().includes(t));
370
+ const liveTopics = () => feedTopics().includes(focus) ? [focus] : feedTopics();
371
  const shown = () => mode === "live" ? feedTopics() : browseTopics();
372
 
373
  /* ---------- topic pickers ---------- */
 
449
  const passCount = t => items.length ? items.filter(it => sc(it, t) >= thr(t)).length : null;
450
  function renderToggles(){
451
  const feed = feedTopics();
452
+ const active = liveTopics().length === 1 && feed.length > 1 ? liveTopics()[0] : null;
453
+ $("feedTopics").innerHTML = (feed.length > 1 ? `<button class="ft" data-focus="" aria-pressed="${!active}" title="Show all your topics together">all</button><span class="note">·</span>` : "")
454
+ + (feed.length ? feed.map(t => `<button class="ft" data-focus="${esc(t)}" aria-pressed="${t === active}" title="Show only ${esc(t)}">${esc(t)}</button>`).join('<span class="note">·</span>')
455
  : `<span class="note">No topics chosen.</span>`) + `<span class="note">(${feed.length} of ${topicNames().length})</span>`;
456
  const bt = browseTopics();
457
  $("browseTopics").innerHTML = bt.length ? bt.map(t => `<span class="chipx">${esc(t)}<button data-rm="${esc(t)}" aria-label="Remove ${esc(t)}" title="Remove">×</button></span>`).join("")
 
464
  <button class="link" data-cal="${esc(t)}">calibrate</button></div>`; }).join("")
465
  : `<p class="note" style="padding:6px 0">Choose topics with <b>choose</b>.</p>`);
466
  }
467
+ $("feedTopics").addEventListener("click", e => {
468
+ const b = e.target.closest("[data-focus]"); if (!b) return;
469
+ focus = b.dataset.focus || null; if (focus) liveAll = false;
470
+ renderToggles(); renderControls(); reload();
471
+ });
472
  $("browseTopics").addEventListener("click", e => { const t = e.target.dataset.rm; if (!t) return; browseSel = browseSel.filter(x => x !== t); store.set("browseSel", browseSel); renderToggles(); reload(); });
473
  function openCal(t){ calTopic = t; renderCal(); $("calPanel").scrollIntoView({block: "nearest"}); }
474
  $("tlist").addEventListener("click", e => { const b = e.target.closest("[data-cal]"); if (b) openCal(b.dataset.cal); });
 
510
  try {
511
  let d;
512
  if (mode === "live"){
513
+ d = (await api("live", {topics: liveTopics(), thresholds: thrMap(liveTopics()), limit: PAGE, offset, since: prevVisit || "", all_items: liveAll, card_only: cardOnly}));
514
  lastLive = d;
515
  } else {
516
  d = (await api("browse", {topics: browseTopics(), thresholds: thrMap(browseTopics()), days: bDays, sources: bSources,
 
546
  if (ws && !ws.error){
547
  const srcs = Object.values(ws.sources || {});
548
  const scoredH = srcs.reduce((a, x) => a + (x.last_hour?.scored || 0), 0);
549
+ const delayOf = source => { const l = ws.sources?.[source]?.latency_modified_to_scored_s, v = l?.last_hour_p50 ?? l?.p50; return v ? `${source}s ${Math.max(1, Math.round(v / 60))} min` : null; };
550
+ const delays = ["model", "dataset"].map(delayOf).filter(Boolean).join(", ");
551
  const stale = ws.updated_at && Date.now() - Date.parse(ws.updated_at) > 15 * 60000;
552
  worker = `worker ${stale ? `silent since ${ago(ws.updated_at)}` : esc(ws.state || "running")} · ${scoredH} scored in the last hour`
553
  + (ws.queue?.total != null ? ` · queue ${ws.queue.total.toLocaleString()}` : "")
554
+ + (delays ? ` · typical delay after a change: ${delays}` : "");
555
  }
556
  $("stream").textContent = `stream · ${st.last_24h.toLocaleString()} items changed in 24 h · ${worker}`;
557
  }
 
618
  <span class="state">${status}</span></div>` : ""}
619
  </article>`;
620
  }
621
+ // One owner can publish dozens of near-identical repos in an hour. Show the first of each owner in the loaded list,
622
+ // with a button for the rest. A text search keeps every item, because it may be looking for that owner.
623
+ function collapseByOwner(list){
624
+ if (mode === "browse" && $("q").value.trim()) return {visible: list, hidden: new Map()};
625
+ const seen = new Set(), hidden = new Map(), visible = [];
626
+ for (const it of list){
627
+ const owner = it.source === "paper" ? null : it.id.split("/")[0];
628
+ if (!owner || expandedOwners.has(owner) || !seen.has(owner)){ visible.push(it); if (owner) seen.add(owner); }
629
+ else hidden.set(owner, (hidden.get(owner) || 0) + 1);
630
+ }
631
+ return {visible, hidden};
632
+ }
633
  function renderFeed(){
634
  if (!CAT) return;
635
  $("feedCount").textContent = items.length ? `${items.length} of ${total.toLocaleString()}` : "";
 
641
  }
642
  let html = "", marked = false;
643
  const newCount = mode === "live" ? items.filter(isNew).length : 0;
644
+ const {visible, hidden} = collapseByOwner(items);
645
+ visible.forEach((it, i) => {
646
  if (mode === "live" && prevVisit && !marked && !isNew(it) && i > 0){
647
  html += `<div class="divider">earlier than your last visit (${hhmm(prevVisit)})</div>`; marked = true;
648
  }
649
  html += itemHTML(it);
650
+ const owner = it.id.split("/")[0], more = it.source === "paper" ? 0 : hidden.get(owner);
651
+ if (more){ html += `<button class="link owner-more" data-owner="${esc(owner)}">+${more} more from ${esc(owner)}</button>`; hidden.delete(owner); }
652
  });
653
  if (mode === "live" && newCount) html = `<div class="divider">${lastLive?.new_since ?? newCount} new since your last visit</div>` + html;
654
  $("feed").innerHTML = html;
 
674
  delete drafts[k]; delete draftRows[k]; saveConfirmed(); afterMarks(k);
675
  }
676
  $("feed").addEventListener("click", e => {
677
+ const moreBtn = e.target.closest("[data-owner]");
678
+ if (moreBtn){ expandedOwners.add(moreBtn.dataset.owner); renderFeed(); return; }
679
  const el = e.target.closest(".item"); if (!el) return;
680
  const cal = e.target.closest("[data-cal]"); if (cal){ openCal(cal.dataset.cal); return; }
681
  const k = el.dataset.k, it = byKey.get(k), tg = e.target.closest(".tg"), a = e.target.closest("[data-a]")?.dataset.a;
 
814
  const dpr = window.devicePixelRatio || 1;
815
  cv.width = W * dpr; cv.height = H * dpr; cv.style.height = H + "px";
816
  const ctx = cv.getContext("2d"); ctx.setTransform(dpr, 0, 0, dpr, 0, 0);
817
+ Object.assign(flow, {ctx, C, lanes, W, H, laneTop, laneH, notY, mainY: 34, xEnd: W - 26, stations: stations.map(([f, s]) => [W * f, W < 560 && typeof s === "string" ? s.split(" · ")[0] : s]),
818
  parts: [], byId: new Map(), pile: {}, begin: performance.now()});
819
  flow.xLast = flow.stations[flow.stations.length - 1][0];
820
  }
 
975
  for (const v of ["cat48-v1"]) Object.assign(manualThr, store.get("thr-" + v, {}));
976
  store.set(thrKey(), manualThr);
977
  }
978
+ $("status").textContent = `Loading scored items… (catalogue ${CAT.catalogue_version})`;
979
  refitMarks(); renderControls(); renderAll(); await reload(); pollEvents();
980
  }
981
  init();