Spaces:
Running on CPU Upgrade
Running on CPU Upgrade
Filter news out of historical newspapers; add topic focus and owner collapse; show delay per source
Browse files- README.md +7 -3
- app.py +23 -3
- index.html +48 -16
README.md
CHANGED
|
@@ -16,6 +16,7 @@ Follow new Hugging Face Hub models, datasets and Daily Papers by topic, or brows
|
|
| 16 |
## What it does
|
| 17 |
|
| 18 |
- **Live:** the newest items for the topics you pick, newest first, with a marker for what is new since your last visit.
|
|
|
|
| 19 |
- **Browse:** pick one or more topics, a time range, sources and a search term. Sort by topic score, newest, likes or
|
| 20 |
downloads. A summary shows the counts per source and what else those items are about.
|
| 21 |
- **Confirm:** tick the topics that fit an item and press Confirm. After 3 confirmations for a topic, its threshold is
|
|
@@ -29,7 +30,8 @@ Follow new Hugging Face Hub models, datasets and Daily Papers by topic, or brows
|
|
| 29 |
- The scores are stored in the bucket `davanstrien/hub-firehose` (private for now), mounted in this Space at `/data`.
|
| 30 |
A background worker process in this Space (cpu-upgrade, CPU only, lower priority than the page) watches the Hub for
|
| 31 |
new and updated models and datasets plus HF Daily Papers, checks whether the card changed, and scores changed items
|
| 32 |
-
with GLiNER2.5-Decide.
|
|
|
|
| 33 |
- The app is FastAPI served by Granian (Docker Space, one app process). It supervises exactly one ingest worker
|
| 34 |
(a file lock enforces this). JSON API: `GET /api/catalogue`, `/api/status`, `/api/live`, `/api/events`,
|
| 35 |
`/api/browse` (list params repeated, e.g. `?topics=robotics&topics=video`; `thresholds` is a JSON object;
|
|
@@ -41,7 +43,8 @@ Follow new Hugging Face Hub models, datasets and Daily Papers by topic, or brows
|
|
| 41 |
|
| 42 |
- Every view has a shareable URL, e.g.
|
| 43 |
`https://huggingface.co/spaces/davanstrien/topic-feed-lab?view=browse&topics=robotics&days=7&sources=dataset&sort=likes`
|
| 44 |
-
(params: `view`, `topics` comma-separated, `items=all`, `days`, `sources`, `sort`,
|
|
|
|
| 45 |
Your thresholds and confirmations stay in your browser and are not part of the link.
|
| 46 |
- **Card only:** repos with a README but no data files (datasets) or weight files (models) yet are hidden by default;
|
| 47 |
tick "card-only" to include them. The worker records this at scoring time (`has_files`) and re-checks a card-only
|
|
@@ -54,4 +57,5 @@ Follow new Hugging Face Hub models, datasets and Daily Papers by topic, or brows
|
|
| 54 |
- Likes and downloads are a snapshot from scoring time.
|
| 55 |
- Papers have dates, not times; "today" means the latest paper day.
|
| 56 |
- Custom topics are not available yet (they need the model; planned for the worker).
|
| 57 |
-
-
|
|
|
|
|
|
| 16 |
## What it does
|
| 17 |
|
| 18 |
- **Live:** the newest items for the topics you pick, newest first, with a marker for what is new since your last visit.
|
| 19 |
+
Pick one topic to see it alone. When one owner has several items, the first is shown with a button for the rest.
|
| 20 |
- **Browse:** pick one or more topics, a time range, sources and a search term. Sort by topic score, newest, likes or
|
| 21 |
downloads. A summary shows the counts per source and what else those items are about.
|
| 22 |
- **Confirm:** tick the topics that fit an item and press Confirm. After 3 confirmations for a topic, its threshold is
|
|
|
|
| 30 |
- The scores are stored in the bucket `davanstrien/hub-firehose` (private for now), mounted in this Space at `/data`.
|
| 31 |
A background worker process in this Space (cpu-upgrade, CPU only, lower priority than the page) watches the Hub for
|
| 32 |
new and updated models and datasets plus HF Daily Papers, checks whether the card changed, and scores changed items
|
| 33 |
+
with GLiNER2.5-Decide. Models usually appear within a minute or two of a change; datasets are found by a slower
|
| 34 |
+
sweep and can take up to about half an hour. The page loads the scores into memory and picks up new files every 20 s.
|
| 35 |
- The app is FastAPI served by Granian (Docker Space, one app process). It supervises exactly one ingest worker
|
| 36 |
(a file lock enforces this). JSON API: `GET /api/catalogue`, `/api/status`, `/api/live`, `/api/events`,
|
| 37 |
`/api/browse` (list params repeated, e.g. `?topics=robotics&topics=video`; `thresholds` is a JSON object;
|
|
|
|
| 43 |
|
| 44 |
- Every view has a shareable URL, e.g.
|
| 45 |
`https://huggingface.co/spaces/davanstrien/topic-feed-lab?view=browse&topics=robotics&days=7&sources=dataset&sort=likes`
|
| 46 |
+
(params: `view`, `topics` comma-separated, `focus` for one feed topic in Live, `items=all`, `days`, `sources`, `sort`,
|
| 47 |
+
`q`, `how=open`, `cardonly=1`).
|
| 48 |
Your thresholds and confirmations stay in your browser and are not part of the link.
|
| 49 |
- **Card only:** repos with a README but no data files (datasets) or weight files (models) yet are hidden by default;
|
| 50 |
tick "card-only" to include them. The worker records this at scoring time (`has_files`) and re-checks a card-only
|
|
|
|
| 57 |
- Likes and downloads are a snapshot from scoring time.
|
| 58 |
- Papers have dates, not times; "today" means the latest paper day.
|
| 59 |
- Custom topics are not available yet (they need the model; planned for the worker).
|
| 60 |
+
- The model gives "historical newspapers" high scores for general news. The topic counts only when an item's name,
|
| 61 |
+
card summary or tags have a historical marker, which misses items whose text has none and keeps some news with one.
|
app.py
CHANGED
|
@@ -67,6 +67,13 @@ NARROW = {} # broad index -> [narrow indices]; a broad topic's effective score
|
|
| 67 |
for b, n in CAT["pairs"]:
|
| 68 |
NARROW.setdefault(L[b], []).append(L[n])
|
| 69 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 70 |
|
| 71 |
def _ns(col):
|
| 72 |
"""ISO-8601 string column -> Int64 ns since epoch (UTC), null when unparsable."""
|
|
@@ -174,9 +181,20 @@ class Store:
|
|
| 174 |
pl.concat_str([pl.col("title").fill_null(""), pl.col("id").fill_null(""), pl.col("snippet").fill_null(""),
|
| 175 |
pl.col("tags").list.join(" ").fill_null("")], separator=" ").str.to_lowercase()
|
| 176 |
.alias("_search"))
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 177 |
eff = raw.copy()
|
| 178 |
for b, ns in NARROW.items():
|
| 179 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 180 |
self.data = Data(meta, raw, eff, self.data.version + 1)
|
| 181 |
|
| 182 |
def _read_status(self):
|
|
@@ -405,7 +423,7 @@ def items_payload(d, rows):
|
|
| 405 |
trend = TRENDING.array(d)[rows]
|
| 406 |
for i, it in enumerate(page):
|
| 407 |
it["snip"] = it.pop("snippet")
|
| 408 |
-
it["tags"] = (it["tags"] or [])[:8]
|
| 409 |
it["t"] = iso_ns(int(t[i]))
|
| 410 |
it["created"] = iso_ns(int(created[i]))
|
| 411 |
it["modified"] = iso_ns(int(modified[i]))
|
|
@@ -479,7 +497,8 @@ def status_payload():
|
|
| 479 |
def catalogue() -> dict:
|
| 480 |
"""Topic catalogue: labels, notes, theme groups, broad/narrow pairs, default thresholds."""
|
| 481 |
return {k: CAT[k] for k in ("catalogue_version", "labels", "notes", "groups", "pairs", "thresholds")} | {
|
| 482 |
-
"thresholds_v1": CAT.get("thresholds_v1"), "custom_topics_enabled": CUSTOM_TOPICS_ENABLED
|
|
|
|
| 483 |
|
| 484 |
|
| 485 |
def status() -> dict:
|
|
@@ -724,6 +743,7 @@ async def homepage():
|
|
| 724 |
# ---------------------------------------------------------------- JSON API
|
| 725 |
# Lists (topics, sources) are repeated query params: ?topics=robotics&topics=video. thresholds is a JSON object.
|
| 726 |
# Item scores come as `s`: a list of 48 raw scores in catalogue label order (the page applies max(broad, narrow)).
|
|
|
|
| 727 |
@app.get("/api/catalogue")
|
| 728 |
def api_catalogue():
|
| 729 |
return cached_json("catalogue", {}, catalogue, max_age=3600)
|
|
|
|
| 67 |
for b, n in CAT["pairs"]:
|
| 68 |
NARROW.setdefault(L[b], []).append(L[n])
|
| 69 |
|
| 70 |
+
# The model gives "historical newspapers" high scores for general news too, and it cannot separate the two: real
|
| 71 |
+
# newspaper corpora and news fine-tunes both reach 0.9+. A score for this topic counts only when the item's own text
|
| 72 |
+
# has a historical marker, and it lifts its broad topic only when it passes this topic's own threshold.
|
| 73 |
+
HISTORICAL_TOPIC = "historical newspapers"
|
| 74 |
+
HISTORICAL_MARKERS = (r"histor|histór|histoire|geschicht|archiv|digiti[sz]|centur|\b1[5-9]\d\d\b|librar(y|ies)\b|"
|
| 75 |
+
r"periodical|gazette\b|old ?(print|book)|\bocr\b")
|
| 76 |
+
|
| 77 |
|
| 78 |
def _ns(col):
|
| 79 |
"""ISO-8601 string column -> Int64 ns since epoch (UTC), null when unparsable."""
|
|
|
|
| 181 |
pl.concat_str([pl.col("title").fill_null(""), pl.col("id").fill_null(""), pl.col("snippet").fill_null(""),
|
| 182 |
pl.col("tags").list.join(" ").fill_null("")], separator=" ").str.to_lowercase()
|
| 183 |
.alias("_search"))
|
| 184 |
+
# Prefixed tags such as library:transformers are left out: they would match the markers on almost every model.
|
| 185 |
+
marker_text = pl.concat_str(
|
| 186 |
+
[pl.col("title").fill_null(""), pl.col("id").fill_null(""), pl.col("snippet").fill_null(""),
|
| 187 |
+
pl.col("tags").list.eval(pl.element().filter(~pl.element().str.contains(":"))).list.join(" ").fill_null("")],
|
| 188 |
+
separator=" ").str.to_lowercase().str.contains(HISTORICAL_MARKERS)
|
| 189 |
+
has_marker = meta.select(marker_text.alias("m"))["m"].fill_null(False).to_numpy()
|
| 190 |
+
raw[~has_marker, L[HISTORICAL_TOPIC]] = 0.0
|
| 191 |
eff = raw.copy()
|
| 192 |
for b, ns in NARROW.items():
|
| 193 |
+
carried = raw[:, ns].copy()
|
| 194 |
+
for col, n in enumerate(ns):
|
| 195 |
+
if LABELS[n] == HISTORICAL_TOPIC:
|
| 196 |
+
carried[raw[:, n] < DEFAULT_THR[n], col] = 0.0
|
| 197 |
+
eff[:, b] = np.max(np.column_stack([raw[:, b], carried]), axis=1)
|
| 198 |
self.data = Data(meta, raw, eff, self.data.version + 1)
|
| 199 |
|
| 200 |
def _read_status(self):
|
|
|
|
| 423 |
trend = TRENDING.array(d)[rows]
|
| 424 |
for i, it in enumerate(page):
|
| 425 |
it["snip"] = it.pop("snippet")
|
| 426 |
+
it["tags"] = [tag for tag in (it["tags"] or []) if ":" not in tag][:8]
|
| 427 |
it["t"] = iso_ns(int(t[i]))
|
| 428 |
it["created"] = iso_ns(int(created[i]))
|
| 429 |
it["modified"] = iso_ns(int(modified[i]))
|
|
|
|
| 497 |
def catalogue() -> dict:
|
| 498 |
"""Topic catalogue: labels, notes, theme groups, broad/narrow pairs, default thresholds."""
|
| 499 |
return {k: CAT[k] for k in ("catalogue_version", "labels", "notes", "groups", "pairs", "thresholds")} | {
|
| 500 |
+
"thresholds_v1": CAT.get("thresholds_v1"), "custom_topics_enabled": CUSTOM_TOPICS_ENABLED,
|
| 501 |
+
"narrow_floor": {HISTORICAL_TOPIC: CAT["thresholds"][HISTORICAL_TOPIC]}}
|
| 502 |
|
| 503 |
|
| 504 |
def status() -> dict:
|
|
|
|
| 743 |
# ---------------------------------------------------------------- JSON API
|
| 744 |
# Lists (topics, sources) are repeated query params: ?topics=robotics&topics=video. thresholds is a JSON object.
|
| 745 |
# Item scores come as `s`: a list of 48 raw scores in catalogue label order (the page applies max(broad, narrow)).
|
| 746 |
+
# "historical newspapers" is 0 in `s` for items without a historical marker (see HISTORICAL_MARKERS).
|
| 747 |
@app.get("/api/catalogue")
|
| 748 |
def api_catalogue():
|
| 749 |
return cached_json("catalogue", {}, catalogue, max_age=3600)
|
index.html
CHANGED
|
@@ -60,7 +60,9 @@ input,select{font:inherit;font-size:14px;background:transparent;color:var(--ink)
|
|
| 60 |
.presets{font-size:13px;color:var(--muted)}
|
| 61 |
@media (max-width:640px){.erow{grid-template-columns:20px minmax(0,1fr) auto}.erow .desc{grid-column:2/3;grid-row:2}}
|
| 62 |
.summary{display:flex;flex-wrap:wrap;gap:4px 10px;align-items:baseline}
|
| 63 |
-
.summary .ft{font-size:14px}
|
|
|
|
|
|
|
| 64 |
.summary .ft .c{font-size:12px;color:var(--faint);margin-left:3px}
|
| 65 |
.tl{display:grid;border-top:1px solid var(--line)}
|
| 66 |
.tr{display:grid;grid-template-columns:minmax(0,1fr) 2rem 1.6rem 1.6rem 4.6rem auto;gap:8px;align-items:baseline;padding:6px 0;border-bottom:1px solid var(--line);font-size:13.5px}
|
|
@@ -187,10 +189,10 @@ input[type=search]{min-width:14rem}
|
|
| 187 |
<p>A small open model, <a href="https://huggingface.co/fastino/GLiNER2.5-Decide" target="_blank" rel="noopener">fastino/GLiNER2.5-Decide</a>, reads each item once and gives it a score from 0 to 1 for each of 48 topics, in one call. It runs on CPU. It is a classifier, not an LLM: it does not write text.</p>
|
| 188 |
<ul>
|
| 189 |
<li><b>What is scored:</b> new and updated public Hub models and datasets that have a real card (at least a few sentences), and papers from Hugging Face Daily Papers. The first week also includes arXiv computer-science papers from a one-off backfill. Quantized and converted models take their base model's topics. Items with no README or an empty template card are skipped.</li>
|
| 190 |
-
<li><b>When it shows up:</b> a worker process in this Space follows the Hub, checks whether each card changed, scores changed items and stores the scores
|
| 191 |
<li><b>Thresholds:</b> an item is tagged with a topic when its score reaches that topic's threshold. The defaults are set so each topic tags about as many items as a larger LLM did on a reference sample.</li>
|
| 192 |
<li><b>Your confirmations:</b> tick or untick the topics on an item and press Confirm. After 3 ✓ for a topic, its threshold is fitted to your confirmations. You can also set it by hand under calibrate. This is stored in your browser only.</li>
|
| 193 |
-
<li><b>Caveats:</b> the topics are zero-shot, so some tags are wrong, and broad topics catch loosely related items. Likes and downloads are a snapshot from when the item was scored.</li>
|
| 194 |
</ul>
|
| 195 |
</details>
|
| 196 |
</div>
|
|
@@ -226,7 +228,7 @@ input[type=search]{min-width:14rem}
|
|
| 226 |
<div class="feedhead"><h2 id="listTitle">Live</h2><span class="note row" style="gap:10px">
|
| 227 |
<span class="seg" id="scopeSeg" role="group" aria-label="Which items"></span>
|
| 228 |
<label class="status" title="Repos whose card is up but that have no data or weight files yet"><input type="checkbox" id="coLive"> card-only</label>
|
| 229 |
-
<span id="feedCount"></span><button class="link" id="flowBtn" aria-expanded="
|
| 230 |
<div class="ticker" id="tickerBox"><span class="pulse" id="pulse"></span><span id="ticker">Checking the worker…</span></div>
|
| 231 |
<div class="stream mono" id="stream"></div>
|
| 232 |
<div class="flow" id="flowBox">
|
|
@@ -234,7 +236,7 @@ input[type=search]{min-width:14rem}
|
|
| 234 |
<div class="row note mono" style="justify-content:space-between"><span id="flowStats"></span><button class="link" id="replay">replay</button></div>
|
| 235 |
</div>
|
| 236 |
<div class="sumbox" id="sumbox" hidden></div>
|
| 237 |
-
<div id="feed"><div class="empty">Loading
|
| 238 |
<button class="btn ghost small more" id="more" hidden>Show more</button>
|
| 239 |
</section>
|
| 240 |
|
|
@@ -287,6 +289,8 @@ const fmt = n => n == null ? "" : n >= 1e6 ? (n / 1e6).toFixed(1) + "M" : n >= 1
|
|
| 287 |
|
| 288 |
/* ---------- state ---------- */
|
| 289 |
let CAT = null, narrowOf = {};
|
|
|
|
|
|
|
| 290 |
let mode = store.get("mode", "live");
|
| 291 |
let selected = store.get("selected", ["agents and tool use", "cultural heritage and history", "documents and OCR"]);
|
| 292 |
let browseSel = store.get("browseSel", ["robotics"]);
|
|
@@ -297,10 +301,10 @@ let confirmed = store.get("confirmed", {});
|
|
| 297 |
const prevVisit = store.get("lastVisit", null); // shown as "new since your last visit"
|
| 298 |
let items = [], total = 0, offset = 0, lastLive = null, lastBrowse = null, calTopic = null, editing = null;
|
| 299 |
let marksThr = {};
|
| 300 |
-
let liveAll = store.get("liveAll", false), flowOpen = store.get("flowOpen",
|
| 301 |
|
| 302 |
/* ---------- deep links ----------
|
| 303 |
-
?view=live|browse&topics=a,b&items=your|all&days=7&sources=model,dataset&sort=likes&q=...&how=
|
| 304 |
URL params set this page view only; saved preferences change only when the user edits something. */
|
| 305 |
const SPACE_URL = "https://huggingface.co/spaces/davanstrien/topic-feed-lab"; // forwards its query to this app
|
| 306 |
(function readUrl(){
|
|
@@ -312,6 +316,7 @@ const SPACE_URL = "https://huggingface.co/spaces/davanstrien/topic-feed-lab";
|
|
| 312 |
if (u.has("sources")) bSources = list("sources").filter(x => ["model", "dataset", "paper"].includes(x));
|
| 313 |
if (["score", "newest", "likes", "downloads", "trending"].includes(u.get("sort"))) bSort = u.get("sort");
|
| 314 |
if (u.has("how")) flowOpen = u.get("how") !== "closed";
|
|
|
|
| 315 |
if (u.has("cardonly")) cardOnly = u.get("cardonly") === "1";
|
| 316 |
if (u.has("q")) queueMicrotask(() => { $("q").value = u.get("q"); });
|
| 317 |
})();
|
|
@@ -320,7 +325,7 @@ function viewQuery(){
|
|
| 320 |
q.set("view", mode);
|
| 321 |
const ts = mode === "live" ? selected : browseSel;
|
| 322 |
if (ts.length) q.set("topics", ts.join(","));
|
| 323 |
-
if (mode === "live"){ if (liveAll) q.set("items", "all"); if (
|
| 324 |
else {
|
| 325 |
q.set("days", bDays);
|
| 326 |
if (bSources.length && bSources.length < 3) q.set("sources", bSources.join(","));
|
|
@@ -351,7 +356,8 @@ const saveConfirmed = () => store.set("confirmed", confirmed);
|
|
| 351 |
const thrKey = () => "thr-" + (CAT?.catalogue_version || "none");
|
| 352 |
const key = it => it.source + ":" + it.id;
|
| 353 |
const raw = (scores, t) => scores?.[t] ?? 0;
|
| 354 |
-
const
|
|
|
|
| 355 |
const sc = (it, t) => eff(it.scores, t);
|
| 356 |
const judged = (k, t) => { const r = confirmed[k]; return r && r.considered.includes(t) ? r.topics.includes(t) : undefined; };
|
| 357 |
const defaultThr = t => CAT?.thresholds?.[t] ?? 0.21;
|
|
@@ -361,6 +367,7 @@ const thrMap = ts => Object.fromEntries(ts.map(t => [t, thr(t)]));
|
|
| 361 |
const topicNames = () => CAT?.labels || [];
|
| 362 |
const feedTopics = () => selected.filter(t => topicNames().includes(t));
|
| 363 |
const browseTopics = () => browseSel.filter(t => topicNames().includes(t));
|
|
|
|
| 364 |
const shown = () => mode === "live" ? feedTopics() : browseTopics();
|
| 365 |
|
| 366 |
/* ---------- topic pickers ---------- */
|
|
@@ -442,7 +449,9 @@ function refitMarks(){ marksThr = {}; for (const t of topicNames()) marksThr[t]
|
|
| 442 |
const passCount = t => items.length ? items.filter(it => sc(it, t) >= thr(t)).length : null;
|
| 443 |
function renderToggles(){
|
| 444 |
const feed = feedTopics();
|
| 445 |
-
|
|
|
|
|
|
|
| 446 |
: `<span class="note">No topics chosen.</span>`) + `<span class="note">(${feed.length} of ${topicNames().length})</span>`;
|
| 447 |
const bt = browseTopics();
|
| 448 |
$("browseTopics").innerHTML = bt.length ? bt.map(t => `<span class="chipx">${esc(t)}<button data-rm="${esc(t)}" aria-label="Remove ${esc(t)}" title="Remove">×</button></span>`).join("")
|
|
@@ -455,6 +464,11 @@ function renderToggles(){
|
|
| 455 |
<button class="link" data-cal="${esc(t)}">calibrate</button></div>`; }).join("")
|
| 456 |
: `<p class="note" style="padding:6px 0">Choose topics with <b>choose</b>.</p>`);
|
| 457 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 458 |
$("browseTopics").addEventListener("click", e => { const t = e.target.dataset.rm; if (!t) return; browseSel = browseSel.filter(x => x !== t); store.set("browseSel", browseSel); renderToggles(); reload(); });
|
| 459 |
function openCal(t){ calTopic = t; renderCal(); $("calPanel").scrollIntoView({block: "nearest"}); }
|
| 460 |
$("tlist").addEventListener("click", e => { const b = e.target.closest("[data-cal]"); if (b) openCal(b.dataset.cal); });
|
|
@@ -496,7 +510,7 @@ async function reload(append = false, poll = false){
|
|
| 496 |
try {
|
| 497 |
let d;
|
| 498 |
if (mode === "live"){
|
| 499 |
-
d = (await api("live", {topics:
|
| 500 |
lastLive = d;
|
| 501 |
} else {
|
| 502 |
d = (await api("browse", {topics: browseTopics(), thresholds: thrMap(browseTopics()), days: bDays, sources: bSources,
|
|
@@ -532,11 +546,12 @@ function renderStatus(st){
|
|
| 532 |
if (ws && !ws.error){
|
| 533 |
const srcs = Object.values(ws.sources || {});
|
| 534 |
const scoredH = srcs.reduce((a, x) => a + (x.last_hour?.scored || 0), 0);
|
| 535 |
-
const
|
|
|
|
| 536 |
const stale = ws.updated_at && Date.now() - Date.parse(ws.updated_at) > 15 * 60000;
|
| 537 |
worker = `worker ${stale ? `silent since ${ago(ws.updated_at)}` : esc(ws.state || "running")} · ${scoredH} scored in the last hour`
|
| 538 |
+ (ws.queue?.total != null ? ` · queue ${ws.queue.total.toLocaleString()}` : "")
|
| 539 |
-
+ (
|
| 540 |
}
|
| 541 |
$("stream").textContent = `stream · ${st.last_24h.toLocaleString()} items changed in 24 h · ${worker}`;
|
| 542 |
}
|
|
@@ -603,6 +618,18 @@ function itemHTML(it){
|
|
| 603 |
<span class="state">${status}</span></div>` : ""}
|
| 604 |
</article>`;
|
| 605 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 606 |
function renderFeed(){
|
| 607 |
if (!CAT) return;
|
| 608 |
$("feedCount").textContent = items.length ? `${items.length} of ${total.toLocaleString()}` : "";
|
|
@@ -614,11 +641,14 @@ function renderFeed(){
|
|
| 614 |
}
|
| 615 |
let html = "", marked = false;
|
| 616 |
const newCount = mode === "live" ? items.filter(isNew).length : 0;
|
| 617 |
-
|
|
|
|
| 618 |
if (mode === "live" && prevVisit && !marked && !isNew(it) && i > 0){
|
| 619 |
html += `<div class="divider">earlier than your last visit (${hhmm(prevVisit)})</div>`; marked = true;
|
| 620 |
}
|
| 621 |
html += itemHTML(it);
|
|
|
|
|
|
|
| 622 |
});
|
| 623 |
if (mode === "live" && newCount) html = `<div class="divider">${lastLive?.new_since ?? newCount} new since your last visit</div>` + html;
|
| 624 |
$("feed").innerHTML = html;
|
|
@@ -644,6 +674,8 @@ function saveItem(k, positives){
|
|
| 644 |
delete drafts[k]; delete draftRows[k]; saveConfirmed(); afterMarks(k);
|
| 645 |
}
|
| 646 |
$("feed").addEventListener("click", e => {
|
|
|
|
|
|
|
| 647 |
const el = e.target.closest(".item"); if (!el) return;
|
| 648 |
const cal = e.target.closest("[data-cal]"); if (cal){ openCal(cal.dataset.cal); return; }
|
| 649 |
const k = el.dataset.k, it = byKey.get(k), tg = e.target.closest(".tg"), a = e.target.closest("[data-a]")?.dataset.a;
|
|
@@ -782,7 +814,7 @@ function flowSetup(stations){
|
|
| 782 |
const dpr = window.devicePixelRatio || 1;
|
| 783 |
cv.width = W * dpr; cv.height = H * dpr; cv.style.height = H + "px";
|
| 784 |
const ctx = cv.getContext("2d"); ctx.setTransform(dpr, 0, 0, dpr, 0, 0);
|
| 785 |
-
Object.assign(flow, {ctx, C, lanes, W, H, laneTop, laneH, notY, mainY: 34, xEnd: W - 26, stations: stations.map(([f, s]) => [W * f, s]),
|
| 786 |
parts: [], byId: new Map(), pile: {}, begin: performance.now()});
|
| 787 |
flow.xLast = flow.stations[flow.stations.length - 1][0];
|
| 788 |
}
|
|
@@ -943,7 +975,7 @@ async function init(){
|
|
| 943 |
for (const v of ["cat48-v1"]) Object.assign(manualThr, store.get("thr-" + v, {}));
|
| 944 |
store.set(thrKey(), manualThr);
|
| 945 |
}
|
| 946 |
-
$("status").textContent = `Loading
|
| 947 |
refitMarks(); renderControls(); renderAll(); await reload(); pollEvents();
|
| 948 |
}
|
| 949 |
init();
|
|
|
|
| 60 |
.presets{font-size:13px;color:var(--muted)}
|
| 61 |
@media (max-width:640px){.erow{grid-template-columns:20px minmax(0,1fr) auto}.erow .desc{grid-column:2/3;grid-row:2}}
|
| 62 |
.summary{display:flex;flex-wrap:wrap;gap:4px 10px;align-items:baseline}
|
| 63 |
+
.summary .ft{font-size:14px;font-family:inherit;color:inherit;background:none;border:0;border-bottom:1px solid transparent;padding:0 0 1px;cursor:pointer}
|
| 64 |
+
.summary .ft[aria-pressed="true"]{border-bottom-color:var(--ink);font-weight:600}
|
| 65 |
+
.owner-more{display:block;margin:-6px 0 14px}
|
| 66 |
.summary .ft .c{font-size:12px;color:var(--faint);margin-left:3px}
|
| 67 |
.tl{display:grid;border-top:1px solid var(--line)}
|
| 68 |
.tr{display:grid;grid-template-columns:minmax(0,1fr) 2rem 1.6rem 1.6rem 4.6rem auto;gap:8px;align-items:baseline;padding:6px 0;border-bottom:1px solid var(--line);font-size:13.5px}
|
|
|
|
| 189 |
<p>A small open model, <a href="https://huggingface.co/fastino/GLiNER2.5-Decide" target="_blank" rel="noopener">fastino/GLiNER2.5-Decide</a>, reads each item once and gives it a score from 0 to 1 for each of 48 topics, in one call. It runs on CPU. It is a classifier, not an LLM: it does not write text.</p>
|
| 190 |
<ul>
|
| 191 |
<li><b>What is scored:</b> new and updated public Hub models and datasets that have a real card (at least a few sentences), and papers from Hugging Face Daily Papers. The first week also includes arXiv computer-science papers from a one-off backfill. Quantized and converted models take their base model's topics. Items with no README or an empty template card are skipped.</li>
|
| 192 |
+
<li><b>When it shows up:</b> a worker process in this Space follows the Hub, checks whether each card changed, scores changed items and stores the scores. Models usually appear within a minute or two of the change; datasets are found by a slower sweep and can take up to about half an hour. The page reads the stored scores.</li>
|
| 193 |
<li><b>Thresholds:</b> an item is tagged with a topic when its score reaches that topic's threshold. The defaults are set so each topic tags about as many items as a larger LLM did on a reference sample.</li>
|
| 194 |
<li><b>Your confirmations:</b> tick or untick the topics on an item and press Confirm. After 3 ✓ for a topic, its threshold is fitted to your confirmations. You can also set it by hand under calibrate. This is stored in your browser only.</li>
|
| 195 |
+
<li><b>Caveats:</b> the topics are zero-shot, so some tags are wrong, and broad topics catch loosely related items. The model also gives “historical newspapers” high scores for general news, so that topic counts only when the item's name, card summary or tags have a historical marker such as “archive” or a year from 1500 to 1999. Likes and downloads are a snapshot from when the item was scored.</li>
|
| 196 |
</ul>
|
| 197 |
</details>
|
| 198 |
</div>
|
|
|
|
| 228 |
<div class="feedhead"><h2 id="listTitle">Live</h2><span class="note row" style="gap:10px">
|
| 229 |
<span class="seg" id="scopeSeg" role="group" aria-label="Which items"></span>
|
| 230 |
<label class="status" title="Repos whose card is up but that have no data or weight files yet"><input type="checkbox" id="coLive"> card-only</label>
|
| 231 |
+
<span id="feedCount"></span><button class="link" id="flowBtn" aria-expanded="false">how it works</button></span></div>
|
| 232 |
<div class="ticker" id="tickerBox"><span class="pulse" id="pulse"></span><span id="ticker">Checking the worker…</span></div>
|
| 233 |
<div class="stream mono" id="stream"></div>
|
| 234 |
<div class="flow" id="flowBox">
|
|
|
|
| 236 |
<div class="row note mono" style="justify-content:space-between"><span id="flowStats"></span><button class="link" id="replay">replay</button></div>
|
| 237 |
</div>
|
| 238 |
<div class="sumbox" id="sumbox" hidden></div>
|
| 239 |
+
<div id="feed"><div class="empty">Loading scored items…</div></div>
|
| 240 |
<button class="btn ghost small more" id="more" hidden>Show more</button>
|
| 241 |
</section>
|
| 242 |
|
|
|
|
| 289 |
|
| 290 |
/* ---------- state ---------- */
|
| 291 |
let CAT = null, narrowOf = {};
|
| 292 |
+
let focus = null; // Live: show one of the feed topics only
|
| 293 |
+
const expandedOwners = new Set(); // owners whose hidden items the user has opened
|
| 294 |
let mode = store.get("mode", "live");
|
| 295 |
let selected = store.get("selected", ["agents and tool use", "cultural heritage and history", "documents and OCR"]);
|
| 296 |
let browseSel = store.get("browseSel", ["robotics"]);
|
|
|
|
| 301 |
const prevVisit = store.get("lastVisit", null); // shown as "new since your last visit"
|
| 302 |
let items = [], total = 0, offset = 0, lastLive = null, lastBrowse = null, calTopic = null, editing = null;
|
| 303 |
let marksThr = {};
|
| 304 |
+
let liveAll = store.get("liveAll", false), flowOpen = store.get("flowOpen", false), cardOnly = store.get("cardOnly", false);
|
| 305 |
|
| 306 |
/* ---------- deep links ----------
|
| 307 |
+
?view=live|browse&topics=a,b&focus=a&items=your|all&days=7&sources=model,dataset&sort=likes&q=...&how=open&cardonly=1
|
| 308 |
URL params set this page view only; saved preferences change only when the user edits something. */
|
| 309 |
const SPACE_URL = "https://huggingface.co/spaces/davanstrien/topic-feed-lab"; // forwards its query to this app
|
| 310 |
(function readUrl(){
|
|
|
|
| 316 |
if (u.has("sources")) bSources = list("sources").filter(x => ["model", "dataset", "paper"].includes(x));
|
| 317 |
if (["score", "newest", "likes", "downloads", "trending"].includes(u.get("sort"))) bSort = u.get("sort");
|
| 318 |
if (u.has("how")) flowOpen = u.get("how") !== "closed";
|
| 319 |
+
if (u.has("focus")) focus = u.get("focus");
|
| 320 |
if (u.has("cardonly")) cardOnly = u.get("cardonly") === "1";
|
| 321 |
if (u.has("q")) queueMicrotask(() => { $("q").value = u.get("q"); });
|
| 322 |
})();
|
|
|
|
| 325 |
q.set("view", mode);
|
| 326 |
const ts = mode === "live" ? selected : browseSel;
|
| 327 |
if (ts.length) q.set("topics", ts.join(","));
|
| 328 |
+
if (mode === "live"){ if (liveAll) q.set("items", "all"); if (flowOpen) q.set("how", "open"); if (focus) q.set("focus", focus); }
|
| 329 |
else {
|
| 330 |
q.set("days", bDays);
|
| 331 |
if (bSources.length && bSources.length < 3) q.set("sources", bSources.join(","));
|
|
|
|
| 356 |
const thrKey = () => "thr-" + (CAT?.catalogue_version || "none");
|
| 357 |
const key = it => it.source + ":" + it.id;
|
| 358 |
const raw = (scores, t) => scores?.[t] ?? 0;
|
| 359 |
+
const carried = (scores, n) => raw(scores, n) >= (CAT?.narrow_floor?.[n] ?? 0) ? raw(scores, n) : 0;
|
| 360 |
+
const eff = (scores, t) => Math.max(raw(scores, t), ...(narrowOf[t] || []).map(n => carried(scores, n))); // broad = max(broad, narrow)
|
| 361 |
const sc = (it, t) => eff(it.scores, t);
|
| 362 |
const judged = (k, t) => { const r = confirmed[k]; return r && r.considered.includes(t) ? r.topics.includes(t) : undefined; };
|
| 363 |
const defaultThr = t => CAT?.thresholds?.[t] ?? 0.21;
|
|
|
|
| 367 |
const topicNames = () => CAT?.labels || [];
|
| 368 |
const feedTopics = () => selected.filter(t => topicNames().includes(t));
|
| 369 |
const browseTopics = () => browseSel.filter(t => topicNames().includes(t));
|
| 370 |
+
const liveTopics = () => feedTopics().includes(focus) ? [focus] : feedTopics();
|
| 371 |
const shown = () => mode === "live" ? feedTopics() : browseTopics();
|
| 372 |
|
| 373 |
/* ---------- topic pickers ---------- */
|
|
|
|
| 449 |
const passCount = t => items.length ? items.filter(it => sc(it, t) >= thr(t)).length : null;
|
| 450 |
function renderToggles(){
|
| 451 |
const feed = feedTopics();
|
| 452 |
+
const active = liveTopics().length === 1 && feed.length > 1 ? liveTopics()[0] : null;
|
| 453 |
+
$("feedTopics").innerHTML = (feed.length > 1 ? `<button class="ft" data-focus="" aria-pressed="${!active}" title="Show all your topics together">all</button><span class="note">·</span>` : "")
|
| 454 |
+
+ (feed.length ? feed.map(t => `<button class="ft" data-focus="${esc(t)}" aria-pressed="${t === active}" title="Show only ${esc(t)}">${esc(t)}</button>`).join('<span class="note">·</span>')
|
| 455 |
: `<span class="note">No topics chosen.</span>`) + `<span class="note">(${feed.length} of ${topicNames().length})</span>`;
|
| 456 |
const bt = browseTopics();
|
| 457 |
$("browseTopics").innerHTML = bt.length ? bt.map(t => `<span class="chipx">${esc(t)}<button data-rm="${esc(t)}" aria-label="Remove ${esc(t)}" title="Remove">×</button></span>`).join("")
|
|
|
|
| 464 |
<button class="link" data-cal="${esc(t)}">calibrate</button></div>`; }).join("")
|
| 465 |
: `<p class="note" style="padding:6px 0">Choose topics with <b>choose</b>.</p>`);
|
| 466 |
}
|
| 467 |
+
$("feedTopics").addEventListener("click", e => {
|
| 468 |
+
const b = e.target.closest("[data-focus]"); if (!b) return;
|
| 469 |
+
focus = b.dataset.focus || null; if (focus) liveAll = false;
|
| 470 |
+
renderToggles(); renderControls(); reload();
|
| 471 |
+
});
|
| 472 |
$("browseTopics").addEventListener("click", e => { const t = e.target.dataset.rm; if (!t) return; browseSel = browseSel.filter(x => x !== t); store.set("browseSel", browseSel); renderToggles(); reload(); });
|
| 473 |
function openCal(t){ calTopic = t; renderCal(); $("calPanel").scrollIntoView({block: "nearest"}); }
|
| 474 |
$("tlist").addEventListener("click", e => { const b = e.target.closest("[data-cal]"); if (b) openCal(b.dataset.cal); });
|
|
|
|
| 510 |
try {
|
| 511 |
let d;
|
| 512 |
if (mode === "live"){
|
| 513 |
+
d = (await api("live", {topics: liveTopics(), thresholds: thrMap(liveTopics()), limit: PAGE, offset, since: prevVisit || "", all_items: liveAll, card_only: cardOnly}));
|
| 514 |
lastLive = d;
|
| 515 |
} else {
|
| 516 |
d = (await api("browse", {topics: browseTopics(), thresholds: thrMap(browseTopics()), days: bDays, sources: bSources,
|
|
|
|
| 546 |
if (ws && !ws.error){
|
| 547 |
const srcs = Object.values(ws.sources || {});
|
| 548 |
const scoredH = srcs.reduce((a, x) => a + (x.last_hour?.scored || 0), 0);
|
| 549 |
+
const delayOf = source => { const l = ws.sources?.[source]?.latency_modified_to_scored_s, v = l?.last_hour_p50 ?? l?.p50; return v ? `${source}s ${Math.max(1, Math.round(v / 60))} min` : null; };
|
| 550 |
+
const delays = ["model", "dataset"].map(delayOf).filter(Boolean).join(", ");
|
| 551 |
const stale = ws.updated_at && Date.now() - Date.parse(ws.updated_at) > 15 * 60000;
|
| 552 |
worker = `worker ${stale ? `silent since ${ago(ws.updated_at)}` : esc(ws.state || "running")} · ${scoredH} scored in the last hour`
|
| 553 |
+ (ws.queue?.total != null ? ` · queue ${ws.queue.total.toLocaleString()}` : "")
|
| 554 |
+
+ (delays ? ` · typical delay after a change: ${delays}` : "");
|
| 555 |
}
|
| 556 |
$("stream").textContent = `stream · ${st.last_24h.toLocaleString()} items changed in 24 h · ${worker}`;
|
| 557 |
}
|
|
|
|
| 618 |
<span class="state">${status}</span></div>` : ""}
|
| 619 |
</article>`;
|
| 620 |
}
|
| 621 |
+
// One owner can publish dozens of near-identical repos in an hour. Show the first of each owner in the loaded list,
|
| 622 |
+
// with a button for the rest. A text search keeps every item, because it may be looking for that owner.
|
| 623 |
+
function collapseByOwner(list){
|
| 624 |
+
if (mode === "browse" && $("q").value.trim()) return {visible: list, hidden: new Map()};
|
| 625 |
+
const seen = new Set(), hidden = new Map(), visible = [];
|
| 626 |
+
for (const it of list){
|
| 627 |
+
const owner = it.source === "paper" ? null : it.id.split("/")[0];
|
| 628 |
+
if (!owner || expandedOwners.has(owner) || !seen.has(owner)){ visible.push(it); if (owner) seen.add(owner); }
|
| 629 |
+
else hidden.set(owner, (hidden.get(owner) || 0) + 1);
|
| 630 |
+
}
|
| 631 |
+
return {visible, hidden};
|
| 632 |
+
}
|
| 633 |
function renderFeed(){
|
| 634 |
if (!CAT) return;
|
| 635 |
$("feedCount").textContent = items.length ? `${items.length} of ${total.toLocaleString()}` : "";
|
|
|
|
| 641 |
}
|
| 642 |
let html = "", marked = false;
|
| 643 |
const newCount = mode === "live" ? items.filter(isNew).length : 0;
|
| 644 |
+
const {visible, hidden} = collapseByOwner(items);
|
| 645 |
+
visible.forEach((it, i) => {
|
| 646 |
if (mode === "live" && prevVisit && !marked && !isNew(it) && i > 0){
|
| 647 |
html += `<div class="divider">earlier than your last visit (${hhmm(prevVisit)})</div>`; marked = true;
|
| 648 |
}
|
| 649 |
html += itemHTML(it);
|
| 650 |
+
const owner = it.id.split("/")[0], more = it.source === "paper" ? 0 : hidden.get(owner);
|
| 651 |
+
if (more){ html += `<button class="link owner-more" data-owner="${esc(owner)}">+${more} more from ${esc(owner)}</button>`; hidden.delete(owner); }
|
| 652 |
});
|
| 653 |
if (mode === "live" && newCount) html = `<div class="divider">${lastLive?.new_since ?? newCount} new since your last visit</div>` + html;
|
| 654 |
$("feed").innerHTML = html;
|
|
|
|
| 674 |
delete drafts[k]; delete draftRows[k]; saveConfirmed(); afterMarks(k);
|
| 675 |
}
|
| 676 |
$("feed").addEventListener("click", e => {
|
| 677 |
+
const moreBtn = e.target.closest("[data-owner]");
|
| 678 |
+
if (moreBtn){ expandedOwners.add(moreBtn.dataset.owner); renderFeed(); return; }
|
| 679 |
const el = e.target.closest(".item"); if (!el) return;
|
| 680 |
const cal = e.target.closest("[data-cal]"); if (cal){ openCal(cal.dataset.cal); return; }
|
| 681 |
const k = el.dataset.k, it = byKey.get(k), tg = e.target.closest(".tg"), a = e.target.closest("[data-a]")?.dataset.a;
|
|
|
|
| 814 |
const dpr = window.devicePixelRatio || 1;
|
| 815 |
cv.width = W * dpr; cv.height = H * dpr; cv.style.height = H + "px";
|
| 816 |
const ctx = cv.getContext("2d"); ctx.setTransform(dpr, 0, 0, dpr, 0, 0);
|
| 817 |
+
Object.assign(flow, {ctx, C, lanes, W, H, laneTop, laneH, notY, mainY: 34, xEnd: W - 26, stations: stations.map(([f, s]) => [W * f, W < 560 && typeof s === "string" ? s.split(" · ")[0] : s]),
|
| 818 |
parts: [], byId: new Map(), pile: {}, begin: performance.now()});
|
| 819 |
flow.xLast = flow.stations[flow.stations.length - 1][0];
|
| 820 |
}
|
|
|
|
| 975 |
for (const v of ["cat48-v1"]) Object.assign(manualThr, store.get("thr-" + v, {}));
|
| 976 |
store.set(thrKey(), manualThr);
|
| 977 |
}
|
| 978 |
+
$("status").textContent = `Loading scored items… (catalogue ${CAT.catalogue_version})`;
|
| 979 |
refitMarks(); renderControls(); renderAll(); await reload(); pollEvents();
|
| 980 |
}
|
| 981 |
init();
|