A working recommendation structure. A freshness gap. Semantic quality still needs real-model evidence. This dashboard separates those three conclusions.
Local audit · No deployment{{ report.tests.counts.get('skipped',0) }} skipped · {{ report.tests.counts.get('failed',0)+report.tests.counts.get('error',0) }} failures/errors.
Entire local suite, including existing tests. This count measures correctness checks, not recommendation accuracy.
The architecture is defensible. We cannot yet say the actual papers are consistently good for real readers.
No invented 8/10 score. Independent relevance judgments are absent.
Citation popularity is not live trending. Refresh can reveal unseen archive papers without discovering new releases.
A zero-citation paper can be excluded by the current remote popularity query.
Preserve distinct interests and their quotas. Add external discovery before ranking, with explicit source provenance and candidate readiness.
{{ case.save_counts|join(' / ') }} saved examples across the simulated interests.
First run: {{ case.first_page_counts[0]|join(' / ') }} papers on page one. Each color is a different simulated interest.
Real Ward, quota, heuristic and MMR functions; generated 1024-dimensional vectors. The dominant profile deliberately favors one topic.
{{ case.first_page_counts }}{{ case.limitation }}
Structural stress result: {% set failures=report.structural|selectattr('status','equalto','fail')|list %}{{ 'Some scenarios failed; inspect the runs above.' if failures else 'All three simulated profiles retained their interests across 60 seeded runs.' }} This is a controlled composition test, not an end-to-end semantic evaluation.
A Daily Papers adapter and scheduled worker now persist dated source observations in separate local SQLite. Metadata enrichment, compatible local embeddings, freshness checks, bounded retries and concurrent-worker exclusion are implemented.
Shadow evaluation only: it is not wired into the serving feed. The app now owns scheduling when HF_DISCOVERY_MODE=shadow; /healthz/discovery reports collection, preparation, and serving separately. Missing shared model dependencies preserve candidate retries. These changes have not been deployed. Production indexing and measured trend growth remain next steps. Inspect the explicitly synthetic end-to-end comparison →
Inspected public Daily Papers endpoint ↗HF can bootstrap useful candidates while ResearchIT has little traffic. Votes are an attention signal, not a save label for every user.
Keep source collection separate from the ranker. Reduce HF's contribution only when a source-removal experiment preserves fresh coverage and judged relevance. Training alone will never tell us tomorrow's releases.
{{ report.discovery_pipeline|tojson(indent=2) }}The worker writes only its separate shadow database. No production candidate indexing has occurred.
{{ report.hf_observed_sample.capture_method }} Observed {{ report.hf_observed_sample.observed_on }}. These have not been verified as indexed in ResearchIT.
| Paper | Publication date | Votes observed | What this establishes |
|---|---|---|---|
| {{ row.paper.title }} {{ row.paper.id }} | {{ row.paper.publishedAt[:10] }} | {{ row.paper.upvotes }} | Real candidate metadata; no measured trend velocity or relevance grade. |
One source snapshot shows current attention, not acceleration. HF primarily covers AI research; it cannot replace candidate sources for every ResearchIT category.
There is not enough evidence to answer yes or no. The suite separates model semantics, stored-vector integrity, retrieval coverage, and final ranking so we can locate the cause of a poor result.
{{ report.semantics.get('reason',report.semantics.get('note','')) }}
Six authored query/positive/hard-negative cases: calibration, retrieval, compression, robotics, medical imaging and safety. Real encoding only; no fallback to random vectors.
{% for r in report.semantics.get('results',[]) %}{{ r.case }}: {{ r.status }} margin {{ r.get('margin','n/a') }}
{% endfor %}On personalization features 20–30, across {{ report.model.get('trees','?') }} trees in the bundled model.
{{ report.model.get('note','Model unavailable') }} The configured scorer is {{ report.environment.scorer }}. A model file with many trees is not evidence of personalized learning.
| Runtime probe | Status | Evidence / limitation |
|---|---|---|
| {{ name.replace('_',' ') }} | {{ value.status }} | {{ value.get('reason',value.get('note','Results available in the JSON evidence export.')) }}{% if value.get('points') is not none %} · {{ value.points }} points{% endif %} |
| External probes | {{ report.live.status }} | {{ report.live.reason }} |
{{ report.environment }}
Stored-vector probes and real HTTP feed collection are executable with reachable configured services. Exact-vs-approximate probes are executable with reachable stores; independently judged ablations still need a labeled dataset.
HTTP refresh and history tests use isolated SQLite with mocked retrieval. Algorithm stress tests use real code with synthetic geometry. The source adapter uses deterministic HTTP fixtures. None is labelled as live relevance.
{{ report.tests.cases|length }} recorded cases. Live/browser marked tests excluded.
| Case | Suite | Status |
|---|---|---|
| {{ t.name }} | {{ t.group }} | {{ t.status }} |
{{ report.tests.deselected_note }} Pytest exit code: {{ report.tests.exit_code }}. This dashboard reports failures rather than hiding them.
{{ report.judged_quality|tojson(indent=2) }}{% else %}No human judgment file supplied. Precision, NDCG and relevance recall are not evaluable, rather than zero or 100%.
{% endif %}The runner accepts --judgments /path/to/judgments.json. It requires evaluation timestamps, rejects future source observations, refuses incomplete top-page labels, and reports pooled recall rather than pretending the entire corpus was judged.
.venv/bin/python scripts/run_recommendation_audit.py --live --encode
Writes HTML, JSON and JUnit results into reports/recommendations/. Uses temporary user storage; Turso replication stays disabled. External probes read public papers and configured collections. Encoding can download model weights when its runtime is installed.
Without flags, it runs local checks and marks external/semantic measurements not run. Missing prerequisites are not counted as passing quality checks.
Schedule source snapshots. Canonicalize and ingest new papers, including zero-citation papers. Track which have searchable metadata and compatible vectors.
First priorityGather dated candidate pools and reader briefs. Blindly judge current, content-only, HF-only and combined feeds. Use an untouched time window and report relevance by interest.
Release gateRecord helpful discoveries, saves, dismissals and returns with source attribution. Keep views separate from approval. Trial reviewed video explanations after paper discovery works.
After baseline evidence