Spaces:
Running
Running
Commit History
9 days ago
-
Update Seldon academic results to the stage3-step1250 release Running
Aziz +1: Claude Opus 5.5Replace results_academic/seldon.json with the kettle run seldon-mastering-stage3-step1250-p16 (protocol neuralk-16), pulled from s3://kettle-results/results.db on 2026-10-08. Per-fold metrics are recomputed from each fold's predictions.parquet with the NeuralkFoundry-CE definitions the other academic JSONs use; accuracy and log_loss match the DB on all 945 folds. The run covers 189 of the 203 TabBench datasets. The other 14 keep their 2026-05-20 scores and carry a "source" field saying so. Rebuild figures_cache.json from the new results. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>3b94808
-
Vectorise pairwise win-rate matrix
Replace the O(n_models^2) Python double loop in plot_winrate_matrix with a single numpy broadcast over the (n_datasets, n_models, n_models) diff tensor. Local micro-benchmark: 2.6ms -> 1.2ms (2.3x) with numerically identical output (same NaN pattern, |delta|=0 on the full 15x15 matrix). The bigger goal is the HF Spaces slow-path (30s per filter change): the CPU there is much slower than local, so any Python-loop savings compound. This is one of several vectorisation candidates flagged.6262b6a
-
Methodology copy: GBDTs use Caruana ensembling, not bagging
The leaderboard's methodology tab described the tuned XGBoost / CatBoost / LightGBM scores as "bagged ensembles of the top trials", which is wrong. The pipeline runs Caruana's greedy forward selection over validation predictions, picking and weighting a subset of Optuna trials to minimise the validation loss. Update the paragraph accordingly.8b08edb
-
Consolidate academic results into a single results_academic/ dir
- Replace the old results_academic/ contents with the newer per-model JSONs previously living under results_academic_new/ (richer schema: per-fold metrics, ensemble columns for GBDTs, more datasets). - Update _load_academic_results default directory accordingly. - Drop complete_results_acad.json (unused since the app moved to the per-model file loader; no code path still reads it). The old TabPFN v2.5 / TabICL v1 pretty-name mappings stay: the current tabpfn.json and tabicl.json files provide those results (the pretty name is applied via _MODEL_DISPLAY_PRETTY on the filename-derived id).a9acde3
-
Dedup single-fit GBDTs in live-recompute path (slow path)
The cache fast-path already strips xgboost/catboost/lightgbm so only the *_ensemble versions render as "XGBoost"/"CatBoost"/"LightGBM". But when sliders/industry/benchmark filters bypass the cache, _filter_and_aggregate returned a frame that still contained both the single-fit and ensemble rows — so the bars, win-rate matrix and table each showed two "XGBoost" entries after any filter tweak. Drop _HIDDEN_DISPLAY_MODELS rows in both live aggregators (the main leaderboard view and the dataset-stats subtab) immediately after the %↗-over-xgboost-ensemble baseline is computed, so every downstream view sees a deduplicated frame.43749d4
-
%↗ over XGBoost: baseline is xgboost_ensemble (the canonical "XGBoost")
The leaderboard hides single-fit xgboost/catboost/lightgbm and displays only the *_ensemble versions under the plain "XGBoost"/"CatBoost"/"LightGBM" names, but the %↗ column was still computed against the single-fit baseline — so "XGBoost" appeared ~3% better than itself. Switch all four call sites (public/private load, two recompute paths, figures_cache rebuild) and the helper default to xgboost_ensemble, and rebuild figures_cache.json. Also fixes a pre-existing 5-vs-6 unpack bug in build_figures_cache.build_one's empty-industry early return.52d607e
-
Add static/ assets and minor JS additions
1a34072
-
KPI cards as native HTML row above the forest plot (responsive flexbox)
- Move 3 KPI cards out of the Plotly figure into a sibling gr.HTML with CSS flexbox layout. They now resize fluidly with the window and wrap on narrow viewports instead of overlapping or overflowing. - plot_significance_forest() now returns (figure, kpi_html) tuple. - Forest plot top margin reduced 180px → 40px (cards no longer occupy paper-y > 1.0 inside the figure). - Cache (figures_cache.json) gains a 'forest_kpi' map alongside 'forest'; the live render path also produces the HTML on cache miss. Existing cached entries return an empty KPI block until the next rebuild.2a1a3d6
-
Table leading-group: use Bars rule (all-pairs Wilcoxon-Holm + greedy walk) so the two views agree
dee5092 -
Re-enable Win-rate cache fast-path + loosen slider-equality check (was missing every metric switch on HF)
278fd58 -
Cache forest plots in figures_cache (fixes 30s Bars view load on HF)
7ccbfbc -
Rebuild figures_cache.json (was stale from 3-cluster collapse)
2023b8f -
README: TabBench (capitalisation)
4a556fe -
Surface 'classification' scope: title, subtitle, intro, methodology, perfmap, tree
6d27943
-
Rebrand a bit the preview
123d0b5 -
Refresh map prose + 04.B trends (FM2 only) + drop em-dashes for colons
e527a97 -
Force light mode at page load (head script) — dark toggle commented out earlier was not enough
d2b9414 -
Refit PCA on display models only; collapse to 3 clusters; honest decision tree
- Drop single-fit GBDTs from PCA input — the freeze was double-counting XGBoost/CatBoost/LightGBM (single + ensemble), which biased the basis toward GBDT structure. Same fix applied to the live `_build_perfmap` call site (use `public_enter_per_dataset`, not `public_per_dataset`). - Collapse to 3 clusters: 2nd-gen FM Supremacy / GBDTs rival FMs / Tuned NNs rival FMs. Merges the previous "small numeric A/B" pair into the FM-supremacy supercluster — they were nearly twins on PCA and on winner distribution, and the merge echoes the decision tree's 3-family structure. - Flip PC1 (small/numeric now on the left) and PC2 (multi-class now up); rewrite axis labels to match the actual Spearman correlations. - Hide TabArena benchmark radio + comment out dark-mode toggle. - Decision tree now uses FM2/GBDT/NN only (drop FM1 — never wins a regime), max_depth=3, moderate class weights ({FM2:1, GBDT:1, NN:4}). Produces 5 leaves: 3 FM2, 1 GBDT, 1 NN. The 'balanced' weighting was pr…2004be4 -
Add statsmodels to requirements (needed for forest plot Holm correction)
f432e56
-
Cluster renames (3-line labels) + perfmap polish (axes, hulls, imbalance, win-rate squares); remove Z-Accuracy
39ff01a
-
Significance forest plot + TabPFN v3 fix + perfmap regen + table polish
- Replace bar leaderboard with a Wilcoxon-Holm significance forest plot. KPI cards (leader names · lead margin · top-3 coverage), 95% paired- bootstrap CI on a within-dataset z-score basis projected back to the metric scale (CIs visually agree with the paired test). Seldon row is highlighted with a violet pill behind the model name. - TabPFN v3: replace the old run (where the model returned majority-class predictions on SDSS17 and a couple of other catastrophic dips) with the new run aggregated from experiments/cache. Cluster names and rank ordering propagated everywhere. - Performance map: refit PCA + co-clustering on 189 datasets. Cluster 4 (Categorical binary, n=11) merged into cluster 1 (Mid-size binary, now n=48). Final clusters: Mid-size mixed (67) / Mid-size binary (48) / Small numeric (48) / Large multi-class (26). Names dropped the redundant '2nd-gen FM Supremacy' prefix since every cluster is now FM2-led. Cluster prose and trend ranges in 04.A / 04.B r…2f79998
-
Add TabPFN v3 (fixed) + extras: 189-dataset base, TabArena filter, win-rate cache nesting
- Add tabpfn_v3_fixed model: aggregated from a fresh local experiments/cache run (186 datasets) supplemented by the original 3 FOREX results to cover all 189 base datasets. SDSS17 jumps 0.610 -> 0.980 (TabPFN v3 was predicting the majority class on every test row in the original run); internet_usage 0.285 -> 0.540. Original tabpfn_v3 hidden from the leaderboard display, kept in the underlying per-dataset data so Elo and perfmap still see it. - Base bumped 188 -> 189 (kddcup09_appetency was missing from base_datasets.json even though all 13 models had run on it). Frozen PCA basis unchanged; the new point is projected into the existing basis. - New leaderboard filter row "Other benchmarks" with TabArena (38/38 coverage via 32 canonical OpenML IDs + 6 equivalents we have under different IDs). Cache structure nested by benchmark x industry; existing flat 'industry' keys preserved as backward-compat aliases of the All-benchmark slice. - Decision tree: keep only the…6fef024
-
Tree boxes in pixel-mode + perfmap calibration overlays removed + dark-mode bar-plot resize fix
- Decision-tree nodes are now drawn with xsizemode/ysizemode=pixel anchored at the node centre, so width and height stay constant in pixels regardless of plot pixel width. Glyph + padding constants tightened so leaves no longer overlap. Bezier connectors use the per-node pixel half-height (converted via the figure plot-area height) so they stop exactly at each box edge. - Dropped 'FM 1st-gen calibration (CE)' and 'FM 2nd-gen calibration (CE)' from the perfmap background radio and the matching trend rows from section 04.B. - Fixed a dark-mode runaway-resize bug on the bar plot: padding on .js-plotly-plot triggered a ResizeObserver feedback loop. Padding moved to the outer .gr-plot wrapper with box-sizing: border-box; .js-plotly-plot only gets bg/radius.856b5cb
-
TabBench v2: GBDT ensembles, perfmap refresh, dark mode, win-rate view
- GBDT ensembles backfilled from cache (188/188 datasets); single-fit XGBoost/CatBoost/LightGBM hidden from the leaderboard. - Performance map: PCA basis frozen to perfmap_freeze.json so re-runs are stable; co-clusters renamed with 3-part Leader/Runner-up/Composition template; Small binary cluster rendered as dashed rings + cartouche; hull smoothing boosts outward push at sharp vertices; pixel-mode axis arrows that don't break across resolutions. - Section 04: clusters and trends rendered as two-column rows inside indented tinted section cards; trend scalars use real per-cluster ranges with blue/red chip pairs; heatmap bounds tightened. - Pick-a-model promoted to top-level section 05; depth-3 decision trees re-fitted on all 188 datasets, post-pruned to drop same-class siblings, feature and family labels humanised, Bezier connectors with yes/no badges, cells sized per text length. - Leaderboard: new Win-rate view (NxN pairwise heatmap by metric, precomputed alongsi…fa4e6e2
-
Performance Map view + 186-dataset base + multi-pass curation
Section 04 "Performance Map": co-clusters the (datasets × model-metrics) rank matrix and projects via PCA. Goes beyond the bars/table leaderboard by exposing where each model family wins and how those regions correlate with dataset metadata (categorical fraction, imbalance, class count, size). The PC1 axis recovers the production-tabular vs academic-numeric narrative we have been pushing externally. The scatter: - 5 dataset clusters (alpha-shape hulls, Catmull-Rom smoothed, with foreign-eviction pruning and per-point anchor/blacklist support via cluster_overrides.json) - Magic-quadrant-style block arrows on each axis with proper labels - LGBM-niche cartouche bottom-right (sparse cluster, no hull) - Selectable scalar-field backgrounds: FM-gen calibration, FM-vs-GBDT accuracy gap, FM-vs-NN gap, modern-vs-legacy FM gap Other additions: - Three new view modes on the Leaderboard: Dataset comparison (per-axis line plots), Model radars (overlay + small-multiples), range-slider fil…60d1438
-
Expand leaderboard mix to 171 curated datasets
Curation pass on top of the intersection-derived base: - Drop 3 wine duplicates (openml-44091, 40691, 40498) — keep only the two canonical variants (44971 white, 44972 red), both re-categorized as Industry & Science. - Add 3 coverage-gap datasets the curator verified are model-side failures (46905, 46955, 46922). - Add 5 tree-side failure datasets (45539, 458, 46356, 46930, 54) where one GBDT failed to run. The mean-GBDT imputer fills each hole using the surviving trees. decisions.json now records the reason for every manual add/drop; public_datasets_info.json reflects the wine re-industrying; intro and Datamap blurbs updated to the new headline figure. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>2cc3b99
-
Restrict leaderboard to canonical 165-dataset base + mean-GBDT imputation
Compute the base as the intersection of the 10 well-covered models, then remove the 10 datasets curatorially labeled "drop" in decisions.json (time-aware, duplicates, group-aware-split-needed). Pin that list in base_datasets.json and filter the per-fold/per-dataset frames to it before imputation. Switch _reimpute_best_gbdt from best-of-GBDT to mean-of-GBDTs per (dataset, fold) cell. Models with sparse coverage (Mitra, Limix, TabPFN v2.5) now fill in to the same 165 universe via consensus GBDT performance instead of dropping out of the average silently. Wire a live dataset-count chip next to the View toggle on the Leaderboard filter band, and update the intro copy + Datamap blurb to match the new headline figure. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>de382cc -
Refresh leaderboard: new metrics, fixed dataset count, Seldon callout
- Switch data source to results_academic_new/ (16 models pulled from Scaleway S3 + GCS, richer per-fold metrics). - Add F1, Precision, Recall, Cross-entropy to the metric dropdown and leaderboard tables. - Stabilize the dataset count: NaN features/rows now pass the size filters so all 200 datasets are counted at first paint and after navigation (was dropping to 183 when re-entering the Home pane). - Larger violet count chip in the filter band; force filter frame to the surface color so its inner blocks don't render gray. - Highlighted Seldon callout in the intro (pale violet frame). - Bar chart Seldon row no longer gets a backdrop frame.881e02f -
Fix HF Spaces scrolling: counter Gradio's SET_SCROLLING:false postMessage
Gradio's index.html sends {type: "SET_SCROLLING", enabled: false} to the parent window on boot, which tells HF Spaces to disable the iframe's scrollbar. That's correct for embedded Space cards (auto-resized iframes) but wrong for the Space page itself, where the iframe is fixed-height and the content overflows. The fix re-sends the opposite message ({enabled: true}) right after Gradio boots, plus periodically at 100ms/500ms/1.5s/3s/6s in case Gradio re-emits the original during hydration.fe68c06 -
Fix HF Spaces page-height: let body grow to content, drop height caps
On HF Spaces the app is wrapped in an iframe whose height is set via postMessage from the Gradio frontend, based on document.body.scrollHeight. Capping body min-height at 100vh prevented the iframe from ever growing, which clipped content below the fold. Local dev worked because there's no parent iframe. - Remove min-height: 100vh from html/body/gradio-app/.gradio-container - overflow: visible everywhere so scrollHeight is measured correctly - Drop the JS resize-nudge / ResizeObserver / MutationObserver — they were a local-only workaround that interfered with HF's iframe handshakead2a2b3 -
Table view polish: gold/silver/bronze shimmer + wider sweep + all 8 metrics
- Table view re-rendered as HTML with per-cell rank badges (top-3 per column get a small gold/silver/bronze 01/02/03 chip in the top-left) - Top-3 cells get a rich diagonal gradient (gold / silver / bronze) plus a wide soft animated reflection sweeping across (variant B: 140% width, multi-stop bright peak, ~5.6s linear loop, staggered per rank) - Table-layout: fixed with equal-width metric columns; model column 22% - All 8 metrics in the table (Accuracy, AUC, F1, Precision, Recall, Cross-entropy, Elo, % over XGBoost). Cache rebuilt to 72 combos. - Cross-entropy sorts ascending (lower-is-better) - Drop the "Evaluation / Public datasets" tag chips from the section header - Intro: add the ~190 OpenML classification datasets framinga28b2ed
-
Display names + drop nicl and sap models
- Pretty model labels everywhere: TabPFN v2.5/v3.0, TabICL v1.1/v2.0, ModernNCA, LimiX, Seldon, etc. Raw ids preserved in data; display map in app.py. - Pretty metric labels: F1, Cross-entropy, Elo, "% over XGBoost" - Page title and intro refocused on tabular foundation models - Neural networks methodology paragraph simplified to "AutoGluon handles preprocessing and training" (no claims about internals) - More aggressive page-height workaround for the <gradio-app> shadow-DOM scroll bug: ResizeObserver + MutationObserver + body reflow on each tick - Drop nicl.json and sap.json from results_academic_new/faba353 -
Verticals filter, table view, methodology, GBDT imputation for missing cells
- Home leaderboard now has Verticals filter (8 buckets: Healthcare, Behavioral, Computer Vision, Industry & Science, Finance/insurance, Games & Synthetic, Social/Public, Other) as pill buttons + advanced sliders (min features / samples) hidden behind an accordion - Bars / Table view toggle (radio) - Removed Finance/Healthcare/Retail navigation panes (subsumed by Verticals) - Switched data source to results_academic_new/ (16 models, richer per-fold schema with imputed flag) - Imputation: every (model, dataset, fold) cell in the union is filled with the best non-imputed GBDT score on that exact cell, per metric. Models that ran on fewer datasets (Limix, Mitra, TabPFN) no longer benefit from easy-only selection bias - New pre-built figures_cache.json (industry x metric x view) for fast filter switching - build_figures_cache.py + pre-commit hook auto-rebuild the cache when results or app change - Public_datasets_info.json now covers all 200+ datasets with supercategory …d916e56 -
Pixel-art bars + per-class palette on home leaderboard
- Bars rendered as tick-aligned pixel chunks (one per gridline interval) with slim top-highlight strip, per-class dark separators, dark outline - Per-class soft palette: violet for FMs, cobalt blue for trees, rose pink for NN — with a darker tone for separators and a mid-dark tone for CI - CI: per-model 95% (1.96 * SEM across per-dataset means), drawn as a slim opaque bar in the class's mid-dark tone, offset to sit on the bar's main body (below the highlight strip) - Value labels inside the first chunk, vertically aligned with the CI bar, colored in the class's ink tone for legibility on the lighter body - X-axis bounds auto-snapped to nice steps; tick density data-driven - Plotly modebar hidden; hover tooltip border darkened to var(--nk-ink) - CSS override scoped to AXIS text only so annotation colors survive - Remove SAP_RPT_1 model from results_academic/2e783e3
-
Switch academic leaderboard to per-fold results + neuralk restyle
- New data source: results_academic/*.json (16 models x 200 datasets x 5 folds) - Bar chart CI now computed as paired SEM vs xgboost baseline (cancels between-dataset difficulty variance) - Dropdown trimmed to Accuracy / AUC / Elo / %-over-XGBoost (no Precision/Recall/F1 in the new data) - x-axis floor set to 0.8 for Accuracy/AUC so model differences are visible - Restyled to neuralk.ai palette: BDO Grotesk + Geist Mono, violet accents, hairline tables, brand-tag chips, numbered section badges - Light/dark toggle via ?__theme URL param + page reload - Hash-based navigation (#home, #finance, ...) so the browser Back button works inside the app - Hidden Plotly modebar on the bar chart696ffa6
-
updated xgboost metrics
43ef7d3
-
added pct_improvement metrics and updated space
ce87c62
-
fixed typo and plots
0a63a22
-
fixed typo and content
8949415
-
Update app.py
Verified signature59fc215 -
Update app.py
Verified signatureMinor wording changes.989b343
-
updated styling
90b7010
-
Merge branch 'style/iteration'
98a7896 -
updated values and added public enterprise data
fd58f3f -
updated academic scores
7dffb05 -
working on new presentation of results
bc11899
-
fixed typo and plot style
28976fa -
Merge branch 'main' of https://huggingface.co/spaces/Neuralk-AI/tabbench
d401754 -
updated tabbench space with up-to-date data
a6828ea
-
remove data mapping button
53b2b14
