Spaces:
Running
Running
| # Source audit — arXiv 2603.00039 (CARE) | |
| | Field | Value | | |
| |---|---| | |
| | Title | CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation | | |
| | Authors | Jitian Zhao, Changho Shin, Tzu-Heng Huang, Satya Sai Srinath Namburi GNVV, Frederic Sala | | |
| | OpenReview | https://openreview.net/forum?id=3WPDFjZ1UT | | |
| | Source retrieved | `https://ar5iv.labs.arxiv.org/html/2603.00039` | | |
| | Retrieved (UTC) | 2026-08-01T04:20:00Z | | |
| | User-Agent | `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36` | | |
| | SHA-256 of HTML | `2e733a1609e1dd907dd839b9eed8d9fb7d88549f45d40fafbca7af94ee5e77ea` | | |
| | Bytes | 865,686 | | |
| | Official code | https://github.com/SprocketLab/CARE @ `72f5b29a822d9934d31777c10a5c38369884c9dc` | | |
| Reproduce the fetch and the hash: | |
| ```bash | |
| curl -sL -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36" \ | |
| "https://ar5iv.labs.arxiv.org/html/2603.00039" -o ar5iv.html | |
| shasum -a 256 ar5iv.html | |
| ``` | |
| --- | |
| ## Anchors, assumptions and exact quantifiers | |
| ### Table 1 — MAE, continuous scoring (Section 5.1) | |
| | Method | ASSET | FeedbackQA | Review-5K | Summarize | UltraFeedback | Yelp | | |
| |---|---|---|---|---|---|---| | |
| | MV | 31.153 ± 0.000 | 0.822 ± 0.000 | 2.608 ± 0.000 | 1.417 ± 0.000 | 0.851 ± 0.000 | 0.923 ± 0.000 | | |
| | AVG | 33.663 ± 0.000 | 0.830 ± 0.000 | 2.274 ± 0.000 | 1.394 ± 0.000 | 0.686 ± 0.000 | 1.037 ± 0.000 | | |
| | WS | 29.073 ± 0.436 | 0.793 ± 0.009 | 2.593 ± 0.052 | 1.364 ± 0.007 | 0.829 ± 0.009 | 0.977 ± 0.008 | | |
| | UWS | 33.928 ± 0.000 | 0.875 ± 0.000 | 2.602 ± 0.000 | 1.362 ± 0.000 | 0.680 ± 0.000 | 0.987 ± 0.000 | | |
| | **CARE-SVD** | **27.629 ± 0.156** | **0.730 ± 0.002** | **1.957 ± 0.018** | **1.325 ± 0.004** | **0.623 ± 0.006** | **0.694 ± 0.004** | | |
| ### Table 2 — Accuracy, classification / preference (Section 5.1) | |
| | Method | Chatbot-Arena | CivilComments | PKU-BETTER | PKU-SAFER | SHP | Summarize | | |
| |---|---|---|---|---|---|---| | |
| | MV | 0.517 | 0.691 | 0.701 | 0.698 | 0.626 | 0.600 | | |
| | AVG | 0.551 | 0.690 | 0.726 | 0.717 | 0.634 | 0.683 | | |
| | WS | 0.543 | 0.739 | 0.575 | 0.570 | 0.619 | 0.705 | | |
| | UWS | 0.507 | 0.713 | 0.703 | 0.701 | 0.629 | 0.713 | | |
| | Dawid–Skene | 0.546 | 0.735 | 0.551 | 0.548 | 0.612 | 0.705 | | |
| | GLAD | 0.510 | 0.695 | 0.697 | 0.671 | 0.644 | 0.718 | | |
| | MACE | 0.550 | 0.732 | 0.734 | **0.735** | 0.580 | 0.706 | | |
| | CARE-SVD | **0.580** | **0.778** | 0.691 | 0.690 | 0.543 | 0.695 | | |
| | CARE-Tensor | 0.564 | 0.749 | **0.779** | 0.731 | **0.695** | **0.814** | | |
| ### Prose quantifiers (Section 5.1) | |
| > "Specially, it reduces error by up to 26.8% compared to MV on UltraFeedback. | |
| > Averaged across scoring datasets, CARE-SVD yields a 17.37% relative improvement over AVG | |
| > and a 12.75% improvement over MV. | |
| > On classification and preference benchmarks, CARE achieves the best accuracy on 5 of 6 datasets, | |
| > with CARE-Tensor leading on three (PKU-BETTER, SHP, and Summarize), including a 13.4% relative | |
| > improvement in accuracy on Summarize over the strongest baseline." | |
| ### Proposition 4.1 (Section 4) — exact wording | |
| > "Assume `K_HH = diag(d_1, …, d_h)` with `d_1 > ⋯ > d_h > 0` and the columns of `K_JH` are | |
| > **orthogonal**. Then the columns of `K_JH` (equivalently, the latent directions encoded by `L`) | |
| > are identifiable from `L` up to sign and permutation. Moreover, if `K_JH` is perturbed to | |
| > `K̃_JH = K_JH + E`, letting `δ_i` denote the eigengap of `L` at eigenvector `u_i`, and `û_i` the | |
| > eigenvector of the estimated low rank matrix `L̂`, then | |
| > `‖û_i − u_i‖₂ ≲ ‖K_HH^{-1}‖₂ ‖E‖₂ / δ_i` (i ∈ [h])." | |
| **Appendix form.** Assumption D.2 states the columns of `K_JH` are **orthonormal** | |
| (`K_JHᵀ K_JH = I_h`); the proof of Theorem D.3 says explicitly "Under Assumption D.2 | |
| *strengthened to orthonormal columns*". Theorem D.4 gives the explicit constant | |
| `‖ũ_i − s_i u_i‖₂ ≤ 4 ‖K_HH^{-1}‖₂ ‖E‖₂ / δ_i + O(‖E‖₂²)`, `δ_i := min{λ_i, min_{j≠i}|λ_i − λ_j|}`, | |
| and its proof bounds `‖Δ‖₂ ≤ 2‖K_JH‖₂‖K_HH^{-1}‖₂‖E‖₂ + ‖K_HH^{-1}‖₂‖E‖₂²` **and then uses | |
| `‖K_JH‖₂ = 1`, which holds only under orthonormality.** | |
| ### Theorem 4.2 (Section 4) = Theorem D.5 | |
| > "… for any `η > 0`, with probability at least `1 − 2e^{−η}`, | |
| > `max_{i≤h} ‖û_i − u_i‖₂ = O( sqrt(η/n) · 1/(ξ(T) δ) )." | |
| Proof chain (Appendix D.6): Chandrasekaran et al. (2012) Thm 4.1 gives | |
| `‖L̂_n − L*‖₂ ≤ C₁ sqrt(ε/n)/ξ(T)`; CARE then invokes Yu et al. (2015), Corollary 3, for | |
| `‖û_i − u_i‖₂ ≤ 2^{3/2}‖L̂_n − L*‖₂/δ_i`; composing gives the stated rate, inverted to | |
| `n ≥ 8C₁²η/(ξ(T)²δ²α²)`. | |
| **Literal discrepancy audited in Claim 5.** D.5 defines | |
| `δ=min_(1≤i<j≤h)|λ_i-λ_j|`, using only the `h` positive eigenvalues, and writes the | |
| ordinary vector distance `‖û_i-u_i‖₂`. D.4 immediately above instead defines the | |
| full-spectrum gap `δ_i=min{λ_i,min_(j≠i)|λ_i-λ_j|}` and includes sign alignment | |
| `‖ũ_i-s_i u_i‖₂`. Yu Corollary 3 requires adjacent population gaps; for the last positive | |
| eigenvector of rank-h `L*`, this includes `λ_h-λ_(h+1)=λ_h-0`. D.5's proof first invokes | |
| that `δ_i`, then replaces it with the weaker positive-eigenvalue-only `δ`. | |
| | Frozen cited source | SHA-256 | Audited anchor | | |
| |---|---|---| | |
| | Yu, Wang & Samworth, arXiv:1405.0680 | `6f4bc8b6a3cc768097102e54379ca894e92302ab5f9a9516fb55b71116ae0cd4` | Corollary 3 uses adjacent population eigengaps, including the gap to zero | | |
| | Chandrasekaran, Parrilo & Willsky, arXiv:1008.1290 | `3c7c5ae5941a5bee218bb5e38c06b70d2120f00ce0b1623a048fae09a5706716` | Theorem 4.1 minimum-signal condition scales with the noise/regularization level | | |
| The exact sign and eigengap families and the two-model Gaussian Le Cam lower bound are | |
| recomputed by the claim module and independently by a direct 2x2 eigendecomposition and | |
| trace/log-determinant KL calculation. | |
| ### Theorem 4.3 (Section 4) = Theorem D.9 | |
| > "… there exist **universal constants** `C₁, C₂ > 0` such that if | |
| > `n ≥ C₁ σ_max⁶/(δ² π_min²) · p log(p/ε)`, then with probability at least `1 − ε`, | |
| > `max_{q,c} ‖μ̂_qc − μ_qc‖₂ ≤ C₁ (σ_max³/δ) sqrt(p log(p/ε)/n)` and | |
| > `max_{q,c} |π̂_qc − π_qc| ≤ C₂ sqrt(p log(p/ε)/n)." | |
| Proof chain (Appendix D.6): eq. (8) `‖M̂ − M‖_op ≤ C σ_max³ sqrt(p log(p/ε)/n)`; | |
| eq. (10)+(12) mean error `≤ √3 C_dec ‖E‖_op/δ`; eq. (11) weight error `≤ C_π ‖E‖_op`. | |
| ### Computing resources declared by the paper (Appendix E.2) | |
| > "We used a server equipped with an NVIDIA A100 (40GB). Generating LLM judge outputs took up to | |
| > 3 hours per dataset." | |
| This is the concrete blocking capability for the eight Table 1 / Table 2 columns whose judge-score | |
| matrices were not released. | |
| --- | |
| ## What the official repository ships | |
| `SprocketLab/CARE @ 72f5b29` contains `judge_outputs/` for exactly three datasets: | |
| * `judge_outputs/fully_gaussian/asset/` — 11 judge CSVs (Table 1, ASSET) | |
| * `judge_outputs/gaussian_mixture/civilcomments/` — 24 judge CSVs (Table 2, CivilComments) | |
| * `judge_outputs/gaussian_mixture/allenai_preference_test_sets_pku_better.tar.gz` (Table 2, PKU-BETTER) | |
| Its `README.md` documents two run entrypoints, `scripts/fully_gaussian_main.py` (Table 1) and | |
| `scripts/gaussian_mixture_main.py` (Table 2). Both are pure CPU: `torch`/`vllm`/`transformers` are | |
| needed only by `scripts/save_judge_outputs.py`, the judge-generation step. | |