From Errors to Proofs: Minimal-Core-Guided Repair for Neuro-Symbolic Constraint Solving
Paper • 2608.14771 • Published
AI & ML interests
Independent research lab. Agent infrastructure, evaluation, and the systems under LLMs. Correctness where the code executes. Receipts, not punditry.
Recent Activity
View all activity
dipankarsarkar
authored 3
papers about 1 month ago
dipankarsarkar
submitted a
paper to Daily Papers about 1 month ago
dipankarsarkar
posted an update about 1 month ago
Post
703
Privacy is moving ad decisions onto the device. The auction runs locally, next to the context it scores.
The budget still lives on a server.
So every device bids against the balance it saw at its last sync. Between syncs, campaigns spend money they no longer have.
I simulated how much that costs: 50 devices, 36 campaigns, about 300,000 auctions per run, 30 seeds per cell, budgets frozen before evaluation. Even pacing, second-score payment:
- Sync every tick: spend lands 18% over budget (32% with iPinYou-calibrated values).
- Sync every 10 ticks: 5.8 times the budget.
- Sync every 50 ticks: 17.7 times the budget (10.1 times with iPinYou-calibrated values).
Zero-lag sync stays within about 1%. The lag alone does the damage.
The incentive side runs the other way. With second-score payment and zero lag, 98% of auctions are manipulable (92% with iPinYou values). Switch the payment rule to critical-bid and that drops to 0 at every sync interval, in both setups.
Sync fixes the budget. The payment rule, not the sync interval, is what removes manipulability.
All 12 result sets are on the Hub, plus the 50,808 context labels every run samples from.
Paper: When Privacy Moves ML-Mediated Decisions On Device: Information and Incentive Misalignment in Auctions (2609.33312)
Dataset: skelfresearch/on-device-auction-audit
Code: https://github.com/sarkar-dipankar/on-device-auction-audit
If your ads stack moves on device, how often does the device learn its budget?
The budget still lives on a server.
So every device bids against the balance it saw at its last sync. Between syncs, campaigns spend money they no longer have.
I simulated how much that costs: 50 devices, 36 campaigns, about 300,000 auctions per run, 30 seeds per cell, budgets frozen before evaluation. Even pacing, second-score payment:
- Sync every tick: spend lands 18% over budget (32% with iPinYou-calibrated values).
- Sync every 10 ticks: 5.8 times the budget.
- Sync every 50 ticks: 17.7 times the budget (10.1 times with iPinYou-calibrated values).
Zero-lag sync stays within about 1%. The lag alone does the damage.
The incentive side runs the other way. With second-score payment and zero lag, 98% of auctions are manipulable (92% with iPinYou values). Switch the payment rule to critical-bid and that drops to 0 at every sync interval, in both setups.
Sync fixes the budget. The payment rule, not the sync interval, is what removes manipulability.
All 12 result sets are on the Hub, plus the 50,808 context labels every run samples from.
Paper: When Privacy Moves ML-Mediated Decisions On Device: Information and Incentive Misalignment in Auctions (2609.33312)
Dataset: skelfresearch/on-device-auction-audit
Code: https://github.com/sarkar-dipankar/on-device-auction-audit
If your ads stack moves on device, how often does the device learn its budget?
dipankarsarkar
updated a
dataset about 1 month ago
dipankarsarkar
published a
dataset about 1 month ago
dipankarsarkar
updated a
dataset about 1 month ago
dipankarsarkar
published a
dataset about 1 month ago
dipankarsarkar
authored 2
papers about 1 month ago
dipankarsarkar
submitted 2
papers to Daily Papers about 1 month ago
dipankarsarkar
posted an update about 1 month ago
Post
3,181
I audited one of my own evaluations. The ranking did not hold up the way I expected.
Eight open models, one task: infer the structure of a prompt. Then ask again with the identical call. Caching off.
- Agreement between repeated identical calls (mean Jaccard) ranged from 0.39 to 0.96 across models.
- Only 35 of 127 prompt-model cells were perfectly reproducible on every run.
- I bootstrapped the reproducibility ranking over prompts. The two least reproducible models kept their rank in 99% and 86% of resamples. The middle four kept theirs in 27% to 48%.
So the table reliably finds the worst model. It does not reliably find the best.
Reproducible is also not the same as correct. F1 against gold annotations ran from 0.56 to 0.99.
By the audit date, 4 of the 8 model variants had been retired (HTTP 410). The study as specified can no longer be re-run. The saved outputs are what survives, so I published all of them: every inferred structure, every run, the prompts and the annotations.
Paper: How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure (2609.30074)
Dataset: dipankarsarkar/llm-evaluation-self-audit
Code: https://github.com/sarkar-dipankar/llm-evaluation-self-audit
How many of the leaderboard rankings you rely on would survive re-running the same calls?
Eight open models, one task: infer the structure of a prompt. Then ask again with the identical call. Caching off.
- Agreement between repeated identical calls (mean Jaccard) ranged from 0.39 to 0.96 across models.
- Only 35 of 127 prompt-model cells were perfectly reproducible on every run.
- I bootstrapped the reproducibility ranking over prompts. The two least reproducible models kept their rank in 99% and 86% of resamples. The middle four kept theirs in 27% to 48%.
So the table reliably finds the worst model. It does not reliably find the best.
Reproducible is also not the same as correct. F1 against gold annotations ran from 0.56 to 0.99.
By the audit date, 4 of the 8 model variants had been retired (HTTP 410). The study as specified can no longer be re-run. The saved outputs are what survives, so I published all of them: every inferred structure, every run, the prompts and the annotations.
Paper: How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure (2609.30074)
Dataset: dipankarsarkar/llm-evaluation-self-audit
Code: https://github.com/sarkar-dipankar/llm-evaluation-self-audit
How many of the leaderboard rankings you rely on would survive re-running the same calls?
- 2 replies
dipankarsarkar
updated a
dataset about 1 month ago
dipankarsarkar
published a
dataset about 1 month ago
dipankarsarkar
authored a
paper about 1 month ago
dipankarsarkar
updated a
collection 4 months ago
dipankarsarkar
published a
Space 4 months ago
dipankarsarkar
updated a
Space 4 months ago