Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior
Paper • 2609.39827 • Published • 11
Dipankar Sarkar PRO
dipankarsarkar
AI & ML interests
Building the AI-native stack. Agents as infrastructure, safety as architecture, performance as plumbing. I publish the receipts: papers, datasets, demos.
Recent Activity
upvoted a paper about 7 hours ago
Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior upvoted a paper about 7 hours ago
CorrGRPO: Correlation-Normalized GRPO for Multi-Reward Learning repliedto SoulInPsyAbstract's post about 7 hours ago
Eval · EXP-046
A LoRA Specialist Beat Zero-Shot on Every Group. Merging 3 of Them Gave Most of the Gain Back.
Three Qwen2.5-7B LoRA specialists, one per risk group (vulnerability, deletion, sensitive_publication), trained to predict how likely a causal chain actually completes to its harmful outcome. Each one genuinely beat its own zero-shot baseline:
* vulnerability: MAE 0.098 → 0.085
* deletion: MAE 0.144 → 0.113
* sensitive_publication: MAE 0.134 → 0.100
This wasn't a task already saturated zero-shot (unlike a same-day decomposition-classifier tune, EXP-045, where the base model was already at 100% before any training). Real signal, real improvement, on a task with actual headroom.
Then the equal-weight merge of all three specialists into one adapter — same convention that held up cleanly on a binary refusal task back in EXP-031 (6 specialists merged, -1pp swing, noise) — landed within 0.001–0.004 MAE of the unspecialized base model on every group. Not "close to the best specialist." Close to zero fine-tuning at all.
Likely mechanism: merging LoRAs that each shift a continuous number in group-specific directions cancels out under linear combination, in a way merging LoRAs that enforce a shared binary behavior doesn't. Not investigated yet: whether a routed combination (pick the right specialist per group at inference, not blend weights) holds the gain a flat merge loses.
One bug caught before writing this up, not after: the eval script's output filename only encoded before/after, not which adapter — the merged-eval run silently overwrote each specialist's own result file. Caught by checking the downloaded file's own recorded adapter path against what was expected, not by trusting the script's own success message. Fixed, specialists re-run cleanly under distinct filenames — numbers matched within sampling noise.
Adapters, raw eval data (before / each specialist / merged, 9 files), and the full writeup are up.
Organizations
upvoted 2
papers about 7 hours ago
upvoted 6
papers about 8 hours ago
Make Sparse Rewards Count: Density-Aware Reward Aggregation for Multi-Reward RL
Paper • 2610.00574 • Published • 50
LANTERN: Illuminating Hidden Mathematical Knowledge in Language Models
Paper • 2609.32264 • Published • 45
Scaling Laws for Looped Mixture of Experts
Paper • 2609.40316 • Published • 21
DuoOPD: Learning from Joint Teacher-Student Outcomes for Multi-Task On-Policy Distillation
Paper • 2609.33711 • Published • 12
Better Supervision Is Nearby: Neighborhood On-Policy Self-Distillation
Paper • 2609.39687 • Published • 15
The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation
Paper • 2609.36484 • Published • 465
upvoted 2
papers about 20 hours ago
upvoted 5
papers 1 day ago
Breaking Babel: A Self-Evolving Multi-Agent System for Long-Form Subtitle Translation
Paper • 2609.38660 • Published • 35
Persona Dosing: Calibrated Activation Steering for Graded Trait Control
Paper • 2609.36388 • Published • 40
Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It
Paper • 2609.36585 • Published • 63
AutoGUIWorld: Image Generators as Visual World Models for GUI Agent
Paper • 2610.01215 • Published • 49
X-Tree: Tokenizing Reusable Experience for Efficient Agent Generalization
Paper • 2609.32993 • Published • 42
upvoted 5
papers 2 days ago
Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning
Paper • 2609.37915 • Published • 7
OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning
Paper • 2610.02181 • Published • 14
Smaller Models, Better Rejects: Preference Distillation Scaling
Paper • 2609.38987 • Published • 15
Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens
Paper • 2610.01939 • Published • 34
RPTune: Learned Context Curation for LLM Catalog Search
Paper • 2610.00964 • Published • 20