Aayam Bansal's picture

Aayam Bansal PRO

aayambansall
·

AI & ML interests

Reinforcement Learning

Recent Activity

posted an update 28 minutes ago
Today we’re releasing FrontierML, a benchmark for evaluating the autonomous empirical ML R&D capabilities of AI agents. It consists of 25 open-ended ML research problems, each built around an active frontier research direction, such as credit assignment under sparse terminal rewards, self-distillation from expert reasoning when pass@32 = 0, and pretraining language models without backprop. We built FrontierML because most existing benchmarks for ML “research” give the agent a validation set or scoring function that closely tracks its final score. In practice, human researchers rarely have that. Much of empirical research is deciding how to measure progress on a problem and how far to trust those measurements. As agents are deployed to run ambitious ML R&D loops with increasingly abstract goals over ever-longer horizons, they’ll need to form their own representations of progress, often implicitly, before they can make any. Each agent works for 12 hours in Lab0, a virtual research lab we purpose-built for them. It can provision H100s from a shared cluster, generate synthetic data through model APIs, and search 1:1 clones of arXiv and Hugging Face Datasets, much like a human researcher would. The agent has to operationalize progress through its own experiments, constructing local proxies and calibrating how far to trust them, while making progress toward an objective it never observes directly. Its submitted model or method is then scored on a held-out evaluation. Across the five frontier models we evaluated, Claude Fable 5.2 performed best, with a mean normalized score of 0.523 across the 25 tasks, followed by Opus 5.5 and GPT-6 Astra. We also analyzed how agents used the lab, as a first step toward measuring research taste. We’re open-sourcing 15 of the 25 tasks along with their full trajectories. Check out frontierml.ai!
updated a Space about 1 month ago
synsci/README
published a Space about 1 month ago
synsci/README
View all activity

Organizations

Synthetic Sciences's profile picture