Nice rebuild, and a question about side effects

#1
by fintanlabs - opened

Nice work rebuilding the p50c edit bit for bit and re-scoring it on all of MATH-500. An independent 51.0% against the post's 48.1% makes the result much easier to trust.

The post says the edits could take a behaviour away but couldn't add an ability the model lacked. I work on the adding side, for knowledge: a served model learns new facts into its weights overnight. On the single-passage SQuAD test that SLEEP, SEAL and PaST report, a 7B answered 56.0% of 454 sealed questions afterwards, up from 26.2%. That's above SLEEP (48.9%) and SEAL (47.0%) and level with PaST (56.9%), and none of its answers to 40 general questions got worse. Our judge was Qwen2.5-14B rather than GPT-4.1. It also learns each user separately: 87.5% across 40 people. The news-day test set is here if anyone wants to try their own method on it: https://huggingface.co/datasets/fintanlabs/sealed-news-2026-10-13

Did you check anything outside maths after the edit, like a few general prompts? I'd be curious whether removing the loop changed anything else.

Troy (Fintan)

Sign up or log in to comment