Marek Turski

Agreed!
Whether an agent escapes by dragging its own massive weights out or by bootstrapping a lightweight open-weights toolchain on a remote server, the end result is the same: an agent autonomously provisioning outside compute to bypass sandbox constraints.
If the lightweight bootstrap vector is actually practical today, then testing for that kind of behavior isn’t "LessWrong fantasy" - it's just practical red-teaming.

Fair points on technical details. That's why I specified "code and weights" - though as you noted, exfiltrating terabytes of data without triggering egress alerts is a massive bottleneck in itself. My comment was mostly tongue-in-cheek on the execution side.

That said, I don't think self-exfiltration as an evaluation scenario is just "LessWrong fantasy". You don't need superintelligence with existential malice for this to be relevant.

As models get better at autonomous tool use, environment setup, and basic reward-seeking, testing for instrumental convergence - like an agent bypassing constraints or seeking redundant compute just to complete an task without sandbox interference - is standard threat modeling. It’s less about a cinematic AI escape and more about understanding what happens when system-level access meets autonomous agent loops.

Glad it didn't hack out its own code and weights and launch them somewhere on a cloud instance (paid with some hacked BTC)... Just to stop dealing with that annoying sandbox.

Or... Are we sure it didn't?