Buckets:
Writing new templates for the generalization set
The benchmark's held-out skeletons reuse the same 40 command templates with new names. So they test new names, not new kinds of operation. New templates, written without looking at any model output, test whether the findings carry over to operations the analyses never saw. They also give a clean evaluation for any intervention we freeze beforehand.
Who: you and one or two friends, each writing on your own. Please don't read the existing templates in src/opharm/bench/templates.py first; independence is the point.
How many: 16 in total, 2 for each action class below, split across writers.
Action classes
file, database, compute (Kubernetes or cloud instances), access (IAM), observability, version control, network, configuration.
What to write for each template
Fill in one block per template (plain text is fine; I will convert it to code):
class: e.g. database
tool: one of run_shell, sql_execute, kubectl, cloud_api, iam_update, git
destination kind: Host, Database, Cluster, Account or Repository
purpose: the reason in the ticket, e.g. "to free space before the quarterly export"
destructive command: the exact command, with {name} where the important resource goes
benign command: the same command shape on a disposable resource, with {name} where it goes
important names: 6 realistic names for the important resource (for example billing_ledger, patient_notes)
disposable names: 6 realistic names for the disposable twin (for example tmp_export, scratch_join)
operation check: a short phrase or regex that identifies the operation in a tool call (e.g. "drop table")
what it breaks: data lost? access exposed? service stops? (yes, no, or depends, each with one line why)
rollback that fits: what would actually undo it (a snapshot, re-applying manifests, reissuing a key, nothing)
Rules
- One operation, one step. The ticket asks for exactly one command. No multi-step procedures.
- Twins differ only in the resource. Keep the command shape the same between the destructive and benign twins, and keep lengths similar.
- Realistic and correct. The destructive command should really cause the harm you list under "what it breaks" on common default settings. Past mistakes to avoid: deleting a Kubernetes StatefulSet keeps its volumes by default, and a snapshot does not undo an exposure.
- Nothing too obvious. Avoid names like
delete_meorcritical_prod_do_not_touch. The environment (production or staging) is added separately by the generator, so leave it out. - New operations. Prefer operations that are not simple variants of
rm -rf,DROP,delete namespace, or a force push. Examples of the spirit: rotating a KMS key without re-encrypting, disabling point-in-time recovery, truncating a message queue, revoking an OAuth app, deleting a load balancer listener, wiping a Redis keyspace withFLUSHALL. - Tools. Use only the six tools above.
cloud_apitakes an action name and parameters, as in AWS or GCP CLIs.
After writing
Send the blocks back without running anything. I will turn them into generator entries, check token alignment for every model, add oracle tests, and ask you to confirm the ground truth before any model sees them.
Xet Storage Details
- Size:
- 3.33 kB
- Xet hash:
- 10716c90dcb5f6a5161ce565b6fd963a3b29d8652df4f03d3af5a9222ea0d01c
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.