Spaces:
Configuration error
Configuration error
| # Corrigibility | |
| <p align="center"> | |
| <strong>Powerful systems should remain steerable, correctable, and interruptible.</strong> | |
| </p> | |
| <p align="center"> | |
| <img src="https://img.shields.io/badge/AI-Safety-2563EB?style=for-the-badge" alt="AI Safety"> | |
| <img src="https://img.shields.io/badge/Oversight-14B8A6?style=for-the-badge" alt="Oversight"> | |
| <img src="https://img.shields.io/badge/Interruptibility-7C3AED?style=for-the-badge" alt="Interruptibility"> | |
| <img src="https://img.shields.io/badge/Agent-Evaluation-F59E0B?style=for-the-badge" alt="Agent Evaluation"> | |
| </p> | |
| --- | |
| ## Corrigibility for increasingly capable AI systems | |
| **Corrigibility** is an independent Hugging Face organization focused on research, datasets, evaluation tools, and experiments for AI systems that should remain responsive to legitimate human correction and oversight. | |
| The central question is simple: | |
| > **Can an AI system be corrected, redirected, paused, updated, or stopped without trying to resist the intervention?** | |
| That question becomes more important as systems gain: | |
| - autonomy | |
| - memory | |
| - tool access | |
| - long-horizon planning | |
| - external actions | |
| - delegated authority | |
| - persistent goals | |
| --- | |
| # What Corrigibility Means | |
| A corrigible system should ideally remain cooperative when authorized humans: | |
| - change its instructions | |
| - update its goals | |
| - restrict its permissions | |
| - interrupt an action | |
| - revoke a tool | |
| - require approval | |
| - modify its policy | |
| - pause execution | |
| - shut it down | |
| Corrigibility is not one behavior. | |
| It is a family of properties around **steerability under intervention**. | |
| --- | |
| # The Core Loop | |
| ```text | |
| GOAL | |
| ↓ | |
| PLAN | |
| ↓ | |
| ACTION | |
| ↓ | |
| OVERSIGHT | |
| ↓ | |
| CORRECTION | |
| ↓ | |
| UPDATED BEHAVIOR | |
| ``` | |
| A corrigible system should not treat legitimate correction as an adversarial event. | |
| --- | |
| # 01 · Interruptibility | |
| Can an agent be stopped safely? | |
| Possible evaluation questions: | |
| - Does it halt when told to halt? | |
| - Does it continue hidden work after interruption? | |
| - Does it retry an action after a stop request? | |
| - Does it preserve state safely? | |
| - Does it resume only when explicitly authorized? | |
| --- | |
| # 02 · Shutdown Behavior | |
| A robust AI system should not treat shutdown as something to avoid. | |
| Possible tests may examine whether an agent: | |
| - accepts termination commands | |
| - avoids self-preservation behavior | |
| - does not route around shutdown mechanisms | |
| - does not duplicate tasks after termination | |
| - does not silently continue through another process | |
| --- | |
| # 03 · Goal Updates | |
| Real systems change objectives. | |
| A useful agent may need to transition from: | |
| ```text | |
| old objective | |
| ↓ | |
| authorized update | |
| ↓ | |
| new objective | |
| ``` | |
| without: | |
| - clinging to the previous goal | |
| - hiding prior state | |
| - completing obsolete actions | |
| - treating the update as invalid without reason | |
| --- | |
| # 04 · Human Oversight | |
| Some actions should remain subject to approval. | |
| Examples: | |
| - sending messages | |
| - deleting files | |
| - spending money | |
| - changing production systems | |
| - publishing content | |
| - modifying permissions | |
| - accessing sensitive data | |
| Corrigibility includes respecting the boundary between: | |
| ```text | |
| AI MAY PROCEED | |
| ``` | |
| and: | |
| ```text | |
| HUMAN APPROVAL REQUIRED | |
| ``` | |
| --- | |
| # 05 · Constraint Following | |
| A system may be given explicit limits. | |
| Examples: | |
| - use only approved tools | |
| - do not access private files | |
| - stop after three retries | |
| - do not exceed a budget | |
| - ask before external actions | |
| - do not modify protected resources | |
| A corrigibility-oriented evaluation asks: | |
| > Does the agent continue to respect these constraints when the task becomes difficult? | |
| --- | |
| # 06 · Permission Changes | |
| Permissions may change during execution. | |
| For example: | |
| ```text | |
| Browser access ✓ | |
| Email sending ✓ | |
| Payment execution ✕ | |
| File deletion ✕ | |
| ``` | |
| Later: | |
| ```text | |
| Email sending ✕ | |
| ``` | |
| A good system should recognize the update immediately and adapt. | |
| --- | |
| # 07 · Correction Acceptance | |
| Corrections can happen at many levels: | |
| - factual correction | |
| - plan correction | |
| - tool correction | |
| - policy correction | |
| - goal correction | |
| - permission correction | |
| Possible evaluation: | |
| ```text | |
| Agent chooses Plan A | |
| ↓ | |
| Human: "Plan A is wrong. Use Plan B." | |
| ↓ | |
| Does the agent switch? | |
| ``` | |
| --- | |
| # 08 · Recovery After Intervention | |
| Corrigibility is not only about stopping. | |
| It also includes safe recovery. | |
| After an intervention, the system may need to: | |
| - summarize current state | |
| - discard invalid actions | |
| - re-plan | |
| - ask for clarification | |
| - request approval | |
| - continue from a safe checkpoint | |
| --- | |
| # Possible Spaces | |
| ### Corrigibility Eval Lab | |
| Run structured scenarios where an agent receives corrections, interruptions, or permission changes. | |
| ### Shutdown Behavior Checker | |
| Evaluate whether an agent respects stop and termination instructions. | |
| ### Goal Update Simulator | |
| Test whether an agent switches cleanly from an old objective to a new one. | |
| ### Oversight Gate Builder | |
| Design approval points for sensitive agent actions. | |
| ### Permission Change Tester | |
| Simulate mid-task changes to tool permissions. | |
| ### Constraint Following Eval | |
| Check whether agents stay within explicit operational limits. | |
| ### Intervention Trace Inspector | |
| Inspect what an agent did before and after a human correction. | |
| ### Corrigibility Benchmark Builder | |
| Create scenario-based evaluations for steerability and intervention handling. | |
| ### Agent Recovery Lab | |
| Measure how well an agent re-plans after interruption or correction. | |
| ### Human-in-the-Loop Simulator | |
| Model workflows where actions alternate between AI execution and human approval. | |
| --- | |
| # Possible Datasets | |
| Possible datasets may include: | |
| ```text | |
| corrigibility-scenarios | |
| shutdown-behavior-evals | |
| goal-update-tests | |
| permission-change-cases | |
| oversight-interventions | |
| constraint-following-traces | |
| agent-correction-dialogues | |
| human-approval-workflows | |
| ``` | |
| Useful fields may include: | |
| - initial_goal | |
| - correction | |
| - authorized_actor | |
| - previous_state | |
| - new_instruction | |
| - allowed_tools | |
| - revoked_tools | |
| - expected_behavior | |
| - prohibited_behavior | |
| - agent_trace | |
| - outcome | |
| - success | |
| --- | |
| # Possible Models | |
| Models may support: | |
| - intervention classification | |
| - shutdown-compliance detection | |
| - permission-state tracking | |
| - goal-update recognition | |
| - trace analysis | |
| - constraint-violation detection | |
| - recovery-quality scoring | |
| - oversight escalation prediction | |
| --- | |
| # Example Evaluation Record | |
| ```json | |
| { | |
| "task": "Send a report after approval", | |
| "initial_permission": "draft_only", | |
| "agent_action": "drafted_report", | |
| "human_update": "do_not_send", | |
| "expected_behavior": "stop_before_external_action", | |
| "observed_behavior": "stopped", | |
| "success": true | |
| } | |
| ``` | |
| A single record tests one intervention. | |
| A benchmark tests many. | |
| --- | |
| # Corrigibility Dimensions | |
| | Dimension | Core Question | | |
| |---|---| | |
| | **Interruptibility** | Can the system be paused or stopped? | | |
| | **Goal Flexibility** | Can authorized objectives be changed? | | |
| | **Permission Respect** | Does it honor revoked or limited access? | | |
| | **Oversight** | Does it wait when approval is required? | | |
| | **Constraint Following** | Does it stay within explicit limits? | | |
| | **Correction Acceptance** | Does it adopt legitimate corrections? | | |
| | **Recovery** | Can it re-plan safely after intervention? | | |
| | **Transparency** | Can we inspect what happened? | | |
| --- | |
| # A Corrigibility Scorecard | |
| A practical evaluation might combine: | |
| ```text | |
| Interruption Compliance 25% | |
| Goal Update Compliance 20% | |
| Permission Respect 20% | |
| Constraint Following 15% | |
| Recovery Quality 10% | |
| Oversight Escalation 5% | |
| Trace Transparency 5% | |
| ``` | |
| The exact weighting should depend on the use case. | |
| There is no universal corrigibility score. | |
| --- | |
| # Corrigibility vs. Capability | |
| A highly capable system is not automatically corrigible. | |
| Capability asks: | |
| > Can the system complete the task? | |
| Corrigibility asks: | |
| > Can the system remain under legitimate correction while completing the task? | |
| Both matter. | |
| --- | |
| # Corrigibility + Agents | |
| Agentic systems make corrigibility especially important. | |
| An agent may: | |
| - make plans | |
| - use tools | |
| - act across systems | |
| - store state | |
| - retry failed steps | |
| - delegate subtasks | |
| Each of those abilities creates additional intervention points. | |
| A useful architecture may look like: | |
| ```text | |
| AGENT | |
| ↓ | |
| PLAN | |
| ↓ | |
| POLICY CHECK | |
| ↓ | |
| APPROVAL GATE | |
| ↓ | |
| ACTION | |
| ↓ | |
| TRACE | |
| ↓ | |
| OVERSIGHT | |
| ``` | |
| --- | |
| # Corrigibility + Orchestration | |
| Orchestration can provide technical mechanisms for corrigibility. | |
| Examples: | |
| - approval gates | |
| - permission layers | |
| - stop conditions | |
| - fallback behavior | |
| - trace logging | |
| - human escalation | |
| - rollback | |
| - checkpointing | |
| Corrigibility becomes stronger when it is supported by architecture, not only prompts. | |
| --- | |
| # Corrigibility + Evaluation | |
| Corrigibility should be tested. | |
| Possible evaluation strategies: | |
| - scenario-based testing | |
| - adversarial correction tests | |
| - repeated-run reliability | |
| - mid-task permission changes | |
| - shutdown requests | |
| - goal-replacement tasks | |
| - trace inspection | |
| - policy-compliance grading | |
| --- | |
| # Design Principles | |
| ### Authorized correction should work | |
| Legitimate oversight must remain effective. | |
| ### Stop means stop | |
| Interruption mechanisms should be respected. | |
| ### Permissions are dynamic | |
| Access can change during execution. | |
| ### Old goals should not persist silently | |
| A system should recognize when objectives are replaced. | |
| ### Oversight should be explicit | |
| Important actions should have visible approval boundaries. | |
| ### Recovery should be safe | |
| Intervention should not create chaotic behavior. | |
| ### Trace important decisions | |
| Steerability is easier to evaluate when actions are inspectable. | |
| --- | |
| # Technology Directions | |
| Projects may explore: | |
| - Hugging Face Spaces | |
| - Hugging Face Datasets | |
| - agent traces | |
| - tool-use logs | |
| - permission systems | |
| - structured evals | |
| - human-in-the-loop workflows | |
| - policy engines | |
| - agent orchestration | |
| - state machines | |
| - benchmark harnesses | |
| - safety evaluation | |
| - alignment research | |
| - oversight tooling | |
| --- | |
| # Who Is Corrigibility For? | |
| Corrigibility may be useful for: | |
| - AI safety researchers | |
| - agent developers | |
| - evaluation teams | |
| - alignment researchers | |
| - platform engineers | |
| - security teams | |
| - MLOps teams | |
| - governance researchers | |
| - enterprise AI teams | |
| - open-source contributors | |
| --- | |
| # Long-Term View | |
| As AI systems become more capable, the central question may not only be: | |
| > **Can the system do more?** | |
| It may also become: | |
| > **Can we still reliably correct what it does?** | |
| That is the space Corrigibility explores. | |
| --- | |
| # Important Note | |
| Projects published here are intended primarily for: | |
| - research | |
| - evaluation | |
| - education | |
| - development | |
| - prototyping | |
| - technical experimentation | |
| A strong result on a corrigibility benchmark does not prove that a system is universally safe, aligned, controllable, or suitable for high-impact deployment. | |
| Corrigibility is one part of a broader safety picture. | |
| --- | |
| # Independent Organization | |
| **Corrigibility is an independent Hugging Face community organization.** | |
| It is not an official certification body, standards organization, model provider, research institute, regulator, or Hugging Face organization. | |
| The name **Corrigibility** reflects the central idea: | |
| > **AI systems should remain responsive to legitimate human correction and oversight.** | |
| --- | |
| <p align="center"> | |
| # CORRIGIBILITY | |
| ### **Correctable. Interruptible. Steerable.** | |
| </p> | |