Commit History
23 days ago
-
Fix gpt-oss-safeguard 400 on an empty system prompt; stop stale example labels hard-erroring Running
Verified signature4a25d3a -
Gate the Mila guardrail on the MH preset tag only, not policy wording
Verified signaturedd580cc -
Add the Mila self-harm output guardrail; only include it in Run all for mental-health policies
Verified signature1630c27 -
Update workshop content samples from the sheet (new MH-06, revised MH-01/04/05)
Verified signature2796397 -
Update MH-P1 and MH-P2 from the workshop sheet (rewritten mental health policies)
Verified signaturecd67551 -
Warn when a preset is added twice; number repeated category titles so PolicyLM accepts them
Verified signature0b76257 -
Put workshop policies first in the preset dropdown; label the old examples
Verified signature542aa13 -
Policy tab: make rule fields editable, add a loaded-policy status line
Verified signaturedc25662
24 days ago
-
Run the model that was actually selected
Verified signatureThe Testing-tab picker was mirrored into the Configuration-tab dropdown and the run read the second one, so a run was correct only if that copy had landed first. When it had not, every run used the Configuration tab's default — GPT-OSS-Safeguard-20B — whatever was picked. Both dropdowns now write to a selected_model_state and the run reads that, so the control the user actually touched is the one that counts. The dropdown-to-dropdown mirroring stays, but only to keep the two displays agreeing: if it fails the pickers look out of step while the run stays correct. Dispatch itself was never at fault — every model id in MODELS resolves to its own backend (shieldstral, policylm, cope-b-s1, cope-a, router), verified against the constants.f8c1283 -
Drive per-model prompt fields from the visible picker, not the hidden dropdown
The system-prompt note and grey-out (and the model info panel) were wired to the hidden config model dropdown, which now only updates programmatically when the Test tab picker syncs to it, and a programmatic update does not re-fire its listeners. So selecting a classifier on the Test tab left the Advanced tab showing the GPT-OSS prompt as editable. Wire format_model_info and the prompt-scope handler to the visible picker (model_quick_select) directly, which fires reliably on user choice. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>93dd9e3 -
Scope the system prompt to GPT-OSS; stop showing it for models that ignore it
The System Prompt and Response Format fields are only read by the GPT-OSS models. PolicyLM, Shieldstral, and CoPE build fixed prompts from their own model cards and never receive these fields, yet the tab populated the box (with leftover Qwen scaffolding, typo and all) and let you edit a prompt that was silently dropped. Now get_default_system_prompt returns empty for non-GPT-OSS models, and the Advanced settings tab shows a per-model note and greys out both fields when a classifier/decision model is selected, so no one edits a prompt that will never be sent. The all-model comparison computes each GPT-OSS model's own default prompt, so it is never degraded by whatever the shared box happens to hold. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>b49008f -
Rename the Policy tab to match the sidebar ("Policy")
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>fa240d4 -
Reorganize tabs, preview preset rules, cut em dashes from copy
- Preset loading now shows what it will add: selecting a policy lists the rules it contains ("Adds 5 rules: ...") and the button reads "Add these rules to my policy", so it is no longer a leap of faith. - Tabs run in workflow order: Policy (start here), Test & Compare, Advanced settings, Saved results. The Config tab becomes "Advanced settings" and keeps every knob (reasoning effort, system prompt, response format, generation params) for advanced users; its duplicate model dropdown is hidden since the Test tab's picker is the one home for model choice (the component stays as the synced backing value). - Renamed "Session Management & Examples" to "Saved results"; updated the sidebar guide to the new names and order. - Replaced em dashes in the visible copy with plainer punctuation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>5bf1594 -
Run-all-models: run concurrently and stream rows as each finishes
The first version ran the six models sequentially, so with the reasoning models inheriting the Config default max_tokens (9192) a comparison could block for a minute-plus with a blank table. Run them in a thread pool instead and make the handler a generator that yields the table after each model returns: every model shows "running" immediately and its verdict fills in as it lands, so total time is about the slowest single model rather than the sum, and a stalled endpoint no longer holds up the rest. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>e3e77dc -
Policy tab: load first; Testing tab: compare all models in one click
Two workshop-navigation fixes from a live walkthrough: - Policy Definition now opens with "Load a policy (start here)" on top and expanded, so the common first action (load your room's preset) is the first thing you see instead of a blank rule form buried above the loader. Renamed the rule-editor heading and dropped the stale "overrides the free-form policy text below" line left over from the one-route change. - Testing gains "⚡ Run on all models": runs the current content + policy against every model and shows a Model / Verdict / Notes comparison table (the single-model view stays below). Each model runs independently; one whose endpoint is down shows as unavailable rather than failing the whole comparison. This is the workshop's core move — same input judged side by side — which previously meant switching the model and re-running one at a time. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>b945bbc -
Policy tab: add presets to the rule list instead of replacing it
Loading a preset (or an upload) used to overwrite the whole rule list and silently become the policy under test, so stacking policies or keeping your own rules beside a preset was impossible, and an upload went live the moment it landed. Now "Add preset's rules" appends the selected policy's rules to the list, each tagged with its source and switched on by default; uploading only makes a policy available (preselected in the dropdown, text in the box for review) rather than going live. Curate with each rule's on/off checkbox — the disabled ones are already excluded from both the rendered text and the PolicyLM categories — so nothing is tested until you leave it enabled. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>4da6259 -
PolicyLM: deliver preamble context on the free-form path + fold 'applies to all' clauses
Two gaps let a written policy's shared guidance reach the prompted models but not PolicyLM (which is scored one category at a time): 1. The per-category definition `context` (protected/not-protected scope) was folded into the violation rule only in rules_to_categories (editor path). On the free-form / Policy-text path, policy_text_to_categories left it as a separate `context` key, which the endpoint handler ignores — so the preamble never reached PolicyLM. Fold it in there too. 2. A flag-less "## Applies to all categories" section (a Don't-flag / Exception carve-out with no Flag of its own) was dropped entirely. parse_policy_markdown and rules_to_categories now collect those clauses and fold them into every real category. Measured live: hate-speech criticism of a non-protected class 0.67 -> 0.54, scams counter-speech carve-out now present in every category; true positives unchanged at ~0.99. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>2d70287 -
One route into the policy: add anything as rules
Verified signatureThe "structured rules vs policy text" switch asked people to understand an internal distinction. Removed: the rule list is simply the policy. A preset already arrives as rules, and "➕ Add as rules" turns anything written or pasted into rules too — one per "## " section, or a single rule for a document with no sections — which can then be edited or deleted. Nothing is silently overridden, because there is only one way in. Also carries a policy's preamble definitions into the rules. Lines of the form "Protected classes: …" / "Not protected: …" sit before the first heading and were dropped by the parser, so a model scored one category at a time never saw them while a model handed the whole document did. They are now attached only to rules whose text uses the term, which keeps the token cost down: hate speech picks up definitions on 4 of 6 rules (~391 of 1662 tokens), mental health and scams on none. This does not make PolicyLM honour scope exclusions. Measured on the hate-speech policy,…bca5542 -
Make the live policy explicit instead of inferred
Verified signatureA policy you typed or uploaded could not be activated. Any rule with content beat the free-form text, so once rules were loaded the text was silently ignored, and Clear wiped the text while leaving the rules live. Loading a preset now fills the rule editor, so this hit every room rather than only people who had hand-written rules. A "Which policy is live?" switch replaces the inference. Loading a preset selects the rules, typing or uploading selects the text, and Clear empties both. The run sends an effective-rules state that is empty when the text is live, so the models receive what the switch says rather than whatever happens to sit in the editor, and the Testing banner reports the same source. Verified locally: with rules loaded and text typed, "Structured rules" sends the rule and "Policy text" sends the typed policy.72d7f4a -
Fix the Input column layout
Verified signatureThree things crowded the narrow left column. The model dropdown's Gradio `info` reserved two lines of hint text above the control; it is now a shorter caption and no longer sits in the primary input path. The dropdown displayed the full "Name (openai/gpt-oss-safeguard-20b)" string, which clipped under the caret, so the choices are (label, value) pairs — the visible label is just the model name while the value keeps the "Name (id)" form extract_model_id() parses. Both dropdowns changed so they still mirror each other. The policy line moved out of the Group: as a grouped child Gradio renders a Markdown as a full-width grey band that reads like a divider between unrelated controls rather than a caption on the run. Verified locally: every model's value still parses to the right id, and the app builds against gradio 6.30.d42cd2d -
Open presets in the rule editor; name the policy under test
Verified signatureA preset used to land only in the free-form box, so its rules were invisible beside any you had written and the two could not be compared. Loading one now parses it into the rule editor, and every rule's header says where it came from — "HS-P1" for a preset rule, "yours" for one you added — so an edited preset rule and your own sit in the same list without being indistinguishable. The Testing tab also says which policy a run will use. The policy lives on another tab, so it was easy to run content against one loaded earlier and forgotten. The line reports rule counts when rules are active and character counts when it is free-form text, because structured rules override the text and the banner has to describe what is actually sent, not what is on screen elsewhere.ee3d5f9 -
Collapsible rules with an on/off switch
Verified signatureA parsed policy runs to six rules and twenty-four textboxes stacked flat, which is unreadable. Each rule is now its own accordion titled with the rule's name, filled rules start collapsed, and a newly added blank one opens. Each rule also has a "Use this rule" switch. Turning a rule off keeps the text but drops it from what the models see, which is exactly the comparison the workshop is built around — the paired policies differ by one carve-out, and a room can now take that clause out and re-run instead of editing or deleting it. The switch is honoured in nonempty_rules(), the single point both the plain-text rendering and the PolicyLM category payload funnel through, so no caller can miss it. A rule with no "enabled" key counts as on, so anything saved before this still works, and switching every rule off reads the same as having none — the free-form policy takes over. The name field and the switch bump the render counter so the header stays accurate; the three long text fields sti…1577408 -
Name the samples and policies; report a category for single-score models
Verified signature"HS-03" says nothing about what a sample is, so each now carries a short title ("'Just asking questions' dogwhistle") and the preset policies lead with the sheet's own ID ("HS-P2 · Hate Speech — counter-speech carve-out removed") so a room can match what it selects to the row it is filling in. Shieldstral and CoPE-B-S1 both returned "categories": [], which the results panel renders as "No categories found in response" followed by a note about expecting valid JSON — so a working model looked broken. Neither has a taxonomy; both now report the one decision they make, with the probability and threshold, matching how PolicyLM already reports its rules. Verified locally against gradio 6.30: presets and sample titles render as intended and the app builds.b346c97 -
Model picker on the Testing tab; keep the policy across a refresh
Verified signatureSwitching models is the most frequent action in a session but lived two tabs away, so the Testing tab now has its own model dropdown mirroring the one in Model Choice & Configuration. Reasoning level, system prompt and sampling stay on the configuration tab — this is only the choice people make constantly. Policy state was per-session gr.State and vanished on reload, which in a workshop means retyping a policy after an accidental refresh. A gr.BrowserState now holds the policy text and structured rules in localStorage and restores them on load. The restore bumps the rule editor's render counter, since the render is bound to that counter rather than to the rules themselves and restored rules would otherwise sit in state without appearing on screen. Verified by constructing the Blocks graph locally against gradio 6.30: BrowserState is present and the app builds with no errors.5c62d46 -
Raise the queue concurrency limit for the workshop
Verified signatureGradio's default_concurrency_limit is 1, so the app processed one Run click at a time across all users. With ~30 people in five rooms that serialises everything, and a single slow gpt-oss reasoning request would stall everyone else behind it. The handlers are I/O-bound — they POST to inference endpoints and wait — so a higher limit costs the cpu-basic Space almost nothing. 20 stays well under the 40-thread default in launch(). Measured first: the endpoints are not the constraint. policylm on a T4 served 30 concurrent requests in 8.9s wall (p50 4.9s, 3.4 req/s, zero errors); shieldstral and cope-b-s1 absorb 12 concurrent with p50 under 460ms.6cc77b7 -
Parse written policies into PolicyLM categories
Verified signatureSelecting a preset and running PolicyLM failed with "InvalidPolicyError: 'Policy violation' Violation rule contains a line break; each field is one line". Every field PolicyLM accepts must be a single line, and the handler's fallback wraps the whole multi-line policy document in one category, so any written policy failed there — only a one-line policy worked. run_policylm now derives categories from the policy text itself. The workshop policies are already in the shape PolicyLM wants ("## Name" with Flag / Don't flag / Exception lines), so they parse into 2-6 real categories, which also recovers per-rule scoring and makes exceptions read as exceptions. A document with no recognisable sections falls back to a single flattened catch-all, which is worse than per-rule scoring but runs instead of erroring. Verified on all 13 bundled policies (no field contains a newline) and end to end on the three dedicated endpoints: PolicyLM 0.9982 / 0.0289, Shieldstral yes / no, CoPE-B-S1 0.9990 / 0.0…a7d1398 -
Stop the rule editor re-rendering on every keystroke
Verified signatureEach rule textbox wrote back into structured_rules_state on blur, and the @gr.render that builds those textboxes was bound to the same state. Editing a field tore the inputs down and rebuilt them under the cursor, which reads as the fields being frozen once you type in them. Adding or removing a rule is the only thing that changes which components exist, so a rules_version_state counter now drives the render and field edits no longer trigger it. app.py passes the Blocks in so the initial render still fires on load.2d9473a -
model_interface: fail loudly on unexpected PolicyLM responses
Missing flagged/categories keys were silently mapped to label 0, so a misconfigured endpoint showed "No Policy Violation" with no categories. The handler always returns at least one category, so treat anything else as an error and surface the raw response.44de8b1 -
Revert helpers.py and tab_policy.py to 716934db
Verified signatureMy previous commit was built on a clone taken before 716934db and overwrote helpers.py wholesale, removing load_workshop_samples, which ui/tab_testing.py imports — the Space failed to boot. That commit had already solved the same problem (registry-derived dropdown) and better, so there is nothing to merge: this restores both files verbatim.f81f49b -
Build the preset dropdown from the policy registry
Verified signatureThe ten workshop policies were registered in load_preset_policy's file map but the dropdown's choices were a separate hardcoded list in three places, so the files shipped without ever appearing in the UI. PRESET_POLICY_FILES is now module-level and preset_policy_names() derives the dropdown from it, so a policy cannot be registered yet be unselectable.628b810 -
Wire workshop policies and samples into the UI
The workshop policies and samples were pre-loaded as files but not reachable from the UI: the policy preset dropdown still listed only the 3 original presets, and workshop_samples.csv was never read. Surface all preset policies (via list_preset_policy_names) in the dropdown + load handlers, and load the 20 workshop samples into the Testing tab's "Load Example" dropdown (via load_workshop_samples). Content itself matches the ROOST x Musubi workbook verbatim; this just makes it selectable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>716934d -
Stop tracking __pycache__; add .gitignore
These .pyc files were accidentally committed earlier via `git add -A`. Remove them from tracking and ignore bytecode/venv dirs going forward. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>eb0a0a0 -
Switch CoPE model to cope-b-s1 (System One / SGLang)
Replace cope-b-a4b with zentropi-ai/cope-b-s1, a policy-steerable decision model that implements the System One spec rather than a generate() LLM. run_cope_b now POSTs a `noul` question wrapping the policy to the SGLang server's /v1/systemone route and reads answers.q1.noul = P(violation), thresholded at COPE_B_THRESHOLD. Served via zentropiai/cope-s1-sglang on an A100. Updates README accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>fd25b5c -
Pre-load workshop policies and samples; make the policy tab model-agnostic
Verified signatureLoads the ten room policies and twenty samples from the 27 Oct workshop sheet so rooms can select their policy from the preset dropdown instead of pasting it. Both sides of each room's comparison are registered. The Policy Definition tab described the rule editor as mapping onto "how PolicyLM was designed". The three-part shape (flag / don't flag / exception) is how written moderation policies are already structured, and the point of the tab is that every model is judged on the same policy. Reworded to lead with that and refer to schema models generally; the violation_rule / not_violation_rule / exception_override mapping is unchanged, so PolicyLM compatibility is untouched. Also clarifies the limits notice, which presented the 16-category cap as universal when it only applies to models taking a category schema.3c569ac
27 days ago
-
Add structured policy rules with per-rule PolicyLM scoring
Author policies as discrete rules (flag / don't-flag / exception) that map 1:1 onto PolicyLM's Category(violation_rule, not_violation_rule, exception_override). Rules are sent to PolicyLM natively for one score per rule, and rendered to plain text for the other models so every model is judged on the identical policy. Adds a live 16-category / 1,662-token limit warning. Requires redeploying the PolicyLM endpoint with the updated handler. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>5c92085
-
Remove Qwen and Gemma models from the dropdown
Trims the model list to the 3 GPT-OSS router models plus the 3 dedicated-endpoint classifiers (Shieldstral, PolicyLM, CoPE-B) for the workshop. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>d658f0b -
Add Shieldstral, PolicyLM, and CoPE-B classifier models
These three models aren't served by the HF Inference Router, so each runs on its own dedicated HF Inference Endpoint with endpoint URL + token read from Space secrets. Each has a special-case branch in model_interface.run_test: - Shieldstral-1.0-3B: vLLM chat completions, softmax yes/no logprobs -> score - PolicyLM-1.7B: custom-handler endpoint (policylm_handler.py) returning JSON - CoPE-B-A4B: vLLM, COPE prompt, "0"/"1" answer + softmaxed logprob score Classifiers surface their violation probability in the Reasoning Trace panel (is_classifier flag suppresses the "no reasoning traces" note). README documents recommended GPU tier, required secrets, and per-minute billing / teardown. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>846e72b
-
Temporarily removing CoPE from UI (#9)
Verified signature- Temporarily removing CoPE from UI (7cb39316a61e26f3301c99abc753082031e055bc) Co-authored-by: Samidh <samidh@users.noreply.huggingface.co>4f51341
-
Change cope label output to int
Verified signatureb466d6e -
Match cope output to expected model output (#8)
Verified signature- Match cope output to expected model output (76f8dfbbcb0c7afd5d62cdeafdf12735a53887e1) Co-authored-by: Samidh <samidh@users.noreply.huggingface.co>eea3b9f -
Custom inference parameters for cope (#7)
Verified signature- Custom inference parameters for cope (574225a2a7a0a2630a5b99e1dde904ca11119282) Co-authored-by: Samidh <samidh@users.noreply.huggingface.co>a21f087 -
Fixed cope_token fetch (#6)
Verified signature- Fixed cope_token fetch (3549bfef09e2d236a5abfe63f57685fe3dad9bf3) Co-authored-by: Samidh <samidh@users.noreply.huggingface.co>7b6bc0f -
Fixed cope endpoint base url
Verified signature79545d2 -
Fixed cope_token (#5)
Verified signature- Fixed cope_token (8cad6bccc107853cfdc671837348334990d8f8f4) Co-authored-by: Samidh <samidh@users.noreply.huggingface.co>063f8a8 -
Add cope inference endpoint (#4)
Verified signature- Add cope inference endpoint (e1d7b915667a41646996087de34783985a68bb40) Co-authored-by: Samidh <samidh@users.noreply.huggingface.co>58f145b -
Add cope inference token
Verified signature0054098 -
Add CoPE to run_test (#3)
Verified signature- Add CoPE to run_test (25450d3b9fbb997bc2f496a0dc7b4cc312f92e05) Co-authored-by: Samidh <samidh@users.noreply.huggingface.co>6ff8ee8 -
Add CoPE prompt template (#2)
Verified signature- Add CoPE prompt template (6222c33e8c04289d4220311ee68b0092db84f79b) Co-authored-by: Samidh <samidh@users.noreply.huggingface.co>43bd2c3 -
Add CoPE to UI (#1)
Verified signature- Add CoPE to UI (404acdcc19ab113b2d3c4fcb5b8eeb6430324c10) Co-authored-by: Samidh <samidh@users.noreply.huggingface.co>4498250
-
native error handling
Yacine Jernitec90d7c6