Commit History
23 days ago
-
Warn when a preset is added twice; number repeated category titles so PolicyLM accepts them
Verified signature0b76257
24 days ago
-
PolicyLM: deliver preamble context on the free-form path + fold 'applies to all' clausesβ
Two gaps let a written policy's shared guidance reach the prompted models but not PolicyLM (which is scored one category at a time): 1. The per-category definition `context` (protected/not-protected scope) was folded into the violation rule only in rules_to_categories (editor path). On the free-form / Policy-text path, policy_text_to_categories left it as a separate `context` key, which the endpoint handler ignores β so the preamble never reached PolicyLM. Fold it in there too. 2. A flag-less "## Applies to all categories" section (a Don't-flag / Exception carve-out with no Flag of its own) was dropped entirely. parse_policy_markdown and rules_to_categories now collect those clauses and fold them into every real category. Measured live: hate-speech criticism of a non-protected class 0.67 -> 0.54, scams counter-speech carve-out now present in every category; true positives unchanged at ~0.99. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>2d70287 -
One route into the policy: add anything as rulesβ
Verified signatureThe "structured rules vs policy text" switch asked people to understand an internal distinction. Removed: the rule list is simply the policy. A preset already arrives as rules, and "β Add as rules" turns anything written or pasted into rules too β one per "## " section, or a single rule for a document with no sections β which can then be edited or deleted. Nothing is silently overridden, because there is only one way in. Also carries a policy's preamble definitions into the rules. Lines of the form "Protected classes: β¦" / "Not protected: β¦" sit before the first heading and were dropped by the parser, so a model scored one category at a time never saw them while a model handed the whole document did. They are now attached only to rules whose text uses the term, which keeps the token cost down: hate speech picks up definitions on 4 of 6 rules (~391 of 1662 tokens), mental health and scams on none. This does not make PolicyLM honour scope exclusions. Measured on the hate-speech policy,β¦bca5542 -
Collapsible rules with an on/off switchβ
Verified signatureA parsed policy runs to six rules and twenty-four textboxes stacked flat, which is unreadable. Each rule is now its own accordion titled with the rule's name, filled rules start collapsed, and a newly added blank one opens. Each rule also has a "Use this rule" switch. Turning a rule off keeps the text but drops it from what the models see, which is exactly the comparison the workshop is built around β the paired policies differ by one carve-out, and a room can now take that clause out and re-run instead of editing or deleting it. The switch is honoured in nonempty_rules(), the single point both the plain-text rendering and the PolicyLM category payload funnel through, so no caller can miss it. A rule with no "enabled" key counts as on, so anything saved before this still works, and switching every rule off reads the same as having none β the free-form policy takes over. The name field and the switch bump the render counter so the header stays accurate; the three long text fields stiβ¦1577408 -
Parse written policies into PolicyLM categoriesβ
Verified signatureSelecting a preset and running PolicyLM failed with "InvalidPolicyError: 'Policy violation' Violation rule contains a line break; each field is one line". Every field PolicyLM accepts must be a single line, and the handler's fallback wraps the whole multi-line policy document in one category, so any written policy failed there β only a one-line policy worked. run_policylm now derives categories from the policy text itself. The workshop policies are already in the shape PolicyLM wants ("## Name" with Flag / Don't flag / Exception lines), so they parse into 2-6 real categories, which also recovers per-rule scoring and makes exceptions read as exceptions. A document with no recognisable sections falls back to a single flattened catch-all, which is worse than per-rule scoring but runs instead of erroring. Verified on all 13 bundled policies (no field contains a newline) and end to end on the three dedicated endpoints: PolicyLM 0.9982 / 0.0289, Shieldstral yes / no, CoPE-B-S1 0.9990 / 0.0β¦a7d1398 -
Pre-load workshop policies and samples; make the policy tab model-agnosticβ
Verified signatureLoads the ten room policies and twenty samples from the 27 Oct workshop sheet so rooms can select their policy from the preset dropdown instead of pasting it. Both sides of each room's comparison are registered. The Policy Definition tab described the rule editor as mapping onto "how PolicyLM was designed". The three-part shape (flag / don't flag / exception) is how written moderation policies are already structured, and the point of the tab is that every model is judged on the same policy. Reworded to lead with that and refer to schema models generally; the violation_rule / not_violation_rule / exception_override mapping is unchanged, so PolicyLM compatibility is untouched. Also clarifies the limits notice, which presented the 16-category cap as universal when it only applies to models taking a category schema.3c569ac
27 days ago
-
Add structured policy rules with per-rule PolicyLM scoringβ
Author policies as discrete rules (flag / don't-flag / exception) that map 1:1 onto PolicyLM's Category(violation_rule, not_violation_rule, exception_override). Rules are sent to PolicyLM natively for one score per rule, and rendered to plain text for the other models so every model is judged on the identical policy. Adds a live 16-category / 1,662-token limit warning. Requires redeploying the PolicyLM endpoint with the updated handler. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>5c92085