Commit History
-
Redesign CTR calibration for marketer workflow — metric input upfront, not buried Running
Eric XuUnverified signatureMove "Anchor to a real metric" from a hidden details section inside eval results to a first-class input field next to the goal. Default to CTR with a dropdown for other metrics (conversion rate, open rate, revenue, custom). - Auto-apply calibration when eval completes if metric value is set - Debounced re-apply on metric value/unit changes - Table headers use actual metric name ("CTR Impact", not "Metric Impact") - Keep Helps/Hurts columns alongside metric columns (was replacing them) - Clear calibration re-renders gradient table back to score-only view - Fix score:0 truthiness bug in frontend mean calculation - Guard against division by zero in linear scaling (mean_score > 0) - Tighten backend score filtering to isinstance check9166125
-
Add optional metric calibration to web UI
ClaudeUnverified signatureAfter evaluation, users can optionally anchor SGO scores to a real-world metric (CTR, conversion rate, revenue, etc.) by entering the current known value. The gradient table then shows predicted metric deltas alongside score deltas. - Single anchor: linear scaling (metric = k * score) - Multiple anchors: Platt scaling via Newton's method - Collapsible UI panel in step 1 after evaluation results - Calibration data included in downloaded reports - Generic naming ("metric") not CTR-specific https://claude.ai/code/session_0141cbZmdz7ziFkNsQbq7z5Ya8d5d4c
-
Add downloadable report + switch default model to gpt-oss-120b
Eric Xu +1: Claude Opus 4.6 (1M context)After any phase (eval, counterfactual, bias audit), users can download a comprehensive markdown report with all results. Default eval model changed from gpt-4o-mini to gpt-oss-120b for better quality. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>5589a80
-
Hide badges until loaded — no more 'checking...' on first paint
Eric XuUnverified signature41b60b4 -
Hide BYOK API key setup on HF Spaces — server key handles everything
Eric XuUnverified signature93afd8d -
Fix: use custom X-LLM headers instead of Authorization (HF proxy intercepts it)
Eric XuUnverified signaturedff6d17 -
Add realistic example templates: product, resume, pitch, policy, listing, blog
Eric XuUnverified signatureEach template has synthetic but plausible content so users see what a real SGO input looks like — specific numbers, names, tradeoffs, and limitations. Added 3 new categories: Policy, Listing, Blog Post.5c12db7 -
Show elapsed seconds on Auto-fill buttons while thinking
Eric XuUnverified signature7f2f206 -
Fix port binding: simplify startup, remove timeout on segments, clean SIGTERM
Eric XuUnverified signatureb4e9b32 -
Show 'Thinking...' on Auto-fill buttons and placeholders during LLM call
Eric XuUnverified signature64a6a10 -
Add elapsed timer, pulsing progress bar, timeouts with fallbacks for long LLM calls
Eric XuUnverified signatured00cf52 -
Security hardening: rate limiting, no creds in URLs, path traversal fix
Eric XuUnverified signatureCritical fixes from security audit: - Move API key from query params to Authorization header (never logged) - Add per-IP rate limiting (20 LLM requests/hour) - Cap parallel workers at 10 to prevent abuse - Fix Nemotron path traversal (must be within project or /tmp) - Disable uvicorn access log to prevent credential leakage - Remove filesystem paths from /api/config response - Fix Dockerfile: add git for HF Spaces build - Update privacy claim to accurately describe key handling9ea2205 -
Pass LLM credentials per-request, never store server-side
Eric XuUnverified signatureAPI key flows from browser JS memory → query params → backend middleware → LLM call. Nothing is stored in server memory or disk. Each request is self-contained. Users can: bring their own key (any provider), or use a server-default key set via HF Spaces Secrets.2f01b87
-
Improve persona setup UX: hide folder on Spaces, add progress bar, better errors
Eric XuUnverified signature7454acd -
Clarify API key is never stored or logged
Eric XuUnverified signaturea2b6fe1 -
Show base URL and model fields by default — don't lock to any provider
Eric XuUnverified signatured323191 -
Add HuggingFace Spaces deployment support
Eric XuUnverified signature- Dockerfile for HF Spaces (port 7860, Python 3.11) - Per-user API key input in UI (no .env needed for hosted mode) - Detect SPACE_ID env var, auto-bind 0.0.0.0:7860 - Persona dataset dropdown: USA, Japan, India, Singapore, Brazil, France (all from nvidia/Nemotron-Personas collection) - HF Spaces metadata in README frontmatter - /api/config/api-key endpoint for browser-based key entry - /api/nemotron/setup accepts dataset choice parameterc9d61d3 -
UX copy overhaul: intuitive labels, consistent terminology, plain language
Eric XuUnverified signatureBased on Codex UX review (gpt-5.3-codex, xhigh): Naming: - "Entity" → "What should the panel review?" - "Directions for your goal" → "Test what to change next" - "Bias Audit" → "Check panel realism" - "Suggest" → "Auto-fill" - "Evaluate" → "Build panel and review" Terminology (standardized on "panel"): - "evaluators/personas/audience" → "panel members" throughout - "positive/neutral/negative" → "would say yes/unsure/would say no" Academic language removed: - "persuadable middle" → "on the fence" - "Framing effect" → "Does wording change the score?" - "Authority bias" → "Do trust signals change the score?" - "Probes" → "Checks" Progress messages rewritten around user progress, not system actions. Error messages now tell users what to do next.e56613d -
Fix XSS, VJP contract, binding, session security, name collisions, and error handling
Eric XuUnverified signature1. XSS: Add esc() helper to sanitize all user/LLM data interpolated into innerHTML (logStep, gradient table details, eval log, bias audit table). 2. VJP contract: Backend analyze_gradient() now returns structured ranked data alongside text; frontend renders from backend rankings instead of recomputing unweighted averages. 3. Bind uvicorn to 127.0.0.1 instead of 0.0.0.0. 4. Session-bound tickets: store sid in counterfactual ticket and verify on stream pickup. 5. Name collision: use composite key (name_user_id) for cohort_map in both web app and scripts to avoid persona lookup collisions; propagate user_id through _evaluator dict. 6. Counterfactual error handling: add try-catch around fut.result() matching the evaluate stream pattern.ff413e9 -
Remove redundant text output — table with expandable rows is the interface
Eric XuUnverified signaturecb7e365 -
Make gradient table rows expandable with per-evaluator details
Eric XuUnverified signaturef64206f -
Remove personal/dating examples from all placeholders and prompts
Eric XuUnverified signature560bae5 -
Show change label instead of ID in counterfactual progress log
Eric XuUnverified signaturef92f674 -
Move counterfactual params from GET query to POST+ticket to avoid log leakage
Eric XuUnverified signaturea08e54e -
Add Suggest buttons for goal and audience instead of black-box auto-fill
Eric XuUnverified signature13165af -
Make entity, goal, and audience first-class inputs
Eric XuUnverified signatureThree equal inputs define the SGO spec: - Entity: what you're putting out there - Goal: what outcome you want - Audience: who evaluates it Goal and audience are auto-inferred from entity via LLM if left blank, but visible and editable so the user always sees and controls the spec. Audience context now drives Nemotron demographic filtering directly. - Add /api/infer-spec endpoint - Promote audience from hidden Advanced Options to main form - Auto-fill on Evaluate if blank, show in progress logca21edb -
Extract demographic filters from audience context for Nemotron cohorts
Eric XuUnverified signatureThe Nemotron path was ignoring audience context entirely (empty filters), returning random Americans regardless of what the user specified. Now the LLM extracts structured filters (sex, age range, city, state, education, occupation) from the audience context and passes them to filter_personas. "Women in Chicago aged 25-45" → {sex: Female, city: Chicago, state: IL, age_min: 25, age_max: 45} → only matching Nemotron personas are sampled. Extracted filters are shown in the progress log for transparency.61872af -
Auto-generate changes from concerns, one-click directions flow
Eric XuUnverified signatureStep 2 is now fully automated: user clicks "Directions for your goal" and the system extracts concerns from the persuadable middle, generates candidate changes via LLM, then probes them — all with live progress. - Add /api/suggest-changes endpoint: LLM generates actionable changes from evaluation concerns + goal - Frontend: remove manual change editor, replace with auto-flow that shows each phase in the progress log - Changes tested are viewable under a details disclosure95b845b -
Reframe analysis around persuadable middle, not universal appeal
Eric XuUnverified signatureSGO does not try to please everyone. The analysis now segments evaluators into champions (8+), persuadable (4-7), and not-for-them (1-3). Only the persuadable middle drives optimization — the rest is context. - evaluate.py: NPS-style segmentation in analysis output. Concerns are drawn from the persuadable middle only. Low scorers shown as "not your audience" rather than problems to solve. - counterfactual.py: gradient header says "Priority Actions" and explains it only consults persuadable evaluators. - Web UI: "Find what to change" → "Directions for your goal". Step desc explains the methodology. - README: clarify that SGO focuses on moving the almost-convinced, not pleasing everyone.88e3978 -
Add goal-weighted gradient (VJP) — optimize toward objectives, not universal appeal
Eric XuUnverified signatureThe semantic gradient is now a vector-Jacobian product: each evaluator is weighted by their relevance to the user's stated goal. Uniform: nabla_j = (1/n) sum_i J_ij (what pleases everyone) VJP: nabla_j = sum_i v_i * J_ij (what moves toward goal) Without a goal, behavior is unchanged (uniform weights). With a goal, the LLM scores each evaluator's relevance (0-1) and the gradient prioritizes changes that matter to the right audience. - counterfactual.py: add --goal flag, compute_goal_weights(), VJP in analyze_gradient() - web: add goal input field, pass through to counterfactual endpoint - README: update math section with VJP formulation9198e06 -
Hint larger panel size when below 40
Eric XuUnverified signature2934f67 -
Use Nemotron personas when available, add dataset setup UI
Eric XuUnverified signatureWeb interface now prefers census-grounded Nemotron personas (1M dataset) over LLM-generated ones. Checks common paths on startup; if not found, shows a setup panel where user provides a path — loads existing data or downloads from HuggingFace (~2GB). - Add /api/nemotron/setup endpoint (load or download to given path) - Add /api/config nemotron_available field - Cohort generation uses stratified sampling from Nemotron when available - Progress log shows data source (census-grounded vs LLM-generated)d94e0d2 -
Remove dating/relationship references from all files
Eric XuUnverified signature92cf501 -
Use generic placeholder for audience context
Eric XuUnverified signature780903d -
Fix checkbox layout, default bias calibration on, simplify label
Eric XuUnverified signaturee90920a -
Simplify web UX: one-click evaluate, no segment configuration
Eric XuUnverified signatureUsers paste their entity and click Evaluate. The system automatically: 1. Analyzes the entity to determine appropriate evaluator segments 2. Generates personas for each segment 3. Runs evaluation with live progress All intermediate steps (segment selection, cohort generation, persona details) are shown in the progress log so users see what's happening under the hood, but don't need to configure anything. Segment config, panel size, bias calibration, and audience context are available under Advanced Options for power users.92e2b88 -
Auto-suggest context-aware segments from entity via LLM
Eric XuUnverified signatureThe cohort step now calls the LLM to suggest segments that match the entity domain (dating profile → partner types, product → buyer personas, resume → hiring managers). Replaces hardcoded product-focused defaults. - Add /api/suggest-segments endpoint - Auto-call on step 2 entry, with Re-suggest button for manual refresh0694cd1 -
Add bias audit to web interface, make web the primary entry point, switch to MIT license
Eric XuUnverified signature- Web: add Step 5 (bias audit) with framing/authority/order probes, SSE streaming progress, and auto-enable bias calibration when over-biased - Web: add bias calibration toggle to evaluation step - Backend: add /api/bias-audit/stream endpoint, pass bias_calibration param through evaluate endpoint - README: web interface is now the primary Quick Start path - License: switch from CC-BY-4.0 to MIT8b7e05d -
Add web interface for SGO pipeline
Eric XuUnverified signatureFastAPI backend + single-page frontend that wraps the full SGO pipeline: - Entity input with templates (product/resume/pitch) - LLM-generated cohort builder with configurable segments - Streaming evaluation with real-time progress via SSE - Counterfactual gradient computation with visual table Run with: uv run --extra web python web/app.py25a1f95