Commit History
-
Redesign CTR calibration for marketer workflow — metric input upfront, not buried Running
Eric XuUnverified signatureMove "Anchor to a real metric" from a hidden details section inside eval results to a first-class input field next to the goal. Default to CTR with a dropdown for other metrics (conversion rate, open rate, revenue, custom). - Auto-apply calibration when eval completes if metric value is set - Debounced re-apply on metric value/unit changes - Table headers use actual metric name ("CTR Impact", not "Metric Impact") - Keep Helps/Hurts columns alongside metric columns (was replacing them) - Clear calibration re-renders gradient table back to score-only view - Fix score:0 truthiness bug in frontend mean calculation - Guard against division by zero in linear scaling (mean_score > 0) - Tighten backend score filtering to isinstance check9166125
-
Add optional metric calibration to web UI
ClaudeUnverified signatureAfter evaluation, users can optionally anchor SGO scores to a real-world metric (CTR, conversion rate, revenue, etc.) by entering the current known value. The gradient table then shows predicted metric deltas alongside score deltas. - Single anchor: linear scaling (metric = k * score) - Multiple anchors: Platt scaling via Newton's method - Collapsible UI panel in step 1 after evaluation results - Calibration data included in downloaded reports - Generic naming ("metric") not CTR-specific https://claude.ai/code/session_0141cbZmdz7ziFkNsQbq7z5Ya8d5d4c -
Replace naive gradient descent with Newton's method for Platt scaling
ClaudeUnverified signature2 parameters on a tiny dataset — Newton's method with analytic Hessian converges in ~5-10 iterations instead of blindly running 10k steps. https://claude.ai/code/session_0141cbZmdz7ziFkNsQbq7z5Y0a66d61 -
Explore CTR calibration: map SGO scores to click-through rate predictions
ClaudeUnverified signatureAdds a research doc explaining how SGO's existing evaluation outputs (scores, actions, bias calibration, counterfactual gradient) can be calibrated to predict CTR using Platt scaling with a small set of anchor creatives with known CTRs. Includes a reference script (ctr_calibrate.py) that fits a calibration function from anchor runs and converts counterfactual score deltas into CTR deltas. https://claude.ai/code/session_0141cbZmdz7ziFkNsQbq7z5Y109fc16
-
Add downloadable report + switch default model to gpt-oss-120b
Eric Xu +1: Claude Opus 4.6 (1M context)After any phase (eval, counterfactual, bias audit), users can download a comprehensive markdown report with all results. Default eval model changed from gpt-4o-mini to gpt-oss-120b for better quality. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>5589a80
-
Add SSE keepalive pings every 15s to prevent proxy timeout disconnects
Eric XuUnverified signaturecc9ff58
-
Hide badges until loaded — no more 'checking...' on first paint
Eric XuUnverified signature41b60b4 -
Hide BYOK API key setup on HF Spaces — server key handles everything
Eric XuUnverified signature93afd8d -
Fix: use custom X-LLM headers instead of Authorization (HF proxy intercepts it)
Eric XuUnverified signaturedff6d17 -
Add realistic example templates: product, resume, pitch, policy, listing, blog
Eric XuUnverified signatureEach template has synthetic but plausible content so users see what a real SGO input looks like — specific numbers, names, tradeoffs, and limitations. Added 3 new categories: Policy, Listing, Blog Post.5c12db7 -
Two-model approach: 7B for setup calls, 72B for evaluation
Eric XuUnverified signatureSetup calls (infer-spec, suggest-segments, suggest-changes, extract-filters) use Qwen2.5-7B-Instruct (~0.5s each). Evaluation and counterfactual probes use the main model (Qwen2.5-72B-Instruct, ~5s each) where quality matters.c5a1119 -
Add 45s timeout + 2 retries on LLM client to prevent Space restarts
Eric XuUnverified signature8a25fc1 -
Show elapsed seconds on Auto-fill buttons while thinking
Eric XuUnverified signature7f2f206 -
Create user 1000 with writable HOME for HF Spaces git config
Eric XuUnverified signature6226068 -
Add wget to Dockerfile (required by HF Spaces dev tools)
Eric XuUnverified signaturee810ce8 -
Use /data/nemotron for persistent storage on HF Spaces
Eric XuUnverified signature4798be2 -
Fix port binding: start.sh kills zombie, add app logging
Eric XuUnverified signature618f739 -
Fix port binding: simplify startup, remove timeout on segments, clean SIGTERM
Eric XuUnverified signatureb4e9b32 -
Show 'Thinking...' on Auto-fill buttons and placeholders during LLM call
Eric XuUnverified signature64a6a10 -
Fix port binding crash: kill zombie process and enable SO_REUSEADDR
Eric XuUnverified signature2a7b950 -
Add elapsed timer, pulsing progress bar, timeouts with fallbacks for long LLM calls
Eric XuUnverified signatured00cf52 -
Remove session restore — was only for debugging
Eric XuUnverified signatureb6efc9e -
Fix startup: use lifespan instead of deprecated on_event
Eric XuUnverified signaturebf0d091 -
Set rate limit to 2 runs per hour
Eric XuUnverified signatured81d8b6 -
Rate limit by pipeline runs (10/hr), not individual LLM calls
Eric XuUnverified signature815de7e -
Security hardening: rate limiting, no creds in URLs, path traversal fix
Eric XuUnverified signatureCritical fixes from security audit: - Move API key from query params to Authorization header (never logged) - Add per-IP rate limiting (20 LLM requests/hour) - Cap parallel workers at 10 to prevent abuse - Fix Nemotron path traversal (must be within project or /tmp) - Disable uvicorn access log to prevent credential leakage - Remove filesystem paths from /api/config response - Fix Dockerfile: add git for HF Spaces build - Update privacy claim to accurately describe key handling9ea2205 -
Pass LLM credentials per-request, never store server-side
Eric XuUnverified signatureAPI key flows from browser JS memory → query params → backend middleware → LLM call. Nothing is stored in server memory or disk. Each request is self-contained. Users can: bring their own key (any provider), or use a server-default key set via HF Spaces Secrets.2f01b87
-
Improve persona setup UX: hide folder on Spaces, add progress bar, better errors
Eric XuUnverified signature7454acd -
Clarify API key is never stored or logged
Eric XuUnverified signaturea2b6fe1 -
Show base URL and model fields by default — don't lock to any provider
Eric XuUnverified signatured323191 -
Add HuggingFace Spaces deployment support
Eric XuUnverified signature- Dockerfile for HF Spaces (port 7860, Python 3.11) - Per-user API key input in UI (no .env needed for hosted mode) - Detect SPACE_ID env var, auto-bind 0.0.0.0:7860 - Persona dataset dropdown: USA, Japan, India, Singapore, Brazil, France (all from nvidia/Nemotron-Personas collection) - HF Spaces metadata in README frontmatter - /api/config/api-key endpoint for browser-based key entry - /api/nemotron/setup accepts dataset choice parameterc9d61d3 -
UX copy overhaul: intuitive labels, consistent terminology, plain language
Eric XuUnverified signatureBased on Codex UX review (gpt-5.3-codex, xhigh): Naming: - "Entity" → "What should the panel review?" - "Directions for your goal" → "Test what to change next" - "Bias Audit" → "Check panel realism" - "Suggest" → "Auto-fill" - "Evaluate" → "Build panel and review" Terminology (standardized on "panel"): - "evaluators/personas/audience" → "panel members" throughout - "positive/neutral/negative" → "would say yes/unsure/would say no" Academic language removed: - "persuadable middle" → "on the fence" - "Framing effect" → "Does wording change the score?" - "Authority bias" → "Do trust signals change the score?" - "Probes" → "Checks" Progress messages rewritten around user progress, not system actions. Error messages now tell users what to do next.e56613d -
Fix XSS, VJP contract, binding, session security, name collisions, and error handling
Eric XuUnverified signature1. XSS: Add esc() helper to sanitize all user/LLM data interpolated into innerHTML (logStep, gradient table details, eval log, bias audit table). 2. VJP contract: Backend analyze_gradient() now returns structured ranked data alongside text; frontend renders from backend rankings instead of recomputing unweighted averages. 3. Bind uvicorn to 127.0.0.1 instead of 0.0.0.0. 4. Session-bound tickets: store sid in counterfactual ticket and verify on stream pickup. 5. Name collision: use composite key (name_user_id) for cohort_map in both web app and scripts to avoid persona lookup collisions; propagate user_id through _evaluator dict. 6. Counterfactual error handling: add try-catch around fut.result() matching the evaluate stream pattern.ff413e9 -
Fix production issues: session overwrite, error handling, memory leaks
Eric XuUnverified signatureCritical fixes from audit: - Fix entity_text overwrite: cohort endpoint no longer creates/overwrites sessions — frontend creates session first, then uploads cohort - Add try-catch around fut.result() in all SSE stream workers - Add TTL cleanup for _cf_pending tickets (expire after 10min) - Store goal/audience in session for consistency - Remove unused asyncio.get_event_loop() call - Add SESSION_MAX_AGE_HOURS constant for future cleanup6dcf8e5 -
Remove redundant text output — table with expandable rows is the interface
Eric XuUnverified signaturecb7e365 -
Make gradient table rows expandable with per-evaluator details
Eric XuUnverified signaturef64206f -
Remove personal/dating examples from all placeholders and prompts
Eric XuUnverified signature560bae5 -
Show change label instead of ID in counterfactual progress log
Eric XuUnverified signaturef92f674 -
Fix 422 on counterfactual stream — remove **kwargs from endpoint
Eric XuUnverified signature4958ab7 -
Move counterfactual params from GET query to POST+ticket to avoid log leakage
Eric XuUnverified signaturea08e54e -
Fix missing compute_goal_weights import
Eric XuUnverified signature33a4f71 -
Add Suggest buttons for goal and audience instead of black-box auto-fill
Eric XuUnverified signature13165af -
Make entity, goal, and audience first-class inputs
Eric XuUnverified signatureThree equal inputs define the SGO spec: - Entity: what you're putting out there - Goal: what outcome you want - Audience: who evaluates it Goal and audience are auto-inferred from entity via LLM if left blank, but visible and editable so the user always sees and controls the spec. Audience context now drives Nemotron demographic filtering directly. - Add /api/infer-spec endpoint - Promote audience from hidden Advanced Options to main form - Auto-fill on Evaluate if blank, show in progress logca21edb -
Extract demographic filters from audience context for Nemotron cohorts
Eric XuUnverified signatureThe Nemotron path was ignoring audience context entirely (empty filters), returning random Americans regardless of what the user specified. Now the LLM extracts structured filters (sex, age range, city, state, education, occupation) from the audience context and passes them to filter_personas. "Women in Chicago aged 25-45" → {sex: Female, city: Chicago, state: IL, age_min: 25, age_max: 45} → only matching Nemotron personas are sampled. Extracted filters are shown in the progress log for transparency.61872af -
Restore saved sessions from disk on server startup
Eric XuUnverified signaturea81335f -
Auto-generate changes from concerns, one-click directions flow
Eric XuUnverified signatureStep 2 is now fully automated: user clicks "Directions for your goal" and the system extracts concerns from the persuadable middle, generates candidate changes via LLM, then probes them — all with live progress. - Add /api/suggest-changes endpoint: LLM generates actionable changes from evaluation concerns + goal - Frontend: remove manual change editor, replace with auto-flow that shows each phase in the progress log - Changes tested are viewable under a details disclosure95b845b -
Reframe analysis around persuadable middle, not universal appeal
Eric XuUnverified signatureSGO does not try to please everyone. The analysis now segments evaluators into champions (8+), persuadable (4-7), and not-for-them (1-3). Only the persuadable middle drives optimization — the rest is context. - evaluate.py: NPS-style segmentation in analysis output. Concerns are drawn from the persuadable middle only. Low scorers shown as "not your audience" rather than problems to solve. - counterfactual.py: gradient header says "Priority Actions" and explains it only consults persuadable evaluators. - Web UI: "Find what to change" → "Directions for your goal". Step desc explains the methodology. - README: clarify that SGO focuses on moving the almost-convinced, not pleasing everyone.88e3978 -
Add goal-weighted gradient (VJP) — optimize toward objectives, not universal appeal
Eric XuUnverified signatureThe semantic gradient is now a vector-Jacobian product: each evaluator is weighted by their relevance to the user's stated goal. Uniform: nabla_j = (1/n) sum_i J_ij (what pleases everyone) VJP: nabla_j = sum_i v_i * J_ij (what moves toward goal) Without a goal, behavior is unchanged (uniform weights). With a goal, the LLM scores each evaluator's relevance (0-1) and the gradient prioritizes changes that matter to the right audience. - counterfactual.py: add --goal flag, compute_goal_weights(), VJP in analyze_gradient() - web: add goal input field, pass through to counterfactual endpoint - README: update math section with VJP formulation9198e06 -
Hint larger panel size when below 40
Eric XuUnverified signature2934f67 -
Fix panel size overflow — hard cap Nemotron cohort to requested total
Eric XuUnverified signature15f45fd