Spaces:
Running
Running
Commit History
-
perf(deploy): scope TS connection to demo DB + monthly-rotating database (250s -> <1s table imports)β
Root cause of the ~250s-per-call table TML imports (found 2026-08-10, corrects the July "server-side, not reducible" conclusion): ThoughtSpot table import scans all external metadata VISIBLE TO THE CONNECTION. Our connections had no database property, so every import scanned ~507 Snowflake DBs. Measured on the same instance with the same trivial table: unscoped 246s, scoped to DEMOBUILD (1,003 schemas of demo debris) 33s, scoped to a fresh tiny DB 0.7s. Prod secloud showed identical ~250s unscoped β never instance-specific. - thoughtspot_deployer: create_connection_tml/create_connection_with_reconcile now REQUIRE a database and emit a `database` property in the connection TML. - snowflake_auth: get_demo_database() -> <SNOWFLAKE_DATABASE>_<YYYY_MM> (memoized per process); ensure_demo_database() creates it on connect. Session params + cdw_connector + env injection (supabase_client) all use the rotated DB so write and deploy paths agree. Admin setting stays the base name. - chat_β¦21ee86d
-
merge: unify MCP (async TML import + concurrency + mcp_server) into integrationβ
- keep app README frontmatter (gradio) on trunk; MCP docker frontmatter stays on hf-mcp-deploy - build_demo.py: blueprint LLM cap/timeout reconciled to 16384 / 300s (superset of both sides) - adds mcp_server.py, tests/e2e_mcp.py, MCP deps; async-only TML import pathc3e3f8b -
merge: liveboard-quality into integration/mcp-lb
ee1ce2a -
liveboard quality: fix flat-72 defects + Tixr-style layoutβ
Reshape the MCP + enhance golden layout to fix the 5 structural grader defects and adopt the SE's Tixr Executive Analytics template. golden_layout.py: - note tile: full company name (was single-letter "N"/"L"), deduped; dropped by default (keep_overview_note flag) so the KPI row leads - colors: color ADVANCED_* charts by dimension via custom_chart_config "slice-with-color" (real per-category colors + legends; was all-blue, since seriesColors/chartLevelColorConfig don't drive the new engine) - per-tile tile_brand_color cycling + tile_kpi_color on KPIs - force CHART_MODE where a chart renders as a data grid - clean raw display names (Dim / Usd / Pct / doubled words) - trend detection: time-only series -> dedicated Performance Trends - geo: GEO_* -> its own full-width row right under the KPI row - asymmetric group sizing (5/4/3, 7/5) + Tixr styling (kpi L font, hidden group descriptions) liveboard_creator.py / thoughtspot_deployer.py: thread logo_url/url into build_dashboard_plaβ¦f7f2b52 -
chore: README, e2e grader, share-permission fix, sprint + quality resultsβ
- README: chat_interface.py is the full app (App tab primary), app.py entry point; updated structure/config - tests/e2e_quality.py: date-aware grader, COMPLETENESS rubric rewrite, --ts-environment override, hard-fail on missing dropdown label - thoughtspot_deployer.py: share-permission 400 fix β notify_on_share=false + empty message to satisfy the GraphQL $message contract - sprint_2026_04.md: progress updates - tests/quality_results: run artifacts incl. 2026-07-28 82.9/B run Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>fcde472
-
fix(deploy): connectivity guard β scope model to one connected join graph (fixes 13122)β
A ThoughtSpot model must form a single connected join graph; a disconnected graph is rejected on import with schema-validation error 13122. Two real builds hit this: one with two disjoint fact stars (underwriting + HR, no shared dim), one with an orphan date dimension (DIM_TIME_PERIOD, no fact FK). The deployer's only 13122 handling was to re-import without model-table joins, which can't fix a connectivity problem β so both failed twice and gave up (no model -> no liveboard). Add _select_model_component(): compute connected components of the FK graph and scope the MODEL to the primary component (the star with the most facts). Orphan tables with no joins are dropped; secondary fact stars are set aside and surfaced in results['set_aside_components'] / warnings (not silently lost). All tables are still created in Snowflake + as TS logical tables (Batch 1/2 unchanged) β only the model composition is scoped. Applied to both the primary model build and the no-joins retry. Shared deploy_allβ¦d0b4398
-
refactor(deploy): make TML import async-only (remove flag + sync branches)β
Both table phases now call import_tmls_async directly β no TS_TML_IMPORT_MODE flag, no sync path selection. The old sync closures (_import_tml_chunk, _import_tmls_chunked, _resolve_/_verify_*_after_timeout) are now unreferenced dead code; they'll be excised in a follow-up once one async build is validated end-to-end (kept in git history as the fallback). Shared deploy_all β app + MCP. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>c19146f -
feat(deploy): async TML import mode (TS_TML_IMPORT_MODE) β eliminates gateway 504sβ
Adds ThoughtSpotDeployer.import_tmls_async: submit to the async TML import API, poll the light status endpoint to a terminal state, then resolve name->guid from the connection β SAME return shape as the sync path, so deploy_all's downstream processing is unchanged. Both table-create (Batch 1) and joins (Batch 2) branch on TS_TML_IMPORT_MODE (default "sync"). Reliability/observability fix, not a speed fix (server-side ~4-5 min/phase unchanged). Shared code: the app and MCP both call deploy_all, so both get it. Sync path intact for instant rollback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>aac7ab6
-
fix: log the actual (resolved) model in semantics step, not stale llm_modelβ
The progress line printed the raw upstream llm_model (e.g. 'gpt-5.5') before ModelSemanticUpdater ran it through resolve_model_name(), which currently force-routes everything to claude-sonnet-4-6 (TEMPORARY_MODEL_OVERRIDE). Log updater.llm_model so the message reflects the model actually used. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>60fd9b1
-
Log deployed code version
mike boone8cea964 -
Improve backup liveboard path logging
mike boone5f0f576
-
Honor explicit custom dataset contracts
mike boonea6d4bca -
fix: retry transient ThoughtSpot auth failures
mike boone4489319
-
fix: reconcile ThoughtSpot connection creation
mike boone34f7221
-
fix: accept valid undefined ThoughtSpot answer tokens
mike boone70a4175 -
fix: detect invalid answer readiness before MCP liveboards
mike booned3049d4 -
fix: extend ThoughtSpot 504 polling to 15 minutes
mike boonec244364 -
fix: recover liveboard creation after MCP answer failures
mike boone827eb22 -
fix: recover tables from connection details
mike boone25fbe18 -
fix: validate recovered ThoughtSpot tables by connection
mike boonede46562 -
fix: import time for model readiness polling
mike booneb018de9 -
fix: recover table creates via connection-scoped search
mike boone0acfbde -
fix: normalize ThoughtSpot TML import object responses
mike boonee343cfd -
fix: harden ThoughtSpot table polling and model readiness
mike boone7aff9ba
-
fix: verify ThoughtSpot TML imports after 504
mike boonec04c93f -
Merge remote-tracking branch 'origin/develop' into develop
mike boone8997cc3 -
test: harden e2e quality diagnostics
mike boone8e5d114 -
Merge remote-tracking branch 'hf-test/main' into developβ
mike boone# Conflicts: # thoughtspot_deployer.pydb6ab4f -
fix: harden dataset and ThoughtSpot deployment recovery
mike booneb210533 -
fix: wait for delayed ThoughtSpot table imports
mike boone7f49e13 -
chore: log table tml import calls
mike boone3d06bfd -
fix: harden ThoughtSpot table import recovery
mike boonecdded3f -
fix: restore MCP-first liveboard answer flow
mike boonec3449ec
-
fix: tighten dataset quality and TS import retries
mike boone889a00d
-
fix: recover ThoughtSpot imports after gateway timeouts
mike boonea61ad4f
-
fix: enforce demo-safe data and liveboard layout
mike boonec3f1d91 -
fix: improve run diagnostics and ts recovery
mike boonee758f65 -
test: add dataset and liveboard prep seams
mike boonebb493ae
-
fix: use selected LLM for model semantics
mike boone24febbb -
fix: clarify dataset-first progress and import json scope
mike boone94f1813 -
fix: chunk ThoughtSpot table imports
mike boone3f6d9dc
-
feat: add deployment notifications and tested demo updates
mike boone3c65377
-
feat: add dataset-first generation path
mike boone7a5c5fe
-
fix: model GUID fallback + grader schema fixβ
- deployer: search TS by name when TML import omits id_guid (fixes Morgan Stanley GUID miss) - deployer: log ts_url in 'model created' session_log so test harness can read ThoughtSpot base URL - test: run_ai_grading now accepts schema_override param (was silently dropped before) - test: fall back to DEMOBUILD + session_logs schema when TML parse can't extract db/schema - test: GUID resolution from session_logs (primary), page HTML (fallback) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>3d87dc7 -
fix: handle non-numeric/dict values in Snowflake inserts (22018 errors)β
legitdata_bridge.py: - Numeric columns: return NULL if value is dict/list/non-numeric (was crashing with 22018 invalid value for cast) - VARIANT/ARRAY/OBJECT columns: serialize non-string values to JSON - VARCHAR/TEXT: serialize dict/list values to JSON string (truncated) - Better first-failed-row diagnostics in error logging thoughtspot_deployer.py: add ts_url to model-created log entry tests/e2e_quality.py: data_size β Medium; improved accordion/dropdown helpers with JS native setter for Svelte reactivity Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>2453228
-
feat: AI viz title humanization + KPI fix + chart varietyβ
- Add _humanize_viz_titles(): one LLM call converts raw TS column-name titles (e.g. "Netsalesamount by Regionname") to business-readable labels ("Revenue by Region"); falls back silently on error - Wire humanization as Step 6.5 in enhance_mcp_liveboard() before re-import; pass llm_model through from thoughtspot_deployer - Fix KPI conversion (Step 3.5): skip "by X" dimensional breakdowns β "Averagesellingprice by Category Weekly" no longer lands in Key Metrics - Chart variety: multi-dim breakdowns ("by X and Y") β STACKED_COLUMN; single-dim categorical breakdowns now catch LINE charts too (MCP frequently generates LINE for categorical, blocking donut conversion) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>1c63c48 -
feat: tag v2 type fix, claude-opus-4-6, viz titles sprint item, DATES fixβ
- Fix tag assignment: try v2 API first with correct type mapping (PINBOARD_ANSWER_BOOK β LIVEBOARD for v2); v1 is fallback only - Add claude-opus-4-6 to UI_MODEL_CHOICES in llm_config.py - Add Priority 6 (AI Viz Titles) to sprint doc - DATES generator fix: date column detection handles PascalCase naming (FullDate, FullDate.upper() = FULLDATE β FULL_DATE was breaking detection) - DATES config: 730 rows regardless of dim_table_size - Quality test improvements: schema fallback, TML limit expansion, liveboard viz count propagation, two-level timeout for MCP blocking Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>479257e
-
fix: remove all THOUGHTSPOT_URL admin fallbacks + data quality improvementsβ
- liveboard_creator.py: _get_direct_api_session/_get_answer_direct now accept ts_url param; create_liveboard_from_model_mcp uses ts_client.base_url β fixes empty liveboards after admin setting deletion - thoughtspot_deployer.py: remove get_admin_setting('THOUGHTSPOT_URL') fallback in __init__ - chat_interface.py: replace all get_admin_setting('THOUGHTSPOT_URL') with self.settings.get('thoughtspot_url'); remove THOUGHTSPOT_URL from admin_checks; fix fact_table_size default 1000β5000 - legitdata/config.py: add 'standard' preset (5k fact, 200 dim rows) - legitdata/generator.py: DATES table always spans 730 days (2 years) regardless of dim_table_size - legitdata/sourcer/generic.py: add DAYSONHAND (15-120), lead time, safety stock realistic ranges - tests/e2e_quality.py: better GUID extraction (full HTML), empty liveboard detection, pipeline completion fix, TEST_TARGET_URL required Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>e2d85b8 -
fix: remove matrix_config from fallback create_liveboard_from_model callβ
Function signature doesn't accept matrix_config β caused fallback to crash with unexpected keyword argument error. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>37d59da
-
feat: march sprint β logging, run history, chat fixes, deploy sub-stagesβ
- Comprehensive stage + sub-stage logging throughout pipeline (DDL, deploy, TS) - Run history grid: use case truncation with click-to-expand detail panel - awaiting_use_case stage fix to prevent company re-extraction on use case responses - ThoughtSpot deployer verbose sub-stage logging (connection, model tag/share, semantics, liveboard) - DDL start/complete/fail logging - Per-session state isolation for prompt and session loggers - Semantics batching, tag v2 fallback, 5-min MCP timeout, 60s Anthropic timeout - Numeric column guard in LegitData - Sprint doc, CLAUDE.md, and test suite updates Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>2b5a90d