dex-demo / docs /DESIGN.md
data-literate's picture
chore: add workspace config, update CI, gitignore, README
de32734
|
Raw History Blame Contribute Delete
10.2 kB

DEX Studio β€” Design Document

1. Purpose

DEX Studio is a local, Python-first, open-source UI application that provides a single control plane for end-to-end data projects powered by TheDataEngineX/DEX.

It does not fork or rebrand upstream DEX. It is a separate tool that connects to a local DEX engine instance and unifies workflows in one place:

Project Setup β†’ Ingestion β†’ Medallion Pipelines β†’ ML/AI β†’ Serving β†’ Observability

2. Architecture

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                        DEX Studio (NiceGUI)                      β”‚
β”‚                                                                   β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
β”‚  β”‚  Pages    β”‚  β”‚ Components β”‚  β”‚   State / Storage            β”‚  β”‚
β”‚  β”‚  (views)  β”‚  β”‚ (shared)   β”‚  β”‚   (NiceGUI app.storage)     β”‚  β”‚
β”‚  β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚
β”‚        β”‚               β”‚                        β”‚                 β”‚
β”‚  β”Œβ”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”‚
β”‚  β”‚                     Studio Core                              β”‚  β”‚
β”‚  β”‚  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                β”‚  β”‚
β”‚  β”‚  β”‚  Config   β”‚  β”‚  Client  β”‚  β”‚   Theme    β”‚                β”‚  β”‚
β”‚  β”‚  β”‚  (YAML)   β”‚  β”‚  (httpx) β”‚  β”‚  (CSS)     β”‚                β”‚  β”‚
β”‚  β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                β”‚  β”‚
β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                              β”‚  HTTP (localhost:8000)
                              β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                    DEX Engine (FastAPI)                           β”‚
β”‚  /health  /ready  /startup  /metrics                             β”‚
β”‚  /api/v1/data/*   /api/v1/warehouse/*   /api/v1/system/*         β”‚
β”‚  /api/v1/models   /api/v1/models/{name}  /api/v1/predict         β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Why NiceGUI + Native Mode

Requirement NiceGUI Capability
Python-first, no JS build step βœ… Pure Python components
Real URL routing (/quality, /models) βœ… @ui.page decorator
Websocket for live updates βœ… Built-in, auto-managed
Desktop-native window βœ… native=True via pywebview
Built on FastAPI βœ… Same stack as DEX engine
Component model (cards, tables, tabs) βœ… Quasar + custom components
Auth middleware (future) βœ… FastAPI middleware support

Why NOT Streamlit

  • No real URL routing (query params only)
  • No websocket support (polling via rerun loop)
  • No background tasks (script re-executes on every interaction)
  • Performance degrades with complex layouts
  • Can't embed into existing FastAPI infrastructure

3. Project Structure

dex-studio/
β”œβ”€β”€ pyproject.toml                  # Hatchling build, deps
β”œβ”€β”€ .python-version                 # 3.11
β”œβ”€β”€ config/
β”‚   └── default.yaml                # Default config template
β”œβ”€β”€ src/
β”‚   └── dex_studio/
β”‚       β”œβ”€β”€ __init__.py             # Package + version
β”‚       β”œβ”€β”€ cli.py                  # CLI entry (dex-studio command)
β”‚       β”œβ”€β”€ app.py                  # NiceGUI app bootstrap
β”‚       β”œβ”€β”€ config.py               # YAML config loading
β”‚       β”œβ”€β”€ client.py               # httpx DEX API client
β”‚       β”œβ”€β”€ theme.py                # CSS theme + colour palette
β”‚       β”œβ”€β”€ components/
β”‚       β”‚   β”œβ”€β”€ sidebar.py          # Navigation sidebar
β”‚       β”‚   β”œβ”€β”€ status_card.py      # Health status cards
β”‚       β”‚   β”œβ”€β”€ metric_card.py      # Metric display cards
β”‚       β”‚   └── page_layout.py      # Sidebar + content wrapper
β”‚       └── pages/
β”‚           β”œβ”€β”€ overview.py         # Dashboard (/)
β”‚           β”œβ”€β”€ health.py           # Health probes (/health)
β”‚           β”œβ”€β”€ data_quality.py     # Medallion quality (/quality)
β”‚           β”œβ”€β”€ lineage.py          # Lineage explorer (/lineage)
β”‚           β”œβ”€β”€ ml_models.py        # Model registry (/models)
β”‚           └── settings.py         # Connection config (/settings)
β”œβ”€β”€ tests/
β”‚   β”œβ”€β”€ conftest.py
β”‚   └── unit/
β”‚       β”œβ”€β”€ test_config.py
β”‚       β”œβ”€β”€ test_client.py
β”‚       └── test_cli.py
└── docs/
    └── DESIGN.md                   # This document

4. Configuration

Config is loaded with this priority (highest wins):

  1. CLI arguments (--url, --token, --theme)
  2. Environment variables (DEX_STUDIO_API_URL, DEX_STUDIO_API_TOKEN, …)
  3. Project-local .dex-studio.yaml
  4. User-level ~/.dex-studio/config.yaml
  5. Built-in defaults
# ~/.dex-studio/config.yaml
api_url: "http://localhost:8000"
api_token: null
timeout: 10.0
theme: dark
poll_interval: 5.0
window_width: 1400
window_height: 900

5. Lockstep Development Model

Each Studio page drives corresponding DEX engine API endpoints. When Studio needs data that DEX doesn't expose yet, the engine API must be extended first.

Studio Page DEX Engine Endpoints Required Status
Overview GET /, /health, /startup, /api/v1/data/quality, /api/v1/system/config βœ… Available
Health GET /health, /ready, /startup βœ… Available
Data Quality GET /api/v1/data/quality, /api/v1/data/quality/{layer}, /api/v1/data/sources, /api/v1/warehouse/layers βœ… Available
Lineage GET /api/v1/warehouse/lineage/{event_id} βœ… Available
ML Models GET /api/v1/models, /api/v1/models/{name}, POST /api/v1/predict πŸ”§ Router exists but was unmounted β€” fixed in lockstep
Settings (local only β€” no API calls) βœ… N/A

Lockstep Rule

No Studio page without a DEX API endpoint. No DEX API endpoint without a Studio page.

Future pages (pipelines, ingestion, drift detection) require new DEX engine endpoints. Those endpoints and pages are developed together in coordinated PRs across both repos.

6. DEX Engine Changes (v0.3.x Lockstep)

ML Router Mounted

The ML router (ml_router in dataenginex.api.routers.ml) existed but was never include_router()'d in the CareerDEX app. Fixed by adding:

from dataenginex.api.routers.ml import ml_router
app.include_router(ml_router)

This activates 3 previously dead endpoints:

  • POST /api/v1/predict
  • GET /api/v1/models
  • GET /api/v1/models/{name}

7. Technology Choices

Choice Rationale
NiceGUI Python-first, real routing, websockets, built on FastAPI
pywebview (native mode) Desktop-native window without Electron overhead
httpx Async HTTP client, connection pooling, same API as requests
Pydantic Config validation, matches DEX engine patterns
PyYAML Config file format
Hatchling Same build backend as DEX engine
Separate repo Different release cadence, users, and dependency trees

8. Phased Roadmap

Phase 0 β€” Foundation (v0.1.0) ← current

  • Project scaffold + CI
  • httpx client with health check
  • YAML config system (file + env + CLI)
  • 6 pages: Overview, Health, Quality, Lineage, ML Models, Settings
  • DEX engine lockstep: mount ML router

Phase 1 β€” Data Operations (v0.2.0)

  • Pipeline trigger page (needs POST /api/v1/pipelines/run on DEX engine)
  • Ingestion management (needs POST /api/v1/data/ingest)
  • Live pipeline log streaming (NiceGUI websocket)

Phase 2 β€” ML Workflows (v0.3.0)

  • Experiment comparison page
  • Drift detection alerts (needs GET /api/v1/ml/drift)
  • Model promotion workflow

Phase 3 β€” Observability (v0.4.0)

  • Prometheus metrics charts (needs structured GET /api/v1/metrics JSON endpoint)
  • Log viewer (needs GET /api/v1/logs)
  • Trace explorer (needs GET /api/v1/traces)

Phase 4 β€” Project Management (v0.5.0)

  • Project scaffolder (dex-studio init)
  • Config editor (edit DEX engine config from Studio)
  • Multi-instance support (connect to multiple DEX engines)

9. Non-Goals (v0.1.0)

  • Cloud deployment β€” Studio is local-first; no hosted SaaS version
  • Multi-user auth β€” single-user desktop app
  • Data editing β€” read-only dashboard; mutations are pipeline triggers only
  • Replacing Grafana β€” Studio shows DEX-specific views, not generic metrics dashboards
  • Package on PyPI β€” distribute via GitHub releases for now