dex-demo / docs /DESIGN.md
data-literate's picture
chore: add workspace config, update CI, gitignore, README
de32734
|
Raw History Blame Contribute Delete
10.2 kB
# DEX Studio β€” Design Document
## 1. Purpose
DEX Studio is a **local, Python-first, open-source UI application** that provides a single
control plane for end-to-end data projects powered by [TheDataEngineX/DEX](https://github.com/TheDataEngineX/DEX).
It does **not** fork or rebrand upstream DEX. It is a separate tool that connects to a local
DEX engine instance and unifies workflows in one place:
```
Project Setup β†’ Ingestion β†’ Medallion Pipelines β†’ ML/AI β†’ Serving β†’ Observability
```
## 2. Architecture
```
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ DEX Studio (NiceGUI) β”‚
β”‚ β”‚
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
β”‚ β”‚ Pages β”‚ β”‚ Components β”‚ β”‚ State / Storage β”‚ β”‚
β”‚ β”‚ (views) β”‚ β”‚ (shared) β”‚ β”‚ (NiceGUI app.storage) β”‚ β”‚
β”‚ β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β”‚ β”‚ β”‚ β”‚ β”‚
β”‚ β”Œβ”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚
β”‚ β”‚ Studio Core β”‚ β”‚
β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚
β”‚ β”‚ β”‚ Config β”‚ β”‚ Client β”‚ β”‚ Theme β”‚ β”‚ β”‚
β”‚ β”‚ β”‚ (YAML) β”‚ β”‚ (httpx) β”‚ β”‚ (CSS) β”‚ β”‚ β”‚
β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚ HTTP (localhost:8000)
β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ DEX Engine (FastAPI) β”‚
β”‚ /health /ready /startup /metrics β”‚
β”‚ /api/v1/data/* /api/v1/warehouse/* /api/v1/system/* β”‚
β”‚ /api/v1/models /api/v1/models/{name} /api/v1/predict β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
```
### Why NiceGUI + Native Mode
| Requirement | NiceGUI Capability |
|--------------------------------------|-------------------------------------------|
| Python-first, no JS build step | βœ… Pure Python components |
| Real URL routing (`/quality`, `/models`) | βœ… `@ui.page` decorator |
| Websocket for live updates | βœ… Built-in, auto-managed |
| Desktop-native window | βœ… `native=True` via pywebview |
| Built on FastAPI | βœ… Same stack as DEX engine |
| Component model (cards, tables, tabs)| βœ… Quasar + custom components |
| Auth middleware (future) | βœ… FastAPI middleware support |
### Why NOT Streamlit
- No real URL routing (query params only)
- No websocket support (polling via rerun loop)
- No background tasks (script re-executes on every interaction)
- Performance degrades with complex layouts
- Can't embed into existing FastAPI infrastructure
## 3. Project Structure
```
dex-studio/
β”œβ”€β”€ pyproject.toml # Hatchling build, deps
β”œβ”€β”€ .python-version # 3.11
β”œβ”€β”€ config/
β”‚ └── default.yaml # Default config template
β”œβ”€β”€ src/
β”‚ └── dex_studio/
β”‚ β”œβ”€β”€ __init__.py # Package + version
β”‚ β”œβ”€β”€ cli.py # CLI entry (dex-studio command)
β”‚ β”œβ”€β”€ app.py # NiceGUI app bootstrap
β”‚ β”œβ”€β”€ config.py # YAML config loading
β”‚ β”œβ”€β”€ client.py # httpx DEX API client
β”‚ β”œβ”€β”€ theme.py # CSS theme + colour palette
β”‚ β”œβ”€β”€ components/
β”‚ β”‚ β”œβ”€β”€ sidebar.py # Navigation sidebar
β”‚ β”‚ β”œβ”€β”€ status_card.py # Health status cards
β”‚ β”‚ β”œβ”€β”€ metric_card.py # Metric display cards
β”‚ β”‚ └── page_layout.py # Sidebar + content wrapper
β”‚ └── pages/
β”‚ β”œβ”€β”€ overview.py # Dashboard (/)
β”‚ β”œβ”€β”€ health.py # Health probes (/health)
β”‚ β”œβ”€β”€ data_quality.py # Medallion quality (/quality)
β”‚ β”œβ”€β”€ lineage.py # Lineage explorer (/lineage)
β”‚ β”œβ”€β”€ ml_models.py # Model registry (/models)
β”‚ └── settings.py # Connection config (/settings)
β”œβ”€β”€ tests/
β”‚ β”œβ”€β”€ conftest.py
β”‚ └── unit/
β”‚ β”œβ”€β”€ test_config.py
β”‚ β”œβ”€β”€ test_client.py
β”‚ └── test_cli.py
└── docs/
└── DESIGN.md # This document
```
## 4. Configuration
Config is loaded with this priority (highest wins):
1. CLI arguments (`--url`, `--token`, `--theme`)
1. Environment variables (`DEX_STUDIO_API_URL`, `DEX_STUDIO_API_TOKEN`, …)
1. Project-local `.dex-studio.yaml`
1. User-level `~/.dex-studio/config.yaml`
1. Built-in defaults
```yaml
# ~/.dex-studio/config.yaml
api_url: "http://localhost:8000"
api_token: null
timeout: 10.0
theme: dark
poll_interval: 5.0
window_width: 1400
window_height: 900
```
## 5. Lockstep Development Model
Each Studio page drives corresponding DEX engine API endpoints.
When Studio needs data that DEX doesn't expose yet, the engine API must be extended first.
| Studio Page | DEX Engine Endpoints Required | Status |
|-----------------|-------------------------------------------------------------|------------|
| Overview | `GET /`, `/health`, `/startup`, `/api/v1/data/quality`, `/api/v1/system/config` | βœ… Available |
| Health | `GET /health`, `/ready`, `/startup` | βœ… Available |
| Data Quality | `GET /api/v1/data/quality`, `/api/v1/data/quality/{layer}`, `/api/v1/data/sources`, `/api/v1/warehouse/layers` | βœ… Available |
| Lineage | `GET /api/v1/warehouse/lineage/{event_id}` | βœ… Available |
| ML Models | `GET /api/v1/models`, `/api/v1/models/{name}`, `POST /api/v1/predict` | πŸ”§ Router exists but was unmounted β€” **fixed in lockstep** |
| Settings | (local only β€” no API calls) | βœ… N/A |
### Lockstep Rule
> **No Studio page without a DEX API endpoint. No DEX API endpoint without a Studio page.**
Future pages (pipelines, ingestion, drift detection) require new DEX engine endpoints.
Those endpoints and pages are developed together in coordinated PRs across both repos.
## 6. DEX Engine Changes (v0.3.x Lockstep)
### ML Router Mounted
The ML router (`ml_router` in `dataenginex.api.routers.ml`) existed but was never
`include_router()`'d in the CareerDEX app. Fixed by adding:
```python
from dataenginex.api.routers.ml import ml_router
app.include_router(ml_router)
```
This activates 3 previously dead endpoints:
- `POST /api/v1/predict`
- `GET /api/v1/models`
- `GET /api/v1/models/{name}`
## 7. Technology Choices
| Choice | Rationale |
|---------------------------|----------------------------------------------------------|
| **NiceGUI** | Python-first, real routing, websockets, built on FastAPI |
| **pywebview (native mode)** | Desktop-native window without Electron overhead |
| **httpx** | Async HTTP client, connection pooling, same API as requests |
| **Pydantic** | Config validation, matches DEX engine patterns |
| **PyYAML** | Config file format |
| **Hatchling** | Same build backend as DEX engine |
| **Separate repo** | Different release cadence, users, and dependency trees |
## 8. Phased Roadmap
### Phase 0 β€” Foundation (v0.1.0) ← current
- Project scaffold + CI
- httpx client with health check
- YAML config system (file + env + CLI)
- 6 pages: Overview, Health, Quality, Lineage, ML Models, Settings
- DEX engine lockstep: mount ML router
### Phase 1 β€” Data Operations (v0.2.0)
- Pipeline trigger page (needs `POST /api/v1/pipelines/run` on DEX engine)
- Ingestion management (needs `POST /api/v1/data/ingest`)
- Live pipeline log streaming (NiceGUI websocket)
### Phase 2 β€” ML Workflows (v0.3.0)
- Experiment comparison page
- Drift detection alerts (needs `GET /api/v1/ml/drift`)
- Model promotion workflow
### Phase 3 β€” Observability (v0.4.0)
- Prometheus metrics charts (needs structured `GET /api/v1/metrics` JSON endpoint)
- Log viewer (needs `GET /api/v1/logs`)
- Trace explorer (needs `GET /api/v1/traces`)
### Phase 4 β€” Project Management (v0.5.0)
- Project scaffolder (`dex-studio init`)
- Config editor (edit DEX engine config from Studio)
- Multi-instance support (connect to multiple DEX engines)
## 9. Non-Goals (v0.1.0)
- **Cloud deployment** β€” Studio is local-first; no hosted SaaS version
- **Multi-user auth** β€” single-user desktop app
- **Data editing** β€” read-only dashboard; mutations are pipeline triggers only
- **Replacing Grafana** β€” Studio shows DEX-specific views, not generic metrics dashboards
- **Package on PyPI** β€” distribute via GitHub releases for now