drift-quant / docs /frontend /FRONTEND_REDESIGN.md
deveshc23
feat: Add Free Gradio Space support, Git LFS tracking for data artifacts, and Quant Allocator
3e33e2e
|
Raw History Blame Contribute Delete
19.6 kB
# Frontend Redesign & Quantitative Research Workstation Architecture
**Project:** Drift Quant β€” NSE Quantitative Research Lab
**Document Version:** 1.0.0
**Status:** Approved for Implementation
**Target Environment:** React 19 + TypeScript + Vite + Tailwind/Modern CSS + FastAPI Backend
---
## 1. Executive Summary & Audit Findings
The initial version of Drift Quant proved the core data-integrity foundation: streaming inspection of a 260 MB ZIP archive (17,808,743 raw 15-minute observations across 488 symbols from 2019 to 2025), a verifiable audit pipeline, next-bar execution simulation, and four persisted holdout experiments.
However, the user interface was a single-view, read-only dashboard displaying only the fixed 2024–2025 holdout period across 10 hardcoded symbols. It lacked interactivity, exploration capabilities, dynamic backtesting, and multi-faceted quantitative analytics.
### Why the Previous Dashboard Was Restricted
1. **Static Experiment Index:** `scripts/run_nse_holdout_backtests.py` ran once with fixed parameters (`START = 2024-01-01`, `END = 2025-07-15`, 10 symbols) and saved to `artifacts/experiments/index.json`.
2. **Missing Market Data Endpoints:** The API only had 5 endpoints reading static files in `artifacts/experiments/` and `dataset_audit.json`. There was no endpoint to retrieve historical OHLCV bars, search symbols, or stream date ranges.
3. **No Execution API:** Backtests could only be launched via CLI scripts; there was no API to parameterize or trigger simulations.
4. **Single Monolithic View:** All components were crammed into `App.tsx` with a single Recharts line chart and no page routing.
### Data Audit Verification: 2019–2025 vs. 2024–2025 Holdout
The full dataset and holdout periods are strictly distinguished across the application:
| Dimension | Full Archive Dataset | Fixed Holdout Experiment |
| :--- | :--- | :--- |
| **Start Date** | `2019-01-01 09:15:00` | `2024-01-01 00:00:00` |
| **End Date** | `2025-07-14 11:00:00` | `2025-07-15 00:00:00` |
| **Total Rows** | 17,808,743 observations | 93,008 valid observations |
| **Symbol Coverage** | 488 NSE symbols | 10 selected large-cap symbols |
| **Invalid OHLC** | 72 rows (excluded) | 2 rows (excluded) |
| **Zero-Volume Bars** | 34,867 bars | Counted in symbol accounting |
| **Intraday Gaps** | 4,894 unexpected gaps | Monitored in quality metrics |
| **Purpose** | Full historical research universe | Out-of-sample holdout verification |
---
## 2. Visual Design System
Drift Quant is redesigned as a **dark-first, high-density institutional quantitative research terminal**, inspired by Bloomberg Terminal, FactSet, and Koyfin.
### Design Principles
1. **Information Density with Breathing Room:** Compact tables, dense metric grids, and maximized chart canvas area.
2. **Semantic Color Palette:**
- **Backgrounds:** Slate dark `#0a0d14` (root), `#111622` (surfaces/cards), `#171e2e` (hover/active), `#222b3d` (borders).
- **Text:** High-contrast white `#f1f5f9` (headers, values), muted slate `#94a3b8` (labels, subtitles), faint `#64748b` (metadata).
- **Returns / P&L:** Emerald `#10b981` (positive returns, profit), Rose/Crimson `#ef4444` (negative returns, drawdown), Cyan `#06b6d4` (benchmark/neutral), Amber `#f59e0b` (warnings, cost friction).
3. **Typography:** Monospace numbers (`JetBrains Mono`, `Fira Code`, or `ui-monospace`) for tabular financial figures with fixed-width glyph alignment; clean sans-serif (`Inter`, `system-ui`) for UI labels.
4. **Currency & Accounting Precision:**
- All monetary figures explicitly display Indian Rupee (`β‚Ή`) symbols with Indian numbering formatting (`β‚Ή12.45L`, `β‚Ή1.50Cr`, or `β‚Ή1,24,500.00`).
- Percentage returns (`+17.11%`, `-39.86%`) are strictly segregated from absolute currency amounts.
- Basis points (`bps`) are explicitly labelled on costs and slippage.
---
## 3. Global Architecture & Navigation
The application is structured into a persistent **collapsible left navigation rail**, a **compact global top bar**, and **ten dedicated analytical sections**:
```mermaid
flowchart TD
App[Drift Quant Workstation] --> TopBar[Global Header: System Status, Active Universe, Quick Stats]
App --> NavRail[Collapsible Navigation Rail]
App --> Content[Active Page Viewport]
NavRail --> P1[1. Overview Cockpit]
NavRail --> P2[2. Market Explorer]
NavRail --> P3[3. Strategy Lab]
NavRail --> P4[4. Backtest Results]
NavRail --> P5[5. Experiment Comparison]
NavRail --> P6[6. Portfolio Analytics]
NavRail --> P7[7. Trade Explorer]
NavRail --> P8[8. Data Quality & Coverage]
NavRail --> P9[9. Research Reports]
NavRail --> P10[10. System Health]
```
### Page Breakdown
### 1. Overview Cockpit
- **Firm-Level KPI Bar:** Total symbols (488), historical rows (17.8M), dataset span (2019–2025), completed experiments, best strategy by Sharpe, portfolio return, annualized volatility, max drawdown, total trades, total transaction costs.
- **Interactive Multi-Strategy Equity Chart:** Compare net equity curves of all saved experiments on normalized or absolute scales.
- **Drawdown Underwater Plot:** Synchronized drawdown curves highlighting maximum peak-to-trough drops and recovery durations.
- **Performance by Period:** Monthly returns matrix and calendar heatmap.
- **Strategy Leaderboard:** Sortable, filterable table of experiments with metric toggles.
- **Data Coverage & Quality Timeline:** Visual observation distribution across years and symbols.
- **Recent Experiments Feed:** Quick-access links with execution parameters and results.
### 2. Market Explorer
- **Instrument Selector:** Full searchable catalog of 488 symbols with instant filtering by sector, row count, and data health status.
- **Interactive Candlestick & Volume Chart:** High-performance OHLC candlestick chart with zoom/pan, crosshairs, time-resolution aggregations (15m, 1h, 1D), and indicator overlays:
- Exponential Moving Average (EMA 9, 21, 50, 200).
- Intraday VWAP Proxy (cumulative session typical price * volume).
- Average True Range (ATR 14).
- **Date Range Selector:** Presets (1M, 3M, 6M, 1Y, 3Y, 5Y, Full 2019–2025) and custom date picker.
- **Historical Observations Table:** Paginated raw bar records with timestamp, open, high, low, close, volume, and CSV export.
- **Data Quality Drawer:** Specific symbol statistics (zero-volume bars, invalid OHLC, intraday gaps).
### 3. Strategy Lab (Interactive Backtest Configurator)
- **Strategy Catalog:** Detailed documentation for each registered strategy:
- `buy_and_hold_baseline`
- `intraday_ema_momentum`
- `vwap_proxy_reversion`
- `atr_trend`
- `sector_relative_pairs` *(New)*
- `sector_cross_momentum` *(New)*
- **Configuration Controls:**
- Date ranges (with explicit validation against dataset availability).
- Universe selector (Holdout 10, Sector Baskets, or Custom symbol selection).
- Initial capital (default β‚Ή10,00,000), position sizing rule, max gross exposure.
- Transaction cost assumptions (default 10.0 bps) and slippage assumptions (default 5.0 bps).
- Strategy-specific parameters (fast/slow periods, deviation thresholds, ATR multiples).
- **Execution Engine:** Local async job queue with real-time progress, status transitions (Queued β†’ Running β†’ Completed / Failed), and instant link to generated Backtest Results.
### 4. Backtest Results
- **Summary Header:** Strategy version, dataset hash, simulation dates, capital, total net return, Sharpe, Sortino, max drawdown, win rate, profit factor, average win/loss, average holding time.
- **Net & Gross Equity Curves:** Interactive chart showing portfolio equity before and after transaction costs.
- **Drawdown Analysis:** Depth, duration, and time spent underwater.
- **Monthly & Yearly Heatmap:** Return breakdown by year and month.
- **Trade Analysis & Exit Reason Breakdown:** Donut/bar charts of win vs. loss, exits by rule vs. trailing stop vs. end-of-data liquidation.
- **Symbol Attribution:** Contribution by symbol (best and worst performing assets in the basket).
- **Configuration & Artifact Provenance:** Complete auditable JSON snapshot, command line invocation, and output file paths.
### 5. Experiment Comparison
- **Multi-Experiment Selector:** Select 2 to 6 experiments for side-by-side evaluation.
- **Normalized Cumulative Returns:** Base 100 or percentage return plot over common timeline.
- **Comparative Metrics Table:** Side-by-side matrix of return, Sharpe, Sortino, volatility, drawdown, trade count, turnover, and fees.
- **Configuration Difference Matrix:** Visual diff highlighting discrepancies in universe, dates, costs, or execution rules, with prominent incomparability warnings.
### 6. Portfolio & Risk Analytics
- **Asset Allocation & Weights:** Stacked area or time-series weight chart showing cash vs. equity allocation.
- **Gross & Net Exposure:** Leverage and cash buffer utilization over time.
- **Sector Concentration:** Exposure broken down by industry (Banking, IT, Energy, FMCG, Auto, etc.).
- **Rolling Volatility:** 30-day and 60-day annualized rolling volatility curves.
- **Correlation Matrix:** Inter-asset return correlation heatmap for active universe.
### 7. Trade Explorer
- **Comprehensive Ledger:** Full trade history across selected experiment.
- **Filters:** Symbol search, date range, direction (long), outcome (profitable vs. unprofitable), exit reason.
- **Summary Metrics:** Total trades, total gross P&L, total transaction costs, net P&L, profit factor.
- **Trade Inspection Modal:** Detailed bar-level breakdown of the trade entry and exit with price context.
- **CSV Export:** Direct download of the filtered trade ledger.
### 8. Data Quality & Coverage
- **Firm-Level Dataset Audit:** Total rows (17.8M), unique symbols (488), start/end dates.
- **Data Integrity Audit:**
- 72 Invalid OHLC records (detailed explanation of failure criteria).
- 34,867 Zero-volume bars.
- 4,894 Intraday unexpected gaps.
- 0 Duplicate timestamps / regressions.
- **Yearly Observation Bar Chart:** Bar volume by calendar year (2019 to 2025).
- **Per-Symbol Audit Table:** Filterable by symbol with individual gap counts, zero-volume counts, and bar counts.
- **Material Disclosures:** Transparent disclosure regarding absence of confirmed corporate actions, split/dividend adjustments, and historical NSE 500 survivorship bias.
### 9. Research Reports
- **Methodology Documentation:** Explicit explanation of next-bar execution rules, fee assumptions, and annualization factors.
- **Validation Archive:** Saved reports from initial and subsequent holdout runs.
- **Bias Analysis:** Overfitting warnings, multiple hypothesis testing disclosures.
### 10. System Health & Diagnostics
- **Backend Service Health:** Uptime, memory usage, API latency, artifact repository path.
- **Market Data Storage Health:** Status of SQLite cache, JSON caches, and ZIP member accessibility.
- **Active Job Monitor:** Current backtest worker state and execution log viewer.
---
## 4. Better Strategy Design: Sector-Relative Quantitative Strategies
The existing holdout experiments revealed severe weaknesses in naive intraday momentum, mean-reversion, and breakout strategies on 15-minute bars:
- `intraday_ema_momentum`: -39.86% return, 1,768 trades, β‚Ή3.32L costs (churned by intraday noise).
- `vwap_proxy_reversion`: -70.58% return, 2,518 trades, β‚Ή4.73L costs (caught in trending intraday moves).
- `atr_trend`: -23.30% return, 1,293 trades, β‚Ή2.43L costs (stop-outs and friction).
With 15 bps per side (30 bps roundtrip), trading 2,000 times destroys 60% of capital purely in transaction friction!
### Solution 1: Sector-Relative Pairs Trading (`sector_pairs_reversion`)
- **Premise:** Indian equities in the same sector share fundamental macroeconomic drivers, interest rate sensitivity, and sector rotation flows.
- **Pair Baskets:**
- Banking: `HDFCBANK` vs `ICICIBANK`, `AXISBANK` vs `SBIN`, `KOTAKBANK` vs `ICICIBANK`.
- Technology: `TCS` vs `INFY`, `WIPRO` vs `HCLTECH`.
- FMCG: `ITC` vs `HINDUNILVR`.
- Energy: `RELIANCE` vs `ONGC`.
- **Mechanism:** Computes rolling log price spread $S_t = \ln(P_A) - \ln(P_B)$ over a lookback window (e.g. 80 bars = ~2 trading days). Standardizes to Z-score:
$$Z_t = \frac{S_t - \mu_S}{\sigma_S}$$
- **Execution Rules (Long-Only):**
- When $Z_t < -1.75$: Stock A is significantly undervalued relative to Stock B. Enter long Stock A.
- When $Z_t > +1.75$: Stock B is significantly undervalued relative to Stock A. Enter long Stock B.
- Exit when spread normalizes ($|Z_t| < 0.25$) or on adverse divergence stop ($|Z_t| > 3.5$).
- **Turnover Control:** Minimum holding time filter (prevents rapid intraday switching), slashing transaction costs by 80%.
### Solution 2: Sector Cross-Sectional Momentum with Regime Filter (`sector_cross_momentum`)
- **Premise:** Sector leaders exhibit persistence over multi-day horizons.
- **Mechanism:** For each sector group, ranks constituent stocks by their intermediate-term returns (e.g. 50 bars = 1.5 days). Long only the top-ranked stock if the sector index is above its 100-bar moving average.
- **Turnover Control:** Rebalancing only occurs when rank shifts persist for at least 8 bars.
---
## 5. API Data Contracts & Endpoint Specifications
To power the workstation, the following backend endpoints are added to `src/drift_quant/api/app.py`:
```
GET /health -> System status, cache state, experiment count
GET /api/v1/dataset/summary -> Firm-level dataset summary (17.8M rows, 488 symbols)
GET /api/v1/dataset/symbols -> List of all 488 symbols with date coverage, row counts, sectors, data quality
GET /api/v1/dataset/coverage-timeline -> Yearly and monthly observation counts
GET /api/v1/market/bars -> Historical OHLCV bars (?symbol=RELIANCE&start=...&end=...&resolution=15m|1h|1d)
GET /api/v1/market/indicators -> Computed indicators (EMA, VWAP, ATR) for a symbol
GET /api/v1/strategies -> Catalog of registered strategies with JSON schemas
POST /api/v1/backtest/run -> Submit backtest job (validated strategy & parameters)
GET /api/v1/jobs/{job_id} -> Query job status (queued, running, completed, failed)
GET /api/v1/experiments -> List persisted experiments with summary metrics
GET /api/v1/experiments/{id} -> Full experiment detail & configuration
GET /api/v1/experiments/{id}/equity -> Downsampled or full net/gross equity series
GET /api/v1/experiments/{id}/drawdown -> Drawdown series with peak, trough, duration
GET /api/v1/experiments/{id}/trades -> Filterable, paginated trade ledger
GET /api/v1/experiments/{id}/attribution -> Per-symbol P&L attribution and win rates
GET /api/v1/experiments/{id}/monthly -> Monthly and yearly return matrix
GET /api/v1/experiments/{id}/risk -> Advanced risk metrics (VaR, Sortino, exposure)
GET /api/v1/experiments/compare -> Comparative multi-experiment summary
GET /api/v1/reports/{id}/export-csv -> CSV download for trades or equity curve
```
---
## 6. Frontend Component Hierarchy
```text
frontend/src/
β”œβ”€β”€ api/
β”‚ β”œβ”€β”€ client.ts # Strongly-typed fetch client with error handling & cancellation
β”‚ └── types.ts # Full TypeScript models matching backend schemas
β”œβ”€β”€ components/
β”‚ β”œβ”€β”€ common/
β”‚ β”‚ β”œβ”€β”€ Badge.tsx # Tone badges (positive, negative, warning, neutral)
β”‚ β”‚ β”œβ”€β”€ Button.tsx # Styled buttons (primary, secondary, ghost, danger)
β”‚ β”‚ β”œβ”€β”€ Card.tsx # Analytical panel containers with titles & actions
β”‚ β”‚ β”œβ”€β”€ DatePicker.tsx # Date range selector with presets
β”‚ β”‚ β”œβ”€β”€ EmptyState.tsx # Informative empty states with guidance
β”‚ β”‚ β”œβ”€β”€ ErrorState.tsx # Actionable error banners
β”‚ β”‚ β”œβ”€β”€ KpiCard.tsx # Standardized financial KPI card with tone & format
β”‚ β”‚ β”œβ”€β”€ LoadingSkeleton.tsx # Skeleton loaders for charts, cards, and tables
β”‚ β”‚ β”œβ”€β”€ Modal.tsx # Accessible overlay dialog
β”‚ β”‚ └── Tooltip.tsx # Contextual financial definitions
β”‚ β”œβ”€β”€ charts/
β”‚ β”‚ β”œβ”€β”€ CandlestickChart.tsx # High-density OHLC candlestick + volume + overlays
β”‚ β”‚ β”œβ”€β”€ EquityChart.tsx # Cumulative equity curve (net vs gross vs benchmark)
β”‚ β”‚ β”œβ”€β”€ DrawdownChart.tsx # Underwater drawdown curve
β”‚ β”‚ β”œβ”€β”€ MonthlyHeatmap.tsx # Calendar month/year return matrix
β”‚ β”‚ └── AllocationChart.tsx # Exposure & position weights over time
β”‚ β”œβ”€β”€ layout/
β”‚ β”‚ β”œβ”€β”€ TopBar.tsx # Global header with system health & active universe
β”‚ β”‚ β”œβ”€β”€ Sidebar.tsx # Collapsible navigation rail with active route
β”‚ β”‚ └── Shell.tsx # Master application shell
β”œβ”€β”€ pages/
β”‚ β”œβ”€β”€ OverviewPage.tsx # Firm research overview cockpit
β”‚ β”œβ”€β”€ MarketExplorerPage.tsx # Deep 2019-2025 market data explorer & candles
β”‚ β”œβ”€β”€ StrategyLabPage.tsx # Interactive backtest configurator & job runner
β”‚ β”œβ”€β”€ BacktestResultsPage.tsx # In-depth single experiment analytics
β”‚ β”œβ”€β”€ ExperimentComparePage.tsx # Multi-experiment side-by-side comparison
β”‚ β”œβ”€β”€ PortfolioPage.tsx # Portfolio allocation, concentration & risk
β”‚ β”œβ”€β”€ TradeExplorerPage.tsx # Filterable trade ledger & trade modal
β”‚ β”œβ”€β”€ DataQualityPage.tsx # Dataset audit, gaps, zero-volume records
β”‚ β”œβ”€β”€ ResearchReportsPage.tsx # Methodology & validation documentation
β”‚ └── SystemHealthPage.tsx # API health, SQLite cache status, job queue
β”œβ”€β”€ utils/
β”‚ β”œβ”€β”€ formatters.ts # Currency (β‚Ή INR), percentages, numbers, dates
β”‚ └── math.ts # Downsampling, rolling metrics, attribution
β”œβ”€β”€ App.tsx # Main router and global context provider
β”œβ”€β”€ main.tsx # React entry point
└── styles.css # Polished design system tokens and styling
```
---
## 7. Implementation Roadmap
1. **Step 1:** Build market data cache (SQLite + JSON) and verify full 2019–2025 coverage across liquid sectors.
2. **Step 2:** Implement the new `sector_pairs_reversion` and `sector_cross_momentum` strategies in `src/drift_quant/backtesting/strategies.py` and register them.
3. **Step 3:** Extend backend FastAPI endpoints (`app.py`, `repository.py`) to serve market bars, symbol catalog, backtest execution runner, attribution, and risk.
4. **Step 4:** Write comprehensive backend tests for new endpoints, strategy algorithms, and caching logic.
5. **Step 5:** Build the frontend API client layer (`api/client.ts`, `api/types.ts`) and formatting utilities with complete INR (β‚Ή) conventions.
6. **Step 6:** Construct the workstation UI components (Sidebar, TopBar, KpiCards, Financial Charts, Data Tables).
7. **Step 7:** Implement the 10 dedicated workstation pages and connect to live backend data.
8. **Step 8:** Optimize build bundles with dynamic code-splitting and evaluate bundle size warnings.
9. **Step 9:** Execute full frontend tests and verification, run backend test suite, and document results.
10. **Step 10:** Update `docs/frontend/FRONTEND_GUIDE.md`, root `README.md`, and API reference.