docs/quant_research_platform.md

Quant research platform — implementation and operation

This platform separates retained market data, candidate research, portfolio allocation, execution assumptions, and actual account PnL. A feature or an ML model is a candidate, not an established Alpha. Nothing in this workflow submits an order or promotes a model.

What is implemented across the 12 work areas

Area	Implemented and reviewable	Evidence still needed
1. Data foundation	Historical permissions probe; explicit IEX/SIP downloads; pagination; timezone, session calendar, source and file checks	Alpaca credentials; point-in-time dataset audit. The original four-symbol OHLCV was freshly re-queried from Yahoo and matched exactly; this is a same-vendor check
2. Broader universe	Separate 30-symbol research panel; multi-symbol historical downloader and experiment runner	30 symbols now have 34 retained full sessions (79,560 bars); historical membership and delisted names remain absent. Today's panel is subject to selection/survivorship bias
3. Alpha registry	Candidate definitions, feature names, source snapshots, package versions and all successful/unavailable trials retained	Stable incremental predictive evidence across independent periods
4. Price/volume research	Causal returns, VWAP deviation, relative volume, peer residual returns, time of day, overnight gap, seasonal volume	Longer history and cross-universe tests
5. Factor residual research	Past-only market residual feature when SPY is supplied; shrunk joint covariance and statistical PCA common risk	SPY is present in the expanded 30-symbol experiment; industry classification and factor-neutral validation remain absent
6. Genuine L1	IEX capture and historical quote/trade import; six measured L1 features; corrected OFI, quote age and direction classification	No actual L1 files present; quote and trade coverage, timestamp/reception-lag and cost effectiveness are untested
7. ML comparison	Same input families compared with Ridge, shallow tree, LightGBM; prior-session fitting and frozen hyperparameters	Calibrated uncertainty and longer forward performance; no reason yet to add deep learning
8. Validation	Independent exchange calendar, completed-bar/next-open ledger, daily training, identical costs, input/output hashes, descriptive paired daily differences	These reused dates are exploratory; no untouched holdout, multiple-testing-adjusted Alpha claim or long-run Sharpe claim
9. Portfolio	Correlation-aware joint allocation; separate uncertainty/common-risk ablations; transaction-cost penalty; integer shares and exposure diagnostics	Future outcomes; actual borrow availability, financing and execution constraints
10. Execution costs	Matching objective and ledger cost; fixed-order repricing kept separate; actual-fill TCA import	Actual fills and reference quotes needed to estimate spread, latency and impact; free IEX midpoint is not consolidated NBBO
11. PnL attribution	Gross/cost/net by symbol, direction, day, hour; cash/equity reconciliation; separate actual account flow adjustment	Broker exports and strategy-tagged fills needed for actual strategy PnL
12. Lab and presentation	Read-only verified comparison API and Lab UI; source archive; frozen future-decision logs; reproducible report and truthful resume text	Future observations and reconciled forward equity curve. The recorder is a command, not a running service

Free data workflow

Alpaca's official FAQ distinguishes free live IEX from paid live SIP. It also states historical SIP requests can be made without a subscription when end is at least 15 minutes old. This is an entitlement to verify with the user's account, not proof that a request has succeeded. No subscription upgrade is performed.

Source: https://docs.alpaca.markets/us/docs/market-data-faq

Configure ALPACA_API_KEY and ALPACA_SECRET_KEY in backend/.env locally. Do not put secrets in reports, commits, command arguments, or chat. The data tools read credentials directly and do not import the broker application configuration.

From the repository root:

python3 scripts/probe_research_permissions.py --output reports/history_probe_new.json
python3 scripts/download_yahoo_research_bars.py --universe30 --start 2026-07-27 --end 2026-09-12 --output data/yahoo_bars_new
python3 scripts/download_research_history.py --universe30 --start 2026-07-01T00:00:00Z --end 2026-09-12T00:00:00Z --feed sip --kind bars --output data/research_bars_new
python3 scripts/download_research_history.py --symbols SNDK TSLA MSTR NVDA --start 2026-09-08T13:30:00Z --end 2026-09-11T20:00:00Z --feed sip --kind quotes --output data/research_quotes_new
python3 scripts/download_research_history.py --symbols SNDK TSLA MSTR NVDA --start 2026-09-08T13:30:00Z --end 2026-09-11T20:00:00Z --feed sip --kind trades --output data/research_trades_new
python3 scripts/capture_alpaca_l1.py --universe liquid_us_iex_30 --feed iex

Quote/trade history can be large. Test a short interval first. An interrupted download is labelled incomplete and cannot silently become a complete research interval. Historical quote and trade directories can be passed together via --l1-dir. Do not mix IEX and SIP events into one feature series. L1 sizes retain their source units (quote round lots, trades shares); no invented L2/L3 levels are produced. The six L1 inputs are spread, displayed-size imbalance, microprice displacement, normalized OFI, classified-trade imbalance, and quote-update count. Unknown trade direction stays missing. The default prior-quote tolerance is a documented 30 seconds; it is a research assumption to evaluate, not a discovered market constant. OFI formula reference: https://arxiv.org/abs/1011.6402

Reproducible experiment

OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 python3 scripts/run_unified_research.py --bars reports/quant_audit_20260912/bars --symbols SNDK TSLA MSTR NVDA --start 2026-08-31 --end 2026-09-11 --costs 2 5 --equity 100000 --output reports/my_new_experiment

For the broader panel, pass the corresponding symbols and downloaded bar directory. For genuine L1, append --l1-dir data/research_quotes_new data/research_trades_new and supply enough prior sessions of both feeds for training. Four evaluation days alone are not enough to cover prior training dates. Missing required L1 makes the candidate unavailable, rather than turning missing order flow into a zero prediction.

The universe, 1-bar horizon, Ridge alpha 10, risk aversion 50, gross .95 and per-symbol .70 are explicit research specifications retained in the registry. Tree/LightGBM settings are also frozen in the archived source. The research and shadow portfolio solver is OSQP, solving the same convex objective. Default shared live solver behavior is retained; these research candidates are not deployed. A 30-stock SLSQP failure is retained as an incomplete experiment, and all final comparisons are rerun with OSQP rather than combining solver versions. Source: https://osqp.org/docs/interfaces/python.html . They are not alleged industry constants or optimized universal answers. Portfolio uncertainty and PCA risk coefficients are research preferences, not probabilities. All candidates use the same gross/per-symbol limits, next-open execution and scheduled 15:55 liquidation. The replay currently supports full 390-minute sessions; early-close requests fail with an explanation instead of silently dropping those dates.

Compare context Ridge against uncertainty-only, common-risk-only, and combined portfolio variants to isolate allocation effects while keeping forecasts fixed. Compare Ridge/tree/LightGBM with the same context features to isolate model choice. Changes in model and cost settings change trades: higher assumed costs can coincidentally produce higher retrospective PnL through a different order path. Fixed-order cost stress is the relevant calculation when asking how an existing path fares at higher costs.

All outputs are simulations using retained Yahoo OHLCV when using the bundled input. Hashes establish local consistency, not independent proof of vendor accuracy or point-in-time availability. A fresh same-vendor requery matched all original four-stock OHLCV values. Yahoo reported four dividend events and no splits in the expanded panel. Intraday-flat PnL does not receive overnight dividends; ex-dividend gaps remain raw model inputs, not a corporate-event Alpha. Borrow restrictions, financing, latency, market impact, intrabar drawdown and actual fills are absent. Do not relabel these as live PnL. The API /api/research/platform serves the final retained bundle read-only; dataset selects four_two_weeks, four_recent_week, or thirty_two_weeks. The API server itself retains existing broker startup behavior, so the offline workflow never starts it. The UI build is verified separately.

Future observation

python3 scripts/shadow_research.py register --directory reports/new_forward_trial --symbols SNDK TSLA MSTR NVDA --candidate context_robust --cost-bps 5
python3 scripts/shadow_research.py record --directory reports/new_forward_trial --bars data/current_bars --account-snapshot data/current_account.json
python3 scripts/shadow_research.py verify --directory reports/new_forward_trial

current_account.json must contain actual or clearly identified paper-account timestamp, equity, and shares for every registered symbol. Example schema:

{"timestamp":"<current timezone-aware timestamp>","equity":100000,"shares":{"SNDK":0,"TSLA":0,"MSTR":0,"NVDA":0},"account_kind":"paper"}

The timestamp placeholder must be replaced with the time of a real snapshot. The recorder requires synchronized completed bars after registration and fresh account/data snapshots; it logs predictions and target weights, archives inputs, and verifies a hash chain. It does not fill orders or invent paper equity changes. A stale observation is not a live trading halt: this is validation of research labels in a separate offline tool. No forced-stop or loss-lockout logic was added to trading. Registration alone creates zero future observations. Run during market hours with new data; collect actual paper fills/equity separately. A scheduled daemon and actual forward execution are not running. Candidate source or package changes require a new registration so experiments remain distinguishable.

Actual PnL and transaction-cost analysis

python3 scripts/reconcile_research_pnl.py account --snapshots data/account_equity.csv --output reports/actual_pnl_new.json
python3 scripts/reconcile_research_pnl.py tca --fills data/actual_fills.csv --quotes data/reference_quotes.csv --output reports/tca_new.csv

Account CSV columns: timestamp,equity,external_flow. External flow means signed deposits minus withdrawals since the preceding snapshot. PnL equals equity change minus external flows and includes unrealized PnL; it is not automatically attributable to this strategy. Fill CSV: symbol,fill_time,quantity,fill_price,commission with signed quantity. Quote CSV: symbol,timestamp,bid_price,ask_price,feed. TCA uses strictly prior same-feed quotes and reports missing references instead of inventing zero slippage. Slippage is a price comparison; do not subtract it again from account equity PnL.

Resume wording supported by the implementation

“Built an auditable multi-asset quantitative research platform with causal feature construction, prior-session model fitting, covariance-aware allocation, explicit execution-cost accounting, and reconciled PnL attribution. Compared Ridge, tree and LightGBM candidates against simple baselines; implemented genuine L1 ingestion and frozen forward-observation logs with reproducible data and source artifacts.”

Do not claim institutional profitability, a proven Alpha, live L1 advantage, or weekly $10,000 earnings before the corresponding independent evidence exists.

Counting features versus established Alpha

The retained selected live-policy artifact has 12 price/volume input features for SNDK, TSLA, MSTR and NVDA. The new offline registry has 12 candidate configurations including two baselines; it is not evidence of 12 independent Alphas. Peer features extend the input set to 16; context features to 19; non-SPY names can have a 20th market-residual input when SPY exists. Genuine L1 contributes six measured features, some algebraically related (microprice, spread and imbalance). These counts cannot be added up as independent predictive edges. No stable net-of-cost Alpha is yet established from these reused short intervals.

Only Ridge error diagnostics are used by uncertainty-aware allocation. Tree and LightGBM runs do not use their linear residual diagnostic field for allocation; that field must not be read as calibrated tree/boosting predictive uncertainty.