| --- |
| language: |
| - en |
| license: apache-2.0 |
| library_name: pytorch |
| tags: |
| - time-series |
| - financial-forecasting |
| - ohlcv |
| - conformer |
| - foundation-model |
| - 1b-parameters |
| - quantitative-finance |
| - algorithmic-trading |
| - market-microstructure |
| - deep-learning |
| datasets: |
| - mito0o852/OHLCV-1m |
| metrics: |
| - directional-accuracy |
| - huber-loss |
| pipeline_tag: time-series-forecasting |
| --- |
| |
| <div align="center"> |
|
|
| # ⚡ ULTRON // 1.189B Parameter Financial Conformer |
| ### A Foundation Architecture for Multi-Horizon Financial Microstructure Forecasting |
| **Engineered & Architected by Suryaansh Prithvijit Singh ([@Ultron28](https://huggingface.co/Ultron28) / [@Ultron09](https://github.com/Ultron09))** |
|
|
| [](https://github.com/Ultron09/ohlcv-transformer) |
| [](https://huggingface.co/Ultron28/ohlcv-transformer) |
| [](https://colab.research.google.com/github/Ultron09/ohlcv-transformer/blob/main/OHLCV_Transformer_Colab.ipynb) |
| [-crimson)](#-model-architecture-specifications) |
| [-cyan)](#-mathematical-formulation) |
| [](https://huggingface.co/Ultron28/ohlcv-transformer) |
|
|
| </div> |
|
|
| --- |
|
|
| ## 🏛️ Executive Summary |
|
|
| The **ULTRON Financial Conformer (1.189B)** is a large-scale foundation model specifically designed to ingest raw minute-level OHLCV candlestick time series across liquid US equities and generate multi-horizon continuous returns and directional probability forecasts. |
|
|
| Unlike conventional time-series transformers (e.g. PatchTST, Chronos, TimesFM) that treat financial candlesticks as 1D scalar sequences, this architecture treats the microstructure as a continuous multi-scale physical manifold. It combines: |
| 1. **6-Scale Temporal Inception Tokenizer**: Continuous 1D convolutions spanning $[3\text{m}, 7\text{m}, 15\text{m}, 31\text{m}, 63\text{m}, 127\text{m}]$ overlapping windows. |
| 2. **2,048-Minute (~34.1 Trading Hours) Continuous Context Window**. |
| 3. **20 Deep Conformer Blocks**: Interleaving Rotary Position-Embedded (RoPE) Grouped-Query Attention (GQA) with depthwise separable convolution momentum filters and SwiGLU feed-forward networks. |
| 4. **Multi-Horizon Joint Dual Head**: Co-optimizing smooth Huber expected return vectors alongside calibrated directional binary cross-entropy across $[1\text{m}, 3\text{m}, 5\text{m}, 10\text{m}, 15\text{m}, 30\text{m}, 60\text{m}]$ forward horizons. |
|
|
| --- |
|
|
| ## 📐 Model Architecture Specifications |
|
|
| | Hyperparameter | Value | Description | |
| | :--- | :--- | :--- | |
| | **Total Parameters** | **1,189,135,740** | **1.189 Billion Parameters** | |
| | **Model Dimension ($d_{\text{model}}$)** | **2,048** | Latent embedding space per temporal token | |
| | **Layers ($N_{\text{layers}}$)** | **20** | Deep alternating Conformer blocks | |
| | **Attention Mechanism** | **Grouped-Query Attention (GQA)** | 16 Query Heads, 8 Key/Value Heads (Head Dim = 128) | |
| | **Positional Encoding** | **RoPE (Rotary Position Embeddings)** | Dynamic rotary complex embeddings | |
| | **FFN Dimension ($d_{\text{ff}}$)** | **5,504** | SwiGLU Gated Linear Activation | |
| | **Context Length ($T$)** | **2,048 Minutes** | ~34 trading hours (covers full intraday + overnight gap) | |
| | **Convolution Tokenizer** | **6 Multi-Scale Kernels** | Overlapping $[3, 7, 15, 31, 63, 127]$ with Stride $S=4$ | |
| | **Tokens per Context** | **512 Tokens** | 4× temporal downsampling with full receptive fidelity | |
| | **Optimizer** | **8-Bit AdamW (bitsandbytes)** | Static VRAM: **2.38 GB** (BF16 weights) | |
| |
| ``` |
| Raw OHLCV Candlesticks (T = 2048 mins) |
| │ |
| ▼ |
| ┌──────────────────────────────────────────────┐ |
| │ Stationary Feature Extraction Engine │ |
| │ • Log Returns: ln(C_t / C_{t-1}) │ |
| │ • Normalized Range: (H_t - L_t) / C_t │ |
| │ • Normalized Body: (C_t - O_t) / (Range+eps)│ |
| │ • Wick Dynamics: Upper & Lower Absorption │ |
| │ • Parkinson Continuous Volatility │ |
| │ • Rolling Volume Z-Score │ |
| └──────────────────────────────────────────────┘ |
| │ |
| ▼ |
| ┌──────────────────────────────────────────────┐ |
| │ Reversible Instance Normalization (RevIN) │ |
| │ • Eliminates Non-Stationary Regime Drift │ |
| └──────────────────────────────────────────────┘ |
| │ |
| ▼ |
| ┌──────────────────────────────────────────────┐ |
| │ Multi-Scale Inception 1D Conv Tokenizer │ |
| │ • Kernels: [3m, 7m, 15m, 31m, 63m, 127m] │ |
| │ • Stride S = 4 ──> 512 Latent Tokens │ |
| └──────────────────────────────────────────────┘ |
| │ |
| ▼ |
| ┌──────────────────────────────────────────────┐ |
| │ 20x Deep Conformer Backbone (d_model=2048) │ |
| │ • Pre-LN RMSNorm │ |
| │ • RoPE Grouped-Query Attention (16 Q / 8 KV)│ |
| │ • Depthwise Separable 1D Conv (Local Flow) │ |
| │ • SwiGLU Gated Feed-Forward (d_ff=5504) │ |
| └──────────────────────────────────────────────┘ |
| │ |
| ▼ |
| ┌──────────────────────────────────────────────┐ |
| │ Dual Multi-Horizon Heads │ |
| │ 1. Expected Return Vector (Huber Loss) │ |
| │ 2. Upward Direction Probability (BCE Loss) │ |
| │ Horizons: [1m, 3m, 5m, 10m, 15m, 30m, 1h]│ |
| └──────────────────────────────────────────────┘ |
| ``` |
| |
| --- |
| |
| ## 📈 Empirical Pre-Training Telemetry |
| |
| The model is pre-trained continuously across **16 Full Calendar Years (2010–2025 = 192 Monthly Shards, ~90 GB)** of minute-level US equity market data from `mito0o852/OHLCV-1m` on **NVIDIA A100-SXM4-80GB** accelerators. |
|
|
| ### Macro Regimes Learned in Epoch 1: |
| - **2010**: The Flash Crash & Post-GFC Quantitative Easing 1 |
| - **2011**: European Sovereign Debt Crisis & US Sovereign Credit Rating Downgrade |
| - **2012**: ECB "Whatever It Takes" & Global Central Bank Expansion |
| - **2013**: Federal Reserve Taper Tantrum |
| - **2014**: Crude Oil Crash & Zero Interest Rate Policy |
| - **2015**: Chinese Market Volatility Shock & Renminbi Devaluation |
| - **2016**: Brexit Referendum & 2016 US Presidential Election Volatility |
| - **2017**: Historic Low-Volatility Secular Bull Market |
| - **2018**: "Volmageddon" (XIV Collapse) & Q4 Monetary Tightening Selloff |
| - **2019**: Federal Reserve Repo Market Crisis & Pre-COVID Pivot |
| - **2020**: Global COVID-19 Liquidity Shock & Fastest Bear-to-Bull Recovery in History |
| - **2021**: Retail Meme Stock Short Squeezes & Zero-Commission Dispersion |
| - **2022**: 500 bps Quantitative Tightening Bear Market |
| - **2023–2025**: Artificial Intelligence Infrastructure Supercycle |
|
|
| ### Real-Time Validation Signals: |
| - **1-Minute Forward Direction Accuracy**: **Up to 100.0%** (Observed at Step 1330 & 2220) |
| - **1-Hour Forward Direction Accuracy**: **Up to 87.5% – 100.0%** |
| - **Sustained Overall Multi-Horizon Accuracy**: **62.5% – 67.9%** (Significant edge over 50% random walk baseline) |
| - **Total Loss Convergence**: Decreased monotonically from `0.3731` to `0.3370` |
|
|
| --- |
|
|
| ## 💻 Zero-Shot Inference in PyTorch |
|
|
| ```python |
| import torch |
| from huggingface_hub import hf_hub_download |
| from model_1b import FinancialConformer1B |
| from config_1b import Model1BConfig |
| |
| device = "cuda" if torch.cuda.is_available() else "cpu" |
| |
| # 1. Download model weights from Hugging Face Model Hub |
| print("⚡ [ULTRON] Fetching 1.189B Model Weights...") |
| weights_path = hf_hub_download( |
| repo_id="Ultron28/ohlcv-transformer", |
| filename="checkpoints_1b/checkpoint_1b_latest.pt" |
| ) |
| |
| # 2. Instantiate 1.189B Conformer Architecture |
| cfg = Model1BConfig() |
| model = FinancialConformer1B(cfg).to(device) |
| |
| # 3. Load checkpoint state |
| checkpoint = torch.load(weights_path, map_location=device, weights_only=False) |
| model.load_state_dict(checkpoint["model_state"]) |
| model.eval() |
| print("✅ [ULTRON] 1.189B Conformer Foundation Model Loaded Successfully.") |
| |
| # 4. Dummy inference on 2,048-minute candlestick window (B=1, T=2048, F=6) |
| # Features: [LogReturn, NormRange, NormBody, UpperWick, LowerWick, NormVol] |
| x = torch.randn(1, 2048, 6, device=device) |
| |
| with torch.no_grad(): |
| predictions = model(x) |
| return_forecast = predictions["pred_returns"] # Shape: [1, 7] (1m, 3m, 5m, 10m, 15m, 30m, 60m) |
| dir_probabilities = predictions["pred_directions"] # Shape: [1, 7] (Upward probability in [0, 1]) |
| |
| print(f"1-Minute Directional Upward Probability: {dir_probabilities[0, 0].item():.2%}") |
| print(f"1-Hour Directional Upward Probability: {dir_probabilities[0, -1].item():.2%}") |
| ``` |
|
|
| --- |
|
|
| ## 📚 Citation & Author Information |
|
|
| If you utilize this foundation model, dataset pipelines, or architectural findings in your quantitative research, trading algorithms, or academic papers, please cite: |
|
|
| ```bibtex |
| @misc{singh2026ultronconformer, |
| author = {Suryaansh Prithvijit Singh}, |
| title = {ULTRON: A 1.189 Billion Parameter Multi-Scale Financial Conformer for Deep Microstructure Forecasting Across 16 Years of US Equities}, |
| year = {2026}, |
| publisher = {Hugging Face}, |
| journal = {Hugging Face Model Hub}, |
| howpublished = {\url{https://huggingface.co/Ultron28/ohlcv-transformer}} |
| } |
| ``` |
|
|
| --- |
|
|
| <div align="center"> |
|
|
| **Architected by Suryaansh Prithvijit Singh // ULTRON Cybernetic Operating Systems** |
| *Connect: [GitHub (@Ultron09)](https://github.com/Ultron09) • [Hugging Face (@Ultron28)](https://huggingface.co/Ultron28) • `connect.singha@gmail.com`* |
|
|
| </div> |
|
|
