Add SDK example notebooks + link them from the model card

README.md

@@ -47,6 +47,19 @@ CHARM is served behind an API and consumed through the open-source [`c3-charm`](
 
 > **Dual-model serving.** A CHARM server can be backed by two independent checkpoints — one for embeddings (`/predict`, `client.embeddings`) and a separate one for forecasting (`/forecast`, `client.prediction`). They may differ in architecture, patch size, and embedding dimension, so read per-model properties from `client.model_info()` rather than hardcoding.
 
+### Example notebooks
+
+Runnable notebooks live in [`notebooks/`](./notebooks) in this repo. Open them on the Hub, or download the folder and run locally after `pip install c3-charm[toolkit]` (set `CHARM_BASE_URL` / `CHARM_API_KEY` first). The classification and reconstruction/forecasting demos read the small sample datasets in [`notebooks/data/`](./notebooks/data).
+
+| Notebook | What it covers |
+|---|---|
+| [`getting_started.ipynb`](./notebooks/getting_started.ipynb) | Client setup, first embeddings call, inspecting `model_info()`. |
+| [`charm_toolkit_demo.ipynb`](./notebooks/charm_toolkit_demo.ipynb) | Tour of the `charm_toolkit` — datasets, precompute, trainer, and each task head. |
+| [`demo_forecasting.ipynb`](./notebooks/demo_forecasting.ipynb) | Zero-shot quantile forecasting and the embedding-based `ForecastingModel` head (weather data). |
+| [`demo_reconstruction.ipynb`](./notebooks/demo_reconstruction.ipynb) | Backcast reconstruction and the `ReconstructionModel` head for anomaly detection (weather data). |
+| [`demo_classification.ipynb`](./notebooks/demo_classification.ipynb) | Time-series classification with the `ClassificationModel` head (BasicMotions data). |
+| [`demo_retrieval_anomaly_detection.ipynb`](./notebooks/demo_retrieval_anomaly_detection.ipynb) | Embedding retrieval + kNN / zero-shot anomaly-detection recipes. |
+
 ### Installation
 
 ```bash

notebooks/charm_toolkit_demo.ipynb

@@ -0,0 +1,237 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "\n",
+    "# Using the Time Series Dataset Utilities\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setup"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import torch\n",
+    "import numpy as np\n",
+    "from sklearn.preprocessing import StandardScaler\n",
+    "import matplotlib.pylab as plt\n",
+    "\n",
+    "from charm_toolkit.Data import Data\n",
+    "from charm_toolkit.BaseTimesSeries import (\n",
+    "    UnwindowedDataset,\n",
+    "    PreWindowedDataset,\n",
+    "    LazyWindowedDataset,\n",
+    "    ForecastingDataset,\n",
+    "    train_test_split,\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "###  Create an UnwindowedDataset\n",
+    "\n",
+    "We can use the `UnwindowedDataset` class as a simple container for **raw, unwindowed time series** of shape\n",
+    "`(T, C)`, where `T` is the number of time steps and `C` is the number of channels. Key features of this class\n",
+    "\n",
+    "- Stores the full time series in `self.data` as a `torch.Tensor` of shape `(T, C)`.\n",
+    "- Optionally applies a provided `StandardScaler` at construction time; otherwise it keeps\n",
+    "  the data as-is (but you can later call `apply_scaler` from `BaseTimeSeriesDataset`).\n",
+    "- `__len__()` returns the **number of time steps** `T`.\n",
+    "- **Does not support direct indexing** via `__getitem__` (it raises an error instead).\n",
+    "  Instead, it is meant to be converted into task-specific datasets:\n",
+    "  - `.window(window_size, stride)` → returns a `LazyWindowedDataset` that creates\n",
+    "    sliding windows on-the-fly.\n",
+    "  - `.forecasting(train_horizon, test_horizon)` → returns a `ForecastingDataset`\n",
+    "    that produces (input, target) pairs for forecasting."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "T, C = 100, 3\n",
+    "data = np.random.randn(T, C).astype(np.float32)\n",
+    "descriptions = [\"temperature\", \"pressure\", \"vibration\"]\n",
+    "\n",
+    "unwindowed_ds = UnwindowedDataset(\n",
+    "    data=data,\n",
+    "    dataset_name=\"demo_unwindowed\",\n",
+    "    descriptions=descriptions\n",
+    ")\n",
+    "\n",
+    "print(\"Raw dataset shape:\", unwindowed_ds.data.shape)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Scaling\n",
+    "You can now apply scaling before splitting to avoid leakage as below"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "unwindowed_ds.apply_scaler()\n",
+    "print(\"First 5 scaled rows:\\n\", unwindowed_ds.data[:5])\n",
+    "\n",
+    "original = unwindowed_ds.inverse_transform(unwindowed_ds.data[:5])\n",
+    "print(\"Inverse transformed:\\n\", original)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Train/Test Split\n",
+    "You can also split data sequentially or randomly for training, validation and testing.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "train_ds, test_ds = train_test_split(unwindowed_ds, train_ratio=0.8)\n",
+    "\n",
+    "print(\"Train length:\", len(train_ds))\n",
+    "print(\"Test length:\", len(test_ds))\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "### Lazy-windowed Dataset\n",
+    "It is commonly needed to create windowed versions of a time series data in a sliding fashion. We can do so on the fly from an UnwindowedDataset. In order to do so in a memory efficient manner for long time series, we do this lazily as below."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "window_size = 10\n",
+    "stride = 5\n",
+    "\n",
+    "lazy_win_train = train_ds.window(window_size=window_size, stride=stride)\n",
+    "\n",
+    "print(\"Number of windows:\", len(lazy_win_train))\n",
+    "\n",
+    "# Inspect a window\n",
+    "sample_window = lazy_win_train[0]\n",
+    "print(\"Window shape:\", sample_window['ts_array'].shape)\n",
+    "print(\"Descriptions:\", sample_window['descriptions'])\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Using All Together: Split → Window → Forecast\n",
+    "\n",
+    "We can now put everything together. Below is an example of this.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# 1. Split\n",
+    "train_ds, test_ds = train_test_split(unwindowed_ds, train_ratio=0.7)\n",
+    "\n",
+    "# 2. Lazy windowing\n",
+    "lazy_train = train_ds.window(window_size=15, stride=5)\n",
+    "lazy_test = test_ds.window(window_size=15, stride=5)\n",
+    "\n",
+    "print(\"Lazy train windows:\", len(lazy_train))\n",
+    "print(\"Lazy test windows:\", len(lazy_test))\n",
+    "\n",
+    "# 3. Forecasting\n",
+    "forecast_train = ForecastingDataset(train_ds, train_horizon=15, test_horizon=5)\n",
+    "forecast_test = ForecastingDataset(test_ds, train_horizon=15, test_horizon=5)\n",
+    "\n",
+    "print(\"Forecast train pairs:\", len(forecast_train))\n",
+    "print(\"Forecast test pairs:\", len(forecast_test))\n",
+    "\n",
+    "\n",
+    "\n",
+    "# Take the first forecasting pair\n",
+    "x, y = forecast_test[0]\n",
+    "x_np = x['ts_array'][0].numpy()\n",
+    "y_np = y['ts_array'][0].numpy()\n",
+    "\n",
+    "time_input = np.arange(len(x_np))\n",
+    "time_target = np.arange(len(x_np), len(x_np) + len(y_np))\n",
+    "\n",
+    "plt.figure(figsize=(10,4))\n",
+    "\n",
+    "# Plot input sequence\n",
+    "for i, desc in enumerate(x['descriptions']):\n",
+    "    plt.plot(time_input, x_np[:, i], label=f\"{desc} (input)\", linestyle='-')\n",
+    "\n",
+    "# Plot target sequence\n",
+    "for i, desc in enumerate(y['descriptions']):\n",
+    "    plt.plot(time_target, y_np[:, i], label=f\"{desc} (target)\", linestyle='--')\n",
+    "\n",
+    "plt.title(\"Forecasting Example: Input vs Target\")\n",
+    "plt.xlabel(\"Time Step\")\n",
+    "plt.ylabel(\"Value\")\n",
+    "plt.legend()\n",
+    "plt.show()\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "charm",
+   "language": "python",
+   "name": "charm"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

notebooks/demo_classification.ipynb

@@ -0,0 +1,507 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "62b712f6-5a23-4de9-af1d-9cb94191f111",
+   "metadata": {},
+   "source": [
+    "# Time Series Classsification with Charm Embeddings\n",
+    "\n",
+    "This notebook walks through a full example of **time-series Classsification** using:\n",
+    "\n",
+    "- A synthetic multi-channel sinewave dataset  \n",
+    "- Your **CharmClient** for embeddings  \n",
+    "- Your custom **ClasssificationModel**  \n",
+    "- A clean dataset pipeline with sliding windows  \n",
+    "- The training loop powered by your custom **Trainer**  \n",
+    "- A visualization callback that plots classsification during training  \n",
+    "\n",
+    "The goal is pedagogical: understand *every single component* of the library.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "97f74300-819e-4a66-8fe7-048c4fe8976a",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Setup: Imports & Charm Client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6c96e1cb-2d56-487c-af33-9fb7eef2d7cf",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from charm import CharmClient\n",
+    "from dotenv import load_dotenv\n",
+    "import os\n",
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "import torch\n",
+    "from torch.utils.data import DataLoader\n",
+    "import matplotlib.pyplot as plt\n",
+    "\n",
+    "# # Load environment variables from .env file\n",
+    "# load_dotenv()\n",
+    "\n",
+    "# # Get API key and base URL from environment variables\n",
+    "# api_key = os.getenv(\"CHARM_API_KEY\", \"sk-charm-default-key\")\n",
+    "# base_url = os.getenv(\"CHARM_BASE_URL\", \"EMBEDDING_MODEL_ENDPOINT\")\n",
+    "\n",
+    "api_key = \"sk-charm-default-key\"\n",
+    "base_url = \"EMBEDDING_MODEL_ENDPOINT\"\n",
+    "\n",
+    "# Create a client for embedding extraction\n",
+    "client = CharmClient(\n",
+    "    base_url=base_url,\n",
+    "    api_key=api_key,\n",
+    "    timeout=60,\n",
+    "    max_retries=3\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "92b27303-c19d-4d9a-9cbb-180832c97171",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Dataset: Basic Motions\n",
+    "\n",
+    "For this tutorial notebook, we use the `Basic Motions` dataset. We have seeded this dataset under the `data` forlder. Below we load and process this dataset."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3f988b4a-8fc8-4e5c-a3b2-c0241909af8f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df = pd.read_csv(\"data/basicmotions.csv\")\n",
+    "\n",
+    "# channel names\n",
+    "channel_names = df.columns[2:]\n",
+    "\n",
+    "# get raw numeric data\n",
+    "data_np = df[channel_names].values\n",
+    "\n",
+    "# convert to torc\n",
+    "tdata = torch.tensor(data_np)\n",
+    "\n",
+    "# reconstruct (B, T, C)\n",
+    "B = df[\"sample_id\"].nunique()\n",
+    "T = df[df[\"sample_id\"] == 0].shape[0]  # time steps per sample\n",
+    "C = len(channel_names)\n",
+    "\n",
+    "\n",
+    "data = tdata.reshape(B, T, C)\n",
+    "\n",
+    "labels = df.drop_duplicates(\"sample_id\").sort_values(\"sample_id\")[\"label\"].tolist()\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8c6a76d2-dda8-44e8-9688-5a9da10f455c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "\n",
+    "# We will have to convert the labels to IDs \n",
+    "\n",
+    "# Create mapping\n",
+    "label_to_id = {label: i for i, label in enumerate(sorted(set(labels)))}\n",
+    "\n",
+    "# Convert\n",
+    "labels = [label_to_id[l] for l in labels]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c4b88dd3-6281-4988-a657-ef0509640027",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Classifier Dataset Splits\n",
+    "\n",
+    "To prepare our dataset for training, validation and testing, we use the function `create_classification_datasets`. This function prepares train/validation/test splits for time-series\n",
+    "classification tasks. It assumes each time series has a single label. The function:\n",
+    "\n",
+    "1. **Wraps the raw data and labels**  \n",
+    "   Creates a `PreWindowedDataset` where each sample is a full time series plus its\n",
+    "   channel descriptions and classification label.\n",
+    "\n",
+    "2. **Splits the dataset**  \n",
+    "   Divides the data into train/val/test sets using either sequential or random splitting (here random)\n",
+    "   (`train_ratio`, `val_ratio`).\n",
+    "\n",
+    "3. **Builds classification-ready datasets**  \n",
+    "   Wraps each split in a `ClassificationDataset`, ensuring the model receives data in the\n",
+    "   correct structure expected by the `ClassificationModel`.\n",
+    "\n",
+    "4. **Returns the three dataset splits**  \n",
+    "   Ready to be used with PyTorch `DataLoader`s."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cbc47cb0-7e23-4789-ab41-6fccc50e1e05",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from charm_toolkit.Datasets import create_classification_datasets, collator\n",
+    "\n",
+    "train_ds, val_ds, test_ds = create_classification_datasets(\n",
+    "    raw_data=data,                                # [B, T, C] tensor; here [80, 100, 6]\n",
+    "    labels=labels,                                # List of labels, one per time series.\n",
+    "    descriptions=list(channel_names),             # list of channel names, e.g., [\"ch0\", \"ch1\"]\n",
+    "    \n",
+    "    train_ratio=0.7,                              # fraction of data used for training (0.7 → 70%)\n",
+    "    val_ratio=0.15,                               # fraction of data used for validation (0.15 → 15%)\n",
+    "    sequential=False,                             # if True, splits are sequential, not shuffled\n",
+    ")\n",
+    "\n",
+    "\n",
+    "train_loader = DataLoader(train_ds, batch_size=32, collate_fn=collator, num_workers=1)\n",
+    "val_loader   = DataLoader(val_ds, batch_size=32, collate_fn=collator, num_workers=1)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e48673a2-3f4d-4336-b562-9e1f433378ee",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Instantiating the Model\n",
+    "\n",
+    "We use the `ClassificationModel` class to build a classifier based on the CHARM embeddings. Given raw time series (`ts_array` + `descriptions`), it\n",
+    "first obtains embeddings of shape **[B, T_p, C, D]**, where:\n",
+    "\n",
+    "- **T_p** — number of temporal patches  \n",
+    "- **C** — channels  \n",
+    "- **D** — embedding dimension  \n",
+    "\n",
+    "The model then applies **two stages of pooling** to convert these embeddings into a fixed-size\n",
+    "feature vector:\n",
+    "\n",
+    "#### Temporal pooling (`pooling_over_t`)\n",
+    "- **\"mean\"** – average over temporal patches  \n",
+    "- **\"last\"** – use the final temporal patch  \n",
+    "- **\"flatten\"** – keep all patches (most expressive)\n",
+    "\n",
+    "#### Channel pooling (`pooling_over_channels`)\n",
+    "- **\"mean\"** – average across channels  \n",
+    "- **\"flatten\"** – keep all channels\n",
+    "\n",
+    "These pooling options produce feature vectors ranging from compact (`[B, D]`) to highly\n",
+    "expressive (`[B, T_p × C × D]`), depending on how much structure you want to retain.\n",
+    "\n",
+    "The pooled vector is passed through a **classifier head**, chosen via `classifier_type`:\n",
+    "- **\"linear\"** – single fully connected layer  \n",
+    "- **\"mlp\"** – 2-layer MLP with GELU + dropout  \n",
+    "- **\"svm\"** – differentiable SVM-like head (linear or RBF kernel)  \n",
+    "- or a custom `nn.Module`\n",
+    "\n",
+    "The model outputs **logits of shape `[B, num_classes]`**."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fa09e6b6-75b2-467a-9746-6d2c13df8585",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from charm_toolkit.models import ClassificationModel\n",
+    "\n",
+    "model = ClassificationModel(\n",
+    "    embedding_client=client,  \n",
+    "    num_classes=len(set(labels)),\n",
+    "    # Number of classes to predict.\n",
+    "    \n",
+    "    hidden_dim=128,              \n",
+    "    # Used only for MLP-style classifier.\n",
+    "    # Controls width of intermediate layers.\n",
+    "\n",
+    "    dropout=0.1,                 \n",
+    "    # Dropout applied\n",
+    "\n",
+    "    pooling_over_t= \"mean\",\n",
+    "    # Pooling mode for temporal dimension (\"flatten\", \"mean\", \"last\")\n",
+    "\n",
+    "    pooling_over_channels=\"mean\",\n",
+    "    # Pooling mode for channel dimension (\"flatten\", \"mean\")\n",
+    "\n",
+    "    classifier_type=\"mlp\",\n",
+    "    # Type of classifier to use (\"linear\", \"mlp\", \"svm\") or a custom module.\n",
+    "    # Default MLP\n",
+    "\n",
+    "    kernel=None,\n",
+    "    # Optional: Kernel type for SVM classifier. \"linear\" and \"rbf\" supported.\n",
+    "    \n",
+    "    gamma =0.1,\n",
+    "    # Gamma parameter for RBF kernel in SVM\n",
+    "    \n",
+    "    T=None,\n",
+    "    # Expected temporal dimension (required if pooling_over_t=\"flatten\")\n",
+    "\n",
+    "    num_channels=None,\n",
+    "    # Number of channels (required if pooling_over_channels=\"flatten\")\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3cc41797-cca5-4b35-97c4-aacfe6de2fd9",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Setting Up the Trainer\n",
+    "\n",
+    "Having defined the model architecture, we can now train it. The `Trainer` class provides a lightweight, flexible training loop for PyTorch models,\n",
+    "including support for validation, early stopping, gradient accumulation, gradient clipping,\n",
+    "learning-rate scheduling, callbacks, and basic visualization snapshots.\n",
+    "\n",
+    "A `Trainer` instance is created with:\n",
+    "- a model (`nn.Module`)\n",
+    "- `train_loader`, and optionally `val_loader` / `test_loader`\n",
+    "- training hyperparameters such as learning rate, weight decay, epochs,\n",
+    "  gradient accumulation steps, and early-stopping parameters.\n",
+    "\n",
+    "During training (`fit()`):\n",
+    "1. **Each epoch streams batches** from the `DataLoader` (no full-dataset loading).\n",
+    "2. For each batch, it:\n",
+    "   - runs a forward pass\n",
+    "   - computes the loss\n",
+    "   - performs optional gradient accumulation (`accum_steps`)\n",
+    "   - applies gradient clipping (`max_grad_norm`)\n",
+    "   - updates the optimizer\n",
+    "3. After each epoch, it:\n",
+    "   - evaluates on the validation set (if provided)\n",
+    "   - steps the scheduler (e.g., `ReduceLROnPlateau`)\n",
+    "   - triggers any user-defined callbacks\n",
+    "   - checks early-stopping conditions (`patience`, `min_delta`)\n",
+    "4. The best model (lowest validation loss) is stored and restored at the end.\n",
+    "\n",
+    "During validation, the trainer also stores a small number of snapshots (targets, predictions, inputs) for visualization. The `evaluate()` method runs a full validation/testing pass and returns the average loss."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5721f474-5b7f-49cb-9df9-c97bf75f5645",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from charm_toolkit.Trainer import Trainer\n",
+    "from charm_toolkit.Plots import PlotCallback\n",
+    "\n",
+    "plot_cb = PlotCallback(\n",
+    "    every=10,           \n",
+    "    # Plot every N steps\n",
+    "\n",
+    "    show_class=True,    \n",
+    "    # Plot original vs classified\n",
+    "\n",
+    "    n_samples=3,        \n",
+    "    # number of samples to plot\n",
+    "\n",
+    "    channels=[0,1],         \n",
+    "    # int or list of channels to visualize\n",
+    ")\n",
+    "\n",
+    "\n",
+    "trainer = Trainer(\n",
+    "    model=model,\n",
+    "    train_loader=train_loader,\n",
+    "    val_loader=val_loader,\n",
+    "    test_loader=None,  \n",
+    "    # optional — only needed if you call trainer.test()\n",
+    "\n",
+    "    lr=1e-3,             \n",
+    "    # learning rate. Default:1e-3\n",
+    "\n",
+    "    weight_decay=1e-4,   \n",
+    "    # L2 regularization. Default:1e-4\n",
+    "\n",
+    "    epochs=1000,         \n",
+    "    # maximum training epochs.  Default:1000\n",
+    "\n",
+    "    accum_steps=1,       \n",
+    "    # gradient accumulation:\n",
+    "    #   if >1: effectively increases batch size without increasing memory\n",
+    "    # Default: 1\n",
+    "\n",
+    "    patience=20,          \n",
+    "    # early stopping patience. Default: 5\n",
+    "\n",
+    "    min_delta=1e-4,      \n",
+    "    # minimum improvement required for early stopping. Default: 1e-4\n",
+    "\n",
+    "    max_grad_norm=5.0,   \n",
+    "    # gradient clipping for stability.  Default: 5\n",
+    "\n",
+    "    device=None,         \n",
+    "    # default: \"cuda\" if available else \"cpu\"\n",
+    "\n",
+    "    save_dir=\"./classification_checkpoints\",\n",
+    "    # where best checkpoints are saved\n",
+    "\n",
+    "    callbacks=[plot_cb], \n",
+    "    # supports ANY number of callbacks:\n",
+    "    #   - plotting\n",
+    "    #   - logging (wandb, tensorboard)\n",
+    "    #   - checkpointing\n",
+    "    #   - custom user hooks\n",
+    "\n",
+    "    criterion=torch.nn.CrossEntropyLoss(),      \n",
+    "    # default: MSELoss\n",
+    "    # user may pass:\n",
+    "    #   nn.L1Loss()\n",
+    "    #   nn.HuberLoss()\n",
+    "    #   custom callable loss\n",
+    "\n",
+    "    optimizer=None,      \n",
+    "    # default: AdamW\n",
+    "    # user may pass:\n",
+    "    #   torch.optim.SGD\n",
+    "    #   torch.optim.Adam\n",
+    "    #   any custom optimizer\n",
+    "\n",
+    "    scheduler=None,      \n",
+    "    # default: ReduceLROnPlateau\n",
+    "    # user may pass:\n",
+    "    #   StepLR\n",
+    "    #   OneCycleLR\n",
+    "    #   custom scheduler\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c708a47e-6175-42da-a783-5dccfdfb0b37",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# How training is triggered!\n",
+    "trainer.fit()\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "eeddcadd-3d3b-4dcf-b798-c428b9d30415",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Using the Trained Model for Inference\n",
+    "With our trained PyTorch model, we can now perform inference on the test dataset and compute basic evaluation metrics to assess forecasting performance."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "19d9b8fc-d26d-4290-9573-72b1c0289db4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import seaborn as sns\n",
+    "from sklearn.metrics import confusion_matrix, classification_report\n",
+    "\n",
+    "# ---- Build class names from label_to_id dict ----\n",
+    "class_names = list(label_to_id.keys())     \n",
+    "num_classes = len(class_names)\n",
+    "test_dataloader=DataLoader(test_ds, batch_size=32, collate_fn=collator, num_workers=1)\n",
+    "\n",
+    " # ---- Collect predictions ----\n",
+    "model.eval()\n",
+    "\n",
+    "all_targets = []\n",
+    "all_preds = []\n",
+    "\n",
+    "with torch.no_grad():\n",
+    "    for batch in test_dataloader:\n",
+    "        x, y = batch[0], batch[1]\n",
+    "        logits = model(x)\n",
+    "\n",
+    "        # Convert logits → predicted class\n",
+    "        pred = torch.argmax(logits, dim=1)\n",
+    "        all_targets.append(y.cpu())\n",
+    "        all_preds.append(pred.cpu())\n",
+    "\n",
+    "y_true = torch.cat(all_targets).numpy()\n",
+    "y_pred = torch.cat(all_preds).numpy()\n",
+    "\n",
+    " # ---- Metrics ----\n",
+    "cm = confusion_matrix(y_true, y_pred)\n",
+    "\n",
+    "report = classification_report(\n",
+    "    y_true,\n",
+    "    y_pred,\n",
+    "    target_names=class_names,\n",
+    "    zero_division=0  # avoid warnings if some class missing\n",
+    ")\n",
+    "print(\"\\n=== Classification Report ===\")\n",
+    "print(report)\n",
+    "\n",
+    "# ---- Plot confusion matrix ----\n",
+    "\n",
+    "plt.figure(figsize=(6, 6))\n",
+    "sns.heatmap(\n",
+    "    cm,\n",
+    "    annot=True,\n",
+    "    fmt=\"d\",\n",
+    "    cmap=\"Blues\",\n",
+    "    xticklabels=class_names,\n",
+    "    yticklabels=class_names\n",
+    ")\n",
+    "plt.title(\"Confusion Matrix\")\n",
+    "plt.xlabel(\"Predicted\")\n",
+    "plt.ylabel(\"True\")\n",
+    "plt.tight_layout()\n",
+    "plt.show()\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fa86a188-0924-4766-8840-dd2bfca2dee6",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "charm",
+   "language": "python",
+   "name": "charm"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

notebooks/demo_forecasting.ipynb

@@ -0,0 +1,771 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "62b712f6-5a23-4de9-af1d-9cb94191f111",
+   "metadata": {},
+   "source": [
+    "# Time Series Forecasting with Charm Embeddings\n",
+    "\n",
+    "This notebook walks through a full example of **time-series Forecasting** using:\n",
+    "\n",
+    "- A synthetic multi-channel sinewave dataset  \n",
+    "- Your **CharmClient** for embeddings  \n",
+    "- Your custom **ForecastingModel**  \n",
+    "- A clean dataset pipeline with sliding windows  \n",
+    "- The training loop powered by your custom **Trainer**  \n",
+    "- A visualization callback that plots forecasting during training  \n",
+    "\n",
+    "The goal is pedagogical: understand *every single component* of the library.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "97f74300-819e-4a66-8fe7-048c4fe8976a",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "### Setup: Imports & Charm Client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6c96e1cb-2d56-487c-af33-9fb7eef2d7cf",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from charm import CharmClient\n",
+    "from dotenv import load_dotenv\n",
+    "import os\n",
+    "import numpy as np\n",
+    "import torch\n",
+    "from torch.utils.data import DataLoader\n",
+    "import matplotlib.pyplot as plt\n",
+    "import torch.nn as nn\n",
+    "import numpy as np\n",
+    "\n",
+    "# # Load environment variables from .env file\n",
+    "# load_dotenv()\n",
+    "\n",
+    "# # Get API key and base URL from environment variables\n",
+    "# api_key = os.getenv(\"CHARM_API_KEY\", \"sk-charm-default-key\")\n",
+    "# base_url = os.getenv(\"CHARM_BASE_URL\", \"EMBEDDING_MODEL_ENDPOINT\")\n",
+    "\n",
+    "api_key = \"sk-charm-default-key\"\n",
+    "base_url = \"EMBEDDING_MODEL_ENDPOINT\"\n",
+    "\n",
+    "# Create a client for embedding extraction\n",
+    "client = CharmClient(\n",
+    "    base_url=base_url,\n",
+    "    api_key=api_key,\n",
+    "    timeout=60,\n",
+    "    max_retries=3\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "92b27303-c19d-4d9a-9cbb-180832c97171",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Dataset: Weather\n",
+    "\n",
+    "Here we rely on the `Weather` dataset. This dataset has been seeded in the `data` folder. Let's start by visualizing this data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e08fa2e7-173a-4911-baae-bebd274af93d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import torch\n",
+    "import numpy as np\n",
+    "import matplotlib.pyplot as plt\n",
+    "from itertools import islice\n",
+    "\n",
+    "# Load the dataset\n",
+    "path = \"data/weather.csv\"\n",
+    "df = pd.read_csv(path)\n",
+    "\n",
+    "# Extract data values (exclude time/index column)\n",
+    "data_np = df.iloc[:, 1:].values\n",
+    "\n",
+    "# Convert to PyTorch t\n",
+    "tdata = torch.tensor(data_np)\n",
+    "\n",
+    "# Get channel names\n",
+    "channel_names = list(df.columns[1:])\n",
+    "\n",
+    "print(\"Channel names:\", channel_names)\n",
+    "print(\"Tensor shape:\", tdata.shape) \n",
+    "\n",
+    "# Plotting: separate y-axis for each channel\n",
+    "plt.style.use(\"default\")\n",
+    "num_channels = 5  # Number of channels to plot\n",
+    "colors = plt.cm.tab10(np.linspace(0, 1, num_channels))\n",
+    "\n",
+    "fig, axes = plt.subplots(num_channels, 1, figsize=(15, 3*num_channels), sharex=True)\n",
+    "\n",
+    "for i, (channel, color) in enumerate(islice(zip(channel_names, colors), num_channels)):\n",
+    "    ax = axes[i]\n",
+    "    ax.plot(data_np[:, i], label=channel, color=color, linewidth=1.4, alpha=0.9)\n",
+    "    ax.set_ylabel(channel, fontsize=12)\n",
+    "    ax.grid(True, linestyle=\"--\", alpha=0.3)\n",
+    "    ax.legend(loc=\"upper right\", fontsize=10)\n",
+    "\n",
+    "axes[-1].set_xlabel(\"Time Index\", fontsize=12)\n",
+    "fig.suptitle(\"Weather Data (First 5 Channels)\", fontsize=16)\n",
+    "plt.tight_layout(rect=[0, 0, 1, 0.96])\n",
+    "plt.show()\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c4b88dd3-6281-4988-a657-ef0509640027",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Sliding-Window Forecasting Dataset\n",
+    "\n",
+    "To prepare our dataset for training, validation and testing, we use the function `create_forecasting_datasets`. This function prepares train/validation/test splits for forecasting tasks.\n",
+    "It takes raw time series and produces windowed input–target pairs for supervised learning. The function\n",
+    "\n",
+    "1. **Wraps the raw data**  \n",
+    "   Creates a base dataset where each sample is a full time series instance along with its\n",
+    "   channel descriptions.\n",
+    "\n",
+    "2. **Splits into train/val/test**  \n",
+    "   Uses sequential splitting based on `train_ratio` and `val_ratio`.\n",
+    "\n",
+    "3. **Applies windowing for forecasting**  \n",
+    "   Converts each split into a `ForecastingDataset`, where:\n",
+    "   - the input is a window of fixed length (`input_size`)\n",
+    "   - the target is the next `horizon` time steps.\n",
+    "\n",
+    "4. **Returns datasets ready for DataLoaders**  \n",
+    "   Each output dataset serves as input for training a `ForecastingModel`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cbc47cb0-7e23-4789-ab41-6fccc50e1e05",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from charm_toolkit.Datasets import create_forecasting_datasets, collator\n",
+    "\n",
+    "train_ds, val_ds, test_ds = create_forecasting_datasets(\n",
+    "    raw_data       = tdata,          # Tensor of shape [T, C], e.g. [4096, 3]\n",
+    "    descriptions   = channel_names,  # List of channel names, e.g. [\"ch0\", \"ch1\"]\n",
+    "\n",
+    "    train_horizon  = 96,             # Number of time steps used as model input\n",
+    "    test_horizon   = 96,             # Number of time steps to forecast\n",
+    "\n",
+    "    train_ratio    = 0.70,           # 70% of data for training\n",
+    "    val_ratio      = 0.15,           # 15% of data for validation (rest → test)\n",
+    "\n",
+    "    sequential     = True,           # Use sequential (not shuffled) splits\n",
+    "    scale          = True            # Scale using training statistics\n",
+    "                                    # Access via: train_ds.dataset.scaler\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "793532f6-874d-493c-838e-ffec6f25d6af",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Zero-shot forecasting\n",
+    "\n",
+    "Zero-shot forecasting evaluates the model’s ability to predict future values **without any additional training or fine-tuning on the target series**. The model is given only a historical context window and the natural-language descriptions of each channel, and it must extrapolate the future based on the representations it has already learned.\n",
+    "\n",
+    "In this setting, we select a single example from the test set and provide:\n",
+    "\n",
+    "* the **context window** (`ts_array`), containing the past observations\n",
+    "* the **channel descriptions**, used by Charm to generate semantic embeddings\n",
+    "* a desired **forecast horizon** (`target_len`)\n",
+    "\n",
+    "The prediction is produced by calling the Charm prediction API. The model returns a **probabilistic forecast** in the form of multiple quantiles for each channel and time step. These predictions are automatically **denormalized**, allowing direct comparison with the ground-truth future values."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ff3d6080-49a2-4a2d-a905-c6811fef2a48",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "idx = 10                      # index of the sample to evaluate\n",
+    "target_len = 96               # number of future time steps to predict\n",
+    "\n",
+    "ts_array = test_ds[idx][0]['ts_array'].tolist()   # input time-series\n",
+    "descriptions = test_ds[idx][0]['descriptions']    # descriptions\n",
+    "\n",
+    "response = client.prediction.create(              # run model prediction\n",
+    "    ts_array=ts_array,\n",
+    "    descriptions=descriptions,\n",
+    "    target_len=target_len,\n",
+    "    return_tensors=\"np\"\n",
+    ")\n",
+    "\n",
+    "preds = response.denormalized_predictions.squeeze(0)  # model output (denormalized)\n",
+    "true = test_ds[idx][1].squeeze(0)                     # ground-truth target values\n",
+    "context = test_ds[idx][0]['ts_array'].squeeze(0)      # original input context\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6f1781ae-31b0-475a-a5dd-c04888852b3a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Configuration\n",
+    "n_channels = 3                     # number of time-series channels to plot\n",
+    "T_ctx, C = context.shape     \n",
+    "T, _, Q = preds.shape  \n",
+    "\n",
+    "# Time axes\n",
+    "t_ctx = np.arange(T_ctx)          \n",
+    "t_fut = np.arange(T_ctx, T_ctx + T)  \n",
+    "\n",
+    "# Quantile indices\n",
+    "q_low, q_high = 1, 19               # lower/upper quantiles (e.g. 10th and 90th)\n",
+    "q_med = Q // 2                     # median quantile index\n",
+    "\n",
+    "# Create subplots (one per channel)\n",
+    "fig, axes = plt.subplots(n_channels, 1, figsize=(14, 3 * n_channels), sharex=True)\n",
+    "if n_channels == 1:\n",
+    "    axes = [axes]\n",
+    "\n",
+    "for ch in range(n_channels):\n",
+    "    ax = axes[ch]\n",
+    "\n",
+    "    # 1. Plot context (historical input)\n",
+    "    ax.plot(t_ctx, context[:, ch], color=\"black\", lw=2, label=\"Context\")\n",
+    "\n",
+    "    # 2. Plot quantile uncertainty fan\n",
+    "    for i in range(q_low, q_med):\n",
+    "        # Alpha increases as quantiles approach the median\n",
+    "        alpha_factor = (i - q_low) / (q_med - q_low)\n",
+    "        alpha = 0.05 + 0.20 * alpha_factor\n",
+    "\n",
+    "        # Shade area between symmetric quantiles (e.g., 10th–90th)\n",
+    "        ax.fill_between(\n",
+    "            t_fut,\n",
+    "            preds[:, ch, i],\n",
+    "            preds[:, ch, Q - 1 - i],\n",
+    "            color=\"red\",\n",
+    "            alpha=0.1,              # low alpha; overlap creates gradient effect\n",
+    "            lw=0\n",
+    "        )\n",
+    "\n",
+    "    # 3. Plot median prediction, ground truth, and context divider\n",
+    "    ax.plot(t_fut, preds[:, ch, q_med], color=\"red\", ls=\"--\", lw=2, label=\"Median Pred\")\n",
+    "    ax.plot(t_fut, true[:, ch], color=\"green\", lw=2, label=\"Ground Truth\")\n",
+    "    ax.axvline(T_ctx - 1, color=\"gray\", ls=\":\", alpha=0.5)  # end of context\n",
+    "\n",
+    "    ax.set_title(f\"Channel {ch}\")\n",
+    "    if ch == 0:\n",
+    "        ax.legend(loc=\"upper left\")\n",
+    "\n",
+    "# Final plot formatting\n",
+    "axes[-1].set_xlabel(\"Time\")\n",
+    "plt.tight_layout()\n",
+    "plt.show()\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5f83fd48-3039-451d-8e1b-b81ee7a7f1c0",
+   "metadata": {},
+   "source": [
+    "## Moving on...\n",
+    "While zero-shot forecasting highlights the model’s ability to generalize without adaptation, it is only one possible usage mode. In many practical settings, better performance can be achieved by linear probing, where the Charm embeddings are kept frozen and a lightweight forecasting head is trained on task-specific data. This allows the model to retain its general semantic understanding while efficiently adapting to the statistical structure of a particular dataset."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7e1733b6-f84e-4f82-aba2-0884dd8ecb7e",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Precomputing Embeddings\n",
+    "\n",
+    "It is possible to precompute the embeddings and save them into disk. This will make training faster and allow us to train different models or heads and experiment **without** having to compute the embeddings again! To this end we use the function `precompute_dataset_embeddings` that computes embeddings for an entire dataset **once** and stores them efficiently in a memory-mapped file on disk. The function works in three steps\n",
+    "\n",
+    "1. **Infer embedding shape**  \n",
+    "   It embeds a small initial batch to determine the full output shape and initializes a\n",
+    "   `memmap` file on disk to store all embeddings.\n",
+    "\n",
+    "2. **Process dataset in batches to avoid memory issues**  \n",
+    "   It iterates through the dataset in chunks (`memory_batch_size`), computes embeddings\n",
+    "   asynchronously using the model’s embedding client, and writes each batch directly into\n",
+    "   the correct slice of the memmap file.\n",
+    "\n",
+    "3. **Flush and return shape**  \n",
+    "   Each batch is saved immediately, and the function returns the full embedding shape for\n",
+    "   later use."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f22c2b26-9810-4076-97e2-1e315cb5341b",
+   "metadata": {
+    "jupyter": {
+     "source_hidden": true
+    }
+   },
+   "outputs": [],
+   "source": [
+    "from charm_toolkit.EmbeddingUtils import precompute_dataset_embeddings, PrecomputedEmbeddingsDataset\n",
+    "\n",
+    "# Precompute embeddings\n",
+    "emb_shape_train = precompute_dataset_embeddings(\n",
+    "    client=client,\n",
+    "    dataset=train_ds,\n",
+    "    output_path=\"./outputs/train_embeddings.pt\"\n",
+    ")\n",
+    "emb_shape_val = precompute_dataset_embeddings(\n",
+    "    client=client,\n",
+    "    dataset=val_ds,\n",
+    "    output_path=\"./outputs/val_embeddings.pt\"\n",
+    ")\n",
+    "emb_shape_test = precompute_dataset_embeddings(\n",
+    "    client=client,\n",
+    "    dataset=test_ds,\n",
+    "    output_path=\"./outputs/test_embeddings.pt\"\n",
+    ")\n",
+    "\n",
+    "\n",
+    "# # # Create datasets with precomputed embeddings\n",
+    "train_dataset_with_embeddings = PrecomputedEmbeddingsDataset(\n",
+    "    train_ds,\n",
+    "    \"./outputs/train_embeddings.pt\",\n",
+    "    emb_shape_train\n",
+    ")\n",
+    "\n",
+    "val_dataset_with_embeddings = PrecomputedEmbeddingsDataset(\n",
+    "    val_ds,\n",
+    "    \"./outputs/val_embeddings.pt\",\n",
+    "    emb_shape_val\n",
+    ")\n",
+    "test_dataset_with_embeddings = PrecomputedEmbeddingsDataset(\n",
+    "    test_ds,\n",
+    "    \"./outputs/test_embeddings.pt\",\n",
+    "    emb_shape_test\n",
+    ")\n",
+    "\n",
+    "# Create dataloaders\n",
+    "train_loader = DataLoader(train_dataset_with_embeddings, batch_size=512, shuffle=True, num_workers=8,collate_fn=collator)\n",
+    "val_loader = DataLoader(val_dataset_with_embeddings, batch_size=512, shuffle=False, num_workers=8,collate_fn=collator)\n",
+    "test_loader = DataLoader(test_dataset_with_embeddings, batch_size=512, shuffle=False, num_workers=8,collate_fn=collator)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ce497a8a-00dc-4c32-b6aa-de29fe8bb404",
+   "metadata": {},
+   "source": [
+    "### Skipping Precomputation (Small Datasets Only)\n",
+    "\n",
+    "For small datasets, you may choose **not** to precompute embeddings at all.\n",
+    "Instead, the model can compute embeddings **on-the-fly** during training and evaluation.\n",
+    "This avoids writing anything to disk and keeps the workflow simpler — but it also makes each training epoch slower because the model recomputes embeddings every time.\n",
+    "\n",
+    "This approach is **only recommended for small tasks** or quick experimentation.\n",
+    "For medium or large datasets, precomputing is strongly preferred to avoid unnecessary compute overhead.\n",
+    "\n",
+    "**To skip precomputation entirely**, simply create your dataloaders from the original datasets:\n",
+    "\n",
+    "```python\n",
+    "# Create dataloaders without precomputing embeddings\n",
+    "train_loader = DataLoader(train_ds, batch_size=512, shuffle=True,\n",
+    "                          num_workers=8, collate_fn=collator)\n",
+    "\n",
+    "val_loader = DataLoader(val_ds, batch_size=512, shuffle=False,\n",
+    "                        num_workers=8, collate_fn=collator)\n",
+    "\n",
+    "test_loader = DataLoader(test_ds, batch_size=512, shuffle=False,\n",
+    "                         num_workers=8, collate_fn=collator)\n",
+    "```\n",
+    "\n",
+    "With this setup, embeddings are computed dynamically each time a batch is passed through the model."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e48673a2-3f4d-4336-b562-9e1f433378ee",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Instantiating the Model\n",
+    "\n",
+    "We use the `ForecastingModel` class to create a forecasting model that predicts future time steps of a time series using embeddings from CHARM. Given an input window of length `input_size`,\n",
+    "the model:\n",
+    "\n",
+    "1. **Computes embeddings**  \n",
+    "   Calls the embedding client to obtain embeddings (or uses pre-computed ones) of shape **[B, T_p, C, D]**, where `T_p`\n",
+    "   is the number of temporal patches.\n",
+    "\n",
+    "2. **Pools embeddings over time** (`mode`):\n",
+    "   - **\"last\"** – use the final temporal patch  \n",
+    "   - **\"avg\"** – average across all patches  \n",
+    "   - **\"none\"** – flatten all patches (most expressive)\n",
+    "\n",
+    "3. **Feeds pooled embeddings into forecasting heads**  \n",
+    "   Depending on `per_channel`:\n",
+    "   - A **separate head per channel**, or  \n",
+    "   - A **shared head** across channels.  \n",
+    "   Heads can be `\"linear\"`, `\"mlp\"`, or a custom module.\n",
+    "\n",
+    "4. **Outputs forecasts**  \n",
+    "   Produces predictions of shape **[B, horizon, C]**, then rescales using the stored\n",
+    "   input mean and std."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fa09e6b6-75b2-467a-9746-6d2c13df8585",
+   "metadata": {
+    "jupyter": {
+     "source_hidden": true
+    }
+   },
+   "outputs": [],
+   "source": [
+    "from charm_toolkit.models import ForecastingModel\n",
+    "\n",
+    "model = ForecastingModel(\n",
+    "    embedding_client=client,     \n",
+    "\n",
+    "    head=\"linear\",      \n",
+    "    # Supported options:\n",
+    "    #   \"linear\" → simple Linear decoder\n",
+    "    #   \"mlp\"    → 2-layer MLP decoder\n",
+    "    #   nn.Module instance → fully custom decoder\n",
+    "\n",
+    "    input_size=96,\n",
+    "    # Number of time steps to use as input. Should be the same one as used in \n",
+    "    # 'create_forecasting_datasets' for train_horizon\n",
+    "    \n",
+    "    hidden_dim=128,              \n",
+    "    # Used only for MLP-style forecaster.\n",
+    "    # Controls width of intermediate layers.\n",
+    "\n",
+    "    dropout=0.0,                 \n",
+    "    # Dropout applied\n",
+    "\n",
+    "    horizon= 96,\n",
+    "    # Horizon to forecast.  Should be the same one as used in \n",
+    "    # 'create_forecasting_datasets' for test_horizon\n",
+    "\n",
+    "    mode= \"last\",\n",
+    "    # Pooling mode for embeddings (\"last\", \"avg\", \"none\")\n",
+    "\n",
+    "    per_channel = True,\n",
+    "    # Whether to use separate forecasting heads for each channel.\n",
+    "\n",
+    "    num_channels= 21,\n",
+    "    # Number of channels in the time series (required if per_channel=True)\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3cc41797-cca5-4b35-97c4-aacfe6de2fd9",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Setting Up the Trainer\n",
+    "\n",
+    "Having defined the model architecture, we can now train it. The `Trainer` class provides a lightweight, flexible training loop for PyTorch models,\n",
+    "including support for validation, early stopping, gradient accumulation, gradient clipping,\n",
+    "learning-rate scheduling, callbacks, and basic visualization snapshots.\n",
+    "\n",
+    "A `Trainer` instance is created with:\n",
+    "- a model (`nn.Module`)\n",
+    "- `train_loader`, and optionally `val_loader` / `test_loader`\n",
+    "- training hyperparameters such as learning rate, weight decay, epochs,\n",
+    "  gradient accumulation steps, and early-stopping parameters.\n",
+    "\n",
+    "During training (`fit()`):\n",
+    "1. **Each epoch streams batches** from the `DataLoader` (no full-dataset loading).\n",
+    "2. For each batch, it:\n",
+    "   - runs a forward pass\n",
+    "   - computes the loss\n",
+    "   - performs optional gradient accumulation (`accum_steps`)\n",
+    "   - applies gradient clipping (`max_grad_norm`)\n",
+    "   - updates the optimizer\n",
+    "3. After each epoch, it:\n",
+    "   - evaluates on the validation set (if provided)\n",
+    "   - steps the scheduler (e.g., `ReduceLROnPlateau`)\n",
+    "   - triggers any user-defined callbacks\n",
+    "   - checks early-stopping conditions (`patience`, `min_delta`)\n",
+    "4. The best model (lowest validation loss) is stored and restored at the end.\n",
+    "\n",
+    "During validation, the trainer also stores a small number of snapshots (targets, predictions, inputs) for visualization. The `evaluate()` method runs a full validation/testing pass and returns the average loss."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5721f474-5b7f-49cb-9df9-c97bf75f5645",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from charm_toolkit.Trainer import Trainer\n",
+    "from charm_toolkit.Plots import PlotCallback\n",
+    "\n",
+    "plot_cb = PlotCallback(\n",
+    "    every=3,           \n",
+    "    # Plot every N steps\n",
+    "    #TODO\n",
+    "    show_forecast=True,    \n",
+    "    # Plot original vs forecasted\n",
+    "\n",
+    "    n_samples=2,        \n",
+    "    # number of samples to plot\n",
+    "\n",
+    "    channels=[1,2,3],         \n",
+    "    # int or list of channels to visualize\n",
+    ")\n",
+    "\n",
+    "\n",
+    "trainer = Trainer(\n",
+    "    model=model,\n",
+    "    train_loader=train_loader,\n",
+    "    val_loader=val_loader,\n",
+    "    test_loader=None,  \n",
+    "    # optional — only needed if you call trainer.test()\n",
+    "\n",
+    "    lr=1e-2,             \n",
+    "    # learning rate. Default:1e-3\n",
+    "\n",
+    "    weight_decay=1e-4,   \n",
+    "    # L2 regularization. Default:1e-4\n",
+    "\n",
+    "    epochs=1000,         \n",
+    "    # maximum training epochs.  Default:1000\n",
+    "\n",
+    "    accum_steps=1,       \n",
+    "    # gradient accumulation:\n",
+    "    #   if >1: effectively increases batch size without increasing memory\n",
+    "    # Default: 1\n",
+    "\n",
+    "    patience=10,          \n",
+    "    # early stopping patience. Default: 5\n",
+    "\n",
+    "    min_delta=1e-4,      \n",
+    "    # minimum improvement required for early stopping. Default: 1e-4\n",
+    "\n",
+    "    max_grad_norm=5.0,   \n",
+    "    # gradient clipping for stability.  Default: 5\n",
+    "\n",
+    "    device=None,         \n",
+    "    # default: \"cuda\" if available else \"cpu\"\n",
+    "\n",
+    "    save_dir=\"./forecasting_checkpoints\",\n",
+    "    # where best checkpoints are saved\n",
+    "\n",
+    "    callbacks=[plot_cb], \n",
+    "    # supports ANY number of callbacks:\n",
+    "    #   - plotting\n",
+    "    #   - logging (wandb, tensorboard)\n",
+    "    #   - checkpointing\n",
+    "    #   - custom user hooks\n",
+    "\n",
+    "    criterion=None,      \n",
+    "    # default: MSELoss\n",
+    "    # user may pass:\n",
+    "    #   nn.L1Loss()\n",
+    "    #   nn.HuberLoss()\n",
+    "    #   custom callable loss\n",
+    "\n",
+    "    optimizer=None,      \n",
+    "    # default: AdamW\n",
+    "    # user may pass:\n",
+    "    #   torch.optim.SGD\n",
+    "    #   torch.optim.Adam\n",
+    "    #   any custom optimizer\n",
+    "\n",
+    "    scheduler=None,      \n",
+    "    # default: ReduceLROnPlateau\n",
+    "    # user may pass:\n",
+    "    #   StepLR\n",
+    "    #   OneCycleLR\n",
+    "    #   custom scheduler\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c708a47e-6175-42da-a783-5dccfdfb0b37",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# How training is triggered!\n",
+    "trainer.fit()\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7d10f91e-7bf9-4950-b64a-ad2fea60f2a9",
+   "metadata": {},
+   "source": [
+    "## Using our Model for Inference\n",
+    "By default, our models use the training statistics to standardize the data for stable training dynamics.\n",
+    "Each dataset contains a scaler, which we can use to transform predictions back into the original data space."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b17609b9-91d5-4863-a70c-f383ab355380",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import random\n",
+    "import matplotlib.pyplot as plt\n",
+    "\n",
+    "plt.style.use(\"seaborn-v0_8-darkgrid\")\n",
+    "\n",
+    "n = 3\n",
+    "channels_to_plot = [1, 2, 3]\n",
+    "\n",
+    "# Random indices\n",
+    "random.seed(42)\n",
+    "indices = random.sample(range(len(test_ds)), n)\n",
+    "\n",
+    "model.eval()\n",
+    "for idx in indices:\n",
+    "\n",
+    "    # ---- load sample ----\n",
+    "    x, y = test_ds[idx]\n",
+    "    context = x[\"ts_array\"]\n",
+    "\n",
+    "    pred_norm = model(x)\n",
+    "\n",
+    "    # ---- numpy ----\n",
+    "    y_np    = y.squeeze(0).cpu().numpy()\n",
+    "    pred_np = pred_norm.squeeze(0).cpu().detach().numpy()\n",
+    "    ctx_np  = context.squeeze(0).cpu().numpy()\n",
+    "\n",
+    "    # ---- unnormalize ----\n",
+    "    scaler = getattr(test_ds.dataset, \"scaler\", None)\n",
+    "    if scaler is not None:\n",
+    "        ctx_un  = scaler.inverse_transform(ctx_np)\n",
+    "        y_un    = scaler.inverse_transform(y_np)\n",
+    "        pred_un = scaler.inverse_transform(pred_np)\n",
+    "    else:\n",
+    "        ctx_un, y_un, pred_un = ctx_np, y_np, pred_np\n",
+    "\n",
+    "    # ---- time axis ----\n",
+    "    T_ctx = ctx_un.shape[0]\n",
+    "    T_fut = y_un.shape[0]\n",
+    "    t_ctx = range(-T_ctx, 0)\n",
+    "    t_fut = range(0, T_fut)\n",
+    "\n",
+    "    # ---- create subplots ----\n",
+    "    fig, axes = plt.subplots(\n",
+    "        len(channels_to_plot),\n",
+    "        1,\n",
+    "        figsize=(11, 4 * len(channels_to_plot)),\n",
+    "        sharex=True\n",
+    "    )\n",
+    "\n",
+    "    # If only one axis, wrap it into a list\n",
+    "    if len(channels_to_plot) == 1:\n",
+    "        axes = [axes]\n",
+    "\n",
+    "    colors = [\"#1f77b4\", \"#d62728\", \"#2ca02c\", \"#9467bd\"]\n",
+    "\n",
+    "    # ---- plot each channel separately ----\n",
+    "    for i, (ax, ch) in enumerate(zip(axes, channels_to_plot)):\n",
+    "\n",
+    "        col = colors[i % len(colors)]\n",
+    "\n",
+    "        # context\n",
+    "        ax.plot(t_ctx, ctx_un[:, ch], color=col, alpha=0.6,\n",
+    "                label=f\"Context (ch{ch})\", linewidth=1.8)\n",
+    "\n",
+    "        # target\n",
+    "        ax.plot(t_fut, y_un[:, ch], color=col,\n",
+    "                label=f\"Target (ch{ch})\", linewidth=2.2)\n",
+    "\n",
+    "        # forecast\n",
+    "        ax.plot(t_fut, pred_un[:, ch], linestyle=\"--\", color=col,\n",
+    "                label=f\"Forecast (ch{ch})\", linewidth=2)\n",
+    "\n",
+    "        # ---- aesthetics (preserving your original style) ----\n",
+    "        ax.axvline(0, color=\"black\", linestyle=\":\", linewidth=1.5)\n",
+    "\n",
+    "        ymax = ax.get_ylim()[1]\n",
+    "        ax.text(-0.5, ymax * 0.95, \"Context\",\n",
+    "                ha=\"right\", va=\"top\", fontsize=10)\n",
+    "        ax.text(0.5, ymax * 0.95, \"Forecast\",\n",
+    "                ha=\"left\", va=\"top\", fontsize=10)\n",
+    "\n",
+    "        ax.set_ylabel(\"Value\", fontsize=12)\n",
+    "        ax.set_title(f\"Channel {ch}\", fontsize=14)\n",
+    "\n",
+    "        ax.legend(loc=\"upper left\", ncol=2, fontsize=10, frameon=True)\n",
+    "\n",
+    "    # ---- global labels ----\n",
+    "    fig.suptitle(f\"Forecasting Example {idx}\", fontsize=16, y=1.02)\n",
+    "    fig.supxlabel(\"Time\", fontsize=12)\n",
+    "\n",
+    "    plt.tight_layout()\n",
+    "    fig.subplots_adjust(hspace=0.15)\n",
+    "    plt.show()\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b1c24063-ab63-459a-bc34-68b0f0569f11",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "charm",
+   "language": "python",
+   "name": "ftse"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.4"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

notebooks/demo_reconstruction.ipynb

@@ -0,0 +1,710 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "62b712f6-5a23-4de9-af1d-9cb94191f111",
+   "metadata": {},
+   "source": [
+    "# Time Series Reconstruction with Charm Embeddings\n",
+    "\n",
+    "This notebook walks through a full example of **time-series reconstruction** using:\n",
+    "\n",
+    "- A synthetic multi-channel sinewave dataset  \n",
+    "- Your **CharmClient** for embeddings  \n",
+    "- Your custom **ReconstructionModel**  \n",
+    "- A clean dataset pipeline with sliding windows  \n",
+    "- The training loop powered by your custom **Trainer**  \n",
+    "- A visualization callback that plots reconstructions during training  \n",
+    "\n",
+    "The goal is pedagogical: understand *every single component* of the library.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "97f74300-819e-4a66-8fe7-048c4fe8976a",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Setup: Imports & Charm Client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6c96e1cb-2d56-487c-af33-9fb7eef2d7cf",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from charm import CharmClient\n",
+    "from dotenv import load_dotenv\n",
+    "import os\n",
+    "import numpy as np\n",
+    "import torch\n",
+    "from torch.utils.data import DataLoader\n",
+    "import matplotlib.pyplot as plt\n",
+    "import torch.nn as nn\n",
+    "\n",
+    "# Create a client\n",
+    "# Initialize the CHARM client using environment variables\n",
+    "\n",
+    "\n",
+    "# # Load environment variables from .env file\n",
+    "# load_dotenv()\n",
+    "\n",
+    "# # Get API key and base URL from environment variables\n",
+    "# api_key = os.getenv(\"CHARM_API_KEY\", \"sk-charm-default-key\")\n",
+    "# base_url = os.getenv(\"CHARM_BASE_URL\", \"EMBEDDING_MODEL_ENDPOINT\")\n",
+    "\n",
+    "api_key = \"sk-charm-default-key\"\n",
+    "base_url = \"EMBEDDING_MODEL_ENDPOINT\"\n",
+    "\n",
+    "client = CharmClient(\n",
+    "    base_url=base_url,\n",
+    "    api_key=api_key,\n",
+    "    timeout=180,\n",
+    "    max_retries=3\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "92b27303-c19d-4d9a-9cbb-180832c97171",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Dataset: Weather\n",
+    "\n",
+    "Here we rely on the `Weather` dataset. This dataset has been seeded in the `data` folder. Let's start by visualizing this data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "da2ef8b4-d107-4875-970a-51525b9713ea",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import torch\n",
+    "import numpy as np\n",
+    "import matplotlib.pyplot as plt\n",
+    "from itertools import islice\n",
+    "\n",
+    "# Load the dataset\n",
+    "path = \"data/weather.csv\"\n",
+    "df = pd.read_csv(path)\n",
+    "\n",
+    "# Extract data values (exclude time/index column)\n",
+    "data_np = df.iloc[:, 1:].values\n",
+    "\n",
+    "# Convert to PyTorch t\n",
+    "tdata = torch.tensor(data_np)\n",
+    "\n",
+    "# Get channel names\n",
+    "channel_names = list(df.columns[1:])\n",
+    "\n",
+    "print(\"Channel names:\", channel_names)\n",
+    "print(\"Tensor shape:\", tdata.shape) \n",
+    "\n",
+    "# Plotting: separate y-axis for each channel\n",
+    "plt.style.use(\"default\")\n",
+    "num_channels = 5  # Number of channels to plot\n",
+    "colors = plt.cm.tab10(np.linspace(0, 1, num_channels))\n",
+    "\n",
+    "fig, axes = plt.subplots(num_channels, 1, figsize=(15, 3*num_channels), sharex=True)\n",
+    "\n",
+    "for i, (channel, color) in enumerate(islice(zip(channel_names, colors), num_channels)):\n",
+    "    ax = axes[i]\n",
+    "    ax.plot(data_np[:, i], label=channel, color=color, linewidth=1.4, alpha=0.9)\n",
+    "    ax.set_ylabel(channel, fontsize=12)\n",
+    "    ax.grid(True, linestyle=\"--\", alpha=0.3)\n",
+    "    ax.legend(loc=\"upper right\", fontsize=10)\n",
+    "\n",
+    "axes[-1].set_xlabel(\"Time Index\", fontsize=12)\n",
+    "fig.suptitle(\"Weather Data (First 5 Channels)\", fontsize=16)\n",
+    "plt.tight_layout(rect=[0, 0, 1, 0.96])\n",
+    "plt.show()\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c4b88dd3-6281-4988-a657-ef0509640027",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Sliding-Window Reconstruction Dataset\n",
+    "\n",
+    "To prepare our dataset for training, validation and testing, we use the function`create_reconstruction_datasets`. This function prepares train/validation/test splits specifically for\n",
+    "time-series reconstruction tasks. The function\n",
+    "\n",
+    "1. **Wraps the raw time series**  \n",
+    "   Creates an `UnwindowedDataset` that stores the full sequence and its channel\n",
+    "   descriptions.\n",
+    "\n",
+    "2. **Splits the dataset**  \n",
+    "   Divides the data into train/val/test sets using sequential splitting\n",
+    "   (`train_ratio`, `val_ratio`).\n",
+    "\n",
+    "3. **Applies normalization**  \n",
+    "   Fits a scaler on the training set only and applies the same scaling to validation and\n",
+    "   test sets (avoids data leakage).\n",
+    "\n",
+    "4. **Creates windowed reconstruction datasets**  \n",
+    "   Converts each split into a `ReconstructionDataset`, where each sample is a window of\n",
+    "   length `window_size` extracted with a given `stride`.\n",
+    "\n",
+    "5. **Returns three dataset splits**  \n",
+    "   Ready for training a `ReconstructionModel`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cbc47cb0-7e23-4789-ab41-6fccc50e1e05",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from charm_toolkit.Datasets import create_reconstruction_datasets, collator\n",
+    "\n",
+    "train_ds, val_ds, test_ds = create_reconstruction_datasets(\n",
+    "    raw_data     = tdata,           # Tensor [T, C], e.g. [4096, 3]\n",
+    "    descriptions = channel_names,   # List of channel names, e.g. [\"ch0\", \"ch1\"]\n",
+    "\n",
+    "    window_size  = 256,             # Size of each input window\n",
+    "    stride       = 1,               # Step between windows (overlap control)\n",
+    "\n",
+    "    train_ratio  = 0.70,            # 70% for training\n",
+    "    val_ratio    = 0.15,            # 15% for validation\n",
+    "\n",
+    "    sequential   = True,            # True = sequential splits; False = shuffled\n",
+    "\n",
+    "    scale        = True             # Scale using training statistics\n",
+    "                                    # Access via: train_ds.dataset.scaler\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e89a2475-0157-479e-be25-5c0e6df21aa7",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Zero-shot reconstruction\n",
+    "\n",
+    "Zero-shot reconstruction evaluates the model’s ability to **reconstruct a time series without any additional training or fine-tuning** on the target data. The model is given an input sequence together with the natural-language descriptions of each channel, and it must reproduce the signal using the representations it has already learned.\n",
+    "\n",
+    "In this setting, we select a single example from the test set and provide:\n",
+    "\n",
+    "* the **input time series** (`ts_array`)\n",
+    "* the **channel descriptions**, used by Charm to generate semantic embeddings\n",
+    "\n",
+    "The reconstruction is produced by calling the Charm prediction API. The model returns a **probabilistic reconstruction** in the form of multiple quantiles for each channel and time step. These predictions are automatically **denormalized**, allowing direct comparison with the original signal.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "38e3d04d-08d8-4b6e-94eb-27a78b2cc4ca",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "idx = 10\n",
+    "recon_len = -256   # negative => reconstruction instead of forecasting\n",
+    "\n",
+    "ts_array = test_ds[idx][0][\"ts_array\"].tolist()\n",
+    "descriptions = test_ds[idx][0][\"descriptions\"]\n",
+    "\n",
+    "response = client.prediction.create(\n",
+    "    ts_array=ts_array,\n",
+    "    descriptions=descriptions,\n",
+    "    target_len=recon_len,\n",
+    "    return_tensors=\"np\"\n",
+    ")\n",
+    "\n",
+    "# Model output: (T_rec, C, Q)\n",
+    "preds = response.denormalized_predictions.squeeze(0)\n",
+    "\n",
+    "# Full input context\n",
+    "context = test_ds[idx][0][\"ts_array\"].squeeze(0)\n",
+    "\n",
+    "# Ground truth = reconstructed portion of the input\n",
+    "true = context"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4b2e884b-71d3-4c05-8870-b8072726013b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Configuration\n",
+    "n_channels = 3\n",
+    "T_rec, C, Q = preds.shape\n",
+    "\n",
+    "# Time axis (shared)\n",
+    "t = np.arange(T_rec)\n",
+    "\n",
+    "# Quantile indices\n",
+    "q_low, q_high = 0, 20 # 1st, 99th quantiles\n",
+    "q_med = Q // 2\n",
+    "\n",
+    "# Plot\n",
+    "fig, axes = plt.subplots(n_channels, 1, figsize=(14, 3 * n_channels), sharex=True)\n",
+    "if n_channels == 1:\n",
+    "    axes = [axes]\n",
+    "\n",
+    "for ch in range(n_channels):\n",
+    "    ax = axes[ch]\n",
+    "\n",
+    "    # Ground truth signal\n",
+    "    ax.plot(t, true[:, ch], color=\"black\", lw=2, label=\"True Signal\")\n",
+    "\n",
+    "    # Quantile uncertainty fan\n",
+    "    for i in range(q_low, q_med):\n",
+    "        ax.fill_between(\n",
+    "            t,\n",
+    "            preds[:, ch, i],\n",
+    "            preds[:, ch, Q - 1 - i],\n",
+    "            color=\"red\",\n",
+    "            alpha=0.1,\n",
+    "            lw=0\n",
+    "        )\n",
+    "\n",
+    "    # Median reconstruction\n",
+    "    ax.plot(\n",
+    "        t,\n",
+    "        preds[:, ch, q_med],\n",
+    "        color=\"red\",\n",
+    "        ls=\"--\",\n",
+    "        lw=2,\n",
+    "        label=\"Median Reconstruction\"\n",
+    "    )\n",
+    "\n",
+    "    ax.set_title(f\"Channel {ch}\")\n",
+    "    if ch == 0:\n",
+    "        ax.legend(loc=\"upper left\")\n",
+    "\n",
+    "axes[-1].set_xlabel(\"Time\")\n",
+    "plt.tight_layout()\n",
+    "plt.show()\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b2de66bb-2abd-475f-a0b4-5066fd40cbb3",
+   "metadata": {},
+   "source": [
+    "## Moving on...\n",
+    "While zero-shot forecasting highlights the model’s ability to generalize without adaptation, it is only one possible usage mode. In many practical settings, better performance can be achieved by linear probing, where the Charm embeddings are kept frozen and a lightweight forecasting head is trained on task-specific data. This allows the model to retain its general semantic understanding while efficiently adapting to the statistical structure of a particular dataset."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "31a98fe0-a323-415f-9a53-419d844ee6de",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Precomputing Embeddings\n",
+    "\n",
+    "It is possible to precompute the embeddings and save them into disk. This will make training faster and allow us to train different models or heads and experiment **without** having to compute the embeddings again! To this end we use the function `precompute_dataset_embeddings` that computes embeddings for an entire dataset **once** and stores them efficiently in a memory-mapped file on disk. The function works in three steps\n",
+    "\n",
+    "1. **Infer embedding shape**  \n",
+    "   It embeds a small initial batch to determine the full output shape and initializes a\n",
+    "   `memmap` file on disk to store all embeddings.\n",
+    "\n",
+    "2. **Process dataset in batches to avoid memory issues**  \n",
+    "   It iterates through the dataset in chunks (`memory_batch_size`), computes embeddings\n",
+    "   asynchronously using the model’s embedding client, and writes each batch directly into\n",
+    "   the correct slice of the memmap file.\n",
+    "\n",
+    "3. **Flush and return shape**  \n",
+    "   Each batch is saved immediately, and the function returns the full embedding shape for\n",
+    "   later use."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "32f05b73-b663-4c37-9e9f-aa0fd3b496c0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from charm_toolkit.EmbeddingUtils import precompute_dataset_embeddings, PrecomputedEmbeddingsDataset\n",
+    "\n",
+    "# Precompute embeddings\n",
+    "emb_shape_train = precompute_dataset_embeddings(\n",
+    "    client=client,\n",
+    "    dataset=train_ds,\n",
+    "    output_path=\"./outputs/train_embeddings.pt\",\n",
+    "    memory_batch_size=8192\n",
+    ")\n",
+    "emb_shape_val = precompute_dataset_embeddings(\n",
+    "    client=client,\n",
+    "    dataset=val_ds,\n",
+    "    output_path=\"./outputs/val_embeddings.pt\",\n",
+    "    memory_batch_size=8192\n",
+    ")\n",
+    "emb_shape_test = precompute_dataset_embeddings(\n",
+    "    client=client,\n",
+    "    dataset=test_ds,\n",
+    "    output_path=\"./outputs/test_embeddings.pt\",\n",
+    "    memory_batch_size=8192\n",
+    ")\n",
+    "\n",
+    "\n",
+    "# # # Create datasets with precomputed embeddings\n",
+    "train_dataset_with_embeddings = PrecomputedEmbeddingsDataset(\n",
+    "    train_ds,\n",
+    "    \"./outputs/train_embeddings.pt\",\n",
+    "    emb_shape_train,\n",
+    ")\n",
+    "\n",
+    "val_dataset_with_embeddings = PrecomputedEmbeddingsDataset(\n",
+    "    val_ds,\n",
+    "    \"./outputs/val_embeddings.pt\",\n",
+    "    emb_shape_val,\n",
+    ")\n",
+    "test_dataset_with_embeddings = PrecomputedEmbeddingsDataset(\n",
+    "    test_ds,\n",
+    "    \"./outputs/test_embeddings.pt\",\n",
+    "    emb_shape_test,\n",
+    ")\n",
+    "\n",
+    "# Create dataloaders\n",
+    "train_loader = DataLoader(train_dataset_with_embeddings, batch_size=512, shuffle=True, num_workers=1, collate_fn=collator)\n",
+    "val_loader = DataLoader(val_dataset_with_embeddings, batch_size=512, shuffle=False, num_workers=1, collate_fn=collator)\n",
+    "test_loader = DataLoader(test_dataset_with_embeddings, batch_size=512, shuffle=False, num_workers=1, collate_fn=collator)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "94195f07-aafb-4b07-9dcf-d5eaae6b1044",
+   "metadata": {},
+   "source": [
+    "### Skipping Precomputation (Small Datasets Only)\n",
+    "\n",
+    "For small datasets, you may choose **not** to precompute embeddings at all.\n",
+    "Instead, the model can compute embeddings **on-the-fly** during training and evaluation.\n",
+    "This avoids writing anything to disk and keeps the workflow simpler — but it also makes each training epoch slower because the model recomputes embeddings every time.\n",
+    "\n",
+    "This approach is **only recommended for small tasks** or quick experimentation.\n",
+    "For medium or large datasets, precomputing is strongly preferred to avoid unnecessary compute overhead.\n",
+    "\n",
+    "**To skip precomputation entirely**, simply create your dataloaders from the original datasets:\n",
+    "\n",
+    "```python\n",
+    "# Create dataloaders without precomputing embeddings\n",
+    "train_loader = DataLoader(train_ds, batch_size=512, shuffle=True,\n",
+    "                          num_workers=1, collate_fn=collator)\n",
+    "\n",
+    "val_loader = DataLoader(val_ds, batch_size=512, shuffle=False,\n",
+    "                        num_workers=1, collate_fn=collator)\n",
+    "\n",
+    "test_loader = DataLoader(test_ds, batch_size=512, shuffle=False,\n",
+    "                         num_workers=1, collate_fn=collator)\n",
+    "```\n",
+    "\n",
+    "With this setup, embeddings are computed dynamically each time a batch is passed through the model."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e48673a2-3f4d-4336-b562-9e1f433378ee",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Instantiating the Model\n",
+    "\n",
+    "We use the `ReconstructionModel` class to define a reconstruction model that reconstructs the original time series from its embeddings, using an\n",
+    "external embedding client (e.g., CHARM) as the encoder and a small neural network\n",
+    "(reconstructor) as the decoder.\n",
+    "\n",
+    "The model:\n",
+    "\n",
+    "1. **Computes embeddings**  \n",
+    "   Converts the input time series into patch embeddings of shape **[B, T_p, C, D]**.\n",
+    "\n",
+    "2. **Decodes each embedding**  \n",
+    "   Applies a reconstruction head (either `\"linear\"`, `\"mlp\"`, or a custom module) that\n",
+    "   maps each embedding vector of size **D** back to a patch of length `patch_size`.\n",
+    "\n",
+    "3. **Reassembles patches**  \n",
+    "   Rearranges and reshapes the decoded patches back into a full sequence of shape  \n",
+    "   **[B, T, C]**.\n",
+    "\n",
+    "4. **Denormalizes the output**  \n",
+    "   Uses the input mean and standard deviation from the forward pass to return the\n",
+    "   reconstructed time series in the original scale."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "fa09e6b6-75b2-467a-9746-6d2c13df8585",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from charm_toolkit.models import ReconstructionModel\n",
+    "\n",
+    "model = ReconstructionModel(\n",
+    "    embedding_client=client,     \n",
+    "\n",
+    "    reconstructor=\"linear\",      \n",
+    "    # Supported options:\n",
+    "    #   \"linear\" → simple Linear decoder\n",
+    "    #   \"mlp\"    → 2-layer MLP decoder\n",
+    "    #   nn.Module instance → fully custom decoder\n",
+    "    \n",
+    "    hidden_dim=128,              \n",
+    "    # Used only for MLP-style reconstructors.\n",
+    "    # Controls width of intermediate layers.\n",
+    "\n",
+    "    dropout=0.1,                 \n",
+    "    # Dropout applied\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3cc41797-cca5-4b35-97c4-aacfe6de2fd9",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Setting Up the Trainer\n",
+    "\n",
+    "Having defined the model architecture, we can now train it. The `Trainer` class provides a lightweight, flexible training loop for PyTorch models,\n",
+    "including support for validation, early stopping, gradient accumulation, gradient clipping,\n",
+    "learning-rate scheduling, callbacks, and basic visualization snapshots.\n",
+    "\n",
+    "A `Trainer` instance is created with:\n",
+    "- a model (`nn.Module`)\n",
+    "- `train_loader`, and optionally `val_loader` / `test_loader`\n",
+    "- training hyperparameters such as learning rate, weight decay, epochs,\n",
+    "  gradient accumulation steps, and early-stopping parameters.\n",
+    "\n",
+    "During training (`fit()`):\n",
+    "1. **Each epoch streams batches** from the `DataLoader` (no full-dataset loading).\n",
+    "2. For each batch, it:\n",
+    "   - runs a forward pass\n",
+    "   - computes the loss\n",
+    "   - performs optional gradient accumulation (`accum_steps`)\n",
+    "   - applies gradient clipping (`max_grad_norm`)\n",
+    "   - updates the optimizer\n",
+    "3. After each epoch, it:\n",
+    "   - evaluates on the validation set (if provided)\n",
+    "   - steps the scheduler (e.g., `ReduceLROnPlateau`)\n",
+    "   - triggers any user-defined callbacks\n",
+    "   - checks early-stopping conditions (`patience`, `min_delta`)\n",
+    "4. The best model (lowest validation loss) is stored and restored at the end.\n",
+    "\n",
+    "During validation, the trainer also stores a small number of snapshots (targets, predictions, inputs) for visualization. The `evaluate()` method runs a full validation/testing pass and returns the average loss."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5721f474-5b7f-49cb-9df9-c97bf75f5645",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from charm_toolkit.Trainer import Trainer\n",
+    "from charm_toolkit.Plots import PlotCallback\n",
+    "\n",
+    "plot_cb = PlotCallback(\n",
+    "    every=3,           \n",
+    "    # Plot every N steps\n",
+    "\n",
+    "    show_recon=True,    \n",
+    "    # Plot original vs reconstructed\n",
+    "\n",
+    "    n_samples=3,        \n",
+    "    # number of samples to plot\n",
+    "\n",
+    "    channels=[0,1],         \n",
+    "    # int or list of channels to visualize\n",
+    ")\n",
+    "\n",
+    "\n",
+    "trainer = Trainer(\n",
+    "    model=model,\n",
+    "    train_loader=train_loader,\n",
+    "    val_loader=val_loader,\n",
+    "    test_loader=None,  \n",
+    "    # optional — only needed if you call trainer.test()\n",
+    "\n",
+    "    lr=1e-3,             \n",
+    "    # learning rate. Default:1e-3\n",
+    "\n",
+    "    weight_decay=1e-3,   \n",
+    "    # L2 regularization. Default:1e-4\n",
+    "\n",
+    "    epochs=1000,         \n",
+    "    # maximum training epochs.  Default:1000\n",
+    "\n",
+    "    accum_steps=1,       \n",
+    "    # gradient accumulation:\n",
+    "    #   if >1: effectively increases batch size without increasing memory\n",
+    "    # Default: 1\n",
+    "\n",
+    "    patience=5,          \n",
+    "    # early stopping patience. Default: 5\n",
+    "\n",
+    "    min_delta=1e-4,      \n",
+    "    # minimum improvement required for early stopping. Default: 1e-4\n",
+    "\n",
+    "    max_grad_norm=5.0,   \n",
+    "    # gradient clipping for stability.  Default: 5\n",
+    "\n",
+    "    device=None,         \n",
+    "    # default: \"cuda\" if available else \"cpu\"\n",
+    "\n",
+    "    save_dir=\"./reconstruction_checkpoints\",\n",
+    "    # where best checkpoints are saved\n",
+    "\n",
+    "    callbacks=[plot_cb], \n",
+    "    # supports ANY number of callbacks:\n",
+    "    #   - plotting\n",
+    "    #   - logging (wandb, tensorboard)\n",
+    "    #   - checkpointing\n",
+    "    #   - custom user hooks\n",
+    "\n",
+    "    criterion=nn.HuberLoss(),      \n",
+    "    # default: MSELoss\n",
+    "    # user may pass:\n",
+    "    #   nn.L1Loss()\n",
+    "    #   nn.HuberLoss()\n",
+    "    #   custom callable loss\n",
+    "\n",
+    "    optimizer=None,      \n",
+    "    # default: AdamW\n",
+    "    # user may pass:\n",
+    "    #   torch.optim.SGD\n",
+    "    #   torch.optim.Adam\n",
+    "    #   any custom optimizer\n",
+    "\n",
+    "    scheduler=None,      \n",
+    "    # default: ReduceLROnPlateau\n",
+    "    # user may pass:\n",
+    "    #   StepLR\n",
+    "    #   OneCycleLR\n",
+    "    #   custom scheduler\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c708a47e-6175-42da-a783-5dccfdfb0b37",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# How training is triggered!\n",
+    "trainer.fit()\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "62d7e5b0-b909-4397-bc72-07f46148e1d4",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Using the Trained Model for Inference\n",
+    "By default, our models use the training statistics to standardize the data for stable training dynamics.\n",
+    "Each dataset contains a scaler, which we can use to transform predictions back into the original data space."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "28c9d4fb-ca7a-47f8-98ff-3d65af4914ec",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import random\n",
+    "import matplotlib.pyplot as plt\n",
+    "\n",
+    "n = 3\n",
+    "channels_to_plot = [0, 1, 2, 3, 4, 5]\n",
+    "\n",
+    "model.eval()\n",
+    "random.seed(42)\n",
+    "indices = random.sample(range(len(test_ds)), n)\n",
+    "\n",
+    "for idx in indices:\n",
+    "    x, y = test_ds[idx]\n",
+    "    inputs = x[\"ts_array\"]\n",
+    "\n",
+    "    with torch.no_grad():\n",
+    "        pred_norm = model(x)\n",
+    "\n",
+    "    y_np = y.squeeze(0).cpu().numpy()\n",
+    "    pred_np = pred_norm.squeeze(0).cpu().numpy()\n",
+    "\n",
+    "    scaler = getattr(test_ds.dataset, 'scaler', None)\n",
+    "    if scaler is not None:\n",
+    "        y_un = scaler.inverse_transform(y_np)\n",
+    "        pred_un = scaler.inverse_transform(pred_np)\n",
+    "    else:\n",
+    "        y_un, pred_un = y_np, pred_np\n",
+    "\n",
+    "    # ---- plotting ----\n",
+    "    C = len(channels_to_plot)\n",
+    "    fig, axes = plt.subplots(C, 1, figsize=(10, 2*C), sharex=True)\n",
+    "\n",
+    "    for ax, ch in zip(axes, channels_to_plot):\n",
+    "        ax.plot(y_un[:, ch], label=\"target\")\n",
+    "        ax.plot(pred_un[:, ch], '--', label=\"pred\")\n",
+    "        ax.set_title(f\"Channel {ch}\")\n",
+    "        ax.grid(True)\n",
+    "        ax.legend(loc=\"upper right\")\n",
+    "\n",
+    "    fig.suptitle(f\"Example {idx}\", fontsize=14)\n",
+    "    plt.tight_layout()\n",
+    "    plt.show()\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5c8e135e-a165-4785-ab1c-148726b08b50",
+   "metadata": {
+    "jupyter": {
+     "source_hidden": true
+    }
+   },
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "charm",
+   "language": "python",
+   "name": "ftse"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.4"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

notebooks/demo_retrieval_anomaly_detection.ipynb

@@ -0,0 +1,620 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# CHARM — Retrieval & Anomaly Detection"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "jupyter": {
+     "source_hidden": true
+    },
+    "tags": [
+     "hide-input"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import numpy as np\n",
+    "import matplotlib.pyplot as plt\n",
+    "from matplotlib.patches import Patch\n",
+    "from sklearn.manifold import TSNE\n",
+    "\n",
+    "from charm import CharmClient\n",
+    "from charm_toolkit.retrieval import cosine_similarity_matrix, retrieval_metrics\n",
+    "from charm_toolkit.anomaly_detection import (\n",
+    "    sliding_window_embeddings,\n",
+    "    knn_anomaly_scores,\n",
+    "    window_scores_to_pointwise,\n",
+    ")\n",
+    "# Zero-shot recipes use sklearn / pyod directly:\n",
+    "from sklearn.ensemble import IsolationForest\n",
+    "\n",
+    "plt.rcParams.update({\n",
+    "    \"figure.facecolor\": \"white\",\n",
+    "    \"axes.facecolor\": \"#FAFAFA\",\n",
+    "    \"axes.grid\": False,\n",
+    "    \"axes.spines.top\": False,\n",
+    "    \"axes.spines.right\": False,\n",
+    "    \"font.size\": 10,\n",
+    "    \"axes.titlesize\": 11,\n",
+    "    \"axes.titleweight\": \"bold\",\n",
+    "})\n",
+    "\n",
+    "client = CharmClient(\n",
+    "    base_url=os.environ[\"CHARM_BASE_URL\"],\n",
+    "    api_key=os.environ.get(\"CHARM_API_KEY\", \"\"),\n",
+    "    timeout=300,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## 1 — Retrieval\n",
+    "\n",
+    "Embed three classes of synthetic time series, then retrieve the most similar ones.  \n",
+    "Classes: **sine** (oscillation), **spike** (sharp transients), **trend** (ramp)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "jupyter": {
+     "source_hidden": true
+    },
+    "tags": [
+     "hide-input"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "np.random.seed(42)\n",
+    "\n",
+    "T, C, N = 128, 2, 6\n",
+    "PALETTE = {\"sine\": \"#1976D2\", \"spike\": \"#E53935\", \"trend\": \"#43A047\"}\n",
+    "\n",
+    "\n",
+    "def generate_sine(n):\n",
+    "    out = []\n",
+    "    for _ in range(n):\n",
+    "        freq = np.random.uniform(0.5, 2.0)\n",
+    "        phase = np.random.uniform(0, 2 * np.pi)\n",
+    "        t = np.linspace(0, 4 * np.pi, T)\n",
+    "        s = np.stack([np.sin(freq * t + phase + c * 0.5) for c in range(C)], axis=-1)\n",
+    "        out.append(s + np.random.randn(T, C) * 0.05)\n",
+    "    return out\n",
+    "\n",
+    "\n",
+    "def generate_spike(n):\n",
+    "    out = []\n",
+    "    for _ in range(n):\n",
+    "        s = np.random.randn(T, C) * 0.1\n",
+    "        for _ in range(np.random.randint(3, 8)):\n",
+    "            s[np.random.randint(0, T)] += np.random.randn(C) * 5.0\n",
+    "        out.append(s)\n",
+    "    return out\n",
+    "\n",
+    "\n",
+    "def generate_trend(n):\n",
+    "    out = []\n",
+    "    for _ in range(n):\n",
+    "        slope = np.random.uniform(-2, 2, size=(1, C))\n",
+    "        t = np.linspace(0, 1, T).reshape(-1, 1)\n",
+    "        out.append(t * slope + np.random.randn(T, C) * 0.2)\n",
+    "    return out\n",
+    "\n",
+    "\n",
+    "sine_samples = generate_sine(N)\n",
+    "spike_samples = generate_spike(N)\n",
+    "trend_samples = generate_trend(N)\n",
+    "\n",
+    "all_samples = sine_samples + spike_samples + trend_samples\n",
+    "all_labels = [\"sine\"] * N + [\"spike\"] * N + [\"trend\"] * N"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "fig, axes = plt.subplots(1, 3, figsize=(12, 2.2))\n",
+    "for ax, (sample, name) in zip(axes, [(sine_samples[0], \"sine\"), (spike_samples[0], \"spike\"), (trend_samples[0], \"trend\")]):\n",
+    "    ax.plot(sample[:, 0], color=PALETTE[name], linewidth=1.0, alpha=0.9)\n",
+    "    ax.plot(sample[:, 1], color=PALETTE[name], linewidth=1.0, alpha=0.5)\n",
+    "    ax.set_title(name)\n",
+    "    ax.set_xlim(0, T)\n",
+    "    ax.tick_params(left=False, labelleft=False)\n",
+    "plt.tight_layout(pad=1.5)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "response = client.embeddings.create(\n",
+    "    descriptions=[[\"ch_0\", \"ch_1\"]] * len(all_samples),\n",
+    "    ts_array=[s.tolist() for s in all_samples],\n",
+    "    return_tensors=\"np\",\n",
+    "    batch_size=32,\n",
+    ")\n",
+    "# Aggregate (N, n_patches, C, D) → (N, D) by mean over patches and channels.\n",
+    "raw = response.embeds\n",
+    "if raw.ndim == 4:\n",
+    "    embeddings = raw.mean(axis=(1, 2))\n",
+    "else:\n",
+    "    embeddings = raw\n",
+    "print(f\"{embeddings.shape[0]} samples → {embeddings.shape[1]}-d embeddings\")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### t-SNE"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tsne = TSNE(n_components=2, perplexity=5, random_state=0, init=\"pca\", learning_rate=\"auto\")\n",
+    "coords = tsne.fit_transform(embeddings)\n",
+    "\n",
+    "fig, ax = plt.subplots(figsize=(5, 4))\n",
+    "for label in [\"sine\", \"spike\", \"trend\"]:\n",
+    "    mask = np.array(all_labels) == label\n",
+    "    ax.scatter(\n",
+    "        coords[mask, 0], coords[mask, 1],\n",
+    "        c=PALETTE[label], s=70, label=label,\n",
+    "        edgecolors=\"white\", linewidth=0.6, zorder=3,\n",
+    "    )\n",
+    "ax.set_xticks([])\n",
+    "ax.set_yticks([])\n",
+    "ax.legend(frameon=True, fontsize=9, loc=\"best\", framealpha=0.9)\n",
+    "ax.set_title(\"CHARM embeddings — t-SNE\")\n",
+    "plt.tight_layout()\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Retrieval accuracy\n",
+    "\n",
+    "One sample per class as query → retrieve from the remaining corpus."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "query_idx = [0, N, 2 * N]\n",
+    "corpus_idx = [i for i in range(len(all_samples)) if i not in query_idx]\n",
+    "\n",
+    "metrics = retrieval_metrics(\n",
+    "    query_emb=embeddings[query_idx],\n",
+    "    corpus_emb=embeddings[corpus_idx],\n",
+    "    query_labels=[all_labels[i] for i in query_idx],\n",
+    "    corpus_labels=[all_labels[i] for i in corpus_idx],\n",
+    "    k_values=[1, 3, 5],\n",
+    ")\n",
+    "\n",
+    "ks = [1, 3, 5]\n",
+    "hits = [metrics[f\"hit_rate@{k}\"] for k in ks]\n",
+    "precs = [metrics[f\"precision@{k}\"] for k in ks]\n",
+    "\n",
+    "fig, ax = plt.subplots(figsize=(4.5, 2.8))\n",
+    "x = np.arange(len(ks))\n",
+    "w = 0.35\n",
+    "ax.bar(x - w/2, hits, w, label=\"hit rate\", color=\"#1976D2\", edgecolor=\"white\", linewidth=0.5)\n",
+    "ax.bar(x + w/2, precs, w, label=\"precision\", color=\"#90CAF9\", edgecolor=\"white\", linewidth=0.5)\n",
+    "ax.set_xticks(x)\n",
+    "ax.set_xticklabels([f\"@{k}\" for k in ks])\n",
+    "ax.set_ylim(0, 1.15)\n",
+    "ax.set_ylabel(\"score\")\n",
+    "ax.set_title(\"Retrieval quality\")\n",
+    "ax.legend(fontsize=8, loc=\"upper right\")\n",
+    "for i, (h, p) in enumerate(zip(hits, precs)):\n",
+    "    ax.text(i - w/2, h + 0.03, f\"{h:.0%}\", ha=\"center\", fontsize=8)\n",
+    "    ax.text(i + w/2, p + 0.03, f\"{p:.0%}\", ha=\"center\", fontsize=8)\n",
+    "plt.tight_layout()\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## 2 — Anomaly Detection\n",
+    "\n",
+    "3-channel sinusoidal signal. First half is clean (reference), second half has 3 injected anomaly bursts.  \n",
+    "Three approaches:\n",
+    "- **A** — kNN distance on windowed embeddings  \n",
+    "- **B** — Reconstruction error (zero-shot backcast)\n",
+    "- **C** — Forecast error (zero-shot forecast vs actual)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "jupyter": {
+     "source_hidden": true
+    },
+    "tags": [
+     "hide-input"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "np.random.seed(123)\n",
+    "\n",
+    "# Synthetic AD setup: long mostly-normal series + 3 short anomaly bursts.\n",
+    "# Normal density much higher than anomaly density to mimic real-world AD.\n",
+    "T_TOTAL = 2000\n",
+    "CHANNELS = [\"sensor_A\", \"sensor_B\", \"sensor_C\"]\n",
+    "C_AD = len(CHANNELS)\n",
+    "\n",
+    "t = np.linspace(0, 16 * np.pi, T_TOTAL)\n",
+    "series = np.stack([np.sin(t), np.cos(t * 0.7), np.sin(t * 1.3 + 1)], axis=-1)\n",
+    "series += np.random.randn(T_TOTAL, C_AD) * 0.05\n",
+    "\n",
+    "gt = np.zeros(T_TOTAL, dtype=np.int8)\n",
+    "# 3 short anomaly bursts in the test half (60 / 1000 = 6 % anomaly density).\n",
+    "anomaly_intervals = [(1100, 1120), (1500, 1520), (1900, 1920)]\n",
+    "for s, e in anomaly_intervals:\n",
+    "    series[s:e] += np.random.randn(e - s, C_AD) * 3.0\n",
+    "    gt[s:e] = 1\n",
+    "\n",
+    "SPLIT = 1000  # first half = clean train; second half = test with anomalies\n",
+    "WINDOW = 64\n",
+    "STRIDE = 8\n",
+    "\n",
+    "train_s = series[:SPLIT]\n",
+    "test_s = series[SPLIT:]\n",
+    "test_gt = gt[SPLIT:]\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ch_colors = [\"#1565C0\", \"#EF6C00\", \"#2E7D32\"]\n",
+    "\n",
+    "fig, axes = plt.subplots(2, 1, figsize=(12, 3.5))\n",
+    "for c in range(C_AD):\n",
+    "    axes[0].plot(train_s[:, c], color=ch_colors[c], linewidth=0.7, alpha=0.85)\n",
+    "axes[0].set_title(\"Train (clean)\")\n",
+    "axes[0].set_xlim(0, SPLIT)\n",
+    "axes[0].tick_params(labelbottom=False)\n",
+    "\n",
+    "for c in range(C_AD):\n",
+    "    axes[1].plot(test_s[:, c], color=ch_colors[c], linewidth=0.7, alpha=0.85)\n",
+    "for s, e in anomaly_intervals:\n",
+    "    axes[1].axvspan(s - SPLIT, e - SPLIT, color=\"#E53935\", alpha=0.12)\n",
+    "axes[1].set_title(\"Test (anomalies highlighted)\")\n",
+    "axes[1].set_xlim(0, len(test_s))\n",
+    "axes[1].set_xlabel(\"timestep\")\n",
+    "plt.tight_layout(pad=1.0)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### A — kNN distance"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "train_emb = sliding_window_embeddings(client, train_s, CHANNELS, window_size=WINDOW, stride=STRIDE, batch_size=32)\n",
+    "test_emb = sliding_window_embeddings(client, test_s, CHANNELS, window_size=WINDOW, stride=STRIDE, batch_size=32)\n",
+    "\n",
+    "knn_scores = knn_anomaly_scores(test_emb, train_emb, k=5, distance=\"cosine\", aggregation=\"mean\")\n",
+    "knn_pw = window_scores_to_pointwise(knn_scores, WINDOW, STRIDE, len(test_s), method=\"mean\")\n",
+    "knn_pw = (knn_pw - knn_pw.min()) / (knn_pw.max() - knn_pw.min() + 1e-8)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "> **`window_scores_to_pointwise` methods:**\n",
+    "> Each timestep is covered by multiple overlapping windows. The `method` controls how to assign a single score:\n",
+    ">\n",
+    "> | Method | Behavior | Use case |\n",
+    "> |--------|----------|----------|\n",
+    "> | `\"mean\"` | Average of all windows covering the point | Smooth, best for offline evaluation |\n",
+    "> | `\"max\"` | Max score among covering windows | Conservative, catches isolated spikes |\n",
+    "> | `\"last\"` | Score of the most recently *completed* window | Online/streaming — score updates only when a window finishes |\n",
+    "> | `\"center\"` | Score of the window centered on the point | Minimal time-shift, tightest alignment |"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### B — Reconstruction error (backcast)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "n_win = (len(test_s) - WINDOW) // STRIDE + 1\n",
+    "test_windows = [test_s[i * STRIDE : i * STRIDE + WINDOW] for i in range(n_win)]\n",
+    "\n",
+    "bc = client.prediction.create(\n",
+    "    descriptions=[CHANNELS] * n_win,\n",
+    "    ts_array=[w.tolist() for w in test_windows],\n",
+    "    target_len=-WINDOW,\n",
+    "    return_tensors=\"np\",\n",
+    "    batch_size=16,\n",
+    ")\n",
+    "\n",
+    "recon = bc.median\n",
+    "recon_err = ((recon - np.array(test_windows)) ** 2).mean(axis=(1, 2))\n",
+    "recon_pw = window_scores_to_pointwise(recon_err, WINDOW, STRIDE, len(test_s), method=\"mean\")\n",
+    "recon_pw = (recon_pw - recon_pw.min()) / (recon_pw.max() - recon_pw.min() + 1e-8)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### C — Forecast error\n",
+    "\n",
+    "Use the preceding window as context, forecast the next `H` steps, compare with actual."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "H = 16  # forecast horizon\n",
+    "CTX = WINDOW  # context length\n",
+    "\n",
+    "n_fc = (len(test_s) - CTX - H) // STRIDE + 1\n",
+    "fc_contexts = [test_s[i * STRIDE : i * STRIDE + CTX] for i in range(n_fc)]\n",
+    "fc_targets = [test_s[i * STRIDE + CTX : i * STRIDE + CTX + H] for i in range(n_fc)]\n",
+    "\n",
+    "fc = client.prediction.create(\n",
+    "    descriptions=[CHANNELS] * n_fc,\n",
+    "    ts_array=[ctx.tolist() for ctx in fc_contexts],\n",
+    "    target_len=H,\n",
+    "    return_tensors=\"np\",\n",
+    "    batch_size=16,\n",
+    ")\n",
+    "\n",
+    "fc_pred = fc.median  # (n_fc, H, C)\n",
+    "fc_actual = np.array(fc_targets)  # (n_fc, H, C)\n",
+    "fc_err = ((fc_pred - fc_actual) ** 2).mean(axis=(1, 2))\n",
+    "\n",
+    "fc_pw = window_scores_to_pointwise(fc_err, CTX + H, STRIDE, len(test_s), method=\"mean\")\n",
+    "fc_pw = (fc_pw - fc_pw.min()) / (fc_pw.max() - fc_pw.min() + 1e-8)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### D — Zero-shot recipes (no labels, no train split)\n",
+    "\n",
+    "Score the **test embeddings directly** — no labels, no train split.\n",
+    "\n",
+    "1. **IsolationForest** — `sklearn.ensemble.IsolationForest`. One line.\n",
+    "2. **CBLOF** — `pyod.models.cblof.CBLOF` (optional dep).\n",
+    "3. **Bootstrap k-NN** — two steps inline: IsolationForest picks the bottom 70% as a clean reference, then `knn_anomaly_scores` distance to that reference. **Best of the three on TSB-AD-M.**\n",
+    "\n",
+    "We L2-normalize embeddings before scoring so all three share the supervised k-NN cell's cosine geometry.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Run three zero-shot recipes on the TEST embeddings directly.\n",
+    "# We L2-normalize so distances are equivalent to cosine — matching the\n",
+    "# supervised k-NN cell's geometry.\n",
+    "from charm_toolkit.retrieval import l2_normalize\n",
+    "test_emb_n = l2_normalize(test_emb)\n",
+    "\n",
+    "# D1 — IsolationForest: 1-line sklearn.\n",
+    "if_scores = -IsolationForest(n_estimators=200, random_state=0).fit(test_emb_n).score_samples(test_emb_n)\n",
+    "\n",
+    "# D2 — CBLOF (pyod): skipped if pyod unavailable or CBLOF can't split clusters.\n",
+    "try:\n",
+    "    from pyod.models.cblof import CBLOF\n",
+    "    cb_scores = CBLOF(n_clusters=4, random_state=0).fit(test_emb_n).decision_scores_\n",
+    "    have_cblof = True\n",
+    "except (ImportError, ValueError) as e:\n",
+    "    cb_scores, have_cblof = None, False\n",
+    "    print(f\"CBLOF skipped: {type(e).__name__}: {e}\")\n",
+    "\n",
+    "# D3 — Bootstrap k-NN: 2 steps. Pick a clean reference with IsolationForest;\n",
+    "# score every window by k-NN distance to that reference.\n",
+    "suspicion = -IsolationForest(n_estimators=200, random_state=0).fit(test_emb_n).score_samples(test_emb_n)\n",
+    "reference = test_emb_n[suspicion <= np.quantile(suspicion, 0.70)]\n",
+    "bs_scores = knn_anomaly_scores(test_emb_n, reference, k=1, distance=\"cosine\")\n",
+    "\n",
+    "def to_pw(s):\n",
+    "    pw = window_scores_to_pointwise(s, WINDOW, STRIDE, len(test_s), method=\"mean\")\n",
+    "    return (pw - pw.min()) / (pw.max() - pw.min() + 1e-8)\n",
+    "\n",
+    "if_pw = to_pw(if_scores)\n",
+    "cb_pw = to_pw(cb_scores) if have_cblof else None\n",
+    "bs_pw = to_pw(bs_scores)\n",
+    "print(f\"IsolationForest scores  shape={if_pw.shape}, range=[{if_pw.min():.3f}, {if_pw.max():.3f}]\")\n",
+    "if have_cblof:\n",
+    "    print(f\"CBLOF scores            shape={cb_pw.shape}, range=[{cb_pw.min():.3f}, {cb_pw.max():.3f}]\")\n",
+    "print(f\"Bootstrap k-NN scores   shape={bs_pw.shape}, range=[{bs_pw.min():.3f}, {bs_pw.max():.3f}]\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Plot zero-shot anomaly scores alongside the test signal & ground truth.\n",
+    "panels = [\n",
+    "    (\"D1 — IsolationForest (sklearn)\",        if_pw, \"#01579B\", \"#4FC3F7\"),\n",
+    "    (\"D3 — Bootstrap k-NN (charm_toolkit)\",   bs_pw, \"#1B5E20\", \"#66BB6A\"),\n",
+    "]\n",
+    "if have_cblof:\n",
+    "    panels.insert(1, (\"D2 — CBLOF (pyod)\", cb_pw, \"#6A1B9A\", \"#BA68C8\"))\n",
+    "\n",
+    "fig, axes = plt.subplots(len(panels) + 1, 1, figsize=(12, 1.7 * (len(panels) + 1)),\n",
+    "                          sharex=True, gridspec_kw={\"hspace\": 0.35})\n",
+    "\n",
+    "axes[0].plot(test_s[:, 0], color=\"#455A64\", linewidth=0.7)\n",
+    "for s, e in anomaly_intervals:\n",
+    "    axes[0].axvspan(s - SPLIT, e - SPLIT, color=\"#E53935\", alpha=0.12)\n",
+    "axes[0].set_ylabel(\"value\"); axes[0].set_title(\"Test signal (sensor_A)\")\n",
+    "\n",
+    "for ax, (title, pw, line_c, fill_c) in zip(axes[1:], panels):\n",
+    "    ax.fill_between(range(len(pw)), pw, color=fill_c, alpha=0.35)\n",
+    "    ax.plot(pw, color=line_c, linewidth=1.0)\n",
+    "    for s, e in anomaly_intervals:\n",
+    "        ax.axvspan(s - SPLIT, e - SPLIT, color=\"#E53935\", alpha=0.10)\n",
+    "    ax.set_ylabel(\"score\"); ax.set_title(title); ax.set_ylim(-0.05, 1.05)\n",
+    "axes[-1].set_xlabel(\"timestep (test)\")\n",
+    "fig.suptitle(\"Zero-shot anomaly detection on test embeddings (red = ground-truth anomalies)\", y=0.995)\n",
+    "plt.tight_layout()\n",
+    "plt.show()\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Results"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "methods = [\n",
+    "    (\"A — kNN distance (supervised)\",            knn_pw,    \"#E65100\", \"#FF6F00\"),\n",
+    "    (\"B — Reconstruction (backcast)\",            recon_pw,  \"#4A148C\", \"#7C4DFF\"),\n",
+    "    (\"C — Forecast error\",                       fc_pw,     \"#1B5E20\", \"#66BB6A\"),\n",
+    "    (\"D1 — IsolationForest (zero-shot)\",         if_pw,     \"#01579B\", \"#4FC3F7\"),\n",
+    "    (\"D3 — Bootstrap k-NN (zero-shot)\",          bs_pw,     \"#2E7D32\", \"#A5D6A7\"),\n",
+    "]\n",
+    "if have_cblof:\n",
+    "    methods.insert(4, (\"D2 — CBLOF (zero-shot)\", cb_pw, \"#6A1B9A\", \"#BA68C8\"))\n",
+    "\n",
+    "fig, axes = plt.subplots(len(methods) + 1, 1, figsize=(12, 1.7 * (len(methods) + 1)),\n",
+    "                          sharex=True, gridspec_kw={\"hspace\": 0.35})\n",
+    "\n",
+    "axes[0].plot(test_s[:, 0], color=\"#455A64\", linewidth=0.7)\n",
+    "for s, e in anomaly_intervals:\n",
+    "    axes[0].axvspan(s - SPLIT, e - SPLIT, color=\"#E53935\", alpha=0.12)\n",
+    "axes[0].set_ylabel(\"value\"); axes[0].set_title(\"Test signal (sensor_A)\")\n",
+    "\n",
+    "for ax, (title, pw, line_c, fill_c) in zip(axes[1:], methods):\n",
+    "    ax.fill_between(range(len(pw)), pw, color=fill_c, alpha=0.35)\n",
+    "    ax.plot(pw, color=line_c, linewidth=1.0)\n",
+    "    for s, e in anomaly_intervals:\n",
+    "        ax.axvspan(s - SPLIT, e - SPLIT, color=\"#E53935\", alpha=0.10)\n",
+    "    ax.set_ylabel(\"score\"); ax.set_title(title); ax.set_ylim(-0.05, 1.05)\n",
+    "axes[-1].set_xlabel(\"timestep (test)\")\n",
+    "plt.tight_layout()\n",
+    "plt.show()\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Quantitative comparison (AUROC + AvgPrec)\n",
+    "\n",
+    "Compare all six methods on the ground-truth anomaly mask using AUROC and Average Precision. The zero-shot recipes should land at or above the supervised k-NN baseline."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sklearn.metrics import roc_auc_score, average_precision_score\n",
+    "\n",
+    "results = [\n",
+    "    (\"A — kNN distance (supervised)\",         knn_pw),\n",
+    "    (\"B — Reconstruction (backcast)\",         recon_pw),\n",
+    "    (\"C — Forecast error\",                    fc_pw),\n",
+    "    (\"D1 — IsolationForest (zero-shot)\",      if_pw),\n",
+    "    (\"D3 — Bootstrap k-NN (zero-shot)\",       bs_pw),\n",
+    "]\n",
+    "if have_cblof:\n",
+    "    results.insert(4, (\"D2 — CBLOF (zero-shot)\", cb_pw))\n",
+    "\n",
+    "print(f\"{'Method':<38}  {'AUROC':>6}  {'AvgPrec':>8}  {'anom-clean':>10}\")\n",
+    "print(\"-\" * 70)\n",
+    "for name, pw in results:\n",
+    "    auc = roc_auc_score(test_gt, pw)\n",
+    "    ap  = average_precision_score(test_gt, pw)\n",
+    "    sep = pw[test_gt == 1].mean() - pw[test_gt == 0].mean()\n",
+    "    print(f\"{name:<38}  {auc:>6.3f}  {ap:>8.3f}  {sep:>+10.3f}\")\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.4"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

notebooks/getting_started.ipynb

@@ -0,0 +1,311 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Getting Started with CHARM SDK\n",
+    "\n",
+    "This notebook demonstrates how to use the CHARM SDK to generate embeddings from time series data."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Setup\n",
+    "\n",
+    "First, let's import the necessary libraries and set up our client:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import numpy as np\n",
+    "from dotenv import load_dotenv\n",
+    "import matplotlib.pyplot as plt\n",
+    "from charm import CharmClient\n",
+    "import asyncio\n",
+    "import nest_asyncio\n",
+    "nest_asyncio.apply()\n",
+    "\n",
+    "# Create a client\n",
+    "# Initialize the CHARM client using environment variables\n",
+    "\n",
+    "\n",
+    "# # Load environment variables from .env file\n",
+    "# load_dotenv()\n",
+    "\n",
+    "# # Get API key and base URL from environment variables\n",
+    "# api_key = os.getenv(\"CHARM_API_KEY\", \"sk-charm-default-key\")\n",
+    "# base_url = os.getenv(\"CHARM_BASE_URL\", \"EMBEDDING_MODEL_ENDPOINT\")\n",
+    "\n",
+    "api_key = \"sk-charm-default-key\"\n",
+    "base_url = \"EMBEDDING_MODEL_ENDPOINT\"\n",
+    "\n",
+    "client = CharmClient(\n",
+    "    base_url=base_url,\n",
+    "    api_key=api_key\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Generate Synthetic Time Series Data\n",
+    "\n",
+    "Let's create some synthetic time series data to demonstrate the SDK:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def generate_sine_wave(freq, amplitude, phase, length=100):\n",
+    "    \"\"\"Generate a sine wave.\"\"\"\n",
+    "    t = np.linspace(0, 10, length)\n",
+    "    return amplitude * np.sin(2 * np.pi * freq * t + phase)\n",
+    "\n",
+    "# Generate 3 different time series\n",
+    "np.random.seed(42)\n",
+    "ts1 = np.column_stack([\n",
+    "    generate_sine_wave(0.5, 1.0, 0),  # Temperature\n",
+    "    generate_sine_wave(0.2, 2.0, np.pi/4)  # Pressure\n",
+    "])\n",
+    "\n",
+    "# phase shift with respect to ts1\n",
+    "ts2 = np.column_stack([\n",
+    "    generate_sine_wave(0.5, 1.5, np.pi/2),  # Temperature\n",
+    "    generate_sine_wave(0.2, 2.5, np.pi/3)  # Pressure\n",
+    "])\n",
+    "\n",
+    "# amplitude change with respect to ts2\n",
+    "ts3 = np.column_stack([\n",
+    "    generate_sine_wave(0.5, 10, np.pi/2),  # Temperature\n",
+    "    generate_sine_wave(0.2, 20, np.pi/3)  # Pressure\n",
+    "])\n",
+    "\n",
+    "# amplitude, phase, frequency change wrt to all\n",
+    "ts4 = np.column_stack([\n",
+    "    generate_sine_wave(0.7, 0.8, np.pi),  # Temperature\n",
+    "    generate_sine_wave(0.4, 1.8, np.pi/6)  # Pressure\n",
+    "])\n",
+    "\n",
+    "# Add some noise\n",
+    "ts1 += np.random.normal(0, 0.1, ts1.shape)\n",
+    "ts2 += np.random.normal(0, 0.1, ts2.shape)\n",
+    "ts3 += np.random.normal(0, 0.1, ts3.shape)\n",
+    "ts4 += np.random.normal(0, 0.1, ts4.shape)\n",
+    "\n",
+    "# Convert to Python lists for the SDK\n",
+    "ts_array = [ts1.tolist(), ts2.tolist(), ts3.tolist(), ts4.tolist()]\n",
+    "descriptions = [[\"temperature\", \"pressure\"] for _ in range(4)]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's visualize our synthetic time series data:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "plt.figure(figsize=(15, 10))\n",
+    "\n",
+    "for i, ts in enumerate([ts1, ts2, ts3, ts4]):\n",
+    "    plt.subplot(4, 1, i+1)\n",
+    "    plt.plot(ts[:, 0], label='Temperature')\n",
+    "    plt.plot(ts[:, 1], label='Pressure')\n",
+    "    plt.title(f'Time Series {i+1}')\n",
+    "    plt.legend()\n",
+    "    plt.grid(True)\n",
+    "\n",
+    "plt.tight_layout()\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Generate Embeddings\n",
+    "\n",
+    "Now, let's use the CHARM SDK to generate embeddings for our time series data. It is possible to generate the embeddings in a synchronous and asynchronous manner (recommended). "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "async def generate_embeddings():\n",
+    "    async with CharmClient(timeout=120) as client:\n",
+    "        response = await client.embeddings.async_create(\n",
+    "            descriptions=descriptions,\n",
+    "            ts_array=ts_array,\n",
+    "            return_tensors=\"np\",\n",
+    "            max_B_per_request=32,\n",
+    "            concurrency_per_call=4, \n",
+    "        )\n",
+    "        return response\n",
+    "\n",
+    "# Run the async function\n",
+    "response = asyncio.run(generate_embeddings())\n",
+    "\n",
+    "# Print information about the response\n",
+    "print(f\"Model: {response.model}\")\n",
+    "print(f\"Embeddings shape: {response.embeds.shape}\")\n",
+    "\n",
+    "# If there's usage information\n",
+    "if response.usage:\n",
+    "    print(f\"Usage: {response.usage}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Visualize Embeddings\n",
+    "\n",
+    "Let's visualize the embeddings using PCA to reduce them to 2 dimensions:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sklearn.decomposition import PCA\n",
+    "\n",
+    "## Double pooling across time and channels and normalizing\n",
+    "double_pooled_normalized_embeddings = response.embeds.mean(axis=1).mean(axis=1).T/np.linalg.norm(response.embeds.mean(axis=1).mean(axis=1), axis=1)\n",
+    "double_pooled_normalized_embeddings = double_pooled_normalized_embeddings.T\n",
+    "\n",
+    "# Reduce dimensions with PCA\n",
+    "pca = PCA(n_components=2)\n",
+    "embeddings_2d = pca.fit_transform(double_pooled_normalized_embeddings)\n",
+    "\n",
+    "# Plot\n",
+    "plt.figure(figsize=(10, 8))\n",
+    "plt.scatter(embeddings_2d[:, 0], embeddings_2d[:, 1], s=100)\n",
+    "\n",
+    "# Add labels\n",
+    "for i, (x, y) in enumerate(embeddings_2d):\n",
+    "    plt.annotate(f\"TS {i+1}\", (x, y), fontsize=12)\n",
+    "    \n",
+    "plt.title(\"PCA of Time Series Embeddings\")\n",
+    "plt.xlabel(\"PC1\")\n",
+    "plt.ylabel(\"PC2\")\n",
+    "plt.grid(True)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## Using PyTorch Tensors\n",
+    "\n",
+    "The CHARM SDK also supports returning embeddings as PyTorch tensors:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import torch\n",
+    "\n",
+    "# Generate embeddings as PyTorch tensors\n",
+    "response_torch = client.embeddings.create(\n",
+    "    descriptions=descriptions,\n",
+    "    ts_array=ts_array,\n",
+    "    return_tensors=\"torch\"  # Get PyTorch tensor\n",
+    ")\n",
+    "\n",
+    "print(f\"PyTorch tensor type: {type(response_torch.embeds)}\")\n",
+    "print(f\"PyTorch tensor shape: {response_torch.embeds.shape}\")\n",
+    "print(f\"PyTorch tensor device: {response_torch.embeds.device}\")\n",
+    "\n",
+    "# Calculate cosine similarity between embeddings\n",
+    "from torch.nn.functional import normalize\n",
+    "\n",
+    "## Double pooling across time and channels and normalizing\n",
+    "double_pooled_normalized_embeddings = normalize(response_torch.embeds.mean(axis=1).mean(axis=1))\n",
+    "\n",
+    "# Calculate cosine similarity matrix\n",
+    "similarity = torch.mm(double_pooled_normalized_embeddings, double_pooled_normalized_embeddings.t())\n",
+    "\n",
+    "# Plot similarity matrix\n",
+    "plt.figure(figsize=(8, 6))\n",
+    "plt.imshow(similarity.cpu().numpy(), cmap='viridis', vmin=0, vmax=1)\n",
+    "plt.colorbar(label='Cosine Similarity')\n",
+    "plt.title('Cosine Similarity Between Time Series Embeddings')\n",
+    "plt.xticks(range(len(ts_array)), [f'TS {i+1}' for i in range(len(ts_array))])\n",
+    "plt.yticks(range(len(ts_array)), [f'TS {i+1}' for i in range(len(ts_array))])\n",
+    "\n",
+    "# Add text annotations\n",
+    "for i in range(len(ts_array)):\n",
+    "    for j in range(len(ts_array)):\n",
+    "        plt.text(j, i, f'{similarity[i, j]:.2f}', \n",
+    "                 ha='center', va='center', color='white' if similarity[i, j] < 0.7 else 'black')\n",
+    "\n",
+    "plt.tight_layout()\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "charm",
+   "language": "python",
+   "name": "charm"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}