--- language: - en license: mit library_name: tokenledger pipeline_tag: tabular-regression tags: - agent-observability - token-forecasting - cache-optimization - onnx - ai-agents datasets: - ngc1024/tokenledger-events metrics: - mae --- # `tokenledger-forecast`: Agent Token & Tail-Burst Quantile Forecaster A lightweight tabular neural network for predicting per-turn LLM generation length, tail blowout risk (p90), and marginal session costs for autonomous coding agents. - **Developer:** [ngc1024](https://huggingface.co/ngc1024) - **Model Type:** Multi-Quantile Pinball Regressor (`nn.Module` with `PyTorchModelHubMixin`) - **Parameters:** ~2,300 parameters (Ultra-fast client-side execution) - **Deployment Formats:** PyTorch (`model.safetensors`), ONNX (`model.onnx` for WebAssembly / browser) - **Dataset:** [`ngc1024/tokenledger-events`](https://huggingface.co/datasets/ngc1024/tokenledger-events) - **Interactive Space:** [`ngc1024/tokenledger-demo`](https://huggingface.co/spaces/ngc1024/tokenledger-demo) --- ## Why This Model? In autonomous coding agents (Claude Code, Antigravity, Cursor, Aider), **token spending is heavily tail-dominated**: 1. **Median output is modest** (~250 tokens), but the **95th percentile explodes to 3,100+ tokens**, and complex tool actions can spike up to **31,000+ tokens** in a single turn. 2. Standard point regressors predict mean output, failing to alert developers before a 10× budget blowout. 3. Cache evictions trigger sharp cliffs when idle gaps exceed 5 minutes. `tokenledger-forecast` predicts output quantiles (**p10, p50, p90**) directly from active session telemetry (prompt window size, previous turn output, elapsed idle gap, cache hit ratio, and context window fill percentage). --- ## Performance & Benchmark vs. Baselines Evaluated on held-out agent sessions (split strictly by whole thread to eliminate inter-turn data leakage): | Model / Baseline | Mean Absolute Error (MAE) | Tail Coverage (Target 90%) | Runtime | | :--- | :--- | :--- | :--- | | **Dumb Baseline 1 (Constant Median)** | ~1,280 tokens | N/A | < 1 ms | | **Dumb Baseline 2 (Lag-1 Previous Turn)** | ~1,410 tokens | N/A | < 1 ms | | **`tokenledger-forecast` (p50)** | **~480 tokens** | **89.6%** | **< 2 ms (CPU / In-Browser)** | --- ## Quickstart ### Using PyTorch (`huggingface_hub`) ```python from huggingface_hub import PyTorchModelHubMixin import torch # Load model directly from Hugging Face Hub from model import TokenLedgerForecaster model = TokenLedgerForecaster.from_pretrained("ngc1024/tokenledger-forecast") model.eval() # Features: [turn_index_scaled, prompt_k, prev_out_k, gap_log, prev_hit_ratio, window_fill] sample_features = torch.tensor([[0.3, 14.0, 0.25, 3.8, 1.0, 0.7]], dtype=torch.float32) predictions = model(sample_features) # returns [p10, p50, p90] in token units print(f"Predicted p50: {predictions[0, 1].item():.0f} tokens | Tail p90: {predictions[0, 2].item():.0f} tokens") ``` ### In-Browser via ONNX Runtime Web This model includes an optimized `model.onnx` file that runs 100% locally in user browsers with zero server GPU cost: ```javascript import * as ort from "onnxruntime-web"; const session = await ort.InferenceSession.create("https://huggingface.co/ngc1024/tokenledger-forecast/resolve/main/model.onnx"); const inputTensor = new ort.Tensor("float32", new Float32Array([...features]), [1, 6]); const results = await session.run({ context_features: inputTensor }); console.log("Quantiles [p10, p50, p90]:", results.quantiles_p10_p50_p90.data); ``` --- ## Input Features Schema (`features.json`) | Feature | Description | Scaling | | :--- | :--- | :--- | | `turn_index` | Turn count within thread | `min(turn, 200) / 50.0` | | `prompt_tokens_k` | Active context window in thousands of tokens | `prompt / 10,000` | | `prev_output_tokens_k` | Output tokens generated in previous step | `prev_out / 1,000` | | `gap_log_s` | Logarithmic idle gap in seconds | `log(1 + gap_seconds)` | | `prev_cache_hit_ratio` | Cache hit percentage of last step | `0.0` to `1.0` | | `window_fill_ratio` | Context saturation against standard window | `min(2.0, prompt / 200,000)` | --- ## Limitations & Calibration - **Model Agnostic vs Specific:** Trained on agent session distributions. While prompt accumulation patterns generalize well across LLM agent architectures, token scale varies between model families (e.g., Claude 3.5 vs Gemini 1.5/Flash). - **Recalibration:** Use `tokenledger.py` locally to collect your own thread logs and fine-tune using `train_and_export.py`. --- ## Citation & License MIT License. Developed by [ngc1024](https://huggingface.co/ngc1024).