tokenledger-forecast: Agent Token & Tail-Burst Quantile Forecaster

A lightweight tabular neural network for predicting per-turn LLM generation length, tail blowout risk (p90), and marginal session costs for autonomous coding agents.

• Developer: ngc1024
• Model Type: Multi-Quantile Pinball Regressor (nn.Module with PyTorchModelHubMixin)
• Parameters: ~2,300 parameters (Ultra-fast client-side execution)
• Deployment Formats: PyTorch (model.safetensors), ONNX (model.onnx for WebAssembly / browser)
• Dataset: ngc1024/tokenledger-events
• Interactive Space: ngc1024/tokenledger-demo

---

Why This Model?

In autonomous coding agents (Claude Code, Antigravity, Cursor, Aider), token spending is heavily tail-dominated:

1. Median output is modest (~250 tokens), but the 95th percentile explodes to 3,100+ tokens, and complex tool actions can spike up to 31,000+ tokens in a single turn.
2. Standard point regressors predict mean output, failing to alert developers before a 10× budget blowout.
3. Cache evictions trigger sharp cliffs when idle gaps exceed 5 minutes.

tokenledger-forecast predicts output quantiles (p10, p50, p90) directly from active session telemetry (prompt window size, previous turn output, elapsed idle gap, cache hit ratio, and context window fill percentage).

---

Performance & Benchmark vs. Baselines

Evaluated on held-out agent sessions (split strictly by whole thread to eliminate inter-turn data leakage):

Model / Baseline	Mean Absolute Error (MAE)	Tail Coverage (Target 90%)	Runtime
Dumb Baseline 1 (Constant Median)	~1,280 tokens	N/A	< 1 ms
Dumb Baseline 2 (Lag-1 Previous Turn)	~1,410 tokens	N/A	< 1 ms
tokenledger-forecast (p50)	~480 tokens	89.6%	< 2 ms (CPU / In-Browser)

---

Quickstart

Using PyTorch (huggingface_hub)

from huggingface_hub import PyTorchModelHubMixin
import torch

# Load model directly from Hugging Face Hub
from model import TokenLedgerForecaster

model = TokenLedgerForecaster.from_pretrained("ngc1024/tokenledger-forecast")
model.eval()

# Features: [turn_index_scaled, prompt_k, prev_out_k, gap_log, prev_hit_ratio, window_fill]
sample_features = torch.tensor([[0.3, 14.0, 0.25, 3.8, 1.0, 0.7]], dtype=torch.float32)
predictions = model(sample_features)  # returns [p10, p50, p90] in token units
print(f"Predicted p50: {predictions[0, 1].item():.0f} tokens | Tail p90: {predictions[0, 2].item():.0f} tokens")

In-Browser via ONNX Runtime Web

This model includes an optimized model.onnx file that runs 100% locally in user browsers with zero server GPU cost:

import * as ort from "onnxruntime-web";

const session = await ort.InferenceSession.create("https://huggingface.co/ngc1024/tokenledger-forecast/resolve/main/model.onnx");
const inputTensor = new ort.Tensor("float32", new Float32Array([...features]), [1, 6]);
const results = await session.run({ context_features: inputTensor });
console.log("Quantiles [p10, p50, p90]:", results.quantiles_p10_p50_p90.data);

---

Input Features Schema (features.json)

Feature	Description	Scaling
turn_index	Turn count within thread	min(turn, 200) / 50.0
prompt_tokens_k	Active context window in thousands of tokens	prompt / 10,000
prev_output_tokens_k	Output tokens generated in previous step	prev_out / 1,000
gap_log_s	Logarithmic idle gap in seconds	log(1 + gap_seconds)
prev_cache_hit_ratio	Cache hit percentage of last step	0.0 to 1.0
window_fill_ratio	Context saturation against standard window	min(2.0, prompt / 200,000)

---

Limitations & Calibration

• Model Agnostic vs Specific: Trained on agent session distributions. While prompt accumulation patterns generalize well across LLM agent architectures, token scale varies between model families (e.g., Claude 3.5 vs Gemini 1.5/Flash).
• Recalibration: Use tokenledger.py locally to collect your own thread logs and fine-tune using train_and_export.py.

---

Citation & License

MIT License. Developed by ngc1024.