|
Download README.md from ngc1024/tokenledger-forecast: direct link, hf CLI and curl.
- Browser
- Download file 4.65 kB
-
https://huggingface.co/ngc1024/tokenledger-forecast/resolve/main/README.md
- Command line
-
hf download hf://ngc1024/tokenledger-forecast/README.md
-
curl -L -o README.md https://huggingface.co/ngc1024/tokenledger-forecast/resolve/main/README.md
4.65 kB
metadata
language:
- en
license: mit
library_name: tokenledger
pipeline_tag: tabular-regression
tags:
- agent-observability
- token-forecasting
- cache-optimization
- onnx
- ai-agents
datasets:
- ngc1024/tokenledger-events
metrics:
- mae
tokenledger-forecast: Agent Token & Tail-Burst Quantile Forecaster
A lightweight tabular neural network for predicting per-turn LLM generation length, tail blowout risk (p90), and marginal session costs for autonomous coding agents.
- Developer: ngc1024
- Model Type: Multi-Quantile Pinball Regressor (
nn.ModulewithPyTorchModelHubMixin) - Parameters: ~2,300 parameters (Ultra-fast client-side execution)
- Deployment Formats: PyTorch (
model.safetensors), ONNX (model.onnxfor WebAssembly / browser) - Dataset:
ngc1024/tokenledger-events - Interactive Space:
ngc1024/tokenledger-demo
Why This Model?
In autonomous coding agents (Claude Code, Antigravity, Cursor, Aider), token spending is heavily tail-dominated:
- Median output is modest (~250 tokens), but the 95th percentile explodes to 3,100+ tokens, and complex tool actions can spike up to 31,000+ tokens in a single turn.
- Standard point regressors predict mean output, failing to alert developers before a 10× budget blowout.
- Cache evictions trigger sharp cliffs when idle gaps exceed 5 minutes.
tokenledger-forecast predicts output quantiles (p10, p50, p90) directly from active session telemetry (prompt window size, previous turn output, elapsed idle gap, cache hit ratio, and context window fill percentage).
Performance & Benchmark vs. Baselines
Evaluated on held-out agent sessions (split strictly by whole thread to eliminate inter-turn data leakage):
| Model / Baseline | Mean Absolute Error (MAE) | Tail Coverage (Target 90%) | Runtime |
|---|---|---|---|
| Dumb Baseline 1 (Constant Median) | ~1,280 tokens | N/A | < 1 ms |
| Dumb Baseline 2 (Lag-1 Previous Turn) | ~1,410 tokens | N/A | < 1 ms |
tokenledger-forecast (p50) |
~480 tokens | 89.6% | < 2 ms (CPU / In-Browser) |
Quickstart
Using PyTorch (huggingface_hub)
from huggingface_hub import PyTorchModelHubMixin
import torch
# Load model directly from Hugging Face Hub
from model import TokenLedgerForecaster
model = TokenLedgerForecaster.from_pretrained("ngc1024/tokenledger-forecast")
model.eval()
# Features: [turn_index_scaled, prompt_k, prev_out_k, gap_log, prev_hit_ratio, window_fill]
sample_features = torch.tensor([[0.3, 14.0, 0.25, 3.8, 1.0, 0.7]], dtype=torch.float32)
predictions = model(sample_features) # returns [p10, p50, p90] in token units
print(f"Predicted p50: {predictions[0, 1].item():.0f} tokens | Tail p90: {predictions[0, 2].item():.0f} tokens")
In-Browser via ONNX Runtime Web
This model includes an optimized model.onnx file that runs 100% locally in user browsers with zero server GPU cost:
import * as ort from "onnxruntime-web";
const session = await ort.InferenceSession.create("https://huggingface.co/ngc1024/tokenledger-forecast/resolve/main/model.onnx");
const inputTensor = new ort.Tensor("float32", new Float32Array([...features]), [1, 6]);
const results = await session.run({ context_features: inputTensor });
console.log("Quantiles [p10, p50, p90]:", results.quantiles_p10_p50_p90.data);
Input Features Schema (features.json)
| Feature | Description | Scaling |
|---|---|---|
turn_index |
Turn count within thread | min(turn, 200) / 50.0 |
prompt_tokens_k |
Active context window in thousands of tokens | prompt / 10,000 |
prev_output_tokens_k |
Output tokens generated in previous step | prev_out / 1,000 |
gap_log_s |
Logarithmic idle gap in seconds | log(1 + gap_seconds) |
prev_cache_hit_ratio |
Cache hit percentage of last step | 0.0 to 1.0 |
window_fill_ratio |
Context saturation against standard window | min(2.0, prompt / 200,000) |
Limitations & Calibration
- Model Agnostic vs Specific: Trained on agent session distributions. While prompt accumulation patterns generalize well across LLM agent architectures, token scale varies between model families (e.g., Claude 3.5 vs Gemini 1.5/Flash).
- Recalibration: Use
tokenledger.pylocally to collect your own thread logs and fine-tune usingtrain_and_export.py.
Citation & License
MIT License. Developed by ngc1024.