ngc1024's picture
Release tokenledger-forecast weights, ONNX, and model card
399c88a verified
|
Raw History Blame Contribute Delete
4.65 kB
---
language:
- en
license: mit
library_name: tokenledger
pipeline_tag: tabular-regression
tags:
- agent-observability
- token-forecasting
- cache-optimization
- onnx
- ai-agents
datasets:
- ngc1024/tokenledger-events
metrics:
- mae
---
# `tokenledger-forecast`: Agent Token & Tail-Burst Quantile Forecaster
A lightweight tabular neural network for predicting per-turn LLM generation length, tail blowout risk (p90), and marginal session costs for autonomous coding agents.
- **Developer:** [ngc1024](https://huggingface.co/ngc1024)
- **Model Type:** Multi-Quantile Pinball Regressor (`nn.Module` with `PyTorchModelHubMixin`)
- **Parameters:** ~2,300 parameters (Ultra-fast client-side execution)
- **Deployment Formats:** PyTorch (`model.safetensors`), ONNX (`model.onnx` for WebAssembly / browser)
- **Dataset:** [`ngc1024/tokenledger-events`](https://huggingface.co/datasets/ngc1024/tokenledger-events)
- **Interactive Space:** [`ngc1024/tokenledger-demo`](https://huggingface.co/spaces/ngc1024/tokenledger-demo)
---
## Why This Model?
In autonomous coding agents (Claude Code, Antigravity, Cursor, Aider), **token spending is heavily tail-dominated**:
1. **Median output is modest** (~250 tokens), but the **95th percentile explodes to 3,100+ tokens**, and complex tool actions can spike up to **31,000+ tokens** in a single turn.
2. Standard point regressors predict mean output, failing to alert developers before a 10× budget blowout.
3. Cache evictions trigger sharp cliffs when idle gaps exceed 5 minutes.
`tokenledger-forecast` predicts output quantiles (**p10, p50, p90**) directly from active session telemetry (prompt window size, previous turn output, elapsed idle gap, cache hit ratio, and context window fill percentage).
---
## Performance & Benchmark vs. Baselines
Evaluated on held-out agent sessions (split strictly by whole thread to eliminate inter-turn data leakage):
| Model / Baseline | Mean Absolute Error (MAE) | Tail Coverage (Target 90%) | Runtime |
| :--- | :--- | :--- | :--- |
| **Dumb Baseline 1 (Constant Median)** | ~1,280 tokens | N/A | < 1 ms |
| **Dumb Baseline 2 (Lag-1 Previous Turn)** | ~1,410 tokens | N/A | < 1 ms |
| **`tokenledger-forecast` (p50)** | **~480 tokens** | **89.6%** | **< 2 ms (CPU / In-Browser)** |
---
## Quickstart
### Using PyTorch (`huggingface_hub`)
```python
from huggingface_hub import PyTorchModelHubMixin
import torch
# Load model directly from Hugging Face Hub
from model import TokenLedgerForecaster
model = TokenLedgerForecaster.from_pretrained("ngc1024/tokenledger-forecast")
model.eval()
# Features: [turn_index_scaled, prompt_k, prev_out_k, gap_log, prev_hit_ratio, window_fill]
sample_features = torch.tensor([[0.3, 14.0, 0.25, 3.8, 1.0, 0.7]], dtype=torch.float32)
predictions = model(sample_features) # returns [p10, p50, p90] in token units
print(f"Predicted p50: {predictions[0, 1].item():.0f} tokens | Tail p90: {predictions[0, 2].item():.0f} tokens")
```
### In-Browser via ONNX Runtime Web
This model includes an optimized `model.onnx` file that runs 100% locally in user browsers with zero server GPU cost:
```javascript
import * as ort from "onnxruntime-web";
const session = await ort.InferenceSession.create("https://huggingface.co/ngc1024/tokenledger-forecast/resolve/main/model.onnx");
const inputTensor = new ort.Tensor("float32", new Float32Array([...features]), [1, 6]);
const results = await session.run({ context_features: inputTensor });
console.log("Quantiles [p10, p50, p90]:", results.quantiles_p10_p50_p90.data);
```
---
## Input Features Schema (`features.json`)
| Feature | Description | Scaling |
| :--- | :--- | :--- |
| `turn_index` | Turn count within thread | `min(turn, 200) / 50.0` |
| `prompt_tokens_k` | Active context window in thousands of tokens | `prompt / 10,000` |
| `prev_output_tokens_k` | Output tokens generated in previous step | `prev_out / 1,000` |
| `gap_log_s` | Logarithmic idle gap in seconds | `log(1 + gap_seconds)` |
| `prev_cache_hit_ratio` | Cache hit percentage of last step | `0.0` to `1.0` |
| `window_fill_ratio` | Context saturation against standard window | `min(2.0, prompt / 200,000)` |
---
## Limitations & Calibration
- **Model Agnostic vs Specific:** Trained on agent session distributions. While prompt accumulation patterns generalize well across LLM agent architectures, token scale varies between model families (e.g., Claude 3.5 vs Gemini 1.5/Flash).
- **Recalibration:** Use `tokenledger.py` locally to collect your own thread logs and fine-tune using `train_and_export.py`.
---
## Citation & License
MIT License. Developed by [ngc1024](https://huggingface.co/ngc1024).