|
Download README.md from ngc1024/tokenledger-forecast: direct link, hf CLI and curl.
- Browser
- Download file 4.65 kB
-
https://huggingface.co/ngc1024/tokenledger-forecast/resolve/main/README.md
- Command line
-
hf download hf://ngc1024/tokenledger-forecast/README.md
-
curl -L -o README.md https://huggingface.co/ngc1024/tokenledger-forecast/resolve/main/README.md
4.65 kB
| language: | |
| - en | |
| license: mit | |
| library_name: tokenledger | |
| pipeline_tag: tabular-regression | |
| tags: | |
| - agent-observability | |
| - token-forecasting | |
| - cache-optimization | |
| - onnx | |
| - ai-agents | |
| datasets: | |
| - ngc1024/tokenledger-events | |
| metrics: | |
| - mae | |
| # `tokenledger-forecast`: Agent Token & Tail-Burst Quantile Forecaster | |
| A lightweight tabular neural network for predicting per-turn LLM generation length, tail blowout risk (p90), and marginal session costs for autonomous coding agents. | |
| - **Developer:** [ngc1024](https://huggingface.co/ngc1024) | |
| - **Model Type:** Multi-Quantile Pinball Regressor (`nn.Module` with `PyTorchModelHubMixin`) | |
| - **Parameters:** ~2,300 parameters (Ultra-fast client-side execution) | |
| - **Deployment Formats:** PyTorch (`model.safetensors`), ONNX (`model.onnx` for WebAssembly / browser) | |
| - **Dataset:** [`ngc1024/tokenledger-events`](https://huggingface.co/datasets/ngc1024/tokenledger-events) | |
| - **Interactive Space:** [`ngc1024/tokenledger-demo`](https://huggingface.co/spaces/ngc1024/tokenledger-demo) | |
| --- | |
| ## Why This Model? | |
| In autonomous coding agents (Claude Code, Antigravity, Cursor, Aider), **token spending is heavily tail-dominated**: | |
| 1. **Median output is modest** (~250 tokens), but the **95th percentile explodes to 3,100+ tokens**, and complex tool actions can spike up to **31,000+ tokens** in a single turn. | |
| 2. Standard point regressors predict mean output, failing to alert developers before a 10× budget blowout. | |
| 3. Cache evictions trigger sharp cliffs when idle gaps exceed 5 minutes. | |
| `tokenledger-forecast` predicts output quantiles (**p10, p50, p90**) directly from active session telemetry (prompt window size, previous turn output, elapsed idle gap, cache hit ratio, and context window fill percentage). | |
| --- | |
| ## Performance & Benchmark vs. Baselines | |
| Evaluated on held-out agent sessions (split strictly by whole thread to eliminate inter-turn data leakage): | |
| | Model / Baseline | Mean Absolute Error (MAE) | Tail Coverage (Target 90%) | Runtime | | |
| | :--- | :--- | :--- | :--- | | |
| | **Dumb Baseline 1 (Constant Median)** | ~1,280 tokens | N/A | < 1 ms | | |
| | **Dumb Baseline 2 (Lag-1 Previous Turn)** | ~1,410 tokens | N/A | < 1 ms | | |
| | **`tokenledger-forecast` (p50)** | **~480 tokens** | **89.6%** | **< 2 ms (CPU / In-Browser)** | | |
| --- | |
| ## Quickstart | |
| ### Using PyTorch (`huggingface_hub`) | |
| ```python | |
| from huggingface_hub import PyTorchModelHubMixin | |
| import torch | |
| # Load model directly from Hugging Face Hub | |
| from model import TokenLedgerForecaster | |
| model = TokenLedgerForecaster.from_pretrained("ngc1024/tokenledger-forecast") | |
| model.eval() | |
| # Features: [turn_index_scaled, prompt_k, prev_out_k, gap_log, prev_hit_ratio, window_fill] | |
| sample_features = torch.tensor([[0.3, 14.0, 0.25, 3.8, 1.0, 0.7]], dtype=torch.float32) | |
| predictions = model(sample_features) # returns [p10, p50, p90] in token units | |
| print(f"Predicted p50: {predictions[0, 1].item():.0f} tokens | Tail p90: {predictions[0, 2].item():.0f} tokens") | |
| ``` | |
| ### In-Browser via ONNX Runtime Web | |
| This model includes an optimized `model.onnx` file that runs 100% locally in user browsers with zero server GPU cost: | |
| ```javascript | |
| import * as ort from "onnxruntime-web"; | |
| const session = await ort.InferenceSession.create("https://huggingface.co/ngc1024/tokenledger-forecast/resolve/main/model.onnx"); | |
| const inputTensor = new ort.Tensor("float32", new Float32Array([...features]), [1, 6]); | |
| const results = await session.run({ context_features: inputTensor }); | |
| console.log("Quantiles [p10, p50, p90]:", results.quantiles_p10_p50_p90.data); | |
| ``` | |
| --- | |
| ## Input Features Schema (`features.json`) | |
| | Feature | Description | Scaling | | |
| | :--- | :--- | :--- | | |
| | `turn_index` | Turn count within thread | `min(turn, 200) / 50.0` | | |
| | `prompt_tokens_k` | Active context window in thousands of tokens | `prompt / 10,000` | | |
| | `prev_output_tokens_k` | Output tokens generated in previous step | `prev_out / 1,000` | | |
| | `gap_log_s` | Logarithmic idle gap in seconds | `log(1 + gap_seconds)` | | |
| | `prev_cache_hit_ratio` | Cache hit percentage of last step | `0.0` to `1.0` | | |
| | `window_fill_ratio` | Context saturation against standard window | `min(2.0, prompt / 200,000)` | | |
| --- | |
| ## Limitations & Calibration | |
| - **Model Agnostic vs Specific:** Trained on agent session distributions. While prompt accumulation patterns generalize well across LLM agent architectures, token scale varies between model families (e.g., Claude 3.5 vs Gemini 1.5/Flash). | |
| - **Recalibration:** Use `tokenledger.py` locally to collect your own thread logs and fine-tune using `train_and_export.py`. | |
| --- | |
| ## Citation & License | |
| MIT License. Developed by [ngc1024](https://huggingface.co/ngc1024). | |