Hawky-ai/EGG-0.5

EGG 0.5 — public research preview

EGG 0.5 scores whether future aggregate advertising click-through rate will decline. It is a custom encoder classifier, not a conversational model, budget optimizer, causal decision system, or production-approved alert service.

Released 12 October 2026 as an experimental checkpoint. Release notes. A newer CPU tree baseline scores higher than EGG on the same exploratory validation set; this release makes no superiority claim.

Start here: Try the model · How it works · Measured benchmarks

Status

This is a public research preview. The checkpoint is uncalibrated. Small exploratory improvements over a numerical baseline do not establish a reliable advantage. A final untouched/prospective evaluation remains outstanding. No production deployment or automated campaign changes are approved by this release.

Benchmarks and model walkthrough

• Full measured benchmarks: baseline comparisons, metric definitions, uncertainty, epoch history, compute measurements, completed checks and evaluations still outstanding.
• How the model works: preprocessing, input layout, inside ModernBERT, custom scoring head, fine-tuning, inference and operational boundaries.
• Machine-readable aggregate benchmark results.

Validation metric	Logistic baseline	EGG 0.5	Tree baseline	Direction
Average precision	0.246349	0.253716	0.303429	Higher
ROC AUC	0.688473	0.690819	0.725089	Higher
Precision in top 10%	28.44%	28.89%	34.09%	Higher
Recall in top 10%	22.34%	22.70%	26.77%	Higher
Brier score	0.105621	0.106058	0.102134	Lower
Log loss	0.357694	0.360773	0.344764	Lower

These results use the same 4,424 validation examples, not an untouched final test. The fixed tree comparator was added later in research; its gains are exploratory and have no paired uncertainty estimate yet. EGG's small AP gain over logistic regression has an uncertainty interval spanning zero. This checkpoint is not established as superior to numerical baselines. See BENCHMARKS.md for the complete protocol.

Additional two-brand historical benchmark

The frozen checkpoint has now been evaluated on 12,321 September examples across two anonymous brands, alongside logistic and tree baselines. EGG average precision is 0.239025 versus 0.256910 for logistic regression overall. Brand A is weaker; the small Brand B gain is uncertain. See per-brand results and uncertainty. Both brands appeared in training, and the historical period was inspected earlier: this is not prospective or untouched evaluation.

Architecture and training

The base encoder is answerdotai/ModernBERT-base, pinned to revision 8949b909ec900327062f0ebf497f51aef5e6f0c8. EGG adds a type embedding, two pre-normalized Transformer encoder layers, and a shared option-marker scoring head. The complete model has 163,785,217 parameters. Training used cross-entropy, one scoring-head warm-up epoch with the encoder frozen, followed by two full fine-tuning epochs. The best checkpoint was selected by validation negative log-likelihood. It was not trained from scratch or with reinforcement learning.

Input and output

Give the model one metric state per ad, prepared from complete reporting days. It accepts structured numeric text, not a natural-language question. The prediction question and answer choices are added internally by inference.py.

1. Prepare the history

Use the last 1, 3, and 7 complete reporting days, all ending on the same day and using the ad account's reporting timezone. These are nested windows: the one-day window is contained in the three-day window, which is contained in the seven-day window. Exclude the incomplete prediction day.

Field for each window	How to calculate it
impressions	Sum impressions over the window
clicks	Sum inline link clicks, matching the training definition; do not substitute all clicks
ctr_*d_pct	100 × total inline link clicks / total impressions, formatted to six decimal places

Calculate CTR from summed counts, not the average of daily CTRs. Do not turn missing reporting days into zero-activity days. Verify source completeness, metric availability at prediction time, and reporting timezone before preparing an input. No future metrics or outcome label should be supplied.

The training cohort used Meta app-promotion ads with at least 5,000 impressions and 30 inline link clicks in the historical seven-day window, plus complete history. Apply these history eligibility checks in preprocessing. The inference helper checks syntax and numeric consistency; it does not enforce exposure, cohort eligibility, source freshness, or reporting-day completeness. Generalization to Google ads, other objectives, low-volume ads, or unseen brands is not established.

2. Use this exact state format

This example is entirely synthetic. Preserve the field names and their order:

task: ctr_deterioration_3d_v1
history_days: 7
impressions_1d: 1000
clicks_1d: 10
ctr_1d_pct: 1.000000
impressions_3d: 3000
clicks_3d: 30
ctr_3d_pct: 1.000000
impressions_7d: 7000
clicks_7d: 70
ctr_7d_pct: 1.000000

Counts must be nonnegative integers. Clicks cannot exceed impressions, cumulative counts must increase or stay equal across nested windows, and the incremental counts between windows must also be consistent. The helper verifies that supplied CTR reconciles with the counts. When impressions are zero, use unknown for that window's CTR. This parser rule does not override the minimum-exposure eligibility rules above.

Exclude brand names and IDs, campaign/ad/ad-set names and IDs, URLs, creative copy, credentials, and all other free text. Keep any private identifier-to-input mapping outside the model input. Unexpected state fields are rejected. The complete tokenized input must fit 512 tokens; oversized inputs are rejected rather than truncated.

3. Save an input file and run it

The CLI reads a JSON object containing a states list. Each item is a multiline string in the exact format above. This executable Python example creates a valid input.json:

import json
from pathlib import Path

state = """task: ctr_deterioration_3d_v1
history_days: 7
impressions_1d: 1000
clicks_1d: 10
ctr_1d_pct: 1.000000
impressions_3d: 3000
clicks_3d: 30
ctr_3d_pct: 1.000000
impressions_7d: 7000
clicks_7d: 70
ctr_7d_pct: 1.000000"""

Path("input.json").write_text(json.dumps({"states": [state]}, indent=2))

From the downloaded model folder, after installing requirements.txt:

python inference.py --model-dir . --input input.json --device cpu

Alternatively, use the included synthetic input file. For direct Python inference:

from inference import load_model, predict_states

model, tokenizer = load_model(".", device="cpu")
results = predict_states(model, tokenizer, [state])
print(results)

Load the model once and reuse it. Each call accepts 1–32 states and returns one result per state in the same order. Split larger collections into batches. No GPU or production database connection is required to try the example. See TESTING.md for authenticated download and environment setup.

4. Interpret the result

The target is a future three-full-reporting-day aggregate CTR below 75% of the preceding seven-day aggregate CTR. The future window begins at the first local midnight at or after prediction. Historical preparation used a morning prediction convention, excluding the already-started prediction day. Correct timezone and actual input availability must be established for prospective use.

For the synthetic input above, the local CPU checkpoint run returned the following scores (rounded; small numerical differences may occur across environments):

[
  {
    "stable_score": 0.6248005,
    "deteriorating_score": 0.3751995,
    "calibrated": false,
    "deployment_approved": false
  }
]

Output	Meaning
stable_score	Softmax score for future CTR staying at or above the threshold
deteriorating_score	Softmax score for future CTR falling below the threshold
calibrated: false	Scores are not validated real-world event probabilities
deployment_approved: false	This checkpoint remains a research preview

The two scores sum to approximately one. In this example the model favors stable. With historical seven-day CTR of 1%, deterioration means future three-day CTR below 0.75%, a relative drop greater than 25%. Stable does not mean profitable, improving, or free of all performance issues.

Do not interpret a score of 0.375 as a validated 37.5% chance of deterioration. No production alert threshold is established. The model does not generate an explanation, forecast revenue, recommend a budget, or approve pausing an ad. Use experimental scores for human-reviewed analysis, then evaluate predictions frozen before future outcomes become available against the numerical baseline.

Run locally

See the complete testing guide for authenticated download, environment setup and the reproducible acceptance suite.

Use a dedicated Python environment (the packaging reload check ran on CPU; training ran on Python 3.12 with CUDA PyTorch 2.8):

python -m pip install -r requirements.txt
python inference.py --model-dir . --input synthetic_example.json --device cpu
python test_release.py --model-dir .

The custom scoring head is required. This package is not directly compatible with a stock AutoModelForSequenceClassification or a generic hosted inference widget. Review inference.py and load this custom architecture explicitly. A GPU is optional for inference; no hosted GPU service is provisioned by uploading these files.

from inference import load_model, predict_states
import json
model, tokenizer = load_model(".", device="cpu")
example = json.load(open("synthetic_example.json"))
print(predict_states(model, tokenizer, example["states"]))

Verification and limitations

Fresh Hub download: 11/11 functional CPU acceptance tests passed. See the test guide and recorded results. This is an inference-correctness check, not a new predictive benchmark.

The exported safetensors checkpoint was reloaded strictly, checked against original-checkpoint outputs on synthetic examples, and tested to reject unexpected identity fields. The source dataset underwent independent feature/label reconstruction, hash, privacy-allowlist, and temporal-window checks. These establish structural integrity, not real-world predictive validity.

Historical metric availability and account timezone mapping remain unverified. Historical rows may have been revised after the nominal prediction date. Repeated windows from the same ad are correlated. The validation set was used to select the checkpoint, so validation comparisons are exploratory. Fresh predictions must be frozen before observing future outcomes, and outcome windows must mature before evaluation.

Privacy and distribution

Training records, customer identifiers, brand names, campaign/ad/ad-set names, source snapshots, identity maps, credentials, and raw private experiment reports are not included. Only aggregate benchmark results, synthetic examples, code, and model artifacts are published. Numeric inputs and model weights can still encode information about training data. The release excludes customer records and identifiers; no formal guarantee against membership inference or memorization is claimed.

No public redistribution license has been selected for the EGG-specific work or fine-tuned artifact. See BASE_MODEL_ATTRIBUTION.md and any included upstream license for the original base model's terms. Public availability does not itself grant an open-source license for the EGG-specific additions.