markopoloaiinc/Athena-mvrko-4B

Athena (4B): a Large Event Model for shopping behavior

Athena predicts the next thing a real shopper does on a retail page. It reads a short persona, the actions the shopper has taken so far in the session, and the interactive elements on the current page. It returns one action as JSON: the element the shopper clicks, the text they type, or terminate when they leave.

On the OPeRA test split (992 actions from 15 shoppers), Athena predicts 24.50% of actions exactly. That is the highest score among the models we evaluated in one harness. GPT-5.6 scores 22.58%, Claude Sonnet 5 18.35% and Claude Opus 4.8 12.70%, all prompted zero-shot. The 1.92-point lead over GPT-5.6 is within sampling noise. Fine-tuned systems from other groups report higher exact match on OPeRA under different setups; they are listed under Published results on OPeRA.

Athena is the base model of the mvrko simulation track.

• Developer: Markopolo AI
• Release: mvrko-sim-1 (weights commit 5865418)
• Base model: Qwen/Qwen3-4B, fine-tuned with LoRA and merged. This repository holds the full weights; no adapter loading is needed.
• Training data: first-party shopping interaction data owned by Markopolo AI Inc., not released. OPeRA was used for evaluation only.
• Input: persona, prior actions and current page elements, as text
• Output: one next action as JSON (action_type, click_type, semantic_id, input_text)
• Benchmark: OPeRA next-action prediction, strict exact match
• Repository: markopoloaiinc/Athena-mvrko-4B

Score	24.50% exact match on OPeRA test (243 of 992); 95% interval 21.9% to 27.3%
Base	Qwen3-4B, LoRA r=32, α=32, merged; 32,768-token context
Output validity	98.9% of the 992 test outputs match the output schema
Tested with	transformers 5.5.0, torch 2.8.0, bf16, greedy decoding
Not included	Probabilities. Athena predicts one action and is not a calibrated conversion score

---

Results

Our harness: OPeRA test split, 992 actions

Every model in our harness saw the same 992 inputs (test file md5 1e02a30d…) in Athena's input format and was scored by the same strict exact-match scorer. Athena is fine-tuned; GPT-5.6, Claude Sonnet 5 and Claude Opus 4.8 are prompted zero-shot. The GPT-4.1 row is the OPeRA authors' published number, shown for reference. It was not run in our harness and was scored on 902 actions.

Model	Setup	Actions scored	Exact match	Athena's lead (points)	Statistical read
Athena 4B	fine-tuned	992	24.50%
GPT-5.6	prompted, our harness	992	22.58%	+1.92	within noise (z ≈ 1.0)
GPT-4.1	prompted, OPeRA paper	902	21.51%	+2.99	not separated (z ≈ 1.5)
Claude Sonnet 5	prompted, our harness	992	18.35%	+6.15	separated (z ≈ 3.4)
Claude Opus 4.8	prompted, our harness	992	12.70%	+11.80	separated (z ≈ 6.8)

How to read the margins:

• The z values come from an unpaired two-proportion test that treats every action as independent. The 992 actions come from 90 sessions by 15 shoppers, so they are correlated, and the true uncertainty is wider than these values suggest.
• A paired McNemar test on the shared 992 actions is the right test for the GPT-5.6 comparison and has not been run yet. Until it is, read Athena and GPT-5.6 as tied, with Athena nominally ahead.
• The OPeRA paper describes its test set as 992 actions but reports its baselines on 902, so the GPT-4.1 row is not a like-for-like comparison.

Published results on OPeRA

Other groups have trained or prompted models on OPeRA. We have not run these systems in our harness, and their test sets, inputs and context lengths differ from ours, so the numbers are not directly comparable to the table above. They are here so readers can place Athena.

System	Model	Method and inputs	Test set	Exact match
Customer-R1 (Wang et al., 2025)	Qwen2.5-7B-Instruct-1M	SFT then RL; HTML of earlier pages, rationales, survey and interview persona; 65K context	OPeRA test, 902 actions	39.58%
Customer-R1, SFT only	Qwen2.5-7B-Instruct-1M	Same inputs, SFT only	OPeRA test, 902 actions	35.14%
ContextSim (Bougie et al., 2026)	Qwen3-8B	SFT on synthesized reasoning, with simulated daily-life context	Own time-based split	35.37%
IToM (Chen et al., 2026)	Claude Opus 4.6	Prompted with the OPeRA prompt and a persona inferred from each shopper's other sessions	985 actions	25.69%

Fine-tuned models score well above prompted ones across these papers, which is the same pattern our harness shows. Two differences matter when placing Athena. Customer-R1 and ContextSim are trained on OPeRA's own training data, while Athena is trained on Markopolo's first-party data and sees OPeRA only at test time. Athena is also smaller (4B against 7B and 8B), uses a shorter context (32K against 65K) and reads a compact page-element list in place of full page HTML. We have not run ablations that separate these effects.

Where prompted models fail

The OPeRA paper's error analysis (Section 4.4, Tables 6 and 7) covers four prompted models: GPT-4.1, DeepSeek-R1, Claude 3.7 Sonnet and Llama 3.3 70B. For every one of them the largest error type is clicking the wrong element, at 60.8% to 75.8% of the 902 test actions, or 77% to 83% of each model's errors. Outputs outside the action space are rare (0% to 4.1%). The models also seldom predict that a shopper will leave: 4.4% of test actions are terminations, and Claude 3.7 Sonnet and Llama 3.3 never predicted one.

So prompted models produce well-formed actions and choose the wrong target. Picking the element a specific shopper clicks next is what fine-tuning on recorded shopper behavior teaches.

---

Input format

The demo Space formats each input in three sections and ends it with ## Next action:. This example is shortened from a real OPeRA test page:

## Persona
Gender:Female; Age:25-34 years old; City:Blacksburg; Employment status:Student

## Prior actions
{"action_type": "click", "click_type": "other", "semantic_id": "go_to_cart"}

## Current page elements
div[nav_bar]
input[nav_bar.search_input]
a[nav_bar.cart_button]: Cart
div[active_item_list]
a[active_item_list.<product>.product_detail]: Airkeep Car Air Freshener - Ocean Mist Sea Salt
button[active_item_list.<product>.delete]: Delete
span[subtotal]: Subtotal (1 item)
input[proceed_to_checkout]: Proceed to checkout

## Next action:

• Persona: demographic fields from the OPeRA participant survey.
• Prior actions: one JSON action per line in the output schema below, or (none) at the start of a session.
• Current page elements: one line per named element in OPeRA's simplified HTML, written as tag[semantic_id]: visible text. The semantic IDs come from OPeRA's page parser. OPeRA's simplified HTML runs from about 2.8K to 626K characters per action; this list keeps only the named elements.

Output format

One JSON object per prediction. A click looks like this:

{"action_type": "click", "click_type": "product_link", "semantic_id": "active_item_list.<product>.product_detail"}

An input carries semantic_id and input_text; a terminate action carries no target fields.

Field	Values
action_type	click, input, terminate
click_type	Clicks only. One of the 13 OPeRA click types: review, search, product_option, product_link, other, purchase, nav_bar, page_related, quantity, suggested_term, cart_side_bar, cart_page_select, filter
semantic_id	Clicks and inputs. The exact semantic ID of the target element on the current page
input_text	Inputs only. The text typed

Scoring. A prediction counts only if action_type and semantic_id both equal the ground truth, and for inputs input_text must match too. There is no partial credit, so clicking a similar product or a different button for the same purpose scores zero.

---

Quickstart

Tested with transformers 5.5.0 and torch 2.8.0.

import json, torch
from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "markopoloaiinc/Athena-mvrko-4B"
SYSTEM = (
    "You simulate a specific online shopper. Given the shopper's persona, their prior "
    "actions, and the current page's interactive elements, predict their NEXT action.\n"
    "Actions: click{click_type, semantic_id} | input{semantic_id, input_text} | terminate.\n"
    "Output ONLY JSON."
)

tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(
    REPO, dtype=torch.bfloat16, device_map="cuda", attn_implementation="sdpa"
).eval()

# Real OPeRA page states from the Athena demo Space; each ends with "## Next action:"
path = hf_hub_download("mirsakib/athena-demo", "examples.json", repo_type="space")
user_turn = json.load(open(path))[0]

messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": user_turn}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
enc = tok(prompt, return_tensors="pt", truncation=True, max_length=32768).to("cuda")
out = model.generate(**enc, max_new_tokens=96, do_sample=False, pad_token_id=tok.pad_token_id)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))

Two things change results if you get them wrong:

• Use greedy decoding. This repository's generation_config.json carries Qwen3's sampling defaults (temperature 0.6, top_p 0.95, top_k 20). A bare pipeline() or generate() call samples unless you pass do_sample=False.
• Keep the system string and the input layout as shown. This is the prompt the demo Space uses, and small wording changes to a prompt can move exact-match scores.

---

Hardware and deployment

• Precision: bf16 for training and evaluation.
• Weights: 8.04 GB on disk (4.02B parameters, bf16, safetensors).
• KV cache: 144 KiB per token (36 layers × 8 KV heads × 128 dims × keys and values × 2 bytes), or 4.5 GiB at 32,768 tokens.
• GPU: one GPU is enough. A 24 GB card covers full 32K-context inference; inputs of 4K to 8K tokens, enough for most single pages, fit in 16 GB. These figures are calculated from config.json, not measured.
• Headline run: one NVIDIA B300, 32,768-token context, batch size 2, greedy decoding, sdpa attention.

Each prediction is at most 96 output tokens with no reasoning tokens. Inference runs locally in batches, without per-call API round trips or rate limits, and session data stays in your environment.

---

Applications

Application	Needs	Status
Next-action prediction, such as prefetching the page a shopper is likely to open next	Athena	Benchmarked
Session replay: scoring a logged journey one step at a time	Athena	Benchmarked; the 992-action result is a replay of this kind, with the real action applied after each step
Free-running simulation of a whole journey	Athena plus an environment that returns a new page for any action	Not demonstrated. OPeRA contains logged sessions only
Calibrated conversion or intent scores	A different mvrko component (the calibrated intent head, evaluated on AUC and ECE)	Not this model

Athena predicts actions and is scored on exact match. It produces no probabilities, so calibration claims made for other mvrko components do not apply to it.

---

Training details

Setting	Value
Base	Qwen/Qwen3-4B
Training data	First-party shopping interaction data owned by Markopolo AI Inc. (not released). No OPeRA data was used for training
Architecture	Dense decoder: 36 layers, hidden size 2,560, MLP size 9,728, 32 attention heads, 8 KV heads, head dim 128, vocabulary 151,936, tied embeddings
Parameters	4.02B total, 3.63B excluding embeddings
Method	LoRA (r=32, α=32, dropout 0.1), merged into the base weights after training
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Context length	32,768 tokens
Objective	Loss on the next-action tokens only (completion-only masking)
Epochs, learning rate, schedule	1 epoch, 1e-4, cosine with 3% warmup
Precision	bf16 with gradient checkpointing
Hardware	2× NVIDIA B300, DDP via torchrun
Evaluation decoding	Greedy, max_new_tokens=96

---

Evaluation artifacts

The following are available on request through this repository's Community tab:

• the evaluation harness: runner, output parser, scorer and the frozen prompt file
• the observation-space parser that turns OPeRA's simplified HTML into the page-element list
• per-example outputs for Athena, GPT-5.6, Claude Sonnet 5 and Claude Opus 4.8 on all 992 actions
• the run record, which pins transformers 5.5.0, torch 2.8.0+cu129, bf16, greedy decoding and batch size 2

Checks completed:

• A cold-start rerun reproduced 24.50% on the indexed, non-deduplicated test file (md5 1e02a30d…).
• 981 of the 992 outputs (98.9%) validate against the output schema.

---

Limitations

• Fine-tuned against prompted. Athena is fine-tuned for next-action prediction on Markopolo data and the frontier models are prompted zero-shot. The comparison measures what specialization buys, not raw model capability.
• The input format suits Athena. All models in our harness read Athena's compact page-element list, which the frontier models see for the first time. Under OPeRA's own prompt, the IToM paper reports 19.09% for Claude Opus 4.6 with no persona, against 12.70% for Claude Opus 4.8 in our harness. The model versions and test sizes differ, but the gap suggests input format moves prompted scores by several points.
• The GPT-5.6 lead is not significant. 1.92 points at n=992 is a tie with a nominal edge until a paired test says otherwise.
• Published fine-tuned systems score higher. Customer-R1 reports 39.58% on 902 OPeRA test actions with a 7B model trained on OPeRA, a longer context and richer inputs. Athena is not the best published result on OPeRA.
• Small, clustered, narrow test set. The 992 actions come from 15 US shoppers in 90 Amazon sessions, in English. How well Athena carries over to other retailers, countries or languages has not been measured.
• No ablations yet. We have not isolated the effect of the page-element format, the 32K context or the persona.
• Single-step only. Every prediction is made from a logged state. Athena has not been tested as a free-running shopper.
• No calibration. Exact match measures whether the predicted action is right, not how well any confidence is calibrated, and Athena outputs no confidence.

---

Responsible use

Intended uses: research on shopper behavior simulation; testing interface and funnel changes against logged sessions; prefetching likely next pages; scoring logged journeys step by step.

Out of scope:

• decisions about individual people with legal or financial effect, such as credit, insurance or eligibility
• varying prices, offers or treatment by gender, age or other personal attributes in the persona
• identifying or profiling individual shoppers
• designing interfaces meant to mislead or pressure shoppers
• treating a prediction as a probability or a guarantee

Data. Athena was trained on first-party shopping interaction data owned by Markopolo AI Inc., which is not released. OPeRA, used here for evaluation, was collected from recruited, paid US participants under Northeastern University IRB oversight, and its authors masked personal information before release. Apply your own consent and privacy controls to any production session data you send to the model.

Accuracy. About one prediction in four is exactly right on the test set. Treat each output as an estimate of a likely next step.

---

Related work

• Wang, Z., Lu, Y., Li, W., Amini, A., Sun, B., Bart, Y., Lyu, W., Gesi, J., Wang, T., Huang, J., Su, Y., Ehsan, U., Alikhani, M., Li, T. J.-J., Chilton, L., and Wang, D. (2025). OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation. arXiv:2506.05606. The benchmark and its prompted baselines.
• Wang, Z., Lu, Y., Zhang, Y., Huang, J., and Wang, D. (2025). Customer-R1: Personalized Simulation of Human Behaviors via RL-based LLM Agent in Online Shopping. arXiv:2510.07230. SFT and RL with persona conditioning on OPeRA.
• Zhang, Y., et al. (2026). Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learning. ICLR 2026. arXiv:2507.17842. RL with hierarchical rewards; its main results are on a separate proprietary shopping dataset.
• Zhang, Y., et al. (2025). See, Think, Act: Online Shopper Behavior Simulation with VLM Agents. arXiv:2510.19245. Adds page screenshots and uses a reorganized OPeRA split with scroll actions, so its numbers are not comparable to ours.
• Bougie, N., et al. (2026). Beyond Offline A/B Testing: Context-Aware Agent Simulation for Recommender System Evaluation. arXiv:2604.09549. ContextSim.
• Chen, M., Lu, F., Chen, C.-F., Tran, L. V., and Katukuri, J. (2026). Inverse Theory of Mind Modeling for Content Recommendation: From Web Browsing to Dynamic Intelligent Interfaces. RecSys 2026. arXiv:2608.11354. IToM.

---

License

The Athena weights are released under the Apache License 2.0; see LICENSE. The base model, Qwen3-4B, is also Apache-2.0, and its license and notices continue to apply. OPeRA is licensed CC BY 4.0 by its authors, and the example inputs in this repository and the demo Space are derived from it.

Citation

@misc{markopolo2026athena,
  title        = {Athena: a 4B Large Event Model for next-action prediction of online shoppers},
  author       = {{Markopolo AI}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/markopoloaiinc/Athena-mvrko-4B}},
  note         = {Release mvrko-sim-1}
}

@misc{wang2025opera,
  title         = {OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation},
  author        = {Ziyi Wang and Yuxuan Lu and Wenbo Li and Amirali Amini and Bo Sun and Yakov Bart and Weimin Lyu and Jiri Gesi and Tian Wang and Jing Huang and Yu Su and Upol Ehsan and Malihe Alikhani and Toby Jia-Jun Li and Lydia Chilton and Dakuo Wang},
  year          = {2025},
  eprint        = {2506.05606},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2506.05606}
}

Contact

Questions and issues: open a discussion in this repository's Community tab.