Decision Models: Their Limits and Reinforcement Learning

1. Introduction

Recently I keep running into Decision Models, starting with TypeSafe's Jev. They are models that do not generate text: a single forward pass returns a probability for each option. Because they can be called far faster and cheaper than an LLM, demos such as the Mario one below have been published.

A demo of Jev clearing Mario 1-1 (from the README of jev-mario)

Watching the video, it looks as if the model judges the timing of each jump one by one and makes its way to the goal. But when I actually read the inputs, I noticed that in many cases the model is simply being made to answer a question that could be answered mechanically.

So I tried the following two things.

• Can a Decision Model make this kind of look-ahead prediction?
• Could it learn to, through reinforcement learning with GRPO, just as LLMs do?

My point in this article is twofold: current Decision Models are, I think, being presented as considerably more than they are, and GRPO that bundles multiple steps together works for Decision Models too.

2. The hype around Decision Models

2.1 What is a Decision Model?

A Decision Model takes a state, a question and a set of options (candidates), and returns a probability for each option. Since it generates no tokens, a single call is very fast, and the intended use is to hand it the many small decisions an agent has to make.

TypeSafe's official blog introduces Jev as follows.

Think of Jev as a frontier-intelligence function call: unstructured state in, typed probabilistic decisions out.

The same post says Jev achieves "similar levels of intelligence on System One tasks" compared to existing LLMs while being "two orders of magnitude faster". It also features a demo that plays Doom, introduced as showing "real-time intelligence" and what can be done with "code + AI".

On the other hand, a Decision Model gives up reasoning ability in exchange for response speed. This is the key point of this article.

2.2 The prompts that actually go in

The headline of the Mario demo from the introduction (jev-mario) is "TypeSafe Jev plays Super Mario Bros." Watching the video, I think most people would take it that the model judged the timing of the jumps and reached the goal.

So what is actually passed to the model? I ran jev-mario's code as it is and printed the request that is sent to Jev.

The request the Mario demo sends to Jev (printed from jev-mario's code; the colours are added for explanation)

In the same way, there is a demo for T-Rex (Chrome's dinosaur game).

Jev playing T-Rex (cropped from laya-vs-jev)

For the T-Rex demo an example is given in its README, so I show that as well.

The request the T-Rex demo sends to Jev (from its README; the colours are added for explanation)

The prediction of what follows each choice has been computed by code and written into the input.

• For "run right", it says dies in 0.6 s
• For the other options, it says how many tiles Mario advances, and even how many of the 36 continuations survive
• The instructions also explain that each option "says what actually happens if Mario does it"
• In T-Rex, the answer is even marked "Best."

Honestly, with this much written down, I think the choice can already be made mechanically. And yet the model is being asked to make it.

2.3 Reading the inputs of public demos

This did not seem to be limited to Mario, so I checked three public game demos. Putting their headlines, what the code computes beforehand, and the text passed to the model side by side gives the following.

Headlines of public demos, and what their code writes into the input

In every demo, "what happens if this option is chosen" is computed by code first and passed in as text or as a flag.

Incidentally, even in T-Rex, where the task is only to pick the option marked "Best.", Jev's choice matched the code's best action 98.7% of the time, and separately the code stepped in and acted 28 times over 5 courses. (Replacing dangerous answers, and emergency avoidance just before a collision.)

2.4 The fine print

It is not that there are no caveats. But they are written in places far less visible than the videos and the headlines.

On TypeSafe's official blog, the "Nuance" section of the Doom demo says the following.

A non-AI doom bot could play better, but we wanted a bot that was reactive to different representations of game state, and most importantly… following instructions was cool as heck!

The README of jev-mario also says the following.

The harness simulates every option and passes the measured outcome, so the only task left is ranking eleven descriptions.

In other words, what is shown large is "intelligence", and what is written small is "a non-AI bot would play better" and "the only task left is ranking eleven descriptions".

Frankly, I felt that it is fair to call this hype. The hard part is solved by code, and what is left for the model is the job of reading the written outcomes and choosing. I think the presentation goes far beyond the actual ability.

2.5 What happens when code does not write the outcomes

So what happens if the code does not write the outcomes? jev-mario also has a record of letting Jev control Mario from a summary of the state alone, without the outcomes (Direct control). The baselines are a set of if-statement rules that use the same information, and code that just keeps holding "run right + jump". Lining up how far each of them got in 1-1 gives the following.

Distance Mario travelled in 1-1 when the outcomes are not written (made from the README of jev-mario)

When code does not write the outcomes, Jev does worse than the if-statement rules, and worse than just holding the buttons.

3. Verification with ViZDoom

3.1 Setup

I also checked this myself with a different game. I used the Predict Position scenario of ViZDoom. (A Doom-based game, like the official demo.)

• Kill an enemy that moves along the wall with a single rocket
• The rocket takes time to arrive, so you have to aim at where the enemy is going

The model receives a JSON with the enemy's position (its bbox on screen) for the last four decisions and the player's own heading, plus four options (turn left / turn right / shoot / wait).

I compared the following conditions on Jev (typesafe/jev-1.13). The evaluation uses the 128 test episodes, always taking the option with the highest probability.

• Jev (no code assistance): only the state and the four options are passed
• Jev (with code assistance): code computes the outcome of each option (hit or miss) beforehand and writes it into the option before passing it
• Jev (code assistance + explanation): in addition, the instructions explain how to read what was written
• Code only: no Jev; simply takes the option that says it will hit
• Random: picks one of the four options at random

The code used for the measurement is scripts/pp_probe.py.

3.2 Results

First, Jev's play in the same scene (the same seed). The left is without code assistance, the right is with code assistance. (A replay of the recorded actions, at about 1.4× speed. The yellow mark shows the enemy's position and is drawn in for explanation.)

Jev's play in the same scene (left: no code assistance, right: with code assistance)

The options the model sees differ as follows. The number at the right end is the probability Jev returned.

The options Jev sees, and the probabilities it returned

The results over 128 episodes were as follows.

Successes on Predict Position (128 episodes)

Without code assistance Jev scored 0/128, below random (12/128). Jev chose "shoot" in 99% of all decisions. In this game, holding the shoot button from the very start never fires, so in 110 episodes it timed out without firing a single shot. (The 18 episodes in which it did fire were all misses.)

With code assistance it goes up to 112/128, and to 126/128 when the instructions explain how to read it. However, code only also scores 126/128, and every episode Jev succeeded on was among those. In other words, putting the model in the loop did not add a single success.

3.3 Checking with Snake

One could argue that whether the rocket hits cannot be known without knowing the physics of the game. So I used Snake, where all the rules are known, and asked questions at three depths: 200 questions each, with the correct answer split evenly between yes and no. The questions are generated by scripts/pp_snake_probe.py.

• Lookup: is this cell part of the snake's body?
• 1-step prediction: does moving in this direction avoid a collision?
• 5-step prediction: after moving, can the snake make 5 more moves?

For example, a board and questions like the following. (The board is passed as text.)

An example of a Snake board and its questions (the drawing is for explanation)

For comparison, I asked an LLM (DeepSeek-V4-Pro) the same questions in two settings, reasoning on and reasoning off: the prompt stays the same and only the reasoning setting of the API is switched.

Accuracy on Snake by kind of question (solid: reasoning on, dashed: reasoning off)

Jev answers every lookup question correctly, but drops to 63.5% as soon as a 1-step prediction is needed. A few lines of code would get 100% on that question.

DeepSeek with reasoning on gets 100% at 1 step and 99% at 5 steps. What is interesting is that the same model with reasoning off gets 51% at 1 step and 50.5% at 5 steps, no different from random choice.

3.4 Summary so far

This is what the results seem to say.

• Decision Models are good at reading what is written in the input and choosing
• They cannot tell what will happen after a choice unless it is written for them (without it, they lose to if-statements and to holding the buttons)
• What makes the difference is not "LLM or not" but whether there is reasoning

Together with what was inside the demos in section 2, I felt once more that the way current Decision Models are presented can fairly be called hype. The hard part is solved by code, and what is left for the model is the job of reading the written outcomes and choosing. That much can be done with a few lines of code.

A Decision Model has no place to think, so whatever lies beyond a choice has to be either "written into the input" or "learned into the weights". The demos do the former with code. Can the latter be done? That is what I try next.

4. Reinforcement learning for Decision Models with GRPO

For LLM agents, reinforcement learning from the success or failure of a task (GRPO) has become common practice. My idea here is that the same should be possible for Decision Models.

Incidentally, Jev itself is described as being trained with reinforcement learning. (The official blog calls it Reinforcement Learning for Calibrated Decisions, RLCD.) However, how it is actually done has not been published. The community is also trying various ways of doing RLCD, but as far as I can tell no method has been settled on yet. I hope what follows can be read as one example showing that reinforcement learning can also be done this way.

4.1 How it maps onto GRPO for LLMs

What is done is the same as GRPO for LLMs. Only the following two things are replaced.

• One action: the next token → a choice among candidates (four options here)
• One trajectory: one answer → one episode (20–30 decisions)

There is one reward at the end of the episode (1 for a hit, 0 for a miss), and it is handed to every decision in that episode. The picture is: bundle the decisions of many steps together, and update all of them by the final outcome.

The flow of GRPO for a Decision Model

Written as formulas, it is as follows. i indexes the 16 plays of the same seed, t the decisions within a play, r_i is the outcome of that play, and ε is 0.2. (There is no KL term to keep the model close to the original.)

$$A_i=\frac{r_i-\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}(r)}{\mathrm{s}\mathrm{t}\mathrm{d}(r)},{\rho }_{i,t}=\frac{{\pi }_{\theta }(a_{i,t}\mid s_{i,t})}{{\pi }_{\mathrm{o}\mathrm{l}\mathrm{d}}(a_{i,t}\mid s_{i,t})}A\_i\; =\; \backslash frac\{r\_i\; -\; \backslash mathrm\{mean\}(r)\}\{\backslash mathrm\{std\}(r)\},\; \backslash qquad\; \backslash rho\_\{i,t\}\; =\; \backslash frac\{\backslash pi\_\backslash theta(a\_\{i,t\}\; \backslash mid\; s\_\{i,t\})\}\{\backslash pi\_\{\backslash mathrm\{old\}\}(a\_\{i,t\}\; \backslash mid\; s\_\{i,t\})\}$$

$$L=-\frac{1}{N}\sum _{i,t}\min ({\rho }_{i,t}{A}_i,\mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}({\rho }_{i,t},1-\epsilon ,1+\epsilon )A_i)L\; =\; -\backslash frac\{1\}\{N\}\; \backslash sum\_\{i,t\}\; \backslash min\backslash left(\; \backslash rho\_\{i,t\}\; A\_i,\backslash \; \backslash mathrm\{clip\}(\backslash rho\_\{i,t\},\; 1-\backslash varepsilon,\; 1+\backslash varepsilon)\backslash ,\; A\_i\; \backslash right)$$

4.2 Implementation

The core of the implementation is quite simple: scripts/pp_rl.py.

Relative scoring within a group

def group_advantages(rewards):       # 16 outcomes (1 or 0) of one seed
    if min(rewards) == max(rewards):
        return None                  # all the same: nothing to learn
    mean, std = statistics.mean(rewards), statistics.pstdev(rewards)
    return [(r - mean) / (std + 1e-8) for r in rewards]

Loss (the same clipping as PPO)

def clipped_loss(new_logp, old_logp, advantages, clip=0.2):
    ratio = (new_logp - old_logp).exp()   # probability ratio per decision
    return -torch.minimum(ratio * advantages,
                          ratio.clamp(1 - clip, 1 + clip) * advantages)

A Decision Model computes "state + question + option" for every option, so training as it is would be quite heavy. I therefore compute the state and the question only once and let the options share that prefix. About 1,950 tokens per decision become about 530. The code is scripts/pp_tree.py.

4.3 Setup

Jev is only available through an API, so I used NanoJev, a public model of the same type (Qwen3-0.6B with decision heads). The public weights solve Predict Position in only 27 of 128 episodes. The input is the same as "no code assistance" in section 3.

I compared the following.

• SFT: another model, already trained with reinforcement learning on this game, serves as the expert, and its actions are learned as the correct answers (512 episodes, 6,788 decisions)
• GRPO: no expert. The model plays the same 512 seeds itself and learns only from whether it hit (one step is 16 seeds × 16 plays = 256 episodes)
• SFT → GRPO: continues with GRPO from the weights where SFT plateaued (the conditions differ from GRPO alone: it uses 1,600 different seeds)

The evaluation uses 640 episodes in total: the 128 public test episodes and 512 seeds that were not used in training. Everything ran on a single A100 (40 GB).

4.4 Results

The success rate during training was as follows. (512 seeds not used in training, always taking the highest-probability option.)

Success rate during training (left: SFT, right: GRPO)

The first thing that stands out is that SFT plateaus after 400 updates. Continuing for another 800 updates did not improve it.

In contrast, GRPO surpassed SFT from the same starting point and the same seeds, without an expert. The success rates over 640 episodes are 19.4% before training, 68.6% for SFT and 79.1% for GRPO. There were 72 episodes that only SFT succeeded on and 139 that only GRPO succeeded on. (p < 0.00001 by a sign test)

Furthermore, continuing with GRPO from the SFT plateau raised it to 81.4%. I think this means that from the point where imitating the expert stalled, learning from outcomes takes it one step higher.

Here is the play of the three models in the same scene. Before training and SFT shoot at once and miss, while GRPO waits, then shoots and hits. (Of the 128 test episodes, there are 36 in which SFT missed and GRPO hit, and 11 the other way round. This scene is chosen from those 36.)

The three models in the same scene (from the left: before training, SFT, GRPO)

4.5 Known weaknesses

The results were not all good, so here are the weaknesses as well.

Results (success rate over 640 episodes; “2× step” and “Basic” are successes out of 128 episodes)

• What it learns is a way of playing that fits the training conditions: GRPO learned to "wait, then shoot". When the time that passes per decision is doubled, it times out without firing a single shot in 75 of 128 episodes. (When it did fire, its hit rate was 64%, higher than SFT's 45%.) In a different game (Basic) too, all 21 failures were timeouts.
• Training is unstable: around step 20 of GRPO, only the evaluation that takes the highest-probability option collapsed for a while.

5. Conclusion

In this article I tested two things: whether a Decision Model can make look-ahead predictions, and whether it could learn to through reinforcement learning with GRPO, just as LLMs do. The results were roughly as follows.

• In the public demos, code computes what happens after each choice and writes it into the input. There is no point in having the model make the choice, and it is considerable hype.
• On ViZDoom, Jev scores 0/128 without code assistance. With code assistance it scores 126/128, but code only scores the same 126/128.
• For Decision Models too, GRPO that bundles multiple steps into one episode simply works. Without an expert, the success rate went from 19% to 79%, surpassing SFT (69%).
• However, what it learns is a way of playing that fits the training conditions, and it breaks down when the conditions change.

If code computes the outcome and writes it into the input, then what the demo shows is the power of the code. I think what Decision Models are good at is reading what is written and choosing quickly.

On the other hand, I was also able to confirm that when look-ahead is needed, there is the path of having the weights learn it with GRPO. Personally, I think there is more potential in this than in making code like that of the demos ever thicker.

I also think the kind of reinforcement learning shown here can be used effectively in far more varied settings, not only in demos like these. For tasks where the right answer cannot be decided in a single step, such as Physical AI, Computer Use and Browser Use, this approach should be useful. At least in my own experiments, I have confirmed that GRPO is effective for Browser Use as well.

If you have thoughts or comments, I would be glad to hear them on X.
X: https://twitter.com/CurveWeb

Thank you for reading.

The code used for the verification is available below. (It is an unofficial extension based on NanoJev. The commands for training and evaluation are in the README.) I used Claude Code for much of the implementation, analysis and drafting. If you find an error, please let me know and I will correct it.

• Hajime-Y/nanojev-grpo

References

• TianyuCodings/NanoJev
• Introducing System One models and Jev (TypeSafe)
• 4esv/jev-mario
• fhshaik/typesafe-mario
• Contrastive-LM/CLM: T-Rex runner
• DeepSeekMath (the GRPO paper)
• Farama-Foundation/ViZDoom
• thainv0212/sonic_doom (the policy used as the SFT expert)