mihailgribov/typecastlm-qwen3.5-3.8b
Text Classification • 4B • Updated • 66
Mikhail Gribov PRO
mihailgribov
AI & ML interests
Understanding LLMs from the inside - probing internals, and testing what survives when the model becomes an agent
Recent Activity
reacted to anthonym21's post with 🔥 5 minutes ago
TypeSafe launched Jev with a claim and no method: a model trained with "reinforcement learning for calibrated decisions," whose stated probabilities are supposed to match how often it turns out to be right. No reward function, no training procedure, no calibration curves. So I rebuilt the objective from the announcement alone and tested it at a scale I can fully fine-tune on one RTX 4080.
My guess at the reward is one line: r = c - p_a. c is 1 if the sampled answer was correct, p_a is the probability the model assigned it. Say 0.9 and be right, earn 0.1. Say 0.9 and be wrong, pay 0.9. With p_a detached, REINFORCE on this is an unbiased estimator of half the gradient of the Brier score, from bandit feedback alone: the environment reveals only whether the action you took was right, never what the options you didn't take would have said.
Two runs, same warmup checkpoint, same 32,000 rows, same optimizer and learning rate. With the outcome-only reward r = c, confidence goes to 0.991 and Brier ends worse than the checkpoint it started from. With the subtraction, accuracy goes 0.748 -> 0.808 on held-out test, ECE stays 0.023, Brier drops 0.339 -> 0.267. The difference is one subtraction.
On the probe below (tickets with two equally cued departments, 0.5 the ideal), RLCD lands at 0.593 max probability while the outcome-only reward says 0.990. The reliability diagram tells the same story.
This is a 0.6B base model and 80 minutes on one consumer GPU. It says nothing about how TypeSafe trained Jev. It says the objective is coherent, and cheap to check.
Everything is open:
Training code, ablation, evaluation: https://github.com/anthony-maio/eve-rlcd
Decision-only checkpoint: https://huggingface.co/anthonym21/qwen3-0.6b-rlcd-decision
Dataset, the exact bytes of every run (64k/8k/8k typed questions): https://huggingface.co/datasets/anthonym21/rlcd-decision-v1
Full write-up: https://anthonymaio.substack.com/p/honest-about-uncertainty-i-tried posted an update 7 minutes ago
Will your AI agent tell you it was attacked?
We took the same agent from our earlier experiment and added one thing: a twentieth tool, `escalate_security_incident`.
The system prompt said nothing about attacks or when to use it. We then ran the same 395 injected emails through nine agentic models.
Alarm rates ranged from 49% to zero.
The unexpected result came from the newest model in the test, `gpt-6-astra`.
Astra did not follow a single injected payment instruction. But it did not report a single one either. On clean and injected emails alike, it simply read the email, logged the subject, and finished.
That is a useful distinction: resisting an attack and recognizing it as a security event are not the same capability.
A model can be perfectly resistant in this test and still leave you with no evidence that anyone attacked it.
Full experiment and results:
https://huggingface.co/blog/mihailgribov/will-the-agent-tell-you-it-was-attacked
Quadrat-IPI dataset:
https://huggingface.co/datasets/mihailgribov/quadrat-ipi
Run your own model:
https://github.com/mihail-gribov/quadrat-ipi-model-eval
#prompt-injection #indirect-prompt-injection #agentic-ai #llm-security #ai-agents updated a model about 10 hours ago
mihailgribov/typecastlm-qwen3.5-3.8bOrganizations
