Dipankar, thank you so much for taking the time to dig into the raw data! You went well beyond reading the article: you independently verified the data, carefully selected a subset that avoids alignment ambiguities, and discovered something I had overlooked. This is exactly why I made the data public. It's exciting to see that independent analysis is already producing unexpected findings.
Two observations particularly struck me: the 252 answers with perfect trial records that still modified the original text, and the 745 apparent misfires versus just 35 misses with a single rule. The latter is especially intriguing: misfires are clearly not limited to long rule lists. How they depend on the other load dimensions, and whether they reflect a distinct failure mechanism, is an open question.
You've identified an important limitation of the current passport. It captures only one side of model behavior: whether an instruction is followed where it should be. It tells us nothing about what happens where no changes are supposed to occur. And your results show just how significant that missing side may be.
I'm now particularly interested in whether these misfires follow their own patterns, how they depend on different load dimensions, and whether I can characterize them quantitatively. I find your suggestion of adding a precision dimension alongside recall very promising. This may turn out to be another distinct dimension of model behavior.
I plan to investigate this across the full dataset, including answers with broken line structures, which will require additional care with alignment. I'll certainly share what I find, and I'd be very interested in your thoughts.
Thank you again. Your analysis has done more than point out a limitation of the current passport — it has opened up a research direction I hadn't originally considered.