Five Raters, One Rule, Five Different Answers: What Happened When We Measured LLM Annotation Agreement

TL;DR — We handed the same hand-written annotation rule and the same 100 scenes to five raters: one deterministic rule-based detector and four large language models. Against a human reference that marked the rule present in 9 scenes, the five machines returned 0, 1, 40, 72 and 78. Cohen's κ was at or near chance for four of them. The raw agreement numbers looked fine — 74.7% to 86.3% — which is exactly the problem. All labels, the scoring script and the recovery code are public.

---

Conflict of interest. Two disclosures belong at the top, not in a footnote. First, I designed the annotation scheme whose transferability is measured here, and I was the sole human rater in the first of the three studies. The designer of a scheme is not a neutral judge of whether it is learnable. Second, this post was drafted with help from Claude, which is also one of the four LLM raters being scored. A model family cannot be a disinterested narrator of its own score. The scoring itself is deterministic and reproducible from the published files, so this conflict affects framing, not arithmetic.

Why we ran this

Using an LLM as an annotator is now routine. The assumption underneath it is rarely stated: that a rule which is clear to the person who wrote it will be clear to the model that receives it. That assumption is testable, and it seemed worth testing on a task where the rule is genuinely interpretive rather than a keyword match.

The corpus is Objective Projection, a bilingual Turkish–English set of purpose-written narrative scenes. It is annotated against six craft features. Two are prohibitions that reduce to surface patterns — no explicitly named emotion, no simile. Four are positive techniques that require judgement. The most demanding is materialized metaphor: an abstract inner state rendered as a concrete physical detail instead of being named.

That split between surface rules and judgement rules turned out to be the whole story.

The design

Three studies, summarised:

Study 1 (n=120). The rule-based detector against a blind human rater on 120 Turkish scenes. The human rater was me — the scheme's author. Declared, not hidden; it is a limitation, and the second study exists because of it.

Studies 2 and 2b (n=100). A disjoint set of 100 Turkish scenes, labelled by an independent volunteer with no role in the methodology. Those labels were locked to an immutable file before any machine was run. Then five machine raters were scored against them: the detector, plus four LLMs given identical instruction blocks.

What came back

On materialized metaphor, with the human reference at 9 positives out of 100:

The spread is the finding. This is not a leaderboard with a winner. Five systems given the same rule and the same text returned positive rates covering essentially the whole available range. Whatever each of them was doing, they were not doing the same thing.

The one coefficient that looks different, κ = 0.185 for Machine B, rests on a single positive judgement: it said present once and happened to be right. That is not a shared criterion either.

The same pattern appeared against a second, independent human reference. In Study 1, on a disjoint scene set with a different human rater, the detector's agreement on this rule was κ = 0.004 with raw agreement of 51.7% — the lowest raw figure anywhere in the three studies.

The part that should worry anyone using LLM annotation

Raw agreement across the five machine raters ranged from 74.7% to 86.3%. If we had reported only that, this would read as a success story.

It isn't, and the reason is the first kappa paradox (Feinstein & Cicchetti 1990; Byrt, Bishop & Carlin 1993). Across the 100 held-out scenes the human positive counts were 0, 1, 9, 96, 99 and 44. On four of six features, one class covers almost everything, so a rater that always answers with the majority class scores high raw agreement while carrying no information at all.

Practical consequence: you cannot measure an annotation scheme's reliability on the corpus the scheme was applied to. The evaluation subset has to be sampled against the feature's distribution, not against the detector's output — and sampling it by the detector's own labels makes the measurement circular.

The fair objection, and what it costs

A reasonable reader objects: those coefficients are uninformative because the classes are degenerate, so the finding may be an artifact.

Correct for most of the scheme. Exactly one feature, atmosphere contradiction, sits at 44/56 — a distribution that supports an informative coefficient. Here is that feature:

Two raters are genuinely above chance here, and that should be recorded as such: on a well-conditioned feature, transfer is partial rather than absent. But 0.269 is the best coefficient anywhere in the three studies, on the best-conditioned feature in the set. By any conventional verbal scale that is weak.

Two readings, and we decline to pick one

The result is compatible with two explanations, and our data cannot separate them.

(a) The feature is inherently interpretive. Whether a physical detail is doing representational work may have no single right answer, in which case disagreement is a property of the task rather than a failure of any rater (Plank 2022).

(b) The rule was not operationalised well enough. The definition may have been clear to its author and underdetermined for everyone else.

We would like reading (a) to be true; it is the more flattering one for a framework whose central construct is the feature in question. We are not entitled to it. One measurement decides between them, and it is cheap: a second independent human annotator labelling the same 100 held-out scenes. If two humans agree with each other and diverge from the machines, the specification is at fault and can be repaired. If two humans cannot agree either, the rule is interpretive in a way no specification will fix.

That measurement has not been made. If you would like to make it, the scenes and the locked reference labels are in the repository and we will publish the result whichever way it falls.

What is actually in the files

The evaluation/ directory of the dataset contains everything needed to reproduce every number above without re-running a single model:

• blind human reference labels for both studies
• per-scene machine label files where they survive
• the ten verbatim instruction blocks given to each rater
• masked scene-ID mapping plus sampling and shuffle seeds
• a dependency-free scoring script

One honesty note about provenance. Per-scene labels for two of the four LLM raters were lost. Their confusion counts were recovered arithmetically from the surviving per-rule κ and raw agreement values, a procedure that gave a unique solution for eleven of twelve cells. Those counts are derived, not raw, and are marked as such in the paper and in the repository. The recovery also corrected an earlier internal record that gave Machine B's count as approximately two; the correct figure is one.

Links

• Official:leventbulut.com
• Paper: Inter-Rater Reliability of LLM and Rule-Based Annotation for Inferential Narrative Features · also arXiv:2609.13936 [cs.CL]
• Dataset: leventbulut/objective-projection · DOI 10.57967/hf/8960
• Corpus resource paper: 10.5281/zenodo.22841625

Licence: corpus CC BY-NC-ND 4.0. Criticism of the scheme is more useful to me than agreement with it; the comments are open.