Benchmarks
An AI rater is only worth what it can be measured at
Tagaroo's agent applies your coding scheme and returns a severity, a span, and a rationale for every judgment. That design is the reason it can be evaluated like a rater rather than trusted like an oracle—so here is how it is evaluated, what the numbers are, and what they do not cover.
Three arms, because no single one is enough
Each arm answers a different question and fails in a different way. Read together they converge; read alone any one of them misleads—a system can score well on synthetic instances and generalize badly, or correlate nicely with a questionnaire simply by coding liberally.
Synthetic ground truth
Does the agent apply the coding rules correctly?
Short instances with the codings decided in advance, stratified by difficulty (easy through adversarial) and by instance type—single-code, multi-code, distractor-heavy, denial, and true negatives. Some suites are human-authored and some are model-authored and automatically validated; each one says which, because it changes what a score there is worth. A held-out test split is scored; a separate dev split is what the skills are iterated against.
Limit: Synthetic speech is cleaner than real speech. A good score here means the decision rules hold, not that the agent transfers to messy clinical recordings.
Construct validity
Do the aggregate scores behave like the instrument should?
Run the agent over a real corpus and correlate its totals against an established independent measure of the same construct.
Limit: Correlation with a different instrument is not clause-level accuracy, and a higher correlation can be bought with liberal coding—which is why it is never reported alone.
Blind expert annotation
Does it agree with trained humans on the same material?
Trained raters code a stratified sample blind to the system's output; agreement is computed between raters and between the system and the rater consensus.
Limit: The strongest arm and the most expensive. Sample sizes are small, and inter-rater agreement sets a ceiling—the system cannot look more consistent than the humans it is compared with.
Research behind the architecture (preprint, under review)
The method Tagaroo is built on has been evaluated on all three arms and written up. These figures come from that work—a research implementation of the same architecture, not from this product—and are reported here as evidence for the approach.
- Synthetic ground truth
- On 150 held-out synthetic instances covering all seven Gottschalk–Gleser depression subscales, the architecture reached F1 0.84 with sensitivity 0.97. When a coding was detected, downstream attribution was accurate: 96.4% correct subscale, 92.9% correct sub-item, 98.3% correct perspective.
- Construct validity
- Across 189 clinical interviews (DAIC-WOZ), aggregate depression scores correlated with PHQ-8 self-report at r = 0.49—the range the scale's own validation tradition reports for two related-but-distinct depression measures, and well above the 0.19 positive predictive value of the previously published keyword-matching computerization.
- Blind expert annotation
- Three psychology-trained raters coded 48 stratified transcript fragments blind to the system. Their own agreement was Fleiss' κ = 0.718. Against their consensus the system scored F1 0.852 with sensitivity 0.92—comparable to the strongest human pair in the study—and reduced the miss rate threefold versus a zero-shot baseline using the same model.
- What it did not show
- Validation ran on one English corpus of semi-structured interviews with a virtual interviewer. Generalization to other languages, to spontaneous conversation, and to human clinician–patient interaction remains to be demonstrated. Clause-level precision was lower for the subscales with the most fine-grained sub-items, where the raters themselves agreed least.
Gutiérrez E, Zhang Y, Navarro J-B, Barajas A. Translating rating-scale ontologies into LLM agents for ambient clinical phenotyping. npj Digital Medicine—submitted; under review. Software archived at doi:10.5281/zenodo.20435439; code at github.com/Kikolo3000/clinical-skill-architecture. The manuscript is under review and not yet public, so the archived software is what is citable today.
Per-scale results
Coding a behaviour-count instrument and rating a severity scale are different tasks, so there is no single accuracy figure for the agent. Each scale gets its own suite, and its own numbers.
Gottschalk–Gleser Depression Scale—seven subscales
View the scale- Held-out instances:
- 150
- Expected codings:
- 345
- Negatives:
- 32
Labels: human-authored; conversion machine-checked Written for the companion manuscript by its authors and converted into this format by script. Every quoted span is verified to occur verbatim and exactly once in its turn, every code exists in the scale, and every severity is a declared level of it. The conversion itself has not been re-read by a human.
The held-out suite from the companion manuscript, converted to this format unchanged: 150 instances across all seven subscales, stratified by instance type—single-code, multi-subscale, multi-code, distractor-heavy, denial, true negative—and by difficulty. Severity here is the scale's perspective weight, so that column measures whose loss or hostility is being described rather than how intense it is, which is why it scores far higher than the severity column on a clarity-rated instrument like MITI. Transcripts are lower-cased with minimal punctuation, matching the DAIC-WOZ corpus the research pipeline was built for. Two of the manuscript's 345 ground-truth clauses did not occur verbatim in their own transcript; both were unambiguous paraphrases and the suite file records the correction. Scores fall monotonically with difficulty—0.87 easy, 0.84 moderate, 0.75 hard, 0.71 adversarial—which is the gradient a well-stratified suite should produce. The weakest subscales are psychomotor retardation (F1 0.67, and the lowest recall of the seven) and outward hostility (0.69), whose most common error is coding hostility as death-and-mutilation content: violence tends to contain both.
Same 150 instances, research pipeline vs shipped agent
| Metric | Manuscript pipeline | Shipped agent |
|---|---|---|
| Detection F1 | 0.840 | 0.790 |
| Sensitivity | 0.971 | 0.800 |
| Precision | 0.740 | 0.780 |
The shipped agent is a little behind overall and differently shaped: more precise, considerably less sensitive. That is the architecture showing through. The manuscript's pipeline screens every utterance and then codes what it flags, so it catches almost everything and pays for it in false positives; the product makes one pass per turn and is correspondingly conservative. Two-stage screening is the obvious thing to try next, and this is the number it would have to beat.
| Detection F1 | 0.790 |
|---|---|
| Sensitivity | 0.800 (95% CI 0.75–0.84) |
| Precision | 0.780 |
| Detection κ | 0.073 (only 29 true negatives to correct against) |
| Severity agreement on matched codings (weighted κ) | 0.874 (n = 276 of 345) |
| Negatives left clean | 91% |
| Run-to-run agreement (default sampling) | single run |
Not re-hosted here on purpose. These labels belong to a manuscript under review, and a copy served from this domain would be crawled into training corpora—which would spoil the suite for us and for anyone replicating the paper. The instances are in the manuscript's own repository.
MITI 4.2.1—behaviour counts
View the scale- Held-out instances:
- 20
- Expected codings:
- 19
- Negatives:
- 5
Labels: model-authored, automatically validated. Human review pending. The ground truth here was written by claude-opus-5 and checked by the harness. Every quoted span is verified to occur verbatim and exactly once in its turn, every code exists in the scale, and every severity is a declared level of it. No human has read these labels yet, so treat a score on this suite as evidence that the agent agrees with another model's reading of the coding manual—a weaker claim than agreement with a trained rater.
Instances target the manual's hard boundaries—simple versus complex reflection, giving information versus persuading, persuade versus confront—plus adversarial cases: reflections phrased as questions, amplified reflections that must not read as confrontation, and client speech the instrument must ignore.
The severity figure here is overstated, and cannot be corrected without re-running the suite. On 2026-09-09 we found a bug in our own scorer: the weighted-kappa routine accumulated expected agreement only over the rating combinations that actually occurred, leaving out every pair of levels the two raters never produced together. Because the omitted terms are all non-negative, expected agreement came out too small, and the coefficient was therefore always too high—never too low. We proved that direction and confirmed it over 20,000 random rating matrices: 16,975 came out lower under the correction, 3,019 identical, none higher. The Gottschalk–Gleser suite above has been re-scored from its cached answers with the corrected routine and its severity agreement fell from 0.9424 to 0.8744; the PHQ-9 suite is unaffected because its agreement was exactly perfect, which no change to chance agreement can move. This suite could not be re-scored: its cached answers no longer match the current prompt hash, so re-deriving its numbers means paying for the model calls again rather than reading them off disk. The severity figure below (0.129) is therefore still the old, overstated one, and the true value is lower. Detection, sensitivity, precision and F1 are unaffected—the bug was confined to the severity path. We are leaving the number visible with this notice rather than deleting it, because a figure we have told you is wrong in a known direction is more useful than a silent gap.
| Detection F1 | 0.818 |
|---|---|
| Sensitivity | 0.947 (95% CI 0.75–0.99) |
| Precision | 0.720 |
| Detection κ | 0.288 (only 3 true negatives to correct against) |
| Severity agreement on matched codings (weighted κ) | 0.129 (n = 18 of 19) |
| Negatives left clean | 60% |
| Run-to-run agreement (default sampling) | 0.863, 0.880 |
PHQ-9—nine reported symptom items
View the scale- Held-out instances:
- 38
- Expected ratings:
- 43
- Negatives:
- 6
Labels: model-authored, automatically validated. Human review pending. The ground truth here was written by claude-opus-5 and checked by the harness. Every quoted span is verified to occur verbatim and exactly once in its turn, every code exists in the scale, and every severity is a declared level of it. No human has read these labels yet, so treat a score on this suite as evidence that the agent agrees with another model's reading of the coding manual—a weaker claim than agreement with a trained rater.
An evidence-mode suite: the unit is one rating per item for the whole sample, on the PHQ-9 frequency anchors, with quoted spans as supporting evidence rather than the thing being counted. A rating of 'not at all' is the absence of a rating here, not a rating of absence, so items the sample does not report are probed through negative and denial instances rather than as expected zeros. Instances target the boundaries the instrument leaves closest together—loss of interest against low mood, fatigue against sleep, fatigue against psychomotor change—plus the three things it does not score at all: symptoms voiced only by the interviewer, symptoms placed outside the two-week window, and symptoms belonging to somebody else. It was written by Claude Opus and evaluated with Sonnet, so no suite here is graded by the model that wrote it.
A ceiling result: no errors, and the suite was not hard enough to find the limits. The agent rated all 43 items correctly and at the right anchor, on both versions of this suite. Read precisely, that says it makes no errors on samples constructed to be unambiguous—which is a real result and rules out gross failure on clear cases, but is a narrower claim than reliability, because these instances were built to be cases any competent coder would agree on rather than drawn from real clinical speech. What the suite cannot tell us is where the agent's judgement breaks down, because nothing here is close enough to the boundary to find out. We tried to make it harder: the first version let the subject state their own frequency (“more than half the days”), which hands over the very judgement a PHQ-9 anchor exists to test, so we forbade that, enforced it as a mechanical check rather than an instruction, and regenerated until no non-easy span counted days at all—the anchor now has to be inferred from what the subject describes (“I've started leaving a book on the bedside”). The score did not move, which tells us the difficulty was never mainly about phrasing. One further limit worth stating: the instances were written by one model and graded by another reading the same coding manual, so a shared misreading of an anchor would show up as perfect agreement rather than as a disagreement. That is why the strongest evidence on this page comes from the human-authored suite. The figure here still carrying information is evidence localisation at 88%—asked which words support a rating, the agent picks the author's span 38 times out of 43. Getting a discriminating number for this instrument needs human-authored instances or real transcripts, not another round of prompt-tuning.
| Detection F1 | 1.000 |
|---|---|
| Sensitivity | 1.000 (95% CI 0.92–1.00) |
| Precision | 1.000 |
| Detection κ | 1.000 (only 6 true negatives to correct against) |
| Severity agreement on matched codings (weighted κ) | 1.000 (n = 43 of 43) |
| Negatives left clean | 100% |
| Cited the author's supporting span | 88% of 43 correct ratings |
| Run-to-run agreement (default sampling) | single run |
Frequently asked questions
Who wrote the ground truth for these suites?
It varies by suite, and each card says so. The Gottschalk–Gleser suite is the companion manuscript's own held-out set, written by its authors and converted into this format by script. The MITI suite was written by Claude Opus against the MITI 4.2.1 manual and validated automatically—every quoted span verified to occur verbatim and exactly once in its turn, every code checked against the scale, every severity checked against its declared levels—but no human has read those labels yet, and the card says so until one has. We deliberately evaluate with a different model from the one that wrote a suite, so no suite is graded by its own author. A model-authored answer key checked only by machine is a weaker standard than a trained rater, and we would rather label it plainly than let a number imply otherwise.
How accurate is Tagaroo's annotation agent?
It depends on the instrument, and we publish per-scale numbers rather than one headline figure—a behaviour-count scale and a severity-rating scale are different tasks. In the companion manuscript (unpublished, under review) the underlying architecture reached F1 0.85 on the Gottschalk–Gleser depression scale against a three-rater expert consensus, matching the strongest human pair in that study. Measured results for the shipped product are published per scale on this page as each suite is run; where a scale says 'not yet measured', that is exactly what it means.
How accurate is AI at qualitative coding compared with human raters?
It depends on the task and the instrument—there is no single number for AI qualitative coding accuracy, and anyone quoting one is overselling. On a behaviour-count instrument (Gottschalk–Gleser, in the companion preprint, unpublished and under review) the architecture reached F1 0.85 against a three-rater expert consensus, matching the strongest human pair in that study. On a 28-item communication rubric, independently published work puts unaided frontier models at only 27–44% exact agreement with trained raters (arXiv:2501.13957). Those two findings are not in conflict: a tightly operationalised construct with worked boundary cases is a far easier target than a broad multi-item rubric applied cold—which is exactly why Tagaroo keeps a human reviewing every call rather than shipping a score.
Why doesn't Joey show a confidence score for each annotation?
Because we have not earned the right to show one. Language models will happily emit a number between 0 and 1 next to any judgement, and on some modalities ours does, but that number has never been checked against whether the judgement was actually right—it is a self-report, not a calibration. Printing it beside a rationale would be worse than useless: a well-written rationale already makes a rating feel more certain than it is, and a plausible-looking 0.88 on top of it turns a guess into an apparent measurement. In clinical coding that is not a UI nicety, it is a measurement error that survives into a published finding. So the agent gives you the evidence span, the severity, and its reasoning, and the scale's own anchor text sits next to them so you can check the call against the definition rather than against the model's self-assessment. If a confidence number ever appears here it will be measured—per phenomenon and per severity level, against the held-out suites on this page—and it will be labelled as benchmark precision, not as the model's opinion of itself.
How often will these benchmark numbers be updated?
Whenever a suite is run against the shipped agent, and whenever the prompt changes enough to invalidate the last run—there is no fixed schedule. Every result records the model and prompt version it was measured under, so a stale number is visible as stale. 'Not yet measured' means exactly that: the suite is written and validated, but no live run has been published for it yet.
Can I reproduce these benchmarks?
The held-out suites are downloadable from this page, the evaluation code lives in the repository, and every result records the model and the prompt version it was measured against. The suites are synthetic by design, so they contain no patient data and carry no access restrictions.
Why synthetic instances instead of real transcripts?
Two reasons. Clause-level ground truth on real clinical speech requires trained raters, which is the bottleneck the whole product exists to relieve; and real clinical transcripts cannot be redistributed, so nobody could check our numbers. Synthetic instances with known codings can be published in full. The trade-off is stated plainly: they test whether the coding rules hold, not whether the agent survives messy real speech—which is what the expert-annotation arm is for.
Does a good benchmark score mean I can skip reviewing the agent's output?
No, and the product does not offer that. Every annotation arrives as a proposal with its evidence span and rationale for a human to accept, edit, or reject. The benchmarks tell you how much work the first pass saves and where it tends to err; they are not a licence to stop looking.
What happens to these numbers when the prompt changes?
They are invalidated, and we treat that as a hard rule. Every result file records the prompt version it was measured under, and a change to the prompt requires a re-run before the numbers are shown again.
Measure it on your own data
The most useful benchmark is the one you run yourself: code a session by hand, have the agent code it too, and read the agreement. That workflow is in the product, not a separate exercise.
Try it free