Benchmarks
An AI rater is only worth what it can be measured at
Tagaroo's agent applies your coding scheme and returns a severity, a span, and a rationale for every judgment. That design is the reason it can be evaluated like a rater rather than trusted like an oracle — so here is how it is evaluated, what the numbers are, and what they do not cover.
Three arms, because no single one is enough
Each arm answers a different question and fails in a different way. Read together they converge; read alone any one of them misleads — a system can score well on synthetic instances and generalize badly, or correlate nicely with a questionnaire simply by coding liberally.
Synthetic ground truth
Does the agent apply the coding rules correctly?
Short instances written by hand with the codings known in advance, stratified by difficulty (easy through adversarial) and by instance type — single-code, multi-code, distractor-heavy, denial, and true negatives. A held-out test split is scored; a separate dev split is what the skills are iterated against.
Limit: Synthetic speech is cleaner than real speech. A good score here means the decision rules hold, not that the agent transfers to messy clinical recordings.
Construct validity
Do the aggregate scores behave like the instrument should?
Run the agent over a real corpus and correlate its totals against an established independent measure of the same construct.
Limit: Correlation with a different instrument is not clause-level accuracy, and a higher correlation can be bought with liberal coding — which is why it is never reported alone.
Blind expert annotation
Does it agree with trained humans on the same material?
Trained raters code a stratified sample blind to the system's output; agreement is computed between raters and between the system and the rater consensus.
Limit: The strongest arm and the most expensive. Sample sizes are small, and inter-rater agreement sets a ceiling — the system cannot look more consistent than the humans it is compared with.
Research behind the architecture (preprint, under review)
The method Tagaroo is built on has been evaluated on all three arms and written up. These figures come from that work — a research implementation of the same architecture, not from this product — and are reported here as evidence for the approach.
- Synthetic ground truth
- On 150 held-out synthetic instances covering all seven Gottschalk–Gleser depression subscales, the architecture reached F1 0.84 with sensitivity 0.97. When a coding was detected, downstream attribution was accurate: 96.4% correct subscale, 92.9% correct sub-item, 98.3% correct perspective.
- Construct validity
- Across 189 clinical interviews (DAIC-WOZ), aggregate depression scores correlated with PHQ-8 self-report at r = 0.49 — the range the scale's own validation tradition reports for two related-but-distinct depression measures, and well above the 0.19 positive predictive value of the previously published keyword-matching computerization.
- Blind expert annotation
- Three psychology-trained raters coded 48 stratified transcript fragments blind to the system. Their own agreement was Fleiss' κ = 0.718. Against their consensus the system scored F1 0.852 with sensitivity 0.92 — comparable to the strongest human pair in the study — and reduced the miss rate threefold versus a zero-shot baseline using the same model.
- What it did not show
- Validation ran on one English corpus of semi-structured interviews with a virtual interviewer. Generalization to other languages, to spontaneous conversation, and to human clinician–patient interaction remains to be demonstrated. Clause-level precision was lower for the subscales with the most fine-grained sub-items, where the raters themselves agreed least.
Gutiérrez E, Zhang Y, Navarro J-B, Barajas A. Translating rating-scale ontologies into LLM agents for ambient clinical phenotyping. npj Digital Medicine — submitted; under review. Software archived at doi:10.5281/zenodo.20435439; code at github.com/Kikolo3000/clinical-skill-architecture. The manuscript is under review and not yet public, so the archived software is what is citable today.
Per-scale results
Coding a behaviour-count instrument and rating a severity scale are different tasks, so there is no single accuracy figure for the agent. Each scale gets its own suite, and its own numbers.
No in-product measurements published yet. The suites below are written and validated, and the evaluation harness runs the shipped agent end to end. Results appear here once each suite has been run against the live agent — we would rather show nothing than show a number we have not measured.
MITI 4.2.1 — behaviour counts
View the scale- Held-out instances:
- 20
- Expected codings:
- 19
- Negatives:
- 5
Instances target the manual's hard boundaries — simple versus complex reflection, giving information versus persuading, persuade versus confront — plus adversarial cases: reflections phrased as questions, amplified reflections that must not read as confrontation, and client speech the instrument must ignore.
Not yet measured against the live agent.
Download the held-out suite (YAML)Frequently asked questions
How accurate is Tagaroo's annotation agent?
It depends on the instrument, and we publish per-scale numbers rather than one headline figure — a behaviour-count scale and a severity-rating scale are different tasks. In the companion manuscript (unpublished, under review) the underlying architecture reached F1 0.85 on the Gottschalk–Gleser depression scale against a three-rater expert consensus, matching the strongest human pair in that study. Measured results for the shipped product are published per scale on this page as each suite is run; where a scale says 'not yet measured', that is exactly what it means.
How accurate is AI at qualitative coding compared with human raters?
It depends on the task and the instrument — there is no single number for AI qualitative coding accuracy, and anyone quoting one is overselling. On a behaviour-count instrument (Gottschalk–Gleser, in the companion preprint, unpublished and under review) the architecture reached F1 0.85 against a three-rater expert consensus, matching the strongest human pair in that study. On a 28-item communication rubric, independently published work puts unaided frontier models at only 27–44% exact agreement with trained raters (arXiv:2501.13957). Those two findings are not in conflict: a tightly operationalised construct with worked boundary cases is a far easier target than a broad multi-item rubric applied cold — which is exactly why Tagaroo keeps a human reviewing every call rather than shipping a score.
How often will these benchmark numbers be updated?
Whenever a suite is run against the shipped agent, and whenever the prompt changes enough to invalidate the last run — there is no fixed schedule. Every result records the model and prompt version it was measured under, so a stale number is visible as stale. 'Not yet measured' means exactly that: the suite is written and validated, but no live run has been published for it yet.
Can I reproduce these benchmarks?
The held-out suites are downloadable from this page, the evaluation code lives in the repository, and every result records the model and the prompt version it was measured against. The suites are synthetic by design, so they contain no patient data and carry no access restrictions.
Why synthetic instances instead of real transcripts?
Two reasons. Clause-level ground truth on real clinical speech requires trained raters, which is the bottleneck the whole product exists to relieve; and real clinical transcripts cannot be redistributed, so nobody could check our numbers. Synthetic instances with known codings can be published in full. The trade-off is stated plainly: they test whether the coding rules hold, not whether the agent survives messy real speech — which is what the expert-annotation arm is for.
Does a good benchmark score mean I can skip reviewing the agent's output?
No, and the product does not offer that. Every annotation arrives as a proposal with its evidence span and rationale for a human to accept, edit, or reject. The benchmarks tell you how much work the first pass saves and where it tends to err; they are not a licence to stop looking.
What happens to these numbers when the prompt changes?
They are invalidated, and we treat that as a hard rule. Every result file records the prompt version it was measured under, and a change to the prompt requires a re-run before the numbers are shown again.
Measure it on your own data
The most useful benchmark is the one you run yourself: code a session by hand, have the agent code it too, and read the agreement. That workflow is in the product, not a separate exercise.
Try it free