Auditable Clinical Ratings: Score From the Transcript

A rating-scale total is only as trustworthy as the process that produced it. Two raters can hand you the same MADRS score of 24 and mean completely different things by it: one added up ten items after quoting the line in the interview that justified each rating, the other formed a global impression and reverse-engineered a number that felt about right. Only the first is an auditable clinical rating—a score whose every point can be traced back to the words that produced it. This piece is about how to turn the second kind of scoring into the first, and why regulators, trial managers, and reliability-obsessed methodologists should care.

What is an auditable clinical rating?

An auditable clinical rating is a score in which each item can be traced to the specific evidence that produced it. Put the standard in one line: a defensible MADRS score is one where every item points to a quoted line in the interview. The total is not a judgment handed down from the rater’s overall sense of the patient; it is the sum of item ratings, each anchored to a passage a second reader can inspect.

That distinction sounds pedantic until you try to reconstruct someone else’s score six months later. With a global impression, the number is all you have—the reasoning evaporated the moment the rater wrote it down. With span-grounded scoring, the number arrives with its own audit trail: item 1 was a 3 because of this sentence, item 7 was a 2 because of that one. The score becomes checkable, and a checkable score is a defensible one.

This is the difference between a measurement and an opinion. A well-run MADRS or Hamilton depression scale already defines anchors for each item; auditable scoring simply insists that you record which words in the transcript met each anchor before you commit to the rating.

Why do global clinical impressions drift?

Global impressions drift because they compress a whole interview into a single number without leaving a record of how. The Clinical Global Impressions scale is the purest example: it asks the clinician one question—“considering your total clinical experience, how mentally ill is the patient?”—and is completed in under a minute, capturing impressions that its own authors describe as transcending “mere symptom checklists” (Busner & Targum, 2007). That is efficient, and for tracking a single patient over time it is useful. As a research measurement, its provenance is thin: nothing in the score tells you what drove it.

The evidence that this matters is not subtle. In a meta-analysis of 136 studies across health and behavior, mechanical (formal, statistical) combination of data was about 10% more accurate than holistic clinical prediction, and the superiority held regardless of the judges’ experience (Grove et al., 2000). The finding that should stop a trial designer cold: clinical prediction did relatively worse exactly when the input was clinical interview data—the setting rating scales live in.

Watching how raters actually behave fills in the mechanism. When reviewers evaluated 104 audiotaped HAM-D interviews, 39% were conducted in ten minutes or less and most were rated fair or unsatisfactory for adequacy of the information obtained (Engelhardt et al., 2006). A cursory interview leaves the rater no choice but to fill the gaps with impression.

And when Kobak and colleagues transcribed 30 pairs of interviews and had independent assessors code why two raters disagreed, the disagreements sorted into five sources—information, observation, interpretation, criterion, and subject variance—with interpretation variance the most common (Kobak et al., 2009). Most rater disagreement is not about what the patient said; it is about what the rater made of it.

How do you build an auditable clinical rating from a transcript?

You build an auditable clinical rating by inverting the usual order: find the evidence first, then rate it. The workflow is four steps, and it is the same whether a human or an assisted tool does the coding.

1. Read for evidence, not for a verdict. For each item on the scale, locate the span in the subject’s speech (or the interviewer’s recorded observation) that bears on it. Highlight it. If nothing in the transcript speaks to an item, that item has no evidence—which is itself a finding.
2. Anchor the span to the item’s rating scale. Match the quoted passage to the instrument’s defined anchors. The MADRS Reported Sadness item, for instance, runs 0–6 from “occasional sadness in keeping with the circumstances” up to “continuous or unvarying sadness”; the span decides which anchor applies.
3. Rate the item, attaching the rationale. Record the item score together with the span and a one-line justification. This is the “rationale” the trial-methods literature keeps asking for—the artifact that lets a reviewer reconstruct the decision (Kobak et al., 2009).
4. Sum the items to a total. The total is now a derived quantity, not a first impression. For the MADRS it is the sum of ten items; for a self-report screen like the PHQ-9 it is the sum of nine.

The payoff is that the total and its parts are separable. A reviewer can accept nine of your ten MADRS items and challenge only the one whose span does not support its rating—a conversation you simply cannot have about a gestalt.

Property	Global impression (gestalt)	Span-grounded item scoring
What is recorded	A single number	A number + the span + a rationale per item
Can a reviewer reconstruct it?	No—reasoning is gone	Yes—each item points to a quoted line
Diagnosing rater disagreement	Hard; you argue about the total	Item-level; you compare the spans
Regulatory provenance	Weak (not attributable)	Strong (attributable, reconstructable)
Typical failure mode	Interpretation drift, halo effects	Time cost; needs a coding tool
ML/training value	Label only	Label + rationale (span)

A worked example: scoring depressed mood from the evidence

Consider a short synthetic exchange from a depression interview. The task is to score the relevant MADRS items from the evidence, not from an overall read of the patient.

Interviewer: How would you describe your mood over the past week?

Subject: Flat, mostly. It doesn’t lift even when something good happens—my daughter visited and I felt nothing. I’m awake by 4 most mornings and just lie there. I keep thinking things won’t get better, but I’d never do anything about it.

Coded span by span, four MADRS items have evidence, and each rating carries its quoted justification.

MADRS item (0–6)	Evidence span (synthetic)	Rating
Reported Sadness	"Flat, mostly. It doesn't lift even when something good happens"	4
Inability to Feel	"my daughter visited and I felt nothing"	4
Reduced Sleep	"I'm awake by 4 most mornings and just lie there"	3
Pessimistic Thoughts	"I keep thinking things won't get better"	3
Suicidal Thoughts	"I'd never do anything about it"	0

The subtotal for these five items is 14; a real rating would code the remaining five items (apparent sadness, inner tension, reduced appetite, concentration difficulties, lassitude) the same way and sum all ten. The point is not the number—it is that the number is now inspectable. Rate the items in the tool below and watch the total, severity band, and remission flag update as the evidence changes.

MADRS score simulator

Load exampleReset

Rate each item on the 0-6 MADRS anchors: 0 = symptom absent, 6 = most severe. Defined descriptions sit at 0, 2, 4, and 6; the odd values (1, 3, 5) mark intermediate states.

1. 1.Apparent sadnessObserved dejection in voice, expression, and posture
   0123456
2. 2.Reported sadnessThe subject's own account of low mood
   0123456
3. 3.Inner tensionEdginess, ill-defined discomfort, dread, or panic
   0123456
4. 4.Reduced sleepReduced duration or depth vs. the subject's normal
   0123456
5. 5.Reduced appetiteLoss of appetite or of the pleasure of eating
   0123456
6. 6.Concentration difficultiesTrouble collecting or sustaining thoughts
   0123456
7. 7.LassitudeDifficulty starting and carrying out activities
   0123456
8. 8.Inability to feelReduced interest; loss of the capacity to feel
   0123456
9. 9.Pessimistic thoughtsGuilt, self-reproach, inferiority, remorse
   0123456
10. 10.Suicidal thoughtsLife not worth living; wishes for death
    0123456

Total score

0 / 60

Severity band

Normal / symptom-free

Remission (<=10)

Yes

Item 10 flag

None

Severity bands follow Snaith et al. (1986); the remission target of a total of 10 or below follows Hawley et al. (2002). This tool illustrates how the MADRS is scored; it is an educational aid, not a diagnostic instrument, and a score is a measure of severity, not a diagnosis. Adjust the items above (or load the example) to see the score update.

Tagaroo’s curated MADRS exposes all ten of the published items (Montgomery & Åsberg, 1979), so the card above and the instrument agree one-to-one. The Hamilton scale is different: the published HDRS-17 has 17 scored items, while Tagaroo’s HAM-D card curates the 11 content-localizable items—the ones a subject can actually report in an interview—and omits several purely observational and objective-measurement items (anxiety-somatic, gastrointestinal and genital symptoms, weight loss) while collapsing the three insomnia items into one. If you report a HAM-D total, say the HDRS-17 total; the curated set is for span-level evidence coding, not for regenerating the 17-item sum.

For the full item walk-throughs, the MADRS scoring guide and the Hamilton Depression Rating Scale guide cover every anchor.

How does span-grounded scoring cut cross-site rater drift?

Span-grounded scoring cuts drift because it makes the largest source of disagreement—interpretation—visible and checkable. Measurement error is not a rounding footnote in trials; Kobak and colleagues argued it is a leading reason antidepressant trials fail outright, by manufacturing placebo response and burying the drug signal in noise (Kobak et al., 2007). If you cannot see why two sites score the same presentation differently, you cannot fix it.

The cost of that invisibility is quantifiable. In a study pitting site raters against blinded centralized raters on the same patients, 35% of patients admitted by site raters would have been ineligible under central raters, and mean placebo-arm change was 7.52 for site raters versus 3.18 for central raters (Kobak et al., 2010). A gap that size is enough to sink an otherwise real treatment effect.

Two findings show where the fix lives. First, competence is not seniority: across 1,241 raters scoring videotaped HAM-A, HAM-D, and YMRS interviews, clinical experience alone did not confer rating competency, whereas repeated training did (Targum, 2006). Second, calibration beats experience—rater agreement on the same interviews was highest for experienced and calibrated raters (ICC 0.93), and experienced-but-uncalibrated raters (0.55) actually trailed inexperienced ones (0.77) (Kobak et al., 2009).

Span-grounded scoring is calibration made durable: when every item cites its evidence, a central reviewer can adjudicate the interpretation directly instead of re-interviewing the patient. It is also where inter-rater work pays off, because you can compute Cohen’s kappa and related agreement coefficients item by item and see exactly which item, and which span, the coders read differently.

What do regulators expect from a defensible score?

Regulators already think in terms of provenance, even if they do not use the word “span.” The FDA’s data-integrity guidance defines the ALCOA standard: data supporting a regulated decision should be attributable, legible, contemporaneously recorded, original or a true copy, and accurate, with the metadata needed to reconstruct the activity (FDA Data Integrity guidance, 2018). A rating-scale total with no trail back to the interview is, in this vocabulary, neither attributable nor reconstructable.

The clinical-outcome-assessment framework points the same way. The FDA classifies the HAM-D and MADRS as clinician-reported outcomes (ClinROs)—the COA type chosen “if clinical judgment is required to interpret an observation”—and expects a clearly defined scoring rule plus evidence that scores are “not overly influenced by measurement error” (FDA COA guidance, 2025). The guidance carefully separates a COA, a COA score, and an endpoint, which means the score must be produced by a standardized, documented process, not improvised per rater.

Span-grounded item scoring is one concrete way to meet that expectation. It gives each ClinRO item a documented, attributable derivation, and it turns “the rater judged severity” into “the rater rated item 3 as a 4 on the basis of this quoted line.” That is the provenance an auditor can follow.

Where item-level evidence scoring falls short

Auditable scoring is not free, and pretending otherwise would undercut the point. Coding an evidence span for every item is slower than forming an impression, and without a purpose-built tool the rationales scatter across notebooks and margins where no reviewer will ever find them. The method also assumes the interview captured the evidence in the first place; a ten-minute interview that never probed sleep leaves the sleep item genuinely ungroundable, and no coding discipline can conjure evidence that was never elicited (Engelhardt et al., 2006).

Some constructs resist span-grounding by nature. Observational items—psychomotor retardation, apparent sadness read from affect rather than words—live in what the rater sees, not in the transcript, so a text-only span cannot fully justify them; this is exactly why the curated HAM-D set is smaller than the HDRS-17.

And auditability improves provenance, not validity: a perfectly documented total from a poorly chosen scale is still measuring the wrong thing. For that decision, matching the instrument to the question, compare a clinician-rated scale against a self-report screen before you commit. Auditable scoring makes a score defensible; it does not make a bad scale good.

From a number to a defensible clinical rating

The practical upshot: a rating-scale total is a claim, and a claim without evidence is worth less than one that carries its own. An auditable clinical rating attaches the evidence to the claim—every item pointing back to the line that produced it—so a reviewer, a central rater, or an auditor can check the work instead of trusting it. The literature says the payoff is real: structured, item-summed scoring beats gestalt on average (Grove et al., 2000), the drift it prevents is expensive (Kobak et al., 2010), and the provenance it produces is exactly what regulators ask for (FDA, 2018; 2025).

References

• Busner, J., & Targum, S. D. (2007). The Clinical Global Impressions Scale: Applying a research tool in clinical practice. Psychiatry (Edgmont), 4(7), 28–37.
• Engelhardt, N., Feiger, A. D., Cogger, K. O., et al. (2006). Rating the raters: Assessing the quality of Hamilton Rating Scale for Depression clinical interviews. doi:10.1097/01.jcp.0000194621.61868.7c
• Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000). Clinical versus mechanical prediction: A meta-analysis. doi:10.1037/1040-3590.12.1.19
• Hamilton, M. (1960). A rating scale for depression. doi:10.1136/jnnp.23.1.56
• Kobak, K. A., Kane, J. M., Thase, M. E., & Nierenberg, A. A. (2007). Why do clinical trials fail? The problem of measurement error in clinical trials. doi:10.1097/jcp.0b013e31802eb4b7
• Kobak, K. A., Brown, B., Sharp, I. R., et al. (2009). Sources of unreliability in depression ratings. doi:10.1097/jcp.0b013e318192e4d7
• Kobak, K. A., Leuchter, A. F., DeBrota, D. J., et al. (2010). Site versus centralized raters in a clinical depression trial. doi:10.1097/jcp.0b013e3181d20912
• Montgomery, S. A., & Åsberg, M. (1979). A new depression scale designed to be sensitive to change. doi:10.1192/bjp.134.4.382
• Targum, S. D. (2006). Evaluating rater competency for CNS clinical trials. doi:10.1097/01.jcp.0000219049.33008.b7
• U.S. Food and Drug Administration. (2018). Data Integrity and Compliance With Drug CGMP: Questions and Answers (Guidance for Industry).
• U.S. Food and Drug Administration. (2025). Patient-Focused Drug Development: Selecting, Developing, or Modifying Fit-for-Purpose Clinical Outcome Assessments (Guidance for Industry).

If you code depression or mania symptoms from interviews, Tagaroo turns the MADRS, the Hamilton scale, and the PHQ-9 into guided, evidence-anchored workflows where each item rating is bound to its transcript span and inter-rater reliability is computed as your coders work. Score from the transcript, not from a gestalt, and the total defends itself.