Enrique Gutiérrez, PhD

Founder, Tagaroo

Enrique Gutiérrez is the founder of Tagaroo and an Associate Professor of Computer Science (PhD), working at the intersection of algorithm design, applied mathematics, and machine learning. He builds Tagaroo, an AI-guided annotation workspace for clinical transcripts, images, and any structured labeling task, and writes about inter-rater reliability, rating scales, and the practical mechanics of running annotation projects.

LinkedIn ORCID Google Scholar

Writes about: Inter-rater reliability · Clinical rating scales · Transcript annotation · Annotation project methodology · Machine learning · Algorithm design

How articles here are researched, checked, and corrected is documented in theeditorial policy.

Articles

97 articles

Two rising speech-shapes on a horizontal sequence line — one faint and dotted, one solid — each meeting an aperture that either widens or narrows, illustrating VR-CoDES cues versus concerns and providing versus reducing space.

Methods & Frameworks

VR-CoDES Coding System: Cues, Concerns, and Responses

How the VR-CoDES coding system codes patient emotional cues versus concerns and the provider response, using the providing-space vs reducing-space axis.

Jul 22, 2026Read

A left-to-right row of annotation tick-marks that begins crisp and evenly spaced and gradually flattens into a sagging waveform, with two gaps where the marks recover and one coral mark flagging the decline — an abstract depiction of the vigilance decrement over a labeling session.

Annotation Operations & Ethics

Annotation Fatigue: Why Label Quality Drops Over Time

Annotation fatigue erodes label quality within a session. See the vigilance-decrement evidence and the breaks, caps, and catch-trials that hold it steady.

Jul 22, 2026Read

Abstract illustration of a long row of identical tick-marks that crowd together and fade, then recover their even spacing after a calm gap marked by a single coral spark, evoking repetitive annotation work and a restorative break.

Annotation Operations & Ethics

Annotator Burnout: Causes, Warning Signs, and Prevention

Annotator burnout has clear causes and warning signs. Learn the drivers, symptoms to watch for, and the workload and rotation practices that prevent it.

Jul 22, 2026Read

Two horizontal confidence-interval ranges centered on the same point estimate—a wide uncertain interval above and a narrow confident one below—over abstract ruler tick-marks.

Reliability & Agreement

Confidence Interval for Kappa, Alpha, and ICC, Done Right

Why a bare agreement number misleads, and how to build a confidence interval for kappa, alpha, and the ICC. See analytic vs bootstrap methods.

Jul 22, 2026Read

An editorial illustration contrasting two annotation modes: on the right a speech waveform feeds a single calm gauge with a few underlined supporting bands (evidence mode), while above, the same waveform is chopped into many separately bracketed, individually tagged segments (instance mode), one accented in coral.

Annotation Operations & Ethics

Evidence Mode vs Instance Mode: Evidence-Based Annotation

Evidence mode rates the whole transcript once and cites its supporting spans; instance mode tags each occurrence. How the choice reshapes reliability math.

Jul 22, 2026Read

Five equal upright columns on a shared baseline, each marked with an abstract checklist tick and joined by fine ledger lines, one tick highlighted in coral — the five Fairwork principles rendered as an auditable standard.

Annotation Operations & Ethics

Ethical Data Annotation: A Practical Fair-Work Standard

Ethical data annotation means fair pay, conditions, contracts, management, and representation. Audit any vendor against the Fairwork standard.

Jul 22, 2026Read

Abstract editorial illustration of a grid of blurred, grayscale content cards behind a translucent protective shield, with one card gently revealed to show neutral tick-marks, and a single coral reveal control.

Annotation Operations & Ethics

Sensitive Content Annotation: A Humane, Practical Protocol

Concrete protections for sensitive content annotation: informed consent, exposure limits, grayscale and blur defaults, and debriefing. Get the protocol.

Jul 22, 2026Read

Methods & Frameworks

Localize a Rating Scale: Annotating in a Second Language

Localize a rating scale for coding in a second language: what back-translation validates, what it misses, and how to re-anchor the coding examples.

Jul 22, 2026Read

Three transparent annotation layers hovering in parallel over one shared speech waveform, their bracket and tick marks aligning over the same segment, one coral mark where all three overlap.

Annotation Operations & Ethics

Multi-Layer Annotation: Stacking Scales on One Transcript

Multi-layer annotation stacks several coding schemes and rating scales on one transcript. Keep layers independent, handle overlaps, score each separately.

Jul 22, 2026Read

Two columns of abstract tick-marks linked by thin green lines showing matching coder labels, a teal agreement band wrapped by a coral confidence-interval bracket, above a faint dashed chance-floor line.

Reliability & Agreement

Percent Agreement, Reconsidered: When It's Actually Fine

Raw percent agreement isn't useless. When to chance-correct, when kappa misleads, and when agreement plus a confidence interval is the honest report.

Jul 22, 2026Read

Annotation Operations & Ethics

Rating Scale vs Coding Scheme: Which Does Your Study Need?

Rating scale vs coding scheme: how dimensional severity and categorical coding change your unit of analysis, reliability statistic, and annotation UI.

Jul 22, 2026Read

Several rows of annotator tick-marks converge through a funnel into one consolidated mark, with one faded row down-weighted and a single coral mark branching off to a separate node representing expert adjudication.

Annotation Operations & Ethics

Annotation Consensus Methods: Majority Vote to Dawid-Skene

How to turn many annotator labels into one gold label: majority vote, Dawid-Skene, MACE, STAPLE, and expert adjudication, with a table for when each fits.

Jul 21, 2026Read

A speech waveform on a warm off-white field, segmented by two annotators into bracketed spans in green and teal; where their brackets overlap but do not align, the shared stretch glows coral, and a small tick-scale measures the gap between the two boundary marks.

Reliability & Agreement

Span-Level Agreement: Why Kappa Fails, Use F1 and IoU

Cohen's kappa needs a fixed item set that span and NER annotation never has. Measure span-level agreement with pairwise F1 and IoU instead—see how.

Jul 21, 2026Read

A slender balance beam tipping toward one of two blank answer cards, beside a vertical rubric ladder of anchor tick-marks, with two rating tracks that run together and diverge at one coral-marked point.

Tools & Comparisons

RLHF Data Annotation: A Playbook for Preference & Eval

An RLHF data annotation playbook: pairwise preference vs Likert rubrics, reward-model data, eval sets, and rater agreement. See the decision guide.

Jul 21, 2026Read

Four abstract document shapes—a chat bubble, a contract page, a survey form, and a speech bubble—each showing the same bracketed, highlighted text span, illustrating one annotation method applied across domains.

Image & Multimodal Annotation

Text Annotation Use Cases: Support, Legal & Survey Text

How teams apply text annotation use cases beyond the clinic: support-ticket intent, legal clause tagging, and survey coding, with span evidence and IRR.

Jul 21, 2026Read

A single labeled tile multiplies across a grid, each column stacking into taller bars, while a small branching path splits into parallel coder lanes that converge at a marked total on the right.

Methods & Frameworks

Annotation Cost Calculator: Budget and Timeline Math

An annotation cost calculator: turn items, labels per item, time, wage, redundancy, and QA overhead into a labeling budget and timeline. See the math.

Jul 21, 2026Read

Four nested brackets around a single tagged data span, each bracket a different color, showing consent, privacy law, licensing, and IP as layers that stack on one annotation.

Annotation Operations & Ethics

Data Annotation Consent & Licensing: HIPAA, GDPR & IP

Data annotation consent explained: HIPAA de-identification vs a BAA, GDPR special-category data, who owns the labels, and how to license a dataset.

Jul 21, 2026Read

Several overlapping translucent contour outlines of the same abstract shape form a band of expert disagreement, with one coral outline nested inside the band, illustrating a segmentation device that agrees with the panel as closely as experts agree with each other.

Image & Multimodal Annotation

Ground Truth for Medical AI: Reference Standards & the FDA

How ground truth for medical AI is built for FDA review: reference standards, expert panels, MRMC reader studies, and the SegAgree agreement idea.

Jul 21, 2026Read

Two diverging colored coding paths over a segmented speech waveform converge onto a single labeled span once a small dated decision-log card is placed between them.

Methods & Frameworks

How to Write Annotation Guidelines That Reduce Disagreement

Write annotation guidelines that reduce annotator disagreement: operational definitions, positive and negative examples, and an edge-case decision log.

Jul 21, 2026Read

A single speech waveform resolving into four stacked, time-aligned annotation tracks—transcription ticks, two-color speaker-diarization segments, bracketed event spans, and a prosody contour—showing audio becoming layered structured labels.

Image & Multimodal Annotation

Audio Annotation: From Transcription to Labeled Events

Audio annotation turns raw speech into labeled data: transcription, speaker diarization, timestamped events, and prosody tags. Compare the types.

Jul 21, 2026Read

Abstract editorial illustration of several highlighted transcript spans on the left, each linked by a thin ink line to a segment of a vertical stacked bar on the right, showing a rating-scale total built from tagged evidence rather than a fading gestalt impression.

Rating Scales

Auditable Clinical Ratings: Score From the Transcript

Build an auditable clinical rating by tying every scale item to a quoted line, not a global impression. See how span-grounded scoring works.

Jul 21, 2026Read

Two vertical tick-mark columns compared side by side: a shorter eighteen-mark column (the BPRS) whose marks reappear inside a taller thirty-mark column grouped into three clusters (the PANSS), illustrating how the PANSS subsumes and extends the BPRS.

Rating Scales

BPRS vs PANSS: The Two Standard Psychosis Rating Scales

BPRS vs PANSS compared: item overlap, the positive-negative-general structure, trial usage, and score linking. See which psychosis scale to use.

Jul 21, 2026Read

A single block of abstract statement-marks viewed through two side-by-side analytic lenses, one casting a long checklist of ticks and the other a short checklist whose single coral mark points the opposite way.

Annotation Operations & Ethics

CBCA vs Reality Monitoring: Two Credibility Methods

CBCA vs reality monitoring: how two statement-credibility methods compare on criteria, accuracy, and court limits—and why neither one detects lies.

Jul 21, 2026Read

Flat-vector illustration of an abstract dark radiographic panel with three translucent finding-labels and one bounding box drawn over it, beside a legend of three state-markers—filled, hollow, and half-filled—for positive, negative, and uncertain labels, the half-filled uncertain marker picked out in coral.

Image & Multimodal Annotation

Chest X-ray Annotation: A Worked Example for AI Datasets

A finding-by-finding chest X-ray annotation example: build a CheXpert-style schema, handle uncertainty labels, and pick image-level vs region labels.

Jul 21, 2026Read

One column of rating nodes read by two slightly offset rulers and a fan of dial gauges whose needles point to different values, illustrating how different ICC forms give different reliability numbers from the same data.

Reliability & Agreement

Which ICC to Use: Choosing Model, Type, and Definition

Which ICC to use for continuous ratings: choose the model (1, 2, or 3), single vs average measures, and absolute agreement vs consistency, from one table.

Jul 21, 2026Read

A single speech waveform read three ways—a plain self-dial, a finely-notched clinician's caliper, and a bracketed tagged span—illustrating self-report, clinician-rated, and transcript-based measurement of the same interview.

Methods & Frameworks

Clinician-Rated vs Self-Report Scales: A Methods Trade-off

Clinician-rated vs self-report scales compared: who rates, the bias each carries, reliability and cost trade-offs, and transcript coding as a third path.

Jul 21, 2026Read

A column of tagged statement text-spans linked to a checklist of abstract criteria marks, some ticked and some empty, with one criterion highlighted in coral.

Annotation Operations & Ethics

Criteria-Based Content Analysis: The 19 CBCA Criteria

Criteria-based content analysis (CBCA) rates 19 content criteria to judge whether a statement reads as experience-based—an aid, not a lie test. See all 19.

Jul 21, 2026Read

Abstract illustration of one off-color tile at the top spawning a widening downward cascade of green and teal label-tiles, depicting a single upstream data error compounding downstream.

Annotation Operations & Ethics

Data Cascades: Why Fixing Labels Beats Tuning Models

Data cascades are compounding downstream failures from upstream data problems. 92% of AI teams hit them—here's why fixing labels beats tuning the model.

Jul 21, 2026Read

Rating Scales

Depression Rating Scales Compared: HAM-D, MADRS, PHQ-9

Depression rating scales compared: HAM-D, MADRS, PHQ-9, and content analysis by rater, item count, and change-sensitivity. See which fits your study.

Jul 21, 2026Read

Two overlapping abstract blobs in green and teal whose shared intersection is highlighted in coral, with a faint dashed line measuring the distance between their boundaries.

Image & Multimodal Annotation

Dice vs IoU vs Hausdorff: Segmentation Metrics Explained

Dice vs IoU, the Jaccard index, and Hausdorff distance compared: formulas, the Dice-IoU identity, and when overlap metrics mislead. See which to report.

Jul 21, 2026Read

A flat-vector medical scan passing through a cleaning gate that strips a row of header tags and blacks out burned-in corner marks before the image enters an annotation pipeline, with a single coral marker on the cleaned frame.

Image & Multimodal Annotation

DICOM Annotation: De-Identify and Prepare Data for AI

A DICOM annotation and data-prep guide: strip header and burned-in pixel PHI, window images for display, and store labels as DICOM-SEG, SR, or JSON.

Jul 21, 2026Read

Abstract tick-marks falling onto a horizontal measurement axis, forming one tight green consensus cluster, a separate teal cluster at a boundary line, and one lone coral outlier, showing disagreement as a distribution rather than a single point.

Annotation Operations & Ethics

Annotator Disagreement Is Signal, Not Noise to Discard

Forcing one gold label can throw away real information. See why annotator disagreement is signal, not noise, and when to reconcile versus preserve it.

Jul 21, 2026Read

A single document page separating into stacked translucent layers — faint text tick-marks at the base, green region outlines, teal bracketed text spans, and a small grid of cells at the top joined by connecting lines, with one coral link highlighted — illustrating the OCR-to-structure document annotation stack.

Image & Multimodal Annotation

Document Annotation: Layout, Entities & Key-Value Fields

How document annotation works for ML: layout regions, document entities, key-value pairs, and table structure, plus the datasets. See the tasks.

Jul 21, 2026Read

Two connected dials fed by a stack of coins: the quantity dial swings high while the quality dial stays flat at center, illustrating that higher pay raises output but not accuracy.

Annotation Operations & Ethics

Does Pay Improve Annotation Quality? What Research Shows

Does pay improve annotation quality? Higher pay lifts speed, participation, and fairness—but rarely accuracy. See what really drives label quality.

Jul 21, 2026Read

One large diagnostic-label shape unfolding into many small bracketed, color-tagged spans over an abstract transcript band, with a single coral-highlighted tag — showing a diagnosis resolved into observable annotatable phenomena.

Methods & Frameworks

DSM Criteria Annotation: From Diagnosis to Taggable Spans

Turn DSM-5 and ICD-11 criteria into observable, taggable phenomena: a worked MDD-to-PHQ-9 mapping and why you annotate symptoms, not a diagnosis.

Jul 21, 2026Read

A single precise expert tick-mark resolves a waveform on the right, while a converging cluster of many small crowd tick-marks on the left agrees on a simple signal but scatters, highlighted in coral, on a tangled one.

Annotation Operations & Ethics

Expert vs Crowd Annotation: When Expertise Really Matters

Expert vs crowd annotation: aggregated non-experts match experts on many tasks, but clinical judgment breaks the ratio. Get the decision guide.

Jul 21, 2026Read

A grid of small flat label tags, most in green and teal, with a single coral tag highlighted by a sweeping lens as the detected label error, and two near-identical tags mismatched in color.

Annotation Operations & Ethics

How to Find Label Errors in Dataset Quality Audits

Label errors hide in almost every dataset. Learn how to find label errors in dataset audits with disagreement, model confidence, and confident learning.

Jul 21, 2026Read

Two side-by-side abstract anxiety gauges—a short seven-notch self-report slider beside a taller fourteen-node clinician gauge split into psychic and somatic bands, one somatic node accented—reading toward a shared threshold line.

Rating Scales

GAD-7 vs HAM-A: Self-Report vs Clinician-Rated Anxiety

GAD-7 vs HAM-A compared: the 7-item self-report anxiety screen against the 14-item clinician-rated severity endpoint. See which anxiety scale to use.

Jul 21, 2026Read

A stylized overhead map of fields, water, roads, and building blocks overlaid with a tiling grid; several tiles lift away as labeled chips, one marked with a single coral accent for a rare feature, with small corner crosshair ticks suggesting georeferencing.

Image & Multimodal Annotation

Geospatial Image Annotation: Labeling Beyond the Everyday

Geospatial image annotation covers satellite, aerial, and microscopy imagery—where huge rasters, rare classes, and expert-only labels change the rules.

Jul 21, 2026Read

A stream of small tag shapes passing through a fine sieve that lifts out one broken coral tag while an intact green tag flows through, illustrating annotation quality checks that catch bad work without discarding legitimate disagreement.

Annotation Operations & Ethics

Gold Questions & Honeypots: Annotation QA That Works

Gold questions, honeypots, and attention checks catch bad annotations—if thresholds spare legitimate disagreement. See how each QA method works.

Jul 21, 2026Read

Two identical gauges read the same heavily skewed pile of tokens: the left needle collapses toward zero while the right needle holds high, illustrating how one chance correction breaks under lopsided prevalence while another stays stable.

Reliability & Agreement

Gwet's AC1 and AC2: A Fix for Kappa's Prevalence Paradox

Why Cohen's kappa collapses under skewed prevalence, and how Gwet's AC1 and AC2 chance-correction fixes it—with the formula and a worked example.

Jul 21, 2026Read

Abstract flat-vector illustration of one, two, and three-plus translucent reading plates over a scan region converging through a merge node into a single consensus outline, with one coral thread escalating a lone disagreement to an adjudication marker.

Image & Multimodal Annotation

How Many Annotators Medical Images Need: A Consensus Guide

How many annotators medical images need depends on clinical risk, from spot-checking low-risk labels to blind multi-reader reads with adjudication.

Jul 21, 2026Read

Several thin ink lines converge from the left onto a single teal square, merging into one green curve that rises and flattens toward a ceiling line, with one coral line pointing off-target — depicting many noisy annotator labels aggregating into one item with diminishing returns.

Annotation Operations & Ethics

How Many Annotators Per Item? Redundancy and the Cost Curve

How many annotators per item do you need? One label is risky. Compare majority vote, Dawid-Skene, and MACE, and see where extra labels stop paying off.

Jul 21, 2026Read

Abstract editorial illustration of a polished automated interface panel held up from below by rows of many faint, identical ghosted worker figures, with one figure highlighted in coral, on a warm off-white background.

Annotation Operations & Ethics

The Human Cost of Data Labeling: Ghost Work Behind AI

The human cost of data labeling: the underpaid, invisible ghost workers behind AI, the toll of the work, and what responsible sourcing looks like.

Jul 21, 2026Read

A stream of identical task tiles fans through a sorting junction into three lanes—many small dots, a few medium marks, and one detailed precision mark—with a single coral-highlighted tile routed to the expert mark, illustrating matching annotator background to task.

Annotation Operations & Ethics

Who Should Annotate Your Data? A Selection Framework

Who should annotate your data? Match annotator background to the task—expert, trained non-expert, or crowd—using a clear selection framework and matrix.

Jul 21, 2026Read

The same stylized bird annotated four ways in sequence—a coarse bounding box, a polygon outline, a filled pixel mask, and a set of connected keypoints—showing annotation precision rising from left to right.

Image & Multimodal Annotation

Image Annotation Types: Cost vs Precision, Explained

Bounding box, polygon, semantic vs instance vs panoptic masks, or keypoints? Compare image annotation types by cost, precision, and downstream task.

Jul 21, 2026Read

A single stylized street object approached by two annotators whose colored label-tags diverge, then converge onto one class after a small decision-log card is added between them.

Image & Multimodal Annotation

Image Labeling Guidelines Annotators Can Actually Apply

Write image labeling guidelines annotators actually apply: mutually exclusive vs overlapping classes, visual examples, and clear when-in-doubt rules.

Jul 21, 2026Read

Several translucent, slightly offset outlines of one shape overlaid so their shared center forms a solid green-teal consensus region, with a single coral tick-mark on the most disputed boundary.

Image & Multimodal Annotation

Image Segmentation Agreement: Dice, IoU, and STAPLE

Why Cohen's kappa fails on pixels, and how Dice, IoU, and the STAPLE algorithm measure image segmentation agreement across annotators. See the math.

Jul 21, 2026Read

Three linked nodes in a repeating loop—an initiation, a response, and a larger third node whose two faint paths fork between closing the loop and opening outward—illustrating the IRF exchange and what the feedback move reveals.

Annotation Operations & Ethics

IRF Classroom Discourse: Coding Teacher-Student Talk

IRF classroom discourse is the three-move Initiation-Response-Feedback exchange behind most teacher talk. Learn to code it and read the F move.

Jul 21, 2026Read

An abstract grid of paired tick-marks where most pairs align in green and teal, a few sit slightly offset to show disagreement, one cell is left blank to represent a missing rating, and a single coral mark highlights the lone divergence.

Reliability & Agreement

Krippendorff's Alpha: Nominal, Ordinal & Missing Data

How to compute Krippendorff's alpha: the observed-over-expected disagreement formula for nominal, ordinal, and missing data, with a worked example.

Jul 21, 2026Read

Abstract flat-vector illustration of a medical scan flowing through an annotation pipeline, where several overlapping translucent region outlines converge into one clean consensus outline, with a single coral marker at the point of agreement.

Image & Multimodal Annotation

Medical Image Annotation: A Complete Guide for AI Teams

Medical image annotation, end to end: annotation types, who should label, consensus and adjudication, DICOM de-identification, and QA tied to risk.

Jul 21, 2026Read

A segmented speech waveform tipping a two-pan balance — more change-talk tiles on one side, fewer sustain-talk tiles on the other — illustrating that the balance of client language, not the raw count, carries the predictive signal.

Methods & Frameworks

Change Talk vs Sustain Talk: Coding Client Language

Change talk vs sustain talk, coded: how the MISC classifies client language, why the balance predicts outcomes, and how to segment it reliably.

Jul 21, 2026Read

Two facing reading lenses examine the same speech-transcript ribbon: one resolves it into the client's bracketed change-and-sustain-talk tags, the other into the clinician's counted behavior tallies, with a single coral spark at the utterance both capture.

Methods & Frameworks

MISC vs MITI: Coding the Client vs the Clinician in MI

MISC vs MITI, disambiguated: the MITI codes the clinician's MI fidelity; the MISC codes the client's change and sustain talk. Compare purpose and use.

Jul 21, 2026Read

A balance scale weighing two stacks of tally-marks against each other beside two proportioned bars in a roughly two-to-one ratio, illustrating MI-consistent versus MI-inconsistent behavior counts and the reflection-to-question ratio.

Methods & Frameworks

MITI Motivational Interviewing: A Fidelity Coding Guide

How MITI motivational interviewing coding works: behavior counts, four global scores, and the R:Q ratio and %complex reflections that grade MI fidelity.

Jul 21, 2026Read

A grid of image-frame cards where faint dashed model-drawn outlines are being corrected into crisp solid outlines, with three uncertain cards lifted and flagged by a single coral tick for labeling next.

Image & Multimodal Annotation

Model-Assisted Labeling for Images: Label Less, Cover More

Model-assisted labeling lets a model draft image labels so annotators correct, not create. How pre-labeling and active learning cut work honestly.

Jul 21, 2026Read

Abstract flat-vector illustration of three horizontal tracks—a line of text, a strip of image frames, and an audio waveform—running left to right, with fine vertical guide lines tying the same moment across all three, and one coral tick marking where they align.

Image & Multimodal Annotation

Multimodal Annotation for Foundation Models: One Schema

Annotate aligned text, image, and audio in one schema for foundation models: cross-modal alignment, unified taxonomies, and QA that keeps labels in sync.

Jul 21, 2026Read

Abstract flat-vector illustration of a medical scan on the left and a column of text on the right, with a single fine coral line drawn from one phrase across to the exact region it describes on the image, and paler lines linking other phrases to other regions.

Image & Multimodal Annotation

Multimodal Annotation: Linking Images to Their Reports

How to build multimodal image-text annotation datasets: pair reports to images, ground report phrases to regions, and link findings across modalities.

Jul 21, 2026Read

Abstract flat-vector illustration of a whole-slide pathology image as a three-level resolution pyramid, zooming from a tiled full slide down to a cluster of cells, with a single coral ring marking one cell.

Image & Multimodal Annotation

Whole-Slide Image Annotation: A Guide for Pathology AI

Whole-slide image annotation for computational pathology: the gigapixel challenge, the slide-to-cell label hierarchy, and pathologist-in-the-loop QA.

Jul 21, 2026Read

A small tray of a few labeled sample tiles feeds a loop that measures, forks into two coder paths, converges, and only then opens a gate to a much larger grid of unlabeled tiles.

Methods & Frameworks

How to Run a Pilot Annotation Round Before You Scale

Run a pilot annotation round before you scale: scope, sample size, double-coding, measuring agreement, and revising the codebook. See the checklist.

Jul 21, 2026Read

Abstract diagram of a reliability study: a rising grid of subject-and-rater nodes beside a stack of confidence-interval brackets that narrow as the sample grows, with one target-width bracket highlighted in coral.

Reliability & Agreement

Sample Size for Inter-Rater Reliability: Subjects and Raters

Set the sample size for inter-rater reliability: how many subjects and raters pin kappa or an ICC to a target CI width. See the planning tables.

Jul 21, 2026Read

A methods paragraph shown as a row of bracketed blank slots being filled with green and teal blocks beside a checklist of tick-marks, with one slot marked by a coral confidence-interval bracket.

Reliability & Agreement

How to Report Inter-Rater Reliability: A Methods Template

A fill-in-the-blanks methods template for how to report inter-rater reliability: coefficient, raters, unit of analysis, CI, and software, per GRRAS.

Jul 21, 2026Read

A vertical rating scale with tick-marks expands rightward into a grid of codebook entry cards, each showing a color tag, a definition line, and an include/exclude mark, one card highlighted in coral.

Methods & Frameworks

Rating Scale to Codebook: Operationalize Any Instrument

A step-by-step method to go from rating scale to codebook: turn each item into a code, anchors, examples, and decision rules coders apply consistently.

Jul 21, 2026Read

Two paths from a cluster of scattered marks: one drops pieces through a coarse filter, the other guides the same marks through aligned gates into an even, calibrated row, one mark highlighted in coral.

Annotation Operations & Ethics

Improve Annotator Quality: Screen and Train, Don't Filter

To improve annotator quality, screen candidates for aptitude and train them before coding—filtering bad labels later loses. See the evidence.

Jul 21, 2026Read

A long-tailed bar distribution: tall common bars filled with a uniform machine-stamped pattern for synthetic labels, thinning into a hand-inked tail where a single coral tick marks a human-labeled rare case.

Annotation Operations & Ethics

Synthetic Data vs Human Annotation: When to Use Which

Synthetic data vs human annotation: when LLM-generated labels are cheap and good, where they miss the tail, and how to build a hybrid you can trust.

Jul 21, 2026Read

Two nested frames of tagged speech spans, a smaller set sitting inside a larger set, illustrating the SAPS positive formal-thought-disorder subscale in relation to Andreasen's TLC scale.

Rating Scales

TLC vs SAPS Thought Disorder: Coding Disorganized Speech

TLC vs SAPS thought disorder, compared: the 18-item TLC glossary vs the 8-item SAPS positive-FTD subscale, with an overlap grid and when to use each.

Jul 21, 2026Read

An abstract decision-fork diagram in which five differently shaped marker groups—scattered dots, a graded staircase of ticks, a smooth curve, a bracketed span, and a small grid—flow down branching channels into separate collection bins, with one branch highlighted in coral.

Reliability & Agreement

Which Inter-Rater Reliability Coefficient Should You Use?

Which inter-rater reliability coefficient to use, by data type, rater count, and missing data—a decision guide to kappa, alpha, AC1, and ICC.

Jul 21, 2026Read

A sorting junction routing annotation cards into four lanes — supervised demonstrations, preference ranking, red-teaming, and rubric scoring — each toward a different tool.

Tools & Comparisons

RLHF Annotation Tools: Match the Task to the Right Tool

Compare RLHF annotation tools by task—SFT demos, preference ranking, red-teaming, eval rubrics—and find which tool fits each. See the matrix.

Jul 19, 2026Read

Sixteen annotation-tool tags sorted into five labeled columns, one highlighted, illustrating a market segmented by use case.

Tools & Comparisons

Best Data Annotation Tools in 2026: An Honest Guide

Compare the best data annotation tools of 2026 by open-source status, pricing, modalities, and workforce—one master table, honest picks by use case.

Jul 19, 2026Read

A speech waveform resolving into a six-row checklist of clinical annotation requirements, with a small shield marking the compliance row.

Tools & Comparisons

Clinical Transcript Annotation Tools: A Buyer's Shortlist

Which clinical transcript annotation tool fits psychiatric, therapy, or SLP research? Compare options on HIPAA, speaker roles, severity scales, and IRR.

Jul 19, 2026Read

Three framed image tiles marked with bounding boxes and polygons — one open at a corner, two sealed with a clasp — with a single coral auto-label spark, contrasting open-source and enterprise computer-vision annotation tools.

Tools & Comparisons

CVAT vs Labelbox vs V7: Computer-Vision Tools (2026)

CVAT vs Labelbox vs V7 for computer-vision annotation: open-source vs enterprise, auto-labeling, DICOM, and pricing models, compared. See which fits.

Jul 19, 2026Read

On the left six plain emotion-category chips; on the right an eight-spoke radial wheel with intensity gradients and dyad links, one spoke accented in coral, with a small downward marker between them — contrasting Ekman's six discrete emotions with Plutchik's wheel.

Clinical Phenomena

Ekman vs Plutchik: Which Emotion Model to Label With?

Ekman vs Plutchik for annotation: 6 discrete emotions or 8 with intensities and dyads, and why more categories lower agreement. See which to pick.

Jul 19, 2026Read

Three parallel text-ribbon panels split into bracketed span-tags, joined by a curved active-learning loop arrow, one span accented in coral — contrasting three text-annotation tools.

Tools & Comparisons

Label Studio vs Prodigy vs doccano (2026 Comparison)

Label Studio vs Prodigy vs doccano for text annotation: open-source status, active learning, team QA, and pricing model, compared. See which one fits.

Jul 19, 2026Read

Two side-by-side gauges — a lean ten-mark column and a denser seventeen-mark column with heavier somatic segments at its base — joined by a faint crosswalk arc, with one coral mark on the shared remission cutoff, contrasting MADRS and HAM-D.

Rating Scales

MADRS vs HAM-D: Which Depression Scale Should You Use?

MADRS vs HAM-D compared: items, somatic weighting, sensitivity to change, score conversion, and remission cutoffs. See which depression scale to use.

Jul 19, 2026Read

A heavy legacy analysis instrument on the right beside three lighter, simpler tool shapes on the left, illustrating alternatives to a single monolithic qualitative software package.

Tools & Comparisons

The Best NVivo Alternatives for Qualitative Research 2026

Looking for an NVivo alternative? Compare the best free, modern, and AI-guided qualitative coding tools by price, platform, and clinical fit.

Jul 19, 2026Read

A balance beam weighing a cluster of small open modular blocks against a single sealed monolith block, its fulcrum marked in coral — the trade-off between open-source and commercial annotation tools.

Tools & Comparisons

Open-Source vs Commercial Annotation Tools: 2026 Guide

Open-source vs commercial annotation tools: the real trade-offs in total cost, support, compliance, and lock-in, plus a decision framework for your team.

Jul 19, 2026Read

One ribbon of interview speech splitting into four parallel channels that each sort it into coded blocks, illustrating a four-way qualitative software comparison.

Tools & Comparisons

Qualitative Data Analysis Software Comparison (2026)

A fair, four-way qualitative data analysis software comparison of NVivo, ATLAS.ti, MAXQDA and Dedoose: pricing, AI coding and built-in IRR, side by side.

Jul 19, 2026Read

Rating Scales

The Brief Psychiatric Rating Scale (BPRS), Explained

The Brief Psychiatric Rating Scale (BPRS) rates psychopathology on 7-point items. See the scoring, the factor structure, BPRS vs PANSS, and how to code it.

Jul 15, 2026Read

$A straight line of reasoning refracted through a faceted prism and emerging bent, illustrating how cognitive distortions bias an appraisal.$ Clinical Phenomena

Cognitive Distortions List: 10 CBT Patterns, Coded

The cognitive distortions list, explained: all 10 CBT thinking errors with definitions and examples, where the list came from, and why they resist coding.

Jul 15, 2026Read

Six minimal geometric face-shapes in a row, illustrating Ekman's six basic emotions as a label set.

Clinical Phenomena

Ekman's Basic Emotions: The Six, and How to Label Them

Ekman's basic emotions are six universal states: anger, disgust, fear, happiness, sadness, surprise. See why they anchor most emotion datasets.

Jul 15, 2026Read

A patient's speech bubble opening beside a six-rung response ladder, illustrating the ECCS opportunity-and-response structure.

Methods & Frameworks

The Empathic Communication Coding System, Explained

The Empathic Communication Coding System codes empathy in clinical talk as a two-step sequence: patient opportunities, then a 0–6 provider response ladder.

Jul 15, 2026Read

Two stacked columns of tick-marks, psychic over somatic, illustrating the HAM-A's two-factor anxiety structure.

Rating Scales

Hamilton Anxiety Rating Scale: Psychic vs Somatic HAM-A

The Hamilton Anxiety Rating Scale (HAM-A) is a 14-item clinician measure. See the psychic and somatic items, the scoring, the cutoffs, and how to code it.

Jul 15, 2026Read

A story arc drawn as a ribbon in six segments with the evaluation high-point marked, illustrating Labov's narrative structure.

Methods & Frameworks

Labov and Waletzky Narrative Analysis: The Six Parts

Labov and Waletzky narrative analysis breaks an oral story into six parts. See each part, why evaluation is the heart, and how reliably it can be coded.

Jul 15, 2026Read

Five funnels from wide-open to narrow, illustrating the NICHD prompt continuum from invitation to suggestive.

Methods & Frameworks

The NICHD Protocol: Coding Interviewer Prompt Types

The NICHD protocol classifies a forensic interviewer's prompts from open to suggestive. See the five types, why open prompts win, and how to code them.

Jul 15, 2026Read

A patient-involvement gauge reading low, illustrating that observed shared decision-making is consistently low.

Methods & Frameworks

The OPTION Scale: Measuring Shared Decision-Making

The OPTION scale rates how far a clinician involves the patient in a decision. See the behaviors it scores, why real scores are low, and how to code it.

Jul 15, 2026Read

An abstract eight-petal radial wheel with graded tones, illustrating Plutchik's eight primary emotions and their intensities.

Clinical Phenomena

Plutchik's Wheel of Emotions: 8 Primaries, Explained

Plutchik's wheel of emotions maps eight primaries into four opposing pairs, with intensity rings and dyads. See how to label emotion in text.

Jul 15, 2026Read

A sparse speech waveform on a wide pale field, mostly flat silence with a few short bracketed segments, illustrating poverty of speech as a measured absence.

Clinical Phenomena

Poverty of Speech and Alogia: Coding the SANS Subscale

Poverty of speech is the core of alogia. See the four SANS alogia items, how they differ from poverty of content, and how to code each from a transcript.

Jul 15, 2026Read

A line of connected nodes representing a train of thought derailing onto an oblique branch, illustrating positive formal thought disorder.

Clinical Phenomena

Positive Formal Thought Disorder: The SAPS Subscale

Positive formal thought disorder is disorganized speech you can see. Meet the SAPS FTD items, how they differ from the TLC and SANS, and how to code them.

Jul 15, 2026Read

Speech-bubble shapes sorting into five distinct channels, illustrating Searle's five classes of speech act.

Methods & Frameworks

Searle's Speech Acts: The Five Illocutionary Types

Searle's speech acts sort every utterance into five illocutionary types. See the classes, direction of fit, examples, and the tie to dialogue-act tagging.

Jul 15, 2026Read

A speech thread that visibly derails onto a tangent, illustrating disordered thought, language, and communication.

Clinical Phenomena

The Thought Language and Communication Scale, Explained

The Thought Language and Communication scale (TLC) is Andreasen's glossary of disordered speech. See the items, the reliability data, and how to code them.

Jul 15, 2026Read

An argument as a supported scaffold with one dashed ghosted beam for the implicit warrant, illustrating the Toulmin model.

Methods & Frameworks

The Toulmin Model of Argumentation: The Six Elements

The Toulmin model of argumentation breaks an argument into six functional parts. See the elements, a worked example, and why warrants are hardest to code.

Jul 15, 2026Read

Two columns of tick-marks linked by green lines over a faint 2×2 grid, with a shaded "chance" band being subtracted to reveal the smaller chance-corrected agreement, one grid cell accented in coral — illustrating Cohen's kappa.

Reliability & Agreement

Cohen's Kappa: How to Compute and Interpret Agreement

How to compute Cohen's kappa and weighted kappa by hand, read the result honestly, and know when to use Krippendorff's alpha, ICC, or Gwet's AC1 instead.

Jul 13, 2026Read

Seven waveforms rising from calm to agitated, illustrating the GAD-7's seven anxiety items and its severity gradient.

Rating Scales

The GAD-7 Anxiety Scale: Items, Cutoffs & Transcript Coding

The GAD-7 is a seven-item anxiety screen. See the 5/10/15 severity bands, the ≥10 cutoff, and how to code it from an interview transcript.

Jul 13, 2026Read

A speech waveform segmented into colored, bracketed spans over a faint punch-card texture, illustrating clause-level content analysis that tags and scores spoken language.

Methods & Frameworks

Gottschalk-Gleser Content Analysis: Coding Affect in Speech

How Gottschalk-Gleser content analysis scores affect in speech, clause by clause—the seven depression subscales, the weighted coding, and its NLP legacy.

Jul 13, 2026Read

Rating Scales

Hamilton Depression Rating Scale (HAM-D): A Coding Guide

How the Hamilton Depression Rating Scale (HAM-D) is scored and coded: the 17 items, the 0–52 range, remission and response cutoffs, and its known flaws.

Jul 13, 2026Read

A descending line chart whose ten points cross a threshold line, illustrating the MADRS's sensitivity to change and its remission cutoff.

Rating Scales

MADRS Explained: 10 Items, Scoring Bands & Remission

MADRS scoring explained: the 10 clinician-rated items, the 0-6 anchors, severity bands, and the response and remission cutoffs. See how it's coded.

Jul 13, 2026Read

Nine short four-step item bars feeding into one horizontal severity gauge with graded bands, a single coral marker at the cutoff line—an abstract depiction of PHQ-9 items, 0–3 anchors, and severity bands.

Rating Scales

PHQ-9 Scoring Explained: Items, Cutoffs, and Coding

PHQ-9 scoring made clear: what the nine items measure, the 0-3 anchors, the severity bands, the ≥10 cutoff, and how to code a PHQ-9 from an interview.

Jul 13, 2026Read

Rating Scales

Young Mania Rating Scale (YMRS): Rating Mania from Speech

The Young Mania Rating Scale (YMRS) explained: its 11 items, the four double-weighted items, severity bands, and how to rate mania from interview speech.

Jul 13, 2026Read