Multimodal Annotation: Linking Images to Their Reports

Multimodal annotation is labeling an image and the text that describes it as one linked object, so a model can learn the correspondence between them—which words in a radiology report point to which region of the scan. It is the data problem sitting under every medical vision-language model, and it stays invisible until you try to build the dataset yourself.

The scale of the opportunity shows up in a single number. MIMIC-CXR released 377,110 chest radiographs paired with their de-identified free-text reports (Johnson et al., 2019), and that image-text pairing is what turned report writing—a byproduct of routine care—into supervision for models. But a pile of paired images and reports is not an annotated dataset. The value comes from three increasingly precise links: pairing the report to the study, grounding a phrase to a region, and linking an entity across both.

What is multimodal annotation?

Multimodal annotation is labeling data that spans more than one modality—here, a medical image and its report—so the annotations in each modality are explicitly tied together. Instead of labeling the image and tagging the text as two separate jobs, you record how the two line up.

That linking happens at three levels of precision, and it helps to name them before you build. Pairing is the coarsest: this report belongs to this study. Grounding is spatial: this phrase describes this region of this image. Entity linking is semantic: the finding named in the text, the region in the pixels, and the term in your vocabulary all refer to the same thing.

Most teams start and stop at pairing because it’s nearly free. The reproducible value, and the harder work, lives in the other two.

Why pair medical images with their reports?

You pair images with reports because the report is free, expert-written supervision. A radiologist already described the findings in prose during routine care, so pairing the report to the study converts that description into labels without a separate annotation campaign.

That single idea is why paired corpora anchor the field. MIMIC-CXR assembled 377,110 images across 227,835 studies, each with the contemporaneous report a practicing radiologist wrote during care, all de-identified for release (Johnson et al., 2019). Downstream, the pattern generalizes: Flamingo showed that training on large web corpora of interleaved images and text yields few-shot visual reasoning (Alayrac et al., 2022), and LLaVA-Med adapted the recipe to medicine using figure-caption pairs mined from PubMed Central plus instruction-tuning data (Li et al., 2023).

Here’s the catch most pipelines gloss over. Report-level pairing is weak supervision: the report describes the whole study, not any single pixel, and a sentence like “no acute cardiopulmonary process” carries no localizable label at all. Pairing gets you a training signal; it does not get you a region.

What is grounding, and why does it matter?

Grounding is linking a specific phrase in a report to the exact image region it describes—drawing the line from the words “small right pleural effusion” to the box around that effusion on the radiograph. It is the difference between knowing a finding was mentioned and knowing where it is.

That distinction is what makes grounded data worth the extra effort. A public benchmark makes the task concrete: MS-CXR provides curated phrase-and-bounding-box pairs across eight cardiopulmonary findings, each pairing a sentence from a report with the region it refers to, reviewed and edited by board-certified radiologists (Boecking et al., 2022).

Two payoffs justify the cost. Grounded labels give a model localizable supervision, so it can learn to point rather than only to classify. And they make the model auditable: a system that highlights the right region for the phrase it generated can be checked against the annotation in a way a free-text caption never can. Grounding is where a multimodal dataset stops being a bag of pairs and starts being evidence.

How do you link entities across modalities?

Entity linking binds the clinical entities named in a report—findings, anatomy, and their relations—to the image regions they occupy and to a shared vocabulary, so “effusion” in the text, the region in the pixels, and the ontology term all point to one thing. It is the layer that keeps a dataset internally consistent.

RadGraph is the reference for the text side. It defines an information-extraction schema of four entity types—Anatomy, and Observation split into Definitely Present, Uncertain, and Definitely Absent—plus three relations (suggestive_of, located_at, modify), with 500 MIMIC-CXR reports annotated by board-certified radiologists yielding 14,579 entities and 10,889 relations (Jain et al., 2021). Notice that presence is built into the entity type: the schema refuses to treat a mentioned finding and a present finding as the same label.

Chest ImaGenome carries that structure onto the image. It is a scene-graph dataset automatically derived from MIMIC-CXR that connects report attributes to 29 anatomical locations with bounding-box coordinates, adds over 670,000 localized comparison relations across sequential exams, and ships a manually validated gold standard for 500 patients (Wu et al., 2021). A scene graph is entity linking made spatial: every node is a region, and the edges are the relations the report described.

How do you build a multimodal annotation dataset?

Build it in four layers, coarse to fine: pair the report to the study, annotate the image regions, extract the entities and phrases from the report, then draw the grounding links between them. Each layer depends on the one before it, and you de-identify before any of them.

Layer	What you produce	Example artifact	Where it breaks
1. Pair	Report tied to the correct study and images	study ↔ report mapping	Wrong study linked; prior exams merged in
2. Region	Boxes or masks on the image	One bounding box per finding	Boundary disagreement between annotators
3. Extract	Entities, phrases, and presence from the text	RadGraph-style entities + relations	Negation and uncertainty silently dropped
4. Ground	Links from a phrase or entity to a region	Phrase ↔ box, MS-CXR style	Non-localizing phrases forced onto a box

De-identification comes first and is non-negotiable, because a report and a burned-in image label are two more places protected health information hides—clean the DICOM header and the pixels before anything else, as covered in our DICOM annotation and data-prep guide. Layer two is ordinary region work; if you’re deciding between boxes, masks, and global labels, the trade-offs are laid out in image annotation types explained. Layers three and four are where multimodal annotation earns its name.

A worked grounding example

Take one synthetic report sentence and run it through the layers. Suppose the impression reads: “Small layering right pleural effusion; no pneumothorax. Cardiomediastinal silhouette stable compared to prior.” Three phrases, three different fates.

Report phrase (synthetic)	Type	Grounds to a region?	How to encode it
small layering right pleural effusion	Finding, present	Yes—right costophrenic angle	Box + entity 'effusion', marked present
no pneumothorax	Finding, absent	No	Entity 'pneumothorax', marked absent—never a positive label
silhouette stable compared to prior	Comparison to a prior exam	No	A relation to another study, not a region in this image

The lesson is in the two phrases that don’t ground. A pipeline that boxes “every finding named” would either invent a pneumothorax region or, worse, label the study positive for a finding the report explicitly denied. Encoding absence and comparison as their own structure—rather than dropping them or flattening them into labels—is what separates a usable multimodal dataset from a noisy one.

This is exactly why RadGraph’s schema makes “Definitely Absent” a first-class entity type (Jain et al., 2021). For a fuller, finding-by-finding treatment on real chest films, see our chest X-ray annotation worked example.

Where curated text schemes come in

Grounding is span annotation that crosses a modality boundary, so it needs what span annotation of text always needed: a curated scheme with defined terms. Without a controlled vocabulary, “effusion,” “fluid,” and “blunted costophrenic angle” become three labels for one thing, and your grounding links stop being comparable across annotators.

Clinical text annotation solved this problem decades ago, and multimodal projects can borrow the pattern wholesale. The Scale for the Assessment of Thought, Language and Communication (TLC) is, at heart, a span-annotation instrument: each item is an operationally defined utterance type an annotator highlights in speech (Andreasen, 1986). The Montgomery-Åsberg Depression Rating Scale (MADRS) works the same way for severity, tying each rating to the passage of interview speech that justifies it (Montgomery & Åsberg, 1979). Both are curated schemes over text—a defined term, bound to a span of evidence.

Grounding a report phrase to an image region is the same discipline with the modality swapped: a defined finding, bound to the evidence that supports it, wherever that evidence lives. The engine that codes a transcript against a scale is the same one that tags a finding on an image—which is the bridge from our clinical transcript annotation tools to the image side. Designing that controlled vocabulary well is its own craft; the mutually-exclusive-class discipline lives in image labeling taxonomy design.

Where does multimodal annotation break down?

Multimodal annotation breaks at the seams between modalities, and the failures are predictable enough to plan around. Four recur.

Reports don’t localize. A radiologist writes to communicate an impression, not to draw boxes, so many true sentences map to no single region. Forcing every phrase onto a box manufactures spatial labels the text never justified.

Negation and uncertainty get dropped. “No pneumothorax” and “possible early edema” are not positive findings, and a schema that only records mentions turns both into errors—which is precisely why RadGraph separates Definitely Present, Uncertain, and Definitely Absent (Jain et al., 2021).

Automatically derived labels are noisy. Chest ImaGenome’s local labels come from a rule-based NLP and atlas-based bounding-box pipeline, which is powerful at scale but imperfect—so its manually validated gold standard covers only 500 patients out of the full set (Wu et al., 2021). Treat machine-derived grounding as a strong prior a human confirms, not as truth.

Regions are where annotators disagree most. Two experts rarely draw the same box, so grounded datasets need multiple reads and an agreement measure. How many annotators, and how to reconcile them, is its own question—see how many annotators medical images need, and use model-assisted image labeling to pre-draw regions a human then corrects rather than starting from a blank frame.

Where Tagaroo fits

Tagaroo is a schema-first annotation workspace: you define a coding scheme once, then tag against it across text and images, with inter-rater reliability and evidence built into the workflow rather than bolted on. The design bet is that multimodal projects live or die on the scheme and the correspondence between modalities—so the same engine that links a TLC item to a span of transcript speech is the one that links a finding to a report phrase and an image region.

Be clear about the boundary. If your work is industrial-scale pixel segmentation of DICOM volumes with specialized auto-annotation, a dedicated computer-vision platform is the right call, and the medical image annotation guide covers that end of the spectrum. Tagaroo’s strength is guided, schema-driven annotation where every label is anchored to the evidence that justifies it—which is exactly the shape of a grounding link.

On data handling, medical reports and images are protected health information, and Tagaroo takes the de-identify-first path this article argues for rather than positioning itself as a PHI processor. Its terms require you to strip direct identifiers—DICOM header fields and burned-in pixel text alike—before upload; its anonymous trial mode runs entirely in the browser, so trial data never leaves your machine; and the stack is EU-hosted and GDPR-oriented. It is not a medical device, and any AI pre-label is a first pass for a qualified human to review, never a diagnosis. If your project must process raw PHI in the cloud under a signed Business Associate Agreement, put that question to any vendor—including this one—before uploading identifiable studies; the specifics are in Tagaroo’s privacy policy.

The practical upshot

Multimodal annotation is three links, coarse to fine—pair, ground, link—and the value concentrates in the two most teams skip. Grounding turns a report into localizable, auditable evidence; entity linking keeps that evidence internally consistent; and a curated scheme is what makes both reproducible across annotators. Get the schema right and the correspondence between modalities takes care of itself.

If you want a workspace that treats the scheme and the evidence link as first-class from label one, start by building a coding scheme in Tagaroo—then extend the same discipline from transcripts to the images that sit beside them.

References

• Johnson AEW, Pollard TJ, Berkowitz SJ, et al. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data. 2019;6:317. doi:10.1038/s41597-019-0322-0
• Boecking B, Usuyama N, Bannur S, et al. Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing. ECCV 2022; MS-CXR phrase-grounding benchmark. arXiv:2204.09817
• Jain S, Agrawal A, Saporta A, et al. RadGraph: Extracting Clinical Entities and Relations from Radiology Reports. NeurIPS 2021 Datasets and Benchmarks Track. arXiv:2106.14463
• Wu JT, Agu NN, Lourentzou I, et al. Chest ImaGenome Dataset for Clinical Reasoning. NeurIPS 2021 Datasets and Benchmarks Track. arXiv:2108.00316
• Li C, Wong C, Zhang S, et al. LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day. NeurIPS 2023 Datasets and Benchmarks Track. arXiv:2306.00890
• Alayrac JB, Donahue J, Luc P, et al. Flamingo: a Visual Language Model for Few-Shot Learning. NeurIPS 2022. arXiv:2204.14198
• Andreasen NC. The Scale for the Assessment of Thought, Language, and Communication (TLC). Schizophrenia Bulletin. 1986;12(3):473-482. doi:10.1093/schbul/12.3.473
• Montgomery SA, Åsberg M. A new depression scale designed to be sensitive to change. British Journal of Psychiatry. 1979;134:382-389. doi:10.1192/bjp.134.4.382