Document Annotation: Layout, Entities & Key-Value Fields

A scanned form is two documents at once: the text a human reads, and the geometry that tells them what the text means. Strip the layout away and “Total” floating next to “$1,240.00” collapses into two unrelated strings. Document annotation is the work of labeling both at the same time—the words, the regions they sit in, the entities they name, and the links between them—so a model can read a page the way a person does.

Get the label set and the ordering right and downstream extraction is almost easy; get them wrong and no model recovers. This guide walks the full stack, task by task, with the datasets and numbers behind each one.

What is document annotation?

Document annotation is the labeling of a document’s content and structure so a machine can turn an unstructured page—a PDF, a scan, a photographed form—into structured, queryable data. It spans four linked layers: the text and its coordinates from OCR, the layout regions that group that text, the entities the text names, and the relations between those entities. Unlike plain text annotation, every label carries a position on the page, because where a word sits often decides what it means.

That makes a document genuinely multimodal: it is text and image at the same time, which is why the field borrows tools from both NLP and computer vision. The dual nature is the whole difficulty, and it rewards being handled deliberately rather than as an afterthought, as we argue in multimodal image-text annotation. A page is not a paragraph with pictures; it is a spatial object whose meaning lives in the arrangement.

The document annotation stack: OCR to semantics

The document annotation stack is an ordered pipeline where each labeling task anchors to the output of the one before it, so the sequence matters as much as the tasks. Work it top to bottom:

1. OCR / text layer. Recover the characters and their positions—words with bounding boxes and reading coordinates. For born-digital PDFs this comes from the file; for scans it comes from an OCR engine. Everything downstream attaches to this.
2. Layout / region labeling. Draw and classify page regions: title, paragraph, list, figure, table, header, footer. This is bounding-box labeling on the page image.
3. Document entities (document NER). Label spans of the recovered text with types—name, date, invoice number, diagnosis—each tied to its box.
4. Key-value / form pairs. Link entities into relations: this value belongs to that key, so “Total” points to its amount.
5. Table structure. Reconstruct rows, columns, spanning cells, and headers, then the content of each cell.
6. Reading order and relations. Order the regions into a coherent sequence and connect captions to figures, footnotes to their references.

You can annotate any level directly—draw a table region by hand—or bootstrap it from a model and correct, but the dependency order holds either way. An entity label is meaningless until the text and its box exist, and a key-value link is meaningless until both entities do.

Document annotation tasks at a glance

The core document annotation tasks map cleanly onto that stack, and each has a canonical benchmark that fixes what “good” looks like. Read down the last column to pick a task; the sections below unpack each one.

Task	What you label	Label / unit example	Benchmark dataset	Downstream use
Layout / region	Page regions as boxes + class	"Title", "Table", "Figure", "List-item"	DocLayNet (80,863 pages, 11 classes)	Document parsing, reading order
Document entities (NER)	Typed spans of recovered text	"question", "answer", "header"	FUNSD (199 forms, 9,743 entities)	Field detection, redaction
Key-value / form pairs	Links between two entities	key → value (Total → $1,240.00)	SROIE (1,000 receipts; 4 key fields)	Invoice / form extraction
Table structure	Rows, columns, spanning cells	Cell grid + header cells	PubTabNet (568k tables, TEDS)	Turning tables into data
Text + layout pretraining	Tokens paired with 2D positions	token + (x, y) box	LayoutLM (text + layout + image)	Jointly powers all of the above

Costs and consistency rise as you move down that table, from coarse region boxes to linked key-value pairs and full table grids. It is the same precision-versus-cost ladder as image annotation types, applied to a page instead of a photo.

How do you label document layout and regions?

You label layout by drawing a bounding box around each region of the page and assigning it a class—Title, Paragraph, List-item, Figure, Table, Header, Footer. It is object detection on a document image, and it is the layer that gives every later label somewhere to live. The choice that decides everything is how many classes you define and where their boundaries fall.

The public datasets show two ways to get there. PubLayNet reached over 360,000 pages by automatically matching the XML and PDF versions of more than a million PubMed Central articles, labeling five classes—text, title, list, figure, table—with no manual drawing (Zhong et al., 2019). DocBank pushed the automatic approach to 500,000 pages with token-level labels across 12 semantic unit types, using weak supervision from arXiv LaTeX source (Li et al., 2020).

Automatic labeling is cheap but inherits the source’s bias: both datasets are scientific papers, so a model trained on them stumbles on invoices, contracts, and forms. DocLayNet was built to fix exactly that—80,863 pages hand-annotated across 11 classes drawn from diverse sources, deliberately varied in layout (Pfitzmann et al., 2022). The practical read: bootstrap from an auto-labeled corpus if your documents look like papers, but budget human labeling when they don’t.

What is document entity annotation (document NER)?

Document entity annotation, or document NER, labels spans of the recovered text with types that matter to the task—question, answer, header, invoice number, date—while keeping each span tied to its box on the page. It is named-entity recognition with coordinates, and it is where a page stops being blocks and starts being fields. The canonical benchmark is FUNSD, which marks 9,743 semantic entities as question, answer, header, or other across 199 real scanned forms (Jaume et al., 2019).

This layer is also where de-identification lives. To redact a clinical or legal record you first annotate the entities to remove, which is why entity labeling on sensitive documents should run on de-identified or access-controlled copies (see Tagaroo’s privacy policy). The label set here is narrower and more opinionated than layout classes, because it encodes what your specific extraction task actually cares about.

How do you annotate key-value and form fields?

Key-value annotation links entities into pairs: you connect each value to the key it answers, so “Total” points to “$1,240.00” and “Date of birth” points to the date beside it. It turns a bag of typed spans into a filled form, and it is the labeling that sits behind receipt, invoice, and form extraction. FUNSD includes these links as relations between semantic entities, and SROIE reduces the whole problem to four target fields—company, date, address, and total—across 1,000 scanned receipts (Jaume et al., 2019; Huang et al., 2019).

Linking is the error-prone part, especially in multi-column forms where several keys and values compete for the same neighborhood. A mislinked pair is a classic quiet defect: the entity type is right, the value is right, but the value is bound to the wrong key. That is exactly the kind of error a systematic pass for label errors in your dataset is built to surface, because it never trips a simple field-level check.

How do you annotate table structure?

Table-structure annotation reconstructs a table’s grid—which cells exist, how rows and columns align, which cells span multiple rows or columns, which are headers—and then the content of each cell. A table is not just another region; it is a nested structure, and it is the hardest layer to get right because errors compound. One merged cell read wrong shifts every value after it.

PubTabNet is the reference dataset here, with 568,000 table images paired with the structured HTML representation of each table, built by matching the XML and PDF of PubMed Central articles (Zhong et al., 2020). It also introduced the metric the task now uses: Tree-Edit-Distance-based Similarity, which models each table as an HTML tree and scores the edit distance between prediction and truth, precisely so that a cell landing in the wrong column costs more than a character typo.

Why does layout matter for a document model?

Layout matters because position carries meaning that the text alone throws away: the same string means different things as a column header, a page footer, or a form label. The field’s central result is that modeling text and 2D position together beats modeling text alone. LayoutLM was the first framework to jointly pretrain on text and layout (and optional image features), and the payoff was direct—adding position raised form-understanding F1 on FUNSD from 70.72 to 79.27, and receipt understanding from 94.02 to 95.24 (Xu et al., 2020).

The lesson for annotation is blunt: your labels must preserve position. Every entity keeps its box, every region keeps its coordinates, and reading order is recorded rather than assumed. Discard that and you throw away the signal the strongest document models are built to use.

Where does document annotation go wrong?

Document annotation goes wrong in a few predictable places: fuzzy region boundaries, inconsistent reading order in multi-column pages, span edges that annotators draw differently, and links attached to the wrong key. These are agreement problems, and you measure them the same way you measure any coding task—by having people label the same pages and quantifying how often they match.

DocLayNet’s authors did exactly that. They double- and triple-annotated a subset specifically to measure inter-annotator agreement, and found their best detection models still trailed human agreement by roughly 10% (Pfitzmann et al., 2022). The human ceiling, not a leaderboard number, is the real target. Track the gap with the same inter-rater reliability coefficients you would use on a rating scale, and you will know whether a low score means a weak model or an ambiguous label set.

Most of these failures are guideline failures, not annotator failures. A precise definition of where a Table region ends, or which value a floating label owns, prevents the disagreement before it happens—which is the entire purpose of annotation guidelines that work. Write the boundary rules first, pilot them, then scale.

Where does Tagaroo fit?

Tagaroo is not an OCR engine or a layout-detection model, and it does not draw boxes on page images; for pixels and PDFs at scale, use the document-AI tools named above. Where it fits is the part every one of these tasks shares: defining the label set and getting people to apply it consistently. Choosing region classes is the same decision whether you land on PubLayNet’s 5, DocLayNet’s 11, or DocBank’s 12—more classes capture more nuance and invite more disagreement.

The clearest parallel is a clinical coding scheme. Andreasen’s TLC scale is, structurally, a document-style label set: twelve operationally defined categories, each with a boundary an annotator must apply the same way every time. Whether the label is “Derailment” on a transcript or “Table” on a page, the discipline is identical—define the term, pilot it, measure agreement, revise. A finer instrument like the MADRS buys more resolution at the cost of more rater disagreement, the same trade as an 11-class layout scheme versus a 5-class one.

Tagaroo turns any such scheme into a guided workflow with inter-rater reliability computed as annotators work. That is the same discipline a serious document annotation program needs, minus the pixels.

How do you run a document annotation project?

Run a document annotation project by working the stack in order and fixing your label set and quality metric before the first page is labeled. A short checklist:

1. Get the text layer solid first. Bad OCR poisons every downstream label; verify word boxes and transcripts before anyone types an entity.
2. Write the label set as definitions with boundaries, not just names. “Table” needs a rule for where it ends; “answer” needs a rule for multi-line values.
3. Pilot on a handful of pages, double-annotate, and measure agreement. Fix the guideline where raters diverge, then scale.
4. Annotate in dependency order—regions, then entities, then links, then tables—so each layer has something to attach to.
5. Choose the metric that matches the task: mean average precision for regions, F1 for entities, and TEDS for tables (Zhong et al., 2020).
6. Route expensive work with a model in the loop. Draft regions or fields automatically and have people verify, using one of the best data annotation tools for the pixel layer.

The practical upshot: document annotation is less a modeling problem than a taxonomy-and-agreement problem wearing a computer-vision costume. Pick the smallest label set that answers your question, preserve every entity’s position, annotate the stack in order, and measure agreement the way you would on any coding scheme. If your real problem is a structured label set and consistent human judgment rather than raw pixels, that is what Tagaroo is for—start by turning a scheme like the TLC into a guided, reliability-tracked workflow.

References

• Jaume G, Ekenel HK, Thiran JP. FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents. ICDAR-OST 2019. arXiv:1905.13538
• Xu Y, Li M, Cui L, Huang S, Wei F, Zhou M. LayoutLM: Pre-training of Text and Layout for Document Image Understanding. KDD 2020. arXiv:1912.13318
• Zhong X, Tang J, Jimeno Yepes A. PubLayNet: Largest Dataset Ever for Document Layout Analysis. ICDAR 2019. arXiv:1908.07836
• Li M, Xu Y, Cui L, Huang S, Wei F, Li Z, Zhou M. DocBank: A Benchmark Dataset for Document Layout Analysis. COLING 2020. DOI · arXiv:2006.01038
• Pfitzmann B, Auer C, Dolfi M, Nassar AS, Staar P. DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis. KDD 2022. DOI · arXiv:2206.01062
• Zhong X, ShafieiBavani E, Jimeno Yepes A. Image-based table recognition: data, model, and evaluation (PubTabNet). ECCV 2020. arXiv:1911.10683
• Huang Z, Chen K, He J, Bai X, Karatzas D, Lu S, Jawahar CV. ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction (SROIE). ICDAR 2019. arXiv:2103.10213

If your annotation problem is a structured label set on transcripts or records rather than raw pixels, Tagaroo turns coding schemes like the TLC and the MADRS into guided, reliability-tracked workflows.