Forms carry a category of information that most extraction systems never see: the marks people make by hand. Inspectors check boxes, adjusters circle applicable conditions, and patients tick, cross, or scribble over their responses in ways that no two documents share. PulseBench-Select is our frontier evaluation of how well foundation models and other vision systems actually read those marks, and today we are publishing the results.
The Benchmark
We evaluated Pulse against five other systems on a corpus of several hundred document pages spanning multiple form types and mark styles. Every system was tested on the same documents, and results were scored against human-validated ground truth. The benchmark consists of 485 pages, with a total of 1706 selections sourced from publicly available PDFs and synthetically generated selections.
We report two sets of metrics. Micro metrics aggregate across every individual selection decision in the corpus, so a document with 20 marks contributes 20 data points. Macro metrics average per document first and then average those document-level scores, which means every document counts equally regardless of how many marks it contains. Macro scores are more sensitive to how a system behaves on harder or sparser documents, and a system that selectively abstains on difficult pages will appear stronger in macro than it deserves.*
Micro metrics
The micro and macro results tell the same story from two angles. On micro, Pulse's F1 of 0.76 is more than double the next best result. On macro, Pulse's recall of 0.80 means it correctly identifies selections on the vast majority of documents, while GPT-4o manages only 0.19.
Systems that were not built for this task tend to either abstain silently or return selections only when the mark is so unambiguous that it barely needed detecting in the first place.
Two numbers in the Pulse row are worth dwelling on specifically. Micro precision of 0.774 means roughly 4 in 5 selections Pulse reports are correct, and micro recall of 0.753 means Pulse finds about 3 in 4 of all marked selections in the corpus. No other system in the benchmark achieves micro recall above 0.32, so even the strongest competitor misses more than 2 in 3 marks. For workflows where a missed selection has downstream consequences, whether that is a compliance finding not captured or a medical preference not recorded, that gap is the difference between automation that works and automation that cannot be trusted.
Macro metrics
What This Unlocks
The practical effect is that a category of form data that was previously invisible to automated extraction pipelines becomes readable. This includes compliance checklists where inspectors mark findings by hand, insurance claim forms where adjusters circle applicable conditions, healthcare intake documents where patients tick, cross, or circle their responses, and survey instruments where respondents annotate their answers in any of a dozen different ways.
Every one of those workflows currently requires a human to read the original document and reconcile it against whatever the extraction system returned. With mark detection, the extraction output reflects what the document actually says, including what the pen said.
Mark detection is available now via the Pulse API. Contact the team to enable it on your extraction pipeline.
* Macro F1 is omitted. Scores are undefined for documents where a provider returned zero predictions.
