how does ocr work
How OCR Turns a Scan Into Text
OCR is not one operation. It is a pipeline of five, and almost every disappointing result can be traced to which stage went wrong.
01
Stage one: clean up the image
Recognition works on shapes, so anything that distorts shapes has to go first. The page is deskewed — a scan is rarely square, and a two-degree rotation is enough to smear the line-finding that comes next. Then it is binarised: every pixel becomes black or white, with the threshold chosen adaptively so a page that is brighter at one edge than the other does not lose half its text.
Then despeckling removes isolated specks of noise, which otherwise get recognised as full stops and commas.
This stage is why scan quality dominates OCR accuracy far more than the recogniser does. A shadow across the page defeats thresholding, and a curved page from a bound book cannot be deskewed because the distortion is not a rotation.
02
Stage two: work out the layout
The engine finds connected components — contiguous runs of black pixels — and groups them. Nearby components on a shared baseline become a word; words in a row become a line; stacked lines with consistent spacing become a text block. Large white gutters split the page into columns.
This is also where images, rules and tables are identified and set aside, so the recogniser is not asked to read a photograph as if it were type.
Layout analysis decides reading order, and reading order is what most people notice when it fails. A newsletter with a pull quote in the middle of a column, or an invoice where the totals sit in a floating box, is where an engine tuned for a page of prose starts producing text in an order nobody wrote.
03
Stage three: recognise the characters
Older OCR compared each character's shape against a library of templates, feature by feature: count the holes, find the stroke junctions, measure the aspect ratio. It was fast, brittle, and needed a separate template set for every typeface.
Modern engines run a neural network over the line as a whole — typically an LSTM, which reads a sequence of narrow vertical slices left to right and outputs a sequence of characters. This matters because it removes the need to segment characters before recognising them. Deciding where one letter ends and the next begins is itself hard: `rn` and `m` are nearly identical at low resolution, and touching letters in a tight font have no gap to cut at. A sequence model sidesteps the problem by never making that cut.
Every character comes out with a confidence score, which is what a good OCR tool exposes and what the next stage uses.
04
Stage four: use language to fix what vision got wrong
The recogniser sees shapes and nothing else, so it will confidently return `l\/lanager`, `0ffice` or `rnodern`. A dictionary and a character-sequence model catch these: the engine knows that in English `rn` at the start of a word is vanishingly rare while `m` is common, and it knows `Manager` is a word while `l\/lanager` is not.
This is why setting the right language matters so much, and why it is the single most effective adjustment available. An engine set to English reading a German page has a dictionary that fights every compound noun. It is also why proper nouns, part numbers, serial codes and addresses are the least accurate things on any page — they are precisely the strings the language model cannot help with, and it sometimes actively corrects them into something wrong.
05
Stage five: write it back invisibly
Here is the part that makes a searchable PDF work, and it is a genuinely elegant trick.
PDF has a text rendering mode operator, `Tr`. Mode 0 fills the glyphs normally. Mode 3 draws nothing at all — the text is placed on the page, occupies its position, and is completely invisible.
So the OCR output is written into the page as real text at mode 3, positioned so each recognised word sits exactly over the pixels it came from, with the original scanned image still drawn underneath. The page looks precisely as it did before. But search finds the words, selection highlights them in the right places, copy returns real characters, and a screen reader can read the document aloud.
This is why a searchable scan is barely larger than the scan it came from: the added text layer is a few kilobytes against megabytes of image. And it is why selecting text on an OCR'd scan sometimes highlights a slightly wrong rectangle — the invisible word is positioned from the recogniser's estimate of where the pixels were.
- The image stays
- You keep the exact appearance of the original scan. Nothing is redrawn.
- The text is real
- Search, copy, selection and screen readers all work on it.
- Errors stay hidden
- A misrecognised word is invisible, so the page never looks wrong — it just fails to be found by search.
- Size barely changes
- Kilobytes of text against megabytes of image.