Document processing traditionally relied upon OCR, templates and machine learning to get the job done. Whether your document was a structured layout such as a form or a semi-structured layout such as an invoice, intelligent document processing used 20-year-old tech to recognise and extract the required meaning and data.
With the rise of generative AI and large language models came a focus on using new technology to solve old problems:
How do we recognise, classify and extract data from incoming documents?
Whatever you opinion on probabilistic vs deterministic data extraction, AI in document processing is here to stay.
One of the major stumbling blocks to using AI to extract meaning from documents is the loss of context and structure. Documents are mostly formatted for rendering nicely into a design that someone can see, read and understand using eyes and a brain. Not for feeding into a large language model. AI pipelines want clean, structured inputs. Parsers reconstruct what they can. Reading order is guessed, tables are flattened, figures disappear and metadata is stripped away. Pulling text from a document and throwing it at an LLM and expecting to get accurate, consistent understanding back is an exercise in frustration.
So how do we best make a document machine readable?
DocLang gives documents a standard structure that machines can actually understand.
Every part of the document keeps its meaning, position and reading order. Tables keep their structure. Headings know what level they are. Figures stay where they belong.
The important bit is that the model does not have to guess what it is looking at.
The invoice shows the difference clearly. A normal parser may return the words, but lose the relationships between the heading, table, totals and chart.
DocLang keeps that structure, layout and reading order intact. It does not decide that something is an invoice or understand the business meaning for you but it does give the AI a better representation to work from.
Now we understand the benefits of DocLang, how do we create DocLang output at scale? Abbyy FineReader.