Residential Magazine

Visual Confirmation

Similar technology to facial-recognition software can be used to process loan documents

By John Martin

Nearly every mortgage file generates hundreds of pages of documents with information that needs to be double-checked and verified. Obtaining consistent, reliable information from these documents is key for audits and compliance monitoring, among other things. 

Mortgage companies have relied on automation to aid in processing these files, but it has often proven expensive and ineffective. Fortunately, another way of vetting these documents — already in use in other industries — may virtually eliminate this as an obstacle. Originators will want to pay close attention to an approach that could speed up loan processing and help to close deals sooner.

For many years, mortgage companies have expended significant human effort to augment automation in extracting information from loan-file documents. Automation often starts loan files on this journey from data to useful information, but the last miles of the trip are invariably covered on foot. And for many problematic documents, the whole journey is a tedious, manual effort.

There are two underlying causes for this. One is the large number of diverse mortgage documents. Think about it: There are literally tens of thousands of cities, counties, businesses and other entities that create these documents, which are usually captured in an image-only format.

The second cause is that most automation programs use optical character recognition technology. This is a fancy way of saying these programs try to convert text characters into codes that can be used in data processing. It’s a situation where the underlying documents themselves are diverse and shifting, and the textual representations of these documents are often marginal at best. 

Mortgage companies spend endless hours trying to do this task. Any time new document types are added, companies must test to make sure the new document types don’t cause either false positives (i.e., assigning a classification to a document that should be called something else) or false negatives (i.e., missing a classification). This effort becomes exponentially more difficult — and more expensive — with a greater number of document types. There may, however, be a better way.

Classifying visually

The latest approach to handling these documents is visual-classification technology. It works like facial recognition or fingerprint classification by initially grouping similar documents based on their overall visual appearance, not by any combination of key terms. One huge differentiator compared to many text-based systems is that the clusters of visually similar documents are self-forming. 

This means that no human resources or upfront effort is needed to develop or maintain text-based classification rules or templates. When new document types start appearing in the document stream, subject-matter experts are notified. And when the new classifications are assigned, documents that are in that cluster (or are later placed in the cluster) receive the same classification.

Visual classification was initially developed for use in the oil and gas industry, which has unique challenges due to documents that can be centuries old or have odd dimensions, such as engineering diagrams that can be multiple feet in width and length. At the other extreme, some documents such as delivery tickets can be quite small — only a few inches in length and width. Techniques and approaches developed in dealing with such diverse documents are largely reusable when dealing with documents found in mortgage files.

Sorting documents

Solid, dependable document classification can answer some basic questions about a loan file. For instance, it will show whether there is a note, deed, mortgage insurance, property insurance, credit report, credit-report authorization and closing disclosure. 

Most loan-management systems or systems of record need specific data elements, including the borrower’s name, Social Security number, name of employer, current address, date of birth, etc. These data elements are attributes displayed on many different documents, and they need to be extracted to either populate or validate information contained in the system of record.

Accurate attribution is dependent on valid, consistent document classification because specific data elements are normally associated with specific document types. For example, a credit report should have the borrower’s name and Social Security number, but not the name of the property appraisal company.  

Knowing what document types are present lets you know what data elements are either present or missing. To continue the example, you wouldn’t have a loan analyst look for the name of the appraisal company on a credit report because it’s not missing from the report — it was never expected to be there in the first place. Proper classification lets companies focus human resources where they are needed.

Clustering groups

Visual classification works at the level of individual graphical elements or glyphs, as well as at the page level. At the character level, visual-classification technology groups or clusters individual glyphs based on visual appearance. 

All the characters in the cluster are assigned the appropriate text value. Because the classification is persistent, companies do not need to perform this assigning of text values to the character-level glyphs because that work has already been done.

Most mortgage-file documents use labels to identify the data elements contained within (e.g., “applicant” on a loan application or “borrower” on a closing disclosure). Visual classification can output both the labels and the data elements. A visual-classification system also can normalize the labels so that the same label will be used to identify the same data types.

Most loan-management systems track information at the loan level, not the document level. For example, there is one interest rate for the loan, but interest rates show up on multiple documents, such as loan estimates, closing disclosures and notes. 

Loan files can have dozens of earlier drafts of these documents before the final versions. With data extracted from each document, mortgage companies can apply business rules about what information to use and what information to manually evaluate. 

For example, if there are different rates on the disclosure and the note, a loan analyst might be tasked to evaluate the variance.

Validating data

The technology underlying visual classification enables multiple levels of data validation. Visual classification knows the page and page-coordinate information of every glyph or graphical element. Mortgage companies can do high-level validation of data contained in a mortgage system of record by searching the loan file for occurrences of these values. 

For example, if a file indicates the interest rate is 3.75%, this rate should appear on multiple documents and the business can develop its own rules about threshold values that would trigger a loan analyst’s review. For instance, in a U.S. Department of Veterans Affairs loan file with 700 pages, 3.75% should appear at least “X” number of times. Mortgage companies also can use data derived from visual classification to develop more nuanced validation (e.g., to report the types of documents in which the search term was located, or to report if other values appeared for the labels associated with the search term). To illustrate this point, if the search was for “Allison B. Brown,” the search could report back that this term appeared as paired with the label “borrower,” and that another term that appeared paired to “borrower” was “A. Britta Brown.”

● ● ●

Too many automation software programs rely on optical character recognition technology. Often, these programs are ineffective and require back-office employees to do tedious and costly work to verify the information. A new approach that visually classifies documents and the information on the page may prove to be a better way of doing this work. 


  • John Martin

    John Martin is CEO and founder of BeyondRecognition LLC, which develops visual-classification technology for documents from a variety of industries. Martin’s earlier development work resulted in a U.S. patent for an electronic discovery system, and he has served as a consulting and a testifying trial expert on e-discovery. Visit

You might also like...