Human-In-The-Loop is not enough. The human must be efficient!

Why a Human-in-the-Loop (HITL) screen is not a simple yes/no feature for Intelligent Document Processing

No IDP solution can guarantee 100% data extraction accuracy. So a human needs to look at the data. Many IDP vendors have added HITL-applications into their solutions to allow for manual review and correction. However, just “having it” is not enough. It is surprising how few vendors have a really sophisticated HITL application. Most claim that “stopping the document based on confidence” is enough. This means, if a field was only 67% certain, a human needs to look at it. But such a confidence score is arbitrary. What does it mean, if a deep neural network extracted a value with 67% certainty? Can I trust that? Can I trust 99% certainty? Shouldn’t I review all fields to be safe?

Let’s look at how some of the big cloud players describe their HITL interfaces

Google Document AI

• Confidence threshold filters limit the number of documents going through HITL.

• Labeler pool management, including task assignments and efficiency analytics by task and labeler.

• UI cues and features that reduce labeler handling time per document.

• Analytics and metrics by task and labeler, so you can streamline HITL operations.

Amazon Textract

“Choose a confidence threshold for your application, and all predictions with a confidence below the threshold are automatically sent to human reviewers for validation. You can also specify which key-value pairs should be sent for human review or you can send randomly selected documents for review as well. “

Amazon A2I human in the loop interface

Confidence is not enough

Arbitrary confidence values don’t help to decide what needs to be reviewed. E.g. there could be a typo on an invoice that puts the invoice date into the future. Any IDP’s extraction engine can extract such a date with 100% confidence. That means the engine is 100% certain it extracted what is printed. But it is still wrong because the invoice date must be in the past. So the first thing you need before the HITL even comes into play is business rules. Your IDP platform must allow you to define rules to verify extracted data. Do all the amounts add up? Is the DOB of an adult subscriber 18 years in the past? Is the due date after the invoice date?

Presenting only what is really needed to human operators

Business rules help to trust the system. Extraction confidence alone cannot achieve that. Now that the operators look at the really important fields, the actual HITL interface becomes important. Think about it: You automate 80% of your extraction so only 20% of the documents need to be reviewed. That’s a lot of time saved, so mission accomplished? No! The remaining 20% can be really expensive to review. It makes a big difference if the operator spends 1 or 5 minutes on a document. That’s potentially another 80% saved!

The bleakest HITL applications, and that’s unfortunately most of them in the market, just display the fields as a long list. You need to click on each uncertain field and key it in. They fulfill the RFP checkbox “Have HITL application? YES”. But that’s not enough.

Must-have HITL features

The most advanced HITL interfaces have features that boost operator performance immensely. Here is what we think are mandatory HITL features for high document volume IDP projects. If your vendor of choice doesn’t have any of them, keep looking!

• Automatic navigation. When you correct or confirm a field it should take you to the next field in the list that is uncertain. You shouldn’t have to scroll and look for it.

• Highlighting. When a field is edited, it should be highlighted in the image so you know where it was found. When the current field is not visible in the image, the UI should automatically scroll and zoom it into focus.

• Image interaction. The UI should allow you to click and lasso values in the document image instead of keying them in.

• Autocomplete. When typing, existing words and phrases in the document should be suggested as you type.

• Lookups. Instead of typing entire addresses, you should have a lookup form that populates the data from a database.

• Business rules. They should not only be applied during automatic extraction, but every time you change a field so you can be sure you didn’t again mistype.

• Formatting. You shouldn’t have to worry about typing a date or amount in the correct format when manually keying it. Whatever format you type or whatever you click in the image, the final formatting should be done by the UI automatically and instantly.

• Keyboard friendly. Some of your operators are keyboard wizards. It should be possible to do the entire review and correction without a mouse

• Line item support. Correcting line items can be a lot of manual effort. If the IDP missed an entire table with 300 line items, an operator can easily spend hours keying in the data. Good HITL applications allow you to lasso a table in the image or infer line items from one or two examples that you manually key in. “I showed you a line item, now do the rest, machine!”

Thinking beyond the document

Most modern IDP products that focus on simplicity are often too simple for advanced use cases. If you process documents that belong to a case, most of the newer IDP products cannot help because they all focus on one document at a time. But what if you need to assure that the customer ID is the same on all documents in a case? What if you need to add up amounts across all documents in a folder? The most advanced IDP products support such features. Their HITL interface shows entire batches or cases and displays cross-document data in fields that can be corrected and reviewed as well. You need to think about whether you need such features and make sure your RFPs cover this to find the best vendor.

So, make the lives of your human operators easy and fun! Don’t burden them with a clunky UI because even those 20% remaining documents that the AI didn’t get right can be a real pain to correct.