Introduction
In our earlier article, we explored the reference app released by Microsoft for Document Automation using the Form Processing model of AI Builder.
The app that is a ready-to-use sample app demonstrates the process of reading the form submitted through email and extracting the data submitted in the form and presenting it to the users in a Power Apps Canvas App for further manual verification and processing by the users.
In this part, we will use the flows to not only extract the data submitted using the form but also go ahead and automate the process for verifying the submissions if the confidence score returned is on the higher side.
Note: At the time of writing the blog this feature is not available in all regions. We have used an environment in the US region (.crm) for the purpose of this article.
Pre-requisite
The form processing model has already been trained on the sample forms that need to be processed. You can follow the steps explained in the earlier article to train the model.
For this example, we are using the model that has been trained to extract Name, Age, and Address from the forms submitted.
AI builder form processing in action
Let us walk through a scenario where we need to verify the document submitted to SharePoint against a predefined and trained AI model in AI builder. If the AI model is unable to confidently read the data extracted from the form, we will send an approval to the manager to review the document manually.
Start with a new Cloud flow and we will watch the ‘When a file is created’ trigger of the SharePoint connector.
Connect to the SharePoint site and set the document library to watch.
The next step is to read the file content. If the trigger does not return the actual file content use the Get file content action from the SharePoint connector and pass the Identifier that you receive in the earlier trigger as a parameter to identify the file to be read.
Now that we have the content, we pass it to the AI builder model to process the file. Look for the AI builder connector and choose the ‘Process and save information from forms’ action as shown below:
This action requires the file content, file type and model to be used for processing.
Choose your model from those listed in the dropdown. This will list all published models in the current environment.
In this case, we have hardcoded the form type to be PDF.
For the Form – we need to provide the actual file content. We get this as the output of the Get file content action. Choose file content from there as shown in the screenshot below:
This action returns the result of the fields trained by the selected model. Our model was trained to identify Name, Age, and Address from the form. It, therefore, returns the following for each of the fields:
The Name value has the actual name included in the form and the Name confidence score provides the accuracy level of the extraction. You see that this one is rated as 0.36 which means the accuracy level is low and the model wasn’t really sure of the data extracted.
We will read the confidence score of each of the fields and if any of these is less than 0.75, we will send out an approval request with the link to the file.
For the approval request, we provide the following;
Type – If you would like any one user to approve or do you want approval from all users that this request is being sent to.
Title – Title for the approval message.
Assigned to – The users to which the approval request needs to be sent. You can provide more than one user here and the message will be sent out to all the users listed here.
Item Link – Use the dynamic properties to select the link to the item from the SharePoint connector.
The approval request will be received on Flow mobile app.
The approvals can be viewed on the device as follows:
And can also be seen in the action items section of the flow portal.
This way you can quickly design a flow and automate the document verification process using the form processing model of AI builder.
Modifying the process in the Microsoft Document Automation app
If we go back to the Microsoft Document Automation app, we can modify the flows in this app to a similar logic as explained above.
Once the app has been installed, you will find the Document automation base kit solution listed in your solutions. This solution has all the assets shipped as part of the Document Automation app. We will specifically focus on the Power Automate Flows used to automate the processes.
1. Document Automation Email Importer
This flow triggers when a new mail is received and it reads the attachment files and creates a new record in the custom Document Processing table and uploads the file there.
2. Document Automation Processor
This flow triggers when the record is created in the Document Processing table after successful execution of the flow listed above.
This flow picks up the file that has been uploaded, reads the AI model to use that is stored in the configuration record, and then calls the Predict custom action of the AI model table and passes the file and model to it for processing.
We could edit this flow to process the file saved in the Document Automation Processing table using the AI builder action and logic similar to the one explained above and if the validation fails, update the Document Automation Processing record status to Manual Review or mark it Validated if the confidence score is above 0.75 for each of the extracted fields.
3. Document Automation Validator
This flow updates the status of the record in the Document Processing Table, we can skip this flow as we are handling the same in the above flow itself.
