A typical task for auditors is reviewing business and annual financial statements for completeness and consistency. Until now, auditors had to manually go through the documents to identify the relevant regulations for each section. This process is very repetitive and time-consuming, as there can be several hundred to thousands of potential requirements to meet, depending on the application area. In this article, we show how AI-based solutions can already automate parts of the auditing process today.
What is ALI and what can the tool do?
In collaboration with PricewaterhouseCoopers GmbH, Fraunhofer IAIS developed a solution aimed at automating the auditing process. This involves methods from Machine Learning (ML) and natural language processing (NLP), combined with the expertise of auditors.
“Automated List Inspection” (ALI) is a recommendation system that assigns text passages from unstructured documents, such as the appendix of an annual financial statement, to the corresponding legal regulations. ALI’s reference point is checklists that include all relevant legal requirements that the documents under review must meet. The recommendation system works in both directions: it can display both the relevant text passages for each legal requirement as well as the relevant legal requirements for each text passage. Additionally, it independently evaluates whether non-legally required information is also relevant and highlights this information. The final decision on which suggestion is correct is left to the auditors.
In the auditing practice, it is essential to stay up-to-date with current legal changes. To ensure that ALI continues to function smoothly with changes to the checklists, the system was designed to be non-specific. This way, it can be adjusted to changes in the checklists. The tool has been successfully used by PwC in everyday business since 2019, significantly speeding up the audit process (Link in German).
A screenshot of our recommendation system. The left column lists the checklist requirements; the middle column contains our system’s suggestions, sorted by relevance; and the right column shows the entire report, with the selected section highlighted.
Data protection & the development of ALI
There are often challenges when integrating AI solutions into the day-to-day operations of companies. The documents to be processed contain sensitive information, such as locations, names of individuals or companies, which allows their use and processing only for authorized persons and purposes. Typically, the use of the data for training algorithms is not included. Generally, regulations (especially GDPR) seem to be the biggest challenge for users to successfully implement ML or AI solutions.
To ensure the protection of sensitive information, we developed the Anonymizer tool in collaboration with PwC alongside ALI. This tool automatically identifies and anonymizes sensitive data in the relevant documents. An anonymization function was integrated into the ALI tool so that sensitive data is automatically recognized and obscured when verifying the correctness and compliance of financial documents. For those interested in learning more about the Anonymizer tool and its functionality, there is a blog post titled “AI in the financial sector – automated anonymization of financial reports“.
How does ALI work?
ALI’s main task is to automate the process of checking documents for compliance with legal regulations. Assuming the audit process follows a fundamental system, an ML algorithm can be trained to understand this system. Three components of the tool are particularly important: text preprocessing, data representation, and the underlying ranking model.
Text preprocessing standardizes natural language for the algorithm. Beyond the usual text preprocessing steps such as removing numbers, stemming, and lemmatization, application-specific preprocessing steps were performed. These steps include normalizing currency units, dates, citations of laws, and recognizing and protecting specific terms.
ALI can represent the given documents in different ways: n-grams, bag-of-words, and neural language models. This component serves to parse the given data and represent each legal requirement and its associated text passages for further text processing steps.
The last essential component is the ranking model. The ranking model assigns text passages to the respective requirements in descending order of relevance. This was trained with the available annotated data in both “unsupervised” and “supervised” manners. Unsupervised Machine Learning models have the advantage of being reusable with minimal effort when the checklist changes. However, we found that the performance of unsupervised models is inferior to that of supervised models.
The next steps towards automated auditing with ALI (BERT)
In summary, the use of AI for auditors offers unprecedented opportunities. ALI is the first step towards a significantly more efficient auditing process. Future research will focus on optimizing and further developing the tool, as outlined below:
A key task of ALI is checking the documents for consistency. It is important that relevant information not only exists but is also complete, correct, and consistently found in different parts of the documents. To meet these requirements, Fraunhofer IAIS is currently working on a consistency check module for ALI to further automate the qualitative analysis of the reports.
Besides the consistency check, user experience is another aspect that can be improved. ALI follows a ranking paradigm that suggests a fixed number of legal requirements per text passage. It does not consider whether a lower or higher number of legal requirements might be relevant for the respective text passage. This reduces clarity and ultimately the efficiency of the tool.
To improve user experience, we aim to use a (multi-label) classification approach instead of the current ranking approach, dynamically adjusting the number of presented suggestions to the respective text passages. We are using a pre-trained BERT language model, which allows ALI to be optimized and adapted more easily end-to-end. With BERT as the architecture, a universally applicable model can be developed in the future, usable for various checklists, fields, and different languages (here is the link to the recently published paper). The future holds exciting developments!
