Artificial intelligence¹ audit? This question is becoming real every day as AI acceleration continues in the consumer and enterprise spaces. In 2021, the awareness around AI by enterprises is increasing and this means evaluating this technology to understand the implications on the business bottom-line.

The explosion of big data and AI translates to vast amounts of information that need scrutiny to ensure companies use data for productive roles. Developing AI audit standards is critical as every business strives to understand the value of data and making them competitive in this age of digital transformation.

Governance and compliance around AI systems² has not come of age. However, enterprises continue to develop approaches for understanding and executing AI capabilities in their operations. A KPMG poll conducted in 2020 found that the C-suite prefers AI audits even after the pandemic to ensure their businesses thrive with this technology. That said, implementing an AI audit is an uphill task that requires solid IT procedures that enable management understand the workings of the AI technology in their operations.  

AI auditing is a step towards informing the C-suite about the value of artificial intelligence to the business. For example, the AI audit³ procedure should educate CEO’s on the importance of leveraging this technology as disruption happens everywhere. Accordingly, the AI audit exposes the risks involved to the business and stakeholders such as customers and developing safeguard controls.  

This paper will explore the checklist you need when auditing your AI systems:

1. AI Business Outcomes

To succeed with AI in your business, you need to understand the influence of this technology to your operations. Executives⁴ often complain that despite adopting AI technology, they still do not see the benefits in areas such as optimization, market growth and competitiveness. In other cases, business managers claim AI investments do not justify their expenditure. This is an example of a checklist every business manager should have when undertaking an AI audit. 

I recently interviewed VeIjko Krunic, the author of Succeeding with AI on the HumAIn Podcast and his words ring true with our discussion today. According to Krunic, businesses must act on the results of the AI technology and not let them gather dust on the shelves. When conducting a business AI audit, executives must find out the benefits that AI brings to the table and take note of them.

2. Data Sources

Why the data source? The data source should come into consideration when conducting an AI business audit. Some organizations derive data from within their departments but others use information from third parties. An AI audit requires clear definition of data sources⁵ if the audit is going to be successful and reflect the accurate state of technology used in the business.

There are instances where companies obtain data from outside sources and without proper vetting of the data; this becomes a problem for the business. A good audit approach in an organization means understanding your data sources and assurance that the information used for the audit is accurate. This is a good practice used in industries including banking, retail and healthcare.

3. Scope of AI Accuracy

Any audit should have the goal of establishing the accuracy of AI systems. Let us be honest. AI accuracy changes with adjustments in algorithms and data. At the same time, bias in artificial intelligence is a matter of concern that should encourage companies to evaluate the accuracy of their AI systems. This includes continuous evaluation and assessment of algorithms and data⁶ since vulnerabilities occur within AI systems after given periods.

For example, the IT department should explore data, queries and AI algorithms as these tell whether the AI systems are accurate or need modifications to improve them. An audit of AI systems with errors produces inaccurate results and business managers should not leave anything to chance. A minor error within the AI system could make or break the audit process and everything comes down to ensuring that data and queries are up to date.

4. Clean Data

This is an important checklist when conducting an AI audit. As saying goes ‘There is no AI without clean data’ and this includes ETL, data discards and data normalization. The IT department has a responsibility of ensuring that data used within the system is clean without errors. AI systems with unclean data have high chances of producing inaccurate results compared to clean data and enterprises should verify that data used is clean.

5. Privacy of Data

In this age of digital transformation, data privacy remains a sensitive topic as companies and consumers set expectations about the safety of their information. The C-suite⁷ in collaboration with technology leaders in the organization has a responsibility of ensuring data compliance and privacy. The rule of thumb for good AI systems is data privacy and the IT department should conduct an analysis to ensure implementation of privacy standards. Legal stipulations surrounding data usage and privacy protect the interests of consumers and organizations because of protecting sensitive information and data. Good AI systems account for the privacy factor and this should be a priority for every IT department assessing technology resources. 

6. IT Security and Controls

The value of any AI system includes the nature of security controls applied to data used in the system. In this era of malware attacks, enterprises face challenges developing security standards. The growth of AI technology brings opportunities as well as risks in equal measure. The same AI technology could give external hackers an opportunity to penetrate the system and cause damage.

The explosion of unstructured data from edge computing and IOT⁸ requires enterprises to up their security game to reduce incidences of attack. AI systems rely on tight security systems and the IT department should lead the way in fast tracking security around AI systems. Additionally, the idea of having devices from external parties with their own security protocols could serve as a loophole to the security status of AI systems of the organization.

Is your AI System Audit Ready?

Every organization must set procedures to accommodate changes in AI systems and developing best practices around collaborating with automated systems. AI systems come under risk many times during the life cycle of the organization and keeping tabs of these changes is a necessity to support AI projects. Organizations that neglect AI audit systems face risks because of surprises including malware attacks that happen without their knowledge. This is a costly approach that organizations should avoid and instead engage in continuous evaluation of AI systems.

A sustainable AI systems audit is pivotal for enterprises in their quest to harness the benefits of AI technology and understand KPI’s associated with this technology. The support from top-level executives creates a strong support for implementation of AI systems audit and this in turn pushes the business a step further. Overall, the C-suite and technology players within the enterprise should keep the conversation going by developing concrete polices around auditing of AI systems and overseeing their implementation. 

Works Cited

¹Artificial intelligence, ²AI Systems, ³AI Audit, ⁴Executives, ⁵Data Sources, ⁶Algorithms and Data, ⁷The C-Suite, ⁸Edge Computing and IOT