As society rapidly accelerates to the ubiquitous adoption of artificial intelligence (AI) we need to be sure that these systems can be trusted, that is we need a robust approach to the assurance of their behaviour. We need to be certain that whether its Netflix recommending your next movie or a Tesla car autonomously driving you to the cinema for your next movie, there that is no risk to the community.
The Government of NSW has recently published their AI Assurance Framework that makes significant progress towards ensuring risks are minimised. It does though not adequality address a key component. There is no robust consideration in the Framework of its application to the engineering (and acceptance) of AI-augmented autonomous systems such as the Tesla car.
The NSW Government AI Assurance Framework
The NSW Government AI Assurance Framework assists agencies in designing, building and using AI-enabled products and solutions. Coming into effect in March 2022, the framework is intended to be used to identify specific risks that arise from the use of AI technologies and to ensure these technologies are used ethically and appropriately. The Framework is intended to be used before deploying custom AI systems, customisable AI systems, and for projects developed using generic AI platforms. History has shown these AI systems may be subject to biases affecting their decision-making capability and may cause unintentional damage and the framework aims to enable these AI systems to be used safely across government with the right safeguards in place and ethical considerations4.
The framework consists of a series of question/answer pairs and requires the user to apply the following principles to their proposed AI project: Community Benefit, Fairness, Privacy, Transparency, Accountability. While this framework contains many valid and useful considerations that should be applied to evaluating a project’s overall benefit, there remain several areas where it falls short. Looking at the framework from the perspective of an engineer that is implementing an AI-enabled solution, there exist a number of additional considerations that are not well covered.
An illustrative application is the implementation of AI technologies within transport networks.
Consider the following.
- The Sydney metro has a fully automated and driverless train system connecting Tallawong to Chatswood (1, 2).
- It stretches 36 km and consists of three stations (1, 2).
- 30 kms of extension are the works, for the train system to reach the city centre by 2024 (1, 2).
Lessons to learn
On the day of opening, one metro train overshot a platform and then failed to open its doors which caused delays across the line (3). This is only one example of automation system failure, and we can easily imagine others. What if someone was to run in front of the moving autonomous train? How will the system handle other unexpected events? If it is using machine learning models (for instance, as a computer vision capability), how do we ensure the model was trained on all the relevant operational scenarios?
These unresolved questions around system assurance become even more important when once we start to apply autonomous control technologies to transport that operates in complex, less homogenous environments. Consider all the unpredictable events that would be encountered by autonomous buses and cars operating on city streets. How can we ensure their safe operation at all times?
Current efforts to develop self-driving cars rely heavily on machine learning models trained on large datasets. Even with huge amounts of training data (collected via many thousands of hours of operation on city streets), a key learning by the companies developing these systems is that operational edge cases (rare events) present a major challenge to safe operation of autonomous vehicles. In addition, the unpredictability of AI due to the non-deterministic and evolving performance and leads to a difficulty in placing trust in the system and its interactions with its operating environment. Safe operation is clearly a primary requirement to evaluate the community benefit introducing any new autonomous system.
When we think about these issues, we are presented with a set of considerations that need to be made towards physical harm and safety of life, with implications for the integration of AI system in safety critical environments. A more comprehensive AI Assurance framework would aim to identify a set of risks from a technical and engineering perspective – while still ensuring the principles discussed earlier (e.g. ethics, privacy) are being adhered to.
Achieving reliable and trustworthy AI systems with assurance
Achieving reliable, trustworthy AI systems requires assurance of the algorithm and data being used to train the AI system. The algorithm used needs to be transparent in a way where decisions made by the AI model are explainable (where possible). For instance, if the decision made by the model differs at various times through the evolution of the model over time, can that be explained?
Moreover, given the operation of an AI model within a complex and critical environment, and the randomness associated with the decision model, it is important to check the assumptions around the performance of the model and ensure that the system is fail-safe. Hence, the performance adequacy of the model should be a core focus. As a result, the operational and environmental requirements of an AI system need to be well understood through incorporation of engineering design principles which include system assurance and testing.
While the NSW AI Assurance framework is a step towards the right direction, further efforts need to be realised to ensure effective utilisation of AI systems in safety critical environments.
Shoal understands that assurance of AI systems is a key component of the engineering lifecycle for these emerging systems and is currently working on in this space through its ASSURED-M and SHARC projects.
Read further articles on AI