M.Sc. Lalli Myllyaho defends his doctoral thesis "On Quality Assurance of Intelligent Systems" on Friday the 18th of August 2023 at 12 o'clock in the University of Helsinki Physicum building, Auditorium E204 (Gustaf Hällströmin katu 2, 2nd floor). His opponent is Professor Tapio Salakoski (University of Turku, Finland) and custos Professor Tomi Männistö (University of Helsinki). The defence will be held in Finnish.
The thesis of Lalli Myllyaho is a part of research done in the Department of Computer Science and in the Empiricial Software Engineering group at the University of Helsinki. His supervisors have been Professor Tomi Männistö and Professor Jukka K. Nurminen (University of Helsinki) as well as Professor Tommi Mikkonen (University of Jyväskylä).
On Quality Assurance of Intelligent Systems
Systems utilizing artificial intelligence provide us with numerous opportunities, allowing these intelligent systems to adapt to new situations and contexts. At the same time, this adaptability raises uncertainties concerning system dependability, such as reliability and security, of these systems. Some of the techniques are easily implementable with little domain knowledge, which combined with the difficulty to test and oversee the intelligent systems has made the quality assurance a pressing issue.
In this dissertation, we study the methods used to validate intelligent systems, fault tolerance in these systems, and potential new error detection approaches. The goal is to classify and describe validation methods and fault tolerance patterns that are used to ensure the dependability of intelligent systems, along with introducing a novel error detection technique for further development.
Multiple research methods were used. A systematic literature review was conducted to study how practical intelligent systems are validated in the research literature. An interview study was conducted to study how experienced software architects approach fault tolerance in intelligent systems. Two of the design patterns described in the interview study were implemented in a design science artifact in an industrial use case to see their effectiveness. The new error detection approach called node co-activations was studied in an experimental setup.
The validation methods were synthesized into a taxonomy consisting of trial, simulation, model-centred validation, and expert opinion. A set of fault-tolerant design patterns were synthesized based on the interviews with the software architects. The implemented approaches showed that they can make results of even simple, crude models useful in an actual industrial setting. The rare co-activations show potential in detecting concept drift and a new kind of data entering the system.
In conclusion, intelligent systems have problems that are out of traditional software testing's reach, and require specialised approaches. Quality assurance of intelligent systems should include considering the entire system, and not just the intelligent components. This can be achieved by validation and introducing fault tolerance into the system. Further research and development is needed to answer the growing complexity of the intelligent systems, and the growth's implications on error detection, infeasibility of validating entire intended environments, and the expanding role of intelligence in new software.
Availability of the dissertation
An electronic version of the doctoral dissertation will be available on the e-thesis site of the University of Helsinki at http://urn.fi/URN:ISBN:978-951-51-9366-7.
Printed copies will be available on request from Lalli Myllyaho: lalli.myllyaho@helsinki.fi.