Why can’t AI say ‘I don’t know’?

Artificial intelligence models like ChatGPT can be powerful tools, but they’re also often overconfident. AI tools are poor at estimating the uncertainty in their predictions and most aren’t able to say ‘I don’t know’ in response to an unfamiliar situation.

‘This overconfidence becomes an issue in safety-critical applications, such as self-driving cars, where poor decision-making can lead to severe consequences,’ says Lassi Meronen, a doctoral researcher at Aalto University who’s working on ways to address this problem. ‘For example, a deep learning model could be helpful in automatic medical diagnosis from imaging data if it could accurately determine 90% of the cases, leaving only 10% as uncertain cases that require checking by a doctor. But if the model can’t tell which cases are uncertain, the doctor would need to go through all the cases anyway.

Meronen’s doctoral thesis focused on making deep learning models better at estimating the uncertainty in their predictions, which would let them be used more widely in safety-critical applications.

‘My main results stemmed from building mathematical connections that bring the benefits of principled probabilistic models into deep learning models,’ says Meronen.

Improving the uncertainty estimates would also allow deep learning models to distinguish between complex and simpler inputs, says Meronen. This would help save energy by using greater computational resources only when they’re needed – with the more complex inputs. This smart allocation of resources is important because deep learning models are getting bigger and take more and more energy to run.

Cooperation with Saab

Meronen’s thesis, which he recently defended, included several high-impact scientific articles, two of which were accepted to the most prestigious academic conference in AI research, the Conference on Neural Information Processing Systems (NeurIPS) in 2019 and 2021. His research also involved close collaboration with Saab under the auspices of a ten-year strategic research partnership with Aalto, which was launched in 2017 and expanded in 2021 to accommodate more research areas. The partnership is worth 23 million euros and has already resulted in dozens of scientific publications and several patent applications. 

‘When we were considering a new research partner, Aalto University stood out,’ says Petteri Alinikula CTO at Saab Finland. ‘Aalto has exactly the knowhow that Saab needs, and their attitude towards industrial cooperation is excellent. Our goal is to train at least 20 new PhDs during this program.’ 

Meronen’s collaboration with Saab has focused on safety-critical deep learning models. 

‘Saab deals with many applications that could benefit from the use of deep learning models. However, most of these applications are safety-critical, which makes research like mine quite relevant. In my collaboration with Saab, I’ve worked on automatic radar signal classification that can be used to recognise aircraft, for example. Such a system must be good at estimating uncertainty, since overconfident and incorrect classification of an aircraft could have severe consequences,’ says Meronen.

This news item was originally published on the Aalto University website on 15.12.2023