I am happy to announce that our article “Probabilistic Attention based on Gaussian Processes for Deep Multiple Instance Learning” (Arne Schmidt, Pablo Morales-Álvarez and Rafael Molina) was accepted in the journal IEEE Transactions on Neural Networks and Learning Systems, one of the best journals in the field of artificial intelligence. It is available as a preprint under https://arxiv.org/abs/2302.04061. In the article, we propose a novel attention mechanism for Multiple Instance Learning (MIL) based on probabilistic deep learning. We use Gaussian processes to regress each attention weight instead of deterministic matrix multiplication, as in common attention mechanisms. The Attention Gaussian Process (AGP) module provides efficient and precise attention weights and additionally estimates the uncertainty of the predictions.
As obtaining huge labeled datasets in digital pathology is a huge challenge, this novel MIL algorithm is another step forward toward AI algorithms that can be trained with limited data. In our experiments, the algorithm successfully predicts the Gleason Score of Prostate cancer biopsies with only global labels of each whole slide image available. Most importantly, the probabilistic uncertainty indicates the risk that a prediction is wrong – a very important feature for medical applications.
I wonder if probabilistic attention mechanisms can improve transformers. They are based on attention mechanisms and still largely rely on deterministic attention functions!
The proposed probabilistic Attention mechanism based on Gaussian processes.
Arne Schmidt – ESR8