Once I developed the whole pipeline regarding the images, I was able to start developing the Deep Learning models for my own research, with the objective of automatically extracting relevant features from the WSIs for the diagnosis and prognosis of the TNBC.
Since I started my journey as part of the CLARIFY project in February 2021, I have strengthened several skills that I already had and I have learnt new ones related to my research. During the first months, I had my first contact with the field of Medical and Computational Pathology, which was completely new for me until that moment.
One of the first tasks that I had to do was to develop the Annotation Protocol for the Triple Negative Breast Cancer (TNBC) Whole Slide Images (WSIs) together with ESR11 Umay Kiraz. This first task allowed me to involve myself into the digitalized images from the TNBC biopsies which, together with the annotations made by ESR11, I would later have to use for training and testing my Deep Learning models. The result was a 23-page document which established the guidelines on how to annotate the TNBC WSIs, including descriptions of all the tissue types that could be found, two versions of the Annotation Protocol (Exhaustive and Non-Exhaustive) and descriptions about the Pathology TNBC grading and staging.
Using the aforementioned Annotation Protocol, ESR 11 exhaustively annotated 9 TNBC WSIs, with special focus on the mitoses. As a result, we came up with an initial dataset of annotations which I would use to develop a Mitoses Detection Deep Learning Model using Convolutional Neural Networks (CNNs). In order to be able to develop such a model, I had to learn to deal with the annotations from the database and process the WSIs using Python and MATLAB for extracting the patches and creating the annotation masks. This kind of images are not trivial to process as they are considerably heavy and they include different levels, according to the different magnifications that pathologists have to work with when using the microscope.
Once I developed the whole pipeline regarding the images (conversion, upload to the annotation tool, recovery or the annotations, translation of the annotations to masks, bounding box and patch extraction), I was able to start developing the Deep Learning models for my own research, with the objective of automatically extracting relevant features from the WSIs for the diagnosis and prognosis of the TNBC.
The first research line that we decided to tackle was the automatic Mitoses Detection in the WSIs, as the Mitotic Activity Index (MAI) has a very important prognostic factor in TNBC and for the pathologists, the mitotic count can end up being a quite tedious process. Therefore, for the first approach I built, trained and tested a CNN (UNet) in TensorFlow trained only with patches that contained mitoses, which was able to successfully segment the mitoses in the patches extracted from the TNBC WSIs. However, in a real scenario it is important to also consider the patches that don´t contain mitoses, and that was our next step. In this case using PyTorch, a different Python library for developing Deep Learning models, we also developed a model able to detect and segment mitoses using patches with and without mitoses.
The following objectives correspond to reconstruct the WSIs and the predictions from our model, in order to be able to provide the pathologist with a heatmap overlaid on the WSI which will show which areas of the WSI are more probable to containing mitoses hotspots, making the process of coming up with the MAI in a more efficient and less time-consuming way. Also, other objectives considered for the future include being able to recognize other tissue types (i.e. Tumor Infiltrating Lymphocytes) and to be able to relate these image features to the prognostic outcome of the patient and to being able to estimate if a patient could be a potential candidate for different types of treatments.
“Claudio Fernandez (ESR6) and Umay Kiraz (ESR11) together during the first day of Claudio´s secondment in the Stavanger University Hospital in Norway with the WSI scanner.”
Claudio Fernández – ESR6.