A MULTI-STAGE MACHINE LEARNING FRAMEWORK FOR PROSTATE CANCER DIAGNOSIS BASED ON MPMRI IMAGING

Prostate cancer is one of the most common malignancies among men worldwide. Early and accurate diagnosis of the pathology plays a critical role in improving treatment outcomes. This thesis presents a comprehensive Machine Learning and Deep Learning ((ML)/(DL)) framework for prostate cancer diagnosis: it integrates zonal prostate segmentation, lesion segmentation, Prostate Imaging Reporting and Data System (PI-RADS) classification, and cancer detection using multi-parametric Magnetic Resonance Imaging (mp-MRI). The analysis relies on data from T2-weighted imaging (T2W), Apparent Diffusion Coefficient (ADC) imaging, and Diffusion-weighted Imaging (DWI), along with morphologic and biomarker-related clinical information. The proposed multi-stage workflow aims to support radiologists, reduce diagnostic variability, and possibly serve as an educational tool for medical trainees. The studied Deep Learning models were based on U-Net architectures. For peripheral zone (PZ) and central gland (CG) segmentation, we relied on the Prostate158 dataset of 3T mp-MRIs: we trained on 90 scans, validated on 25, and tested on 24. We considered Attention-Res-UNet, Vanilla-Net, and V-Net individually and also as an ensemble. Meta-Net and YOLO-V8 were also evaluated. We achieved high performance in providing anatomically precise delineations: YOLO- V8 achieved the best results, with a DSC of 89% for the central gland and 73% for peripheral zone segmentation. For lesion segmentation, we developed a dedicated dataset of 311 mp-MRI cases collected at the Centro Diagnostico Italiano (CDI), Milan, including 58 PI-RADS 3 and 253 PI-RADS 4–5 cases. The data set comprised T2W, ADC, and DWI sequences with manually annotated masks and was preprocessed by normalization and registration. Four deep learning architectures, U-Net, Dense U-Net, Attention U-Net, and LSTM U-Net, were evaluated using both single-modality and multi- input strategies. The Dense U-Net achieved the best performance, with a DSC of 69% on ADC images for PI-RADS 4–5 and 68% for PI-RADS 3–5. PI-RADS is a standardized scoring system used by radiologists to categorize prostate lesions based on their likelihood of clinically significant cancer. The PI-RADS machine learning classifier improved diagnostic consistency by extracting discriminative imaging features. Three approaches were evaluated for automated PI-RADS 3–5 classification using T2W, DWI, and ADC sequences: (1) hand-crafted radiomic features from manually segmented lesions, (2) a fully automated lesion and zonal segmentation pipeline, and (3) a custom convolution neural net- work learning high-level features from ADC images and masks. ADC-derived features performed best, with an ensemble model achieving 77% accuracy Accuracy (Acc), 83% AUC, and 0.618 Matthews Correlation Coefficient Matthews Correlation Coefficient (MCC), with PI-RADS 5 most reliably classified (AUC 94%), while PI-RADS 3 remained the most challenging. The cancer detection stage integrated imaging and clinical parameters, such as age, prostate-specific antigen (Prostate-Specific Antigen (PSA)) levels, and biopsy results, to enhance malignancy prediction. In total, 345 patients were included in the study, with data collected from Trita Hospital in Tehran, Iran. Using ADC images, the proposed model achieved the highest performance, with an accuracy of 83%, AUC of 87%, and a Matthews Correlation Coefficient of 0.638. A key contribution of this research is the creation of an end-to-end clinically inspired workflow that connects all diagnostic stages and includes, for practical deployment, a user-friendly 3D Slicer plugin that we contributed to developing. Although the size of the data set and the lack of multicenter validation still represent limitations of the work, the results highlight the potential of multimodal AI-driven approaches to improve diagnostic accuracy, standardize reporting, and help personalized patient care. This work establishes a foundation for future developments in AI-assisted prostate cancer diagnosis and clinical decision support systems.