Multi-Modal Deep Learning Architecture for Medical Signal and Image Fusion in Clinical Decision Support

Abstract

Clinical diagnosis is becoming more based on heterogeneous information like medical images and physiological signals, but the majority of current clinical decision support systems are analyzed in isolation, resulting in a piecemeal interpretation and poorer diagnostic dependability. This paper suggests a multi-modal deep learning system that would incorporate both medical imaging and biomedical signal information into one system. The model is a combination of CNN-based spatial feature encoding given the MRI images and Bi-LSTM-based temporal modelling involving ECG signals, and an attention-guided cross-modal fusion mechanism. The MRI brain tumor and MIT-BIH Arrhythmia datasets were publicly available and were experimentally evaluated with a training validation test split of 70:15:15. The proposed model had an accuracy of 95.1%, precision of 0.95, recall of 0.94, F1-score of 0.94, and AUC of 0.97, which was better than image-only (88.4) and signal-only (84.9) baselines by 6-10%. The attention-based fusion showed a significant enhancement in the discriminative ability and reduced false predictions. The findings have shown that multi-modal learning can be used to increase robustness, reliability, and clinical applicability. This paper contributes to the relevance of integrated cross-modal design in enhancing smart healthcare decision support systems.