Clinical decision-making in oncology involves multi-modal data such as radiology and clinical factors. In recent years, several computer-aided multi-modal decision systems have been developed to predict the recurrence of hepatocellular carcinoma (HCC) after hepatectomy, but these models simply concatenate features naively at the feature level, which would create redundancy and hinder model performance. Particularly, we found that integrating deep multi-modal models based on tensor fusion, which can better handle relevant information between different modalities and reduce model redundancy to obtain better results. This finding suggests the presence of independent, complementary prognostic information between radiology and clinical modalities. Hence we propose a multi-modal learning network that uses a tensor-based fusion approach with clinical parameters, radiomics features and corresponding complex interrelationships between pathological data to predict postoperative early recurrence of single hepatocellular carcinoma. In order to maximize the information gathered from each modality, we introduced a multi-modal fusion loss function based on orthogonal loss to assist multi-modal training. With IRB’s approval, we collected 176 cases with radiomics (MRI) and pathological features diagnosed by experienced clinicians, and established an approach based on tensor fusion that takes pathological findings and postoperative early recurrence as ground truth. Training with 140 cases and tested with 36 cases, the proposed network achieved the AUC of 0.883, which showed great potential in predicting postoperative early HCC recurrence. The ablation experimental results also showed that the auxiliary loss function had a statistically significant improvement in model performance (p<0.01).
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.