摘要 :
Deep learning methods have achieved significant results in various fields. Due to the success of these methods, many researchers have used deep learning algorithms in medical analyses. Using multimodal data to achieve more accurat...
展开
Deep learning methods have achieved significant results in various fields. Due to the success of these methods, many researchers have used deep learning algorithms in medical analyses. Using multimodal data to achieve more accurate results is a successful strategy because multimodal data provide complementary information. This paper first introduces the most popular modalities, fusion strategies, and deep learning architectures. We also explain learning strategies, including transfer learning, end-to-end learning, and multitask learning. Then, we give an overview of deep learning methods for multimodal medical data analysis. We have focused on articles published over the last four years. We end with a summary of the current state-of-the-art, common problems, and directions for future research.
收起
摘要 :
Many different direct volume rendering methods have been developed to visualize 3D scalar fields on uniform rectilinear grids. However, little work has been done on rendering simultaneously various properties of the same 3D region...
展开
Many different direct volume rendering methods have been developed to visualize 3D scalar fields on uniform rectilinear grids. However, little work has been done on rendering simultaneously various properties of the same 3D region measured with different registration devices or at different instants of time. The demand for this type of visualization is rapidly increasing in scientific applications such as medicine in which the visual integration of multiple modalities allows a better comprehension of the anatomy and a perception of its relationships with activity. This paper presents different strategies of direct multimodal volume rendering (DMVR). It is restricted to voxel models with a known 3D rigid alignment transformation. The paper evaluates at which steps of the rendering pipeline the data fusion must be realized in order to accomplish the desired visual integration and to provide fast re-renders when some fusion parameters are modified. In addition, it analyses how existing monomodal visualization algorithms can be extended to multiple datasets and it compares their efficiency and their computational cost.
收起
摘要 :
Electronic Health Records (EHRs) significantly enhance clinical decision-making, particularly in safe and effective medication recommendation based on complex patient data. Current methods, while encoding each medical event indivi...
展开
Electronic Health Records (EHRs) significantly enhance clinical decision-making, particularly in safe and effective medication recommendation based on complex patient data. Current methods, while encoding each medical event individually with domain-specific knowledge, inadequately harness multi-source domain knowledge and neglect the interrelations among various medical codes, the influence of historical patient visits, and the relevance of similar patient trajectories. To address these limitations, we present PROMISE, a multimodal medication recommendation framework that progressively learns patient representations from specific health states to a comprehensive view. PROMISE integrates domain knowledge into modality-specific encoders to improve local and global patient representations, facilitating enhanced medication recommendations through the interaction of patient representations from various modalities. Specifically, within the code modality, PROMISE utilizes encoding of EHR hypergraphs to learn patient representations featuring structured information. Simultaneously, in the text modality, it acquires patient representations with semantic information by encoding clinical texts obtained from tables. Our framework surpasses state-of-the-art baselines with up to 2.06% and 1.97% improvements on key metrics within the MIMIC-Ⅲ and Ⅳ datasets, respectively, confirming its effectiveness and superiority.
收起
摘要 :
Medical imaging is fundamental to modern healthcare, and its widespread use has resulted in the creation of image databases, as well as picture archiving and communication systems. These repositories now contain images from a dive...
展开
Medical imaging is fundamental to modern healthcare, and its widespread use has resulted in the creation of image databases, as well as picture archiving and communication systems. These repositories now contain images from a diverse range of modalities, multidimensional (three-dimensional or time-varying) images, as well as co-aligned multimodality images. These image collections offer the opportunity for evidence-based diagnosis, teaching, and research; for these applications, there is a requirement for appropriate methods to search the collections for images that have characteristics similar to the case(s) of interest. Content-based image retrieval (CBIR) is an image search technique that complements the conventional text-based retrieval of images by using visual features, such as color, texture, and shape, as search criteria. Medical CBIR is an established field of study that is beginning to realize promise when applied to multidimensional and multimodality medical data. In this paper, we present a review of state-of-the-art medical CBIR approaches in five main categories: two-dimensional image retrieval, retrieval of images with three or more dimensions, the use of nonimage data to enhance the retrieval, multimodality image retrieval, and retrieval from diverse datasets. We use these categories as a framework for discussing the state of the art, focusing on the characteristics and modalities of the information used during medical image retrieval.
收起
摘要 :
The brachial plexus is a complex network of peripheral nerves that enables sensing from and control of the movements of the arms and hand. Nowadays, the coordination between the muscles to generate simple movements is still not we...
展开
The brachial plexus is a complex network of peripheral nerves that enables sensing from and control of the movements of the arms and hand. Nowadays, the coordination between the muscles to generate simple movements is still not well understood, hindering the knowledge of how to best treat patients with this type of peripheral nerve injury. To acquire enough information for medical data analysis, physicians conduct motion analysis assessments with patients to produce a rich dataset of electromyographic signals from multiple muscles recorded with joint movements during real-world tasks. However, tools for the analysis and visualization of the data in a succinct and interpretable manner are currently not available. Without the ability to integrate, compare, and compute multiple data sources in one platform, physicians can only compute simple statistical values to describe patients behavior vaguely, which limits the possibility to answer clinical questions and generate hypotheses for research. To address this challenge, we have developed Motion Browser, an interactive visual analytics system which provides an efficient framework to extract and compare muscle activity patterns from the patients limbs and coordinated views to help users analyze muscle signals, motion data, and video information to address different tasks. The system was developed as a result of a collaborative endeavor between computer scientists and orthopedic surgery and rehabilitation physicians. We present case studies showing physicians can utilize the information displayed to understand how individuals coordinate their muscles to initiate appropriate treatment and generate new hypotheses for future research.
收起
摘要 :
The recent pandemic has revealed the urgent need for lung disease diagnosis at early stages in humans. Deep learning-based automatic diagnosis methods typically rely on single-modality data such as medical imaging. However, analys...
展开
The recent pandemic has revealed the urgent need for lung disease diagnosis at early stages in humans. Deep learning-based automatic diagnosis methods typically rely on single-modality data such as medical imaging. However, analysis of single modality data is not so reliable to diagnose the disease at its early stages. Clinical data, blood tests together with imaging methods are very powerful and reliable sources to detect the presence of disease in the human body. This study attempts to use medical imaging data with clinical information to develop a multimodal fusion approach to detect lung disease. Two architectures of multimodal network based on late and intermediate fusion is proposed. Besides, an approach of adapting batch size is also introduced. Experiments show that the performance of intermediate fusion is better than the late fusion model with both direct and adaptive batch size approach.
收起
摘要 :
Modern medical information retrieval systems are paramount to manage the insurmountable quantities of clinical data. These systems empower health care experts in the diagnosis of patients and play an important role in the clinical...
展开
Modern medical information retrieval systems are paramount to manage the insurmountable quantities of clinical data. These systems empower health care experts in the diagnosis of patients and play an important role in the clinical decision process. However, the ever-growing heterogeneous information generated in medical environments poses several challenges for retrieval systems. We propose a medical information retrieval system with support for multimodal medical case-based retrieval. The system supports medical information discovery by providing multimodal search, through a novel data fusion algorithm, and term suggestions from a medical thesaurus. Our search system compared favorably to other systems in 2013 ImageCLEFMedical. (C) 2014 Elsevier Ltd. All rights reserved.
收起
摘要 :
With the rapid technological advances in acquiring data from diverse platforms in cancer research, numerous large scale omics and imaging data sets have become available, providing high-resolution views and multifaceted descriptio...
展开
With the rapid technological advances in acquiring data from diverse platforms in cancer research, numerous large scale omics and imaging data sets have become available, providing high-resolution views and multifaceted descriptions of biological systems. Simultaneous analysis of such multimodal data sets is an important task in integrative systems biology. The main challenge here is how to integrate them to extract relevant and meaningful information for a given problem. The multimodal data contains more information and the combination of multimodal data may potentially provide a more complete and discriminatory description of the intrinsic characteristics of pattern by producing improved system performance than individual modalities. In this regard, some recent advances in multimodal big data analysis for cancer diagnosis are reported in this article.
收起
摘要 :
Insufficient training data is a common barrier to effectively learn multimodal information interactions and question semantics in existing medical Visual Question Answering (VQA) models. This paper proposes a new Asymmetric Cross ...
展开
Insufficient training data is a common barrier to effectively learn multimodal information interactions and question semantics in existing medical Visual Question Answering (VQA) models. This paper proposes a new Asymmetric Cross Modal Attention network called ACMA, which constructs an image-guided attention and a question-guided attention to improve multimodal interactions from insufficient data. In addition, a Semantic Understanding Auxiliary (SUA) in the question-guided attention is newly designed to learn rich semantic em-beddings for improving model performance on question understanding by integrating word-level and sentence-level information. Moreover, we propose a new data augmentation method called Multimodal Augmented Mixup (MAM) to train the ACMA, denoted as ACMA-MAM. The MAM incorporates various data augmentations and a vanilla mixup strategy to generate more non-repetitive data, which avoids time-consuming artificial data annotations and improves model generalization capability. Our ACMA-MAM outperforms state-of-the-art models on three publicly accessible medical VQA datasets (VQA-Rad, VQA-Slake, and Path VQA) with accuracies of 76.14 %, 83.13 %, and 53.83 % respectively, achieving improvements of 2.00 %, 1.32 %, and 1.59 % accordingly. Moreover, our model achieves F1 scores of 78.33 %, 82.83 %, and 51.86 %, surpassing the state-of-the-art models by 2.80 %, 1.15 %, and 1.37 % respectively.
收起
摘要 :
In most developing countries, it has become a severe challenge for the limited medical resources and outdated healthcare technology to meet the high demand of large population. From the perspective of social development, this unba...
展开
In most developing countries, it has become a severe challenge for the limited medical resources and outdated healthcare technology to meet the high demand of large population. From the perspective of social development, this unbalanced healthcare system in developing counties has also exacerbated the contradiction between physicians and patients, particularly those suffering from malignant diseases (such as prostate cancer). Rapid improvements in artificial intelligence, computing power, parallel operation, and data storage management have contributed significantly to a credible medical data decision-making on the detection, diagnosis, treatment, and prognosis of malignant diseases. Consequently, to address these existing problems in the current healthcare field of developing countries, this paper proposes a novel big medical data decision-making model exploiting fuzzy inference logic for prostate cancer in developing countries, constructing an intelligent medical system for disease detection, medical data analysis and fusion, treatment recommendations, and risk management. Based on 1 933 535 items of hospitalization information from over 8000 prostate cancer cases in China, the experimental results demonstrate that the intelligent medical system could be adopted to assist physicians and medical specialists in coming up with a more dependable diagnosis scheme.
收起
原文获取