Search

B-cell epitope identification plays a vital role in the development of vaccines, therapies, and diagnostic tools. Currently, molecular docking tools in B-cell epitope prediction are heavily influenced by empirical parameters and require significant computational resources, rendering a great challenge to meet large-scale prediction demands. When predicting epitopes from antigen-antibody complex, current artificial intelligence algorithms cannot accurately implement the prediction due to insufficient protein feature representations, indicating novel algorithm is desperately needed for efficient protein information extraction. In this paper, we introduce a multimodal model called WUREN (Whole-modal Union Representation for Epitope predictioN), which effectively combines sequence, graph, and structural features. It achieved AUC-PR scores of 0.213 and 0.193 on the solved structures and AlphaFold-generated structures, respectively, for the independent test proteins selected from DiscoTope3 benchmark. Our findings indicate that WUREN is an efficient feature extraction model for protein complexes, with the generalizable application potential in the development of protein-based drugs. Moreover, the streamlined framework of WUREN could be readily extended to model similar biomolecules, such as nucleic acids, carbohydrates, and lipids.

In the realm of edge AI systems where deep learning is paramount, protecting the intellectual property (IP) of multimodal neural network models is crucial. Current watermarking solutions often bypass the intricacies of multimodal models and the unique constraints of edge environments. Addressing this, a novel watermarking scheme specifically devised for multimodal neural networks is introduced, marking a significant stride in securing these models against IP theft and unauthorized use. A discrete watermark is ingeniously embedded within each modality of a multimodal model, synthesizing a comprehensive watermark that spans the entire model. This method ensures IP protection across varied data types without hampering the model's performance or imposing undue computational demands, making it ideal for resource-limited edge devices. By leveraging the redundancies inherent in multimodal data, watermarks are embedded efficiently, maintaining model integrity and operational effectiveness. A robust verification mechanism is implemented, accurately identifying watermark presence across modalities with minimal computational overhead. Empirical validation on a benchmark dataset demonstrates the method's efficacy in embedding watermarks discreetly while preserving the model's original task performance, showing a 1 to 4% increase in watermark detection rates and a 6 to 10% reduction in false positives compared to existing approaches. This positions the scheme as an effective strategy for IP protection in multimodal neural network models, especially suited for the computational economy required in edge AI systems. The work advances neural network watermarking and addresses the urgent need for scalable IP protection solutions in the evolving AI landscape. [ABSTRACT FROM AUTHOR]

Introduction: Obstructive Sleep Apnea-Hypopnea Syndrome (OSAHS) is a common sleep-related breathing disorder that significantly impacts the daily lives of patients. Currently, the diagnosis of OSAHS relies on various physiological signal monitoring devices, requiring a comprehensive Polysomnography (PSG). However, this invasive diagnostic method faces challenges such as data fluctuation and high costs. To address these challenges, we propose a novel data-driven Audio-SemanticMulti-Modalmodel forOSAHS severity classification (i.e., ASMM-OSA) based on patient snoring sound characteristics. Methods: In light of the correlation between the acoustic attributes of a patient's snoring patterns and their episodes of breathing disorders, we utilize the patient's sleep audio recordings as an initial screening modality. We analyze the audio features of snoring sounds during the night for subjects suspected of having OSAHS. Audio features were augmented via PubMedBERT to enrich their diversity and detail and subsequently classified for OSAHS severity using XGBoost based on the number of sleep apnea events. Results: Experimental results using the OSAHS dataset from a collaborative university hospital demonstrate that our ASMM-OSA audio-semanticmultimodal model achieves a diagnostic level in automatically identifying sleep apnea events and classifying the four-class severity (normal, mild, moderate, and severe) of OSAHS. Discussion: Our proposed model promises new perspectives for non-invasive OSAHS diagnosis, potentially reducing costs and enhancing patient quality of life. [ABSTRACT FROM AUTHOR]

Solar radio bursts are intense radio radiation sources that occur during the energy-release process and represent a hot topic in solar-physics and space-weather research. In this paper, we present a multimode prediction model for daily solar radio bursts. The model uses deep learning and machine learning to obtain data information from different dimensions and to establish the relationship between the characteristics of the solar active region on the solar surface and solar radio bursts. For this model, we use data from the Solar and Heliospheric Observatory (SOHO)/Michelson Doppler Imager (MDI) total solar magnetic map, the Royal Observatory of Belgium World Data Centre in Brussels, and NOAA sunspot parameters (including number, area, and type of sunspots) as inputs. The output results are then compared with the list of solar radio bursts recorded by the Radio Solar Telescope Network (RSTN) to determine whether solar radio bursts are present and to determine the key parameters for determining radio bursts. Based on 5449 days of observational data, we find that the prediction accuracy of the model is 0.898 ± 0.011, and that the number of sunspots is a key parameter in determining the occurrence of solar radio bursts. Specifically, when the number of sunspots is greater than 15, the probability of occurrence of solar radio bursts is greater than 90%. We have identified the key parameters and thresholds for determining solar radio bursts and highlighted the key parameters for space-weather prediction. In addition, the prediction model can also be used for predicting in other fields. [ABSTRACT FROM AUTHOR]

The problem of sentiment analysis of tourism data focuses on the analysis of the multimodal characteristics of the data generated digitally by tourists on each platform or social network. Generally, their opinions have multimodal characteristics, since they combine text, images or numbers (ratings), which represents an important challenge in sentiment analysis that requires new models or multimodal data classification techniques. This work proposes a multimodal sentiment analysis model for data in Spanish in the tourism domain composed of four main phases (extraction, classification, fusion, visualization), and a transversal phase to evaluate the quality of the multimodal sentiment analysis process. Thus, the multimodal sentiment analysis model integrates a data quality model to improve multimodal sentiment analysis tasks, but in addition, the linguistic resource "SenticNet 5" is adapted to Spanish. The model was validated by applying various classification metrics, and the classification results were compared to a manually labeled dataset (TASS) using two machine learning classification algorithms. The first was Random Forest, where the manually labeled dataset has a 50% F1 score compared to the adapted SenticNet automatically generated dataset, which has a 71% F1 score measure and a 70% accuracy. The classification generated by SenticNet is 21% higher than that of the TASS data set. The second algorithm applied was Support Vector Machine (SVM), which classified the SenticNet-generated dataset with an F1 score of 72% versus the manually created dataset with 57.7% (14.3% more effective). In the fusion tests of the multimodal sentiment inputs, the accuracy results for text were 65%, for images 33%, and the fusion of both was 71%. In general, it was identified that the opinions made by users composed of text in Spanish and images improve polarity identification if an independent classification is carried out, and then apply a polarity fusion process. [ABSTRACT FROM AUTHOR]

Objective Surgical outcome prediction is challenging but necessary for postoperative management. Current machine learning models utilize pre- and post-op data, excluding intraoperative information in surgical notes. Current models also usually predict binary outcomes even when surgeries have multiple outcomes that require different postoperative management. This study addresses these gaps by incorporating intraoperative information into multimodal models for multiclass glaucoma surgery outcome prediction. Materials and methods We developed and evaluated multimodal deep learning models for multiclass glaucoma trabeculectomy surgery outcomes using both structured EHR data and free-text operative notes. We compare those to baseline models that use structured EHR data exclusively, or neural network models that leverage only operative notes. Results The multimodal neural network had the highest performance with a macro AUROC of 0.750 and F1 score of 0.583. It outperformed the baseline machine learning model with structured EHR data alone (macro AUROC of 0.712 and F1 score of 0.486). Additionally, the multimodal model achieved the highest recall (0.692) for hypotony surgical failure, while the surgical success group had the highest precision (0.884) and F1 score (0.775). Discussion This study shows that operative notes are an important source of predictive information. The multimodal predictive model combining perioperative notes and structured pre- and post-op EHR data outperformed other models. Multiclass surgical outcome prediction can provide valuable insights for clinical decision-making. Conclusions Our results show the potential of deep learning models to enhance clinical decision-making for postoperative management. They can be applied to other specialties to improve surgical outcome predictions. [ABSTRACT FROM AUTHOR]

Specific emitter identification (SEI) refers to the task of distinguishing similar emitters, especially those of the same type and transmission parameters, which is one of the most critical tasks of electronic warfare. However, SEI is still a challenging task when a feature has low physical representation. Feature representation largely determines the recognition results. Therefore, this article expects to move toward robust feature representation for SEI. Efficient multimodal strategies have great potential for applications using multimodal data and can further improve the performance of SEI. In this research, we introduce a multimodal emitter identification method that explores the application of multimodal data, time-series radar signals, and feature vector data to an enhanced transformer, which employs a conformer block to embed the raw data and integrates an efficient multimodal feature representation module. Moreover, we employ self-knowledge distillation to mitigate overconfident predictions and reduce intra-class variations. Our study reveals that multimodal data provide sufficient information for specific emitter identification. Simultaneously, we propose the CV-CutMixOut method to augment the time-domain signal. Extensive experiments on real radar datasets indicate that the proposed method achieves more accurate identification results and higher feature discriminability. [ABSTRACT FROM AUTHOR]

ObjectiveThe purpose of this study is to develop a multimodal model based on artificial intelligence to assist clinical doctors in the early diagnosis of necrotizing enterocolitis in newborns.MethodsThis study is a retrospective study that collected the initial laboratory test results and abdominal x-ray image data of newborns (non-NEC, NEC) admitted to our hospital from January 2022 to January 2024.A multimodal model was developed to differentiate multimodal data, trained on the training dataset, and evaluated on the validation dataset. The interpretability was enhanced by incorporating the Gradient-weighted Class Activation Mapping (GradCAM) analysis to analyze the attention mechanism of the multimodal model, and finally compared and evaluated with clinical doctors on external datasets.ResultsThe dataset constructed in this study included 11,016 laboratory examination data from 408 children and 408 image data. When applied to the validation dataset, the area under the curve was 0.91, and the accuracy was 0.94. The GradCAM analysis shows that the model's attention is focused on the fixed dilatation of the intestinal folds, intestinal wall edema, interintestinal gas, and portal venous gas. External validation demonstrated that the multimodal model had comparable accuracy to pediatric doctors with ten years of clinical experience in identification.ConclusionThe multimodal model we developed can assist doctors in early and accurate diagnosis of NEC, providing a new approach for assisting diagnosis in underdeveloped medical areas.

Medical image data often face the problem of data scarcity and costly annotation processes. To overcome this, our study introduces a novel transfer learning method for medical image classification. We present a multimodal learning framework that incorporates the pre-trained PubMedCLIP model and multimodal feature fusion. Prompts of different complexities are combined with images as inputs to the proposed model. Our findings demonstrate that this approach significantly enhances image classification tasks while reducing the burden of annotation costs. Our study underscores the potential of PubMedCLIP in revolutionizing medical image analysis through its prompt-based approach and showcases the value of multi-modality for training robust models in healthcare. Code is available at:https://github.com/HongJapan/MTL_prompt_medical.git.

Moving traces are essential data for target detection and associated behavior recognition. Previous studies have used time–location sequences, route maps, or tracking videos to establish mathematical recognition models for behavior recognition. The multimodal approach has seldom been considered because of the limited modality of sensing data. With the rapid development of natural language processing and computer vision, the multimodal model has become a possible choice to process multisource data. In this study, we have proposed a mathematical model for aircraft behavior recognition with joint data manners. The feature abstraction, cross-modal fusion, and classification layers are included in the proposed model for obtaining multiscale features and analyzing multimanner information. Attention has been placed on providing self- and cross-relation assessments on the spatiotemporal and geographic data related to a moving object. We have adopted both a feedforward network and a softmax function to form the classifier. Moreover, we have enabled a modality-increasing phase, combining longitude and latitude sequences with related geographic maps to avoid monotonous data. We have collected an aircraft trajectory dataset of longitude and latitude sequences for experimental validation. We have demonstrated the excellent behavior recognition performance of the proposed model joint with the modality-increasing phase. As a result, our proposed methodology reached the highest accuracy of 95.8% among all the adopted methods, demonstrating the effectiveness and feasibility of trajectory-based behavior recognition. [ABSTRACT FROM AUTHOR]

Purpose: Retinopathy of prematurity (ROP) is one of the leading causes of blindness in children. Although the role of oxygen in the pathophysiology of ROP is well established, a precise understanding of the dynamic relationship between oxygen exposure ROP incidence and severity is lacking. The purpose of this study was to evaluate the correlation between time-dependent oxygen variables and the onset of ROP. Design: Retrospective cohort study. Participants: Two hundred thirty infants who were born at a single academic center and met the inclusion criteria were included. Infants are mainly born between January 2011 and October 2022. Methods: Patient data were extracted from electronic health records (EHRs), with sufficient time-dependent oxygen data. Clinical outcomes for ROP were recorded as none/mild or moderate/severe (defined as type II or worse). Mixed-effects linear models were used to compare the 2 groups in terms of dynamic oxygen variables, such as daily average and the coefficient of variation (COV) fraction of inspired oxygen (FiO2). Support vector machine (SVM) and long-short-term memory (LSTM)-based multimodal models were trained with fivefold cross-validation to predict which infants would develop moderate/severe ROP. Gestational age (GA), birth weight, and time-dependent oxygen variables were used to develop predictive models. Main Outcome Measures: Model cross-validation performance was evaluated by computing the mean area under the receiver operating characteristic (AUROC) curve, precision, recall, and F1 score. Results: We found that both daily average and COV of FiO2 were associated with more severe ROP (adjusted P

Network general knowledge course is a meaningful way to cultivate comprehensive quality talents in Chinese higher education based on the modern information technology environment, and the optimization of network general knowledge course teaching is directly related to whether the network general knowledge course can be effectively implemented, which affects the overall effect of talent cultivation. In this paper, starting from the purpose and goal of humanities general education courses, the optimization direction of teaching general education courses using artificial intelligence is clarified. Skeletal sequence feature extraction is used to obtain students’ general education classroom behavior data, and a stackable multimodal attention module consisting of a self-attention layer and a multimodal co-attention layer is designed for feature fusion and action prediction based on multimodal characteristics. After data integration and processing, the HGCL-LG network model is proposed to complete the diagnosis of students’ general knowledge cognition. The trend of interest assessment and multimodal model calculation value is consistent. In contrast, the student’s interest attention in general knowledge classroom above 0.33 accounts for more than 40% of the overall information, and the interest attention is good. Students’ scores on the objective questions of general knowledge are good; the average score range is between 1.5 and 2 points, the degree of dispersion is small, and the mastery of basic knowledge of general knowledge is good.

Gyrokinetic simulations are required for the quantitative calculation of fluxes due to turbulence, which dominates over other transport mechanisms in tokamaks. However, nonlinear gyrokinetic simulations are computationally expensive. A multimodal convolutional neural network model that reads images and values generated by nonlinear gyrokinetic simulations and predicts electrostatic turbulent heat fluxes was developed to support efficient runs. The model was extended to account for squared electrostatic potential fluctuations, which are proportional to the fluxes in the quasilinear model, as well as images containing fluctuating electron and ion distribution functions and fluctuating electrostatic potentials in wavenumber space. This multimodal model can predict the time and electron and ion turbulent heat fluxes corresponding to the input data. The model trained on the Cyclone base case data successfully predicted times and fluxes not only for its test data, but also for the completely different and unknown JT‐60U case, with high accuracy. The predictive performance of the model depended on the similarity of the linear stability of the case used to train the model to the case being predicted. [ABSTRACT FROM AUTHOR]

As deep learning research continues to advance, interpretability is becoming as important as model performance. Conducting interpretability studies to understand the decision-making processes of deep learning models can improve performance and provide valuable insights for humans. The interpretability of visual question answering (VQA), a crucial task for human–computer interaction, has garnered the attention of researchers due to its wide range of applications. The generation of natural language explanations for VQA that humans can better understand has gradually supplanted heatmap representations as the mainstream focus in the field. Humans typically answer questions by first identifying the primary objects in an image and then referring to various information sources, both within and beyond the image, including prior knowledge. However, previous studies have only considered input images, resulting in insufficient information that can lead to incorrect answers and implausible explanations. To address this issue, we introduce multiple references in addition to the input image. Specifically, we propose a multimodal model that generates natural language explanations for VQA. We introduce outside knowledge using the input image and question and incorporate object information into the model through an object detection module. By increasing the information available during the model generation process, we significantly improve VQA accuracy and the reliability of the generated explanations. Moreover, we employ a simple and effective feature fusion joint vector to combine information from multiple modalities while maximizing information preservation. Qualitative and quantitative evaluation experiments demonstrate that the proposed method can generate more reliable explanations than state-of-the-art methods while maintaining answering accuracy. [ABSTRACT FROM AUTHOR]

The distribution coefficient (K d) plays a crucial role in predicting the migration behavior of radionuclides in the soil environment. However, K d depends on the complexities of geological and environmental factors, and existing models often do not reflect the unique soil properties. We propose a multimodal technique to predict K d values for radionuclide adsorption in soils surrounding nuclear facilities in Republic of Korea. We integrated and trained three sub-networks reflecting different data domains: soil adsorption factors for physicochemical conditions, X-ray fluorescence (XRF) data, and X-ray diffraction (XRD) spectra for inherent soil properties. Our multimodal model achieved high performance, with a coefficient of determination (R2) of 0.84 and root mean squared error (RMSE) of 0.89 for natural log-transformed K d. This is the first study to develop a multimodal model that simultaneously incorporates inherent soil properties and adsorption factors to predict K d. We investigated influential peaks in XRD spectra and also revealed that pH and calcium oxide (CaO) were significant variables in soil adsorption factors and XRF data, respectively. These results promote the use of a multimodal model to predict K d values by integrating data from different domains, providing a cost-effective and novel approach to elucidate the mechanisms of radionuclide adsorption in soil. [Display omitted] • Soil composition near nuclear facilities was identified through sorption experiments. • A multimodal machine learning was developed for various sorption data modalities. • The multimodal model estimated the distribution coefficient effectively. • SHAP was employed to investigate the importance of variables affecting the results. • Grad-CAM algorithms was utilized to visualize influential XRD spectrum bands. [ABSTRACT FROM AUTHOR]

Objective. To develop a multimodal model that combines multiphase contrast-enhanced computed tomography (CECT) imaging and clinical characteristics, including experts’ experience, to preoperatively predict lymph node metastasis (LNM) in pancreatic cancer patients. Methods. We proposed a new classifier fusion strategy (CFS) based on a new evidential reasoning (ER) rule (CFS-nER) by combining nomogram weights into a previous ER rule-based CFS. Three kernelled support tensor machine-based classifiers with plain, arterial, and venous phases of CECT as the inputs, respectively, were constructed. They were then fused based on the CFS-nER to construct a fusion model of multiphase CECT. The clinical characteristics were analyzed by univariate and multivariable logistic regression to screen risk factors, which were used to construct correspondent risk factor-based classifiers. Finally, the fusion model of the three phases of CECT and each risk factor-based classifier were fused further to construct the multimodal model based on our CFS-nER, named MMM-nER. This study consisted of 186 patients diagnosed with pancreatic cancer from four clinical centers in China, 88 (47.31%) of whom had LNM. Results. The fusion model of the three phases of CECT performed better overall than single and two-phase fusion models; this implies that the three considered phases of CECT were supplementary and complemented one another. The MMM-nER further improved the predictive performance, which implies that our MMM-nER can complement the supplementary information between CECT and clinical characteristics. The MMM-nER had better predictive performance than based on previous classifier fusion strategies, which presents the advantage of our CFS-nER. Conclusion. We proposed a new CFS-nER, based on which the fusion model of the three phases of CECT and MMM-nER were constructed and performed better than all compared methods. MMM-nER achieved an encouraging performance, implying that it can assist clinicians in noninvasively and preoperatively evaluating the lymph node status of pancreatic cancer. [ABSTRACT FROM AUTHOR]

Human 3D skeleton-based action recognition has received increasing interest in recent years. Inspired by the excellent ability of the multi-modal model, some pioneer attempts to employ diverse modalities, i.e., skeleton and language, to construct the skeleton-language model and have shown compelling results. Yet, these attempts model the data representation as deterministic point estimation, ignoring a key issue that descriptions of similar motions are uncertain and ambiguous, which brings about restricted comprehension of complex concept hierarchies and impoverished cross-modal alignment reliability. To tackle this challenge, this paper proposes a new Uncertain Skeleton-Language Learning Framework (USLLF) to capture the semantic ambiguity among diverse modalities in a probabilistic manner for the first time. USLLF consists of both inter- and intra-modal uncertainties. Specifically, first, we integrate the language (text) generated by ChatGPT with the generic skeleton-based network and develop a deterministic multi-modal baseline, which can be easily achieved via any off-the-shelf skeleton and text encoders. Then, based on this baseline, we explicitly model the intra-modal (skeleton/language) uncertainties as the Gaussian distributions using the new uncertainty networks capable of learning the distributional embeddings of modalities. Following this, these embeddings are aligned and formulated as inter-modal (skeleton-language) uncertainty using both the contrastive and negative log-likelihood objectives to alleviate the cross-modal alignment error. Experimental results on NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets show that our approach outperforms the proposed baseline and achieves comparable performance with a high inference efficiency compared to the state-of-the-art methods. Besides, we also deliver insightful analyses on how learned uncertainty reduces the impact of uncertain and ambiguous data on model performance. [ABSTRACT FROM AUTHOR]

Heart failure, as a critical symptom or terminal stage of assorted heart diseases, is a world-class public health problem. Establishing a prognostic model can help identify high dangerous patients, save their lives promptly, and reduce medical burden. Although integrating structured indicators and unstructured text for complementary information has been proven effective in disease prediction tasks, there are still certain limitations. Firstly, the processing of single branch modes is easily overlooked, which can affect the final fusion result. Secondly, simple fusion will lose complementary information between modalities, limiting the network's learning ability. Thirdly, incomplete interpretability can affect the practical application and development of the model. To overcome these challenges, this paper proposes the MDL-HFP multimodal model for predicting patient prognosis using the MIMIC-III public database. Firstly, the ADASYN algorithm is used to handle the imbalance of data categories. Then, the proposed improved Deep&Cross Network is used for automatic feature selection to encode structured sparse features, and implicit graph structure information is introduced to encode unstructured clinical notes based on the HR-BGCN model. Finally, the information of the two modalities is fused through a cross-modal dynamic interaction layer. By comparing multiple advanced multimodal deep learning models, the model's effectiveness is verified, with an average F1 score of 90.42% and an average accuracy of 90.70%. The model proposed in this paper can accurately classify the readmission status of patients, thereby assisting doctors in making judgments and improving the patient's prognosis. Further visual analysis demonstrates the usability of the model, providing a comprehensive explanation for clinical decision-making. [ABSTRACT FROM AUTHOR]

Patient-specific quality assurance (QA) for Volumetric Modulated Arc Therapy (VMAT) plans is routinely performed in the clinical. However, it is labor-intensive and time-consuming for medical physicists. QA prediction models can address these shortcomings and improve efficiency. Current approaches mainly focus on single cancer and single modality data. They are not applicable to clinical practice. To assess the accuracy of QA results for VMAT plans, this paper presents a new model that learns complementary features from the multi-modal data to predict the gamma passing rate (GPR). According to the characteristics of VMAT plans, a feature-data fusion approach is designed to fuse the features of imaging and non-imaging information in the model. In this study, 690 VMAT plans are collected encompassing more than ten diseases. The model can accurately predict the most VMAT plans at all three gamma criteria: 2%/2 mm, 3%/2 mm and 3%/3 mm. The mean absolute error between the predicted and measured GPR is 2.17%, 1.16% and 0.71%, respectively. The maximum deviation between the predicted and measured GPR is 3.46%, 4.6%, 8.56%, respectively. The proposed model is effective, and the features of the two modalities significantly influence QA results. [ABSTRACT FROM AUTHOR]

Foundation models (FM), which are large-scale artificial intelligence (AI) models that can complete a range of tasks, represent a paradigm shift in AI. These versatile models encompass large language models, vision-language models, and multimodal models. Although these models are often trained for broad tasks, they have been applied either out of the box or after additional fine tuning to tasks in medicine, including dermatology. From addressing administrative tasks to answering dermatology questions, these models are poised to have an impact on dermatology care delivery. As FMs become more ubiquitous in health care, it is important for clinicians and dermatologists to have a basic understanding of how these models are developed, what they are capable of, and what pitfalls exist. In this paper, we present a comprehensive yet accessible overview of the current state of FMs and summarize their current applications in dermatology, highlight their limitations, and discuss future developments in the field., (Copyright © 2024 The Authors. Published by Elsevier Inc. All rights reserved.)

Objective: The purpose of this study is to develop a multimodal model based on artificial intelligence to assist clinical doctors in the early diagnosis of necrotizing enterocolitis in newborns., Methods: This study is a retrospective study that collected the initial laboratory test results and abdominal x-ray image data of newborns (non-NEC, NEC) admitted to our hospital from January 2022 to January 2024.A multimodal model was developed to differentiate multimodal data, trained on the training dataset, and evaluated on the validation dataset. The interpretability was enhanced by incorporating the Gradient-weighted Class Activation Mapping (GradCAM) analysis to analyze the attention mechanism of the multimodal model, and finally compared and evaluated with clinical doctors on external datasets., Results: The dataset constructed in this study included 11,016 laboratory examination data from 408 children and 408 image data. When applied to the validation dataset, the area under the curve was 0.91, and the accuracy was 0.94. The GradCAM analysis shows that the model's attention is focused on the fixed dilatation of the intestinal folds, intestinal wall edema, interintestinal gas, and portal venous gas. External validation demonstrated that the multimodal model had comparable accuracy to pediatric doctors with ten years of clinical experience in identification., Conclusion: The multimodal model we developed can assist doctors in early and accurate diagnosis of NEC, providing a new approach for assisting diagnosis in underdeveloped medical areas., Competing Interests: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest., (© 2024 Cui, Changrong, Maomin, Hui and Xiuxiang.)

B-cell epitope identification plays a vital role in the development of vaccines, therapies, and diagnostic tools. Currently, molecular docking tools in B-cell epitope prediction are heavily influenced by empirical parameters and require significant computational resources, rendering a great challenge to meet large-scale prediction demands. When predicting epitopes from antigen-antibody complex, current artificial intelligence algorithms cannot accurately implement the prediction due to insufficient protein feature representations, indicating novel algorithm is desperately needed for efficient protein information extraction. In this paper, we introduce a multimodal model called WUREN (Whole-modal Union Representation for Epitope predictioN), which effectively combines sequence, graph, and structural features. It achieved AUC-PR scores of 0.213 and 0.193 on the solved structures and AlphaFold-generated structures, respectively, for the independent test proteins selected from DiscoTope3 benchmark. Our findings indicate that WUREN is an efficient feature extraction model for protein complexes, with the generalizable application potential in the development of protein-based drugs. Moreover, the streamlined framework of WUREN could be readily extended to model similar biomolecules, such as nucleic acids, carbohydrates, and lipids., Competing Interests: The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (© 2024 The Authors.)

Introduction: Obstructive Sleep Apnea-Hypopnea Syndrome (OSAHS) is a common sleep-related breathing disorder that significantly impacts the daily lives of patients. Currently, the diagnosis of OSAHS relies on various physiological signal monitoring devices, requiring a comprehensive Polysomnography (PSG). However, this invasive diagnostic method faces challenges such as data fluctuation and high costs. To address these challenges, we propose a novel data-driven Audio-Semantic Multi-Modal model for OSAHS severity classification (i.e., ASMM-OSA) based on patient snoring sound characteristics., Methods: In light of the correlation between the acoustic attributes of a patient's snoring patterns and their episodes of breathing disorders, we utilize the patient's sleep audio recordings as an initial screening modality. We analyze the audio features of snoring sounds during the night for subjects suspected of having OSAHS. Audio features were augmented via PubMedBERT to enrich their diversity and detail and subsequently classified for OSAHS severity using XGBoost based on the number of sleep apnea events., Results: Experimental results using the OSAHS dataset from a collaborative university hospital demonstrate that our ASMM-OSA audio-semantic multimodal model achieves a diagnostic level in automatically identifying sleep apnea events and classifying the four-class severity (normal, mild, moderate, and severe) of OSAHS., Discussion: Our proposed model promises new perspectives for non-invasive OSAHS diagnosis, potentially reducing costs and enhancing patient quality of life., Competing Interests: XT was employed by INF Technology (Shanghai) Co., Ltd. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest., (Copyright © 2024 Qiu, Wang, Li, Tong, Tan, Yang, Tao and Huang.)

In the early building design stage, machine learning-based surrogate models demonstrate significant advantages in rapidly evaluating design performance compared to conventional simulation software. However, due to network limitations, current models struggle to incorporate comprehensive building information, leading to undesired generalizability in adapting to design variations. This study proposes a multimodal Generative Adversarial Network (GAN)-based surrogate model, representing building features with combined multimodal data of images and vectors. Geometric features on the plan and façade are translated into grayscale maps, indicating the potential for daylight reception from each viewpoint, while material properties and weather features are extracted as vectors, such as reflectance, transmittance, sun position and sky conditions. To satisfy the demands of multimodal inputs, the multimodal feature fusion block, the vector-based feature encoding block, and the feature reinforcement block are introduced. These components aim to promote the deep integration of information from different modalities, balance the influences of features across different dimensional scales, and ensure the integrity of the information. The model's validity was verified on the RPLAN database, predicting illumination distribution at the whole-building hierarchy instead of the single-room hierarchy. The results indicate that the multimodal GAN has a Mean Squared Error of 8.129, a Mean Absolute Percentage Error of 0.135, and a Structural Similarity Index of 0.907 on the test set. Meanwhile, this approach saved 73.03% of computing time compared to simulation methods. Boasting strengths in generalizability, speed, and accuracy, the model provides effective support for daylight organization in the early stage of building design, without being confined by variable design scenarios. [ABSTRACT FROM AUTHOR]

Tissue-to-blood partition coefficients ( P tb ) are key parameters for assessing toxicokinetics of xenobiotics in organisms, yet their experimental data were lacking. Experimental methods for measuring P data from diverse sources, and their applicability domain (AD) was limited. The current study developed a multimodal model capable of processing and integrating textual (categorical features) and numerical information (molecular descriptors/fingerprints) to simultaneously predict tb values across various species, tissues, blood matrices, and measurement methods. Artificial neural network algorithms with embedding layers were used for the multimodal modeling. The corresponding unimodal models were developed for comparison. Results showed that the multimodal model outperformed unimodal models. To enhance the reliability of the model, a method considering categorical features, weighted molecular similarity density, and weighted inconsistency in molecular activities of structure-activity landscapes was used to characterize the AD. The model constrained by the AD exhibited better prediction accuracy for the validation set, with the determination coefficient, root mean-square error, and mean absolute error being 0.843, 0.276, and 0.213 log units, respectively. The multimodal model coupled with the AD characterization can serve as an efficient tool for internal exposure assessment of chemicals.P tb data from diverse sources, and their applicability domain (AD) was limited. The current study developed a multimodal model capable of processing and integrating textual (categorical features) and numerical information (molecular descriptors/fingerprints) to simultaneously predict P tb values across various species, tissues, blood matrices, and measurement methods. Artificial neural network algorithms with embedding layers were used for the multimodal modeling. The corresponding unimodal models were developed for comparison. Results showed that the multimodal model outperformed unimodal models. To enhance the reliability of the model, a method considering categorical features, weighted molecular similarity density, and weighted inconsistency in molecular activities of structure-activity landscapes was used to characterize the AD. The model constrained by the AD exhibited better prediction accuracy for the validation set, with the determination coefficient, root mean-square error, and mean absolute error being 0.843, 0.276, and 0.213 log units, respectively. The multimodal model coupled with the AD characterization can serve as an efficient tool for internal exposure assessment of chemicals.

Purpose: Retinopathy of prematurity (ROP) is one of the leading causes of blindness in children. Although the role of oxygen in the pathophysiology of ROP is well established, a precise understanding of the dynamic relationship between oxygen exposure ROP incidence and severity is lacking. The purpose of this study was to evaluate the correlation between time-dependent oxygen variables and the onset of ROP., Design: Retrospective cohort study., Participants: Two hundred thirty infants who were born at a single academic center and met the inclusion criteria were included. Infants are mainly born between January 2011 and October 2022., Methods: Patient data were extracted from electronic health records (EHRs), with sufficient time-dependent oxygen data. Clinical outcomes for ROP were recorded as none/mild or moderate/severe (defined as type II or worse). Mixed-effects linear models were used to compare the 2 groups in terms of dynamic oxygen variables, such as daily average and the coefficient of variation (COV) fraction of inspired oxygen (FiO 2 ). Support vector machine (SVM) and long-short-term memory (LSTM)-based multimodal models were trained with fivefold cross-validation to predict which infants would develop moderate/severe ROP. Gestational age (GA), birth weight, and time-dependent oxygen variables were used to develop predictive models., Main Outcome Measures: Model cross-validation performance was evaluated by computing the mean area under the receiver operating characteristic (AUROC) curve, precision, recall, and F1 score., Results: We found that both daily average and COV of FiO 2 were associated with more severe ROP (adjusted P  < 0.001). With fivefold cross-validation, the multimodal LSTM models had higher performance than the best static models (SVM using GA and 3 average FiO 2 features) and SVM models trained on GA alone (mean AUROC = 0.89 ± 0.04 vs. 0.86 ± 0.05 vs. 0.83 ± 0.04)., Conclusions: The development of severe ROP might not only be influenced by oxygen exposure but also by its fluctuation, which provides direction for future study of pathophysiological factors associated with severe ROP development. Additionally, we demonstrated that multimodal neural networks can be a method to extract useful information from time-series data, which may be a valuable methodology for the investigation of other diseases using EHR data., Financial Disclosures: Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article., (© 2023 Published by Elsevier Inc. on behalf of the American Academy of Ophthalmology.)

Circular RNAs (circRNAs) have been found to have the ability to encode proteins through internal ribosome entry sites (IRESs), which are essential RNA regulatory elements for cap-independent translation. Identification of IRES elements in circRNA is crucial for understanding its function. Previous studies have presented IRES predictors based on machine learning techniques, but they were mainly designed for linear RNA IRES. In this study, we proposed DeepCIP (Deep learning method for CircRNA IRES Prediction), a multimodal deep learning approach that employs both sequence and structural information for circRNA IRES prediction. Our results demonstrate the effectiveness of the sequence and structure models used by DeepCIP in feature extraction and suggest that integrating sequence and structural information efficiently improves the accuracy of prediction. The comparison studies indicate that DeepCIP outperforms other comparative methods on the test set and real circRNA IRES dataset. Furthermore, through the integration of an interpretable analysis mechanism, we elucidate the sequence patterns learned by our model, which align with the previous discovery of motifs that facilitate circRNA translation. Thus, DeepCIP has the potential to enhance the study of the coding potential of circRNAs and contribute to the design of circRNA-based drugs. DeepCIP as a standalone program is freely available at https://github.org/zjupgx/DeepCIP., Competing Interests: Declaration of competing interest The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Yunguang Tong is the founder of Aoming (Hangzhou) Biomedical Co., Ltd. All other authors declare no competing financial interests., (Copyright © 2023 Elsevier Ltd. All rights reserved.)

This article examines theoretical frameworks and models that focus on the pedagogical aspects of online education. After a review of learning theory as applied to online education, a proposal for an integrated Multimodal Model for Online Education is provided based on pedagogical purpose. The model attempts to integrate the work of several other major theorists and model builders such as Anderson (2011). [ABSTRACT FROM AUTHOR]

Various psychological approaches are effective in maintaining and maximizing functioning in Alzheimer's disease. The most efficacious approaches have been based on a combination of individualized, interdisciplinary, and holistic treatment. Such a treatment approach is comparable to Lazarus's multimodal model, which has also been referred to as “systematic eclecticism.” The multimodal model is especially well suited for people experiencing dementia, a progressive disorder. [ABSTRACT FROM PUBLISHER]