Search

Industrial collaborative robots play an essential role in smart manufacturing because they improve productivity while also ensuring workplace safety. However, the development of prognostic and health management systems to ensure the reliability of these robots has been a major challenge due to the lack of fault data. This paper proposed a digital twin scheme based on the fusion of the robot kinematic and dynamic models' information down to the powertrains (i.e., the joints motor, and gear) along with the control algorithms and uncertainty accommodation based upon deep learning. The presented digital twin concept has the potential to propel simulation-based fault prediction. We also highlight and discuss challenges and opportunities around the development of the hybrid digital twin for condition monitoring of industrial collaborative robots. [ABSTRACT FROM AUTHOR]

Production sequencing methodology using DeepLearning Seq2Seq-LSTM is applied to a tire production case study in Quebec, Canada. Production and demand data are used to predict the most likely product sequences to operate. The comparison of 4 forecasting models, differing in consideration of demand and a statistical component, leads to nearly 70% of good prediction when all machines are studied and 10 production scenarios are considered. This performance reaches 92% for a specific class of machines. The analysis of the forecasts by class of machine allows highlighting 2 factors influencing the performance of the models, namely the ratio of product/machine by class and the total number of available records. The forecasts of possible production scenarios can then be used in a digital twin to evaluate a reasonable number of options and develop a decision support system for production sequencing. [ABSTRACT FROM AUTHOR]

The meticulous reconstruction of three-dimensional (3D) braided composite materials serves as a crucial foundation for achieving high-fidelity simulations. Nonetheless, the transition from tomographic images to a 3D mesh entails a laborious and time-intensive process. To address this, an integrated procedure based on artificial intelligence is proposed for reconstructing meshes from tomograms. The initial stage of the process involves employing artificial intelligence techniques to segment complex contours and optimize high-dimensional contours. This facilitates the input of high-quality images needed to reconstruct accurate digital twins with strong convergence. The subsequent reconstruction phase integrates various calculations, including shape interpolation, contour extraction, 3D surface reconstruction, 3D mesh reconstruction, and element data interpolation. During this process, optimization objectives are set to minimize the deviation between the digital twin's surface and the actual surface, as well as to optimize the aspect ratio of the element mesh. Upon completion of the aforementioned steps, high-quality input files suitable for finite element calculations are directly generated. Ultimately, the proposed method utilizes the reconstructed finite element model for mechanical analysis, and the results are found to be in good agreement with experimental tests. This method offers an efficient and rapid way to achieve high-quality reconstruction of complex digital twins. [Display omitted] • The proposed procedure to reconstruct meshes from tomograms is integrated, enabling batch processing. • The deep learning models are employed to segment the intricate yarns contours of 3D5D braided composites. • The Ramer–Douglas–Peucker algorithm is used to capture high-dimensional contour features and improve convergence. • A decimating filter employing the quadric-based edge-collapsed strategy improves volumetric meshing. [ABSTRACT FROM AUTHOR]

Reliable traffic infrastructure is a key factor for any country's economy. However, aging bridges often require renovation or reconstruction. Promising approaches for enhancing asset management and cost reduction are digital twins and predictive maintenance strategies. However, the creation of geometric-semantic as-is models as a basis for digital twins currently involves labor-intensive manual data capture and modeling. Modern deep learning models, such as Kernel Point Convolution (KPConv) show promising results in reducing the time needed to create digital twins by semantically segmenting point cloud data but have so far been hindered by the lack of a reliable quality measure, which can predict when the model's prediction can be trusted. This is especially viable in the construction industry, where objects and sensors may vary widely between projects and contractors. In this work, we present Bayesian neural networks, implemented through Variational Inference and Monte-Carlo dropout as approaches to conducting inference with KPConv, which show improvements in the confidence estimation and out-of-domain (OOD) detection in the scans of typical infrastructure point clouds. When confronted with different domain shifts to the test data, such as a change of scanning device and introduction of unseen classes the proposed model showed a 25.3% decrease in expected calibration error (ECE) and a 4.82 increase in OOD detection in terms of outlier-intersection-over-union (O-IoU) on average with respect to a deterministic baseline. • Bayesian neural networks applied in point cloud semantic segmentation. • Training stochastic models with Variational Inference improve calibration. • Out of domain detection was improved with a single stochastic layer. • Model robustness to domain shift from mobile to terrestrial laser scanner. [ABSTRACT FROM AUTHOR]

The thermal errors in ball screws present complex spatial-temporal characteristics and exhibit persistent long-term memory, significantly affecting the machining accuracy of the whole machine tool in applications. Deep learning has emerged as a promising approach to predict these errors using thermal data. However, the scarcity of extensive thermal datasets, typically due to the prohibitive costs and time required for experimental acquisition, hampers the comprehensive exploitation of spatial-temporal and long-term memory attributes. Moreover, the current thermal error compensation system does not support the deployment of deep learning-based thermal error models due to its weak real-time performance. These limitations lead to suboptimal prediction accuracy and inadequate error compensation. Addressing this challenge, a novel spatial-temporal interactive integration network is proposed, and this is a sophisticated model that synergistically blends a time memory gate and a spatial-temporal fusion gate within an advanced attention-based spatial-temporal graph convolutional framework. Our network ingeniously leverages spatial data to anchor long-term memory while employing temporal data for selective memory feature extraction, facilitating a refined integration of spatial-temporal features. Concurrently, the integration of gated recurrent units and a time attention layer meticulously extracts and enhances temporal features, bolstered by our innovative time memory gate, which is adept at handling small-sample scenarios and refining thermal error predictions. Critically, the digital twin system for thermal error compensation is constructed to improve the system's real-time performance. The integration of this network into the digital twin system marks a pivotal advancement in thermal error compensation. Our empirical results underscore the system's remarkable robustness and superior predictive accuracy, demonstrating a significant reduction in positioning and machining errors (over 90% and 80% respectively) even with constrained thermal data inputs. These advancements not only delineate a substantial leap in predictive accuracy but also underscore our contribution to reducing operational costs and enhancing the efficacy of machine tools. [Display omitted] • Spatial-temporal fusion gate is proposed to extract spatiotemporal features. • STIIN is proposed to extract spatiotemporal features of thermal error. • Digital twin system for thermal error compensation is established. • STIIN model is embedded into digital twin system. • Robust compensation is realized with small-volume thermal information data. [ABSTRACT FROM AUTHOR]

In this article, we present a deep learning-based surrogate model for spatio-temporal prediction of cavitating fluid flow. Specifically, we introduce a finite element-inspired rotation equivariant hypergraph neural network for inferring and predicting dynamical behaviors of cavitating flow. We generate ground-truth spatial–temporal data by simulating a full-order variational system based on homogeneous mixture-based cavitation theory. We consider the flow past a NACA0012 hydrofoil to examine the predictive ability of the proposed graph neural network for cavitation dynamics. Results demonstrate that the network achieves stabilized and accurate temporal predictions of the system states, successfully forecasting the evolution patterns of individual cavitation events. Additionally, comparisons of predicted fluid loading coefficients are in good agreements with the ground-truth values. We also discuss some challenges encountered in the long-term prediction of flow patterns across multiple cavitation events. The proposed framework has implications for design optimization, active control and development of a physics-based digital twin of a cavitating marine propeller. [Display omitted] • Finite element-inspired graph neural network for cavitation dynamics. • Stabilized and accurate spatio-temporal predictions of system states. • Successful forecasting of individual cavitation events. • Good comparison of predicted force coefficients with ground truth values. [ABSTRACT FROM AUTHOR]

Large volumes of 3D parametric datasets, such as building information modeling (BIM), are the foundation for developing and applying smart city and digital twin technologies. Those datasets are also considered valuable tools for efficiently managing rebuilt structures during the operation and maintenance stages. Nevertheless, current approaches developed for the scan-to-BIM process rely on manual or semi-automatic procedures and insufficiently leverage semantic data in point clouds. These methods struggle to accurately represent large-scale indoor complex layouts and extract details from irregular-shaped unstructured elements, causing inefficiencies in BIM model generation. To address these issues, we propose an innovative scan-to-BIM framework based on deep learning algorithms and raw point cloud data, enabling the automatic generation of 3D models for both structured and unstructured indoor elements. Initially, we propose an enhanced deep learning neural network to improve the point clouds' semantic segmentation accuracy. Subsequently, an efficient workflow is developed to reconstruct 3D building models of structured indoor scenes. The proposed workflow can reconstruct large-scale data with multiple room layouts of Manhattan or non-Manhattan structures and reconstruct 3D models automatically by using a BIM parametric algorithm implemented in Revit software. Moreover, we introduce a robust method for unstructured elements to automatically generate corresponding 3D BIM models, even when the incorporating semantic information is incomplete. The proposed approach was evaluated on synthetic and real data for different scales and complexities of indoor scenes. The results of the experiments demonstrate that the improved model significantly enhances the overall semantic segmentation accuracy compared to the baseline models. The proposed scan-to-BIM framework is efficient for indoor element 3D reconstruction, achieving precision, recall, and F-score values ranging from 96% to 99%. The generated BIM models are competitive with traditional methods regarding model completeness and geometric accuracy. • We propose an innovative scan-to-BIM framework based on deep learning algorithms: • We propose an enhanced deep-learning neural network to improve the point clouds' semantic segmentation accuracy. • A novel and efficient workflow is developed to reconstruct 3D building models of structured indoor scenes. It can reconstruct large-scale data with multiple room layouts in 3D model automatically by using a BIM parametric • For unstructured elements, we propose a robust method to automatically generate 3D BIM models, even when the semantic information is incomplete. [ABSTRACT FROM AUTHOR]

• A digital twin-assisted dual transfer method was proposed with information and model adaptation for bearing fault diagnosis. • DTa-DT aims to simultaneously synthesize information transfer and model transfer for domain transfer error minimization. • DTd-IT method was proposed to information transfer with low distribution difference of twin data adapted. • DAd-MT method, composed of DBTN, was proposed to model transfer with feature distribution difference of domain-model adapted. • Experiments and ablation study verified that the proposed DTa-DT method can be well applied to small sample learning. Rolling bearing fault diagnosis is of great importance to the safety management of mechanical equipment. The scarcity of labelled fault data makes it difficult to adequately perform the training process of intelligent diagnosis models, and this will result in these intelligent models not being effectively and widely used in practice. Although some recent studies have verified that the addition of dynamic model response to the training process will greatly improve the ability of the model with low cost and high efficiency, it is still stuck in poor effect caused by large information distribution difference between dynamic model response and real measured data. Focusing on this issue, a digital twin-assisted dual transfer (DTa-DT) method with information and model adaptation was proposed for rolling bearing fault diagnosis. Different from the traditional digital-analogue driven transfer methods, the proposed DTa-DT aims to simultaneously synthesize data information transfer and feature model transfer together with domain transfer error minimization. In particular, it should be noted that the DTa-DT architecture consists of a dual transfer learning process, including digital twin-driven information transfer (DTd-IT) and digital-analogue-driven model transfer (DAd-MT), where the information is collaborated with the model to improve the integrated transfer diagnosis effect under sampling. On one aspect, with the employment of bearing dynamic model responses, DTd-IT is innovatively designed to establish the transfer of dynamic information and measured information. The information distribution difference between these twin data and real measured data is effetely adjusted with the introduced actual inference components, where the twin data with low information distribution difference can be well fusion generated by the information transfer digital twin (ITDT) model. On the other aspect, considering the truth that there are still small sample cases of real measured data and information distribution differences will affect the quality of the twin data, a digital-analogue driven model transfer (DAd-MT) method is further proposed, where the deep branch transfer network (DBTN) model with improved convolutional neural network (CNN) is used to achieve an accurate fault diagnosis effect with the help of digital twin data. Experiments and wear analysis verified that the proposed DTa-DT can significantly reduce the distribution difference between the dynamic model response and the real measured data, thus achieving low-cost and efficient rolling bearing transfer diagnosis compared to other ten state-of-the-art deep learning models. It can be predicted that the proposed dual transfer architecture provides more opportunities for the practical application of intelligent fault diagnosis under small sample sizes. [ABSTRACT FROM AUTHOR]

Digital twin has become a critical technical solution for satellite in-orbit service and maintenance. The majority of existing research in this field has focused on constructing system-level digital twins, with various typical applications that primarily describe the operating condition of satellite subsystems. However, achieving a more meticulous portrayal of in-orbit satellites necessitates the inclusion of physical field information, e.g., the temperature field serves as a fundamental basis for characterizing the thermodynamic state and plays a crucial role in precise thermal management of a satellite. The main challenge for realizing thorough state detection and real-time response lies in the cross-level model fusion of system simulation and physical field calculation. To overcome this problem, this paper proposes a Hybrid Digital Twin (HDT) framework that seamlessly integrates the digital twin at the system-level with its physical field-level counterpart, enabling a comprehensive and detailed representation of modeling objects. Specifically, at the physical field-level, deep learning methods are utilized to construct an accurate surrogate model, offering a lightweight, computationally efficient and accelerated execution alternative to the Finite Element Analysis models. At the system-level, multi-domain unified modeling technology is employed to ensure both efficiency and accuracy of coupling simulation. Moreover, a co-simulation approach is developed on the basis of the Functional Mock-up Interface, overcoming the bottleneck of cross-level information interaction and computing time synchronization. The practical utility of the proposed framework is verified by a real-world engineering problem, i.e., satellite temperature field perception and attitude control in the context of the harsh space environment. Whole process engineering experiments demonstrate the effectiveness of the HDT, with mean absolute errors below 1 K for temperature field reconstruction and within 1° for attitude control. [ABSTRACT FROM AUTHOR]

Given the pervasive use of deep learning models across various domains, ensuring model security has emerged as a critical concern. This paper examines backdoor attacks, a form of security threat that compromises model output by poisoning the training data. Our investigation specifically addresses backdoor attacks on object detection models, vital for security-sensitive applications like autonomous driving and smart city systems. Consequently, such attacks on object detection models could pose significant risks to human life and property. Consequently, backdoor attacks on object detection could pose serious threats to human life and property. To elucidate this security risk, we propose and experimentally evaluate five backdoor attack methods for object detection models. The key findings are: (1) Unnecessary Object Generation : a globally embedded trigger creating false objects in the target class; (2) Partial Misclassification : a trigger causing specific class misclassification; (3) Global Misclassification : a trigger reclassifying all objects into the target class; (4) Specific Object Vanishing : a trigger causing non-detection of certain objects; (5) Object Position Shifting : a trigger causing bounding box shifts for a specific class. To assess attack effectiveness, we introduced the Attack Success Rate (ASR), which can surpass 1 in object detection tasks, thus providing a more accurate reflection of the attack impact. Experimental outcomes indicate that the ASR values of these varied backdoor attacks frequently approach or surpass 1, demonstrating our method's capacity to impact multiple objects simultaneously. Additionally, to augment trigger stealth, we introduce Backdoor Attack with Wavelet Embedding (BAWE), which discreetly embeds triggers as image watermarks in training data. This embedding method yields more natural triggers with enhanced stealth. Highly stealthy triggers are less detectable, significantly increasing the likelihood of attack success and efficacy. We have developed a Transformer-based network architecture, diverging from traditional neural network frameworks. Our experiments across various object detection datasets highlight the susceptibility of these models and the high success rate of our approaches. This vulnerability poses significant risks to digital twin systems utilizing object detection technology. Our methodology not only enhances trigger stealth but also suits dense predictive tasks and circumvents current neural network backdoor attack detection methods. The experimental findings expose key challenges in the security of object detection models, particularly when integrated with digital twins, offering new avenues for backdoor attack research and foundational insights for devising defense strategies against these attacks. • We embed watermarks as invisible triggers in training data using wavelet transform. • Our method is suited for dense prediction tasks like object detection. • We use watermarks as triggers and are not limited to standard neural network. [ABSTRACT FROM AUTHOR]

Accurate and efficient inspection of rebar dimensions has proven to be a persistent challenge for researchers and practitioners. This paper introduces a semantically enriched 3D model-based system that employs computer vision and deep learning for location-aware identification and tracking of rebar issues. The system comprises four modules: (A) digital twin generation, (B) segmentation, (C) inspection, and (D) issue identification and tracking. The generation module constructs 3D models from rebar structures. The segmentation and inspection modules analyze the 3D models, enriching them with semantic information. The issue identification and tracking module exchanges information between the semantically enriched 3D models and the building information models across time. An experiment on a column rebar cage is conducted. A precision of over 90% and a recall of over 97% are reported in 3D instance segmentation. Diameter inspection achieves an accuracy of 95.5% for large-size rebars. Spacing inspection achieves a mean relative error of 0.98%. The defective spacing is identified and tracked. • Fuses deep learning and computer vision to construct 3D semantic rebar model. • BIM- and digital-twin-based information system fulfills rebar quality control. • Autonomous rebar inspection pipeline directly generates issue reports from images. [ABSTRACT FROM AUTHOR]

This paper establishes a digital twin (DT) enabled model that aims to achieve real-time monitoring and control to enhance the overall performance of tunnel boring machines (TBM) during excavation. Through the Internet of Things (IoT) sensors that are installed on TBM, the actual operating status is transferred from the physical object to the digital model. Deep learning models that integrated graph convolutional network (GCN) and long short-term memory (LSTM) networks are employed for reliable TBM performance estimation. An online multi-objective optimization (MOO) approach is then proposed to determine the optimal operating parameters for advanced control. Real data from Singapore's Circle Line 6 C885 project is utilized as a case study to examine the applicability of the developed digital twin model. TBM's penetration rate, over-excavation ratio, energy consumption, and tool wear are selected as the objectives (O 1 to O 4) for optimization. The results indicate that (1) The digital twin model can provide a reliable estimation of TBM's performance indicators with a high coefficient of determination (R 2) values. (2) TBM's overall performance can be effectively improved through the proposed online optimization method, with an overall improvement of 21.12%. (3) The proposed online optimization method is more practical and creates better improvement than the conventional MOO approach. This study addresses the existing research gaps of lacking a reliable method to guide TBM operation with enhanced performances for the major objectives. It also contributes to proposing a novel intelligent approach that integrates DT, deep learning, and an online-NSGA-II method for TBM operation optimization during tunnel excavation. • A digital twin enabled approach is proposed for real-time intelligent control for TBM operation. • An MOO approach that integrates deep-learning and online optimization is established. • Data from a realistic tunnel project in Singapore is used to demonstrate as a case study. • TBM's efficiency, safety, energy consumption and tool wear are considered as objectives. • An overall improvement of 21.12% is achieved by implementing the proposed method. [ABSTRACT FROM AUTHOR]

• Numerical simulation modeling is complex and computationally intensive. • Digital twin deployment requires lightweight and responsive systems. • Process sensor data drives deep learning algorithms to implement digital twins. • Generative adversarial networks enable monitoring of heat distribution changes. • Results contribute to weld quality assurance and blade structure optimization. In this paper, a novel method of using deep learning to construct a digital twin of the thermos-compression bonding blade's temperature field is presented. Usually, a heat transfer model based on numerical analysis can calculate the heat distribution in the welding process, but its complex modeling process and heavy computational effort are not appropriate for the deployment of the digital twin concept. The approach proposed in this paper is distinctive because it directly converts the collected process sensor data into the bonding blade's temperature field images and the bonding surface's temperature distribution curves for monitoring the welding process. This is achieved by training the generator and discriminator of the conditional depth convolutional generative adversarial network separately. The network is then capable of predicting the bonding blade's heat distribution within a certain range of welding parameters. Compared to methods based on numerical simulation, this approach is far more effective. Training and testing datasets were created to assess the suggested method's efficacy. Multi-scale structural similarity index, structural similarity index, peak signal-to-noise ratio, difference hash and mean absolute error were used to evaluate the simulations' quality. The results show that the simulation's results and the actual data agree quite well. The temperature changes that occur during the welding process may be quickly and accurately simulated using the proposed technology. [ABSTRACT FROM AUTHOR]

• Deep features are extracted from the CT images using deep convolutional neural network model with dense layers. • Hyper edges are constructed for each images by using extracted deep features. A novel hypergraph convolutional neural network is constructed and trained by representational learning. • Evaluated the performance of the trained neural network with test data and attained ideal accuracy, precision, sensitivity, specificity, recall and F1-score. Kidney failure, also called as end-stage renal disease is the final stage of chronic kidney disease(CKD). Kidney failure cannot be revered and is the life-threatening if left untreated. Hence there is a demand for the early and automatic detection of CKD from radiology images based on deep learning techniques. This work focuses on the diagnosis of the predominant CKDs such as kidney stone, cyst and tumor from CT scan images. CT scan abdomen and urogram images are used to construct the digital-twin for the diagnosis of CKD in the robust manner. The proposed work is based on the dataset which contains 12,446 unique CT scan images, within it in which the cyst contains 3,709, normal 5,077, stone 1,377, and tumor 2,283. Deep features are extracted from these images and generated hyper edges from these features. These features are used to construct hypergraph representing the renal images. These hypergraphs are used in hypergraph convolutional neural network for representational learning. Finally, the model is validated by hold-out dataset. Deep learning metrics including precision, recall, accuracy, and FI score are used to validate the proposed model. It outperforms compared to other state of art algorithms with a superior validation accuracy of 99.71. The proposed model is a robust digital-twin for the early diagnosis of kidney diseases. It will assist the nephrologist for the better prognosis of kidney related abnormalities. [ABSTRACT FROM AUTHOR]

The digital twin concept represents an appealing opportunity to advance condition-based and predictive maintenance paradigms for civil engineering systems, thus allowing reduced lifecycle costs, increased system safety, and increased system availability. This work proposes a predictive digital twin approach to the health monitoring, maintenance, and management planning of civil engineering structures. The asset-twin coupled dynamical system is encoded employing a probabilistic graphical model, which allows all relevant sources of uncertainty to be taken into account. In particular, the time-repeating observations-to-decisions flow is modeled using a dynamic Bayesian network. Real-time structural health diagnostics are provided by assimilating sensed data with deep learning models. The digital twin state is continually updated in a sequential Bayesian inference fashion. This is then exploited to inform the optimal planning of maintenance and management actions within a dynamic decision-making framework. A preliminary offline phase involves the population of training datasets through a reduced-order numerical model and the computation of a health-dependent control policy. The strategy is assessed on two synthetic case studies, involving a cantilever beam and a railway bridge, demonstrating the dynamic decision-making capabilities of health-aware digital twins. • We present a digital twin framework for the SHM of civil structures. • The asset-twin dynamical system is encoded through a probabilistic graphical model. • Sensor recordings are assimilated with artificial neural networks. • A reduced-order model is employed offline to generate training vibration responses. • Our strategy allows predictive maintenance and management planning of structures. [ABSTRACT FROM AUTHOR]

Decoupling the physical world and providing standardized service interfaces is still challenging when developing Location Based Services (LBS). This lack also hinders the possibility of developing Intelligent services on top of LBS architectures. In this paper, we propose a multi-layer Digital Twin-based architecture that aims to enable the development of machine learning-based Intelligent LBS (I-LBS) that are able to adapt, evolve, and perform Continual Learning (CL). The platform uses Digital Twins to ensure physical abstraction and provide cyber–physical knowledge to the I-LBSs, which is defined as an execution graph of operation modules. Finally, we simulated a use-case for this platform in the complex scenario of Healthcare organization and management where the I-LBS classifies allowed/not allowed trajectories of users inside a real-existing hospital scenario depending on their role in the organization. The use case is implemented as a Deep Learning-based reconstruction task of high-resolution trajectories processed by the DT architecture that also deploys the I-LBS. The platform is evaluated in terms of physical complexity and computational time on the DT side and on both a traditional machine learning setting and a replay-based CL one for the intelligence side to demonstrate the flexibility and adaptability features introduced by the components for dynamic or unseen scenarios. [ABSTRACT FROM AUTHOR]

This work proposes a new method for the detection, localization, and classification of grid-connected photovoltaic (PV) array faults. Line-to-line, open-module, shorted-module, open-string, and shorted-string faults as well as partial shading conditions are studied. The proposed method has two stages, which are (1) detection and localization of faults and (2) classification of faults. In the first stage, detection and localization are performed using a digital twin (DT) by analyzing the current ratio of each PV array. The measured DC (direct current) power after the operation of the DC/DC boost converter in the physical object is firstly converted into a 2-dimensional image using a recurrence plot (RP) and is then inputted to the second stage. A deep learning-based shifted windows (swin) transformer optimized by particle swarm optimization (PSO) is used in the classification stage, eliminating the need for model tuning by trial-and-error. A PV system of ten arrays with 49 kW is studied. The coefficients of determination (R2) between the results of the digital object (digital twin) and the physical object for different scenarios demonstrate the accuracy and success of the digital twin. R2 values of 0.99988 for varying irradiation with constant temperature and 0.97923 for constant irradiation with varying temperature indicate strong correlations between the digital and physical objects, further confirming the applicability of the digital twin in PV fault detection. The comparative evaluation of the PSO-optimized swin transformer against classical machine learning algorithms and state-of-the-art convolutional neural networks reveals the superior performance of the proposed method. It achieves an outstanding classification accuracy of 98.55%, demonstrating its ability to effectively classify various types of PV faults. The results of the area under the curve (AUC) of the receiver operating characteristic (ROC) curve, which measures the trade-off between true positive rate (TPR) and false positive rate (FPR), further illustrate the effectiveness of the proposed method for PV fault classification. • A model-based digital twin is used to detect and locate faults in a PV system. • Shifted Window Transformer with few trained variables is used to classify PV faults. • Topology of deep learning model is optimized by particle swarm optimization. [ABSTRACT FROM AUTHOR]

Digital twins (DTs) have attracted widespread attention in academia and industry in recent years. It can accurately reflect the physical world in real-time, enabling online monitoring, control, and prediction operations. Their foundation is super-real-time computing and high data representation capabilities. However, current DTs do not achieve 3D super-real-time computing. This study proposes a novel 3D computational method for solving fluid–solid coupling problems in a super-real-time. The method is based on a mixed solution framework that combines traditional numerical methods with deep learning operators. Specifically, the method employs multi-core CPU parallel acceleration to solve the solid equations while leveraging the computing power of GPU to solve the fluid equations. The fluid–solid coupling is achieved through information exchange between the GPU and the multi-core CPU. In addition, the proposed method introduces a new deep learning operator framework based on the DeepONET. The framework is accompanied by a database structure that facilitates model training and validation and a loss function that guides the training. The space nuclear reactor, an improved TOPAZ-II system, was selected to demonstrate its feasibility. Four non-training transient conditions were simulated to test the generalization performance. The results show that the proposed method achieves an average error between the calculated results and reference values below 2.5%, with the average error of thermodynamic parameters below 1.5%. The average deviation between system parameter peak values during the transient process and the reference value was less than 5 s. The result meets the acceptable error level and satisfies the super-real-time requirements with a time acceleration ratio of approximately 1.17, which is 60 times faster than traditional numerical methods. The results demonstrate the accuracy and efficiency of the proposed method for DT. [ABSTRACT FROM AUTHOR]

Preparing the infrastructure for analyzing, recognizing, and characterizing microorganisms in the Metaverse can transform the fields of biology, medicine, and drug discovery. Accordingly, the realization of digital twins of microorganisms, such as viruses, fungi, algae, bacteria, protozoa, archaea, and multicellular animal parasites, can streamline the applicability of the Metaverse and similar emerging technologies like Cyber-Physical Healthcare Systems (CPHS). This is why a new approach to the digital twinning of bacteria has been presented in this research. This method of digital twinning can revolutionize the research and study of bacteria because it allows us to separate useful and harmful bacterial species, increasing the efficiency of treatment noticeably. This innovative method can easily be used by clinics, medical centers, or even by private users physically and virtually and can be adapted to every centralized or decentralized Metaverse Platform. To determine the proper treatment, biologists have always tried to identify the correct bacterial species that prompted a bacterial infection. They use various indicators, such as the bacterial cell's shape and the size of the colony formed by the bacteria, to classify different types of bacteria with different biochemistries and shapes. However, it is challenging because of the extensive similarities between some species. For instance, such similarities exist between Staphylococcus aureus and Staphylococcus saprophyticus, which has caused numerous false diagnostic reports by operators. Wrong species reports bring about treatment failure and increase antibiotic resistance issues. Therefore, the digital twins of bacteria cover all of their identifiable characteristics and overcome many limitations to study them. In this approach, DTL techniques, including MobileNetV2, EfficientNetV2-S, and ResNet-50, have been employed to build digital twins of bacteria species. A hybrid dataset was used for training and evaluation. Among the models, EfficientNetV2-S exhibited the best performance, with a validation accuracy of 99.58% and a test accuracy of 99.33%. The results showed the ability of deep learning models to make bacteria digital twins based on image processing from different labs, thus assisting experts in speeding up the process and reducing diagnostic errors. In addition, in the mispredicted cases, the correct species was among the first three choices of the model. Therefore, not only can experts use DTL approaches to speed up the digital twin realization of microorganisms in the Metaverse, but they can also use these methods to reduce diagnostic errors. • Introducing a technique to streamline digital twinning of microorganisms using CNN. • Describing machine learning digital twins of microorganisms in the Metaverse. • Presenting an exact, economical digital twinning approach via smartphone photography. • Providing a comprehensive bacteria species dataset for future method development. [ABSTRACT FROM AUTHOR]

In the Architecture, Engineering, and Construction (AEC) sector, digital twins rely on precise 3D models to convey digital information about physical structures in a virtual space. However, due to the vulnerability to measurement errors in weak-textured regions, 3D point clouds generated by conventional photometric consistency or deep learning-based Multi-View Stereo (MVS) algorithms are often incomplete or inaccurate. Therefore, this paper integrates the consistency constraint across multiple views and the inferential capacity of deep learning to propose a novel approach for refining the missing regions of the depth maps generated by photo-consistency based MVS algorithms. The proposed solution involves a cost volume pyramid-based depth completion (CVP-DC) network with three multi-level pyramid structures, which sequentially estimates and completes depth maps in a coarse-to-fine manner. A dataset that consists of input images and the corresponding depth maps generated by photo-consistency based MVS algorithms, along with output ground truth depth maps, is developed using an open DTU MVS dataset. CVP-DC demonstrates competitive performance when tested on the public DTU MVS dataset, outperforming existing MVS algorithms in terms of both completeness and accuracy. Additionally, experimental studies are conducted utilizing UAV-collected RTK (Real-Time Kinematic) images of an outdoor bridge pier to reconstruct point clouds with absolute scales. Experimental validations demonstrate the effectiveness and applicability of the proposed approach in filling uneven and incomplete depth maps, thereby enhancing the completeness of the generated point clouds. The proposed approach holds promise for establishing precise 3D models for the digital twin of the AEC sector. [ABSTRACT FROM AUTHOR]

Aortic stenosis (AS) is the most prevalent heart valve disease in western countries that poses a significant public health challenge due to the lack of a medical treatment to prevent valve calcification. Given the aging population demographic, the prevalence of AS is projected to rise, resulting in a progressively significant healthcare and economic burden. While surgical aortic valve replacement (SAVR) has been the gold standard approach, the less invasive transcatheter aortic valve replacement (TAVR) is poised to become the dominant method for high- and medium-risk interventions. Computational simulations using patient-specific models, have opened new research avenues for optimizing emerging devices and predicting clinical outcomes. The traditional techniques of generating digital replicas of patients' aortic root, native valve, and calcification are time-consuming and labor-intensive processes requiring specialized tools and expertise in anatomy. Alternatively, deep learning models, such as the U-Net architecture, have emerged as reliable and fully automated methods for medical image segmentation. Two-dimensional U-Nets have been shown to produce comparable or more accurate results than trained clinicians' manual segmentation while significantly reducing computational costs. In this study, we have developed a fully automatic AI tool capable of reconstructing the digital twin geometry and analyzing the calcification distribution on the aortic valve. The developed automatic segmentation package enables the modeling of patient-specific anatomies, which can then be used to simulate virtual interventional procedures, optimize emerging prosthetic devices, and predict clinical outcomes. • Aortic stenosis, the most common valve disease, lacks a preventative treatment for valve calcification. • Computer models for TAVR enhancement hindered by digital twin generation burden. • Our AI tool automatically creates digital replicas of case-specific anatomy from CT. • Digital twin allows calcification distribution mapping, informing diagnosis/therapy. • Our tool feeds virtual surgery and device optimization to improve patient outcome. [ABSTRACT FROM AUTHOR]

• We develop a clinically interpretable dual-stream AI model for the automatic diagnosis of fetal distress. • We combine clinical knowledge traction to quantize numerous clinical diagnostic indicators of authoritative guidelines. • We employ the TreeSHAP-based white box interpreter and CNN-based case tracking model to enhance clinical interpretability. • Our approach achieves a high precision pathological diagnosis of 0.963, which provides solutions for future AI-aided diagnosis systems. Clinically, Fetal Heart Rate (FHR)-based intelligent cardiotocography classification to diagnose fetal well-being is of utmost importance for obstetricians and gynecologists. However, the current Fetal Distress Diagnosis (FDD) algorithms based on Artificial Intelligence (AI) methods are designed as black box models that usually result in poor performance on clinical interpretation and lack explicit diagnostic evidence. This paper focuses on two aspects: enhancing the accuracy of FD and providing explanations for the model's decisions. Specifically, we propose a novel clinically interpretable dual-stream AI architecture for FDD. The two streams embed multi-feature category representations and the original FHR into the Digital Twin model (DT) and the deep convolutional neural Networks-based Case Tracking model (CTNet), respectively, thus marking as DT-CTNet. And the former makes the input-outcome dependency transparent, while the latter provides the samples that are the most similar to the object being diagnosed. By this design, not only is high-precision FDD achievable, but a transparent explanation with global, local and instantiation interpretation can be obtained. To the best of our knowledge, this work is the first to provide an interpretable diagnosis of fetal distress. The proposed method is comprehensively evaluated from five aspects: classification model accuracy, explanation of the classification model decision, consistency of the evidence explanation, pathological difference in visual analysis, and a comparison with state-of-the-art methods. Experimental results, which are performed on a challenging dataset, show that the proposed fetal distress diagnosis method outperforms the state-of-the-art methods and has a high accuracy (0.963). [ABSTRACT FROM AUTHOR]

Leak detection in natural gas pipelines is an extremely important and persistent problem in the oil and gas industry. The construction of accurate physical models of pipelines is limited by the high complexity, unavailable closed-form solution, and strict experienced personnel requirements. Moreover, industrial automation has the common problems of large amount of data with little information. This paper proposes a data-driven digital twin (DT) method for leak detection as a new paradigm solution to these challenges. From the perspective of knowledge-based data-driven, a DT pipeline learning and updating scheme based on normal data directly from operational data during the entity pipelines life-cycle is proposed to enhance DT adaptability. A DT-driven leak detection method is proposed, making effective use of data interaction and fusion of DT. The effectiveness and performance of the proposed approach is illustrated by deploying the DT pipeline in a simulated leak scenario of a real running natural gas pipeline. © 2012 Published by Elsevier Ltd. Selection and/or peer-review under responsibility of Global Science and Technology Forum Pte Ltd [ABSTRACT FROM AUTHOR]

Early failure detection is required for Fused Deposition Modelling (FDM) 3D printers to reduce material waste. Typically, such systems are created based on images captured during printing or sensor data for tracking the extruder's movement. This work presents a novel approach to sensor data-driven fault diagnosis, utilizing Artificial Intelligence (AI) technology to investigate the temperature imbalance in the extruder and printing surface. First, a Lightweight Convolutional Neural Network (LCNN) is proposed to detect faults from sensory data. The model's architecture concatenates the CNN layer to extract additional features, improving the model's performance while maintaining a lightweight configuration suitable for real-time monitoring systems. Second, the concept of Digital Twin (DT) technology for FDM 3D printer fault detection is introduced. The DT creates a virtual representation of a physical object, and its functionality is validated by examining the network's latency and System Overhead (SO) as the number of clients increases. The simulation results show that the proposed LCNN with a DT environment can effectively monitor, detect, and control the physical workplace with an F1-Score of 0.9981 and an average latency of 995. 4253 ms. Additionally, this research contributes to the development of future technologies for virtual condition monitoring of 3D printer abnormalities, which will be essential for intelligent and autonomous factories. [ABSTRACT FROM AUTHOR]

The Industrial 5.0 Model integrates enabling technologies such as deep learning, digital twins, and the meta-universe with new development concepts. However, model and data security may pose challenges for developing zero-defect production and other industrial manufacturing industries. To address this issue, we generate adversarial examples using a one-pixel attack in adversarial machine learning, which can fool the defect detection classification model. The traditional one-pixel attack based on the Differential Evolution (DE) algorithm has limited global search ability. Therefore, we use a novel algorithm called T eaching and L earning-based M oth- F lame O ptimization (TLMFO), which enhances the global search performance and improves the attack effectiveness. We evaluate TLMFO on benchmark functions and attacks on Cifar10 and ImageNet datasets, and compare it with MFO and DE. The results show that TLMFO outperforms both MFO and DE in terms of accuracy and speed of convergence. Moreover, TLMFO achieves notably better attack effectiveness than DE under targeted and untargeted attacks on the Cifar10 dataset and under-targeted attacks on the ImageNet dataset. Our research confirms that safety prevention is a link worth considering in developing Industry 5.0. [ABSTRACT FROM AUTHOR]

Digital twins are undergoing growth that enables highly informative and scaleable interaction between physical objects and virtual twins, which is of great significance to the life cycle analysis of composites. Real-time fine evolution is challenging due to vast combinations of input features and high-resolution calculated variables. Here, we systematically demonstrate an AI-based methodology for digital twinning of complex composite structures. First, three types of deep neural networks are created with optionally used autoencoders as surrogate models, with architectures and data processing inspired by the rule-of-mixture of composites. Second, the prediction accuracy and efficiency are evaluated quantitatively and qualitatively, demonstrating the feasibility of predicting 3D displacement and stress fields directly from sensing data of temperature, pressure and loading displacement, and the optimal architecture is selected to be the evolving digital twin. Finally, the real-time interactive experiments are relayed to the digital twin and demonstrated that it can interact with physical objects and evolve online with high accuracy. These results indicate that the computational time can be reduced by 3∼6 orders of magnitude with high information intensity and scalability compared with conventional numerical and experimental methods, which opens up the avenues for the cost-effective and efficient development of digital twin services for composites. [Display omitted] [ABSTRACT FROM AUTHOR]

Finite element method (FEM) based high-fidelity simulation can be computationally demanding and time-consuming as engineering problems become more complicated. It is thus necessary to develop a surrogate model that only requires a small amount of computational time but retains sufficient accuracy. A graph neural network (GNN) based framework is proposed as a general surrogate model for FEM to simulate the Von Mises stress distribution. The mesh body is embedded to a graph and a novel global attribute representation is introduced to capture the geometry and boundary conditions while overcoming the common problem of over smoothing in graph deep learning. The challenge to deal with varying geometry and boundary conditions is overcome by the proposed model and thus it outperforms existing methods such as proper orthogonal decomposition (POD) and greedy algorithm in terms of generalization. Numerical experiments are given to demonstrate the capability of the model developed and the results show that the proposed model not only accurately predicts the stress distribution but also speed-ups hundreds of times faster compared to a FEM-based simulator, enabling real-time structural response analysis for the application of digital twin and structural health monitoring. It is indicated that GNNs can be a powerful tool for resolving complex physical problems, thereby assisting in advancing science and enhancing engineering productivity. • A surrogate model is developed to accelerate the numerical simulation. • Global attributes are proposed to mitigate potential issues with graph deep learning. • Accurate prediction of physical quantities of interest is achieved. • Significant improvements in prediction speed over numerical simulation are achieved. • Powerful generalizability is obtained through the use of graph neural networks. [ABSTRACT FROM AUTHOR]

Current procedures for the rapid inspection of buildings and infrastructure are subjective, time-consuming, and cumbersome to document, necessitating new technologies to automate the process and eliminate these shortcomings. Fortunately, recent developments in imaging devices and artificial intelligence, such as computer vision, provide the necessary tools for this, though they are not yet integrated into infrastructure applications. In this paper, we propose an end-to-end pipeline that generates damage-augmented digital twins for buildings at LOD3, including geometrical information as well as data pertaining to damage condition and its characterization. Our framework incorporates multiple-view images to (1) create a level of detail model, (2) segment damage information, and (3) characterize damage. The core of the method is the structure from motion, which is used to reconstruct the building scene, and machine-learning models that segment and characterize damage. In contrast to current practices, our method does not require manual intervention, generates lightweight models, and can be applied to a wide range of assets. The results generated with our pipeline represent a significant step towards an automated infrastructure damage assessment. We intend to expand our work in the future to include real-time applications and applications to other types of infrastructure. Codes and data sets are publicly available (https://github.com/eesd-epfl/DADT%5fbuildings and https://doi.org/10.5281/zenodo.7767478). • A novel framework to generate Damage Augmented Digital Twins (DADT) of buildings. • State-of-the-art solutions for 3D models, crack segmentation and characterization. • Bench-marking data set for evaluating the automated generation of DADT of buildings. [ABSTRACT FROM AUTHOR]

The shop floor has always been an important application object for the digital twin. It is well known that production, process, and product are the core business of the shop floor. Therefore, the digital twin shop floor covers multi-dimensional information and multi-scale application scenarios. In this paper, the digital twin shop floor is constructed according to the modeling method of the complex digital twin proposed in Part I. The digital twin shop floor is firstly divided into several simple digital twins that focus on scenarios of different scales. Two simple application scenarios are constructed, including tool wear prediction and spindle temperature prediction. Main functions in different application scenarios, such as data acquisition, data processing, and data visualization, are implemented and encapsulated as components to construct simple digital twins. Secondly, ontology models, knowledge graphs, and message queues are used to assemble these simple digital twins into the complex digital twin shop floor. And two complex application scenarios are constructed, including machining geometry simulation considering spindle temperature and production scheduling considering tool wear. The implementation of the complex digital twin shop floor demonstrates the feasibility of the proposed modeling method. [ABSTRACT FROM AUTHOR]

• Transfer learning-based variable-fidelity approach for real-time plume tracking. • Case study of gas release and dispersion from LNG terminal is conducted. • Our approach saves 70% high-fidelity computations with desirable accuracy. Deep learning has been used to track the real-time flammable plume of natural gas. However, a large volume of high-fidelity data is required to train the deep learning model for sufficient accuracy in congested industrial environments, which can be computationally prohibitive. This study proposes a transfer learning-based variable-fidelity approach for real-time plume tracking. A Gaussian dispersion model was applied to efficiently generate a large volume of low-fidelity data, which is then used to pre-train the deep learning model. A limited number of high-fidelity simulations were conducted by solving the Navier-Stokes equation to fine-tune the pre-trained model. A case study demonstrated our proposed approach could reduce high-fidelity computations by 72% while ensuring prediction accuracy with R2 =0.96 for released plume area estimation in congested chemical facilities. Optimal number of frozen layers, learning rate and the number of high-fidelity simulations required were determined to ensure adequate efficiency for this approach. This study provides an efficient alternative to improve the generalization of deep learning for real-time plume area estimation for large-scale congested chemical plants. [ABSTRACT FROM AUTHOR]

Digitalization of Nuclear Power Plants (NPPs) is critical for their safe and effective operation and maintenance. Development of Digital Twins (DTs) of NPP legacy assets and subsystems is key to achieving this goal. Doing this effectively requires a framework for intelligent allocation of limited resources. This framework is developed here by synthesizing emerging best practices with NPP operators' needs for legacy assets management. Within the framework, a pipeline employs deep-learning object detection to read and locate equipment tags in images. It computes their locations in the corresponding 3D point clouds and then relates that data to an asset management system. The pipeline is premised on preservation and augmentation of existing NPP asset management processes that preclude options such as RFID tags or barcodes. It is a significant step toward more efficient development of DTs of legacy assets. The contributions are framed in the context of a typical Canadian legacy NPP. • A framework that defines the key aspects of legacy assets for Digital Twins. • Automatic link between photographic records and asset management software. • Performance of tag detection and optical character recognition on real images. • Pipeline for efficient linking of assets in point clouds to management systems. • Identification of asset locations using legacy asset tags in nuclear power plants. [ABSTRACT FROM AUTHOR]

The paper proposes a surrogate model for liquid moulding of structural composites. A methodology is presented to simulate the dual-phase Darcy's flow in a heterogeneous porous medium. The approach is an encoder–decoder that receives as input a matrix of permeabilities and produces two scalar fields that represent the pressure and front flow. This model is trained with synthetic data generated with a computer fluid dynamics simulator. In this context, the lack of robustness of models trained with the popular L 2 and L 1 losses is highlighted and several enhancements to these baseline approaches are introduced. First, the study provides a piece-wise power-logarithmic loss that improves training in the presence of the bimodal distribution of error residuals produced by the dual-phase flow predictions. A non-uniform sampling strategy for the selection of time training snapshots is also included, which contributes to improve the prediction accuracy. The estimation of the front flow field is further refined with a multi-task training strategy. The introduction of these improvements in the baseline models reduce the relative error of the pressure and front flow fields by more than 50%, performing these simulations in a record time of 50 ms. The surrogate model is further evaluated as a digital twin to predict – in a real experiment – the location and spatial extent of race-tracking channels and regions with dissimilar degrees of permeability. [ABSTRACT FROM AUTHOR]

Using digital material twins reveals the fabric architectures of woven composites and details the morphological features of fiber tows and defects, which makes a unique approach to accessing the mechanical performance of woven composites, especially the damage and failure behaviors. However, the current major challenges are accurately and efficiently segmenting low-contrast digital images, such as μCT images or MRI images, and reconstructing 3D braided structures. In this paper, a ResL-U-Net convolutional neural network is proposed, in which we use the leaky-ReLU activation function to improve efficiency. We use the residual structure to prevent network degradation and obtain excellent robustness and accuracy. The initial manufacturing defects are implanted into the finite element model by three-dimensional spatial mapping, which ensures the authenticity of defects on the one hand and high-quality mesh on the other. The damage evolution and fracture features of the GFRP material are investigated based on digital material twins; simultaneously, an in-situ CT tensile experiment is tailored to verify the simulations. It is shown that the simulations by the material twins demonstrated the mechanical performances of the GFRP very well, especially on the damage locations and material failure patterns. [ABSTRACT FROM AUTHOR]

Earthwork operations are crucial parts of most construction projects. Heavy construction equipment and workers are often required to work in limited workspaces simultaneously. Struck-by accidents resulting from poor worker and equipment interactions account for a large proportion of accidents and fatalities on construction sites. The emerging technologies based on computer vision and artificial intelligence offer an opportunity to enhance construction safety through advanced monitoring utilizing site cameras. A crucial pre-requisite to the development of safety monitoring applications is the ability to identify accurately and localize the position of the equipment and its critical components in 3D space. This study proposes a workflow for excavator 3D pose estimation based on deep learning using RGB images. In the proposed workflow, an articulated 3D digital twin of an excavator is used to generate the necessary data for training a 3D pose estimation model. In addition, a method for generating hybrid datasets (simulation and laboratory) for adapting the 3D pose estimation model for various scenarios with different camera parameters is proposed. Evaluations prove the capability of the workflow in estimating the 3D pose of excavators. The study concludes by discussing the limitations and future research opportunities. [ABSTRACT FROM AUTHOR]

• Combining DT and hierarchical DL to realize intelligent identification of structural damage. • Study an establishment method of high-fidelity digital twin model based on offline data and online data-driven. • Develop a program for automatic establishing damage database of cable dome structure. • Propose a DL framework which can identify the damage type, location and degree with high accuracy. • The approach can provide a new basis for structural damage identification with broad application prospects. Accurate identification of structural damage is the most critical step in structural health monitoring. Traditional damage identification strategies are easily interfered by environmental and human factors, resulting in low-accuracy of identification. In this light, a combined digital twin (DT) and hierarchical deep learning (DL) approach for intelligent damage identification in cable dome structures is proposed in this paper. Based on actual engineering cases, a DT model that accurately maps the physical structure of the cable dome is constructed using APDL based on data. A cable dome structure damage sample database is then automatically established through the large-scale finite element analysis of DT. Finally, the damage features of the data samples are extracted using the hierarchical DL framework proposed in this study. Accuracy verification based on cable force confirms that the established DT model can accurately reflect the mechanical state of the physical structure. The identification results of the trained network on a test set demonstrate that the proposed framework can intelligently identify the damage type, damage location, and damage degree in the cable dome structure with a high accuracy and strong robustness. The proposed intelligent damage identification approach is feasible and reliable and can provide a new basis for structural damage identification with broad application prospects. [ABSTRACT FROM AUTHOR]

• A model using graph neural network is proposed to predict state points and performance metrics. • A configuration representation method based on thermodynamic graph is developed. • GNN can extract structure features from different graphs of three SCO 2 power systems. • GNN achieves excellent accuracy and efficiency and outperforms classical data-driven models. Considering the increasing energy consumption and greenhouse gas emissions, the Supercritical CO 2 (S-CO 2) power system has attracted more and more attention. Due to the expensive computation resource and time cost, data-based solutions for performance prediction are urged. The surrogate model by machine learning is a promising alternative, but it only focuses on the objective functions and ignores the importance of topological structures and physical states of cycles. Aiming at providing a comprehensive model to predict physical states as well as thermodynamic characteristics, a deep learning method based on graph neural network (GNN) are devised in this paper. With the modeling calculation results as training dataset, a well-trained model can accurately reconstruct the physical states consisting of temperature, pressure, enthalpy, entropy (relative error of most samples <5 %) and exergy as well as thermal and exergy efficiency (relative error of most samples <5 %). Moreover, this model shows superior performance compared with traditional machine learning models including Regression Tree, Artificial Neural Network (ANN), Support Vector Machine (SVM) and Gaussian Process Regression (GPR). Finally, the comparison between different training sizes demonstrate the model can help reduce sampling costs for complex systems. Overall, the presented deep learning model can provide a reliable and competitive choice for the digital twin of S-CO 2 power system and other power systems. [ABSTRACT FROM AUTHOR]

Urban underground infrastructure is a critical component in cities to provide essential services to residents. Research efforts have been made to facilitate different activities of underground infrastructure projects using various methods, particularly digital technologies. To obtain deeper insights from existing research and provide directions for future research, this study conducts a comprehensive review of research on underground infrastructure construction and Operation & Maintenance (O&M) with a focus on digital technologies. The in-depth review was conducted based on 145 publications from the perspective of locating and mapping, construction and coordination, as well as O&M. Consequently, critical limitations and challenges are revealed, such as the lack of as-built and as-is information, the requirement of data quality and quantity for deep learning methods, the lack of fully automated robotic systems, etc. Afterwards, a status matrix was presented to identify the level of different digital technologies being studied and their future application potential for key activities of underground infrastructure projects. In the end, future research trends are proposed, including (1) digital twinning of underground infrastructure, (2) quality and uncertainty of inspection data, (3) data generation and semi-supervised learning, (4) predictive maintenance, and (5) fully automated robotic systems for inspection and maintenance. This study contributes to the body of knowledge by identifying the challenges and limitations of existing studies through a systematic review, providing a clear view of the achievements and potentials of digital technologies for underground infrastructure, and proposing future research directions to facilitate digital transformation in this area. [Display omitted] • Conducted a systematic review of research on construction and maintenance of underground infrastructure. • Identified achievements, limitations and challenges of previous research. • Presented a status matrix of digital technologies for underground infrastructure. • Proposed future research directions to improve digitalisation in underground infrastructure. [ABSTRACT FROM AUTHOR]

• Novel pipeline for automatically reconstruct LOD3 models of free-standing buildings. • New metrics for evaluating the performance of a LOD reconstruction. • Benchmark dataset for evaluating the reconstruction of SfM as well as LOD3 models. • Deep learning models for facade and opening segmentation of free-standing masonry buildings. This paper describes a pipeline for automatically generating level of detail (LOD) models (digital twins), specifically LOD2 and LOD3, from free-standing buildings. Our approach combines structure from motion (SfM) with deep-learning-based segmentation techniques. Given multiple-view images of a building, we compute a three-dimensional (3D) planar abstraction (LOD2 model) of its point cloud using SfM techniques. To obtain LOD3 models, we use deep learning to perform semantic segmentation of the openings in the two-dimensional (2D) images. Unlike existing approaches, we do not rely on complex input, pre-defined 3D shapes or manual intervention. To demonstrate the robustness of our method, we show that it can generate 3D building shapes from a collection of building images with no further input. For evaluating reconstructions, we also propose two novel metrics. The first is a Euclidean–distance-based correlation of the 3D building model with the point cloud. The second involves re-projecting 3D model facades onto source photos to determine dice scores with respect to the ground-truth masks. Finally, we make the code, the image datasets, SfM outputs, and digital twins reported in this work publicly available in github.com/eesd-epfl/LOD3_buildings and doi.org/10.5281/zenodo.6651663. With this work we aim to contribute research in applications such as construction management, city planning, and mechanical analysis, among others. [ABSTRACT FROM AUTHOR]

[Display omitted] • Laser scanning and photogrammetric technologies are fused in the pipeline. • Semantic information in images is mapped to 3D laser-scanned point clouds. • Small objects of di_erent classes are detected in the creation of digital twins. • Text information is recognised and used to enrich digital twins. This paper addresses the challenge of enriching geometric digital twins of buildings, with a particular emphasis on capturing small but important entities from the electrical and the fire-safety domain, such as signs, sockets, switches, smoke alarms, etc. Unlike most previous research that focussed on structural elements and processed laser point clouds and images separately, we propose a novel method that fuses laser scanning and photogrammetry methods to capture the relevant objects, recognise them in 2D images and then map these to a 3D space. The considered object classes include electrical elements (light switch, light, speaker, socket, elevator button), safety elements (emergency switch, smoke alarm, fire extinguisher, escape sign), plumbing system elements (pipes), and other objects with useful information (door sign, board). Semantic information like class labels is extracted by applying AI-based image segmentation and then mapped to the 3D point cloud, segmenting the point cloud into point clusters. We subsequently fit geometric primitives to the point clusters and extract text information by AI-based text detection and recognition. The final output of our proposed method is an information-rich digital twin of buildings that contains geometric information, semantic information such as object categories and useful text information which is valuable in many aspects, like condition monitoring, facility maintenance and management. In summary, the paper presents a nearly fully-automated pipeline to enrich a geometric digital twin of buildings with details and provides a comprehensive case study. [ABSTRACT FROM AUTHOR]

Registering images to a building information model is an effective approach of associating as-built component status to as-designed information. However, registering a single still image to a digital twin is difficult due to pose ambiguities caused by the limited field of view in images. Several recent studies have started focusing on leveraging panoramic images to address problems, such as partial occlusions, repetitive facility components and textureless views, which could cause registration failures in existing image-based registration workflows. Even though having a bigger field of view helps in locating where the image was taken, registering a panoramic image to a building information model is still challenging due to inconsistent visual appearances, such as model vs. image visual differences caused by temporary objects in a scene, lighting condition changes, and different levels of details. In this paper, we present a novel method that registers panoramic images to a digital twin in a hierarchical way: the proposed method first performs rough registration through image retrieval using semantic segmentation, then localizes the image with a fine registration through minimizing semantic reprojection errors. Compared to existing methods, the developed method has the following contributions: 1) semantic-based image retrieval makes the rough registration process robust to lighting condition changes, texture differences, and temporary objects, 2) semantic-based image retrieval allows for bi-directional queries between a model and images, 3) reprojection-based fine registration further reduces the localization error due to the dimension reduction of features during image retrieval. Though the proposed method is developed for panoramic images, it can be generalized to monocular images at the cost of localization accuracy. The developed method was evaluated on a real-world academic building and a synthetic dataset, and the results showed that the proposed method can localize a panorama in less than a second and achieve sub-meter level localization error. • A new panorama-to-BIM registration method is developed. • A new image retrieval approach is developed for hierarchical registration. • Semantic features help address visual differences between BIM and images. • Accuracy and robustness of image-to-BIM registration are improved. [ABSTRACT FROM AUTHOR]

Spatial locations of personnel, equipment, and materials are constantly changing as construction projects progress. The dynamic nature of the construction industry affects workers' performance of identifying hazards. Even though a great deal of effort has been made to improve construction safety, the construction industry still witnesses a high accident rate. In order to complement the existing body of knowledge relating to construction safety, this paper integrates Digital Twin (DT), Deep Learning (DL), and Mixed Reality (MR) technologies into a newly developed real-time visual warning system, which enables construction workers to proactively determine their safety status and avoid accidents. Next, system tests were conducted under three quasi-on-site scenarios, and the feasibility was proven in terms of synchronising construction activities over a large area and visually representing hazard information to its users. These evidenced merits of the development testing scenarios can improve workers' risk assessment accuracy, reinforce workers' safety behaviour, and provide a new perspective for construction safety managers to analyse construction safety status. • The proposed system generates simulated virtual construction sites based on the digital twin concept. • The virtual construction sites integrate BIM models and live video data for hazard identification. • Wearable mixed reality devices can present real-time hazard information to individual workers intuitively. • Three preliminary experiments have demonstrated the feasibility of the proposed system. [ABSTRACT FROM AUTHOR]

By constructing digital twins (DT) of an integrated energy system (IES), one can benefit from DT's predictive capabilities to improve coordinations among various energy converters, hence enhancing energy efficiency, cost savings and carbon emission reduction. This paper is motivated by the fact that practical IESs suffer from multiple uncertainty sources, and complicated surrounding environment. To address this problem, a novel DT-based day-ahead scheduling method is proposed. The physical IES is modelled as a multi-vector energy system in its virtual space that interacts with the physical IES to manipulate its operations. A deep neural network is trained to make statistical cost-saving scheduling by learning from both historical forecasting errors and day-ahead forecasts. Case studies of IESs show that the proposed DT-based method is able to reduce the operating cost of IES by 63.5%, comparing to the existing forecast-based scheduling methods. It is also found that both electric vehicles and thermal energy storages play proactive roles in the proposed method, highlighting their importance in future energy system integration and decarbonisation. • A deep learning and digital twins (DT) based scheduling method is proposed. • Physical IES performance is addressed by its virtual replica's predictive capability. • A machine learning method is proposed to address multiple uncertainty challenges. • Historical U.K. datasets are used for performance evaluations. • Results demonstrate the Digital Twin model reduces the extra operating cost by 63.5%. [ABSTRACT FROM AUTHOR]