Search

2D-3D registration is increasingly being applied in various scientific and engineering scenarios. However, due to appearance differences and cross-modal discrepancies, it is demanding for image and point cloud registration methods to establish correspondences, making 2D-3D registration highly challenging. To handle these problems, we propose a novel and automatic solution for 2D-3D registration in Manhattan world based on line primitives, which we denote as VPPnL. Firstly, we derive the rotation matrix candidates by establishing the vanishing point coordinate system as the link of point cloud principal directions to camera coordinate system. Subsequently, the RANSAC algorithm, which accounts for the clustering of parallel lines, is employed in conjunction with the least-squares method for translation vectors estimation and optimization. Finally, a nonlinear least-squares graph optimization method is carried out to optimize the camera pose and realize the 2D-3D registration and point colorization. Experiments on synthetic data and real-world data illustrate that our proposed algorithm can address the problem of 2D-3D direct registration in the case of Manhattan scenes where images are limited and sparse. [ABSTRACT FROM AUTHOR]

High-speed biplanar videoradiography can derive the dynamic bony translations and rotations required for joint cartilage contact mechanics to provide insights into the mechanical processes and mechanisms of joint degeneration or pathology. A key challenge is the accurate registration of 3D bone models (from MRI or CT scans) with 2D X-ray image pairs. Marker-based or model-based 2D–3D registration can be performed. The former has higher registration accuracy owing to corresponding marker pairs. The latter avoids bead implantation and uses radiograph intensity or features. A rigorous new method based on projection strategy and least-squares estimation that can be used for both methods is proposed and validated by a 3D-printed bone with implanted beads. The results show that it can achieve greater marker-based registration accuracy than the state-of-the-art RSA method. Model-based registration achieved a 3D reconstruction accuracy of 0.79 mm. Systematic offsets between detected edges in the radiographs and their actual position were observed and modeled to improve the reconstruction accuracy to 0.56 mm (tibia) and 0.64 mm (femur). This method is demonstrated on in vivo data, achieving a registration precision of 0.68 mm (tibia) and 0.60 mm (femur). The proposed method allows the determination of accurate 3D kinematic parameters that can be used to calculate joint cartilage contact mechanics. [ABSTRACT FROM AUTHOR]

Kinematic tracking of native anatomy from stereo-radiography provides a quantitative basis for evaluating human movement. Conventional tracking procedures require significant manual effort and call for acquisition and annotation of subject-specific volumetric medical images. The current work introduces a framework for fully automatic tracking of native knee anatomy from dynamic stereo-radiography which forgoes reliance on volumetric scans. The method consists of three computational steps. First, captured radiographs are annotated with segmentation maps and anatomic landmarks using a convolutional neural network. Next, a non-convex polynomial optimization problem formulated from annotated landmarks is solved to acquire preliminary anatomy and pose estimates. Finally, a global optimization routine is performed for concurrent refinement of anatomy and pose. An objective function is maximized which quantifies similarities between masked radiographs and digitally reconstructed radiographs produced from statistical shape and intensity models. The proposed framework was evaluated against manually tracked trials comprising dynamic activities, and additional frames capturing a static knee phantom. Experiments revealed anatomic surface errors routinely below 1.0 mm in both evaluation cohorts. Median absolute errors of individual bone pose estimates were below 1.0 ∘ or mm for 15 out of 18 degrees of freedom in both evaluation cohorts. Results indicate that accurate pose estimation of native anatomy from stereo-radiography may be performed with significantly reduced manual effort, and without reliance on volumetric scans. [ABSTRACT FROM AUTHOR]

2D-3D registration is increasingly being applied in various scientific and engineering scenarios. However, due to appearance differences and cross-modal discrepancies, it is demanding for image and point cloud registration methods to establish correspondences, making 2D-3D registration highly challenging. To handle these problems, we propose a novel and automatic solution for 2D-3D registration in Manhattan world based on line primitives, which we denote as VPPnL. Firstly, we derive the rotation matrix candidates by establishing the vanishing point coordinate system as the link of point cloud principal directions to camera coordinate system. Subsequently, the RANSAC algorithm, which accounts for the clustering of parallel lines, is employed in conjunction with the least-squares method for translation vectors estimation and optimization. Finally, a nonlinear least-squares graph optimization method is carried out to optimize the camera pose and realize the 2D-3D registration and point colorization. Experiments on synthetic data and real-world data illustrate that our proposed algorithm can address the problem of 2D-3D direct registration in the case of Manhattan scenes where images are limited and sparse.

In this study, we proposed a high‐density three‐dimensional (3D) tunnel measurement method, which estimates the pose changes of cameras based on a point set registration algorithm regarding 2D and 3D point clouds. To detect small deformations and defects, high‐density 3D measurements are necessary for tunnel construction sites. The line‐structured light method uses an omnidirectional laser to measure a high‐density cross‐section point cloud from camera images. To estimate the pose changes of cameras in tunnels, which have few textures and distinctive shapes, cooperative robots are useful because they estimate the pose by aggregating relative poses from the other robots. However, previous studies mounted several sensors for both the 3D measurement and pose estimation, increasing the size of the measurement system. Furthermore, the lack of 3D features makes it difficult to match point clouds obtained from different robots. The proposed measurement system consists of a cross‐section measurement unit and a pose estimation unit; one camera was mounted for each unit. To estimate the relative poses of the two cameras, we designed a 2D–3D registration algorithm for the omnidirectional laser light, and implemented hand‐truck and unmanned aerial vehicle systems. In the measurement of a tunnel with a width of 8.8 m and a height of 6.4 m, the error of the point cloud measured by the proposed method was 162.8 and 575.3 mm along 27 m, respectively. In a hallway measurement, the proposed method generated less errors in straight line shapes with few distinctive shapes compared with that of the 3D point set registration algorithm with Light Detection and Ranging. [ABSTRACT FROM AUTHOR]

High-speed biplanar videoradiography can derive the dynamic bony translations and rotations required for joint cartilage contact mechanics to provide insights into the mechanical processes and mechanisms of joint degeneration or pathology. A key challenge is the accurate registration of 3D bone models (from MRI or CT scans) with 2D X-ray image pairs. Marker-based or model-based 2D–3D registration can be performed. The former has higher registration accuracy owing to corresponding marker pairs. The latter avoids bead implantation and uses radiograph intensity or features. A rigorous new method based on projection strategy and least-squares estimation that can be used for both methods is proposed and validated by a 3D-printed bone with implanted beads. The results show that it can achieve greater marker-based registration accuracy than the state-of-the-art RSA method. Model-based registration achieved a 3D reconstruction accuracy of 0.79 mm. Systematic offsets between detected edges in the radiographs and their actual position were observed and modeled to improve the reconstruction accuracy to 0.56 mm (tibia) and 0.64 mm (femur). This method is demonstrated on in vivo data, achieving a registration precision of 0.68 mm (tibia) and 0.60 mm (femur). The proposed method allows the determination of accurate 3D kinematic parameters that can be used to calculate joint cartilage contact mechanics.

Objective: Today, cerebrovascular disease has become an important health hazard. Therefore, it is necessary to perform a more accurate and less time-consuming registration of preoperative three-dimensional (3D) images and intraoperative two-dimensional (2D) projection images which is very important for conducting cerebrovascular disease interventions. The 2D-3D registration method proposed in this study is designed to solve the problems of long registration time and large registration errors in 3D computed tomography angiography (CTA) images and 2D digital subtraction angiography (DSA) images. Methods: To make a more comprehensive and active diagnosis, treatment and surgery plan for patients with cerebrovascular diseases, we propose a weighted similarity measure function, the normalized mutual information-gradient difference (NMG), which can evaluate the 2D-3D registration results. Then, using a multi-resolution fusion optimization strategy, the multi-resolution fused regular step gradient descent optimization (MR-RSGD) method is presented to attain the optimal value of the registration results in the process of the optimization algorithm. Result: In this study, we adopt two datasets of the brain vessels to validate and obtain similarity metric values which are 0.0037 and 0.0003, respectively. Using the registration method proposed in this study, the time taken for the experiment was calculated to be 56.55s and 50.8070s, respectively, for the two sets of data. The results show that the registration methods proposed in this study are both better than the Normalized Mutual (NM) and Normalized Mutual Information (NMI). Conclusion: The experimental results in this study show that in the 2D-3D registration process, to evaluate the registration results more accurately, we can use the similarity metric function containing the image gray information and spatial information. To improve the efficiency of the registration process, we can choose the algorithm with gradient optimization strategy. Our method has great potential to be applied in practical interventional treatment for intuitive 3D navigation. [ABSTRACT FROM AUTHOR]

Background: Clinical outcomes of kinematically aligned total knee arthroplasty (KA-TKA) have been reported as comparable or superior to those of mechanically aligned TKA (MA-TKA). However, cruciate-retaining prostheses have mostly been used for KA-TKA. This study used medial pivot knee prostheses for KA-TKA, and knee kinematics after KA-TKA were assessed and compared with those after MA-TKA. Methods: Thirteen knees in 9 patients undergoing primary TKA (8 KAs, 5 MAs) were subjected to two-dimensional (2D) to three-dimensional (3D) registration analysis at 1 year postoperatively. Each patient performed weight-bearing activities, and movements were recorded as intermittent digital radiographic images. Three-dimensional implant positions during activities were analyzed for anterior-posterior translation in the sagittal plane, condylar liftoff and mediolateral translation in the coronal plane, and femoral rotation in the axial plane. Results: Posterior translation of the lateral femoral condyle from 0° to 100° was larger in KA-TKA than in MA-TKA (P = .006). The degrees of condylar liftoff and mediolateral translation were comparable between TKAs. Total external rotation of the femoral component relative to tibial component was significantly greater for KA-TKA (7.7 ± 5.2°) than for MA-TKA (1.3 ± 3.3°; P = .03). The kinematic path of the femoral component revealed greater medial pivoting motion in KA-TKA than in MA-TKA. Conclusions: KA-TKA using a medial pivot knee prosthesis successfully reproduced the medial pivot pattern and achieved larger femoral external rotation relative to the tibia than MA-TKA. KA-TKA was able to maximize the primary concept of the medial pivot knee prosthesis.

Accurate camera localization in existing LiDAR maps is promising since it potentially allows exploiting strengths of both LiDAR-based and camera-based methods. However, effective methods that robustly address appearance and modality differences for 2D–3D localization are still missing. To overcome these problems, we propose the I2D-Loc, a scene-agnostic and end-to-end trainable neural network that estimates the 6-DoF pose from an RGB image to an existing LiDAR map with local optimization on an initial pose. Specifically, we first project the LiDAR map to the image plane according to a rough initial pose and utilize a depth completion algorithm to generate a dense depth image. We further design a confidence map to weight the features extracted from the dense depth to get a more reliable depth representation. Then, we propose to utilize a neural network to estimate the correspondence flow between depth and RGB images. Finally, we utilize the BPnP algorithm to estimate the 6-DoF pose, calculating the gradients of pose error and optimizing the front-end network parameters. Moreover, by decoupling the intrinsic camera parameters out of the end-to-end training process, I2D-Loc can be generalized to the images with different intrinsic parameters. Experiments on KITTI, Argoverse, and Lyft5 datasets demonstrate that the I2D-Loc can achieve centimeter localization performance. The source code, dataset, trained models, and demo videos are released at https://levenberg.github.io/I2D-Loc/. [ABSTRACT FROM AUTHOR]

Intensity-based 2D-3D registration methods are commonly used in musculoskeletal research and image-guided therapy to align 2D X-ray images with 3D CT scans. However, their success rate (SR) is limited by local optimization methods, which often cause the optimization of the underlying cost function to get stuck at a local minimum, resulting in false alignments. Global optimization methods aim to mitigate this problem, but despite their increasing popularity, the existing literature lacks consensus on which one is the most appropriate. In this work, we compare 11 global and 4 local optimization methods on thousands of typical registration examples of single- and dual-plane fluoroscopy, including three datasets of varying complexity. In addition, we evaluate the differences between global and local methods, determine the best overall method, and validate its suitability for real clinical data. The results demonstrate that global methods that require a large number of function evaluations (NFEV) are generally the most robust. Furthermore, hyperparameter tuning can significantly improve their performance and is generalizable across datasets. Evolutionary strategy (ES) is the best-performing optimization method in our study, achieving a mean SR of ∼95% for all test models, ∼77% for the knee bones, and ∼95-100% for cerebral angiograms when using dual-plane registration setup. Nevertheless, in cases where good initialization is available, local methods are still preferable due to their reduced NFEV. The use of global optimization improves the overall robustness and ease-of-use of 2D-3D registration, potentially accelerating its adaptation in routine medical practice and biomedical research., Competing Interests: Declaration of competing interest The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Marko Leskovar reports financial support was provided by German Research Foundation. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2024 The Authors. Published by Elsevier Ltd.. All rights reserved.)

A digital twin city maps a virtual three-dimensional (3D) city model to the geographic information system, constructs a virtual world, and integrates real sensor data to achieve the purpose of virtual–real fusion. Focusing on the accuracy problem of vision sensor registration in the virtual digital twin city scene, this study proposes a 2D-3D registration method under inverse depth and structural semantic constraints. First, perspective and inverse depth images of the virtual scene were obtained by using perspective view and inverse-depth nascent technology, and then the structural semantic features were extracted by the two-line minimal solution set method. A simultaneous matching and pose estimation method under inverse depth and structural semantic constraints was proposed to achieve the 2D-3D registration of real images and virtual scenes. The experimental results show that the proposed method can effectively optimize the initial vision sensor pose and achieve high-precision registration in the digital twin scene, and the Z-coordinate error is reduced by 45%. An application experiment of monocular image multi-object spatial positioning was designed, which proved the practicability of this method, and the influence of model data error on registration accuracy was analyzed. [ABSTRACT FROM AUTHOR]

Abstract Background This study aimed to assess the error of different registration techniques and imaging modalities for fusion imaging of the aorta in a standardized setting using a anthropomorphic body phantom. Materials and methods A phantom with the 3D printed vasculature of a patient suffering from an infrarenal aortic aneurysm was constructed. Pulsatile flow was generated via an external pump. CTA/MRA of the phantom was performed, and a virtual 3D vascular model was computed. Subsequently, fusion imaging was performed employing 3D-3D and 2D-3D registration techniques. Accuracy of the registration was evaluated from 7 right/left anterior oblique c-arm angulations using the agreement of centerlines and landmarks between the phantom vessels and the virtual 3D virtual vascular model. Differences between imaging modalities were assessed in a head-to-head comparison based on centerline deviation. Statistics included the comparison of means ± standard deviations, student’s t-test, Bland-Altman analysis, and intraclass correlation coefficient for intra- and inter-reader analysis. Results 3D-3D registration was superior to 2D-3D registration, with the highest mean centerline deviation being 1.67 ± 0.24 mm compared to 4.47 ± 0.92 mm. The highest absolute deviation was 3.25 mm for 3D-3D and 6.25 mm for 2D-3D registration. Differences for all angulations between registration techniques reached statistical significance. A decrease in registration accuracy was observed for c-arm angulations beyond 30° right anterior oblique/left anterior oblique. All landmarks (100%) were correctly positioned using 3D-3D registration compared to 81% using 2D-3D registration. Differences in accuracy between CT and MRI were acceptably small. Intra- and inter-reader reliability was excellent. Conclusion In the realm of registration techniques, the 3D-3D method proved more accurate than did the 2D-3D method. Based on our data, the use of 2D-3D registration for interventions with high registration quality requirements (e.g., fenestrated aortic repair procedures) cannot be fully recommended. Regarding imaging modalities, CTA and MRA can be used equivalently.

Deep-learning methods provide a promising approach for measuring in-vivo knee joint motion from fast registration of two-dimensional (2D) to three-dimensional (3D) data with a broad range of capture. However, if there are insufficient data for training, the data-driven approach will fail. We propose a feature-based transfer-learning method to extract features from fluoroscopic images. With three subjects and fewer than 100 pairs of real fluoroscopic images, we achieved a mean registration success rate of up to 40%. The proposed method provides a promising solution, using a learning-based registration method when only a limited number of real fluoroscopic images is available.

Abstract Objectives 3D preoperative planning of lower limb osteotomies has become increasingly important in light of modern surgical technologies. However, 3D models are usually reconstructed from Computed Tomography data acquired in a non-weight-bearing posture and thus neglecting the positional variations introduced by weight-bearing. We developed a registration and planning pipeline that allows for 3D preoperative planning and subsequent 3D assessment of anatomical deformities in weight-bearing conditions. Methods An intensity-based algorithm was used to register CT scans with long-leg standing radiographs and subsequently transform patient-specific 3D models into a weight-bearing state. 3D measurement methods for the mechanical axis as well as the joint line convergence angle were developed. The pipeline was validated using a leg phantom. Furthermore, we evaluated our methods clinically by applying it to the radiological data from 59 patients. Results The registration accuracy was evaluated in 3D and showed a maximum translational and rotational error of 1.1 mm (mediolateral direction) and 1.2° (superior-inferior axis). Clinical evaluation proved feasibility on real patient data and resulted in significant differences for 3D measurements when the effects of weight-bearing were considered. Mean differences were 2.1 ± 1.7° and 2.0 ± 1.6° for the mechanical axis and the joint line convergence angle, respectively. 37.3 and 40.7% of the patients had differences of 2° or more in the mechanical axis or joint line convergence angle between weight-bearing and non-weight-bearing states. Conclusions Our presented approach provides a clinically feasible approach to preoperatively fuse 2D weight-bearing and 3D non-weight-bearing data in order to optimize the surgical correction.

Purpose: Estimation of the accuracy of 2D‐3D registration is paramount for a correct evaluation of its outcome in both research and clinical studies. Publicly available datasets with standardized evaluation methodology are necessary for validation and comparison of 2D‐3D registration techniques. Given the large use of 2D‐3D registration in biomechanics, we introduced the first gold standard validation dataset for computed tomography (CT)‐to‐x‐ray registration of the hip joint, based on fluoroscopic images with large rotation angles. As the ground truth computed with fiducial markers is affected by localization errors in the image datasets, we proposed a new methodology based on uncertainty propagation to estimate the accuracy of a gold standard dataset. Methods: The gold standard dataset included a 3D CT scan of a female hip phantom and 19 2D fluoroscopic images acquired at different views and voltages. The ground truth transformations were estimated based on the corresponding pairs of extracted 2D and 3D fiducial locations. These were assumed to be corrupted by Gaussian noise, without any restrictions of isotropy. We devised the multiple projective points criterion (MPPC) that jointly optimizes the transformations and the noisy 3D fiducial locations for all views. The accuracy of the transformations obtained with the MPPC was assessed in both synthetic and real experiments using different formulations of the target registration error (TRE), including a novel formulation of the TRE (uTRE) derived from the uncertainty analysis of the MPPC. Results: The proposed MPPC method was statistically more accurate compared to the validation methods for 2D‐3D registration that did not optimize the 3D fiducial positions or wrongly assumed the isotropy of the noise. The reported results were comparable to previous published works of gold standard datasets. However, a formulation of the TRE commonly found in these gold standard datasets was found to significantly miscalculate the true TRE computed in synthetic experiments with known ground truths. In contrast, the uncertainty‐based uTRE was statistically closer to the true TRE. Conclusions: We proposed a new gold standard dataset for the validation of CT‐to‐X‐ray registration of the hip joint. The gold standard transformations were derived from a novel method modeling the uncertainty in extracted 2D and 3D fiducials. Results showed that considering possible noise anisotropy and including corrupted 3D fiducials in the optimization resulted in improved accuracy of the gold standard. A new uncertainty‐based formulation of the TRE also appeared as a good alternative to the unknown true TRE that has been replaced in previous works by an alternative TRE not fully reflecting the gold standard accuracy. [ABSTRACT FROM AUTHOR]

A digital twin city maps a virtual three-dimensional (3D) city model to the geographic information system, constructs a virtual world, and integrates real sensor data to achieve the purpose of virtual–real fusion. Focusing on the accuracy problem of vision sensor registration in the virtual digital twin city scene, this study proposes a 2D-3D registration method under inverse depth and structural semantic constraints. First, perspective and inverse depth images of the virtual scene were obtained by using perspective view and inverse-depth nascent technology, and then the structural semantic features were extracted by the two-line minimal solution set method. A simultaneous matching and pose estimation method under inverse depth and structural semantic constraints was proposed to achieve the 2D-3D registration of real images and virtual scenes. The experimental results show that the proposed method can effectively optimize the initial vision sensor pose and achieve high-precision registration in the digital twin scene, and the Z-coordinate error is reduced by 45%. An application experiment of monocular image multi-object spatial positioning was designed, which proved the practicability of this method, and the influence of model data error on registration accuracy was analyzed.

Purpose: Performing a transrectal ultrasound (TRUS) prostate biopsy is at the heart of the current prostate cancer detection procedure. With today's two‐dimensional (2D) live ultrasound (US) imaging equipment, this task remains complex due to the poor visibility of cancerous tissue on TRUS images and the limited anatomical context available in the 2D TRUS plane. This paper presents a rigid 2D/3DUS registration method for navigated prostate biopsy. This allows continuous localization of the biopsy trajectory during the procedure. Methods: We proposed an organ‐based approach to achieve real‐time rigid registration without the need for any probe localization device. The registration method combines image similarity and geometric proximity of detected features. Additions to our previous work include a multi‐level approach and the use of a rejection rate favouring the best matches. Their aim is to increase the accuracy and time performances. These modifications and their in‐depth evaluation on real clinical cases and comparison to this previous work are described. We performed static and dynamic evaluations along biopsy trajectories on a very large amount of data acquired under uncontrolled routine conditions. The computed transforms are compared to a ground truth obtained either from corresponding manually detected fiducials or from an already evaluated registration method. Results: All results show that the current method outperforms its previous version, both in terms of accuracy (the average error reported here is 12 to 17% smaller depending on the experiment) and processing time (from 20 to 60 times faster compared to the previous implementation). The dynamic registration experiment demonstrates that the method can be successfully used for continuous tracking of the biopsy location w.r.t the prostate at a rate that varies between 5 and 15 Hz. Conclusions: This work shows that on the fly 2D/3DUS registration can be performed very efficiently on biopsy trajectories. This allows us to plan further improvements in prostate navigation and a clinical transfer. [ABSTRACT FROM AUTHOR]

Background: This study aimed to assess the error of different registration techniques and imaging modalities for fusion imaging of the aorta in a standardized setting using a anthropomorphic body phantom. Materials and methods: A phantom with the 3D printed vasculature of a patient suffering from an infrarenal aortic aneurysm was constructed. Pulsatile flow was generated via an external pump. CTA/MRA of the phantom was performed, and a virtual 3D vascular model was computed. Subsequently, fusion imaging was performed employing 3D-3D and 2D-3D registration techniques. Accuracy of the registration was evaluated from 7 right/left anterior oblique c-arm angulations using the agreement of centerlines and landmarks between the phantom vessels and the virtual 3D virtual vascular model. Differences between imaging modalities were assessed in a head-to-head comparison based on centerline deviation. Statistics included the comparison of means ± standard deviations, student's t-test, Bland-Altman analysis, and intraclass correlation coefficient for intra- and inter-reader analysis. Results: 3D-3D registration was superior to 2D-3D registration, with the highest mean centerline deviation being 1.67 ± 0.24 mm compared to 4.47 ± 0.92 mm. The highest absolute deviation was 3.25 mm for 3D-3D and 6.25 mm for 2D-3D registration. Differences for all angulations between registration techniques reached statistical significance. A decrease in registration accuracy was observed for c-arm angulations beyond 30° right anterior oblique/left anterior oblique. All landmarks (100%) were correctly positioned using 3D-3D registration compared to 81% using 2D-3D registration. Differences in accuracy between CT and MRI were acceptably small. Intra- and inter-reader reliability was excellent. Conclusion: In the realm of registration techniques, the 3D-3D method proved more accurate than did the 2D-3D method. Based on our data, the use of 2D-3D registration for interventions with high registration quality requirements (e.g., fenestrated aortic repair procedures) cannot be fully recommended. Regarding imaging modalities, CTA and MRA can be used equivalently. [ABSTRACT FROM AUTHOR]

Objectives: 3D preoperative planning of lower limb osteotomies has become increasingly important in light of modern surgical technologies. However, 3D models are usually reconstructed from Computed Tomography data acquired in a non-weight-bearing posture and thus neglecting the positional variations introduced by weight-bearing. We developed a registration and planning pipeline that allows for 3D preoperative planning and subsequent 3D assessment of anatomical deformities in weight-bearing conditions. Methods: An intensity-based algorithm was used to register CT scans with long-leg standing radiographs and subsequently transform patient-specific 3D models into a weight-bearing state. 3D measurement methods for the mechanical axis as well as the joint line convergence angle were developed. The pipeline was validated using a leg phantom. Furthermore, we evaluated our methods clinically by applying it to the radiological data from 59 patients. Results: The registration accuracy was evaluated in 3D and showed a maximum translational and rotational error of 1.1 mm (mediolateral direction) and 1.2° (superior-inferior axis). Clinical evaluation proved feasibility on real patient data and resulted in significant differences for 3D measurements when the effects of weight-bearing were considered. Mean differences were 2.1 ± 1.7° and 2.0 ± 1.6° for the mechanical axis and the joint line convergence angle, respectively. 37.3 and 40.7% of the patients had differences of 2° or more in the mechanical axis or joint line convergence angle between weight-bearing and non-weight-bearing states. Conclusions: Our presented approach provides a clinically feasible approach to preoperatively fuse 2D weight-bearing and 3D non-weight-bearing data in order to optimize the surgical correction. [ABSTRACT FROM AUTHOR]

Purpose: This paper aims to obtain a texture dental model with real images and improve the rendering effect of the dental model. Design/methodology/approach: The paper proposes a semiautomatic method to construct a realistic dental model with real images based on two-dimensional/three-dimensional (2D/3D) registration. First, a 3D digital dental model and three intraoral images are obtained by a 3D scanner and digital single-lens reflex camera. Second, the camera projection poses for every intraoral images are calculated by using the single-objective optimization algorithm. Third, with camera poses, the preliminary projection texture mapping is performed; besides, the seam between two textures is marked. Finally, the marked regions are fused based on the image pyramid to eliminate obvious seams. Findings: The paper provides a method to construct a realistic dental model. The method can map three intraoral images to the dental model. The experimental results show that the textured dental model without obvious distortion, dislocation and seams is constructed with simple interactions. Originality/value: The proposed method can be applied to the digital smile design system to improve the communication efficiency between doctors, patients and technicians. [ABSTRACT FROM AUTHOR]

Registration between the intraoperative 2D digital subtraction angiography (DSA) and the pre-operative 3D computed tomography angiography (CTA) is useful in the guidance of vascular interventional surgery (VIS). Because of the flow and diffusion of contrast in vessels, not all the vessels are visible in the DSA, which leads to difficulty in registration. Although the conventional mutual information (MI) and the normalized MI (NMI) perform well in orthopedic surgery, it cannot perform as well in VIS because of the missing vessel in DSA. In this paper, a novel similarity measure, WLMI (Weighted Local Mutual Information), is proposed to perform 2D-3D registration, which uses the patches selected in DSA to find the best match in the DRRs (digital radiography reconstruction) with weighted MI. At first, we choose several ways to measure the richness of block information to select small and scattered patches in the DSA image. Subsequently, different weights are assigned to these patches according to their information abundance by calculating the local MI. With the proposed WLMI, the 2D-3D registration experiments are conducted with the synthetic and real DSA images to CTA vessel model. In synthetic registration, the optimal way to measure information abundance and the optimal weight parameter value are selected. The registration with intact and partially absent vessels are compared with the conventional methods using MAE, mTREproj, and IoU metrics. In the registration of real DSA to CTA model, only IoU is used to compare the difference since the ground-truth cannot be derived. In synthetic registration with vessel excalation test, the mTREproj is 2.9 mm and the IoU is 96.7%. In a real registration test, the IoU reaches 81.3%. Both the synthetic and real DSA registration results show that the proposed WLMI can cope with the problem that partial vessel is invisible in the DSA, and outperformed the conventional methods such as MI and NMI. Therefore, the WLMI is more suitable to be used in the 2D/3D registration task in VIS.

This paper introduces a method for dynamic 3D registration of female pelvic organs using 2D dynamic magnetic‐resonance images (MRIs). The aim is to provide a better knowledge and understanding of pathologies such as prolapsus or abnormal mobility of tissues. 2D dynamic MRI sequences are commonly used in nowadays clinical routines in order to evaluate the dynamic of organs, but due to the limited view, subjectivity related to human perception cannot be avoided in the diagnoses. A novel method for 2D/3D registration is proposed combining 3D finite element models with a priori knowledge of boundary conditions, in order to provide a 3D extrapolation of the dynamic of the organs observed in a single 2D MRI slice. The method is applied to the four main structures of the female pelvic floor (bladder, vagina, uterus and rectum), providing a full 3D visualisation of the organs' displacements. The methodology is evaluated with two patient‐specific data sets of volunteers presenting no pelvic pathology, and a sensitivity study is performed using synthetic data. The resulting simulations provide an estimation of the dynamic 3D shape of the organs facilitating diagnosis compared with 2D sequences. Moreover, the method follows a protocol compatible with current clinical constraints presenting this way potential short term medical applications. [ABSTRACT FROM AUTHOR]

Pelvic sagittal inclination (PSI) significantly affects the femoral head coverage by the acetabulum in patients with developmental dysplasia of the hip (DDH), while no reports have quantified PSI in DDH patients in the supine and standing positions. Furthermore, little is known about how PSI changes after periacetabular osteotomies. Herein, PSI in the supine and standing positions was quantified in DDH patients preoperatively and postoperatively. Twenty‐five patients with DDH who had undergone periacetabular osteotomies were analyzed. The preoperative PSI and the PSI 2 years after surgery were measured in the supine and standing positions using the image registration technique between radiographs and computed tomographic images. The percentage of patients who showed PSI changes of more than 10° from the supine to the standing position was quantified. PSI changed 8.2 ± 5.0° posteriorly from the supine to the standing position during the preoperative period. Posterior pelvic tilt of more than 10° was found in nine cases (36%). Two years after periacetabular osteotomies, the postural PSI change was 7.1 ± 3.9° posteriorly. When the preoperative and postoperative PSI values were compared, PSI in the standing position did not differ (p = 0.20). Similarly, the amount of PSI change from the supine to standing position was not significantly different (p = 0.26). In conclusion, posterior pelvic tilt in the standing position was found preoperatively in symptomatic DDH patients, and it remained for 2 years after periacetabular osteotomies. This postural change in PSI does not seem to influence the outcome of periacetabular osteotomy. However, during preoperative planning, surgeons should recognize that acetabular anteversion or anterior acetabular coverage differs between the supine and standing positions in some patients with DDH. © 2019 Orthopaedic Research Society. Published by Wiley Periodicals, Inc. J Orthop Res 38:578–587, 2020 [ABSTRACT FROM AUTHOR]

Precise measurement of joint-level motion from stereo-radiography facilitates understanding of human movement. Conventional procedures for kinematic tracking require significant manual effort and are time intensive. The current work introduces a method for fully automatic tracking of native knee kinematics from stereo-radiography sequences. The framework consists of three computational steps. First, biplanar radiograph frames are annotated with segmentation maps and key points using a convolutional neural network. Next, initial bone pose estimates are acquired by solving a polynomial optimization problem constructed from annotated key points and anatomic landmarks from digitized models. A semidefinite relaxation is formulated to realize the global minimum of the non-convex problem. Pose estimates are then refined by registering computed tomography-based digitally reconstructed radiographs to masked radiographs. A novel rendering method is also introduced which enables generating digitally reconstructed radiographs from computed tomography scans with inconsistent slice widths. The automatic tracking framework was evaluated with stereo-radiography trials manually tracked with model-image registration, and with frames which capture a synthetic leg phantom. The tracking method produced pose estimates which were consistently similar to manually tracked values; and demonstrated pose errors below 1.0 degree or millimeter for all femur and tibia degrees of freedom in phantom trials. Results indicate the described framework may benefit orthopaedics and biomechanics applications through acceleration of kinematic tracking. [ABSTRACT FROM AUTHOR]

Purpose . To evaluate the performance of an automated 2D-3D bone registration algorithm incorporating a grayscale compression method for quantifying patient position errors in non-coplanar radiotherapy. Methods . An automated 2D-3D registration incorporating a grayscale compression method to segment bone structures was proposed. Portal images containing only bone structures (Portalbone) and digitally reconstructed radiographs containing only bone structures (DRRbone) were used for registration. First, the portal image was filtered by a high-pass finite impulse response (FIR) filter. Then the grayscale range of the filtered portal image was compressed. Thresholds were determined based on the difference in gray values of bone structures in the filtered and compressed portal image to obtainPortalbone.Another threshold was applied to generateDRRbonewhen the CT image uses the ray-casting algorithm to generate DRR images. The compression performance was assessed by registering theDRRbonewith thePortalboneobtained by compressing the portal image into various grayscale ranges. The proposed registration method was quantitatively and visually validated using (1) a CT image of an anthropomorphic head phantom and its portal images obtained in different poses and (2) CT images and pre-treatment portal images of 20 patients treated with non-coplanar radiotherapy. Results . Mean absolute registration errors for the best compression grayscale range test were 0.642 mm, 0.574 mm, and 0.643 mm, with calculation times of 50.6 min, 42.2 min, and 49.6 min for grayscale ranges of 0-127, 0-63 and 0-31, respectively. For the accuracy validation (1), the mean absolute registration errors for couch angles 0°, 45°, 90°, 270°, and 315° were 0.694 mm, 0.839 mm, 0.726 mm, 0.833 mm, and 0.873 mm, respectively. Among the six transformation parameters, the translation error in the vertical direction contributed the most to the registration errors. Visual inspection of the patient registration results revealed success in every instance. Conclusions . The implemented grayscale compression method successfully enhances and segments bone structures in portal images, allowing for accurate determination of patient setup errors in non-coplanar radiotherapy., (© 2024 IOP Publishing Ltd.)

In the context of 2D/3D registration, this paper introduces an approach that allows for matching features detected in two different modalities, photographs, and 3D models, by using a common 2D representation. More precisely, 2D images are matched with a set of depth images representing the 3D model. After introducing the concept of Curvilinear Saliency, which is related to curvature estimation, we propose a new ridge and valley detector for depth images rendered from 3D models. A variant of this detector is adapted to photographs, first by considering multi-scale features and second by integrating the focus curve principle. Finally, a registration algorithm determines the correct view of the 3D model and, thus, the pose of the photograph. This approach relies on the Histogram of Curvilinear Saliency (HCS), an adaptation of the Histogram of Oriented Gradients (HOG) to the proposed features in 2D and 3D. The presented results highlight both the quality of the features detected in terms of repeatability and the interest of the approach for registration and pose estimation. [ABSTRACT FROM AUTHOR]

This paper addresses the problem of registering a known structured 3D scene, typically a 3D scan, and its metric Structure-from-Motion (SfM) counterpart. The proposed registration method relies on a prior plane segmentation of the 3D scan. Alignment is carried out by solving either the point-to-plane assignment problem, should the SfM reconstruction be sparse, or the plane-to-plane one in case of dense SfM. A Polynomial Sum-of-Squares optimization theory framework is employed for identifying point-to-plane and plane-to-plane mismatches, i.e. outliers, with certainty. An inlier set maximization approach within a Branch-and-Bound search scheme is adopted to iteratively build potential inlier sets and converge to the solution satisfied by the largest number of assignments. Plane visibility conditions and vague camera locations may be incorporated for better efficiency without sacrificing optimality. The registration problem is solved in two cases: (i) putative correspondences (with possibly overwhelmingly many outliers) are provided as input and (ii) no initial correspondences are available. Our approach yields outstanding results in terms of robustness and optimality. [ABSTRACT FROM AUTHOR]

The in vivo kinematic characteristics of the tarsal joints during gait stance phase were still unclear in adult‐acquired foot deformity (AAFD). This study included seven healthy subjects (14 feet) and 12 stage II AAFD patients (14 feet). The 3D models of tarsal bones were reconstructed based on CT scan. Each subject took standard gait on the single fluoroscopy system. Continuous lateral fluoroscopic images were collected. The key postures during the stance phase were selected. The 2D‐3D registration technique was applied to explore the spatial motions of the tarsal joints in 6 degrees of freedom (DOF). During the whole stance phase, the AAFD talo‐navicular joint (TNJ) exhibited ROM of 13 ± 6° in the sagittal plane while the normal subjects showed ROM of 7 ± 3° (p = 0.004). In AAFD, the subtalar joint (STJ) demonstrated 19 ± 8° and 7 ± 3° of motion in coronal and horizontal plane, respectively, while the normal subjects showed 14 ± 4°(p = 0.031) and 11 ± 3° (p = 0.014) of motion, respectively. Additionally STJ of AAFD patients showed significantly less dorsiflexion during the weight acceptance and showed significantly less external rotation both during the weight acceptance and single limb support of stance phase. In conclusion, for stage II AAFD patients, the talonavicular joint and the subtalar joint showed hypermobility in dorsi/planterflexion and inversion/eversion, respectively, during the gait stance phase while the internal/external rotation of the subtalar joint was reduced. The current study improves our understanding of the pathological kinematics of the tarsal complex in AAFD patients. Notice should be taken about these tarsal joints mobility in AAFD during clinical practice. © 2017 Orthopaedic Research Society. Published by Wiley Periodicals, Inc. J Orthop Res 37:477–482, 2019. [ABSTRACT FROM AUTHOR]

Screening colonoscopy is an important clinical application for several 3D computer vision techniques, including depth estimation, surface reconstruction, and missing region detection. However, the development, evaluation, and comparison of these techniques in real colonoscopy videos remain largely qualitative due to the difficulty of acquiring ground truth data. In this work, we present a C olonoscopy 3 D V ideo D ataset (C3VD) acquired with a high definition clinical colonoscope and high-fidelity colon models for benchmarking computer vision methods in colonoscopy. We introduce a novel multimodal 2D-3D registration technique to register optical video sequences with ground truth rendered views of a known 3D model. The different modalities are registered by transforming optical images to depth maps with a Generative Adversarial Network and aligning edge features with an evolutionary optimizer. This registration method achieves an average translation error of 0.321 millimeters and an average rotation error of 0.159 degrees in simulation experiments where error-free ground truth is available. The method also leverages video information, improving registration accuracy by 55.6% for translation and 60.4% for rotation compared to single frame registration. 22 short video sequences were registered to generate 10,015 total frames with paired ground truth depth, surface normals, optical flow, occlusion, six degree-of-freedom pose, coverage maps, and 3D models. The dataset also includes screening videos acquired by a gastroenterologist with paired ground truth pose and 3D surface models. The dataset and registration source code are available at https://durr.jhu.edu/C3VD. • 2D-3D registration that leverages temporal information in optical colonoscopy videos. • Conditional Generative Adversarial Network used to transform frames to depth domain. • High-fidelity silicone phantom models with 3D ground truth. • Public dataset for quantitative benchmarking of computer vision tasks in colonoscopy. [Display omitted] [ABSTRACT FROM AUTHOR]

Highlights • 3D rib motion was automatically recovered from x-ray video and one time-phase CT. • Local contrast normalization improved the cost function space facilitating robustness in the optimization. • An articular motion model of the rib constrained the optimization to obtain physically plausible rib motion. • The improved accuracy was demonstrated in both simulation and real-image experiments. • Correlation between improvement of the cost function space and robustness was analyzed by real-image experiments. Graphical abstract Abstract Dynamic chest radiography (2D x-ray video) is a low-dose and cost-effective functional imaging method with high temporal resolution. While the analysis of rib-cage motion has been shown to be effective for evaluating respiratory function, it has been limited to 2D. We aim at 3D rib-motion analysis for high temporal resolution while keeping the radiation dose at a level comparable to conventional examination. To achieve this, we developed a method for automatically recovering 3D rib motion based on 2D-3D registration of x-ray video and single-time-phase computed tomography. We introduce the following two novel components into the conventional intensity-based 2D–3D registration pipeline: (1) a rib-motion model based on a uniaxial joint to constrain the search space and (2) local contrast normalization (LCN) as a pre-process of x-ray video to improve the cost function of the optimization parameters, which is often called the landscape. The effects of each component on the registration results were quantitatively evaluated through experiments using simulated images and real patients' x-ray videos obtained in a clinical setting. The rotation-angle error of the rib and the mean projection contour distance (mPCD) were used as the error metrics. The simulation experiments indicate that the proposed uniaxial joint model improved registration accuracy. By searching the rotation axis along with the rotation angle of the ribs, the rotation-angle error and mPCD significantly decreased from 2.246 ± 1.839° and 1.148 ± 0.743 mm to 1.495 ± 0.993° and 0.742 ± 0.281 mm, compared to simply applying De Troyer's model. The real-image experiments with eight patients demonstrated that LCN improved the cost function space; thus, robustness in optimization resulting in an average mPCD of 1.255 ± 0.615 mm. We demonstrated that an anatomical-knowledge based constraint and an intensity normalization, LCN, significantly improved robustness and accuracy in rib-motion reconstruction using chest x-ray video. [ABSTRACT FROM AUTHOR]

We propose a robust and direct 2D-3D registration method for camera synchronization. Once the cameras are synchronized—or for synchronous setups—we also propose a visual odometry framework that benefits from both 2D and 3D acquisitions. Our method does not require a precise set of 2D-to-3D correspondences, handles occlusions and works when the scene is only partially known. It is carried out through a 2D-3D based initial motion estimation followed by a constrained nonlinear optimization for motion refinement. The problems of occlusion and that of missing scene parts are handled by comparing the image-based reconstruction and 3D sensor measurements. The results of our experiments demonstrate that the proposed framework allows to obtain a good initial motion estimate and a significant improvement through refinement. [ABSTRACT FROM AUTHOR]

Summary: Localizing a histological section in the three‐dimensional dataset of a different imaging modality is a challenging 2D‐3D registration problem. In the literature, several approaches have been proposed to solve this problem; however, they cannot be considered as fully automatic. Recently, we developed an automatic algorithm that could successfully find the position of a histological section in a micro computed tomography (μCT) volume. For the majority of the datasets, the result of localization corresponded to the manual results. However, for some datasets, the matching μCT slice was off the ground‐truth position. Furthermore, elastic distortions, due to histological preparation, could not be accounted for in this framework. In the current study, we introduce two optimization frameworks based on normalized mutual information, which enabled us to accurately register histology slides to volume data. The rigid approach allocated 81 % of histological sections with a median position error of 8.4 μm in jaw bone datasets, and the deformable approach improved registration by 33 μm with respect to the median distance error for four histological slides in the cerebellum dataset. [ABSTRACT FROM AUTHOR]

Purpose: We present a hybrid 2D-3D ultrasound (US) rigid registration method for navigated prostate biopsy that enables continuous localization of the biopsy trajectory during the exam.Methods: Current clinical computer-assisted biopsy systems use either sensor-based or image-based approaches. We combine the advantages of both in order to obtain an accurate and real-time navigation based only on an approximate localization of the US probe. Starting with features extracted in both 2D and 3D images, our method introduces a variant of the iterative closest point (ICP) algorithm. Among other differences to ICP, a combination of both the euclidean distance of feature positions and the similarity distance of feature descriptors is used to find matches between 2D and 3D features. The evaluation of the method is twofold. First, an analysis of variance on input parameters is conducted to estimate the sensitivity of our method to their initialization. Second, for a selected set of their values, the target registration error (TRE) was calculated on 29,760 (resp. 4000) registrations in two different experiments. It was obtained using manually identified anatomical fiducials.Results: For 160 US volumes, from 20 patients, recorded during routine biopsy procedures performed in two hospitals by six operators, the mean TRE was 3.91±3.22 mm (resp. 4.37±2.62 mm).Conclusion: This work allows envisioning further developments for prostate navigation and their clinical transfer. [ABSTRACT FROM AUTHOR]

Purpose: We present and evaluate a fully automated 2D–3D intensity‐based registration framework using a single limited field‐of‐view (FOV) 2D kV radiograph and a 3D kV CBCT for 3D estimation of patient setup errors during brain radiotherapy. Methods: We evaluated two similarity measures, the Pearson correlation coefficient on image intensity values (ICC) and maximum likelihood measure with Gaussian noise (MLG), derived from the statistics of transmission images. Pose determination experiments were conducted on 2D kV radiographs in the anterior–posterior (AP) and left lateral (LL) views and 3D kV CBCTs of an anthropomorphic head phantom. In order to minimize radiation exposure and exclude nonrigid structures from the registration, limited FOV 2D kV radiographs were employed. A spatial frequency band useful for the 2D–3D registration was identified from the bone‐to‐no‐bone spectral ratio (BNBSR) of digitally reconstructed radiographs (DRRs) computed from the 3D kV planning CT of the phantom. The images being registered were filtered accordingly prior to computation of the similarity measures. We evaluated the registration accuracy achievable with a single 2D kV radiograph and with the registration results from the AP and LL views combined. We also compared the performance of the 2D–3D registration solutions proposed to that of a commercial 3D–3D registration algorithm, which used the entire skull for the registration. The ground truth was determined from markers affixed to the phantom and visible in the CBCT images. Results: The accuracy of the 2D–3D registration solutions, as quantified by the root mean squared value of the target registration error (TRE) calculated over a radius of 3 cm for all poses tested, was ICCAP: 0.56 mm, MLGAP: 0.74 mm, ICCLL: 0.57 mm, MLGLL: 0.54 mm, ICC (AP and LL combined): 0.19 mm, and MLG (AP and LL combined): 0.21 mm. The accuracy of the 3D–3D registration algorithm was 0.27 mm. There was no significant difference in mean TRE for the 2D–3D registration algorithms using a single 2D kV radiograph with similarity measure and image view point. There was no significant difference in mean TRE between ICCLL, MLGLL, ICC (AP and LL combined), MLG (AP and LL combined), and the 3D–3D registration algorithm despite the smaller FOV used for the 2D–3D registration. While submillimeter registration accuracy was obtained with both ICC and MLG using a single 2D kV radiograph, combining the results from the two projection views resulted in a significantly smaller (P≤0.05) mean TRE. Conclusions: Our results indicate that it is possible to achieve submillimeter registration accuracy with both ICC and MLG using either single or dual limited FOV 2D kV radiographs of the head in the AP and LL views. The registration accuracy suggests that the 2D–3D registration solutions presented are suitable for the estimation of patient setup errors not only during conventional brain radiation therapy, but also during stereotactic procedures and proton radiation therapy where tighter setup margins are required. [ABSTRACT FROM AUTHOR]

X-ray imaging is the conventional method for diagnosing the orthopedic condition of a patient. Computerized Tomography(CT) scanning is another diagnostic method that provides patient’s 3D anatomical information. However, both methods have limitations when diagnosing the whole leg; X-ray imaging does not provide 3D information, and normal CT scanning cannot be performed with a standing posture. Obtaining 3D data regarding the whole leg in a standing posture is clinically important because it enables 3D analysis in the weight bearing condition. Based on these clinical needs, a hardware-based bi-plane X-ray imaging system has been developed; it uses two orthogonal X-ray images. However, such methods have not been made available in general clinics because of the hight cost. Therefore, we proposed a widely adaptive method for 2D X-ray image and 3D CT scan data. By this method, it is possible to threedimensionally analyze the whole leg in standing posture. The optimal position that generates the most similar image is the captured X-ray image. The algorithm verifies the similarity using the performance of the proposed method by simulation-based experiments. Then, we analyzed the internal-external rotation angle of the femur using real patient data. Approximately 10.55 degrees of internal rotations were found relative to the defined anterior-posterior direction. In this paper, we present a useful registration method using the conventional X-ray image and 3D CT scan data to analyze the whole leg in the weight-bearing condition. [ABSTRACT FROM AUTHOR]

Background: Stem anteversion angle is important in the combined anteversion theory to avoid implant impingement after total hip arthroplasty (THA). However, anatomic measurements of stem anteversion angle may not represent functional anteversion of the femur if the femur undergoes axial rotation. Herein, the femoral rotational angle (FRA) was measured in supine and standing positions before and after THA to evaluate the difference between anatomic and functional measurements.Methods: A total of 191 hips (174 patients) treated with THA for osteoarthritis were analyzed in this retrospective, case-controlled study. The FRA was measured as the angle between the posterior condylar line and the line through the bilateral anterior superior iliac spines (positive for external rotation) and was measured preoperatively and postoperatively in supine and standing positions with computed tomography segmentation and landmark localization of the pelvis and the femur followed by intensity-based 2D-3D registration. The number of cases in which the absolute FRA remained <15° in both positions was also calculated.Results: The average ± standard deviation preoperative FRA was 0.3° ± 8.3° in the supine position and -4.5° ± 8.8° during standing; the postoperative FRA was -3.8° ± 9.0° in supine and -14.3° ± 8.3° during standing. There were 134 cases (70%) in which the preoperative absolute FRA remained <15° in both positions while only 85 hips (45%) remained <15°, postoperatively.Conclusion: Substantial variability was seen in the FRA, especially during the postoperative period. These results suggest that the anatomic stem anteversion angle may not represent the functional anteversion of the femur. [ABSTRACT FROM AUTHOR]