Search

Background: Over the last two decades, augmented reality (AR) has been used as a visualization tool in many medical fields in order to increase precision, limit the radiation dose, and decrease the variability among operators. Here, we report the first in vivo study of a novel AR system for the guidance of percutaneous interventional oncology procedures. Methods: Eight patients with 15 liver tumors (0.7–3.0 cm, mean 1.56 + 0.55) underwent percutaneous thermal ablations using AR guidance (i.e., the Endosight system). Prior to the intervention, the patients were evaluated with US and CT. The targeted nodules were segmented and three-dimensionally (3D) reconstructed from CT images, and the probe trajectory to the target was defined. The procedures were guided solely by AR, with the position of the probe tip was subsequently confirmed by conventional imaging. The primary endpoints were the targeting accuracy, the system setup time, and targeting time (i.e., from the target visualization to the correct needle insertion). The technical success was also evaluated and validated by co-registration software. Upon completion, the operators were assessed for cybersickness or other symptoms related to the use of AR. Results: Rapid system setup and procedural targeting times were noted (mean 14.3 min; 12.0–17.2 min; 4.3 min, 3.2–5.7 min, mean, respectively). The high targeting accuracy (3.4 mm; 2.6–4.2 mm, mean) was accompanied by technical success in all 15 lesions (i.e., the complete ablation of the tumor and 13/15 lesions with a >90% 5-mm periablational margin). No intra/periprocedural complications or operator cybersickness were observed. Conclusions: AR guidance is highly accurate, and allows for the confident performance of percutaneous thermal ablations.

An efficient and accurate forest sample plot survey is of great significance to understand the current status of forest resources at the stand or regional scale and the basis of scientific forest management. Close-range photogrammetry (CRP) technology can easily and quickly collect sequence images with high overlapping to reconstruct the 3D model of forest scenes and extract the individual tree parameters automatically and, therefore, can greatly improve the efficiency of forest investigation and has great application potential in forestry visualization management. However, it has some issues in practical forestry applications. First, the imaging quality is affected by the illumination in the forest, resulting in difficulty in feature matching and low accuracy of parameter extraction. Second, the efficiency of 3D forest model reconstruction is limited under complex understory vegetation or the topographic situation in the forest. In addition, the density of point clouds by dense matching directly affects the accuracy of individual tree parameter extraction. This research collected the sequence images of sample plots of four tree species by smartphones in Gaofeng Forest Farm in Guangxi and Wangyedian Forest Farm in Mongolia to analyze the effects of image enhancement, feature detection and dense point cloud algorithms on the efficiency of 3D forest reconstruction and accuracy of individual tree parameter extraction, then proposed a strategy of 3D reconstruction and parameter extraction suitable for different forest scenes. First, we compared the image enhancement effects of median–Gaussian (MG) filtering, single-scale retinex (SSR) and multi-scale retinex (MSR) filtering algorithms. Then, an improved algorithm combining Harris corner detection with speeded-up robust features (SURF) feature detection (Harris+SURF) is proposed, and the feature matching effect is compared with that of a scale invariant feature transform (SIFT) operator. Third, according to the morphological characteristics of the trees in the sequence images, we used the iterative interpolation algorithm of a planar triangulation network based on geometric constraints (GC-based IIPTN) to increase the density of point clouds and reconstruct the 3D forest model, and then extract the position and DBH of the individual trees. The results show that MSR image enhancement can significantly increase the number of matched point pairs. The improved Harris+SURF method can reduce the reconstruction time of the 3D forest model, and the GC-based IIPTN algorithm can improve the accuracy of individual tree parameter extraction. The extracted position of the individual tree is the same as the measured position with the bias within 0.2 m. The accuracy of extracted DBH of Eucalyptus grandis, Taxus chinensis, Larix gmelinii and Pinus tabuliformis is 94%, 95%, 96% and 90%, respectively, which proves that the proposed 3D model reconstruction method based on image enhancement has great potential for tree position and DBH extraction, and also provides effective support for forest resource investigation and visualization management in the future.

High-speed railway in China has undergone rapid development in recent years. Technology for structure measurement is recognized as an important aspect of steel rail quality inspection. The shape of the rail welding base is mainly gauged using mechanical contact measurement technology. Matching this technology with the escalating demands for quality inspection of steel rails is a challenging task. In this paper, a structured light measurement approach is proposed and employed which intersects the rail through the structural light plane projected by inner and outer laser sensors. These sensors form a laser light stripe curve containing the profile information of the rail surface. By processing laser light bars using image analysis and contour reduction techniques, 3-D profiles of the rails are generated. The laser plane is fitted using the coordinate points of the laser light bars converted between the image and world coordinate systems. A new calibration method based on the parallel moving target for line-structured light is proposed to improve the calibration and address the issue caused by the limited number of the extracted calibration points during free target calibration. The rail profile, including the geometric dimension, straightness, and twist of the rail, are then measured. By comparing the measured profile with the standard contour of the rails, the rail wear can then be quantitatively assessed. The experimental results show that the proposed measurement system outperforms the traditional single index detection.

Precision and planning are key to reconstructive surgery. Augmented reality (AR) can bring the information within preoperative computed tomography angiography (CTA) imaging to life, allowing the surgeon to ‘see through’ the patient’s skin and appreciate the underlying anatomy without making a single incision. This work has demonstrated that AR can assist the accurate identification, dissection and execution of vascular pedunculated flaps during reconstructive surgery. Separate volumes of osseous, vascular, skin, soft tissue structures and relevant vascular perforators were delineated from preoperative CTA scans to generate three-dimensional images using two complementary segmentation software packages. These were converted to polygonal models and rendered by means of a custom application within the HoloLens™ stereo head-mounted display. Intraoperatively, the models were registered manually to their respective subjects by the operating surgeon using a combination of tracked hand gestures and voice commands; AR was used to aid navigation and accurate dissection. Identification of the subsurface location of vascular perforators through AR overlay was compared to the positions obtained by audible Doppler ultrasound. Through a preliminary HoloLens-assisted case series, the operating surgeon was able to demonstrate precise and efficient localisation of perforating vessels.

The global scale of Point Cloud Data (PCD) generated through monocular photo/videogrammetry is unknown, and can be calculated using at least one known dimension of the scene. Measuring one or more dimensions for this purpose induces a manual step in the 3D reconstruction process; this increases the effort and reduces the speed of reconstructing scenes, and induces substantial human error in the process due to the high level of measurement accuracy needed. Other ways of measuring such dimensions are based on acquiring additional information by either using extra sensors or specific classes of objects existing in the scene; we found that these solutions are not simple, cost effective or general enough to be considered practical for reconstructing both indoor and outdoor built infrastructure scenes. To address the issue, in this paper, we propose a novel method for automatically calculating the absolute scale of built infrastructure PCD. We use a pre-measured cube for outdoor scenes and a sheet of paper for indoor environments as the calibration patterns. Assuming that the dimensions of these objects are known, the proposed method extracts the objects’ corner points in 2D video frames using a novel algorithm. The extracted corner points are then matched between the consecutive frames. Finally, the corresponding corner points are reconstructed along with other features of the scenes to determine the real world scale. To evaluate the performance of the method, ten indoor and ten outdoor cases were selected and the absolute-scale PCD for each case was computed. Results illustrated the proposed algorithm is able to reconstruct the predefined objects with a high success rate while the generated absolute scale PCD is sufficiently accurate.

International audience; Three-dimensional (3D) structure of a wide range of biological macromolecular assemblies can be computed from two-dimensional images collected by transmission electron microscopy. This information integrated with other structural data (e.g., from X-ray crystallography or nuclear magnetic resonance) helps structural biologists understand the function of macromolecular complexes. Single-particle analysis (SPA) is a method used for studies of complexes whose structure and dynamics can be analyzed in isolation. To reconstruct the 3D structure, SPA methods use a large number of images of randomly oriented individual complexes. When the angular distribution of single-particle orientations samples Fourier space completely and the population is structurally homogeneous, a resolution of the reconstruction of 0.4-1 nm can be achieved. Such high resolutions are possible thanks to the high number of images and the correction of the Contrast Transfer Function (CTF) of the microscope. One of the standard SPA approaches is the refinement of a preliminary 3D model using iterative projection matching combined with CTF correction. We describe a protocol for the refinement of a preliminary model using CTF correction by Wiener filtering of volumes from focal series of experimental images. This protocol combines potentially best features of two other protocols proposed in the field.

Electron tomography is a method for obtaining three-dimensional structural information from electron micrographs. It can be applied to a wide range of samples that can be prepared for transmission electron microscopy (TEM)—may they be of biological origin like e.g. cryo or thin plastic sections of cells and tissue or of material science origin like solid catalysts and electronic devices. Electron tomography data acquisition can be regarded as a kind of super-stereology. Instead of images taken from only two different vantage points, a tomographic data set might consist of more than 151 images taken over an angular range of 150?. A strong advantage of electron tomography is that it does not depend on averaging over unit cells or particles or on the assumption and exploitation of symmetry in samples, as is the case for methods like angular reconstitution, nuclear magnetic resonance spectroscopy, electron and X-ray crystallography. Electron tomography can thus be applied to truly unique structures. In this publication we discuss three new methods, that make electron tomography data acquisition faster, more accurate and opens it for novel acquisition modes. In chapter 2 we describe a method for automated electron tomography data collection that reduces acquisition time by a factor of five, enabling data collection in less than an hour. The method includes a pre-calibration step—measurement and correction of image shift and defocus change at low magnification to detect displacements that might be as large as several µm—before image acquisition. This step avoids switching back and forth between high and low magnification during image acquisition, which is a very time consuming step regarding the electron magnetic lens stabilizations needed. The method is based on the fact that these dislocations occur due to a displacement of the feature of interest from the eucentric height, a displacement of the optical axis from the tilt axis and some movements intrinsic to the sample holder/stage combination and are thus almost predictable. In chapter 3 we discuss a method for the correction of autofocusing errors due to specimen tilt, as this would disable accurate defocus prediction in e.g. pre-calibration tomography. Defocus determination using the beam-tilt method can be hampered for low magnifications and high-tilt angles due to a defocus ramp in the images. The method for the correct cross-correlation of two images of a tilted sample acquired under tilted-beam conditions is a modification of the cosine stretch used in the alignment of images acquired under different tilt angles. Electron tomography is not restricted to the use of transmission electron microscopy. Image acquisition in high angular annular dark-field scanning transmission electron microscopy (HAADF-STEM) mode makes the method readily available for e.g. crystalline samples, which cannot be acquired in TEM mode as they show angle dependent diffraction contrast. As it was shown that nm-sized inclusions in crystalline material can be detected by this method we have applied the approach to the detection of ultrasmall immuno-gold labels absorbed to heavy-metal stained plastic sections of biological material. Our initial experiments discussed in chapter 4 provide good evidence that HAADF-STEM electron tomography could be a useful tool for the accurate three-dimensional immuno-localization of proteins.

Electron tomography is a method for obtaining three-dimensional structural information from electron micrographs. It can be applied to a wide range of samples that can be prepared for transmission electron microscopy (TEM)—may they be of biological origin like e.g. cryo or thin plastic sections of cells and tissue or of material science origin like solid catalysts and electronic devices. Electron tomography data acquisition can be regarded as a kind of super-stereology. Instead of images taken from only two different vantage points, a tomographic data set might consist of more than 151 images taken over an angular range of 150?. A strong advantage of electron tomography is that it does not depend on averaging over unit cells or particles or on the assumption and exploitation of symmetry in samples, as is the case for methods like angular reconstitution, nuclear magnetic resonance spectroscopy, electron and X-ray crystallography. Electron tomography can thus be applied to truly unique structures. In this publication we discuss three new methods, that make electron tomography data acquisition faster, more accurate and opens it for novel acquisition modes. In chapter 2 we describe a method for automated electron tomography data collection that reduces acquisition time by a factor of five, enabling data collection in less than an hour. The method includes a pre-calibration step—measurement and correction of image shift and defocus change at low magnification to detect displacements that might be as large as several µm—before image acquisition. This step avoids switching back and forth between high and low magnification during image acquisition, which is a very time consuming step regarding the electron magnetic lens stabilizations needed. The method is based on the fact that these dislocations occur due to a displacement of the feature of interest from the eucentric height, a displacement of the optical axis from the tilt axis and some movements intrinsic to the sample holder/stage combination and are thus almost predictable. In chapter 3 we discuss a method for the correction of autofocusing errors due to specimen tilt, as this would disable accurate defocus prediction in e.g. pre-calibration tomography. Defocus determination using the beam-tilt method can be hampered for low magnifications and high-tilt angles due to a defocus ramp in the images. The method for the correct cross-correlation of two images of a tilted sample acquired under tilted-beam conditions is a modification of the cosine stretch used in the alignment of images acquired under different tilt angles. Electron tomography is not restricted to the use of transmission electron microscopy. Image acquisition in high angular annular dark-field scanning transmission electron microscopy (HAADF-STEM) mode makes the method readily available for e.g. crystalline samples, which cannot be acquired in TEM mode as they show angle dependent diffraction contrast. As it was shown that nm-sized inclusions in crystalline material can be detected by this method we have applied the approach to the detection of ultrasmall immuno-gold labels absorbed to heavy-metal stained plastic sections of biological material. Our initial experiments discussed in chapter 4 provide good evidence that HAADF-STEM electron tomography could be a useful tool for the accurate three-dimensional immuno-localization of proteins.