Search

For Intelligence, Surveillance, Reconnaissance (ISR) missions of manned and unmanned air systems typical electrooptical payloads provide high-definition video data which has to be exploited with respect to relevant ground targets in real-time by automatic/assisted target recognition software. Airbus Defence and Space is developing required technologies for real-time sensor exploitation since years and has combined the latest advances of Deep Convolutional Neural Networks (CNN) with a proprietary high-speed Support Vector Machine (SVM) learning method into a powerful object recognition system with impressive results on relevant high-definition video scenes compared to conventional target recognition approaches. This paper describes the principal requirements for real-time target recognition in high-definition video for ISR missions and the Airbus approach of combining an invariant feature extraction using pre-trained CNNs and the high-speed training and classification ability of a novel frequency-domain SVM training method. The frequency-domain approach allows for a highly optimized implementation for General Purpose Computation on a Graphics Processing Unit (GPGPU) and also an efficient training of large training samples. The selected CNN which is pre-trained only once on domain-extrinsic data reveals a highly invariant feature extraction. This allows for a significantly reduced adaptation and training of the target recognition method for new target classes and mission scenarios. A comprehensive training and test dataset was defined and prepared using relevant high-definition airborne video sequences. The assessment concept is explained and performance results are given using the established precision-recall diagrams, average precision and runtime figures on representative test data. A comparison to legacy target recognition approaches shows the impressive performance increase by the proposed CNN+SVM machine-learning approach and the capability of real-time high-definition video exploitation.

We propose an image reconstruction method of digital holography for getting more accurate reconstruction. Digital holography provides both the light amplitude and the phase of a specimen through recording the interferogram. Since the Fresenl diffraction can be efficiently implemented by the Fourier transform, zero padding technique can be applied to obtain more accurate information. In this work, we report the method of frequency domain zero padding (FDZP). Both in computer-simulation and in experiment made with a USAF 1951 resolution chart and target, the FDZD gave the more accurate rconstruction images. Even though, the FDZD asks more processing time, with the help of graphics processing unit (GPU), it can find good applications in digital holography for 3-D profile imaging.

We propose a method for computing realistic computer-generated holograms (CGHs) of three-dimensional (3D) objects, where we benefit from well-established graphical processing units (GPUs) and computer graphics techniques to handle occlusion, shading and parallax effects. The graphics render provides a 2D perspective image including occlusion and shading effects. We also extract the depth map data of the scene. The intensity values and 3D positions of object points are extracted by combining the rendered intensity image and the depth map (Z-buffer) image. We divide the depth range into several planes and quantize the depth value of 3D image points to the nearest plane. In the CGH computation part, we perform proper Fresnel transformations of these planar objects and sum them up to create the hologram corresponding to the particular viewpoint. We then repeat the entire procedure for all possible viewpoints and cover the hologram area. The experimental results show that the technique is capable of performing high quality reconstructions in a fast manner., European Research Council under the European Union Seventh Framework Programme (FP7) ERC

Collinear holographic data storage system is a promising candidate for next-generation storage technique. Numerical simulation plays a vital role in the process of revealing physical insight into the effectiveness of proposed methods and providing guidance for further system optimization. In this work, we demonstrated a GPU accelerated numerical model for image formation in collinear holographic data storage system. An average 125 times speedup with 99.8% accuracy was achieved with our accelerated model compared to conventional CPU based simulation. Applications of our model for collinear holographic data storage system such as wavelength drift compensation and noise study were demonstrated.

In this paper the authors present a solution based on a small and lightweight computing platform equipped with a graphics processing unit (GPU) which allows the possibility of performing a real-time fully focused SAR algorithm. The presented system is dedicated for airborne SAR applications including small SAR systems dedicated for medium-sized unmanned aerial vehicle (UAV) platforms. The proposed solution also reduces the need for a storage system. In the paper real SAR results obtained using a Frequency Modulation Continuous Wave (FMCW) radar demonstrator operating at 35 GHz carrier frequency with 1GHz bandwidth are presented. As a radar carrier an airborne platform was used. The presented SAR radar demonstrator was developed by the Warsaw University of Technology in cooperation with the Air Force Institute of Technology, Warsaw, Poland.

The use of Graphics Processing Unit (GPU) for computational work has revolutionized how complex electromagnetic problems are solved. Complex problems which required supercomputers in the past for analysis can now be tackled and solved using personal computers by channeling the computational work towards GPUs instead of the traditional computer Central Processing Unit (CPU). Finite-Difference Time-Domain (FDTD) analysis, which is a computationally expensive method of solving electromagnetic problems is highly parallel in nature and can be readily executed in a GPU. We have developed an algorithm for three dimensional FDTD analysis of optical devices with micro and nano-structures using Compute Unified Device Architecture (CUDA). The developed algorithm exploits the benefits of multiple cores of GPU chips and boosts the speed of simulation without sacrificing its accuracy. We achieved a 25-fold speed up of simulation using CUDA compared to MATLAB code in CPU.

The newly developed smartphone application, named RINO, in this study allows measuring absolute dynamic displacements and processing them in real time using state-of-the-art smartphone technologies, such as high-performance graphics processing unit (GPU), in addition to already powerful CPU and memories, embedded high-speed/ resolution camera, and open-source computer vision libraries. A carefully designed color-patterned target and user-adjustable crop filter enable accurate and fast image processing, allowing up to 240fps for complete displacement calculation and real-time display. The performances of the developed smartphone application are experimentally validated, showing comparable accuracy with those of conventional laser displacement sensor.

This paper presents parallel optimizations in the en-face (C-scan) optical coherence tomography (OCT) display. Compared with the cross-sectional (B-scan) imagery, the production of en-face images is more computationally demanding, due to the increased size of the data handled by the digital signal processing (DSP) algorithms. A sequential implementation of the DSP leads to a limited number of real-time generated en-face images. There are OCT applications, where simultaneous production of large number of en-face images from multiple depths is required, such as real-time diagnostics and monitoring of surgery and ablation. In sequential computing, this requirement leads to a significant increase of the time to process the data and to generate the images. As a result, the processing time exceeds the acquisition time and the image generation is not in real-time. In these cases, not producing en-face images in real-time makes the OCT system ineffective. Parallel optimization of the DSP algorithms provides a solution to this problem. Coarse-grained central processing unit (CPU) based and fine-grained graphics processing unit (GPU) based parallel implementations of the conventional Fourier domain (CFD) OCT method and the Master-Slave Interferometry (MSI) OCT method are studied. In the coarse-grained CPU implementation, each parallel thread processes the whole OCT frame and generates a single en-face image. The corresponding fine-grained GPU implementation launches one parallel thread for every data point from the OCT frame and thus achieves maximum parallelism. The performance and scalability of the CPU-based and GPU-based parallel approaches are analyzed and compared. The quality and the resolution of the images generated by the CFD method and the MSI method are also discussed and compared.

The backprojection-filtration (BPF) algorithm has become a good solution for local reconstruction in cone-beam computed tomography (CBCT). However, the reconstruction speed of BPF is a severe limitation for clinical applications. The selective-backprojection filtration (S-BPF) algorithm is developed to improve the parallel performance of BPF by selective backprojection. Furthermore, the general-purpose graphics processing unit (GP-GPU) is a popular tool for accelerating the reconstruction. Much work has been performed aiming for the optimization of the cone-beam back-projection. As the cone-beam back-projection process becomes faster, the data transportation holds a much bigger time proportion in the reconstruction than before. This paper focuses on minimizing the total time in the reconstruction with the S-BPF algorithm by hiding the data transportation among hard disk, CPU and GPU. And based on the analysis of the S-BPF algorithm, some strategies are implemented: (1) the asynchronous calls are used to overlap the implemention of CPU and GPU, (2) an innovative strategy is applied to obtain the DBP image to hide the transport time effectively, (3) two streams for data transportation and calculation are synchronized by the cudaEvent in the inverse of finite Hilbert transform on GPU. Our main contribution is a smart reconstruction of the S-BPF algorithm with GPU’s continuous calculation and no data transportation time cost. a 5123 volume is reconstructed in less than 0.7 second on a single Tesla-based K20 GPU from 182 views projection with 5122 pixel per projection. The time cost of our implementation is about a half of that without the overlap behavior.

A real-time algorithm for single image dehazing is presented. The algorithm is based on calculation of local neighborhoods of a hazed image inside a moving window. The local neighborhoods are constructed by computing rank-order statistics. Next the dark-channel-prior approach is applied to the local neighborhoods to estimate the transmission function of the scene. By using the suggested approach there is no need for applying a refining algorithm to the estimated transmission such as the soft matting algorithm. To achieve high-rate signal processing the proposed algorithm is implemented exploiting massive parallelism on a graphics processing unit (GPU). Computer simulation results are carried out to test the performance of the proposed algorithm in terms of dehazing efficiency and speed of processing. These tests are performed using several synthetic and real images. The obtained results are analyzed and compared with those obtained with existing dehazing algorithms.

In this paper we solve the problem of pose recognition of a 3D object in non-uniformly illuminated and noisy scenes. The recognition system employs a bank of space-variant correlation filters constructed with an adaptive approach based on local statistical parameters of the input scene. The position and orientation of the target are estimated with the help of the filter bank. For an observed input frame, the algorithm computes the correlation process between the observed image and the bank of filters using a combination of data and task parallelism by taking advantage of a graphics processing unit (GPU) architecture. The pose of the target is estimated by finding the template that better matches the current view of target within the scene. The performance of the proposed system is evaluated in terms of recognition accuracy, location and orientation errors, and computational performance.

Image registration techniques are used among different scientific fields, like medical imaging or optical metrology. The straightest way to calculate shifting between two images is using the cross correlation, taking the highest value of this correlation image. Shifting resolution is given in whole pixels which cannot be enough for certain applications. Better results can be achieved interpolating both images, as much as the desired resolution we want to get, and applying the same technique described before, but the memory needed by the system is significantly higher. To avoid memory consuming we are implementing a subpixel shifting method based on FFT. With the original images, subpixel shifting can be achieved multiplying its discrete Fourier transform by a linear phase with different slopes. This method is high time consuming method because checking a concrete shifting means new calculations. The algorithm, highly parallelizable, is very suitable for high performance computing systems. GPU (Graphics Processing Unit) accelerated computing became very popular more than ten years ago because they have hundreds of computational cores in a reasonable cheap card. In our case, we are going to register the shifting between two images, doing the first approach by FFT based correlation, and later doing the subpixel approach using the technique described before. We consider it as ‘brute force’ method. So we will present a benchmark of the algorithm consisting on a first approach (pixel resolution) and then do subpixel resolution approaching, decreasing the shifting step in every loop achieving a high resolution in few steps. This program will be executed in three different computers. At the end, we will present the results of the computation, with different kind of CPUs and GPUs, checking the accuracy of the method, and the time consumed in each computer, discussing the advantages, disadvantages of the use of GPUs.

Conventionally, the 3D object reconstructed from computational integral imaging technique consists of focused and off-focused areas. The reconstructed object with off-focused points will affect the high-level image analysis such as object classification, recognition, and tracking. Therefore, it is necessary to develop a method to remove the off-focused points for further object analysis. For each point in 3D space, we assume that its intensity values on all the 2D elemental images captured with integral imaging system are similar. Consequently, each focused point on reconstructed depth slice image will share the sample points on elemental images with similar intensity values while the sample points of each off-focused point will have large varied intensity values. If the variance of these sample points on elemental images is larger than a pre-defined threshold, the corresponding point on the reconstructed depth slice image can be estimated as the off-focused point and removed. However, each point on the reconstructed image doing the similar processing sequentially will make the computation burden, especially to multiple depth slice images reconstruction. In this paper, we overview a method to reconstruct the multiple depth slice images with only focused parts in parallel using graphic processing unit (GPU). Experimental results show that this method can reconstruct the multiple depth slice images without off-focused points in a much faster speed on GPU than that on CPU.

We present a Graph Cut based image segmentation that was implemented on a Graphics Processing Unit for acceleration of processing retinal images acquired with OCT. We applied this work to generate a retinal thickness map, and for retinal layer segmentation to enhance the visualization of vasculature networks from distinct retinal capillary beds during acquisition using speckle variance OCT.

Digital breast tomosynthesis (DBT) provides 3D images which remove tissue overlapping and enables better cancer detection. Stationary DBT (s-DBT) uses a fixed X-ray source array to eliminate image blur associated with the x-ray tube motion and provides better image quality as well as faster scanning speed. For limited angle tomography, it is known that iterative reconstructions generally produces better images with fewer artifacts. However classical iterative tomosynthesis reconstruction methods are considerably slower than the filtered back-projection (FBP) reconstruction. The linear x-ray source array used in s-DBT enables a computationally more efficient volume reconstruction using adapted fan beam slice sampling, which transforms the 3-D cone beam reconstruction to a series of 2-D fan beam slice reconstructions. In this paper, we report the first results of the adapted fan-beam volume reconstruction (AFVR) for the s-DBT system currently undergoing clinical trial at UNC, using a simultaneous algebraic reconstruction technique (SART). An analytic breast phantom is used to quantitatively analyze the performance of the AFVR. Image quality of a CIRS biopsy phantom reconstructed using the AFVR method are compared to that using FBP algorithm with a commercial package. Our results show a significant reduction in memory usage and an order of magnitude speed increase in reconstructing speed using AFVR compared to that of classical 3-D cone beam reconstruction. We also observed that images reconstructed by AFVR with SART had a better sharpness and contrast compared to that using FBP. Preliminary results on patient images demonstrates the improved detectability of the s-DBT system over the mammography. By utilizing parallel computing with graphics processing unit (GPU), it is expected that the AFVR method will enable iterative reconstruction technique to be practical for clinical applications.

We present a system for registering the coordinate frame of an endoscope to pre- or intra- operatively acquired CT data based on optimizing the similarity metric between an endoscopic image and an image predicted via rendering of CT. Our method is robust and semi-automatic because it takes account of physical constraints, specifically, collisions between the endoscope and the anatomy, to initialize and constrain the search. The proposed optimization method is based on a stochastic optimization algorithm that evaluates a large number of similarity metric functions in parallel on a graphics processing unit. Images from a cadaver and a patient were used for evaluation. The registration error was 0.83 mm and 1.97 mm for cadaver and patient images respectively. The average registration time for 60 trials was 4.4 seconds. The patient study demonstrated robustness of the proposed algorithm against a moderate anatomical deformation.

In this paper, we systematically demonstrate two real-time CS SD OCT systems based on a conventional desktop having three GPUs. The first one takes fast Fourier transform (FFT) as the sensing technique and under-sampled linear wavenumber spectral sampling as input data, while the second one uses non-uniform fast Fourier transform (NUFFT) and under-sampled nonlinear wavenumber spectral sampling, respectively. The maximum reconstruction speed of 72k and 33.5k A-line/s were achieved for these two systems, respectively, with A-scan size 2048. It is >100 times faster than the C++ implementation and >400 times faster than the MATLAB implementation. Finally, we present real-time dispersion compensated image reconstruction for both systems.

This paper presents a handheld micro-injector system using common-path swept source optical coherence tomography (CP-SSOCT) as a distal sensor with highly accurate injection-depth-locking. To achieve real-time, highly precise, and intuitive freehand control, the system used graphics processing unit (GPU) to process the oversampled OCT signal with high throughput and a smart customized motion monitoring control algorithm. A performance evaluation was conducted with 60-insertions and fluorescein dye injection tests to show how accurately the system can guide the needle and lock to the target depth. The evaluation tests show our system can guide the injection needle into the desired depth with 4.12μm average deviation error while injecting 50nƖ of fluorescein dye.

An improved parallel algorithm for holograms of three-dimensional objects was presented. According to the physical characteristics and mathematical properties of the original ray tracing algorithm for computer generated holograms (CGH), using transform approximation and numerical analysis methods, we extract parts of ray tracing algorithm which satisfy parallelization features and implement them on graphics processing unit (GPU). Meanwhile, through proper design of parallel numerical procedure, we did parallel programming to the two-dimensional slices of three-dimensional object with CUDA. According to the experiments, an effective method of dealing with occlusion problem in ray tracing is proposed, as well as generating the holograms of 3D objects with additive property. Our results indicate that the improved algorithm can effectively shorten the computing time. Due to the different sizes of spatial object points and hologram pixels, the speed has increased 20 to 70 times comparing with original ray tracing algorithm.

The partitioned predictive vector quantization (PPVQ) algorithm is known for its high compression ratio for lossless compression of the ultraspectral sounder data with high spatial and spectral resolutions. With the advent of the multicore technologies, parallelization of several parts of the algorithm has been explored in previous work using a compute unified device architecture (CUDA) aided environment on the Graphics Processing Unit (GPU). Recently the Intel Many Integrated Core (MIC) architecture on a coprocessor is introduced which shows promise in handling more divergent workloads as needed in PPVQ. Therefore we will explore the parallel performance of the MIC-aided implementation. With parallelization of the two most time-consuming modules of linear prediction and vector quantization in PPVQ, the total processing time of an AIRS granule can be compressed in less than 7.5 seconds which is equivalent to a speedup of ~8.8x. The use of MIC for PPVQ compression is thus promising as a low-cost and effective compression solution for ultraspectral sounder data for ground rebroadcast use.

In this paper, we investigate the use of random selection (RS) and random projection (RP) for hyperspectral image analysis, which are data-independent and computationally more efficient than other widely used dimensionality reduction methods. Both anomaly detection and target detection are considered. Due to the random nature, multiple runs of RS or RP are conducted followed by decision fusion to ensure a stable output. Parallel implementations using graphics processing unit (GPU) and clusters are also investigated. The experimental results demonstrated that both RS and RP are capable of providing better target detection performance after decision fusion, while the overall computing time can be greatly decreased with parallel implementations.

A local adaptive algorithm for single image dehazing is presented. The algorithm is able to estimate a dehazed image from an observed hazed scene by solving an objective function whose parameters are adapted to local statistics of the hazed image inside a moving window. The proposed objective function is based on a trade-off among several local rank order statistics of the dehazed signal and the mean-squared-error between the hazed and dehazed signals. In order to achieve a high-rate signal processing, the proposed algorithm is implemented in a graphics processing unit (GPU) exploiting massive parallelism. Experimental results obtained with a laboratory prototype are presented, discussed, and compared with those results obtained with existing single image dehazing methods in terms of objective metrics and computational complexity.

A reliable method for real-time target tracking is presented. The method is based on an interest point detector and a bank of locally adaptive correlation filters. The point detector is used to identify local regions in the observed scene around potential location of the target. The bank of correlation filters is employed to reliably detect the target and accurately estimate its position within the scene, by processing the local regions identified by the detector. Using information of past state estimates of the target the proposed algorithm predicts the state of the target in the next frame in order to perform a fast and accurate target tracking by focusing signal processing only on small regions of the scene in each frame. In order to achieve a real-time operation performance the proposed algorithm is implemented in a graphics processing unit. Experimental results obtained with the proposed method are presented, discussed, and compared with those obtained with a similar state-of-the-art target tracking algorithm.

Light interactions with matter is of remarkable complexity. An adequate modeling of global illumination is a vastly studied topic since the beginning of computer graphics, and still is an unsolved problem. The rendering equation for global illumination is based of refraction and reflection of light in interaction with matter within an environment. This physical process possesses a high computational complexity when implemented in a digital computer. The appearance of an object depends on light interactions with the surface of the material, such as emission, scattering, and absorption. Several image-synthesis methods have been used to realistically render the appearance of light incidence on an object. Recent global illumination algorithms employ mathematical models and computational strategies that improve the efficiency of the simulation solution. This work presents a review the state of the art of global illumination algorithms and focuses on the efficiency of the solution in a computational implementation in a graphics processing unit. A reliable system is developed to simulate realistics scenes in the context of real-time object recognition under different lighting conditions. Computer simulations results are presented and discussed in terms of discrimination capability, and robustness to additive noise, when considering several lighting model reflections and multiple light sources.

The Daniel K. Inouye Solar Telescope (DKIST) Data Handling System (DHS) provides the technical framework and building blocks for developing on-summit instrument quality assurance and data reduction pipelines. The DKIST Visible Broadband Imager (VBI) is a first light instrument that alone will create two data streams with a bandwidth of 960 MB/s each. The high data rate and data volume of the VBI require near-real time processing capability for quality assurance and data reduction, and will be performed on-summit using Graphics Processing Unit (GPU) technology. The VBI data processing pipeline (DPP) is the first designed and developed using the DKIST DHS components, and therefore provides insight into the strengths and weaknesses of the framework. In this paper we lay out the design of the VBI DPP, examine how the underlying DKIST DHS components are utilized, and discuss how integration of the DHS framework with GPUs was accomplished. We present our results of the VBI DPP alpha release implementation of the calibration, frame selection reduction, and quality assurance display processing nodes.

We present a novel approach to use mountain drainage patterns for GPS-Denied navigation of small unmanned aerial systems (UAS) such as the ScanEagle, utilizing a down-looking fixed focus monocular imager. Our proposal allows extension of missions to GPS-denied mountain areas, with no assumption of human-made geographic objects. We leverage the analogy between mountain drainage patterns, human arteriograms, and human fingerprints, to match local drainage patterns to Graphics Processing Unit (GPU) rendered parallax occlusion maps of geo-registered radar returns (GRRR). Details of our actual GPU algorithm is beyond the subject of this paper, and is planned as a future paper. The matching occurs in real-time, while GRRR data is loaded on-board the aircraft pre-mission, so as not to require a scanning aperture radar during the mission. For recognition purposes, we represent a given mountain area with a set of spatially distributed mountain minutiae, i.e., details found in the drainage patterns, so that conventional minutiae-based fingerprint matching approaches can be used to match real-time camera image against template images in the training set. We use medical arteriography processing techniques to extract the patterns. The minutiae-based representation of mountains is achieved by first exposing mountain ridges and valleys with a series of filters and then extracting mountain minutiae from these ridges/valleys. Our results are experimentally validated on actual terrain data and show the effectiveness of minutiae-based mountain representation method. Furthermore, we study how to select landmarks for UAS navigation based on the proposed mountain representation and give a set of examples to show its feasibility. This research was in part funded by Rockwell Collins Inc.

Efficient onboard satellite hyperspectral image compression represents a necessity and a challenge for current and future space missions. Therefore, it is mandatory to provide hardware implementations for this type of algorithms in order to achieve the constraints required for onboard compression. In this work, we implement the Lossy Compression for Exomars (LCE) algorithm on an FPGA by means of high-level synthesis (HSL) in order to shorten the design cycle. Specifically, we use CatapultC HLS tool to obtain a VHDL description of the LCE algorithm from C-language specifications. Two different approaches are followed for HLS: on one hand, introducing the whole C-language description in CatapultC and on the other hand, splitting the C-language description in functional modules to be implemented independently with CatapultC, connecting and controlling them by an RTL description code without HLS. In both cases the goal is to obtain an FPGA implementation. We explain the several changes applied to the original Clanguage source code in order to optimize the results obtained by CatapultC for both approaches. Experimental results show low area occupancy of less than 15% for a SRAM-based Virtex-5 FPGA and a maximum frequency above 80 MHz. Additionally, the LCE compressor was implemented into an RTAX2000S antifuse-based FPGA, showing an area occupancy of 75% and a frequency around 53 MHz. All these serve to demonstrate that the LCE algorithm can be efficiently executed on an FPGA onboard a satellite. A comparison between both implementation approaches is also provided. The performance of the algorithm is finally compared with implementations on other technologies, specifically a graphics processing unit (GPU) and a single-threaded CPU.

In the vision from a mobile infrared platform, the background scene keeps changing due to the moving camera, which results in more background clutters and noise. In this condition, it is difficult to track the target depending on brightness or gradient information. In this paper, we present a new tracking algorithm which segments the moving target according to the difference of velocity field between the target and background. In the proposed algorithm, the phase correlation method is introduced to transform two adjacent frames into one coordinate to remain a relatively static background. And then the Horn-Schunck optical flow is calculated in the tracking window to estimate the velocity field of the target. Finally we introduce the particle filter algorithm to estimate the location of the target, which has a more robust performance by optimizing the transition probability of particles with features of optical flow. Our algorithm can be organized in parallel processing mode enabled for GPU (graphics processing unit). By parallel computing, our algorithm is computationally efficient that it can work in real time. Experimental results show that the proposed algorithm can track the infrared target in real time on a moving platform.

Intra-fractional respiratorymotion during radiotherapy is one of themain sources of uncertainty in dose application creating the need to extend themargins of the planning target volume (PTV). Real-time tumormotion tracking by 2D/3D registration using on-board kilo-voltage (kV) imaging can lead to a reduction of the PTV. One limitation of this technique when using one projection image, is the inability to resolve motion along the imaging beam axis. We present a retrospective patient study to investigate the impact of paired portal mega-voltage (MV) and kV images, on registration accuracy. We used data from eighteen patients suffering from non small cell lung cancer undergoing regular treatment at our center. For each patient we acquired a planning CT and sequences of kV and MV images during treatment. Our evaluation consisted of comparing the accuracy of motion tracking in 6 degrees-of-freedom(DOF) using the anterior-posterior (AP) kV sequence or the sequence of kV-MV image pairs. We use graphics processing unit rendering for real-time performance. Motion along cranial-caudal direction could accurately be extracted when using only the kV sequence but in AP direction we obtained large errors. When using kV-MV pairs, the average error was reduced from 3.3 mm to 1.8 mm and the motion along AP was successfully extracted. The mean registration time was of 190±35 ms . Our evaluation shows that using kVMV image pairs leads to improved motion extraction in 6 DOF. Therefore, this approach is suitable for accurate, real-time tumor motion tracking with a conventional LINAC.

Synthetic aperture (SA) ultrasound imaging system produces highly accurate axial and lateral displacement estimates; however, low frame rates and large data volumes can hamper its clinical use. This paper describes a real-time SA imaging based ultrasound elastography system that we have recently developed to overcome this limitation. In this system, we implemented both beamforming and 2D cross-correlation echo tracking on Nvidia GTX 480 graphics processing unit (GPU). We used one thread per pixel for beamforming; whereas, one block per pixel was used for echo tracking. We compared the quality of elastograms computed with our real-time system relative to those computed using our standard single threaded elastographic imaging methodology. In all studies, we used conventional measures of image quality such as elastographic signal to noise ratio (SNRe). Specifically, SNRe of axial and lateral strain elastograms computed with real-time system were 36 dB and 23 dB, respectively, which was numerically equal to those computed with our standard approach. We achieved a frame rate of 6 frames per second using our GPU based approach for 16 transmits and kernel size of 60 × 60 pixels, which is 400 times faster than that achieved using our standard protocol.

A general-purpose graphics processing unit (GPGPU) has been used for improving computing power in medical ultrasound imaging systems. Recently, a mobile GPU becomes powerful to deal with 3D games and videos at high frame rates on Full HD or HD resolution displays. This paper proposes the method to implement ultrasound signal processing on a mobile GPU available in the high-end smartphone (Galaxy S4, Samsung Electronics, Seoul, Korea) with programmable shaders on the OpenGL ES 2.0 platform. To maximize the performance of the mobile GPU, the optimization of shader design and load sharing between vertex and fragment shader was performed. The beamformed data were captured from a tissue mimicking phantom (Model 539 Multipurpose Phantom, ATS Laboratories, Inc., Bridgeport, CT, USA) by using a commercial ultrasound imaging system equipped with a research package (Ultrasonix Touch, Ultrasonix, Richmond, BC, Canada). The real-time performance is evaluated by frame rates while varying the range of signal processing blocks. The implementation method of ultrasound signal processing on OpenGL ES 2.0 was verified by analyzing PSNR with MATLAB gold standard that has the same signal path. CNR was also analyzed to verify the method. From the evaluations, the proposed mobile GPU-based processing method has no significant difference with the processing using MATLAB (i.e., PSNR

Rapid estimation of blood velocity and visualization of complex flow patterns are important for clinical use ofdiagnostic ultrasound. This paper presents real-time processing for two-dimensional (2-D) vector flow imaging which utilizes an off-the-shelf graphics processing unit (GPU). In this work, Open Computing Language (OpenCL) is used to estimate 2-D vector velocity flow in vivo in the carotid artery. Data are streamed live from a BK Medical 2202 Pro Focus UltraView Scanner to a workstation running a research interface software platform. Processing data from a 50 millisecond frame of a duplex vector flow acquisition takes 2.3 milliseconds seconds on an Advanced Micro Devices Radeon HD 7850 GPU card. The detected velocities are accurate to within the precision limit of the output format of the display routine. Because this tool was developed as a module external to the scanner’s built-in processing, it enables new opportunities for prototyping novel algorithms, optimizing processing parameters, and accelerating the path from development lab to clinic.

We present group coordinate descent algorithms for edge-preserving image denoising that are particularly well-suited to the graphics processing unit (GPU). The algorithms decouple the denoising optimization problem into a set of iterated, independent one-dimensional problems. We provide methods to handle both differentiable regularizers and the absolute value function using the majorize-minimize technique. Specifically, we use quadratic majorizers with Huber curvatures for differentiable potentials and a duality approach for the absolute value function. Preliminary experimental results indicate that the algorithms converge remarkably quickly in time.

Here we present an automated microscope capable of 3D multi-color single molecule localization of individual messenger RNA molecules in a wide range of cell types. We have implemented astigmatic imaging with a cylindrical lens to improve z-localization, and a maximum likelihood estimator on a graphics processing unit to improve localization precision and speed. This microscope will aid in gene expression analysis by its capability to perform high throughput imaging of thick cells and tissues while still maintaining sufficient z resolution to resolve single RNA transcripts in three dimensions. Enhanced z-localization allows for resolving membrane localized and co-localized transcripts.

High-resolution tomography is of great importance to many areas of biomedical imaging, but with it comes several apparent tradeoffs such as a narrowing depth-of-field and increasing optical aberrations. Overcoming these challenges has attracted many hardware and computational solutions. Hardware solutions, though, can become bulky or expensive and computational approaches can require high computing power or large processing times. This study demonstrates memory efficient implementations of interferometric synthetic aperture microscopy (ISAM) and computational adaptive optics (CAO) – two computational approaches for overcoming the depthof- field limitation and the effect of optical aberrations in optical coherence tomography (OCT). Traditionally requiring lengthy post processing, here we report implementations of ISAM and CAO on a single GPU for real-time in vivo imaging. Real-time, camera-limited ISAM processing enabled reliable acquisition of stable data for in vivo imaging, and CAO processing on the same GPU is shown to quickly correct static aberrations. These algorithmic advances hold the promise for high-resolution volumetric imaging in time-sensitive situations as well as enabling aberrationfree cellular-level volumetric tomography.

Photoacoustic tomography (PAT) offers structural and functional imaging of living biological tissue with highly sensitive optical absorption contrast and excellent spatial resolution comparable to medical ultrasound (US) imaging. We report the development of a fully integrated PAT and US dual-modality imaging system, which performs signal scanning, image reconstruction and display for both photoacoustic (PA) and US imaging all in a truly real-time manner. The backprojection (BP) algorithm for PA image reconstruction is optimized to reduce the computational cost and facilitate parallel computation on a state of the art graphics processing unit (GPU) card. For the first time, PAT and US imaging of the same object can be conducted simultaneously and continuously, at a real time frame rate, presently limited by the laser repetition rate of 10 Hz. Noninvasive PAT and US imaging of human peripheral joints in vivo were achieved, demonstrating the satisfactory image quality realized with this system. Another experiment, simultaneous PAT and US imaging of contrast agent flowing through an artificial vessel was conducted to verify the performance of this system for imaging fast biological events. The GPU based image reconstruction software code for this dual-modality system is open source and available for download from http://sourceforge.net/projects/pat realtime .

Polyethylene terephthalate (PET) preforms are massively produced nowadays with the purpose of producing food and beverages packaging and liquid containers. Some varieties of these preforms are produced as multilayer structures, where very thin inner film(s) act as a barrier for nutrients leakage. The knowledge of the thickness of this thin inner layer is important in the production line. The quality control of preforms production requires a fast approach and normally the thickness control is performed by destructive means out of the production line. A spectral domain optical coherence tomography (SD-OCT) method was proposed to examine the thin layers in real time. This paper describes a nondestructive approach and all required signal processing steps to characterize the thin inner layers and also to improve the imaging speed and the signal to noise ratio. The algorithm was developed by using graphics processing unit (GPU) with computer unified device architecture (CUDA). This GPU-accelerated white light interferometry technique nondestructively assesses the samples and has high imaging speed advantage, overcoming the bottlenecks in PET performs quality control.

Parallel hyperspectral unmixing problem is considered in this paper. A semisupervised approach is developed under the linear mixture model, where the abundance’s physical constraints are taken into account. The proposed approach relies on the increasing availability of spectral libraries of materials measured on the ground instead of resorting to endmember extraction methods. Since Libraries are potentially very large and hyperspectral datasets are of high dimensionality a parallel implementation in a pixel-by-pixel fashion is derived to properly exploits the graphics processing units (GPU) architecture at low level, thus taking full advantage of the computational power of GPUs. Experimental results obtained for real hyperspectral datasets reveal significant speedup factors, up to 164 times, with regards to optimized serial implementation.

A real-time tracking system based on adaptive correlation filtering and state prediction is proposed. The system is able to estimate at high-rate the position of multiple targets within the observed scene by taking into account information of past and present scene-frames. The position of the targets in the current frame is estimated with the help of a bank of composite correlation filters applied to several small regions taken from the observed scene. These small regions are updated in each frame according to information from a state predictor based on the motion model of targets in a twodimensional plane. The proposed system is implemented on a graphics processing unit to take advantage of massive parallelism. Computer simulation results obtained with the proposed system are presented and discussed in terms of tracking accuracy and real-time operation efficiency.

This paper proposes a Graphics Processing Unit (GPU) implementation of the stereo correspondence matching using Phase-Only Correlation (POC). The use of high-accuracy stereo correspondence matching based on POC makes it possible to measure accurate 3D shape of an object using stereo vision, while the drawback of POCbased approach is its high computational cost. Addressing this problem, we propose a GPU implementation of POC-based correspondence matching. Through a set of experiments using a variety of GPUs, we demonstrate that the proposed implementation exhibits high-speed and high-efficiency computation of POC-based stereo correspondence matching compared with the CPU implementation. We also apply the proposed implementation to two types of 3D measurement application and demonstrate their effectiveness.

Recent advances in technology have enabled the acquisition of high-resolution 3D models in real-time though the use of structured light scanning techniques. While these advances are impressive, they require large amounts of computing power, thus being limited to using large desktop computers with high end CPUs and sometimes GPUs. This is undesirable in making high-resolution real-time 3D scanners ubiquitous in our mobile lives. To address this issue, this work describes and demonstrates a real-time 3D scanning system that is realized on a mobile device, namely a laptop computer, which can achieve speeds of 20fps 3D at a resolution of 640x480 per frame. By utilizing a graphics processing unit (GPU) as a multipurpose parallel processor, along with a parallel phase shifting technique, we are able to realize the entire 3D processing pipeline in parallel. To mitigate high speed camera transfer problems, which typically require a dedicated frame grabber, we make use of USB 3.0 along with direct memory access (DMA) to transfer camera images to the GPU. To demonstrate the eectiveness of the technique, we experiment with the scanner on both static geometry of a statue and dynamic geometry of a deforming material sample in front of the system.

With the progress of 3D technology, the huge computing capacity for the real-time autostereoscopic display is required. Because of complicated sub-pixel allocating, masks providing arranged sub-pixels are fabricated to reduce real-time computation. However, the binary mask has inherent drawbacks. In order to solve these problems, weighted masks are used in displaying based on partial sub-pixel. Nevertheless, the corresponding computations will be tremendously growing and unbearable for CPU. To improve calculating speed, Graphics Processing Unit (GPU) processing with parallel computing ability is adopted. Here the principle of partial sub-pixel is presented, and the texture array of Direct3D 10 is used to increase the number of computable textures. When dealing with a HD display and multi-viewpoints, a low level GPU is still able to permit a fluent real time displaying, while the performance of high level CPU is really not acceptable. Meanwhile, after using texture array, the performance of D3D10 could be double, and sometimes be triple faster than D3D9. There are several distinguishing features for the proposed method, such as the good portability, less overhead and good stability. The GPU display system could also be used for the future Ultra HD autostereoscopic display.

Back-projection algorithms are probably the fastest approach to reconstruct an image from a set of optoacoustic (photoacoustic) data set. However, standard implementations of back-projection formulae are still not adequate for real-time (greater than 5 frames per second) visualization of three-dimensional structures. This is due to the fact that the number of voxels one needs to reconstruct in three-dimensions is orders of magnitude larger than the number of pixels in two dimensions. Herein we describe a parallel implementation of optoacoustic signal processing and back-projection reconstruction in an attempt to achieve real-time visualization of structures with three-dimensional optoacoustic tomographic systems. For this purpose, the parallel computation power of a graphics processing unit (GPU) is utilized. The GPU is programmed with OpenCL, a programming language for heterogenous platforms. We showcase that with the implementation suggested in this work imaging at frame rates up to 50 high-resolution three-dimensional images per second is achievable.

Guided projectile engagement scenarios often involve impact area constraints, in which it may be less desirable to incur miss distance on one side of a target or within a specified boundary near the target area. Current projectile guidance schemes such as impact point predictors cannot handle these constraints within the guidance loop, and may produce dispersion patterns that are insensitive to these constraints. In this paper, a new projectile guidance law is proposed that leverages real-time Monte Carlo impact point prediction to continually evaluate the probability of violating impact area constraints. The desired aim point is then adjusted accordingly. Real-time Monte Carlo simulation is enabled within the feedback loop through use of graphics processing units (GPU’s), which provide parallel pipelines through which a dispersion pattern can routinely be predicted. The result is a guidance law that can achieve minimum miss distance while avoiding impact area constraints. The new guidance law is described and formulated as a nonlinear optimization problem which is solved in real-time through massively-parallel Monte Carlo simulation. An example simulation is shown in which impact area constraints are enforced and the methodology of stochastic guidance is demonstrated. Finally, Monte Carlo simulations are shown which demonstrate the ability of the stochastic guidance scheme to avoid an arbitrary set of impact area constraints, generating an impact probability density function that optimally trades miss distance within the restricted impact area. The proposed guidance scheme has applications beyond smart weapons to include missiles, UAV’s, and other autonomous systems.

This paper details a Video SAR (Synthetic Aperture Radar) mode that provides a persistent view of a scene centered at the Motion Compensation Point (MCP). The radar platform follows a circular flight path. An objective is to form a sequence of SAR images while observing dynamic scene changes at a selectable video frame rate. A formulation of backprojection meets this objective. Modified backprojection equations take into account changes in the grazing angle or squint angle that result from non-ideal flight paths. The algorithm forms a new video frame relying upon much of the signal processing performed in prior frames. The method described applies an appropriate azimuth window to each video frame for window sidelobe rejection. A Cardinal Direction Up (CDU) coordinate frame forms images with the top of the image oriented along a given cardinal direction for all video frames. Using this coordinate frame helps characterize a moving target’s target response. Generation of synthetic targets with linear motion including both constant velocity and constant acceleration is described. The synthetic target video imagery demonstrates dynamic SAR imagery with expected moving target responses. The paper presents 2011 flight data collected by General Atomics Aeronautical Systems, Inc. (GA-ASI) implementing the video SAR mode. The flight data demonstrates good video quality showing moving vehicles. The flight imagery demonstrates the real-time capability of the video SAR mode. The video SAR mode uses a circular shift register of subapertures. The radar employs a Graphics Processing Unit (GPU) in order to implement this algorithm.

This work extends the multi-histogram volume rendering framework proposed by Kniss et al. [1] to provide rendering results based on the impression of overlaid triangles on a graph of image intensity versus gradient magnitude. The developed method of volume rendering allows for greater emphasis to boundary visualization while avoiding issues common in medical image acquisition. For example, partial voluming effects in computed tomography and intensity inhomogeneity of similar tissue types in magnetic resonance imaging introduce pixel values that will not reflect differing tissue types when a standard transfer function is applied to an intensity histogram. This new framework uses developing technology to improve upon the Kniss multi-histogram framework by using Java, the GPU, and MIPAV, an open-source medical image processing application, to allow multi-histogram techniques to be widely disseminated. The OpenGL view aligned texture rendering approach suffered from performance setbacks, inaccessibility, and usability problems. Rendering results can now be interactively compared with other rendering frameworks, surfaces can now be extracted for use in other programs, and file formats that are widely used in the field of biomedical imaging can be visualized using this multi-histogram approach. OpenCL and GLSL are used to produce this new multi-histogram approach, leveraging texture memory on the graphics processing unit of desktops to provide a new interactive method for visualizing biomedical images. Performance results for this method are generated and qualitative rendering results are compared. The resulting framework provides the opportunity for further applications in medical imaging, both in volume rendering and in generic image processing.

To make Quantitative Radiology a reality in routine radiological practice, computerized automatic anatomy recognition (AAR) becomes essential. Previously, we presented a fuzzy object modeling strategy for AAR. This paper presents several advances in this project including streamlined definition of open-ended anatomic objects, extension to multiple imaging modalities, and demonstration of the same AAR approach on multiple body regions. The AAR approach consists of the following steps: (a) Collecting image data for each population group G and body region B. (b) Delineating in these images the objects in B to be modeled. (c) Building Fuzzy Object Models (FOMs) for B. (d) Recognizing individual objects in a given image of B by using the models. (e) Delineating the recognized objects. (f) Implementing the computationally intensive steps in a graphics processing unit (GPU). Image data are collected for B and G from our existing patient image database. Fuzzy models for the individual objects are built and assembled into a model of B as per a chosen hierarchy of the objects in B. A global recognition strategy is used to determine the pose of the objects within a given image I following the hierarchy. The recognized pose is utilized to delineate the objects, also hierarchically. Based on three body regions tested utilizing both computed tomography (CT) and magnetic resonance (MR) imagery, recognition accuracy for non-sparse objects has been found to be generally sufficient ( 3 to 11 mm or 2-3 voxels) to yield delineation false positive (FP) and true positive (TP) values of < 5% and ≥ 90%, respectively. The sparse objects require further work to improve their recognition accuracy.

This paper presents initial performance results for a software toolkit that implements GPU-based parallel computation of digitally reconstructed radiographs (DRRs) from volumetric imaging data for 2D-3D registration. The computational parallelism is achieved using NVIDIA’s CUDA implementation of general purpose computing on the graphics processing unit. The sample volumetric imaging data shown here is from CT imaging of a cadaveric foot, but the toolkit can be applied equally well to other volumetric imaging data. An efficient implementation requires launching hundreds of simultaneous, independent computational threads and fast thread access to the global memory where they need to read and write data. We have implemented fast DRR generation by launching a computational thread for each pixel in the image, and achieve efficient memory access by using 3D texture memory to store the volumetric data and constant memory to store global information such as intensifier coordinates. The Thrust software library was used to store individual bone DRRs, which enables efficient memory transfer and use of built-in device operators during image compositing and similarity quantification. By storing individual DRRs, the toolkit can support independent kinematics for up to 32 segmented objects. We show that the algorithm scales with the number of processors and compare timings for three commercially available GPUs. Here we present our initial fast DRR computations to demonstrate that the toolkit can produce useful results for a full 160 × 339 × 439 stack of floating point density data on a high resolution 1152 × 896 pixel screen in 1.3 ms and on a 512 × 512 pixel screen in less than 0.6 ms.

We describe a novel dual-functional optical coherence tomography (OCT) system with both a 3-D OCT real time scanner and a fiber probe using a sapphire ball lens for imaging and sensing the critical structures of the temporal bone. To prevent injury to facial nerve, 3-D visualization links anatomic landmarks to 3-D map of critical intracochlear structures. We used a graphics processing unit to boost the computing and 3-D rendering performance of swept source OCT. Both the intracochlear structures and facial nerve trunk of cadaveric human temporal bones are clearly identified with 3-D OCT volumetric rendering.

Spectroscopic optical coherence tomography (OCT) is an extension of the standard backscattering intensity analysis of OCT. It enables depth resolved monitoring of molecular and structural differences of tissue. One drawback of most methods to calculate the spectroscopic data is the long processing time. Also systematic and stochastic errors make the interpretation of the results challenging. Our approach combines modern signal processing tools with powerful graphics processing unit (GPU) programming. The processing speed for the spectroscopic analysis is nearly 3 mega voxel per second. This allows us to analyze multiple B-Scans in a few seconds and to display the results as a three dimensional data set. Our algorithm contains the following steps in addition to the conventional processing for frequency domain OCT: a quality map to exclude noisy parts of the data, spectral analysis by short time Fourier transform, feature reduction by Principal Component Analysis, unsupervised pattern recognition with K-means and rendering of the gray scale backscattering OCT data which is superimposed with a color map that is based on the results of the pattern recognition algorithm. Our set up provides a spectral range from 650-950nm and is optimized to suppress chromatic errors. In a proof-of-principle attempt, we already achieved additional spectroscopic contrast in phantom samples including scattering microspheres of different sizes and ex vivo biological tissue. This is an important step towards a system for real time spectral analysis of OCT data, which would be a powerful diagnosis tool for many diseases e.g. cancer detection at an early stage.