Search

Video frame interpolation is a challenging problem that involves various scenarios depending on the variety of foreground and background motions, frame rate, and occlusion. Therefore, generalizing across different scenes is difficult for a single network with fixed parameters. Ideally, one could have a different network for each scenario, but this will be computationally infeasible for practical applications. In this work, we propose MetaVFI, an adaptive video frame interpolation algorithm that uses additional information readily available at test time but has not been exploited in previous works. We initially show the benefits of test-time adaptation through simple fine-tuning of a network and then greatly improve its efficiency by incorporating meta-learning. Thus, we obtain significant performance gains with only a single gradient update without introducing any additional parameters. Moreover, the proposed MetaVFI algorithm is model-agnostic which can be easily combined with any video frame interpolation network. We show that our adaptive framework greatly improves the performance of baseline video frame interpolation networks on multiple benchmark datasets.

Small object detection is the main challenge for image detection of unmanned aerial vehicles (UAVs), especially with small pixel ratios and blurred boundaries. In this paper, a one-stage detector (SF-SSD) is proposed with a new spatial cognition algorithm. The deconvolution operation is introduced to a feature fusion module, which enhances the representation of shallow features. These more representative features prove effective for small-scale object detection. Empowered by a spatial cognition method, the deep model can re-detect objects with less-reliable confidence scores. This enables the detector to improve detection accuracy significantly. Both between-class similarity and within-class similarity are fully exploited to suppress useless background information. This motivates the proposed model to take full use of semantic features in the detection process of multi-class small objects. A simplified network structure can improve the speed of object detection. The experiments are conducted on a newly collected dataset (SY-UAV) and the benchmark datasets (CARPK and PUCPR+). To further demonstrate the effectiveness of the spatial cognition module, a multi-class object detection experiment is conducted on the Stanford Drone dataset (SDD). The results show that the proposed model achieves high frame rates and better detection accuracies than the state-of-the-art methods, which are 90.1% (CAPPK), 90.8% (PUCPR+), and 91.2% (SDD).

Fetal congenital heart disease (CHD) is the most common type of fatal congenital malformation. Fetal four-chamber (FC) view is a significant and easily accessible ultrasound (US) image among fetal echocardiography images. Automatic detection of four fetal heart chambers considerably contributes to the early diagnosis of fetal CHD. Furthermore, robust and discriminative features are essential for detecting crucial visualizing medical images, especially fetal FC views. However, it is an incredibly challenging task due to several key factors, such as numerous speckles in US images, the fetal four chambers with small size and unfixed positions, and category confusion caused by the similarity of cardiac chambers. These factors hinder the process of capturing robust and discriminative features, hence destroying the fetal four chambers' precise detection. Therefore, we propose an intelligent feature learning detection system (FLDS) for FC views to detect the four chambers. A multistage residual hybrid attention module (MRHAM) presented in this paper is incorporated in the FLDS for learning powerful and robust features, helping FLDS accurately locate the four chambers in the fetal FC views. Extensive experiments demonstrate that our proposed FLDS outperforms the current state-of-the-art, including the precision of 0.919, the recall of 0.971, the F1 score of 0.944, the mAP of 0.953, and the frames per second (FPS) of 43. In addition, our proposed FLDS is also validated on other visualizing nature images such as the PASCAL VOC dataset, achieving a higher mAP of 0.878 while input size is 608 * 608.

Object detection is the most critical and foundational sensing module for the autonomous movement platform. However, most of the existing deep learning solutions are based on GPU servers, which limits their actual deployment. We present an efficient multi-sensor fusion based object detection model that can be deployed on the off-the-shelf edge computing device for the vehicle platform. To achieve real-time target detection, the model eliminates a large number of invalid point clouds through ground filtering algorithm, and then adds texture information (fused from camera image) through point cloud coloring to enhance features. The proposed PV-EncoNet efficiently encodes both the spatial and texture features of each colored point through point-wise and voxel-wise encoding, and then predicts the position, heading and class of the objects. The final model can achieve about 17.92 and 24.25 Frame per Second (FPS) on two different edge computing platforms, and the detection accuracy is comparable with the state-of-the-art models on the KITTI public dataset (i.e., 88.54% for cars, 71.94% for pedestrians and 73.04% for cyclists). The robustness and generalization ability of the PV-EncoNet for the 3D colored point cloud detection task is also verified by deploying it on the local vehicle platform and testing it on real road conditions.

Autonomous unmanned aerial vehicles (UAVs) are essential for detecting and tracking specific events, such as automatic navigation. The intelligent monitoring of people's social distances in crowds is one of the most significant events caused by the coronavirus. The virus is spreading more quickly among the crowds, and the disease cycle continues in congested areas. Due to the error that occurs when humans monitor their activity, an automated model is required to alert to social distance violations in crowds. As a result, this article proposes a two-step framework based on autonomous UAV videos, including human tracking and deep learning-based recognition of the crowd's social distance. The deep architecture is a modified-fast and lightweight ShuffleNet learning structure. First, the Kalman filter is used to determine the positions of individuals, and then the modified ShuffleNet is used to refine the bounding boxes obtained and determine the social distance. The social distance is calculated using the initial refinement of the bounding box obtained during the tracking step and the scale in frames of the human body. The observed average accuracy, average processing time (APT), and processed frame per second (FPS) for three congestion datasets were 97.5%, 84 milliseconds, and 11.5 FPS, respectively. Real-time decision-making was achieved by reducing the size and resolution of the frames. Additionally, the frames were re-labeled to reduce the computational complexity associated with detecting social distancing. The experimental results demonstrated that the proposed method could operate more quickly and accurately on various resolution frames of UAV videos with difficult conditions. IEEE

Most of siamese network and correlation filter (CF) based trackers usually employ the context regression scheme to achieve appealing performance in both accuracy and efficiency. However, they are prone to drifting in challenging situations exhibiting occlusion, out-of-view and large-scale variations due to the lack of failure correction ability in these regressors. Particle filter based trackers can help to recover from tracking failures since several particles of high confidence about the target can be remained for the probability estimation of next frames, but need the large numbers of particles for each frame. In this paper, we propose a generic dynamic particle filter framework, which can reasonably control the number of particles in different scenarios, to improve the robustness of siamese and CF trackers by jointing the target classifier to relieve drifting. Our fundamental insight is that general scenarios are processed efficiently by the context regressor with few particles, while special scenarios are handled effectively by the target classifier with many particles. We propose a novel measure to determine whether to adopt the regressor and few particles to estimate target states with high measure scores, or increase the number of particles to prevent drifting with the proposed multi-template matching strategy in the classifier. In extensive experiments on eight large-scale benchmarks including OTB, UAV, TC128, VOT2017, VOT2019, LaSOT, TrackingNet and Got-10k, the proposed framework enables many basic siamese and CF trackers to operate at least over 24 frames per second and achieve superior tracking performance than themself, as well as the comparable accuracy with the state-of-the-art trackers. Furthermore, our framework is flexible and still has great potential for improvement and generalization.

Multi-object tracking (MOT) is a crucial technology for security surveillance, which is computationally intensive due to the requirement of processing a large number of video streams within low latency in practice. The input video streams of MOT are processed on a cloud computing center with abundant computational capability, posing heavy pressures on delivering video streams to the cloud. Recent advances in the Internet of Things (IoT) technology provide edge-computing-based solutions for video analytics at scale. However, the gap between MOT’s high computational capability demand and IoT devices’ resource-constrained nature remains significant. In this paper, a resource-efficient multi-object tracking method (REMOT) is proposed for real-time surveillance on IoT embedded devices, including an affinity measurement based on an appearance model with angular triplet loss and a motion association that substitutes the time-consuming graph-based data association stage. Considering the trade-off between latency and accuracy, we design an optimization strategy on the parallel processing of deep learning models’ layers to accelerate the inference speed with less accuracy loss. Besides, we employ a model compression strategy for model size reduction. Experiments on MOT16 and MOT17 benchmarks demonstrate that REMOT reduces 2.4x latency compared with the original implementation and achieves a running speed of 81 frames per second (fps) on an embedded device with only a marginal accuracy loss (6%), which meets the requirements of real-time processing and low-latency response for surveillance.

Environmental context prediction is important for wearable robotic applications, such as terrain-adaptive control. System efficiency is critical for wearable robots, in which system resources (e.g., processors and memory) are highly constrained. This article aims to address the system efficiency of real-time environmental context prediction for lower limb prostheses. First, we develop an uncertainty-aware frame selection strategy that can dynamically select frames according to lower limb motion and uncertainty captured by Bayesian neural networks (BNNs) for environment prediction. We further propose a dynamic Bayesian gated recurrent unit (D-BGRU) network to address the inconsistent frame rate which is a side effect of the dynamic frame selection. Second, we investigate the effects on the tradeoff between computational complexity and environment prediction accuracy of adding additional sensing modalities (e.g., GPS and an on-glasses camera) into the system. Finally, we implement and optimize our framework for embedded hardware, and evaluate the real-time inference accuracy and efficiency of classifying six types of terrains. The experiments show that our proposed frame selection strategy can reduce more than 90% of the computations without sacrificing environment prediction accuracy, and can be easily extended to the situation of multimodality fusion. We achieve around 93% prediction accuracy with less than one frame to be processed per second. Our model has 6.4 million 16-bit float numbers and takes 44 ms to process each frame on a lightweight embedded platform (NVIDIA Jetson TX2).

This paper provides an end-to-end latency model of real-time generated live videos composed of sub-models of capturing, encoding, network, decoding, rendering, and display refreshing. We build the model by deeply analyzing each sub part's latency with parameters like video's Spatial (frame scale s), Temporal (frame rate t), Amplitude (quantization step-size q) Resolutions (STAR), Compressed Frame Bit Numbers (nF), and other necessary parameters. The latency of 640 combinations of STAR for eight representative video sequences is measured to fit accurate models. We primarily focus on processing procedures, whose latency is becoming the principal affection, as network latency decreases due to the its development. We then introduce a baseline video sequence set to reduce the complexity when fitting the encoding and decoding sub-models by predicting the model parameters on new hardware from parameters of a known one, and improve models' robustness crossing different scenarios. These sub-models correlate well with the realistic latency data, with the average Pearson Correlation Coefficient (PCC) of 0.99 and the average Spearman's Rank Correlation Coefficient (SRCC) of 0.98, according to an independent validation test. Furthermore, we validate our end-to-end latency model in a simulated end-to-end live video transmission from generating to displaying, introducing the latency as a strict condition in adaptive bit-rate selection. The simulation shows that our model could significantly increase time during which the latency can satisfy the given tolerance at a low cost.

Sequence image target tracking has two requirements of accuracy and speed. Aiming at the problem of slow speed of convolutional neural network target tracking algorithm, an improved algorithm for sequence image target tracking is designed by using a fusion of Faster MDNet and optical flow method. The target algorithm designed in this paper is improved on the basis of Multi-Domain Convolutional Neural Network (MDNet). The optical flow method is used to obtain the primary selection box first, which can effectively deal with the challenges brought by scale change and in-plane rotation, and has good tracking speed and performance. The objective algorithm of this paper is used to explore the Marangoni Effect in the melting process of solid iron tailings in molten blast furnace slag. Because the main component of iron tailings is silica, and the melting point of silica is the highest, so taking silica as the research object, an intelligent analysis of Marangoni Effect in high temperature melting process of silica particles is realized. The results show that: the target algorithm achieves the real-time standard (FPS > 24), the average Distance Precision (DP) is 97.3%, the average Success Rate (SR) is 98.02%, the average Center Location Error (CLE) is 6.2684, and the average Overlap Precision (OP) coverage is 63.86%. The target algorithm has certain advantages in five evaluation indexes: distance precision, success rate, center location error, overlap precision and frames per second.

Distracted driving is a problem which yearly causes a large amount of road traffic crashes with high rates of fatalities and injured persons. Recently, car manufacturers started to integrate driver monitoring systems to detect visual distraction. This paper proposes a method to extend such systems by driver posture classification to detect driver cell phone usage and food consumption. Such an extension can be beneficial since systems that focus on the detection of visual distraction mainly rely on head pose and gaze information. Thus, distraction caused by cell phone usage or food consumption can not be detected by these systems when the driver is looking to the road ahead. To robustly detect those types of manual and cognitive distraction, different deep learning models were trained and evaluated based on a new image dataset which was captured by two infrared cameras to ensure that a large range of head angles can be covered by the system. Separate Convolutional Neural Networks (CNNs) were trained and evaluated for the dataset of the left and the right camera to optimize the classification accuracy. The trained CNNs revealed a competitive test accuracy of 92.88% and 90.36% for the left and the right camera, respectively. In inference mode, the models achieve a frame rate of 44Hz and 28Hz for the left and the right camera, respectively. The combination of the classification output of both networks revealed a test accuracy of 92.54%.

The ultra-high-definition 16K Virtual Reality (VR) video is coming to ages with more ”real” virtual experience and less cybersickness. However, the huge bitrate and decoding overhead would overwhelm today’s network and mobile hardware. The widely-known Field-of-View (FoV) adaptation streaming method still has severe bitrate wastes and decoding overhead as it delivers FoV areas with grid-like static tiles. Inspired by this, we present a novel ShiftTile-traCking (STC) streaming scheme, which crops and delivers tiles by tracking FoV movement. It is equivalent to deliver an FoV planar video instead of VR videos. This would save huge bit-rate and reduce decoding complexity. We mainly entail three contributions. 1) To reduce projection distortions, a novel FoV-centric sphere projection is proposed, which projects VR videos with the center of users’ FoVs. 2) To cover diverse FoV movement trajectories with a limited number of tiles, we propose an optimal tiling algorithm by trajectory clustering. 3) To be resilient to FoV prediction errors, we propose an accuracy-sensitive streaming algorithm, which scales FoV areas by the prediction accuracy. The evaluation shows that under the same network conditions, STC improves up to 1.3dB V-PSNR, reduces up to 13.2% buffering ratio, and achieves 60% faster decoding speed (61.5 frames per second) compared with state-of-the-art solutions.

DVFS (dynamic voltage and frequency scaling) is a system-level technique that adjusts voltage and frequency levels of CPU/GPU at runtime to balance energy efficiency and high performance. DVFS has been studied for many years, but it is considered still challenging to realize a DVFS that performs ideally for mobile devices for two main reasons: i) an optimal power budget distribution between CPU and GPU in a power-constrained platform can only be defined by the application performance, but conventional DVFS implementations are mostly application-agnostic; ii) mobile platforms experience dynamic thermal environments for many reasons such as mobility and holding methods, but conventional implementations are not adaptive enough to such environmental changes. In this work, we propose a deep reinforcement learning-based frequency scaling technique, zTT. zTT learns thermal environmental characteristics and jointly scales CPU and GPU frequencies to maximize the application performance in an energy-efficient manner while achieving zero thermal throttling. Our evaluations for zTT implemented on Google Pixel 3a and NVIDIA JETSON TX2 platform with various applications show that zTT can adapt quickly to changing thermal environments, consistently resulting in high application performance with energy efficiency. In a high-temperature environment where a rendering application with the default mobile DVFS fails to keep producing more than a target frame rate, zTT successfully manages to do so even with 23.9% less average power consumption.

Gaming video streaming services have grown tremendously in the past few years, with higher resolutions, higher frame rates and HDR gaming videos getting increasingly adopted among the gaming community. Since gaming content as such is different from non-gaming content, it is imperative to evaluate the performance of the existing encoders to help understand the bandwidth requirements of such services, as well as further improve the compression efficiency of such encoders. Towards this end, we present in this paper GamingHDRVideoSET, a dataset consisting of eighteen 10-bit UHD-HDR gaming videos and encoded video sequences using four different codecs, together with their objective evaluation results. The dataset is available online at [to be added after paper acceptance]. Additionally, the paper discusses the codec compression efficiency of most widely used practical encoders, i.e., x264 (H.264/AVC), x265 (H.265/HEVC) and libvpx (VP9), as well the recently proposed encoder libaom (AV1), on 10-bit, UHD-HDR content gaming content. Our results show that the latest compression standard AV1 results in the best compression efficiency, followed by HEVC, H.264, and VP9., 14 pages, 8 figures, submitted to IEEE journal

For embedded, mobile and edge-computing intelligent applications, this brief proposes a low-cost real-time neuromorphic hardware system of spiking Extreme Learning Machine (ELM) equipped with on-chip triplet-based reward-modulated spike-timing-dependent plasticity (R-STDP) learning capability. Our design employs a time-step pipelined dual-core architecture consisting of parallel computing unit arrays to improve processing speed, as well as a trace-assisting learning mechanism and on-the-fly hidden layer weight re-generators to significantly reduce hardware resource costs. Our architecture is scalable to different spiking ELM sizes under different tradeoffs among processing speed, recognition accuracy and resource costs. Tests showed that the on-chip triplet R-STDP learning capability can help to achieve relatively high recognition accuracies on our hardware system. An FPGA prototype with low logic and memory resource consumption was implemented, achieving 93% and 78.5% recognition accuracies on the MNIST and Fashion-MNIST image datasets, respectively, at a speed of 30 frames per second (fps) for inference and 22.5 fps for on-chip learning.

Autonomous mobile systems such as vehicles or robots are equipped with multiple sensor modalities including Lidar, RGB, and Radar. The fusion of multi-modal information can enhance task accuracy but indiscriminate sensing and fusion in all modality increases demand on available system resources. This paper presents a task-driven approach to input fusion that minimizes utilization of resource-heavy sensors and demonstrates its application to Visual-Lidar fusion for object tracking and path planning. Proposed spatiotemporal sampling algorithm activates Lidar only at regions-of-interest identified by analyzing visual input and reduces the Lidar ‘base frame rate’ according to kinematic state of the system. This significantly reduces Lidar usage, in terms of data sensed/transferred and potentially power consumed, without severe reduction in performance compared to both a baseline decision-level fusion and state-of-the-art deep multi-modal fusion.

Recent years have witnessed the prevalence of using the Siamese network for RGB-T tracking because of its remarkable success in RGB object tracking. Despite their faster than real-time speeds, existing RGB-T Siamese trackers suffer from low accuracy and poor robustness, compared to other state-of-the-art RGB-T trackers. To address such issues, a new complementarity- and distractor-aware RGB-T tracker based on Siamese network (referred to as SiamCDA) is developed in this paper. To this end, several modules are presented, where the feature pyramid network (FPN) is incorporated into the Siamese network to capture the cross-level information within unimodal features extracted from the RGB or the thermal images. Next, a complementarity-aware multi-modal feature fusion module (CA-MF) is specially designed to capture the cross-modal information between RGB features and thermal features. In the final bounding box selection phase, a distractor-aware region proposal selection module (DAS) further enhances the robustness of our tracker. On top of the technical modules, we also build a large-scale, diverse synthetic RGB-T tracking dataset, containing more than 4831 pairs of synthetic RGB-T videos and 12K synthetic RGB-T images. Extensive experiments on three RGB-T tracking benchmark datasets demonstrate the outstanding performance of our proposed tracker with a tracking speed over 37 frames per second (FPS).

Non-line-of-sight (NLOS) imaging systems reconstruct hidden scenes using computational methods based on indirect light that diffusely reflected from relay walls. Due to the computation and memory requirements of reconstruction algorithms, real-time NLOS imaging for room-size scenes based on non-confocal data has long been challenging. This paper proposes a field programmable gate array (FPGA) accelerator for the recently proposed Rayleigh-Sommerfeld Diffraction (RSD)-based NLOS reconstruction method. In the proposed accelerator design, ring sampling and radius sampling techniques are proposed to reduce the memory requirements by reconstructing the RSD kernels with a set of kernel bases and ring sampling coefficients during the runtime. Based on that, a customized hardware architecture and the corresponding FPGA design for real-time RSD-based NLOS reconstruction is further proposed. Implementation results show that the proposed FPGA accelerator is capable of reconstructing NLOS scenes at 25 frames per second (FPS), running at a relatively slow clock frequency of 50 MHz. To the best knowledge of the authors, this is the first real-time enabled FPGA accelerator for room-size NLOS imaging with a resolution of 128x 128.

Ultrafast ultrasound imaging remains an active area of interest in the ultrasound community due to its ultra-high frame rates. Recently, a wide variety of studies based on deep learning have sought to improve ultrafast ultrasound imaging. Most of these approaches have been performed on radio frequency (RF) signals. However, in-phase/quadrature (I/Q) digital beamformers are now widely used as low-cost strategies. In this work, we used complex convolutional neural networks for reconstruction of ultrasound images from I/Q signals. We recently described a convolutional neural network architecture called ID-Net, which exploited an inception layer designed for reconstruction of RF diverging-wave ultrasound images. In the present study, we derive the complex equivalent of this network; i.e., the Complex-valued Inception for Diverging-wave Network (CID-Net) that operates on I/Q data. We provide experimental evidence that CID-Net provides the same image quality as that obtained from RF-trained convolutional neural networks; i.e., using only three I/Q images, the CID-Net produces high-quality images that can compete with those obtained by coherently compounding 31 RF images. Moreover, we show that CID-Net outperforms the straightforward architecture that consists of processing the real and imaginary parts of the I/Q signal separately, which thereby indicates the importance of consistently processing the I/Q signals using a network that exploits the complex nature of such signals.

In video-based dynamic point cloud compression (V-PCC), 3D point clouds are projected into patches, and then the patches are padded into 2D images suitable for the video compression framework. However, the patch projection-based method produces a large number of empty pixels; the far and near components are projected to generate different 2D images (video frames), respectively. As a result, the generated video is with high resolutions and double frame rates, so the V-PCC has huge computational complexity. This paper proposes an occupancy map guided fast V-PCC method. Firstly, the relationship between the prediction coding and block complexity is studied based on a local linear image gradient model. Secondly, according to the V-PCC strategies of patch projection and block generation, we investigate the differences of rate-distortion characteristics between different types of blocks, and the temporal correlations between the far and near layers. Finally, by taking advantage of the fact that occupancy maps can explicitly indicate the block types, we propose an occupancy map guided fast coding method, in which coding is performed on the different types of blocks. Experiments have tested typical dynamic point clouds, and shown that the proposed method achieves an average 43.66% time-saving at the cost of only 0.27% and 0.16% Bjontegaard Delta (BD) rate increment under the geometry Point-to-Point (D1) error and attribute Luma Peak-Signal-Noise-Ratio (PSNR), respectively.

Vision-based Marine ship detection has been explored for many years, effectively improving maritime transportation management. However, many ship detection methods get troubles on the movable platform, which are summarized as follows: 1) high time efficiency is needed to handle the rapid changing of maritime scenes so that ships can be detected successfully on movable platform; 2) different appearances caused by sizes and viewpoints enlarge the intra-class distance of ships, which exceed the representation capability of some features like histogram of oriented gradients (HOG) with fixed size. For addressing these issues, we propose a rapid ship detection method based on multi-size gradient features and multi-branch support vector machine (SVM) in a ``coarse-to-fine'' manner that can be applied on a movable platform. The proposed multi-size gradient features are used to represent the ships with different sizes, including the coarse and fine gradient features. To speed up the detection process in a sliding way, the coarse ship locations are firstly generated only based on the coarse gradient features, which highly reduces the computational cost. Then, the multi-size gradient features extracted from these locations are determined by a multi-branch SVM model which is designed to deal with features of different dimensions and improve the precision of ship position with fine gradient features. The proposed method is tested on the changing background where ships have different sizes. Experimental results show that the proposed method obtains average precision (AP) of 0.795 and its detection speed achieves 60.6 frame per second, which achieve real-time performance and satisfactory detection precision.

This article introduces a pulsed laser direct time-of-flight (dTOF) flash light detection and ranging (LiDAR) sensor fabricated in 0.18-μm HV CMOS technology. The chip includes 32x 32 macro pixels and 1024 time-to-digital converters (TDCs). A noise filtering circuit with different threshold (Nth) configuration is adopted in each macro pixel [formed by four single-photon avalanche diodes (SPADs)], which can suppress strong background light (BG) induced pile-up. To verify the imaging function and effectiveness of the noise filtering circuit, two systems are implemented (System1 for indoor imaging measurement and System2 for outdoor distance measurement). With the help of the noise filtering circuit and a reasonable signal-to-background noise ratio (SBR), the maximum detection range outdoors with reasonable accuracy can be greatly extended (from 12m @ Nth=1 of System2 to more than 20m @ Nth=2 of System2 under 70klux of background noise). The counter in the noise filtering circuit can be reused to get intensity information. A robust 13-bit TDC with a reliable reset is introduced. Thanks to a dedicated START/STOP logic and a Schmitt trigger, large TDC quantization errors can be avoided. It achieves a 200ps resolution (LSB) and exhibits an INLp-p of 3.55LSB and a DNLp-p of 0.53LSB. The maximum inter-frame rate can reach 270kfps with 16 IOs operating at speed of 500MHz. Combining 9k inter-frames to get one frame, a frame rate of 30 is achieved for an indoor 3-D imaging. For outdoor measurement, more laser pulses should be accumulated.

Electrical resistance tomography (ERT) is an inferential imaging technique that has shown promising results for enabling large-area tactile sensors constructed from a piezoresistive sheet. The performance of such sensors is improved by increasing the number of electrodes, but the number of measurements and the computational cost also increase. In this paper, we propose a new measurement algorithm for ERT-based tactile sensors: it adaptively changes the measurement pattern to be optimal for the present external stimulus. Regions of normal pressure are first detected by a base measurement pattern that maximizes the distinguishability of local conductivity changes. When a new contact is detected, a set of local patterns is selectively recruited near the pressed region to acquire more detailed information. For fast and parallel execution, the proposed algorithm is implemented with a field-programmable gate array (FPGA). It is validated through indentation experiments on an ERT-based sensor that has 32 electrodes. The optimized base pattern of 100 measurements enabled a frame rate five times faster than before. Transmitting only detected contact events reduced the idle data rate to 0.5% of its original value. The pattern adapted to new contacts with a latency of only 80 microseconds and an accuracy of 99.5%, enabling efficient, high-quality real-time reconstruction of complex multi-contact conditions.

Surface defect inspection is a key step to ensure the quality of the hot rolled steel surface. However, current advanced detection methods have high precision but low detection speed, which hinders the application of the detector in actual production. In this work, a real-time detection network (RDN) focusing on both speed and accuracy is proposed to solve the problem of steel surface defect detection. RDN takes ResNet-dcn, a modular encoding and decoding network with light weights, as the basic convolutional architecture whose backbone is pre-trained on ImageNet. To improve the detection accuracy, a skip layer connection module (SCM) and a pyramid feature fusion module (PFM) are involved into RDN. On the standard dataset NEU-DET, the proposed method can achieve the state-of-the-art recognition speed of 64 frames per second(FPS) and the mean average precision of 80.0% on a single GPU, which fully meets the requirements of the detection accuracy and speed in the actual production line.

This paper presents an end-to-end object-detection accelerator that processes raw Bayer images to generate detection results. The accelerator utilizes histogram of oriented gradients (HOG) features in combination with a support vector machine (SVM) classifier. The proposed HOG for raw images (HOGR) skips the image signal processors which consume a significant amount of power (about 2.5X of an accelerator). The proposed architecture temporally partitions the algorithms and time multiplexes the logic such that the accelerator works on the same frequency with the image sensor using affordable resources. The prototype is verified using 1080p (1920x 1080) and VGA (640x 480) raw videos on Altera Arria10 and Cyclone IV field programmable gate array (FPGA) platforms. The accelerator can process 1080p raw videos with a 12-scale pyramid at 60 frames per second (FPS) under the pixel frequency of corresponding image sensor (148.5 MHz), consuming 510 Kbit block memory. To the best knowledge of the authors, this is the first end-to-end HOG+SVM accelerator that takes raw Bayer images as input, skips the ISP pipeline for resource optimization from the system perspective, and is synchronized with the image sensor.

Achieving high rendering quality in the visualization of large particle data, for example from large-scale molecular dynamics simulations, requires a significant amount of sub-pixel super-sampling, due to very high numbers of particles per pixel. Although it is impossible to super-sample all particles of large-scale data at interactive rates, efficient occlusion culling can decouple the overall data size from a high effective sampling rate of visible particles. However, while the latter is essential for domain scientists to be able to see important data features, performing occlusion culling by sampling or sorting the data is usually slow or error-prone due to visibility estimates of insufficient quality. We present a novel probabilistic culling architecture for super-sampled high-quality rendering of large particle data. Occlusion is dynamically determined at the sub-pixel level, without explicit visibility sorting or data simplification. We introduce confidence maps to probabilistically estimate confidence in the visibility data gathered so far. This enables progressive, confidence-based culling, helping to avoid wrong visibility decisions. In this way, we determine particle visibility with high accuracy, although only a small part of the data set is sampled. This enables extensive super-sampling of (partially) visible particles for high rendering quality, at a fraction of the cost of sampling all particles. For real-time performance with millions of particles, we exploit novel features of recent GPU architectures to group particles into two hierarchy levels, combining fine-grained culling with high frame rates.

In this paper, we present a real-world case study on deploying a face recognition application, using MTCNN detector and FaceNet recognizer. We report the challenges faced to decide on the best deployment strategy. We propose three inference architectures for the deployment, including cloud-based, edge-based, and hybrid. Furthermore, we evaluate the performance of face recognition inference on different cloud-based and edge-based GPU platforms. We consider different types of Jetson boards for the edge, and various GPUs for the cloud. We also investigate the effect of deep learning model optimization using TensorRT and TFLite compared to a standard Tensorflow GPU model, and the effect of input resolution. We provide a benchmarking study for all these devices in terms of frame per second, execution times, energy and memory usages. After conducting a total of 294 experiments, the results demonstrate that the TensorRT optimization provides the fastest execution on all cloud and edge devices, at the expense of a significantly larger energy consumption (up to +40% and +35% for edge and cloud devices respectively, compared to Tensorflow). Whereas TFLite is the most efficient framework in terms of memory and power consumption, while providing significantly less (-4% to -62%) processing acceleration than TensorRT.

The semi-global matching (SGM) algorithm in stereo vision is a well-known depth-estimation method since it can generate dense and robust disparity maps. However, the real-time processing and low power dissipation, the specifications of the Internet-of-Thing (IoT) applications, are challenging for their computational complexity. In this paper, we propose a hardware-oriented SGM algorithm with pixel-level pipeline and region-optimized cost aggregation for high-speed processing and low hardware-resource usage. Firstly, the matching costs in a region are integrated with an optimization strategy to significantly reduce memory usage and improve the processing speed of the cost aggregation. Then, a two-layer parallel two-stage pipeline (TPTP) architecture, which enables pixel-level processing, is designed to calculate two directions (0° and 135°) aggregation to further solve the crucial computational bottleneck of the SGM algorithm. Finally, the architecture is demonstrated on a low-cost XILINX Spartan-7 device and an advanced Stratix-V FPGA device for VGA (640x 480) depth estimation. The experimental results show that the proposed architecture with compact hardware architecture also ensures accuracy. The pixel-level pipeline architecture enables a processing speed of 355 frames per second (fps) at 109MHz on the Spartan-7 FPGA device and 508 fps at 156MHz on the Stratix-V FPGA. Besides, the coprocessor respectively achieves an energy efficiency of 4.74 nJ/pixel with a power dissipation of 517mW and 4.29nJ/pixel with a power dissipation of 669mW on these two FPGAs.

This paper first provides a new Viewport-based OmniDirectional Video Quality Assessment (VOD-VQA) database, which includes eighteen salient viewport videos extracted from the original OmniDirectional Videos (ODVs), and corresponding 774 impaired samples generated by compressing the raw viewports using a variety of combinations of its Spatial (frame size s), Temporal (frame rate t), and Amplitude (quantization stepsize q) Resolutions (STAR). Total 160 subjects have assessed the processed viewport videos rendered on the head mounted display (HMD) when they stabilize their fixations. We then have formulated an analytical model to connect the perceptual quality of a compressed viewport video with its STAR variables, noted as the QVPSTAR index. All four model parameters can be predicted using linearly weighted content features, making the proposed metric generalized to various contents. This model correlates well with the mean opinion scores (MOSs) collected for processed viewport videos, having both the Pearson Correlation Coefficient and Spearman’s Rank Correlation Coefficient (SRCC) at 0.95 according to an independent validation test, yielding the state-of-the-art performance in comparison to those popular objective metrics (e.g., WS-PSNR [1], WMS-SSIM [2], VMAF [3], FSIM [4], and VSI [5]). Furthermore, this viewport-based quality index QVPSTAR is extended to infer the overall ODV quality, a.k.a., QODVSTAR, by linearly weighing the saliency-aggregated qualities of salient viewports and the quality of quick-scanning (or non-salient) area. Experiments have shown that inferred QODVSTAR can accurately predict the MOS with competitive performance to the state-of-the-art algorithm using another four independent and third-party ODV assessment datasets. All related materials are made publicly accessible at https://vision.nju.edu.cn/20/86/c29466a467078/page.htm for reproducible research.

Most previous studies focus on using complex deep neural networks to learn diverse features of massive synthetic data. In more realistic situations with a limited number of data pairs, complex networks not only have higher computational complexity, increasing training time and reducing inference speed, but also tend to over-fit a small amount of training data, thus having weak generalization capability. To address the aforementioned problem, we propose a lightweight architecture for real-time velocity inversion in realistic situations, the bilateral inversion network (BiInNet). BiInNet uses lightweight ResNet18, ShuffleNetV2, and modified MobileNetV2 as backbones, taking into account the inversion accuracy and inference speed. To reduce the redundant information in common shot gathers and focus on graphical property features which are strongly correlated with velocity, the intermediate results of velocity analysis, semblances, and interval velocity models are prepared as data pairs. Numerical experiments show that BiInNet can infer interval velocity models in real-time, with frames per second (FPS) up to 76.90 when ResNet18 is used as the backbone. Moreover, BiInNet achieves the best inversion accuracy on more realistic fold models, fault models, salt models, and noisy dataset (NFOMD) when adopting ShuffleNetV2 as the backbone, which illustrates that BiInNet can be applied to velocity inversion tasks of different geological structures and is robust to noise. Adopting transfer learning to fine-tune pretrained model, BiInNet is effectively applicable to velocity reversal models and field data, which further demonstrates the reliability of the proposed method and provides a practical velocity inversion scheme when the field data pairs are insufficient.

Discriminative correlation filters (DCF)-based trackers have been increasingly applied to visual tracking due to their high precision while running at high frame rates. However, most recent DCF-based methods solely concentrate on learning the correlation filter with spatial information and thus do not have sufficient descriptive power to discriminate the target from the background in the complex circumstances, such as full occlusion (OCC) and rapid target variation. In this article, we introduce a novel tracking framework that exploits the relationship between the target and its spatiotemporal context to improve tracking accuracy and robustness. Especially, we present our spatiotemporal context model in a hierarchical way, where each layer of the context pyramid is a spatial correlation filter learned from different temporal instances. For gaining an accurate spatiotemporal model, we propose an optimization fusion approach that can adaptively and efficiently learn the effect of each hierarchical layer and exploit these multiple temporal levels of correlation filters for visual tracking. Moreover, an adaptive model update strategy for correlation filters is introduced into the framework to dynamically select proper hierarchical layers, which boosts the temporal diversity of the target appearance, while radically reduces the number of model parameters and guarantees the real-time performance of the tracking method. The experimental results show that, with conventional handcrafted features, our tracker achieves the best success rates among available state-of-the-art trackers with handcrafted features, and provides state-of-the-art performance comparable to those of deep-learning-based trackers on OTB-2013, OTB-2015, VOT-2016, and UAV-20L benchmarks but runs significantly faster than deep trackers.

It has long been recognized that synthetic aperture radar (SAR) images suffer in many applications from the speckle noise. Video SAR has a high frame rate of imaging and contains redundant information among frames. The temporal redundancy in video SAR images has been found useful for suppressing the speckle noise. However, the motion and the local differences between frames make it difficult to employ the temporal redundancy to suppress speckle noise. This article presents a video SAR image despeckling framework based on a new unsupervised training strategy referred to as DualNoise2Noise. This developed framework consists of a registration network and a denoising network. The registration network first compensates for the motion between two adjacent frames in video SAR in real time. After image registration, two adjacent frames with random speckle noise can be considered as the observations of the same region with local differences. The denoising network adopts the DualNoise2Noise training strategy to suppress speckle noise by using the temporal redundancy and to remove the negative impact caused by the local differences. The proposed approach has been used to process the real video SAR data, and the experimental results are convincing.

Modern autonomous vehicles are required to perform various visual perception tasks for scene construction and motion decision. The multiobject tracking and instance segmentation (MOTS) are the main tasks since they directly influence the steering and braking of the car. Implementing both tasks using a multitask learning neural network presents significant challenges in performance and complexity. Current work on MOTS devotes to improve the precision of the network with a two-stage tracking by detection model, which is difficult to satisfy the real-time requirement of autonomous vehicles. In this article, a real-time multitask network named YolTrack based on one-stage instance segmentation model is proposed to perform the MOTS task, achieving an inference speed of 29.5 frames per second (fps) with slight accuracy and precision drop. The YolTrack uses ShuffleNet V2 with feature pyramid network (FPN) as a backbone, from which two decoders are extended to generate instance segments and embedding vectors. Segmentation masks are used to improve the tracking performance by performing logic AND operation with feature maps, proving that foreground segmentation plays an important role in object tracking. The different scales of multiple tasks are balanced by the optimized geometric mean loss during the training phase. Experimental results on the KITTI MOTS data set show that YolTrack outperforms other state-of-the-art MOTS architectures in real-time aspect and is appropriate for deployment in autonomous vehicles.

Accurate identification of table grapes is crucial to the harvesting process of the grape picking robot. This paper proposes an efficient grape detection model, YOLO-Grape, to solve the problem of unrecognition or decreased recognition accuracy caused by the complicated growth environment, shadows of branches and leaves, and overlapping grapes. To improve the network recognition accuracy, a down-sampling fusion structure is integrated into the network, and the Mish activation function is used. Meanwhile, an attention mechanism is added to the network, and the non-maximum suppression (NMS) function is replaced with the soft non-maximum suppression (Soft-NMS) function, thereby reducing the missing of predicted boxes due to overlapping grapes. Besides, the depthwise separable convolution is introduced to improve the detection speed of the network. In addition, transfer learning is used in the training process to improve the detection accuracy and generalization ability of the model. On the test data set of 700 grape images, the experimental results show that YOLO-Grape achieves an F1-score of 90.47%, a mAP of 91.08% and a detection speed of 81 frames per second. Compared with Faster-RCNN(Resnet50), SSD300, YOLOv4, and YOLOv4-tiny, the mAP of the YOLO-Grape model is increased by 1.67%, 2.28%, 0.84%, and 6.69%, respectively. The average recognition speeds of the YOLO-Grape model were 31.15, 3.38 and 6.45 times of Faster-RCNN(Resnet50), SSD300, and YOLOv4 respectively. Through four sets of comparative experiments, it is found that the proposed YOLO-Grape model achieves high recognition accuracy for occluded grapes, meeting the requirements of grape picking robots for real-time detection of multiple varieties of table grapes in complex situations.

In recent times, the service providers and the operators are facing the problems like visual quality degradations and frequent interruptions during video streaming, due to higher dynamism of network conditions. The video parameters like occupancy, playback quality, underflow/overflow buffer, and rate switching frequency/amplitude affect the quality of user’s experience. In the present scenario, numerous adaptive streaming protocols are developed to provide continuous service to the user’s under complex network condition with heterogeneous devices. In this research article, linear discriminant regression technique is proposed based on cognitive computing in order to predict the user’s quality of experience over software-defined network. By varying the network and the objective parameters, initially the mean opinion score is collected from the users. Further, a new architecture is developed on the basis of linear discriminant regression technique that uses the mean opinion score under different network conditions to predict the expected mean opinion score. The simulation results showed that the linear discriminant regression technique achieved better video quality compared to linear regression technique in terms of peak signal-to-noise ratio, structural similarity index, and video quality metric by varying the video resolutions, bit-rates, and frame rates.

Modern image sensor devices regularly aim for low-noise and high frame-rate performances, among other key characteristics. However, not all sensor applications require both low-noise and fast output image acquisition as part of the same sensor. In slow object motion, at low-light illumination levels (such as for night vision), low-noise image performance is necessary but only at a reasonable sensor frame rate. This paper presents a theoretical analysis of the most common pixel readout methods in modern imagers, at two distinct operation modes. One mode is the classical active pixel sensor circuit and the other is a more recent operation mode based on a pulsed biasing scheme introduced for specific application cases, in which the sensor speed is not of much concern. From this perspective, this paper aims to compare the theoretical output thermal noise powers of both kinds of readouts in order to identify which readout method is the read option most suitable for low-light vision without the need to employ advanced and complex techniques to obtain a low-noise imager. In addition to the theoretical results, simulation results are presented as well, as a means to validate the theoretical work.

Condition assessment is an essential process to comprehend the condition of the water pipelines and facilitate the maintenance as well as the renewal plans. Nowadays, varied in-pipe inspection platforms equipped with closed-circuit cameras are employed to capture the internal condition of the water pipelines. However, the automated platform often faces the challenge to negotiate with the installed valves during the inspection. To ensure continuous inspection, the platform needs identify the valve automatically and activate the control mechanism to pass through it. Thus, the valves need to be detected to facilitate the negotiation and ensure that the control mechanism can take an action in time. This paper focuses on real-time valve detection using Jetson TX2™ and a lightweight algorithm, namely YOLOv3-tiny. The performance of the implementation is compared with state-of-the-art real-time detection models. The experimental results demonstrate that YOLOv3-tiny has a high detection speed in frame per second for valve detection and outperforms the state-of-the-art real-time algorithms. Hence, the deployment YOLOv3-tiny into the embedded system will aid the automated platform to accomplish the uninterrupted inspection and enhance the capability for the condition assessment of the water pipelines.

Two-way dynamic focusing in synthetic transmit aperture (STA) beamforming can benefit high-quality ultrasound imaging with higher lateral spatial resolution and contrast resolution. However, STA requires the complete dataset for beamforming in a relatively low frame rate and transmit power. This paper proposes a deep-learning architecture to achieve high frame rate STA imaging with two-way dynamic focusing. The network consists of an encoder and a joint decoder. The encoder trains a set of binary weights as the apodizations of the high-frame-rate plane wave transmissions. In this respect, we term our network ApodNet . The decoder can recover the complete dataset from the acquired channel data to achieve dynamic transmit focusing. We evaluate the proposed method by simulations at different levels of noise and in-vivo experiments on the human biceps brachii and common carotid artery. The experimental results demonstrate that ApodNet provides a promising strategy for high frame rate STA imaging, obtaining comparable lateral resolution and contrast resolution with four-times higher frame rate than conventional STA imaging in the in-vivo experiments. Particularly, ApodNet improves contrast resolution of the hypoechoic targets with much shorter computational time when compared with other high-frame-rate methods in both simulations and in-vivo experiments.

The raindrops on the glass will affect driving safety, such as rear-view camera, outside mirror and windshield, etc. This article proposed a robust raindrop detection using deep learning on embedded platform with AI accelerator for real-time implementation. A training model is established through a convolution neural network (CNN)-like architecture to classify the images by the vehicle camera into three classes: no rain, heavy rain, and light rain. The classification results are used to control the speed of the motor to implement an automatic wiper control system. The training model, ResNet, is used to classify the image with good tradeoff between the computational cost and accuracy. For real-time application, the camera module on the Google Coral Dev board on embedded system platform is used to test the video stream and to estimate the performance of this system. Results show that the recognition accuracy reaches 95%, and the processing speed can achieve 20 frames per second (fps) on the embedded system.

Background The correct identification of pills is very important to ensure the safe administration of drugs to patients. Here, we use three current mainstream object detection models, namely RetinaNet, Single Shot Multi-Box Detector (SSD), and You Only Look Once v3(YOLO v3), to identify pills and compare the associated performance. Methods In this paper, we introduce the basic principles of three object detection models. We trained each algorithm on a pill image dataset and analyzed the performance of the three models to determine the best pill recognition model. The models were then used to detect difficult samples and we compared the results. Results The mean average precision (MAP) of RetinaNet reached 82.89%, but the frames per second (FPS) is only one third of YOLO v3, which makes it difficult to achieve real-time performance. SSD does not perform as well on the indicators of MAP and FPS. Although the MAP of YOLO v3 is slightly lower than the others (80.69%), it has a significant advantage in terms of detection speed. YOLO v3 also performed better when tasked with hard sample detection, and therefore the model is more suitable for deployment in hospital equipment. Conclusion Our study reveals that object detection can be applied for real-time pill identification in a hospital pharmacy, and YOLO v3 exhibits an advantage in detection speed while maintaining a satisfactory MAP.

Graphics processing units (GPUs) power today’s fastest supercomputers, are the dominant platform for deep learning, and provide the intelligence for devices ranging from self-driving cars to robots and smart cameras. They also generate compelling photorealistic images at real-time frame rates. GPUs have evolved by adding features to support new use cases. NVIDIA’s GeForce 256, the first GPU, was a dedicated processor for real-time graphics, an application that demands large amounts of floating-point arithmetic for vertex and fragment shading computations and high memory bandwidth. As real-time graphics advanced, GPUs became programmable. The combination of programmability and floating-point performance made GPUs attractive for running scientific applications. Scientists found ways to use early programmable GPUs by casting their calculations as vertex and fragment shaders. GPUs evolved to meet the needs of scientific users by adding hardware for simpler programming, double-precision floating-point arithmetic, and resilience.

Drain blockage is a crucial problem in the urban environment. It heavily affects the ecosystem and human health. Hence, routine drain inspection is essential for urban environment. Manual drain inspection is a tedious task and prone to accidents and water-borne diseases. This work presents a drain inspection framework using convolutional neural network (CNN) based object detection algorithm and in house developed reconfigurable teleoperated robot called ‘Raptor’. The CNN based object detection model was trained using a transfer learning scheme with our custom drain-blocking objects data-set. The efficiency of the trained CNN algorithm and drain inspection robot Raptor was evaluated through various real-time drain inspection field trial. The experimental results indicate that our trained object detection algorithm has detect and classified the drain blocking objects with 91.42% accuracy for both offline and online test images and is able to process 18 frames per second (FPS). Further, the maneuverability of the robot was evaluated from various open and closed drain environment. The field trial results ensure that the robot maneuverability was stable, and its mapping and localization is also accurate in a complex drain environment.

Aggregate is an important component of asphalt mixtures, and its shape has a significant influence on the road quality. In this study, a single industrial camera was used to collect images of aggregate particles during falling; then, their morphologies obtained in multiple views were analyzed. Using the equivalent geometric model, four shape characterization parameters—area variety factor, minor diameter variety factor, maximum elongation factor, and Strip-Block area variety factor—were proposed to compose the multi-view shape feature. On this basis, a general regression neural network was adopted to realize the classification of aggregate particles. The results show that the aggregate classification is slightly different when using different equivalent geometric models, while the aggregate shape can be effectively classified. The accuracy of aggregate classification can be improved by fusing parameters from different equivalent models using principal component analysis; another way is through increasing the frame rate of image collection that may increase the number of views. In general, the findings indicate that the proposed detection method can be applied to actual road engineering, which is of great significance to guarantee pavement quality.

This article presents new developments in a multiframe blind deconvolution algorithm for imaging low earth orbiting objects during flyover. The foundational aspects of the algorithm rely on the constrained maximum likelihood (ML) formulation in the presence of Poisson noise, previously developed in Schulz et al. (2018). The new algorithm achieves real time evaluation capability at over two frames per second, which is allowed by two novel aspects. First, the prototype algorithms are transferred to highly parallelized computations on graphical processing units (GPUs) with CUDA implementation that reduce the computational time of a single iteration by a factor of up to 100. Second, new numerical optimization strategies are developed and demonstrated to accelerate the convergence of the algorithm by a factor of 5 to 10. Several other new capabilities are also demonstrated in this article. We derive and implement a modified variation of the algorithm that achieves subpixel resolution, which is shown effective on real and simulated data. Finally, a new post-processing visual enhancement technique is proposed with several examples, which in part helps deal with the dynamic range degradation due to glint.

After over 30 years of advances, multiphoton microscopy (MPM) is now instrumental in a wide range of in vivo biological imaging applications. However, it has, until recently, remained not achievable or affordable to meet the unmet need for fast monitoring of biological dynamics and large-scale examination of biological heterogeneity. Only within the past few years have new strategies emerged to empower MPM at a speed that was once inconceivable, notably at kilohertz two-dimensional (2D) frame rate, and 3D rate or beyond. This Review highlights the latest high-speed innovations and discusses the potential of their synergism with other advanced, but less speed-centric MPM toolboxes. Recognizing these prospects and challenges could inspire new approaches for reprioritizing imaging speed in future MPM developments. This Review summarizes the latest state-of-the-art technologies for high-speed multiphoton (fluorescence) microscopy, especially at kilohertz 2D frame rate, and 3D video rate or beyond—a speed regime that was generally inconceivable until very recently, as well as the prospects and challenges of these emerging technologies.

Camshift algorithm tracking is susceptible to interference when a tracking object is occluded or when its hue is similar to the background. An improved Camshift object-tracking algorithm combining AKAZE (Accelerated-KAZE) feature matching and Kalman filtering is proposed. First, the video channel is converted for processing. Second, AKAZE is used to match the object feature points and Kalman filtering is used to predict the next position. Then different scenes are judged by the threshold and the Camshift and Kalman tracking algorithms are used for object tracking, respectively. Finally, the improved Camshift algorithm is used to test the moving object in a variety of situations and compared with the traditional Camshift algorithm and the Kalman filter improved Camshift algorithm. Experimental results show that the improved joint tracking algorithm can continue tracking under full occlusion. The effective frame rate of recognition is increased by about 20%, and the single-frame image processing time is less than 35 ms, which can meet the real-time tracking requirements.

Scanning imaging can be used to obtain wide field‐of‐view, high‐resolution remote‐sensing images with a telescope lens. However, the motion between the image and the sensor causes blurring during scanning, which reduces the image quality. In this paper, a motion compensation method based on multi‐actuator control with modal switching is proposed. In this method, the tracking controller realizes scanning and compensation; the cooperative controller reduces the relative motion that causes motion blur, and modal switching solves the travel limitation of the linear actuator. Further, to improve the control performance, an adaptive sliding mode controller is proposed and combined with a disturbance observer. The stability theorems for multi‐actuator and modal switching are studied. Experimental results show that the accuracy, robustness, and frame rate of the proposed method improved compared with those of traditional methods. Compared with the non‐cooperative structure using a PD controller, the root mean square error is reduced by 76.61%.

Cataract causes more than half of all blindness worldwide. The most effective treatment is surgery, where cataract is often replaced by intraocular lens (IOL). Beyond saving vision, toric IOL implantation is becoming increasingly popular to correct corneal astigmatism. It is important to precisely position and align the axis of IOL during surgery to achieve optimal post-operative astigmatism correction. Comparing with conventional manual marking, automated markerless IOL alignment can be faster, more accurate and non-invasive. Here we propose a framework for computer-assisted intraoperative IOL positioning and alignment based on detection and tracking. Firstly, the iris boundary was segmented and the eye center was determined. A statistical sampling method was developed to segment iris and generate training labels, and both conventional algorithms and deep convolutional neural network (CNN) methods were evaluated. Then, regions of interests (ROIs) containing high density of scleral capillaries were used for tracking eye rotations. Both correlation filter and CNN methods were evaluated for tracking. Cumulative errors during long-term tracking were corrected using a reference image. Validation studies against manual labeling using 7 clinical cataract surgical videos demonstrated that the proposed algorithm achieved an average position error around 0.2 mm, an axis alignment error of $^{\circ}$ , and a frame rate of > 25 FPS, and can be potentially used intraoperatively for markerless IOL positioning and alignment during cataract surgery.

The ability to detect and segment moving objects in a scene is essential for building consistent maps, making future state predictions, avoiding collisions, and planning. In this paper, we address the problem of moving object segmentation from 3D LiDAR scans. We propose a novel approach that pushes the current state of the art in LiDAR-only moving object segmentation forward to provide relevant information for autonomous robots and other vehicles. Instead of segmenting the point cloud semantically, i.e., predicting the semantic classes such as vehicles, pedestrians, roads, etc., our approach accurately segments the scene into moving and static objects, i.e., also distinguishing between moving cars vs. parked cars. Our proposed approach exploits sequential range images from a rotating 3D LiDAR sensor as an intermediate representation combined with a convolutional neural network and runs faster than the frame rate of the sensor. We compare our approach to several other state-of-the-art methods showing superior segmentation quality in urban environments. Additionally, we created a new benchmark for LiDAR-based moving object segmentation based on SemanticKITTI. We published it to allow other researchers to compare their approaches transparently and we furthermore published our code., Accepted by RA-L with IROS 2021

When using the current method to compress the low frame rate video animation video, there is no frame rate compensation for the video image, which cannot eliminate the artifacts generated in the compression process, resulting in low definition, poor quality, and low compression efficiency of the compressed low frame rate video animation video. In the context of new media, the linear function model is introduced to study the frame rate video animation video compression algorithm. In this paper, an adaptive detachable convolutional network is used to estimate the offset of low frame rate video animation using local convolution. According to the estimation results, the video frames are compensated to eliminate the artifacts of low frame rate video animation. After the frame rate compensation, the low frame rate video animation video is divided into blocks, the CS value of the image block is measured, the linear estimation of the image block is carried out by using the linear function model, and the compression of the low frame rate video animation video is completed according to the best linear estimation result. The experimental results show that the low frame rate video and animation video compressed by the proposed algorithm have high definition, high compression quality under different compression ratios, and high compression efficiency under different compression ratios.