Search

Network Intrusion detection systems are essential for the protection of advanced communication networks. Originally, these systems were hard-coded to identify specific signatures, patterns and rule violations; now artificial intelligence and machine learning algorithms provide promising alternatives. However, in the literature, various outdated datasets as well as a plethora of different evaluation metrics are used to prove algorithm efficacy. To enable a global comparison, this study compiles algorithms for different configurations to create common ground and proposes two new evaluation metrics. These metrics, the detection score and the identification score, together reliably present the performance of a network intrusion detection system to allow for practical comparison on a large scale. Additionally, we present a workflow to process raw packet flows into input features for machine learning. This framework quickly implements different algorithms for the various datasets and allows systematic performance comparison between those algorithms. Our experimental results, matching and surpassing the state-of-the-art, indicate the potential of this approach. As raw traffic input features are much easier and cheaper to extract when compared to traditional features, they show promise for application in real-time deep learning-based systems. ispartof: Ieee Access vol:9 pages:63995-64015 status: published

Motivated by an increasing interest from startups in embedded Artificial Intelligence (AI) and by their limited expertise, the AITIA Project targets the development of embedded AI techniques for industrial applications. This extended abstract presents the motivation and the solutions being developed towards four use cases: smart sensors, network intrusion detection, driver-assistance systems, and Industry 4.0.

Traditionally, network intrusion detection systems identify attacks based on signatures, rules, events or anomaly detection. More and more research investigates the application of deep learning techniques for this purpose. Deep learning significantly increases detection performance, and can abolish the need for expert knowledge-intensive feature extraction. The use of deep learning for network intrusion detection also has a major disadvantage, however, as it is not deployed yet in real-time implementations. In this paper, we propose two approaches that facilitate the transition towards functional real-time implementations: (1) the use of flow buckets to collect raw traffic-based features, and (2) the acceleration of neural network architectures for intrusion detection using the Xilinx FINN toolchain for FPGAs. We obtain promising results that show our flow bucket approach does not deteriorate detection performance when compared to traditional approaches, and we lay a foundation to further build on with respect to accelerating deep learning algorithms for network intrusion detection on FPGA.

With the evolvement of various drone platforms, the constraints of such embedded systems are getting more and more strict and specific. As the low-level technology, correct behaviours and decent performance of an RTOS play a vital role in guaranteeing such constraints. This paper studies the real-time performance and behaviour of two popular RTOSs used in drones, presenting both quantitative and qualitative results. We also exploited the source code to locate the origin of possible misbehaviours. By analysing the results, we draw several conclusions on the comparison of NuttX and ChibiOS, which can be useful for drone developers and users. NuttX has a vital defect in the implementation of priority inheritance. And in most tests, ChibiOS beats NuttX, therefore, it is the winner in this comparison.

Automating sheet steel visual inspection can improve quality and reduce costs during its production. While many manufacturers still rely on manual or traditional inspection methods, deep learning-based approaches have proven their efficiency. In this paper, we go beyond the state-of-the-art in this domain by proposing a multi-task model that performs both pixel-based defect segmentation and severity estimation of the defects in one two-branch network. Additionally, we show how incorporation of the production process parameters improves the model’s performance. After manually constructing a real-life industrial dataset, we first implemented and trained two single-task models performing the defect segmentation and severity estimation tasks separately. Next, we compared this to a multi-task model that simultaneously performs the two tasks at hand. By combining the tasks into one model, both segmentation tasks improved by 2.5% and 3% mIoU, respectively. In the next step, we extended the multi-task model using sensor fusion with process parameters. We demonstrate that the incorporation of the process parameters resulted in a further mIoU increase of 6.8% and 2.9% for the defect segmentation and severity estimation tasks, respectively.

Object detection models are usually trained and evaluated on highly complicated, challenging academic datasets, which results in deep networks requiring lots of computations. However, a lot of operational use-cases consist of more constrained situations: they have a limited number of classes to be detected, less intra-class variance, less lighting and background variance, constrained or even fixed camera viewpoints, etc. In these cases, we hypothesize that smaller networks could be used without deteriorating the accuracy. However, there are multiple reasons why this does not happen in practice. Firstly, overparameterized networks tend to learn better, and secondly, transfer learning is usually used to reduce the necessary amount of training data. In this paper, we investigate how much we can reduce the computational complexity of a standard object detection network in such constrained object detection problems. As a case study, we focus on a well-known single-shot object detector, YoloV2, and combine three different techniques to reduce the computational complexity of the model without reducing its accuracy on our target dataset. To investigate the influence of the problem complexity, we compare two datasets: a prototypical academic (Pascal VOC) and a real-life operational (LWIR person detection) dataset. The three optimization steps we exploited are: swapping all the convolutions for depth-wise separable convolutions, perform pruning and use weight quantization. The results of our case study indeed substantiate our hypothesis that the more constrained a problem is, the more the network can be optimized. On the constrained operational dataset, combining these optimization techniques allowed us to reduce the computational complexity with a factor of 349, as compared to only a factor 9.8 on the academic dataset. When running a benchmark on an Nvidia Jetson AGX Xavier, our fastest model runs more than 15 times faster than the original YoloV2 model, whilst increasing the accuracy by 5% Average Precision (AP). ispartof: JOURNAL OF IMAGING vol:7 issue:4 ispartof: location:Switzerland status: published

The assessment of gaze behaviour is essential for understanding the psychology of communication. Mobile eye-tracking glasses are useful to measure gaze behaviour during dynamic interactions. Eye-tracking data can be analysed by using manually annotated areas-of-interest. Computer vision algorithms may alternatively be used to reduce the amount of manual effort, but also the subjectivity and complexity of these analyses. Using additional re-identification (Re-ID) algorithms, different participants in the interaction can be distinguished. The aim of this study was to compare the results of manual annotation of mobile eye-tracking data with the results of a computer vision algorithm. We selected the first minute of seven randomly selected eye-tracking videos of consultations between physicians and patients in a Dutch Internal Medicine out-patient clinic. Three human annotators and a computer vision algorithm annotated mobile eye-tracking data, after which interrater reliability was assessed between the areas-of-interest annotated by the annotators and the computer vision algorithm. Additionally, we explored interrater reliability when using lengthy videos and different area-of-interest shapes. In total, we analysed more than 65 min of eye-tracking videos manually and with the algorithm. Overall, the absolute normalized difference between the manual and the algorithm annotations of face-gaze was less than 2%. Our results show high interrater agreements between human annotators and the algorithm with Cohen’s kappa ranging from 0.85 to 0.98. We conclude that computer vision algorithms produce comparable results to those of human annotators. Analyses by the algorithm are not subject to annotator fatigue or subjectivity and can therefore advance eye-tracking analyses.

Reality TV shows have gained popularity, motivating many production houses to bring new variants for us to watch. Compared to traditional TV shows, reality TV shows have spontaneous unscripted footage. Computer vision techniques could partially replace the manual labour needed to record and process this spontaneity. However, automated real-world video recording and editing is a challenging topic. In this paper, we propose a system that utilises state-of-the-art video and audio processing algorithms to, on the one hand, automatically steer cameras, replacing camera operators and on the other hand, detect all audiovisual action cues in the recorded video, to ease the job of the film editor. This publication has hence two main contributions. The first, automating the steering of multiple Pan-Tilt-Zoom PTZ cameras to take aesthetically pleasing medium shots of all the people present. These shots need to comply with the cinematographic rules and are based on the poses acquired by a pose detector. Secondly, when a huge amount of audio-visual data has been collected, it becomes labour intensive for a human editor retrieve the relevant fragments. As a second contribution, we combine state-of-the-art audio and video processing techniques for sound activity detection, action recognition, face recognition, and pose detection to decrease the required manual labour during and after recording. These techniques used during post-processing produce meta-data allowing for footage filtering, decreasing the search space. We extended our system further by producing timelines uniting generated meta-data, allowing the editor to have a quick overview. We evaluated our system on three in-the-wild reality TV recording sessions of 24 hours (× 8 cameras) each taken in real households.

The trend of increased use of lithium-ion batteries, challenges the cost-effectiveness and safety of manual battery separation during the end-of-life treatment of Waste Electric and Electronic Equipment (WEEE). Therefore, the need for novel techniques to separate and sort batteries from WEEE is increasingly important. For this reason, the presented research investigates the potential to facilitate the development of novel techniques for battery extraction and sorting by examining the technical feasibility of predicting the presence, location, and type of batteries inside electronic devices with a deep learning object detection network using X-Ray images of the internal structure of WEEE. To determine the required X-ray imaging parameters, 532 electronic devices were arbitrarily collected from a recycling facility. From each product, two X-Ray Transmission (XRT) images were captured at two different X-Ray source configurations. Results obtained with the limited dataset are promising, demonstrating a 91% true positive rate and only a 6% false positive rate for classifying battery-containing devices. Moreover, a precision of 89% and a recall of 81% are demonstrated for battery detection, and an average precision of 85% and an average recall of 76% are demonstrated to distinguish amongst the following six battery technologies: cylindrical nickel-metal hydride or nickel-cadmium, cylindrical alkaline, cylindrical zinc-carbon, cylindrical lithium-ion, pouch lithium-ion, and button cell batteries. These results demonstrate the potential of using deep learning object detection on XRT-generated images for both automated battery extraction and sorting, regardless of the condition or shape of the products. ispartof: Resources Conservation And Recycling vol:168 status: published

New achievements in Artificial Intelligence (AI) and Machine Learning (ML) are reported almost daily by the big companies. While those achievements are accomplished by fast and massive data processing techniques, the potential of embedded machine learning, where intelligent algorithms run in resource-constrained devices rather than in the cloud, is still not understood well by the majority of the industrial players and Small and Medium Entereprises (SMEs). Nevertheless, the potential embedded machine learning for processing high-performance algorithms without relying on expensive cloud solutions is perceived as very high. This potential has led to a broad demand by industry and SMEs for a practical and application-oriented feasibility study, which helps them to understand the potential benefits, but also the limitations of embedded AI. To address these needs, this paper presents the approach of the AITIA project, a consortium of four Universities which aims at developing and demonstrating best practices for embedded AI by means of four industrial case studies of high-relevance to the European industry and SMEs: sensors, security, automotive and industry 4.0.

© 2018 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved. The uprising trend of deep learning in computer vision and artificial intelligence can simply not be ignored. On the most diverse tasks, from recognition and detection to segmentation, deep learning is able to obtain state-of-the-art results, reaching top notch performance. In this paper we explore how deep convolutional neural networks dedicated to the task of object detection can improve our industrial-oriented object detection pipelines, using state-of-the-art open source deep learning frameworks, like Darknet. By using a deep learning architecture that integrates region proposals, classification and probability estimation in a single run, we aim at obtaining real-time performance. We focus on reducing the needed amount of training data drastically by exploring transfer learning, while still maintaining a high average precision. Furthermore we apply these algorithms to two industrially relevant applications, one being the detection of promotion boards in eye tracking data and the other detecting and recognizing packages of warehouse products for augmented advertisements. ispartof: pages:209-217 ispartof: VISIGRAPP 2018 - Proceedings of the 13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications vol:5 pages:209-217 ispartof: VISIGRAPP 2018 location:Madeira, Portugal date:27 Jan - 29 Jan 2018 status: published

Reality TV shows that follow people in their day-to-day lives are not a new concept. However, the traditional methods used in the industry require a lot of manual labour and need the presence of at least one physical camera man. Because of this, the subjects tend to behave differently when they are aware of being recorded. This paper will present an approach to follow people in their day-to-day lives, for long periods of time (months to years), while being as unobtrusive as possible. To do this, we use unmanned cinematographically-aware cameras hidden in people's houses. Our contribution in this paper is twofold: First, we create a system to limit the amount of recorded data by intelligently controlling a video switch matrix, in combination with a multi-channel recorder. Second, we create a virtual camera man by controlling a PTZ camera to automatically make cinematographically pleasing shots. Throughout this paper, we worked closely with a real camera crew. This enabled us to compare the results of our system to the work of trained professionals., 4 pages, 6 figures

Using hand gestures to answer a call or to control the radio while driving a car, is nowadays an established feature in more expensive cars. High resolution time-of-flight cameras and powerful embedded processors usually form the heart of these gesture recognition systems. This however comes with a price tag. We therefore investigate the possibility to design an algorithm that predicts hand gestures using a cheap low-resolution thermal camera with only 32×24 pixels, which is light-weight enough to run on a low-cost processor. We recorded a new dataset of over 1300 video clips for training and evaluation and propose a lightweight low-latency prediction algorithm. Our best model achieves 95.9% classification accuracy and 83% mAP detection accuracy while its processing pipeline has a latency of only one frame.

In this paper, we investigate the feasibility of automatic small object detection, such as vehicles and vessels, in satellite imagery with a spatial resolution between 0.3 and 0.5 m. The main challenges of this task are the small objects, as well as the spread in object sizes, with objects ranging from 5 to a few hundred pixels in length. We first annotated 1500 km2, making sure to have equal amounts of land and water data. On top of this dataset we trained and evaluated four different single-shot object detection networks: YOLOV2, YOLOV3, D-YOLO and YOLT, adjusting the many hyperparameters to achieve maximal accuracy. We performed various experiments to better understand the performance and differences between the models. The best performing model, D-YOLO, reached an average precision of 60% for vehicles and 66% for vessels and can process an image of around 1 Gpx in 14 s. We conclude that these models, if properly tuned, can thus indeed be used to help speed up the workflows of satellite data analysts and to create even bigger datasets, making it possible to train even better models in the future. ispartof: Remote Sensing vol:12 issue:7 status: Published online

This preface acts as an introduction to the special issue on Advanced Machine Vision. It highlights the goal of this special issue on Advanced Machine Vision as well as discusses the reviewing process. On top of that, it highlights the selected submissions and describes them briefly.

To answer the ever-increasing demand for computational power, coarse-grain reconfigurable architectures (CGRAs) experience a revival. As they provide efficient, dedicated datapaths to match the dataflow graphs of a wide range of applications, CGRAs successfully manage high performance tasks. Yet, modern CGRAs access data through classical memory hierarchies or global scratchpad memories, failing to take advantage of data locality. This reveals memory bandwidth as a key bottleneck for these reconfigurable architectures. This letter introduces a heterogeneous 2-D CGRA which distributes data buffers across its computational grid. Together with a single-cycle data forwarding path, this allows the CGRA to better take advantage of data locality, in order to minimize data transfers. A 28-nm CMOS instantiation of this concept realizes the mapping and execution of a variety of compute intensive kernels, such as a real-time 512-point FFT or $5\times 5$ convolutional filter at high power efficiency. The CGRA demonstrates a peak energy efficiency of 584.9 GOPS/W during a 103-tap FIR filter, marking a $2.9\times $ improvement over state-of-the-art 2-D CGRAs.

In open-set face recognition - as opposed to closed-set face recognition - it is possible that the identity of a given query is not present in the gallery set. In that case, the identity of the query can only be correctly classified as "unknown" when the similarity with the gallery faces is below a threshold that was determined a priori. However, in many use-cases, the set of queries contains multiple instances of the same identity, whether or not this identity is represented in the gallery. Thus, the set of query faces lends itself to identity clustering that could yield representative instances for unknown identities. By augmenting the gallery with these instances, we can make an open-set face recognition problem more closed. In this paper, we show that this method of Cluster-Inferred Gallery Augmentation (CIGA) does indeed improve the quality of open-set face recognition. We evaluate the addition of CIGA for both a private dataset of images taken in a school context and the public LFW dataset, showing a significant improvement in both cases. Moreover, an implementation of the suggested approach along with our experiments are made publicly available on https://gitlab.com/florisdf/acpr2019. ispartof: pages:15-26 ispartof: Lecture Notes in Computer Science vol:12047 pages:15-26 ispartof: The 5th Asian Conference on Pattern Recognition (ACPR 2019) location:Auckland, New Zealand date:26 Nov - 29 Nov 2019 status: published

Shopping behaviour analysis through counting and tracking of people in shop-like environments offers valuable information for store operators and provides key insights in the stores layout (e.g.frequently visited spots). Instead of using extra staff for this, automated on-premise solutions are preferred. These automated systems should be cost-effective, preferably on lightweight embedded hardware, work in very challenging situations (e.g. handling occlusions) and preferably work real-time. We solve this challenge by implementing a real-time Tensor RT optimized YOLOv3-based pedestrian detector, on a Jetson TX2 hardware platform. By combining the detector with a sparse optical flow tracker we assign a unique ID to each customer and tackle the problem of loosing partially occluded customers. Our detector-tracker based solution achieves an average precision of 81.59% at a processing speed of 10 FPS. Besides valuable statistics, heat maps of frequently visited spots are extracted and used as an overlay on the video stream. ispartof: pages:541-553 ispartof: ACIVS: International Conference on Advanced Concepts for Intelligent Vision Systems vol:abs/2007.04942 pages:541-553 ispartof: Advanced Concepts for Intelligent Vision Systems location:Auckland, New-Zealand date:10 Feb - 14 Feb 2020 status: published

Training deep learning models on embedded devices is typically avoided since this requires more memory, computation and power over inference. In this work, we focus on lowering the amount of memory needed for storing all activations, which are required during the backward pass to compute the gradients. Instead, during the forward pass, static Synthetic Gradient Modules (SGMs) predict gradients for each layer. This allows training the model in a feed-forward manner without having to store all activations. We tested our method on a robot grasping scenario where a robot needs to learn to grasp new objects given only a single demonstration. By first training the SGMs in a meta-learning manner on a set of common objects, during fine-tuning, the SGMs provided the model with accurate gradients to successfully learn to grasp new objects. We have shown that our method has comparable results to using standard backpropagation.

Re-identification neural networks are widely used in numerous applications such as crowd control, crime investi- gations, safety systems and even in most smartphones to unlock the phone with a picture of the owner. These techniques are mostly used to re-identify faces or persons but in this paper we investigate the possibility to adapt these to also re-identify similar looking objects such as diamonds. Since polished diamonds are very similar to the naked eye, it is difficult to distinguish one diamond from another. We have indications that diamonds are sometimes switched by trained switchers with fake or less expensive stones, while they pretend to inspect the stone. A solution to this is diamond fingerprinting. We therefore propose a technique to generate a unique ID for each stone, which allows to re-identify the diamond solely based on an image of the gem. Since each diamond is assigned a unique ID it is even possible to keep track of the diamonds over time. This allows the seller to verify his stones before and after trading while switchers don’t stand a chance. For this task we trained and adapted a classification network optimized for both speed and accuracy. ispartof: pages:2020-2025 ispartof: 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA) pages:2020-2025 ispartof: 18th IEEE International Conference on Machine Learning and Applications - ICMLA 2019 location:Boca Raton, Florida, USA status: published

In this paper, we present a room occupancy sensing solution with unique properties: (i) It is based on an omnidirectional vision camera, capturing rich scene info over a wide angle, enabling to count the number of people in a room and even their position. (ii) Although it uses a camera-input, no privacy issues arise because its extremely low image resolution, rendering people unrecognisable. (iii) The neural network inference is running entirely on a low-cost processing platform embedded in the sensor, reducing the privacy risk even further. (iv) Limited manual data annotation is needed, because of the self-training scheme we propose. Such a smart room occupancy rate sensor can be used in e.g. meeting rooms and flex-desks. Indeed, by encouraging flex-desking, the required office space can be reduced significantly. In some cases, however, a flex-desk that has been reserved remains unoccupied without an update in the reservation system. A similar problem occurs with meeting rooms, which are often under-occupied. By optimising the occupancy rate a huge reduction in costs can be achieved. Therefore, in this paper, we develop such system which determines the number of people present in office flex-desks and meeting rooms. Using an omnidirectional camera mounted in the ceiling, combined with a person detector, the company can intelligently update the reservation system based on the measured occupancy. Next to the optimisation and embedded implementation of such a self-training omnidirectional people detection algorithm, in this work we propose a novel approach that combines spatial and temporal image data, improving performance of our system on extreme low-resolution images.

Quantization of neural networks has been one of the most popular techniques to compress models for embedded (IoT) hardware platforms with highly constrained latency, storage, memory-bandwidth, and energy specifications. Limiting the number of bits per weight and activation has been the main focus in the literature. To avoid major degradation of accuracy, common quantization methods introduce additional scale factors to adapt the quantized values to the diverse data ranges, present in full-precision (floating-point) neural networks. These scales are usually kept in high precision, requiring the target compute engine to support a few high-precision multiplications, which is not desirable due to the larger hardware cost. Little effort has yet been invested in trying to avoid high-precision multipliers altogether, especially in combination with 4 bit weights. This work proposes a new quantization scheme, based on power-of-two quantization scales, that works on-par compared to uniform per-channel quantization with full-precision 32 bit quantization scales when using only 4 bit weights. This is done through the addition of a low-precision lookup-table that translates stored 4 bit weights into nonuniformly distributed 8 bit weights for internal computation. All our quantized ImageNet CNNs achieved or even exceeded the Top-1 accuracy of their full-precision counterparts, with ResNet18 exceeding its full-precision model by 0.35%. Our MobileNetV2 model achieved state-of-the-art performance with only a slight drop in accuracy of 0.51%. ispartof: MDPI electronics, special issue "Intelligent IoT Systems" vol:10 issue:22 status: Published online

In a variety of research fields, including linguistics, human-computer interaction research, psychology, sociology and behavioral studies, there is a growing interest in the role of gestural behavior related to speech and other modalities. The analysis of multimodal communication requires high-quality video data and detailed annotation of the different semiotic resources under scrutiny. In the majority of cases, the annotation of hand position, hand motion, gesture type, etc. is done manually, which is a time-consuming enterprise requiring multiple annotators and substantial resources. In this paper we present a semi-automatic alternative, in which the focus lies on minimizing the manual workload while guaranteeing highly accurate annotations. First, we discuss our approach, which consists of several processing steps such as identifying the hands in images, calculating motion of the hands, segmenting the recording in gesture and non-gesture events, etc. Second, we validate our approach against two existing corpora in terms of accuracy and usefulness. The proposed approach is designed to provide annotations according to the McNeill (1992) gesture space and the output is compatible with annotation tools such as ELAN. Article available online: http://rdcu.be/ybFA ispartof: Language Resources and Evaluation vol:52 issue:2 pages:433-460 status: published

© Springer Nature Switzerland AG 2019. Drones are becoming a necessary and invaluable tool in many industries, as well as in emergency response situations. They can assist emergency-services in hazardous situations to get better situational awareness. This may lead to an improved rescue-coordination, increased personal safety for agents in the field and less personal, physical and financial damages as a result of a faster and better intervention. Photo-realistic 3D models generated from the drone video data, for example, can provide situational awareness as it is easier to understand the scene by visualizing it in 3D. The 3D model can be viewed from different perspectives and provides an instant overview of the situation. In contrast to SLAM which is fast but sparse, and SfM-MVS which is dense but slow, we present a pipeline that produces a dense photo-realistic 3D model of the event site in near real time by fusing oblique images with pre-recorded, publicly available LiDAR datasets. ispartof: pages:139-150 ispartof: Representations, Analysis and Recognition of Shape and Motion from Imaging Data vol:842 issue:Communications in Computer and Information Science pages:139-150 ispartof: RFMI location:Aussois, France date:17 Dec - 20 Dec 2017 status: published

Adversarial attacks on machine learning models have seen increasing interest in the past years. By making only subtle changes to the input of a convolutional neural network, the output of the network can be swayed to output a completely different result. The first attacks did this by changing pixel values of an input image slightly to fool a classifier to output the wrong class. Other approaches have tried to learn "patches" that can be applied to an object to fool detectors and classifiers. Some of these approaches have also shown that these attacks are feasible in the real-world, i.e. by modifying an object and filming it with a video camera. However, all of these approaches target classes that contain almost no intra-class variety (e.g. stop signs). The known structure of the object is then used to generate an adversarial patch on top of it. In this paper, we present an approach to generate adversarial patches to targets with lots of intra-class variety, namely persons. The goal is to generate a patch that is able successfully hide a person from a person detector. An attack that could for instance be used maliciously to circumvent surveillance systems, intruders can sneak around undetected by holding a small cardboard plate in front of their body aimed towards the surveillance camera. From our results we can see that our system is able significantly lower the accuracy of a person detector. Our approach also functions well in real-life scenarios where the patch is filmed by a camera. To the best of our knowledge we are the first to attempt this kind of attack on targets with a high level of intra-class variety like persons., Accepted for CVPR Workshop: CV-COPS 2019

The recent trend towards high frame rate, high resolution image processing necessitates a solution for the growing Von Neumann bottleneck. Bandwidth pressure is relieved in traditional image processors by tiling the image and storing pixel patches in a local scratchpad memory [1]. Yet, this only partially alleviates the memory bottleneck, due to repeated fetches for overlapping patches and the scratchpad access becoming the main bottleneck (Fig. 2.2.1). Streaming architectures [2] eliminate this overhead by streaming the image pixels into an array of processing elements and transferring intermediate data results to a next processing element instead of sending them back to memory. Crucial in the efficiency of such streaming image processing architectures is the line buffering strategy, required to merge intermediate results in the stream with previously computed results. Existing state-of-the-art architectures either have dedicated, application-specific line buffer instances which are inflexible towards various image processing workloads [1], or achieve flexibility by routing streams back to centralized scratchpad memories, again resulting in a memory bottleneck around the centralized buffers, and introducing the need for inefficient on-chip-networks between the processing elements [3]. This paper introduces the concept of flexible, instruction-programmable streaming processing with embedded configurable-delay FIFOs to enable line buffering for a wide range of computer vision algorithms on a single platform. Providing a deep reconfigurable pipeline, this design facilitates the mapping and execution of complex vision tasks, like dense optic flow in real-time (30fps VGA) at low power (10.7mW), marking A $5.8\times$ improvement over the state-of-the-art.

Drones have proven to be useful in many industry segments such as security and surveillance, where e.g. on-board real-time object tracking is a necessity for autonomous flying guards. Tracking and following suspicious objects is therefore required in real-time on limited hardware. With an object detector in the loop, low latency becomes extremely important. In this paper, we propose a solution to make object detection for UAVs both fast and super accurate. We propose a multi-dataset learning strategy yielding top eye-sky object detection accuracy. Our model generalizes well on unseen data and can cope with different flying heights, optically zoomed-in shots and different viewing angles. We apply optimization steps such that we achieve minimal latency on embedded on-board hardware by fusing layers, quantizing calculations to 16-bit floats and 8-bit integers, with negligible loss in accuracy. We validate on NVIDIA's Jetson TX2 and Jetson Xavier platforms where we achieve a speed-wise performance boost of more than 10x., Comment: Conference MVA2019, Source code: https://gitlab.com/EAVISE/jetnet

© Springer Nature Switzerland AG 2019. In this paper, we propose a framework for reconstructing a compact geometric model from point clouds of building interiors. Geometric reconstruction of indoor scenes is especially challenging due to clutter in the scene, such as furniture and cabinets. The clutter may (partially) hide the structural components of the interior. The proposed framework is able to cope with this clutter by using a hypothesizing and selection strategy, in which candidate faces are firstly generated by intersecting the extracted planar primitives. Secondly, an optimal subset of candidate faces is selected by optimizing a binary labeling problem. We formulate the selection problem as a continuous quadratic optimization problem, allowing us to incorporate a cost function specifically for indoor scenes. The obtained polygonal surface is not only 2-manifold but also oriented, meaning that the surface normals of each polygon are consistently oriented towards the interior. All adjacent and coplanar faces that were selected, are merged into a single face in order to obtain a final geometric model that is as compact as possible. This compact model of the room uses less memory and allows for faster processing when used in virtual reality applications. The method of Nan et al. was used as a starting point for our proposed framework. Finally, as opposed to other state-of-the-art interior modeling approaches, the only input that is required, is the point cloud itself. We do not rely on viewpoint information, nor do we assume constrained input environments with a 2.5D or, more restrictively, a Manhattan-world structure. To demonstrate the practical applicability of our proposed method, we performed various experiments on actual scan data of building interiors. ispartof: pages:459-472 ispartof: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) vol:11129 LNCS pages:459-472 ispartof: 15th European Conference on Computer Vision (ECCV) location:GERMANY, Munich date:8 Sep - 14 Sep 2018 status: published

In this paper we present an overview of the contributed work presented at the UAVision2018 ECCV workshop. This workshop focused on real-time image processing on-board of Unmanned Aerial Vehicles (UAVs). For such applications the computational complexity of state-of-the-art computer vision algorithms often conflicts with the need for real-time operation and the extreme resource limitations of the hardware. Apart from a summary of the accepted workshop papers, this work also aims to identify common challenges and concerns which were addressed by multiple authors during the workshop, and their proposed solutions.

In this work, we use a ceiling-mounted omni-directional camera to detect people in a room. This can be used as a sensor to measure the occupancy of meeting rooms and count the amount of flex-desk working spaces available. If these devices can be integrated in an embedded low-power sensor, it would form an ideal extension of automated room reservation systems in office environments. The main challenge we target here is ensuring the privacy of the people filmed. The approach we propose is going to extremely low image resolutions, such that it is impossible to recognise people or read potentially confidential documents. Therefore, we retrained a single-shot low-resolution person detection network with automatically generated ground truth. In this paper, we prove the functionality of this approach and explore how low we can go in resolution, to determine the optimal trade-off between recognition accuracy and privacy preservation. Because of the low resolution, the result is a lightweight network that can potentially be deployed on embedded hardware. Such embedded implementation enables the development of a decentralised smart camera which only outputs the required meta-data (i.e. the number of persons in the meeting room).

Several new algorithms for camera-based fall detection have been proposed, with the aim of reliably alerting caregivers about older persons' falls at home. These algorithms have been evaluated almost exclusively using brief segments of video data captured in artificial environments under optimal conditions and with falls simulated by actors. By contrast, we collected real-life video data recorded over several months at seven older persons' residences. Here, we report on our fall-detection algorithm based on the state-of-the-art, and we present an analysis of the real-life video data. The performance of our detection algorithm was compared with the performance of three previously reported algorithms that used a publicly available simulation data set. All four algorithms produced similar results when using the simulated data. However, the performance of our algorithm degraded drastically when evaluating falls in the real-life data. The false alarm rate was especially high, showing that some challenges still need to be met to make the system sufficiently robust to deploy in real-world situations. We conclude that using more realistic data sets that include longer video recordings and a broad range of activities are essential to reveal weaknesses in fall-detection algorithms.

In this paper we investigate the benefit of using depth information on top of normal RGB for camera-based pedestrian detection. Indeed, depth sensing is easily acquired using depth cameras such as a Kinect or stereo setups. We investigate the best way to perform this sensor fusion with a special focus on lightweight single-pass CNN architectures, enabling real-time processing on limited hardware. We implement different network architectures, each fusing depth at different layers of our network. Our experiments show that midway fusion performs the best, outperforming a regular RGB detector substantially in accuracy. Moreover, we prove that our fusion network is better at detecting individuals in a crowd, by demonstrating that it has both a better localization of pedestrians and is better at handling occluded persons. The resulting network is computationally efficient and achieves real-time performance on both desktop and embedded GPUs. ispartof: pages:524-529 ispartof: AVSS Workshop - MSS pages:524-529 ispartof: IEEE International Conference on Advanced Video and Signal-based Surveillance (AVSS) - MSS Workshop location:Auckland, New Zealand date:27 Nov - 30 Nov 2018 status: published

A traditional re-identification pipeline consists of a detection and re-identification step, i.e. a person detector is run on an input image to get a cutout which is then sent to a separate re-identification system. In this work we combine detection and re-identification into one single pass neural network. We propose an architecture that can do re-identification simultaneously with detection and classification. The effect of our modification has only a negligible impact on detection accuracy, and adds the calculation of re-identification vectors at virtually no cost. The resulting re-identification vector is strong enough to be used in speed sensitive applications which can benefit from an additional re-identification vector in addition to detection. We demonstrate this by using it as detection and re-identification input for a real-time person tracker. Moreover, unlike traditional detection + re-id pipelines our single-pass network’s computational cost is not dependent on the number of people in the image. ispartof: pages:49-54 ispartof: IEEE International Conference on Advanced Video and Signal-based Surveillance pages:49-54 ispartof: 15th IEEE International Conference on Advanced Video and Signal-based Surveillance location:Auckland, New Zealand date:27 Nov - 30 Nov 2018 status: published

© 2019, Springer Nature Singapore Pte Ltd. Mobile eye-tracking systems have been available for about a decade now and are becoming increasingly popular in different fields of application, including marketing, sociology, usability studies and linguistics. While the user-friendliness and ergonomics of the hardware are developing at a rapid pace, the software for the analysis of mobile eye-tracking data in some points still lacks robustness and functionality. With this paper, we investigate which state-of-the-art computer vision algorithms may be used to automate the post-analysis of mobile eye-tracking data. For the case study in this paper, we focus on mobile eye-tracker recordings made during human-human face-to-face interactions. We compared two recent publicly available frameworks (YOLOv2 and OpenPose) to relate the gaze location generated by the eye-tracker to the head and hands visible in the scene camera data. In this paper we will show that the use of this single pipeline framework provides robust results, which are both more accurate and faster than previous work in the field. Moreover, our approach does not rely on manual interventions during this process. ispartof: pages:499-509 ispartof: INFORMATION SCIENCE AND APPLICATIONS 2018, ICISA 2018 vol:514 pages:499-509 ispartof: iCatse International Conference on Information Science and Applications (ICISA) location:Hong Kong, HONG KONG date:25 Jun - 27 Jun 2018 status: published

Object detection using a boosted cascade of weak classifiers is a principle that has been used in a variety of applications, ranging from pedestrian detection to fruit counting in orchards, and this with a high average precision. In this work we prove that using both the approach suggest by Viola & Jones and the adapted approach by Dollár yields promising results on coconut tree detection in aerial images. However with the rise of robust deep learning algorithms for both detection and classification, and the significant drop in hardware costs, we wonder if it is feasible to apply deep learning to solve the task of fast and robust coconut tree localization in aerial imagery. We examine both pure classification-based and detection-based deep neural networks. By doing so we prove that deep learning is indeed a feasible alternative for coconut tree detection with a high average precision, although caution should be taken and one should keep attention to known issues with the selected architectures. ispartof: pages:230-241 ispartof: Proceedings of the 13th international joint conference on computer vision, imaging and computer graphics theory and applications vol:5 pages:230-241 ispartof: VISAPP location:Funchal, Madeira, Portugal date:27 Jan - 29 Jan 2018 status: published

In this paper we investigate the best way to automatically compose a photo album for an individual child from a large collection of photographs taken during a school year. For this, we efficiently combine state-of-the-art identification algorithms to select relevant photos, with an aesthetics estimation algorithm to only keep the best images. For the identification task, we achieved 86% precision for 86% recall on a real-life dataset containing lots of specific challenges of this application. Indeed, playing children appear in non-standard poses and facial expressions, can be dressed up or have their faces painted etc. In a top-1 sense, our system was able to correctly identify 89.2% of the faces in close-up. Apart from facial recognition, we discuss and evaluate extending the identification system with person re-identification. To select out the best-looking photos from the identified child photos to fill the album with, we propose an automatic assessment technique that takes into account the aesthetic photo quality as well as the emotions in the photos. Our experiments show that this measure correlates well with a manually labeled general appreciation score. ispartof: pages:321-332 ispartof: Advanced Concepts for Intelligent Vision Systems vol:11182 pages:321-332 ispartof: Advanced Concepts for Intelligent Vision Systems location:Poitiers date:24 Sep - 27 Sep 2018 status: published

In this paper, we demonstrate a system that automates the process of recording video lectures in classrooms. Through special hard-ware (lecturer and audience facing cameras and microphone arrays), we record multiple points of view of the lecture. Person detection and tracking, along with recognition of different human actions are used to digitally zoom in on the lecturer, and alternate focus between the lecturer and the slides or the blackboard. Audio sound source localization, along with face detection and tracking, is used to detect questions from the audience, to digitally zoom in on the member of the audience asking the question and to improve the quality of the sound recording. Finally, an automatic video editing system is used to naturally switch between the different video streams and to compose a compelling end product. We demonstrate the working system in two classrooms, over two 2-hour lectures, given by two lecturers. Hulens D., Aerts B., Chakravarty P., Diba A., Goedemé T., Roussel T., Zegers J., Tuytelaars T., Van Eycken L., Van Gool L., Van hamme H., Vennekens J., ''The CAMETRON lecture recording system: High quality video recording and editing with minimal human supervision'', Lecture notes in computer science, vol. 10704, pp. 518-530 (24th international conference on multiMedia modeling - MMM 2018, February 5-7, 2018, Bangkok, Thailand). ispartof: pages:518-530 ispartof: MultiMedia Modeling vol:10704 pages:518-530 ispartof: International conference on multimedia modeling - MMM 2018 location:Bangkok, Thailand date:5 Feb - 7 Feb 2018 status: published

The need for fast and robust pedestrian detection in various applications is growing every day. The addition of a thermal camera could help solving this problem resulting in higher detection accuracy in day but especially during night and bad weather conditions. Using convolutional neural networks, the leading technology in the field of object detection and classification, we propose a network architecture and training method for an accurate real-time multispectral pedestrian detector. We select a regression based single-pass network architecture with pre-trained weights from the Pascal VOC 2007 dataset. The network is then transfer-learned without changing the architecture but taking as input three image channels composed from information of the four available image channels (RGB+T). In our experiments we compare the results of different input-channel compositions and select a top performing combination. Our results show that this simple approach easily outperforms the improved ACF+T+THOG detector, coming close to the accuracy of other state-of-the-art multispectral CNNs with a log-average miss-rate of 31.2% measured on the KAIST multispectral benchmark dataset. Our main contribution: it runs as fast as 80FPS, estimated 10\(\times \) faster than the closest competitors.

In this paper, we demonstrate and evaluate a method to perform real-time object detection on-board a UAV using the state of the art YOLOv2 object detection algorithm running on an NVIDIA Jetson TX2, an GPU platform targeted at power constrained mobile applications that use neural networks under the hood. This, as a result of comparing several cutting edge object detection algorithms. Multiple evaluations we present provide insights that help choose the optimal object detection configuration given certain frame rate and detection accuracy requirements. We propose how this setup running on-board a UAV can be used to process a video feed during emergencies in real-time, and feed a decision support warning system using the generated detections. ispartof: pages:2110-2118 ispartof: The IEEE International Conference on Computer Vision (ICCV) vol:2018-January pages:2110-2118 ispartof: International Workshop on Computer Vision for UAVs (UAVision 2017) location:Venice, Italy date:28 Oct - 28 Oct 2017 status: published

In this paper we present a framework that is able to reliably and completely autonomously detect abnormal behavior in surveillance images. As input, we rely solely on a long-wave infrared (LWIR) image sensor. Our abnormal behavior detection pipeline consists of two consecutive stages. In a first stage, we perform efficient and fast pedestrian detection and tracking. In a second step, the detected paths are fed into a semi-supervised classifier that detects abnormal behavior. As test-case we recorded a unique real-life LWIR train station dataset -- which will be made publicly available -- containing natural occurrences of both normal and abnormal behavior. Our experiments indicate that our proposed framework achieves excellent accuracy results at real-time processing speeds. ispartof: Proceedings of the 14th IEEE International Conference on Advanced Video and Signal based Surveillance (AVSS) ispartof: The IEEE International Conference on Advanced Video and Signal based Surveillance (AVSS) location:Lecce, Italy date:29 Aug - 1 Sep 2017 status: published

Recent advances in computer vision have almost solved the problem of in the wild face detection, using complex techniques like convolutional neural networks. On the contrary many open source computer vision frameworks like OpenCV have not yet made the switch to these complex techniques and tend to depend on well established algorithms for face detection, like the cascade classification pipeline suggested by Viola and Jones. However the accuracy of these open source face detection models on public datasets like FDDB stays rather low, mainly due to the relatively high number of false positive detections produced. We propose several adaptations to the current existing LBP/AdaBoost cascade classification pipeline of OpenCV. This is done by improving the training sample generation and annotation procedure, and by applying an active learning strategy. This boosts the accuracy of in the wild face detection on the FDDB dataset quite drastically, closing the gap towards the accuracy levels gained by state-of-the-art CNN-based face detectors. The proposed changes allow us to provide a new face detection model to the OpenCV framework, achieving a remarkably high precision at an acceptable recall rate, offering a model that ensures detected faces are actually faces, a critical requirement for further processing pipelines like person identification, etc. ispartof: pages:396-404 ispartof: Proceedings of VISAPP 2017 vol:5 pages:396-404 ispartof: VISAPP location:Porto, Portugal date:5 Jan - 5 Jan 2017 status: published

The use of time-of-flight (ToF) cameras in industrial applications has become increasingly popular due to the camera’s reduced cost and its ability to provide real-time depth information. Still, one of the main drawbacks of these cameras has been their limited field of view. We therefore propose a technique to fuse the views of multiple ToF cameras. By mounting two cameras side by side and pointing them away from each other, the horizontal field of view can be artificially extended. The combined views can then be used for object detection. The main advantages of our technique is that the calibration is fully automatic and only one shot of the calibration target is needed. Furthermore, no overlap between the views is required. ispartof: pages:203-207 ispartof: Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications vol:4 pages:203-207 ispartof: VISAPP location:Porto, Portugal date:27 Feb - 1 Mar 2017 status: published

Unmanned Aerial Vehicles (UAV) enable numerous agricultural applications such as terrain mapping, monitor crop growth, detecting areas with diseases and so on. For these applications a UAV flies above the terrain and has a global view of the plants. When the individual fruits or plants have to be examined, an oblique view is better, e.g. via an inspection-camera mounted on expensive all-terrain wheeled robots that drive through the orchard. However, in this paper we aim to autonomously navigate through the orchard with a low-cost UAV and cheap sensors (e.g. a webcam). Evidently, this is challenging since every orchard or even every corridor looks different. For this we developed a vision-based system that detects the center and end of the corridor to autonomously navigate the UAV towards the end of the orchard without colliding with the trees. Furthermore extensive experiments were performed to prove that our algorithm is able to navigate through the orchard with high accuracy and in real-time, even on embedded hardware. A connection with a ground station is thus unnecessary which makes the UAV fully autonomous. Hulens D., Vandersteegen M., Goedemé T., ''Real-time vision-based UAV navigation in fruit orchards'', Proceedings 12th international conference on computer vision theory and applications - VISAPP 2017, 6 pp., February 27 - March 1, 2017, Porto, Portugal. ispartof: pages:617-622 ispartof: Proceedings VISAPP 2017 vol:4 pages:617-622 ispartof: 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP) location:Porto, Portugal date:27 Feb - 1 Mar 2017 status: published

© 2017 Glen Debard et al. More than thirty percent of persons over 65 years fall at least once a year and are often not able to get up again. Camera-based fall detection systems can help by triggering an alarm when falls occur. Previously we showed that real-life data poses significant challenges, resulting in high false alarm rates. Here, we show three ways to tackle this. First, using a particle filter combined with a person detector increases the robustness of our foreground segmentation, reducing the number of false alarms by 50%. Second, selecting only nonoccluded falls for training further decreases the false alarm rate on average from 31.4 to 26 falls per day. But, most importantly, this improvement is also shown by the doubling of the AUC of the precision-recall curve compared to using all falls. Third, personalizing the detector by adding several days containing only normal activities, no fall incidents, of the monitored person to the training data further increases the robustness of our fall detection system. In one case, this reduced the number of false alarms by a factor of 7 while in another one the sensitivity increased by 17% for an increase of the false alarms of 11%. Debard G., Mertens M., Goedemé T., Tuytelaars T., Vanrumste B., ''Three ways to improve the performance of real-life camera-based fall detection systems'', Journal of sensors, vol. 2017 - Article ID 8241910, 15 pp., October 23, 2017. ispartof: Journal of Sensors vol:2017 status: published

Photogrammetric acquisition with unmanned aerial vehicles (UAV) has grown extensively over the last couple of years. Such mobile platforms and their processing software have matured, resulting in a market which offers off-the-shelf mapping solutions to surveying companies and geospatial enterprises. Different approaches in platform type and optical instruments exist, though its resulting products have similar specifications. To demonstrate differences in acquisitioning practice, a case study over an open mine was flown with two different off-the-shelf UAVs (a fixed-wing and a multi-rotor). The resulting imagery is analyzed to clarify the differences in collection quality. We look at image settings, and stress the fact of photographic experience if manual setting are applied. For mapping production it might be safest to set the camera on automatic. Furthermore, we try to estimate if blur is present due to image motion. A subtle trend seems to be present, for the fast flying platform though its extent is of similar order to the slow moving one. It shows both systems operate at their limits. Finally, the lens distortion is assessed with special attention to chromatic aberration. Here we see that through calibration such aberrations could be present, however detecting this phenomena directly on imagery is not straightforward. For such effects a normal lens is sufficient, though a better lens and collimator does give significant improvement.

In this paper we propose a low-cost high-speed imaging line scan system. We replace an expensive industrial line scan camera and illumination with a custom-built set-up of cheap off-the-shelf components, yielding a measurement system with comparative quality while costing about 20 times less. We use a low-cost linear (1D) image sensor, cheap optics including a LED-based or LASER-based lighting and an embedded platform to process the images. A step-by-step method to design such a custom high speed imaging system and select proper components is proposed. Simulations allowing to predict the final image quality to be obtained by the set-up has been developed. Finally, we applied our method in a lab, closely representing the real-life cases. Our results shows that our simulations are very accurate and that our low-cost line scan set-up acquired image quality compared to the high-end commercial vision system, for a fraction of the price., 2015 International Conference on Image Processing Theory, Tools and Applications (IPTA)