Search

Technological advances have changed our conception of cameras as boxes that capture images into the notion of networked, smart devices that analyze their environment and trigger decisions. Since smart cameras have become key enabling components for cyberphysical systems, trust in these devices is gaining importance.

Crowd surveillance plays a key role to ensure safety and security in public areas. Surveillance systems traditionally rely on fixed camera networks, which suffer from limitations, as coverage of the monitored area, video resolution and analytic performance. On the other hand, a smart camera network provides the ability to reconfigure the sensing infrastructure by incorporating active devices such as pan-tilt-zoom (PTZ) cameras and UAV-based cameras, thus enabling the network to adapt over time to changes in the scene. We propose a new decentralised approach for network reconfiguration, where each camera dynamically adapts its parameters and position to optimise scene coverage. Two policies for decentralised camera reconfiguration are presented: a greedy approach and a reinforcement learning approach. In both cases, cameras are able to locally control the state of their neighbourhood and dynamically adjust their position and PTZ parameters. When crowds are present, the network balances between global coverage of the entire scene and high resolution for the crowded areas. We evaluate our approach in a simulated environment monitored with fixed, PTZ and UAV-based cameras.

Crowd surveillance will play a fundamental role in the coming generation of video surveillance systems, in particular for improving public safety and security. However, traditional camera networks are mostly not able to closely survey the entire monitoring area due to limitations in coverage, resolution and analytics performance. A smart camera network, on the other hand, offers the ability to reconfigure the sensing infrastructure by incorporating active devices such as pan-tilt-zoom (PTZ) cameras and UAV-based cameras, which enable the adaptation of coverage and target resolution over time. This paper proposes a novel decentralized approach for dynamic network reconfiguration, where cameras locally control their PTZ parameters and position, to optimally cover the entire scene. For crowded scenes, cameras must deal with a trade-off among global coverage and target resolution to effectively perform crowd analysis. We evaluate our approach in a simulated environment surveyed with fixed, PTZ, and UAV-based cameras.

Smart cameras perform on-board image analysis, adapt their algorithms to changes in their environment, and collaborate with other networked cameras to analyze the dynamic behavior of objects. A proposed computational framework adopts the concepts of self-awareness and self-expression to more efficiently manage the complex tradeoffs among performance, flexibility, resources, and reliability. The Web extra at http://youtu.be/NKe31_OKLz4 is a video demonstrating CamSim, a smart camera simulation tool, enables users to test self-adaptive and self-organizing smart-camera techniques without deploying a smart-camera network.

We study heterogeneity among nodes in self-organizing smart camera networks, which use strategies based on social and economic knowledge to target communication activity efficiently. We compare homogeneous configurations, when cameras use the same strategy, with heterogeneous configurations, when cameras use different strategies. Our first contribution is to establish that static heterogeneity leads to new outcomes that are more efficient than those possible with homogeneity. Next, two forms of dynamic heterogeneity are investigated: nonadaptive mixed strategies and adaptive strategies, which learn online. Our second contribution is to show that mixed strategies offer Pareto efficiency consistently comparable with the most efficient static heterogeneous configurations. Since the particular configuration required for high Pareto efficiency in a scenario will not be known in advance, our third contribution is to show how decentralized online learning can lead to more efficient outcomes than the homogeneous case. In some cases, outcomes from online learning were more efficient than all other evaluated configuration types. Our fourth contribution is to show that online learning typically leads to outcomes more evenly spread over the objective space. Our results provide insight into the relationship between static, dynamic, and adaptive heterogeneity, suggesting that all have a key role in achieving efficient self-organization.

Smart cameras are considered as key sensors in Internet of Things (IoT) applications ranging from home to city scales. Since these cameras often capture highly sensitive information, security is a major concern. An elliptic curve (EC) based signcryption achieves resource-efficiency by performing data encryption and signing in a single step. In this work, we present aggregate-signcryption which extends the EC-based signcryption approach to a cluster-based multi-camera setup. The signcrypted data from the smart cameras within a cluster is aggregated on a specific node called cluster head. Aggregatesigncryption reduces the communication overhead and requires fewer steps for the unsigncryption as compared to individual signcryption.

Cameras and other sensors are increasingly deployed for private space monitoring applications such as home monitoring, assisted/enhanced living and child monitoring. Since these cameras capture highly sensitive information and transfer it over public communication infrastructures, security and privacy is a major concern. This work presents a secure camera device along with a secure data delivery and archiving solution for private space monitoring applications using untrusted public cloud storage services. Integrity, authenticity, confidentiality and freshness of captured data are protected on-board using physically unclonable functions (PUF). The protection holds true for entire lifetime of the data until it is consumed by an authorized end-user. Experimental results obtained from our Zynq7010 SoC based prototype shows that the device is able to secure videos with 30 frames per second at 640×480 resolution with marginal overhead. The presented solution is not limited to visual sensing but can be applied to a wide range of pervasive sensing and secure data delivery scenarios.

In this paper we present an algorithm, which enables distributed visual sensor networks to autonomously calibrate the network and dynamically build clusters to achieve cooperative object tracking based on state estimation. A main focus is thereby on resource-awareness and -efficiency, since we aim for low-power embedded smart camera networks. We do not require any human intervention or a-priori information about the network topology to achieve calibration and tracking. Camera nodes first estimate relative positions and orientations and then use the common coordinate system to enable cooperative state estimation. For that purpose, cameras dynamically build clusters depending on their available resources. New nodes joining the network are discovered and failing nodes do not prevent others from their tasks. Compared to other methods, our approach is not only able to handle sensor measurement errors but also faulty camera positions gathered during the network calibration process.

Cameras are expected to become key sensor devices for various internet of things (IoT) applications. Since cameras often capture highly sensitive information, security is a major concern. Our approach towards data security for smart cameras is rooted on protecting the captured images by signcryption based on elliptic curve cryptography (ECC). Signcryption achieves resource-efficiency by performing data signing and encryption in a single step. By running the signcryption on the sensing unit, we can relax some security assumptions for the camera host unit which typically runs a complex software stack. We introduce our system architecture motivated by a typical case study for camera-based IoT applications, evaluate security properties and present performance results of an ARM-based implementation.

In this article we present an approach to object tracking handover in a network of smart cameras, based on self-interested autonomous agents, which exchange responsibility for tracking objects in a market mechanism, in order to maximise their own utility. A novel ant-colony inspired mechanism is used to learn the vision graph, that is, the camera neighbourhood relations, during runtime, which may then be used to optimise communication between cameras. The key benefits of our completely decentralised approach are on the one hand generating the vision graph online, enabling efficient deployment in unknown scenarios and camera network topologies, and on the other hand relying only on local information, increasing the robustness of the system. Since our market-based approach does not rely on a priori topology information, the need for any multicamera calibration can be avoided. We have evaluated our approach both in a simulation study and in network of real distributed smart cameras.

Smart cameras are key sensors in Internet of Things (IoT) applications and often capture highly sensitive information. Therefore, security and privacy protection is a key concern. This paper introduces a lightweight security approach for smart camera IoT applications based on elliptic-curve (EC) signcryption that performs data signing and encryption in a single step. We deploy signcryption to efficiently protect sensitive data onboard the cameras and secure the data transfer from multiple cameras to multiple monitoring devices. Our multi-sender/multi-receiver approach provides integrity, authenticity, and confidentiality of data with decryption fairness for multiple receivers throughout the entire lifetime of the data. It further provides public verifiability and forward secrecy of data. Our certificateless multi-receiver aggregate-signcryption protection has been implemented for a smart camera IoT scenario, and the runtime and communication effort has been compared with single-sender/single-receiver and multi-sender/single-receiver setups.

By bringing together advances in computer vision, embedded computing, image sensors, and networks, researchers are working toward more automated analysis of large-scale camera networks' data.

Distributed smart cameras (DSC) are an emerging technology for a broad range of important applications including smart rooms, surveillance, entertainment, tracking, and motion analysis. By having access to many views and through cooperation among the individual cameras, these DSCs have the potential to realize many more complex and challenging applications than single-camera systems. This article focuses on the system-level software required for efficient streaming applications on single smart cameras as well as on networks of DSCs. Embedded platforms with limited resources do not provide middleware services well known on general-purpose platforms. Our software framework supports transparent intra- and interprocessor communication while keeping the memory and computation overhead very low. The software framework is based on a publisher--subscriber architecture and provides mechanisms for dynamically loading and unloading software components as well as for graceful degradation in case of software- and hardware-related faults. The software framework has been completely implemented and tested on our embedded smart cameras consisting of an ARM-based network processor and several digital signal processors. Two case studies demonstrate the feasibility of our approach.

Distributed smart cameras (DSCs) are real-time distributed embedded systems that perform computer vision using multiple cameras. This new approach has emerged thanks to a confluence of simultaneous advances in four key disciplines: computer vision, image sensors, embedded computing, and sensor networks. Processing images in a network of distributed smart cameras introduces several complications. However, we believe that the problems DSCs solve are much more important than the challenges of designing and building a distributed video system. We argue that distributed smart cameras represent key components for future embedded computer vision systems and that smart cameras will become an enabling technology for many new applications. We summarize smart camera technology and applications, discuss current trends, and identify important research challenges.

Recently much research has been conducted in visual sensor networks. Compared to traditional sensor networks, vision networks differ in various aspects such as the amount of data to be processed and transmitted, the requirements on quality-of-service, and the level of collaboration among the sensor nodes. This paper deals with sensor fusion on visual sensor networks. We focus here on methods for fusing data from various distributed sensors and present a generic framework for fusion on embedded sensor nodes. This paper extends our previous work on distributed smart cameras and presents our approach toward the transformation of smart cameras into a distributed, embedded multisensor network. Our generic fusion model has been completely implemented on a distributed embedded system. It provides a middleware which supports automatic mapping of our fusion model to the target hardware. This middleware features dynamic reconfiguration to support modification of the fusion application at runtime without loss of sensor data. The feasibility and reusability of the I-SENSE concept is demonstrated with experimental results of two case studies: vehicle classification and bulk good separation. Qualitative and quantitative benefits of multilevel information fusion are outlined in this article.

When visual sensor networks are composed of cameras which can adjust the zoom factor of their own lens, one must determine the optimal zoom levels for the cameras, for a given task. This gives rise to an important trade-off between the overlap of the different cameras' fields of view, providing redundancy, and image quality. In an object tracking task, having multiple cameras observe the same area allows for quicker recovery, when a camera fails. In contrast having narrow zooms allow for a higher pixel count on regions of interest, leading to increased tracking confidence. In this paper we propose an approach for the self-organisation of redundancy in a distributed visual sensor network, based on decentralised multi-objective online learning using only local information to approximate the global state. We explore the impact of different zoom levels on these trade-offs, when tasking Omni directional cameras, having perfect 360-degree view, with keeping track of a varying number of moving objects. We further show how employing decentralised reinforcement learning enables zoom configurations to be achieved dynamically at runtime according to an operator's preference for maximising either the proportion of objects tracked, confidence associated with tracking, or redundancy in expectation of camera failure. We show that explicitly taking account of the level of overlap, even based only on local knowledge, improves resilience when cameras fail. Our results illustrate the trade-off between maintaining high confidence and object coverage, and maintaining redundancy, in anticipation of future failure. Our approach provides a fully tunable decentralised method for the self-organisation of redundancy in a changing environment, according to an operator's preferences.

Recent advances in computing, communication, and sensor technology are pushing the development of many new applications. This trend is especially evident in pervasive computing, sensor networks, and embedded systems. Smart cameras, one example of this innovation, are equipped with a high-performance onboard computing and communication infrastructure, combining video sensing, processing, and communications in a single embedded device. By providing access to many views through cooperation among individual cameras, networks of embedded cameras can potentially support more complex and challenging applications - including smart rooms, surveillance, tracking, and motion analysis - than a single camera. We designed our smart camera as a fully embedded system, focusing on power consumption, QoS management, and limited resources. The camera is a scalable, embedded, high-performance, multiprocessor platform consisting of a network processor and a variable number of digital signal processors (DSPs). Using the implemented software framework, our embedded cameras offer system-level services such as dynamic load distribution and task reconfiguration. In addition, we combined several smart cameras to form a distributed embedded surveillance system that supports cooperation and communication among cameras.

Generally, resource-awareness plays a key role in wireless sensor networks due the limited capabilities in processing, storage and communication. In this paper we present a resource-aware cooperative state estimation facilitated by a dynamic cluster-based protocol in a visual sensor network (VSN). The VSN consists of smart cameras, which process and analyze the captured data locally. We apply a state estimation algorithm to improve the tracking results of the cameras. To design a lightweight protocol, the final aggregation of the observations and state estimation are only performed by the cluster head. Our protocol is based on a marketbased approach in which the cluster head is elected based on the available resources and a visibility parameter of the object gained by the cluster members. We show in simulations that our approach reduces the costs for state estimation and communication as compared to a fully distributed approach. As resource-awareness is the focus of the cluster-based protocol we can accept a slight degradation of the accuracy on the object’s state estimation by a standard deviation of about 1.48 length units to the available ground truth.

We have developed a self-aware multi-camera multi-object tracking application for distributed heterogeneous smart camera networks. This application is able to track multiple persons through the network whereby the cameras are able to transfer the tracking responsibility between each other in a self-organizing manner without any central control. This application runs on top of a fully distributed middleware, designed for extensibility and robustness as well as heterogeneous hardware and network technologies. Our demonstrator has been deployed in a laboratory environment consisting of different hardware platforms and network technologies.

Smart cameras are considered as emerging technology for the realization of various smart environments ranging from room to city scales. Since these cameras capture images that potentially reveal sensitive information about individuals, appropriate protection mechanisms are required. In this paper we discuss the key security and privacy protection domains in smart camera networks and present our TrustEYE.M4 platform which embeds privacy protection close to the image sensor and provides strong separation between hardware-supported protection and flexible scene analytics.

Dataflow-oriented processing is an attractive paradigm for smart camera applications. In this paper we present a dataflow-oriented middleware system to support the development of such applications on resource-limited distributed systems. Our main contribution includes the reduction of the overall processing delay by delaying the data generation and by the just-in-time transfer of data elements. A side effect of this delayed data transfer is also a reduction of the memory requirement for the communication links. We introduce the basic algorithm and present experimental results of our middleware system running in a distributed environment.

Visual privacy in video-based applications such as surveillance, assisted living or home monitoring is a highly active research topic. It is critical to protect the privacy of monitored people without severely limiting the utility of the system. We present a resource-aware cartooning privacy protection filter which converts raw images into abstracted frames where the privacy revealing details are re-moved. Cartooning can be applied either to entire images or pre-selected sensitive regions of interest. We provide an adaptation mechanism to our cartooning technique where the operator can easily change the filter intensity. The feasibility of this new approach is demonstrated by its deployment to real-world embedded smart cameras. We evaluate privacy protection and utility of cartooning with the PEViD data set and compare it with the two widely-used privacy filters: blurring and pixelation.

The 11 articles in this special issue on distributed smart cameras are divided into the following three categories: architecture and networks; distributed and collaborative image processing; and applications and security issues.

In this work, we propose the cubature Kalman filter (CKF) based distributed object tracking algorithm in a visual sensor network (VSN). A VSN consists of several distributed smart cameras having the ability to process and analyze the retrieved data locally. The first objective is to optimize the tracking process within the VSN through the CKF. Under the conditions of non-linear motion and observation model, the CKF based method features a considerably better tracking accuracy than the extended Kalman filter (EKF) based method in terms of the mean square error (MSE). Although, the particle filter (PF) based method shows better performance than the CKF, it is computationally very complex. The second objective is to optimize the object tracking by aggregating the tracking results from multiple cameras. Assuming the VSN is a multi-camera network with overlapping field of views (FOVs), cameras having the same object in their FOV exchange their local estimates of the object's position and velocity. During the estimation process, each of the participating cameras aggregates the received states via a consensus algorithm. Thus, the object's real state is more accurately predicted by the resulting joint state than it would be by processing only a single camera's observation.

In this paper we study the self-organising behaviour of smart camera networks which use market-based handover of object tracking responsibilities to achieve an efficient allocation of objects to cameras. Specifically, we compare previously known homogeneous configurations, when all cameras use the same marketing strategy, with heterogeneous configurations, when each camera makes use of its own, possibly different marketing strategy. Our first contribution is to establish that such heterogeneity of marketing strategies can lead to system wide outcomes which are Pareto superior when compared to those possible in homogeneous configurations. However, since the particular configuration required to lead to Pareto efficiency in a given scenario will not be known in advance, our second contribution is to show how online learning of marketing strategies at the individual camera level can lead to high performing heterogeneous configurations from the system point of view, extending the Pareto front when compared to the homogeneous case. Our third contribution is to show that in many cases, the dynamic behaviour resulting from online learning leads to global outcomes which extend the Pareto front even when compared to static heterogeneous configurations. Our evaluation considers results obtained from an open source simulation package as well as data from a network of real cameras.

Smart cameras allow pre-processing of video data on the camera instead of sending it to a remote server for further analysis. Having a network of smart cameras allows various vision tasks to be processed in a distributed fashion. While cameras may have different tasks, we concentrate on distributed tracking in smart camera networks. This application introduces various highly interesting problems. Firstly, how can conflicting goals be satisfied such as cameras in the network try to track objects while also trying to keep communication overhead low? Secondly, how can cameras in the network self adapt in response to the behavior of objects and changes in scenarios, to ensure continued efficient performance? Thirdly, how can cameras organise themselves to improve the overall network's performance and efficiency? This paper presents a simulation environment, called CamSim, allowing distributed self-adaptation and self-organisation algorithms to be tested, without setting up a physical smart camera network. The simulation tool is written in Java and hence allows high portability between different operating systems. Relaxing various problems of computer vision and network communication enables a focus on implementing and testing new self-adaptation and self-organisation algorithms for cameras to use.

Embedded smart cameras are gaining in popularity for a number of real-Time outdoor surveillance applications. However, there are still challenges, i.e., computational latency, variation in illumina ...

This paper presents a novel method for video-based traffic state detection on motorways performed on smart cameras. Camera calibration parameters are obtained from the known length of lane markings. Mean traffic speed is estimated from Kanade-Lucas-Tomasi (KLT) optical flow method using a robust outlier detection. Traffic density is estimated using a robust statistical counting method. Our method has been implemented on an embedded smart camera and evaluated under different road and illumination conditions. It achieves a detection rate of more than 95% for stationary traffic.

A recent trend in smart camera networks is that they are able to modify the functionality during runtime to better reflect changes in the observed scenes and in the specified monitoring tasks. In this paper we focus on different configuration methods for such networks. A configuration is given by three components: (i) a description of the camera nodes, (ii) a specification of the area of interest by means of observation points and the associated monitoring activities, and (iii) a description of the analysis tasks. We introduce centralized, distributed and proprioceptive configuration methods and compare their properties and performance.

A novel level of service (LOS) estimation approach based on the extraction of three local visual features is presented. The feature set comprises KLT motion vectors and Sobel edges, and is fed into a Gaussian radial-basis-function (GRBF) network to classify the prevailing LOS. The whole approach is designed and implemented to run on smart cameras in real-time and has been evaluated with a comprehensive set of real-world training and test video data from a national motorway. The evaluations in daylight environments have shown an average accuracy of LOS classification of 86.2% on an Atom-based smart camera, with a maximum reachable processing frame rate of 12.5 frames/sec. Incorrect classified samples differed from the ground truth by only one level. The comparisons are done with observation data from sensors utilizing a combination of Doppler radar, ultrasound, and passive infrared technologies.

In this paper we propose an approach based on self-interested autonomous cameras, which exchange responsibility for tracking objects in a market mechanism, in order to maximise their own utility. A novel ant-colony inspired mechanism is used to grow the vision graph during runtime, which may then be used to optimise communication between cameras. The key benefits of our completely decentralised approach are on the one hand generating the vision graph online which permits the addition and removal cameras to the network during runtime and on the other hand relying only on local information, increasing the robustness of the system. Since our market-based approach does not rely on a priori topology information, the need for any multi-camera calibration can be avoided.

Camera systems are used in many applications including video surveillance for crime prevention and investigation, traffic monitoring on highways or building monitoring and automation. With the shift from analog towards digital systems, the capabilities of cameras are constantly increasing. Today's smart camera systems come with considerable computing power, large memory, and wired or wireless communication interfaces. With onboard image processing and analysis capabilities, cameras not only open new possibilities but also raise new challenges. Often overlooked are potential security issues of the camera system. The increasing amount of software running on the cameras turns them into attractive targets for attackers. Therefore, the protection of camera devices and delivered data is of critical importance. In this work we present an embedded camera prototype that uses Trusted Computing to provide security guarantees for streamed videos. With a hardware-based security solution, we ensure integrity, authenticity, and confidentiality of videos. Furthermore, we incorporate image timestamping, detection of platform reboots, and reporting of the system status. This work is not limited to theoretical considerations but also describes the implementation of a prototype system. Extensive evaluation results illustrate the practical feasibility of the approach.

This paper describes an approach for integrating cognition, real-time communication and control. Our test setup is a crane system whose area of operation is observed by distributed smart cameras analyzing the crane's environment. The position and motion information of all detected objects are transferred to the crane controller via a real-time network. With this information the controller can optimize the trajectory of the load. We present preliminary results of a single camera setting and give an outlook for the multi camera data fusion process.

The majority of research in the field of camera networks is targeted at distributed and cooperative processing, advanced computer vision algorithms or the development of embedded, ubiquitous camera systems. Privacy and security are topics that are often overlooked or considered as an afterthought. With the digitalization of visual surveillance, data can easily be stored and accessed. This raises the question how confidential data can be protected, authenticity and integrity can be ensured and access can be restricted. This work discusses security and privacy issues relevant in the context of visual surveillance and camera networks. We try to capture the state of the art on these aspects in the available literature and highlight areas that require special consideration. We present a concept of a privacy-preserving camera system based on Trusted Computing. In our system-level approach, we consider privacy and security as primary goals without limiting the overall usefulness of a camera system.

Pervasive smart cameras (PSC) are an emerging technology with the goal of providing user-centric and ubiquitous visual sensor networks. System autonomy and resource-awareness are challenging requirements making local image processing for event-based communication a necessary design constraint. In this paper we present the distributed event space (DES)---a middleware service for the resource-aware management of distributed event data. The DES architecture is based on a decentralized tuple space which describes local events by tuples consisting of position and time of the events as well as a set of features describing the detected objects. The DES supports prompt distribution of detected local events and application-specific functions for sophisticated event filtering. We present a distributed tracking application to demonstrate its applicability.

Security and privacy protection are critical issues forpublic acceptance of camera networks. Smart cameras,with onboard image processing, can be used to identifyand remove privacy sensitive image regions. Existing approaches,however, only address isolated aspects withoutconsidering the integration with established security technologiesand the underlying platform. This work tries to fillthis gap and presents TrustCAM, a security-enhanced smartcamera. Based on Trusted Computing, we realize integrityprotection, authenticity and confidentiality of image data.Multiple levels of privacy protection, together with accesscontrol, are supported. Impact on overall system performanceis evaluated on a real prototype implementation.

Camera networks are used for a variety of applications including surveillance, traffic monitoring or elderly care. The shift from analog towards fully digitized systems has considerably increased their capabilities. With large-scale deployments of smart cameras and visual sensor networks, public awareness of privacy issues is increasing. Researchers are addressing these concerns by introducing privacy preserving technologies like content scrambling and encryption. Today's systems however do not provide mechanisms that allow monitored people to verify that a camera system is behaving as advertised by its operators. In this work, we propose to use Trusted Computing to enhance the security of camera systems and, by enabling user-based attestation, give users a simple and intuitive way to check the trustworthiness of cameras.

Visual sensor networks bring together the research domains of smart cameras and wireless sensor networks. Focusing on power consumption and longterm deployments, typical sensor nodes come with limited computing power and low communication performance. A smart camera that performs onboard video analysis and exchanges data with adjacent nodes has higher computation and communication requirements. Therefore, a main challenge in visual sensor networks is to find a tradeoff between performance and power consumption. This work presents a camera prototype equipped with a low- and a high-performance radio allowing to trade communication performance for power consumption by selecting the appropriate device. Radio power consumption is evaluated for different modes of operation and a power model for clusters of dual-radio cameras is presented.

Pervasive Smart Cameras are embedded computer vision systems bringing together the research areas of smart cameras and wireless sensor networks. Besides traditional areas like video surveillance or traffic monitoring, small, cheap and powerful camera systems open a wide range of potential new applications including assisted living, home automation or entertainment. At the same time, widespread deployment of cameras introduces several security challenges. Using wireless networking and being mounted at remote locations, smart cameras are an attractive target for attackers. Another issue of crucial importance when it comes to the acceptance of camera systems is user privacy. In this work we explore the use of Trusted Computing concepts to enhance security of an experimental smart camera system. Additionally, we discuss required and achievable performance based on evaluations on our prototype platform.

Advances in sensors, networking and embedded computing have driven a paradigm shift in the control of camera networks. The processing of the image data has migrated from central servers to the camera network following fully distributed or clustered approach. In clustering, the data is analyzed within a group of cameras, and the abstracted data is then forwarded to other clusters. In this paper we present a flexible and scalable software tool that supports clustering in smart camera networks. This tool supports the dataflow-oriented processing within a group of cameras. A plug-in-mechanism facilitates flexibility and scalability; data processing can be migrated between the cluster node and the smart cameras. We demonstrate the feasibility of our tool in a multi-camera person tracking case study

Clustering is an important concept for structuring large networks of cameras. In this research we investigate the various trade offs of clustering in networks of smart cameras. Major research questions in this context are (i) how to model clustering, (ii) how to deal with the heterogeneity in large camera networks, and (iii) how to integrate clustering in real-world networks. We have developed a flexible and scalable software suite that supports clustering in camera networks. We present first results in a multi-camera person tracking case study.

Visual sensor networks are an emerging research area with the goal of using cameras as pervasive and affordable sensing and processing devices. This paper presents a pervasive smart camera platform which is built from off-the-shelf hardware and software components. The hardware platform is comprised of an OMAP 3530 processor, 128 MB RAM and various interfaces for connecting sensors and peripherals. A dual-radio wireless network allows to trade communication performance for power consumption. The software architecture is built upon standard Linux and supports dataflow oriented application development by dynamically instantiating and connecting functions blocks. Data is transferred between blocks via shared memory for high throughput. We present a performance evaluation of our smart camera platform as well as a multi-camera object classification system to demonstrate the capabilities and applicability of our approach.

Smart cameras are embedded systems that perform on-board video content analysis and only report detected events instead of permanently streaming videos. Visual sensor networks aim at integrating smart cameras with wireless sensor networks. Camera sensors have higher requirements regarding computing power and communication bandwidth than those typically used in wireless sensor network applications. Consequently, power management is an even more critical issue. This work in progress presents an attempt to address this by combining high and low power radios as well as high and low performance computing systems in one single platform. This allows to control power consumption by selectively enabling only required components.

Smart camera networks are real-time distributed embedded systems that perform computer vision using multiple cameras. This new approach has emerged thanks to a confluence of simultaneous advances in four key disciplines: computer vision, image sensors, embedded computing, and sensor networks. In this chapter, we briefly review and classify smart camera platforms and networks into single smart cameras, distributed smart camera systems, and wireless smart camera networks. We elaborate the vision of pervasive smart camera networks and identify major research challenges. As the technology for these networks advances, we expect to see many new applications open up—transforming traditional multi-camera systems into pervasive smart camera networks.

Smart cameras represent an interesting research field that has evolved over the last decade. In this chapter we focus on the integration of multiple, potentially heterogeneous smart cameras into a distributed system for computer vision and sensor fusion. Because an important aspect of distributed systems is the system-level software, also called middleware, we discuss the middleware requirements of distributed smart cameras and the services the middleware must provide. In our opinion, a middleware following the agent-oriented paradigm allows us to build flexible and self-organizing applications that encourage a modular design.

Having seen increased interest from the research community, smart camera systems have gone through a number of evolutionary steps like from single cameras to distributed smart camera systems with collaboration features. This work aims at defining a taxonomy to classify these systems based on their platform capabilities, the degree of distributed processing as well as system autonomy aspects covering self-configuration and mobility. Existing camera systems are classified according to the proposed taxonomy. Besides capturing the design space for smart cameras, the main contribution of this paper is an overview of the research challenges for the envisioned class of pervasive smart camera systems. As defined in this work, future pervasive smart camera systems will be visual sensor networks targeted at end-user applications where special emphasis is put on unobtrusiveness of the cameras as well as simple deployment supported by self configuration capabilities.

This paper investigates on middleware for distributed smart cameras. We describe DSCAgents, our agent-oriented lightweight middleware system. The design goal is to provide a modular and flexible middleware that takes into account the underlying hardware platform and also supports collaborative image processing. Mobile agents are used to model image processing tasks and to manage the whole smart camera network. The agent-oriented approach simplifies application development and domain specific services unburden programmers from implementing the same functionality for each application again. Mobility of agents allows to build highly dynamic and adaptive systems where image processing tasks can move from one camera to another during operation. The evaluation of DSCAgents shows reasonable performance while keeping the resource requirements low.

In the recent past, much effort has been put into the development of distributed vision systems with smart cameras as key components. Smart cameras combine video sensing, processing and communication within a single embedded device and provide sufficient on-board infrastructure to carry out high-level video analysis tasks. Networks of smart cameras help to overcome some hard problems inherent to single-camera systems by providing multiple views of a scene. This paper reports on an improved, agent-oriented middleware for embedded smart cameras. Each image processing task is represented by an agent resident on a smart camera within the network. Agents are able to move from one camera to another as needed during run-time. An agent is comprised of the high-level application logic and the image processing algorithm which is executed on the processing unit. The presented middleware is also designed for distributed image processing where two or more cameras can cooperate for a single task. In the paper we discuss the requirements for such an agent-oriented middleware capable of supporting distributed image processing. Further, we describe the architecture of our middleware implementation. The evaluation of our current middleware implementation shows significant performance improvements compared to our previous Java-based implementation.

In the proposed project we are working towards multi-sensor smart cameras, i.e., we augment vision-based cameras by additional sensors such as infrared and audio and, thus, transform a single smart camera into an embedded multi-sensor node. Our software framework for embedded online data fusion, called I-SENSE, which supports data fusion on different levels of data abstraction is presented. Further our fusion model is presented with the focus set on four main parts, namely (i) the acoustic and visual feature extraction, (ii) feature based data fusion and the feature selection algorithm, (iii) feature based decision modeling based on support vector machines (SVM) and (iv) decision modeling based on a modified Dempster-Shafer approach is discussed. Finally we demonstrate the feasibility of our multilevel data fusion approach with experimental results of our "vehicle classification" case study.

Networks of embedded smart cameras are an emerging technology for a broad range of pervasive computing applications including smart rooms, intelligent infrastructures and security. Smart cameras combine video sensing, processing and communication within a single embedded device and provide sufficient onboard infrastructure to perform high-level video analysis tasks. This paper deals with middleware services required for the efficient deployment and operation of distributed smart cameras. We focus here on services for autonomous and dynamic reconfiguration. Dynamic reconfiguration refers to the exchange of software tasks as well as the alteration of QoS-levels these tasks provide during runtime. Dynamic reconfiguration provides several advantages over statically configured networks including (i) modification of functionality after deployment and during runtime, (ii) adaptation of the network to changes in its internal and external state, and (Hi) better exploitation of the available resources. We have developed the services for dynamic reconfiguration using policies. Policies help to specify rules for the reconfiguration process. By evaluation the policy the new task-level configuration of the network is computed. The reconfiguration is implemented using mobile agents in order to achieve a flexible and scalable middleware service. Our policy-based middleware is demonstrated by a surveillance application.