Search

Remotely Operated Vehicles (ROVs) are essential to human-operated underwater expeditions in the deep sea. However, piloting an ROV to safely interact with live ecosystems is an expensive and cognitively demanding task, requiring extensive maneuvering and situational awareness. Immersive Virtual Reality (VR) Head-Mounted Displays (HMDs) could address some of these challenges. This paper investigates how VR HMDs influence operator performance through a novel telepresence system for piloting ROVs in real-time. We present an empirical user study [N=12] that examines common midwater creature capture tasks, comparing Stereoscopic-VR, Monoscopic-VR, and Desktop teleoperation conditions. Our findings indicate that Stereoscopic-VR can outperform Monoscopic-VR and Desktop ROV capture tasks, effectively doubling the efficacy of operators. We also found significant differences in presence, task load, usability, intrinsic motivation, and cybersickness. Our research points to new opportunities towards VR with ROVs.

As artificial agents proliferate into society, there will be more and more situations in which they need to communicate their capabilities to humans, including what they can “see.” Humans do this with each other by using mental models of human capabilities coupled with social cues that enable situation awareness. In order to design better agents, we analyze video games, which have been communicating to humans about agent visual perception for decades. We present preliminary findings from a qualitative thematic analysis of sight cues in video games. We focus on three cue properties that warrant further study: whether the cue specifies the perceiver, whether the stimulus’ primary purpose is to be a sight cue, and the communication of non-binary sighting information. We close with an additional call for future work: on the effects of using multiple sight cues in combination.

Australia has witnessed heavy wildfires since 2019, which were especially serious in eastern Victoria. What's worse, climate change can make situations more serious. Surprisingly the unmanned drones can help a lot in dealing with wildfires, which are divided into two types: one for surveillance and situation awareness (SSA) and another for extending the communication range limited by circumstances. This paper aims to provide purchase and deployment plans of both kinds of drones to the government of Victoria taking the fires on one of the mountains as examples. To achieve that, we establish three models. Firstly, a multi-objective programming model is used to make the plan reasonable comprehensively. Considering the safety of firemen, mutual interference of different drones, and consumption of drones, we build the model to study the best solution under constraints. After learning the basic size and frequency of wildfires in Victoria, we can construct the drones' system better. The second model is established to optimize the locations of repeater drones. We take our basic assumptions into consideration and put forward the possible locations of repeater drones. Based on new assumptions, we obtain the optimal results of repeater drones. The third model is established to obtain the patrol route of SSA drones, considering the safety of firemen and the capability of drones, then SSA drones' number and patrol mode are confirmed through boustrophedon decomposition. Then we can make the cost of the wildfire rescue system minimum.

Resource management for geo-distributed infrastructures is challenging due to the scarcity and non-uniformity of edge resources, as well as the high client mobility and workload surges inherent to situation awareness applications. Due to their centralized nature, state-of-the-art schedulers that work well in datacenters lack the performance and feature requirements of such applications. We present OneEdge, a hybrid control plane that enables autonomous decision-making at edge sites for localized, rapid single-site application deployment. Edge sites handle mobility, churn, and load spikes, by cooperating with a centralized controller that allows coordinated multi-site scheduling and dynamic reconfiguration. OneEdge's scheduling decisions are driven by each application's end-to-end service level objective (E2E SLO) as well as the specific requirements of situation awareness applications. OneEdge's novel distributed state management combines autonomous decision-making at the edge sites for rapid localized resource allocations with decision-making at the central controller when multi-site application deployment is needed. Using a mix of applications on multi-region Azure instances, we show that, in contrast to centralized or fully distributed control planes, OneEdge caters to the unique requirements of situation awareness applications. Compared to a centralized control plane, OneEdge reduces deployment latency by 66% for single-site applications, without compromising E2E SLOs.

We propose ModularHMD, a new mobile head-mounted display concept, which adopts a modular mechanism and allows a user to perform ad-hoc peripheral interaction with real-world devices or people during VR experiences. ModularHMD is comprised of a central HMD and three removable module devices installed in the periphery of the HMD cowl. Each module has four main states: occluding, extended VR view, video see-through (VST), and removed/reused. Among different combinations of module states, a user can quickly setup the necessary HMD forms, functions, and real-world visions for ad-hoc peripheral interactions without removing the headset. For instance, an HMD user can see her surroundings by switching a module into the VST mode. She can also physically remove a module to obtain direct peripheral visions of the real world. The removed module can be reused as an instant interaction device (e.g., touch keyboards) for subsequent peripheral interactions. Users can end the peripheral interaction and revert to a full VR experience by re-mounting the module. We design ModularHMD’s configuration and peripheral interactions with real-world objects and people. We also implement a proof-of-concept prototype of ModularHMD to validate its interactions capabilities through a user study. Results show that ModularHMD is an effective solution that enables both immersive VR and ad-hoc peripheral interactions.

In this late-breaking work, we describe the design of an interface for displaying streaming vital sign data on a digital checklist used in the emergency medical setting of pediatric trauma resuscitation. We used feedback from interviews and participatory design workshops with clinicians to develop two prototypes of the streaming vital sign interface. We evaluated these prototypes in design-walkthroughs, finding that clinicians preferred the design displaying trend graphs for all four vital signs at once. We discuss how streaming data interfaces on interactive mobile devices can be used to provide situational awareness while unobtrusively supporting different levels of clinical experience.

In (partially) automated vehicles, users will sometimes have to take over control due to system failure or reach of an operational driving domain end. In such scenarios, the user becomes the driver and has to quickly gain situational awareness. With advanced sensory, an automated vehicle could aid the user in building situational awareness by providing information. A literature analysis found no commonly conveyed information. Therefore, to evaluate the effects of four different abstraction levels (high, medium-high, medium, and low) and used modalities (visual vs. visual+auditory) on situational awareness and accompanying usability scores, we evaluated eight systems and a baseline without information display. In the between-subjects online monitor-based study (N=225), we found that while subjective measures are higher and a warning is required, providing abstract information does not improve objective situational awareness, and only being provided with visual information was preferred.

Digital companions and conversational agents are becoming increasingly popular in our everyday lives. Natural language interfaces play an important role in ubiquitous computing: voice assistants are used to control smart home devices and smartphone applications; chatbots serve as an interface to solve tasks and acquire information easily. However, misunderstandings due to non-standard language, expressions that serve social functions without conveying information, or a lack of situational awareness still pose problems for these interfaces. Humans are able to prevent or repair communication failures by imitating their conversation partner’s lexical choices, sentence structures, and overall language style; a mechanism known as Linguistic Alignment. In this paper we present different strategies to easily integrate an alignment effect in natural language interfaces. We implemented a chatbot that imitates alignment and tested it in an online user study with 75 participants. Our results show that alignment helps to decrease user frustration and perceived task workload.

External Human-Machine Interfaces (eHMIs) are proposed to address the communication gap between automated vehicles (AVs) and other road users. However, existing concepts of eHMIs are often limited in their ability to communicate in a clear, unambiguous, and scalable way, which hinders their effectiveness. eHMIs typically communicate by showing the vehicle’s driving intention, situational awareness, or its path/ trajectory; none of which is independently able to sufficiently achieve effective and unambiguous communication. We created a novel eHMI concept which combines these three communication elements in one unified design that (1) promises a more seamless, two-way interaction between AVs and other road users, and (2) is capable of a better scalable, high-resolution communication, i.e., facilitating communication beyond one-on-one communication while mitigating ambiguity regarding the specific recipient of the communication message. This addresses some of the limitations of existing eHMIs and paves a path towards a novel approach in eHMI design for future AVs.

Research shows that emotions have a substantial influence on the cognitive processes of humans and in the context of driving, can negatively influence driving performance. Drivers’ interaction with in-vehicle agents can improve their emotional state and can lead to increased road safety. Language is another important aspect that influences human behavior and information processing. This study aims to explore the effects of native and secondary-language processing on emotional drivers’ situation awareness, driving performance, and subjective perception by conducting a within-subject simulation study. Twenty-four young drivers drove three different laps with a native-language speaking agent, secondary-language speaking agent, and no agent. The study results are indicative of the importance of native-language processing in the context of driving. Native-language agent condition resulted in improved driving performance and heightened situation awareness. The study results and discussions have theoretical and practical design implications and are expected to help foster future work in this domain.

Automated driving seems promising to reduce crashes caused by human error. However, in the transition towards automated driving, a human is still required in some automation levels in some circumstances. Specifically, in conditional automation or SAE Level 3, a human needs to be able to continue the driving task any time the vehicle requests it. This means that throughout the L3 automated driving, this ”fallback-ready user” needs to remain in a state to continue driving, even when they are engaged in other tasks, such as watching a movie. We designed three interventions with the aim to increase their fallback-readiness and have tested them in a high-fidelity video driving simulation study. In this video, we present and describe the interventions, the study design and the setup to test the interventions.

Navigation and situation awareness are amongst the most important aspects of collaborative analysis in 3D environments. This paper investigates the latest technology of mixed reality to improve the team navigation through effective real-time communication. We develop a cross-platform collaboration system, supporting different types of devices and operating systems, including Microsoft HoloLens and iOS devices. Our system provides essential coordination and information sharing functions by leveraging on device sensors and Vuforia API of image marker to localize users inside a building. We provide a set of essential building navigation, visualization, and interaction methods to support joint tasks in the physical building environments among participants with mobile devices in real-time. We have performed a user study to evaluate different devices used in coordination tasks. Our results demonstrate the effects of immersive visualization for improving 3D navigation and coordination.

Manual assembly tasks require workers to precisely assemble parts in 3D space in parallel steps. Often, additional time pressure further increases the complexity of these parallel tasks (e.g. in adhesive bonding processes). The performance in parallel tasks is heavily influenced by the capacity of the working memory, which is often overwhelmed if rules and states of too many tasks have to be remembered or perceived. Therefore, we propose to use Augmented Reality (AR) to investigate how visual assistance with parallel tasks can affect worker performance in time and spatial dependent process steps. In a user study, we compare three conditions: AR instructions presented (a) One task, (b) Two tasks, and (c) Four tasks per step on a tablet. For instructions we used selected work steps from a standardized adhesive bonding process as a representative for common time-critical assembly tasks. Our results show that instructions with multiple displayed tasks simultaneously per step can improve the process time but also increase the error rate and task load. The work instructions with less displayed tasks per work step showed better subjective results among participants, which may increase motivation, especially among less experienced workers. Our results help designers and developers to design assistance systems for time-critical and simultaneously executable assembly tasks, while considering process times, error rate and task load.

The amount of public data being generated on a daily basis has grown exponentially in the last few years and continues to increase at incredible speed. Most of this data is unstructured and includes text in different formats, in different languages, from many different sources; images, video, audio, and data from sensors. A lot of that data contains information about events happening all over the world, many of which require emergency response. Detecting events in public data, in real time, is therefore critical in many applications: from getting information to first responders as quickly as possible, to creating situational awareness in such emergency situations, as getting the right information to the right places as quickly as possible is critical in saving lives. When an event is ongoing, information on what is happening can be critical in making decisions to keep people safe and take control of the particular situation unfolding. First responders have to quickly make decisions that include what resources to deploy and where. Fortunately, in most emergencies, people use social media to publicly share information. At the same time, sensor data is increasingly becoming available. In order to do this, efficient computational approaches must detect and deliver the right information to the right destination. This tutorial will cover techniques at the state-of-the art to detect events in real-time from large-scale heterogeneous sources. We will focus on NLP, Computer Vision, and Anomaly Detection techniques. We will give specific examples and discuss relevant future research directions in Machine Learning, NLP, Computer Vision and other fields relevant to real time event detection. We will also discuss applications of event detection.

Neuromorphic cameras are well suited to detect the motion of propellers (blades) on Unmanned Aerial Systems (UAS), or drones. In this paper, we introduce the concept of a virtual fence which is a low-cost networked situational awareness device to quickly alert that a drone has entered the zone. Neuromorphic cameras significantly reduce the amount of data that must be processed as opposed to conventional cameras. Processing is required only when events are generated. Those events can be generated by a drone, by another low altitude airborne object (projectiles or birds), or by variations in the background. We propose two complementary algorithms that allow us to differentiate the signature of propeller blades from other events. Those algorithms exploit the periodic nature of propellers’ signal and the presence of sub-harmonics in the detected signal. Those sub-harmonics are introduced in the signal when a camera pixel misses some high-frequency events. We also show how to adjust the optics of the camera so as to reduce the contrast of background events, simplifying the categorization task. A prototype of a system consuming, during normal operations, 5.14 W with a battery autonomy of to 27 hours is presented. This prototype can detect drones up to an altitude of 9 m using a DAVIS 346 from IniVation with a field of view of about 70 degrees. Based on the actual improvement in resolution of current and next generation neuromorphic cameras, it is expected that the range of detection will increase and the virtual fence concept could be deployed operationally in the next few years., ICONS 2021: International Conference on Neuromorphic Systems 2021, July 27-29, 2021, Knoxville, Tennessee [Virtual Event], Series: International Conference Proceeding Series (ICPS)

The digitized environments are particularly vulnerable to various attacks. In such a situation of a security attack, detecting and responding to attacks require effective actions. One of the most significant ways to improve resilience to security attacks is to obtain accurate and timely situational aspect of the security awareness. The efficient production and utilization of situation information is achieved by sharing information with other actors in the information sharing network quickly and reliably without compromising the confidential information of one's own organization. At the same time, it should also be possible to avoid a flood of irrelevant information in the sharing network, which wastes resources and slows down the implementation of security measures. In our study, we have investigated how security-related information can be shared online as efficiently as possible by building a security information sharing topology based on the two most widely used network optimization algorithms. In the article, we present a model of an information sharing network, in which three different parameters have been used to optimize the network topology: the activity level of organization, the similarity of information systems between different actors and the requirement for the level of information privacy generally in the organization.

Supervising anesthesiologists oversee junior anesthesiologists in several operating rooms to instruct and provide advice, and hence ensure high patient safety. Supervisors benefit from good situation awareness but accessing information about multiple patients can be challenging. In a user-centered design process, we have developed an interface concept for a head-worn display (HWD) to support multi-patient monitoring. The design includes data about the patients’ physiology, vital status, vital trends, and medications. We conducted an exploratory evaluation of a prototype in a high-fidelity medical simulation with six supervising anesthesiologists. Our results show that the supervisors’ situation awareness of all simulated operations improved because participants noticed and responded to multiple events over the course of the scenario. However, the supervisors reported increased mental workload due to the constant availability of patient data. We discuss insights about the HWD display, the design process, and the experience and views of staff on using the HWD.

An approach to simulating and analysing sensor events in a boxed beef supply chain is presented. The simulation component reflects our industrial partner’s transport routes and parameters under normal and abnormal conditions. The simulated transport events are fed into our situational awareness system for detecting temperature anomalies or potential box tampering. The situational awareness system features a logic-based modeling language and an inference engine that tolerates incomplete or erroneous observations. The paper describes the approach and experimental results in more detail.

The Resource Forecast and Ramp Visualization for Situational Awareness (RAVIS) is an innovative, open-source tool for visualizing variable renewable resource forecasts and alerts for significant up- and down-ramps in renewable generation and the consequent system net load. The modular dashboard of RAVIS contains configurable panes for viewing probabilistic time-series forecasts, ramp event alerts on the look-ahead timeline, and spatially resolved resource sites, and forecasts. For comprehensive situational awareness, the tool can add additional data layers from simulation and independent system operator (ISO) market clearing data---including electric line utilization, nodal price, and available generation flexibility---in response to continuously updated renewable forecasts. This paper introduces the RAVIS technology suite employed to provide optimum visualization and flexible design characteristics. The paper also illustrates some use cases of the tool using site-specific solar power forecast data obtained from the IBM Watt-Sun forecasting platform for the California ISO and Midcontinent ISO footprints. The source code for RAVIS is public (https://github.com/ravis-nrel/ravis), and the intended users are forecasters, utility planners, ISO operators, and researchers.

Digital inequality has been intensively studied in recent decades, due to the considerable social significance of this phenomenon. Research has struggled with finding quality and profound ways to measure digital literacy of people from different social groups, due to the dynamic character of digital technology, which results in ever-changing, nuanced types of digital inequality. This study proposes an innovative method for examining how users approach and accomplish digital tasks by introducing eye tracking for measuring user scan patterns, gaze and attention during completion of tasks. Eye tracking as a measurement of attention and focus reflects the processes that occur while users are performing required tasks, and therefore may be a useful tool to comprehend digital literacy. We apply this innovative methodology in a repeated measures observation study of digital skills of low-skilled participants in a computer introductory course. 19 participants were requested to perform several online tasks before and after completing the course. The paper describes the results, which demonstrate that although participants’ skills have improved, the improvement is manifest in basic, trivial uses, while advance uses, such as understanding of efficient searching, or using the Internet in sophisticated ways as an environmental resource supporting one's situational awareness, are only slightly improved, as data based on tracking user gaze and mouse movements reveals.

The availability of large quantities of data has given an impulse for methods and techniques to extract unseen useful knowledge and process it in a fast and scalable manner. In order to extract the most complete possible knowledge from the continuous data stream, it is necessary to use the heterogeneous data sources and process information from multiple modalities. The systems that utilize multimodal data must take advantage of the up-to-date approaches for data storage, usage, cleaning and storage. Neural networks and machine learning approaches are widely used for data-heavy software where pattern extractions and predictions need to be conducted while serverless computing frameworks are being increasingly used for machine learning solutions to optimize cost and speed of such systems. This work presents a framework for processing data from multimodal sources where the task of feature and pattern extraction is performed on a serverless computing platform. The use cases for public safety solutions to increase situational awareness are described and compared with other implementation approaches.

The Australian wildfires that lasted for several months have caused serious damage to Australia's ecological environment and other aspects. The Victorian government used drones for surveillance and situational awareness, which played a positive role in handling the fire.In this paper, the cellular automata model is used to simulate the occurrence and spread of forest fires, and the frequency and magnitude of fires can be simulated by controlling related parameters. By combining the data of the Victorian fire in previous years, our teams think of dividing the importance of regions, setting different levels of attention in different regions.

Modern vehicles are using AI and increasingly sophisticated sensor suites to improve Advanced Driving Assistance Systems (ADAS) and support automated driving capabilities. Heads-Up-Displays (HUDs) provide an opportunity to visually inform drivers about vehicle perception and interpretation of the driving environment. One approach to HUD design may be to reveal to drivers the vehicle’s full contextual understanding, though it is not clear if the benefits of additional information outweigh the drawbacks of added complexity, or if this balance holds across drivers. We designed and tested an Augmented Reality (AR) HUD in an online study (N = 298), focusing on the influence of HUD visualizations on drivers’ situation awareness and perceptions. Participants viewed two driving scenes with one of three HUD conditions. Results were nuanced: situation awareness declined with increasing driving context complexity, and contrary to expectation, also declined with the presence of a HUD compared to no HUD. Significant differences were found by varying HUD complexity, which led us to explore different characterizations of complexity, including counts of scene items, item categories, and illuminated pixels. Our analysis finds that driving style interacts with driving context and HUD complexity, warranting further study.

Autonomous vehicles could improve mobility, safety, and inclusion in traffic. While this technology seems within reach, its successful introduction depends on the intended user’s acceptance. A substantial factor for this acceptance is trust in the autonomous vehicle’s capabilities. Visualizing internal information processed by an autonomous vehicle could calibrate this trust by enabling the perception of the vehicle’s detection capabilities (and its failures) while only inducing a low cognitive load. Additionally, the simultaneously raised situation awareness could benefit potential take-overs. We report the results of two comparative online studies on visualizing semantic segmentation information for the human user of autonomous vehicles. Effects on trust, cognitive load, and situation awareness were measured using a simulation (N=32) and state-of-the-art panoptic segmentation on a pre-recorded real-world video (N=41). Results show that the visualization using Augmented Reality increases situation awareness while remaining low cognitive load.

In various control systems, automation is implemented to increase efficiency and safety. With increased automation, it becomes increasingly relevant to view the automation as a team member, rather than as a tool. In best cases, human-automation teamwork keeps workload within acceptable limits, increase situation awareness, and keeps the operator in the control loop. However, human-automation teamwork will only flourish if the automation is developed with the human operator in mind. Therefore, investigations of the current experiences and expectations regarding automation and teamwork are important for the development of automation. Through a questionnaire among Air Traffic Controllers (ATCOs), the present study aims to investigate how ATCOs perceive automation and safety in current and future air traffic control systems and the importance of different teamwork factors for human-human and human-automation collaboration. The results indicate that the ATCOs believe that safety will increase in the future along with increasing automation as long as the automation is working as expected. The ATCOs expressed a fear of deskilling and losing situation awareness with automation, a fear associated with a new ATCO role of monitoring the system and take over when the automation fails. The results suggest that design for human-automation teamwork aspects that ATCOs value, such as adaptability or mutual performance monitoring, could be a way forward.

Surgical time-outs are designed to promote situation awareness, teamwork, and error prevention. The pre-incision time-out in particular aims to facilitate shared mental models prior to incision. Objective, unbiased measures to confirm its effectiveness are lacking. We hypothesized that providers’ mental workload would reveal team psychophysiological mirroring during a formal, well-executed pre-incision time-out. Heart rate variability was collected during cardiac surgery cases from the surgeon, anesthesiologist, and perfusionist. Data were analyzed for six cases from patient arrival until sternal closure. Annotation of surgical phases was completed according to previously developed standardized process models of aortic valve replacement and coronary artery bypass graft procedures, producing thirteen total surgical phases. Statistical analysis revealed significant main effects. Tukey HSD post hoc tests revealed significant differences across provider roles within various phases, including Anesthesia Induction, Heparinization, Initiation of Bypass, Aortic Clamp and Cardioplegia, Anastomoses or Aortotomy, Separation from Bypass, and Sternal Closure. Despite these observed differences between providers over various surgical phases, the Pre-incision Time-out phase revealed almost negligible differences across roles. This preliminary work supports the utility of the pre-incision safety checklist to focus the attention of surgical team members and promote shared team mental models, measured via psychophysiological mirroring, using an objective mental workload measure. Future studies should investigate the relationship between psychophysiological mirroring among surgical team members and the effectiveness of the pre-incision time-out checklist.

Teleoperations requires both a robust set of controls and the right balance of sensory data to allow task completion without overwhelming the user. Previous work has mostly focused on using depth cameras that fail to provide situational awareness. We have developed a teleoperation system that integrates 360° camera data in addition to the more standard depth data. We infer depth from the 360° camera data and use that to render it in VR, which allows for six degree of freedom viewing. We use a virtual gantry control mechanism, and also provide a menu with which the user can choose which rendering schemes to render the robot's environment with. We hypothesize that this approach will increase the speed and accuracy with which the user can teleoperate the robot.

One focus of augmented reality (AR) in robotics has been on enriching the interface for human-robot interaction. While such an interface is often made intuitive to interact with, it invariably imposes novel objects into the environment. In situations where the human already has a focus, such as in a human-robot teaming task, these objects can potentially overload our senses and lead to degraded teaming performance. In this paper, we propose using AR objects to solely augment natural objects to avoid disrupting our natural senses while adding critical information about the current situation. In particular, our case study focuses on addressing the limited field of view of humans by incorporating persistent virtual shadows of robots for maintaining situation awareness in proximal human-robot teaming tasks.

Edge computing is now widely applied in industrial IoT to enable fast response and resolution of services for end-user applications in the interconnected world of things. However, edge computing devices are highly vulnerable to attacks due to their proximity to ends with large amounts of private user information and rich digital assets. The purpose of this paper is to propose a situational awareness approach for edge computing devices to protect the security of edge devices. An example is implemented, and the results show that the situational awareness model can accurately detect the attacks encountered by edge devices.

Situational awareness is significant for cyber security, which can help researchers and security analysts obtain the network security situation comprehensively and accurately. Advanced Persistent Threat (APT) attack could cause severe consequences to cyberspace and detecting such attacks have become a very important part of cyber security situational awareness. Some APT attacks may belong to a same group, many countries and organizations have established databases for APT groups, such as adopting knowledge graph (KG) to represent the knowledge. However, cyberspace security knowledge varies by temporal and spatial characteristics, such as the attack technologies are updated very frequently, traditional KG cannot represent such knowledge timely. To address this problem, the MDATA (Multi-dimensional Data Association and inTelligent Analysis) model is proposed in [1], which is a supplement and improvement to traditional KG. In this paper, we introduce an APT group knowledge model based on MDATA, which adds spatial-temporal characteristics of the APT groups. We also analyze how this knowledge model could help address the challenges of APT attack awareness.

With the development of information technology, the scale of the network continues to expand, and the security issues continue to increase. The complexity of the network security situation is increasing and the importance of the network security situational awareness continues to increase. The data sources are wide, the type and the number are large in a large-scale network environment currently. This paper comprehensively studies the network security situational awareness technology based on multi-source heterogeneous data. In addition, this paper introduces the origin, concept and the model of the network situational awareness, as well as the characteristics of network security situational awareness based on multi-source heterogeneous data fusion. Finally, the key technologies in the security situational awareness such as the extraction of situational elements, multi-source data fusion, situation assessment, situation prediction and visual display are introduced.

situational awareness technology can effectively improve the operational efficiency of grid and strongly guarantee the safe and stable operation of grid. Its important forecasting and decision support ability are indispensable parts of the smart grid. This article integrates a variety of professional systems, including power marketing system and power acquisition system, combines seasonal weather, social and economic information, and establishes a typical industry power consumption matrix data set and anti-electricity stealing diagnosis model, and proposes a multi-dimensional intelligent situational awareness solution of electricity stealing, which will improve the level of intelligent situational awareness of electricity stealing.

In recent years, both electronic jamming and electronic anti-jamming technologies have been greatly developed.Since traditional electronic jamming system is a static interference strategy, it has only several fixed interference modes, resulting the jamming strategy could'nt be changed autonomously as the environment changes. The COV19 virus that broke out in 2019 has spread across the world, causing many people to die. Viruses invading the human body also have the similar function to electronic jamming. In this paper, we transfer the mechanism of viruses invading the human immune system to the field of electronic jamming, acheaving a more intelligent autonomous electronic jamming strategy. The new modern cognitive electronic jamming system could realize the real-time situational awareness and make much more intelligent electronic strategy.

When a natural disaster occurs, emergency responders need information and real-time imagery in order to make better decisions and save time. Unmanned aerial vehicles (UAVs) used by emergency services, such as police officers or firefighters, can rapidly provide situational awareness over a large area, reducing the time and number of researchers required to locate and rescue people, thus reducing the costs and risks of search and rescue missions. This work considers the problem of completely overfly an area just affected by an earthquake with the objective of opportunely direct rescue teams. It provides a complete model that tries to keep into account all the main real-life issues in two different realistic scenarios.

International audience; The use of robotic surgical systems disrupts existing team dynamics inside operating rooms and constitutes a major challenge for the development of crucial non-technical skills such as situation awareness (SA). Techniques for assessing SA mostly rely on subjective assessments and questionnaires; few leverage multimodal measures combining physiological, behavioural, and subjective indicators. We propose a conceptual model relating SA with mental workload, stress and communication, supported by measurable behaviours and physiological signals. To validate this model, we collect subjective, behavioural, and physiological data from surgical teams performing radical prostatectomy using robotic surgical systems. Statistical analyses will be performed to establish relationships between SA, subjective assessment of stress and mental workload, communication processes, and the surgeons' physiological signals.

We demonstrate AHOM, a system that can Actively Observe Hazards via Monitoring Social Media Streams. AHOM proposes an active way to include the human in the loop of hazard information ac-quisition for social media. Different from state of the art, it supports bi-directional interaction between social media data processing system and social media users, which leads to the establishment of deeper and more accurate situational awareness of hazard events. We demonstrate how AHOM utilizes Twitter streams and bi-directional information exchange with social media users for enhanced hazard observation.

Humans interact with other humans or the world through information from various channels including vision, audio, language, haptics, etc. To simulate intelligence, machines require similar abilities to process and combine information from different channels to acquire better situation awareness, better communication ability, and better decision-making ability. In this talk, we describe three projects. In the first study, we enable a robot to utilize both vision and audio information to achieve better user understanding [1]. Then we use incremental language generation to improve the robot's communication with a human. In the second study, we utilize multimodal history tracking to optimize policy planning in task-oriented visual dialogs. In the third project, we tackle the well-known trade-off between dialog response relevance and policy effectiveness in visual dialog generation. We propose a new machine learning procedure that alternates from supervised learning and reinforcement learning to optimum language generation and policy planning jointly in visual dialogs [2]. We will also cover some recent ongoing work on image synthesis through dialogs.

In many intelligent systems, a network of agents collaboratively perceives the environment for better and more efficient situation awareness. As these agents often have limited resources, it could be greatly beneficial to identify the content overlapping among camera views from different agents and leverage it for reducing the processing, transmission and storage of redundant/unimportant video frames. This paper presents a consensus-based distributed multi-agent video fast-forwarding framework, named DMVF, that fast-forwards multi-view video streams collaboratively and adaptively. In our framework, each camera view is addressed by a reinforcement learning based fast-forwarding agent, which periodically chooses from multiple strategies to selectively process video frames and transmits the selected frames at adjustable paces. During every adaptation period, each agent communicates with a number of neighboring agents, evaluates the importance of the selected frames from itself and those from its neighbors, refines such evaluation together with other agents via a system-wide consensus algorithm, and uses such evaluation to decide their strategy for the next period. Compared with approaches in the literature on a real-world surveillance video dataset VideoWeb, our method significantly improves the coverage of important frames and also reduces the number of frames processed in the system., To appear at ACM Multimedia 2020

Using a smartphone while walking in urban traffic is dangerous. Pedestrians might become distracted and have to split their attention between traffic, walking and using the mobile device. The increasing level of automation in vehicles introduces novel challenges and opportunities for pedestrian-vehicle interaction, e.g external displays attached to automated vehicles. However, these approaches are hardly scalable and fail to provide clear information in a multi-user environment. We investigate whether a smartphone app could provide individual guidance to enhance pedestrian safety in future traffic. To this end, we tested three app concepts in a user study (N=24) and found that on-screen guidance increases the frequency of successful crossing decisions significantly. In addition, all participants indicated that they would use the proposed system, preferably with an unobtrusive colored bar indicating the safest crossing decision in real-time. Integrating smartphones into the interaction between vehicles and pedestrians could increase situational awareness while crossing roads, solve the scalability problem and thus foster pedestrian safety in future traffic scenarios.

The organization is experiencing higher risk in running their business today due to the rapid rise of malware threats. It is critical for the organization to understand and aware of the level of threats that the emerging malware is capable of. This research introduces Mal-TScore: a threat score framework based on features extracted from static and dynamic analysis features. The resulting threat score could help the organization to reinforce the strategic and decision making process to improve cyber situational awareness of the emerging malware threats.

Autonomous Vehicles (AVs) are facilitating the development of a diverse set of applications, from human-less delivering to alternative mobility services. There are a variety of challenges for AVs that might be assessed and solved by remote operators, such as sensor data ambiguity, temporary changes to infrastructure, or unexpected interventions by other road users. With this paper, we propose a design space to support the development of appropriate user interfaces for remote situational awareness and teleoperation. The design space is envisioned to discover and evaluate design alternatives before the implementation and to provide a systematic approach towards the development of novel interfaces.

Augmented Reality (AR) headsets are being employed in industrial settings (e.g., the oil industry); however, there has been little work on how information should be presented in these headsets, especially in the context of situational awareness. We present a study examining three different presentation styles (Display, Environment, Mixed Environment) for textual secondary information in AR headsets. We found that the Display and Environment presentation styles assisted in perception and comprehension. Our work contributes a first step to understanding how to design visual information in AR headsets to support situational awareness.

As automated systems continue to be integrated in everyday vehicles, drivers have an opportunity to engage in non-driving related tasks (NDRTs) while the automation is responsible for the driving task. However, current research on NDRTs is limited and has not explored the use of virtual reality (VR) in an automated vehicle. To understand how this technology may be implemented in this environment, it is critical to investigate related constructs like situation awareness, perceived risk, or presence. However, current measures for these constructs are not suitable for use in VR. This workshop aims to address this gap bringing together researchers to begin the development of new measures for these constructs. Creating new measures is the first step toward effectively and accurately assessing the use of VR in the automated vehicle context.

Despite the rapid progress of Autonomous Vehicle (AV) technology, remote human situational assessment continues to be required. Herewith, remote operation is introducing several challenges, as limited perception and difficulty in maintaining Situation Awareness (SA). In this regard, this research provides first-hand SA requirements for remote teleoperation-based interfaces. Complementary to a previous literature review on requirements for Human-Machine Interface for unmanned systems, we conducted two (N = 18, N = 10) user studies. To ascertain the views of the users, we employed two methodologies, in-depth interviews, and traditional statistical analysis to find out specific preferences. We collected a total of 80 statements that we could cluster over 12 categories, presenting a comprehensive overview of SA user requirements. The research is envisioned to be used by others as a tool to help the development of AV teleoperation-based interfaces.

In conditionally automated driving, drivers do not have to constantly monitor their vehicle but they must be able to take over control when necessary. In this paper, we assess the impact of instructions about limitations of automation and the presentation of context-related information through a mobile application on the situation awareness and takeover performance of drivers. We conducted an experiment with 80 participants in a fixed-base driving simulator. Participants drove for an hour in conditional automation while performing secondary tasks on a tablet. Besides, they had to react to five different takeover requests. In addition to the assessment of behavioral data (e.g. quality of takeover), participants rated their situation awareness after each takeover situation. Instructions and context-related information on limitations combined showed encouraging results to raise awareness and improve takeover performance.

Small unmanned autonomous vehicles (UAVs) swarms are becoming ubiquitous in a number of applications (e.g., surveying, monitoring, and situational awareness). Indoor environments may contain metal equipment that temporarily disrupts radio reception. During these momentary interruptions, a small UAV needs to be able to broadcast a 'heartbeat' to indicate that it is not damaged or lost. Considering alternative messaging modalities, we observe that light-based methods require line-of sight, which is not guaranteed when UAVs are moving through a cluttered environment, while a naive sound-based method is easily drowned out by the UAV's own loud motor and propeller noise. We present Bleep, a side-channel messaging system that modulates UAV motors sounds to enable multiple UAVs to communicate when radio is unavailable. Bleep accomplishes this by 1) embedding linear chirps within the PWM frequency while maintaining the controllability of the UAV and 2) decoding the received sounds utilizing chirp multiplication to enhance signal detection in loud motor noise. Our chirp-based encoding allows multiple simultaneous transmissions to be detected and decoded in the presence of loud sounds from multiple UAV motors. Through real-world experiments, we show that Bleep is able to achieve over 95% signal detection and decoding accuracy with simultaneous transmissions.

People new to making and makerspaces often struggle with identifying what tools are available and where they are, understanding how to operate the tools, and predicting how their decisions will affect their final product. From literature on novices and our interviews with expert makers, we identified situation awareness support as one possible way to address some of the challenges faced by novices. We present a set of design goals intended to scaffold situation awareness in a makerspace, and MakeAware, a prototype system we implemented based on those design goals. MakeAware provides a combination of environmental cues, information about the project process, and background knowledge. In a preliminary evaluation, we found MakeAware can help novices make conscious choices during a project and put more emphasis on planning, thereby exhibiting traits associated with having situation awareness while making.

Cycling offers significant health and environmental benefits, but safety remains a critical issue. We need better tools and design processes to develop on-bicycle notification interfaces, for example, for hazard warnings, and to overcome design challenges associated with the cycling context. We present a physical computing toolkit that supports the rapid exploration and co-design of on-bicycle interfaces. Physical plug-and-play interaction modules controlled by an orchestration interface allow participants to explore different tangible and ambient interaction approaches on a budget cycling simulator. The toolkit was assessed by analysing video recordings of two group design workshops (N=8) and twelve individual design sessions (N=12). Our results show that the toolkit enabled flexible transitions between ideation and out-of-the-box thinking, prototyping, and immediate evaluation. We offer insights on how to design physical computing toolkits that offer low-cost, 'good enough' simulation while allowing for free and safe exploration of on-bicycle notification interfaces.

Connected products and systems are becoming popular, but they seldom provide direct and intuitive communication to the users. In this study, we applied Disney's animation principles to design the expressivity with LED lights and speakers commonly embedded in electronic devices. We explored the subtle transitions of brightness and controlled the timing to compose individual and system-level behaviors with multiple devices. The designs were evaluated and improved through three iterations. In the main study, we recruited 16 designer participants to investigate whether lights and sounds could be intuitively interpreted as what the system wanted to convey. The result shows that group light behaviors could evoke meanings that are highly similar to the intents of the system. When the acoustic accompaniments were provided, participants could better perceive the presence of devices. We concluded with six sets of light behaviors that are capable of expressing smart devices and systems' intents intuitively and unobtrusively.

Autonomous vehicles are expected to arrive sooner than expected. Autonomous vehicles of higher automation rely on both on-board and on-road deployed sensory data. Advanced approaches to improve the situational awareness of autonomous vehicle suggest to implement federated learning, where the raw data need to be transferred from vehicles to edges or clouds and vice-versa. This consequently generates dynamically varying communication link demands. The envisioned new era of autonomous driving demands the strong interplay of key stakeholders like: city authorities and communication network providers. In this paper, we study this relationship, where the traffic efficiency on different road segments may be achieved by incentivizing the autonomous vehicles through better communication resources on alternate routes. We model profit functions of the involved stakeholder. To carryout experiments, we use real traffic data of 8 months, which were collected through sensors deployed at Ernst-Reuter-Platz, Berlin, Germany. We developed an extensive validation framework to validate the approach, which comprises of SUMO, network simulator, and contributed modules. Results show that proposed approach achieves the traffic efficiency and help network operators to use the under-utilized network resources on the alternate paths.