Search

One challenge of any smart city is the management of crowded events (concerts, protests, marathons, etc.). For civil servants in charge of management, in-advance attendance prevision, real-time situational awareness and its evolution forecasting are crucial to resource assignment. These massive events put under stress public resources, organization and safety of smart cities. In this paper, we describe an ongoing effort to model urban layout, sensors deployed, and citizen information (from social networks and smartphone application) to deal with these situations. We use the concept of a digital twin applied to a city by modelling different flows of information which are integrated with a 3D virtual representation with forecasting possibilities. The main contribution of this paper is the architecture proposed and GUI using the augmented virtuality concept. The main purpose of our proposal is to facilitate the knowledge of the situation and the management of this type of event., Uno de los retos de cualquier ciudad inteligente es la gestión de eventos multitudinarios (conciertos, protestas, maratones, etc.). Para los funcionarios encargados de la gestión, la previsión de asistencia con antelación, el conocimiento de la situación en tiempo real y la previsión de su evolución son cruciales para la asignación de recursos. Estos eventos masivos ponen en tensión los recursos públicos, la organización y la seguridad de las ciudades inteligentes. En este artículo, describimos un esfuerzo en curso para modelar el trazado urbano, los sensores desplegados y la información de los ciudadanos (procedente de las redes sociales y de las aplicaciones de los teléfonos inteligentes) para hacer frente a estas situaciones. Utilizamos el concepto de gemelo digital aplicado a una ciudad mediante la modelización de diferentes flujos de información que se integran con una representación virtual en 3D con posibilidades de previsión. La principal aportación de este trabajo es la arquitectura propuesta y la interfaz gráfica de usuario que utiliza el concepto de virtualidad aumentada. El objetivo principal de nuestra propuesta es facilitar el conocimiento de la situación y la gestión de este tipo de eventos.

Virtual Reality (VR) Head-Mounted Display (HMD) technology immerses a user in a computer generated virtual environment. However, a VR HMD also blocks the users’ view of their physical surroundings, and so prevents them from using their mobile phones in a natural manner. In this paper, we present a novel Augmented Virtuality (AV) interface that enables people to naturally interact with a mobile phone in real time in a virtual environment. The system allows the user to wear a VR HMD while seeing his/her 3D hands captured by a depth sensor and rendered in different styles, and enables the user to operate a virtual mobile phone aligned with their real phone. We conducted a formal user study to compare the AV interface with physical touch interaction on user experience in five mobile applications. Participants reported that our system brought the real mobile phone into the virtual world. Unfortunately, the experiment results indicated that using a phone with our AV interfaces in VR was more difficult than the regular smartphone touch interaction, with increased workload and lower system usability, especially for a typing task. We ran a follow-up study to compare different hand visualizations for text typing using the AV interface. Participants felt that a skin-colored hand visualization method provided better usability and immersiveness than other hand rendering styles.

The tamarind forest is a virtual reality (VR) installation that explores shifts of perception through a somatic experience in a virtual space. As part of a larger show (kimosphere no.4) that explored immersion and an entwinement of real and virtual spaces, The Tamarind forest experiments with augmented virtuality and invites the audience to pay attention to their physical body while taking a journey through a virtual forest. This paper describes the piece, the technology and the design process as well as the main takeaways and reflections.

Concurrent with autonomous robots, teleoperation gains importance in industrial applications. This includes remote intervention scenarios and human-robot-cooperation during complex or harmful tasks operated from a distance. For an efficient teleoperation, an interface achieving a good user experience and relieving workload of the operators is decisive. Whereas virtual reality allows a spatial integration, environment reconstruction is often based on point clouds in combination with pre-known object models. Challenges exist in spatial representation, through bandwidth limitations and sensor interferences. Furthermore, the potential of knowledge augmentation for workload improvement is often not fully utilized. To overcome these challenges, we present a hybrid environment reconstruction method, which is adaptive regarding the integration of different data sources as well as their derived semantic information levels. For evaluation, a user study is performed. Thereby known procedures to measure usability, immersion, user experience and operator workload serve for evaluation and comparison.

The intersection between music and Extended Reality (XR) has grown significantly over the past twenty years, amounting to an established area of research today. The use of XR technologies represents a fundamental paradigm shift for various musical contexts as they disrupt traditional notions of musical interaction by enabling performers and audiences to interact musically with virtual objects, agents, and environments. This article both surveys and expands upon the knowledge accumulated in existing research in this area to build a foundation for future works that bring together Music and XR. To this end, we created a freely available dataset of 260 publications in this space and conducted an in-depth analysis covering 199 works in the last decade. We conducted this analysis using a list of conceptual dimensions belonging to technical, artistic, perceptual and methodological domains. This review of the literature is complemented with a set of interviews with domain experts with the goal of establishing a definition for the emergent field of Musical XR, i.e., the field of music in Extended Realities. Based on the results of the conducted review, a research agenda for the field is proposed.

PurposeThis paper presents a reclassification of markers for mixed reality environments that is also applicable to the use of markers in robot navigation systems and 3D modelling. In the case of Augmented Reality (AR) mixed reality environments, markers are used to integrate computer generated (virtual) objects into a predominantly real world, while in Augmented Virtuality (AV) mixed reality environments, the goal is to integrate real objects into a predominantly virtual (computer generated) world. Apart from AR/AV classifications, mixed reality environments have also been classified by reality; output technology/display devices; immersiveness as well as by visibility of markers.Design/methodology/approachThe approach adopted consists of presenting six existing classifications of mixed reality environments and then extending them to define new categories of abstract, blended, virtual augmented, active and smart markers. This is supported with results/examples taken from the joint Mixed Augmented and Virtual Reality Laboratory (MAVRLAB) of the Ulster University, Belfast, Northern Ireland; the Abdus Salam International Centre for Theoretical Physics (ICTP), Trieste, Italy and Santasco SrL, Regio Emilia/Milan, Italy.FindingsExisting classification of markers and mixed reality environments are mainly binary in nature and do not adequately capture the contextual relationship between markers and their use and application. The reclassification of markers into abstract, blended and virtual categories captures the context for simple use and applications while the categories of augmented, active and smart markers captures the relationship for enhanced or more complex use of markers. The new classifications are capable of improving the definitions of existing simple marker and markerless mixed reality environments as well as supporting more complex features within mixed reality environments such as co-location of objects, advanced interactivity, personalised user experience.Research limitations/implicationsIt is thought that applications and devices in mixed reality environments when properly developed and deployed enhances the real environment by making invisible information visible to the user. The current work only marginally covers the use of internet of things (IoT) devices in mixed reality environments as well as potential implications for robot navigation systems and 3D modelling.Practical implicationsThe use of these reclassifications enables researchers, developers and users of mixed reality environments to select and make informed decisions on best tools and environment for their respective application, while conveying information with additional clarity and accuracy. The development and application of more complex markers would contribute in no small measure to attaining greater advancements in extending current knowledge and developing applications to positively impact entertainment, business and health while minimizing costs and maximizing benefits.Originality/valueThe originality of this paper lies in the approach adopted in reclassifying markers. This is supported with results and work carried out at the MAV Reality Laboratory of Ulster University, Belfast–UK, the Abdus Salam International Centre for Theoretical Physics (ICTP), Trieste-Italy and Santasco SrL, Regio Emilia, Milan–Italy. The value of present research lies in the definitions of new categories as well as the discussions of how they improve mixed reality environments and application especially in the health and education sectors.

This article discusses the interim results of research under the RFBR project No. 19-29-14153 "Fundamentals of transformation of the content and methods of general education as a result of the use of the technology of augmented virtuality by students (on the example of teaching computer science)", which analyzed the technology of augmented reality in relation to other immersive technologies, real and virtual educational materials, as well as significant features of specialized school education. The article highlights various approaches to supplementing the virtual space with real objects, as well as the areas of their application in the framework of teaching schoolchildren at the profile level.

In the paper, the author notes that the development of modern technologies, including artificial intelligence, unmanned transport, robotics, portable and embedded digital devices, already has a great impact on the daily life of a person and can fundamentally change the existing social order in the near future.Virtual reality as a technology was born in the cross-section of research in the field of three-dimensional computer graphics and human-machine interaction. The spectrum of mixed reality includes the real world itself, the one that is before our eyes, the world of augmented reality — an improved reality that results from the introduction of sensory data into the field of perception in order to supplement information about the surrounding world and improve the perception of information; the world of virtual reality, which is created using technologies that provide full immersion in the environment. In some studies, augmented virtuality is also included in the spectrum, which implies the addition of virtual reality with elements of the real world (combining the virtual and real world).The paper substantiates the conclusion that in the near future both the legislator and judicial practice will have to find a balance between the interests of the creators of virtual worlds and virtual artists exclusive control over their virtual works, on the one hand, and society in using these virtual works and their development, on the other hand. It is necessary to allow users to participate, interact and create new forms of creative expression in the virtual environment.The author concludes that a broader interpretation of the fair use doctrine should be applied in this area, especially for those virtual worlds and virtual objects that imitate the real world and reality. However, it is necessary to distinguish between cases where the protection of such objects justifies licensing and those where it is advisable to encourage unrestricted use of the results for the further development of new technologies.

In virtual environments, spatial discrepancies between visual and haptic scenes negatively impact user performance and experience. This paper shows how spatial discrepancies due to pose differences can occur in a haptic augmented virtuality system with an encountered-type haptic display. To mitigate this problem, we propose visual guidance, an algorithm that dynamically manipulates the visual scene to compensate for discrepancies. The effectiveness of this algorithm was verified in a pair of studies involving a button pressing task and spatial discrepancies between ±150 mm and ±40°. Experimental results show that discrepant trials using the technique yield error rates and a number of speed peaks (representing the number of targeting movements) that are comparable to those attained in trials with zero spatial discrepancy. This result was also achieved without requiring a dedicated adaptation or training process, ensuring the algorithm can be used immediately by users. A pair of follow-up studies also indicates the algorithm has little impact on subjective ratings of simulator sickness, suggesting that sporadic use of the algorithm will not negatively affect user’s experience of a virtual environment. We believe that the visual guidance algorithm presented in this paper can be used to create more useful and compelling experiences in various haptic training applications incorporating encountered-type haptic displays.

Mixed Reality, MR, is defined as the integration of real and virtual worlds to create an environment which allows co-existing of physical and digital objects enabling users to interact with both. MR is a cross of reality, virtual reality including augmented reality and augmented virtuality. With the advent of wireless technologies and ever-increasing integration of the virtual world with the real world, MR has gained more significance in the last 5 years. Also, because of its real-life applications in education, gaming, entertainment, medicine, and military. But no technology comes without risks and shortcomings. Thus, MR also faces many threats and attacks which are broadly classified into 5 types namely, input, output, data, user interaction and device protection some of which include latency, data collection, physical device threats, user threats etc. This paper presents a comprehensive analysis of these threats on MR and solutions to mitigate the effects of it.

© 2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., In this study, we focus on the egocentric segmentation of arms to improve self-perception in Augmented Virtuality (AV). The main contributions of this work are: $i$ ) a comprehensive survey of segmentation algorithms for AV; $ii$ ) an Egocentric Arm Segmentation Dataset (EgoArm), composed of more than 10, 000 images, demographically inclusive (variations of skin color, and gender), and open for research purposes. We also provide all details required for the automated generation of groundtruth and semi-synthetic images; $iii$ ) the proposal of a deep learning network to segment arms in AV; $iv$ ) a detailed quantitative and qualitative evaluation to showcase the usefulness of the deep network and EgoArm dataset, reporting results on different real egocentric hand datasets, including GTEA Gaze+, EDSH, EgoHands, Ego Youtube Hands, THU-Read, TEgO, FPAB, and Ego Gesture, which allow for direct comparisons with existing approaches using color or depth. Results confirm the suitability of the EgoArm dataset for this task, achieving improvements up to 40% with respect to the baseline network, depending on the particular dataset. Results also suggest that, while approaches based on color or depth can work under controlled conditions (lack of occlusion, uniform lighting, only objects of interest in the near range, controlled background, etc.), deep learning is more robust in real AV applications

В статье представлен анализ понятий «дополненная реальность» и «дополненная виртуальность» - компонентов континуума «виртуальность-реальность». Рассмотрены подходы к внедрению иммерсивных технологий в учебный процесс школы.

Beyond conventional automated tasks, autonomous robot capabilities aside to human cognitive skills are gaining importance. This comprises goods commissioning and material supply in intralogistics as well as material feeding and assembly operations in production. Deep learning-based computer vision is considered as enabler for autonomy. Currently, the effort to generate specific datasets is challenging. Adaptation of new components often also results in downtimes. The objective of this paper is to propose an augmented virtuality (AV) based RGBD data annotation and refinement method. The approach reduces required effort in initial dataset generation to enable prior system commissioning and enables dataset quality improvement up to operational readiness during ramp-up. In addition, remote fault intervention through a teleoperation interface is provided to increase operational system availability. Several components within a real-world experimental bin-picking setup serve for evaluation. The results are quantified by comparison to established annotation methods and through known evaluation metrics for pose estimation in bin-picking scenarios. The results enable to derive accurate and more time-efficient data annotation for different algorithms. The AV approach shows a noticeable reduction in required effort and timespan for annotation as well as dataset refinement.

Accidents resulting from poorly planned or set up work environments are a major concern within the construction industry. While traditional education and training of personnel offer well-known approaches for establishing safe work practices, serious games in virtual reality (VR) are used more often as a complementary approach for active, personalized learning experiences. Their designs yet have to take full advantage of how trainees can potentially manipulate and interact with virtual objects. In addition, little construction safety research has focused on generating and analyzing the inherent data that can be collected about the trainees’ actions in the virtual environment. The objective analysis of their performance in the virtual environment offers precise feedback to sensitize their work behavior later in real practice. This research presents a novel framework for the generation and assessment of the trainees’ data in augmented virtuality (AV). The proposed approach is tested in a virtual work environment consisting of multiple stages and hazards that are consistent within today’s construction sites and workshops. A real angle grinder has been reworked and repurposed as an interactive AV controller to further enhance immersion. Results on the performance in the proposed system and the experiences of two groups of volunteering participants are presented and discussed. An outlook presents future avenues towards enhancing existing construction safety education and focus points on correlating objective tracking data with self-assessment.

Актуальность проблемы исследования обусловлена нарастающим влиянием технологий виртуальной реальности и дополненной виртуальности на многие образовательные процессы. Новые средства развиваются стремительно. На смену традиционному процессу приходит высокотехнологичная среда, в которой учащиеся могут погрузиться в виртуальные миры, имитирующие различные процессы и явления, воспринимаемые с помощью органов чувств так, будто они происходят в реальной жизни. В связи с этим цель исследования — выявление дидактического потенциала технологий виртуальной реальности и дополненной виртуальности, а также роли преподавателя, использующего эти технологии в условиях широкой трансформации образования. Задачи исследования: 1) выявить наиболее важные преимущества описываемых технологий; 2) обосновать влияние технологий виртуальной реальности и дополненной виртуальности на восприятие учебной информации учащимися; 3) показать меняющуюся роль преподавателя в условиях использования высокотехнологичной образовательной среды., The relevance of the research problem is due to the growing influence of virtual reality and augmented virtuality technologies on many educational processes. New tools are developing rapidly. The traditional process is being replaced by a high-tech environment in which students can immerse themselves in virtual worlds that simulate various processes and phenomena perceived with the help of the senses as if they were happening in real life. In this regard, the purpose of the study is to identify the didactic potential technologies of virtual reality and augmented virtuality, as well as the role of a teacher using these technologies in the context of a broad transformation of education. Research objectives: 1) identify the most important advantages of the described technologies; 2) to substantiate the impact of virtual reality and augmented virtuality technologies on the perception of educational information by students; 3) show the changing role of the teacher in the context of using a high-tech educational environment., ВЕСТНИК МОСКОВСКОГО ГОРОДСКОГО ПЕДАГОГИЧЕСКОГО УНИВЕРСИТЕТА. СЕРИЯ: ИНФОРМАТИКА И ИНФОРМАТИЗАЦИЯ ОБРАЗОВАНИЯ, Выпуск 2 (60) 2022

In this paper we want to discuss the future potential of Serious Virtual Reality (VR) Simulations and VR Serious Games. This contribution is the foundation of the VENUS workshop presented on ISMAR 2022. We will provide insights into recent VR projects to lay a foundation for in-depth discussions with participants of the workshop. Furthermore, we want to shed light on needed research in the area of VR frameworks, further strengthening the subconscious transfer of knowledge through Serious Games and the involvement of tangible objects to increase immersion and presence.

Trabalho de projeto Submitted by Anabela Novais (anabelanovais@eseig.ipp.pt) on 2021-11-22T10:41:02Z No. of bitstreams: 1 DM_LuisaGuedes_2021.pdf: 12822398 bytes, checksum: 7a29ffae7f9ff37051cca0fe45460477 (MD5) Made available in DSpace on 2021-11-22T10:41:03Z (GMT). No. of bitstreams: 1 DM_LuisaGuedes_2021.pdf: 12822398 bytes, checksum: 7a29ffae7f9ff37051cca0fe45460477 (MD5) Previous issue date: 2021-10-29

Interaction and augmentation strongly depend on the context of use and the respective human aspects. In this paper, we present the use of Mixed Reality (MR) for the creation of experience prototypes and the evaluation of mobile applications in a (simulated) usage context. The novelty is the integration of real mobile devices into MR according to the augmented virtuality principle and the mirroring of the screen from the real device to the virtual replica. This preserves the user experience and the full range of functions when using the mobile device in MR allowing it to be operated as in reality. In the MR environment it is possible to simulate usage contexts and to investigate the user experience with mobile applications. First, we show the technical concept and then we demonstrate the approach with a prototype smartphone app for a mobility assistant in two scenarios.

New opportunities for immersive storytelling experiences have arrived through the technology in mobile phones, including the ability to overlay or register digital content on a user’s real world surroundings, to greater immerse the user in the world of the story. This raises questions around the methods and freedom to interact with the digital elements, that will lead to a more immersive and engaging experience. To investigate these areas the Augmented Virtuality (AV) mobile phone application Home Story was developed for iOS devices. It allows a user to move and interact with objects in a virtual environment displayed on their phone, by physically moving in the real world, completing particular actions to progress a story. A mixed methods study with Home Story either guided participants to the next interaction, or offered them increased agency to choose what object to interact with next. Virtual objects could also be interacted with in one of three ways; imagining the interaction, an embodied interaction using the user’s free hand, or a virtual interaction performed on the phone’s touchscreen. Similar levels of immersion were recorded across both study conditions suggesting both can be effective, though highlighting different issues in each case. The embodied free hand interactions proved particularly memorable, though further work is required to improve their implementation, arising from their novelty and lack of familiarity.

The paper presents the results of a study on improving the safety of navigation, during maneuvers in restricted areas, with the use of an augmented virtuality navigation information display. The augmented virtuality solution has been developed by one of the authors and has been described in a previous paper. A set of simulation scenarios has been proposed to study various maneuvers with different types of ships. Models of existing areas and existing ships have been used so the study can be recreated and reevaluated with different types of interfaces. The study is focused on comparing safe maneuvering areas with different information sources used. The results showed that augmented virtuality solution can indeed decrease the safe maneuvering areas and thus increase the safety of navigation for maneuvers in restricted areas.

International audience; Helicopter landing on a ship is a visually regulated "rendezvous" task during which pilots must use fine control to land a powerful rotorcraft on the deck of a moving ship tossed by the sea while minimizing the energy at impact. Although augmented reality assistance can be hypothesized to improve pilots' performance and the safety of landing maneuvers by guiding action toward optimal behavior in complex and stressful situations, the question of the optimal information to be displayed to feed the pilots' natural information-movement coupling remains to be investigated. Novice participants were instructed to land a simplified helicopter on a ship in a virtual reality simulator while minimizing energy at impact and landing duration. The wave amplitude and related ship heave were manipulated. We compared the benefits of two types of visual augmentation whose design was based on either solving cockpit-induced visual occlusion problems or strengthening the online regulation of the deceleration by keeping the current tau-dot variable around an ideal value of-0.5 to conduct smooth and efficient landing. Our results showed that the second augmentation, ecologically grounded, offers benefits at several levels of analysis. It decreases the landing duration, improves the control of the helicopter displacement, and sharpens the sensitivity to changes in tau-dot. This underlines the importance for designers of augmented reality systems to collaborate with psychologists to identify the relevant perceptual-motor strategy that must be encouraged before designing an augmentation that will enhance it.

Haptic augmented virtuality systems can provide users with a highly immersive haptic experience by visuo-haptic collocation of physical objects with their corresponding virtual objects. The collocation can be accomplished by an encounter-type haptic display system where a robotic device provides a physical prop that represents a whole or partial virtual object in real-time and in the space occupied by the virtual object in the virtual environment. For efficient and accurate visuo-haptic collocation, this paper proposes a novel yet simple, efficient and accurate calibration method. The method employs a high-accuracy position device such as an industrial robot that can place physical objects quickly and accurately. The transformation between the reference frames of the real (robot) and virtual worlds is obtained by the simple motion of the robot’s end-effector equipped with a real object that is in turn tracked by a VR tracker system such as the HTC VIVE tracker. An evaluation of the proposed method indicates that it is possible to achieve acceptable accuracy with a few data points as compared to existing calibration methods that require a large number of data points.

In this paper, we report findings from a systematic mapping study, conducted to review the existing literature on collaborative educational environments incorporating mixed reality technologies. There is increasing interest in mixed reality technologies in education, especially with the introduction of new over head mounted displays (OHMDs), such as HoloLens, Oculus Rift, and HTC Vive. with the consideration of areas, such as education, dynamic technology, and complex environments, a research area is identified. We carried out an extensive review of the literature from 2007 to 2017 and conducted an analysis of the works on mixed reality technologies and its subcategories applied to collaborative education environments. Results highlighted the lack of research across the mixed reality spectrum, especially in the augmented virtuality subcategory, as well as technical limitations such as response time in the development of mixed reality technologies for collaborative environments. Furthermore, the difficulty of teaching professionals to replicate mixed reality experiments in real environments, due to the technical skills required, was identified. The main contribution of this paper is the discussion of the current works with visualization of the present state of the area, which is aimed to encourage educators to develop mixed reality artefacts and conduct further research to support collaborative educational environments.

In order to be used for skill development and skill maintenance, virtual environments require accurate simulation of the physical phenomena involved in the process of the task being trained. The accuracy needs to be conveyed in a multimodal fashion with varying parameterisations still being quantified, and these are a function of task, prior knowledge, sensory efficacy and human perception. Virtual reality (VR) has been integrated from a didactic perspective in many serious games and shown to be effective in the pedological process. This paper interrogates whether didactic processes introduced into a VR serious game, by taking advantage of augmented virtuality to modify game attributes, can be effective for both beginners and experts to a task. The task in question is subjective performance in a clay pigeon shooting simulation. The investigation covers whether modified game attributes influence skill and learning in a complex motor task and also investigates whether this process is applicable to experts as well as beginners to the task. VR offers designers and developers of serious games the ability to provide information in the virtual world in a fashion that is impossible in the real world. This introduces the question of whether this is effective and transfers skill adoption into the real world and also if a-priori knowledge influences the practical nature of this information in the pedagogic process. Analysis is conducted via a between-subjects repeated measure ANOVA using a $$2 \times 2$$ 2 × 2 factorial design to address these questions. The results show that the different training provided affects the performance in this task ($$N=57$$ N = 57 ). The skill improvement is still evidenced in repeated measures when information and guidance is removed. This effect does not exist under a control condition. Additionally, we separate by an expert and non-expert group to deduce if a-priori knowledge influences the effect of the presented information, it is shown that it does not.

Mixed reality (MR) research aims to develop technologies that inputting or mixing the real world objects into computer generated three dimensional (3D) virtual space. The mixed reality, including augmented reality (AR) and augmented virtuality (AV), produces an environment in which the real objects are superimposed on user’s view of the virtual environment or the virtual objects are superimposed on user’s view of the real environment. Mixed reality has received a great deal of attention as a new method for displaying information or increasing the reality of virtual environments. Many research results have been reported and demonstrated [P. Biermann and B. Jung, 2004 Shintaro Ono et al., 2005]. Recently, some new assessments have been developed that such new model of virtual systems which are different from traditional form is efficient for exciting user’s sense [Yi Cai et al., 1997 Raphael Grasset, 2005]. However, most of such large-scale assessments still remain at the level of viewing computer graphics (CG)-generated virtual objects. It has revealed that such model of virtual system is no longer functional or fascinate based on virtual reality techniques only. The new, innovative, and mixed reality based approaches are required for operation or exhibition in 21st century. That means the new virtual reality model is focusing on cooperation-centred, real object-based operation. The features of such operation are mainly real or mixed, natural interactive, three dimensional with stereo vision, and collaborative. Mixed reality environments are defined by Milgram as those in which real world and virtual world objects are presented together on a single display [Milgram P. and Kishino, 1994]. The single user based mixed reality interfaces have been developed for computer aided instruction [S. Feiner et al., 1993], manufacture [Cruz-Neira et al., 1992] and medical visualization [Bajura et al., 1992]. Recently, Seon-Min et al. [Seon-Min et al., 2006] presents a method for merging a live video stream of multiple users into a shared virtual space. These applications have shown that mixed reality interfaces can enable users to interact with the real world in ways never before possible. Furthermore, the combination of mixed reality and network communication is becoming a more interesting research subject. Although mixed reality techniques have proven enough valuable in single user applications, there has been less research on group collaborative applications. We believe that mixed reality is ideal for collaborative interfaces because it addresses two major issues in three dimensional computer supported collaborative work

Background Mixed reality (MR), the computer-supported augmentation of a real environment with virtual elements, becomes ever more relevant in the medical domain, especially in urology, ranging from education and training over surgeries. We aimed to review existing MR technologies and their applications in urology. Methods A non-systematic review of current literature was performed using the PubMed-Medline database using the medical subject headings (MeSH) term “mixed reality”, combined with one of the following terms: “virtual reality”, “augmented reality”, ‘’urology’’ and “augmented virtuality”. The relevant studies were utilized. Results MR applications such as MR guided systems, immersive VR headsets, AR models, MR-simulated ureteroscopy and smart glasses have enormous potential in education, training and surgical interventions of urology. Medical students, urology residents and inexperienced urologists can gain experience thanks to MR technologies. MR applications are also used in patient education before interventions. Conclusions For surgical support, the achievable accuracy is often not sufficient. The main challenges are the non-rigid nature of the genitourinary organs, intraoperative data acquisition, online and multimodal registration and calibration of devices. However, the progress made in recent years is tremendous in all respects and the gap is constantly shrinking., Highlights ∙ MR, including AV and AR, is an intriguing technology with tremendous potential in urology field. ∙The main challenges lie in intraoperative data acquisition, online and multimodal registration and calibration of devices and data, appropriate display hardware, as well as cooperative devices and tools in the operation theatres. ∙Medical experts should feel encouraged to experience MR solutions and to communicate their specific needs and effects they aim at.

CAPES - Coordenação de Aperfeiçoamento de Pessoal de Nível Superior CEMIG - Companhia Energética de Minas Gerais CNPq - Conselho Nacional de Desenvolvimento Científico e Tecnológico FAPEMIG - Fundação de Amparo a Pesquisa do Estado de Minas Gerais A inspeção remota e o controle supervisório são requisitos críticos para fábricas modernas, vigilância de civis, sistemas de energia e outras áreas. Para reduzir o tempo da tomada de decisão, os operadores precisam de uma elevada consciência da situação em campo, o que implica em uma grande quantidade de dados a serem apresentados, mas com menor carga sensorial possível. Estudos recentes sugerem a adoção de técnicas de visão computacional para inspeção automática, e a Realidade Virtual (VR) como uma alternativa às interfaces tradicionais do SCADA. Entretanto, apesar de fornecer uma boa representação do estado da subestação, os ambientes virtuais carecem de algumas informações de campo, provenientes de câmeras e microfones. Como essas duas fontes de dados (VR e dispositivos de captura) não são integrados em uma única solução, perde-se a oportunidade de usar VR como uma ferramenta de inspeção remota conectada ao SCADA, durante a operação e rotinas de respostas a desastres. Este trabalho trata de um método para aumentar ambientes virtuais de subestações com imagens de campo, permitindo aos operadores a rápida visualização de uma representação virtual do entorno da área monitorada. O ambiente resultante é integrado com uma máquina de inferência estados por imagens, comparando continuamente os estados inferidos com aqueles reportados pela base SCADA. Na ocasião de uma discrepância, um alarme é gerado e possibilita que a câmera virtual seja imediatamente teletransportada para a região afetada, acelerando o processo de retomada do sistema. A solução se baseia em uma arquitetura cliente-servidor e permite múltiplas câmeras presentes em múltiplas subestações. Os resultados dizem respeito à qualidade do registro 2D–3D e à taxa de renderização para um cenário simples. As métricas quantitativas coletadas sugerem bons níveis de registro e estimativa de pose de câmera, além de uma taxa ótima de renderização para fins de inspeção de equipamentos em subestações. Remote inspection and supervisory control are critical features for smart factories, civilian surveillance, power systems, among other domains. For reducing the time to make decisions, operators must have both a high situation awareness, implying a considerable amount of data to be presented, and minimal sensory load. Recent research suggests the adoption of computer vision techniques for automatic inspection, as well as virtual reality (VR) as an alternative to traditional SCADA interfaces. Nevertheless, although VR may provide a good representation of a substation’s state, it lacks some real-time information, available from online field cameras and microphones. Since these two sources of information (VR and field information) are not integrated into one single solution, we miss the opportunity of using VR as a SCADA-aware remote inspection tool, during operation and disaster-response routines. This work discusses a method to augment virtual environments of power substations with field images, enabling operators to promptly see a virtual representation of the inspected area's surroundings. The resulting environment is integrated with an image-based state inference machine, continuously checking the inferred states against the ones reported by the SCADA database. Whenever a discrepancy is found, an alarm is triggered and the virtual camera can be immediately teleported to the affected region, speeding up system reestablishment. The solution is based on a client-server architecture and allows multiple cameras deployed in multiple substations. Our results concern the quality of the 2D–3D registration and the rendering framerate for a simple scenario. The collected quantitative metrics suggest good camera pose estimations and registrations, as well as an arguably optimal rendering framerate for substations' equipment inspection. Tese (Doutorado)