Search

We present a solution to egocentric 3D body pose estimation from monocular images captured from downward looking fish-eye cameras installed on the rim of a head mounted VR device. This unusual viewpoint leads to images with unique visual appearance, with severe self-occlusions and perspective distortions that result in drastic differences in resolution between lower and upper body. We propose an encoder-decoder architecture with a novel multi-branch decoder designed to account for the varying uncertainty in 2D predictions. The quantitative evaluation, on synthetic and real-world datasets, shows that our strategy leads to substantial improvements in accuracy over state of the art egocentric approaches. To tackle the lack of labelled data we also introduced a large photo-realistic synthetic dataset. xR-EgoPose offers high quality renderings of people with diverse skintones, body shapes and clothing, performing a range of actions. Our experiments show that the high variability in our new synthetic training corpus leads to good generalization to real world footage and to state of theart results on real world datasets with ground truth. Moreover, an evaluation on the Human3.6M benchmark shows that the performance of our method is on par with top performing approaches on the more classic problem of 3D human pose from a third person viewpoint., 14 pages. arXiv admin note: substantial text overlap with arXiv:1907.10045

Today, neuronavigation is widely used in daily clinical routine to perform safe and efficient surgery. Augmented reality (AR) interfaces can provide anatomical models and preoperative planning contextually blended with the real surgical scenario, overcoming the limitations of traditional neuronavigators. This study aims to demonstrate the reliability of a new-concept AR headset in navigating complex craniotomies. Moreover, we aim to prove the efficacy of a patient-specific template-based methodology for fast, non-invasive, and fully automatic planning-to-patient registration. The AR platform navigation performance was assessed with an in-vitro study whose goal was twofold: to measure the real-to-virtual 3D target visualization error (TVE), and assess the navigation accuracy through a user study involving 10 subjects in tracing a complex craniotomy. The feasibility of the template-based registration was preliminarily tested on a volunteer. The TVE mean and standard deviation were 1.3 and 0.6 mm. The results of the user study, over 30 traced craniotomies, showed that 97% of the trajectory length was traced within an error margin of 1.5 mm, and 92% within a margin of 1 mm. The in-vivo test confirmed the feasibility and reliability of the patient-specific template for registration. The proposed AR headset allows ergonomic and intuitive fruition of preoperative planning, and it can represent a valid option to support neurosurgical tasks.

Summarization: Input methods for interaction in smartphone-based virtual and mixed reality (VR/MR) are currently based on uncomfortable head tracking controlling a pointer on the screen. User fixations are a fast and natural input method for VR/MR interaction. Previously, eye tracking in mobile VR suffered from low accuracy, long processing time, and the need for hardware add-ons such as anti-reflective lens coating and infrared emitters. We present an innovative mobile VR eye tracking methodology utilizing only the eye images from the front-facing (selfie) camera through the headset’s lens, without any modifications. Our system first enhances the low-contrast, poorly lit eye images by applying a pipeline of customised low-level image enhancements suppressing obtrusive lens reflections. We then propose an iris region-of-interest detection algorithm that is run only once. This increases the iris tracking speed by reducing the iris search space in mobile devices. We iteratively fit a customised geometric model to the iris to refine its coordinates. We display a thin bezel of light at the top edge of the screen for constant illumination. A confidence metric calculates the probability of successful iris detection. Calibration and linear gaze mapping between the estimated iris centroid and physical pixels on the screen results in low latency, real-time iris tracking. A formal study confirmed that our system’s accuracy is similar to eye trackers in commercial VR headsets in the central part of the headset’s field-of-view. In a VR game, gaze-driven user completion time was as fast as with head-tracked interaction, without the need for consecutive head motions. In a VR panorama viewer, users could successfully switch between panoramas using gaze. Presented on: ACM Transactions on Applied Perception

The main goal of this paper is to investigate the effect of different types of self-representations through floating members (hands vs. hands + feet), virtual full body (hands + feet vs. full-body avatar), walking fidelity (static feet, simulated walking, real walking), and number of tracking points used (head + hands, head + hands + feet, head + hands + feet + hip) on the sense of presence and embodiment through questionnaires. The sample consisted of 98 participants divided into a total of six conditions in a between-subjects design. The HTC Vive headset, controllers, and trackers were used to perform the experiment. Users were tasked to find a series of hidden objects in a virtual environment and place them in a travel bag. We concluded that (1) the addition of feet to floating hands can impair the experienced realism ( $$p=0.039$$ ), (2) both floating members and full-body avatars can be used without affecting presence and embodiment ( $$p>0.05$$ ) as long as there is the same level of control over the self-representation, (3) simulated walking scores of presence and embodiment were similar when compared to static feet and real walking tracking data ( $$p>0.05$$ ), and (4) adding hip tracking overhead, hand and feet tracking (when using a full-body avatar) allows for a more realistic response to stimuli ( $$p=0.002$$ ) and a higher overall feeling of embodiment ( $$p=0.023$$ ).

Low-cost virtual-reality (VR) head-mounted displays (HMDs) with the integration of smartphones have brought the immersive VR to the masses, and increased the ubiquity of VR. However, these systems are often limited by their poor interactivity. In this paper, we present GestOnHMD, a gesture-based interaction technique and a gesture-classification pipeline that leverages the stereo microphones in a commodity smartphone to detect the tapping and the scratching gestures on the front, the left, and the right surfaces on a mobile VR headset. Taking the Google Cardboard as our focused headset, we first conducted a gesture-elicitation study to generate 150 user-defined gestures with 50 on each surface. We then selected 15, 9, and 9 gestures for the front, the left, and the right surfaces respectively based on user preferences and signal detectability. We constructed a data set containing the acoustic signals of 18 users performing these on-surface gestures, and trained the deep-learning classification pipeline for gesture detection and recognition. Lastly, with the real-time demonstration of GestOnHMD, we conducted a series of online participatory-design sessions to collect a set of user-defined gesture-referent mappings that could potentially benefit from GestOnHMD.

An important aspect in safety–critical domains is Situational Awareness (SA) where operators consolidate data into an understanding of the situation that needs to be updated dynamically as the situation changes over time. Among existing measures of SA, only physiological measures can assess the cognitive processes associated with SA in real-time. Some studies showed promise in detecting cognitive states associated with SA in complex tasks using brain signals (e.g. electroencephalogram/EEG). In this paper, an analytical methodology is proposed to identify EEG signatures associated with SA on various regions of the brain. A new data set from 32 participants completing the SA test in the PEBL is collected using a 32-channel dry-EEG headset. The proposed method is tested on the new data set and a correlation is identified between the frequency bands of β ( 12 - 30 Hz ) and γ ( 30 - 45 Hz ) and SA. Also, activation of neurons in the left and right hemisphere of the parietal and temporal lobe is observed. These regions are responsible for the visuo-spatial ability and memory and reasoning tasks. Among the presented results, the highest achieved accuracy on test data is 67 % .

For more than three centuries we have been watching and studying microscopic phenomena behind a microscope. We discovered that cells live in a physical environment whose predominant factors are no longer those of our scale and for which we lack a direct experience and consequently a deep intuition. Here we demonstrate a new instrument which, by integrating holographic and virtual reality technologies, allows the user to be completely immersed in a dynamic virtual world which is a simultaneous replica of a real system under the microscope. We use holographic microscopy for fast 3D imaging and real-time rendering on a virtual reality headset. At the same time, hand tracking data is used to dynamically generate holographic optical traps that can be used as virtual projections of the user hands to interactively grab and manipulate ensembles of microparticles or living motile cells.

Virtual reality (VR) experiences are receiving increasing attention in education and training. Some VR setups can deliver immersive VR training (e.g., on multiple projected screens), while others can deliver nonimmersive VR training (e.g., on standard desktop monitors). Recently, consumer VR headsets make it possible to deliver immersive VR training with six-degrees-of-freedom tracking of trainees’ head as well as hand controllers, while most smartphones can deliver nonimmersive VR training without the need for additional hardware. Previous studies compared immersive and nonimmersive VR setups for training, highlighting effects on performance, learning, presence, and engagement, but no study focused on contrasting procedural training with (immersive) VR headsets and (nonimmersive) smartphones. This article conducts a comparison of these two VR setups in the aviation safety domain. The considered training concerned door opening procedures in different aircraft and included a virtual instructor. In addition, we compared the two VR setups with the traditional printed materials used in the considered domain, i.e., safety cards. Results show that both VR setups allowed gaining and retaining more procedural knowledge than printed materials, and led to higher confidence in performing procedures. However, only the VR headset was considered to be significantly more usable than the printed materials, and presence was higher with the VR headset than the smartphone. The VR headset turned out to be important also for engagement and satisfaction, which were higher with the VR headset than both the printed materials and the smartphone. We discuss the implications of these results.

This paper describes the possibilities of supporting the teaching of neural tissue biology and biophysics through experiments with a simple, commonly available electroencephalography headset. Data are transmitted over a Bluetooth virtual serial port and can be analyzed in several ways by students or used solely as a potential motivational factor for teaching otherwise challenging and abstract curriculum about the human brain.

This paper presents a method for endoscope’s autonomous positioning by a robotic endoscope holder for minimally invasive surgery. The method improves human-robot cooperation in robot-assisted surgery by allowing the endoscope holder to acknowledge the surgeon’s view projection and navigate the camera without manual control. The real-time prediction of next desired camera location is estimated using segmented instrument’s tip locations from endoscope video and surgeon’s attention focus given by tracked virtual reality headset. To tackle the issue of real-time surgical instrument segmentation for more precise instrument tip localization, we propose the YOLOv3 and ResNet Combined Neural Network. The method showed an 86.6% IoU across MICCAI’17 Endovis datasets with 30 frames per second processing speed. The proposed pipeline was implemented in ROS on Ubuntu with visualization running under Windows operating system in Unity3D. The simulation demonstrates the robotic arm, endoscope, and surgical environment visualized in 3D in the virtual reality headset to provide a stable view of the endoscope and improve the surgeon’s perception of the operating environment.

Brain Computer Interfaces (BCIs) allow a user to control a computer application by acquiring brain activity using Electro Enchephalo Gram (EEG). Non-invasive practical EEG headset available in the market brings more and more research in the field of BCI. BCI applications such as speller device is used both in entertainment field and health industry for typing using brain without using hands. There are a variety of paradigms available for designing a BCI based speller device. Here the primary design goals are to analyze the following parameters: speed in terms of the information transfer rate (ITR) and accuracy of word selection and increasing user friendliness. The common requirement for each method is stimulus design. In most cases the stimulus designs are responsible for the performance factors. This paper analyses the design of those stimuli used in the speller device and the associated Event Related Potential (ERP). The design parameters common to all the methodologies and performance parameters are also systematically discussed. The character selection time with each type of stimuli also has been compared.

This paper presents an Augmented Reality (AR) application that aims to facilitate the operator’s work in an industrial, human-robot collaboration environment with mobile robots. In such a flexible environment, with robots and humans working and moving in the same area, the ease of communication between the two sides is critical and prerequisite. The developed application provides the user with handy tools to interact with the mobile platform, give direct instructions to it and receive information about the robot’s and the broader system’s state, through an AR headset. The communication between the headset and the robot is achieved through a ROS based system, that interconnects the resources. The discussed tool has been deployed and tested in a use case inspired from the automotive industry, assisting the operators during the collaborative assembly tasks.

In this study, we propose the use of electroencephalography (EEG), electrooculography (EOG), and kinematic motion data captured through wearable sensors to classify emotional states, while individuals are playing a serious computer game (Whack-a-Mole). Twenty-one participants wore an OpenBCI headset and JINS MEME eyewear while playing the Whack-a-Mole game at three levels of difficulty. We used a variety of classifiers [i.e., a support vector machine (SVM), logistic regression (LR), random forest (RF), and ensemble classifier (EC)] to classify the participants’ emotional states based on their EEG, EOG, and kinematic motion data. The classifiers were trained using the International Affective Picture System (IAPS). The EC and RF showed the best results in terms of their overall performance. Using tenfold cross-validation for all the subjects, the accuracies obtained were 73% for Arousal and 80% for Valence. Our results suggest that EEG and EOG biosignals, as well as kinematic motion data acquired using off-the-shelf wearable sensors in combination with machine-learning techniques such as EC, can be used to classify emotional states, while the individuals were playing the Whack-a-Mole game.

The software keyboard is widely used on digital devices such as smartphones, computers, and tablets. The software keyboard operates via touch, which is efficient, convenient, and familiar to users. However, some emerging technology devices such as AR/VR headsets and smart TVs do not support touch-based text entry. In this paper, we present QwertyRing, a technique that supports text entry on physical surfaces using an IMU (Inertial Measurement Unit) ring. Users wear the ring on the middle phalanx of the index finger and type on any desk-like surface, as if there is a QWERTY keyboard on the surface. While typing, users do not focus on monitoring the hand motions. They receive text feedback on a separate screen, e.g., an AR/VR headset or a digital device display, such as a computer monitor. The basic idea of QwertyRing is to detect touch events and predict users' desired words by the orientation of the IMU ring. We evaluate the performance of QwertyRing through a five-day user study. Participants achieved a speed of 13.74 WPM in the first 40 minutes and reached 20.59 WPM at the end. The speed outperforms other ring-based techniques [24, 30, 45, 68] and is 86.48% of the speed of typing on a smartphone with an index finger. The results show that QwertyRing enables efficient touch-based text entry on physical surfaces.

Virtual Reality (VR) has shown promising potentials in many applications, such as e-business, healthcare, and social networking. Rich information regarding user's activities and their online accounts is stored in VR devices. If it is carelessly unattended, then attackers, including insiders, can make use of the stored information to, for example, perform in-app purchases at the legitimate owner's expenses. Current solutions, mostly following schemes designed for general personal devices, have been proved vulnerable to shoulder-surfing attacks due to the sight blocking caused by the headset. Although there have been efforts trying to fill this gap, they either rely on some highly advanced equipment, such as electrodes to read brainwaves, or introduce heavy cognitive load that has users perform a series of cumbersome authentication tasks. Therefore, an authentication method for VR devices that is robust and convenient is in dire need. In this paper, we present the design, implementation, and evaluation of a two-factor user authentication scheme, BlinKey, for VR devices that are equipped with an eye tracker. A user's secret passcode is a set of recorded rhythms when he/she blinks, together with the unique pupil size variation pattern. We call this passcode as a blinkey, which can be jointly characterized by knowledge-based and biometric features. To examine the performances, BlinKey is implemented on an HTC Vive Pro with a Pupil Labs eye tracker. Through extensive experimental evaluations with 52 participants, we show that our scheme can achieve the average EER as low as 4.0% with only 6 training samples. Besides, it is robust against various types of attacks. BlinKey also exhibits satisfactory usability in terms of login attempts, memorability, and impact of user motions. We also carry out questionnaire-based pre-/post-studies. The survey result indicates that BlinKey is well accepted as a user authentication scheme for VR devices.

The Microsoft HoloLens, a mixed reality head-mounted display (HMD), has been demonstrated in domains such as medicine, engineering, and manufacturing. In order to interact with the device, voice input may be required. Given this range of environments, it is necessary to understand the impact of noise on voice dictation speed and accuracy. In this study, we evaluated the dictation feature of the HoloLens through speed (WPM), accuracy (WER), perceived workload, and perceived usability at three different noise levels: 40 dB, 55 dB, and 70 dB. No differences were found across noise levels in speed (67-75 WPM), or perceived workload. Accuracy and perceived usability worsened in the 70 dB noise condition. Only 37.5% of participants were able to successfully dictate in the 70 dB condition. This study shows that if the HoloLens is to be accepted in environments with high noise levels, improvements to dictation need to be made.

nondestructive evaluation (NDE) is an analysis technique used to evaluate the properties of a material, component, structure or system without causing damage. In this paper, we introduce a novel mixed reality system for NDE training. In particular, we model and simulate the inspected object and the inspection probe. The operator trainees with the wearable headsets are able to move and zoom the inspected object in a mixed environment, i.e., reality with virtual objects overlaid. In addition, the trainees use their gaze, gesture, and voice to control and interact with the virtual objects within the NDE training session. They can also access the help manual in order to follow the training instruction. The system is successfully operated on HoloLens, the state-of-the-art mixed reality headset. Evaluational results demonstrate that the use of mixed reality training provides significant benefit for the potential technician trainees.

New generation communication technologies and advanced deep learning models present a tremendous opportunity to develop fast, accurate, and seamless distributed systems in different sectors including the healthcare sector. in this article, we suggest a smart healthcare framework consisting of a pathology detection system, which is developed using deep learning. The pathology can be detected from electroencephalogram signals of a subject. in the framework, a smart EEG headset captures EEG signals and sends them to a mobile edge computing server. The server preprocesses the signals and transmits them to a cloud server. The cloud server does the main processing using deep learning and decides on whether the subject has pathology or not. Clients and stakeholders of the framework are connected via an authentication manager located in the cloud server. Experiment results on a publicly available database confirm the appropriateness of the proposed framework.

Communications inside an aircraft cockpit are currently based on wired connections especially for the audio headsets used by the pilots. A wireless headset would be an advantage in terms of comfort and flexibility but the use of classical radio frequencies is limited by interference and security issues. Optical wireless communication technology is an option for headset connectivity. Indeed, as optical beams are confined, this technology provides robustness against the risk of hacking, thus increasing security. In addition, the use of optical waves ensures the absence of radio-frequency disturbances. Using simulation, this article presents a thorough study of the optical wireless channel behavior inside the cockpit of an aircraft by considering a headset worn by a pilot possibly in motion and an access point at the ceiling. The impact of the characteristics of the environment model, such as the level of geometric description, the reflectivity of materials and for the first time, the ambient noise induced by the sun, is highlighted. System performance is evaluated in terms of optimal half-power angles and the necessary optical power of the light sources.

In this paper we evaluate the accuracy, recording interference, and articulatory quality of two different ultrasound probe stabilization headsets: a metallic Ultrasound Stabilisation Headset (USH) and UltraFit, a recently developed headset that is 3D printed in Nylon. To evaluate accuracy, we recorded three native speakers of German with different head sizes using an optical marker tracking system that provides sub-millimeter tracking accuracy (NaturalPoint OptiTrack Expression). The speakers had to read C1V1C2V1/2 non-words (to diminish lexical influences) in three conditions: wearing the USH headset, wearing the UltraFit headset, and without a headset. To estimate the relative headset movement, we measured the movement between tracked points on the probe, headset, and speaker’s nose. By also tracking visual marker points on the speaker’s lip and chin, we compared the movement of the outer articulators with and without a headset and, thereby, measured how the headsets interfere with the articulatory space of the speaker. Additionally, we computed the differences in tongue profiles at the acoustic midpoint of V1 under the three conditions and evaluated the articulatory recording quality with a distance index and an area index. In the final evaluation, we also compared formant measurements of recordings with and without headsets. With this objective evaluation we provide a systematic analysis of different headsets for Ultrasound Tongue Imaging (UTI) and also contribute to the discussion of using UTI stabilization headsets for recording natural speech. We show that both headsets have a similar accuracy, with the USH performing slightly better overall but introducing the largest error for one speaker, and that the UltraFit headset shows more flexibility during recordings. Each headset influences the lip opening differently. Concerning the tongue movement, there are no significant differences between different sessions, showing the stability of both headsets during the recordings. Acoustic analysis of formant differences in vowels revealed that the USH headset has a larger influence on formant production than the UltraFit headset.

Eye-tracking with gaze estimation is a key element in many applications, ranging from foveated rendering and user interaction to behavioural analysis and usage metrics. For virtual reality, eye-tracking typically relies on near-eye cameras that are mounted in the VR headset. Such methods usually involve an initial calibration to create a mapping from eye features to a gaze position. However, the accuracy based on the initial calibration degrades when the position of the headset relative to the users’ head changes; this is especially noticeable when users readjust the headset for comfort or even completely remove it for a short while. We show that a correction of such shifts can be achieved via 2D drift vectors in eye space. Our method estimates these drifts by extracting salient cues from the shown virtual environment to determine potential gaze directions. Our solution can compensate for HMD shifts, even those arising from taking off the headset, which enables us to eliminate reinitialization steps.

Background: The affective domain describes learning associated with feelings, values, and emotions. Although this domain was first described in the late 1960s, a significant emphasis on skills and knowledge still remains. This prominence is evident despite global concerns about standards of practice and the place for empathy and care in nursing education. The emergence of newer technologies, such as 360° video, provides an opportunity to revisit the design and delivery of affective learning. Method: Using the taxonomy of the affective domain, a 360° video viewed through a virtual reality headset was developed to give nursing and medical students insight on the patient experience and an orientation to the operating room. Results: Careful planning and consideration of how the affective domain can inform design of learning and teaching materials resulted in an engaging and informative learning experience. Being aware of how each level of the domain informs the next is essential to develop effective affective teaching plans. Conclusion: Affective learning traditionally has been difficult to deliver and assess. Technology such as 360° video provides an engaging, reproducible, and consistent platform for delivering a domain of education. [ J Nurs Educ . 2020;59(7):409–412.]

Brain-computer interface (BCI) technology represents a fast-growing field of research and applications for disabled and healthy people, which is a direct communication pathway to translate the neural information into an active command. Owing to the complicated headset structure, low accuracies, extended training periods, and nonstationary noises, BCI still has many challenges that should be dealt with for further facilitation of BCI technology use in daily life. In this study, a simplified synchronized hybrid BCI system is proposed for multiple command control by the electroencephalograph (EEG) signals in the motor cortex. This system can detect the single motor imagery (MI) task, single steady-state visually evoked potential (SSVEP) task, and hybrid MI + SSVEP tasks simultaneously (total ten mental tasks) via 2 EEG channels with high accuracy. The fast independent component analysis algorithm is employed to hybrid signals for obtaining clear EEG signals resulting from denoising. Feature extraction is performed by the wavelet transform, which is extracted by the features in the frequency and time domains. Furthermore, a four-layer convolutional neural network (CNN) is used as a classifier to distinguish different mental tasks. Finally, the hybrid MI + SSVEP system with a simple structure achieves a high accuracy of 95.56%. Additionally, the single MI-based and the SSVEP-based BCI system obtain the classification accuracy of 90.16% and 93.21%, respectively. Experimental results indicate that the synchronized hybrid BCI system could achieve multiple command control with a simple structure. In comparison with the single MI-based and the SSVEP-based BCI system, the hybrid MI + SSVEP BCI system shows a stable performance and higher efficiency. The proposed investigation provides a new method for the multiple command control by a hybrid BCI system. Also, the proposed BCI system offers the possibility of friendly utilization for disabled people because of its reliability, ease of use, and simplified headset structure.

Augmented reality technology is attracting increased attention, given the popularity of the smartphone. The utilisation of the smartphone with AR technology is also enabling mobile augmented reality (MAR) to become accessible. Stereoscopic vision offers users the benefit of depth perception, which can help improve user memory. Therefore, this study aims to investigate the effect on knowledge retention using MAR with stereoscopic vision. An experiment to compare a stereoscopic-based MAR application and a paper-based manual was designed to test the participants’ knowledge retention on a product part and disassembly process. The developed MAR adopted a smartphone and headset case to provide a stereoscopic view to the user. The experiment consisted of both pre-test and post-test phases where the pre-test phase examined the participant’s knowledge about the product, and the post-test phase focused on how much information the participants could recall. The post-test phase was conducted after 48 h following the pre-test phase since long-term memory required several hours in which to stabilise. The results showed that the MAR group had an advantage over the paper-based manual group for both information retentions having a mean score of 8.89 from the MAR group, compared with 6.33 for the paper-based manual group. Moreover, the MAR group was observed to have fewer errors in the post-test with a mean score of 0.53, whereas the paper-based achieved a score of 2.2. This result indicated that the MAR group and the paper-based manual group had equivalent performance in the completion time without the assistance tool. In addition, MAR had a better mean score for the subjective feedback (usefulness = 4.47, ease of use 4.18 and satisfaction = 4.07).

We present an end-to-end system for real-time environment capture, 3D reconstruction, and stereoscopic view synthesis on a mobile VR headset. Our solution allows the user to use the cameras on their VR headset as their eyes to see and interact with the real world while still wearing their headset, a feature often referred to as Passthrough. The central challenge when building such a system is the choice and implementation of algorithms under the strict compute, power, and performance constraints imposed by the target user experience and mobile platform. A key contribution of this paper is a complete description of a corresponding system that performs temporally stable passthrough rendering at 72 Hz with only 200 mW power consumption on a mobile Snapdragon 835 platform. Our algorithmic contributions for enabling this performance include the computation of a coarse 3D scene proxy on the embedded video encoding hardware, followed by a depth densification and filtering step, and finally stereoscopic texturing and spatio-temporal up-sampling. We provide a detailed discussion and evaluation of the challenges we encountered, as well as algorithm and performance trade-offs in terms of compute and resulting passthrough quality.;AB@The described system is available to users as the Passthrough+ feature on Oculus Quest. We believe that by publishing the underlying system and methods, we provide valuable insights to the community on how to design and implement real-time environment sensing and rendering on heavily resource constrained hardware.

Almost all smartphone-based personal sound amplification applications (PSAPs) do not provide amplification specifications as well as medical hearing aid (HA) products. Therefore, the main purpose of this study is to develop an HA-simulator and investigate the possible fitting range of this smartphone-based PSAP app equipped with a commercial headset, and to evaluate whether the aided outcome provides reasonable benefits for individuals with hearing loss by analyzing the Speechmap and speech intelligibility index (SII). An HA-simulator with a four-channel compression structure was developed and evaluated using a smartphone equipped with Apple EarPods. Measurement methods specified by the ANSI S3.22 were used to evaluate the electroacoustic performance of the proposed system. The Speechmap method and SII were used to check if the amplified speech for five different hearing loss configurations. These ANSI S3.22 and Speechmap measurement results were then compared with a digital HA under the same test conditions. The results demonstrated that the proposed HA-simulator system provided an adequate output sound level for patients with mild-to-moderate hearing loss and a tolerable group delay for the user while providing an electroacoustic performance comparable to that of a digital HA based on ANSI S3.22 test criteria. The amplified speech signals presented by Speechmap and the corresponding SII values indicated that the proposed HA-simulator provided benefits comparable to those of medical HA. These results indicate that a smartphone-based HA simulator could provide suitable performance to that of a medical HA, and it potentially aid individuals with hearing loss.

International audience; This paper focuses on the dimensioning of a very bright full color 10 μm-pitch light-emitting device (LED) microdisplay for avionics application. Starting from the specifications of head-mounted display to be used in an augmented reality optical system, a theoretical approach is proposed that enables predicting the specifications of the main technology building blocks enteringinto the microdisplay manufacturing process flow. By taking into account various material and technological parameters, kept as realistic as possible, it is possible to assess the feasibility of a very bright LED microdisplay (1 Mcd/m2 full white) and to point out the main limitations. The theoretical specifications are then compared with the technical results obtained so far in the framework of the H2020 Clean Sky “HILICO” project. It shows that 350 000 cd/m2 of white emission may be accessible with the present gallium nitride (GaN)-micro-LED technology provided a color conversion solution with stable external quantum efficiency of 30% is available. Beyond such level of luminance, the inherent limitations of driving circuit (4 V, 15 μA per pixel) commandsworking with materials enabling higher external quantum efficiency (EQE). In particular, 10-μm-pitch micro-LEDs with electroluminescence EQE of 15% and color conversion EQE approaching 60% are needed, opening the way to future challenging material and technology research developments.

PurposeThe purpose of this paper is to provide an insight into how augmented reality (AR) technologies are being applied to robotics.Design/methodology/approachFollowing an introduction and a brief historical background to AR, this first provides examples of AR applications in robot programming. It then gives examples of recent research into AR-based robot teleoperation. Research activities involving the virtual fixtures (VF) technique are then discussed and finally, brief conclusions are drawn.FindingsBecause AR concepts were first investigated in the 1990s, applications involving robotics have been widely studied. Programming with the aid of AR devices, such as the HoloLens headset, can be simplified and AR methods, including the VF technique, can improve the accuracy and speed of teleoperation, manipulation and positional control tasks. They can also provide visual or haptic feedback which leads to more intuitive operation and significantly reduces the cognitive load on the operator.Originality/valueThis provides an insight into the growing role of AR in robotics by providing examples of recent research in a range of applications.

Emotion recognition from electroencephalography (EEG) has been an important research direction in affective brain–computer interactions (ABCI). An integration of EEG-based emotion recognition algorithms in ABCI can make the users experience more complete and engaging. In this paper, we propose a new data representation of electroencephalogram (EEG), which transforms 1D chain-like EEG vector sequences into 2D mesh-like matrix sequences. The mesh structure of the matrix at each time point corresponds to the electrodes’ location of EEG headset, which could better represent the spatial correlation of EEG signals among multiple physically adjacent electrodes. Then, the sliding window is used to divide the 2D matrix sequences into segments containing equal time points, and each segment is seen as an EEG sample integrating the temporal and spatial correlation of raw EEG recordings. We also propose both cascaded and parallel hybrid convolution recurrent neural networks to accurately predict the emotional category of each EEG sample. Extensive binary emotion classification experiments in valence and arousal are carried out on a large scale DEAP dataset (32 subjects, 9,830,400 EEG recordings). The experimental results demonstrate that the classification accuracies of both proposed hybrid networks on our spatial–temporal EEG representation achieve over 93%, which outperform the most recent baseline methods and other deep learning models in within-subject validation scenario. Our proposed methods and hybrid models effectively improve the accuracy and robustness of EEG emotion classification, which can be applied to develop affective BCI applications using consumer multi-electrode EEG headset.

This paper introduces an intelligent system that can manipulate an industrial robot using the electroencephalogram signals of human brains to perform collaborative assembly tasks. The system is initiated by capturing the brain signals using a wearable headset, and the signals are then filtered to remove any possible artifact. Consequently, the process continues by identifying the brain signals patterns using a classifier based on pre-recorded samples. The classifier’s output determines the proper matching of the robot command that is intended by the human. To validate the results, an industrial collaborative assembly scenario of a car manifold is examined as a case study.

Research in multimodal interfaces aims to provide immersive solutions and to increase overall human performance. A promising direction is to combine auditory, visual and haptic interaction between the user and the simulated environment. However, no extensive comparison exists to show how combining audiovisuohaptic interfaces would affect human perception and by extent reflected on task performance. Our paper explores this idea and presents a thorough, full-factorial comparison of how all combinations of audio, visual and haptic interfaces affect performance during manipulation. We evaluated how each combination affects the performance in a study ( $N=25$ ) consisting of manipulation tasks with various difficulties. The overall performance was assessed using both subjective measures, by assessing cognitive workload and system usability, and objective measurements, by incorporating time and spatial accuracy-based metrics. The results showed that regardless of task complexity, the combination of stereoscopic-vision with the virtual reality headset increased performance across all measurements by 40%, compared to monocular-vision from a generic display monitor. Besides, using haptic feedback improved outcomes by 10% and auditory feedback accounted for approximately 5% improvement.

In this article, we explore the performance of optical wireless technology for ensuring audio communications inside an aircraft cockpit. One advantage is that, unlike radio frequencies, opaque objects block optical signals and, therefore, signals cannot pass through walls. This can reduce security risks against eavesdropping and hacking of the physical layer, which is one of the main concerns in the aviation environment. However, optical wireless technology faces some issues, including range limitation and sensitivity to blockages. To study the achievable performances, we propose a modeling of the channels for the uplink and the downlink between the headsets of the four pilots of an Airbus A350 and the access point at the cockpit ceiling. A ray-tracing approach associated with a Monte-Carlo method takes into account the 3D geometric model of the cockpit, the presence of the pilots and their movements. We show that using spatial diversity for headset transceivers can improve performance. Using IEEE 802.11 medium access control mechanism to ensure multi-user communication, the approach highlights the trade-offs between power and delay for a successful communication, linked to the maximum achievable data rate for a given performance level.

A room-scale virtual reality (VR) configuration allows users to physically walk within an interactive area while reflecting their real-world motion in the virtual environment. Room-scale VR provides developers of serious games and virtual simulations the opportunity to develop more realistic applications that are highly interactive and engaging. Despite these benefits, room-scale VR hardware is generally more expensive due to motion tracking and headset costs, also requiring a potentially large open physical space for the player to walk around in. Here we provide a quantitative and qualitative comparison between the usability, performance, and engagement of traditional seated VR with a room-scale variation within the scope of a medical-based epidural preparation serious game. This is a first-step in examining the differences between the two configurations and determining whether the additional requirements and resources associated with a room-scale VR configuration are warranted. Although our quantitative results reveal limited differences, the room scale VR configuration led to higher immersion. Our qualitative results indicate that the hardware being used and the requirements imposed upon it by the desired fidelity of the simulation must be carefully considered; higher fidelity implies greater computational requirements that may not be readily available to the average computer user.

Voice-over-IP (VoIP) software are among the most widely spread and pervasive software, counting millions of monthly users. However, we argue that people ignore the drawbacks of transmitting information along with their voice, such as keystroke sounds—as such sound can reveal what someone is typing on a keyboard. In this article, we present and assess a new keyboard acoustic eavesdropping attack that involves VoIP, called Skype & Type ( S&T ). Unlike previous attacks, S&T assumes a weak adversary model that is very practical in many real-world settings. Indeed, S&T is very feasible, as it does not require (i) the attacker to be physically close to the victim (either in person or with a recording device) and (ii) precise profiling of the victim’s typing style and keyboard; moreover, it can work with a very small amount of leaked keystrokes. We observe that leakage of keystrokes during a VoIP call is likely, as people often “multi-task” during such calls. As expected, VoIP software acquires and faithfully transmits all sounds, including emanations of pressed keystrokes, which can include passwords and other sensitive information. We show that one very popular VoIP software (Skype) conveys enough audio information to reconstruct the victim’s input—keystrokes typed on the remote keyboard. Our results demonstrate that, given some knowledge on the victim’s typing style and keyboard model, the attacker attains top-5 accuracy of 91.7% in guessing a random key pressed by the victim. This work extends previous results on S&T , demonstrating that our attack is effective with many different recording devices (such as laptop microphones, headset microphones, and smartphones located in proximity of the target keyboard), diverse typing styles and speed, and is particularly threatening when the victim is typing in a known language.

Robotic laparoscopic surgery has provided various benefits, but during the surgery, the surgeons are experiencing uncomfortable positioning issue which leads to neck pain and back pain. So, for an enhanced & efficient ergonomic robot-assisted surgery, a novel virtual reality (VR)-based endoscope control system (ECS) is proposed in this research. The overall system in this study is composed of a da Vinci research kit (dVRK), 4-degree-of-freedom ECS, three-dimensional endoscope, and VR headset with a built-in attitude and heading reference system (AHRS) module. Also, the proposed VR headset could replace a stereo viewer in the dVRK which could reduce the size of surgical robot system. Furthermore, the three dimensional endoscope could be controlled by a built-in AHRS module in the VR headset. The proposed system has been verified with four novice volunteers. They showed rapid learning through the peg-transfer task. Additionally, a collision avoidance strategy for VR-based ECS control was developed and verified by performing computer-based simulations. The ergonomic VR-based ECS proposed in this research could greatly reduce surgeon’s pains in both neck and back which potentially reduce surgeon’s workload during the surgery.

Recent advances in technology bring about novel operating environments where the role of human participants is reduced to passive observation. While opening new frontiers in productivity and lifestyle, such environments also create hazards related to the inability of human individuals to maintain focus and concentration during passive control tasks. A passive brain-computer interface for monitoring mental attention states of human individuals (focused, unfocused, and drowsy) by using electroencephalographic (EEG) brain activity imaging and machine learning data analysis methods is developed in this work. An EEG data processing pipeline and a machine learning mental state detection algorithm using the Support Vector Machine (SVM) method were designed and compared with k-Nearest Neighbor and Adaptive Neuro-Fuzzy System methods. To collect 25 h of EEG data from 5 participants, a classic EEG headset was modified. We found that the changes in EEG activity in frontal and parietal lobes occurring at 1–5 Hz and 10–15 Hz frequency bands were associated with the changes in individuals’ attention state. We demonstrated the ability to use such changes to identify individuals’ attention state with 96.70% (best) and 91.72% (avg.) accuracy in experimental settings using a version of continuous performance task with SVM-based mental state detector. The findings help guide the design of future systems for monitoring the state of human individuals by means of EEG brain activity data.

Augmented Reality (AR) offers new capabilities for blurring the boundaries between physical reality and digital media. However, the capabilities of integrating web contents and AR remain underexplored. This paper presents an AR web browser with an integrated context-aware AR-to-Web content recommendation service named as A2W browser, to provide continuously user-centric web browsing experiences driven by AR headsets. We implement the A2W browser on an AR headset as our demonstration application, demonstrating the features and performance of A2W framework. The A2W browser visualizes the AR-driven web contents to the user, which is suggested by the content-based filtering model in our recommendation system. In our experiments, 20 participants with the adaptive UIs and recommendation system in A2W browser achieve up to 30.69% time saving compared to smartphone conditions. Accordingly, A2W-supported web browsing on workstations facilitates the recommended information leading to 41.67% faster reaches to the target information than typical web browsing.

The utilization of robotic systems has been increasing in the last decade. This increase has been derived by the evolvement in the computational capabilities, communication systems, and the information systems of the manufacturing systems which is reflected in the concept of Industry 4.0. Furthermore, the robotics systems are continuously required to address new challenges in the industrial and manufacturing domain, like keeping humans in the loop, among other challenges. Briefly, the keeping humans in the loop concept focuses on closing the gap between humans and machines by introducing a safe and trustworthy environment for the human workers to work side by side with robots and machines. It aims at increasing the engagement of the human as the automation level increases rather than replacing the human, which can be nearly impossible in some applications. Consequently, the collaborative robots (Cobots) have been created to allow physical interaction with the human worker. However, these cobots still lack of recognizing the human emotional state. In this regard, this paper presents an approach for adapting cobot parameters to the emotional state of the human worker. The approach utilizes the Electroencephalography (EEG) technology for digitizing and understanding the human emotional state. Afterwards, the parameters of the cobot are instantly adjusted to keep the human emotional state in a desirable range which increases the confidence and the trust between the human and the cobot. In addition, the paper includes a review on technologies and methods for emotional sensing and recognition. Finally, this approach is tested on an ABB YuMi cobot with commercially available EEG headset. publishedVersion

Virtual Reality (VR) headsets can open opportunities for users to accomplish complex tasks on large virtual displays using compact and portable devices. However, interacting with such large virtual displays using existing interaction techniques might cause fatigue, especially for precise manipulation tasks, due to the lack of physical surfaces. To deal with this issue, we explored the design of VXSlate, an interaction technique that uses a large virtual display as an expansion of a tablet. We combined a user’s head movements as tracked by the VR headset, and touch interaction on the tablet. Using VXSlate, a user head movements positions a virtual representation of the tablet together with the user’s hand, on the large virtual display. This allows the user to perform fine-tuned multi-touch content manipulations. In a user study with seventeen participants, we investigated the effects of VXSlate on users in problem-solving tasks involving content manipulations at different levels of difficulty, such as translation, rotation, scaling, and sketching. As a baseline for comparison, off-the-shelf touch-controller interactions were used. Overall, VXSlate allowed participants to complete the task with completion times and accuracy that are comparable to touch-controller interactions. After an interval of use, VXSlate significantly reduced users’ time to perform scaling tasks in content manipulations, as well as reducing perceived effort. We reflected on the advantages and disadvantages of VXSlate in content manipulation on large virtual displays and explored further applications within the VXSlate design space.

We propose a new method for speaker diarization that can handle overlapping speech with 2+ people. Our method is based on compositional embeddings [1]: Like standard speaker embedding methods such as x-vector [2], compositional embedding models contain a function f that separates speech from different speakers. In addition, they include a composition function g to compute set-union operations in the embedding space so as to infer the set of speakers within the input audio. In an experiment on multi-person speaker identification using synthesized LibriSpeech data, the proposed method outperforms traditional embedding methods that are only trained to separate single speakers (not speaker sets). In a speaker diarization experiment on the AMI Headset Mix corpus, we achieve state-of-the-art accuracy (DER=22.93%), slightly higher than the previous best result (23.82% from [3]).

In this work we introduce the Thinking Cap, a wearable system designed to resemble the Sorting Hat from Harry Potter, fitted with a commercially available Electroencephalography (EEG) headset and a Bluetooth speaker. The Thinking Cap can inform the user about their brain activity in real-time via a speaker. We designed and conducted a study with 48 elementary and middle schoolers to investigate the influence of a BCI device and perceived magic on the development of growth mindset in children. Our results suggest that interacting with the Thinking Cap has a positive impact on children's mindset, which was expressed through their communicated beliefs and task-based behaviors.

Virtual and augmented reality offer comparative performance in terms of remote usability testing to lab-based co-located settings. However, direct contact with a researcher is still required to provide setup, troubleshoot, and training. In this paper, we present Surrogate-Aloud as a remote ideation and usability method that establishes a surrogate relationship between participants and a facilitating researcher through video conferencing. The researcher wears a VR headset and shares their viewpoint through video conferencing to a remote participant, who applies think-aloud protocol to express movement and interaction commands to be executed by the researcher, alongside their thought process as they interact with virtual prototypes or scenarios. We conducted a preliminary study to evaluate the Surrogate-Aloud method for remote usability evaluation and ideation of a new instructional technique with volumetric recordings. Results show that Surrogate-Aloud leverages the surrogate’s technical expertise and enables sufficient capability to conduct truly remote usability evaluation and ideation.

Virtual reality (VR) technology is a promising tool in physical rehabilitation. Research indicates that VR-supported rehabilitation is beneficial for task-specific training, multi-sensory feedback, diversified rehabilitation tasks, and patient motivation. Our first goal was to create a biomechatronics laboratory with a VR setup for increasing immersion and a motion platform to provide realistic feedback to patients. The second goal was to investigate possibilities to replicate features of the biomechatronics laboratory in a home-based training system using commercially available components. The laboratory comprises of a motion platform with 6-degrees-of-freedom (Rexroth eMotion), fitted with a load cell integrated treadmill, and an Oculus Quest virtual reality headset. The load cells provide input for data collection, as well as VR motion control. The home-based rehabilitation system consists of a Nintendo Wii Balance Board and an Oculus Rift virtual reality headset. User studies in the laboratory and home environment used direct observation techniques and self-reported attitudinal research methods to assess the solution’s usability and user experience. The findings indicate that the proposed VR solution is feasible. Participants using the home-based system experienced more cybersickness and imbalance compared to those using the biomechatronics laboratory solution. Future studies will look at a setup that is safe for first patient studies, and exercises to improve diagnosis of patients and progress during rehabilitation.

We carried out an experiment to examine emotional effect associated with images displayed in small field of view (FOV) telescope like virtual screen environment with VR-HMDs. We manipulate FOV of a virtual camera in VR to change the view of contents. Decreasing FOV makes view of angle narrow, and therefore, the view in VR looks like telescope, in which environment the view greatly moves with even small HMD movement. With smaller value of FOV, the view in VR is more sensitive to HMD movement and requires subjects to keep concentrating not to move their heads to watch target contents properly. Stronger emotional responses were observed with smaller value of FOV in the experiment. The result indicates the possibility of controlling effectiveness of the same contents with the same VR-HMD. This result is expected to improve a wide range of medical usage of VR contents.

Despite its marked success in recent years, it is still not clear how Virtual Reality (VR) can assist architects at the early stages of ideation and design. In this paper, we approach VR to build and explore maquettes at different scales in early design stages. To this end we developed a VR environment where user interactions are supported by untethered, easy to operate, peripherals, using a mobile virtual reality headset to provide virtual immersion and simplified geometric information to create voxel-based maquettes. Usability studies with laypeople suggest that the proposed system is both easier to use and more effective [better suited] than current CAD software to rapidly create simplified models. Additionally, tests with architects have shown the system's potential to improve their toolset. This is partly due to VR combining real-time performance with immersive exploration of the content, where body-scale relationships become visible to support the creative process, allowing architects to become both builders and explores of spatial constructs.

Objective To determine viability of drowsiness detection, researchers study the feasibility of photoplethysmogram (PPG) data collection from the geography of the aviation headset, correlating to electrocardiogram (ECG) reference. Background Fatigue has been a probable cause, contributing factor, or a finding in 20% of transportation incidents and accidents studied between January 2001 and December 2012. This operational hazard is particularly troublesome within aviation and airline operations. Method PPG and ECG data were collected synchronously from Federal Aviation Administration (FAA) commercially rated pilots during flight simulation in the window of circadian low (WOCL). Valid PPG and ECG data from 14 participants were analyzed, which yielded approximately 2 hr of data per participant for fatigue-related analysis. Results The results of the study demonstrate clear trends toward decreased heart rate for both ECG and PPG and suggest progression of drowsiness between four separate periods (T1, T2, T3, and T4) selected during the study; however, the mean heart rate change from T1 to T4 was statistically significant. Conclusion The results suggest that ECG and PPG data can be an important tool to observe conditions where drowsiness or fatigue may add risk to the operation. In addition, the data show high correlation between ECG and PPG data, further suggesting that a simpler PPG sensor, mounted within the geography of the aviation headset, may streamline the operationalization of important physiological data. Application Incorporation of PPG sensors and associated signal processing methods into facilitating equipment, such as the aviation headset, may add a layer to operational safety.

With the fast penetration of commercial Virtual Reality (VR) and Augmented Reality (AR) systems into our daily life, the security issues of those devices have attracted significant interests from both academia and industry. Modern VR/AR systems typically use head-mounted devices (i.e., headsets) to interact with users, and often store private user data, e.g., social network accounts, online transactions or even payment information. This poses significant security threats, since in practice the headset can be potentially obtained and accessed by unauthenticated parties, e.g., identity thieves, and thus cause catastrophic breach. In this paper, we propose a novel GaitLock system, which can reliably authenticate users using their gait signatures. Our system doesn't require extra hardware, e.g., fingerprint sensors or retina scanners, but only uses the on-board inertial measurement units (IMUs) equipped in almost all mainstream VR/AR headsets to authenticate the legitimate users from intruders, by simply asking them to walk a few steps. To achieve that, we propose a new gait recognition model Dynamic-SRC, which combines the strength of Dynamic Time Warping (DTW) and Sparse Representation Classifier (SRC), to extract unique gait patterns from the inertial signals during walking. We implement GaitLock on Google Glass (a typical AR headset), and extensive experiments show that GaitLock outperforms the state-of-the-art systems significantly in recognition accuracy ($>$>98 percent success in 5 steps), and is able to run in-situ on the resource-constrained VR/AR headsets without incurring high energy cost.

Short-range wireless technologies are known to transmit voice, audio, image, and video messages in real time. Energy consumption and transmission reach are critical in such networks, especially for portable and power autonomous devices. The purpose of the Voice over Zigbee technology is to provide a competitive offering that excels in these performance aspects. Due to the CSMA-CA mechanism implemented in the 802.15.4 layer, a well-designed strategy must be considered in Zigbee to create a robust, reliable, and full-duplex conversation. In past efforts, we proved that the radio channel of Zigbee has enough bandwidth to support a full-duplex conversation with narrow-band voice codecs. Our embedded implementation of the Speex voice codec targeted the development of a low-cost, ultralow-power, long-range wireless headset using Zigbee technology to transmit voice in full-duplex mode for use with leading PC VoIP programs. Furthermore, we presented the real environment performance evaluation and power consumption tests involving the developed headset prototype. Talk time is comparable to Bluetooth including at the power budget, the codec processing, and analogue audio interface, but its deep-sleep lifetime more than doubles the Bluetooth performance. This was one of the very few successful efforts to port a voice codec on an ultralow-power DSP for use with power sensitive Zigbee applications, which is highly cited in the literature, proving additionally that using an open-source codec can deliver similar voice quality, reducing the total system cost. The current paper elaborates on the embedded parallelism of the Speex implementation and the exploitation of the DSP architectural parallelism which critically enabled the Voice over Zigbee application on the ultralow-power DSP platform. Another significant contribution of this work is towards understanding and resolving the challenges faced when trying to achieve good quality transmission of media over constrained devices and networks. The way to new ultralow-power voice-related Zigbee and constrained network applications is open.

International audience; The way doctors deliver bad news has a significant impact on the therapeutic process. In order to facilitate doctor's training, we have developed an embodied conversational agent simulating a patient to train doctors to break bad news. In this article, we present an evaluation of the virtual reality training platform comparing the users' experience depending on the virtual environment displays: a PC desktop, a virtual reality headset, and four wall fully immersive systems. The results of the experience, including both real doctors and naive participants, reveal a significant impact of the environment display on the perception of the user (sense of presence, sense of co-presence, perception of the believability of the virtual patient), showing, moreover, the different perceptions of the participants depending on their level of expertise.

This paper aims to develop a speller system based on a bipolar single-channel electroencephalogram with sufficient accuracy. The proposed system consists of a custom-designed headset, a new virtual keyboard with 58 characters, special symbols, and digits, and a five-target steady-state visual-evoked potential (SSVEP)-based brain–computer interface (BCI) utilizing one-dimensional convolutional neural network (1-D CNN) for SSVEP frequency detection. The deep learning model is implemented and trained under the training mode before being applied in the operation mode of the system. To validate the proposed model, we acquire the training dataset with numerous testing conditions, including different frequency resolutions of the feature and different time-window lengths of analysis. Two types of features based on the frequency domain are investigated to compare their performances in terms of classification accuracy of the model. The experimental results from eight subjects shows that on average, the proposed model can classify five-class SSVEP data with a high accuracy of 99.2%. The proposed BCI is then employed in an online experiment of spelling the word “SPELLER” using 2-s time window. Consequently, the system achieves an average accuracy of 97.4% and an information transfer rate of 49 ± 7.7 bpm, showing the practicality and feasibility of implementing a reliable single-channel SSVEP-based speller utilizing 1-D CNN.