Resource allocation for multi-sensory virtual environments

Fidelity is of key importance if virtual environments are to be used as authentic representations of real life scenarios. However, simulating the multitude of senses that comprise the human sensory system is a computationally challenging task. With limited computational resources it is essential to distribute these carefully in order to simulate the most ideal perceptual experience for the user. This thesis investigates this balance of resources across multiple scenarios where combined audio, visual and olfactory cues are delivered to the user. Starting with bi-modal virtual environments where audio and visual stimuli are delivered to the users, a subjective experimental study, denoted as E1, was undertaken where participants (N = 35) allocated five fixed resource budgets for adjusting the quality of the displayed graphics and acoustics stimuli. In this experiment, increasing the quality of one of the sensory stimuli decreased the quality of the other. Findings demonstrate that participants allocate more resources to graphics, however as the computational budget is increased the allocation ratio between graphics and acoustics decreases significantly. Based on the results, an audio-visual quality prediction model is proposed and successfully validated against previously untested budgets and scenarios. The introduction of realistic olfactory stimuli is considered necessary if multisensory virtual environments are to be used as genuine representations of real life experiences. The estimation and delivery of smell impulses includes many challenges and significantly differs from the methods used for computing and displaying auditory and visual cues. Furthermore, the absence of a quality metric for assessing olfactory stimuli makes the introduction of an olfactory quality scale in the resource allocation framework significantly challenging. The work presented in this thesis investigates whether better spatial discretisation of the computational domain, a frequently used technique for increasing numerical stability in fluid transport simulations, can be used as a successful smell quality metric that can elicit a perceptual impact to the users of the virtual environment. In this context, better spatial discretisation levels are evaluated based on the estimation of the Just Noticeable Difference (JND) threshold for smell intensity using an experimental study (N = 20) and implemented in two phases. This experiment is denoted as E2 throughout this thesis. Findings demonstrate that the JND threshold is larger than the concentration differences given by progressively accurate smell transport simulations. This outcome enables computational savings from avoiding exhaustive smell transport simulations that provide no perceptual benefit to the user. Having considered the limitations associated with assessing smell impulses in terms of quality, a third experimental study is proposed, denoted as E3, and is exploited for resource allocation in tri-modal virtual environments. The experimental layout of E3 (N = 25) builds on the framework proposed in E1 including the delivery of physically accurate smell impulses to the user. The display of the smell bursts is implemented in a binary fashion (two levels or ON/OFF smell) along with the quality levels for the senses of vision and hearing as selected in E1. The smell concentration level presented in this experiment follows from the results of the JND threshold estimation for odour concentration presented in the psychophysics study E2. In conclusion, the research presented, shows that human preference criteria can be fully exploited in the design and delivery of multi-sensory virtual experiences. Experimental data can be used to derive computationally inexpensive prediction models that direct resource allocation in rendering pipelines where diverse sensory stimuli are simulated and delivered to the users.