Search

The mental contents of perception and imagery are thought to be encoded in hierarchical representations in the brain, but previous attempts to visualize perceptual contents have failed to capitalize on multiple levels of the hierarchy, leaving it challenging to reconstruct internal imagery. Recent work showed that visual cortical activity measured by functional magnetic resonance imaging (fMRI) can be decoded (translated) into the hierarchical features of a pre-trained deep neural network (DNN) for the same input image, providing a way to make use of the information from hierarchical visual features. Here, we present a novel image reconstruction method, in which the pixel values of an image are optimized to make its DNN features similar to those decoded from human brain activity at multiple layers. We found that our method was able to reliably produce reconstructions that resembled the viewed natural images. A natural image prior introduced by a deep generator neural network effectively rendered semantically meaningful details to the reconstructions. Human judgment of the reconstructions supported the effectiveness of combining multiple DNN layers to enhance the visual quality of generated images. While our model was solely trained with natural images, it successfully generalized to artificial shapes, indicating that our model was not simply matching to exemplars. The same analysis applied to mental imagery demonstrated rudimentary reconstructions of the subjective content. Our results suggest that our method can effectively combine hierarchical neural representations to reconstruct perceptual and subjective images, providing a new window into the internal contents of the brain., Competing Interests: The authors declare no conflict of interest.

Dreaming is a subjective experience during sleep that is often accompanied by vivid perceptual and emotional contents. Because of its fundamentally subjective nature, the objective study of dream contents has been challenging. However, since the discovery of rapid eye movements during sleep, scientific knowledge on the relationship between dreaming and physiological measures including brain activity has accumulated. Recent advances in neuroimaging analysis methods have made it possible to uncover direct links between specific dream contents and brain activity patterns. In this review, we first give a historical overview on dream researches with a focus on the neurophysiological and behavioral signatures of dreaming. We then discuss our recent study in which visual dream contents were predicted, or decoded, from brain activity during sleep onset periods using machine learning-based pattern recognition of functional MRI data. We suggest that advanced analytical tools combined with neural and behavioral databases will reveal the relevance of spontaneous brain activity during sleep to waking experiences.