Search

Visual images perceived by humans can be reconstructed from their brain activity. However, the visualization (externalization) of mental imagery remains a challenge. In this study, we demonstrated that the visual image reconstruction method proposed in the seminal study by Shen et al. (2019) heavily relied on low-level visual information decoded from the brain and could not efficiently utilize semantic information that would be recruited during mental imagery. To address this limitation, we extended the previous method to a Bayesian estimation framework and introduced the assistance of semantic information into it. Our proposed framework successfully reconstructed both seen (i.e., directly captured by the human eye) and imagined images from the brain activity. These results suggest that our framework would provide a technology for directly investigating the subjective contents of the brain.

異なる脳機能イメージング法であるfMRIとPETで撮像画像に共通する成分はあるのだろうか。共通する成分を抽出できれば、それを利用しfMRI画像とPET画像を相互に変換することが可能となる。本研究では、共通成分の抽出・相互変換を試みるため、深層学習により拡張した正順相関分析法を実装した。本発表において、シミュレーションデータに適応した予備結果を報告し、実データ適用に向けた課題・拡張について議論する。, 量子生命科学会 第4回大会

The sensory cortex is characterized by general organizational principles such as topography and hierarchy. However, measured brain activity given identical input exhibits substantially different patterns across individuals. Although anatomical and functional alignment methods have been proposed in functional magnetic resonance imaging (fMRI) studies, it remains unclear whether and how hierarchical and fine-grained representations can be converted between individuals while preserving the encoded perceptual content. In this study, we trained a method of functional alignment called neural code converter that predicts a target subject’s brain activity pattern from a source subject given the same stimulus, and analyzed the converted patterns by decoding hierarchical visual features and reconstructing perceived images. The converters were trained on fMRI responses to identical sets of natural images presented to pairs of individuals, using the voxels on the visual cortex that covers from V1 through the ventral object areas without explicit labels of the visual areas. We decoded the converted brain activity patterns into the hierarchical visual features of a deep neural network using decoders pre-trained on the target subject and then reconstructed images via the decoded features. Without explicit information about the visual cortical hierarchy, the converters automatically learned the correspondence between visual areas of the same levels. Deep neural network feature decoding at each layer showed higher decoding accuracies from corresponding levels of visual areas, indicating that hierarchical representations were preserved after conversion. The visual images were reconstructed with recognizable silhouettes of objects even with relatively small numbers of data for converter training. The decoders trained on pooled data from multiple individuals through conversions lead to a slight improvement over those trained on a single individual. These results demonstrate that the hierarchical and fine-grained representation can be converted by functional alignment, while preserving sufficient visual information to enable inter-individual visual image reconstruction.Highlights● Neural code converters convert brain activity patterns across individuals with moderate conversion accuracy and learn reasonable visual area correspondences at the same level between individuals.● The converted brain activity patterns can be decoded into hierarchical DNN features to reconstruct visual images, even though the converter is trained on a small number of data samples.● The information of hierarchical and fine-scale visual features that enable visual image reconstruction are preserved after functional alignment.

Machine learning algorithms specialized for neural data have allowed the extraction of information encoded in the brain. As an example, in previous studies, the images human subjects see were reconstructed from their brain activity measured via functional magnetic resonance imaging (fMRI) [1]. However, the application of those machine learning algorithms to high-resolution fMRI data, which may become mainstream in the near future, is limited due to their high computational cost. To solve this problem, scalable machine learning algorithms are being designed by utilizing computational techniques developed in the field of quantum computation [2,3]. In this report, taking one of the popular statistical methods, principal component analysis (PCA), as an example, we show that machine algorithms can be approximated with the use of such quantum-inspired techniques. The computational time and approximation performance of quantum-inspired PCA are demonstrated. The main results of this report have been presented in a previous paper by the author [3]. References 1G. Shen, T. Horikawa, K. Majima, Y. Kamitani, PLOS Computational Biology, Volume 15, e1006633 (2019). 2E. Tang, Physical Review Letters, Volume 127, 060503 (2021). 3N. Koide-Majima, K. Majima, Neural Networks, Volume 135, 55–67 (2021)., QUANTUM INNOVATION 2021

Principal component analysis (PCA) is a widely used statistical tool for extracting low-dimensional structures underlying multivariate data. However, its application to high-dimensional data is limited due to its large computational time. While the conventional PCA algorithm requires polynomial time, using a quantum-inspired algorithm as a subroutine, we have implemented an algorithm that approximates it with computational time proportional to the logarithm of the input dimensionality. The computational efficiency and performance of the implemented algorithm, quantum-inspired PCA, are experimentally evaluated on synthetic and real datasets., 第31回 日本神経回路学会 全国大会（JNNS2021）

Introduction: Using stimuli (e.g., images, videos, products) labeled by a number of raters has recently become common in brain decoding analysis, where subjective emotion/impression for stimuli felt by the population is predicted from brain responses. However, there remains no established method for constructing a decoder using such multi-rater labels. In previous studies, the variability across multiple raters was assumed to reflect noise, and the answers for a binary judgment were averaged across raters. Then, the average scores (i.e., empirical probabilities) for individual stimuli were predicted using standard regression methods. While this procedure is a simple and popular approach, it is not appropriate because most of these regression methods ignore the fact that probability is the variable to be predicted. To address this in an appropriate manner, we present a new framework in this study. Methods: Here, we assume that individual answers for a binary judgment about a stimulus follow a Bernoulli distribution. We then predicted the probability of positive answers from the human functional magnetic resonance imaging (fMRI) response to the stimulus using probit regression. We also introduced sparse regularization into probit regression (sparse probit regression) to prevent overfitting. Results: In both simulation and real fMRI data analysis, sparse probit regression more accurately predicted the probabilities of positive answers for individual stimuli than probit regression without sparse regularization, indicating that sparseness results in better decoding performance. Sparse probit regression also outperformed linear regression using the same type of sparse regularization, reflecting the advantage of our appropriate treatment of probability. Discussion & Conclusion: Our results suggest that our framework using sparse probit regression provides an effective method for the population prediction of emotion/impression assessment based on brain activity., 第5回ヒト脳イメージング研究会

主成分分析は多変量データから重要な低次元成分を抽出する統計手法である．しかし，主成分分析のアルゴリズムは特異値分解に基づくため，次元数（変量の数）が数百万を超えるデータには，計算時間の問題からしばしば適用が困難となる．我々は近年発案された「量子インスパイアアルゴリズム」を用い，計算時間を次元数の対数オーダーに抑えつつ，主成分分析を近似するアルゴリズムを計算機実装した．本報告において，複数の人工データ・実データを用いてその計算時間と性能を評価した結果を紹介する．, 第44回量子情報技術研究会（QIT44）

Sensory perception and memory recall generate different conscious experiences. Although externally and internally driven neural activities signifying the same perceptual content overlap in the sensory cortex, their distribution in the prefrontal cortex (PFC), an area implicated in both perception and memory, remains elusive. Here, we test whether the local spatial configurations and frequencies of neural oscillations driven by perception and memory recall overlap in the macaque PFC using high-density electrocorticography and multivariate pattern analysis. We find that dynamically changing oscillatory signals distributed across the PFC in the delta-, theta-, alpha-, and beta-band ranges carry significant, but mutually different, information predicting the same feature of memory-recalled internal targets and passively perceived external objects. These findings suggest that the frequency-specific distribution of oscillatory neural signals in the PFC serves cortical signatures responsible for distinguishing between different types of cognition driven by external perception and internal memory.

Summary Achievement of human-level image recognition by deep neural networks (DNNs) has spurred interest in whether and how DNNs are brain-like. Both DNNs and the visual cortex perform hierarchical processing, and correspondence has been shown between hierarchical visual areas and DNN layers in representing visual features. Here, we propose the brain hierarchy (BH) score as a metric to quantify the degree of hierarchical correspondence based on neural decoding and encoding analyses where DNN unit activations and human brain activity are predicted from each other. We find that BH scores for 29 pre-trained DNNs with various architectures are negatively correlated with image recognition performance, thus indicating that recently developed high-performance DNNs are not necessarily brain-like. Experimental manipulations of DNN models suggest that single-path sequential feedforward architecture with broad spatial integration is critical to brain-like hierarchy. Our method may provide new ways to design DNNs in light of their representational homology to the brain., Graphical abstract, Highlights • A measure for brain-like hierarchy is proposed to characterize DNNs • Encoding/decoding with human fMRI quantifies the hierarchical correspondence • Among representative DNN models, high-performance models are not brain-like • Critical factors for brain-like hierarchy are explored, Neuroscience; Neural networks; Human-centered computing

The inferior temporal cortex (ITC) contains neurons selective to multiple levels of visual categories. However, the mechanisms by which these neurons collectively construct hierarchical category percepts remain unclear. By comparing decoding accuracy with simultaneously acquired electrocorticogram (ECoG), local field potentials (LFPs), and multi-unit activity in the macaque ITC, we show that low-frequency LFPs/ECoG in the early evoked visual response phase contain sufficient coarse category (e.g., face) information, which is homogeneous and enhanced by spatial summation of up to several millimeters. Late-induced high-frequency LFPs additionally carry spike-coupled finer category (e.g., species, view, and identity of the face) information, which is heterogeneous and reduced by spatial summation. Face-encoding neural activity forms a cluster in similar cortical locations regardless of whether it is defined by early evoked low-frequency signals or late-induced high-gamma signals. By contrast, facial subcategory-encoding activity is distributed, not confined to the face cluster, and dynamically increases its heterogeneity from the early evoked to late-induced phases. These findings support a view that, in contrast to the homogeneous and static coarse category-encoding neural cluster, finer category-encoding clusters are heterogeneously distributed even outside their parent category cluster and dynamically increase heterogeneity along with the local cortical processing in the ITC.

Prepared movements are more efficient than those that are not prepared for. Although changes in cortical activity have been observed prior to a forthcoming action, the circuits involved in motor preparation remain unclear. Here, we use in vivo two-photon calcium imaging to uncover changes in the motor cortex during variable waiting periods prior to a forepaw reaching task in mice. Consistent with previous reports, we observed a subset of neurons with increased activity during the waiting period; however, these neurons did not account for the degree of preparation as defined by reaction time (RT). Instead, the suppression of activity of distinct neurons in the same cortical area better accounts for RT. This suppression of neural activity resulted in a distinct and reproducible pattern when mice were well prepared. Thus, the selective suppression of network activity in the motor cortex may be a key feature of prepared movements.

Canonical correlation analysis (CCA) is a technique to find statistical dependencies between a pair of multivariate data. However, its application to high dimensional data is limited due to the resulting time complexity. While the conventional CCA algorithm requires polynomial time, we have developed an algorithm that approximates CCA with computational time proportional to the logarithm of the input dimensionality using quantum-inspired computation. The computational efficiency and approximation performance of the proposed quantum-inspired CCA (qiCCA) algorithm are experimentally demonstrated. Furthermore, the fast computation of qiCCA allows us to directly apply CCA even after nonlinearly mapping raw input data into very high dimensional spaces. Experiments performed using a benchmark dataset demonstrated that, by mapping the raw input data into the high dimensional spaces with second-order monomials, the proposed qiCCA extracted more correlations than linear CCA and was comparable to deep CCA and kernel CCA. These results suggest that qiCCA is considerably useful and quantum-inspired computation has the potential to unlock a new field in which exponentially large dimensional data can be analyzed., Comment: 45 pages, 9 figures. This article will appear in Neural Networks

The mental contents of perception and imagery are thought to be encoded in hierarchical representations in the brain, but previous attempts to visualize perceptual contents have failed to capitalize on multiple levels of the hierarchy, leaving it challenging to reconstruct internal imagery. Recent work showed that visual cortical activity measured by functional magnetic resonance imaging (fMRI) can be decoded (translated) into the hierarchical features of a pre-trained deep neural network (DNN) for the same input image, providing a way to make use of the information from hierarchical visual features. Here, we present a novel image reconstruction method, in which the pixel values of an image are optimized to make its DNN features similar to those decoded from human brain activity at multiple layers. We found that our method was able to reliably produce reconstructions that resembled the viewed natural images. A natural image prior introduced by a deep generator neural network effectively rendered semantically meaningful details to the reconstructions. Human judgment of the reconstructions supported the effectiveness of combining multiple DNN layers to enhance the visual quality of generated images. While our model was solely trained with natural images, it successfully generalized to artificial shapes, indicating that our model was not simply matching to exemplars. The same analysis applied to mental imagery demonstrated rudimentary reconstructions of the subjective content. Our results suggest that our method can effectively combine hierarchical neural representations to reconstruct perceptual and subjective images, providing a new window into the internal contents of the brain., Author summary Machine learning-based analysis of human functional magnetic resonance imaging (fMRI) patterns has enabled the visualization of perceptual content. However, prior work visualizing perceptual contents from brain activity has failed to combine visual information of multiple hierarchical levels. Here, we present a method for visual image reconstruction from the brain that can reveal both seen and imagined contents by capitalizing on multiple levels of visual cortical representations. We decoded brain activity into hierarchical visual features of a deep neural network (DNN), and optimized an image to make its DNN features similar to the decoded features. Our method successfully produced perceptually similar images to viewed natural images and artificial images (colored shapes and letters), whereas the decoder was trained only on an independent set of natural images. It also generalized to the reconstruction of mental imagery of remembered images. Our approach allows for studying subjective contents represented in hierarchical neural representations by objectifying them into images.

Brain decoding with multivariate classification and regression has provided a powerful framework for characterizing information encoded in population neural activity. Classification and regression models are respectively used to predict discrete and continuous variables of interest. However, cognitive and behavioral parameters that we wish to decode are often ordinal variables whose values are discrete but ordered, such as subjective ratings. To date, there is no established method of predicting ordinal variables in brain decoding. In this study, we present a new algorithm, sparse ordinal logistic regression (SOLR), that combines ordinal logistic regression with Bayesian sparse weight estimation. We found that, in both simulation and analyses using real functional magnetic resonance imaging data, SOLR outperformed ordinal logistic regression with non-sparse regularization, indicating that sparseness leads to better decoding performance. SOLR also outperformed classification and linear regression models with the same type of sparseness, indicating the advantage of the modeling tailored to ordinal outputs. Our results suggest that SOLR provides a principled and effective method of decoding ordinal variables.

Neurons in high-level visual areas respond to more complex visual features with broader receptive fields (RFs) compared to those in low-level visual areas. Thus, high-level visual areas are generally considered to carry less information regarding the position of seen objects in the visual field. However, larger RFs may not imply loss of position information at the population level. Here, we evaluated how accurately the position of a seen object could be predicted (decoded) from activity patterns in each of six representative visual areas with different RF sizes (V1–V4, LOC, and FFA). We collected fMRI responses while human subjects viewed a ball randomly moving in a two-dimensional field. To estimate population RF sizes of individual fMRI voxels, RF models were fitted for individual voxels in each brain area. The voxels in higher visual areas showed larger estimated RFs than those in lower visual areas. Then, the ball’s position in a separate session was predicted by maximum likelihood regression (support vector regression, SVR) to predict the position. We found that regardless of the difference in RF size, all visual areas showed similar prediction accuracies, especially on the horizontal dimension. Higher areas showed slightly lower accuracies on the vertical dimension, which appears to be attributed to the narrower spatial distributions of the RFs centers. The results suggest that much of position information is preserved in population activity through the hierarchical visual pathway regardless of RF sizes, and is potentially available in later processing for recognition and behavior.Significance statementHigh-level ventral visual areas are thought to achieve position invariance with larger receptive fields at the cost of the loss of precise position information. However, larger receptive fields may not imply loss of position information at the population level. Here, multivoxel fMRI decoding reveals that high-level visual areas are predictive of an object’s position with similar accuracies to low-level visual areas, especially on the horizontal dimension, preserving the information potentially available for later processing.

Highly localized neuronal spikes in primate temporal cortex can encode associative memory; however, whether memory formation involves area-wide reorganization of ensemble activity, which often accompanies rhythmicity, or just local microcircuit-level plasticity, remains elusive. Using high-density electrocorticography, we capture local-field potentials spanning the monkey temporal lobes, and show that the visual pair-association (PA) memory is encoded in spatial patterns of theta activity in areas TE, 36, and, partially, in the parahippocampal cortex, but not in the entorhinal cortex. The theta patterns elicited by learned paired associates are distinct between pairs, but similar within pairs. This pattern similarity, emerging through novel PA learning, allows a machine-learning decoder trained on theta patterns elicited by a particular visual item to correctly predict the identity of those elicited by its paired associate. Our results suggest that the formation and sharing of widespread cortical theta patterns via learning-induced reorganization are involved in the mechanisms of associative memory representation., Episodic or declarative memory is thought to be encoded in the ensemble firing of spatially distributed neurons. Here the authors use high-density electrical recordings to show that some areas in the primate temporal cortex develop patterns of theta activity that are similar for pairs of remembered objects.

The intrinsic toxicity of lepidopteran-active Bt proteins necessitates assessment of non-target risks associated with environmental release of transgenic crops expressing these proteins. Principles of ecological risk assessment provide a means for assessing non-target risks when information regarding exposure to the toxin and species-specific effects are lacking. This is shown for the case of Bt Cry1F maize release in Japan, where off-field pollen dissemination and effect on butterfly species is of concern. The specific ecological entity of concern for the assessment of the non-target impact of Cry1F maize pollen was Yamato-shijimi (pale grass blue butterfly), Pseudozizeeria maha (Kollar), a commonly occurring, susceptible species. Yamato-shijimi is widely adapted in Japan where it occurs in both rural and metropolitan settings, corresponding to the distribution and habitat of katabami (Oxalis corniculata (L.)), the larval host plant. The northern extent of Yamato-shijimi habitat lies to the south of major maize production regions in Japan, but exposure may occur elsewhere where maize and Yamato-shijimi co-occur. Screening level assessment of potential adverse effects to Yamato-shijimi in the field environment considered the probability for spatial-temporal co-occurrence of the life stages of concern (1st and 2nd instars) and the stressor (Cry1F protein expressed in maize pollen) at environmentally relevant concentrations. In the event of exposure to maize pollen, early instars of Yamato-shijimi feed exclusively on the underside of katabami leaves, which further limits the portion of the butterfly population that would be exposed. Projected levels of exposure to Cry1F pollen are below the toxicity level of concern and, thus, indicate negligible risk. Most sensitive species characterization (intergenera sensitivity) similarly shows negligible risk to other Japanese butterfly species of concern when distributed beyond the maize field or field margin.

We successfully established a mass production system for an influenza virus-like particle (VLP) vaccine using a synthetic H5 hemagglutinin (HA) gene codon-optimized for the silkworm. A recombinant baculovirus containing the synthetic gene was inoculated into silkworm pupae. Four days after inoculation, the hemagglutination titer in homogenates from infected pupae reached a mean value of 0.8 million hemagglutination units (HAU), approximately 2,000 μg HA protein per pupa, more than 50-fold higher than that produced with an embryonated chicken egg. VLPs ranging from 30 nm to 300 nm in diameter and covered with a large number of spikes were detected in the homogenates. The spikes were approximately 14 nm long, similar to an authentic influenza HA spike. Detailed electron micrographs indicated that the VLP spike density was similar to that of authentic influenza virus particles. The results clearly show that the expression of a single HA gene can efficiently produce VLPs in silkworm pupae. When chickens were immunized with the pupae homogenate, the hemagglutination inhibition titer in their sera reached values of 2,048-8,192 after approximately 1 month. This is the first report demonstrating that a large amount of VLP vaccine could be produced by single synthetic HA gene in silkworm pupae. Our system might be useful for future vaccine development against other viral diseases.

We isolated a Drosophila fickleP (ficP) mutant with a shortened copulatory duration and reduced adult-stage life span. The reduced copulatory duration is ascribable to incomplete fusion of the left and right halves of the apodeme that holds the penis during copulation. ficP is an intronic mutation occurring in the Btk gene, a gene which encodes two forms (type 1 and type 2) of a Bruton’s tyrosine kinase (Btk) family cytoplasmic tyrosine kinase as a result of alternative exon usage. The ficP mutation prevents the formation of the type 2 isoform but leaves expression of the type 1 transcript intact. Ubiquitous overexpression of the wild-type cDNA by using a heat shock 70 promoter during the late larval or pupal stages rescued the life span and genital defects in the mutant, respectively, establishing the causal relationship between the ficP phenotypes and the Btk gene mutation. The stage specificity of the rescuing ability suggests that the Btk gene is required for the development of male genitalia and substrates required for adult survival.

The genome of the nucleopolyhedrovirus (NPV) (T3 strain) pathogenic for Bombyx mori (Bm) was sequenced and analysed. The BmNPV genome was 128,413 nucleotides long with a G+C content of 40% and contained 136 open reading frames (ORFs) encoding predicted proteins of over 60 amino acids. Although phenotypically different, the genome organizations of BmNPV and Autographa californica multinucleocapsid NPV (AcMNPV) were closely related. The BmNPV genome was over 90% identical to about three-quarters of the genome of AcMNPV. The relatedness of predicted amino acid sequences of corresponding ORFs between BmNPV and AcMNPV was about 90%. However, the BmNPV genome lacked homologues of the following AcMNPV ORFs: Ac3 (conotoxin), Ac7 (orf603), Ac48 (etm), Ac49 (pcna), Ac70 (hcf-1), Ac86 (pnk/pnl) and Ac134 (p94). In addition, BmNPV contained five ORFs related to Ac2. A high frequency of multiple 3 bp insertions was also found within BmNPV and AcMNPV coding sequences.

How visual object categories are represented in the brain is one of the key questions in neuroscience. Studies on low-level visual features have shown that relative timings or phases of neural activity between multiple brain locations encode information. However, whether such temporal patterns of neural activity are used in the representation of visual objects is unknown. Here, we examined whether and how visual object categories could be predicted (or decoded) from temporal patterns of electrocorticographic (ECoG) signals from the temporal cortex in five patients with epilepsy. We used temporal correlations between electrodes as input features, and compared the decoding performance with features defined by spectral power and phase from individual electrodes. While using power or phase alone, the decoding accuracy was significantly better than chance, correlations alone or those combined with power outperformed other features. Decoding performance with correlations was degraded by shuffling the order of trials of the same category in each electrode, indicating that the relative time series between electrodes in each trial is critical. Analysis using a sliding time window revealed that decoding performance with correlations began to rise earlier than that with power. This earlier increase in performance was replicated by a model using phase differences to encode categories. These results suggest that activity patterns arising from interactions between multiple neuronal units carry additional information on visual object categories.

The goal of brain-machine interface (BMI) research is to interpret brain signals in order to control an external device. Substantial progress toward this goal has been achieved over the last decade. Currently, BMI algorithms can translate neural signals into motor commands that reproduce arm-reaching and hand-grasping movements in artificial actuators, thereby promising the restoration of limb mobility in paralyzed people. In one study, a tetraplegic human subject used a clinical neuromotor prosthesis to restore his communication and mobility. Furthermore, a recently developed neural decoding technology provides an effective means to read out mental states from human brain activity. Decoding of mental states could be used for direct human-human communication outside the brain's normal pathways. However, for BMI practical, long-term stability of signal interpretation is required. Unfortunately, the classical invasive BMI methods suffer from poor long-term stability because of deterioration in signal quality. Two new approaches to long-term BMI applications are showing promising results in maintaining signal quality. One is the use of newly developed electrodes that are less harmful to neural tissues, and the other is the use of electrocorticograms (ECoGs), which measure population activity of neurons with electrodes placed on the surface of the brain. Both these new technologies facilitate clearer signals from the brain and greater stability of brain signals over time. In this review, we summarize the previous BMI approaches and shed light upon the new advances that may enable long-term BMI use.