Search

Pose estimation algorithms are shedding new light on animal behavior and intelligence. Most existing models are only trained with labeled frames (supervised learning). Although effective in many cases, the fully supervised approach requires extensive image labeling, struggles to generalize to new videos, and produces noisy outputs that hinder downstream analyses. We address each of these limitations with a semi-supervised approach that leverages the spatiotemporal statistics of unlabeled videos in two different ways. First, we introduce unsupervised training objectives that penalize the network whenever its predictions violate smoothness of physical motion, multiple-view geometry, or depart from a low-dimensional subspace of plausible body configurations. Second, we design a new network architecture that predicts pose for a given frame using temporal context from surrounding unlabeled frames. These context frames help resolve brief occlusions or ambiguities between nearby and similar-looking body parts. The resulting pose estimation networks achieve better performance with fewer labels, generalize better to unseen videos, and provide smoother and more reliable pose trajectories for downstream analysis; for example, these improved pose trajectories exhibit stronger correlations with neural activity. We also propose a Bayesian post-processing approach based on deep ensembling and Kalman smoothing that further improves tracking accuracy and robustness. We release a deep learning package that adheres to industry best practices, supporting easy model development and accelerated training and prediction. Our package is accompanied by a cloud application that allows users to annotate data, train networks, and predict new videos at scale, directly from the browser.

A key computation in building adaptive internal models of the external world is to ascribe sensory signals to their likely cause(s), a process of causal inference (CI). CI is well studied within the framework of two-alternative forced-choice tasks, but less well understood within the cadre of naturalistic action–perception loops. Here, we examine the process of disambiguating retinal motion caused by self- and/or object-motion during closed-loop navigation. First, we derive a normative account specifying how observers ought to intercept hidden and moving targets given their belief about (i) whether retinal motion was caused by the target moving, and (ii) if so, with what velocity. Next, in line with the modelling results, we show that humans report targets as stationary and steer towards their initial rather than final position more often when they are themselves moving, suggesting a putative misattribution of object-motion to the self. Further, we predict that observers should misattribute retinal motion more often: (i) during passive rather than active self-motion (given the lack of an efference copy informing self-motion estimates in the former), and (ii) when targets are presented eccentrically rather than centrally (given that lateral self-motion flow vectors are larger at eccentric locations during forward self-motion). Results support both of these predictions. Lastly, analysis of eye movements show that, while initial saccades toward targets were largely accurate regardless of the self-motion condition, subsequent gaze pursuit was modulated by target velocity during object-only motion, but not during concurrent object- and self-motion. These results demonstrate CI within action–perception loops, and suggest a protracted temporal unfolding of the computations characterizing CI.This article is part of the theme issue ‘How the brain controls decision making in a multisensory world’.

Progress in neuroscience is hindered by poor reproducibility of mouse behavior. Here we show that in a visual decision making task, reproducibility can be achieved by automating the training protocol and by standardizing experimental hardware, software, and procedures. We trained 101 mice in this task across seven laboratories at six different research institutions in three countries, and obtained 3 million mouse choices. In trained mice, variability in behavior between labs was indistinguishable from variability within labs. Psychometric curves showed no significant differences in visual threshold, bias, or lapse rates across labs. Moreover, mice across laboratories adopted similar strategies when stimulus location had asymmetrical probability that changed over time. We provide detailed instructions and open-source tools to set up and implement our method in other laboratories. These results establish a new standard for reproducibility of rodent behavior and provide accessible tools for the study of decision making in mice.

The integration of information across sensory modalities is dependent on the spatiotemporal characteristics of the stimuli that are paired. Despite large variation in the distance over which events occur in our environment, relatively little is known regarding how stimulus-observer distance affects multisensory integration. Prior work has suggested that exteroceptive stimuli are integrated over larger temporal intervals in near relative to far space, and that larger multisensory facilitations are evident in far relative to near space. Here, we sought to examine the interrelationship between these previously established distance-related features of multisensory processing. Participants performed an audiovisual simultaneity judgment and redundant target task in near and far space, while audiovisual stimuli were presented at a range of temporal delays (i.e., stimulus onset asynchronies). In line with the previous findings, temporal acuity was poorer in near relative to far space. Furthermore, reaction time to asynchronously presented audiovisual targets suggested a temporal window for fast detection—a range of stimuli asynchronies that was also larger in near as compared to far space. However, the range of reaction times over which multisensory response enhancement was observed was limited to a restricted range of relatively small (i.e., 150 ms) asynchronies, and did not differ significantly between near and far space. Furthermore, for synchronous presentations, these distance-related (i.e., near vs. far) modulations in temporal acuity and multisensory gain correlated negatively at an individual subject level. Thus, the findings support the conclusion that multisensory temporal binding and gain are asymmetrically modulated as a function of distance from the observer, and specifies that this relationship is specific for temporally synchronous audiovisual stimulus presentations. Electronic supplementary material The online version of this article (10.1007/s00221-018-5274-7) contains supplementary material, which is available to authorized users.

Introduction Diagnosis and prediction of recovery in the acute phase of disorders of consciousness (DOC) are critical for subsequent medical decisions. However, reliable assessment remains elusive due to the inability of current validated clinical scales to take into account motor and drive deficits. Recently, the Motor Behaviour Tool (MBT), a novel clinical scale, has been developed to address this caveat (1). In this context, neuroimaging and brain-computer interface (BCI) have also been proposed to improve the diagnosis and prognosis of these patients.(2). Objectives This pilot study aims to investigate the diagnostic and prognostic value of two electroencephalography (EEG)-based paradigms in patients with DOC, i.e., coma, Unresponsive Awareness Syndrome (UWS) or Minimally Conscious State (MCS). As a second step, we will employ them as evidence to further establish the added value of the MBT (i.e., assessment of minimal responses suggesting remaining conscious processing) combined with the Coma Recovery Scale-Revised (CRS-R), a standardized validated scale commonly used to assess consciousness in this population (3). Patients & Methods Acute DOC patients undergo CRS-R and MBT assessment prior to two EEG paradigms. Firstly, a motor attempt EEG-BCI coupled with Functional Electrical Stimulation (FES) is used (4). We hypothesize that replacing the need for overt movements with motor attempt can alleviate the tendency of CRS-R to underestimate the level of awareness in case of cognitive-motor dissociation (CMD) (5). In addition, a second EEG protocol presents patients with FES-tactile (T), auditory (A), and audio-tactile (AT) stimuli both in actionable and non-actionable space. EEG evoked potentials observed in the actionable space are expected to show a non-linear addition of sensory stimuli (i.e., A+T ≠ AT) indicating multisensory integration and the capacity of conscious processing (6). Results Pending elaborate analysis, preliminary findings show (Fig. 1) that BCI accuracy is significantly above chance only for a patient who was diagnosed as UWS by the CRS-R evaluation, but exhibited a motor behavior classified as CMD confirmed by the MBT tool, and not for one in real UWS (same diagnosis based on CRS-R and MBT), implying the presence of the hypothesized relation between motor EEG correlates and awareness (7). Average EEG evoked potentials of 8 patients during the second EEG paradigm highlight a difference between within (solid line) vs. outside (dashed line) the actionable space (Fig. 2), suggesting awareness-dependent modulation. Future analyses will explore correlations of such EEG descriptors with the clinical outcomes. Conclusion EEG correlates extracted from these EEG paradigms are promising tools for diagnosis of DOC and may supplement current clinical scales to help the validation of new tools like the MBT. References J. M. Pignat, E. Mauron, J. Jöhr, C. Gilart De Keranflec'h, D. Van De Ville, M. G. Preti, D. E. Meskaldji, V. Hörnberg, S. Laureys, B. Draganski, R. Frackowiak, K. Diserens, Outcome prediction of consciousness disorders in the acute stage based on a complementary motor behavioural tool, PLOS ONE 11(6), e0156882 (2016). A. M. Owen, M. R. Coleman, M. Boly, M. H. Davis, S. Laureys, J. D. Pickard, Detecting awareness in the vegetative state, Science, 313, 1402 (2006). J. T. Giacino, K. Kalmar, J. Whyte, The JFK Coma Recovery Scale-Revised: measurement characteristics and diagnostic utility, Arch. Phys. Med. Rehabil. 85(12), 2020-9 (2004). T. Corbet, R. Leeb, A. Biasiucci, S. Perdikis, J. del R. Millán et al. BCI-NMES therapy enhances effective connectivity in the damaged hemisphere in stroke patients. 6th International Brain-Computer Interface Meeting, Asilomar, California, USA (2016). N. D. Schiff, Cognitive motor dissociation following severe brain injuries, JAMA Neurol. 72(12), 1413–1415 (2015). J. P. Noel, C. Pfeiffer, O. Blanke, A. Serino, Full body peripersonal space as the space of the bodily self, Cognition 144, 49-57 (2015). A. M . Goldfine, J. D. Victor, M. M. Conte, J. C. Bardin, N. D. Schiff, Determination of awareness in patients with severe brain injury using EEG power spectral analysis, Clin. Neurophysiol. 122(11), 2157-68 (2011).

Interactions between the body and the environment occur within the peripersonal space (PPS), the space immediately surrounding the body. The PPS is encoded by multisensory (audio-tactile, visual-tactile) neurons that possess receptive fields (RFs) anchored on the body and restricted in depth. The extension in depth of PPS neurons’ RFs has been documented to change dynamically as a function of the velocity of incoming stimuli, but the underlying neural mechanisms are still unknown. Here, by integrating a psychophysical approach with neural network modeling, we propose a mechanistic explanation behind this inherent dynamic property of PPS. We psychophysically mapped the size of participant’s peri-face and peri-trunk space as a function of the velocity of task-irrelevant approaching auditory stimuli. Findings indicated that the peri-trunk space was larger than the peri-face space, and, importantly, as for the neurophysiological delineation of RFs, both of these representations enlarged as the velocity of incoming sound increased. We propose a neural network model to mechanistically interpret these findings: the network includes reciprocal connections between unisensory areas and higher order multisensory neurons, and it implements neural adaptation to persistent stimulation as a mechanism sensitive to stimulus velocity. The network was capable of replicating the behavioral observations of PPS size remapping and relates behavioral proxies of PPS size to neurophysiological measures of multisensory neurons’ RF size. We propose that a biologically plausible neural adaptation mechanism embedded within the network encoding for PPS can be responsible for the dynamic alterations in PPS size as a function of the velocity of incoming stimuli. NEW & NOTEWORTHY Interactions between body and environment occur within the peripersonal space (PPS). PPS neurons are highly dynamic, adapting online as a function of body-object interactions. The mechanistic underpinning PPS dynamic properties are unexplained. We demonstrate with a psychophysical approach that PPS enlarges as incoming stimulus velocity increases, efficiently preventing contacts with faster approaching objects. We present a neurocomputational model of multisensory PPS implementing neural adaptation to persistent stimulation to propose a neurophysiological mechanism underlying this effect.

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce "Lightning Pose," an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry, and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post-hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We release a cloud application that allows users to label data, train networks, and predict new videos directly from the browser., Competing Interests: Competing Interests Statement Robert S. Lee assisted in the initial development of the cloud application as a solution architect at Lightning AI in Spring-Summer 2022. He left the company in August 2022 and continues to hold shares. The remaining authors declare no competing interests.

Natural behaviors occur in closed action-perception loops and are supported by dynamic and flexible beliefs abstracted away from our immediate sensory milieu. How this real-world flexibility is instantiated in neural circuits remains unknown. Here we have macaques navigate in a virtual environment by primarily leveraging sensory (optic flow) signals, or by more heavily relying on acquired internal models. We record single-unit spiking activity simultaneously from the dorsomedial superior temporal area (MSTd), parietal area 7a, and the dorso-lateral prefrontal cortex (dlPFC). Results show that while animals were able to maintain adaptive task-relevant beliefs regardless of sensory context, the fine-grain statistical dependencies between neurons, particularly in 7a and dlPFC, dynamically remapped with the changing computational demands. In dlPFC, but not 7a, destroying these statistical dependencies abolished the area's ability for cross-context decoding. Lastly, correlation analyses suggested that the more unit-to-unit couplings remapped in dlPFC, and the less they did so in MSTd, the less were population codes and behavior impacted by the loss of sensory evidence. We conclude that dynamic functional connectivity between prefrontal cortex neurons maintains a stable population code and context-invariant beliefs during naturalistic behavior with closed action-perception loops., Competing Interests: Competing Interests Statement No author had any competing interests of any kind.

A key computation in building adaptive internal models of the external world is to ascribe sensory signals to their likely cause(s), a process of Bayesian Causal Inference (CI). CI is well studied within the framework of two-alternative forced-choice tasks, but less well understood within the cadre of naturalistic action-perception loops. Here, we examine the process of disambiguating retinal motion caused by self- and/or object-motion during closed-loop navigation. First, we derive a normative account specifying how observers ought to intercept hidden and moving targets given their belief over (i) whether retinal motion was caused by the target moving, and (ii) if so, with what velocity. Next, in line with the modeling results, we show that humans report targets as stationary and steer toward their initial rather than final position more often when they are themselves moving, suggesting a misattribution of object-motion to the self. Further, we predict that observers should misattribute retinal motion more often: (i) during passive rather than active self-motion (given the lack of an efference copy informing self-motion estimates in the former), and (ii) when targets are presented eccentrically rather than centrally (given that lateral self-motion flow vectors are larger at eccentric locations during forward self-motion). Results confirm both of these predictions. Lastly, analysis of eye-movements show that, while initial saccades toward targets are largely accurate regardless of the self-motion condition, subsequent gaze pursuit was modulated by target velocity during object-only motion, but not during concurrent object- and self-motion. These results demonstrate CI within action-perception loops, and suggest a protracted temporal unfolding of the computations characterizing CI.