Search

An important model system for understanding genes, neurons and behavior, the nematode worm C. elegans naturally moves through a variety of complex postures, for which estimation from video data is challenging. We introduce an open-source Python package, WormPose, for 2D pose estimation in C. elegans, including self-occluded, coiled shapes. We leverage advances in machine vision afforded from convolutional neural networks and introduce a synthetic yet realistic generative model for images of worm posture, thus avoiding the need for human-labeled training. WormPose is effective and adaptable for imaging conditions across worm tracking efforts. We quantify pose estimation using synthetic data as well as N2 and mutant worms in on-food conditions. We further demonstrate WormPose by analyzing long (∼ 8 hour), fast-sampled (∼ 30 Hz) recordings of on-food N2 worms to provide a posture-scale analysis of roaming/dwelling behaviors., Author summary Recent advances in machine learning have enabled the high-resolution estimation of bodypoint positions of freely behaving animals, but manual labeling can render these methods imprecise and impractical, especially in highly deformable animals such as the nematode C. elegans. Such animals also frequently coil, resulting in complicated shapes whose ambiguity presents difficulties for standard pose estimation methods. Efficiently solving coiled shapes in C. elegans, exhibited in a variety of important natural contexts, is the primary limiting factor for fully automated high-throughput behavior analysis. WormPose provides pose estimation that works across imaging conditions, naturally complements existing worm trackers, and harnesses the power of deep convolutional networks but with an image generator to automatically provide precise image-centerline pairings for training. We apply WormPose to on-food recordings, finding a near absence of deep δ-turns. We also show that incoherent body motions in the dwell state, which do not translate the worm, have been misidentified as an increase in reversal rate by previous, centroid-based methods. We expect that the combination of a body model and image synthesis demonstrated in WormPose will be both of general interest and important for future progress in precise pose estimation in other slender-bodied and deformable organisms.

From cells in tissue, to bird flocks, to human crowds, living systems display a stunning variety of collective behaviors. Yet quantifying such phenomena first requires tracking a significant fraction of the group members in natural conditions, a substantial and ongoing challenge. We present a comprehensive, computational method for tracking an entire colony of the honey bee Apis mellifera using high-resolution video on a natural honeycomb background. We adapt a convolutional neural network (CNN) segmentation architecture to automatically identify bee and brood cell positions, body orientations and within-cell states. We achieve high accuracy (~10% body width error in position, ~10° error in orientation, and true positive rate > 90%) and demonstrate months-long monitoring of sociometric colony fluctuations. These fluctuations include ~24 h cycles in the counted detections, negative correlation between bee and brood, and nightly enhancement of bees inside comb cells. We combine detected positions with visual features of organism-centered images to track individuals over time and through challenging occluding events, recovering ~79% of bee trajectories from five observation hives over 5 min timespans. The trajectories reveal important individual behaviors, including waggle dances and crawling inside comb cells. Our results provide opportunities for the quantitative study of collective bee behavior and for advancing tracking techniques of crowded systems., Honey bee colonies are hard to automatically monitor due to the high number of visually similar members which move rapidly and whose numbers change over time. Here, the authors report a method for markerless tracking of a bee colony by adapting convolutional neural networks for detection and tracking.

Data related to thestudyMarkerless tracking of an entire insect colony. The dataset comprises detection and trajectory information from five beehive recordings at 10 fps 5 min segments. Tracking pipeline is described in ourgithub repositoryandsample datawas prepared to run the pipeline.

Data related to thestudyMarkerless tracking of an entire insect colony. The dataset comprises detection and trajectory information from five beehive recordings at 10 fps 5 min segments. Tracking pipeline is described in ourgithub repositoryandsample datawas prepared to run the pipeline. We include also manuscript Supplementary Movies in this dataset: M1Background extracted from hive L1. Each frame represents 12 h of the original recording. M2Background extracted from hive L5. Each frame represents 12 h of the original recording, predicted brood cells are marked in red. M3-M7. Example bee trajectories with their corresponding trajectories through the space of visual features. The originally 64-dimensional vectors of visual features representing each bee instance are projected into 3D with the use of t-Distributed Stochastic Neighbor Embedding (t-SNE). Analogous to the plot 3B, 10 most recent representations of the tracked bee are marked in red dots and representations of all other bees from three most recent video frames are marked in yellow dots. Examples are from hives S1-S5, respectively. M8-M12. Short video snippets provided to visually illustrate the proportion of tracked individuals in each hives. All reference trajectories in hives: S1-S5, respectively. M13-M15. Bees performing waggle dance, whose trajectories are shown in Fig. 4D. M16-M18. Bees visiting multiple comb cells, whose trajectories are shown in Fig. 4F.

An important model system for understanding genes, neurons and behavior, the nematode worm C. elegans naturally moves through a variety of complex postures, for which estimation from video data is challenging. We introduce an open-source Python package, WormPose, for 2D pose estimation in C. elegans, including self-occluded, coiled shapes. We leverage advances in machine vision afforded from convolutional neural networks and introduce a synthetic yet realistic generative model for images of worm posture, thus avoiding the need for human-labeled training. WormPose is effective and adaptable for imaging conditions across worm tracking efforts. We quantify pose estimation using synthetic data as well as N2 and mutant worms in on-food conditions. We further demonstrate WormPose by analyzing long (∼ 10 hour), fast-sampled (∼ 30 Hz) recordings of on-food N2 worms to provide a posture-scale analysis of roaming/dwelling behaviors.

We present a comprehensive, computational method for tracking an entire colony of the honey beeApis melliferausing high-resolution video on a natural honeycomb background. We adapt a convolutional neural network (CNN) segmentation architecture to automatically identify bee and brood cell positions, body orientations and within-cell states. We achieve high accuracy (~10% body width error in position, ~10° error in orientation, and true positive rate > 90%) and demonstrate months-long monitoring of sociometric colony fluctuations. We combine extracted positions with rich visual features of organism-centered images to track individuals over time and through challenging occluding events, recovering ~79% of bee trajectories from five observation hives over a span of 5 minutes. The resulting trajectories reveal important behaviors, including fast motion, comb-cell activity, and waggle dances. Our results provide new opportunities for the quantitative study of collective bee behavior and for advancing tracking techniques of crowded systems.

Tracking large numbers of densely-arranged, interacting objects is challenging due to occlusions and the resulting complexity of possible trajectory combinations, as well as the sparsity of relevant, labeled datasets. Here we describe a novel technique of collective tracking in the model environment of a 2D honeybee hive in which sample colonies consist of $N\sim10^3$ highly similar individuals, tightly packed, and in rapid, irregular motion. Such a system offers universal challenges for multi-object tracking, while being conveniently accessible for image recording. We first apply an accurate, segmentation-based object detection method to build initial short trajectory segments by matching object configurations based on class, position and orientation. We then join these tracks into full single object trajectories by creating an object recognition model which is adaptively trained to recognize honeybee individuals through their visual appearance across multiple frames, an attribute we denote as pixel personality. Overall, we reconstruct ~46% of the trajectories in 5 min recordings from two different hives and over 71% of the tracks for at least 2 min. We provide validated trajectories spanning 3000 video frames of 876 unmarked moving bees in two distinct colonies in different locations and filmed with different pixel resolutions, which we expect to be useful in the further development of general-purpose tracking solutions., 13 pages, 4 main and 9 supplementary figures as well as a link to supplementary movies

The original version of this Article omitted from the author list the fourth author Alexander S. Mikheyev, who is from the Ecology and Evolution Unit, OIST Graduate University, Okinawa, Japan, and the Research School of Biology, Australian National University, Canberra, Australia. The third author Yoann Portugal has the following additional affiliation: Ecology and Evolution Unit, OIST Graduate University, Okinawa, Japan. The fourth author Alexander S. Mikheyev and the fifth author Greg J. Stephens declare equal contributions. Consequently, the Acknowledgements, which formerly read “We thank Michael Iuzzolino, Dieu My thanh Nguyen, Orit Peleg, and Michael Smith for comments on the manuscript and code testing. This work was supported by the Okinawa Institute of Science and Technology Graduate University”, have been corrected to “We are grateful to Takahashi Ikemiya for maintaining the experimental bee colonies. We thank Michael Iuzzolino, Dieu My Thanh Nguyen, Orit Peleg, and Michael Smith for comments on the manuscript and code testing. This work was supported by the Okinawa Institute of Science and Technology Graduate University. Additional funding was provided by KAKENHI grants 16H06209 and 16KK0175 from the Japan Society for the Promotion of Science to AM”. Additionally, the Author Contributions, which formerly read “Y.P. performed the bee work and devised the imaging setup, L. H. devised the labeling tool, K.B. performed method development and data analysis, K.B. and G.S. designed the study and wrote the manuscript”, has been corrected to “Y.P. performed the bee work, Y.P. and A.M. devised the imaging setup, L.H. devised the labeling tool, K.B. performed method development and data analysis, K.B., A.M., and G.S. designed the study, K.B. and G.S. wrote the manuscript”. This has been corrected in both the PDF and HTML versions of the Article. The original version of the Supplementary information associated with this Article contained an error in the description of Supplementary Table 2, which incorrectly read “All imaging data in this study were collected in 2019”. The correct version states “2018” in place of “2019”. The HTML has been updated to include a corrected version of the Supplementary information.

Time-lapse microscopy is routinely used to follow cells within organoids, allowing direct study of division and differentiation patterns. There is an increasing interest in cell tracking in organoids, which makes it possible to study their growth and homeostasis at the single-cell level. As tracking these cells by hand is prohibitively time consuming, automation using a computer program is required. Unfortunately, organoids have a high cell density and fast cell movement, which makes automated cell tracking difficult. In this work, a semi-automated cell tracker has been developed. To detect the nuclei, we use a machine learning approach based on a convolutional neural network. To form cell trajectories, we link detections at different time points together using a min-cost flow solver. The tracker raises warnings for situations with likely errors. Rapid changes in nucleus volume and position are reported for manual review, as well as cases where nuclei divide, appear and disappear. When the warning system is adjusted such that virtually error-free lineage trees can be obtained, still less than 2% of all detected nuclei positions are marked for manual analysis. This provides an enormous speed boost over manual cell tracking, while still providing tracking data of the same quality as manual tracking.