Search

Camera-based passive dietary intake monitoring is able to continuously capture the eating episodes of a subject, recording rich visual information, such as the type and volume of food being consumed, as well as the eating behaviors of the subject. However, there currently is no method that is able to incorporate these visual clues and provide a comprehensive context of dietary intake from passive recording (e.g., is the subject sharing food with others, what food the subject is eating, and how much food is left in the bowl). On the other hand, privacy is a major concern while egocentric wearable cameras are used for capturing. In this article, we propose a privacy-preserved secure solution (i.e., egocentric image captioning) for dietary assessment with passive monitoring, which unifies food recognition, volume estimation, and scene understanding. By converting images into rich text descriptions, nutritionists can assess individual dietary intake based on the captions instead of the original images, reducing the risk of privacy leakage from images. To this end, an egocentric dietary image captioning dataset has been built, which consists of in-the-wild images captured by head-worn and chest-worn cameras in field studies in Ghana. A novel transformer-based architecture is designed to caption egocentric dietary images. Comprehensive experiments have been conducted to evaluate the effectiveness and to justify the design of the proposed architecture for egocentric dietary image captioning. To the best of our knowledge, this is the first work that applies image captioning for dietary intake assessment in real-life settings.

A detailed assessment of smoking behavior under free-living conditions is a key challenge for health behavior research. A number of methods using wearable sensors and puff topography devices have been developed for smoking and individual puff detection. In this paper, we propose a novel algorithm for automatic detection of puffs in smoking episodes by using a combination of Respiratory Inductance Plethysmography and Inertial Measurement Unit sensors. The detection of puffs was performed by using a deep network containing convolutional and recurrent neural networks. Convolutional neural networks (CNN) were utilized to automate feature learning from raw sensor streams. Long Short Term Memory (LSTM) network layers were utilized to obtain the temporal dynamics of sensor signals and classify sequence of time segmented sensor streams. An evaluation was performed by using a large, challenging dataset containing 467 smoking events from 40 participants under free-living conditions. The proposed approach achieved an F1-score of 78% in leave-one-subject-out cross-validation. The results suggest that CNN-LSTM based neural network architecture sufficiently detect puffing episodes in free-living condition. The proposed model be used as a detection tool for smoking cessation programs and scientific research.

Automatic food portion size estimation (FPSE) with minimal user burden is a challenging task. Most of the existing FPSE methods use fiducial markers and/or virtual models as dimensional references. An alternative approach is to estimate the dimensions of the eating containers prior to estimating the portion size. In this article, we propose a wearable sensor system (the automatic ingestion monitor integrated with a ranging sensor) and a related method for the estimation of dimensions of plates and bowls. The contributions of this study are: 1) the model eliminates the need for fiducial markers; 2) the camera system [automatic ingestion monitor version 2 (AIM-2)] is not restricted in terms of positioning relative to the food item; 3) our model accounts for radial lens distortion caused due to lens aberrations; 4) a ranging sensor directly gives the distance between the sensor and the eating surface; 5) the model is not restricted to circular plates; and 6) the proposed system implements a passive method that can be used for assessment of container dimensions with minimum user interaction. The error rates (mean ± std. dev) for dimension estimation were 2.01% ± 4.10% for plate widths/diameters, 2.75% ± 38.11% for bowl heights, and 4.58% ± 6.78% for bowl diameters.

The science and tools of measuring energy intake and output in humans have rapidly advanced in the last decade. Engineered devices such as wearables and sensors, software applications, and Web-based tools are now ubiquitous in both research and consumer environments. The assessment of energy expenditure in particular has progressed from reliance on self-report instruments to advanced technologies requiring collaboration across multiple disciplines, from optics to accelerometry. In contrast, assessing energy intake still heavily relies on self-report mechanisms. Although these tools have improved, moving from paper-based to online reporting, considerable room for refinement remains in existing tools, and great opportunities exist for novel, transformational tools, including those using spectroscopy and chemo-sensing. This report reviews the state of the science, and the opportunities and challenges in existing and emerging technologies, from the perspectives of 3 key stakeholders: researchers, users, and developers. Each stakeholder approaches these tools with unique requirements: researchers are concerned with validity, accuracy, data detail and abundance, and ethical use; users with ease of use and privacy; and developers with high adherence and utilization, intellectual property, licensing rights, and monetization. Cross-cutting concerns include frequent updating and integration of the food and nutrient databases on which assessments rely, improving accessibility and reducing disparities in use, and maintaining reliable technical assistance. These contextual challenges are discussed in terms of opportunities and further steps in the direction of personalized health.Statement of Significance: This article is the first to discuss the status and challenges of current and emerging technology tools designed to measure individual food intake, eating behavior, and physical activity through the perspectives of 3 stakeholders: researchers, users, and developers. The objective of this work is to bring together experts to address interdisciplinary and cross-cutting issues with the shared mission of improving the measurement of energy intake and expenditure.

The science and tools of measuring energy intake and output in humans have rapidly advanced in the last decade. Engineered devices such as wearables and sensors, software applications, and Web-based tools are now ubiquitous in both research and consumer environments. The assessment of energy expenditure in particular has progressed from reliance on self-report instruments to advanced technologies requiring collaboration across multiple disciplines, from optics to accelerometry. In contrast, assessing energy intake still heavily relies on self-report mechanisms. Although these tools have improved, moving from paper-based to online reporting, considerable room for refinement remains in existing tools, and great opportunities exist for novel, transformational tools, including those using spectroscopy and chemo-sensing. This report reviews the state of the science, and the opportunities and challenges in existing and emerging technologies, from the perspectives of 3 key stakeholders: researchers, users, and developers. Each stakeholder approaches these tools with unique requirements: researchers are concerned with validity, accuracy, data detail and abundance, and ethical use; users with ease of use and privacy; and developers with high adherence and utilization, intellectual property, licensing rights, and monetization. Cross-cutting concerns include frequent updating and integration of the food and nutrient databases on which assessments rely, improving accessibility and reducing disparities in use, and maintaining reliable technical assistance. These contextual challenges are discussed in terms of opportunities and further steps in the direction of personalized health.

At present, the sustainability of a sensor becomes a hindrance due to energy storage’s limited lifetime. In this case, self-powered sensors are proposed to solve this issue. While self-powered sensors are promising, there are still some challenges to overcome before they can be deployed on a large scale. Therefore, this Special Issue aims to solicit original papers with discussions on self-powered sensors, as well as their architectures, applications, and challenges.

Iris recognition has expanded from controlled settings to uncontrolled settings (on the move, from a distance) where blur is more likely to be present in the images. More research is needed to quantify the impact of blur on iris recognition. In this paper we study the effect of out-of-focus blur on iris recognition performance from images captured with out-of-focus blur produced at acquisition. A key aspect to this study is that we are able to create a range of blur based on changing focus of the camera during acquisition. We quantify the produced out-of-focus blur based on the Laplacian of Gaussian operator and compare it to the gold standard of the modulation transfer function (MTF) of a calibrated black/white chart. The sharpness measure uses an unsegmented iris images from a video sequence with changing focus and offers a good approximation of the standard MTF. We examined the effect of the 9 blur levels on iris recognition performance. Our results have shown that for moderately blurry images (sharpness at least 50%) the drop in performance does not exceed 5% from the baseline (100% sharpness).

Actigraphy offers one of the best-known alternatives to polysomnography for sleep–wake identification. The advantages of actigraphy include high accuracy, simplicity of use and low intrusiveness. These features allow the use of actigraphy for determining sleep–wake states in such highly sensitive groups as infants. This study utilizes a motion sensor (accelerometer) for a dual purpose: to determine an infant's position in the crib and to identify sleep–wake states. The accelerometer was positioned over the sacral region on the infant's diaper, unlike commonly used attachment to an ankle. Opposed to broadly used discriminant analysis, this study utilized logistic regression and neural networks as predictors. The accuracy of predicted sleep–wake states was established in comparison to the sleep–wake states recorded by technicians in a polysomnograph study. Both statistical and neural predictors of this study provide an accuracy of approximately 77–92% which is comparable to similar studies achieving prediction rates of 85–95%, thus validating the suggested methodology. The results support the use of body motion as a simple and reliable method for determining sleep–wake states in infants. Nonlinear mapping capabilities of the neural network benefit the accuracy of sleep–wake state identification. Utilization of the accelerometer for the dual purpose allows us to minimize intrusiveness of home infant monitors.

Actigraphy offers one of the best-known alternatives to polysomnography for sleep-wake identification. The advantages of actigraphy include high accuracy, simplicity of use and low intrusiveness. These features allow the use of actigraphy for determining sleep-wake states in such highly sensitive groups as infants. This study utilizes a motion sensor (accelerometer) for a dual purpose: to determine an infant's position in the crib and to identify sleep-wake states. The accelerometer was positioned over the sacral region on the infant's diaper, unlike commonly used attachment to an ankle. Opposed to broadly used discriminant analysis, this study utilized logistic regression and neural networks as predictors. The accuracy of predicted sleep-wake states was established in comparison to the sleep-wake states recorded by technicians in a polysomnograph study. Both statistical and neural predictors of this study provide an accuracy of approximately 77-92% which is comparable to similar studies achieving prediction rates of 85-95%, thus validating the suggested methodology. The results support the use of body motion as a simple and reliable method for determining sleep-wake states in infants. Nonlinear mapping capabilities of the neural network benefit the accuracy of sleep-wake state identification. Utilization of the accelerometer for the dual purpose allows us to minimize intrusiveness of home infant monitors.

Rapid weight gain during infancy increases the risk of obesity. Given that infant feeding may contribute to rapid weight gain, it would be useful to develop objective tools which can monitor infant feeding behavior. This paper presents an objective method for examining infant sucking count during meals. A piezoelectric jaw motion sensor and a video camera were used to monitor jaw motions of 10 infants during a meal. Videotapes and sensor signals were annotated by two independent human raters, counting the number of sucks in each 10 second epoch. Annotated data were used as a gold standard for the development of the computer algorithms. The sensor signal was de-noised and normalized prior to computing the per-epoch sucking counts. A leave-one-out cross-validation scheme resulted in a mean error rate of -9.7% and an average intra-class correlation coefficient value of 0.86 between the human raters and the algorithm.