Search

Recent research has employed deep learning to detect intake gestures from inertial sensor and video camera data. However, the fusion of these modalities has not been attempted. The present research explores the potential of fusing the outputs of two individual deep learning inertial and video intake gesture detection models (i.e., score-level and decision-level fusion) using the test sets from two publicly available multimodal datasets: (1) OREBA-DIS recorded from 100 participants while consuming food served in discrete portions and (2) OREBA-SHA recorded from 102 participants while consuming a communal dish. We first assess the potential of fusion by contrasting the performance of the individual models in intake gesture detection. The assessment shows that fusing the outputs of individual models is more promising on the OREBA-DIS dataset. Subsequently, we conduct experiments using different score-level and decision-level fusion approaches. Our results from fusion show that the score-level fusion approach of max score model performs best of all considered fusion approaches. On the OREBA-DIS dataset, the max score fusion approach ( $F_{1} =0.871$ ) outperforms both individual video ( $F_{1} =0.855$ ) and inertial ( $F_{1} =0.806$ ) models. However, on the OREBA-SHA dataset, the max score fusion approach ( $F_{1} =0.873$ ) fails to outperform the individual inertial model ( $F_{1} =0.895$ ). Pairwise comparisons using bootstrapped samples confirm the statistical significance of these differences in model performance ( $p \lt $ .001).

Smartphone-enabled image and voice recordings provide new avenues for capturing a person’s dietary intake. This can help to reduce the burdens of traditional dietary assessment methods (e.g., journals, observation), facilitating accessibility to under-represented populations. However, there is little research into how one can effectively process the image-voice data captured in this way for accurate dietary assessment. Building on an interdisciplinary team of experts in computing and nutrition, the goal of the present study is to co-design a web-based content management system to facilitate dietary assessment of image-voice dietary intake data. This led to the formulation and evaluation of five design principles for semi-automated dietary assessment of image-voice food records. Integrated into a wider project, the system was evaluated with actual dietary intake data collected in free-living conditions in Cambodia, and generative feedback provided by dietary assessment analysts. We discuss the challenges and experiences of building a system to process image-voice food records, and present the resulting design principles to aid future development.

Traditional methods of capturing people’s dietary intake are complex and labour-intensive, requiring a high level of literacy and time. Speech recording has potential to reduce these barriers, and recent technological advances have greatly increased the viability of this approach. The aim of this paper is to establish the current state of research on the usage of speech records in dietary assessment. To this end, we performed a systematic literature review and summarised the current state of research along a conceptual framework that captures the components involved in using speech records for dietary assessment. Six databases from the nutrition and computing domains were interrogated, resulting in 21 relevant papers. Speech recording in an unstructured format was preferred when compared against other methods by all three studies reporting comparisons. High technological satisfaction and ease of use were noted by all eight studies reporting user acceptance. When recording data, 78% of studies focused on collecting prospective food records. The choice of device reflected this, with 15 of 18 studies reporting a form of handheld, portable collection device intended to be always available. To process data, nine studies performed automated speech transcription achieving an average accuracy of 83%, seven of which utilized a readily available commercial service. Of the five studies that used natural language processing to further automate analysis, an average accuracy of 82% was reported. Further research is required to adapt these prototypes to address practical challenges in dietary assessment and monitoring (e.g. self-monitoring for low-literacy users).

2D images can be used to capture food intake data in nutrition studies. Estimates of food volume from these images are required for nutrient analysis. Although 3D image capture is possible, it is not commonplace. Additionally, nutrition studies often require multiple food images taken by non-expert users, typically collected using mobile phones, due to their convenience. Current 2D image to 3D volume approaches are restricted by the need for prescribed camera placement, image metadata analysis and/or significant computational resources. A new method is presented combining 2D image capture and automated 3D scene projection with manual placement and resizing of wire mesh objects. 2D images, with a reference object, are taken on low specification mobile phones. 3D scene projection is calculated by twinning a cuboid in 3D space to the reference object in the 2D image. A manually selected 3D wire mesh object is then positioned over the target food item and manually transformed to improve accuracy. The virtual wire mesh object is then projected into the 3D scene and the volume of the target food item calculated. The whole process is computationally light and runs in real-time as an app on a standard Apple iPad. Based on a user study with 60 participants, experimental evaluations of volume estimates over regular shape and ground truth food items demonstrate that this approach provides acceptable accuracy. We demonstrate that the accuracy of estimates can be increased by combining multiple independent estimates.

Abstract In a Dutch auction, an item is offered for sale at a set maximum price. The price is then gradually lowered over a fixed interval of time until a bid is made, securing the item for the bidder at the current price. Bidders must trade-off between certainty and price: bid early to secure the item and you pay a premium; bid later at a lower price but risk losing to another bidder. These properties of Dutch auctions provide new opportunities to study competitive decision-making in a group setting. We developed a novel computerised Dutch auction platform and conducted a set of experiments manipulating volatility (fixed vs varied number of items for sale) and price reduction interval rate (step-rate). Triplets of participants ( $$N=66$$ N = 66 ) competed with hypothetical funds against each other. We report null effects of step-rate and volatility on bidding behaviour. We developed a novel adaptation of prospect theory to account for group bidding behaviour by balancing certainty and subjective expected utility. We show the model is sensitive to variation in auction starting price and can predict the associated changes in group bid prices that were observed in our data.

Automatic detection of intake gestures is a key element of automatic dietary monitoring. Several types of sensors, including inertial measurement units (IMU) and video cameras, have been used for this purpose. The common machine learning approaches make use of labeled sensor data to automatically learn how to make detections. One characteristic, especially for deep learning models, is the need for large datasets. To meet this need, we collected the Objectively Recognizing Eating Behavior and Associated Intake (OREBA) dataset. The OREBA dataset aims to provide comprehensive multi-sensor data recorded during the course of communal meals for researchers interested in intake gesture detection. Two scenarios are included, with 100 participants for a discrete dish and 102 participants for a shared dish, totalling 9069 intake gestures. Available sensor data consist of synchronized frontal video and IMU with accelerometer and gyroscope for both hands. We report the details of data collection and annotation, as well as details of sensor processing. The results of studies on IMU and video data involving deep learning models are reported to provide a baseline for future research. Specifically, the best baseline models achieve performances of F1=0.853 for the discrete dish using video and F1=0.852 for the shared dish using inertial data.

Wrist-worn inertial measurement units have emerged as a promising technology to passively capture dietary intake data. State-of-the-art approaches use deep neural networks to process the collected inertial data and detect characteristic hand movements associated with intake gestures. In order to clarify the effects of data preprocessing, sensor modalities, and sensor positions, we collected and labeled inertial data from wrist-worn accelerometers and gyroscopes on both hands of 100 participants in a semi-controlled setting. The method included data preprocessing and data segmentation, followed by a two-stage approach. In Stage 1, we estimated the probability of each inertial data frame being intake or non-intake, benchmarking different deep learning models and architectures. Based on the probabilities estimated in Stage 1, we detected the intake gestures in Stage 2 and calculated the F1 score for each model. Results indicate that top model performance was achieved by a CNN-LSTM with earliest sensor data fusion through a dedicated CNN layer and a target matching technique (F1 = .778). As for data preprocessing, results show that applying a consecutive combination of mirroring, removing gravity effect, and standardization was beneficial for model performance, while smoothing had adverse effects. We further investigate the effectiveness of using different combinations of sensor modalities (i.e., accelerometer and/or gyroscope) and sensor positions (i.e., dominant intake hand and/or non-dominant intake hand).