Search

Developing interactive lifelog retrieval systems is a growing research area. There are many international competitions for lifelog retrieval that encourage researchers to build effective systems that can address the multimodal retrieval challenge of lifelogs. The Lifelog Search Challenge (LSC) was first organised in 2018 and is currently the only interactive benchmarking evaluation for lifelog retrieval systems. Participating systems should have an accurate search engine and a user-friendly interface that can help users to retrieve relevant content. In this paper, we upgrade our previous MyScéal, which was the top performing system in LSC'20 and LSC'21, and present E-MyScéal for LSC'22, which includes a completely different search engine. Instead of using visual concepts for retrieval such as MyScéal, the new E-MyScéal employs an embedding technique that facilitates novice users who are not familiar with the concepts. Our experiments show that the new search engine can find relevant images in the first place in the ranked list, four a quarter of the LSC'21 queries (26%) by using just the first hint from the textual information need. Regarding the user interface, we still keep the simple non-faceted design as in the previous version but improve the event view browsing in order to better support novice users.

BackgroundOver the past decade, the wide availability and small size of different types of sensors, together with the decrease in pricing, have allowed the acquisition of a substantial amount of data about a person’s life in real time. These sensors can be incorporated into personal electronic devices available at a reasonable cost, such as smartphones and small wearable devices. They allow the acquisition of images, audio, location, physical activity, and physiological signals among other data. With these data, usually denoted as lifelog data, we can then analyze and understand personal experiences and behaviors. This process is called lifelogging.ObjectiveThe objective of this paper was to present a narrative review of the existing literature about lifelogging over the past decade. To achieve this goal, we analyzed lifelogging applications used to retrieve relevant information from daily digital data, some of them with the purpose of monitoring and assisting people with memory issues and others designed for memory augmentation. We aimed for this review to be used by researchers to obtain a broad idea of the type of data used, methodologies, and applications available in this research field.MethodsWe followed a narrative review methodology to conduct a comprehensive search for relevant publications in Google Scholar and Scopus databases using lifelog topic–related keywords. A total of 411 publications were retrieved and screened. Of these 411 publications, 114 (27.7%) publications were fully reviewed. In addition, 30 publications were manually included based on our bibliographical knowledge of this research field.ResultsFrom the 144 reviewed publications, a total of 113 (78.5%) were selected and included in this narrative review based on content analysis. The findings of this narrative review suggest that lifelogs are prone to become powerful tools to retrieve memories or increase knowledge about an individual’s experiences or behaviors. Several computational tools are already available for a considerable range of applications. These tools use multimodal data of different natures, with visual lifelogs being one of the most used and rich sources of information. Different approaches and algorithms to process these data are currently in use, as this review will unravel. Moreover, we identified several open questions and possible lines of investigation in lifelogging.ConclusionsThe use of personal lifelogs can be beneficial to improve the quality of our life, as they can serve as tools for memory augmentation or for providing support to people with memory issues. Through the acquisition and analysis of lifelog data, lifelogging systems can create digital memories that can be potentially used as surrogate memory. Through this narrative review, we understand that contextual information can be extracted from lifelogs, which provides an understanding of the daily life of a person based on events, experiences, and behaviors.

Facing a great deal of Lifelog data and complex information, how to extract valuable information and making it easily understood for users is the most urgent and important issue. This paper introduces a method to find personal trajectories from a Lifelog dataset named informative lifelogging which the author collected over the past 8 years. After introducing the Lifelog system and the data structure, this paper propose a method to find personal trajectory in the dataset. Experiment result in the last chapter shows the correctness of this method. The method proposed in this paper provides some way of managing and retrieving the Lifelog data efficiently.

The authors began collecting personal informative lifelogging data in 2011. The data collected in this article is a discontinuous data set, each of which includes one or more images, a GPS message, a description of a location, and a description of the text, we call this informative lifelogging. However, for the purposes automatically building a story from a huge collection of unstructured egocentric data presents major challenges. This paper first introduces the structure and characteristics of the collected data and uses the DB-scan algorithm to classify the data. Then a model for generating a story is proposed, and a model of story generation based on a story template is proposed in the model. The author implemented a complete software system through code, described a story generation model, and gave the key algorithm to generate stories. Through this system, 418 stories were generated automatically, of which 62% of the stories were particularly accurate. The experimental results verify that it is feasible to automatically generate stories based on personal Informative lifelogging data.

Lifelogging is the process of digital tracking of person’s daily experiences for a variety of purposes. In recent years, lifelogging has become an increasingly popular area of research due to not only the growing demands from many applications, such as wellbeing, entertainment, healthcare systems, and intelligent environments, but also to the advances in device technologies that offer the promise to record and store large volumes of personal data using inexpensive tools. However, getting insights from egocentric experience using huge deluge of unlabeled and unstructured data continues to pose major challenges. A large number of research have been conducted in recent years to cover these challenges but there is still a lack of studies that provide a comprehensive survey of the available literature, while most of the existing lifelogging surveys generally focus on only one aspect. This review highlights the advances of state-of-the-art in lifelogging from different angles, including its research history, current applications, activity recognition techniques, moment retrieval, storytelling, privacy and security issues, as well as challenges and future research trends.

An aging society characterized by rising numbers of people in need of assistance and care poses tremendous challenges for care services, hospitals, families, and the whole society. Using assisting ...

Automatic analysis of our daily lives and activities through a first-person lifelog camera provides us with opportunities to improve our life rhythms or to support our limited visual memories. Notably, to express the visual experiences, the task of generating captions from first-person lifelog images has been actively studied in recent years. First-person images involve scenes approximating what users actually see; therein, the visual cues are not enough to express the user’s context since the images are limited by his/her intention. Our challenge is to generate lifelog captions using a meta-perspective called “fourth-person vision”. The “fourth-person vision” is a novel concept which complementary exploits the visual information from the first-, second-, and third-person perspectives. First, we assume human–robot symbiotic scenarios that provide a second-person perspective from the camera mounted on the robot and a third-person perspective from the camera fixed in the symbiotic room. To validate our approach in this scenario, we collect perspective-aware lifelog videos and corresponding caption annotations. Subsequently, we propose a multi-perspective image captioning model composed of an image-wise salient region encoder, an attention module that adaptively fuses the salient regions, and a caption decoder that generates scene descriptions. We demonstrate that our proposed model based on the fourth-person concept can greatly improve the captioning performance against single- and double-perspective models.

The existing action recognition systems mainly focus on generalized methods to categorize human actions. However, the generalized systems cannot attain the same level of recognition performance for new users mainly due to the high variance in terms of human behavior and the way of performing actions, i.e. activity handling. The use of personalized models based on similarity was introduced to overcome the activity handling problem, but the improvement was found to be limited as the similarity was based on physiognomies rather than the behavior. Moreover, human interaction with contextual information has not been studied extensively in the domain of action recognition. Such interactions can provide an edge for both recognizing high-level activities and improving the personalization effect. In this paper, we propose the context-aware personalized human activity recognition (CAPHAR) framework which computes the class association rules between low-level actions/sensor activations and the contextual information to recognize high-level activities. The personalization in CAPHAR leverages the individual behavior process using a similarity metric to reduce the effect of the activity handling problem. The experimental results on the “daily lifelog” dataset show that CAPHAR can achieve at most 23.73% better accuracy for new users in comparison to the existing classification methods.

Recently, the “lifelogging” and “lifelog” terminologies are frequently used to represent the activity of continuously recording people's everyday experiences, and the dataset contained these recorded experiences, respectively. Hence, providing an excellent tool to retrieve life moments from lifelogs to fast and accurately bring a memory back to a human when required, has become a challenging but exciting task for researchers. In this paper, a new method to meet this challenge by utilizing the hypothesis that “a sequence of images taken during a specific period can share the same context and content” is introduced. This manuscript per the authors introduces a new system to overcome the drawbacks. The experimental results confirm the high productivity of the proposed method in both stable and accuracy aspects as well as the advantage of having an interactive schema to push the accuracy when there is a conflict between a query and how to interpret such a query. Especially, this system requires only a few interactive steps to retrieve relevant images with high accuracy completely.

Vehicular lifelogging provides us with the opportunity to digitally record life associated with our journeys on the road. Driving on the road is an integral part of life, and it should be given unparalleled attention to log life bits on the road for many purposes, including safety, entertainment, memory augmentation, and analyzing driving behaviors. Various technologies have been used to assist people in capturing live events such as wearable devices, biometric devices, fitness devices, non-visual wearable, and unwearable devices. These devices have played a vital role in capturing the life events of individuals. However, little attention has been given to capturing and recording events related to a person's life on the road. By using a vehicle's integrated sensors along with other auxiliary devices, a vehicle can very conveniently capture personal as well as vehicle-related data of the driver. This research aims to propose a framework, termed as AutoLog, which is implemented in the Android platform and can be installed on any android-based smartphones/tablets or a vehicle's infotainment system. AutoLog is capable of logging the driver activities and vehicle dynamics along with several varying environmental contexts. The proposed solution is evaluated through an empirical study by collecting data from 63 drivers. Different tests, i.e., descriptive tabulation, Cronbach alpha, Kendall's tau-b, and Principal Component Factor Analysis (PCFA) have been carried out to analyze the data. These tests are significant, and the findings show that the use of AutoLog application has a positive perception in terms of attitude, ease of use, user satisfaction, and intention to use. Besides, the findings also prove that the proposed solution helped the drivers in improving risky driving behavior.

Describing people's lifestyle has become a hot topic in the field of artificial intelligence. Lifelogging is described as the process of collecting personal activity data describing the daily behaviour of a person. Nowadays, the development of new technologies and the increasing use of wearable sensors allow to automatically record data from our daily living. In this paper, we describe our developed automatic tools for the analysis of collected visual data that describes the daily behaviour of a person. For this analysis, we rely on sequences of images collected by wearable cameras, which are called egocentric photo-streams. These images are a rich source of information about the behaviour of the camera wearer since they show an objective and first-person view of his or her lifestyle.

Contains fulltext : 219674.pdf (Publisher’s version ) (Open Access) Lifelog photo review is considered to enhance the recall of personal events. While a sizable body of research has explored the neural basis of autobiographical memory (AM), there is limited neural evidence on the retrieval-based enhancement effect on event memory among older adults in the real-world environment. This study examined the neural processes of AM as was modulated by retrieval practice through lifelog photo review in older adults. In the experiment, blood-oxygen-level dependent response during subjects' recall of recent events was recorded, where events were cued by photos that may or may not have been exposed to a priori retrieval practice (training). Subjects remembered more episodic details under the trained relative to non-trained condition. Importantly, the neural correlates of AM was exhibited by (1) dissociable cortical areas related to recollection and familiarity, and (2) a positive correlation between the amount of recollected episodic details and cortical activation within several lateral temporal and parietal regions. Further analysis of the brain activation pattern at a few regions of interest within the core remember network showed a training_condition x event_detail interaction effect, suggesting that the boosting effect of retrieval practice depended on the level of recollected event details.

The article is addressed to the consideration of the subject organization of lyrics by I. Kholin, a representative of the Lianozovo literary group, who became widely known as the author of the book of poems “Residents of the Barack”. Already in the presented, debut for the poet, composition there were outlined the types of speech widely presented to him, in which the subject is likened to a life-log camera, and the focus of attention is shifted from “Self” to “you”, from internal to external to the speaking world. The term “life-logging”, meaning the automatic fixation of a person's life using a video medium fixed to the body, is used in the article as a literary metaphor, “highlighting” the difficulty of personal speaking and self-expression of the protagonist. However, just as the environment of a life logger who does not fall into his own lens gives him a sufficient (if not exhaustive) idea, the life realities in the subject's field of view can tell about his inner world no less than he by himself. The analysis of a number of poems (the most detailed is considered “Fences. Garbage cans. Posters. Advertising”) allows you to demonstrate the intersubjective nature of life-logic “optics”. The latter is used by Kholin in three different forms: as the “dissolution” of the speaker in the text, as the construction of the statement on behalf of the syncretic subject and as the priority of the “other” over the “self” when creating the verbal “self-portrait” of the hero. The impossibility of distancing from hostile reality, but also the inadequacy of selfdetermination in its conditions, testify to the formation of a kenotic model of the artist's relationship with reality. In Kholin's poetry, the lyrical subject is not only a detached viewer, but also a protagonist who fully shares his sins and suffering with the world.

In this paper, we introduce LifeSeeker 4.0 - an interactive lifelog retrieval system developed for the fifth annual Lifelog Search Challenge (LSC'22). In LifeSeeker 4.0, we focus on enhancing our previous system to allow users who have little to no knowledge of underlying system functioning and lifelog data to use it with ease by employing a Contrastive Language-Image Pre-training (CLIP) model. Furthermore, we have exploited the music metadata to facilitate searches that may incorporate emotion. Event clustering is also improved in this version to increase user experience by reducing the occurrence of repeated images, and hence decreasing the search time.

For the fifth time since 2018, the Lifelog Search Challenge (LSC) facilitated a benchmarking exercise to compare interactive search systems designed for multimodal lifelogs. LSC'22 attracted nine participating research groups who developed interactive lifelog retrieval systems enabling fast and effective access to lifelogs. The systems competed in front of a hybrid audience at the LSC workshop at ACM ICMR'22. This paper presents an introduction to the LSC workshop, the new (larger) dataset used in the competition, and introduces the participating lifelog search systems.

Smartphone-based Lifelog (automatically annotating the users' daily experience from multisensory streams on smartphones) is in great need. Accurate positioning under any situation is one of the most significant techniques for a desirable Lifelog. This paper proposes to detect location-related activities and use the activity information to improve positioning accuracy. In the proposed system, a human activity recognition module is developed to extract location-related activities from multisensory streams of smartphones. After that, the proposed system integrates activity information with PDR-based positioning results in a context-based map-matching framework. The developed system can be used for both outdoor and indoor scenarios. Moreover, the developed indoor positioning method is used to determine the positions of calibration points automatically in an auto-calibration Wi-Fi positioning system. The proposed methods achieve 3.1-m accuracy in outdoor and average 2.2-m accuracy in indoor situations.

As the process of creating and sharing data about ourselves becomes more prevalent, researchers have access to increasingly rich data about human behavior. Framed as a fictional paper published at some point in the not-so-distant future, this design fiction draws from current inquiry and debate into the ethics of using public data for research, and speculatively extends this conversation into even more robust and more personal data that could exist when we design new technologies in the future. By looking to how the precedents of today might impact the practices of tomorrow, we can consider how we might design policies, ethical guidelines, and technologies that are forward-thinking.

Building an interactive retrieval system for lifelogging contains many challenges due to massive multi-modal personal data besides the requirement of accuracy and rapid response for such a tool. The Lifelog Search Challenge (LSC) is the international lifelog retrieval competition that inspires researchers to develop their systems to cope with the challenges and evaluates the effectiveness of their solutions. In this paper, we upgrade our previous Myscéal 2.0 and present Myscéal 2.0 system for the LSC'21 with the improved features inspired by the novice users experiments. The experiments show that a novice user achieved more than half of the expert score on average. To mitigate the gap of them, some potential enhancements were identified and integrated to the enhanced version.

In Japan, the number of college students suffering from changes in the communicative environment due to the COVID-19 pandemic is increasing. However, many college students do not use on-campus counseling services even though they are available. The reason for this is students’ psychological barriers to counseling. Therefore, we designed a chatbot to reduce the psychological barrier and recommend counseling services to college students. We defined dialogue tasks for maintaining mental health, i.e., lifelogging, confiding, and recommending counseling services. We also defined the required and optional slots for each dialogue task. We used crowdsourcing to collect an example dataset of ideal dialogues between chatbots and users. From the collected data, we categorized the chatbot's responses and design a task- and slot-based response-selection method that is based on Aspect-based Sentiment Analysis.

Nowadays, almost everyone holds some form or other of a personal life archive. Automatically maintaining such an archive is an activity that is becoming increasingly common, however without automatic support the users will quickly be overwhelmed by the volume of data and will miss out on the potential benefits that lifelogs provide. This research is to build an effective and efficient lifelog information retrieval system, which can address the challenges of organising and searching personal life archives, using advanced IR models and ranking approaches. The main contributions of this thesis are as follows. Firstly, lifelog data is defined and a first generation of lifelog datasets are constructed based on our proposed process. Secondly, a baseline search engine is proposed and developed for lifelog data collection, which aims to make the lifelog data searchable, organizable and serves as a baseline comparison for other lifelog retrieval systems that took part in the various benchmarking activities we organised. Thirdly, research on investigating and enriching image concept features using human-object interaction(HOI) annotations are presented and a causality score between different HOIs are calculated and automatic / manual HOI features are compared to validate which feature is most valuable for lifelog moment retrieval. Finally, the approaches presented use pre-trained textual and visual-semantic embeddings for the lifelog retrieval problem and enhance retrieval performance with a high-level semantic mapping of user information. In summary, this thesis primarily contributes to the field of building an effective and efficient lifelog information retrieval system and delivers a number of pioneering contributions.

Filming video blogs, which is shortened to vlog, becomes a popular way for people to record their life experiences in recent years. In this work, we present a novel task that is aimed at extracting life events from videos and constructing personal knowledge bases of individuals. In contrast to most existing researches in the field of computer vision that focus on identifying low-level script-like activities such as moving boxes, our goal is to extract life events where high-level activities like moving into a new house are recorded. The challenges to be tackled include: (1) identifying which objects in a given scene related to the life events of the protagonist we concern, and (2) determining the association between an extracted visual concept and a more high-level description of a video clip. To address the research issues, we construct a video life event extraction dataset VidLife by exploiting videos from the TV series The Big Bang Theory, in which the plot is around the daily lives of several characters. A pilot multitask learning model is proposed to extract life events given video clips and subtitles for storing in the personal knowledge base.

The multi-modal and interrelated nature of lifelog data makes it well suited for graph-based representations. In this paper, we present the second iteration of LifeGraph, a Knowledge Graph for Lifelog Data, initially introduced during the 3rd Lifelog Search Challenge in 2020. This second iteration incorporates several lessons learned from the previous version. While the actual graph has undergone only small changes, the mechanisms by which it is traversed during querying as well as the underlying storage system which performs the traversal have been changed. The means for query formulation have also been slightly extended in capability and made more efficient and intuitive. All these changes have the aim of improving result quality and reducing query time.

The multimodal nature of lifelog data collections poses unique challenges for multimedia management and retrieval systems. The Lifelog Search Challenge (LSC) offers an annual evaluation platform for such interactive retrieval systems. They compete against one another in finding items of interest within a set time frame. In this paper, we present the multimedia retrieval system vitrivr-VR, the latest addition to the vitrivr stack, which participated in the LSC in recent years. vitrivr-VR leverages the 3D space in virtual reality (VR) to offer novel retrieval and user interaction models, which we describe with a special focus on design decisions taken for the participation in the LSC.

Device-to-Device (D2D) communications in this paper refer to autonomous direct communications among end devices without central control. Wireless networks based on autonomous D2D communications are of resiliency feature. We developed a lifelog gathering system using such autonomous D2D communications towards elderly health monitoring in depopulated area, where creating wireless networks with coverage over entire area is costly. The developed system consists of three types of terminals, i.e., sensing terminal (s-terminal), delivering terminal (d-terminal), and gathering terminal (g-terminal). The s-terminal collects lifelog at objective’s house. The d-terminal can be placed in a moving car and it receives lifelog data when running within communication range of the objective’s house. Moreover, the lifelog is forwarded to the g-terminal when the d-terminal enters the communication range of facilities where the g-terminal is in operation. We conducted field experiments in a mountainous village. The measured results show that the developed system can successfully gather lifelog data even under non line-of-sight environments. Examples of observation from the gathered lifelog data are also presented.

With the advance of science and technology, people are used to recording their daily life events via writing blogs, uploading social media posts, taking photos, or filming videos. Such rich repository personal information is useful for supporting human living assistance, such as information recall service. The main challenges are how to store and manage personal knowledge from various sources, and how to provide support for people who may have difficulty recalling past experiences. In this position paper, we propose a research agenda on personal knowledge mining from various sources of lifelogs, personal knowledge base construction, and information recall for assisting people to recall their experiences. Ten research questions are formulated.

Image retrieval from a given text query (text-to-image retrieval) is one of the most essential systems, and it is effectively utilized for databases (DBs) on the Web. To make them more versatile and familiar, a retrieval system that is adaptive even for personal DBs such as images in smartphones and lifelogging devices should be considered. In this paper, we present a novel text-to-image retrieval system that is specialized for personal DBs. With the cross-modal scheme and the question-answering scheme, the developed system enables users to obtain the desired image effectively even from personal DBs. Our demo is available at https://sites.google.com/view/ir-questioner/.

The Lifelog Search Challenge (LSC) is an annual benchmarking challenge for comparing approaches to interactive retrieval from multi-modal lifelogs. LSC'21, the fourth challenge, attracted sixteen participants, each of which had developed interactive retrieval systems for large multimodal lifelogs. These interactive retrieval systems participated in a comparative evaluation in front of an online live-audience at the LSC workshop at ACM ICMR'21. This overview presents the motivation for LSC'21, the lifelog dataset used in the competition, and the participating systems.

In a society dominated by mobile phones and still increasing media collections, Interactive Learning is slowly becoming the favored paradigm for managing these collections. Still, however, no scaling Interactive Learning system exists on a mobile phone. In this paper, we present XQC, an Interactive Learning platform with a user interface that fits most modern smartphones, and scales to large media collections.

Since computers were first used to address the challenge of managing information rather than performing calculations computing arithmetic values, or even before that since the time that MEMEX was designed by Vannevar Bush in the 1940s, we have been building systems that help people like us to find information accurately and quickly. These systems have grown to be technological marvels, discovering and indexing information almost as soon as it appears available online and making it available to billions of people for searching and delivery within fractions of a second, and across a range of devices. Yet it is well known that half the time people are actually searching for things that they once knew but have since forgotten, or can't remember where they found that information first time around, and need to re-find it. As our science of information seeking and information discovery has progressed, we rarely ask why people forgot those things in the first place. If we were allowed to jump back in time say 50 years, and to re-start the development of information retrieval as a technology then perhaps we would be build systems that help us to remember and to learn, rather than trying to plug the gap and find information for us when we forget. In separate but parallel and sometimes overlapping developments, the analysis and indexing of visual information -- images and video -- has also made spectacular progress mostly within the last decade. Using automated processes we can detect and track objects, we can describe visual content as tags or even as text captions, we can now generate realistic high quality visual content using machine learning and we can compute high-level abstract features of visual content like salience, aesthetics, and even memorability. One of the areas where information management/retrieval with its 50 years of technological progress meets computer vision with its recent decade of spectacular development is in lifelogging. At this intersection we can apply computer vision techniques to analyse and index visual lifelogs generated from wearable cameras, for example, in order to support lifelog search and browsing tasks. But we should ask ourselves whether this really is the right way for us to use our lifelogs. Memory is one of the core features that make us what we are yet it is fragile and only partly understood. We have no real control over what we remember and what we forget and when we really do need to remember something that could be important then we make ham-fisted efforts to consciously over-ride our natural tendency to forget. We do this, for example, rehearsing and replaying information, building on the Ebbinghaus principle of repeated conscious reviewing to overcome transience which is the general deterioration of memory over time. In this presentation I will probe deeper into memory, recall, recognition, memorability and memory triggers and how our lifelogs could really act as memory prosthetics, visual triggers for our own natural memory. This will allow us to ask whether the lifelog challenges that we build and run in events such as this Annual Lifelog Search Challenge meeting are appropriately framed and whether they are taking us in the direction where lifelogs are genuinely useful to a wide population rather than to a niche set of people. Finally I will address the frightening scenario where everything for us is potentially remembered and whether or not we want that to actually happen.

In this paper, we present the interactive lifelog retrieval engine developed for the LSC’21 comparative benchmarking challenge. The LifeSeeker 3.0 interactive lifelog retrieval engine is an enhanced version of our previous system participating in LSC’20 - LifeSeeker 2.0. The system is developed by both Dublin City University and the Ho Chi Minh City University of Science. The implementation of LifeSeeker 3.0 focuses on searching and filtering by text query using a weighted Bag-of-Words model with visual concept augmentation and three weighted vocabularies. The visual similarity search is improved using a bag of local convolutional features; while improving the previous version’s performance, enhancing query processing time, result displaying, and browsing support.

SOMHunter represents a modern light-weight framework for known-item search in datasets of visual data like images or videos. The framework combines an effective W2VV++ text-to-image search approach, a traditional Bayesian like model for maintenance of relevance scores influenced by positive examples, and several types of exploration and exploitation displays. With this initial setting in 2020, already the first prototype of the system turned out to be highly competitive in comparison with other state-of-the-art systems at Video Browser Showdown and Lifelog Search Challenge competitions. In this paper, we present a new version of the system further extending the list of visual data search capabilities. The new version combines localized text queries with collage queries tested at VBS 2021 in two separate systems by our team. Furthermore, the new version of SOMHunter will integrate also the new CLIP text search model recently released by OpenAI. We believe that all the extensions will improve chances to effectively initialize the search that can continue with already supported browsing capabilities.

We present LifeMon, a new lifelog retrieval prototype targeting LSC. LifeMon is based around the MongoDB document store, which is one of a host of scalable NoSQL systems developed over the last two decades, with a semi-structured data model that seems well matched with lifelog requirements. Preliminary results indicate that the system is efficient and that novice users can successfully use it to solve some LSC tasks.

In this paper, we describe an extended version of Voxento which is an interactive voice-based retrieval system for lifelogs that has been developed to participate in the fourth Lifelog Search Challenge LSC’21, at ACM ICMR’21. Voxento provides a spoken interface to the lifelog dataset, which facilitates a novice user to interact with a personal lifelog using a range of vocal commands and interactions. For the version presented here, Voxento has been enhanced with new retrieval features and better user interaction support. In this paper, we introduce these new features, which include dynamic result filtering, predefined interactive responses and the development of a new retrieval API. Although Voxento was proposed for wearable technologies such as Google Glass or interactive devices like smart TVs, the version of Voxento presented here uses a desktop computer in order to participate in the LSC’21 competition. In the current Voxento iteration, the user has the option to enable voice interaction or use standard text-based retrieval.

With a huge collection of photos and video clips, it is essential to provide an efficient and easy-to-use system for users to retrieve moments of interest with a wide variation of query types. This motivates us to develop and upgrade our flexible interactive retrieval system for visual lifelog exploration. In this paper, we briefly introduce version 2 of our system with the following main features. Our system supports multiple modalities for interaction and query processing, including visual query by meta-data, text query and visual information matching based on a joint embedding model, scene clustering based on visual and location information, flexible temporal event navigation, and query expansion with visual examples. With the flexibility in system architecture, we expect our system can easily integrate new modules to enhance its functionalities.

Exquisitor is a scalable media exploration system based on interactive learning. To satisfy a user's information need, the system asks the user for feedback on media items and uses that feedback to interactively construct a classifier, that is in turn used to identify the next potentially relevant set of media items. To facilitate effective exploration of a collection, the system offers filters to narrow the scope of exploration, search functionality for finding good examples for the classifier, and support for timeline browsing of videos or image sequences. For this year's Lifelog Search Challenge, we have enhanced Exquisitor to better support tasks with a temporal component, by adding features that allow the user to build multiple classifiers and merge the classifier results, using both traditional set operators and advanced temporal operators.

The major challenge in visual lifelog retrieval is the semantic gap between textual queries and visual concepts. This paper presents our work on the Lifelog Search Challenge 2021 (LSC'21), an annual comparative benchmarking activity for comparing approaches to interactive retrieval from multimodal lifelogs. We propose LifeConcept, an interactive lifelog search system that is aimed at accelerating the retrieval process and retrieving more precise results. In this work, we introduce several new features such as the number of people, location cluster, and object with color. Moreover, we obtain visual concepts from the images with computer vision models and propose a concept recommendation method to reduce the semantic gap. In this way, users can efficiently set up the related conditions for their requirements and search the desired images with appropriate query terms based on the suggestion.

In this paper, we present the interactive lifelog retrieval engine developed for the LSC'21 comparative benchmarking challenge. The LifeSeeker 3.0 interactive lifelog retrieval engine is an enhanced version of our previous system participating in LSC'20 - LifeSeeker 2.0. The system is developed by both Dublin City University and the Ho Chi Minh City University of Science. The implementation of LifeSeeker 3.0 focuses on searching and filtering by text query using a weighted Bag-of-Words model with visual concept augmentation and three weighted vocabularies. The visual similarity search is improved using a bag of local convolutional features; while improving the previous version's performance, enhancing query processing time, result displaying, and browsing support.

Since its first iteration in 2018, the Lifelog Search Challenge (LSC) continues to rise in popularity as an interactive lifelog data retrieval competition, co-located at the ACM International Conference on Multimedia Retrieval (ICMR). The goal of this annual live event is to search a large corpus of lifelogging data for specifically announced memories using a purposefully developed tool within a limited amount of time. As long-standing participants, we present our improved lifeXplore -- a retrieval system combining chronologic day summary browsing with interactive combinable concept filtering. Compared to previous versions, the tool is improved by incorporating temporal queries, advanced day summary features as well as usability improvements.

The personal digital twin extends to individual persons, a concept that originated in engineering to twin complex machines with a digital simulation containing a model of its functions to monitor its past and present behaviour, and repair, correct, improve or otherwise ensure its optimal operation. Several independent trends in technological developments are seen to converge towards the elaboration of the digital replication of individual human data and life history, notably in health industries. Among the main ones, we consider the ubiquitous distribution of digital assistants, the rapid progress of machine learning concurrent with the exponential growth of ‘personal’ Big Data and the incipient interest in developing lifelogs. The core hypothesis here is that among the psychological effects of the digital transformation, the externalization of cognitive faculties such as memory, planning and judgement, the decision-making processes located within the human person are also emigrating to digital functions, perhaps as a prelude to a later re-integration within the person via brain–computer interfaces. The paper concludes with ethical considerations about these ongoing developments. This article is part of the theme issue ‘Towards symbiotic autonomous systems’.

We address in this work the topic of broadcasting one’s visual reality, captured by the video cameras embedded in mobile and wearable devices, to a remote audience. We discuss several designs of such life broadcasting systems, which we position at the intersection of lifelogging, Alternate Reality, and Cross-Reality technology. To this end, we introduce the “Alternate Reality Broadcast-Time” matrix for the broadcasting and consumption of alternate realities, in which designs of systems that implement sharing and consumption of personal visual realities can be positioned, characterized, and compared. These design options range from simple video streaming over the web using conventional video protocols to mediated and augmented reality, to audio narration and vibrotactile rendering of concepts automatically detected from video captured by wearable cameras. To demonstrate the usefulness of our broadcast-time matrix, we describe three prototypes implementing lifelogging, concept recognition from video, augmented and mediated vision, and vibrotactile feedback. Our contributions open the way toward new applications that blend lifelogging, consumption of multimedia alternate realities, and XR technology to empower users with richer opportunities for self-expression and new means to connect with the followers of events in their lives.

Self-awareness is an essential concept in physiology and psychology. Accurate overall self-awareness benefits the development and well being of an individual. The previous research studies on self-awareness mainly collect and analyze data in the laboratory environment through questionnaires, user study, or field research study. However, these methods are usually not real-time and unavailable for daily life applications. Therefore, we propose a new direction of utilizing lifelog for self-awareness. Lifelog records about daily activities are used for analysis, prediction, and intervention on individual physical and psychological status, which can be automatically processed in real-time. With the help of lifelog, ordinary people are able to understand their condition more precisely, get effective personal advice about health, and even discover physical and mental abnormalities at an early stage. As the first step on using lifelog for self-awareness, we learn from the traditional machine learning problems, and summarize a schema on data collection, feature extraction, label tagging, and model learning in the lifelog scenario. The schema provides a flexible and privacy-protected method for lifelog applications. Following the schema, four topics were conducted: sleep quality prediction, personality detection, mood detection and prediction, and depression detection. Experiments on real datasets show encouraging results on these topics, revealing the significant relation between daily activity records and physical and psychological self-awareness. In the end, we discuss the experiment results and limitations in detail and propose an application, Lifelog Recorder, for multi-dimensional self-awareness lifelog data collection.

The Lifelog Search Challenge (LSC) is a venue where retrieval system researchers compete in solving tasks to retrieve the correct image from a lifelog collection. At LSC 2021, we introduce the PhotoCube system as a new competitor. PhotoCube is an interactive media retrieval system that considers media items to exist in a hypercube in multidimensional metadata space. To solve tasks, users explore the contents of the hypercube by dynamically (a) applying a variety of filters and (b) projecting the hypercube to a three-dimensional cube that is visualised on screen.

In this paper we describe the first iteration of the ViRMA prototype system, a novel approach to multimedia analysis in virtual reality and inspired by the M3 data model. Weevaluated our approach via the Lifelog Search Challenge (LSC) to serve as a benchmark against other multimedia analytics systems.

This paper presents a conversational lifelog mining system, ConvLogMiner, which detects personal life events from the human online conversation in real-time. Given a daily conversation of two speakers, ConvLogMiner identifies the new life events specific to each speaker that occur in the latest utterances. The lifelogs mined by our system are useful to provide complementary information to support lifestyle analysis and memory assistance service.