Search

In this paper, we present the rationale and the ideas behind META4RS, a museum itinerary recommender system. The system leverages deep learning techniques to acquire data about the visitor’s position while ensuring her anonymity. Moreover, the visitor’s appraisal of the artwork she observes is inferred implicitly based on the emotional reactions she expresses while watching a given artwork. We are not aware of any such recommender system proposed in the research literature. However, this system should ensure several advantages: (i) it is non-intrusive since it makes use of simple badges and off-the-shelf cameras while ensuring the anonymity of the visitor; (ii) it is independent of the type of museum; (iii) it offers personalized itineraries to visitors based on their implicitly inferred interests and preferences. Specifically, we illustrate the background and describe the architecture of the proposed system, discussing the steps required for its implementation. We also provide details of what has already been done and what remains to be done, outlining the open problems.

The awareness of the visitor’s role centrality in museums has led to the constant development of techniques and methodologies aimed at improving the visitor’s experience. In this context, Augello et al. suggest the four pillars to enhance the user experience: User Localization, User Understanding, Multimodal Interaction, and Gamification. As part of our research activities, we have focused on the latter two pillars. More specifically, User Localization allows museum curators and staff to understand the user movements in the museum and how much time she spends in front of the artworks. This information can help personalize the user experience to make it more and more satisfying. To collect this information, we believe that rather than using traditional techniques, such as beacons or IR, it is better to leverage deep learning techniques using cameras, thus supporting subsequent analyzes such as an automatic analysis of the user’s face. In particular, analyzing the user’s face allows us to grasp her reactions in front of the artworks and establish whether she may have liked that artwork or not, and if she may want to see similar artworks. The aim of this research is to understand how to use localization systems and automatic face analysis to build a user model that could allow museum staff to customize her experience. The first results obtained regarding an innovative localization system, based on deep learning, show the potential of this technique, which permits the extraction of information from faces through the same cameras. As regards this second part, techniques based on deep neural networks, such as Convolutional and Recurrent Neural Networks, are being tested to identify the best approach to extract implicit feedback from users’ faces without making them fill out long questionnaires that require active participation and steal time from the visit.

Although ubiquitous and fast access to the Internet allows us to admire objects and artworks exhibited worldwide from the comfort of our home, visiting a museum or an exhibition remains an essential experience today. Current technologies can help make that experience even more satisfying. For instance, they can assist the user during the visit, personalizing her experience by suggesting the artworks of her higher interest and providing her with related textual and multimedia content. To this aim, it is necessary to automatically acquire information relating to the active user. In this paper, we show how a deep neural network-based approach can allow us to obtain accurate information for understanding the behavior of the visitor alone or in a group. This information can also be used to identify users similar to the active one to suggest not only personalized itineraries but also possible visiting companions for promoting the museum as a vehicle for social and cultural inclusion.

Current image/video acquisition and analysis techniques allow for not only the identification and classification of objects in a scene but also more sophisticated processing. For example, there are video cameras today able to capture micro facial expressions, namely, facial expressions that occur in a fraction of a second. Such micro expressions can provide useful information to define a person's emotional state. In this article, we propose to use these features to collect useful information for designing and implementing increasingly effective interactive technologies. In particular, facial micro expressions could be used to develop interfaces capable of fostering the social and cultural inclusion of users belonging to different realities and categories. The preliminary experimental results obtained by recording the reactions of individuals while observing artworks demonstrate the existence of correlations between the action units (i.e., single components of the muscular movement in which it is possible to break down facial expressions) and the emotional reactions of a sample of users, as well as correlations within some homogeneous groups of testers.