Search

Your search keyword '"Palotti, Joao"' showing total 123 results
Start Over Author "Palotti, Joao"

Refine your results

more »

more »

more »

more »
123 results on '"Palotti, Joao"'

1. Multilingual Detection of Personal Employment Status on Twitter

Author

Tonneau, Manuel, Adjodah, Dhaval, Palotti, João, Grinberg, Nir, and Fraiberger, Samuel

Subjects
Computer Science - Computation and Language
Abstract

Detecting disclosures of individuals' employment status on social media can provide valuable information to match job seekers with suitable vacancies, offer social protection, or measure labor market flows. However, identifying such personal disclosures is a challenging task due to their rarity in a sea of social media content and the variety of linguistic forms used to describe them. Here, we examine three Active Learning (AL) strategies in real-world settings of extreme class imbalance, and identify five types of disclosures about individuals' employment status (e.g. job loss) in three languages using BERT-based classification models. Our findings show that, even under extreme imbalance settings, a small number of AL iterations is sufficient to obtain large and significant gains in precision, recall, and diversity of results compared to a supervised baseline with the same number of labels. We also find that no AL strategy consistently outperforms the rest. Qualitative analysis suggests that AL helps focus the attention mechanism of BERT on core terms and adjust the boundaries of semantic expansion, highlighting the importance of interpretable models to provide greater control and visibility into this dynamic learning process., Comment: ACL 2022 main conference. Data and models available at https://github.com/manueltonneau/twitter-unemployment

Published
2022
Full Text
View/download PDF

2. Towards Accurate and Efficient Sleep Period Detection Using Wearable Devices

Author

Jokar, Fatemeh, Azzopardi, George, Palotti, Joao, Goos, Gerhard, Founding Editor, Hartmanis, Juris, Founding Editor, Bertino, Elisa, Editorial Board Member, Gao, Wen, Editorial Board Member, Steffen, Bernhard, Editorial Board Member, Yung, Moti, Editorial Board Member, Tsapatsoulis, Nicolas, editor, Lanitis, Andreas, editor, Pattichis, Marios, editor, Pattichis, Constantinos, editor, Kyrkou, Christos, editor, Kyriacou, Efthyvoulos, editor, Theodosiou, Zenonas, editor, and Panayides, Andreas, editor

Published
2023
Full Text
View/download PDF

6. Health Lifestyle Data-Driven Applications Using Pervasive Computing

Author

Fernandez-Luque, Luis, Aupetit, Michaël, Palotti, Joao, Singh, Meghna, Fadlelbari, Ayman, Baggag, Abdelkader, Khowaja, Kamran, Al-Thani, Dena, Lovell, Nigel H., Advisory Editor, Oneto, Luca, Advisory Editor, Piotto, Stefano, Advisory Editor, Rossi, Federico, Advisory Editor, Samsonovich, Alexei V., Advisory Editor, Babiloni, Fabio, Advisory Editor, Liwo, Adam, Advisory Editor, Magjarevic, Ratko, Advisory Editor, Househ, Mowafa, editor, Kushniruk, Andre W., editor, and Borycki, Elizabeth M., editor

Published
2019
Full Text
View/download PDF

7. Overview of the CLEF eHealth Evaluation Lab 2019

Author

Kelly, Liadh, Suominen, Hanna, Goeuriot, Lorraine, Neves, Mariana, Kanoulas, Evangelos, Li, Dan, Azzopardi, Leif, Spijker, Rene, Zuccon, Guido, Scells, Harrisen, Palotti, João, Goos, Gerhard, Founding Editor, Hartmanis, Juris, Founding Editor, Bertino, Elisa, Editorial Board Member, Gao, Wen, Editorial Board Member, Steffen, Bernhard, Editorial Board Member, Woeginger, Gerhard, Editorial Board Member, Yung, Moti, Editorial Board Member, Crestani, Fabio, editor, Braschler, Martin, editor, Savoy, Jacques, editor, Rauber, Andreas, editor, Müller, Henning, editor, Losada, David E., editor, Heinatz Bürki, Gundula, editor, Cappellato, Linda, editor, and Ferro, Nicola, editor

Published
2019
Full Text
View/download PDF

8. CLEF eHealth 2019 Evaluation Lab

Author

Kelly, Liadh, Goeuriot, Lorraine, Suominen, Hanna, Neves, Mariana, Kanoulas, Evangelos, Spijker, Rene, Azzopardi, Leif, Li, Dan, Jimmy, Palotti, João, Zuccon, Guido, Hutchison, David, Editorial Board Member, Kanade, Takeo, Editorial Board Member, Kittler, Josef, Editorial Board Member, Kleinberg, Jon M., Editorial Board Member, Mattern, Friedemann, Editorial Board Member, Mitchell, John C., Editorial Board Member, Naor, Moni, Editorial Board Member, Pandu Rangan, C., Editorial Board Member, Steffen, Bernhard, Editorial Board Member, Terzopoulos, Demetri, Editorial Board Member, Tygar, Doug, Editorial Board Member, Goos, Gerhard, Founding Editor, Hartmanis, Juris, Founding Editor, Azzopardi, Leif, editor, Stein, Benno, editor, Fuhr, Norbert, editor, Mayr, Philipp, editor, Hauff, Claudia, editor, and Hiemstra, Djoerd, editor

Published
2019
Full Text
View/download PDF

10. Overview of the CLEF eHealth Evaluation Lab 2018

Author

Suominen, Hanna, Kelly, Liadh, Goeuriot, Lorraine, Névéol, Aurélie, Ramadier, Lionel, Robert, Aude, Kanoulas, Evangelos, Spijker, Rene, Azzopardi, Leif, Li, Dan, Jimmy, Palotti, João, Zuccon, Guido, Hutchison, David, Series Editor, Kanade, Takeo, Series Editor, Kittler, Josef, Series Editor, Kleinberg, Jon M., Series Editor, Mattern, Friedemann, Series Editor, Mitchell, John C., Series Editor, Naor, Moni, Series Editor, Pandu Rangan, C., Series Editor, Steffen, Bernhard, Series Editor, Terzopoulos, Demetri, Series Editor, Tygar, Doug, Series Editor, Weikum, Gerhard, Series Editor, Bellot, Patrice, editor, Trabelsi, Chiraz, editor, Mothe, Josiane, editor, Murtagh, Fionn, editor, Nie, Jian Yun, editor, Soulier, Laure, editor, SanJuan, Eric, editor, Cappellato, Linda, editor, and Ferro, Nicola, editor

Published
2018
Full Text
View/download PDF

11. CLEF 2017 eHealth Evaluation Lab Overview

Author

Goeuriot, Lorraine, Kelly, Liadh, Suominen, Hanna, Névéol, Aurélie, Robert, Aude, Kanoulas, Evangelos, Spijker, Rene, Palotti, João, Zuccon, Guido, Hutchison, David, Series editor, Kanade, Takeo, Series editor, Kittler, Josef, Series editor, Kleinberg, Jon M., Series editor, Mattern, Friedemann, Series editor, Mitchell, John C., Series editor, Naor, Moni, Series editor, Pandu Rangan, C., Series editor, Steffen, Bernhard, Series editor, Terzopoulos, Demetri, Series editor, Tygar, Doug, Series editor, Weikum, Gerhard, Series editor, Jones, Gareth J.F., editor, Lawless, Séamus, editor, Gonzalo, Julio, editor, Kelly, Liadh, editor, Goeuriot, Lorraine, editor, Mandl, Thomas, editor, Cappellato, Linda, editor, and Ferro, Nicola, editor

Published
2017
Full Text
View/download PDF

12. Does Online Evaluation Correspond to Offline Evaluation in Query Auto Completion?

Author

Bampoulidis, Alexandros, Palotti, João, Lupu, Mihai, Brassey, Jon, Hanbury, Allan, Hutchison, David, Series editor, Kanade, Takeo, Series editor, Kittler, Josef, Series editor, Kleinberg, Jon M., Series editor, Mattern, Friedemann, Series editor, Mitchell, John C., Series editor, Naor, Moni, Series editor, Pandu Rangan, C., Series editor, Steffen, Bernhard, Series editor, Terzopoulos, Demetri, Series editor, Tygar, Doug, Series editor, Weikum, Gerhard, Series editor, Jose, Joemon M, editor, Hauff, Claudia, editor, Altıngovde, Ismail Sengor, editor, Song, Dawei, editor, Albakour, Dyaa, editor, Watt, Stuart, editor, and Tait, John, editor

Published
2017
Full Text
View/download PDF

13. Fixed-Cost Pooling Strategies Based on IR Evaluation Measures

Author

Lipani, Aldo, Palotti, Joao, Lupu, Mihai, Piroi, Florina, Zuccon, Guido, Hanbury, Allan, Hutchison, David, Series editor, Kanade, Takeo, Series editor, Kittler, Josef, Series editor, Kleinberg, Jon M., Series editor, Mattern, Friedemann, Series editor, Mitchell, John C., Series editor, Naor, Moni, Series editor, Pandu Rangan, C., Series editor, Steffen, Bernhard, Series editor, Terzopoulos, Demetri, Series editor, Tygar, Doug, Series editor, Weikum, Gerhard, Series editor, Jose, Joemon M, editor, Hauff, Claudia, editor, Altıngovde, Ismail Sengor, editor, Song, Dawei, editor, Albakour, Dyaa, editor, Watt, Stuart, editor, and Tait, John, editor

Published
2017
Full Text
View/download PDF

14. Overview of the CLEF eHealth Evaluation Lab 2016

Author

Kelly, Liadh, Goeuriot, Lorraine, Suominen, Hanna, Névéol, Aurélie, Palotti, João, Zuccon, Guido, Hutchison, David, Series editor, Kanade, Takeo, Series editor, Kittler, Josef, Series editor, Kleinberg, Jon M., Series editor, Mattern, Friedemann, Series editor, Mitchell, John C., Series editor, Naor, Moni, Series editor, Pandu Rangan, C., Series editor, Steffen, Bernhard, Series editor, Terzopoulos, Demetri, Series editor, Tygar, Doug, Series editor, Weikum, Gerhard, Series editor, Fuhr, Norbert, editor, Quaresma, Paulo, editor, Gonçalves, Teresa, editor, Larsen, Birger, editor, Balog, Krisztian, editor, Macdonald, Craig, editor, Cappellato, Linda, editor, and Ferro, Nicola, editor

Published
2016
Full Text
View/download PDF

15. Assessors Agreement: A Case Study Across Assessor Type, Payment Levels, Query Variations and Relevance Dimensions

Author

Palotti, Joao, Zuccon, Guido, Bernhardt, Johannes, Hanbury, Allan, Goeuriot, Lorraine, Hutchison, David, Series editor, Kanade, Takeo, Series editor, Kittler, Josef, Series editor, Kleinberg, Jon M., Series editor, Mattern, Friedemann, Series editor, Mitchell, John C., Series editor, Naor, Moni, Series editor, Pandu Rangan, C., Series editor, Steffen, Bernhard, Series editor, Terzopoulos, Demetri, Series editor, Tygar, Doug, Series editor, Weikum, Gerhard, Series editor, Fuhr, Norbert, editor, Quaresma, Paulo, editor, Gonçalves, Teresa, editor, Larsen, Birger, editor, Balog, Krisztian, editor, Macdonald, Craig, editor, Cappellato, Linda, editor, and Ferro, Nicola, editor

Published
2016
Full Text
View/download PDF

18. Overview of the CLEF eHealth Evaluation Lab 2015

Author

Goeuriot, Lorraine, Kelly, Liadh, Suominen, Hanna, Hanlen, Leif, Névéol, Aurélie, Grouin, Cyril, Palotti, João, Zuccon, Guido, Hutchison, David, Series editor, Kanade, Takeo, Series editor, Kittler, Josef, Series editor, Kleinberg, Jon M., Series editor, Mattern, Friedemann, Series editor, Mitchell, John C., Series editor, Naor, Moni, Series editor, Pandu Rangan, C., Series editor, Steffen, Bernhard, Series editor, Terzopoulos, Demetri, Series editor, Tygar, Doug, Series editor, Weikum, Gerhard, Series editor, Mothe, Josanne, editor, Savoy, Jacques, editor, Kamps, Jaap, editor, Pinel-Sauvagnat, Karen, editor, Jones, Gareth, editor, San Juan, Eric, editor, Capellato, Linda, editor, and Ferro, Nicola, editor

Published
2015
Full Text
View/download PDF

19. Diagnose This If You Can : On the Effectiveness of Search Engines in Finding Medical Self-diagnosis Information

Author

Zuccon, Guido, Koopman, Bevan, Palotti, João, Hutchison, David, Series editor, Kanade, Takeo, Series editor, Kittler, Josef, Series editor, Kleinberg, Jon M., Series editor, Kobsa, Alfred, Series editor, Mattern, Friedemann, Series editor, Mitchell, John C., Series editor, Naor, Moni, Series editor, Nierstrasz, Oscar, Series editor, Pandu Rangan, C., Series editor, Steffen, Bernhard, Series editor, Terzopoulos, Demetri, Series editor, Tygar, Doug, Series editor, Weikum, Gerhard, Series editor, Hanbury, Allan, editor, Kazai, Gabriella, editor, Rauber, Andreas, editor, and Fuhr, Norbert, editor

Published
2015
Full Text
View/download PDF

22. Overview of the ShARe/CLEF eHealth Evaluation Lab 2014

Author

Kelly, Liadh, Goeuriot, Lorraine, Suominen, Hanna, Schreck, Tobias, Leroy, Gondy, Mowery, Danielle L., Velupillai, Sumithra, Chapman, Wendy W., Martinez, David, Zuccon, Guido, Palotti, João, Hutchison, David, Series editor, Kanade, Takeo, Series editor, Kittler, Josef, Series editor, Kleinberg, Jon M., Series editor, Kobsa, Alfred, Series editor, Mattern, Friedemann, Series editor, Mitchell, John C., Series editor, Naor, Moni, Series editor, Nierstrasz, Oscar, Series editor, Pandu Rangan, C., Series editor, Steffen, Bernhard, Series editor, Terzopoulos, Demetri, Series editor, Tygar, Doug, Series editor, Weikum, Gerhard, Series editor, Kanoulas, Evangelos, editor, Lupu, Mihai, editor, Clough, Paul, editor, Sanderson, Mark, editor, Hall, Mark, editor, Hanbury, Allan, editor, and Toms, Elaine, editor

Published
2014
Full Text
View/download PDF

25. Benchmark on a large cohort for sleep-wake classification with machine learning techniques

Author

Palotti, Joao, Mall, Raghvendra, Aupetit, Michael, Rueschman, Michael, Singh, Meghna, Sathyanarayana, Aarti, Taheri, Shahrad, and Fernandez-Luque, Luis

Subjects
Quality of life, Epidemiology, Machine learning, Health care, Health services and systems, lcsh:R858-859.7, lcsh:Computer applications to medicine. Medical informatics, Article, Fatigue
Abstract

Accurately measuring sleep and its quality with polysomnography (PSG) is an expensive task. Actigraphy, an alternative, has been proven cheap and relatively accurate. However, the largest experiments conducted to date, have had only hundreds of participants. In this work, we processed the data of the recently published Multi-Ethnic Study of Atherosclerosis (MESA) Sleep study to have both PSG and actigraphy data synchronized. We propose the adoption of this publicly available large dataset, which is at least one order of magnitude larger than any other dataset, to systematically compare existing methods for the detection of sleep-wake stages, thus fostering the creation of new algorithms. We also implemented and compared state-of-the-art methods to score sleep-wake stages, which range from the widely used traditional algorithms to recent machine learning approaches. We identified among the traditional algorithms, two approaches that perform better than the algorithm implemented by the actigraphy device used in the MESA Sleep experiments. The performance, in regards to accuracy and F1 score of the machine learning algorithms, was also superior to the device’s native algorithm and comparable to human annotation. Future research in developing new sleep-wake scoring algorithms, in particular, machine learning approaches, will be highly facilitated by the cohort used here. We exemplify this potential by showing that two particular deep-learning architectures, CNN and LSTM, among the many recently created, can achieve accuracy scores significantly higher than other methods for the same tasks.Other Information Published in: npj Digital Medicine License: https://creativecommons.org/licenses/by/4.0See article on publisher's website: http://dx.doi.org/10.1038/s41746-019-0126-9

Published
2022
Full Text
View/download PDF

28. Monitoring of the Venezuelan exodus through Facebook's advertising platform

Author

Palotti, Joao, Adler, Natalia, Morales-Guzman, Alfredo, Villaveces, Jeffrey, Sekara, Vedran, Garcia Herranz, Manuel, Al-Asad, Musa, and Weber, Ingmar

Subjects
Computer and Information Sciences, Facebook, Science, Social Sciences, Colombia, Graduates, Geographical locations, Education, Sociology, Advertising, Educational Attainment, Marketing, Refugees, Correction, Social Communication, Emigration and Immigration, South America, Venezuela, Communications, Social Networks, People and Places, Educational Status, Medicine, Population Groupings, Social Media, Network Analysis, Brazil, Research Article
Abstract

Venezuela is going through the worst economical, political and social crisis in its modern history. Basic products like food or medicine are scarce and hyperinflation is combined with economic depression. This situation is creating an unprecedented refugee and migrant crisis in the region. Governments and international agencies have not been able to consistently leverage reliable information using traditional methods. Therefore, to organize and deploy any kind of humanitarian response, it is crucial to evaluate new methodologies to measure the number and location of Venezuelan refugees and migrants across Latin America. In this paper, we propose to use Facebook's advertising platform as an additional data source for monitoring the ongoing crisis. We estimate and validate national and sub-national numbers of refugees and migrants and break-down their socio-economic profiles to further understand the complexity of the phenomenon. Although limitations exist, we believe that the presented methodology can be of value for real-time assessment of refugee and migrant crises world-wide.

Published
2020

30. Making Sense of Sleep

Author

Zhai, Bing, primary, Perez-Pozuelo, Ignacio, additional, Clifton, Emma A. D., additional, Palotti, Joao, additional, and Guan, Yu, additional

Published
2020
Full Text
View/download PDF

33. Overview of the CLEF 2018 Consumer Health Search Task

Author

Jimmy, Zuccon, Guido, Palotti, Joao, Goeuriot, Lorraine, Kelly, Liadh, Jimmy, Zuccon, Guido, Palotti, Joao, Goeuriot, Lorraine, and Kelly, Liadh

Abstract

This paper details the collection, systems and evaluation methods used in the CLEF 2018 eHealth Evaluation Lab, Consumer Health Search (CHS) task (Task 3). This task investigates the effectiveness of search engines in providing access to medical information present on the Web for people that have no or little medical knowledge. The task aims to foster advances in the development of search technologies for Consumer Health Search by providing resources and evaluation methods to test and validate search systems. Built upon the the 2013-17 series of CLEF eHealth Information Retrieval tasks, the 2018 task considers both mono- and multilingual retrieval, embracing the Text REtrieval Conference (TREC) -style evaluation process with a shared collection of documents and queries, the contribution of runs from participants and the subsequent formation of relevance assessments and evaluation of the participants submissions. For this year, the CHS task uses a new Web corpus and a new set of queries compared to the previous years. The new corpus consists of Web pages acquired from the CommonCrawl and the new set of queries consists of 50 queries issued by the general public to the Health on the Net (HON) search services. We then manually translated the 50 queries to French, German, and Czech; and obtained English query variations of the 50 original queries. A total of 7 teams from 7 different countries participated in the 2018 CHS task: CUNI (Czech Republic), IMS Unipd (Italy), MIRACL (Tunisia), QUT (Australia), SINAI (Spain), UB-Botswana (Botswana), and UEvora (Portugal).

Published
2018

34. Clef 2017 task overview: The IR task at the ehealth evaluation lab

Author

Palotti, Joao, Zuccon, Guido, Jimmy, Jimmy, Lupu, Mihai, Goeuriot, Lorraine, Kelly, Liadh, Hanbury, Allan, Vienna University of Technology (TU Wien), Queensland University of Technology [Brisbane] (QUT), Université Grenoble Alpes [2016-2019] (UGA [2016-2019]), Modélisation et Recherche d’Information Multimédia [Grenoble] (MRIM ), Laboratoire d'Informatique de Grenoble (LIG ), Institut polytechnique de Grenoble - Grenoble Institute of Technology (Grenoble INP )-Centre National de la Recherche Scientifique (CNRS)-Université Grenoble Alpes [2016-2019] (UGA [2016-2019])-Institut polytechnique de Grenoble - Grenoble Institute of Technology (Grenoble INP )-Centre National de la Recherche Scientifique (CNRS)-Université Grenoble Alpes [2016-2019] (UGA [2016-2019]), and Dublin City University [Dublin] (DCU)

Subjects
[INFO.INFO-IR]Computer Science [cs]/Information Retrieval [cs.IR], ComputingMilieux_MISCELLANEOUS
Abstract

International audience

Published
2017

35. MM

Author

Palotti, Joao, primary, Zuccon, Guido, additional, and Hanbury, Allan, additional

Published
2018
Full Text
View/download PDF

38. Fixed budget pooling strategies based on fusion methods

Author

Shin, D, Lencastre, M, Lipani, Aldo, Lupu, Mihai, Palotti, Joao, Zuccon, Guido, Hanbury, Allan, Shin, D, Lencastre, M, Lipani, Aldo, Lupu, Mihai, Palotti, Joao, Zuccon, Guido, and Hanbury, Allan

Abstract

The empirical nature of Information Retrieval (IR) mandates strong experimental practices. The Cranfield/TREC evaluation paradigm represents a keystone of such experimental practices. Within this paradigm, the generation of relevance judgments has been the subject of intense scientific investigation. This is because, on one hand, consistent, precise and numerous judgements are key to reduce evaluation uncertainty and test collection bias; on the other hand, however, relevance judgements are costly to collect. The selection of which documents to judge for relevance (known as pooling) has therefore great impact in IR evaluation. In this paper, we contribute a set of 8 novel pooling strategies based on retrieval fusion methods. We show that the choice of the pooling strategy has significant effects on the cost needed to obtain an unbiased test collection; we also identify the best performing pooling strategy according to three evaluation measure.

Published
2017

39. Fixed-cost pooling strategies based on IR evaluation measures

Author

Song, D, Watt, S, Tait, J, Hauff, C, Joemon M, M J, Albakour, D, Altingovde, I S, Lipani, Aldo, Palotti, Joao, Lupu, Mihai, Piroi, Florina, Zuccon, Guido, Hanbury, Allan, Song, D, Watt, S, Tait, J, Hauff, C, Joemon M, M J, Albakour, D, Altingovde, I S, Lipani, Aldo, Palotti, Joao, Lupu, Mihai, Piroi, Florina, Zuccon, Guido, and Hanbury, Allan

Abstract

Recent studies have reconsidered the way we operationalise the pooling method, by considering the practical limitations often encountered by test collection builders. The biggest constraint is often the budget available for relevance assessments and the question is how best – in terms of the lowest pool bias – to select the documents to be assessed given a fixed budget. Here, we explore a series of 3 new pooling strategies introduced in this paper against 3 existing ones and a baseline. We show that there are significant differences depending on the evaluation measure ultimately used to assess the runs. We conclude that adaptive strategies are always best, but in their absence, for top-heavy evaluation measures we can continue to use the baseline, while for P@100 we should use any of the other non-adaptive strategies.

Published
2017

40. CLEF 2017 eHealth Evaluation Lab overview

Author

Jones, G J F, Kelly, L, Mandl, T, ferro, N, Lawless, S, Gonzalo, J, Goeuriot, L, Cappellato, L, Goeuriot, Lorraine, Kelly, Liadh, Suominen, Hanna, Neveol, Aurelie, Robert, Aude, Kanoulas, Evangelos, Spijker, Rene, Palotti, Joao, Zuccon, Guido, Jones, G J F, Kelly, L, Mandl, T, ferro, N, Lawless, S, Gonzalo, J, Goeuriot, L, Cappellato, L, Goeuriot, Lorraine, Kelly, Liadh, Suominen, Hanna, Neveol, Aurelie, Robert, Aude, Kanoulas, Evangelos, Spijker, Rene, Palotti, Joao, and Zuccon, Guido

Abstract

In this paper we provide an overview of the fifth edition of the CLEF eHealth evaluation lab. CLEF eHealth 2017 continues our evaluation resource building efforts around the easing and support of patients, their next-of-kins, clinical staff, and health scientists in understanding, accessing, and authoring eHealth information in a multilingual setting. This year’s lab offered three tasks: Task 1 on multilingual information extraction to extend from last year’s task on French corpora, Task 2 on technologically assisted reviews in empirical medicine as a new pilot task, and Task 3 on patient-centered information retrieval (IR) building on the 2013-16 IR tasks. In total 32 teams took part in these tasks (11 in Task 1, 14 in Task 2, and 7 in Task 3). We also continued the replication track from 2016. Herein, we describe the resources created for these tasks, evaluation methodology adopted and provide a brief summary of participants of this year’s challenges and results obtained. As in previous years, the organizers have made data and tools associated with the lab tasks available for future research and development.

Published
2017

42. Query variations and their effect on comparing information retrieval systems

Author

Zhai, C, Mukhopadhyay, S, Zuccon, Guido, Palotti, Joao, Hanbury, Allan, Zhai, C, Mukhopadhyay, S, Zuccon, Guido, Palotti, Joao, and Hanbury, Allan

Abstract

We explore the implications of using query variations for evaluating information retrieval systems and how these variations should be exploited to compare system effectiveness. Current evaluation approaches consider the availability of a set of topics (information needs), and only one expression of each topic in the form of a query is used for evaluation and system comparison. While there is strong evidence that considering query variations better models the usage of retrieval systems and accounts for the important user aspect of user variability, it is unclear how to best exploit query variations for evaluating and comparing information retrieval systems. We propose a framework for evaluating retrieval systems that explicitly takes into account query variations. The framework considers both the system mean effectiveness and its variance over query variations and topics, as opposed to current approaches that only consider the mean across topics or perform a topic-focused analysis of variance across systems. Furthermore, the framework extends current evaluation practice by encoding: (1) user tolerance to effectiveness variations, (2) the popularity of different query variations, and (3) the relative importance of individual topics. These extensions and our findings make information retrieval comparisons more aligned with user behaviour.

Published
2016

43. Ranking health web pages with relevance and understandability

Author

Aslam, J, Zobel, J, Ruthven, I, Palotti, Joao, Goeuriot, Lorraine, Zuccon, Guido, Hanbury, Allan, Aslam, J, Zobel, J, Ruthven, I, Palotti, Joao, Goeuriot, Lorraine, Zuccon, Guido, and Hanbury, Allan

Abstract

We propose a method that integrates relevance and understandability to rank health web documents. We use a learning to rank approach with standard retrieval features to determine topical relevance and additional features based on readability measures and medical lexical aspects to determine understandability. Our experiments measured the effectiveness of the learning to rank approach integrating understandability on a consumer health benchmark. The findings suggest that this approach promotes documents that are at the same time topically relevant and understandable.

Published
2016

44. Overview of the CLEF eHealth Evaluation Lab 2016

Author

Fuhr, N, Balog, K, Ferro, N, Larsen, B, Quaresma, P, Goncalves, T, Macdonald, C, Cappellato, L, Kelly, Liadh, Goeuriot, Lorraine, Suominen, Hanna, Neveol, Aurelie, Palotti, Joao, Zuccon, Guido, Fuhr, N, Balog, K, Ferro, N, Larsen, B, Quaresma, P, Goncalves, T, Macdonald, C, Cappellato, L, Kelly, Liadh, Goeuriot, Lorraine, Suominen, Hanna, Neveol, Aurelie, Palotti, Joao, and Zuccon, Guido

Abstract

In this paper we provide an overview of the fourth edition of the CLEF eHealth evaluation lab. CLEF eHealth 2016 continues our evaluation resource building efforts around the easing and support of patients, their next-of-kins and clinical staff in understanding, accessing and authoring eHealth information in a multilingual setting. This year’s lab offered three tasks: Task 1 on handover information extraction related to Australian nursing shift changes, Task 2 on information extraction in French corpora, and Task 3 on multilingual patient-centred information retrieval considering query variations. In total 20 teams took part in these tasks (3 in Task 1, 7 in Task 2 and 10 in Task 3). Herein, we describe the resources created for these tasks, evaluation methodology adopted and provide a brief summary of participants to this year’s challenges and some results obtained. As in previous years, the organizers have made data and tools associated with the lab tasks available for future research and development.

Published
2016

45. Assessors agreement: A case study across assessor type, payment levels, query variations and relevance dimensions

Author

Fuhr, N, Balog, K, Ferro, N, Larsen, B, Quaresma, P, Goncalves, T, Macdonald, C, Cappellato, L, Palotti, Joao, Zuccon, Guido, Bernhardt, Johannes, Hanbury, Allan, Goeuriot, Lorraine, Fuhr, N, Balog, K, Ferro, N, Larsen, B, Quaresma, P, Goncalves, T, Macdonald, C, Cappellato, L, Palotti, Joao, Zuccon, Guido, Bernhardt, Johannes, Hanbury, Allan, and Goeuriot, Lorraine

Abstract

Relevance assessments are the cornerstone of Information Retrieval evaluation. Yet, there is only limited understanding of how assessment disagreement influences the reliability of the evaluation in terms of systems rankings. In this paper we examine the role of assessor type (expert vs. layperson), payment levels (paid vs. unpaid), query variations and relevance dimensions (topicality and understandability) and their influence on system evaluation in the presence of disagreements across assessments obtained in the different settings. The analysis is carried out in the context of the CLEF 2015 eHealth Task 2 collection and shows that disagreements between assessors belonging to the same group have little impact on evaluation. It also shows, however, that assessment disagreement found across settings has major impact on evaluation when topical relevance is considered, while it has no impact when understandability assessments are considered.

Published
2016

46. The IR Task at the CLEF eHealth evaluation lab 2016: User-centred health information retrieval

Author

Balog, K, Ferro, N, Macdonald, C, Cappellato, L, Zuccon, Guido, Palotti, Joao, Goeuriot, Lorraine, Kelly, Liadh, Lupu, Mihai, Pecina, Pavel, Muller, Henning, Budaher, Julie, Deacon, Anthony, Balog, K, Ferro, N, Macdonald, C, Cappellato, L, Zuccon, Guido, Palotti, Joao, Goeuriot, Lorraine, Kelly, Liadh, Lupu, Mihai, Pecina, Pavel, Muller, Henning, Budaher, Julie, and Deacon, Anthony

Published
2016

47. ShARe/CLEF eHealth evaluation lab 2014, task 3: user-centred health information retrieval

Author

Goeuriot, Lorraine, Kelly, Liadh, Li, Wei B., Palotti, Joao, Zuccon, Guido, Hanbury, Allan, Jones, Gareth J.F., and Mueller, Henning

Subjects
Evaluation, Medical information retrieval, Information retrieval
Abstract

This paper presents the results of task 3 of the ShARe/CLEF eHealth Evaluation Lab 2014. This evaluation lab focuses on improving access to medical information on the web. The task objective was to investigate the eect of using additional information such as a related discharge summary and external resources such as medical ontologies on the eectiveness of information retrieval systems, in a monolingual (Task 3a) and in a multilingual (Task 3b) context. The participants were allowed to submit up to seven runs for each language (English, Czech, French, German), one mandatory run using no additional information or external resources, and three each using or not using discharge summaries.

Published
2014

Catalog

Books, media, physical & digital resources
See catalog results