Search

In embodiment design, insights about the relations of the embodiment of a system to its behavior and functions are essential. These insights are gained through system analysis. For successful system analysis, the derivation of hypotheses for causes of system behavior is a crucial task. In this study, we investigate, how qualitative modelling can support this derivation of hypotheses. Modelling activities and derivation of hypotheses were investigated in ten engineering design projects as subsequent case studies by using the Grounded Theory approach. The projects were conducted by research institutes in cooperation with various industrial partners. In these projects, the Contact and Channel Approach was used for qualitative modelling.. Modelling activities conducted in these projects were identified, combined into concepts, cross-validated and summarized into nine categories assigned to three clusters. The first cluster is “modelling support” and contains the categories $\textit{keep it simple, make it clear, explicate your mental model}$ and $\textit{shift the effort}$. The second cluster is “model verification” and contains the categories $\textit{how me don’t tell me, link your models, combine your information}$ and $\textit{verify your parameters}$. The third cluster is “guidelines for hypotheses” and contains the category $\textit{link your hypotheses}$. With the clusters, we propose an approach that can support the derivation of hypotheses from qualitative modelling. They present a basis for further research on supporting the derivation of hypotheses and can also be used directly as recommendations for action for hypothesis-driven qualitative modelling.

Inhalt Bei der Entwicklung von Produkten treten häufig technische Probleme auf, die schwierig vorhersehbar waren und deren Ursachen schwierig zu identifizieren sind. Für die Entwicklung von technischen Lösungen muss zunächst die Problemursache identifiziert werden. In diesem Beitrag werden in einer Fallstudie vier Herausforderungen erhoben, welche bei der Identifikation von Problemursachen in der Konstruktion auftreten: (1) Die Eingrenzung auf die tatsächliche Problemursache, (2) unbemerkte Fehlinterpretation diskutierter Zusammenhänge sowie (3) das Übersehen von möglichen Problemursachen und Indizien. Diese Herausforderungen führen zur (4) Problemlösung ohne ausreichende Problemanalyse und damit zu unnötigen Iterationen, da Lösungen auf Grundlage falscher Annahmen entwickelt werden. Auf Basis der Beobachtungen in der Fallstudie werden Empfehlungen für die methodische Unterstützung der Problemanalyse abgeleitet.

Grand challenges have become the de facto standard for benchmarking image analysis algorithms. While the number of these international competitions is steadily increasing, surprisingly little effort has been invested in ensuring high quality design, execution and reporting for these international competitions. Specifically, results analysis and visualization in the event of uncertainties have been given almost no attention in the literature. Given these shortcomings, the contribution of this paper is two-fold: (1) we present a set of methods to comprehensively analyze and visualize the results of single-task and multi-task challenges and apply them to a number of simulated and real-life challenges to demonstrate their specific strengths and weaknesses; (2) we release the open-source framework challengeR as part of this work to enable fast and wide adoption of the methodology proposed in this paper. Our approach offers an intuitive way to gain important insights into the relative and absolute performance of algorithms, which cannot be revealed by commonly applied visualization techniques. This is demonstrated by the experiments performed in the specific context of biomedical image analysis challenges. Our framework could thus become an important tool for analyzing and visualizing challenge results in the field of biomedical image analysis and beyond.

Inhalt Treten Probleme am technischen System auf, muss die Problemursache verstanden werden, bevor konstruktive Maßnahmen umgesetzt werden. Sonst entstehen Konstruktionen, die nicht das Problem lösen. Bei der Problemanalyse können unbemerkte Denkfehler auftreten, die eine Identifikation der tatsächlichen Ursache verhindern. Zur Überwindung eines besonders relevanten Denkfehlers, des Bestätigungsfehlers, wird in anderen Fachdisziplinen die Methode Analysis of Competing Hypotheses (ACH) angewendet. Dieser Beitrag beschreibt die Anpassung und Evaluation der ACH-Methode innerhalb einer Schulung, um Herausforderungen bei der Problemanalyse in der Konstruktion zu überwinden. Die Methode wurde von den Teilnehmern als sehr gut bewertet und unterstützte sie bei der Problemlösung.

The requirements on validity for studies in design research are very high. Therefore, this paper aims at identifying challenges that occur when setting up studies and suggests solution strategies to address them. Three different institutes combining their experience discussed several studies in a workshop. Resulting main challenges are to find a suitable task, to operationalise the variables and to deal with a high analysis effort per participant. Automation in data evaluation and a detailed practical guideline on studies in design research are considered necessary.

Recent developments in data science in general and machine learning in particular have transformed the way experts envision the future of surgery. Surgical Data Science (SDS) is a new research field that aims to improve the quality of interventional healthcare through the capture, organization, analysis and modeling of data. While an increasing number of data-driven approaches and clinical applications have been studied in the fields of radiological and clinical data science, translational success stories are still lacking in surgery. In this publication, we shed light on the underlying reasons and provide a roadmap for future advances in the field. Based on an international workshop involving leading researchers in the field of SDS, we review current practice, key achievements and initiatives as well as available standards and tools for a number of topics relevant to the field, namely (1) infrastructure for data acquisition, storage and access in the presence of regulatory constraints, (2) data annotation and sharing and (3) data analytics. We further complement this technical perspective with (4) a review of currently available SDS products and the translational progress from academia and (5) a roadmap for faster clinical translation and exploitation of the full potential of SDS, based on an international multi-round Delphi process.

With the impact of artificial intelligence (AI) algorithms on medical research on the rise, the importance of competitions for comparative validation of algorithms, so-called challenges, has been steadily increasing, to a point at which challenges can be considered major drivers of research, particularly in the biomedical image analysis domain. Given their importance, high quality, transparency, and interpretability of challenges is essential for good scientific practice and meaningful validation of AI algorithms, for instance towards clinical translation. This mini-review presents several issues related to the design, execution, and interpretation of challenges in the biomedical domain and provides best-practice recommendations. PATIENT SUMMARY: This paper presents recommendations on how to reliably compare the usefulness of new artificial intelligence methods for analysis of medical images.

Grand challenges have become the de facto standard for benchmarking image analysis algorithms. While the number of these international competitions is steadily increasing, surprisingly little effort has been invested in ensuring high quality design, execution and reporting for these international competitions. Specifically, results analysis and visualization in the event of uncertainties have been given almost no attention in the literature. Given these shortcomings, the contribution of this paper is two-fold: (1) we present a set of methods to comprehensively analyze and visualize the results of single-task and multi-task challenges and apply them to a number of simulated and real-life challenges to demonstrate their specific strengths and weaknesses; (2) we release the open-source framework challengeR as part of this work to enable fast and wide adoption of the methodology proposed in this paper. Our approach offers an intuitive way to gain important insights into the relative and absolute performance of algorithms, which cannot be revealed by commonly applied visualization techniques. This is demonstrated by the experiments performed in the specific context of biomedical image analysis challenges. Our framework could thus become an important tool for analyzing and visualizing challenge results in the field of biomedical image analysis and beyond.

Design method validation is an important research activity in method development. But the design method's applicability is often left out in validation, although it is a necessary step and it enables further validation steps. Currently, there is a lack of support for the assessment of design method applicability. We present an approach to provide support for iterative studies on three aspects of design method applicability. We evaluate the approach in a single case study. In the study, we investigate the applicability of a method for model building in analysis, in three iterative steps of the approach. Application of the approach produced valuable insights on the method applicability. The results show how aspects of applicability can be assessed using the presented approach. Therefore, the approach can contribute to the standardisation of design method validation, especially for early stages of validation, which is currently discussed in the community.

Validation of design methods is a challenge in design research, as there is a lack of common methodology. This contribution investigates how validation of design methods is currently conducted in o...

Design methods are seldom used in engineering design practice. The presented study aims at finding the alternative strategies for situations with a need for methodological support. Semi-structured interviews were conducted with ten experienced design engineers to identify causes of and strategies for those situations. Three strategy clusters could be identified: generating information, experience and method application. As the individual's and the team's experience compete with the application of methods, they are seen as reasons for a lack of method application.

Highlights • First guideline for reporting biomedical image analysis challenges • Includes checklist for authors and reviewers, Graphical abstract, The number of biomedical image analysis challenges organized per year is steadily increasing. These international competitions have the purpose of benchmarking algorithms on common data sets, typically to identify the best method for a given problem. Recent research, however, revealed that common practice related to challenge reporting does not allow for adequate interpretation and reproducibility of results. To address the discrepancy between the impact of challenges and the quality (control), the Biomedical Image Analysis ChallengeS (BIAS) initiative developed a set of recommendations for the reporting of challenges. The BIAS statement aims to improve the transparency of the reporting of a biomedical image analysis challenge regardless of field of application, image modality or task category assessed. This article describes how the BIAS statement was developed and presents a checklist which authors of biomedical image analysis challenges are encouraged to include in their submission when giving a paper on a challenge into review. The purpose of the checklist is to standardize and facilitate the review process and raise interpretability and reproducibility of challenge results by making relevant information explicit.

Accurate segmentations in medical images are the foundations for various clinical applications. Advances in machine learning-based techniques show great potential for automatic image segmentation, but these techniques usually require a huge amount of accurately annotated reference segmentations for training. The guiding hypothesis of this paper was that crowd-algorithm collaboration could evolve as a key technique in large-scale medical data annotation. As an initial step toward this goal, we evaluated the performance of untrained individuals to detect and correct errors made by three-dimensional (3-D) medical segmentation algorithms. To this end, we developed a multistage segmentation pipeline incorporating a hybrid crowd-algorithm 3-D segmentation algorithm integrated into a medical imaging platform. In a pilot study of liver segmentation using a publicly available dataset of computed tomography scans, we show that the crowd is able to detect and refine inaccurate organ contours with a quality similar to that of experts (engineers with domain knowledge, medical students, and radiologists). Although the crowds need significantly more time for the annotation of a slice, the annotation rate is extremely high. This could render crowdsourcing a key tool for cost-effective large-scale medical image annotation.

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future., Comment: Article published in Nature Communications: https://rdcu.be/bRmNr

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future., Biomedical image analysis challenges have increased in the last ten years, but common practices have not been established yet. Here the authors analyze 150 recent challenges and demonstrate that outcome varies based on the metrics used and that limited information reporting hampers reproducibility.

This paper introduces Surgical Data Science as an emerging scientific discipline. Key perspectives are based on discussions during an intensive two-day international interactive workshop that brought together leading researchers working in the related field of computer and robot assisted interventions. Our consensus opinion is that increasing access to large amounts of complex data, at scale, throughout the patient care process, complemented by advances in data science and machine learning techniques, has set the stage for a new generation of analytics that will support decision-making and quality improvement in interventional medicine. In this article, we provide a consensus definition for Surgical Data Science, identify associated challenges and opportunities and provide a roadmap for advancing the field.

Im Bereich der biomedizinischen Bildanalyse werden vermehrt offentliche Wettbewerbe (Challenges) durchgefuhrt, die den Vergleich von Methoden unter denselben Bedingungen ermoglichen. Ergebnisse aus solchen Challenges gewinnen zur Bewertung von Forschungsresultaten – z.B. im Reviewprozess von Publikationen– immer mehr an Bedeutung. Demgegenuber steht eine mangelnde Qualit atskontrolle im Challengedesign. Dieser Beitrag beruht auf der Hypothese, dass eine unzureichende Qualitatskontrolle zu einer geringen Aussagekraft der Challengeergebnisse fuhren kann. Basierend auf dem Validierungsprotokoll von Jannin et al. [1] wurden samtliche biomedizinischen Challenges des Kollektivs Grand Challenges in Biomedical Image Analysis“ [2] bis zum Jahr 2016 erfasst und systematisch analysiert. Wir prasentieren die Analyseergebnisse hinsichtlich der Vollstandigkeit des Reportings und des Einflusses verschiedener Entscheidungen im Challengedesign auf das finale Ranking der Teilnehmer. Unsere Analyse demonstriert die Notwendigkeit einer Qualitatskontrolle, welche dazu beitragen sollte, dass Rankings nachvollziehbar sowie reproduzierbar sind und die Aussagef ahigkeit erhoht wird.

Machine learning algorithms are gaining increasing interest in the context of computer-assisted interventions. One of the bottlenecks so far, however, has been the availability of training data, typically generated by medical experts with very limited resources. Crowdsourcing is a new trend that is based on outsourcing cognitive tasks to many anonymous untrained individuals from an online community. In this work, we investigate the potential of crowdsourcing for segmenting medical instruments in endoscopic image data. Our study suggests that (1) segmentations computed from annotations of multiple anonymous non-experts are comparable to those made by medical experts and (2) training data generated by the crowd is of the same quality as that annotated by medical experts. Given the speed of annotation, scalability and low costs, this implies that the scientific community might no longer need to rely on experts to generate reference or training data for certain applications. To trigger further research in endoscopic image processing, the data used in this study will be made publicly available.

Due to the emergence of experiments in product development and yet still missing systematic adjustment of the generic experimental design procedure to test design methods. a facilitation is required. A concept map to depict the most important elements to consider when validating design methods in an experiment was developed systematically to guide method developers. Combining literature findings with the analysis of six cases in academia and an intensive exchange of researchers between different institutions resulted in the concept map. Future research potentials in the definition of variables. gaining objective results and managing the required tools are outlined. The concept map is available online for further feedback from the community.