Search

Entity Resolution (ER) is a crucial process in the field of data management and integration. The primary goal of ER is to identify different profiles (or records) that refer to the same real-world entity across databases. The challenging problem is that labeling a large sample of profiles can be very expensive and time-consuming. Active Machine Learning (ActiveML) addresses this issue by selecting the most representative or informative profiles pairs to be labeled. The informativeness is determined by the capacity to diminish the uncertainty of the model. Conversely, representativeness evaluates whether a selected instance effectively reflects the overall input patterns of unlabeled data. Traditional ActiveML techniques typically rely on one strategy, Which may severely restrict the performance of the ActiveML process and lead to slow convergence. Especially in ER problems with a lack of initial training data. In this paper, we overcame this issue by inventing an approach for balancing the two above strategies. The implemented solution named EBEES (Epsilon-based Balancing Exploration and Exploitation Strategy), Which contains two variations: Adaptive- ϵ and ϵ -decreasing. We evaluated the EBEES on twelve datasets. Comparing the EBEES strategy against the state-of-the-art methods, without an initial training data, showed an enhanced performance in terms of F1-score, model stability, and rapid convergence. [ABSTRACT FROM AUTHOR]

Bibliographic databases are used to measure the performance of researchers, universities and research institutions. Thus, high data quality is required and data duplication is avoided. One of the weaknesses of the threshold-based approach in duplication detection is the low accuracy level. Therefore, another approach is required to improve duplication detection. This study proposes a method that combines threshold-based and rule-based approaches to perform duplication detection. These two approaches are implemented in the comparison stage. The cosine similarity function is used to create weight vectors from the features. Then, the comparison operator is used to determine whether the pair of records are grouped as duplication or not. Three research databases: Web of Science (WoS), Scopus, and Google Scholar (GS) on the Science and Technology Index (SINTA) database are investigated. Rule 4 and Rule 5 provide the best performance. For WoS dataset, the accuracy, precision, recall, and F1-measure values were 100.00%. For Scopus dataset, the accuracy and precision values were 100.00%, recall: 98.00%, and the F1-measure value is 98.00%. For GS dataset, the accuracy value was 100.00%, precision: 99.00%, recall: 97.00%, and the F1-measure value is 98.00%. The proposed method is potential tool for accurate detection on duplication records in publication databases. [ABSTRACT FROM AUTHOR]

A systematic review is a type of literature review that aims to collect and analyse all available evidence from the literature on a particular topic. The process of screening and identifying eligible articles from the vast amounts of literature is a time‐consuming task. Specialised software has been developed to aid in the screening process and save significant time and labour. However, the most suitable software tools that are available often come with a cost or only offer either a limited or a trial version for free. In this paper, we report the release of a new software application, Catchii, which contains all the important features of a systematic review screening application while being completely free. It supports a user at different stages of screening, from detecting duplicates to creating the final flowchart for a publication. Catchii is designed to provide a good user experience and streamline the screening process through its clean and user‐friendly interface on both computers and mobile devices. All in all, Catchii is a valuable addition to the current selection of systematic review screening applications. It enables researchers without financial resources to access features found in the best paid tools, while also diminishing costs for those who have previously relied on paid applications. Catchii is available at https://catchii.org. [ABSTRACT FROM AUTHOR]

Stakeholders in software projects use issue trackers like JIRA or Bugzilla to capture and manage issues, including requirements, feature requests, and bugs. To ease issue navigation and structure project knowledge, stakeholders manually connect issues via links of certain types that reflect different dependencies, such as Epic-, Block-, Duplicate-, or Relate- links. Based on a large dataset of 16 JIRA repositories, we study the commonalities and differences in linking practices and link types across the repositories. We then investigate how state-of-the-art machine learning models can predict common link types. We observed significant differences across the repositories and link types, depending on how they are used and by whom. Additionally, we observed several inconsistencies, e.g., in how Duplicate links are used. We found that a transformer model trained on titles and descriptions of linked issues significantly outperforms other optimized models, achieving an encouraging average macro F1-score of 0.64 for predicting nine popular link types across all repositories (weighted F1-score of 0.73). For the specific Subtask- and Epic- links, the model achieves top F1-scores of 0.89 and 0.97, respectively. If we restrict the task to predict the mere existence of links, the average macro F1-score goes up to 0.95. In general, the shorter issue text, possibly indicating precise issues, seems to improve the prediction accuracy with a strong negative correlation of - 0.73. We found that Relate-links often get confused with the other links, which suggests that they are likely used as default links in unclear cases. Our findings particularly on the quality and heterogeinity of issue link data have implications for researching and applying issue link prediction in practice. [ABSTRACT FROM AUTHOR]

Duplicate Bug Report Detection (DBRD) is one of the famous problems in software triage systems like Bugzilla. There are two main approaches to this problem, including information retrieval and machine learning. The second one is more effective for validation performance. Duplicate detection needs feature extraction, which is a time-consuming process. Both approaches suffer runtime issues, because they should check the new bug report to all bug reports in the repository, and it takes a long time for feature extraction and duplicate detection. This study proposes a new two-step classification approach which tries to reduce the search space of the bug repository search space in the first step and then check the duplicate detection using textual features. The Mozilla and Eclipse datasets are used for experimental evaluation. The results show that overall, 87.70% and 89.01% validation performance achieved averagely for accuracy and F1-measure, respectively. Moreover, 95.85% and 87.65% of bug reports can be classified in step one very fast for Eclipse and Mozilla datasets, respectively, and the other one needs textual feature extraction until it can be checked by the traditional DBRD approach. An average of 90% runtime improvement is achieved using the proposed method.

Clustering method is a technique used for comparisons reduction between the candidates records in the duplicate detection process. The process of clustering records is affected by the quality of data. The more error-free the data, the more efficient the clustering algorithm, as data errors cause data to be placed in incorrect groups. Window algorithms suffer from the window size. The larger the window, the greater the number of unnecessary comparisons, and the smaller the window size may prevent the detection of duplicates that are supposed to be within the window. In this paper, we propose a data pre-processing method that increases the efficiency of window algorithms in grouping similar records together. In addition, the proposed method also deal s with the window size problem. In the proposed method, high-rank attributes are selected and then preparators are applied to the selected traits. A compensation algorithm is implemented to reduce the problem of missing and distorted sort keys. Two datasets (compact disc database (CDDB) and MusicBrainz) were used to test duplicates detection algorithms. The duplicates detection toolkit(DuDe) was used as a benchmark for the proposed method. Experiments showed that the proposed method achieved a high rate of accuracy in detecting duplicates. In addition, the proposed method. [ABSTRACT FROM AUTHOR]

ESystems based on EHRs (Electronic health records) have been in use for many years and their amplified realizations have been felt recently. They still have been pioneering collections of massive volumes of health data. Duplicate detections involve discovering records referring to the same practical components, indicating tasks, which are generally dependent on several input parameters that experts yield. Record linkage specifies the issue of finding identical records across various data sources. The similarity existing between two records is characterized based on domain-based similarity functions over different features. De-duplication of one dataset or the linkage of multiple data sets has become a highly significant operation in the data processing stages of different data mining programmes. The objective is to match all the records associated with the same entity. Various measures have been in use for representing the quality and complexity about data linkage algorithms, and many other novel metrics have been introduced. An outline of the problem existing in the measurement of data linkage and de-duplication quality and complexity is presented. This article focuses on the reprocessing of health data that is horizontally divided among data custodians, with the purpose of custodians giving similar features to sets of patients. The first step in this technique is about an automatic selection of training examples with superior quality from the compared record pairs and the second step involves training the reciprocal neuro-fuzzy inference system (RANFIS) classifier. Using the Optimal Threshold classifier, it is presumed that there is information about the original match status for all compared record pairs (i.e., Ant Lion Optimization), and therefore an optimal threshold can be computed based on the respective RANFIS. Febrl, Clinical Decision (CD), and Cork Open Research Archive (CORA) data repository help analyze the proposed method with evaluated benchmarks with current techniques. [ABSTRACT FROM AUTHOR]

Purpose: Literature reports of adverse drug events can be replicated across multiple companies, resulting in extreme duplication (defined as a majority of reports being duplicates) in the FDA Adverse Event Reporting System (FAERS) database because they can escape legacy duplicate detection algorithms routinely deployed on that data source. Literature reference field, added to in 2014, could potentially be utilized to identify replicated reports. FAERS does not enforce adherence to the Vancouver referencing convention, thus the same article may be referenced differently leading to duplication. The objective of this analysis is to determine if variations of the same literature references observed in FAERS can be resolved with text normalization and fuzzy string matching. Methods: We normalized the literature references recorded in the FAERS database through the first quarter of 2021 with a rule‐based algorithm so that they better conform to the Vancouver convention. Levenshtein distance was then utilized to merge sufficiently similar normalized literature references together. Results: Normalization of literature references increases the percentage that can be parsed into author, title, and journal from 61.74% to 93.93%. We observe that about 98% of pairs within groups do have a Levenshtein similarity of the title above the threshold. The extreme duplication ranged from 66% to 87% with a median of 72% of reports being duplicates and often involved addictovigilance scenarios. Conclusions: We have shown that these normalized references can be merged via fuzzy string matching to improve enumeration of all the individual case safety reports that refer to the same article. Inclusion of the PubMed ID and adherence to the Vancouver convention could facilitate identification of duplicates in the FAERS dataset. Awareness of this phenomenon may improve disproportionality analysis, especially in areas such as addictovigilance. [ABSTRACT FROM AUTHOR]

Duplicate Bug Report Detection (DBRD) is one of the famous problems in software triage systems like Bugzilla. There are two main approaches to this problem, including information retrieval and machine learning. The second one is more effective for validation performance. Duplicate detection needs feature extraction, which is a time-consuming process. Both approaches suffer runtime issues, because they should check the new bug report to all bug reports in the repository, and it takes a long time for feature extraction and duplicate detection. This study proposes a new two-step classification approach which tries to reduce the search space of the bug repository search space in the first step and then check the duplicate detection using textual features. The Mozilla and Eclipse datasets are used for experimental evaluation. The results show that overall, 87.70% and 89.01% validation performance achieved averagely for accuracy and F1-measure, respectively. Moreover, 95.85% and 87.65% of bug reports can be classified in step one very fast for Eclipse and Mozilla datasets, respectively, and the other one needs textual feature extraction until it can be checked by the traditional DBRD approach. An average of 90% runtime improvement is achieved using the proposed method. [ABSTRACT FROM AUTHOR]

Software projects reckon on the bug tracking systems to guide software maintenance activities. The critical information about the nature of the crash is carried by the bug reports which are submitted to bug repositories. This information is in free form text format and is submitted by users or developers. A large amount of bug reports gets collected in bug repositories. Out of these submitted bugs, many reports are mere identical of the already existing bugs. Furthermore, not all non-duplicate bugs are reproducible in nature. This paper introduces DENATURE, a two step framework for detecting duplication and identifying bug type. The proposed framework will help to minimize time and developer's effort utilized in resolution of bug reports which will further improvise overall software quality. Information retrieval techniques are used for finding duplicate bugs and machine learning classification techniques are used for identifying the type of bug report. Through experiments, we found that the proposed framework obtained prediction accuracy up to 88.81%. [ABSTRACT FROM AUTHOR]

Recently, open-source repositories have grown rapidly due to volunteer contributions worldwide. Collaboration software platforms have gained popularity as thousands of external contributors have contributed to open-source repositories. Although data de-duplication decreases the size of backup workloads, this causes poor data locality (fragmentation) and redundant review time and effort. Deep learning and machine learning techniques have recently been applied to identify complex bugs and duplicate issue reports. It is difficult to use, but it increases the risk of developers submitting duplicate pull requests, resulting in additional maintenance costs. We propose a hybrid deep learning technique in this work on the basis of an optimal duplication detection is for pull requests (HDL-ODPRs) in open-source repositories. An algorithm used to extract textual data from pull requests is hybrid leader-based optimization (HLBO), which increases the accuracy of duplicate detection. Following that, we compute the similarities between pull requests by utilizing the multiobjective alpine skiing optimization (MASO) algorithm, which provides textual, file-change, and code-change similarities. For pull request duplicate detection, a hybrid deep learning technique (named GAN-GS) is introduced, in which the global search (GS) algorithm is used to optimize the design metrics of the generative adversarial network (GAN). The proposed HDL-ODPR model is validated against the public standard benchmark datasets, such as DupPR-basic and DupPR-complementary data. According to the simulation results, the proposed HDL-ODPR model can achieve promising results in comparison with existing state-of-the-art models. [ABSTRACT FROM AUTHOR]

Industrial software maintenance is critical but burdensome. Activities such as detecting duplicate bug reports are often performed manually. Herein an automated duplicate bug report detection system improves maintenance efficiency using vectorization of the contents and deep learning–based sentence embedding to calculate the similarity of the whole report from vectors of individual elements. Specifically, sentence embedding is realized using Sentence-BERT fine tuning. Additionally, its performance is experimentally compared to baseline methods to validate the proposed system. The proposed system detects duplicate bug reports more effectively than existing methods.

We consider the problem of duplicate detection in noisy and incomplete data:given a large data set in which each record has multiple entries (attributes),detect which distinct records refer to the same real world entity. This task iscomplicated by noise (such as misspellings) and missing data, which can lead torecords being different, despite referring to the same entity. Our methodconsists of three main steps: creating a similarity score between records,grouping records together into "unique entities", and refining the groups. Wecompare various methods for creating similarity scores between noisy records,considering different combinations of string matching, term frequency-inversedocument frequency methods, and n-gram techniques. In particular, we introducea vectorized soft term frequency-inverse document frequency method, with anoptional refinement step. We also discuss two methods to deal with missing datain computing similarity scores. We test our method on the Los Angeles Police Department Field Interview Carddata set, the Cora Citation Matching data set, and two sets of restaurantreview data. The results show that the methods that use words as the basicunits are preferable to those that use 3-grams. Moreover, in some (butcertainly not all) parameter ranges soft term frequency-inverse documentfrequency methods can outperform the standard term frequency-inverse documentfrequency method. The results also confirm that our method for automaticallydetermining the number of groups typically works well in many cases and allowsfor accurate results in the absence of a priori knowledge of the number ofunique entities in the data set.

The data management process is characterised by a set of tasks where data quality management (DQM) is one of the core components. Data quality, however, is a multidimensional concept, where the nature of the data quality issues is very diverse. One of the most widely anticipated data quality challenges, which becomes particularly vital when data come from multiple data sources which is a typical situation in the current data-driven world, is duplicates or non-uniqueness. Even more, duplicates were recognised to be one of the key domain-specific data quality dimensions in the context of the Internet of Things (IoT) application domains, where smart grids and health dominate most. Duplicate data lead to inaccurate analyses, leading to wrong decisions, negatively affect data-driven and/or data processing activities such as the development of models, forecasts, simulations, have a negative impact on customer service, risk and crisis management, service personalisation in terms of both their accuracy and trustworthiness, decrease user adoption and satisfaction, etc. The process of determination and elimination of duplicates is known as deduplication, while the process of finding duplicates in one or more databases that refer to the same entities is known as Record Linkage. To find the duplicates, the data sets are compared with each other using similarity functions that are usually used to compare two input strings to find similarities between them, which requires quadratic time complexity. To defuse the quadratic complexity of the problem, especially in large data sources, record linkage methods, such as blocking and sorted neighbourhood, are used. In this paper, we propose a six-step record linkage deduplication framework. The operation of the framework is demonstrated on a simplified example of research data artifacts, such as publications, research projects and others of the real-world research institution representing Research Information Systems (RIS) domain. To make the proposed framework usable we integrated it into a tool that is already used in practice, by developing a prototype of an extension for the well-known DataCleaner. The framework detects and visualises duplicates thereby identifying and providing the user with identified redundancies in a user-friendly manner allowing their further elimination. By removing the redundancies, the quality of the data is improved therefore improving analyses and decision-making. This study makes a call for other researchers to take a step towards the "golden record" that can be achieved when all data quality issues are recognised and resolved, thus moving towards absolute data quality. [ABSTRACT FROM AUTHOR]

Abstract Background The identity management is a central component in medical research. Patients are recruited from various sites, which requires an error tolerant record linkage method, to ensure that patients are registered only once. In large research projects or institutions, the identity management has to deal with several thousands or millions of patients. In environments with large numbers of patients the register process could lead to high runtimes caused by record linkage. The Central Biomaterial Bank of the Charité (ZeBanC) searched for an identity management solution, which can handle millions of patients in large research projects with an acceptable performance. The goal of this paper was to simulate the registration of several million patients using the E-PIX service at Charité – Universitätsmedizin Berlin. The E-PIX service was evaluated in terms of needed runtimes, memory requirements, and processor utilization. A total of at least 20 million patients had to be registered. The runtimes to register patients into databases with various sizes should be examined, and the maximum number of patients, which the E-PIX service could handle, should be determined. Methods Tools were set up or developed to measure the needed runtimes, the memory used and the processor usage to register patients into various sizes of databases. To generate runtimes close to reality, modified patient data based on transposed real patient data were used for the simulation. The transposed patient data were sent to E-PIX to measure the runtimes of the registration process. This measurement was repeated for various database sizes. Results E-PIX is suitable to manage multi-million patients within a dataset. With the given hardware, it was possible to register a total of more than 30 million patients. It was possible to register more than 16 thousand patients per day into this database. Conclusions The E-PIX tool fulfills the requirements of the Charité to be used for large research projects. The use of E-PIX is intended for the research context in the Charité.

Automatic news repository collection systems involve a news crawler that extracts news from different news portals, subsequently, these news need to be processed to figure out the category of a news article e.g. sports, politics, showbiz etc. In this process there are two main challenges first one is to place a news article under the right category of news, while the second one is to detect a duplicate news, i.e. when the news are being extracted from multiple sources, it is highly probable to get the same news from many different portals, resulting into duplicate news; failing to which may result into inconsistent statistics obtained after pre-processing the news text. This problem becomes more pertinent when we deal with human loss news involving crime, accident etc. related news articles. As the system may count the same news many times resulting into misleading statistics. In order to address these problems, this research presents the following contributions. Firstly, a news corpus comprising of human loss news of different categories has been developed by gathering data from different sources of well-known and authentic news websites. The corpus also includes a number of duplicate news. Secondly, a comparison of different classification approaches has been conducted to empirically find out the best suitable text classifier for the categorization of different sub-categories of human loss news. Lastly, methods have been proposed and compared to detect duplicate news from the corpus by involving different pre-processing techniques and widely used similarity measures, cosine similarity, and Jaccard's coefficient. The results show that conventional text classifiers are still relevant and perform well in text classification tasks as MNB has given 89.5% accurate results. While, Jaccard coefficient exhibits much better results than Cosine similarity for duplicate news detection with different pre-processing variations with an average accuracy of 83.16%.

Recently, open-source repositories have grown rapidly due to volunteer contributions worldwide. Collaboration software platforms have gained popularity as thousands of external contributors have contributed to open-source repositories. Although data de-duplication decreases the size of backup workloads, this causes poor data locality (fragmentation) and redundant review time and effort. Deep learning and machine learning techniques have recently been applied to identify complex bugs and duplicate issue reports. It is difficult to use, but it increases the risk of developers submitting duplicate pull requests, resulting in additional maintenance costs. We propose a hybrid deep learning technique in this work on the basis of an optimal duplication detection is for pull requests (HDL-ODPRs) in open-source repositories. An algorithm used to extract textual data from pull requests is hybrid leader-based optimization (HLBO), which increases the accuracy of duplicate detection. Following that, we compute the similarities between pull requests by utilizing the multiobjective alpine skiing optimization (MASO) algorithm, which provides textual, file-change, and code-change similarities. For pull request duplicate detection, a hybrid deep learning technique (named GAN-GS) is introduced, in which the global search (GS) algorithm is used to optimize the design metrics of the generative adversarial network (GAN). The proposed HDL-ODPR model is validated against the public standard benchmark datasets, such as DupPR-basic and DupPR-complementary data. According to the simulation results, the proposed HDL-ODPR model can achieve promising results in comparison with existing state-of-the-art models.

In this paper, we describe various application scenarios for archive management, broadcast/stream analysis, media search and media forensics which require the detection and accurate localization of unknown partial audio matches within items and datasets. We explain why they cannot be addressed with state-of-the-art matching approaches based on fingerprinting, and propose a new partial matching algorithm which can satisfy the relevant requirements. We propose two distinct requirement sets and hence two variants / settings for our proposed approach: One focusing on lower time granularity and hence lower computational complexity, to be able to deal with large datasets, and one focusing on fine-grain analysis for small datasets and individual items. Both variants are tested using distinct evaluation sets and methodologies and compared with a popular audio matching algorithm, thereby demonstrating that the proposed algorithm achieves convincing performance for the relevant application scenarios beyond the current state-of-the-art. [ABSTRACT FROM AUTHOR]

While organizations in the current era of big data are generating massive volumes of data, they also need to ensure that its quality is maintained for it to be useful in decision-making purposes. The problem of dirty data plagues every organization. One aspect of dirty data is the presence of duplicate data records that negatively impact the organization's operations in many ways. Many existing approaches attempt to address this problem by using traditional data cleansing methods. In this paper, we address this problem by using an in-house crowdsourcing-based framework, namely, DedupCrowd. One of the main obstacles of crowdsourcing-based approaches is to monitor the performance of the crowd, by which the integrity of the whole process is maintained. In this paper, a statistical quality control-based technique is proposed to regulate the performance of the crowd. We apply our proposed framework in the context of a contact center, where the Customer Service Representatives are used as the crowd to assist in the process of deduplicating detection. By using comprehensive working examples, we show how the different modules of the DedupCrowd work not only to monitor the performance of the crowd but also to assist in duplicate detection.

The data management process is characterised by a set of tasks where data quality management (DQM) is one of the core components. Data quality, however, is a multidimensional concept, where the nature of the data quality issues is very diverse. One of the most widely anticipated data quality challenges, which becomes particularly vital when data come from multiple data sources which is a typical situation in the current data-driven world, is duplicates or non-uniqueness. Even more, duplicates were recognised to be one of the key domain-specific data quality dimensions in the context of the Internet of Things (IoT) application domains, where smart grids and health dominate most. Duplicate data lead to inaccurate analyses, leading to wrong decisions, negatively affect data-driven and/or data processing activities such as the development of models, forecasts, simulations, have a negative impact on customer service, risk and crisis management, service personalisation in terms of both their accuracy and trustworthiness, decrease user adoption and satisfaction, etc. The process of determination and elimination of duplicates is known as deduplication, while the process of finding duplicates in one or more databases that refer to the same entities is known as Record Linkage. To find the duplicates, the data sets are compared with each other using similarity functions that are usually used to compare two input strings to find similarities between them, which requires quadratic time complexity. To defuse the quadratic complexity of the problem, especially in large data sources, record linkage methods, such as blocking and sorted neighbourhood, are used. In this paper, we propose a six-step record linkage deduplication framework. The operation of the framework is demonstrated on a simplified example of research data artifacts, such as publications, research projects and others of the real-world research institution representing Research Information Systems (RIS) domain. To make the proposed framework usable we integrated it into a tool that is already used in practice, by developing a prototype of an extension for the well-known DataCleaner. The framework detects and visualises duplicates thereby identifying and providing the user with identified redundancies in a user-friendly manner allowing their further elimination. By removing the redundancies, the quality of the data is improved therefore improving analyses and decision-making. This study makes a call for other researchers to take a step towards the “golden record” that can be achieved when all data quality issues are recognised and resolved, thus moving towards absolute data quality.

Communication and coordination between open source software (OSS) developers who do not work physically in the same location have always been the challenging issues. The pull-based development model, as the state-of-the-art collaborative development mechanism, provides high openness and transparency to improve the visibility of contributors' work. However, duplicate contributions may still be submitted by more than one contributor to solve the same problem due to the parallel and uncoordinated nature of this model. If not detected in time, duplicate pull-requests can cause contributors and reviewers to waste time and energy on redundant work. In this paper, we propose an approach combining textual and change similarities to automatically detect duplicate contributions in the pull-based model at submission time. For a new-arriving contribution, we first compute textual similarity and change similarity between it and other existing contributions. And then our method returns a list of candidate duplicate contributions that are most similar to the new contribution in terms of the combined textual and change similarity. The evaluation shows that 83.4% of the duplicates can be found in average when we use the combined textual and change similarity compared with 54.8% using only textual similarity and 78.2% using only change similarity. [ABSTRACT FROM AUTHOR]

Several tasks related to geographical information retrieval and to the geographical information sciences involve toponym matching, that is, the problem of matching place names that share a common referent. In this article, we present the results of a wide-ranging evaluation on the performance of different string similarity metrics over the toponym matching task. We also report on experiments involving the usage of supervised machine learning for combining multiple similarity metrics, which has the natural advantage of avoiding the manual tuning of similarity thresholds. Experiments with a very large dataset show that the performance differences for the individual similarity metrics are relatively small, and that carefully tuning the similarity threshold is important for achieving good results. The methods based on supervised machine learning, particularly when considering ensembles of decision trees, can achieve good results on this task, significantly outperforming the individual similarity metrics.

In keyword-driven testing (KDT), having duplicate actions in the test script is perhaps the most common bad smell. Once the target user interface is changed, a KDT script with duplicate actions can be difficult to maintain. Thus, detecting and removing duplicate actions is an important task. However, so far, no KDT testing tools support automated duplicate detection. This paper proposes a method and tool, called DDT (Duplicate script Detection Tool), for the tester to quickly identify duplicate actions. Two detection algorithms based on Longest Repeated Substring (LRS) and Longest Common Subsequence (LCS) are presented. In addition, DDT provides a keyword extraction feature that can automatically remove duplicate actions. Our experimental results show that there are 21-42% of duplicate actions in a typical KDT script, DDT can detect these duplicate actions in 3-5 seconds, and up to 58-81% of these duplicate actions should be refactored. [ABSTRACT FROM AUTHOR]

Background: The identity management is a central component in medical research. Patients are recruited from various sites, which requires an error tolerant record linkage method, to ensure that patients are registered only once. In large research projects or institutions, the identity management has to deal with several thousands or millions of patients. In environments with large numbers of patients the register process could lead to high runtimes caused by record linkage. The Central Biomaterial Bank of the Charité (ZeBanC) searched for an identity management solution, which can handle millions of patients in large research projects with an acceptable performance. The goal of this paper was to simulate the registration of several million patients using the E-PIX service at Charité - Universitätsmedizin Berlin. The E-PIX service was evaluated in terms of needed runtimes, memory requirements, and processor utilization. A total of at least 20 million patients had to be registered. The runtimes to register patients into databases with various sizes should be examined, and the maximum number of patients, which the E-PIX service could handle, should be determined.Methods: Tools were set up or developed to measure the needed runtimes, the memory used and the processor usage to register patients into various sizes of databases. To generate runtimes close to reality, modified patient data based on transposed real patient data were used for the simulation. The transposed patient data were sent to E-PIX to measure the runtimes of the registration process. This measurement was repeated for various database sizes.Results: E-PIX is suitable to manage multi-million patients within a dataset. With the given hardware, it was possible to register a total of more than 30 million patients. It was possible to register more than 16 thousand patients per day into this database.Conclusions: The E-PIX tool fulfills the requirements of the Charité to be used for large research projects. The use of E-PIX is intended for the research context in the Charité. [ABSTRACT FROM AUTHOR]

In this paper, a comprehensive performance analysis of duplicate data detection techniques for relational databases has been performed. The research focuses on traditional SQL based and modern bloom filter techniques to find and eliminate records which already exist in the database while performing bulk insertion operation (i.e. bulk insertion involved in the loading phase of the Extract, Transform, and Load (ETL) process and data synchronization in multisite database synchronization). The comprehensive performance analysis was performed on several data sizes using SQL, bloom filter, and parallel bloom filter. The results show that the parallel bloom filter is highly suitable for duplicate detection in the database. [ABSTRACT FROM AUTHOR]