Search

Apache Spark programming framework provides an effective and vigorous open-source solution for Big data testing and management. It offers a rich set of big data applications through a programming interface. Testing procedures deliver optimum solutions to complex business problems like analyzing large data volumes. A typical test engineer demands manual to automatic testing which assists the testing strategy with minimal effort. Big data testing means, verification, and validation of data while storing and processing it. This research paper demonstrates the challenges with new high-performance analytics for simpler and faster testing processing of relevant data. It enables, timely and accurate insights using Big data testing predict analytics and can manage large quantities of Structured, Semi-structured, and Unstructured data forms with spark analysis. These methods are evaluated using Extract, Transformation, Loading, and Apache spark procedures. The proposed Big data testing method has statistically outperformed conventional procedures.

Online experiments are ubiquitous. As the scale of experiments has grown, so has the complexity of their design and implementation. In response, firms have developed software frameworks for designing and deploying online experiments. Ensuring that experiments in these frameworks are correctly designed and that their results are trustworthy---referred to as *internal validity*---can be difficult. Currently, verifying internal validity requires manual inspection by someone with substantial expertise in experimental design. We present the first approach for statically checking the internal validity of online experiments. Our checks are based on well-known problems that arise in experimental design and causal inference. Our analyses target PlanOut, a widely deployed, open-source experimentation framework that uses a domain-specific language to specify and run complex experiments. We have built a tool, PlanAlyzer, that checks PlanOut programs for a variety of threats to internal validity, including failures of randomization, treatment assignment, and causal sufficiency. PlanAlyzer uses its analyses to automatically generate *contrasts*, a key type of information required to perform valid statistical analyses over experimental results. We demonstrate PlanAlyzer's utility on a corpus of PlanOut scripts deployed in production at Facebook, and we evaluate its ability to identify threats to validity on a mutated subset of this corpus. PlanAlyzer has both precision and recall of 92% on the mutated corpus, and 82% of the contrasts it automatically generates match hand-specified data., 30 pages, hella long

This paper describes briefly about the concept of serverless cloud computing model, its usage in IT industries and its benefits. In the traditional model the developer is responsible for resource allocation, managing servers and owning of servers, and it included three models based upon the service such as IaaS, PaaS and SaaS. In IaaS (Infrastructure as a Service) the content storage and accessing of network is carried out by the cloud provider, SaaS (Software as a Service) here different software’s are provided to the user as a service, PaaS (Platform as a Service), the developer gets access to certain services for carrying out organizing process and run it accordingly. In serverless cloud computing, the developer need not worry about owning, management, and maintenance of servers as it is carried out by the cloud service provider. Hence by using this model, the time that is needed for a system to reach the market is very much reduced and is cost effective. Serverless architecture includes three categories namely, AWS Lambda, Azure, and Google cloud. It also includes certain challenges such as it cannot be used in the case where a process takes longer time to run and it is discussed below in this paper.

A frequent pattern is a substructure that appears in a database with frequency (aka. support) no less than a user-specified threshold, while a closed pattern is one that has no super-pattern that has the same support. Here, a substructure can refer to different structural forms, such as itemsets, subsequences, subtrees, and subgraphs, and mining such substructures is important in many real applications such as product recommendation and feature extraction. Currently, there lacks a general programming framework that can be easily customized to mine different types of patterns, and existing parallel and distributed solutions are IO-bound rendering CPU cores underutilized. Since mining frequent and/or closed patterns are NP-hard, it is important to fully utilize the available CPU cores. This paper presents such a general-purpose framework called PrefixFPM. The framework is based on the idea of prefix projection which allows a divide-and-conquer mining paradigm. PrefixFPM exposes a unified programming interface to users who can readily customize it to mine their desired patterns. We have adapted the state-of-the-art serial algorithms for mining patterns including subsequences, subtrees, and subgraphs on top of PrefixFPM, and extensive experiments demonstrate an excellent speedup ratio of PrefixFPM with the number of CPU cores.

During the implementation process of identification systems, multiple factors need to be considered at the same time. Due to the large amount of calculation and the randomness of the signal, the automatic identification capability of the machine is currently poor, especially for continuous signal. In order to solve this problem, a neuron fuzzy identification system was described based on a complex nonlinear mathematical model, which was designed from both hardware and software aspects of the system. The hardware architecture diagram was constructed based on the S3C:2440 microprocessor. The main modules include the power module, acquisition module, storage module, and output module. The DSP/BIOS system was used to construct the software framework diagram of the neuron fuzzy identification system to describe the identification process of video data stream, image data stream and control signal data. The software algorithm was designed based on the neuron fuzzy theory to establish a fuzzy similarity matrix and the best identification result was found by the maximum and minimum methods. The experimental results show that the designed system has higher recognition ability. When the frame length reaches 40 frames, the recognition rate increases to a larger value, the recognition rate is 75%, and the information is more accurate.

Aiming at the problem of DDoS attack detection in internet of things (IoT) environment, statistical and machine-learning algorithms are proposed to model and analyze the network traffic of DDoS attack. Docker-based virtualization platform is designed and configured to collect IoT network traffic data. Then the packet-level, flow-level, and second-level network traffic datasets are generated, and the importance of features in different traffic datasets are sorted. By SKlearn and TensorFlow machine-learning software framework, different machine learning algorithms are researched and compared. In packet-level DDoS attack detection, KNN algorithm achieves the best results; the accuracy is 92.8%. In flow-level DDoS attack detection, the voting algorithm achieves the best results; the accuracy is 99.8%. In second-level DDoS attack detection, the RNN algorithm behaves best results; the accuracy is 97.1%. The DDoS attack detection method combined with statistical analysis and machine-learning can effectively detect large-scale DDoS attacks on the internet of things simulation experimental environment.

Substantial research is available on detectinginfluencerson social media platforms. In contrast, comparatively few studies exists on the role ofonline activists, defined informally as users who actively participate in socially-minded online campaigns. Automatically discovering activists who can potentially be approached by organisations that promote social campaigns is important, but not easy, as they are typically active only locally, and, unlike influencers, they are not central to large social media networks. We make the hypothesis that such interesting users can be found on Twitter within temporally and spatially localisedcontexts. We define these as small but topical fragments of the network, containing interactions about social events or campaigns with a significant online footprint. To explore this hypothesis, we have designed an iterative discovery pipeline consisting of two alternating phases of user discovery and context discovery. Multiple iterations of the pipeline result in a growing dataset of user profiles for activists, as well as growing set of online social contexts. This mode of exploration differs significantly from prior techniques that focus on influencers, and presents unique challenges because of the weak online signal available to detect activists. The paper describes the design and implementation of the pipeline as a customisable software framework, where user-defined operational definitions of online activism can be explored. We present an empirical evaluation on two extensive case studies, one concerning healthcare-related campaigns in the UK during 2018, the other related to online activism in Italy during the COVID-19 pandemic.

Differential privacy offers a formal framework for reasoning about the privacy and accuracy of computations on private data. It also offers a rich set of building blocks for constructing private data analyses. When carefully calibrated, these analyses simultaneously guarantee the privacy of the individuals contributing their data, and the accuracy of the data analysis results, inferring useful properties about the population. The compositional nature of differential privacy has motivated the design and implementation of several programming languages to ease the implementation of differentially private analyses. Even though these programming languages provide support for reasoning about privacy, most of them disregard reasoning about the accuracy of data analyses. To overcome this limitation, we present DPella, a programming framework providing data analysts with support for reasoning about privacy, accuracy, and their trade-offs. The distinguishing feature of DPella is a novel component that statically tracks the accuracy of different data analyses. To provide tight accuracy estimations, this component leverages taint analysis for automatically inferring statistical independence of the different noise quantities added for guaranteeing privacy. We evaluate our approach by implementing several classical queries from the literature and showing how data analysts can calibrate the privacy parameters to meet the accuracy requirements, and vice versa.

Robotic manipulation is fundamental to many realworld applications; however, it is an unsolved problem that remains a very active research area. New algorithms for robot perception and control are frequently proposed by the research community. These methods must be thoroughly evaluated in realistic conditions before they can be adopted by industry. This process can be extremely time consuming, mainly due to the complexity of integrating different hardware and software components.

Using large, multi-national datasets for high-performance medical imaging AI systems requires innovation in privacy-preserving machine learning so models can train on sensitive data without requiring data transfer. Here we present PriMIA (Privacy-preserving Medical Image Analysis), a free, open-source software framework for differentially private, securely aggregated federated learning and encrypted inference on medical imaging data. We test PriMIA using a real-life case study in which an expert-level deep convolutional neural network classifies paediatric chest X-rays; the resulting model’s classification performance is on par with locally, non-securely trained models. We theoretically and empirically evaluate our framework’s performance and privacy guarantees, and demonstrate that the protections provided prevent the reconstruction of usable data by a gradient-based model inversion attack. Finally, we successfully employ the trained model in an end-to-end encrypted remote inference scenario using secure multi-party computation to prevent the disclosure of the data and the model. Gaining access to medical data to train AI applications can present problems due to patient privacy or proprietary interests. A way forward can be privacy-preserving federated learning schemes. Kaissis, Ziller and colleagues demonstrate here their open source framework for privacy-preserving medical image analysis in a remote inference scenario.

Deep learning has sparked a renaissance in computer systems and architecture. Despite the breakneck pace of innovation, there is a crucial issue concerning the research and industry communities at large: how to enable neutral and useful performance assessment for machine learning (ML) software frameworks, ML hardware accelerators, and ML systems comprising both the software stack and the hardware. The ML field needs systematic methods for evaluating performance that represents real-world use cases and useful for making comparisons across different software and hardware implementations. MLPerf answers the call. MLPerf is an ML benchmark standard driven by academia and industry (70+ organizations). Built out of the expertise of multiple organizations, MLPerf establishes a standard benchmark suite with proper metrics and benchmarking methodologies to level the playing field for ML system performance measurement of different ML inference hardware, software, and services.

Edge computing technologies facilitate the deployment of services on nearby edge servers with a large number of end users and their mobile devices to fulfill personalized demands. Owing to frequent changes in user mobility and demands, service systems deployed in an edge–cloud environment must continuously adapt to ensure that the quality of service (QoS) perceived by the end users is maintained at a stable and satisfactory level. As it is difficult for system operation engineers to manually deal with such frequent and large-scale evolution due to problems of cost and efficiency, self-adaptation of the system is essential. In this paper, we present a programming framework for microservices (EPF4M) and an infrastructure for self-adaptive microservice systems (EI4MS) for the cloud–edge environment based on microservice architecture. Our study follows a “monitoring–analyzing–planning–execution” control loop that empowers the service systems to redeploy the services according to changes in the QoS. A two-phase strategy is adopted to minimize the side effects of the loop on the performance of the service system. A prototype of this framework and infrastructure has been open-sourced and verified through experiments conducted in a real cloud–edge environment. The results demonstrate the usefulness and advantages of our approach.

Cognitive modeling tools have been widely used by researchers and practitioners to help design, evaluate, and study computer user interfaces (UIs). Despite their usefulness, large-scale modeling tasks can still be very challenging due to the amount of manual work needed. To address this scalability challenge, we propose CogTool+, a new cognitive modeling software framework developed on top of the well-known software tool CogTool. CogTool+ addresses the scalability problem by supporting the following key features: (1) a higher level of parameterization and automation; (2) algorithmic components; (3) interfaces for using external data; and (4) a clear separation of tasks, which allows programmers and psychologists to define reusable components (e.g., algorithmic modules and behavioral templates) that can be used by UI/UX researchers and designers without the need to understand the low-level implementation details of such components. CogTool+ also supports mixed cognitive models required for many large-scale modeling tasks and provides an offline analyzer of simulation results. In order to show how CogTool+ can reduce the human effort required for large-scale modeling, we illustrate how it works using a pedagogical example, and demonstrate its actual performance by applying it to large-scale modeling tasks of two real-world user-authentication systems.

Hadoop Distributed File System is used for storage along with a programming framework MapReduce for processing large datasets allowing parallel processing. The process of handling such complex and vast data and maintaining the performance parameters up to certain level is a difficult task. Hence, an improvised mechanism is proposed here that will enhance the job scheduling capabilities of Hadoop and optimize allocation and utilization of resources. Significantly, an aggregator node is added to the default HDFS framework architecture to improve the performance of Hadoop Name node. In this paper, four entities viz., the name node, secondary name node, aggregator nodes, and data nodes have been modified. Here, the aggregator node assigns jobs to data node, while Name node tracks aggregator nodes. Also, based on the job size and expected execution time, an improvised ant colony optimization method is developed for scheduling jobs.In the end, the results demonstrate notable improvisation over native Hadoop and other approaches.

In proton therapy, the knowledge of the proton stopping power, i.e. the energy deposition per unit length within human tissue, is essential for accurate treatment planning. One suitable method to directly measure the stopping power is proton computed tomography (pCT). Due to the proton interaction mechanisms in matter, pCT image reconstruction faces some challenges: the unique path of each proton has to be considered separately in the reconstruction process adding complexity to the reconstruction problem. This study shows that the GPU-based open-source software toolkit TIGRE, which was initially intended for X-ray CT reconstruction, can be applied to the pCT image reconstruction problem using a straight line approach for the proton path. This simplified approach allows for reconstructions within seconds. To validate the applicability of TIGRE to pCT, several Monte Carlo simulations modeling a pCT setup with two Catphan® modules as phantoms were performed. Ordered-Subset Simultaneous Algebraic Reconstruction Technique (OS-SART) and Adaptive-Steepest-Descent Projection Onto Convex Sets (ASD-POCS) were used for image reconstruction. Since the accuracy of the approach is limited by the straight line approximation of the proton path, requirements for further improvement of TIGRE for pCT are addressed.

The work is devoted to the research that was carried out within the framework of computer vision problems applicable to the analysis of images and video information with vehicles. We solve the problem of classifying vehicles. We analyze the drawbacks of Haar features and convolutional neural networks and test the obtained networks using the key point method; we construct an integral algorithm that includes several networks, and we further validate it on a large number of real photographs and types of vehicles. Next, we solve the task to develop a software framework for tracking vehicles by analyzing adjacent photographs from a video sequence. After that, we consider the tracking task in more detail. We analyze modern tracking algorithms using machine learning and describe our implemented tracker with support for the appearance of obstacles between the camera and a moving vehicle. As a result, we propose algorithms and open-source software that, after being configured for specific cameras, can be used in traffic analysis systems.

Big data is the high-volume, high-variety data which involves data storage, data management, and data analysis that presents a wide view of business possibility for real-time data, sensor data, and streaming data over the web. Big data relies on technology, analysis, and mythology where technology deals with computation power, accuracy, linking, and large datasets; analysis is to find patterns by analyzing large datasets to discover hidden information; and mythology is the wrong beliefs that large datasets give insight knowledge of data that is not obtained by small datasets. In this paper, the authors analyzed the major benefits the organization see from employing contract workers using map reduce programming framework.

As a part of the future upgrade program of the Multi-Purpose Detector (MPD) experiment at the Nuclotron-Based Ion Collider Facility (NICA) complex, an Inner Tracking System (ITS) made of Monolitic Active Pixel Sensors (MAPSs) is proposed between the beam pipe and the Time Projection Chamber (TPC). It is expected that the new detector will enhance the experimental potential for the reconstruction of short-lived particles—in particular, those containing the open charm particle. To study the detector performance and select its best configuration, a track reconstruction approach based on a constrained combinatorial search was developed and implemented as a software toolkit called Vector Finder. This paper describes the proposed approach and demonstrates its characteristics for primary and secondary track finding in ITS, ITS-to-TPC track matching and hyperon reconstruction within the MPD software framework. The results were obtained on a set of simulated central gold–gold collision events at sNN=9 GeV with an average multiplicity of ∼1000 charged particles in the detector acceptance produced with the Ultra-Relativistic Quantum Molecular Dynamics (UrQMD) generator.

Domain-specific programming frameworks are usually effective to simplify the development of large-scale applications on supercomputers. This paper introduces a parallel programming framework named JCOGIN for particle transport on combinatorial geometry. JCOGIN provides a combinatorial geometry data model and a patch-based parallel computing model to manage the data distribution in parallel computing and implements the hybrid parallelism of the domain decomposition and the particle parallelism on MPI/OpenMP to overcome the bottleneck of huge memory demand and long computational time. The application programming interface of JCOGIN can support users to quickly develop their parallel particle transport applications. Based on this framework, users only need to write serial codes for large-scale numerical simulations on modern supercomputers. The parallel efficiency of applications based on JCOGIN can reach up to 80% on hundreds of thousands of CPU cores.

解法が確立していない問題を解くためのソフトウェア開発では，プログラムの仕様の変更が頻繁に起こることがある．本論文は，頻繁な仕様変更に対応しながら正しくて効率的なプログラムを開発するためのフレームワークを提案する．本研究の特徴は，プログラムの部品を書き換えルールで表現し，仕様から正しい書き換えルールを生成・集積してプログラムを構成することである．このことによってプログラムの効率と正しさの両立が可能になる．フレームワークの適用例としてコンテナ・プリマーシャリング問題の最適解を高速列挙するプログラムの開発プロセスを扱う．, Frequent revision of program specifications can occur in software development to solve problems of which well-established solutions have yet to be found. This paper proposes a framework for software development to efficiently enumerate optimal solutions to combinatorial optimization problems; the framework copes with the frequent specification changes. A peculiarity of this study is that we represent each program component as a correct rewriting rule, which is generated from a specification and accumulates in a program. The peculiarity enables us to manage both efficiency and correctness of a program. This paper demonstrates a software development process to efficiently enumerate optimal solutions to a container pre-marshalling problem as an application example of the ramework., 情報科学, Information Sciences, 論文

This paper presents an extension to a software framework designed to evaluate the efficiency of different software and hardware-accelerated virtual switches, each commonly adopted on Linux to provide virtual network connectivity to containers in high-performance scenarios, like in Network Function Virtualization (NFV). We present results from the use of our tools, showing the performance of multiple high-performance networking frameworks on a specific platform, comparing the collected data for various key metrics, namely throughput, latency and scalability, with respect to the required computational power.