Search

In the era of rapid experimental expansion data analysis needs are rapidly outpacing the capabilities of small institutional clusters and looking to integrate HPC resources into their workflow. We propose one way of reconciling on-demand needs of experimental analytics with the batch managed HPC resources within a system that dynamically moves nodes between an on-demand cluster configured with cloud technology (OpenStack) and a traditional HPC cluster managed by a batch scheduler (Torque). We evaluate this system experimentally both in the context of real-life traces representing two years of a specific institutional need, and via experiments in the context of synthetic traces that capture generalized characteristics of potential batch and on-demand workloads. Our results for the real-life scenario show that our approach could reduce the current investment in on-demand infrastructure by 82% while at the same time improving the mean batch wait time almost by an order of magnitude (8x).

Interest has been increasing in mining information from a huge amount of data for various purposes such as business processes, scientific discoveries, and security. MINDS (Minnesota INtrusion Detection System) is a tool utilizing data mining techniques to identify both known and unknown network intrusions. While performing well with a high degree of accuracy in detecting intrusions, MINDS is intrinsically limited due to its centralized design. At the same time, there are increasing needs for coordinating multiple distributed sites to find correlations of suspicious events and for sharing distributed anomaly data in different administrative domains since network intrusions become more sophisticated and harder to identify by a single site analysis. In this work, we consider a distributed model of MINDS to eliminate the limitations that the current MINDS suffers from due to its centralized nature. We utilize Grid technologies to achieve the distribution of services. In addition, we develop security frameworks for authentication and access control since it is important to preserve privacy and sensitivity of data regardless of distribution. To verify the feasibility of our approach, we also present our experimental results conducted in a local testbed with four nodes and in a wide-area setting in PlanetLab with 20 nodes geographically dispersed.

Geo-distributed cloud storage systems must tame complexity at many levels: uniform APIs for storage access, supporting flexible storage policies that meet a wide array of application metrics, handling uncertain network dynamics and access dynamism, and operating across many levels of heterogeneity both within and across data-centers. In this paper, we present an integrated solution called Wiera. Wiera extends our earlier cloud storage system, Tiera, that is targeted to multi-tiered policy-based single cloud storage, to the wide-area and multiple data-centers (even across different providers). Wiera enables the specification of global data management policies built on top of local Tiera policies. Such policies enable the user to optimize for cost, performance, reliability, durability, and consistency, both within and across data-centers, and to express their tradeoffs. A key aspect of Wiera is first-class support for dynamism due to network, workload, and access patterns changes. Wiera policies can adapt to changes in user workload, poorly performing data tiers, failures, and changes in user metrics (e.g., cost). Wiera allows unmodified applications to reap the benefits of flexible data/storage policies by externalizing the policy specification. As far as we know, Wiera is the first geo-distributed cloud storage system which handles dynamism actively at run-time. We show how Wiera enables a rich specification of dynamic policies using a concise notation and describe the design and implementation of the system. We have implemented a Wiera prototype on multiple cloud environments, AWS and Azure, that illustrates potential benefits from managing dynamics and in using multiple cloud storage tiers both within and across data-centers.

As the number of services and the size of data involved in workflows increases, centralized orchestration techniques are reaching the limits of scalability. When relying on web services without third-party data transfer, a standard orchestration model needs to pass all data through a centralized engine, which results in unnecessary data transfer and the engine to become a bottleneck to the execution of a workflow. As a solution, this paper presents and evaluates Circulate, an alternative service-oriented architecture which facilitates an orchestration model of central control in combination with a choreography model of optimized distributed data transport. Extensive performance analysis through the PlanetLab framework is conducted on a web service-based implementation over a range of Internet-scale configurations which mirror scientific workflow environments. Performance analysis concludes that our architecture's optimized model of data transport speeds up the execution time of workflows, consistently outperforms standard orchestration and scales with data and node size. Furthermore, Circulate is a less-intrusive solution as individual services do not have to be reconfigured in order to take part in a workflow.

Network service-based computation is a promising paradigm for both scientific and engineering, and enterprise computing. The network service allows users to focus on their application and obtain services when needed, simply by invoking the service across the network. In this paper, we show that an adaptive, general-purpose run-time infrastructure in support of effective resource management can be built for a wide range of high-end network services running in a single-site cluster and in a Grid. The primary components of the run-time infrastructure are: (1) dynamic performance prediction; (2) adaptive intra-site resource management; and (3) adaptive inter-site resource management. The novel aspect of our approach is that the run-time system is able to dynamically select the most appropriate performance predictor or resource management strategy over time. This capability not only improves the performance, but also makes the infrastructure reusable across different high-end services. To evaluate the effectiveness and applicability of our approach, we have transformed two different classes of high-end applications-data parallel and distributed applications-into network services using the infrastructure. The experimental results show that the network services running on the infrastructure significantly reduce the overall service times under dynamically varying circumstances.

As the number of services and the size of data involved in workflows increases, centralised orchestration techniques are reaching the limits of scalability. In the classic orchestration model, all data passes through a centralised engine, which results in unnecessary data transfer, wasted bandwidth and the engine to become a bottleneck to the execution of a workflow. This paper presents and evaluates the Circulate architecture which maintains the robustness and simplicity of centralised orchestration, but facilitates choreography by allowing services to exchange data directly with one another. Circulate could be realised within any existing workflow framework, in this paper, we focus on WS-Circulate, a Web services based implementation. Taking inspiration from the Montage workflow, a number of common workflow patterns (sequence, fan-in and fan-out), input to output data size relationships and network configurations are identified and evaluated. The performance analysis concludes that a substantial reduction in communication overhead results in a 2---4 fold performance benefit across all patterns. An end-to-end pattern through the Montage workflow results in an 8 fold performance benefit and demonstrates how the advantage of using the Circulate architecture increases as the complexity of a workflow grows.

This paper addresses the inherent unreliability and instability of worker nodes in large-scale donation-based distributed infrastructures such as peer-to-peer and grid systems. We present adaptive scheduling techniques that can mitigate this uncertainty and significantly outperform current approaches. In this work, we consider nodes that execute tasks via donated computational resources and may behave erratically or maliciously. We present a model in which reliability is not a binary property, but a statistical one based on a node's prior performance and behavior. We use this model to construct several reputation-based scheduling algorithms that employ estimated reliability ratings of worker nodes for efficient task allocation. Our scheduling algorithms are designed to adapt to changing system conditions, as well as nonstationary node reliability. Through simulation, we demonstrate that our algorithms can significantly improve throughput while maintaining a very high success rate of task completion. Our results suggest that reputation-based scheduling can handle a wide variety of worker populations, including nonstationary behavior, with overhead that scales well with system size. We also show that our adaptation mechanism allows the application designer fine-grain control over the desired performance metrics.

In today's batch queue HPC cluster systems, the user submits a job requesting a fixed number of processors. The system will not start the job until all of the requested resources become available simultaneously. When cluster workload is high, large sized jobs will experience long waiting time due to this policy. In this paper, we propose a new approach that dynamically decomposes a large job into smaller ones to reduce waiting time, and lets the application expand across multiple subjobs while continuously achieving progress. This approach has three benefits: (i) application turnaround time is reduced, (ii) system fragmentation is diminished, and (iii) fairness is promoted. Our approach does not depend on job queue time prediction but exploits available backfill opportunities. Simulation results have shown that our approach can reduce application mean turnaround time by up to 48%.

Many Cloud applications exploit the diversity of storage options in a data center to achieve desired cost, performance, and durability tradeoffs. It is common to see applications using a combination of memory, local disk, and archival storage tiers within a single data center to meet their needs. For example, hot data can be kept in memory using ElastiCache, and colder data in cheaper, slower storage such as S3, using Amazon as an example. For user-facing applications, a recent trend is to exploit multiple data centers for data placement to enable better latency of access from users to their data. The conventional wisdom is that co-location of computation and storage within the same data center is a key to application performance, so that applications running within a data center are often still limited to access local data. In this paper, using experiments on Amazon, Microsoft, and Google clouds, we show that this assumption is false, and that accessing data in nearby data centers may be faster than local access at different or even same points in the storage hierarchy. This can lead to not only better performance, but also reduced cost, simpler consistency policies and reconsidering data locality in multiple DCs environment. This argues for an expansion of cloud storage tiers to consider non-local storage options, and has interesting implications for the design of a distributed storage system.

In this paper, we explore the trade-offs and opportunities in porting a high-throughput Grid computing middleware to a high-performance service oriented environment. We present the limitations of the Grid computing middleware when operating in such a performance sensitive enviroment and suggest ways of overcoming these limitations. We focus on exploiting the computation and communiction heterogeneity of the Grid resources to meet the peformance requirements of services, and present several approaches of work distribution that deal with this hetergeneity. We also present a heuristic for finding the best decomposition of work and present algorithms for each of the approaches which we evaluate on a PlanetLab tesbed. The results validate the heuristic and indicate that a significant improvement in performance can be achieved by making the Grid computing middleware aware of the heterogeneity in the underlying infrastructure. The results also provide some useful insights into selecting a work distribution policy based on the dynamic status of the Grid computing environment.

This paper presents a new paradigm for parallel job scheduling called integrated scheduling or iScheduling. The iScheduler is an application-aware job scheduler as opposed to a general-purpose system scheduler. It dynamically controls resource allocation among a set of competing applications, but unlike a traditional job scheduler, it can interact directly with an application during execution to optimize resource allocation. An iScheduler may add or remove resources from a running application to improve the performance of other applications. Such fluid resource management can support both improved application and system performance. We propose a framework for building iSchedulers and evaluate the concept on several workload traces obtained both from supercomputer centers and from a set of real parallel jobs. The results indicate that iScheduling can improve both waiting time and overall turnaround time substantially for these workload classes, outperforming standard policies such as backfilling and moldable job scheduling.

This paper presents the design of a new system architecture, Virtual Service Grid (VSG), for delivering high-performance network services. The VSG is based on the concept of the virtual service which provides location, replication, and fault transparency to clients accessing remotely deployed high-end services. One of the novel features of the virtual service is the ability to self-scale in response to client demand. The VSG exploits network and service information to make adaptive dynamic replica selection, creation, and deletion decisions. We describe the VSG architecture, middleware, and replica management policies. We have deployed the VSG on a wide-area Internet testbed to evaluate its performance. The results indicate that the VSG can deliver efficient performance for a wide range of client workloads, both in terms of reduced response time and in the utilization of system resources. Copyright © 2002 John Wiley & Sons, Ltd.

Today, centralized data-centers or clouds have become the de-facto platform for data-intensive computing in the commercial, and increasingly, scientific domains. This is because clouds such as Amazon AWS and Microsoft Azure offer large amounts of monetized co-located computation and storage well suited to typical processing tasks such as batch analytics. However, many Big Data applications rely on data that is geographically distributed, and is not collocated with the centralized computational resources provided by clouds. Examples of such applications include analysis of user data such as blogs, video feeds taken from geographically separated cameras, monitoring and log analysis of server and content distribution network (CDN) logs, and scientific data collected from distributed instruments and sensors. Such applications lead to a number of challenges for efficient data analytics in today's cloud platforms. First, in many applications, data is both large and widely distributed and data upload may constitute a non-trivial portion of the execution time. Second, centralized cloud resources present a single point of failure and network partitions between the data sources and the cloud can also lead to service disruptions. Third, the cost to transport, store, and process data may be outside of the budget of the small-scale application designer or end-user. The paper present Nebula: a dispersed edge cloud infrastructure that provides both computation and data storage to address the above challenges. The use of edge resources is attractive for several reasons. First, there is an increasing amount of computing and storage resources available on the edge, as evidenced by their use in several volunteer computing, filesharing as well as content delivery (CDN) environments. This capacity is likely to increase further with the provision of powerful multi-core, multi-node desktop and home machines coupled with increasing amount of high bandwidth Internet connectivity. Second, edge resources provide locality to data and users naturally, and hence, can be exploited easily for insitu processing. Finally, if cost is an issue, then volunteer edge resources can be utilized at a relatively low cost.

In this paper, we present our vision for data-driven cloud-based mobile computing. We identify the concept of Region of interest (RoI) that reflects the profile of the user in how they access information or interact with applications. Such information enables a series of data-driven optimizations: filtering, aggregation, and speculation, that go beyond the well-researched benefit of mobile outsourcing. These optimizations can improve performance, reliability, and energy usage. A novel aspect of our approach is to exploit the unique ability of the cloud to collect and analyze large amounts of user profile data, cache shared data, and even enable sharing of computations, across different mobile users. We implement two exemplar mobile-cloud applications on an Android/Amazon Elastic Cloud Compute (EC2)-based mobile outsourcing platform, that utilize the RoI abstraction for data-driven optimizations. We presentresults driven by workload traces derived from Twitter feeds and Wikipedia document editing to illustrate the opportunities of using such optimizations.

Cloud providers offer an array of storage services that represent different points along the performance, cost, and durability spectrum. If an application desires the composite benefits of multiple storage tiers, then it must manage the complexity of different interfaces to these storage services and their diverse policies. We believe that it is possible to provide the benefits of customized tiered cloud storage to applications without compromising simplicity using a lightweight middleware. In this paper, we introduce Tiera, a middleware that enables the provision of multi-tiered cloud storage instances that are easy to specify, flexible, and enable a rich array of storage policies and desired metrics to be realized. Tiera's novelty lies in the first-class support for encapsulated tiered cloud storage, ease of programmability of data management policies, and support for runtime replacement and addition of policies and tiers. Tiera enables an application to realize a desired metric (e.g., low latency or low cost) by selecting different storage services that constitute a Tiera instance, and easily specifying a policy, using event and response pairs, to manage the life cycle of data stored in the instance. We illustrate the benefits of Tiera through a prototype implemented on the Amazon cloud. By deploying unmodified MySQL database engine and a TPC-W Web bookstore application on Tiera, we are able to improve their respective throughputs by 47% -- 125% and 46% -- 69%, over standard deployments. We further show the flexibility of Tiera in achieving different desired application metrics with minimal overhead.

Computational grids are an exciting new technology that blends high-performance computing, distributed systems and operating systems. Grids provide a coherent software infrastructure that permits the seamless integration of wide-area resources, such as computers, instruments, devices and data archives, to solve large-scale scientific and engineering problems. In grids, the wide-area network appears to be a large virtual computer. Computational grids are a powerful technology that has engendered a new paradigm for high-performance distributed computing. Teaching about grids is crucial for enabling the future grid workforce and increasing its productivity. Developing high-quality grid courseware is difficult; instructors at the University of Minnesota have provided a helpful Web site based on their experience with teaching a metacomputing class. Grid researchers and educators need to pool their talents and resources to develop effective shareable courseware. To aid this effort, university instructors are developing a repository for grid courseware.

Mentat is an object-oriented parallel processing system designed to simplify the task of writing portable parallel programs for parallel machines and workstation networks. The Mentat compiler and run-time system work together to automatically manage the communication and synchronization between objects. The run-time system marshals member function arguments, schedules objects on processors, and dynamically constructs and executes large-grain data dependence graphs. In this article we present the Mentat run-time system. We focus on three aspects—the software architecture, including the interface to the compiler and the structure and interaction of the principle components of the run-time system; the run-time overhead on a component-by-component basis for two platforms, a Sun SparcStation 2 and an Intel Paragon; and an analysis of the minimum granularity required for application programs to overcome the run-time overhead.

International audience; By exploiting idle time on volunteer machines, desktop grids provide a way to execute large sets of tasks with negligible maintenance and low cost. Although desktop grids are attractive for their scalability and low cost, relying on external resources may compromise the correctness of application execution due to the well-known unreliability of nodes. In this paper, we consider a very challenging threat model: correlated errors caused either by organized groups of cheaters that may collude to produce incorrect results, or by buggy or so-called "unofficial" clients. By using a previously described on-line algorithm for detecting collusion and characterizing the participant behaviors, we propose a scheduling and result certification algorithm that tackles collusion. Using several real-life traces, we show that our approach minimizes both replication overhead and the number of incorrectly certified results.

International audience; By exploiting idle time on volunteer machines, desktop grids provide a way to execute large sets of tasks with negligible maintenance and low cost. Although desktop grids are attractive for cost-conscious projects, relying on external resources may compromise the correctness of application execution due to the wellknown unreliability of nodes. In this paper, we consider the most challenging threat model: organized groups of cheaters that may collude to produce incorrect results. We propose two on-line algorithms for detecting collusion and characterizing the participant behaviors. Using several real-life traces, we show that our approach is accurate and efficient in identifying collusion and in estimating group behavior.

It is generally accepted that the ability to develop large-scale distributed applications that are extensible and independent of infrastructure details has lagged seriously behind other developments in cyberinfrastructure. As the sophistication and scale of distributed infrastructure increases, the complexity of successfully developing and deploying distributed applications increases both quantitatively and in qualitatively newer ways. In this paper we trace the evolution of a representative set of “state-of-the-art” distributed applications and production infrastructure; in doing so we aim to provide insight into the evolving sophistication of distributed applications — from simple generalizations of legacy static high-performance to applications composed of multiple loosely-coupled and dynamic components. The ultimate aim of this work is to highlight that even accounting for the fact that developing applications for distributed infrastructure is a difficult undertaking, there are suspiciously few novel and interesting distributed applications that utilize production Grid infrastructure. Along the way, we aim to provide an appreciation for the fact that developing distributed applications and the theory and practise of production Grid infrastructure have often not progressed in phase. Progress in the next phase and generation of distributed applications will require stronger coupling between the design and implementation of production infrastructure and the theory of distributed applications, including but not limited to explicit support for distributed application usage modes and advances that enable distributed applications to scale-out.

Efficiently executing large-scale, data-intensive workflows such as Montage must take into account the volume and pattern of communication. When orchestrating data-centric workflows, centralised servers common to standard workflow systems can become a bottleneck to performance. However, standards-based workflow systems that rely on centralisation, e.g., Web service based frameworks, have many other benefits such as a wide user base and sustained support.This paper presents and evaluates a light-weight hybrid architecture which maintains the robustness and simplicity of centralised orchestration, but facilitates choreography by allowing services to exchange data directly with one another. Furthermore our architecture is standards compliment, flexible and is a non-disruptive solution; service definitions do not have to be altered prior to enactment. Our architecture could be realised within any existing workflow framework, in this paper, we focus on a Web service based framework.Taking inspiration from Montage, a number of common workflow patterns (sequence, fan-in and fan-out), input-output data size relationships and network configurations are identified and evaluated. The performance analysis concludes that a substantial reduction in communication overhead results in a 2-4 fold performance benefit across all patterns. An end-to-end pattern through the Montage workflow results in an 8 fold performance benefit and demonstrates how the advantage of using our hybrid architecture increases as the complexity of a workflow grows.

An open distributed system provides a best-effort guarantee on the quality of service provided to applications. This has worked well for throughput-based applications of the kind typically executed in Condor or BOINCstyle environments. For other applications, the absence of timeliness of correctness guarantees limit the utility or appeal of this environment. Computational results that are too late or erroneous are not usable to the application. We present techniques designed to efficiently promote usable computing in open distributed systems.

When orchestrating data-centric workflows as are commonly found in the sciences, centralised servers can become a bottleneck to the performance of a workflow; output from service invocations are normally transferred via a centralised orchestration engine, when they should be passed directly to where they are needed at the next service in the workflow. To address this performance bottleneck, this paper presents a lightweight hybrid workflow architecture and concrete API, based on a centralised control flow, distributed data flow model. Our architecture maintains the robustness and simplicity of centralised orchestration, but facilitates choreography by allowing services to exchange data directly with one another, reducing data that needs to be transferred through a centralised server. Furthermore our architecture is standards compliment, flexible and is a non-disruptive solution; service definitions do not have to be altered prior to enactment.

We describe our project that marries data mining together with Grid computing. Specifically, we focus on one data mining application - the Minnesota Intrusion Detection System (MINDS), which uses a suite of data mining based algorithms to address different aspects of cyber security including malicious activities such as denial-of-service (DoS) traffic, worms, policy violations and inside abuse. MINDS has shown great operational success in detecting network intrusions in several real deployments. In sophisticated distributed cyber attacks using a multitude of wide-area nodes, combining the results of several MINDS instances can enable additional early-alert cyber security. We also describe a Grid service system that can deploy and manage multiple MINDS instances across a wide-area network.

Large-scale donation-based distributed infrastructures need to cope with the inherent unreliability of participant nodes. A widely-used work scheduling technique in such environments is to redundantly schedule the outsourced computations to a number of nodes. We present the design and implementation of RIDGE, a reliability-aware system which uses a node's prior performance and behavior to make more effective scheduling decisions. We have implemented RIDGE on top of the BOINC distributed computing infrastructure and have evaluated its performance on a live PlanetLab testbed. Our experimental results show that RIDGE is able to match or surpass the throughput of the best BOINC configuration by automatically adapting to the characteristics of the underlying environment. In addition, RIDGE is able to provide much lower workunit makespans compared to BOINC. RIDGE is also able to produce significantly lower communication makespans for downloading clients. Collectively, the results suggest that RIDGE has great promise for service-oriented environments with time constraints.

This project addresses the problem of dynamic service adaptation in the grid. The need for adaptation arises due to both resource and service demand uncertainty. With NGS support, we are tackling several key issues in this problem space: (1) a dynamic OGSA-based grid service architecture to support dynamic service hosting - where to host and re-host a service within the grid in response to service demand and resource fluctuation (2005), (2) resource management middleware that dynamically decides how many resources to allocate to a request and where a request should run [B. Lee et al., (2003)][J. B. Weissman et al., (2003)], (3) a dynamic leasing framework that decides how many resources to lease to a service to handle future requests [D. England et al., (2004)][D. England et al., (2005)], (4) and a new model of service robustness that can describe the sensitivity of a service to grid fluctuations[D. England et al., (2005)]. In this short report, we present material on (1) and (3) in the interest of brevity. Papers describing all aspects of the project are contained in the bibliography.

We introduce a formal model for deployment and hosting of a dynamic grid service wherein the service provider must pay a resource provider for the use of computational resources. Our model produces policies that balance the number of required resources with the desire to keep the cost of hosting the service to a minimum. The two components of cost that we consider are the deployment cost and the cost to keep the service active, which we view as a lease. We cast the problem in a dynamic programming framework and we are able to show that the model makes good leasing decisions in the face of such uncertainties as random demand for the service and random execution times of service requests. The results show that the policies obtained from the model reduce the cost of hosting a service and significantly reduce the variance of that cost.

This paper presents an architecture and implementation for a dynamic OGSA-based grid service architecture that extends GT3 to support dynamic service hosting - where to host and re-host a service within the grid in response to service demand and resource fluctuation. Our model goes beyond current OGSI implementations in which the service is presumed to be pre-installed at all sites (and only service instantiation is dynamic). In dynamic virtual organizations (VOs), we believe dynamic service hosting provides an important flexibility. Our model also defines several new adaptive grid service classes that support adaptation at multiple levels. Dynamic service deployment allows new services to be added or replaced without taking down a site for reconfiguration and allows a VO to respond effectively to dynamic resource availability and demand. The preliminary results suggest that the cost of dynamic installation, deployment, and invocation, is tolerable.

Reputation systems have been a hot topic in the peer-to-peer community for several years. In a services-oriented distributed computing environment like the grid, reputation systems can be utilized by clients to select between competing service providers. In this paper, we selected several existing reputation algorithms and adapted them to the problem of service selection in a grid-like environment. We performed a quantitative comparison of both the accuracy and overhead associated with these techniques under common scenarios. The results indicate that using a reputation system to guide service selection can significantly improve client satisfaction with minimal overhead. In addition, we show that the most appropriate algorithm depends on the kinds of anticipated attacks. A new algorithm we've proposed appears to be the approach of choice if clients can misreport service ratings.

Advances in packaging and interface technologies have made it possible for software components to be shared across the network through encapsulation and offered as network services. They allow end-users to focus on their applications and obtain remote services when needed simply by invoking them across the network. Many groups have built significant infrastructures for providing domain-specific high performance services. However, transforming high performance applications into network services is labor intensive and time consuming because there is little existing infrastructure to utilize. In this paper, we propose a software toolkit and runtime infrastructure for rapid deployment of network services.

Partitioning data parallel computations across a network of heterogeneous workstations is a difficult problem for the user. We have developed a runtime partitioning method for choosing the number and type of processors to apply to a data parallel computation, and a decomposition of the data domain in order to achieve reduced completion time. The partitioning method utilizes information about the problem in the form of callback functions and uses a set of topology-specific communication functions to estimate communication costs. We show that the method makes effective partitioning decisions and has runtime overhead that is easily tolerated. In particular we show that for two implementations of a canonical stencil application, minimum elapsed times are obtained for a range of problem sizes on a network of heterogeneous workstations. >

Metacomputing is the seamless application of geographically separated distributed computing resources to user applications. We consider the scheduling of meta applications; applications consisting of multiple components that may communicate and interact over the course of the application. Components may be schedulable computations, remote servers or databases, remote instruments, humans in the loop, etc. We divide applications into three categories-concurrent, parallel, and pipeline. Concurrent is the classic meta application in which a set of components each running in a single site are executing concurrently and exchanging data. Parallel is a special case of concurrent in which a component is replicated and distributed across multiple sites. Pipeline applications consist of components connected in a chain like fashion.

This paper describes the current state of Prophet-a system that provides run-time scheduling support for parallel applications in heterogeneous workstation networks. Prior work on Prophet demonstrated that scheduling SPMD applications could be effectively automated with excellent performance. Enhancements have been made to Prophet to broaden its use to other application types including parallel pipelines, and to make more effective use of dynamic system state information to further improve performance. The results indicate that both SPMD and parallel pipeline applications can be scheduled to produce reduced completion time by exploiting the application structure and run-time information.

With the explosive expansion of the Internet as the global information infrastructure, coupled with rapid advances in wireless technologies, a new vision for computing is emerging. A seamless, anywhere, anytime, untethered, pervasive computing paradigm is touted as the next major paradigm shift in the way we compute and communicate. In order to realize this vision, the complexity of this environment must be masked from its consumers (users) and producers alike (service providers). We believe that a middleware infrastructure that provides appropriate high-level abstractions for both users and service-providers is needed. Such a middleware infrastructure must allow powerful services to be easily deployed and permit users to interact with such services in a personalized, yet effortless manner. In this position paper, we introduce the notion of a "smart service" - services with which users can dynamically interact in an automated and personalized manner, and can be composed to form "smart environments." Smart environments present an integrated view of services to the user to enable much higher-level, domain-specific interactions and to support user customization. We present a system architecture for building smart services and smart environments, and the middleware research challenges that must be addressed to realize this architecture and the core abstractions.

The conventional wisdom in the scientific computing community is that the best way to solve large-scale numerically intensive scientific problems on today's parallel MIMD computers is to use Fortran or C programmed in a data-parallel style using low-level message-passing primitives. This approach inevitably leads to nonportable codes and extensive development time, and restricts parallel programming to the domain of the expert programmer. We believe that these problems are not inherent to parallel computing but are the result of the programming tools used. We will show that comparable performance can be achieved with little effort if better tools that present higher level abstractions are used. The vehicle for our demonstration is a 2D electromagnetic finite element scattering code we have implemented in Mentat, an object-oriented parallel processing system. We briefly describe the application. Mentat, the implementation, and present performance results for both a Mentat and a hand-coded parallel Fortran version.