Search

This paper shares our experience in supporting and running the Open vSwitch (OVS) software switch, as part of the NSX product for enterprise data center virtualization used by thousands of VMware customers. Starting in 2009, the OVS design split its code between tightly coupled kernel and userspace components. This split was necessary at the time for performance, but it caused maintainability problems that persist today. In addition, in-kernel packet processing is now much slower than newer options. To solve the problems caused by the user/kernel split, OVS must adopt a new architecture. We describe two possibilities that we explored, but did not adopt, one because it gives up compatibility with drivers and tools that are important to virtual data center operators, the other because it performs poorly. Instead, we endorse a third approach, based on a new Linux socket type called AF\_XDP, which solves the maintainability problem in a compatible, performant way. The new code is already merged into the mainstream OVS repository. We include a thorough performance evaluation and a collection of lessons learned.

VMware NSX virtualizes network functionality in a manner anal- ogous to how hypervisors virtualize compute resources. To do this, NSX must faithfully recreate virtual versions of network compo- nents, such as switches, routers, and firewalls. As this functionality becomes commoditized, NSX must move "up the stack" to provide more advanced features, such as load-balancers, IDS/IPS (intrusion detection and prevention systems), and DPI (deep packet inspec- tion) for classification. NSX is designed to work in all types of deployments-even those without any other VMware software. It integrates with ESXi, Linux KVM, and Hyper-V hypervisors; it is even being made to work on systems without a hypervisor, such as containers and third- party clouds. Each of these platforms has its own native forwarding plane. For the best user experience, all of the forwarding planes should provide the same behavior, but the disparate implemen- tations make this difficult in practice. As network functions be- come more complex and as NSX supports more forwarding planes, both duplication of effort and undesirable diversity of behavior in- creases. We propose a new approach to building advanced network func- tions in NSX. Under this approach, identical code runs on all of NSX's supported platforms. Applications will run at or near native performance, but with better security and identical cross-platform behavior. We demonstrate this by writing a single application to provide DPI functionality that runs in the fast paths of each of NSX's primary platforms: ESXi, Linux, and Edge gateway appli- ance. We evaluate the performance and correctness of our imple- mentation on the three platforms.

Hypervisors use software switches to steer packets to and from virtual machines (VMs). These switches frequently need upgrading and customization—to support new protocol headers or encapsulations for tunneling and overlays, to improve measurement and debugging features, and even to add middlebox-like functions. Software switches are typically based on a large body of code, including kernel code, and changing the switch is a formidable undertaking requiring domain mastery of network protocol design and developing, testing, and maintaining a large, complex codebase. Changing how a software switch forwards packets should not require intimate knowledge of its implementation. Instead, it should be possible to specify how packets are processed and forwarded in a high-level domain-specific language (DSL) such as P4, and compiled to run on a software switch. We present PISCES, a software switch derived from Open vSwitch (OVS), a hard-wired hypervisor switch, whose behavior is customized using P4. PISCES is not hard-wired to specific protocols; this independence makes it easy to add new features. We also show how the compiler can analyze the high-level specification to optimize forwarding performance. Our evaluation shows that PISCES performs comparably to OVS and that PISCES programs are about 40 times shorter than equivalent changes to OVS source code.

As anyone who has operated a large network can attest, enterprise networks are difficult to manage. That they have remained so despite significant commercial and academic efforts suggests the need for a different network management paradigm. Here we turn to operating systems as an instructive example in taming management complexity. In the early days of computing, programs were written in machine languages that had no common abstractions for the underlying physical resources. This made programs hard to write, port, reason about, and debug. Modern operating systems facilitate program development by providing controlled access to high-level abstractions for resources (e.g., memory, storage, communication) and information (e.g., files, directories). These abstractions enable programs to carry out complicated tasks safely and efficiently on a wide variety of computing hardware. In contrast, networks are managed through low-level configuration of individual components. Moreover, these configurations often depend on the underlying network; for example, blocking a user’s access with an ACL entry requires knowing the user’s current IP address. More complicated tasks require more extensive network knowledge; forcing guest users’ port 80 traffic to traverse an HTTP proxy requires knowing the current network topology and the location of each guest. In this way, an enterprise network resembles a computer without an operating system, with network-dependent component configuration playing the role of hardware-dependent machine-language programming. What we clearly need is an “operating system” for networks, one that provides a uniform and centralized programmatic interface to the entire network. Analogous to the read and write access to various resources provided by computer operating systems, a network operating system provides the ability to observe and control a network. A network operating system does not manage the network itself; it merely provides a programmatic interface. Applications implemented on top of the network operating system perform the actual management tasks. The programmatic interface should be general enough to support a broad spectrum of network management applications. Such a network operating system represents two major conceptual departures from the status quo. First, the network operating system presents programs with a centralized programming model; programs are written as if the entire network were present on a single machine (i.e., one would use Dijkstra to compute shortest paths, not Bellman-Ford). This requires (as in [3, 8, 14] and elsewhere) centralizing network state. Second, programs are written in terms of high-level abstractions (e.g., user and host names), not low-level configuration parameters (e.g., IP and MAC addresses). This allows management directives to be enforced independent of the underlying network topology, but it requires that the network operating system carefully maintain the bindings (i.e., mappings) between these abstractions and the low-level configurations. Thus, a network operating system allows management applications to be written as centralized programs over highlevel names as opposed to the distributed algorithms over low-level addresses we are forced to use today. While clearly a desirable goal, achieving this transformation from distributed algorithms to centralized programming presents significant technical challenges, and the question we pose here is: Can one build a network operating system at significant scale?

Pintos is an instructional operating system, complete with documentation and ready-made, modular projects that introduce students to the principles of multi-programming, scheduling, virtual memory, and filesystems. By allowing students to run their work product on actual hardware, while simultaneously benefiting from debugging and dynamic analysis tools provided in simulated and emulated environments, Pintos increases student engagement. Unlike tailored versions of commercial or open source OS such as Linux, Pintos is designed from the ground up from an educational perspective. It has been used by multiple institutions for a number of years and is available for wider use.

As sensitive data lifetime (i.e. propagation and duration in memory) increases, so does the risk of exposure. Unfortunately, this issue has been largely overlooked in the design of most of today's operating systems, libraries, languages, etc. As a result, applications are likely to leave the sensitive data they handle (passwords, financial and military information, etc.) scattered widely over memory, leaked to disk, etc. and left there for an indeterminate period of time. This greatly increases the impact of a system compromise.Dealing with data lifetime issues is currently left to application developers, who largely overlook them. Security-aware developers who attempt to address them (e.g. cryptographic library writers) are stymied by the limitations of the operating systems, languages, etc. they rely on. We argue that data lifetime is a systems issue which must be recognized and addressed at all layers of the software stack.

We present a flexible architecture for trusted computing, called Terra, that allows applications with a wide range of security requirements to run simultaneously on commodity hardware. Applications on Terra enjoy the semantics of running on a separate, dedicated, tamper-resistant hardware platform, while retaining the ability to run side-by-side with normal applications on a general-purpose computing platform. Terra achieves this synthesis by use of a trusted virtual machine monitor (TVMM) that partitions a tamper-resistant hardware platform into multiple, isolated virtual machines (VM), providing the appearance of multiple boxes on a single, general-purpose platform. To each VM, the TVMM provides the semantics of either an "open box," i.e. a general-purpose hardware platform like today's PCs and workstations, or a "closed box," an opaque special-purpose platform that protects the privacy and integrity of its contents like today's game consoles and cellular phones. The software stack in each VM can be tailored from the hardware interface up to meet the security requirements of its application(s). The hardware and TVMM can act as a trusted party to allow closed-box VMs to cryptographically identify the software they run, i.e. what is in the box, to remote parties. We explore the strengths and limitations of this architecture by describing our prototype implementation and several applications that we developed for it.

This paper shows how to quickly move the state of a running computer across a network, including the state in its disks, memory, CPU registers, and I/O devices. We call this state a capsule . Capsule state is hardware state, so it includes the entire operating system as well as applications and running processes.We have chosen to move x 86 computer states because x 86 computers are common, cheap, run the software we use, and have tools for migration. Unfortunately, x 86 capsules can be large, containing hundreds of megabytes of memory and gigabytes of disk data. We have developed techniques to reduce the amount of data sent over the network: copy-on-write disks track just the updates to capsule disks, "ballooning" zeros unused memory, demand paging fetches only needed blocks, and hashing avoids sending blocks that already exist at the remote end. We demonstrate these optimizations in a prototype system that uses VMware GSX Server virtual, machine monitor to create and run x 86 capsules. The system targets networks as slow as 384 kbps.Our experimental results suggest that efficient capsule migration can improve user mobility and system management. Software updates or installations on a set of machines can be accomplished simply by distributing a capsule with the new changes. Assuming the presence of a prior capsule, the amount of traffic incurred is commensurate with the size of the update or installation package itself. Capsule migration makes it possible for machines to start running an application within 20 minutes on a 384 kbps link, without having to first install the application or even the underlying operating system. Furthermore, users' capsules can be migrated during a commute between home and work in even less time.

In this paper, we describe a collaborative system specifically designed to address problems faced by distributed (or virtual) teams. TeamSCOPE (Team Software for a Collaborative Project Environment) is a web-based work environment that has emerged from a research project studying the communication needs of internationally distributed engineering design teams. The paper begins by outlining some of the needs of virtual teams. An integrative framework that focuses on facilitation of group members' awareness of group activities, communications and resources is proposed. These needs and awareness requirements are then translated into a set of collaborative system design goals which have guided the implementation of TeamSCOPE. The features of TeamSCOPE are briefly reviewed, and some preliminary observations from early users are provided. We conclude by noting some of the new features planned for TeamSCOPE based on our early trials.