Search

IBM has designed special cryptographic processors for its servers for more than 25 years. These began as very simple devices, but over time the requirements have become increasingly complex, and there has been a never-ending demand for increased speed. This paper describes the PCIXCC, the new coprocessor introduced in the IBM z990 server. In many ways, PCIXCC is a watershed design. For the first time, a single product satisfies all requirements across all IBM server platforms. It offers the performance demanded by today's Web servers, it supports the complex and specialized cryptographic functions needed in the banking and finance industry, and # uses packaging technology that leads the world in resistance to physical or electrical attacks against its secure processes and the secret data it holds. Furthermore, it is programmable and highly flexible, so that its function can be easily modified to meet new requirements as they appear. These features are possible because of innovative design in both the hardware and embedded software for the card. This paper provides an overview of that design.

The IBM eServer[TM] zSeries[R] Model z990 offers customers significant new opportunity for server growth while preserving and enhancing server availability. The z990 provides vertical growth capability by introducing the concurrent addition of processor/memory books and horizontal growth in channels by the use of extended virtualization technology. In order to continue to support the zSeries legacy for high availability and continuous reliable operation, the z990 delivers significant new features for reliability, availability, and serviceability (RAS). This paper describes these new capabilities, in each case presenting the value of the feature, both in terms of enhancing the self-management capability of the server and its availability.

This paper describes a new kind of initial program loading (IPL) for IBM zSeries[R] servers. The new IPL protocol expands the set of input/output devices that can be used during IPL to include Small Computer System Interface (SCSI) Fibre Channel protocol (FCP) disk devices (SCSI disks). We begin by describing several new challenges resulting from the use of SCSI disks during IPL, followed by a brief overview of new concepts we have applied to the IPL process to overcome these challenges. We continue with a step-by-step description of the processes executed during SCSI IPL, the tools used, the disk format, the parameters required, and related topics. Since SCSI IPL is supported for virtual machines instantiated by the z/VM[R] operating system, some unique features of this capability are described. Finally, we describe a variation of SCSI IPL that enables the contents of memory to be dumped onto a SCSI disk.

The performance of large servers is to a high degree determined by their I/O subsystems. In the z990 server, nearly all of the components in the I/O path have been considerably improved in performance, capability, and cost. A 2-GB/s enhanced self-timed interface (eSTI) was introduced which is capable of absorbing the ever-increasing data rates of modem high-speed adapters. The I/O bandwidth available from a single node (three memory bus adapter, or MBA, chips, each with four eSTI ports) now equals 48 GB/s. As a consequence, both the MBA chip and the STI multiplexer switch (STI switch) chip had to be completely redesigned. In addition to these two chips, this paper describes the eSTI design itself and the Sweep chip, which integrates the function of four bidirectional adapter chips, one switch chip, and a clock chip.

The IBM eServer[TM] z990 introduces a new mode for cooling multichip processor modules that enables significantly more processors to be refrigerant-cooled than previously. In recent IBM zSeries[R] offerings, including G4, G5, G6, and z900, chip junctions in a single multichip module (MCM) located in a central electronic complex (CEC) frame were cooled for reliability and performance benefits, using refrigerant technology, to temperatures lower than those achievable with air cooling. In the z990 system, a hybrid cooling approach is used, allowing refrigeration to be extended to four MCMs in a single CEC, which makes possible denser systems and greater power efficiency compared with prior modular refrigeration technologies used. In the event of a malfunction of the primary refrigeration cooling system, a backup air-cooling system is automatically engaged until the refrigeration problem is fixed. System sensors monitor the cooling state at all times. When air cooling is required, the chip circuit temperatures increase and the logic clocks are optimally adjusted to match the new junction temperatures. When refrigeration cooling is restored, the clocks are adjusted back to their fast speed. This technique allows the z990 system to match the processor density of direct-air-cooled systems while retaining a system performance and reliability benefit from refrigeration.

In this paper, we describe the challenging first- and second-level packaging technology of a new system packaging architecture for the IBM eServer[TM] z990. The z990 dramatically increases the volumetric processor density over that of the predecessor z900 by implementing a super-blade design comprising four node cards. Each blade is plugged into a common center board, and a blade contains the node with up to sixteen processor cores on the multichip module (MCM), up to 64 GB of memory on two memory cards, and up to twelve self-timed interface (STI) cables plugged into the front of the node. Each glass-ceramic MCM carries 16 chips dissipating a maximum power of 800 W. In this super-blade design, the packaging complexity is increased dramatically over that of the previous zSeries[R] eServer z900 to achieve increased volumetric density, processor performance, and system scalability. This approach permits the system to be scaled from one to four nodes, with full interaction between all nodes using a ring structure for the wiring between the four nodes. The processor frequencies are increased to 1.2 GHz, with a 0.6-GHz nest with synchronous double-data-rate interchip and interblade communication. This data rate over these package connections demands an electrical verification methodology that includes all of the different relevant system components to ensure that the proper signal and power distribution operation is achieved. The signal integrity analysis verifies that crosstalk limits are not exceeded and proper timing relationships are maintained. The power integrity simulations are performed to optimize the hierarchical decoupling in order to maintain the voltage on the power distribution networks within prescribed limits.

The floating-point unit (FPU) of the IBM z990 eServer[TM] is the first one in an IBM mainframe with a fused multiply-add dataflow. It also represents the first time that an SRT divide algorithm (named after Sweeney, Robertson, and Tocher, who independently proposed the algorithm) was used in an IBM mainframe. The FPU supports dual architectures: the zSeries[R] hexadecimal floating-point architecture and the IEEE 754 binary floating-point architecture. Six floating-point formats--including short, long, and extended operands--are supported in hardware. The throughput of this FPU is one multiply-add operation per cycle. The instructions are executed in five pipeline steps, and there are multiple provisions to avoid stalls in case of data dependencies. It is able to handle denormalized input operands and denormalized results without a stall (except for architectural program exceptions). It has a new extended-precision divide and square-root dataflow. This dataflow uses a radix-4 SRT algorithm (radix-2 for square root) and is able to handle divides and square-root operations in multiple floating-point and fixed-point formats. For fixed-point divisions, a new mechanism improves the performance by using an algorithm with which the number of divide iterations depends on the effective number of quotient bits.

The z990 eServer[TM] central electronic complex (CEC) houses four multichip-module-based processor units instead of one, as in the previous-generation z900 eServer. The multichip module (MCM) input/output pin density in z990 processor units is more than twice that of the MCMs in z900 processor units. This increase in packaging density and the consequent tripling of the current drawn by the processor units were accommodated by the first-time use of land grid array (LGA) MCM-to-board interconnections in an IBM zSeries[R] eServer. This was done by using innovative refrigeration cooling of the MCM with air cooling as backup, and by a new mechanical packaging and power distribution scheme. This paper describes the mechanical engineering of the CEC cage, the LGA MCM-to-board interconnection scheme, and the mechanical isolation of the MCM evaporator-heat-sink mass from the LGA contacts. The paper also describes the electrical power and the cooling solutions implemented to meet the more demanding requirements of the denser CEC package.

A high-end eServer[TM] consists of multiple microprocessor chips packaged with additional chips on a multichip module. In conjunction with memory and various I/O cards, this module is mounted on a card called a processor book, and a few of those cards on a board finally represent a major part of the system. Before the first hardware is built, simulations must be performed to verify that all of these components work together. But before we can build the simulation models, we need to find answers to many questions and to specify constraints, such as the scope of the simulation, the representation of the packaging data, the handling of cross-hierarchical connections such as cables, and the handling of passive components such as resistors and capacitors. This system model build should be as flexible as possible. System verification must be done for different system configurations (both single-processor and multiprocessor systems, one-processor-book systems, and multiprocessor-book systems) with or without I/O. Therefore, not only should a configurable model build downsize the model structure, but it should provide the capability to add logic. The requirement to include special logic, such as clock macros or checker logic, is driven by the use of emulation and acceleration technology and by other speed-related elements. This paper discusses these new concepts in eServer development: a configurable simulation model build, the automatic derivation of structural model data from packaging design, and the addition of specific logic without affecting the model structure generated by the previous step.

The IBM eServer[TM] z990 microprocessor implements many features designed to give excellent performance on both newer and traditional mainframe applications. These features include a new superscalar instruction execution pipeline, high-bandwidth caches, a huge secondary translation-lookaside buffer (TLB), and an onboard cryptographic coprocessor. The microprocessor maintains zSeries[R] leadership in RAS (reliability, availability, serviceability) capabilities that include state-of-the-art error detection and recovery.