Coherency Hub Design for Multi-socket Sun Servers with CoolThreads (TM) Technology

This paper describes the micro-architecture of a Coherency Hub (CoHub) ASIC for a 4-socket highly-threaded multiprocessor using Sun's UltraSPARC ¯ T2 Plus processor. UltraSPARC T2 Plus is an 8-core CMT processor in the Sun Servers with CoolThreadsTM Technology family. CoHub enables cost-effective scaling to 4 nodes with a total thread count of 256 and near-linear performance scaling on transaction processing workloads. Extending a 2-node "glueless" system to a 4-node system without processor changes was a key requirement. CoHub broadcasts snoop requests, serializes requests to the same address, and consolidates snoop responses. It communicates with nodes via serial links, using a proprietary link layer implemented over FBDIMM. We present the coherency scheme, ASIC design, transaction flows, and engineering challenges created by 800 MHz operation and 6-stage pipeline budget. We report performance scalability results measured on commercial server benchmarks.