Error handling for the CDF online silicon vertex tracker

The online silicon vertex tracker (SVT) is composed of 104 VME 9U digital boards (of eight different types). Since the data output from the SVT (few MB/s) are a small fraction of the input data (200 MB/s), it is extremely difficult to track possible internal errors by using only the output stream. For this reason, several diagnostic tools have been implemented: local error registers, error bits propagated through the data streams, and the Spy Buffer system. Data flowing through each input and output stream of every board are continuously copied to memory banks named Spy Buffers, which act as built-in logic state analyzers hooked continuously to internal data streams. The contents of all buffers can be frozen at any time (e.g., on error detection) to take a snapshot of all data flowing through each SVT board. The Spy Buffers are coordinated at system level by the Spy Control Board. The architecture, design, and implementation of this system are described.