The DAQ architecture must accommodate operation up to 1 kHz readout rates. Some aspects of DAQ will support this initially since this choice affects the design of the front-end systems and the data collection network. Other aspects of DAQ are scalable and upgradable. This is especially true of the event builder and level 3 trigger nodes.
Front-end issues
To minimize dead-time from the DAQ system, the front-end boards will need
to provide a small amount of event buffering. The readout of these
events must be able to occur in parallel with either recording data between
triggers or with digitizing data from a later trigger. Only the conversion
time of 10
s drives the dead-time. Data transfer to the
crate controller can take 1 ms on average and occurs asynchronously.
Local buffers can overflow if they are not emptied fast enough by the crate controller. A trigger throttling signal needs to be generated from each crate to indicate that the board-level buffers are approaching overflow. This feeds into the event flow control network.
If data conversion time is longer than a level 2 trigger decision time, the
front-end board conversion must be capable of being aborted. Similarly,
a mechanism for flushing level 2 rejected events from local buffers must be
available. It is desirable
that data conversion and event buffering all occur in less than 10 s
to give a dead-time contribution of 1%at 1 kHz. Conversion times as
long as 50
s would not be disastrous.
Crate level issues The packaging of the front-end electronics will probably not be homogeneous. The trigger system and other CLEO-specific electronics will probably be implemented in VME or VXI. At the moment, the most cost effective commercial electronics for time and charge measurements are packaged in FASTBUS.
In either case, a processor will be resident in each crate. Note that most commercial front-end boards are self-sparsifying and that the crate processor will be relieved of this chore. Primarily, it will read an event's worth of data from the front-end boards, assemble them into an event fragment tagged with an event number, buffer it, and transfer it to the event builder/level 3 complex. Data is pulled from the front-end boards by the controller, which works from a list of outstanding events to process. Data is pushed from the crate controller into the data collection network as long as there is buffer space on the receiving end. Our present thinking is that the current overhead for pulling the data from the crates at the request of the event building is too high. This view may change.
As with the front-end boards, there is a finite amount of event buffer space available in the crate controller. A throttling signal needs to be generated from each crate to indicate that the buffer is approaching overflow. This feeds into the event flow control network.
For VME, there is an abundance of commercial processors and
network interfaces. For FASTBUS, the choices are few. The most
modern and flexible FASTBUS based processor is the MIPS-based FNAL FRC.
In both cases, a uni-directional, high-speed data transfer port is required
for the event data and a local area network (LAN) connection is
required for configuration
and control. Whether event flow control can be incorporated into the
LAN is up for discussion. Additionally, VxWorks will become the
real-time operating system of choice since it is required to support
the FRC and more processors and I/O devices are supported under it
than under our present pSOS.
We observe that VME does not provide a mechanism for sparse scan readout, i.e., slave boards cannot signal that all data has been transferred. Of course, this is not a problem in FASTBUS. The impact on the effective back-plane bandwidth has to be carefully understood.
Bandwidth
The present CLEO II readout event size is about 5 kByte. The size of events
for
an upgraded detector will depend on the kind of multi-hit electronics used, the
front-end solutions adopted for the silicon and particle-ID components, and the
overall detector occupancy. A conservative estimate is 25 kByte per event,
spread over tens of front-end crates. The aggregate bandwidth at 1 kHz readout
rate is about 25 MByte/s. The multiplicity of crates means that the required
bandwidths for the crate back-plane and data link from the crate to the event
builder/level 3 complex is modest. The impact of the full bandwidth is on the
design of the event builder. Note that this bandwidth is estimated to be
needed at luminosities approaching
and need not be supported in an event builder at
.
Data collection network A variety of data collection schemes have been considered: extensions to the CLEO II system, packet and channel oriented switches, and central shared memory. For reasons of cost, timeliness, and implementability, the shared memory approach is preferred. It is believed that the CLEO II system can not scale much beyond 100 Hz. ATM and Fibre Channel switch-based systems are not yet well enough established for us to commit to them. Ring-oriented SCI is also early in its life cycle. The useful bandwidth of FDDI may not be adequate. They, along with HIPPI, are also relatively expensive, especially when implemented in each crate. A further restriction is that the use of FASTBUS greatly limits the commercial interfaces available. In-house engineering of ATM or other complex network interfaces is beyond the collaboration's capabilities.
An alternative is to implement a point-to-point link from each crate to a shared memory system. Each point-to-point link terminates in a individual dual-port memory as in CLEO II. The link technology could be a version of the present CLEO II data link, LeCroy ECL-line, or a TAXI-like serial link on copper or fiber at a few hundred megabits per second. The FNAL FRC allows personalization of its data output port. The existence of what is already available for this port or what can be easily implemented will determine the link structure of CLEO III.
The centralized memory system is the terminus of that part of the DAQ system which must support 1 kHz operation with 25 kByte events at the time of commissioning. The rest of the system can be scaled with luminosity and event rate.
Unlike the problem of off-line computing, DAQ on-line computing is primarily an I/O problem requiring modest computing performance-1 Mips/event-but capable of absorbing data at 25 MByte/s and generating 6 to 12 MByte/s of output. Note that these are effective rates which will require at least a factor of two greater peak rates. A distributed level 3 system comprised of workstations requires the development of a full bandwidth event builder, an event distribution scheme, and interface to the workstation. The merging of the output data streams over conventional LAN technology is problematic even with CLEO II's modest output data rates. A single box solution providing good I/O and CPU performance is appropriate here. Multi-processor computers with 1000 Mips and easy-to-interface I/O are available today. The merging of output data is easy since there is no data movement from the box over a network.
We observe that powerful computers such as the SGI
Power Challenge series and the DEC AXP processors with FUTUREBUS+ will
allow the attachment of multiple VME crates. A single node terminus
for the links can serve as event builder and as level 3 processor.
Essentially, we merge the present DAQ90 hardware with the
level 3 workstation.
Initially, processor CPU cycles can move data from the dual-port
memories. As data bandwidths exceed 10 MByte/s, it may become
necessary to add VME-resident data movement engines to DMA the
data into the host memory. A possible configuration of a memory-based
system is shown in Fig. 
along with a switch-based
system.
The cost of the above processor is not included as part of the CLEO III upgrade. The interfaces to the system are included.
This approach limits the custom hardware requirements to the link hardware, possibly the dual-port memories, and the host software. We leverage the computer vendors' ability to interface to VME and to provide the CPU performance in an easily maintainable and programmable environment.