Cache-coherency

CC — NUMA Cache Coherent Non-Uniform Memory Access-machines... 

In Figure 2.15, a single HCA is connected, which provides connectivity to the other nodes on the network. The node shown has a shared memory architecture in which all the sixteen processors have direct access to all the memory. In a shared memory architecture, data from the same memory can be read by several processors and, hence, may be stored in multiple caches if one of the processors then changes its data, the system may have two inconsistent copies of this data, resident in different cache entries. This situation is dealt with by cache-coherency support in hardware, which will invalidate data in other caches whenever the data in the local caches change. Such invalidation generates additional traffic on the network connecting the processors within the node, and this can impact the performance of the computer, especially when the number of processors in the node is large. 

Shared memory computers in which all processors have equal access to all memory in the system are referred to as symmetric multiprocessors (SMP), and may also be called uniform memory access (UMA) computers. In the node shown in Figure 2.15, references to memory may need to pass through one, two, or three crossbar switches, depending on where the referenced memory is located. Thus, this node technically has a nonuniform memory access (NUMA) architecture, and, since the node is cache-coherent, this architecture is called ccNUMA. However, since the crossbar switches in the quad-core AMD Opteron implementation of ccNUMA exhibit high performance, this particular node would typically be considered to be an SMP. 

The definitions of the main invariants we have proved for the correctness of the Runway cache coherence protocol are given in Figure 3. InvCtrl asserts, a cache coherency-state correctness if a line is Dirty (or Private-Clean) at a client it must be Invalid at the others. This invariant is necessary for... 

CGH 93] Edmund Clarke, Orna Grumberg, Hiromi Hiraishi, Somesh Jha, David Long, Ken McMillan, and Linda Ness. Verification of the futurebus+ cache coherence protocol. In D. Agnew, L. Claesen, and R. Camposano, editors. Proceedings of the 11th International Conference on Computer Hardware Description Languages and their Applications, 1993. 

GKMK91] Stein Gjessing, Stein Krogdahl, and Ellen Munthe-Kaas. A top down approach to the formal specification of SCI cache coherence. In Computer Aided Verification, pages 83-91, 1991. LNCS 575. 

SCI was perhaps the first SAN to achieve IEEE standardization and has very good bandwidth and latency characteristics. Existing implementations provide between 3.2- and 8-Gbps peak bandwidth with best latencies below 4 /u,sec. The SCI standard includes protocol for support of distributed shared memory operation. However, most clusters employing SCI use PCI-compatible network control cards (e.g.. Dolphin) that cannot support cross-node cache coherence. Nonetheless, even in distributed memory clusters, it provides an effective network infrastructure. 

---