cache coherence protocol

Abstract. In the last three years or so we at Enterprise Platforms Group at Intel Corporation have been applying formal methods to various problems that arose during the process of defining platform architectures for Intel’s processor families. In this paper we give an overview of some of the problems we have worked on, the results we have obtained, and the lessons we have learned. The last topic is addressed mainly from the perspective of platform architects. 1. Problems and Results Modern computer systems are highly complex distributed systems with many interacting components. Architecturally they are often organized like a computer network into multiple layers: physical layer, link layer, protocol layer, etc. Most of the problems to which we applied formal methods are the formal verification (FV) of intricate protocols in the protocol and link layers. In addition, we also found several novel uses of binary decision diagrams (BDDs) [3] that are worth mentioning. 1.1. Directory-bas...

Assertions are valuable in producing quality software but some assertions like those checking data structure invariants are costly by nature. The authors of this paper propose that costly assertions be queued up for later execution by a free processor. The state of data is preserved using a memory snapshot. Such a scheme will extract further parallelism from a given application and improve its speed up on a parallel machine. Three schemes for end user programs to use memory snapshots are discussed. One is based on memory mapping like mmap. Second scheme uses unused virtual address space bits for snapshot identifier. The third scheme uses a software library. Merits and demerits of user interface schemes are identified. Three methods for backend implementation of memory snapshots are discussed. One is based on physical address space versioning. It proposes modifications to memory controller in a way similar to transactional memory. It goes on to introduce a snapshot aware cache coherence protocol. The second method is based on virtual address space versioning and suggests kernel modifications. The last method is totally supported in a user library. Advantages and disadvantages of different implementations are compared. Evaluation is not extensive, given the large matrix of implementation possibilities. However basic memory controller snapshot capability is implemented in M5 simulator. Additionally, a software library approach is implemented. Most valuable results of this evaluation are the pros and cons of various approaches discussed. The feasibility study of various implementation schemes is also an important groundwork for future research on memory snapshots. Based on these experiments, the authors believe that memory snapshot capability is a powerful tool that allows parallelism where it is not natural without such a capability.

The specification and verification of shared-memory multiprocessor cache coherence protocols is a paradigmatic example of parallel technologies where formal methods can be applied. In this paper we present the specification and verification of a cache protocol and a set of formalisms which are based on 'process theory'. System correctness is not established by simple techniques such as testing and simulation, but 'ensured'in terms of the underlying formalism. In order to manipulate the specification and verify the properties we have used ...

Nowadays, the computational systems (multi and uniprocessors) need to avoid the cache coherence problem. There are some techniques to solve this problem. The MESI cache coherence protocol is one of them. This paper presents a simulator of the MESI protocol which is used for teaching the cache memory coherence on the computer systems with hierarchical memory system and for explaining

To meet the demand for more powerful high-performance shared-memory servers, multiprocessor systems must incorporate efficient and scalable cache coherence protocols, such as those based on directory caches. However, the limited directory cache size of the increasingly larger systems may cause frequent evictions of directory entries and, consequently, invalidations of cached blocks, which severely degrades system performance. A significant percentage of

In a previous paper we presented a method which allows to compute abstractions for parameterized systems modeled in the decidable logic WS1S. These WS1S systems provide an intuitive way to describe parameterized systems of finite state processes. The abstractions can be used to establish properties of the parameterized network. To be able to prove liveness properties, an algorithm is used which enriches the abstract system with fairness constraints. We summarize this verification method and present its application by the verification of both safety and liveness properties of a non-trivial example of a cache coherence protocol, provided by Steve German. Contact Author.

This paper explores a new technique called coherence decoupling , which breaks a traditional cache coherence protocol into two protocols: a Speculative Cache Lookup (SCL) protocol and a safe, backing coherence protocol. The SCL protocol produces a speculative load value, typically from an invalid cache line, permitting the processor to compute with incoherent data. In parallel, the coherence protocol obtains the necessary coherence permissions and the correct value. Eventually, the speculative use of the incoherent data can be verified against the coherent data. Thus, coherence decoupling can greatly reduce --- if not eliminate --- the effects of false sharing. Furthermore, coherence decoupling can also reduce latencies incurred by true sharing. SCL protocols reduce those latencies by speculatively writing updates into invalid lines, thereby increasing the accuracy of speculation, without complicating the simple, underlying coherence protocol that guarantees correctness.The performa...

The evaluation of network performance under real application loads is carried out by detailed time-intensive and resourceintensive simulations. Moreover, the use of ILP processors in cc-NUMA architectures introduces non-deterministic memory accesses; the resulting parallel system must be modeled by a detailed execution-driven simulation, further increasing the evaluation cost. This work introduces a simulation methodology, based on network traces, to estimate the impact that a given network has on the execution time of parallel applications. This methodology allows the study of the network design space with a level of accuracy close to that of execution-driven simulations but with much shorter simulation times. The network trace, extracted from an execution-driven simulation, is processed to substitute the temporal dependencies produced by the simulated network with an estimation of the message dependencies caused by both the application and the applied cache-coherent protocol. This methodology has been tested on two direct networks, with 16 and 64 nodes respectively, running the FFT and Radix applications of the SPLASH2 suite. The trace-driven simulation is 3 to 4 times faster than the execution-driven one with an average error of 4% in total execution time. * This Work is supported in part by TI98-1162-C02-01. approximation to real applications [2]. Notwithstanding, the analysis of the impact that the interconnection network has on system performance can hardly be forecast unless we use realistic application loads, similar to those the network is being designed to support.

In a modern shared memory multiprocessor, it is possible to support more than one protocol for maintaining cache coherence. Possible candidates might be based on the Write-Back/Invalidate, Write-Through/Invalidate, and Write-Update protocols. Hybrid protocols allow the use of different protocols for different data blocks, and dynamic hybrid protocols additionally allow for changes in the choice of protocol during the execution of an application. In this paper, we introduce a set of analytical models for predicting the performance of parallel applications under various cache coherence protocol assumptions. These models can be used at compile time to determine which protocols are to be used for which data blocks, and also to determine when to change protocols in the case of dynamic protocols. Although we focus on tightly-coupled multiprocessor systems, the models apply equally well to loosely-coupled distributed systems such as networks of workstations. Our models are unique in that they lie between a large body of theoretical models that assume independence and a uniform distribution of memory accesses across processors, and a large body of address-trace oriented models that assume the availability of a precise characterization of interleaving behavior of memory accesses. The former are not very realistic, and the latter are not suitable for compile-time and run-time usage. In contrast, our models use a set of parameters that characterize the access behavior of applications well, and can be obtained with advanced compiler technology. We present the models and show how the required parameters can be obtained. We assess the accuracy of our models on 15 parallel applications and show that we can predict performance within a 10% margin. As part of this study, we also show the potential advantage of using dynamic hybrid protocols. 1. The concept of classifying data by how it is accessed (i.e. degree of sharing and access mode) has also been used by other researches [8, 25-27], although their classification differs slightly from ours.

Protection and security are becoming essential requirements in commercial servers. In this paper, we present a fast and efficient method for providing secure memory and cache-to-cache communications in shared memory multiprocessor systems that are becoming enormously popular in designing servers for various applications. Since our scheme is independent of underlying interconnects and cache coherence protocols, we refer to it as Interconnects-Independent Security Enhanced Shared Memory Multiprocessor Systems (I 2 SEMS). The main challenge in designing I 2 SEMS is how to precompute keystreams in a timely manner, which is critical to minimize performance overhead. We achieve this goal by adopting a single system-wide Global Counter Controller (GCC) and three additional components for each processor: a keystream queue, a keystream cache, and a keystream pool. The GCC assigns a unique range of counters as a way to help processors precompute the counters' keystreams. We have implemented I 2 SEMS using Simics with Wisconsin multifacet General Execution-driven Multiprocessor Simulator (GEMS). We tested our design with SPLASH-2 benchmarks on up to 16-processor shared memory multiprocessor systems. Simulation results show that the overall performance slowdown is 4% on average and the keystream hit rate is as high as 78%. The stable keystream hit rate shows that I 2 SEMS works well with both memory-read and memory-write dominant applications. Similar to the conventional cache, a large keystream pool size is beneficial to high hit rates.

The use of model checking for validation requires that models of the underlying system be created. Creating such models is both difficult and error prone and as a result, verification is rarely used despite its advantages. In this paper, we present a method for automatically extracting models from low level software implementations. Our method is based on the use of an extensible compiler system, xg++, to perform the extraction. The extracted model is combined with a model of the hardware, a description of correctness, and an initial state. The whole model is then checked with the Mur model checker. As a case study, we apply our method to the cache coherence protocols of the Stanford FLASH multiprocessor. Our system has a number of advantages. First, it reduces the cost of creating models, which allows model checking to be used more frequently. Second, it increases the effectiveness of model checking since the automatically extracted models are more accurate and faithful to the unde...

The use of model checking for validation requires that models of the underlying system be created. Creating such models is both difficult and error prone and as a result, verification is rarely used despite its advantages. In this paper, we present a method for automatically extracting models from low level software implementations. Our method is based on the use of an extensible compiler system , xg++, to perform the extraction. The extracted model is combined with a model of the hardware, a description of correctness, and an initial state. The whole model is then checked with the Murφ model checker. As a case study, we apply our method to the cache coherence protocols of the Stanford FLASH multiprocessor. Our system has a number of advantages. First, it reduces the cost of creating models, which allows model checking to be used more frequently. Second, it increases the effectiveness of model checking since the automatically extracted models are more accurate and faithful to the un...

We present PREACH, an industrial strength distributed explicit state model checker based on Murphi. The goal of this project was to develop a reliable, easy to maintain, scalable model checker that was compatible with the Murphi specification language. PREACH is implemented in the concurrent functional language Erlang, chosen for its parallel programming elegance. We use the original Murphi front-end to parse the model description, a layer written in Erlang to handle the communication aspects of the algorithm, and also use Murphi as a back-end for state expansion and to store the hash table. This allowed a clean and simple implementation, with the core parallel algorithms written in under 1000 lines of code. This paper describes the PREACH implementation including the various features that are necessary for the large models we target. We have used PREACH to model check an industrial cache coherence protocol with approximately 30 billion states. To our knowledge, this is the largest number published for a distributed explicit state model checker. PREACH has been released to the public under an open source BSD license.

A shared data structure is lock-free if its operations do not require mutual exclusion. If one process is interrupted in the middle of an operation, other processes will not be prevented from operating on that object. In highly concurrent systems, lock-free data structures avoid common problems associated with conventional locking techniques, including priority inversion, convoying, and difficulty of avoiding deadlock. This paper introduces transactional memory, a new multiprocessor architecture that supports lock-free implementations of complex data structures in a simple and efficient way. Transactional memory allows programmers to define customized read-modify-write operations that apply to multiple, independently-chosen words of memory. It is implemented by straightforward extensions to any ownership-based multiprocessor cachecoherence protocol. Simulation results show that transactional memory outperforms the best known locking techniques for simple benchmarks, even in the absence of priority inversion, convoying, and deadlock.

Distributed shared-memory architectures typically employ a directory-based protocol to maintain cache coherence. Identifying sharing patterns in parallel programs and applying specialized optimizations can increase cache-coherence protocol efficiency and yield performance improvements. In this thesis, I propose and study both optimizations to sharing patterns and techniques to identify sharing patterns. The main thrust of the thesis is GLOW, a comprehensive optimization for wide sharing-a sharing pattern that is a serious obstacle to scalability to large numbers of processors. I present GLOW in the form of extensions to the SCI ANSI/IEEE standard. GLOW is implemented in special network switches and incorporates characteristics that are not found together in previous proposals: scalable writes and scalable reads, network locality (by exploiting the abundance of widely-shared data to satisfy requests locally), simplicity, transparency to the base protocol, I dedicate this thesis to the memory of my father Vasileios, my mother Ekaterini and my beloved partner Angeliki. The love of my mother, my brother Efthimios, my sister Eleni and her family, as well as Angeliki's family kept us going through graduate school. I would like to thank my advisor and mentor, professor James R. Goodman for supporting me. I am grateful to professor David A. Wood for his insight, professor Gurindar S. Sohi for his support, and professor Mark D. Hill for his help. Their sound advice helped me to complete my Ph.D. Also, I would like to thank all my friends and colleagues at the University of Wisconsin, espe

Caching has been intensively used in memory and traditional file systems to improve system performance. However, the use of caching in parallel file systems and I/O libraries has been limited to I/O nodes to avoid cache coherence problems. In this paper, we specify an adaptive cache coherence protocol very suitable for parallel file systems and parallel I/O libraries. This model exploits the use of caching, both at processing and I/O nodes, providing performance increase mechanisms as aggressive prefetching and delayed-write techniques. The cache coherence problem is solved by using a dynamic scheme of cache coherence protocols with different sizes and shapes of granularity. The proposed model is very appropriate for parallel I/O interfaces, as MPI-IO. Performance results, obtained on an IBM SP2, are presented to demonstrate the advantages offered by the cache management methods proposed.

The evaluation of network performance under real application loads is carried out by detailed time-intensive and resourceintensive simulations. Moreover, the use of ILP processors in cc-NUMA architectures introduces non-deterministic memory accesses; the resulting parallel system must be modeled by a detailed execution-driven simulation, further increasing the evaluation cost. This work introduces a simulation methodology, based on network traces, to estimate the impact that a given network has on the execution time of parallel applications. This methodology allows the study of the network design space with a level of accuracy close to that of execution-driven simulations but with much shorter simulation times. The network trace, extracted from an execution-driven simulation, is processed to substitute the temporal dependencies produced by the simulated network with an estimation of the message dependencies caused by both the application and the applied cache-coherent protocol. This methodology has been tested on two direct networks, with 16 and 64 nodes respectively, running the FFT and Radix applications of the SPLASH2 suite. The trace-driven simulation is 3 to 4 times faster than the execution-driven one with an average error of 4% in total execution time.