Memory registration caching correctness

Fast and powerful networks are becoming more popular on clusters to support applications including message passing, file systems, and databases. These networks require special treatment by the operating system to obtain high throughput and low latency. In particular, application memory must be pinned and registered in advance of use. However, popular communication libraries such as MPI have interfaces that do not require explicit registration calls from the user, thus the libraries must manage this aspect themselves. Registration caching is a necessary and effective tool to reuse memory registrations and avoid the overheads of pinning and unpinning pages around every send or receive. Current memory registration caching schemes do not take into account the fact that the user has access to a variety of operating system calls that can alter memory layout and destroy earlier cached registrations. The work presented in this paper fixes that problem by providing a mechanism for the operating system to notify the communication library of changes in the memory layout of a process while preserving existing application semantics. This permits the safe and accurate use of memory registration caching.