Lower bounds on information transfer in distributed computations

We derive a lower bound on the interprocessor information transfer required for computing a function in a distributed network configuration. The bound is expressed in terms of the function's derivatives, and we use it to exhibit functions whose computation requires a great deal of interprocess communication. As a sample application, we give lower bounds on information transfer in the distributed computation of some typical matrix operations. Traditional measures of computational complexity, such as the number of primitive operations or memory cells required to compute functions, do not form an adequate framework for assessing the complexity of computations carried out in distributed networks. Even in the relatively straightforward situation of memoryless processors arranged in highly structured configurations, Gentleman 4 has demonstrated that data movement, rather than arithmetic operations, can often be the significant factor in the performance of parallel computations. And for the more general kinds of distributed processing, involving arbitrary network configurations and distributed data bases, the situation is correspondingly more complex. This paper addresses the problem of measuring computational complexity in terms of the interprocess communication required when a computation is distributed among a number of processors. More precisely, we model the distributed computation of functions which depend upon large amounts of data by assuming that the data is partitioned into disjoint subsets, and that a processor is assigned to each subset. Each processor (which we can think of as a node in a computational network) computes some values based on its own "local" data, and transmits these values to other processors, which are able to use them in subsequent local comutations. This "compute locally and share information" procedure is repeated over and over until finally some (predetermined) processor outputs the value of the desired function. In measuring the complexity of such computations we will be concerned, not with the individual local computations, but rather with the total information transfer, i.e., the total number of values which must be transmitted between processors. We derive a lower bound on the total information transfer required for computing a function in a distributed network. The bound is expressed in terms of the function's derivatives, and we use it to exhibit functions whose computation requires a great deal of interprocess communicaion. As a sample application, we give lower bounds on information transfer in the distributed computation of some typical matrix operations.