The other major distinctive model of memory access is termed distributed, for a very good reason:

• Memory is physically distributed among processors; each local memory is directly accessible only by its processor.

  Just as you're used to when buying a plain computer, each component of a distributed memory parallel system is, in most cases, a self-contained environment, capable of acting independently of all other processors in the system. But in order to achieve the true benefits of this system, of course there must be a way for all of the processors to act in concert, which means "control"...

• Synchronization is achieved by moving data (even if it's just the message itself) between processors (communication).

  The only link among these distributed processors is the traffic along the communications network that couples them; therefore, any "control" must take the form of data moving along that network to the processors. This is not all that different from the shared-memory case, in that you still have control information flowing back to processors, but now it's from other processors instead of from a central memory store.

• A major concern is data decomposition -- how to divide arrays among local CPUs to minimize communication

  Here is a major distinction between shared- and distributed-memory: in the former, the processors don't need to worry about communicating with their peers, only with the central memory, while in the latter there really isn't anything but the processors. A single large regular data structure, such as an array, can be left intact within shared-memory, and each cooperating processor simply told which ranges of indices are its to deal with; for the distributed case, once the decision as to index-ranges has been made, the data structure has to be decomposed, i.e., the data within a given set of ranges assigned to a particular processor must be physically sent to that processor in order for the processing to be done, and then any results must be sent back to whichever processor has responsibility for coordinating the final result. And, to make matters even more interesting, it's very common in these types of cases for the boundary values, the values along each "outer" side of each section, to be relevant to the processor which shares that boundary.