|
A distributed memory system is going to be using message passing in order to provide communication among the various nodes, whether or not the applications programmer uses it explicitly. It can even be argued that, since any other model is going to have to be supported in terms of message passing, no other form of communication can have a lower overhead.
- Single program-multiple data (SPMD)
A generalization of the SIMD data-parallel programming, SPMD further loosens the synchronicity bonds that restrict the operation of the various processors and simply specifies what program all processors must run, but not which instruction each should be executing at any particular time. Data distribution, however, is still a key concern; in fact, another commonly used term for SPMD is data-decomposition:
this alludes to the fact that the overall dataset is going to be decomposed, resulting in a part of it going to each of the participating processors, each of which will run the same code against their own individual sections.
- Multiple program-multiple data (MPMD)
Anything running anywhere, on any kind of data. This is the most general programming model, and subsumes all others as special cases.
An important subcategory of MPMD style parallelism is called Master/Worker:
it is exemplified by having a single processor (the master) controlling all of the tasks being given to the other processors (the workers). The workers execute independently of one another, and only communicate with the master in order to pass back results from the last assigned task, and get a new assignment.
- Examples:
- MPI
Message Passing Interface (MPI) is not a separate product, but rather an interface standard, a set of functions and associated capabilities that parallel tool providers can use to guarantee both a certain level of functionality and a mechanism for porting user codes between various software environments. Although a proposed standard, various message passing environments have already ported various amounts of MPI as higher level functions actually supported by their own routines. Click here for more information on MPI.
- IBM MPL
IBM's Message Passing Library, IBM's proprietary message passing facility for the SP2.
- PVM
The defacto standard in public domain message passing support, the Parallel Virtual Machine suite of tools is still the most widely used distributed memory environment.
- P4, Express, Linda, TCGMSG, ...
Other solutions to the general problem of providing parallel support across a distributed domain of processors have been put forward, for example:
- p4: before you ask, this stands for "Portable Programs for Parallel Processing", and is a well-developed library of message-passing functions. One of its most convenient features is its ability to be used by both message-passing systems and shared-memory systems within the same distributed cluster. Click here for more information.
- Express: a commercial suite of parallel programming tools, with its origins in the Cosmic Cube work done by Geoffrey Fox and friends at CalTech in the mid-80's. Probably one the most complete set of analysis tools, and a visual network construction utility, as well. Click here for more information.
- Linda: described in more detail elsewhere in this document in a shared-memory context. The basic operations are amenable to more "traditional" message-passing usage, and the latest version apparently gives the applications programmer more control over "object" (both data-tuple and process) placement within the network.
- TCGMSG: also described elsewhere in this document in terms of shared-memory, its main "claim to fame" is as the lowest-latency message-passing toolkit of all generic (commercial or public-domain) brands ... of course, it's also the "leanest" (read "has the fewest") in terms of features, and don't ever expect to find MPI running on top of it, but if your application has pretty cut-and-dried "send this here" kinds of communication requirements, this may be your best bet.
|
