Parallel Programming Concepts

1.1 Goals
  1. Reduce Wall Clock Time 
    Wallclocks
     

    In the just-stated goal, you'll notice that it isn't simply time that's being reduced, but wall-clock time; other kinds of "time" could have been emphasized, for example CPU time, which is a count of the exact number of CPU cycles spent working on your job, but wall-clock is considered to be the most significant because it's what you-the-researcher want to spend as little of as possible waiting for the solution: your own time. What is considered to be "acceptable" differs with the situation, and involves characteristics of the user, the particular code being run, and the system it's being run on. But in all cases, it is safe to assume that the user is generally going to be more pleased the faster a solution is produced.

    It should be pointed out that reducing the wall-clock time to solution is only one of many possible goals that might be of interest; some others might be:

  2. Cheapest Possible Solution Strategy 
    Cheaper
     

    Running programs costs money; different ways of achieving the same solution could have significantly different costs. If you're in a fiscally tight situation, you may have no reasonable recourse to a parallel strategy if it costs more than your budget allows. At the same time, you may find that running your program in parallel across a large number of workstation-type computers could cost considerably less than submitting it to a large, mainframe-style mega-beast.

  3. Local versus Non-Local Resources 
    Resources
     

    "Locality," here, usually refers to either "geographical locality" or "political locality." The former is just another way of saying that you want all of your processes to be "close" to one another in terms of communications; the latter indicates that you only want to use resources that are administratively open to you. Both can have a bearing on "cost;" e.g., the more communications latency incurred by your application, the longer it will run, and the more you might be charged, while use of resources "owned" by other organizations may also carry charges.

  4. Memory Constraints 
    MemoryConstraints
     

    As researchers become increasingly computationally sophisticated, the complexity of the problems they tackle increases proportionally (some would say superlinearly, that researchers are forever trying to bite off more than their computers can chew). One of the first resources to get exhausted is local memory -- especially for Grand Challenge level projects, the amount of memory available to a single system is rarely going to be sufficient to the computational and data-storage needs encountered during application runs.

    This situation is greatly alleviated by having access to the aggregate memory made available by distributed computing environments: working storage (main memory) requirements can be spread around the various processors engaged in the cooperative computation, and long-term storage (tape and disk) can be accommodated at different locations (indeed, data security can be enhanced by arranging for multiple copies to be maintained at distinct locations in the distributed environment).

You can probably think of others; basically, any limited resource can be considered as the object of optimization, if it is deemed to be the most important quantity to conserve. In most cases involving large-scale computation, however, user time, as measured by the clock on the wall, is considered to be the single most valuable resource to be conserved.