We're now going to discuss some general issues relevant to the construction of well-designed distributed applications which rely on explicit message passing for data- and control-communications. These principles are largely concerned with issues you should be focusing on as you consider the parallelization of your application:
- How is memory going to be used, and from where?
- How will the different parts of the application be coordinated?
- What kinds of operations can be done collectively?
- When should communications be blocking, and when non-blocking?
- What kinds of synchronization considerations need to be addressed, and when?
- What kinds of common problems could be encountered, and how can they be avoided?
As has been mentioned before, and as will be mentioned again:
There's no substitute for a good design ... and the worse your design, the more time you'll spending debugging it. 
