Gains from non-blocking communication
- avoids deadlock
- decreases synchronization overhead
- reduces systems overhead (on some systems)
Best practices
- post non-blocking sends and receives as early as possible
- do waits as late as possible
Mistakes to avoid
Gains from non-blocking communication:
Avoids deadlock
Non-blocking communication avoids the situation where processes are waiting for the same resource or for each other. Deadlock is discussed in detail in MPI Point to Point II.
Decreases synchronization overhead
Non-blocking calls have the advantage that computation can continue almost immediately, even if the message can't be sent. This can reduce synchronization overhead.
Reduces systems overhead (on some systems)
On some machines, the system overhead can be reduced if the message transport can be handled in the background without having any impact on the computations in progress on both the sender and receiver.
A knowledgeable source has the following comments regarding whether or not non-blocking calls do in fact result in a reduction of system overhead:
...I have the strong suspicion that non-blocking calls can actually incur more overhead than blocking ones that result in immediate message transfer. This extra overhead comes from several sources:
- the cost of allocating a request object
- the overhead of doing the interrupt when the data are transferred later
- the cost of querying to determine whether the transfer has completed.
The first cost can be eliminated by using persistent requests (definitely an advanced topic). All these can be small compared to the synchronization overhead that is avoided if useful computations are available to be done.
For information on persistent requests, consult the standard or available texts, once you've gotten some experience with the basics of MPI. The rest of the above comment simply points out that a non-blocking call carries with it much more systems-baggage than does a blocking call, assuming both are going to be satisfied immediately. But if the transaction is not immediately satisfied, then the non-blocking call wins to the extent that "useful computation" can be accomplished prior to its conclusion.
Best practices:
post non-blocking sends and receives as early as possible
do waits as late as possible
Some additional programming is required with non-blocking calls, to test for completion of the communication. It is best to post sends and receives as early as possible, and to wait for completion as late as possible. "Early as possible" above means that the data in the buffer to be sent must be valid, and likewise the buffer to be received into must be available. "Late as possible" means the point where the send buffer is about to be overwritten, or where the received data are needed by the computation.
Mistakes to avoid:
writing to send buffer between MPI_Isend and MPI_Wait
reading from and writing to receive buffer between MPI_Irecv and MPI_Wait
It should be possible to safely read the send buffer after the send is posted, but nothing should be written to that buffer until status has been checked to give assurance that the original message has been sent. Otherwise, the original message contents could be overwritten. NO user reading or writing of the receive buffer should take place between posting a non-blocking receive and determining that the message has been received. The read might give either old data or new (incoming) message data. A write could overwrite the recently arrived message.
