For standard mode, the library implementor specifies the system behavior that will work best for most users on the target system. For MPI/Pro, there are two scenarios, depending on whether the message size is greater or smaller than a threshold value, called the cutoff limit.
Message size less than or equal to cutoff limit
The behavior when the message size is less than or equal to the threshold is shown below:
Click here for Flash animation
- Message is buffered on the receiving side
- System, not user, manages buffer
- Synchronization overhead is minimized for sender
- If buffer is full: call blocks with no error, waits until space is free
In this case, the blocking standard send MPI_Send (S) copies the message over the network into a system buffer on the receiving node. The standard send then returns, and the sending task can continue computation. The system buffer is attached when the program is started -- the user does not need to manage it in any way. There is one system buffer per task that will hold multiple messages. The message will be copied from the system buffer to the receiving task when the receive call is executed.
As with buffered mode, use of a buffer decreases the likelihood of synchronization overhead on the sending task at the price of increased system overhead resulting from the extra copy to the buffer. As always, synchronization overhead can be incurred by the receiving task if a receive is posted first.
Unlike buffered mode, the sending task will not incur an error if the buffer space is exceeded. Instead the sending task will block until the receiving task calls a receive that pulls data out of the system buffer. Thus, synchronization overhead can still be incurred by the sending task.
Message size greater than cutoff limit
When the message size is greater than the threshold, the behavior of the blocking standard send MPI_Send (S) is essentially the same as for synchronous mode.
- No buffering of large messages
- Reduces system overhead
Why does standard mode behavior differ with message size? Small messages benefit from the decreased chance of synchronization overhead resulting from use of the system buffer. However, as message size increases, the cost of copying to the buffer increases, and it ultimately becomes impossible to provide enough system buffer space. Thus, standard mode tries to provide the best compromise.
In MPI/Pro, what is the threshold for this switchover in behavior? It depends on the communication protocol that is being used. On CTC systems, this can be either: SMP (Symmetric Multi-Processor), for multithreading on a single node; or TCP (Transmission Control Protocol), the Internet standard. SMP is specialized but very fast, while TCP is better for heterogeneous environments but can be relatively slow in the absence of special hardware. The relative transfer speeds explain MPI/Pro's default values for the cutoff limits listed in the table below:
Protocol
(MPI_COMM) |
Default
cutoff limit |
Flag to mpirun to
change cutoff limit |
| TCP |
32768 bytes |
-tcp_long xxx(in bytes) |
| SMP |
8248 bytes |
-smp_long xxx(in bytes) |
At cutoff limit, system and synchronization overheads are about equal
At the cutoff limit, the system and synchronization overheads are about equal. Thus, the default cutoff limit is set higher for TCP because it takes longer to post a receive across the network. Note that the cutoff limit can be tuned at runtime with flags to mpirun as shown. However, manipulating these values can lead to undefined behavior.
You have now seen the system behavior for all four communication modes for blocking sends.
