- Step 1: Create requestsMPI_Recv_init (buf1, cnt, tp, src, tag, com, &req[0]);MPI_Send_init (buf2, cnt, tp, dst, tag, com, &req[1]);
- for (i=1; i<BIGNUM; i++)
- {
- MPI_Irecv (buf1, cnt, tp, src, tag, com, &req[0]);
Step 2: Replace receiveMPI_Start (&req[0]);
- MPI_Isend (buf2, cnt, tp, dst, tag, com, &req[1]);
Step 2: Replace sendMPI_Start (&req[1]);
- MPI_Waitall (2, req, status);
- do_work(buf1,buf2);
- }
Step 3: Deallocate requestsMPI_Request_free (&req[0]);MPI_Request_free (&req[1]);
The plain text in the worksheet above shows a computational loop that uses a non-blocking receive and a non-blocking send. Since the loop is repeated many times, and the arguments to the communication routines do not change, this program can use persistent communication to improve performance.
The selection boxes on the worksheet show the steps required to convert the program to use persistent communication for both the send and receive.
Step 1: Create requests
The first step in converting the program is to initialize the persistent communication. This is done outside the loop. The receive is initialized with:
There are four MPI routines for initialization of persistent sends, corresponding to the four communication modes:
- MPI_Send_init (S)
- MPI_Ssend_init (S)
- MPI_Rsend_init (S)
- MPI_Bsend_init (S)
Since the worksheet uses standard mode, the persistent request is created using MPI_Send_init.
The initialization routines create persistent request objects and return handles (req[0] and req[1] in the worksheet). They do not cause any data to be transferred.
The initialization routines (MPI_Send_init and variants, and MPI_Recv_init) have the same argument lists as the non-blocking message-passing calls (MPI_Isend and variants, and MPI_Irecv). When adding the initialization routine, simply "borrow" the argument list from the message-passing call which is to be replaced.
In addition to creating request objects using MPI_Send_init, it is sometimes useful to initialize a request handle to the null request MPI_REQUEST_NULL. This can simplify writing code when all tasks do not do exactly the same thing, or when behavior may vary at runtime.
Step 2: Replace receive and send
The second step in porting the program is to replace the receive and send calls (which create requests and start the communication) with calls that only start the communication, based on pre-existing requests.
There are two start calls:
- MPI_Start (S)
- MPI_Startall (S)
These are non-blocking calls. Therefore, the call to MPI_Waitall at the end of the loop is conserved. In fact, a Wait is required between successive MPI_Start calls that use the same request object -- a request object cannot track two communications at once.
Step 3: Deallocate requests
After the loop exits, the program no longer needs the persistent request objects. Since persistent request objects are not deallocated by Wait or Test, they must be explicitly deallocated with:
