How leaking MPI MPI_Requests hurts your application. Always be sure to use an MPI TEST or WAIT operation on your pending MPI_Requests!

1. Always MPI_TEST or
MPI_WAIT
Jeff Squyres
The MPI Guy
December 2012

2. void sender(void) { void receiver(void) {
MPI_Request request; int ack;
int ack; /* … */
/* … */ MPI_Recv(buffer, …);
MPI_Isend(buffer, …, MPI_Send(&ack, …);
&request); }
MPI_Recv(&ack, …);
}

3. void sender(void) { void receiver(void) {
MPI_Request request; int ack;
int ack; /* … */
/* … */ MPI_Recv(buffer, …);
MPI_Isend(buffer, …, MPI_Send(&ack, …);
&request); }
MPI_Recv(&ack, …);
}

4. void sender(void) { void receiver(void) {
MPI_Request request; int ack;
int ack; /* … */
/* … */ MPI_Recv(buffer, …);
MPI_Isend(buffer, …, MPI_Send(&ack, …);
&request); }
MPI_Recv(&ack, …);
}
Must (eventually) use an MPI TEST or WAIT operation
on the pending request, even if you “know” that it
has already completed

5. But
WHY?

6. A. Because MPI says so.

7. But I don’t WANT TO!

8. A. Because MPI says so.
Don’t fight the Law, son.
The Law will always win.

9. B. Because you’ll leak resources

10. Consider what happens
when you Isend…
MPI_Isend(buffer, …, &request);

11. Consider what happens
when you Isend…
MPI_Isend(buffer, …, &request);
isend_request
MPI allocates a request

12. Consider what happens
when you Isend…
MPI_Isend(buffer, …, &request);
isend_request
MPI allocates a request
MPI performs the send

13. Consider what happens
when you Isend…
resources
MPI_Isend(buffer, …, &request);
isend_request
MPI allocates a request
MPI performs the send
…which may include allocating additional resources

14. Consider what happens
when you Isend…
MPI_Isend(buffer, …, &request);
isend_request
resources
MPI allocates a request
MPI performs the send
…which may include allocating additional resources
Those resources are attached to the request

15. Consider what happens
when you Isend…
MPI_Isend(buffer, …, &request);
isend_request
resources
The send eventually done

16. Consider what happens
when you Isend…
MPI_Isend(buffer, …, &request);
isend_request
resources
The send eventually finishes
The request is marked complete

17. Consider what happens
when you Isend…
MPI_Isend(buffer, …, &request);
isend_request
resources
…but it’s still there!

18. Consider what happens
when you Isend…
MPI_Isend(buffer, …, &request);
isend_request
resources
…but it’s still there!
Eating valuable memory!

19. Consider what happens
when you Isend…
MPI_Isend(buffer, …, &request);
isend_request
resources
…but it’s still there!
Eating valuable memory!
And possibly still holding on to valuable resources!

20. Consider what happens
when you Isend…
MPI_Isend(buffer, …, &request);
MPI_Wait(&request, …); isend_request
resources
…until you invoke a TEST
or WAIT operation that
completes the request

21. Consider what happens
when you Isend…
MPI_Isend(buffer, …, &request);
MPI_Wait(&request, …);
resources
Then the request is freed

22. Consider what happens
when you Isend…
MPI_Isend(buffer, …, &request);
MPI_Wait(&request, …);
Then the request is freed
…including any resources that
weren’t previously released

23. Does it really
MATTER?

25. Individual MPI_Requests
One tiny
request may be small

26. Particularly compared to
One tiny
request how much RAM is available
All of RAM

27. …but every MPI process
is making requests
All of RAM

28. …and they might be making
LOTS of requests
All of RAM

29. You might fill RAM slowly
…but you’ll still fill it
All of RAM

30. Worse: hardware resources might not
be released until the request is complete
isend_request
Remember these? resources

31. Meaning: it’s not just abundant
RAM that you’re consuming
resources

32. Communication hardware = expensive
Therefore, far less resources available
resources resources resources resources
resources resources resources resources
resources resources resources resources
resources resources resources resources
resources resources resources resources
resources resources resources resources
resources resources resources resources

33. Compare: RAM vs.
communication hardware resources
All of RAM

34. Compare: RAM vs.
communication hardware resources
Wasting this may be forgivable
All of RAM

35. Compare: RAM vs.
communication hardware resources
Wasting this may be forgivable
Wasting this is
All of RAM a terrible idea

36. void sender(void) { void receiver(void) {
MPI_Request request; int ack;
int ack; /* … */
/* … */ MPI_Recv(buffer, …);
MPI_Isend(buffer, …, MPI_Send(&ack, …);
&request); }
MPI_Recv(&ack, …);
MPI_Wait(&request, …);
}

37. void sender(void) { void receiver(void) {
MPI_Request request; int ack;
int ack; /* … */
/* … */ MPI_Recv(buffer, …);
MPI_Isend(buffer, …, MPI_Send(&ack, …);
&request); }
MPI_Recv(&ack, …);
MPI_Wait(&request, …);
}
If you “know” the send is already complete, the WAIT
will likely be fast

38. Friends don’t let friends
leak MPI_Requests

39. Thank you.