HP-MPI V2.3 for Linux Release Note

IMPORTANT: A call to MPI_Comm_dup() always terminates all outstanding
communications with failures on the communicator regardless of the presence or
absence of errors. Therefore, the functionality of MPI_Comm_dup() when using
-ha:recover is not standard-compliant in the absence of errors.
When one or more pairs of ranks within a communicator are unable to communicate
because a rank has exited or because the communication layers have returned errors,
a call to MPI_Comm_dup attempts to return the largest communicator containing ranks
that were fully interconnected at some point during the MPI_Comm_dup call. Because
new errors can occur at any time, there is no guarantee that the returned communicator
is completely error free. However, the two ranks in the original communicator that
were unable to communicate before the call are not included in a communicator
generated by MPI_Comm_dup.
Communication failures can partition ranks into two groups, A and B, such that no
rank in group A is able to communicate to any rank in group B and vice versa. This
implies that a call to MPI_Comm_dup() can behave similarly to a call to
MPI_Comm_split(), returning different legal communicators to different callers.
When a larger communicator exists than the largest communicator the rank can join,
it will return MPI_COMM_NULL. However, extensive communication failures such as a
failed switch, can make such knowledge unattainable to a rank and thus result in the
splitting of the communicator.
If the communicator returned by rank A contains rank B, then either the communicator
return by ranks A and B will be identical or rank B will return MPI_COMM_NULL and
any attempt by rank A to communicate with rank B immediately returns
MPI_ERR_EXITED. Therefore, any legal use of communicator return by
MPI_Comm_dup() should not result in a deadlock. Members of the resulting
communicator either agree on their membership or those that disagree are unreachable
to all members so that an attempt to communicate with those members results in a
failure.
Interruptible Collectives
When a failure (host, process, or interconnect) that affects a collective operation occurs,
at least one rank calling the collective returns with an error. It is the responsibility of
the application to initiate a recovery from those ranks by calling MPI_Comm_dup()
on the communicator used by the failed collective. This will ensure that all other ranks
within the collective also exit the collective. It is possible for some ranks to exit
successfully from a collective call while other ranks exit with failure. Ranks which exit
with MPI_SUCCESS will have successfully completed their role in the operation, and
any output buffers will be correctly set. The return value of MPI_SUCCESS does not
indicate that all ranks have successfully completed their role in the operation.
After a failure, one or more ranks must call MPI_Comm_dup(). This ensures that all
future communication on that communicator results in failure for all ranks until each
18 Information About This Release