HP-MPI User's Guide (11th Edition)

Understanding HP-MPI
Scalability
Chapter 3164
Resource usage of TCP/IP communication
HP-MPI has also been tested on large Linux TCP/IP clusters with as
many as 2048 ranks. Because each HP-MPI rank creates a socket
connection to each other remote rank, the number of socket descriptors
required increases with the number of ranks. On many Linux systems,
this requires increasing the operating system limit on per-process and
system-wide file descriptors.
The number of sockets used by HP-MPI can be reduced on some systems
at the cost of performance by using daemon communication. In this case,
the processes on a host use shared memory to send messages to and
receive messages from the daemon. The daemon, in turn, uses a socket
connection to communicate with daemons on other hosts. Using this
option, the maximum number of sockets opened by any HP-MPI process
grows with the number of hosts used by the MPI job rather than the
number of total ranks.
Figure 3-2 Daemon communication
host1
host2
Daemon
process
Daemon
process
A
B
C
D
E
Application
Socket
connection
processes
Outbound/Inbound
shared-memory
fragments