User Manual

Rev 2.3-1.0.1
Mellanox Technologies
183
3.4.1.6.4 Partitioning IPoIB Communication using PKeys
PKeys are used to partition IPoIB communication between the Virtual Machines and the Dom0
by mapping a non-default full-membership PKey to virtual index 0, and mapping the default
PKey to a virtual pkey index other than zero.
The below describes how to set up two hosts, each with 2 Virtual Machines. Host-1/vm-1 will be
able to communicate via IPoIB only with Host2/vm1,and Host1/vm2 only with Host2/vm2.
In addition, Host1/Dom0 will be able to communicate only with Host2/Dom0 over ib0. vm1 and
vm2 will not be able to communicate with each other
, nor with Dom0.
This is done by configuring the virtual-to-physical PKey mappings for all the VMs, such that at
virtual PKey index 0, both vm-1s will have the same pkey and both vm-2s will have the same
PKey (dif
ferent from the vm-1's), and the Dom0's will have the default pkey (different from the
vm's pkeys at index 0).
OpenSM must be used to configure the physical Pkey tables on both hosts.
• The physical Pkey table on both hosts (Dom0) will be configured by OpenSM to be:
index 0 = 0xffff
index 1 = 0xb000
index 2 = 0xb030
• The vm1's virt-to-physical PKey mapping will be:
pkey_idx 0 = 1
pkey_idx 1 = 0
• The vm2's virt-to-phys pkey mapping will be:
pkey_idx 0 = 2
pkey_idx 1 = 0
so that the default pkey will reside on the vms at index 1 instead of at index 0.
The IPoIB QPs are created to use the PKey at index 0. As a result, the Dom0, vm1 and vm2
IPoIB QPs will all use dif
ferent PKeys.
 To partition IPoIB communication using PKeys:
Step 1. Create a file "/etc/opensm/partitions.conf" on the host on which OpenSM runs, con-
taining lines.
Default=0x7fff,ipoib : ALL=full ;
Pkey1=0x3000,ipoib : ALL=full;
Pkey3=0x3030,ipoib : ALL=full;
This will cause OpenSM to configure the physical Port Pkey tables on all physical ports on
the network as follows:
pkey idx | pkey value
---------|---------
0 | 0xFFFF
1 | 0xB000
2 | 0xB030
(the most significant bit indicates if a PKey is a full PKey).
The ",ipoib" causes OpenSM to pre-create IPoIB the broadcast group for the indicated
PKeys.