User Manual

Rev 2.1-1.0.6
Mellanox Technologies
101
If the value under admin_guids/<m> is different that the value under gids/<m>, the request
is still in progress.
4.13.7.2.3Partitioning IPoIB Communication using PKeys
PKeys are used to partition IPoIB communication between the Virtual Machines and the Dom0
by mapping a non-default full-membership PKey to virtual index 0, and mapping the default
PKey to a virtual pkey index other than zero.
The below describes how to set up two hosts, each with 2 Virtual Machines. Host-1/vm-1 will be
able to communicate via IPoIB only with Host2/vm1,and Host1/vm2 only with Host2/vm2.
In addition, Host1/Dom0 will be able to communicate only with Host2/Dom0 over ib0. vm1 and
vm2 will not be able to communicate with each other, nor with Dom0.
This is done by configuring the virtual-to-physical PKey mappings for all the VMs, such that at
virtual PKey index 0, both vm-1s will have the same pkey and both vm-2s will have the same
PKey (different from the vm-1's), and the Dom0's will have the default pkey (different from the
vm's pkeys at index 0).
OpenSM must be used to configure the physical Pkey tables on both hosts.
• The physical Pkey table on both hosts (Dom0) will be configured by OpenSM to be:
• The vm1's virt-to-physical PKey mapping will be:
• The vm2's virt-to-phys pkey mapping will be:
so that the default pkey will reside on the vms at index 1 instead of at index 0.
The IPoIB QPs are created to use the PKey at index 0. As a result, the Dom0, vm1 and vm2
IPoIB QPs will all use different PKeys.
 To partition IPoIB communication using PKeys:
Step 1. Create a file "/etc/opensm/partitions.conf" on the host on which OpenSM runs, con-
taining lines.
This will cause OpenSM to configure the physical Port Pkey tables on all physical ports on
the network as follows:
index 0 = 0xffff
index 1 = 0xb000
index 2 = 0xb030
pkey_idx 0 = 1
pkey_idx 1 = 0
pkey_idx 0 = 2
pkey_idx 1 = 0
Default=0x7fff,ipoib : ALL=full ;
Pkey1=0x3000,ipoib : ALL=full;
Pkey3=0x3030,ipoib : ALL=full;
pkey idx | pkey value
---------|---------
0 | 0xFFFF
1 | 0xB000
2 | 0xB030