User Manual

Rev 2.0-3.0.0

Mellanox Technologies

• The vm2's virt-to-phys pkey mapping will be:

so that the default pkey will reside on the vms at index 1 instead of at index 0.

The IPoIB QPs are created to use the PKey at index 0. As a result, the Dom0, vm1 and vm2

IPoIB QPs will all use different PKeys.

 To partition IPoIB communication using PKeys:

Step 1. Create a file "/etc/opensm/partitions.conf" on the host on which OpenSM runs, contain-

ing lines.

This will cause OpenSM to configure the physical Port Pkey tables on all physical ports on the

network as follows:

(the most significant bit indicates if a PKey is a full PKey).

Step 2. Configure (on Dom0) the virtual-to-physical PKey mappings for the VMs.

Step a. Check the PCI ID for the Physical Function and the Virtual Functions.

Step b. Assuming that on Host1, the physical function displayed by lspci is "0000:02:00.0", and that

on Host2 it is "0000:03:00.0"

On Host1 do the following.

Do not touch the Dom0 mapping table (under <nnnn>:<nn>:00.0). Modify only

tables under 0000:02:00.1 and/or 0000:02:00.2. We assume that vm1 uses VF

0000:02:00.1 and vm2 uses VF 0000:02:00.2

Step c. Configure the virtual-to-physical PKey mapping for the VMs.

pkey_idx 0 = 2

pkey_idx 1 = 0

Default=0x7fff,ipoib : ALL=full ;

Pkey1=0x3000,ipoib : ALL=full;

Pkey3=0x3030,ipoib : ALL=full;

pkey idx | pkey value

---------|---------

0 | 0xFFFF

1 | 0xB000

2 | 0xB030

The ",ipoib" causes OpenSM to pre-create IPoIB the broadcast group for the indicated

PKeys.

lspci | grep Mel

cd /sys/class/infiniband/mlx4_0/iov

0000:02:00.0 0000:02:00.1 0000:02:00.2 ...

1. 0000:02:00.0 contains the virtual-to-physical mapping tables for the physical func-

tion.

0000:02:00.X contain the virt-to-phys mapping tables for the virtual functions.

echo 0 > 0000:02:00.1/ports/1/pkey_idx/1

echo 1 > 0000:02:00.1/ports/1/pkey_idx/0

echo 0 > 0000:02:00.2/ports/1/pkey_idx/1

echo 2 > 0000:02:00.2/ports/1/pkey_idx/0