Intel DMA Coalescing White Paper

The DMA Coalescing max-wait time is
adjustable through the same interfaces.
Methods to Verify Behavior
There are three methods to verify
behavior: at the PCIe Bus Analyzer level,
via Package Cx state residency counters,
and raw wall-power measurements of the
system.
PCIe Bus Analyzer Method
The PCIe bus analyzer method requires
instrumentation of the LAN adapter with
a PCIe interposer and capturing PCIe
trafc while the device is being used. (A
full description of what is required to do
this is beyond the scope of this document.)
After the traces have been captured,
using the analyzer-specic software,
visualize the bus usage based on the PCIe
transactions. An example graph generated
by LeCroy PETracer* appears in Figure 6.
Wall-power Method
The raw wall-power measurement
of the system is relatively straight-
forward with the correct wall-power
measurement equipment—such as a Watts
Up*, Kill a Watt*, or an equivalent power
measurement tool. In general, follow
the platform measurement guidelines
published in the EnergyStar* standards—
specically allowing for a settling time
after the system boots to allow startup
processes to complete and then go idle.
Package Cx State Residency Method
The Package Cx state residency method
requires special software, as well as a
basic background on “C states” on CPUs.
“C states” in ACPI corresponds to various
CPU functional states, similar to what
D-states are for I/O devices, and S-states
are for platforms. For example, C0 means
the CPU is fully operational and executing
instructions. The various higher C-states,
such as C1, C2, and C3, correspond to
lower and lower power states with longer
and longer resume times.
The OS typically requests entry into one
of these C states based on its own internal
heuristics, as well as an exit latency table
provided by the BIOS to the OS.
The BIOS maps processor-specic
C-states to the OS-exposed C-states.
For example, if the OS calls the ACPI “C1”
state, this would likely be mapped to the
Intel C1E power state (where the CPU, on
exit, resumes execution at the lowest-
frequency available, if Enhanced Intel
Speed Step, EIST, is also enabled). If the
OS invokes “C3,” the BIOS could activate
either C3 or C6 depending upon how the
BIOS is congured. Lastly, although the
BIOS may request the CPU go into “C6,”
the CPU may auto-demote, or select a
different, more shallow C state such as
C3, based on device access and interrupt
delivery patterns.
conguration.
Linux* versions 2.6.33 and later support
the required power management hooks to
optimize DMA coalescing. Customizations
of the kernel enhance the effect:
1. Enable ‘tickless’ feature with Tick=1000
and preemption mode=Server, CPU idle–
Power Management support=enabled.
2. Load CPUFREQ module: cpufreq_
ondemand.
3. If possible, disable PCSCD (Smart Card
Daemon).
4. After conguration and boot, run
“turbostat” (of powertop version 2.0 or
later) to verify 80% or greater Package
C3 or Package C6 residency.
Controlling DMA Coalescing
Performance
Disabling Interrupt Moderation will also
disable DMA Coalescing. DMA Coalescing
is disabled by default, but is enabled
through the Performance Options tab in
the Windows* DMIX interface (Figure 5)
and through the command line in Linux.
For example:
modprobe igb [<option>=<VAL1>,<VAL2>,...]
See Table 1. The default value for each
parameter is generally the recommended
setting, unless otherwise noted.
Figure 5
DMAC 0, 250, 500, 1000, 2000, 3000,
4000, 5000, 6000, 7000, 8000,
9000, 10000
0
(disabled)
Enables or disables DMA Coalescing feature. Values are in usec’s
and increase the internal DMA Coalescing feature’s internal
timer. DMA (Direct Memory Access) allows the network device
to move packet data directly to the system’s memory, reduc-
ing CPU usage. However, the frequency and random intervals at
which packets arrive do not enable the system to enter a lower
power state. DMA Coalescing enables the adapter to collect
packets before it initiates a DMA event. This may increase net-
work latency but also increases the chances that the system will
enter a lower power state.
Turning on DMA Coalescing may save energy with kernel 2.6.32
and later. This will impart the greatest chance for your system
to consume less power. DMA Coalescing is effective in helping
potentially saving the platform power only when it is enabled
across all active ports.
InterruptThrottleRate (ITR) should be set to dynamic. When
ITR=0, DMA Coalescing is automatically disabled.
Table 1
5
Intel® I350 Ethernet Controller & DMA Coalescing