Intel® I/O Acceleration Technology (Intel® I/OAT) is a new Intel platform network technology
that accelerates, optimizes and seamlessly scales the TCP network stack on Microsoft Windows Server and Linux operating systems.
Intel® I/OAT is focused on optimizing performance, throughput, and CPU utilization for
applications that
Use TCP protocol.
Have a significant portion of their networking traffic as receive based.
Use application buffer sizes larger than 2 KB in Windows and 4 KB in
Linux.
Applications with larger buffer sizes typically see greater gains with Intel®
I/OAT.
It is recommended that you work with your system and applications to determine
the best settings for Intel® I/OAT in your environment.
This is a platform-level solution for data movement and several pieces are necessary for a system to be Intel® I/OAT capable:
Platform: The platform must be based on the Intel® 5000 Series
Chipset with an option in the BIOS to enable Intel® I/OAT.
NOTE: You must verify whether the
Intel® I/OAT DMA function is enabled in the system BIOS. If it is not
enabled, Intel® I/OAT will not function. If Intel® I/OAT is not shown in the BIOS,
then a BIOS upgrade maybe necessary.
NOTE: Intel® I/OAT is disabled by default in
the BIOS setup menu. To enable Intel® I/OAT in the BIOS,
Enter the BIOS setup menu.
Go to "Integrated Devices".
Select "I/OAT DMA Engine" and enable it.
Operating System: Windows Server 2003 with Service
Pack 2 (SP2 contains the Scalable Networking Pack) or SUSE SLES 10 and Red
Hat RHEL
5.
Network Connectivity Hardware: Intel® I/OAT is available on specific server adapters and network
connections available from Intel and other vendors. If you have multiple network adapters in your system, Intel® I/OAT
will operate automatically on the compatible models, while non-compatible
adapters will pass traffic normally. Supported Intel adapters include the Intel®
10 Gigabit XF SR Server Adapter.
Drivers and Connectivity Software: Intel® I/OAT becomes active
when all required software support is loaded and enabled. For
Windows Server 2003, the IOATDMA driver and NDIS miniport driver version 9.3.31.0, or
greater, must be installed. For SUSE SLES 10 and Red Hat RHEL 5, the Intel IOATDMA module must be loaded.
System Memory: System memory channels should be fully populated
for optimal performance. It is recommended to have a minimum of 4 GB of
system memory or 1 GB per channel. Refer to you system guide for specific
information regarding memory channels.
NOTE: At least one Intel® I/OAT capable port
must be enabled when your system starts up. If no Intel® I/OAT capable ports
are enabled at system startup, the IOATDMA driver will not register with NDIS and Intel® I/OAT capable ports enabled
after system startup will show no DMA activity. Enable an Intel® I/OAT capable
port and reboot to resolve this issue.
Intel® I/OAT performance can be monitored using the Status tab of the Intel®
5000 Series Chipset Driver’s Property Page, which can be found in: Device
Manager --> System Device --> 1A38 - Intel 5000 Series Chipset.
The statistics captured are
listed below. The counters are updated approximately every second to show Intel®
I/OAT
activity.
Counter
Description
DMA Count Channel x
Provides DMACountChannelx for Channel 0, Channel 1, Channel 2 or
Channel 3.
DMA Rate Channel x
Provides the DMARateChannelx.
Total DMA Count
Provides the total count of DMAs requested across all queues.
Total DMA Rate
Provides the rate of all DMA requests across all queues.
Intel® I/OAT performance can also be monitored using the Microsoft supplied
application perfmon.exe. The DMA Count Channel x, DMA Rate Channel x, Total DMA
Count and Total DMA Rate counters must be added to monitor Intel® I/OAT
statistics:
Open perfmon.exe and navigate to the Add Counters page.
Make you computer selection from the Select counters from computer
drop-down menu.
Select IOATDMA_DMA_STATS from the Performance object drop-down menu.
NOTE: IOATDMA_DMA_STATS may not appear in the
object drop-down menu. If this occurs, exit and re-open the Add Counters
page. Repeat steps 2 and 3. IOATDMA_DMA_STATS will not appear in the object
drop-down menu of the Add Counters page for approximately three minutes
after initial installation or a reboot.
Highlight the counter you want to add from the Select counters from list
box and highlight the instance for the counter from the Select instances
from list box.
Click on the Add button.
Linux
In a Linux environment, Intel® I/OAT performance can be monitored using the
statistics provided by the DMA driver. For additional information regarding the
channel entries, see Intel® I/OAT sysfs Interface.
Tuning Your System
For optimal benefits from the Intel® I/OAT network solution, your system memory channels should be fully populated. Intel recommends a
minimum of 4 GB of system memory or 1 GB per channel. Typically, there are two
memory slots per channel. For example, slots 0 and 1 are associated with memory
channel 0. Refer to your system guide for specific information regarding memory
channels.
For a Windows environment, if your system has only one physical dual-core CPU, you will only see two DMA
channels in use even if four ports are enabled in the system using Intel® I/OAT.
In a Windows environment, the following advanced settings must be enabled:
RSS
Offload Receive IP Checksum
Offload Receive TCP Checksum
In a Linux environment, the tcp_dma_copybreak variable represents the lower limit of the size of the
socket reads that will be offloaded to the DMA copy engine when CONFIG_NET_DMA
is enabled. The default value is 4096 bytes. Lowering this limit reduces the
threshold when offloading will be enabled.
The tcp_low_latency system variable (/proc/sys/net/ipv4/tcp_low_latency)
ensures data is forwarded through the TCP stack to the application buffer. When
this variable is set to a value of 1, it disables tcp_prequeue in the Linux IPv4
TCP stack and no offloading will occur.
The following conditions must be met to fully support Intel® I/OAT in a
Windows environment.
Windows Server 2003 SP2 must be installed.
The registry key must be present and set to the appropriate values.
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
"EnableTCPA"=dword:0000 0001
At least one network adapter in the system must be able to perform all three tasks:
Offload TCP and IP
checksum in the hardware
Compute the RSS hash value
for TCP IPv4 packets
Must not report TCP Chimney Offload capability
The received packet must be TCP/IP and larger than 1460 bytes.
The networking application must support pre-post buffers for TCP/IP to place received networking packets.
Windows Restrictions for Intel® I/OAT
In a Windows environment, Intel® I/OAT will not be supported with any of the
following features regardless of the registry settings:
Windows firewall
Internet Protocol Security (IPSec) Policy applied
Internet Protocol Network Address Translation (IPNAT)
Third-party firewalls
NDIS 5.1 intermediate drivers
TCP Options
NOTE:
TCP Window Scaling Option: The TCP Window Scaling Option can
be enabled on either the client or server side or both sides
(tcp1323 set to 1). This still allows Intel® I/OAT offload of incoming
packets. However, you will not see the synchronizing packets
since the Window Scaling Option is only present in these
packets.
TCP Timestamp Option: If you do not see any Intel® I/OAT offload,
verify the TCP Timestamp Option on both the server and client
sides. The tcp1323 option should be set to a value other than 2
or 3. A reboot is required for this registry setting change to
take effect. If you do not have access to the client systems
with the TCP Timestamp Option present, you will not be able to
use Intel® I/OAT offload for incoming packets from these clients since
their packets will have the Timestamp Option present in every
TCP packet.
IP Options
When remote access and routing services are enabled, you must create an
interface of which NAT is a part. If you create a NAT interface, Intel® I/OAT will
not work. The IPSec service is either enabled or disabled regardless of this
feature. If IPSec service is enabled, you must not apply an IPSec policy. If an
IPSec policy is applied, Intel® I/OAT will not work. A reboot is required if a NAT
interface was created or an IPSec policy was applied and then disabled in order
for Intel® I/OAT to function.
Under heavy network loads, the user interface may go into a live lock because
all available system interrupts are being used to process network traffic. By
design, Windows prioritizes the network server request over local GUI activity.
To avoid this situation, you should examine the application and network load on
the server and consider either distributing the load between multiple servers or
upgrading the server hardware.
The ioatdma-x.x.tar.gz does not include the kernel
patch required to enable Intel® I/OAT. Refer to the readme.txt file for more
information.
Support for Intel® I/OAT is included starting with the 2.6.18 kernel. Some
distributions, such as Novell SLES 10, have also integrated the patch into their
kernels. In these cases, integrating the kernel patch is not required for Intel®
I/OAT
support. Thus, use the ioatdma-x.x.tar.gz.
NOTE: If the kernel does support
Intel® I/OAT, skip to the Configuration section.
If the 1A38 chipset integrated system device is not listed under lspci, make
sure it is enabled in the BIOS menu:
--> Integrated Device --> I/OAT DMA Engine
By default, it is disabled.
Configuration
The ioatdma driver is a loadable module. The required steps to make and
install the module are:
Rename the old module to minimize the potential confusion. cd /lib/modules/`uname –r`/kernel/drivers/dma/
mv ioatdma.ko ioatdma_old.ko
Make and install the ioatdma module. make
make install
Ensure that the updated module is used. depmod –a
If you are using the TCP Timestamp Option and do not see any Intel® I/OAT offload,
verify the /proc/sys/net/ipv4/tcp_timestamp setting. It must be set to 0 for
Intel® I/OAT to offload the traffic originated from those clients. No reboot is
required for Linux. If you do not have access to the client systems with the TCP
Timestamp Option present, you will not be able to use Intel® I/OAT offload for incoming
packets from these clients since their packets will have the Timestamp Option
present in every TCP packet.
Enabling and Disabling Intel® I/OAT
The Intel® I/OAT network accelerations are enabled by loading the Intel® I/OAT
DMA engine driver at runtime. The driver module filename is ioatdma.ko. Intel®
I/OAT
support should also be enabled in the kernel (refer to the Integration section
for instructions).
To enable Intel® I/OAT, load the ioatdma driver module:
modprobe ioatdma
Removing the ioatdma module once it has been loaded is not recommended since TCP
receives a hold reference to the ioatdma driver when offloading traffic. However,
if the "Forced module unloading" option is enabled in the kernel, the module
may be unloaded with:
rmmod -f ioatdma
WARNING: The rmmod command may hang the
system. For this reason, the module is listed as "unsafe" in the modules
status listing as viewed with the "lsmod" command.
Intel Intel® I/OAT sysfs Interface
When the Intel® I/OAT driver is properly loaded, directories are created in sysfs, under /sys/class/dma, named dma0chanX, where X is
0, 1, 2 or 3.
Channel Entries
Description
in_use
This will equal 1 if the DMA channel is allocated to a client, such
as the network stack.
bytes_transferred
The total number of bytes transferred by either the DMA engine or
CPU copy.
