Installing Chelsio 10GE Driver on CentOS 5.4


Chelsio is one of the leaders for High Speed, Low Latency 10GE Adapters. On great interest to me is the TCP offloading and the iWARP capability of the Card. To complement this high-end quality cards, you have to use good quality high-end but very low latency from Blade Network Technologies (BNT) 

1. Documentation (Impt)

  1. The various documentation list for Chelsio OEM-IBM Information can be found at my Linux Toolkit Blog.
  2. The most critical page is to go to the Chelsio Drivers Downloads page  for a variety of OS can be found.

 

2. Installing the Drivers on CentOS 5.4

1. I have downgraded the OS from CentOS 5.5 to Centos 5.4 as the offloading bonding is not supported by latest latest CentOS 5.5 kernel

2. Download the Linux drivers under Terminator 3 (T3) family Ethernet Adapter drivers. At this point in writing, the latest Linux driver is cxgb3toe-1.4.1.2.tar.gz (TOE/NIC)

3. Untar the drivers and go into the directory

# tar -zxvf cxgb3toe-1.4.1.2.tar.gz
# cd cxgb3toe-1.4.1.2

4. Follow the instruction on the READ ME found at source directory ( $SOURCE_DIR/cxgb3toe-1.4.1.2)

5. The src directory contains the driver source files for building kernel modules. To build the TOE driver, change to the src/ directory and run:

make
make install

6. Go to the $SOURCE_DIR/cxgb3toe-1.4.1.2 directory/tools directory and copy the ifup-local and perftune.sh to the /sbin directory and perftune.sh will be run each time the interface is enabled.

# cd  /tmp/cxgb3toe-1.4.1.2 directory/tools
# cp ifup-tools /sbin
# cp perftune.sh /sbin

7. Run the performance Tuning

/sbin/perftune.sh

You should see output something like this.

/  > Extract internal utility to /var/tmp/mmapr64.           [ PASS ]
 > IRQ Balance daemon is not running.                      [ PASS ]
 > eth1: PCI-E x8 device using all lanes.                  [ PASS ]
 > eth1: Set IRQ  51 smp_affinity to CPU0.                 [ PASS ]
 > eth1: Set IRQ  59 smp_affinity to CPU1.                 [ PASS ]
 > eth1: Set IRQ  67 smp_affinity to CPU2.                 [ PASS ]
 > eth1: Set IRQ 202 smp_affinity to CPU3.                 [ PASS ]
 > eth1: Set IRQ 210 smp_affinity to CPU4.                 [ PASS ]
 > eth1: Set IRQ 218 smp_affinity to CPU5.                 [ PASS ]
 > eth1: Set IRQ 226 smp_affinity to CPU6.                 [ PASS ]
 > eth1: Set IRQ 234 smp_affinity to CPU7.                 [ PASS ]
 > TOM(toe0): Enable DDP.                                  [ PASS ]
 > TOM(toe0): Set 'delayed_ack=2'.                         [ PASS ]
 > TOM(toe0)[eth4]: Disable TCP timestamps.                [ PASS ]
 [ Set sysctls... ]
 > Set net.core.wmem_max="16777216"                        [ PASS ]
 > Set net.core.rmem_max="16777216"                        [ PASS ]
 > Set net.ipv4.tcp_timestamps="0"                         [ PASS ]
 > Set net.ipv4.tcp_rmem="4096 262144 16777216"            [ PASS ]
 > Set net.ipv4.tcp_wmem="4096 262144 16777216"            [ PASS ]
 > Set net.core.optmem_max="524288"                        [ PASS ]
 > Set net.core.netdev_max_backlog="200000"                [ PASS ]
 [ System tuning is complete. ]

8. Configuring iWARP….Coming your way

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s