Methodology

In our benchmark case, we have mostly generated IPv4 (OSI layer 3 protocol) flows. In addition, some TCP results are available.

The motherboard used was a Supermicro PDSMI-LN4 with a P4 2.8Ghz (HT disable), 1Go of memory.

The kernel (conf=http://xview.net/papers/jumbo_mbufs_freebsd/LSD) & userland were built with CFLAGS=-02 -pipe, the timecounter was ACPI-fast.

To complete those tests, we used a SmartBits 600 unit and the SmartFlow software.

Those tools are failry trivial to setup, “plug-n-play”. The test can be found here: http://xview.net/papers/jumbo_mbufs_freebsd/smb.flo

Categories of results

nopolling	polling	fastforward	tcp
default FreeBSD configuration	Custom kernel with device polling enable per interface; HZ=2000	polling & net.inet.ip.fastforwarding=1	Same as fastforward but with TCP Mesurements

Results are available here: http://xview.net/papers/jumbo_mbufs_freebsd/smartbits_results/

NOTE: 1024, 1500 & 9k correspond to the Frame length (with CRC)

Issues

•We started to install a FreeBSD 6.0-RELEASE system, this one was not able to find some of the onboard nics because of missing device ID in /sys/dev/em/if_em.c. Upgrading to RELENG_6_1 (cvsup on April, the 7th) fixed this part. That is not really surprising, FreeBSD 6.0 was released more than 4 months ago.
•The second one was more anoying and has not been solved at the time of writting: we were not able to set the MTU for the motherboard nics. This appears to be a limitation of FreeBSD. Therefore, we only used only an Intel PRO 1000/MT (the referance board) with 4 copper interfaces, attached to a PCI-X 133Mhz slot. Thus, we were limited to a 2Gbps throughput. I put online the stdout of ‘piconf -l’ (http://xview.net/papers/jumbo_mbufs_freebsd/pciconf_out.txt). The problematic interfaces are em[4-7].

The need for Jumbo Frame

The standard ethernet MTU is 1500, this size has been decided for 10BaseT and were preserved over time for compatibility reasons.

Jumbo Frame offers the ability to send more data inside a packet, lowering the processing cost for a data transmission(interrupts, memory allocation etc...). Besides, I would recommand you to read this paper (http://www.bsdcan.org/2004/papers/NetworkBufferAllocation.pdf) which explains in details how things are working inside the FreeBSD kernel.

The benefits of this technology are often underlined when talking about data transferts, streaming applications and last but not least, iSCSI.

About Jumbo mbufs

Before 6.1-PRERELEASE, the FreeBSD operating system does support Jumbo Frame; nevertheless, since the mbufs were limited to 2048-Byte clusters, each Jumbo Frame needs to be cut to fit inside this size.

This mechanism is, of course, suboptimal.

Andre Opperman have done some work (MFCed in the beginning of ‘06), to support larger mbufs (4k, 9k & 15k).

This is supposed to enhance performances, by removing the overhead of the previous way used to handle jumbo frame.