I would recommand you to read some documentation ([WIKI01] [WIKI02]), if you are not familiar with the concepts involved here. This paper focus on the FreeBSD operating system [HANDBOOK]. You will find more details & similar results for other operating systems on the project’s webpage [REF01]
Why measuring interrupt latency ?
In todays operating systems, it becomes really important to handle interrupt as fast as possible.
Indeed, to increase available bandwith, we need to lower CPU ressource utilisation so to lower the per interrupt cost either by handling interrupt faster or by avoiding “context switching”.
Within the following tests, we tried to categorize the shortest time required to stimulate the FreeBSD kernel as well as how it reacts with reproductible loads. Do we missed some interrupts ? Where is the “breaking point” of the current interrupt handler ? How the system reacts when different interrupts are requesting simultaneously ? How does it manage interrupts priority ? Dans quelles mesures Does the scheduler influences interrupt latency ?
This may give us crutial information on kernel’s behaviour.
Previous experimentations
The “old-school” way to measure interrupt latency was to use the parallel port. Poul-Henning Kamp published, in 1998, his work on this subject [PHK]. Though he concludes his results may suffer “a potentially large systematic error on all the measurements due to a number of different effects”.
Testing methodology
The system used was a Supermicro PDSMI-LN4 motherboard with a Dual-Core P4 @2.8Ghz (HT disable) and 1Go of memory.
Kernel (conf) & userland were built with CFLAGS=-02 -pipe -funroll-loops -ffast-maths, the timecounter was ACPI-fast.
To complete those tests, we used RTBench [REF01].
This tool can be devided into two major parts: the kernel one & the userland
I) Kernel parts
- RTC (Real Time Clock) driver
- Moreover, to workaround , you need to use this kernel patch (i386) [ 6.0-RELEASE | CURRENT ]
II) Userland tools
This includes alllat, the stress testsuite and some automation scripts.
At time of exit, alllat will generate three files with measurements of the Interrupt Service Routine, Driver dispatch latency & the userland process latency. This means we have got measurements results for the bottom half as well as the top half of kernel.
How to use RTBench ?
Categories of results
|
|
|
|
|
|---|
|
|
“new” scheduler which aims to bring the best performance on highly loaded computer & SMP
|
New scheduler designed to handle correctly next generations of CPU (multi-core ones) [OSDI06]
|
|
Those results are available for UP & SMP configurations, here: http://xview.net/papers/interrupt_latency_scheduler/results/
In addition, I provide another test using SpirentCom appliances (Avalanche/Reflector). This bench may generate enough packets to create a significant amount of interrupts on the system.
*Add a schema here ;-)*
Interpretation
Conclusion
We set KTR to analyse the dumps with schedgraph.py. Unfortunately KTR_SCHED does not support ALQ thus it was not possible to get long time running dumps of kernel events [ML06]. You may be able to analyse each ktr.dump on any system with python & tkinter installed using the following command line: python schedgraph.py ktr.dump
I would like to thank Suphathorn Ramabutr, the guy behind RTBench and Gregory Fresnais from Spirent Communication who lend me the appliances.
References:
INTRBench.conf
# INTRBench configuration file
#
# These informations are needed by INTRBench
#
# - Binary of the C program that measures the interrupts latency
$INTRBench_C_PROG = "/tmp/RTBench/bin/alllat";
# - Frequency of interrupt (must be a power of 2)
$INTRBench_FREQ = "1024";
# - Define stress loader and the duration of each stress load
$STRESS_1 = {
ID => "DISK-1",
TITLE => "Disk Stress 1",
BIN => "/tmp/RTBench/bin/disk1",
ARGS => [ "tmp-diskstress1", 100 ],
DURATION => 300, # 5 min
};
$STRESS_2 = {
ID => "DISK-2",
TITLE => "Disk Stress 2",
BIN => "/tmp/RTBench/bin/disk2",
ARGS => [ "diskstress2", 1024 ],
DURATION => 300,
};
$STRESS_3 = {
ID => "SIGNAL-1",
TITLE => "Signal Stress 1",
BIN => "/tmp/RTBench/bin/signal1",
ARGS => [ ],
DURATION => 300,
};
$STRESS_4 = {
ID => "MEM-1",
TITLE => "Memory Stress 1",
BIN => "/tmp/RTBench/bin/mem01",
ARGS => [ ],
DURATION => 300,
};
$STRESS_5 = {
ID => "MEM-2",
TITLE => "Memory Stress 2",
BIN => "/tmp/RTBench/bin/mem02",
ARGS => [ ],
DURATION => 300,
};
$STRESS_6 = {
ID => "MSGQ-1",
TITLE => "Message Queue Stress 1",
BIN => "/tmp/RTBench/bin/msgq01",
ARGS => [ ],
DURATION => 300,
};
$STRESS_7 = {
ID => "PIPE-1",
TITLE => "Unnamed Pipe Stress 1",
BIN => "/tmp/RTBench/bin/pipe01",
ARGS => [ "-w256", "-b2048" ],
DURATION => 300,
};
$STRESS_8 = {
ID => "PIPE-2",
TITLE => "Named Pipe Stress 2",
BIN => "/tmp/RTBench/bin/pipe01",
ARGS => [ "-w256", "-b2048", "-n" ],
DURATION => 300,
};
$STRESS_9 = {
ID => "PIPE-3",
TITLE => "Unnamed Pipe Stress 3",
BIN => "/tmp/RTBench/bin/pipe02",
ARGS => [ "-r256", "-b2048" ],
DURATION => 300,
};
$STRESS_10 = {
ID => "PIPE-4",
TITLE => "Named Pipe Stress 4",
BIN => "/tmp/RTBench/bin/pipe02",
ARGS => [ "-r256", "-b2048", "-n" ],
DURATION => 300,
};
$STRESS_0 = {
ID => "NO-STRESS",
TITLE => "No Stress",
DURATION => 300,
};
@STRESS_SEQ = (
$STRESS_0,
$STRESS_1,
$STRESS_2,
$STRESS_3,
$STRESS_4,
$STRESS_5,
$STRESS_6,
$STRESS_7,
$STRESS_8,
$STRESS_9,
$STRESS_10
);
