fsync 50 times slower after 2.5.27

[Duncan Sands]

I noticed a performance degradation in recent kernels:
fsync takes around 50 times longer in kernels 2.5.28 to
2.5.34 when the system is under heavy load, as compared
to kernels <= 2.5.27.  I noticed this because it makes kmail
unusable.  2.5.34 is the most recent kernel I tested.

"Heavy load" is kernel compile (-j 4) at the same time as
another heavy compile.  The following fsyncs come from
strace -T -p <kmail pid>.  kmail does several fsyncs grouped
together, followed by other stuff, followed by more fsyncs.

Here are some typical fsync groups.  The time for each fsync
is the last number (in seconds).

2.5.27 (similar results for 2.5.26 and 2.4.19):

fsync(17)                               = 0 <0.078718>
fsync(18)                               = 0 <0.099900>
fsync(25)                               = 0 <0.004719>
fsync(11)                               = 0 <0.001747>
fsync(12)                               = 0 <0.001726>
fsync(13)                               = 0 <0.014935>
fsync(14)                               = 0 <0.011553>
fsync(15)                               = 0 <0.002506>
fsync(16)                               = 0 <0.002854>

2.5.28 (similar results for kernels up to 2.5.34):

fsync(17)                               = 0 <0.682749>
fsync(18)                               = 0 <2.142922>
fsync(22)                               = 0 <2.269918>
fsync(24)                               = 0 <1.114331>
fsync(11)                               = 0 <4.092790>
fsync(12)                               = 0 <2.309529>
fsync(13)                               = 0 <0.441093>
fsync(14)                               = 0 <1.730422>
fsync(15)                               = 0 <5.444556>
fsync(16)                               = 0 <1.844690>

The filesystem is ext3 but the same occurs with ext2.
This is a UP x86 (400MHz), no preempt.

Any ideas?  I have looked through the changes between
2.5.27 and 2.5.28 but don't see any obvious culprits...

All the best,

Duncan.