2.4.13-pre5-aa1 O_DIRECT drastic HIGHMEM performance hit

[Marvin Justice]

We're looking into setting up an HD video editing solution with the
following hardware:

dual 1GHz P3, 2GB RAM
Serverworks HeSL based board
dual U160 Adaptec SCSI card
5 68gb Fujitsu drives on each SCSI channel

The 10 drives are software raid0 striped with a 68k chunk size. For an
initial benchmark I got a copy of lmdd and modified it to open files
using O_DIRECT (also hacked it to work on W2K with FILE_FLAG_NO_BUFFER).
Here are the average results in MB/sec for 2.4.13-pre5-aa1 streaming
chunks of data 4MB at a time to RAM where they are immediately
discarded:

                     w           r
no O_DIRECT         136         132
O_DIRECT            111          96

I experimented around a bit and discovered that increasing
KIO_MAX_ATOMIC_IO from 512 to 4096 gave a significant performance boost
for the O_DIRECT read case:
                     w           r
no O_DIRECT         117         124
O_DIRECT            111         165

Next I recompiled without HIGHMEM support and was quite suprised
(numbers are with KIO_MAX_ATOMIC_IO=4096)

                     w           r
no O_DIRECT         138         125
O_DIRECT            221         182

XFS shows similar behavior: (XFS barfs if you change KIO_MAX_ATOMIC_IO
from 512)

                           w         r
O_DIRECT w/ HIGHMEM       114       146
O_DIRECT w/o HIGHMEM      218       248

(Incidentally, W2k reads and writes at a smooth 255 MB/sec on identical
hardware.) 

Are we stuck with a low mem configuration or are there workarounds that
would allow us to stick with the initial 2GB of RAM and still get ~200
MB/sec.
 

--------------------------
Marvin Justice
Software Developer
BOXX Technologies, Inc.
www.boxxtech.com
[e-mail]
(V) [phone]
(F) [phone]