Depending on what parameters you have passed to fsx, it isn't necessarily
going to be doing a lot of I/O. The default for the fsx I have is to max
the file size out at 256kB (on average it will be about half of that), and
you have 1070 instances running, so that agrees with the ~110MB of cache
difference between O_DIRECT and non-O_DIRECT.
Also, in the non-O_DIRECT case fsx will be doing reads from cache and not
disk, so there is no reason to see anything in "bi". The writes may or
may not be a problem, as fsx is "truncate happy", so some large amounts of
data that are "written" are immediately truncated again. For O_DIRECT,
everything is going straight to/from disk, hence much higher IO numbers.
What you should really be checking is how many "ops per second" you are
getting from fsx with and without O_DIRECT. It would be my guess that
the O_DIRECT fsx is actually _slower_ because it is doing more I/O (and
waiting for it to complete). Run each fsx with some fixed number of ops
(-N <num ops>) and see how long it takes for both tests to complete.
> vmstat:
> ======
>
> procs memory swap io system cpu
> r b w swpd free buff cache si so bi bo in cs us sy id
> 1089 0 3 0 2597396 169324 219896 0 0 0 2304 1221 330 68 32 0
>
> vmstat output for O_DIRECT (fsx):
> ===========================
> procs memory swap io system cpu
> r b w swpd free buff cache si so bi bo in cs us sy id
> 36 1034 5 0 2756980 150676 83844 0 0 43269 57496 9446 8861 70 30 0
Cheers, Andreas
-- Andreas Dilger http://sourceforge.net/projects/ext2resize/ http://www-mddsp.enel.ucalgary.ca/People/adilger/- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/