Re: O_DIRECT performance impact on 2.4.18 (was: Re: [PATCH] 2.5.14 IDE 56)

We did some i/o profiling about 6 years ago on a big scientific
 app that had started in fortran and had been rewritten in c++ 
the fortran code used r/w on files and used temp files
the c++ did memmaps and had big data structures - taking advantage of 
memory management.
one thing I thought was interesting is that it was easy to see how a smart
algorithm, not even such a smart one, could adapt i/o to the patterns of
i/o in the fortran code, but the c++ i/o patters were really complex. 

when everything goes into the page cache, it seems like you will loose
information.

On Sun, May 12, 2002 at 09:38:12AM +1000, Lincoln Dale wrote:
> as the person who started this whole thread and made the assertion that 
> copying from A to B is common:
> 
> At 11:35 AM 11/05/2002 -0700, Linus Torvalds wrote:
> >And I personally believe that "generate the data yourself" is actually a
> >very common case. A pure pipe between two places is not what a computer is
> >good at, or what a computer should be used for.
> 
> i think you'd be surprised.  if we include "pipe from disk to network" then 
> a large number of 'server' applications do exactly this.
> webservers do.  fileservers do. http caches do.  streaming-media servers do.
> 
> sure, they may add additional headers on the front and still generate 
> dynamic content in some cases, but the "common case" is 'pipe from disk to 
> network' or 'pipe from network to disk'.
> 'network' is typically TCP but can be UDP (with rate-limiting) in some cases.
> 
> 
> its very good to see this being discussed.  thats a large step forward from 
> many people believing the problem was nonexistent.
> 
> i'm skeptical that continuing to use the page-cache is the correct way to 
> go -- many of these kinds of applications are doing their own form of 
> memory-management and hot-content 'caching' so are happy to manage a 
> few-to-several hundred megabytes of "page cache equivalent" data themselves.
> at least on many of the 2.3.xx linux releases, that was one of the big 
> attractions of 'raw' devices -- they didn't get the box into an OOM situation.
> if 2.5.xx and recent 2.4.xx has the issues of 
> page-cache-doesn't-shrink-fast-enough solved, then its forseeable it will fly.
> 
> 
> cheers,
> 
> lincoln.
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [e-mail]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
---------------------------------------------------------
Victor Yodaiken 
Finite State Machine Labs: The RTLinux Company.
 www.fsmlabs.com  www.rtlinux.com