Re: [PATCH] io stalls (was: -rc7 Re: Linux 2.4.21-rc6)

[Chris Mason]

On Mon, 2003-06-09 at 18:19, Andrea Arcangeli wrote:

> > Anyway, less talk, more code.  Treat this with care, it has only been
> > lightly tested.  Thanks to Andrea and Nick whose patches this is largely
> > based on:
> 
> I spent last Saturday working on this too. This is the status of my
> current patches, would be interesting to compare them. they're not very
> well tested yet though.
> 

I'll try to get some numbers in the morning.

> They would obsoletes the old fix-pausing and the old elevator-lowlatency
> (I was going to release a new tree today, but I delayed it so I fixed
> uml today too first [tested with skas and w/o skas]).
> 
> those backout the rc7 interactivity changes (the only one that wasn't in
> my tree was the add_wait_queue_exclusive, that IMHO would better stay
> for scalability reasons).
> 

I didn't test without _exlusive for the final iteration of my patch, but
in all the early ones using _exclusive improve latencies.  I think
people are reporting otherwise because they have hit the sweet spot for
number of procs going after the requests.  With _exclusive they have a
higher chance of getting starved by a new process coming in, without the
_exclusive, each waiter has a fighting chance of getting to the free
request on their own.  Hopefully we can do better with the _exclusive,
it does seem to scale much better.

Aside from the io in flight calculations, the major difference between
our patches is in __get_request_wait.  Once a process waits once, that
call to __get_request_wait ignores q->full in my code.

I found the q->full checks did help, but as you increased the number of
concurrent readers/writers things broke down to the old high latencies. 
By delaying the point where q->full was cleared, I could make the
latency benefit last for a higher number of procs.  Finally I gave up
and left it set until all the waiters were gone, which seems to have the
most consistent results.

The interesting part was it didn't seem to change the hit in
throughput.  The cost was about the same between the original patch and
my final one, but I need to test more.

-chris