Re: Unifying epoll,aio,futexes etc. (What I really want from epoll)

[John Gardiner Myers]

Matthew D. Hall wrote:

> *  There is a seemingly significant overhead in performing exactly one 
> callback per event.

The "exactly one callback per event" semantics of aio are important for 
cancellation in thread pool environments.  When you're shutting down a 
connection, you need to be able to get to a point where you know no 
other thread is processing or will process an event for the connection, 
so it is safe to free the connection state.

> *  Only one queue per process or kernel thread.

Having a single thread process multiple queues is not particularly 
interesting (unless you have user-space threads or coroutines).  Being 
able to have different threads in the same process process different 
queues is interesting--it permits a library to set up its own queue, 
using its own threads to process it.

> *  No re-arming events.  They must be manually killed.

Rearming events is a useful way to get the correct cancellation 
semantics in thread pool environments.

> -  Should the kernel attempt to prune the queue of "cancelled" events 
> (hints later deemed irrelevant, untrue, or obsolete by newer events)? 

This makes the cancellation semantics much easier to deal with in single 
threaded event loops.  Single threaded cancellation is difficult in the 
current aio interface because in the case where the canceled operation 
already has an undelivered event in the queue, the canceling code has to 
defer freeing the context until it receives that event.

An additional point: In a thread pool environment, you want event wakeup 
to be in LIFO order and use wake-one semantics.  You also want 
concurrency control: don't deliver an event to a waiting thread if that 
pool does not have fewer threads in runnable state than CPUs.