Re: A note on APIC bus latency

[Alan Cox]

> A message exchange (IO-APIC sends an interrupt message; CPU sends 
> back an EOI message) requires from 35 to 48 APIC bus clocks, or 2-3 
> microseconds. That gets to be pretty significant compared to packet 
> times, especially at Gbit speeds, but even at 100 MHz, and is the 
> time required to burst a thousand bytes or more at faster PCI rates.
> 
> It's also likely to be significant for inter-processor interrupts, 
> though I don't know what the implications are here.

The big implication so far has been some extremely horrible to debug 
irq handling bugs where drivers such as the i810 audio assumed that the
disable of an irq on the pci device was immediate once then pci write
and a pci read to force posting completed. 

There are impacts on things like TLB shootdowns where the latency impacts
an SMP crosscall. I'm not sure how bad the impact is on the bigger numa
boxes as I notice Martin uses multiple sends for that

Alan