Re: do_gettimeofday vs. rdtsc in the scheduler

[Vojtech Pavlik]

On Tue, Sep 17, 2002 at 03:02:04PM -0700, James Cleverdon wrote:
> On Tuesday 17 September 2002 02:18 pm, David S. Miller wrote:
> >    From: Alan Cox <[e-mail]>
> >    Date: 17 Sep 2002 22:28:12 +0100
> >
> >    A bus clock - but things like the x440 have more than one bus clock. Its
> >    NUMA. Also the bus clock and rdtsc clock are different - rdtsc is
> >    dependant on the multiplier. Shove a celeron 300 and a celeron 450 in a
> >    BP6 board with tsc on and enjoy
> >
> > That's mostly my point.
> >
> > If the bus clocks differ, then great create some system wide crystal
> > oscillator.  That's a detail, the important bit is that you don't need
> > to go out to the system bus to read the tick value, it must be cpu
> > local to be effective and without serious performance impact.
> > -
> 
> It's more than just a detail.  Sequent's last NUMA system (_not_ the NUMA-Q;  
> never released) did exactly what you suggest.  The midplane card generated 
> the bus clock for all quad modules.  We had requested this feature because it 
> was such a pain dealing with clock drift between nodes in the OS.
> 
> The HW guys were able to give us synchronized bus clocks on a 16-way box, but 
> warned us that it would not be practical on the 256-way.  Too much clock skew 
> at those speeds, or something like that.  I suppose you could trade off 
> interconnect rate for clock sync, but then performance would suffer.
> 
> I don't know how Sun and SGI manage with their larger systems.  Either they 
> don't do clock sync, or they may have to make expensive tradeoffs.
> 
> Interestingly, Intel's IA64 manual does not guarantee that the CPU clock (and 
> thus its TSC register) has anything to do with the bus clock rate.  Maybe 
> they want to dabble with asynchronous logic or multiple clock domains in 
> future CPUs.

The point here is: You don't need a synchronized bus clock. You don't
need synchronized CPU clocks. You need a synchronized system-wide clock
that doesn't drive any bus or CPU, just a simple counter in every CPU
that you can read from inside the CPU. You can pull that pretty far and
to many CPUs. That's what I understand Sun does.

-- 
Vojtech Pavlik
SuSE Labs