Re: Voyager subarchitecture for 2.5.46

[Linus Torvalds]

On 6 Nov 2002, Alan Cox wrote:
>
> On Wed, 2002-11-06 at 15:45, Linus Torvalds wrote:
> > It's clearly stupid in the long run to depend on the TSC synchronization.
> > We should consider different CPU's to be different clock-domains, and just
> > synchronize them using the primitives we already have (hey, people can use
> > ntp to synchronize over networks quite well, and that's without the kind
> > of synchronization primitives that we have within the same box).
> 
> NTP synchronization assumes the clock runs at approximately the same
> speed and that you can 'bend' ticklength to avoid backward steps. Thats
> a really cool idea for the x440 but I wonder how practical it is when we
> have CPU's that keep changing speeds and not always notifying us about
> it either.

Note that you have a _lot_ more flexibility than NTP thanks to the strong 
synchronization that we actually do have between CPU's in the end.

The synchronization just isn't strong enough to allow us to believe that 
the TSC is exactly the _same_. But it is certainly string enough that we 
should be able to do a really good job.

Of course, if the TSC changes speed without telling us, we have problems. 

But that has nothing to do witht he synchronization protocol itself: we 
have problems with that even on a single CPU on laptops right now. Does it 
mean that gettimeofday() gets confused? Sure as hell. But it doesn't get 
any _worse_ from being done separately on multiple CPU's.

(And it _does_ get slightly better. On multiple CPU's with per-CPU time
structures at least you _can_ handle the case where one CPU runs at a
different speed, so at least you could handle the case where one CPU is
slowed down explicitly much better than we can right now).

As an example of something that is simpler in the MP/NUMA world than in 
NTP: we see the processes migrating, and we can fairly trivially do things 
like

 - every gettimeofday() will always save the value we return, along with a 
   sequence number (which is mainly read-only, so it's ok to share among 
   CPU's)

 - every "settimeofday()" will increase the sequence number

 - when the next gettimeofday happens, we can check the sequence number 
   and the old gettimeofday, and verify that we get monotonic behaviour in 
   the absense of explicit date setting. This allows us to handle small 
   problems gracefully ("return the old time + 1 ns" to make it 
   monotonic even when we screw up), _and_ it will also act as a big clue
   for us that we should try to synchronize - so that we basically never 
   need to worry about "should I check the clocks" (where "basically 
   never" may be "we check the clocks every minute or so if nothing else 
   happens")

Basically, I think NTP itself would be _way_ overkill between CPU's, I 
wasn't really suggesting we use NTP as the main mechanism at that level. I 
just suspect that a lot of the data structures and info that we already 
have to have for NTP might be used as help.

		Linus