Re: Whining about NUMA. :) [Was whining about 2.5...]

[Marcelo Tosatti]

On Mon, 8 Oct 2001, Martin J. Bligh wrote:

> >> The worst possible case I can conceive (in the future architectures 
> >> that I know of)  is 4 different levels. I don't think the number of access
> >> speed levels is ever related to the number of processors ?
> >> (users of other NUMA architectures feel free to slap me at this point).
> > 
> > So you're saying that at most any given node is 4 hops away from any
> > other for your arch?
> 
> For the current architecture (well, for NUMA-Q) it's 0 or 1. For future
> architectures, there will be more (forgive me for deliberately not being 
> specific ... I'd have to ask for more blessing first). Up to about 4. Ish.
> 
> Depending on how much extra latency each hop introduces, it may well
> not be worth adding the complexity of differentiating beyond local vs
> remote? At least at first ...
> 
> Do you know how many hops SGI can get, and how much extra latency 
> you introduce? I know we're something like 10:1 ratio at the moment 
> between local and remote. 
> 
> I guess my main point was that the number of levels was more like constant 
> than linear. Maybe for large interconnected switched systems with small 
> switches, it's n log n, but in practice I think log n is small enough to be 
> considered constant (the number of levels of switches).
> 
> >> So I *think* the worst possible case is still linear (to number of nodes) 
> >> in terms of how many classzone type things we'd need? And the number 
> >> of classzone type things any given access would have to search through 
> >> for an access is constant? The number of zones searched would be
> >> (worst case) linear to number of nodes?
> > 
> > That's how we have our stuff coded at the moment, but with classzones you
> > might be able to get that down even further.  For instance, you could have
> > classzones that correspond to the number of hops a set of nodes is from a
> > given node. Having such classzones might make finding nearby memory easier.
> 
> That's what I was planning on ... we'd need m x n classzones, where m
> was the number of levels, and n the number of nodes. Each search would
> obviously be through m classzones. I'll go poke at the current code some more.

You say "numbers of levels" as in each level being a given number of nodes
on that "level" distance ?