Re: [Lse-tech] [RFC] [PATCH] Scalable Statistics Counters

[Paul Jackson]

On Fri, 7 Dec 2001, Dipankar Sarma wrote:
> .... If we extend kmem_cache_alloc() to allocate memory
> in a particular NUMA node, we could simply do this for placing the
> counters -
> 
> static int pcpu_ctr_mem_grow(struct pcpu_ctr_ctl *ctl, int flags)
> {
>         ...
> 
>         /* Get per cpu cache lines for the block */
>         for_each_cpu(cpu) {
>                blkp->lineaddr[cpu] = kmem_cache_alloc_node(ctl->cachep, 
> 						flags, CPU_TO_NODE(cpu));
>                ...
>         }
> 
> This would put the block of counters corresponding to a CPU in
> memory local to the NUMA node.

Rather than baking into each call of kmem_cache_alloc_node()
the CPU_TO_NODE() transformation, how about having a
kmem_cache_alloc_cpu() call that allocates closest to a
specified cpu.

I would prefer to avoid spreading the assumption that for each
cpu there is an identifiable node that has a single memory
that is best for all cpus on that node.  Instead, assume that
for each cpu, there is an identifiable best memory ... and let
the internal implementation of kmem_cache_alloc_cpu() find that
best memory for the specified cpu.

Given this change, the kmem_cache_alloc_cpu() implementation
could use the CpuMemSets NUMA infrastructure that my group is
working on to find the best memory.  With CpuMemSets, the
kernel will have, for each cpu, a list of memories, sorted
by distance from that cpu.  Just take the first memory block
off the selected cpus memory list for the above purpose.

                          I won't rest till it's the best ...
			  Manager, Linux Scalability
                          Paul Jackson <[e-mail]> 1.650.933.1373