Re: userspace irq balancer

[Andrea Arcangeli]

On Tue, May 27, 2003 at 01:34:46AM +0200, Andrea Arcangeli wrote:
> On Mon, May 26, 2003 at 04:26:16PM -0700, Andrew Morton wrote:
> > Andrea Arcangeli <[e-mail]> wrote:
> > >
> > >  	if (IRQ_ALLOWED(phys_id, allowed_mask) && idle_cpu(phys_id))
> > >  		return cpu;
> > 
> > How hard would it be to make this HT-aware?
> > 
> > 	idle_cpu(phys_id) && idle_cpu_siblings(phys_id)
> > 
> > or whatever.
> 
> yeah! that was the obvious next step. as fast path the additional && is
> sure good. Maybe that's enough after all, and we might search only for
> fully idle cpus, however I wouldn't dislike to search for a fallback
> (partially) logical idle cpu if none physical cpu is (fully) idle.

I'm going to try this (if it compiles ;). the ksoftirqd check is the one
for the NAPI workload brought to attention by Dave. the idea is that
statistically the softirq load will follow the hardirq load. Both wants
to go into an idle cpu. But we don't want to mistake the softirq load
for unrelated cpu load. So we don't want to separate a ksoftirqd load
from the irq load or we could keep bouncing over and over again.

For HT I take the trivial approch you mentioned above that is to switch
only if the physical cpu is completely idle.

--- ./arch/i386/kernel/io_apic.c.~1~	2003-05-27 02:45:34.000000000 +0200
+++ ./arch/i386/kernel/io_apic.c	2003-05-27 03:00:32.000000000 +0200
@@ -217,13 +217,18 @@ extern unsigned long irq_affinity [NR_IR
 #define IRQ_ALLOWED(cpu,allowed_mask) \
 		((1UL << cpu) & (allowed_mask))
 
+#define ksoftirqd_is_running(phys_id) (cpu_curr(phys_id) == ksoftirqd_task(phys_id))
+#define __irq_idle_cpu(phys_id) (idle_cpu(phys_id) || ksoftirqd_is_running(phys_id))
+#define irq_idle_cpu(phys_id) (__irq_idle_cpu(phys_id) &&
+			       (smp_num_siblings <= 1 || __irq_idle_cpu(cpu_sibling_map[phys_id])))
+
 static unsigned long move(unsigned int curr_cpu, unsigned long allowed_mask, unsigned long now, int direction)
 {
 	unsigned int cpu = curr_cpu;
 	unsigned int phys_id;
 
 	phys_id = cpu_logical_map(cpu);
-	if (IRQ_ALLOWED(phys_id, allowed_mask) && idle_cpu(phys_id))
+	if (IRQ_ALLOWED(phys_id, allowed_mask) && irq_idle_cpu(phys_id))
 		return cpu;
 
 	goto inside;
@@ -243,7 +248,7 @@ inside:
 		}
 
 		phys_id = cpu_logical_map(cpu);
-	} while (!IRQ_ALLOWED(phys_id, allowed_mask) || !idle_cpu(phys_id));
+	} while (!IRQ_ALLOWED(phys_id, allowed_mask) || !irq_idle_cpu(phys_id));
 
 	return cpu;
 }

> 
> Andrea

Andrea