Re: [BENCHMARK] Corrected gcc3.2 v gcc2.95.3 contest results

Ryan Anderson (ryan@michonline.com)
Mon, 23 Sep 2002 10:02:41 -0400


On Mon, Sep 23, 2002 at 08:30:21PM +1000, Con Kolivas wrote:
> Quoting Ingo Molnar <mingo@elte.hu>:
> > On Mon, 23 Sep 2002, Con Kolivas wrote:
> >
> > how many times are you running each test? You should run them at least
> > twice (ideally 3 times at least), to establish some sort of statistical
> > noise measure. Especially IO benchmarks tend to fluctuate very heavily
> > depending on various things - they are also very dependent on the initial
> > state - ie. how the pagecache happens to lay out, etc. Ie. a meaningful
> > measurement result would be something like:
>
> Yes you make a very valid point and something I've been stewing over privately
> for some time. contest runs benchmarks in a fixed order with a "priming" compile
> to try and get pagecaches etc back to some sort of baseline (I've been trying
> hard to make the results accurate and repeatable).

Well, run contest once, discard the results. Run it 3 more times, and
you should have started the second, third and fourth runs with similar initial conditions.

Or you could run the contest 3 times, rebooting between each run....
(automating that is a little harder, of course.)

IANAS, however.

-- 

Ryan Anderson sometimes Pug Majere - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/