Re: [BENCHMARK] Corrected gcc3.2 v gcc2.95.3 contest results

[Con Kolivas]

Quoting "Richard B. Johnson" <[e-mail]>:

> On Mon, 23 Sep 2002, Ryan Anderson wrote:
> 
> > On Mon, Sep 23, 2002 at 08:30:21PM +1000, Con Kolivas wrote:
> > > Quoting Ingo Molnar <[e-mail]>:
> > > > On Mon, 23 Sep 2002, Con Kolivas wrote:
> > > > 
> > > > how many times are you running each test? You should run them at
> least
> > > > twice (ideally 3 times at least), to establish some sort of
> statistical
> > > > noise measure. Especially IO benchmarks tend to fluctuate very
> heavily
> > > > depending on various things - they are also very dependent on the
> initial
> > > > state - ie. how the pagecache happens to lay out, etc. Ie. a
> meaningful
> > > > measurement result would be something like:
> > > 
> > > Yes you make a very valid point and something I've been stewing over
> privately
> > > for some time. contest runs benchmarks in a fixed order with a "priming"
> compile
> > > to try and get pagecaches etc back to some sort of baseline (I've been
> trying
> > > hard to make the results accurate and repeatable). 
> > 
> > Well, run contest once, discard the results.  Run it 3 more times, and
> > you should have started the second, third and fourth runs with similar
> initial conditions.
> > 
> > Or you could run the contest 3 times, rebooting between each run....
> > (automating that is a little harder, of course.)
> > 
> > IANAS, however.
> > 
> 
> (1)	Obtain statistics from a number of runs.
> (2)	Throw away the smallest and largest.
> (3)	Average whatever remains.
> 
> This works for many "real-world" things because it removes noise-spikes
> that could unfairly poison the average.

That is the system I was considering. I just need to run enough benchmarks to
make this worthwhile though. That means about 5 for each it seems - which may
take me a while. A basic mean will suffice for a measure of central tendency. I
also need to quote some measure of variability. Standard deviation?

Con