Is your idea good? [was: Re: LTT for inclusion into 2.5]

[Larry McVoy]

I don't mean to pick on LTT, I haven't used it, it may be the best thing
since sliced bread.

I can tell you how to present this and any other feature similar to this
in a way which would make me a lot more willing to accept it, which
presupposes I'm doing Linus' job which of course I am not.  However,
it's likely that Linus has similar views but he gets to chime in and
speak for himself.

All of these tools/features/whatever add some cost.  The cost can be 
measured in lots of different ways:

    - lines of code
    - lines of code which can't be configed out
    - call depth increases
    - stack size increases
    - cache foot print increases
    - parallelism (think preempt)
    - interface changes

I suspect there are other metrics and it would be very cool if others would
chime in with their pet peeves.

What would be cool is if there was some way to quantify as much as possible 
of the accepted set of costs so that that could be balanced against the 
value of the change, right?

The one that always gets me is

    "I've added feature XYZ, I benchmarked it with <whatever, usually
    LMbench> and it didn't make a difference"

That is almost certainly misleading.  The real thing you want to do
is quantify the actual costs because there can be non-zero costs that
do not show up in benchmarks.  For example, suppose that the benchmark
neatly fits in the onchip caches and it only uses 1/2 of those caches.
Your change could increase the cache foot print to just fill the caches,
the benchmark says no difference, you declare success and move on.
The problem is that almost all changes are good enough that they match
this description.  Measuring them in isolation doesn't tell us enough.
If I combine two changes, both of which use up 1/2 the cache, there is
no longer any room for anything else in the cache.

I'd love to see a trend where patch requests for any non-trivial patch
included before/after data for the above metrics (and any others that 
people see as useful).  I'd love to see some people taking just one of 
the above and making a tool which measures that metric.  Then we combine
the tools into a "patch measurement suite" and start prefixing patches
with

    Code changes:
	[phone] = -4444	(all code)
	+123 -567 = -444	(all code subject to CONFIG_XYZ)

    Call depth:
	+2 for read()
	+2 for write()
	no change for all other system calls

    Stack size:
	+2099 bytes for read()/write() path

    Cache misses:
	No change for benchmark1, 2, 3
	12,000 data read misses for lat_ctx ....
    
    Etc.

What does the list think of this?