2012.04.06 - measurements

i've wrote about the need for measuring. quite a few times actually… it is so obvious, for an engineer, to perform an experiment to “see what happens”… or is it?

it appears that not that many engineers does that after all. that is why superstitions still lives on. “templates are bad”¹⁾, “exceptions have terrible impact on performance”, “if you use exceptions your program has an errors” and a lot, lot more bullshit bouncing here and there, on the internet and among programmers. some time ago i was “charged” of wasting time, while doing measurements!

if you're sure about your opinions, why not to check it with the proper experiment(s)? and if you have measured this 5 years ago, how come are you sure your results still holds? i highly recommend Bjarne Stroustrup's C++11 style lecture, presented during Going native 2012 conference²⁾. for the matter of this text, pay a special attention to the part from 44:50 to 52:48 of Bjarne's the lecture. since on that particular presentation important graph disappeared, along watching see the C++11 Style presentation, starting with page 43. now you have a time to watch it. i mean it. do it now – i'll wait! ok so if you've seen it – what was YOUR first reaction? what result did YOU expect? well – my expectation was wrong too… the funny thing is, as Bjarne noticed, it was so for everyone. times have changed and CPUs cache is now a crucial resource. if your expectations was that the list will be faster around 100-10k elements – it WAS true… about 10 years ago.

what just happened? knowledge outdated. this is normal. 10 years is quite a time, actually… :) and this is the clue of what i want to share – MEASURE and SAVE your test programs for future! even if you checked and confirmed your opinion it does not mean that a new compiler or a newer CPU won't make the results dramatically different – even counter intuitive.

in my opinion each project should have one more directory, along “src”, “tests”, “doc” and similar, namely “measurements”. this would be the location where a lot of (short) “checker” programs will be held, that check a number of means to solve each problem, team has decided to measure. this can be memory usage, run time or even as simple as output binary size. with rational description (simple documentation) on what and how is being measured, this can be used to periodically ensure that foundations of solutions being used still holds.

think about it as a way of extending an idea of automated testing, for the conceptual (design) decisions. tests are run at least daily, by CI tool. “measurements” tests can be run the same way, but in longer time spans – each time toolchain and/or hardware changed + every half a year (keep in mind, that if you're supposed to do it manually and rarely, you probably will forget about it ;)). cycle is simple: measure, save, re-measure and compare. results can be astonishing…