I have came across this phrase repeated during my career from different people. It has a few forms

• You can’t manage what you can’t measure
• You can’t improve what you can’t measure
• By measuring something it will automagically improve

There is a lot of truth in this.

Why? There are a number of aspects

• If you can measure something you probably have someone looking at it. So it gets visibility and attention.
• People like seeing progress. If you can measure something, you can see your score get better (e.g. it’s nice to see the Out of SLA bug count to go down).
• Measurements make goals more objectively measurable. Goals help put attention on progress.

You can probably double or triple the above list yourself pretty easily.

What does this mean in practice day to day at work?

• If you want to get improved code coverage of test cases, invest in gathering the metrics.
• If you want to get more accurate at software estimates, make sure you capture old ones made so you can look back on them.
• If you want to get the Out of SLA count down, make sure you can display them easily.

You may say the above is not that deep – and you would be right. These were simple examples. Where it gets more interesting is here there is not an easy way to measure something. For example, I would like to improve the skill of writing “good” test cases. But what is good? Is it coverage? Not by itself – you can have great line coverage and poor test coverage. You need to first decide what is good then come up with ways to either directly or indirectly measure your progress.

For example, one indirect measure of effectiveness of test cases would be to measure the number of times a new feature is rolled to site with production bugs being reported in the new feature. This is fairly easy to measure, but is not a perfect measure of “goodness” of test cases.

Another measure might be how many unit tests break if you change one class. One key definition I have for a good unit test is the test ONLY tests the functionality in the class under test – not anything it calls. If the test case includes behavior not in the class under test, then the test case will need reworking whenever the dependent code changes. This is generally bad. The whole purpose of unit tests is to allow code to be refactored with confidence. Having to update lots of test cases of other classes when you change one class is an indication of poor isolation of unit tests. (Note: I am talking unit tests here, not integration or system tests.) But how to measure? Automated measurements are a lot better because they don’t rely on people not being lazy. Automated measurements are also more objective. But this measure may require infrastructure to be built where every commit reruns all unit tests, then the number of changed classes in a commit is compared to the number of broken unit tests. This however is not a good score since all unit tests should be fixed before the code is committed. This is an example of a hard measurement to collect.

Monitoring of production systems is another example of measurement systems. Not knowing what the system is doing is not acceptable. It is necessary to measure and analyze performance trends if we are to forecast growth trends to make sure we have hardware in place in time for future load.

So in summary, measurement is definitely useful. If you think something is important, think about how you are going to measure it. Sometimes measurement is easy. Sometimes it is not. Just because it is hard, this does not mean it is not worth measuring.