Test Metrics to die for

Abhimanyu Grover
April 7, 2011

Yes! there are a few of them you would want to die for. Here are some examples from our Test Collab application in order of their importance (but order varies for different type of organizations):

1. Average time taken per test execution:

This tells you efficiency of testers. And a sudden rise or fall can be a bad news for software managers.

Why is it important?Keeping close eye on this will help managers knowing if something is changed abnormally in the project or if suddenly there’s a communication problem in team or if just testers are not working productively.

2. Defects Reported:

Total number of failures/defects in project reported since creation.

Why is it important?
Every defect reported from your test case management tool is a cost expense which needs to be paid before you deliver. Keeping in close touch with this metric will keep you connected with project’s target date and budget and make smarter decision when need arises. Sudden rise of this metric means your project health is coming down and your developers might not be doing a good job.

3. Average success % per execution:

No of passes / No. of total test cases executed * 100 (Ideal value: 100%)

Why is it important?
This metric tells you what % of cases are passed and failed on average. Higher value indicates the project is going great and there’s a great communication between developers and testers. Rising value indicates team is improving itself and paying more attention to the quality. Declining or a low value is bad sign and that’s when a manager should jump in to take necessary steps to fix it. Make a rule for lowest value in your organization and tell your developers to maintain this. If they don’t jump in to find out the reason.

4. Test Cases:

Total number of test cases in a project.

Why is it important?
A greater number of test cases usually suggest that team has worked well identifying what to test as end-user in a particular feature or the whole project. A greater number early while development and with slow rise during development also suggests that project is in good direction.

5. Test Executions:

Total number of test executions in project. Don’t forget that each execution contains various set of tests.

Why is it important?
At lower level this metrics tell you how much testing has been performed on a project, not in terms of the time but total number of instances. Comparing this value with your other projects can suggest you if there were greater/lesser effort spent on testing. And sometimes, depending on project outcome, you can acknowledge if this worked in your favor or against (even though a project failure can be occur by various reasons).

6. Time spent on testing:

Total time spent on testing in a project.

Why is it important?
For very obvious reasons. This can tell you how much time you’re spending on testing (excluding your test management time though). You can compare this with your planned cost of testing or cost of project. If that tells you that you’re spending 60% of allotted time of a project in testing, you can focus on improving your testing process and thereby reducing the time.

There are several other important metrics which you should not forget:

1. Defects per size: Defects detected / system size
2. Cost to report a defect: Cost of testing / the number of defects reported
3. Defects detected in testing: Defects detected in testing / total system defects
4. Defects detected in production: Defects detected in production/system size
5. Quality of Testing: No of defects found during Testing/(No of defects found during testing + No of acceptance defects found after delivery) *100
6. Effectiveness of testing to business: Loss due to problems / total resources processed by the system.

This list covers some major standard testing metrics. There can be several metrics in your organization. So which metrics do you die for? We’ll appreciate your input.

P.S. See also how we do it in Test Collab: