Bugs / KLOC ratio is useless (OR How to avoid snake oil merchants)

For the past few months, there is a lot of talk about software security metrics, even a book is out One very common metric is the identified bug per (kilo, mega, whatever is applicable to the size of your code base) lines of code. This method is seriously flawed. Why the astute reader may ask? Let’s demonstrate by example. Consider the following (rather contrived) C snippet

/* for the sake of argument let's assume that msg is controlled by the user and is gotten securely */

printf(msg);

/* ... rest of the program ... */

/* then someplace after our evil printf call */

printf(msg);

So Mr. Johnny Security Consultant gets a fat fee to run his favorite static analysis tool. The tool correctly states that there are two format string bugs. If the program is 100 lines of code, then we get a bug/LOC ratio of  0.02. Each of this bug might potentially lead to a complete application compromise.

Mrs Jane Outsourcer then steps in and does a bit of refactoring

/* this is contrived but it does demonstrate the point */

void myprintf(char *msg) {

printf(msg);

}

/* a bit within main() */

myprintf(msg);

/* a bit later in the code */

myprintf(msg);

Johnny charges yet again his fat fee and is happy to write in his report that the application now contains only one critical bug and the ratio has drop to 0.01.

Is the application any safer? No!

Is it easier to fix? In our example, perhaps.

Will a run of the mill static analysis tool fail to state that the application has still two vulnerable points? Yes

Using pure bugs/LOC can lull the development team into a false sense of security, especially if the program is not something trivial like the one above. Software security is a weakest link model, an application containing one critical vulnerability is not safer than one containing ten critical vulnerabilities.

So next time someone tries to sell you some software and spouts some statistics in your face, you will have a counterpoint.