Don’t Juke the Stats

Imagine a police department that closes a case if they figure they can’t solve it. Say this department tosses away criminal complaints if the suspect is already accused of another crime. And they upgrade misdemeanors to felonies to make sure they get solved.

That would be one messed up police force. It might be entertaining as a TV show (but maybe not — imagine Cold Case if every episode consisted of “Wow, this case is really old. Case closed.”)

This is often how bug databases are operated, though. Bug reports are closed because they’re duplicates, because there’s no plan to fix the bug, because the product was released (seriously, I worked on a game project where the publisher marked all bugs as closed when the game shipped), basically because everyone likes seeing bugs closed (except for the bug submitter who didn’t get a fix). And I’ve seen bugs upgraded from minor to critical to raise their priority as other critical bugs got fixed. This isn’t a bug database — it’s a to-do list.

A bug database is supposed to represent the truth of the system. It’s your weapon against wishful thinking (I presented one of my bosses with the open bug count when he kept trying to cram new features into an impending release — that was enough to get him to back down). It’s the history of the product development — each bug is a story that shouldn’t end abruptly with “and they lived happily ever after. The End.” And the bug database is your source of statistics. You should see bug reports spike up and ramp down during a project. But they should be real statistics. To borrow from another TV police drama, The Wire — “Don’t juke the stats.”