What BookScan Measures, and What It Doesn't

I found myself typing a long comment on this post over at The Beat blog -- about Colleen Doran's discovery that BookScan has been horrible at measuring the sales of her books -- so I've ported it over here as well. I think it will make sense out of that context.

Sure, it's true that mainstream newspaper reporters and other ill-informed people state that BookScan covers 75% of the book market as if that means all kinds of books everywhere -- but why should we listen to ill-informed people?

BookScan only covers the US, for starters -- many books will have a substantial life outside of that territory. And it always has been very clear about which accounts report to it and which don't -- it covers most of the largest retailers of books (Wal-Mart is joining as of the beginning of 2013), but does not include the mass of small bookstores, nor does it cover all of the other outlets that sell books sometimes, or sell a few specific niche books. And, as others have said, it doesn't do a good job of covering sales to libraries, either.

So there are publishing areas where BookScan is less useful: craft books, some travel categories, anything that gets picked up in large numbers by non-traditional retailers (like those books in Starbucks), and, yes, graphic novels. What it does cover well, not surprisingly, are the core areas of major publishing: fiction and narrative non-fiction, the kind of books that dominate sales and keep most of the big publishing companies afloat. For the vast majority of those titles, the BookScan rule of thumb (add about 25% to account for libraries and indies) works well, and it's thus the only way publishing people can accurately gauge how books from other publishers are doing. (Nobody in publishing uses BookScan to check out their own books' real sales; we all have internal systems for that.)

The smart publishing people who work in the areas that are less well covered by BookScan know that (and complain about it, of course, since it makes their jobs harder), and they make allowances for those other markets. In the case of the DM, relatively accurate numbers are available separately from Diamond, so anyone looking to sign up a comics project with any hope of hitting a comics-shop audience would have to be an idiot not to check the numbers in that channel. I'm not claiming that those idiots don't exist, of course.

I think what Colleen Doran has really unearthed is that there are stupid, badly informed people -- some of them (at least momentarily) in positions of power at publishing houses. (And many of them writing for major newspapers like the Globe & Mail.) Those people don't understand what BookScan really measures, and use it wrongly. That's not the fault of the tool; it's the fault of the idiot using the tool.

Since she came out of the DM world -- and most of her projects are focused at that world -- it's only to be expected that her books would sell primarily in that world. Since BookScan does not, and has never claimed to, include sales figures in the DM, her numbers are vastly more inaccurate than in the genres that BookScan was made for. That doesn't mean that anyone is lying or misrepresenting things; it means that BookScan is a very bad tool for calculating the sales of products that mostly sell within the DM. And we all should have known that already, since that's not what it measures.

This all seems analogous to looking at a thermometer in your living room and complaining that it doesn't tell you how cold it is outside.