Product Reviews Require Lots of Salt

I have been introspecting (I'm pretty sure that's not a word, but I'll bet everyone knows what I mean) over the last couple of days, and I'd like to offer some observations.

Having been involved in amateur radio for 36 years, and woodworking even longer, and having subscribed to a number of publications in those years (I'm a life subscriber to two amateur radio pubs, and have every issue of Fine Woodworking) I have learned to temper my enthusiasm over product reviews a good deal. I mention amateur radio, because if you folks think we woodworkers spend money on tools, Hah!. I have a transceiver that cost more than a Unisaw. I have an amplifier that cost more than a Felder. Add up the towers, antennas, and other accessories, and we're talking a pretty nice shop. I won't mention the home theater system that is worth three Roblands. I want to be well informed spending those dollars, too.

None of what I will say here is necessarily a direct indictment of any publication's policies, but no one should ignore the very real fact that most magazine revenues obtain from advertising. It is naive at best to assume that a really poor product will get a concomitant review from a publication that curries their advertising dollar, although every publisher known will argue that huge dollars do not assure great reviews. And don't discount the dartboard approach, either. You know, it's Brand X's turn for a good Best Buy rating. Your observations and experience will best shape your confidence in the independency of the reviews.

Moreover, I have observed in countless product reviews that my head nodding deviates from the vertical in inverse proportion to my personal opinion of the product in question. That means that when they say something I like is good I proclaim what an excellent review they have done. If they pan something I don't like, I agree with them. But if they pan something I like or praise something I don't like, I harrumph (I think I've transfigured that into a verb), “that wasn't very well done,” or something to that effect.

Let's face it. We all have buyer's bias. We don't want to think we got taken by any purchase we make, and continually seek reinforcement that we made a good buying decision. Someone mentioned that the pool of knowledge is quite helpful in making those decisions (much the purpose of the various woodworking forums) but they are all based on that same fear of buyer's remorse, with the addition of some real application experience.

While we can't entirely discount that real application experience, don't forget that owning a one-off of any product generally limits our experience to that product, and effectively insulates us from comparison to a potentially better one. That, we hope is the role of the product review. With disposables, such as router bits (as opposed to Unisaws, which are hardly disposable), that knowledge is important, since although we may have had success with an original purchase, we wouldn't mind a better one the next time, if it exists.

My hope for all of us is that we take our product reviews with large grains of salt, accept happily the raves of products we like, question the accuracy of the reviews we don't, but sagely refrain from absolute confidence in their findings.

Someone pointed out that when the test disagrees with a significant body of knowledge extant, the test must be questioned. I concur. I'm not sure how to unring this bell, but here may be another function of the internet. Those of us that have had success with certain products will continue to defend and recommend them, and the dogs of the market will continue to be identified by those that have had them fail. The marketplace will solve this problem.

So, I was happy to see the rags agree with my assessment of the Bosch 1584 and 3915, the P-C Omnijig, Senco nailers, and the mighty PM66. I still don't understand how they could have so missed the target on some of my other fine tools, like my Freud and Jesada bits.

Finally, just remember that those personal biases are pretty strong. After all, they still sell Chevies, Fords, and Chryslers. You can pick your favorite and keep wondering how those other two stay in business.

Last updated: 27 January 2009