Magazine 'Tool Tests'

[russ bransford]

This is sort of a follow-up to Gary Herrmann's thread of 'congrats dino'.

How much stock do you put in a mag's tool test? Are there certain mags you 'follow' and certain ones you stay away from?

I think Dino appreciates the publicity, but it seems that many times the comparisons don't measure up to your experience. Of course right now I'm speaking about the ez guide. for those who have the ez guide and have the seen the Wood mag test, what do you think?

russ

[Dennis McDonaugh]

I have mixed feelings about tool tests. I think the sidebar boxes listing pros and cons of tools are usually pretty accurate in what they describe. However, most tests are done over too short a period of time to identify all the pros and cons you'll identify after using the thing for a year or two. There's usually no question over the over the "Top Tool" or "Best Value" either. Basically, you get what you pay for. Buy Lamello, Festool or Fein and you probably don't need the tool test in the first place. Where you really need help, and aren't likely to get it, is when you want to select a tool from the Portable Cables, DeWalts, Milwaukees and the like. All of these manufactures produce "pro" quality tools and every one of them has produced a clinker or two in the past. The magazines are no good here because they just don't use the tools long enough to discover the PC 7529 had a bad switch setup or the PC 310 Trim router base wasn't perpendicular to the shaft on many of the production routers. I'm not picking on PC, these are just two of the tool mistakes I've bought. I still look at the reviews if I'm contemplating a new tool purchase, but I just use them to weed out the obvious clunkers.

[Jeff Sudmeier]

What Dennis said!! In rare occasions do the mag's find faults with the tools, in other words wearing them out. They will tell you out of the box which is the best, features, power etc... but not long time wear. It would be a much bigger process for them to do that. (Could get a lemon from the company so you have to test more than one of each model, etc).

[russ bransford]

yeah, i agree with the idea of testing over time. it just seems that with many comparisons and sales pitches you end up getting more features than functions. i realize i may be playing with definitions, but in my thinking, salesmen use features and try to be one-up on the competition. but after you have used the tool a while you discover the real functions of the tool.

it may look nice in the pitch or promo but over time you find out some things not mentioned in the comparison that are great and then again some of the mentionables aren't worth repeating.

some tools are so different in their functions that comparing features doesn't do it justice. I think that's what Dino is running into when the ez is compared to other 'similar' items.

I do like some tests in 'tools of the trade'. they seem to have a pretty good way of testing in the real world, and they usually admit their bias'.

[Gary Herrmann]

Some of them especially seem to think very highly of all tools they test. I like the Fine WWing Tool comparison issue, but mostly to be able to compare specs easily.

That said, I posted on EZ Smart because I've been very happy with it, and despite what I just said above, their eval of the EZ Smart agreed with my own. The other 2 piece guides I've used flexed way too much - I couldn't get a straight cut over 8 feet without clamping or something else.

It's like car reviews. Your own experience will wind up being your best guide.

[Jim Becker]

I've had the pleasure of actually watching some of the process that happens at WOOD Magazine when they do tool testing. It's hard work and they are really careful about documenting things and keeping the playing field level. In all of these things, I think they and most other magazines do a good job.

But I also have to agree with the sentiment that testing needs to go beyond the immediate and longer-term commentary is critical to do a complete job. (Something I like about certain groups that do testing on say, automobiles) Of course, with many tools tests, the models change often enough that this might not work out as well as with cars...and it takes 6-9 months from the beginning of a testing or re-testing effort to publication. The lead times are brutal.

[Harry Goodwin]

I just read one of these magazines to which I subscribe and they are showing ways to save money and have people recommending straightening nails, drier sheets used for finishing and others and I am goiing to buy a recommended tool Ha, Ha. They are still great magazines. Harry

[Jim Becker]

Harry that article is a compilation of posts from their online forum by woodworkers "like us" after a question by the "world famous" Buck Nall...a very interesting fellow and friend...about how folks are thrifty around their shops. The examples are not recommendations of the magazine.

[Steve Clardy]

I haven't actually read the article, as I dropped Wood magazine some time ago.

I have Dinos system. No complaints here, just compliments for his system.

Steve

[Dev Emch]

I read them, sometimes, but I dont put a great deal of value into them. The mags are bound to following the sponsors who buy advert space and sponor shows. Thus there is often a lack of objectivity. Also, many times, the top end entries are never reviewed. Of late, I have seen very little comparisons on the made in canada general machines. Often when you see a table saw comparision, you will see the entry from General international and not the General of Canada. Another case in point although there is a lack of electrons here..... Plane reviews often stop at Lie Nielson. What about the rest of the super planes out there? A Holtey, for example, can smoke the pants off a Lie Nielson any day of the week. Yes, holteys are custom made and carry a price tag to beat the band making them not affordable to most. But, they are an option and frquently, its an option not covered. When asked, the mags will say that these items have been left out because then are not in alignment with our reader's demographics. O.K., I can see where they would hold such a belief but shouldnt we be the judge of that? Isnt that not being totally objective to the comparision? So that is why I read them but dont allow them to sway my thinking one way or the other.

[Howard Acheson]

Here is something written by the then Woodworking Products Editor of Wood Magazine responding to a comment about advertiser influence in the tool test results. It addresses many of the issues raised here.

QUOTE

Nothing that has been said so far has changed your opinion that WOOD magazine reviews are somehow influenced by advertising schedules. So I don't expect that I'll change your mind either. But, for those of you who are still listening, let me explain about our tool-testing procedures.

To keep the tests manageable, given the typical 6-page allotment, we must limit the number of models we test to a handful. That may be 6 tools, it may be 10 tools, depending on the complexity of the tool and where we can draw a meaningful line (might be price, might be power, might be capacity, might be a combination of those or others).

Tools are usually acquired 10-11 months before the cover date of the magazine. (Right now, I'm acquiring product to test for our June 2002 issue.) Unlike other magazines that use their editorial staff as testers, we use ndependent testers who are given 1-2 months to complete their tests and
provide the results of the tests.

If we find serious problems with a tool, we report them to the product manager of the manufacturer for their response in the magazine. The article is written from the tester's notes and completed 5-6 months before the cover date. Most of the advertising for a given issue comes in about 2-3 months before the cover date. (It takes about a month for the magazine to actually be color-separated and printed, and the issue usually come out the month before the cover date.)

Neither the manufacturers nor our advertising salespeople know who will get the recommendation(s) at the end of the article. They do know, however, that we'll be reviewing Tablesaws Under $900, for example, and which specific models are in the test.

Frankly, as an editor, I could care less who "wins" any given tool review. My responsibility is to our readers, not the advertisers.

Dave Campbell
Woodworking Products Editor

CLOSE QUOTE

Howie..........

[Steve Rowe]

I've got to agree with Dennis on this one. I place essentially zero stock in magazine reviews of tools, including their editors choice, best value, best buy or whatever. Often, it is difficult to distinguish what is fact from what is the authors opinion or perhaps even fiction or hype. Much of what we as users like (or dislike) about tools is subjective touchy feely stuff that is unique to the individual user. Additionally, we are biased by our own habits and preferences. The same goes for the authors of these articles although often they don't appear to acknowledge this. There are obviously some exceptions to this but, overall very few. I am often amazed at those who post on some forums about they were ready to purchase a particular machine based on some review but then read a different review that reached a different conclusion and now don't know what to do. Sometimes you just have to go with your gut based on your own research and needs and sometimes you make a clunker of a decision and that is the school of hard knocks. Such is life.

[Jim Becker]

Thanks, Howie. Dave is a friend of mine and he's the one who took me through the process when I visited a couple years ago. He's as honest as you could ever find a person.

[Dev Emch]

Thanks Howie for that quote.... Unfortunately, the issues still remain. Clearly the reviews need a structured settlement in order to be reasonable and we accept that. But at the same time, you are excluding options that may be better suited to many readers.

I personally believe strongly that many woodworkers should follow the route layed down by Lou Sansone or myself and others. That does not always mean constructing a stand alone shop of Lou's caliber; however, it does not exclude many with a basement shop from getting "eccletic" (sp?) tool collections. I love that word even though its hard to spell

What really bothers me is that our high schools no longer train folks in woodworking and many newbies to the hobby are drawn in through the mags. Forums such as this really do open the eyes of many as to what is out there and how it all works. Shop tours, tool reviews and project pictures really go a long way to introducing newbies to the hobby. Having said that, the mags, by their own choice and article aritecture, focus the attention of knowledge hungry newbies in a limited direction. If you have ever attended a Woodworking Shows show in an active town, you will know how dynamic this force can be. I have personally seen Lie Nielsen Co sell $4000 to $5000 dollars worth of product in a two to three hour time window on an opening Friday afternoon!

My only point was that too often, the top 20 percent of all tools, both hand tools and power tools, are never covered by these reviews. Show me the last Fine Woodworking review of a martin table saw or martin jointer. Where is the holtey review from wood magazine? Etc. Etc. Thus, we should all take these reviews with a grain of salt knowing that the cream has been skimmed and will not be reviewed. Those reviews fall into the laps of folks who blog them out on forms like this.

[Peter Pedisich]

The things I find most useful about reviews and comparisons is when they bring up stuff like "the handles on the Milwaukee were thinner than the Bosch and more comfortable after an hour of use". This at least lets me decide, hey, my hands are the size of a mountain gorilla's- the Bosch is for me.

I buy based on a few things:

1) Hold it, if it feels right it is.
2) Has this brand been good to you in the past?
3) Make a list of pros and cons and decide what is important and what's nonsense. Only you can make this list. Don't be swayed by the printed word unless it's your own printed words.

Pete