THE BOOK--Playing The Percentages In Baseball

Tuesday, June 09, 2009

Another terrible job of research/writing by a USA Today writer

There was an article in Monday’s USA Today headlined “Pilots in crashes had failed multiple tests.”

The gist of the article is the following:

In nearly every serious regional airline accident during the past 10 years, at least one of the pilots had failed tests of his or her skills multiple times, according to an analysis of federal accident records.

In eight of the nine accidents during that time, which killed 137 people, pilots had a history of failing two or more “check rides,” tests by federal or airline inspectors of pilots’ ability to fly and respond to emergencies. In the lone case in which pilots didn’t have multiple failures since becoming licensed, the co-pilot was fired after the non-fatal crash for falsifying his job application.

I mentioned this before in another thread, but (obviously) a statistic like that is meaningless unless we have some context or a baseline.  The example I always give to illustrate that is the commercial for some car that says, “75% of our cars are still on the road after 10 years.”

Without knowing the rate for other similar cars, that number (75%) means nothing of course.  Wouldn’t it be hilarious if their competition was 80%!

Anyway, I was reading this article this morning and just waiting for the statistics on how many pilots fail these tests and how often for all flights that don’t crash!  It was nowhere to be found!  Plus, even if it is less (the percentage of pilots who fail X number of tests on flights that don’t crash) we also would want to know if the number found in the 9 crashed was statistically significant.  After all, 9 crashed is not a very large sample size.  Plus, the author should certainly tell us what the pilot falsified on his application - it might have been completely irrelevant to his piloting competence.

What a piss-poor article - again.