The world of made-up statistics

 In my mind, it used to be that statistics were based on data. Data existed, and you extrapolated statistics from that data.

In the world of the internet, everyone has a voice. And while generally that's a good thing, sometimes that privilage is abused andthat voice comes with the amazing ability (and willingness) of some people to make up data and statistics.

While it probably has always been this way, our ability to fact-check this information has never been greater. Unfortunately, with media outlets making severe cuts to news crews, cutting investigative journalism, and apparently fact checkers, it almost seems more common for made-up statistics to make their way around the internet and via twitter. One such "stat" is the idea that "Only 1 in 600 pit bulls finds a home." "The offspring of one unspayed cat will, within five years, will have 420,000 cats" is another.

Some of these are commonly used. They've been passed down for years by rescuers in the same way that urban legends are passed down. Most people don't mean to be malicious with their false statistics, but that still doesn't make them true.

But some people ARE malicious with their claims -- and those claims can also be just as made up.  A couple of weeks ago a lawyer's blog popped up in my news feed.  In it was yet another "statistic" that if someone thought about it for even a minute they would realize couldn't possibly be true. But there it was, in all its glory.

"Together, pit bulls and Rottweilers account for half of the 4.7 million dog bites reported in the United States in 2011."

Wow. That's quite the statistic.

Well, after one of my fellow advocates did a little sluething, here is the source for this astounding statistic.

The first, the 4.7 million dog bites, is the per year estimate routinely given by the Center for Disease Control and Humane Society of the United States. The source for this is actually a study from 1994 in which they garnered national "statistics" from 5,300 phone interviews that not only had a small sample size but also just "made up" data for age groups that were not accounted for in the study. I don't necessarily fault the author of the article for not tracking down the source (I do fault the CDC and HSUS for repeating it for the past 2 decades as if it is a reliable stat), it is more than a little dishonest to attribute this estimated number, from a study done nearly 30 years ago, as the exact number of bites that happened in the specific year 2011.

Meanwhile, datapoint in the second part of the "statistic" is apparently Merritt Clifton's 30 year report on "maimings and maulings".  Clifton's data has often come under scrutiny on this blog, and Luisa did magnificent work on this several years ago here, here and here. In this case, the source is a huge miss.

However, even if the data source wasn't bunk, the data she used isn't even related to the first data point. The Clifton Report that was cited featured 30 years worth of data, not one year. And the report never professed to cover all dog bites. At its best, it was only meant to cover the worst-of dog bites (at it's worst, it's completely flawed in both its methodology, its record-keeping, and being completely non-comprehensive). 

So, instead, the author of the blog reported two pieces of information, that were not related in any way shape or form, one with data from the past 30 years, and one that was literally from 30 years ago, and tacked them together for one "statistic", for one year, in 2011.

It's wrong. And I believe done to be malicious, fear-monger, and drum up business.

Accuracy is important. And making up data to create fear-mongering is not accuracy. But unfortunately it does represent the world of made-up statistics that we live in. 

So please everyone, if you are using data from the internet, PLEASE consider your sources wisely.

Comments

[Katzklaw]

thanks for pointing this out. i reported the lawyer's page complaining about the bogus statistic. maybe, hopefully... it will get edited.

[Randy]

Excellent point which I have considered myself. Being a bit older I can remember a time when as a general rule you could "trust" that most data from a reliable news organization could generally be trusted and had been vetted so to speak. However today with all the citizen reporters out there it is easy for false data to spread and the general internet public appears to trust such data and does little back checking. No matter which side of an arguemnt you are on I have seen false or questionable data thrown around as if fact. Guess it is a brave new world and "buyer be ware".

[Joel]

Unfortunately, "truth" is whatever we want to believe.

When presented with a "statistic", if it is something that fits with our view, we ask ourselves "Can I believe this?" If it sounds plausible, we accept it as truth.

If it doesn't fit with our view, we ask ourselves "Must I believe this?", and go on a quest to discredit it.

Best practices in logical thinking would encourage us to test the veracity of any "fact". But it's just not the way humans are wired.

[Brent]

Randy -- I used to be that way with news organizations too. Now I wonder if they've always been wrong, or if we're just now aware of it because there are more options. I've actually seen some citizen reporters who are more accurate than mainstream media -- and yet, the other extreme happens there too.

Joel -- agree on the "can I believe this?" thing. I always call it the "sniff test" -- does it sound plausible? Often things just don't pass the sniff test and make me look into them further (which is where I was on the 1 in 600 pit bulls item I noted above). And this stat simply didn't pass the sniff test because for it to be true, there would have been 2.35 million pit bull and rottweiler bites last year...which would have been about 1 out of every 3 or 4 dogs of these types being involved -- which is just unfathomable.

[Caveat]

Another point re the 4.7 million dog bites: They were not "reported" dog bites in Sacks et al's what-if paper. They were an estimate of the number that could possibly be happening which is very different.

I've even seen this number presented as bites requiring medical attention, which is even more ludicrous.

I just wish some of these silly myths would die a natural death, instead of having to be debunked every single time.

Also, you forgot the scare quotes around "data" when referring to Clifton's incoherent nonsense LOL

[Joel]

I would also add that as idiotic as some of the "facts" are that the dog bite loons like to trot out, efforts to discredit them are compromised when silly facts like these are used by the other side.

[Randy]

Bret-Valid point, maybe the good old days were not much different we just did not know better, lol. While I have not studied it a lot real data in the animal welfare world seems very limited and hence eaisly miss used. I know there are some groups that claim to know the shelter population and hence what can be done (typically by adoptions) to eliminate shelter over crowding. Yet I can find no reliable data on the subject. Evenif it was good data it would not account for the number of strays or ferals which are intakes to the shelter too. As for bites I believe you made the comment that they seem to hit the headlines. True I guess and not to minimize the severity but my guess is deaths due to toxic chemicals in the houshold far exceed the deaths from dog bites yet one never hears about it.

[Jen Brighton]

I get frustrated with my husband when I tell him about a study I read and he won't accept the "facts" until I cite all the references and whether it was a peer-reviewed study. But I guess we all need to ask ourselves the same questions.

As a law office employee, I am shocked that a lawyer would actually list such silly statistics. Where in the world did she get her information to come up with the half comment? DBO, perhaps? Makes me glad I don't work for her! She should be embarrassed.