Response to (Bini) San Antonio Dog Bite Study -- Creating statistics to fit a point of view

Earlier this week, several media outlets began covering some news of a new "research" paper that ran in the April  issue of Annals of Surgery that highlighted 'pit bulls' as being more dangerous than other types of dogs (the link in that sentence goes to the San Antonio Express News -- who actually did a somewhat fair job of reporting on the research).

I debated on this for a bit -- because at first glance, the paper seemed to be a little shaky and I wasn't sure if I should even highlight it or not. But upon reading the paper in its entirety, I realized just how poorly researched the paper was. 

And I think completely highlights how if you go into a research paper trying to find statistical information you can probably create it in a way that supports your point of view.

The "research paper" begins with telling story of three fatal dog attacks in the San Antonio area over the past 15 years.  The stories all involve 'pit bull' type dogs -- but the stories seem to leave out important details -- including that one of the incidents involved a 7 month old child (not 11 months old as the paper indicates) that was left alone with a dog with a bite history and in another that involved a 10 year old girl that went out to rescue a chained up 'pit bull' that was being strangled by it's chain as it got hung up on a fence and during the girl's rescue attempt the girl was bitten by the scared/injured/choking dog.  Some very important details in both of these incidents were left out of the "research" paper to make it seem more like specific "pit bull" related deaths than circumstantial (and tragic) attacks).

At this point, I want highlight an earlier post by Jim Crosby -- who highlights some of the details of those incidents that were ignored by the paper, as well as some glaring errors in the paper -- all of which are worth reading and I don't want to repeat most of his work here.

After one of these incidents, Dr. John Bini of the University Hospital began a 'research' project at his own hospital with the objective (taken straight from the paper): "We postulate that patients admitted to a level I trauma center with dog bites would have severe injuries and the gravest of injuries would be caused by pit bulls".

And if you go out looking to prove something, you might be able to create the statistics that support your point of view.

The 'study' then looks back 15 years worth of hospital data to support its claim.

Over the course of 15 years (Jan 1994 to April 2009), the hospital had 228 dog attacks that fit their criteria. That is roughly 15 dog bites per year that required medical treatment - -roughly one every 3 1/2 weeks. I think this important to put the size of this problem in perspective:  That's one case every 3 1/2 weeks.

Of those 228 victims, breed of dog info was only available for 82 (35%) of the total cases.  At this point any of the 'statistical' analysis breaks down - because the sample size is extremely low. Even the writers of the study note:

"We should state that our study is limited by its restrospective nature and the limited number of cases in which the breed of dog responsible for the attack could be determined. This lack of information may compromise the validity of our results implicating the pit bull as a major culprit in severe dog bites admitted to our trama center." (Emphasis mine).

This, of course, doesn't prevent them from trying to use the data to implicate 'pit bulls' -- in spite of their own warning of having a statistically invalid sample.

Among the statistically flawed findings:

• Of the 228 dog bite cases seen at the trauma center, 29 had 'pit bull' listed as the breed causing the attack (this is less than 2 cases per year), 53 to all other breeds combined where the breed was known, and 146 of unknown breeds.
• The study claims that 'pit bull' victims had fewer hospital-free days than other victims -- with the average 'pit bull' victim having 22.5 hospital-free days (median 28, range 0-29), where non-pit bull bite victims had 28 hospital-free-days (median 28, range 21-29).  They make the same claim that for ICU-free days --with the average 'pit bull' victim having 25.5 ICU-free days (median 30, range 0-30) and non-pit bulls having 29.7 ICU-free days (with a median of 30, range 23-30).   Statistically, it's important to note that the median is exactly the same for both groupings of dogs -- which means that more than half of all incidents, regardless of breed grouping, had zero ICU days and less than 2 hospital days.  The main difference is the range - where one or two statistical outliers that led to 0 ICU-free days caussd a small swing in the averages because of the extremely small sample size.
• The "study" then uses data from Merrit Clifton (you can find out all you need to know about Clifton's 'studies' here, here, here and here.) and then tries to marry that data with the total number of AKC registered dogs to try to create a relative risk factor for all breeds. What the  "study" doesn't mention is that AKC registered dogs only make up an estimated 1.3 million of the 78 million dogs in the US (about 1.5% of all dogs). So not only is that far from a random or representative sample, the 'study' also fails to mention that the ACK doesn't even recognize American Pit Bull Terriers -- the largest of the 'study's' grouping of dogs at 'pit bulls'.  Thus, 'pit bulls' in this case are going to be clearly under-represented -- and yet, they use this data nonetheless to suppor their position.
• The study, to its credit, acknowledges scientific data that pinpoints that there is no such thing as a locking jaw in 'pit bull' type dogs, and acknowledges that jaw strength data indicates that 'pit bulls' bite with less bite force  (235 psi) than Rottweiler (328 psi)s, German Shepherds (238 psi) - -and certainly less than wolves(400 psi) and Lions 600 psi).

The final conclusion states:

"Attacks by pit bulls are associated with higher morbidity rates, higher hospital charges and higher risk of death than are attacks by other breeds of dogs. Strict regulation of pit bulls may substantially reduce the US mortality rates related to dog."

It should specifically be noted here that at no point in the paper does it ever address the actual success or failures of laws that regulate breed of dog ownership -- and thus, they have no idea whether such laws would actually work. In fact, studies in this regard have repeatedly shown that because major attacks are so rare, diverting resources from dogs that have shown signs of aggression toward dogs that are a particular breed have actually DECREASED overall public safety. But this study never once explores that assumption in its conclusion.

And then, there is this little gem from the conclusion:

"These breeds (pit bulls) should be regulated in the same way in which other dangerous species, such as leopards, are regulated."

First off, it should be noted that 'pit bulls' are a type of dog, making up various breeds, within the species of canines. They are not a unique species at all - -and certainly do not in any way resemble Leopards (Leopards mre closely resemble lions, which we've already noted have nearly 3x the bite strength of 'pit bulls'.

Whatever.

Again, there are a lot of major errors in the study - -and I'll refer you to Jim Crosby again for those.

I think the study gives an almost laughable example of what happens when you let people with a set point of view set out to 'prove' their established point of view vs letting the data guide their point of view. In this case they relied heavily on faulty data from Merrit Clifton (and never seemed to question it) and dogsbite.org (which also has credibility issues with misleading claims) and then relies on their own data -- which they acknowledge isn't statistically valid and has a small sample size that allows a couple of statistical outliers to skew their data.

So in spite of the science pointing to something different, they still let skewed "statistical" data change their point of view -- and even drive themselves to the ridiculous conclusion that 'pit bulls' are like leopards and that a premise they never even research, breed restrictions, might help solve the problem.

Meanwhile, one has to wonder why the mounds of peer reviewed research on the limitations of breed-specific legislation,   studies on the failure of breed specific laws, scientific studies on the differences in canine aggression among dog breeds, similar research (with a deeper data set and a different conclusion) in Colorado, an outstanding article on Dog Bite Prevention from the AVMA,  were all ignored in favor of non-peer reviewed information. It's also amazing that there was never a single mention of any discussions with canine behaviorists, hands on experience with dogs, or even talking to the Canine Research Council.

But hey, why bother reading it if it doesn't support your point of view, right? 

Comments

[kmk]

Oh dear, I read the study and came to the same conclusion as another famous Missouri native - "Lies, damned lies, and statistics". there is so much wrong with the study I don't even know where to begin and I'm NOT going to try and compete with Jim Crosby. and how in the name of Sam Hill do seemingly respectable publications get sucked into studies that don't even belong in the National Enquirer????????

This comment is interesting - "These breeds (pit bulls) should be regulated in the same way in which other dangerous species, such as leopards, are regulated."

back in the 80s it was common to compare pit bulls and pit bull bites to big cats and big cat attacks, but those comparisons have been fairly rare more recently. I remember asking someone (Mike Santos, Randall Lockwood, someone else that's equally stupid - can't remember), "So, you'd rather be in the cage with the Bengal Tigers at the Kansas City Zoo than in my livingroom with my pit bulls?"

Good analyis, Brent!

[Brent]

The reason that comparison is similar to what was said in the 80s is because at least 10 of their resources were dated 1999 or before -- with several of them being very early 90s and at least one going all the way back to 1982. Thankfully most of us have learned a few things over the past 30 years...

[kmk]

Oh, good point Brent. I hadn't quite connected those dots re: the dates of the resources. that comparison came up in the "Hearne" case in Overland Park, KS. Someone testified a "pit bull bite" was as dangerous as a leopard or some other big cat. For crying out loud, never mind the difference in size, the infection rates can't even be compared on cat vs. dog bites.

And that was one of the more sensible things that came out in testimony at that trial! Honestly, civil cases are not like criminal cases. If I ever get picked for a civil trial jury I am going to pay my doctor whatever it takes to write a note stating I can't be there due to potential emotional distress. And that's the truth!

I believe the journal that published this "study" should be renamed "The ANALS of Surgery" because someone clearly pulled this information out of their ***.

[Brent]

I actually seriously mis-typed that the first time through. Must have been Freudian. I don't do much for spell checking (as editing your own work is impossible and editors don't work cheaply and I do this for free) --but I did catch that when skimming through and admit that I chuckled a bit.

[Valerie]

Oh well, so much for the peer-review process...

[Zbyszek]

American Stafford Terrier I have. I live with him now 12 years old and my children too. And now I found out that this is a leopard, and I'm stupid, I thought my dog.

[Lpbowlingdvm]

I didn't even have to read it to know the research was flawed. If they wanted actual scientific data, they should have researched the circumstances surrounding the dog bites--and they probably would have found out that the dog was provoked or the person was unsupervised (ie, a child). I'm sorry, but that's not the dog's fault. Their conclusion is that more damage is done from pit bull bites--well duh, look at the power in their jaws!!! This is such bad science I can't believe someone funded it, much less published it. If they wanted actual data they could use, they probably should have had veterinarians do the study. At least most of us aren't biased towards breeds. Frankly, the doctor heading the paper should be ashamed of himself.

[Lpbowlingdvm]

I'm from San Antonio, so I"ll definitely tell my friends and family to stay away from university hospital. If they can't do science, they probably can't do medicine either.

[Brent]

DVM,

The funny thing is, in the "study" they acknowledge that pit bull bites weren't more powerful than other large breed dogs and STILL came to the conclusions they came to. And sure, the bite would be stronger than your average Spaniel, but less so than many other larger breed dogs. That only makes sense.

Meanwhile, as you note, it's amazing the number of people who choose to do studies without any effort to look at circumstances behind bites - -and in this case, specifically left out information on significant causal factors.

[Sarah @ Dog Separation Anxiety Cure]

It is very difficult to find a neutral point of view taking in all of the evidence..

Thanks for debunking this study and drawing attention to the fact that media is mediated and you can find research to support ANY argument! In order to be valid science there should be two teams per study trying to find the opposite of what the other team is looking for. Then we could see which viewpoint had more supporting evidence.

[Spaccio Moncler]

Nice, and thanks for sharing this info with us.Good Luck!