A New Study discussing the reliability of visual breed identification

In a new study that was just released, researcher Victoria Voith once again exposes the challenges of identifying dominant breeds of mixed breed dogs, even by "experts" in the field.  I think the research is important for a lot of reasons (some different from what others have surmised), and worth reviewing as the world of DNA genetics testing continues to challenge what we think we know about canine genetics, and rapidly adding knowledge to the field.

The Study

In the study, more than 900 people who had professions in the field of animal welfare were asked to watch a 1 minute video of 20 dogs that showed the full view of the front and side of the dog, along with the weights, age, and sex of the dogs. All participants noted that they're breed identification was sometimes used for record keeping purposes and were largely made up of kennel staff, Veterinary Assistants, Animal control Field officers, dog trainers, or other animal control or veterinary/behavior triaining staff.

The survey participants then had to answer the following questions:

Do you think the dog is purebred?  yes/no

If yes, what breed to you think it is?

If No, what do you think is the most predominant breed?

What do you think is the second most predominant breed? If you are unsure, write "mix".

The answers were then compared to the DNA results for the dogs.

The Dogs

Before I get into the results for this, I want to discuss the dogs used, and DNA as a whole. It's very important to note that at this point no one is saying that DNA tests are 100% reliable. Even the researchers who are creating the tests note this. Currently, the Mars Wisdom Panel test (which was used for this analysis) says it has a 90% accuracy rate across all first generation crosses. As dogs get further removed from pure heritage, the results get less and less accurate.

It's very important to note in this study that only ONE of the 20 dogs used came back with 50% or more of any one breed -- so only ONE fell within the 90% accuracy range. So, essentially,we cannot have a lot of certainly of the actual DNA results of almost any of the dogs in the study.  However, with the other 19, we can be 90% certain that the really are very mixed breed dogs and not predominently any one breed.  I think this is very important in how the results are analyzed.

The Results

While this study is far from conclusive because of the very small sample set of dogs, I do believe the initial results are very interesting.

Based on the study results, for 7 of the 20 dogs, more than 10% of the responents thought that the dog was "probably" a purebred dog. For three of the 20, 19% or more of the respondents thought the dog was probably a purebred dog (one of these three was the only dog in the study with a mix of 50% of any one breed - Miniature Pincher - and the majority of respondents incorrectly ID'd the dog as predominently Carin Terrier). 

Also of note, there was agreement among 50% or more of the respondents in the predominent breed in only 7 of the dogs -- and in 3 of them, the visual ID did not match ANY of the DNA identification. In the other 13 dogs (65%) there was no majority guess among the panel.

In 14 of the 20 dogs used, less than half of the guesses included a breed that was even detected by the DNA results. And for one of the dogs, exactly zero of the 859 respondents correctly identified a breed of dog in the mix (and this was the dog with highest breed concentration). 

Discussion

It's important to note why this data is important. In the United States, more than 40% of the canine population is mixed breed. 

For decades, the only possible way to identify mixed breed dogs was through visual identification, and many city juristictions have used such identification to influence their animal control policies.

However, we are now getting a lot of science-based data on the efficacy of such ideals, and the results don't support visual breed ID as a basis for policy:

- Visual ID of dogs is highly inaccurate when compared to DNA results

- The Visual ID of mixed breed dogs is highly subjective, as most 'experts' don't agree on the breeds based on looks

- Even those well-educated in the field are susceptible to judgment biases

Thus, statistical data that has often used to promote breed-specific policies is, and has been, based on visual breed identification that is proving itself to be highly subjective and inaccurate. And if public policy-makers want to rely on scientific and accurate information for their decisions, then the best solution will be to focus solely on behavior-based ordinances that target how dogs (and owenrs) BEHAVE, not how they look. 

Here's a link to the entire Voith Study (very detailed)

Here's a link to the National Canine Research Council's overview of the study.

Here's a link to a very nice visual chart that shows the dogs and the study results for the dogs.

Comments

[Peter Masloch]

I'm not sure if this is the same study but here also is one from the Maddie's Shelter Medicine Program at the University of Florida:
http://sheltermedicine.vetmed.ufl.edu/library/research-studies/current-studies/dog-breeds/dna-results/

[Brent]

Thanks Peter. Different study (I wrote about that one here:http://btoellner.typepad.com/kcdogblog/2012/10/what-kind-of-doggie-is-that-in-the-window.html )

Although I think the Voith study is a bit more statistically reliable due to methodology, I'm glad you posted the link because the results are pretty similar in terms of highlighting the inaccuracy of visual identification.

[Stacey]

This study highlights the inaccuracy of BOTH forms of breed identification.

DNA on dog # 13 came back as Alaskan Malamute. Seriously?

Dog #11 looks like a purebred Dutch Shepherd. This is an uncommon breed that shares common ancestry with both German and Belgian shepherds. Many of these dogs have recent import ancestors. This dog could easily be an out-group for the DNA test, making the results invalid.

I really wish someone would do a study with mixed dogs of known ancestry to compare with the DNA results. There are LOTS of sled dogs of known mixes with long pedigrees. Someone needs to go study those dogs. Or maybe Chinooks, or LUA dalmatians. It's not like purpose-bred mixes with known pedigrees are difficult to find.

[Brent]

Obviously there are significant holes when there are breeds missing from the database -- Currently they're focusing on the more popular breeds -- and neither Dutch Shepherd nor Belgian shepherds are in the breed pool. It would make sense though that they if this were a Dutch Shepherd, it would come back with GSD in the results. It's worth noting that it would be VERY odd for someone to own a Dutch Shepherd, or even a Dutch Shepherd mix, without knowing what they owned, given their rarity.

As for #11, it is essentially a mixed breed with no genetic markers. At the most recognizable being 12.5% it is essentially not picking up on much of anything.

The test-makers themselves pretty much have to be testing this on dogs of known ancestory. There would be no way to say "90% reliable" if they weren't. I think they could improve their case by publishing the results (although I realize for competitive reasons they probably never will).

I'd like to see a third party test them on known mixes too. The few samples I've seen have been pretty good on first generation mixes, but pretty poor on more distant mixes (especially at the Gen 3 level). There are some holes of course -- with some genetic markers being more reliable than others. Boxers for instance have very reliable markers. Collies and Australian Shepherds are almost indistinguishable after Gen 1.

Northern breeds would really be interesting. Because they are still close to their working roots, and still largely bred for function (not looks), I have a feeling they would be very muddy in their outcome (similar to Jack Russell and Rat Terriers).

[EmilyS]

two significant problems with this study:

1) look at the list of participant categories. A huge percentage are ACO's, shelter staff (including receptionists) and pet store employees. What qualifies these people as experts? Why were these types chosen? Because the choice predetermines the outcome they wanted. Otherwise, they would have limited their participants to those with actual established expertise, such as breeders and breed judges. All this sample shows is what we already know: most people, including those in dog professions, DONT KNOW how to identify breeds.

2) it assumes the accuracy of DNA to establish the breed's identity. There ARE no third part tests, Brent. There is NO PROOF. The DNA companies assertions about accuracy are just... assertions.

I challenge these authors to do a study where they use dogs of known pedigree, present them to breed experts and then compare the DNA.

As it is, this study doesn't prove ANYTHING. It certainly doesn't validate the growing propaganda that "no one can identify a dog's breed; it's foolish to try; let's treat all dogs only as individuals; let's not assign or assume any breed characteristics (to go along with the anti-breed red herrings that "not all dogs of a breed exhibit every breed trait to the same degree" and "any dog may exhibit a trait typical of some other breed"

This study is NOT in isolation. And if it's not funded by NCRC or its owner AFF, it is certainly being touted by them as something remarkably new and convincing.

The trend to denigrate/minimize breed characteristics is a dangerous one, in my opinion. Rather than leading to open-mindedness in chosing a dog, it can lead to more inappropriate adoptions and more returns. Or worse.

Why not go the opposite way: EDUCATE shelter staff/ACOs etc by involving breed clubs/judges etc. Teach the people who make life/death decisions about dogs to be more accurate about a dog's breed and what that MIGHT mean to a potential adopter. At least until that nirvana where every shelter dog has an environment in which the full range of its behaviors is displayed (unlike the current challenging stressful one most shelter dogs experience... all experience is that it takes a dog "x" amount of time in a home before its personality comes out)

But that of course would require the shelter world to put aside the ingrained "hate breeders" mentality.

[Brent]

Emily,

Did you read the full study?

I did note several of the flaws within the study, and the study itself notes the lack of 100% accuracy of DNA testing. At this point, I suspect your hatred of all things AFF or NCRC at this point is clouding your judgment about the test itself (different from the NCRC paper and the single sheet visual). Heck the person you're most at odds with it seems doesn't even work there any more.

There is nothing in the study that denigrates and minimizes breed characteristics. There isn't a single purebred dog in the entire study, and only one that has more than 50% representation. I don't know of anyone who would stand behind "breed traits" of such a watered down mix (feel free to prove me wrong, I've just never seen it).

The reason for the mix of participants (of which, only about 7% were pet store workers, and it seems most had multiple positions so it's a bit hard to determine) is that most breed identification in this country of dogs of unknown origins is done by animal control staff and shelter workers.

There is no question in my mind that breed and show people have a better ability to identify mixed breed dog breeds than shelter staff or ACOs. Most of the latter don't really have a good feel for what purebred dogs look like (since they see relatively few of them) and certainly aren't as familiar with many of the less common breeds. This is why most shelters in this country are full of "lab mixes", "Pit bulls mixes" and "shepherd mixes". It's why our own shelter has a show community person come in regularly to help us breed ID mixed breed dogs.

The reality is though, that breed ID of mixed breed dogs in this country is largely done by ACOs and shelter staff. This mistaken breed ID is then picked up by the media, used in dog bite reports, and statistics that are often used to target breeds by city councils.

And the data is inherently inaccurate.

I completely agree that there is a lot of work that needs to be done to educate shelter staff/ACOs on breed identification (although, in fairness, if the mix is unknown, then there is no way to really test the education level to know of its accuracy). But we can, no doubt, get better.

But the reality is that there is 25 years worth of false data that is being used as a reason to kill dogs every day, right now. And if this helps debunk that (which it does), without even needing DNA results as part of the study, then I'm in favor.

I will also recognize that I am a bigger believer of the DNA testing than you are. Not because I think it's 100% accurate (it's not) and not because I think its reliable down to 3rd and 4th generation iterations (even the test makers agree it's not). But I think we're learning a lot from the DNA testing, and it's accuracy is unquestionably better than it was 5 years ago, and will continue to get better. I'm all for learning about what it can provide us now, so we can be better equipped to deal with its reality in a few years.

[Jen Brighton]

In all fairness to the study, in section 4.1, Limitations of Study, it does say: "It is possible that the breeds of these 20 dogs in this study are unusually difficult to identify visually. Similar studies should be conducted with other samples of dogs and by other researchers."

I just heard mention on the PBLN blog talk show that even the head of the AKC said he can't identify a breed of dog without a DNA test. I believe the reference was to a collie. It might look like a collie and act like a collie, but without DNA testing he couldn't confirm that it is 100% purebred collie. I looked for a cite to this statement but didn't find one, so this is hearsay at this point. Fred Kray might be able to supply the source.

In defense of shelter workers, it seems to me that they work with many breeds of dogs in settings that some AKC judges don't. Even my local agility instructor who has won many titles, has judged and has been featured in magazines had not had much experience with certain breeds, especially the ones that until recently weren't allowed to participate in agility such as bully breed or mixed breed dogs.

Sounds like we have room for a new career opportunity: Dog Breed ID Professional.

[Stacey]

"It's worth noting that it would be VERY odd for someone to own a Dutch Shepherd, or even a Dutch Shepherd mix, without knowing what they owned, given their rarity."

This is a false assumption. Rare breeds are just assigned as a mix of whatever someone at the shelter thought they looked like. I met someone in the park the other day with a plott hound who was convinced they had a lab/beagle cross. I've met a person with a Swedish vallhund who was convinced they had a corgi/shepherd mix. I have a friend with a Belgian Laekenois. I don't know what a shelter would call him, but it definitely wouldn't be purebred.

Also, Belgian shepherds ARE in the testable pool. The are just separated into three of the four varieties. How honestly can you score the "purity" of a Malinois with Tervueren and Groenendael in it's pedigree anyway? Trick question :) They are all the same breed, except in the AKC's paperwork.

[Emily]

Scott and Fuller did some known breeding tests back in the 60's (crossing pure to pure F1, F2, etc.) and the visual results were... pretty much what we see here, lots of dogs that looked like neither parent or no breed at all. It would certainly be nice to see more data on it, though.

And I gotta say, my "Fancy" friends and clients, including judges and dedicated show folk, seem as all over the map as the rest of us... some of the hard-core breed people will say bitterly that an All-Breed judge (as compared to a specialty judge for their breed) wouldn't know a "good" specimen from a hole in the wall. And that may be the point: using visual ID OR DNA testing as the basis of public policy like BSL when no one can agree on the accuracy of either... hmmm.

[EmilyS]

That reference to an AKC statement about DNA must be incorrect. AKC does NOT use DNA to identify breeds, but only for parentage. (they will outright reject DNA tests to support a limited registration for a dog that is an AKC breed but does not have known AKC parentage)

http://www.akc.org/dna/index.cfm
"12. Can DNA testing determine the breed of a dog?

No. DNA profiling can determine parentage and genetic identity, but not the breed."

[DubV]

Here's a study with a small sample size that is actually done correctly.

http://news.vin.com/VINNews.aspx?articleId=23206

It takes Mars panel and tests dogs of known parentage.

The results are underwhelming. The clawing for DNA tests is based upon people thinking that DNA breed tests are like DNA fingerprinting at crime scenes, which is not the case whatsoever.

That the studies are not done correctly but are trumpeted by many should give you pause.

I saw the problem within a few minutes of looking at the tests using DNA tests and visual ID on dogs, did you, Brent?

[Brent]

Dub, the test referred to in this post was designed to have a different result than the one you linked to (which I've also covered here). This study shows the disparity in breed ID of mixed breed dogs among animal care professionals and DNA results. The one you linked to was designed solely to look at efficacy of DNA tests. Different goals. Different, but not competing outcomes.

[Brent]

"And that may be the point: using visual ID OR DNA testing as the basis of public policy like BSL when no one can agree on the accuracy of either... hmmm."

Agreed. I do think it's interesting that everyone in all groups think they are effective at determining breed, and everyone else is horrible at it. The one thing is certain, they're not all correct, because the disagreement is rampant.

[Pat]

Speaking of trying to guess pedigree by visual exam reminds me of our first dog... He was 1/2 Afghan and 1/2 Bloodhound --- in his prime, he looked more like a Vizsla than anything else...

[DubV]

"This study shows the disparity in breed ID of mixed breed dogs among animal care professionals and DNA results. The one you linked to was designed solely to look at efficacy of DNA tests. Different goals. Different, but not competing outcomes."

With most studies of that type, a premise (whether stated or not) is that DNA testing should be considered more accurate than the visual. Otherwise, what exactly would be the point to show that visual ID and DNA tests differ? It would mean nothing without any reference. The way to do a study like this correctly would be to do visual and dna tests on dogs of known parentage. I thought that much was obvious from my comment. As it is now, this study holds nearly 0 information.

[Brent]

"With most studies of that type, a premise (whether stated or not) is that DNA testing should be considered more accurate than the visual."

I guess I shouldn't be surprised that you misread this study as well given your inability to comprehend anything well.

Read the study. It notes that DNA testing is not 100% accurate. It also notes that humans are not accurate. And essentially the study concludes that there is a wide disparity in breed recognition among humans, and between humans and DNA. So in other words, no matter what system you use (at this point) there is discrepancy. Don't read into something it does not say, nor really imply.

[DubV]

Removed my comment where I pointed out that you misused the accuracy statistic? It was much more cordial than your own. Look up various uses of confusion matrices, seriously. That will help you interpret studies in the future.

"Currently, the Mars Wisdom Panel test (which was used for this analysis) says it has a 90% accuracy rate across all first generation crosses. As dogs get further removed from pure heritage, the results get less and less accurate.

"It's very important to note in this study that only ONE of the 20 dogs used came back with 50% or more of any one breed -- so only ONE fell within the 90% accuracy range."

You applying the 90% accuracy to the results, and moving "back" to reality, is a classic misuse of that type of statistic. If the accuracy stats were calculated as you seem to infer, then what you did was similar to this:

"80% of the kids with a first name starting with S answered all questions correctly on this quiz. This kid answered all questions correctly, therefore there is an 80% chance that his/her first name starts with an S."

I'm dumb though, so don't listen to me.

[Brent]

Dub,

The study applied the 90% accuracy rate. I actually provided the reality that only 1 of the 20 dogs in the pool fell within that criteria so that people would understand that the DNA results weren't reliable (other than, we can realistically determine that they were mostly likely not purebreds or G1 mixes).

What's important to note is that the accuracy of the DNA test is not essential to the conclusions that either the study, or I, made. Not in the slightest.

Your previous comment was removed, like all others in the future will be, because you don't interest yourself in discussion, but just want to argue for arguing's sake or seeking to discredit anything that disrupt your current (misled) point of view. This is a perfect example where you completely have missed the point of the study, and tried to misinterpret my attempt to add clarity to the study.

I don't wish to entertain it any more, and all other future comments from you will most likely be removed.

[Selma]

OK you know my feeling about the mixed breed DNA test: it is a party game, pure and simple. Until someone does a blinded, controlled trial using documented mixes that begin with purebreds, down the line to at least the fifth generation, I will retain my opinion. I have already designed such a study in draft form, maybe I should conduct it myself.

Just because Mars says they have 90% accuracy in F1s doesn't mean it's true. They are selling a product, not doing clnical research. It may be true, but to date, without any published peer-reviewed results around mixed breed testing, it is just an assertion. One would think if they had hard results they'd be trumpeting them from the rooftops to boost sales.

There are some papers out there around DNA testing in purebreds but they aren't making wild claims about accuracy and, since AKC gave the genome gang 10K samples from each breed, it makes a little more sense.

These guys are testing undocumented mutts.

Sorry Brent, but to state that in one case the 90% accuracy applies because the DNA test found one dog with 50%+ of one breed doesn't make any sense. First of all, an F1 is a cross between two purebreds - not one purebred and a bunch of other breeds. Secondly, saying the test is accurate because of the test results makes my head hurt lol.

I wish this shelter research group would either do a proper blinded, controlled study or just stop talking about the DNA aspect altogether because it diminishes the real value of their findings. Plus, all the shelter people are buying into it, big time. That makes me extremely uneasy.

It's bollocks that a mixed breed dog exhibits 'breed' characteristics. Being a mongrel negates the entire concept of breed, not to mention that getting specific characteristics is difficult enough with purebreds. Ask any long-time breeder who doesn't like to tell tall tales.

The value of this study lies in the fact that the very people who are most likely to identify mongrels as a 'breed' or F1, including in terms of BSL, are shown to be unqualified to do so since they disagree with each other much more often than they should, given any kind of industrial standard or measurable result.

[Brent]

Selma,

It is important to note that neither my own conclusion, or the one in the research paper, is reliant in any way on the DNA research being accurate. I do think that given even the lack of the DNA test's ability to pick up breeds in this case makes the DNA part of the study irrelevant, and distracting, from the overall study results (especially given that that is the ONLY thing people are discussing about it).

Meanwhile, I understand why you would want more research on the DNA tests. And I understand why it should come from a third party. But I do know the research behind the tests is based on sound science, and suspect that it is far more accurate than the "party game" assertion. Just how accurate remains to be seen, and likely depends at this point by what breeds are involved as some markers are better defined than others.

[Selma]

That's kind of what I was saying - that they should forget the DNA until there is some evidence because it is indeed distracting from very interesting results - which don't depend on the DNA to be valid.

Here's the thing. 'Markers' aren't little signs that indicate ethnic group, ie, breed, or even visible physical characteristics. It's not the way DNA works.

As one of my friends asked "OK, so are they looking for things such as oxygen uptake in Whippets vs Bulldogs, or metabolic factors in Basset Hounds vs Fox terriers? There's a lot more to DNA than a few physical factors that are visible on the outside." Very true.

There is no way that DNA can identify my ethnic heritage. It can't tell you how tall I am. It doesn't tell you much about me at all, including the way I look, unless you are looking for very specific things that aren't that easy to find.

Familial/parental DNA isn't as accurate as people believe it to be. It doesn't say 'you are the father of that child', it says 'we can eliminate you as the father of that child'. It is only accurate with exclusions, in other words. It's the same with dogs; kennels where lines are fairly close, it is almost impossible to determine which dog(s) sired a litter or part litter.

I know that DNA is very useful and that there have been advances made. I'm a huge fan of science and the scientific method. What I don't like is being asked to take something on faith that is supposedly scientific. It activates my radar.

What I question is the supposed accuracy of a test that is working on an unverifiable subject - a shelter mutt. That smells like snake oil to me. Since dogs are all basically alike genetically anyway, much like humans, I think that believing we can extract multiple breeds, most of which are usually preposterous such as Tibetan mastiff x Italian Greyhound x Plott hound sorts of results, is a bit naive.

I would like to see some actual research into this. I have Mars's patent app and am going through the long list of references but thus far, no dice.

I'm not saying it's not possible. I'm saying that until there is some actual evidence I won't accept it on somebody's assertion.

[Selma]

This is a spiffy little writeup about parental DNA, which should be more accurate than determining mixed ethnic ancestries:

http://www.vgl.ucdavis.edu/services/CatandDogDNATyping.php

[Brent]

Well, for starters, it would be impossible to test the accuracy by working with unknown subjects. You'd HAVE to work with known subjects -- which is why they started with known, registered dogs.

It makes sense. Dogs do appear different. So there would have to be a difference in the genetic sequence for two different breeds of dogs. While MOST will be the same, there would have to be some differential.

By studying the different breeds, you could better identify the different changes in genetic sequences in order to determine breeds. That's how we know from DNA that a banana is not a dog (although, the difference is obviously much greater).

That said, there are a lot of different genes that are different triggers, and how they mix and match in mixed breed dogs would no doubt be VERY challenging. And more challenging in some breeds than in others (for instance, they say the gene sequence in Boxers is very distinct, where the ones for Jack Russels are not, and they say that once you start mixing Border Collies, the gene sequence is almost indistinguishable from Australian Shepherds).

Anyway, there is an awful lot known in this category, and again, I think it's well above "snake oil" but well below "exact science" at this point.

Here are two pretty good articles about it. One from National Geographic that explains some of the sequencing, and another from Patricia McConnell, who's opinion matters to me.

http://ngm.nationalgeographic.com/2012/02/build-a-dog/ratliff-text

http://www.patriciamcconnell.com/theotherendoftheleash/analyzing-the-analysis-wisdom-panel-dna-tests-mixed-breed-dogs

[Selma]

Yeah, National Geographic is not what I'm looking for, and McConnell is not a geneticist - albeit a great writer who knows a lot about dog behaviour. There are lots of opinions out there, just no evidence.

In purebreds, they aren't bad in terms of accuracy so that's not the problem. The majority of these tests are performed on an unverifiable subject - a mutt of unknown provenance. That's why it needs to be tested.

Anyway, I don't want to become the New Dub, so I'll move on. As I said, I don't dispute the science behind DNA. I dispute that you can mix curly and straight hair and say "Irish" and "Swedish".

At some point, somebody will test these kits. You don't need to be a geneticist. You have to start with purebreds, test those, record the results, then mix to create F1s, test, record, then mix via inbreeding and outcrossing among F1s, test, record, etc all the way back to at least gen 5. It really only means you have to create pedigrees for mixed breed dogs, keeping track of the DNA results in each generation and comparing the results against what you know. Simple record-keeping, really.

It would take some money and some time.

[Stacey]

I don’t think we are asking the best questions about the DNA breed test. How about we start with this: What does this test actually test for? It tests for aggregations/patterns in various locations of DNA. That’s it. Noting more. If Mars says dog #13 shares genetic markers with an Alaskan malamute with a 90% confidence rate, it’s probably true. There is no good reason to doubt this.

Next, how does this test, which claims to determine breed based on marker aggregates, correlate to our expectations of breed? When we think of a breed of dog, we think of a specific body type and behavior. Do the dogs tested show the expected correlation in body/behavior to the breed test results?

The last question we need to ask is: Are these results meaningful? If we go back in time to when all these dogs were puppies, and tested them, would the results be helpful predictors of the dogs’ future appearance and/or behavior? Would a consumer feel the results are worth the price of the test?