DID HE OR DIDN’T HE?….JOHN LOTT AND THE MYSTERIOUS SURVEY….The revelation that John Lott has been masquerading on the Internet under the name “Mary Rosh” has given everybody considerable entertainment today, but it has a more serious side too: it is now obvious that he is the kind of person who is willing to engage in a meticulous and lengthy deception in order to defend his work. He has posted comments at both Julian Sanchez’s site and Megan McArdle’s, he has written email to me, he has reviewed his own book on Amazon, and he has posted hundreds of messages on newsgroups, many of them praising his own personal qualities. This is the obsessive behavior of a desperate person, and it means we should now judge his claims extremely skeptically.
The question is, did Lott actually conduct a survey in 1997 regarding defensive gun use, or did he fabricate the results? Recently a person came forward who claimed to have been part of Lott’s 1997 survey, and several people have taken this as evidence that Lott actually did conduct the survey in question. Tim Lambert, Lott’s most dogged critic, says:
[James] Lindgren questioned him carefully about what questions he was asked and they corresponded pretty well to what Lott said they were (with one major discrepency which I will comment on in the next paragraph). The only way this could have happened is if he had conspired with Lott, and given the blundering way Lott has defended himself, I just don’t buy it.
Given Lott’s behavior as Mary Rosh, however, this kind of collusion now seems at least plausible. Basically, we still don’t know.
So that leaves us with the data itself from Lott’s survey, and this is where my main question lies. Here’s a brief summary of the problem:
Lott claims his survey had 2,424 respondents.
James Lindgren estimates that out of this number about 25 would have reported defensive gun use.
Lott claims that of those 25, only 2% actually fired their gun. That’s one-half of a person.
Furthermore, out of the 2%, only one-fourth actually fired at a person (the other three-fourths fired warning shots). That’s one-eighth of a person.
Now, as it happens, I have an explanation of this from Lott himself (writing to me under his Mary Rosh pseudonym): the results were weighted, not raw numbers. This is perfectly reasonable, of course, but when I wrote back saying that the weighting seemed awfully large, here’s what he replied:
Whether it is possible depends upon how finely you do the weighting. If you do something as simple as national weighting, you are right, it would not be likely. But if you are willing to put in the effort to break things down into enough categories it becomes quite likely. I just looked up some different numbers from 2000 to give you a rough idea. In Montana, black males make up .14 percent of the population. In Mississippi, they make up 18.8 percent. That is a difference of 134 fold, quite a bit bigger than your 8/1 ratio. Obviously, this is an extreme difference and the difference that Lott must have come across is only about 1/17th as large. If he broke things down by age in addition to race and gender, I am sure that you could easily get difference much bigger than 134 fold. My impression is that at least on this point Lindgren is “making a mountain out of a mole hill.”
This doesn’t sound convincing to me. In order to get different figures for people who fired at a person versus those who fired warning shots, at least two people must have fired their guns. That’s 8%, which would have to be weighted down 4:1 to arrive at his 2% figure. Of those two, one of them fired at the person and the other fired a warning shot, which means that one of those people would have to be weighted up from one-half to three-fourths and the other would be weighted down from one-half to one-fourth.
I’m conversant with statistics, but I don’t know much about demographic sample weighting in a survey of this nature. So while it seems dubious to me, I just don’t know.
But there are people who read this blog who do know, and here’s my question for them: does this make any sense? Is there any reasonable combination of survey data and weighting that could be applied to a sample this size that would produce Lott’s stated results? And I don’t mean a statistically significant result, I just mean any reasonable weighting that could produce this result at all? How about it?