This morning when I was reading the blogs, the title of Brian Doherty’s article caught my eye: “You Know Less Than You Think About Guns.” Really? Well, it turns out that, if he was addressing me, he was right. His article is a very careful look at the data on guns, crime, violence, and gun laws. It took me about 20 minutes to read and absorb it.

I recommend the whole thing–it’s long, but worth reading.

Some nuggets:

Guns Up, Homicide Rate Down:

This simple point–that America is awash with more guns than ever before, yet we are killing each other with guns at a far lower rate than when we had far fewer guns–undermines the narrative that there is a straightforward, causal relationship between increased gun prevalence and gun homicide. Even if you fall back on the conclusion that it’s just a small number of owners stockpiling more and more guns, it’s hard to escape noticing that even these hoarders seem to be harming fewer and fewer people with their weapons, casting doubt on the proposition that gun ownership is a political crisis demanding action.

In the face of these trend lines–way more guns, way fewer gun murders–how can politicians such as Obama and Hillary Clinton so successfully capitalize on the panic that follows each high profile shooting? Partly because Americans haven’t caught on to the crime drop. A 2013 Pew Research Poll found 56 percent of respondents thought that gun crime had gone up over the past 20 years, and only 12 percent were aware it had declined.

Suicides versus Homicides:

Critics quickly noted that the [National] Journal‘s Libby Isenstein had included suicides among “gun-related deaths” and suicide-irrelevant policies such as stand-your-ground laws among its tally of “gun laws.” That meant that high-suicide, low-homicide states such as Wyoming, Alaska, and Idaho were taken to task for their liberal carry-permit policies. Worse, several of the states with what the Brady Campaign to Prevent Gun Violence considers terribly lax gun laws were dropped from Isenstein’s data set because their murder rates were too low!

The Importance of Outliers in Affecting the Results

Among other anomalies in Fleegler’s research, Hinderaker pointed out that it didn’t include Washington, D.C., with its strict gun laws and frequent homicides. If just one weak-gun-law state, Louisiana, were taken out of the equation, “the remaining nine lowest-regulation states have an average gun homicide rate of 2.8 per 100,000, which is 12.5% less than the average of the ten states with the strictest gun control laws,” he found.

Having a Gun in the Home: Better Not to Be a Renter

The idea that keeping a gun in the home puts owners and their families at elevated risk first rose to prominence in a 1993 New England Journal of Medicine article by Arthur Kellermann and his colleagues. “Although firearms are often kept in homes for personal protection,” they concluded, “this study shows that the practice is counterproductive.”

The study has many flaws. In addition to the predictable failure to establish causality, there’s a more glaring irregularity: Slightly less than half of the murders Kellermann studied were actually committed with a gun (substantially less than the national average in 1993 of around 71 percent). And even in those cases he failed to establish that the gun owners were killed with their own guns. If even a small percentage of them weren’t, given that more than half of the murders were not committed with guns, the causal relevance of the harmed being gun owners is far less clear. (The study found that even more dangerous risks than having a gun at home included living alone, using drugs, or being a renter.)

If You Don’t Run, Use a Gun

In the October 2015 special issue on “gun violence prevention,” Preventive Medicine featured the latest and most thorough attempt to treat the NCVS as the gold standard for measuring defensive gun usage. The study, by Harvard’s Hemenway and Sara J. Solnick of the University of Vermont, broke down the characteristics of the small number of DGUs [Defensive Gun Uses] recorded by the NCVS [National Crime Victimization Survey] from 2007 to 2011. The authors found, among other things, that “Of the 127 incidents in which victims used a gun in self-defense, they were injured after they used a gun in 4.1% of the incidents. Running away and calling the police were associated with a reduced likelihood of injury after taking action; self-defense gun use was not.” That sounds not so great, but Hemenway went on to explain that “attacking or threatening the perpetrator with a gun had no significant effect on the likelihood of the victim being injured after taking self-protective action,” since slightly more people who tried non-firearm means of defending themselves were injured. Thus, for those who place value on self-defense and resistance over running, the use of a weapon doesn’t seem too bad comparatively; Hemenway found that 55.9 percent of victims who took any kind of protective action lost property, but only 38.5 percent of people who used a gun in self-defense did.

John Lott’s Work

The range of contentious issues involved in Lott’s techniques were summed up pretty thoroughly in a sympathetic but critical review of the third** edition in Regulation. The economist Stan Liebowitz of the University of Texas at Dallas wrote: “Should county level data or state level data be used? Should all counties (or states) be given equal weight? What control variables should be included in the regression? What violent crime categories should be used? How should counties that have zero crimes in a category, such as murder, be treated? How much time after passage of a law is enough to determine the effect of RTC laws? What is the appropriate time period for the analysis?”

Lott tried to demonstrate that on the county level, violent crime trends showed signs of improvement in counties that had or passed RTC laws compared to counties that had not, among other things checking both mean crime rates and the slope of crime rates before and after RTC passage. He attempted to control for many handfuls of other variables that might affect crime rates–indeed, some researchers accused him of accounting for too many variables, while others slammed him for failing to account for other factors, like conviction rates or length of prison sentences.

Trying to prove Lott wrong quickly became a cottage industry for others interested in the nexus of guns and public safety. The back-and-forths were so extensive that the latest edition of Lott’s book is nearly twice as long, with his reactions to his critics.

Comparing Connecticut and Rhode Island After Connecticut’s Gun Law

Rhode Island’s murder rate went up unusually after 1997 (the researchers don’t speculate on why that might have been), thus creating some “extra” murders (presuming that choices to murder in Rhode Island would have for some reason created a proportional number of choices to murder in Connecticut) that we can credit Connecticut with having evaded thanks to the more stringent gun law.

But what happens when you extend the analytic period beyond the arbitrary cutoff date of 2005? From 2005 to 2012, Connecticut’s gun murders per 100,000 people increased 66 percent, from 2.05 to 3.41, while Rhode Island’s went down 20 percent, from 1.83 to 1.45. It seems quite premature to take Webster and his team’s counterfactual guess about expected murder rates over one 10-year period as establishing any reliable causal knowledge about the effects of tougher gun purchasing laws. Yet that study was used to help buttress a proposed federal law the week it went public, trying to pressure other states into following Connecticut’s lead on background checks and permits, given what we now “know” about how life-saving that move had been.

Cost-Benefit Analysis Leaving Out a Key Set of Benefits

As Harvard’s [David] Hemenway explained to me, the confidence intervals of the social sciences in colloquial terms demand a belief that the chances are 19 to 1, or at worst 10 to 1, for you being right about your conclusion before you accept it as provisionally verified. Hemenway also believes, given the good he thinks can come from legal interventions about guns, that we don’t need to be that certain we are right for policy work.

But that’s easier to accept if you don’t value any particular benefits to relatively unrestricted private gun ownership–scientific, constitutional, or just personal. Some researchers, particularly in the public health field, act as if there were no values to balance on the other side of the policy goal of making it harder for people to get guns.

The Bottom Line

Whether you consider the associations and causations supposedly demonstrated by gun-related social sciences to be proven beyond whatever level of doubt you see as appropriate, applying those stipulated facts to policy questions can never itself be a purely mathematical or scientific process. It’s politics all the way down, and that politics is less informed by rigorous and certain knowledge than President Obama thinks.