By Ryan P. Burge and Hannah Smothers, Eastern Illinois University

One of my biggest fears in life is writing a grant, receiving a sizable chunk of money, and running a survey only to realize that I overlooked something that may put my entire research design in jeopardy. Once that survey is in the field, there’s not much that can be done to fix a major design problem that wouldn’t cost a lot of money or dramatically limit the sample size.

But, sometimes those little oversights can actually serve as opportunities to test out some statistical techniques that have not been widely adopted by social science. That exercise became the basis for a paper that Hannah Smothers and I recently published at the Secularism and Nonreligion. You can read it here.

The Cooperative Congressional Election Study has quickly become my favorite dataset in social science in part because of its sheer size. The 2016 version had over 64,000 respondents, for instance. It asks a variety of questions about religious belonging and practice and the CCES uses the same general approach to measuring religion in each of its iterations. For instance, respondents are asked to select their current religion from a list of 12 options. The list includes the usual suspects: Protestant, Catholic, Mormon, Buddhists, etc. But then it gives three options for the nones: atheist, agnostic, and nothing in particular. However, for reasons that are not entirely clear, the 2010 CCES left out the atheist option – it simply wasn’t listed on the survey.

That provides social scientists the opportunity to answer a great question: what box do atheists check if they can’t identify as an atheist? The graph below indicates in orange how the distribution of religious groups changed in 2010 compared to the other years of the CCES that are available.

Protestants were slightly higher in 2010 than they were in 2012, but the difference was less than a percentage point, Catholics were also marginally larger as well. However, agnostics went from 4.5% in 2008 to 5.2% in 2010 – maybe some atheists went there. The other group that saw a larger jump was “nothing in particular.” In 2008 they were 14.4%, which leapt to 18.8% in 2010 only to drop back down to 17.4% in 2012. Maybe most of them chose the nothing in particular option.

To further explore where the missing atheists went, we employed a type of machine learning called a random forest to make statistically educated guesses about where the atheists may have ended up in 2010. To help it make those guesses we trained the model on the 2008 and 2012 datasets because they already had atheists labeled correctly. The algorithm is given a number of variables that could potentially help it classify atheists. In this case, we included variables like church attendance, prayer frequency, the importance of religion, born-again status, alongside other demographic factors. Using these factors, the model was able to correctly guess atheists in the training data set 96.3% of the time.

Because we expected about 3.8% of the CCES respondents to identify as atheists in 2010, we then selected the respondents that the random forest model scored as the most likely to be atheists to represent the actual atheists in the 2010 sample. In most ways, this sub-sample of “guessed” atheists looked very similar to the sample of actual atheists in 2008 and 2012. The only real difference was that the random forest model selected a more highly educated sample than that which was represented in 2008 and 2012.

So, what option did these atheists in 2010 select instead? The answer is that 80% of them chose another none category: agnostic or nothing in particular. The third most likely landing spot was checking the “something else” box. Less than 15% of them chose a standard religious tradition like Catholic, Protestant, or Jewish. Most stuck with an identity that is as close as possible to their atheistic outlook.

While this was a fun statistical exercise and gave us the opportunity to use machine learning, it does provide some level of insight into the decision-making process of the non-religious in American society. It’s clear that most atheists won’t fall back on the religion in which they were likely raised. They instead make their decision on what religious group is closest to their perspective when taking the survey, which is clearly agnostics or nothing in particulars. In short, the vast majority of atheists have severed ties with all established religions, which means it’s highly unlikely that they would return to the fold.

Ryan P. Burge teaches political science at Eastern Illinois University in Charleston, Illinois. He can be contacted via Twitter or his personal website.

Hannah Smothers is a graduate student in political science at Eastern Illinois University in Charleston, Illinois.