By Ryan P. Burge

Photo Credit: Huffington Post

I remember before I really got into data analysis, I thought the idea of an open response survey question to be exciting. Instead of forcing the survey respondents into pre-determined categories, they are instead allowed to type (or speak) their response to the survey taker which is recorded verbatim. My excitement for this approach to surveys has been greatly diminished over time. A case in point is the 2016 CCES.

The CCES asks the following question: “What is your present religion, if any?” The survey taker is presented with twelve different response options:

• Protestant
• Roman Catholic
• Mormon
• Eastern or Greek Orthodox
• Jewish
• Muslim
• Buddhist
• Hindu
• Atheist
• Agnostic
• Nothing in Particular
• Something Else

Many of the response options lead to a branching structure. For example, if one indicates that they are Protestant then they are asked, “To which Protestant church or group do you belong?” This in turns leads to another layer of specificity: if one indicates Baptist, then they have the option to choose Southern Baptist, American Baptist, etc.

However, if a respondent chooses the “Something Else” option, they are given the option to type their religious affiliation in a textbox. That’s what I wanted to sift through. Of the 64,600 respondents, 2773 (or 4.2%) entered something into the open response box. Reading through the responses that individuals offer is sometimes maddening, sometimes confusing, and frequently hilarious. Here’s what I found.

There were lots of people who should have chosen the Protestant response item (which was the first one on the list) but didn’t. Over 1200 people did not choose Protestant but instead said they were something else and indicated that they were generically “Christian” or generically “Baptist.” There were also another 200 who were Lutheran (!) or Methodist who did not correctly identify their tradition as Protestant. Now, to be fair, there were some people who were genuinely presented with a challenge. Pagans and Wiccans really had no other place to go, their religious tradition is not listed as one of the choices. However, it’s important to note that there were only 36 Pagans and 27 Wiccans in a sample of nearly 65,000 Americans.

One would guess that it may be possible to go back through and “clean up” some of these stray respondents who should clearly be classified as Protestant, but that’s a process that is fraught with difficulty because there were over 30 different permutations of just the term “Baptist.” The word cloud below demonstrates just how difficult this task would be in sorting out over 500 people who seem to indicate their Baptist denominational affiliation. Yes, there is a “Baptist, Nazarene, Catholic.”

This is also evident in the vast number of spellings and descriptions of those who come from the charismatic tradition that is often labeled as “Pentecostal.” Note that this doesn’t even include a wide variety of individuals who identified as “Apostolic” or “Apostolic/Pentecostal.” For a researcher, the choice of whether to separate these two labels into different religious traditions or combine them under an umbrella term like “charismatic” is fraught with peril.

While a lot of this is head shakingly frustrating[1], because there is no solid way to parse these myriad misspellings and lack of religious awareness, there was one finding that I think should give pause to researchers who study religious demography in the United States. Without a doubt the religious tradition that had the greatest number of variations was the most amorphous one: non-denominational Christians.

In fact, I found 62 unique spelling or punctuation combinations for the same general idea. I wonder if this speaks to something deeper that is going on with that growing movement in the United States. Paul Djupe has previously written on the potential peril regarding the rise of non-denominational churches in the United States and I think that needs to be reiterated here. Many non-denominational churches take great pride in the fact that they shun labels and history. Many pastors constantly tell their congregations that Christianity is a relationship, not a religion. One has to wonder what unintended consequences that has on not just survey responses, but on collective identity, and a firmly held worldview from which to engage the world. It seems possible, maybe even likely, that some individuals believe that their obviously Protestant church is not Protestant because their church is not connected to any sort of established Protestant denomination. If that’s true, then our approach to measuring religious traditions needs to shift to accommodate the fact that they identify with an anti-label.

I can’t end without giving you my top ten favorite responses. Some are funny, some are frustrating, and some make you think a little bit.

1. Ancient Alien Theorist
2. Christian (why is this not an option?)
3. Christian wow you list muslin but not Christian
4. godless heathen
5. I’m an atheist, but my wife is Protestant, so we attend Lutheran services.
6. It’s complicated
7. none of your business
8. Spastic
9. Truth, Logic, and Mathematical Thinking
10. Why

Would you like to peruse the list of responses yourself? Here’s all of them without alteration.

Ryan P. Burge teaches at Eastern Illinois University in Charleston, Illinois. He can be contacted via Twitter or his personal website.

The syntax for this post is available on Github.

---

Note

1. Wordclouds for Methodist and Lutheran are available as well.