Often when we measure things on surveys and other forms we don’t think how the order of the questions might affect the answers we receive. However, a recent article in the Washington Post demonstrates that the order of the questions can matter a lot.

After the 2008 financial collapse, Gallup integrated some questions about politics into a continuing survey, right before some questions asking about overall life satisfaction. After this change, there was a remarkable decrease in people’s satisfaction ratings. At the time, this effect on life satisfaction was attributed to the financial crisis. However, later Gallup removed the politics questions and the satisfaction ratings shot back up to pre-financial crisis levels. They also tried asking a transition question between the political questions and the satisfaction questions. This also eliminated the downward affect on satisfaction.

This is just one more reason we need to be thoughtful as we put together evaluation instruments. We never know how one question might affect the answer to other questions. This just came up on a survey instrument that I was working on. My team wanted to ask a few new questions about mental distress, suicidal ideation and suicide attempts. We decided to split up these sets of questions, kind of like Gallup did with the transition questions so that acknowledging suicidal ideation or suicide attempts did not inadvertently increase the ratings of mental distress, or vice versa, the mental distress ratings did not affect the suicide questions. Whether we are successful or not in avoiding this problem, we will probably never know, but at least we thought about it before we implemented it.