Did our ad spend result in more sales?
(Or why you shouldn’t sleep with your shoes on)
Marketers often use market research surveys to establish whether their strategies are working or to identify the underlying causes or drivers of customer behaviour. However, while it is relatively easy for correlations between variables in a survey to be identified, establishing causation, e.g. that a change in one variable such as advertising spend actually caused an increase in another variable such as sales, is much more difficult.
Incorrectly claiming that one thing caused another just because they are correlated is known as the ‘cum hoc ergo propter hoc’ (Latin for “with this, therefore because of this”) fallacy. In common language, you could express it as:
X occurs in correlation with Y, therefore X causes Y.
In market research surveys, we might see that high overall satisfaction with a service is found among those people who also gave high ratings for ‘friendly service’. We therefore conclude (possibly incorrectly) that it is the friendliness of the service that is driving overall satisfaction.
Unfortunately, however, while correlation is a necessary prerequisite for causation, it is not necessarily proof of causation.
The sub-title of this article asserts that you should not sleep with your shoes on. This is a version of an old saying that can be expressed for our purposes as:
Every time I fall asleep with my shoes on, I end up with a headache.
Therefore, sleeping with your shoes on causes headaches.
Of course, the more likely explanation is that both falling asleep with my shoes on and waking up with a headache were both the result of being drunk.
Rush to find simple explanations
In the case of the customer satisfaction survey, one alternative explanation is that those customers who gave high ratings on both friendliness and overall satisfaction are your more frequent customers. Typically, frequent customers become more satisfied because they learn to ‘work your system’.
In addition, because they are frequent customers, your staff become more familiar with them and chat more with them compared with the less frequent customers. Not surprisingly, the more frequent customers give your staff higher ratings for friendliness. The real cause of both ratings is the frequency of use, which may be due to other variables, e.g. location.
While this potential explanation could be easily checked by cross-referencing the survey data for a relationship between frequency of use, overall satisfaction and friendliness ratings, this form of additional analysis is often overlooked in the rush to find simple satisfaction drivers.
Another form of this fallacy is often seen in social research, when very large-scale surveys or large volumes of data are used. This is because very large surveys and datasets can contain many hidden and/or complex relationships between the data items that are conveniently ignored because they do not directly relate to the question at hand.
For example, the famous ‘Hemline index’ refers to the strong statistical correlation between the fashionable length of women’s skirts and the Dow Jones stockmarket index. The statistics say that when hemlines go up, so does the Dow Jones.
Clearly, however, both of these variables are influenced by many complex factors including world economies and climate. While there may be a highly complex relationship between these two variables, it defies logic to suggest that one directly causes the other.
The direction of the relationship
The final, most common form of the fallacy of assuming causation occurs when we incorrectly assume the direction of the relationship. For example, studies are sometimes quoted as saying that high rates of gun ownership are positively correlated with high rates of violent crime.
Some commentators use these statistics to ‘prove’ that guns cause violent crime. However, it might also be the case that in high crime rate areas, the people who live there purchase more guns because they are more frightened.
David Hume, a famous philosopher, argues that it is impossible to ever prove causation. I am not so strident, however; I would suggest that prior to using correlated survey results to prove that one thing has caused another, you might want to check that the relationship …
- Cannot be explained by a ‘third factor’ that links both variables.
- Is logical and can be reasonably and simply explained.
- Only works in the direction you suggest.
- Is supported by other independent evidence.
- Is repeated over time.
|
