Digital Analytics

In analyzing data for business, they say you’re best off telling a story. You see, when you review data, you determine what happened, and then use that information to synthesize a narrative. This helps the consumers of your analysis understand the results, and it promotes executive buy-in for the job you’re doing as an analyst.

This approach isn’t wrong. In fact, it’s very important to effective communication. Yet, data journalism does have a vulnerability that I want to address, especially when analysts lose sight of what they’re doing when crafting the narrative that they’re going to share within their organization.

Data isn’t telling you a story. Data is the footprints of what happened; it’s the measured impact of change. Now, you can tell some useful, informed stories about what that change might have looked like: people visited a website, customers bought some things, fingers pressed some icons in an app. But each one of these stories needs an element of doubt, especially when the narrative is being told by a single analyst, or a single organization.

Different things can leave similar footprints. My toddler likes to play with my phone, and my wife’s phone, and throws off some interesting data when he does. He likes to use any app he can get into, he makes calls, and he’s found his way to random YouTube videos more than once. Sometimes he knows what he’s doing, but more often than not he’s just pressing icons until something interesting happens. If you were to examine user behaviour on my phone, you might not be able to tell what’s going on, and not everyone who looked at such data would come to the same conclusions about the phone’s owner. Maybe I’m a parent whose phone occasionally finds its way into a child’s hands. Maybe I occasionally erupt into uncontrollable spastic fits. Maybe my phone fell into the tentacles of an alien who is trying to figure out our technology and is systematically exploring the device.

This example is getting away from me, but the point is this: There might be many explanations for why the data you get is the data you get, and not everyone is going to think about that data the same way.

A recent editorial comment in the journal Nature made an argument for crowdsourcing academic research. Its authors put together a test, wherein 29 teams of researchers were given identical data and asked the same question. The data included details about soccer players, referees and their interactions, and the teams were each asked to determine whether or not referees were more likely to give red cards to players with darker skin.

“We found that the overall group consensus was much more tentative than would be expected from a single-team analysis”, the authors write. Since teams employed different analytical methods to their research, they arrived at different conclusions. “Had any one of these 29 analyses come out as a single peer-reviewed publication, the conclusion could have ranged from no race bias in referee decisions to a huge bias.”

The fact that groups of researchers come to different conclusions shouldn’t be startling, but its impact is. In academic journals, this leads to conclusions being established by individual studies, even after peer review, and then informing other research. Meanwhile, media reports take the results of academic studies and sensationalize the news in an effort to attract readership. This compacted bias takes results that are complicated and nuanced, and turns them into a simplified, definitive statement that may influence people’s behaviour.

“Under the current system, strong storylines win out over messy results.”

If this is true in academic research, which is meant to be quite thorough, careful and stringent, consider how such variances play out in business analysis, where the timelines are compressed and methodology isn’t carefully checked over. How do you regain confidence in results?

The lesson here is to be sceptical of any individual analyst’s conclusions, especially if their methodology is unclear and their results are not framed with details about how they were obtained.

To protect against this, an organization should work with multiple analysts, working both in and outside of the organization. There are advantages and biases that come with either position; analysts inside an organization are more likely to be influenced by internal politics and the culture of the group, but on the other hand, they may have more contextual information that sheds light on their results. External analysts are less likely to be swayed by the workings of the organization, and can offer a fresh perspective from their differing vantage point. By employing analysts in both situations and occasionally having them approach the same sets of data, you can mitigate the biases of their backgrounds and approaches. When they tell you the same story, then you can have greater confidence in it. When their stories differ, you gain a more nuanced perspective and protect yourself against making great leaps in the wrong direction when you act on the results.

This is part of why I always thought Napkyn’s Analyst Program was a great idea. I used to be a Senior Analyst at Napkyn, where the company offers dedicated analysts to large enterprises as a managed service. Throughout the development of that program, we were often faced with two challenges: organizations that wanted to have their analysis done in-house, and in-house analysts who felt threatened by the presence of external analytics professionals working with the company. Both of these problems go away when everybody sees the advantages of having multiple teams with different perspectives working with the same data. Outsourcing an analyst is no substitute for having internal teams looking at, consuming and sharing data; nor is it enough to leave every aspect of your business analysis to internal teams.

Finally, back to analysts: don’t always expect the data to tell you a clear story. Data is messy; sometimes that’s because it’s not recorded or organized well, but sometimes it’s because that data represents the footprints of messy actions. As an analyst, it’s not your job to write compelling stories; it’s your job to deliver the truth. If the truth can be told as a true story, that’s fantastic. You will certainly discover facts that can be presented in a compelling narrative, and those insights will be the easiest to act on. But when the stars don’t align, you’re going to have to communicate that results are inconclusive or unclear, because that’s the truth. You should never be delivering a story that’s merely inspired by true events.