The Big Data Fairy Tale

By Roel Castelein

Fairy tales usually start with ‘Once upon a time ...' and end with ‘... And they lived long and happily ever after'. But nobody explains ‘how' the heroes live long and happily ever after. Big data (analytics) promise to transform your business, but just as in fairy tale endings, big data will not explain ‘how' to transform your organization. In my view, big data might spark some behavioral change or open people's minds, but it will not transform organizations. At best, big data evolves organizations. Let's look at the concept and a concrete example to draw conclusions.


What big data analytics does is take a bunch of data, analyze and visualize it, and then derive insights that potentially can improve your organization or business. Based on these insights the actual transformation can begin, but it requires more than just big data. Let's have a look at a classic example of data analytics; the reduction of crime in New York under Mayor Giuliani with the help of CompStat.

CompStat is a data system that maps crime geographically and in terms of emerging criminal patterns, as well as charting officer performance by quantifying criminal apprehensions. The key to success was not the data or analysis, but that the organizational management that used the data and analysis was effective. Processes, structures and accountability were setup to drive the transformation. In weekly meetings, NYPD executives met with local precinct commanders from the eight boroughs in New York to discuss the problems. They devised strategies and tactics to solve problems, reduce crime, and ultimately improve quality of life in their assigned area. CompStat tracked the results of these strategies and tactics, and whether they were successful or not. Precinct commanders were held accountable for the results.

Drawing upon my own experience, I know how difficult an organizational transformation is. Even if you have the data and the analysis that shows things need to change, it requires much more than data analysis. Let's assume that the data uncovers opportunities for improvement, either in reducing cost or in increasing revenue. The next step is to design the changes in processes, in people's roles, in org charts and in the systems. This usually entails a two pronged approach; communicate the change in org charts, processes and roles, and engrain these changes in the systems to track the change results. This tracking creates a feedback loop, necessary to manage the transformation.

Another challenge in the big data transformation message is finding the right people. Ideally the team leading the transformation needs to understand an organization's data, enriched with outside data, then know how to do data analysis, and once the results are there, strategically communicate the change to get everybody on board. Next, the transformation team needs to set up a tracking and feedback process that holds participants accountable for the transformation results. And when participants do not play along, have an escalation process in place, with the possibility for punitive measures.

In the same way that Giuliani fired one of the precinct commanders when he showed up drunk at the first CompStat meeting, big data systems require a complementary management philosophy to ensure whatever transformational insights are derived get implemented and controlled.

So, when the advertisements claim that big data will transform your business, remember that big data brings the potential for transformation, not the actual transformation. That still requires commitment and hard work, just like ‘living long and happily ever after'. That's why they are called fairy tales.