What Happened to Big Data?Seven years ago, I was involved in a corporate workshop headed by a customer’s CIO to discuss that company’s big data strategy. Big data was a relatively new concept back then and the workshop included four speakers who explained how it would transform that company’s business. To my surprise, not one of the speakers made a cogent point about what big data was and why the audience should care about it.
One speaker talked about the “Four Vs” (I’ll let you look that up and decipher it for me because I never really understood the positioning of that definition). Another talked about open-source technologies (primarily Hadoop). Another talked about general data warehousing concepts that had been around for several years. And the fourth talked about analyzing audio files (without discussing why that was relevant for this retailer).
All the speakers talked glowingly about their topics, the exciting future, and the fact that people should sit up and take notice. People in the audience then broke into workshops to figure out how they would leverage big data. Follow up presentations discussed a myriad of analytical opportunities (some, incidentally, were about using existing techniques and technologies). At the end of the day, however, I was left with more questions about big data than when I started.
Now, several years later, after a lot of time and money has been invested, what strikes me is that I rarely hear the term “big data” anymore. So, what happened to it? In my opinion, the problem is that it’s a term that meant everything and nothing. It was a platform. It was certain data types. It was a certain set of analytical methods and tools. It was whatever you wanted it to be.
Unfortunately, what it typically didn’t have was anything to do with how the innovative use of big data and analytics would make your company better. The analytical evolution was certainly underway and creating a monumental shift in thinking, but people became fixated on a nebulous term and not pragmatic results. Whether or not people were directly involved in analytics, they owed it to themselves and their respective organizations to delve deeper into what was behind this concept and how it could be applied. Instead, a great deal of opportunity was lost chasing the hype of big data.
The Definition of InsanityThere’s an old saying, “The definition of insanity is doing the same thing over and over and expecting different results.” We see this play out time and time again in the world of analytics. Gartner has a popular concept called the “Hype Cycle.” In this cycle (see graph below) there’s an initial rush toward a new technology or concept. Expectations are inflated, which ultimately leads to a general letdown (Trough of Disillusionment) and then (hopefully) a reasonable level of common understanding whereby the technology matures and finds its stated value in the market (Plateau of Productivity).
The Gartner Hype Cycle
This scenario plays out repeatedly and is particularly relevant in analytics. With respect to big data, it followed this curve but then, interestingly, dropped altogether before reaching some generally agreed upon level of acceptance. The rationale was presumably because “big data [had] become a part of many hype cycles,” according to Datanami.The reality, in my opinion, is that the ambiguity of the term and lack of tangible value from investments ascribed to it led to its eventual downfall. The frustrating part is that this hype scenario is so prevalent that Gartner, one of the largest business and technology research organizations, identified a need for it in 1995, and its use has grown into all areas of Gartner’s consultancy scope. So, is the problem that vendors and consultants come up with terms like “big data” in the first place, or is it that we, as practitioners and enablers of analytics, keep chasing the next big thing and expect a different result? If it’s the latter, then it clearly falls into the definition of insanity.
Why do we have hype cycles like those touted by Gartner? That’s a broader philosophical discussion but, on some level, it’s because of the inability to accept the responsibility of being intellectually curious when it comes to new analytical concepts (rather than taking hype at face value). Some may not be actual practitioners or developers of advanced analytics, but they still should have a base knowledge of what it is and what it’s not—and challenge partners touting these concepts to explain their relevance in a business context.
Additionally, we should focus on pragmatic aspects of the technology (will it work in my specific environment, at scale, etc.). In doing this, we’re better prepared to cut through the hype and better understand how these concepts can—or cannot—be leveraged to solve real business challenges. Note that the X axis on the graph above is “Time.” Imagine the amount of time that businesses have lost because they traversed the peaks and valleys associated with this cycle. Is it really necessary to chase windmills, or can we do better and reduce or eliminate the time lost to hype by not accepting this phenomenon as “just the way it is with analytics”?