In 2012, the Obama administration unveiled a two hundred million dollar initiative to facilitate the growth of the Big Data industry and demonstrate its potential in large-scale markets. However, the initiative was an apparent failure. Despite government endorsement, Big Data has been unable to develop into an industry in its own right; the analysis and evaluation of billions of data points remains a trend rather than an emerging technology.
Tom Kalil, Deputy Director for Policy at OSTP, released a statement on behalf of the Obama administration in which he declared that, “to launch the initiative, six Federal departments and agencies will announce more than $200 million in new commitments that, together, promise to greatly improve the tools and techniques needed to access, organize, and glean discoveries from huge volumes of digital data”.
The statement went on to say, “we also want to challenge industry, research universities, and nonprofits to join with the Administration to make the most of the opportunities created by Big Data”. Unfortunately, high hopes and large investments did not translate to practical and effective solutions, and the US Government failed to demonstrate the potential of Big Data in the industries they had targeted.
Why the US Government's Initiative Failed
The downfall of the Obama administration’s initiative was undoubtedly the result of the increasingly negative perception of Big Data. For many Big Data represents the impractical and uncritical “fetishization of data”, or the unnecessary aggregation of large sums of information. The US government effectively placed a very large bet on the expectation that Big Data would evolve into an industry in its own right, however, it has distinctly failed to do so. Conversely, other forms or branches of Big Data, such as data visualization used by Netflix and leading technology service providers, have succeeded.
“Big Data is a tool that we have which allows us to make better decisions. I’m not afraid of Big Data.”
Since the launch of the Hype Cycle, technology consulting firm Gartner has failed to include Big Data on its list of emerging technologies. The Gartner report in 2015 extensively covered Virtual Reality (VR) and Artificial Intelligence (AI), based on their data evaluation tools for technologies and hype cycles. However, due to its lack of adoption and maturity, Big Data did not meet the requirements to be considered an emerging technology. In comparison to these other industries, it has shown very limited signs of growth. The incredibly modest demand and enthusiasm for Big Data in the market has meant that many consulting firms like Gartner have continued to dismiss it -- even with large-scale government initiatives.
“Gartner Hype Cycles provide a graphic representation of the maturity and adoption of technologies and applications, and how they are potentially relevant to solving real business problems and exploiting new opportunities”, said the Gartner team. “Gartner Hype Cycle methodology gives you a view of how a technology or application will evolve over time, providing a sound source of insight to manage its deployment within the context of your specific business goals”.
Despite this, it is important to note that Big Data has been embedded into almost all the operations and services of social media, technology and streaming platforms. However, firms like Netflix which actively use Big Data, prefer the term “data visualization” rather than Big Data due to its negative connotation.
How Netflix Has Demonstrated the Potential of Big Data
Popular social media, technology, and streaming platforms like Netflix collect billions of data points on a daily basis in order to offer video recommendations and customized product marketing to their users. Earlier this year, the Netflix development team noted that data visualization is of “paramount importance” to the platform; it allows them to provide users with recommendations for movies, TV shows, and concerts based on their preferences and viewing history.
“The longer you take to find the data, the less valuable it becomes.”
At the Hadoop Summit in 2013, Jeff Magnusson, the manager of data platform architecture at Netflix, offered a rare insight into the company’s utilization of sophisticated data visualization and Big Data technologies. During the presentation entitled “Watching Pigs Fly with the Netflix Hadoop Toolkit”, Magnusson stated that the data should be accessible, easy to discover, and easy to process for everyone. “Whether your dataset is large or small, being able to visualize it makes it easier to explain”, he emphasized. “The longer you take to find the data, the less valuable it becomes.”
Since early 2013, Netflix has continued to develop advanced data visualization tools, which have been a leading factor in the company’s success. As shown in the screenshot of Netflix’s data visualization tool, the Netflix platform has been relying on Big Data to investigate and analyse the consumer base for each Netflix show, the preference of genres based on age, and other important data that is crucial for both the company and the consumer base of Netflix.
The problem faced by large conglomerates in major industries, with regards to utilizing Big Data, is not necessarily technological. Instead, it is a problem with the misrepresentation of Big Data and negative public opinion, which has gotten worse over time. This is in part due to the fact that the average consumer has become significantly more privy to issues surrounding online privacy.
In the long-term, Big Data could eventually solve public policy challenges including privacy. As Economist and Data Scientist, Matthew C. Harding pointed out, we are dealing with really complex problems in Big Data which require a complex interdisciplinary line of approach and behaviour change. Harding has been using Big Data to answer crucial policy questions in Energy/Environment and Health/Nutrition. He believes that it is necessary to find ways to change human behaviour. “Big Data is a tool that we have which allows us to make better decisions”, he stated in the talk. " I’m not afraid of Big Data.”
Our desire to be able to predict outcomes and future scenarios has made us obsessed with collecting data and stockpiling it in huge databases. And in this digital era, we are definitely not short of information to gather. Despite the bad rep that Big Data has garnered over the last few years, this need to know what’s coming and prepare for all situations isn’t new. William John Warner, better known as Cheiro, was a famous Irish astrologist and palmist to the stars in the early 20th century. His predictions included the late success of a heavily indebted Mark Twain, the abdication of Prince of Edward VIII, the bitter downfall of Oscar Wilde and, most chillingly, the tragic fate of the Russian Tsar and his family. Whatever happened to “ignorance is bliss”??
Illustrations by Ilya Martynov
To contact the editor responsible for this story:
Margarita Khartanovich at [email protected]