Big Data Opportunities and Challenges

In 2012, 2.5 quintillion bytes of data were generated daily, and 90% of current data worldwide originated in the past two years (Marcia and Big Data). During 2012, 2.2 million TB of new data are generated each day. In 2010, the market for Big Data was $3.2 billion, and this value is expected to increase to $16.9 billion in 2015. As of July 9, 2012, the amount of digital data in the world was 2.7 ZB; Facebook alone stores, accesses, and analyzes 30 + PB of user-generated data. In 2008, Google was processing 20,000 TB of data daily. To enhance advertising, Akamai processes and analyzes 75 million events per day. Walmart processes over 1 million customer transactions, thus generating data in excess of 2.5 PB as an estimate.

More than 5 billion people worldwide call, text, tweet, and browse on mobile devices. The amount of e-mail accounts created worldwide is expected to increase from 3.3 billion in 2012 to over 4.3 billion by late 2016 at an average annual rate of 6% over the next four years. In 2012, a total of 89 billion e-mails were sent and received daily, and this value is expected to increase at an average annual rate of 13% over the next four years to exceed 143 billion by the end of 2016. In 2012, 730 million users (34% of all e-mail users) were e-mailing through mobile devices. Boston.com reported that in 2013, approximately 507 billion e-mails were sent daily. Currently, an e-mail is sent every 3.5 × 10⁻⁷ seconds. Thus, the volume of data increases per second as a result of rapid data generation.

Growth rates can be observed based on the daily increase in data. Until the early 1990s, annual growth rate was constant at roughly 40%. After this period, however, the increase was sharp and peaked at 88% in 1998. Technological progress has since slowed down. In late 2011, 1.8 ZB of data were created as of that year, according to IDC. In 2012, this value increased to 2.8 ZB. Globally, approximately 1.2 ZB (10²¹) of electronic data are generated per year by various sources. By 2020, enterprise data is expected to total 40 ZB, as per IDC. Based on this estimation, business-to-consumer (B2C) and internet-business-to-business (B2B) transactions will amount to 450 billion per day. Thus, efficient management tools and techniques are required.