Everyday, we create 2.5 quintillion bytes of data, that includes data created by climate sensors, posts and pictures to social networks, videos uploaded to YouTube, phone calls records and GPS records, transaction records and so on. This mammoth data is called Big Data.
Big data includes data sets whose size is too large, too dispersed, too unstructured, and too awkward to for conventional software tools to capture, manage, and process the data within a tolerable, finite time. Big data sizes are a constantly moving target currently ranging from a few dozen terabytes to many petabytes of data in a single data set. Borne out of search companies’ needs to handle massive quantities of information, big data initiatives are going mainstream and transforming product strategies, businesses, organizational roles and entire industries.
Trends
- Growing technology deployments are creating an exponential increase in the volume of data available, but big data is not solely about volume
- Existing analytical techniques and business decision-making processes are overwhelmed by the extreme information management issues—of which, big data is the most immediate, requiring new approaches from both IT and business leadership
- A successful approach to the handling of big data will be a critical business capability in the data-rich environment of the future, delivering significant competitive advantage to organizations
Applications
- In the public sector, real-time detailed information regarding traffic flows, vehicle locations and resource use (such as energy and water) supports the optimization of service delivery and efficiency across a wide range of public services
- In government and public safety, enhanced forecasting of the extent and impact of natural and man-made disasters and improved status has helped optimize the deployment of resources to support and protect the population
- In online businesses, a better understanding of customer preferences and social interactions supports enhanced cross-selling, upselling and recommendation engines
Opportunity
- Big data can provide considerable value by making information usable and transparent at much higher frequency.
- As businesses create and store more transactional data in digital form, they can collect more accurate and detailed performance information on everything from product inventories to number of transactions, and therefore expose vulnerability and boost performance.
- Big data allows a detailed segmentation of customers and therefore more precisely tailored products or services.
- Sophisticated analytics can substantially improve decision making.
- Big data can be used to create innovative after-sales service offerings, thereby improving the development of next-generation products and services.
The deep technology basis of big data represents exactly the type of technology-enabled new business opportunity that offers an almost limitless platform to develop competitive advantage and new revenue. Big data poses a significant dilemma for CEOs and business leaders, because the investment required in emergent technology and the rapidly evolving discovery of the technical issues potentially represent a substantial amount. CIOs, as well as information, data, system and enterprise architects, face significant challenges in addressing the issues surrounding big data, but they should not ignore them. Newer technologies and applications are emerging, and should be investigated to understand their potential value. Hadoop is one such technology that aids simplification of big data, as I elaborated here a few days ago.
The market at large has started to talk about big data as a disruptive force and an immediate problem that is already affecting traditional understanding and business models. However, big data is just the start. In the future, the full range of extreme information management issues—of which volume is just one aspect—will pose even greater challenges, but also enable the emergence of even more significant business opportunities.
The true challenge is not one of big data, but the more-complex issues across all the dimensions of information management. Growing issues around the variety, complexity and velocity of data are equally significant.
