The Age of ‘Big Data’, and what it means today

More data has been created in the last two years than in all preceding history.

What’s So ‘Big’ About Big Data?

In basic terms, big data refers to the enormous bank of information that organizations create and collect. To give a sense of scale, market research firm IDC estimates that the amount of data produced in 2014 was 1.7 megabytes every minute for every person on Earth.

But storing and processing big data is a decades-old practice, and visualizing that great dump of information only tells part of the story. What’s special about today’s big data is that it is largely ‘unstructured,’ which means that it isn’t in a specific database. So an organization’s emails, social media, business transactions, weather reports and balance sheets could all sit together in the same disordered set.

With modern analytics, that deep pool of data becomes an immensely powerful tool. Big data isn’t necessary for finding out the particular cost of an item on a retailer’s shelves, but using analytics, it could tell you how many of those items were bought by a certain type of customer under particular weather conditions, for example. The ability to cross-reference and make links in a sea of information is what truly empowers the data.

If It’s So Big, Why Can’t I See It?

Whether we’re aware of it or not, big data punctuates our lives every day. Wondering which series you might get through next on Netflix, which novel to pick up from Amazon, or which article to read next? So is big data. Using sophisticated algorithms, retailers and marketing firms can pick out and target individual preferences. And while data is often used in a sales context, its uses go far beyond that. The world’s biggest eCommerce platform Alibaba has used customer data to venture into the small business loan market, while Springwise-featured startup Beyond Pricing uses algorithms to find the optimum Airbnb price on any given day.

Beyond business-oriented applications, data could be used to save lives. Hospitals and medical practitioners are increasingly using intelligent devices to capture data from patients, with the rise in connected devices accounting in part for the huge increase in collected data. Nightingale, previously featured on Springwise, is one example of a healthcare platform using data to achieve concrete medical outcomes. The cloud-based app makes use of data analytics to support personalized treatment for children suffering from autism. At a broader level, by capitalizing on opportunities for data collection, the healthcare industry can isolate and target high-risk patients — and ultimately save lives.

In education, too, there are profound ways in which big data is making a difference. With insights on who’s graduating, what grades they’re achieving and what feedback they’re giving, education providers have a much deeper understanding of the individual student. The value of that insight is underlined by education publishing giant Pearson’s recent decision to acquire analytics startup Learning Catalytics in a bid to strengthen its personalized learning offerings. Alongside industry behemoths are startups such as Panorama Education, the data analytics company aiming to measure more abstract aspects of a student’s classroom experience – such as whether they feel safe or valued at school. Both Pearson and Panorama Education are demonstrating the increasing importance of analytics and data in improving educational outcomes.

Too Big to Fail?

This might seem like an unmitigated good news story: abundant data twinned with remarkable analytics represents an opportunity to improve the offerings of industries ranging from healthcare to eCommerce. But this new technology opens the door to a brighter future only if it can be supported by the technologies of the present. Infinite data can only save lives if there’s somewhere to store it. Insights can only be gained if we have the means to process them.

The most immediate concern for advocates of big data is how to store the vast amount of information that is constantly being gathered. The International Data Corporation (IDC) predicts that the digital universe will weigh in at a colossal 44 zettabytes of data by 2020, and where that data goes is a serious question. A recent report from Forrester research found 86% of surveyed businesses are already struggling with networks that aren’t up to the task of modern business demands. Existing IT infrastructure — which includes data centers, networks, workplace equipment, personnel and security — is inflexible, non-standardized and overloaded, presenting a significant stumbling block in the big data revolution.

Intelligent Solutions

This overstretching of IT capabilities, though debilitating, is by no means terminal. Making digital technology platforms more agile is not only possible, but the route to doing so is known. The answer is to create an intelligent infrastructure: one which uses automation wherever possible, has analytics and tracking embedded from the outset rather than tacked on as an afterthought and uses machine learning to heal and optimize itself.

The options for data storage are also improving. The gluttons of big data — Facebook, Google and Amazon — use hyperscale computing environments. These fight scale with scale, using a huge number of servers to process data. For smaller companies, options include cloud-based storage or software-defined networking.

The majority of businesses today are reliant on technology to reach their customers. When the infrastructure supporting that technology falters, so does the service. Upgrading IT systems will help organizations meet the demands placed on them by abundant data, make operations semi-autonomous, and help supply chains become far more resilient and responsive.

Go Big or Go Home

Big data is big business, and capitalizing on it requires a step up for most companies. But the technology is there, and innovation continues to rapidly augment the capacity of storage solutions. Looking at the organizations exploiting big data to its capacity — the likes of Google or Facebook — it’s not hard to see the benefits. It’s now time for smaller businesses to step up and begin reaping the rewards.