Big data fractal

As more and more science-fiction becomes reality (talking phones, self-driving cars…) the more and more we expect machines to understand and think for us. But that ‘singularity’ is still at least 30 years away.

In the meantime, Matti Heikkonen points out that ‘big data’ analysis is very real already and a powerful tool for competitive advantage and decision-making… but the final decisions are still best left to humans.

Human insights

While data can give clues to what is happening, it takes humans to discover why it is happening.

Big data vs human insight

Big data analytics is limited to what is inside the datasets it analyzes. Usually, the aim is to find average behavior trends (and there are a lot of assumptions built into the algorithms). On the other hand, humans naturally only analyze data that is relevant to them, and they incorporate knowledge and influences from outside the source datasets.

Big data analysis can identify trends, whereas humans can determine why those trends exist and what might happen next. That’s much more powerful.

The datasets themselves also have limitations. Transactional data logs people’s behaviors, but we know people often behave unpredictably during times of change. This means we might base major decisions on behavior that that won’t exist in the future.

Gut instincts

Even if the big data analytics are good, would anyone take any notice anyway?

If you watch enough TV, you will know how important gut instincts are to police detectives. It turns out business decision-makers are the same. In a recent US survey, only 10% said they would go against their gut instinct based on data analytics. Most said that contrary data analytics just indicates that they need to do more analytics, not change their mind.

Humans #FTW

Obviously we like machine intelligence at InsightNG, but we also strongly back human intelligence.

It’s not ‘either-or’ – big data and machine intelligence are powerful tools that help humans make much better decisions.


About InsightNG

InsightNG is a smart thinking-tool that helps you gain greater clarity and deeper understanding when making important decisions, completing assignments, or getting through personal challenges.

InsightNG uses contextual intelligence and communities to help guide your discovery journey, but you are always in control.


About Douglas Campbell

Douglas spends a galling amount of time thinking about metadata and helping people find the information they need. Previously he has designed and product-managed search systems for national institutions in New Zealand. He is now working deep down in the nitty-gritty, developing our machine intelligence engine.