Is Big Data like shooting fish in a barrel?

Humans are predictable sorts. While we’re all slightly different, our biology and similarity of experience on the planet conspire to make us more like each other than not. The 2010 French movie Babies showed us that infants from the Mongolian Steppes, African bush, Tokyo and San Francisco all progress in nearly the same way, at the same rates. Does that surprise you? It surprised us.


Professionals are aware of this. But until recent times, precision in predicting how people will behave was a challenge. There wasn’t enough data or enough technology to give data the inspection it needed. Instead, scientists conducted studies and marketers set up focus groups. It was all about getting the right ‘sample’ size and makeup.

Shooting fish in a barrel is a great metaphor for what’s changed. Big Data gives us the ability to process increasingly large volumes of preference and behavioral data, not from studies, but from real people. The more data, the more patterns of similarity emerge. The more refined the patterns, the greater the likelihood that a person’s likes, dislikes, and future actions can be predicted.

We can hit the target every time.

Is it too easy?

So when we agree to give people our information, we’re giving them the power to predict exactly how we’ll respond. We’re giving up, in a real, scientifically proven way, our free will. How many people realize this when they allow a cookie on their computer, or turn on geo location on their phone? How many people are targeted because of extraneous detail that makes them an ‘easy shot’?

When our preference for movies demonstrates our politics, Netflix is the new campaign bumper sticker.

Want to see what others have said? Head to Quora to see the discussion.


Tags: , ,

Categories: Data Analytics / Big Data, Privacy

Author:Chris Taylor

Reimagining the way work is done through big data, analytics, and event processing. There's no end to what we can change and improve. I wear myself out...

Subscribe to the blog

Subscribe and receive an email when new articles are published

One Comment on “Is Big Data like shooting fish in a barrel?”

  1. Kenneth Watman
    September 17, 2012 at 9:02 am #

    Certainly, for more than a century, life insurance companies have based their success on the predictability permitted by a great enough N over a great enough period of time. Put more generally, obviously we could not do a great many things we do without an adequate capability to predict the behavior of large populations. So I don’t doubt your assertion that ‘big Data” means we will get better and better in this respect, but with two caveats.

    First, none of this means we will get any better at predicting an individual’s behavior, any more than physics can predict exactly which subatomic particle will decay. All we can say is that in a given population,we can predict that X individuals will behave a certain way with Y accuracy within a given time term.

    Second, if a chaotic or random-like mechanism is driving the behavior we wish to predict, then we will not know when a particular behavior will occur, only that, if given enough time, it will occur with X accuracy. For example, many experts on North Korea expect that regime will collapse in some way over the next 20 years. But, since the timing and path of such events are very hard to predict, we are likely to be surprised when it does happen.

    I can imagine how Big Data might address these two problems, but I think progress will be much slower than expanding our ability to predict more stable behaviors of populations, like buying behavior and the like.


    Kenneth Watman

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: