Stop Hyping Big Data and Start Paying Attention to ‘Long Data’

Wired: Stop Hyping Big Data and Start Paying Attention to 'Long Data'

Our species can't seem to escape big data. We have more data inputs, storage, and computing resources than ever, so Homo sapiens naturally does what it has always done when given new tools: It goes even bigger, higher, and bolder.

We did it in buildings and now we're doing it in data. Sure, big data is a powerful lens — some would even argue a liberating one — for looking at our world. Despite its limitations and requirements, crunching big numbers can help us learn a lot about ourselves.

But no matter how big that data is or what insights we glean from it, it is still just a snapshot: a moment in time. That's why I think we need to stop getting stuck only on big data and start thinking about long data.

By "long" data, I mean datasets that have massive historical sweep — taking you from the dawn of civilization to the present day. The kinds of datasets you see in Michael Kremer's "Population growth and technological change: one million BC to 1990," which provides an economic model tied to the world's population data for a million years; or in Tertius Chandler's Four Thousand Years of Urban Growth, which contains an exhaustive dataset of city populations over millennia. These datasets can humble us and inspire wonder, but they also hold tremendous potential for learning about ourselves.

Because as beautiful as a snapshot is, how much richer is a moving picture, one that allows us to see how processes and interactions unfold over time? …

… Why does the time dimension matter if we're only interested in current or future phenomena? Because many of the things that affect us today and will affect us tomorrow have changed slowly over time: sometimes over the course of a single lifetime, and sometimes over generations or even eons.

Datasets of long timescales not only help us understand how the world is changing, but how we, as humans, are changing it — without this awareness, we fall victim to shifting baseline syndrome. This is the tendency to shift our "baseline," or what is considered "normal" — blinding us to shifts that occur across generations (since the generation we are born into is taken to be the norm). …

I strongly resonate with this article. Trends and trajectories over extended periods are often far more useful than details of the latest twist or turn in societal development. It is so easy to get lost in the challenges of the moment. When you stand back and look at our moment in time from the standpoint of centuries and millennia, we live in the most astounding age of human flourishing in the planet's history. We are never without challenges, but there is good reason to expect flourishing to improve in coming generations.

The two groups I find the most insufferable are youth who believe their latest insights are the magic solution that brings utopia and grumpy old curmudgeons who mope about, complaining the world is going to hell in a handbasket. Neither has a sense of the longue durée. We need to spend less time with journalists and more time with historians.