The Disappearing Poor


Exploring the incredible worldwide gains in prosperity: Reality Project Episode 1

Every day for 25 years, newspapers could have correctly run the headline “130,000 People Escape Extreme Poverty Yesterday” (source). Instead, as far as I can tell, this has appeared on the front page of a major newspaper precisely 0 times. Maybe this explains the incredibly terrible results recorded by people on this simple question (from Gapminder):

Question 1. of the Gapminder test

When this question was asked in the United States, 5% of humans got the correct answer, C: almost halved. Animals, on the other hand, answer this question with 33% accuracy (because they randomly guess).

How is it possible for people in a supposedly literate society like the US to be so wrong about reality? I don’t like to place blame, but in this case, it’s likely justified: the media has succeeded in a campaign to misinform us about the world. Instead of reporting facts, they have chosen to activate our natural inclination towards xenophobia (fear of outsiders) by convincing us there is a “developed world” with well-off people, and then there is everyone else, a mass of poor hordes that will never be able to rise from poverty.

Fortunately, armed with the right data, we can correct our wrong beliefs about poverty. On doing so, we discoverable a remarkable fact: over the past few decades, people all over the world have undergone a remarkable increase in prosperity, resulting in measurably better living standards for billions of people.

Read More

Announcing The Reality Project

Sunrise from the International Space Station (Source)

Becoming less wrong about the world with data

Strange and wonderful things happen when you put down the newspaper, turn off the tv, and decide to look at actual statistics about the world instead of relying on what you’ve been told. The ground begins to shift beneath your feet as you realize the worldview which the news has been feeding you — that the world is on a downward slide — is, based on almost every single factual measure, in direct opposition to reality. Once we look at the data on topics such as wealth, health, human rights, and environmental protection, we are left with no option but to conclude the world has been on an upward trajectory for most of recorded history, we live in the best time ever for human civilization, and the improvement shows no signs of stopping.

Over the past five years, as I’ve slowly weaned myself from a debilitating news addiction, I’ve experienced what can only be described as an update to the worldview-generating software in my brain. Through tens of thousands of book pages, millions of data points, and countless charts, my view of the world as a dark, dangerous place and humanity as an evil force has reversed. Based on the data, the story of the world has flipped: the history of humanity is not one of decline, but a gradual ascent continuing to today.

I did not start out with the aim of becoming an optimist, but a long-term positive outlook is the inevitable conclusion from a fact-based examination of the world.

As a new year begins, and now that I’ve assimilated enough independent sources of data to be confident reality is really better than we think, I’m starting The Reality Project, a data-driven endeavor to present the world as it is through a series of weekly articles. The stated goal is to become less wrong about the world through data, disposing of our distorted worldviews. If, along the way, the data happen to give you a more optimistic view of humanity, then consider it as a beneficial side effect.

Read More

Analyzing Medium Story Stats With Python

A Python toolkit for data science with Medium article statistics

Medium is a great place to write: no distracting features, a large — yet civil — readership, and, best of all, no advertisements. However, one aspect where it falls short is in the statistics you can see for your articles. Sure, you can go to the stats page, but all you get to see is some plain numbers and a bar chart in an awful shade of green. There’s no in-depth analysis of any kind and no way to make sense of the data generated by your articles.

It’s nice when you reach the point where it’s more than your mom reading your articles.

It’s as if Medium said: “let’s build a great blogging platform, but make it as difficult as possible for writers to get insights from their stats.” Although I don’t care about using stats to maximize views (if I wanted to get the most views, all my articles would be 3-minute lists), as a data scientist, I can’t bear the thought of data going unexamined.

Instead of just complaining about the poor state of Medium’s stats, I decidedto do something about it and wrote a Python toolkit to allow anyone to quickly retrieve, analyze, interpret, and make beautiful, interactive plots of their Medium statistics. In this article, I’ll show how to use the tools, discuss how they work, and we’ll explore some insights from my Medium story stats.

The full toolkit for you to use is on GitHub. You can see a usage Jupyter Notebook on GitHub here (unfortunately interactive plots don’t work on GitHub’s notebook viewer) or in full interactive glory on NBviewer here. Contributions to this toolkit are welcome!

Example plot from Python toolkit for analyzing Medium articles

Read More

The Copernican Principle And How To Use Statistics To Figure Out How Long Anything Will Last


Statistics, the lifetime equation, and when data science will end

The pursuit of astronomy has been a gradual process of uncovering the insignificance of humanity. We started out in the center of the universe with the cosmos literally revolving around us. Then we were rudely relegated to one of 8 planets orbiting the sun, a sun which subsequently was revealed to be just one of billions of stars (and not even a large one) in our galaxy.

This galaxy, the majestic Milky Way, seemed pretty impressive until Hubble discovered that all those fuzzy objects in the sky are billions of other galaxies, each of which has billions of stars (potentially with their own intelligent life). The demotion has only continued in the 21st century, as mathematicians and physicists have concluded the universe is one of an infinity of universes collectively called the multiverse.

Go here to be blown away

Read More

Books Of 2018


1 year, 75 books, innumerable ideas

There’s one aspect of reading to make clear up front: I don’t think reading is an inherently better way to spend one’s time than any other activity. People who brag about how many books they read or say “I never watch tv” as if it makes them superior to people who do frustrate me because there is no reason to judge other people for how they choose to live their life.

Reading books is just another way to pass the time and you can waste time reading (see the books at the bottom of this rankings) just as you can watching television. For me, reading is how I spend my commute, get through exercise (listening to audiobooks), and the time right before I fall asleep, but, I don’t think this makes me better than people who watch movies or go out with friends instead. As a good rule, don’t judge people on how they spend their free time as long as they aren’t harming others.

I say this up front because I don’t want this article to sound like “here’s all the books I read which makes me really smart.” I try to write a few sentences about the books I read to process them, fitting them into my existing worldview, or updating my positions as needed (when the facts change, I change my opinions). While these are originally for myself, I’ve cleaned up my notes (you can access the raw version here) and made them public for anyone curious. This is more personal than my usual data science writing, but, comments, criticisms, and discussion is still encouraged here or on Twitter.

Read More