We review Yelp’s new index for rating local economies:  It’s a good start

For a long time, the only comprehensive and reliable means we’ve had of tracking and comparing economic activity across state and regional economies has been official government statistics, such as those compiled by the Census Bureau and the Bureau of Labor Statistics. While these data have many virtues, there are often significant lags between the time data is collected and the time it is reported, especially for data with high levels of geographic or industry detail. In addition, as a rule, government data strictly protect the confidentiality of individuals, and so suppress data that might the identity or precise location of a particular household or business.

The advent of big data, in the form of massive databases augmented with crowd-sourced information, adds a new dimension to our ability to track and measure local economies. One of the most exciting sources is Yelp, which tracks and publishes user reviews of millions of businesses. Yelp has just introduced its new “local economic outlook” which rates cities and neighborhoods based on their “economic opportunity.” The rankings are based on Yelp’s extensive data, and are summarized in the form of national rankings of cities (and a parallel rankings of the top 50 neighborhoods).

Yelp has done its homework. Ed Glaeser, the dean of the nation’s regional economists, has authored a paper with two other economists testing the validity of Yelp’s business count data against the Census Bureau’s County Business Patterns data (CBP).  CBP is generally only available with a lag of a year or more and so isn’t a good guide to what’s happening right now in regional economies.  Glaeser and his co-authors find that Yelp’s data does a good job of predicting future trends in CBP data.  There is some variability: Yelp’s counts tend to be most accurate in dense, well-educated and higher income areas.  Also–unsurprisingly–Yelp’s coverage and accuracy has been steadily improving over time, and more closely agrees now with official statistics than it did just a few years ago.

It’s fantastic to get this data, but at least in its first iteration Yelp provides only rankings and doesn’t publish specific numerical estimates. We don’t know, for example, whether the cities in the top ten are 1 percent, 10 percent or 50 percent better than the median. All we have, in effect, is a top-to-bottom ranking of cities according to Yelp’s economic opportunity index. As we’ve always maintained, simple rankings that omit scalar data tend to generate a lot of heat, but actually shed little light. The Yelp report would be much more useful and interesting if it reported the actual numerical values of their index, and updated these figures on a monthly or quarterly basis. Yelp’s Carl Bialik tells us via email that they are working to extend the opportunity scores for future quarters, including releasing more raw data. We’re looking forward to this.

Because it is different from most conventional measures of economic activity, and because it is so new, it’s still a bit difficult to know exactly what the opportunity index measures, and whether its survival probabilities reflect short-term or more enduring differences in economic climate. It’s apparent from the methodological explanation (below) that they are looking at the opening and closing of business establishments, and using this information to compute survival probabilities in particular areas. But since they don’t publish numerical values for individual cities, or precisely reveal the formula for computing the index, its hard to interpret the rankings, or see how the opportunity index squares with other widely used measures of local economic activity. As we see more of the detail from the index, and can calibrate it against other measures, we’ll get a better idea of what the index is signaling and what it means.

The Yelp data is a promising and tantalizing look at how big, crowd-sourced data can help us develop a more timely and nuanced understanding of local economic activity. While their current rankings are a good way to promote awareness of the data, we hope they’ll do even more in the months ahead to publish and regularly update specific metro market indicators that others can use.