The quality of rent data varies widely, beware of erratic data sources
Trying to measure average housing costs for neighborhoods across an entire city—let alone the whole country—is an incredibly ambitious task. Not only does it require a massive database of real estate listings, it requires making those listings somehow representative at the level of each neighborhood and city.
For a number of reasons, just taking the average of all the listings you can find is likely to produce extremely skewed results, with numbers much higher than true average home prices. For one, many apartments, especially on the lower end of the market, aren’t necessarily listed in places that are easy to find—or at all. Instead, landlords find tenants with a sign on a fence or streetlight pole, local (and not necessarily English-language) newspapers, or just word of mouth. On top of that, if you have two homes of similar quality but even slightly different prices, you would expect the cheaper one to rent or sell more quickly. As a result, it would spend less time listed than the more expensive home; any given sample of listings, then, would tend to over-represent those more expensive, harder-to-rent homes. (If this doesn’t make sense, read the “visitors to the mall” example here, explaining a similar statistical problem with attempts to measure prison recidivism.)
So we’re sympathetic to anyone taking on this challenge. But that doesn’t mean that organizations who take it on but fall short should be given a pass.
Take, for example, Zumper. Zumper is a website that features rental listings in cities around the country. So far, so good. Zumper has also made a name for itself through its “National Rent Reports”—more or less monthly press releases that claim to track median rental prices around the country. These reports have received copious media coverage, from the Bay Area to Seattle to Nashville to Chicago to Boston to LA to Miami to Denver, and so on.
Some journalists apparently take Zumper’s reports at face value. For example, Wolf Richter of WolfStreet, spotted a “bubble” in apartment markets–based on a look at month over month changes in one bedroom apartment prices starting in late 2017. According to Zumper data, one bedroom rents surged, reaching double digit rates by February 2018, before falling back to less than two percent. According to Richter:
A peculiar phenomenon cropped up last November: The median asking rent for 1-BR apartments suddenly surged by the double-digits, even as the median asking rent for 2-BR apartments was barely edging up. This phenomenon endured for four months but has now collapsed (the phenomenon remains unexplained, though some suspects have been lined up):
But the real suspect here is the quality of Zumper’s data. Not only is there no plausible reason why nationally, one bedroom rents would follow a suddenly different trajectory than two bedroom rents. Averaged over millions of units nationally, there’s no reason why rents should move so abruptly if they’re correctly measured. In addition, no other source has any corroborating increase in one bedroom rents during this time. For reference, Zillow‘s Rental Index shows a very steady rate of increase, with a value of less than 3 percent year over year through 2017 and 2018. Yardi‘s national average apartment rent didn’t change by more than $1 per month between July 2017 and March 2018. The real problem would seem to be Zumper’s methodology, which is severely affected by the composition of units listed for lease on its website. This composition effect, and not any change in market conditions appears to be influencing these results.
Unfortunately, Zumper’s reports also appear to be severely affected by the problems we listed above, and possibly others. We noticed this in Chicago. Back in August, 2015 Zumper’s National Rent Report declared that the median one-bedroom apartment in Chicago cost $1,920—a number that would raise eyebrows among anyone who has actually looked for one-bedroom apartments in that city. A cursory glance at Zumper’s neighborhood-level data reveals issues that should call the entire report into question.
From Zumper’s website.
“Median,” of course, means that half of Chicago’s one-bedroom apartments ought to cost more than $1,920, and half ought to cost less. But according to Zumper’s own data, just three of the city’s 77 neighborhoods had median one-bedroom rents of over $1,920. While apartments are definitely not distributed evenly over the city, so you wouldn’t necessarily expect an even split in terms of neighborhoods, it’s simply not plausible (or supported by, say, the Census) that half of the city’s apartments are in just three of its neighborhoods.
It seems more likely that half of Zumper’s listings are in just three of the city’s (wealthiest) neighborhoods. As of the writing of this article, Zumper claims to have over 4,000 apartments listed in the Near North Side—the most expensive part of the city—and just 11 in Jefferson Park, five in West Garfield Park, and zero in South Lawndale, three of the cheaper neighborhoods.
Nor does it appear that Chicago is the only city with this problem. In Los Angeles, it appears that about 25 neighborhoods have median rents above the supposed citywide median—and about 70 have ones below. In Philadelphia, Zumper’s map shows just 11 neighborhoods with median rental costs at or above the supposed citywide median, and over 40 below; the proportion is similar in San Diego. The skewed distribution of Zumper’s listings is also apparent in these cities: the relatively more expensive Philadelphia neighborhoods of Rittenhouse Square, Center City East, and University City have 99, 219, and 94 apartments listed, respectively, while the less-expensive communities of Elmwood, Kingsessing, and Mill Creek have 19, 25, and 8.
These comparisons likely understate how inaccurate Zumper’s numbers are. After all, if its listings skew towards the higher end of the market, they likely not only oversample wealthier neighborhoods, but also more expensive properties in those neighborhoods, meaning that the true median rent in each neighborhood, not just the city as a whole, is below what Zumper reports.
Comparing Zumper’s citywide medians to estimates from Zillow, which is generally regarded as one of the more accurate estimators of real estate prices, reveals a mixed bag. (We looked at numbers for September, 2015). In some cities, the two sources give roughly similar numbers: Zumper estimates the median listed one-bedroom apartment cost $2,110 in Washington, DC, versus Zillow’s estimate of $2,149; the estimates for Los Angeles are $1,830 and $1,850, respectively. But in many places, they’re quite different. In New York, it’s $3,160 versus $2,300; in Chicago, $1,920 and $1,550.
Zumper responded to our inquiries over Twitter and email. A spokesperson said that Zumper “stands firmly behind [its] rental data.” He added: “We have some of the strongest inventory from which to analyze…. We are reporting on true, asking rents seen in the market, and do not create an algorithm to estimate value.”
Of course, put another way, this is largely our point: Zumper takes the median from its listings, without compensating at all for the fact that its listings are disproportionately concentrated in higher-end neighborhoods. While it may be true that Zumper has a relatively large inventory of rental homes in its database, that’s akin to an online pollster saying that their polls must be accurate because they got so many votes. While quantity matters, at a certain point, quality—representativeness—matters much more.
On Twitter, Zumper’s CEO also told us that the National Rent Report focuses on the median apartment available for rent, and doesn’t claim to take into account apartments that are currently occupied. But as an explanation for Zumper’s concentration of listings in high-end neighborhoods, that doesn’t really pass the smell test: differences in housing turnover between wealthier and less-wealthy communities are several orders of magnitude too small to account for, say, the gap between the number of listings in the Near North Side and Jefferson Park. (Note that the Zillow estimates we described above are also for listed apartments.) Nor are claims that that gap is “in proportion to how many people move” to each of those neighborhoods plausible.
We should note that none of this is really a problem for Zumper’s main business, which is being a database for people looking for a place to rent. But it does mean that they should not be used as a reliable source for rental data, just as journalists shouldn’t report on real estate trends by simply adding up every listing on Craigslist.
Housing affordability issues are real, as we’ve written about here extensively, and the media absolutely should be reporting on home price trends, both locally and nationally. But precisely because these issues are so important, it’s crucial that the data that gets reported is reliable. Until it addresses the problems we’ve brought up here, Zumper’s rent reports are not, and journalists should be aware of that.
“Caveat Rentor” – beware of crazy rent statistics
The quality of rent data varies widely, beware of erratic data sources
Trying to measure average housing costs for neighborhoods across an entire city—let alone the whole country—is an incredibly ambitious task. Not only does it require a massive database of real estate listings, it requires making those listings somehow representative at the level of each neighborhood and city.
For a number of reasons, just taking the average of all the listings you can find is likely to produce extremely skewed results, with numbers much higher than true average home prices. For one, many apartments, especially on the lower end of the market, aren’t necessarily listed in places that are easy to find—or at all. Instead, landlords find tenants with a sign on a fence or streetlight pole, local (and not necessarily English-language) newspapers, or just word of mouth. On top of that, if you have two homes of similar quality but even slightly different prices, you would expect the cheaper one to rent or sell more quickly. As a result, it would spend less time listed than the more expensive home; any given sample of listings, then, would tend to over-represent those more expensive, harder-to-rent homes. (If this doesn’t make sense, read the “visitors to the mall” example here, explaining a similar statistical problem with attempts to measure prison recidivism.)
So we’re sympathetic to anyone taking on this challenge. But that doesn’t mean that organizations who take it on but fall short should be given a pass.
Take, for example, Zumper. Zumper is a website that features rental listings in cities around the country. So far, so good. Zumper has also made a name for itself through its “National Rent Reports”—more or less monthly press releases that claim to track median rental prices around the country. These reports have received copious media coverage, from the Bay Area to Seattle to Nashville to Chicago to Boston to LA to Miami to Denver, and so on.
Some journalists apparently take Zumper’s reports at face value. For example, Wolf Richter of WolfStreet, spotted a “bubble” in apartment markets–based on a look at month over month changes in one bedroom apartment prices starting in late 2017. According to Zumper data, one bedroom rents surged, reaching double digit rates by February 2018, before falling back to less than two percent. According to Richter:
But the real suspect here is the quality of Zumper’s data. Not only is there no plausible reason why nationally, one bedroom rents would follow a suddenly different trajectory than two bedroom rents. Averaged over millions of units nationally, there’s no reason why rents should move so abruptly if they’re correctly measured. In addition, no other source has any corroborating increase in one bedroom rents during this time. For reference, Zillow‘s Rental Index shows a very steady rate of increase, with a value of less than 3 percent year over year through 2017 and 2018. Yardi‘s national average apartment rent didn’t change by more than $1 per month between July 2017 and March 2018. The real problem would seem to be Zumper’s methodology, which is severely affected by the composition of units listed for lease on its website. This composition effect, and not any change in market conditions appears to be influencing these results.
Unfortunately, Zumper’s reports also appear to be severely affected by the problems we listed above, and possibly others. We noticed this in Chicago. Back in August, 2015 Zumper’s National Rent Report declared that the median one-bedroom apartment in Chicago cost $1,920—a number that would raise eyebrows among anyone who has actually looked for one-bedroom apartments in that city. A cursory glance at Zumper’s neighborhood-level data reveals issues that should call the entire report into question.
“Median,” of course, means that half of Chicago’s one-bedroom apartments ought to cost more than $1,920, and half ought to cost less. But according to Zumper’s own data, just three of the city’s 77 neighborhoods had median one-bedroom rents of over $1,920. While apartments are definitely not distributed evenly over the city, so you wouldn’t necessarily expect an even split in terms of neighborhoods, it’s simply not plausible (or supported by, say, the Census) that half of the city’s apartments are in just three of its neighborhoods.
It seems more likely that half of Zumper’s listings are in just three of the city’s (wealthiest) neighborhoods. As of the writing of this article, Zumper claims to have over 4,000 apartments listed in the Near North Side—the most expensive part of the city—and just 11 in Jefferson Park, five in West Garfield Park, and zero in South Lawndale, three of the cheaper neighborhoods.
Nor does it appear that Chicago is the only city with this problem. In Los Angeles, it appears that about 25 neighborhoods have median rents above the supposed citywide median—and about 70 have ones below. In Philadelphia, Zumper’s map shows just 11 neighborhoods with median rental costs at or above the supposed citywide median, and over 40 below; the proportion is similar in San Diego. The skewed distribution of Zumper’s listings is also apparent in these cities: the relatively more expensive Philadelphia neighborhoods of Rittenhouse Square, Center City East, and University City have 99, 219, and 94 apartments listed, respectively, while the less-expensive communities of Elmwood, Kingsessing, and Mill Creek have 19, 25, and 8.
These comparisons likely understate how inaccurate Zumper’s numbers are. After all, if its listings skew towards the higher end of the market, they likely not only oversample wealthier neighborhoods, but also more expensive properties in those neighborhoods, meaning that the true median rent in each neighborhood, not just the city as a whole, is below what Zumper reports.
Comparing Zumper’s citywide medians to estimates from Zillow, which is generally regarded as one of the more accurate estimators of real estate prices, reveals a mixed bag. (We looked at numbers for September, 2015). In some cities, the two sources give roughly similar numbers: Zumper estimates the median listed one-bedroom apartment cost $2,110 in Washington, DC, versus Zillow’s estimate of $2,149; the estimates for Los Angeles are $1,830 and $1,850, respectively. But in many places, they’re quite different. In New York, it’s $3,160 versus $2,300; in Chicago, $1,920 and $1,550.
Zumper responded to our inquiries over Twitter and email. A spokesperson said that Zumper “stands firmly behind [its] rental data.” He added: “We have some of the strongest inventory from which to analyze…. We are reporting on true, asking rents seen in the market, and do not create an algorithm to estimate value.”
Of course, put another way, this is largely our point: Zumper takes the median from its listings, without compensating at all for the fact that its listings are disproportionately concentrated in higher-end neighborhoods. While it may be true that Zumper has a relatively large inventory of rental homes in its database, that’s akin to an online pollster saying that their polls must be accurate because they got so many votes. While quantity matters, at a certain point, quality—representativeness—matters much more.
On Twitter, Zumper’s CEO also told us that the National Rent Report focuses on the median apartment available for rent, and doesn’t claim to take into account apartments that are currently occupied. But as an explanation for Zumper’s concentration of listings in high-end neighborhoods, that doesn’t really pass the smell test: differences in housing turnover between wealthier and less-wealthy communities are several orders of magnitude too small to account for, say, the gap between the number of listings in the Near North Side and Jefferson Park. (Note that the Zillow estimates we described above are also for listed apartments.) Nor are claims that that gap is “in proportion to how many people move” to each of those neighborhoods plausible.
We should note that none of this is really a problem for Zumper’s main business, which is being a database for people looking for a place to rent. But it does mean that they should not be used as a reliable source for rental data, just as journalists shouldn’t report on real estate trends by simply adding up every listing on Craigslist.
Housing affordability issues are real, as we’ve written about here extensively, and the media absolutely should be reporting on home price trends, both locally and nationally. But precisely because these issues are so important, it’s crucial that the data that gets reported is reliable. Until it addresses the problems we’ve brought up here, Zumper’s rent reports are not, and journalists should be aware of that.
Related Commentary