Even the most casual consumer of urban news can’t avoid reading articles about whether rents in their city are up, or down, and how they compare to other cities around their country. Unfortunately, the vast majority of these rent estimates are completely made up.

As we’ve written, the proliferation of these rent stories seems to be driven by a growing number of online real estate startups, who have figured out that they can get free publicity by using their listings to create “median rent reports.” The problem is that, for a number of reasons, even the most comprehensive listing agency will be missing a lot: many apartments aren’t officially listed anywhere, renting by word of mouth or a simple sign hanging on a front gate. Moreover, most of these sites are targeting younger, higher-income renters, which might go some way to explaining why they have hundreds or thousands of listings in some neighborhoods, and only a small handful in others. And even beyond that, the fact that apartments listed above market price will tend to stay unrented longer, while relatively cheaper apartments will get quickly snatched up, will bias any simple average upwards.

Screen Shot 2016-03-23 at 10.47.19 AM

Screen Shot 2016-03-23 at 10.47.31 AM
Simply comparing rental figures from different companies—like these infographics from Abodo and Zumper—shows wildly differing rents.

 

For both journalists and news consumers, then, we’ve put together a short field guide for rent statistics: who makes them, and the advantages or disadvantages of each.

HUD Fair Market Rents

How it’s made: HUD creates “fair market rent” estimates for metropolitan areas in order to determine how much money Housing Choice Vouchers can be worth in each region. The estimates are for rental units at the 40th percentile of price—that is, cheaper than 60 percent of all units—by number of bedrooms, for the entire metro area. The formula is complicated, but involves taking the most recent American Community Survey 5-year estimate, and then adjusting it based on the most recent ACS 1-year estimate for recent movers, a local inflation measure, and forecasted national rent trends.

Advantages: Theoretically, the ACS samples that Fair Market Rents are based on really do contain the entire universe of apartments (with one caveat—see below), removing the problem of units that are never listed. The estimates are also designed to estimate the median price of a newly rented unit, as opposed to all units—which include a number with longstanding tenants who are probably paying less than those who recently moved—like American Community Survey figures.

Disadvantages: ACS numbers come out with quite a bit of lag, so the 2016 Fair Market Rent calculations are based on ACS figures collected no later than 2013—and the 5-year estimate began collection in 2009. The adjustments are meant to make up for that gap, but obviously that introduces quite a bit of room for error. It also makes FMR unhelpful for trying to track up-to-the-minute trends. For a long time, FMRs were also only made at the metropolitan area level, so neighborhood-to-neighborhood comparisons were impossible. More recently, that has changed—see the next entry. Also, FMR is calculated in a way that is designed to filter out subsidized units, which pushes its figures up a bit. Finally, it can’t tell you about relatively low-cost or high-cost housing.

Can answer: What are the most expensive metropolitan areas to begin a new lease in?

Can’t answer: What are the most expensive neighborhoods in my city? Which metropolitan areas have seen the fastest rent growth in the last year?

HUD Small Area Fair Market Rents

How it’s made: HUD began calculating sub-metropolitan area rent estimates in order to make it easier for voucher holders to move into “opportunity areas,” where market prices were too far above the regional median for vouchers to be usable. The “small areas” are ZIP codes, whose Fair Market Rents are otherwise calculated in a similar way as the metro area estimates.

Advantages: Like regular FMRs, coverage of non-listed apartments and an attempt to measure the cost of beginning a lease now, as opposed to including longtime leaseholders. And, obviously, the ability to compare neighborhood-to-neighborhood prices.

Disadvantages: The same time-lag issues as regular Fair Market Rent calculations. Also, as you go to smaller levels of geography, ACS margins of error become much larger. HUD considers an estimate “reliable” if its margin of error is less than half of the value of the estimate itself—that is, if the ACS says a ZIP code’s median rent for one-bedroom apartments is $1,000, HUD considers that “reliable” if the margin of error is less than $500—and while margins of error are much lower than that in most urban neighborhoods, caution is advised. Detecting small differences between neighborhoods, or from one year to the next, isn’t really possible. Still can’t tell you about relatively low-cost or high-cost housing.

Can answer: Broadly speaking, what are the most expensive ZIP codes in my metropolitan area, and how do they compare with the most expensive ZIP codes in another metropolitan area?

Can’t answer: Is the median apartment in this ZIP code $20 more or less expensive than the median apartment in that ZIP code? What does the typical low-income person pay in this ZIP code?

American Community Survey

How it’s made: The Census’ American Community Survey replaced the long-form decennial questionnaire, and surveys US residents every year on a variety of issues, including rent. The responses are rolled up into three main products: 1-year estimates, 3-year estimates, and 5-year estimates. The 3- and 5-year estimates are necessary to increase sample sizes, and create more reasonable margins of error, for smaller geographic areas.

Advantages: Like Fair Market Rents, the ACS theoretically gets around the problem of unlisted apartments. Perhaps most valuably, it publishes more than just median figures, allowing you to compare estimates at, say, the 25th percentile, which gives more of an insight into what lower-income people might be paying. Using IPUMS, you can also cross-reference ACS data to find median rents by particular demographics. Also, because the Census has been using similar data for a number of years, you can do some historical analysis.

Disadvantages: The ACS has quite a time lag—at the moment, the most recent data released is from 2014—and if you’re interested in smaller levels of geography, like neighborhoods or even smaller suburbs, you probably need to use multiyear estimates that include survey responses from over five years ago. Also, because the estimates include all renters, they may be biased downwards compared to the rents facing someone looking to begin a lease.

Can answer: Roughly, how do rents for low-cost apartments compare from one city, or one neighborhood, to the next? How has the geography of high-rent apartments changed over the last 10 or 20 years?

Can’t answer: How much did rents increase in my neighborhood, or my city, last year? What is the median rental price facing people on the rental market today?

Zillow

How it’s made: Zillow’s rental estimates are based on their own listings, but crucially, they also add their own proprietary modeling that adjusts for the changing mix of listings available at any given time.

Advantages: Unlike FMR or the ACS, Zillow’s numbers are based off of nearly real-time data on rental listings, allowing for much more timely estimates. The modeling also introduces a necessary corrective to straight averages from available listings, which can vary wildly based on what happens to be available at any given time. Especially in larger markets, Zillow also makes available lots of breakdowns of the data, including by metro area, ZIP code, number of bedrooms, and “market tier,” allowing you to compare relatively low- or high-cost apartments in different areas. You can download the entire national dataset as a CSV file.

Disadvantages: Because Zillow’s figures are based on listings, they are missing any unlisted units, which will probably tend to bias Zillow’s estimates upwards. Rental estimates are also only available going back to about 2010 in most areas.

Can answer: Roughly speaking, how have rents changed over the last year in my city compared to another city? What are the most expensive parts of my city? How has that changed in the last few years?

Can’t answer: What is the long-term trend in rents in my city? What is the exact median rent for my neighborhood or city?

Most apartment listings services (Zumper, Abodo, etc.)

How it’s made: Many of these services appear to simply use the median rent from their own listings without any adjustments. We say “appear to” because the descriptions of the methodologies that they use to generate their rents are typically quite sketchy, and simply make very vague references to their database listings.

Advantages: Avoid the guesswork that comes with modeling.

Disadvantages: Suffers from all of the biases and problems of raw listings, including a tendency to dramatically over-sample higher-end neighborhoods and apartments because of selection and survivorship biases (see the top of this post for more), and a vulnerability to random changes in the composition of available apartments from one time period to the next.  

Can answer: What is the median price of apartments currently listed by this company?

Can’t answer: Anything else.

Market analysis firms (Yardi Matrix, Rainmaker Insights, CoStar, etc.)

Unlike apartment listings services, whose main product is the ability to search for particular apartments, market analysis firms specialize in giving big-picture snapshots of the real estate market to businesses. Perhaps because their livelihood is their data analysis, they are somewhat less forthcoming about their proprietary methodologies on their websites (and most of their actual data is behind a paywall), and so there isn’t as much we can say about them. We have reached out to several to ask about their methodologies, and if we receive responses, we will update this post with them.


Three questions to ask

Obviously this isn’t a comprehensive list of all the people who might try to tell you what median rents are. But if you’re a journalist (or an enterprising reader), and somebody is trying to tell you that they have rental figures, here are three questions to ask:

  1. Where did the data come from? The answer will generally be either surveys or listings. Surveys have margins of error, which can get quite large, especially for smaller geographic areas; they should be willing to tell you what they are, and how big the samples are. If it’s listings, they should be able to tell you how they dealt with all the biases we’ve talked about that are inherent to listings.
  2. If it’s listing data, is this just an average of all your listings? If the answer is yes, you can safely ignore any “median rent” claims. If not, they should be able to explain how they deal with under-representation of low-rent, unlisted apartments; survivorship bias that over-representation of high-rent apartments that take a long time to lease; and noise in the basket of apartments that happen to be available at any given time.
  3. Why are your numbers different from other sites’? This is the bottom line: Given that there are many different rental figures, they should have a good answer for why theirs are the most accurate. And “we have the most listings” is not a good answer, any more than the most accurate poll is the one with the most respondents. Quality—including ways to address all the pitfalls we’ve brought up in this post—matters more than quantity.