There’s no statistically significant news on young adults in cities in the latest Census release

Pro-tip:  Ignore changes smaller than the margin of error: they’re noise, not signal 

It’s hard to underestimate the journalistic zeal to report a contrarian story. In the world of urbanism, a recent favorite has been “young adults are leaving cities.” We’ve repeatedly debunked previous iterations of this particular species of weed, but unsurprisingly, it’s back.

The Wall Street Journal’s Janet Adamy and Paul Overberg produced a provocative story last week:  Young adults, they claim, are leaving the nation’s cities in “large numbers.”  The on-line story is slugged: “Millennials Continue Their Exodus From Big U.S. Cities.” They write:

Large U.S. cities lost tens of thousands of millennial and younger Gen X residents last year, according to Census figures released Thursday that offer fresh signs of cooling urban growth.  Cities with more than a half million people collectively lost almost 27,000 residents age 25 to 39 in 2018, according to a Wall Street Journal analysis of the figures.

The story drew on just-released data from the 2018 American Community Survey to compute the total number of 25 to 39 year-olds moving into and out of in cities with more than 500,000 residents in the past two years.  The main finding:  according to the WSJ analysis, the number of 25-39 year old residents declined by 27,000 in these cities between 2017 and 2018.

“Tens of thousands” sounds pretty impressive number, unless you understand that there are more than 10 million 25 to 39 year olds living in cities with a population of 500,000 or more. (This basic contextual statistic is not included in the WSJ article.) So a decline of 27,000 represents a decrease on the order of one-quarter of one percent.  That’s neither “large” nor “an exodus.”

More noise than signal:  Smaller change than the margin of error

There’s another reason to pay particular attention to the fraction of percent of change here:  It is so small that it’s statistically indistinguishable from zero.  Because the data are taken from a survey (the American Community Survey samples about 1-2 percent of all US households each year), it has a margin of error.  The margin of error for city-level estimates of age group population is on the order of 4 to 5 percent, meaning that any change of less than that amount is likely to be statistically insignificant.  A one-quarter of one percentage point decline is essentially indistinguishable from statistical noise.

Reporting a much smaller than margin of error change in a population subgroup as showing that young adults are leaving cities is the statistical equivalent of using a winter snowstorm as evidence that the global climate is not warming.

For at least one city, more straightforward Census data show increases

Inasmuch as we’re based in Portland, we follow Portland data closely.  We were surprised that the Wall Street Journal article claimed that Portland was a city that lost population in the 25 to 39 age group:

New York, Chicago, Houston, San Francisco, Las Vegas, Washington and Portland, Ore., were among those that lost large numbers of residents in this age group.

We double-checked the Portland data from the Census website, examining one-year American Community Survey estimates for each of the years 2014 through 2018..  You can download data for any large city in the US at data.census.gov.  We got age-sex population estimates from Table B01001.  These data show that the total number of 25 to 39 year olds was 187,483 in 2017 and 188,322 in 2018.  This represents an increase of 839 persons in this age group; not a decrease as reported by the WSJ.  These one-year survey data show such a small change–about 0.4 percent–that it is not significantly different from zero.  This isn’t surprising–there’s a lot of variability due to random sampling in these figures. What this suggests is that the one-year data provide no basis for a claim that Portland is losing population in this demographic group.

The Census Bureau’s five-year American Community Survey also contains estimates of the number of 25-39 year olds living in the City of Portland.  The aggregated five-year data are preferred because the larger sample size produces estimates that have a smaller margin of error, although they have a noticeable lag. The latest estimates run through 2017; the 5-year data through 2018 will be released later this fall.  These data show a steady increase in the 20-39 year old population in Portland.

We spoke with the Wall Street Journal reporter responsible for the data analysis in this story. The WSJ used a different method for computing population change by age group. They combined data from two different tables (one dealing with in-migration and a second covering out-migration–Tables B07001 and B07041).  By comparing the differences in the two tables (i.e. looking to see whether the number of migrants into a city exceeded the number of migrants out of a city) they computed population change.

The difficulty with this approach is two-fold.  First, because the Census is applied only to current residents of the US, those who move out of the country in a given year aren’t surveyed.  Second, the margin of error of the estimate for in-migrants and out-migrants is much larger because there are fewer than 10 percent of the US population moves each year.  For example, for Portland, Census reports that the margin of error for estimates of in- and out-migrants is between 30 and 40 percent and exceed 100 percent for migrants from other countries.  These very large margins of error make it much more likely that computed changes in point estimates are the result of statistical noise rather than some underlying trend.

Average Margin of Error, 5-year age groups, 20-39
City of Portland
American Community Survey, 2018

Given the much smaller error margins associated with aggregate population estimates, and the simplicity of working with total population figures from year to year, we believe that this is a much more accurate way to portray population change for a particular city.  The Wall Street Journal did not publish data for particular cities (other than New York), nor did it post its data tabulations, so we don’t know if similar differences between the net migration method used by WSJ and the year-over-year total change method we prefer hold for other cities as well.

Mixed signals:  Some cities up, other cities down

While the story headlines a so-called exodus, there is an admission–in its very last paragraph–that several cities are continuing to see an increase in 25 to 39 year olds:

Among the big cities that gained large numbers of young adults were Los Angeles, Phoenix, San Antonio, San Diego, Austin, Seattle, Denver and Columbus.

So apparently, the pattern isn’t so pronounced that eight of America’s largest cities aren’t still gaining young adult residents.  The Wall Street Journal story lists just seven cities that experienced declines in 25 to 39 year-olds by their calculations, so its difficult to argue from what’s presented here that the most common pattern is out-migration. Because the story doesn’t show city level statistics–it contains specific  for only one city (New York)–it’s impossible to tell whether the number of large cities gaining 25 to 39 year olds is more or less than the number that are losing them (again, for the moment, ignoring the fact that many, if not all of these year-over-year changes in the Census point estimates are likely to be statistically insignificant, and may be an artifact of relying on the net migration estimates, rather than the more reliable year-over-year total population change).

If anything, the very limited data on individual cities contradicts the thesis of the article.  Note that nationally, the aggregate net decline in big city 25-39 year old population was 27,000; The authors report that New York City alone lost 38,000.  This implies that outside of New York City, big cities as a whole actually gained population in the 25-39 age group. (If New York lost 38,000, and big cities collectively lost 27,000, then big cities other than New York gained a net of 11,000 residents).

This fact underscores two important points:  First, that contrary to the impression generated by the Wall Street Journal article, there doesn’t appear to be any generalized trend–their main result, if it is accurate, is driven in large part by New York City.  It also appears that by the WSJ’s own analysis, the “trend” has dissipated sharply in the past year, they say net outmigration from this group of large cities has declined by about half from the previous year, from 54,000 in 2017 to 27,000 this year.  If this were an emerging trend, we’d expect it to accelerate, not fall by half.

Second, we’re talking about really small, and certainly statistically insignificant changes:  outside of New York City, the aggregate change of 10,000 persons is just one-tenth of one percent of the nearly 10 million 20-39 year olds living in those cities; well beyond the statistical precision of the American Community Survey.

These very tiny changes and this kind of noisy, uneven data pattern is exactly what one would expect from sample data.  Point estimates of individual variables fluctuate from year to year, and don’t necessarily represent real changes.  It’s particularly unfortunate that the article repeatedly refers to changes as “large” even though there’s no evidence that they are statistically significant, i.e. not discernible from zero. If the population of a particular subgroup were actually un-changed from year to year in all cities, statistically you would expect a survey like the ACS to report higher point estimates for half of cities and lower point estimates for the other half solely because of sampling variability.  Not allowing for the possibility of this high level of statistical noise leads to a conclusion that simply isn’t supported by the data.

In sum, when your survey-based estimates don’t reach a threshold level of statistical significance, when they’re contradicted by other, more straightforward ways of making a similar calculation (ones with a lower margin of error), when even by your own calculations at least eight major cities are recording increases in the young adult population, and when the trend you claim to have spotted has declined by 50 percent in the past year, it’s probably not valid evidence for claiming that there’s an exodus of young adults from the nation’s cities. We understand the journalistic temptation to want to be the first (or among the first) to make a provocative contrarian call about some well-known trend, but in this case the evidence is just too thin to be believable.

In addition, despite the desire in some quarters to treat any tapering off or decline in young adult migration to cities as evidence of dis-enchantment with urban living, or a renewed desire for living in the suburbs, it’s worth noting that the evidence of market prices shows a strong and growing demand to live in cities. Rents have risen fastest and most in the centers of large cities, and the fact that the urban migration of young adults has continued apace, even in the most expensive markets, shows that the chief limit on movement of young adults to cities is a lack of supply of housing, not any diminution of the demand for urbanity. So far from signaling a rekindling of the supposed suburban romance, a slowing (or decline) in the number of young adults in cities most likely indicates that we’re bumping up against the limits of housing supply in great urban neighborhoods; rather than writing off cities, we need to be building more housing in them.

Editor’s Note (October 10):  This post has been revised to correct a typographical error in the paragraph describing the net aggregate increase in young adults in large cities other than New York.