Show Your Work: Getting DOT Traffic Forecasts Out of the Black Box
By Joe Cortright
Traffic projections used to justify highway expansions are often wildly wrong
The recent Wisconsin court case doesn’t substitute better models, but it does require DOTs to show their data and assumptions instead of hiding them
The road less traveled: Wisconsin Highway 23
There’s a lot of high-fiving in the progressive transportation community about last month’s Wisconsin court decision that stopped a proposed highway widening project. The reason? The state Department of Transportation (DOT) used inadequate traffic projections to justify the project.
The plaintiffs in the case were in a celebratory mood. Steve Hiniker, Executive Director of 1000 Friends of Wisconsin said “We have known for years that the state DOT has been using artificially high traffic forecasts to justify a number of highway expansion projects. Now a federal court has validated our claims.” Over at CityLab, Eric Jaffe calls it a court-ordered vindication of the peak car argument: “How Wisconsin residents cried peak car and won.”
But while the decision is hugely encouraging, it’s important to understand that 1,000 Friends of Wisconsin v. US DOT wasn’t a conclusive win for better traffic projections — the case was actually decided on different, much narrower grounds.
The federal district court ruling is really a take down of the opaque “black box” approach most state DOTs use in transportation forecasting. The project in question was a 20-mile long widening–from two lanes to four–of a stretch of state highway 23 between Sheboygan and Fond du Lac. The environmental group sued, charging that the Environmental Impact Statement prepared to justify the project and evaluate alternatives was based on faulty and outdated forecasts that overstated future traffic levels.
The court made it clear that it wasn’t in the business of adjudicating competing claims about the reasonableness of models or modeling assumptions. And it didn’t rule that 1,000 Friends of Wisconsin’s arguments about declining traffic or peak car or lower population projections trumped or invalidated Wisconsin DOTs modeling. What the court did do, however, was say that WisDOT failed to explain how its model worked in a way that the public (and the court) could understand. Essentially, the court ruled that Wisconsin DOT couldn’t use a “black box” to generate its projections — instead it had to present its data, assumptions and methodology in such a way that the public and outsiders could see how the results were produced. Judge Lynn Adelman wrote:
“In short, a reader of the impact statement and the administrative record has no idea how WisDOT applied TAFIS and TDM to produce the actual traffic projections that appear in the impact statement.” (page 12)
The court was unpersuaded by vague and repetitive blandishments offered in defense by the DOT about techniques and the mechanics of modeling methodology. The court specifically found that the DOT staff failed to explain how they arrived at the projected traffic volumes that appear in the impact statement, which seem to conflict with the recent trend of declining traffic volumes. And it found that:
“. . . the defendants repeated and elaborated on their general discussion of how TAFIS and TDM work and did not explain how those tools were applied to arrive at the specific traffic projections that appear in the impact statement.” (page 13).
It appears that the DOT’s position foundered over its inability to answer very basic questions about how a decline in population forecasts and a decline in recorded traffic levels squared with its modeling of future traffic levels. The Wisconsin DOT didn’t explain to the court’s satisfaction why it was sticking with the same level of traffic predicted for 2035, when population growth rate forecasts–which were supposedly a key input to the model–were reduced by two-thirds.
As a legal matter, the court went out of its way to state that it wasn’t about to second guess the methodology and assumptions chosen by the state DOT. Here the court ruled, as other courts have, that unless the methodology is “irrational,” it’s not in violation of the National Environmental Policy Act (NEPA).
While it falls short of a legal vindication of the “peak car” argument, requiring DOTs to open up their “black box” forecasts is still likely to be a devastatingly important ruling. Official DOT traffic forecasts are frequently presented as the product of a special kind of technical alchemy. While model results are clothed with the illusion of precision (“this road will carry 184,200 cars in 2035”), there’s really much, much more ambiguity in the results. To pass muster under NEPA, the process used for calculating future traffic levels will now likely be laid bare.
Those who’ve worked with traffic models know that they’re clumsier, clunkier and more malleable than the precise, hyper-technical image that traffic engineers (or politically appointed transportation agency officials) typically paint of them in the introductions to environmental impact statements. The numerical outputs from computer simulations, for example, are often subjected to “post-processing” — the preferred euphemism for manually changing predicted traffic levels based on the judgment of the modeler (or the desires of the modeler’s client.)
And there’s lots of room for manipulation. In his book, “Toll Road Traffic and Revenue Forecasts” Rob Bain, a pre-eminent international expert on traffic forecasting, lists 21 different ways modelers can inflate traffic forecasts and concludes “it is perfectly possible to inflate the numbers for clients who want inflated numbers” (page 75).
In practice, DOTs have often used traffic forecasts as a sales tool or a rationalization for new projects. Once the traffic modeling generates a sufficiently high number to justify additional capacity, the agencies stick with it in spite of evidence to the contrary. For the proposed $3 billion Columbia River Crossing between Oregon and Washington, the two state DOTs stuck with exaggerated vintage 2005 forecasts in a final environmental impact statement issued in 2013; ignoring actual declines in traffic that had occurred in the intervening years. And as in Wisconsin, they offered no explanation as to why the modelling didn’t change.
For years, we’ve known that DOT traffic forecasting models are frequently wrong and that they regularly over-estimate future traffic and congestion. Multi-billion dollar projects are often predicated on traffic forecasts that fail repeatedly to be borne out by reality. The Sightline Institute showed that for Washington’s SR-520 floating bridge project, the state always forecast a big increase in traffic, even though traffic levels continually declined.
The political acceptance of these kinds of errors is rampant in the industry. The State Smart Transportation Institute analyzed an aggregation of state traffic forecasts prepared annually by the US DOT showed that the 20-year projections overestimated future traffic volumes in every single year the reports could be compared against data on actual miles driven by Americans.
A big part of the reason these flawed forecasts have continued to be made–and not corrected–is that the forecasting process is opaque to outsiders. The federal district court’s ruling in 1000 Friends of Wisconsin v. U.S. DOT should make it much more difficult for highway builders to continue justifying projects based on this kind of “black box” modeling. As the old saying goes: sunlight is often the best disinfectant. Greater transparency in the data and assumptions that underlie traffic forecasts could lead to much wiser decisions about where to invest scarce transportation resources.
Show Your Work: Getting DOT Traffic Forecasts Out of the Black Box
The road less traveled: Wisconsin Highway 23
There’s a lot of high-fiving in the progressive transportation community about last month’s Wisconsin court decision that stopped a proposed highway widening project. The reason? The state Department of Transportation (DOT) used inadequate traffic projections to justify the project.
The plaintiffs in the case were in a celebratory mood. Steve Hiniker, Executive Director of 1000 Friends of Wisconsin said “We have known for years that the state DOT has been using artificially high traffic forecasts to justify a number of highway expansion projects. Now a federal court has validated our claims.” Over at CityLab, Eric Jaffe calls it a court-ordered vindication of the peak car argument: “How Wisconsin residents cried peak car and won.”
But while the decision is hugely encouraging, it’s important to understand that 1,000 Friends of Wisconsin v. US DOT wasn’t a conclusive win for better traffic projections — the case was actually decided on different, much narrower grounds.
The federal district court ruling is really a take down of the opaque “black box” approach most state DOTs use in transportation forecasting. The project in question was a 20-mile long widening–from two lanes to four–of a stretch of state highway 23 between Sheboygan and Fond du Lac. The environmental group sued, charging that the Environmental Impact Statement prepared to justify the project and evaluate alternatives was based on faulty and outdated forecasts that overstated future traffic levels.
The court made it clear that it wasn’t in the business of adjudicating competing claims about the reasonableness of models or modeling assumptions. And it didn’t rule that 1,000 Friends of Wisconsin’s arguments about declining traffic or peak car or lower population projections trumped or invalidated Wisconsin DOTs modeling. What the court did do, however, was say that WisDOT failed to explain how its model worked in a way that the public (and the court) could understand. Essentially, the court ruled that Wisconsin DOT couldn’t use a “black box” to generate its projections — instead it had to present its data, assumptions and methodology in such a way that the public and outsiders could see how the results were produced. Judge Lynn Adelman wrote:
“In short, a reader of the impact statement and the administrative record has no idea how WisDOT applied TAFIS and TDM to produce the actual traffic projections that appear in the impact statement.” (page 12)
The court was unpersuaded by vague and repetitive blandishments offered in defense by the DOT about techniques and the mechanics of modeling methodology. The court specifically found that the DOT staff failed to explain how they arrived at the projected traffic volumes that appear in the impact statement, which seem to conflict with the recent trend of declining traffic volumes. And it found that:
“. . . the defendants repeated and elaborated on their general discussion of how TAFIS and TDM work and did not explain how those tools were applied to arrive at the specific traffic projections that appear in the impact statement.” (page 13).
It appears that the DOT’s position foundered over its inability to answer very basic questions about how a decline in population forecasts and a decline in recorded traffic levels squared with its modeling of future traffic levels. The Wisconsin DOT didn’t explain to the court’s satisfaction why it was sticking with the same level of traffic predicted for 2035, when population growth rate forecasts–which were supposedly a key input to the model–were reduced by two-thirds.
As a legal matter, the court went out of its way to state that it wasn’t about to second guess the methodology and assumptions chosen by the state DOT. Here the court ruled, as other courts have, that unless the methodology is “irrational,” it’s not in violation of the National Environmental Policy Act (NEPA).
While it falls short of a legal vindication of the “peak car” argument, requiring DOTs to open up their “black box” forecasts is still likely to be a devastatingly important ruling. Official DOT traffic forecasts are frequently presented as the product of a special kind of technical alchemy. While model results are clothed with the illusion of precision (“this road will carry 184,200 cars in 2035”), there’s really much, much more ambiguity in the results. To pass muster under NEPA, the process used for calculating future traffic levels will now likely be laid bare.
Those who’ve worked with traffic models know that they’re clumsier, clunkier and more malleable than the precise, hyper-technical image that traffic engineers (or politically appointed transportation agency officials) typically paint of them in the introductions to environmental impact statements. The numerical outputs from computer simulations, for example, are often subjected to “post-processing” — the preferred euphemism for manually changing predicted traffic levels based on the judgment of the modeler (or the desires of the modeler’s client.)
And there’s lots of room for manipulation. In his book, “Toll Road Traffic and Revenue Forecasts” Rob Bain, a pre-eminent international expert on traffic forecasting, lists 21 different ways modelers can inflate traffic forecasts and concludes “it is perfectly possible to inflate the numbers for clients who want inflated numbers” (page 75).
In practice, DOTs have often used traffic forecasts as a sales tool or a rationalization for new projects. Once the traffic modeling generates a sufficiently high number to justify additional capacity, the agencies stick with it in spite of evidence to the contrary. For the proposed $3 billion Columbia River Crossing between Oregon and Washington, the two state DOTs stuck with exaggerated vintage 2005 forecasts in a final environmental impact statement issued in 2013; ignoring actual declines in traffic that had occurred in the intervening years. And as in Wisconsin, they offered no explanation as to why the modelling didn’t change.
For years, we’ve known that DOT traffic forecasting models are frequently wrong and that they regularly over-estimate future traffic and congestion. Multi-billion dollar projects are often predicated on traffic forecasts that fail repeatedly to be borne out by reality. The Sightline Institute showed that for Washington’s SR-520 floating bridge project, the state always forecast a big increase in traffic, even though traffic levels continually declined.
The political acceptance of these kinds of errors is rampant in the industry. The State Smart Transportation Institute analyzed an aggregation of state traffic forecasts prepared annually by the US DOT showed that the 20-year projections overestimated future traffic volumes in every single year the reports could be compared against data on actual miles driven by Americans.
A big part of the reason these flawed forecasts have continued to be made–and not corrected–is that the forecasting process is opaque to outsiders. The federal district court’s ruling in 1000 Friends of Wisconsin v. U.S. DOT should make it much more difficult for highway builders to continue justifying projects based on this kind of “black box” modeling. As the old saying goes: sunlight is often the best disinfectant. Greater transparency in the data and assumptions that underlie traffic forecasts could lead to much wiser decisions about where to invest scarce transportation resources.
Related Commentary