These dynamics have driven up rents and kept asking prices for homes low in most markets. Though the real estate market is, as the saying goes, all about location.

In some of the country’s most economically vibrant locations, like New York City or San Francisco, housing isn’t affordable, especially for middle-class earners. You don’t have to be an economist to come to this conclusion, but Jed Kolko, chief economist at Trulia, released a new report Tuesday that lays out in black and white just how difficult it is for middle-class Americans to achieve homeownership in certain parts of the country.

MORE: After 8 years, the real estate market is finally looking normal again

Kolko studied the 100 largest metro areas in the U.S. and calculated the percentage of listed homes a family earning the median salary (based on location) could afford by spending less than 31% of its income on housing (including mortgage, insurance, and property taxes). You can see the least affordable cities for the middle class in the chart below:

As you can see, the most populous areas of California aren’t great for middle-class folks looking to buy a home, and San Francisco represents the epitome of California’s affordability problem.

Further down the list, there are other metro areas with less publicized issues with housing affordability. For instance, in Boston, only 41% of the homes are affordable for middle-class earners in that area, while fewer than half the homes in Houston, Dallas, and Miami are affordable for the middle class.

The most affordable metro areas tend to be clustered in the Midwest and South, and in cities where the local economy isn’t thriving relative to the rest of the country:

It’s no coincidence that the cities on the list above are places that are shrinking in size and economic clout. Akron, Ohio’s population, for instance, peaked in 1960, when the city was still a major manufacturing hub. And these cities are great examples of why it’s sometimes unwise to talk about the real estate market in America as a whole. Does it matter that housing in America overall looks cheap if homes in the most economically vibrant locations aren’t?

In fact, the lack of affordability in areas that are powering the U.S. economy has some economists worried about America’s capacity for stronger economic growth going forward. Workers earn more money in cities like New York or San Francisco because their labor is more valuable when they are working in close proximity to other valuable workers and because of the wealth of infrastructure contained in large cities. But if cities are unable to grow and add housing supply to meet demand, that means that workers across the country and the overall economy aren’t realizing their potential.

MORE: America’s thorny affordable housing crisis

The problem, as Kolko points out in his report, is that it’s difficult to determine how much the lack of housing supply growth in places like San Francisco and New York is the result of government regulation versus geographical limitations (like the San Francisco Bay). Furthermore, the politics surrounding increasing density are as fraught as they get. Writes Kolko:

While expanding the housing supply would improve affordability, new construction can change the character of a place in many ways: more congestion; a different skyline; an influx of new residents who are richer, poorer, darker, or whiter than current residents; as well the inconvenience during construction itself. Furthermore, not everyone wants to improve affordability. Another name for “housing costs” is “property values,” and current homeowners and landlords want property values to stay high. The beneficiaries of improved affordability are renters (though less so if they’re already protected by rent regulations); people who are priced out of expensive markets and therefore live elsewhere; and future generations – who usually have less political power and influence than homeowners and landlords. Plus, even people who might benefit from greater affordability might care more about preserving the character of their neighborhood.

There are plenty of reasons to resist change in economically vibrant cities, but Kolko’s report underscores the cost of that obstinance: middle-class Americans struggling to get by in the places where they have best chance to thrive.