Luxury stocks are often thought of as a defensive investment. Wealthy people buy fancy handbags and glittery watches even during a recession. But over the past several years, luxury companies branched out by targeting“aspirational buyers” — those with income between $30,000 and $100,000 a year. Though the middle class hardly spends as much per purchase, there are simply more aspiring rich than actual rich. The masses add up to quite a lot.

This has been a lucrative new group for luxury brands, particularly in China where there is now a burgeoning middle class. Middle class accounts for 38% of global luxury goods spending, according to recent estimates from Goldman Sachs. These purchasers in China spend more too, $782 on average versus $241 annually. China’s potential market for middle class buyers has gone from 1 million people in 1995 to 37 million today. Goldman expects it to grow to 256 million in 2025, at a CAGR of 14.1%.

MORE: Buffett pulls ahead in wager against hedge funds

Stocks have reflected this growth. Prada (PRDSY) is up more than 140% since fall 2011. Richemont is up nearly 75%. Overall the Bloomberg European Luxury Goods Index, which includes other stocks like Tod’s and Salvatore Ferragamo, is up more than 50%. The Dow Jones Industrial Average has risen about 30% over the same period.

But critics question if the growth can continue, for several reasons. First, China’s housing boom has helped line some pockets. But the country has more stringent mortgage restrictions, such as recourse loans for individuals and high down payments. These stipulations have curtailed buyers over the past few years, creating an oversupply of building. Any decline in this market could chip away at the buying power of some middle-class Chinese.

That slowdown isn’t showing yet, but UBS has been warning about its risks for a while. “Given the sheer size of the property and construction sector and its heavy role in driving growth, even just having housing demand ‘fade away’ because of changes in policy or asset preferences could already lead to a hard landing of the overall economy,” said China economist Tao Wang in a March 2011 report.

There are other complications as well. Some people were buying luxury products as presents for local officials in exchange for rights to build. But the Chinese government is making strides to cut down on the practice of “gifting.” And retailers have to grapple with other idiosyncrasies. For example, unlike in Japan, historically Chinese luxury goods buyers are mostly men. Though some say this is changing, men may not be quite as loyal — or spontaneous — customers.

MORE: The judge who rules on business

All told, luxury retail sales in China have slowed dramatically over the past several months. In mid-2011, retail sales growth for jewelry, watches, and clocks grew faster every month than retail sales overall, hitting nearly 50% annual growth in mid-2011, according to statistics from the Hong Kong Census & Statistics Department. But by January 2012, growth in this segment slowed to a slower pace than retail sales. In five of the last seven months leading up to October last year, luxury retail sales actually declined.

Goldman Sachs stands by its bullish estimates, even in light of this.

“In 2012, luxury demand in China has slowed, with most companies now reporting mid-single-digit to low-double-digit growth rates in the country,” analyst William Hutchings said in a recent report. “We believe that this slowdown has masked the underlying trends, as there has been a significant transfer of demand from Asia to Europe as the take-off in travel trends has coincided with a weaker euro.”

But what would happen to luxury stocks if the critics were right, and middle-class Chinese consumers suddenly stopped buying these goods? Consider this back-of-the-envelope analysis: Goldman Sachs estimates that the Chinese middle class contribute 7% of global demand for luxury goods. Let’s assume this is the case for Richemont, who declined to comment for this article. The €29 billion ($38.6 billion) company should make €10.2 billion ($13.6 billion) in revenue this year, according to Goldman Sachs, but sales would fall by €700 million ($932 million) if the Chinese middle-class buyer suddenly went away. Let that work its way into valuation, assuming everything else stays the same, and an investor would be buying the stock at 11 times earnings before interest, tax, depreciation, and amortization, versus 10 times. That’s the top of where Richemont has traded for the last several years.

Of course, this analysis isn’t perfect. The middle class probably wouldn’t go away altogether, or all of a sudden. And some early reports from retailers show that fourth quarter sales may have actually picked up. But Goldman also estimates that the Chinese middle class could contribute as much as 17% of global demand by 2025. To the extent that this hyper growth is already being factored into the stock price, it could fall even more dramatically if demand tapered off.

In a somewhat odd break from the norm, companies are sounding the alarm ahead of investors. Last September, Burberry (BBRYF) issued a warning on its profits, partly because sales were slowing in China. “It’s not necessarily being felt by all of our peers, but we’re certainly not alone,” Burberry Chief Financial Officer Stacey Cartwright told the Wall Street Journal at the time. “Yes, we are seeing a slowdown in Asia, and yes, China is a significant contributor to that.”

Last year, the chief executive of Richemont Johann Rupert admitted that the company may be dangerously vulnerable to one market. “I feel like I’m having a black tie dinner on top of a volcano,” said Johann Rupert, executive chairman and CEO. “There is a volcano somewhere, whether it’s this year, in ten years’ time, or in twenty years’ time. We are exposed to China.”