But some investors are questioning the timing of such a strategy now. Many wonder if the recent pullback in luxury sales is a temporary anomaly or the start of a deeper downward spiral.

“If business is softening up, that’s a bad time to go public,” says Howard Davidowitz, chairman of Davidowitz & Associates Inc., a retail consulting and investment banking services firm. “Investors want a growth story.”

The luxury retail market had been on a tear for several years, making luxury retailers, such as Neiman Marcus, potentially attractive investments.

MORE: Just how risky is Amazon’s debt?

During the financial crisis, the sector posted double-digit sales declines for 12 consecutive months before starting to recover in late 2009, according to Michael McNamara, vice president of research and analysis at MasterCard Advisors SpendingPulse, which tracks retail sales in the cash, check and credit card markets. Luxury sales climbed 6% in 2010 and another 9.9% in 2011. They remained resilient in 2012, although the climb has been rockier. Neiman enjoyed similar gains, with revenue rising more than 8% in both fiscal 2011 and 2012 and earnings growing to $31.6 million and $140.1 million respectively (it posted a $668 million loss in 2009).

As the capital markets rebounded, so did luxury spending. “Wealthy people have a higher proportion of their wealth invested in the capital markets and over the last three of four years, they’ve made lots of money,” Davidowitz says.

However, that run hit a wall in November, when the luxury retail sector saw sales abruptly decline 1.6%, says McNamara. Sales took a hit in the wake of hurricane Sandy, the uncertainty leading up to the federal election and, more recently, over concerns about the fiscal cliff.

Neiman, which operates 42 Neiman Marcus stores, two Bergdorf Goodman stores and 39 smaller Last Call and Cusp stores, began seeing sales slow as early as August, although it still posted year-over-year increases in the latest quarter.

MORE: Why stores are finally turning on to WiFi

“Although our sales were strong for the quarter, they were not as robust or as consistent as we would have liked, and the cadence of business throughout the quarter was a little choppy,” said Karen Katz, president and chief executive of the Neiman Marcus Group, during a conference call earlier this month to discuss the company’s fiscal first quarter earnings.

Neiman’s first fiscal quarter of 2013, which ended Oct. 27, doesn’t include the fallout from Hurricane Sandy or the growing jitters about the fiscal cliff, which gained steam in November. Katz blamed the economy and volatile capital markets for the sales slowdown.

“Our customers are very attuned to the fluctuations of the markets, both here and abroad,” she said. “They also closely followed the election and continue to weigh the news from Washington and Congress regarding the economy, the debt crisis, and possible changes to tax laws.”

There are definitely links between stock market performance and luxury sales levels, says McNamara. “When we were coming out of the recession, the bottom of the stock market coincided with the bottom of the luxury retail sales landscape.”

Neiman executives have taken a number of steps over the past two years to boost the company’s sales and outlook, such as bringing in an e-commerce heavy-hitter from Williams-Sonoma to head up the company’s online business, rolling out the Cusp brand to attract hip younger customers, accepting credit cards other than Neiman Marcus and Amex, acquiring a stake in Chinese fashion website Glamour Sales, boosting the company’s social marketing efforts, opening up new outlet stores and recently teaming up with Target to launch a special holiday collection.

Some of these strategies are viewed as hits – others misses. The Chinese website investment is expected to help the company tap the lucrative Chinese luxury market. The Cusp brand and other initiatives aimed at bringing in younger shoppers are also viewed as a plus, as they offer a way to expand beyond the company’s traditionally older customer base.

MORE: The race to kill the plastic gift card

However, rolling out outlet stores and partnering with Target have raised some eyebrows. Swinand isn’t convinced that ramping up outlet stores was wise for the brand. “Everybody is skating to where the puck is instead of where it’s going to be,” he says. “All my retailers are saying they want to grow outlets – Nordstrom, Saks, Coach – but if you’ve been around long enough, it’s a classic Wall Street screw-up where everybody pushes it until it breaks and then they’ll all go – why did we build so many outlets? It’s going to end badly.”

Swinand also believes the outlets along with the Target (TGT) partnership could tarnish Neiman’s luxury brand. “Target competes with WalMart!” says Swinand.

So where does that leave owners TPG and Warburg Pincus? The two have been getting antsy about finding an exit, according to several people familiar with the situation. Traditionally, private equity players have a three-to-five year window on an investment before selling off the entity or doing an IPO to provide returns to shareholders. The recession disrupted those plans with Neiman.

Neiman issued a special dividend to its owners of $435 a share earlier this year at a cost of about $449 million. Although company executives boasted that the dividend reflected their confidence in the company’s growth prospects and outlook, analysts aren’t buying it. Skeptics noted that Neiman had to borrow $150 million to pay for it, and accused the PE owners of lining their pockets at the expense of the company. The dividend boosted the company’s already high debt level.

The runaway success of Michael Kors (KORS), whose stock has more than doubled since its December 2011 IPO, has generated considerable interest in upscale apparel companies, and this could potentially help lure investor interest in a Neiman IPO.

MORE: Wal-Mart CEO: Shoppers weighed down by the fiscal cliff

If an IPO is shelved for now, could another buyer – potentially even another private equity buyer step up? “The economy still sucks, why would you want to do it now?” says Swinand. “Why would one private equity guy find it attractive and another not?”

Some speculate that another retailer, such as Macy’s (M), which has the cash to make such an acquisition and has been expanding its reach into different retail segments, could be a possible suitor at the right price. Davidowitz notes the Macy’s chief executive, Terry Lundgren, was formerly the CEO of Neiman.

However, Swinand says it’s unlikely Macy’s would want to jump in now. “They’ve just been through this huge punch-in-the-gut recession and just did an enormous (effort) to deleverage,” he says. “It makes sense on paper, but I think at this moment right now with the fiscal cliff and European crisis, they wouldn’t want to do that now.”

Taking into account market conditions, debt levels, and Saks’s valuation, Swinand estimates Neiman’s current value around $4.7 billion, which is short of the $5.1 billion paid in 2005.

TPG, Warburg Pincus, and Neiman Marcus all declined to comment for this story.