A global company based in Danbury, Connecticut, IMS (IMS) buys bulk data from pharmacy chains such as CVS (CVS), doctor’s electronic record systems such as Allscripts, claims from insurers such as Blue Cross Blue Shield and from others who handle your health information. The data is anonymized—stripped from the identifiers that identify individuals. In turn, IMS sells insights from its more than half a billion patient dossiers mainly to drug companies.

So-called health care data mining is a growing market—and one largely dominated by IMS. Last week, the company reported 2015 net income of $417 million on revenue of $2.9 billion, compared with a loss of $189 million in 2014 (an acquisition also boosted revenue over the year). “The outlook for this business remains strong,” CEO Ari Bousbib said in announcing the earnings.

The company also said that it bought a subsidiary of Symphony Technology Group, which also owns IMS rival Symphony Health, a privately held company based in Conshohocken, PA. (Lawsuits between IMS and Symphony also ceased as part of the deal.)

For more on health care, watch this Fortune video:

IMS declined or didn’t respond to several interview requests.

In its marketing materials, IMS says it helps advance healthcare by providing “reliable, connected information, and real-world insights.” Pharmaceutical sales and marketing are a key part of IMS’ business, and its data also helps Big Pharma justify prices for drugs by demonstrating their effectiveness.

Some health care and privacy experts worry, however, that such commercial trade largely hidden from the general public carries big privacy risks. “We seem to be spending a disproportionate amount of time hammering government (albeit rightly so) when the big industry ‘hosts’ of our data have so much control,” says Claudia Pagliari, a senior lecturer at the Centre for Population Health Sciences at the University of Edinburgh.

A company with a long history

For a company little known to the public, IMS has been around a long time. German immigrant Ludwig Frohlich created IMS in the mid-1950s hoping to bolster his successful Madison Avenue medical advertising business by producing market reports on how well various drugs sold.

Later, IMS bought copies of prescriptions from drug stores. After pharmacies computerized in the 1980s and 1990s, the data miner compiled profiles on the exact prescribing patterns of individual doctors to help pharmaceutical salespeople target their pitches. By the 1990s, IMS also started gathering medical information from multiple sources on individual patients—whose names are removed from their dossiers in accordance with Health Insurance Portability and Accountability Act (HIPAA) privacy rules.

Privacy concerns growing

IMS tells the public, the source of its patient data, little about how its aggregates patient dossiers. The company is not more forthcoming with the press. In researching an upcoming book on the business of patient data, I’ve reached out to the CEO and top company officials multiple times since 2012. They’ve declined interview requests, and the company didn’t respond to requests for comment for this article.

This reticence concerns some health care and privacy experts, who worry that even when data is anonymized, the records could somehow become identifiable—and personal data vulnerable. Data security and hacking are always a threat.

“Only more sophisticated laws and more extreme corporate penalties are likely to deter the misuse of this data,” says Pagliari. HIPAA doesn’t cover anonymized data.

One way health data miners such as IMS Health could mitigate public concern is to give patients a choice to opt out of the collection of medical information, even if anonymized.

That’s what marketing data giant, Acxiom (ACXM), did, after coming under Congressional and public scrutiny concerning its use of non-medical data. When Scott Howe became Acxiom CEO in 2011, he declared he did not want to work at a company known as “the commercial equivalent of NSA or the super-secret spy guys.”

In 2013, Acxiom started letting individuals view their profiles via a dedicated site and offered the option to opt out. The CEO predicted that 15-20% of the public might do so, but only about four percent have asked to have their names removed, spokeswoman Ines Gutzmer told Fortune.

Today, consumers concerned about the anonymous sharing of their data would have to pick and choose among providers. For example, some health record systems share data anonymously, others do not; if you felt strongly about this you might have to change doctors to one that uses a system that does not share such information. Many insurers also sell claims data, but Aetna, for instance, does not. However you may not have a choice if you get your insurance through your employer.

Since such a process would be cumbersome, having the major data miner IMS allow an opt out would be an important step. IMS boasts of a “Relentless Focus on Confidentiality and Privacy.” It’s time to give the patients who are the sources of the data a voice in this discussion.

Adam Tanner is a fellow at Harvard University’s Institute for Quantitative Social Science and author of “What Stays in Vegas. The World of Personal Data-Lifeblood of Big Business—and the End of Privacy as We Know It.” (Sept. 2014) His next book is “The Big Health Data Bazaar: Uncovering a Multi-Billion Dollar Trade in Our Medical Secrets” (Jan. 2017).