I’ve gone all-in on Amazon’s line of Alexa-powered speakers, installing them throughout my home and buying them for family members. We use them to play music and news, tell jokes and get the weather. And I get to talk to a computer like I’m Captain Picard in Star Trek.

But stories like this one, in Gizmodo last week, suggest that the popular and inexpensive line of voice-activated speakers pose a threat to user privacy. The writer argues that devices like the Amazon Echo and Google Home contain microphones that are “always on” and sending volumes of data back to their parent companies. These speakers might also make it possible for hackers and law enforcement authorities to drop a secret wiretap into your living room, the article says.

The American Civil Liberties Union agrees, inveighing against the Echo and other connected speakers. Boing Boing’s Cory Doctorow writes that they “normalize surveillance.” Even InfoWars conspiracy theorist Alex Jones got into the act last week, interrogating an Echo on his show with hilarious earnestness. “Alexa, do you work for the CIA?… Alexa, you are lying to me!… Alexa, who is Jeff Bezos?”

Now, I don’t mean to defend these companies as much as to rationalize my own enthusiasm for these devices. But all these concerns seem a bit overheated. If the companies are telling the truth about how they operate—and lying about it would draw ire from both government regulators and customers—the privacy threat is not as big of a problem as it might appear.

First, devices like the Echo and Google Home are not really “always on.” They’re in passive listening mode, using a small amount of power for something called “device keyword spotting.” In effect, the companies say, the device is recording about one second of ambient sound, hunting for the acoustic signature of their wake words, “Alexa” or “OK Google,” and then constantly overwriting and discarding that fraction of sound.

When the wake word is uttered, the Echo glows with a blue crown of light. Only then does it send the command to the cloud and fetch a response from Amazon’s servers. Users can look at their personal history in the Alexa smartphone app to see exactly what sound Amazon has sent to its servers. These are typically anodyne commands (“Alexa, play Harry Potter trivia”) or expressions of frustration when the device has misinterpreted a request and is going haywire (“Alexa, STOP!”).

The Gizmodo piece acknowledges this but raises some hypothetical scenarios. What if hackers get a hold of the device and change how it operates? This apparently happened once, when a British security researcher had to physically disassemble the device to demonstrate an Echo vulnerability.

Then there’s the sensational example from earlier in the year when police officials in Bentonville, Arkansas subpoenaed a customer’s Alexa records in connection with a 2015 homicide at his home. (Bloomberg’s Nico Grant and Aki Ito recorded a great episode of our Decrypted podcast about the case.)

Amazon initially fought the subpoena, then acquiesced when the suspect’s lawyers agreed that the data could be turned over. They likely realized that the police, like some of Alexa’s critics, were overestimating the amount of data Alexa might gather.

None of this means we should blindly trust Amazon and Google in other parts of their business, or take our eyes off how these devices evolve. Amazon has talked about handing over customer Alexa transcripts to third party developers. Surely privacy-concerned Echo owners should get a chance to opt out of that program.

For now, though, I am going to keep enjoying this new class of devices. And if I’m having a truly sensitive conversation in my dining room? I’ll try to stifle the impulse to go outside and whisper.